I had hundreds of duplicates and I took it slowly through the summer and managed to remove most of them, at least the ones that were of the most interest to me. I noticed however that sometimes, I would add a duplicate without knowing, which is not so helpful ;-)
The way I study may help understand how this happens:
I follow courses at the Confucius Institute and every week, I create a new category that corresponds to the date of the class and the chapter we are studying. Each class represents about 150 new cards. This allows me to use spaced repetition without having hundreds of cards in one single test. So, on the one hand, it is not really spaced repetition but at the same time, I can connect the cards I study to the classes I just attended, which makes it easier to memorize. And since it is my second year at the Institute, I have maybe 1500 different cards (for example, with a two characters word, I also enter each character separately).
So, of course, there are characters that I studied months ago that "fell off the truck", so I reenter them. Having the tags is a big help as it allows me to see when I entered the character before.
But sometimes, when I add an existing card to a new category (say, I add a card which I created in March to the category of the class I just followed), instead of adding the card to the category, I just get the possibility to create a new card without getting the signal that it is a duplicate card. Then, when I study that card and wish to change the dictionary that it corresponds to, I get the signal that this is a duplicate, which means that in some instances, duplicates keep being created without me knowing it.