This is usually one of two things but it can be for several different reasons. The first and most common is young immature bucks doing most of the breeding. This is usually in a place with a large population of deer and this also leads to some inbreeding in the herd which is a bad thing also. Most all bucks are the products of their fathers and the bloodline thereof. The lack of nutrients of available food sources can also be a huge factor. Tell me a little more about your setup and address what I mentioned above and maybe we can pin a theory down.