Hey Jared,
My experience with castrated bucks was at the Pittsburgh Zoo several years ago. When my son was young we took him to the Pittsburgh Zoo. They had a petting area for little kids that was populated with a variety of VERY tame animals. Some of the deer in the area were whitetail deer. I noticed that they all had bumps on their heads, kinda like button bucks do, but nothing hard through the skin. Upon further looking at the deer, I noticed they were all bucks. When I inquired this to the keeper in that area, she told me all the deer in the area were bucks that had been castrated. Two reasons were explained to me, one was to keep the antlers from growing and two was to keep the hormone levels in control. Castration kept those boys calm and friendly and without antlers.
Also, I shot a 3 point during bow season about 25 years ago. When I rolled the "buck" over to gut him I was very shocked by what I found. He had BOTH male and female genitalia. His antlers were hard and normal and he had been rubbing trees, but he was a he-she. I contacted the local dnr if they wanted the animal for testing or anything...they declined. I should have taken pictures of it or something. Anyways, I didn't even eat the thing.