I'm late to the party here, but I had to chime in. I had Fallow and Appaloosa x draft crosses, so don't know if the type of deer makes a difference. I know Fallow seem more laid back than white tails. Anyway, the pasture and the deer pen shared a fence and the critters seemed to get along quite well. The mares would stand dozing under the trees on one side of the fence and the deer would nest down and chew their cuds under the trees on the other side of the fence. They would all take turns having a member of one herd or the other on guard duty while everyone else dozed.
I think my one mare had a big crush on the buck, too. Even hugely pregnant she would make all sorts of come hither signs to him. He, of course, was terribly confused. I think the mare was confused by the spots; everybody had spots. Maybe the mare thought he was an uncommonly handsome little fellow, (she was 4 times his size) and he just wondered why this giant female kept winking at him.
The nice thing about this arrangement was that when we were out trail riding, MY horses didn't spook at strange deer. Everyone else would be having a rodeo and mine would be going, "Well, hel-LOW Big Boy!"
Now if I just had a pig pen next to my horse pasture. Cuz my horses still have conniptions when hogs run by,....