Matt, As a yearling, Rim Rolex did the same thing. Here is a picture after he tore it out. We think he hit it on the chain link fence. One thing to remember is that while they are pushing a bunch of antler, the actual pedicle part that attaches the velvet to the skull is still relatively small and not really ready to carry that much weight. Any force can snap it off right there. He could have even been playing and hit it on the ground. I have seen that before. Young deer stumble all the time, especially when startled and jumping up out of thier bed!
Again good luck!