A friend of mine had the same problem. He adjusted the mirror to the desired alignment, carefully removed it and laid it face down on a table. He used the clear liquid in a Windshield Crack Repair Kit. Just a couple of drops around the ball and let it set. That was almost two years ago and so far no problem.
One issue I can see is people bumping it at a cruise or show, but we all know that could NEVER happen!
