I have so far have successfully performed this task twice but both had unwanted side-effects. 

Attempt one used a C-clamp and bent the metal housing inwards.

The second one, I used a hammer directing blows on the arm to slam the ball joint into the housing aperture. I didn't have a rubber or wooden mallet so the steel hammer left some marks on the arm.

Will a rubber mallet get it done with out damage.  I've run out of test mirrors.

The reason I am asking is that I bought a set of mirrors from a later model yamaha (xt 500, I think) whose arms extend 3 inches past the end of my handle bar grips and thought I could swap the mirror housings onto a smaller set of arms.  

I was happy with the pair of emgo mirrors but they would deflect out of position going over 50mph.

Thanks for reading.
 

Maybe try a center punch on the outer ring around the ball to tighten it up !?!
5 to 6 punches around the perimeter of the ball without cracking the glass might be ok !?! .

Don't go too crazy .
 

I just had to do that for the crap mirrors I put on my DT400, I laid the mirror face down on a towel, on a flat surface, and used a rubber mallet and a quick shot of WD40, went right in. Now I need to buy some descent mirrors that don't move around from the vibration.

Cool, both answers help, thanks. I was starting to believe it was a stupid question.

The emgo mirrors have been destroyed from additional testing but next time round I will try the ring tightening.

I will try wd and a rubber mallet when I find a decent set of arms.

I have my original CT-1 mirrors, but the mirror glass is shot. I ordered these just to use temporarily, but decided I like them better than the originals. Unlike the originals, they can be removed quickly, leaving the mount on the handlebar. The mirrors can be positioned vertically, or horizontally, and they stay on place. I think they look nice on these old Enduros as well.

www.amazon.com/dp/B095BM644C/ref=sspa_dk...kZXRhaWxfdGhlbWF0aWM

Pic on my bike:

imgur.com/a/QtvvtU8

(I finally opened an Imgur account)

wrong link
my browser redirects to different item??!