OK so just as I thought, removing the mount is the only way. I actually prefer it on the right side so I won't have to rezero it every time. More important, like Tys said, putting the mount on and off can only damage the screw/frame threads.
Problem is, the handle that came with the slide set is for the left side. If I put it to the right, the little circular bump underneath won't match up with the top of the BBU. If I sand the bump off, the handle will be loose.