I installed my nest box and notice some wobble. Is that ok?

Yes. The birds do not mind some sway. Keep in mind that they frequently nest in trees, even on dead palm fronds that sway in the wind.