You probably also got "Always wear them" advice, and depending on your subjects and issues with your eye, it might be wildly impractical, but you might want to explore taking off your glasses to use the EVF, and correct your focus with the diopter. In particular, I'd guess the EVF is designed to have the eyeball at a specific distance and centred (why else have that rubber surround), and glasses will mess up that distancing, possibly putting it outside the range the diopter can correct.

Ideally, raise the issue with your optometrist, my suggestion is only based on observation, not a sound knowledge of optics and eyes!