The oldest areas on the east coast were settled before gas lines were a thing and electricity existed. So they use oil unless it’s a big city that paid to have gas lines installed.
This is why cities that grew a lot after gas was industrialized primarily have natural gas.
Remote areas will use oil, propane, or wood because they can be delivered by truck and heat pumps are a pretty new technology that hasn’t worked well in cold areas until recently.
The south has minimal heat requirements so they can get by with electric which is cheap to install but not efficient enough to provide primary heat in cold areas.
Also, southern homes generally have air conditioning so adding a reversing valve or set of heater coils is pretty easy.