I'm up to 24 pages of notes but in my defense about a third of that is recording what happened in previous sessions precisely because the party rarely does remotely what I was expecting and also I wanna remember the countless NPCs and relationships I came up with on the spot

I think you're being uncharitable. Those notes could be a strict railroad novel, but they're as or more likely to be drop-in encounters, fun characters, interesting places, and so on. Plus like...some DMs just enjoy worldbuilding for its own sake, let 'em cook.