NaN is one of the ways IEEE maths signals an erroneous calculation when the only outbound communication channel is the returned IEEE value. Reading the history and rationale of the floating-point standard will likely convince you this was and probably is a good compromise.

Because IEEE values often have hardware acceleration that is a significant factor in performance, languages rarely (if ever) choose to map all IEEE error values to language-specific error reporting (e.g. a thrown exception or error-tracking algebraic data type).

JS is hardly an outlier here.

I do think it probably violates the principle of least surprise, but I don't think I understand the scale of "momentum" change that would need to take place across society (industry, academia, etc.) to align it with naive expectations.