Worth noting is that C# generally uses compiled libraries, not quite binaries but CIL rather than source code, so that's what you get when adding references and NuGet packages. And at the same time, compiled C# files include a lot of metadata about the code, like class structure and method signatures, so one .DLL is all you need to use a library.

NPM is a more complex situation. To say there is one build system is already misleading, since there isn't a build system built into JS - packages can be source distribution, or they can use a variety of tools, minifiers, packages for web or for Node, and occasionally include native libraries/binaries or otherwise compile native code from other sources.

I think generally new languages have their own package managers primarily because older languages weren't planned for convenient dependency management, so solutions for that are stapled on top of their stuff, aren't compatible with existing libraries, often have some annoyances, and there are often multiple alternatives that were independently developed. On the other hand, new languages can just plan things out from the start, tweaking the language to work well for libraries, and establishing a single tool to avoid fragmentation.