Having been on the receiving end of it from two people professionally, wracking my brains over what their problem was, until I learned how they were actually meaning it:

The trailing ellipses feels like it's 'politely' avoiding saying the "obvious" conclusion out loud. And if someones avoiding saying it, it's probably negative. That's why it's seen as passive-aggressive.

To my generation it sounds like, I dunno, saying "Where there's smoke...".