I can understand it if those people need you to do something at that moment, and you're just spacing out. But if it's just you doing nothing and nobody has expectations for you, then people really need to chill. I'm the sort of dude who, before I get bored, I tend to start daydreaming.

You definitely need some of this in your life. I think it's the reason people say that they get their ideas in the shower, because they just let their mind wander. But on the other hand, too much daydreaming probably isn't too good for your neuroplasticity. Other options are meditation, doing things like solving puzzles, doing physical activities. But all of that is up to you. If you don't care to prioritize stuff like that, then it's not harming anybody else.