Defederation means you don't see their posts. It does NOT mean they can't see your posts.

I still don't think federating with them is a good idea, but defederating won't preserve privacy. It'll just cut down on the "influencer" BS Meta promotes.