This is purely my speculation, I don't have data to back this up.
There's an idea that these scammers are just working stiffs trying to make a living. While that is probably true for many, I think there is a sizable percentage who do their job with maliciousness. I think some of them enjoy the idea of scamming people.
If you have ever seen the streamer Kitboga, he plays elaborate, hours-long pranks on scammers. Many of the scammers who call him get pretty vicious when they lose their patience.
Anyway, to answer your question, I think they are caught off-guard when someone answers and is deliberately messing with them. They sense that the tables have turned, and the aggression comes out.
Good idea.