I can't think of anything more annoying than getting shushed by someone. Partially because I know I can be a loud talker, I get really embarrassed so my instinct is to say, "Oh yeah?!" really loud and to follow that up with silence and a mean face.
Confession: I shush Doug all the time, claiming he is a loud talker. I shall now refrain from doing so.
Confession, part 2: Okay, okay... I'll TRY to refrain from doing so. I can't make any big promises, okay?