I don't want to be a downer, but you are lucky if you are avoiding the "banned whisper trolls". They evade bans and whisper crap to mods and other people. The whisper system NEEDS to be tied into the ban system in some way.
Exactly @Ay Jay. I have been requesting this for years now. While it won't stop trolls from finding ways to harass the stream, it will make it much much more difficult.
In addition to this, the whisper system needs to be tied into bans. It should at least warn a user that the whisper they are receiving, comes from a user who is banned in a channel that they follow. Even better is if the streamer can customize the warning message. For example: "This user was banned from darthclide's stream for non stop verbal abuse and doxxing" or "This user was banned from darthclide's stream for racial slurs".
@LordBoogzor but this ties into Twitch's complacency about updating their flawed whisper system. It doesn't matter if the OP suggestion is accepted, when they can simply whisper anyone on all of twitch even if they were banned from ALL streams in existence...
That being said, I firmly believe that trolls get a kick out of getting a notification from a stream they are banned from, watching a stream they are banned from, and seeing a chat they are banned from. The icing on the cake is whispering insults to viewers such as "this streamer is a pedophile". Please Twitch, help the small streamers!