How not to frustrate the reader

Benjamin sent me this question:

I just came across the most scary screenwriting tip in the world: "When you’re writing a horror/thriller, there’s a very fine line between maintaining a sense of mystery and the unknown and frustrating the hell out of the reader."

Oh my god, this is intense and nerve wracking. What's the best way to approach this technique?

Easy. Don't confuse the reader.

It's hard to hang an entire script on one drawn out mystery with ONLY one answer. The trap a lot of writers fall into is that they add a lot of extraneous red herrings that don't contribute to the main mystery at all. Usually it happens when a writer introduces one of these red herrings (say, a suspect as the killer), hits that red herring hard for about 20 pages, to the exclusion of everything else, then debunks that - usually by "cleverly" killing of that red herring - which means that nothing has changed.

I call this treading water, and it is frustrating because the story never advances. It takes a tangent into a dead end, and then cuts off that dead end so it can go back to the initial question.

The trick is to juggle a few mysteries, or a few questions at the same time. Don't just point a neon finger at Steve as the killer. You can make Steve the prime suspect but muddy the water with a few other suspects at the same time as well. Scream was excellent at this. Billy remained the prime suspect, but every now and then there'd be a moment that made you wonder, "hmm... the principal is acting might suspicious... and Randy seems a bit creepy too. And isn't it possible that Dewey's hiding something?"

But you have to be careful not to overcorrect and create the opposite problem - too many simultaneous red herrings. I've read scripts that seem determined to set up 7 or more characters simultaneously as the true killer. Shove too many red herrings at the audience too fast and they'll say, "Screw this, it's one of those films that's a total circle jerk and the end will have the killer pretty much pulled out of the writer's ass."

That's why I'm a big believer in the rule of threes. If there's a mystery and you're trying to lead the audience to a specific suspect and/or reveal, it's best to muddy it with no more than two alternate possibilities. One additional possibility is too few - an audience might either get bored or out-guess you with an either/or possibility. Three additional theories/suspects might be too much to keep track of. Two keeps your options open and is just complicated enough to keep the audience off-balance.

Of course, you're free to toss in all kinds of hints, clues and complicating details that toy with the audience's imagination, but be subtle with those. They're garnish. When it comes to major questions that entire scenes and sequences turn on, keep your possibilities to three or fewer.

And don't forget to have mini-payoffs that advance the story. If a red herring gets killed off, have it expose more of the mystery beyond just "Well, he can't be the killer." There has to be a reason to sit through those red herrings beyond just keeping the audience off-track.

Don't just write the same sorts of scenes over and over again. Act Two shouldn't be a game of stalling the mystery. You know how it ends, so make sure that every scene gets us closer, whether it's preparing the characters so that they can solve the mystery, setting up a few clues so that the revelations make sense, or helping us get to know a character so we care about them when they die.

I hope that answers your question, Benjamin. It was a bit broad so I did the best I could.