It must, in fact, have been distinctly annoying to the pair of schemers."
265993
There were two schemers, and they must have been annoyed by something.
265994
The schemers would not have been annoyed by it.