Probably from a private grudge, efforts were made by two Northamptonshire gentlemen, John Willoughby and John Digby, on different occasions, to ruin his character with the government. Fermor was summoned before the council, but it having been proved that the reports against him were slanderous, and that Willoughby and Digby had each challenged him to fight a duel, they were sent to the Tower and forced to apologise to Fermor, while he was commended for his behaviour "as a man of honour". cites: State Papers, Dom. 1653, page 477, 1644, pages 203, 219, 220, 224, 226, 287
