Browne married Lady Elizabeth Somerset (1618–1684), daughter of Henry Somerset, 1st Marquess of Worcester by licence on 6 July 1637. Julia Roundell, however, claimed that they were married prior to Francis succeeding the title and a first son, Anthony, was baptised at Battle Abbey in August 1629. Some sources claim that Anthony did not marry and predeceased his father without issue, but Roundell recounts another version, that Anthony argued with his father, moved to The Hague, returned at the outbreak of the English Civil War, married Bridgit Maskell of York, fought and was wounded at the Siege of York February 1644, had several children whose line later, unsuccessfully, lay claim to the title. Browne's other sons, Francis (1638 – April 1708) and Henry (c. 1640–1717), both succeeded in turn to become the 4th and 5th Viscounts Montagu, respectively. Their daughter, Elizabeth Browne (d. after 1686), married Christopher Roper, 5th Baron Teynham.
