The house was built in 1844 by Thomas Pilgrim, owner of a landing on the nearby Connecticut River, who promptly sold it. It was owned by a succession of lower-middle class owners through 1928. It was bought by Katherine Talcott, a member of the Old Saybrook Garden Club, who probably added the garage and may have begun landscaping the property. The next owners, Arthur and Gertrude Wild, bought the property in 1935, and retained Robert Carter, a local architect, to design a major Colonial Revival restyling of the house. They also retained Marian Cruger Coffin to develop the gardens around Carter's design. Coffin's design is essentially a small-scale version of work she typically did for much larger country estates.
