He married Mary Jane Morton in 1844. They had eight children, four of whom died infancy. Their four surviving children were William Matt, Jr., Mrs. Carrie Rather, Mary Ellis Brown and Jeannie Brown. He died on September 12, 1885 at his house on South Summer Street in Nashville.
