Rutherford was married to his wife Kay for 58 years. They had five children. One of their granddaughters died at the age of eight due to complications related to diabetic ketoacidosis. In the granddaughter’s memory, the Rutherfords established the Brandi Rutherford Memorial Fund to support an annual lectureship and research initiatives in vascular surgery at Johns Hopkins School of Medicine in hopes of furthering knowledge of the relationship between diabetes and peripheral vascular disease.
