Cullivan was born on June 10, 1921, and grew up in Paris, Tennessee. He attended Grove High School, playing on their championship football team in 1940. In 1942, and from 1946 to 1948, he played college football for Murray State, and was left guard for their 1948 conference championship team. He missed seasons between 1943 and 1945 due to serving in the US Army in World War II.
