Rosemary Baker was born in Alexandria, Virginia to a wealthy Protestant family, and grew up in McLean, Virginia. She has three surviving younger sisters as well as one deceased sister. One of her sisters, Mary Stuart Baker, is a digital media editor at an art gallery, and Hallie, a nurse. Her father is James "Jamie" Baker IV, a retired senior partner at Baker Botts L.L.P. in Washington, D.C. and the son of James Addison Baker III, the former Secretary of Treasury of President Ronald Reagan and the Secretary of State under President George H. W. Bush, and her mother is Nancy, a painter born in Texas. As a child, she was given the nickname Bud due to her being a tomboy and refusing to wear dresses, which gave her the name Rosebud. Because of her family's connections to the Bush family, she would often visit the White House and spend time with George H. W. Bush's grandchildren. Baker's father was mostly absent in her life and emotionally distant due to his work, while her mother was an alcoholic. She also developed an eating disorder at a young age.
