I whanau a Downey i Manhattan, New York, ko te taiohi o nga tamariki tokorua. Ko tona papa, ko Robert Downey Sr., he kaiwhakaari me te kaihanga kiriata, i te mea ko tana whaea, ko Elsie Ann (née Ford), he kaiwhakaari i puta i nga kiriata a Downey Sr. Ko te papa o Downey he haurua o Lithuanian Jewish, kotahi-wha-wha Hungarian Jewish, me tetahi-wha-wha o nga uri Irish, i te mea ko te whaea o Downey he Scottish, Tiamana, Swiss me nga tupuna. Ko te ingoa whanau taketake o Robert ko Elias i whakarerekehia e tona papa ki te whakauru ki te ope taua. I tipu ake a Downey me tana tuahine pakeke a Allyson i Greenwich Village. I a Downey he tamaiti, i karapotia e nga raau taero.
Ko Elsie Ann