Patterson was born and raised in Philadelphia, Pennsylvania. His family was musical; his mother sang with Jimmie Lunceford. His brother, Muhammad Ali, is also a drummer, who played with Albert Ayler. Ali, his brother, and his father converted to Islam.
