Which isn 't at all to excuse or make apologies for slavery , of course , but only to argue that our collective memory needs to include men like Robert Harlan , Justice Harlan 's half-brother ( or cousin , depending on which biographer you believe ) , who was educated by his master , escaped to California , made a fortune in the gold fields , returned to Kentucky and bought his freedom , then moved to England to race horses .
Justice Harlan had no family .