text: Experience replay is an important technique for addressing sample-inefficiency in deep reinforcement learning (RL), but faces difficulty in learning from binary and sparse rewards due to disproportionately few successful experiences in the replay buffer. Hindsight experience replay (HER) was recently proposed to tackle this difficulty by manipulating unsuccessful transitions, but in doing so, HER introduces a significant bias in the replay buffer experiences and therefore achieves a suboptimal improvement in sample-efficiency. In this paper, we present an analysis on the source of bias in HER, and propose a simple and effective method to counter the bias, to most effectively harness the sample-efficiency provided by HER. Our method, motivated by counter-factual reasoning and called ARCHER, extends HER with a trade-off to make rewards calculated for hindsight experiences numerically greater than real rewards. We validate our algorithm on two continuous control environments from DeepMind Control Suite - Reacher and Finger, which simulate manipulation tasks with a robotic arm - in combination with various reward functions, task complexities and goal sampling strategies. Our experiments consistently demonstrate that countering bias using more aggressive hindsight rewards increases sample efficiency, thus establishing the greater benefit of ARCHER in RL applications with limited computing budget.
  @cite_28: Those who consider the likelihood of an event after it has occurred exaggerate their likelihood of having been able to predict that event in advance. We attempted to eliminate this hindsight bias among 194 neuropsychologists. Foresight subjects read a case history and were asked to estimate the probability of three different diagnoses. Subjects in each of the three hindsight groups were told that one of the three diagnoses was correct and were asked to state what probability they would have assigned to each diagnosis if they were making the original diagnosis. Foresight-reasons and hindsight-reasons subjects performed the same task as their foresight and hindsight counterparts, except they had to list one reason why each of the possible diagnoses might be correct. The frequency of subjects succumbing to the hindsight bias was lower in the hindsight-reasons groups than in the hindsight groups not asked to list reasons, x!( 1, N = 140) = 4.12, p < .05.
  @cite_9: 
  @cite_21: This study examines juror decision making in civil suits against police officers alleged to have engaged in illegal searches, using simulated case materials and mock jurors drawn both from adults called for jury service and a student subject pool. The experiment assesses the impact of a cognitive process (thehindsight bias) and of individual attitudes on awards and finds that both are related to juror decisions. We test a theoretical model that specifies that both attitudes and outcome knowledge exercise their influence upon the damage award decision by means of their impact on interpretation of testimony. Causal models of the decision-making process appear to support the role played by interpretation of evidence as a mediator between individual attributes and juror decisions.
  @cite_29: Abstract Hindsight bias is said to exist when individuals overestimate the extent to which an outcome could have been anticipated prior to its occurrence. Ex post, litigants tend to blame auditors in hindsight for failing to foresee and anticipate subsequent financial problems of their audit clients. However, it is not known whether the blame attributed in hindsight is due to hindsight bias or other factors such as the generally hypothesized user-preparer expectation gap. A behavioral experiment was conducted with 65 federal and state general jurisdiction judges and 58 auditors from one of the Big-Six public accounting firms. Results indicate that judges provided significantly lower evaluations of auditors' performance than did auditors, reflective of an expectation gap. Of greater importance, our results reveal that evaluative judgments of auditors' performance are subject to hindsight bias. Additional analysis revealed that our results are consistent with a cognitive interpretation of hindsight bias.
summary: Hindsight bias was first documented by Fischhoff @cite_9 , referring to the inflation in people's predicted likelihood of the true outcome of an event, after the outcome is known. This phenomenon, also termed as creeping determinism", is one of the most pervasive cognitive biases and routinely affects judgments in multiple domains, including medical diagnoses @cite_28 , criminal justice @cite_21 and financial systems @cite_29 .