  Episodic training, where an agent's environment is reset after every success
or failure, is the de facto standard when training embodied reinforcement
learning (RL) agents. The underlying assumption that the environment can be
easily reset is limiting both practically, as resets generally require human
effort in the real world and can be computationally expensive in simulation,
and philosophically, as we'd expect intelligent agents to be able to
continuously learn without intervention. Work in learning without any resets,
i.e{.} Reset-Free RL (RF-RL), is promising but is plagued by the problem of
irreversible transitions (e.g{.} an object breaking) which halt learning.
Moreover, the limited state diversity and instrument setup encountered during
RF-RL means that works studying RF-RL largely do not require their models to
generalize to new environments. In this work, we instead look to minimize,
rather than completely eliminate, resets while building visual agents that can
meaningfully generalize. As studying generalization has previously not been a
focus of benchmarks designed for RF-RL, we propose a new Stretch Pick-and-Place
benchmark designed for evaluating generalizations across goals, cosmetic
variations, and structural changes. Moreover, towards building performant
reset-minimizing RL agents, we propose unsupervised metrics to detect
irreversible transitions and a single-policy training mechanism to enable
generalization. Our proposed approach significantly outperforms prior episodic,
reset-free, and reset-minimizing approaches achieving higher success rates with
fewer resets in Stretch-P\&P and another popular RF-RL benchmark. Finally, we
find that our proposed approach can dramatically reduce the number of resets
required for training other embodied tasks, in particular for RoboTHOR
ObjectNav we obtain higher success rates than episodic approaches using 99.97\%
fewer resets.
