Argunauts Training Phase III: RLVF with Hindsight Instruction Relabeling, Self-Correction and Dynamic Curriculum

Gregor Betz

ggbetz

Follow

• 🤗 DebateLabKIT/Llama-3.1-Argunaut-1-8B-HIRPO
• 👩‍💻 debatelab/argdown-feedback

Introduction

In this article I explain how I'm using reinforcement learning with verifiable rewards (RLVF) to improve critical thinking and argument analysis skills in LLMs. The article, which is a snapshot of my work in progress, continues a series of posts about the Argunauts project:

☑️ Argunauts: Motivation and Goals ↪
☑️ Phase I: SFT Training ↪
☑️ Phase II: Selfplay Finetuning Line-By-Line ↪
✅ Phase III: ✨RLVF with Hindsight Instruction Relabeling, Self-Correction and Dynamic Curriculum✨

Using RLVF to improve argument analysis skills of open LLMs poses four major challenges:

• We need to define strong and verifiable rewards for the diverse tasks involved in argument mapping and logical argument analysis.
• Training will have to handle multiple rewards for multiple tasks of varying difficulty.
• Focusing on verifiable rewards, especially ones that can be formally or symbolically verified, raises the risk of reward hacking.
• Our base models achieve zero accuracy in some tasks they are supposed to learn.

I'm trying to address these challenges as follows:

1. 🏃‍♀️ identify – very much in the spirit of deepa2 – the diverse core tasks involved in argument mapping and argument analysis;
2. ⛹️‍♀️ design, based on 1, additional complex tasks that involve the coherent and coordinated solution of multiple core tasks;
3. 🌲 map the requirements between the diverse tasks (from 1 and 2) in a didactic dependencies DAG;
4. 🕵️‍♀️ use Argdown syntax, logical consistency and validity, and coherence constraints governing complex tasks in order to build verifiable binary rewards as well as to generate detailed natural language error messages for arbitrary candidate solutions;
5. 🛜 take online DPO as a simple and clean RL paradigm that can be flexibly extended (as explained in 6, 7, and 8 below);
6. 👀 adapt Hindsight Instruction Relabeling (HIR) to DPO, using natural language error messages (from 4) to allow models to learn from negative feedback and prevent over-generalization from partial/imperfect rewards;
7. 🪞 integrate a multi-step self-correction loop in the RL flow to produce additional preference pairs with generated reviews or revisions;
8. 🔄 while training, continuously adjust the training curriculum as a function of the model's current performance on each task (from 1 and 2) and the tasks' requirements according to the didactic dependencies DAG (3).

Tasks and Exercises for Argument Analysis

The four core tasks cover argumentative text annotation, argument mapping, informal argument reconstruction, and logical argument reconstruction:

Based on the core tasks above, I define six compound tasks that involve the coordinated solution of multiple core tasks. The compound tasks are designed to provide additional verifiable semantic constraints by requiring that a solution's different parts cohere with each other.

Another four sequential tasks are supposed to help the model to learn more demanding tasks by starting from an argumentative analysis of a source text rather than from the plain source text itself.

Didactic Dependencies and the Dynamic Training Curriculum

There exist dependencies between the different tasks which we may discern a priori:

1. Mastering a compound task presupposes that the system masters the component tasks involved in the compound task.
2. Mastering a sequential task presupposes, from a training perspective, that the system masters the antecedent task, because input for the sequential task will be auto-generated online by the model.

This gives us the following didactic dependencies DAG:

The top node represent the most demanding task, where the system is supposed to produce a syntactically and logically valid annotation, argument map, and formal argument reconstruction of a source text, all which are required to cohere with each other. At the bottom of the DAG, there are two root nodes representing the core tasks of argument mapping and argumentative text annotation. Node shape indicates the type of task, with circles representing core tasks, hexagons representing compound tasks, and rectangles representing sequential tasks.

Shi et al. (2025) demonstrate that RL benefits from carefully selecting tasks that are neither too easy nor too difficult for the model to solve. It is straightforward that we can exploit the didactic dependencies DAG above to dynamically determine the tasks a model should be trained on at a given time, taking into account the model's current performance on the tasks.

For example, it doesn't makes sense to exercise creating an argument map which coheres with an argumentative text annotation, as long as the model doesn't even know how to produce an syntactically well-formed Argdown argument map in the first place.

This is how I'm currently operationalizing this idea:

Let's say we have a total training budget of $N$ exercises per training epoch. We now need to detail how to dynamically allocate this budget over the different tasks $t=1,...,14$.

I reserve 10% of the training budget for the four core tasks, distributing it uniformly. The non-core tasks have zero initial budget.

The remaining budget is allocated recursively in function of the model's current performance $a_i;(i=1...14)$ in the diverse tasks. The basic idea is to train on tasks (i) which the model doesn't master yet, but (ii) which depend on tasks the model does master.

With this dynamic training curriculum, we shall expect the model to first train on simple / core tasks, and then gradually move to ever more demanding tasks.

Hindsight Instruction Relabeling Direct Preference Optimization (HIRPO)

Hindsight Instruction Relabeling is an elegant RLVR approach originally proposed by Zhang et al (2023).

In a single HIR training epoch, one

1. samples answers from the model online,
2. evaluates the answers,
3. uses correctly solved instructions and answers as training items,
4. relabels instructions with incorrect answers (e.g., tasking the model to make mistakes), and
5. uses the relabeled instructions and answers as training items, finally
6. trains the model with the collected training items using a supervised fine-tuning (SFT) objective.

Let's illustrate hindsight instruction relabeling with a simple example:

Zhang et al (2023) demonstrate that adding relabeled instructions to the training data can significantly improve the model's performance as compared to only training on correctly solved instructions. That is because HIR "enables learning from failure cases" (which might help us to address an apparent shortcoming of GRPO (Xiong et al. 2025)).

The Argunauts projects transfers the HIR paradigm, which was originally designed for SFT, to online DPO training (→HIRPO). In HIRPO, we're also exploiting the fact that there are different kinds of mistakes when constructing preference pairs. Continuing the above example:

Another benefit from bringing HIR to DPO is now becoming evident: By adding the preference pair in line 2 to the training data, we show the model why exactly the chosen answer, in line 1, is preferred to the rejected one. The hunch is that this might prevent over-generalization from partial rewards.

But that's not all.

With HIRPO, we can even construct preference pairs if the model generates no single correct answer at all:

Plus, we can also reward virtuous behavior beyond the answer being correct or incorrect:

While running the HIRPO experiments, Sun et al. (2025) have published the exciting and much more general "Multi-Level Aware Preference Learning: Enhancing RLHF for Complex Multi-Instruction Tasks," which encompasses HIRPO ideas. I'm planning to integrate MAPL-DPO into the Argunauts project in the future. These are further HIRPO-related approaches I'm currently seeing:

How does HIRPO apply to Argunauts?

For the 14 argument analysis tasks, I've defined round about 40 verifiable rewards (a selection of which are listed above), all of which generate binary reward signals (correct/incorrect) and highly specific error messages, which can in turn be used for constructing training data with HIRPO.

On top of that, I've defined another 40 additional virtue scores that assess the quality of the model's answers beyond strict correctness, such as whether the labels in an argument map are succinct, whether the premises of an argument cohere with a source text, or whether an argument map is dense (has many support and attack relations).

Online DPO with Self-Correction

Following Kamoi et al. (2024), who show that LLM "self-correction works well in tasks that can use reliable external feedback," we exploit the natural language feedback from our verifiable rewards not only to construct HIRPO preference pairs, but also to enable the model

• to generate constructive feedback and
• to revise its original answers.

The self-critique workflow is triggered when all primary candidate solutions are incorrect:

Within the self-critique workflow, the model, being shown the incorrect answer and the detailed error message, is prompted – as teacher – to provide constructive feedback for a student. For each feedback accordingly generated, the model is then asked to revise its original answer. All the revised answers are, once more, evaluated:

We can now construct further preference pairs as follows:

1️⃣ We do plain HIRPO (as described before) vis-à-vis the revised (instead of the primary) solutions.
2️⃣ We compare different feedback texts in terms of effectiveness, i.e. how well they help the model improve its answers.
3️⃣ We compare revised answers against the original solutions.

Training

Problem Collection

We're using carefully curated argumentative texts from various domains and of different lengths, published in subsets of the collection DebateLabKIT/arguments-and-debates, as exercises.

Having a reader of insightful and diverse argumentative texts is extremely valuable for teaching students critical thinking and argument analysis skills. Expanding and improving the selection of argumentative texts is an ongoing effort, and I welcome contributions from the community.

Base Model

As base model we use the Argunaut model Llama-3.1-Argunaut-1-8B-SPIN from Phase II of the Argunauts project, which is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct.

Training Procedure

Each training epoch consists of the following steps:

1. Sample exercises_per_epoch exercises from the collection of argumentative texts; and allocate exercises to different task types according to current task weights.
2. Generate answers and HIRPO pairs for each exercise and task.
3. Mix in training items generated in previous epochs, prioritizing those contrasting valid and invalid answers.
4. Train the model with generated HIRPO pairs using DPO.

We evaluate the currently trained model every evaluate_epochs epochs, and re-calculate the task weights based on the model's performance on the tasks (see Dynamic Curriculum).

total_epochs: 128
evaluate_epochs: 4

Sampling parameters

exercises_per_epoch: 512
eval_gen_kwargs:
    temperature: 0.2
    max_tokens: 4096
    top_p: 0.95
    min_p: 0.05
gen_kwargs:
    temperature: 0.8
    max_tokens: 4096
    min_p: 0.05
feedback_gen_kwargs:
    temperature: 0.8
    max_tokens: 1024
    min_p: 0.05

Training parameters

loss_type: sigmoid
num_train_epochs: 2
per_device_train_batch_size: 2
gradient_accumulation_steps: 8
learning_rate: 5.0e-7
lr_scheduler_type: linear
warmup_ratio: 0.5

Frameworks

• trl
• vLLM and vLLM Production Stack
• DeepSpeed
• Spectrum
• Liger Kernels
• argdown-feedback

Training Metrics

Training loss is going down both in each epoch and as we progress from one epoch to another:

The performance in the diverse argument analysis tasks, assessed against exercises from the eval split, improves significantly during HIRPO training:

Noteworthy:

• The model acquires skills it does initially not have whatsoever, such as informal argument reconstruction (infreco, around epoch 4), logical argument analysis (logreco, around epoch 20), and, with less success, coherent argument mapping and annotation (argmap_plus_arganno, around epoch 40).
• Having achieved high levels of performance, metrics keep oscillating characteristically, reflecting a drop in performance as training focus shifts dynamically to other tasks, followed by a rebound once training returns to the task with degraded performance.
• Llama-3.1-Argunaut-1-8B-HIRPO fails to acquire more complex tasks, like coherent annotation and logical analysis (arganno_plus_logreco). I'll try to fix this in future iterations, starting with more precise and detailed instructions / prompts.

The training currculum is dynamically adjusted as the model acquires more and more skills, as shown by the number of training items generated in each epoch:

This also demonstrates that a spiral curriculum, where training returns to previously mastered tasks, is an emerging feature of the training process (compare the oscillating number of infreco or logreco training items).

We may break down the performance in terms of error types being made. Thus, models may fail to generate a valid logical reconstruction of an argumentative text for very different reasons (e.g., misformatted febced code-block, missing conclusion, missing inference data, ill-formed logical formulas, etc.). The following diagram shows the prevalence of different mistakes (issues) made by the model in the logreco task:

We see that the model is drastically improving and starting to avoid specific mistakes well before it achieves even a minimal level of performance in the task (around epoch 20). This is because the model successfully transfers insights from other tasks it has been practicing, or because it learns purely from negative feedback (failure type preference pairs).

Evaluation

Chat experience

Let me share this chat with ggbetz/Llama-3.1-Argunaut-1-8B-HIRPO-dev4-Q4_K_M-GGUF.

[Conclusion~1]: School uniforms deter crime and increase student safety.
    <+ <Subargument 1>： 
        <+ [Conclusion~2]: Implementing school uniforms reduces bullying based on appearance or economic background.
            <+ <Subargument 2>： 
                <+ [Premise]: 83% of teachers think that a good school uniform can prevent bullying based on appearance or economic background.
        <+ [Premise]: Reducing bullying based on appearance or economic background contributes to a sense of belonging and social acceptance among students.
        <+ [Premise]: A sense of belonging and social acceptance among students is a desirable outcome.

(1) 83% of teachers think that a good school uniform can prevent bullying based on appearance or economic background.
(2) Reducing bullying based on appearance or economic background contributes to a sense of belonging and social acceptance among students.
(3) A sense of belonging and social acceptance among students is a desirable outcome.
--
School uniforms deter crime and increase student safety.

(1) 83% of teachers think that a good school uniform can prevent bullying based on appearance or economic background.
(2) Reducing bullying based on appearance or economic background contributes to a sense of belonging and social acceptance among students.
(3) A sense of belonging and social acceptance among students is a desirable outcome.
(4) A positive school environment contributes to a safer school environment.
--------------------------
(5) School uniforms deter crime and increase student safety.

My upshot is that improved argument analysis performance as demonstrated by evals does not translate directly into better performance in chats. ... Probably also shows that, at some point, Argunaut models will have to go through a final, classical instruction finetuning phase.

Benchmarks

🚧 coming soon, contributions welcome

Next steps

• So much to test empirically and evaluate thoroughly... So if you're graduate student or junior researcher looking for great projects with low hanging fruits to work on, please get in touch with me!