library_name: transformers
tags:
- trl
- sft
license: apache-2.0
base_model: Qwen/Qwen2.5-1.5B-Instruct
datasets:
- IsmaelMousa/books
metrics:
- accuracy
- f1
- precision
- recall
- cohen_kappa
- rmse
model-index:
- name: Qwen2.5-1.5B-Instruct-Books-19K
results:
- task:
name: Text Generation
type: text-generation
dataset:
name: IsmaelMousa/books
type: IsmaelMousa/books
config: IsmaelMousa/books
split: train
args: IsmaelMousa/books
metrics:
- name: Accuracy
type: accuracy
value: 0.12
- name: F1
type: f1
value: 0.0973
- name: Precision
type: precision
value: 0.1739
- name: Recall
type: recall
value: 0.1198
- name: Cohen Kappa
type: cohen_kappa
value: -0.1549
- name: RMSE
type: rmse
value: 1.6248
language:
- en
pipeline_tag: text-generation
Qwen2.5-1.5B-Instruct-Books-19K
This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the Books dataset for Essay Grading.
- Workflow: GitHub Repository: https://github.com/IsmaelMousa/automatic-essay-grading.
- Base Model: Qwen2.5-1.5B-Instruct: https://doi.org/10.48550/arXiv.2412.15115.
- Fine-tuning Dataset: Books-19K: https://github.com/IsmaelMousa/Books/19K.
- Task: Automatic Essay Grading (Text Generation).
Dataset
The Books dataset is a synthetic collection of essay-style data points generated using public domain literature and large language model prompting. The dataset comprises a total of 300 entries and is built from six classic books. Four of these: The Life of James Watt, The Life of Julius Caesar, The Moonstone, and North and South; were used during the training phase, while the remaining two: The Life of Napoleon and Sense and Sensibility; were held out for benchmarking purposes. Each book contributed exactly 50 entries, leading to a structured split of 200 training samples and 100 benchmark samples.
All entries were generated using Le Chat Mistral, a model developed by Mistral AI. A carefully crafted prompt was used to ensure each generated entry included a question, a reference answer written by an expert, a student answer meant to simulate a real-world response, a mark scheme outlining the grading criteria, a score between 1 and 4, and a rationale explaining why the score was assigned. The prompt enforced strict quality control: no duplicate questions or answers were allowed, all required fields had to be present, and the scoring range was strictly limited to valid values. The final output was formatted as CSV files to maintain consistency and ensure compatibility with downstream processing.
For more details, the metadata can be accessed at: metadata.
Modeling
The modeling approach for this study was carefully designed to evaluate the performance of different large language models (LLMs) on the automated essay grading task. We selected the Qwen2.5 architecture to represent a range of model sizes: 0.5B, 1.5B, and 3B. Each model was instruction-tuned on the Books dataset in varying sizes, with hyperparameters optimized to balance computational efficiency and performance. The experiments were conducted on GPU-accelerated hardware, leveraging techniques such as gradient checkpointing, flash attention, and mixed-precision training to maximize resource utilization.
Evaluation
The evaluation methodology employed both quantitative metrics and qualitative analysis. For quantitative assessment, we computed accuracy, precision, recall, F1 score, root mean squared error (RMSE), and Cohen's kappa score (CKS) for the scoring task, while using BERT-Score precision, recall, and F1 for rationale evaluation. On a held-out test set of 100 samples. Qualitative examination of models' outputs revealed cases where most of the models correctly identified key aspects of student answers but sometimes failed to properly align its scoring with the rubric criteria.
Evaluation results for score and rationale outputs:
| Aspect | F1 | Precision | Recall | Accuracy | CKS | RMSE |
|---|---|---|---|---|---|---|
| Score | 0.0973 | 0.1739 | 0.1198 | 0.1200 | -0.1549 | 1.6248 |
| Rationale | 0.4921 | 0.4872 | 0.5001 | -- | -- | -- |
Usage
Below is an example of how to use the model with the Hugging Face Transformers library:
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
import torch
checkpoint = "IsmaelMousa/Qwen2.5-1.5B-Instruct-Books-19K"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer .from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint)
assistant = pipeline("text-generation", tokenizer=tokenizer, model=model, device=device)
question = input("Question : ")
reference_answer = input("Reference Answer: ")
student_answer = input("Student Answer : ")
mark_scheme = input("Mark Scheme : ")
system_content = "You are a grading assistant. Evaluate student answers based on the mark scheme. Respond only in JSON format with keys 'score' (int) and 'rationale' (string)."
user_content = ("Provide both a score and a rationale by evaluating the student's answer strictly within the mark scheme range,"
" grading based on how well it meets the question's requirements by comparing the student answer to the reference answer.\n"
f"Question: {question}\n"
f"Reference Answer: {reference_answer}\n"
f"Student Answer: {student_answer}\n"
f"Mark Scheme: {mark_scheme}")
messages = [{"role": "system", "content": system_content}, {"role": "user", "content": user_content}]
inputs = tokenizer.apply_chat_template(messages, tokenize=False)
output = assistant(inputs, max_new_tokens=128, do_sample=False, return_full_text=False)[0]["generated_text"]
print(output)
Frameworks
datasets-3.6.0torch-2.7.0transformers-4.51.3trl-0.17.0scikit-learn-1.6.1bert-score-0.3.13json-repair-0.46.0