Update README.md

README.md

@@ -11,26 +11,57 @@ language:
 - en
 ---
 
-# Uploaded  model
+# Uploaded model
 
-- **Developed by:** forestav
-- **License:** apache-2.0
-- **Finetuned from model :** unsloth/llama-3.2-1b-instruct-bnb-4bit
+- **Developed by:** forestav  
+- **License:** apache-2.0  
+- **Finetuned from model:** [unsloth/llama-3.2-1b-instruct-bnb-4bit](https://huggingface.co/unsloth/llama-3.2-1b-instruct-bnb-4bit)
 
-This model was a further development of a LoRA adapter model trained on unsloth/Llama-3.2-3B-Instruct with the FineTome-100k dataset.
+## Model description
+This model is a refined version of a LoRA adapter trained on the **unsloth/Llama-3.2-3B-Instruct** model using the **FineTome-100k** dataset. The finetuned model uses fewer parameters (1B vs. 3B) to achieve faster training and improved adaptability for specific tasks, such as medical applications.  
 
-Here, we used a smaller model, with 1B parameters instead of 3B. We wanted to have faster training capabilities as well as make it easier to finetune to our specific application, which is easier when using a pretrained model with fewer parameters.
+### Key adjustments:
+1. **Reduced Parameter Count:** The model was downsized to 1B parameters to improve training efficiency and ease customization.
+2. **Adjusted Learning Rate:** A smaller learning rate was used to prevent overfitting and mitigate catastrophic forgetting. This ensures the model retains its general pretraining knowledge while learning new tasks effectively.
 
-We also changed the training arguments slightly. We set a smaller learning rate as fewer parameters makes the model more prone to overfitting. By using a smaller learning rate, we reduce the likelihood of catastrophic forgetting where the model loses the general knowledge it acquired during pretraining, becoming overly specialized for medical advice or forgetting basic language patterns.
-When having a large learning rate, the large weight updates can overwrite pretrained weights, causing the model to lose generalization.
+The finetuning dataset, **ruslanmv/ai-medical-chatbot**, contains only 257k rows, which necessitated careful hyperparameter tuning to avoid over-specialization.
 
-Our finetuning dataset ruslanmv/ai-medical-chatbot only has 257k rows, much smaller than the data used for pretraining. A smaller learning rate ensures that the model learns patterns specific to our data without deviating too far from its pretrained state.
+---
+
+## Hyperparameters and explanations
+
+- **Learning rate:** `2e-5`  
+  A smaller learning rate reduces the risk of overfitting and catastrophic forgetting, particularly when working with models containing fewer parameters.
+
+- **Warm-up steps:** `5`  
+  Warm-up allows the optimizer to gather gradient statistics before training at the full learning rate, improving stability.
+
+- **Per device train batch size:** `2`  
+  Each GPU processes 2 training samples per step. This setup is suitable for resource-constrained environments.
+
+- **Gradient accumulation steps:** `4`  
+  Gradients are accumulated over 4 steps to simulate a larger batch size (effective batch size: 8) without exceeding memory limits.
 
-# Hyperparameters and explanations:
-**Warm up steps: 5.** The warmup steps tell us how many steps we should have a low learning rate prior to training with the normal learning rate. It is important when having the Adam optimizer as it relies on statistics of the gradients, which we do not have unless we have computed some gradients a priori.
+- **Optimizer:** `AdamW with 8-bit Quantization`  
+  - **AdamW:** Adds weight decay to prevent overfitting.  
+  - **8-bit Quantization:** Reduces memory usage by compressing optimizer states, facilitating faster training.
 
-**Per device train batch size: 2.** We take 2 training samples for each GPU (we only had one GPU).
+- **Weight decay:** `0.01`  
+  Standard weight decay value effective across various training scenarios.
 
-This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
+- **Learning rate scheduler type:** `Linear`  
+  Gradually decreases the learning rate from the initial value to zero over the course of training.
+
+---
+
+## Quantization details
+The model is saved in **16-bit GGUF format**, which:  
+- Ensures **100% accuracy retention**.  
+- Trades off speed and memory for improved precision.  
+
+### Training optimization
+Training was accelerated by **2x** using [Unsloth](https://github.com/unslothai/unsloth) in combination with Hugging Face's **TRL library**.
+
+---
 
-[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
+[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)