mahing
/

historical-narrative-generator

@@ -19,6 +19,7 @@ To successfully fine-tune an LLM for this task, I first picked a suitable base m
 **Training Data** <br />
 For this task, I utilized the various first-person sources and historical documents from Project Gutenberg as input data, along with manual searching for certain well-known documents. Project Gutenberg’s main goal is to digitize cultural and historical works, thereby including many biographies and memoirs throughout history that would be perfect in teaching an LLM to build out an accurate narrative from the document’s era. The output corresponding to this input data will be a first-person narrative based on the events in the input data. For example, if the input data is the description of a war, the output can be a soldier’s first-person account of their daily life during the war. The main source of my data wrangling was synthetically generating these first-person narratives using an LLM and testing its output using my knowledge and other LLMs to determine the strength of the response. Doing this was a tedious task, and I finished with approximately 900 document-narrative pairs, which I split up into 700 for the training set, and 100 for the validation and test sets using a random seed of 42.
 **Training Method** <br />
 I chose to use LoRA for my task of creating first-person historical narratives of an era. Based on previous results, few shot prompting sometimes did not capture the improvements I hoped to see from responses. Full fine-tuning would be more computationally intensive than LoRA and does not seem necessary for my task. LoRA is a good balance between the two, only changing some parameters related to my task, and using the data set to update key parameters to help create narratives in a style that better matches the prose of an era and the historical accuracy of it. LoRA can also perform well without a massive training data set because of its low-rank adaptations.
 For my hyperparameter combinations, I chose to use LORA_R = 128, LORA_ALPHA = 128, and LORA_DROPOUT = .1. These hyperparameters had the best qualitative results out of the options I tried. Despite my smaller data set, this approach gave strong first-person narratives that I enjoyed. They included prose from the era, were historically accurate, and even included imagery and entertaining details that I'd expect from a quality response. The results from these hyperparameters exceeded any expectations I had.

 **Training Data** <br />
 For this task, I utilized the various first-person sources and historical documents from Project Gutenberg as input data, along with manual searching for certain well-known documents. Project Gutenberg’s main goal is to digitize cultural and historical works, thereby including many biographies and memoirs throughout history that would be perfect in teaching an LLM to build out an accurate narrative from the document’s era. The output corresponding to this input data will be a first-person narrative based on the events in the input data. For example, if the input data is the description of a war, the output can be a soldier’s first-person account of their daily life during the war. The main source of my data wrangling was synthetically generating these first-person narratives using an LLM and testing its output using my knowledge and other LLMs to determine the strength of the response. Doing this was a tedious task, and I finished with approximately 900 document-narrative pairs, which I split up into 700 for the training set, and 100 for the validation and test sets using a random seed of 42.
 **Training Method** <br />
 I chose to use LoRA for my task of creating first-person historical narratives of an era. Based on previous results, few shot prompting sometimes did not capture the improvements I hoped to see from responses. Full fine-tuning would be more computationally intensive than LoRA and does not seem necessary for my task. LoRA is a good balance between the two, only changing some parameters related to my task, and using the data set to update key parameters to help create narratives in a style that better matches the prose of an era and the historical accuracy of it. LoRA can also perform well without a massive training data set because of its low-rank adaptations.
 For my hyperparameter combinations, I chose to use LORA_R = 128, LORA_ALPHA = 128, and LORA_DROPOUT = .1. These hyperparameters had the best qualitative results out of the options I tried. Despite my smaller data set, this approach gave strong first-person narratives that I enjoyed. They included prose from the era, were historically accurate, and even included imagery and entertaining details that I'd expect from a quality response. The results from these hyperparameters exceeded any expectations I had.