Updated readme

README.md

@@ -1,3 +1,102 @@
+license: apache-2.0
+datasets:
+- amaai-lab/MidiCaps
+tags:
+- music
+- text-to-music
+- symbolic-music
+pipeline_tag: text-to-music
 ---
-license: mit
----
+
+<div align="center">
+
+# text2midi: Generating Symbolic Music from Captions
+
+[Demo](https://huggingface.co/spaces/amaai-lab/text2midi) | [Model](https://huggingface.co/amaai-lab/text2midi) | [Website and Examples](https://github.com/AMAAI-Lab/text2midi) | [Paper](https://arxiv.org/abs/2311.08355) | [Dataset](https://huggingface.co/datasets/amaai-lab/MidiCaps)
+
+[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/amaai-lab/text2midi)
+</div>
+
+**text2midi** is the first end-to-end model for generating MIDI files from textual descriptions. By leveraging pretrained large language models and a powerful autoregressive transformer decoder, **text2midi** allows users to create symbolic music that aligns with detailed textual prompts, including musical attributes like chords, tempo, and style.
+
+🔥 Live demo available on [HuggingFace Spaces](https://huggingface.co/spaces/amaai-lab/text2midi).
+
+<div align="center">
+  <img src="text2midi_architecture.jpg" width="500"/>
+</div>
+
+## Quickstart Guide
+
+Generate symbolic music from a text prompt:
+
+```python
+from transformers import T5Tokenizer
+from model.transformer_model import Transformer
+from miditok import REMI, TokenizerConfig
+from pathlib import Path
+
+device = 'cuda' if torch.cuda.is_available() else 'cpu'
+artifact_folder = 'artifacts'
+
+tokenizer_filepath = os.path.join(artifact_folder, "vocab_remi.pkl")
+# Load the tokenizer dictionary
+with open(tokenizer_filepath, "rb") as f:
+    r_tokenizer = pickle.load(f)
+
+# Get the vocab size
+vocab_size = len(r_tokenizer)
+print("Vocab size: ", vocab_size)
+model = Transformer(vocab_size, 768, 8, 5000, 18, 1024, False, 8, device=device)
+model.load_state_dict(torch.load('/text2midi/artifacts/pytorch_model_140.bin', map_location=device))
+model.eval()
+tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-base")
+
+src = "A melodic electronic song with ambient elements, featuring piano, acoustic guitar, alto saxophone, string ensemble, and electric bass. Set in G minor with a 4/4 time signature, it moves at a lively Presto tempo. The composition evokes a blend of relaxation and darkness, with hints of happiness and a meditative quality."
+inputs = tokenizer(src, return_tensors='pt', padding=True, truncation=True)
+input_ids = nn.utils.rnn.pad_sequence(inputs.input_ids, batch_first=True, padding_value=0)
+input_ids = input_ids.to(device)
+attention_mask =nn.utils.rnn.pad_sequence(inputs.attention_mask, batch_first=True, padding_value=0) 
+attention_mask = attention_mask.to(device)
+output = model.generate(input_ids, attention_mask, max_len=2000,temperature = 1.0)
+output_list = output[0].tolist()
+generated_midi = r_tokenizer.decode(output_list)
+generated_midi.dump_midi("output.mid")
+post_processing("output.mid", "output.mid")
+```
+
+## Installation
+git clone https://github.com/AMAAI-Lab/text2midi
+cd text2midi
+pip install -r requirements.txt
+
+## Datasets
+The MidiCaps dataset is a large-scale dataset of 168k MIDI files paired with rich text captions. These captions contain musical attributes such as key, tempo, style, and mood, making it ideal for text-to-MIDI generation tasks.
+
+## Subjective Evaluation by Expert Listeners
+Model	Dataset	Pre-trained	Overall Match ↑	Chord Match ↑	Tempo Match ↑	Symbolic Quality ↑	Musicality ↑	Text Alignment ↑
+MuseCoco	MidiCaps	✓	4.12	3.02	3.85	3.50	3.20	3.90
+text2midi	MidiCaps	✓	4.85	4.10	4.62	4.25	4.45	4.78
+
+## Training
+To train text2midi, we recommend using accelerate for multi-GPU support. First, configure accelerate by running:
+accelerate config
+
+Then, use the following command to start training:
+accelerate launch train.py \
+--encoder_model="google/flan-t5-large" \
+--decoder_model="configs/transformer_decoder_config.json" \
+--dataset_name="amaai-lab/MidiCaps" \
+--pretrain_dataset="amaai-lab/SymphonyNet" \
+--batch_size=16 \
+--learning_rate=1e-4 \
+--epochs=40 \
+
+## Citation
+If you use text2midi in your research, please cite:
+@misc{bhandari2025text2midi,
+      title={text2midi: Generating Symbolic Music from Captions}, 
+      author={Keshav Bhandari and Abhinaba Roy and Kyra Wang and Geeta Puri and Simon Colton and Dorien Herremans},
+      year={2025},
+      eprint={2311.08355},
+      archivePrefix={arXiv},
+}