YAML Metadata Warning: The pipeline tag "text2text-generation" is not in the official list: text-classification, token-classification, table-question-answering, question-answering, zero-shot-classification, translation, summarization, feature-extraction, text-generation, fill-mask, sentence-similarity, text-to-speech, text-to-audio, automatic-speech-recognition, audio-to-audio, audio-classification, audio-text-to-text, voice-activity-detection, depth-estimation, image-classification, object-detection, image-segmentation, text-to-image, image-to-text, image-to-image, image-to-video, unconditional-image-generation, video-classification, reinforcement-learning, robotics, tabular-classification, tabular-regression, tabular-to-text, table-to-text, multiple-choice, text-ranking, text-retrieval, time-series-forecasting, text-to-video, image-text-to-text, visual-question-answering, document-question-answering, zero-shot-image-classification, graph-ml, mask-generation, zero-shot-object-detection, text-to-3d, image-to-3d, image-feature-extraction, video-text-to-text, keypoint-detection, visual-document-retrieval, any-to-any, video-to-video, other

Model Card for Model ID

Model Description

This model is quantized in 8-bit and trained with question and answer pairs for text-to-SQL tasks using the LoRA PEFT method. It serves as a foundation model for further development in Text-to-SQL Retrieval-Augmented Generation (RAG) applications.

Developed by: Lei-bw
Model type: Causal Language Model
Language(s) (NLP): English
License: bsl-1.0
Finetuned from model: google/gemma-2b
Device to be used: NVIDIA GeForce RTX 3080 Ti (12288MiB)

How to Get Started with the Model

Apply the access to model google/gemma-2b You may submit the approval from google/gemma-2b
Login-in If you are using cli, please use below command to login your huggingface account with your access token key:

huggingface-cli login

If you are using notebook, please use below command to login your huggingface account with your access token key:

from huggingface_hub import login
login()

Install the required library

peft==0.12.0
transformers==4.44.2

Get access to the Lei-bw/text-to-sql-fm model

from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

config = PeftConfig.from_pretrained("Lei-bw/text-to-sql-fm")
base_model = AutoModelForCausalLM.from_pretrained("google/gemma-2b")
model = PeftModel.from_pretrained(base_model, "Lei-bw/text-to-sql-fm")
tokenizer = AutoTokenizer.from_pretrained("Lei-bw/text-to-sql-fm")

Example to use this model:

import torch

# Example usage
eval_prompt = """
How many seniors in the departments are younger than 28?
"""

model_input = tokenizer(eval_prompt, return_tensors="pt")

model.eval()
with torch.no_grad():
    print(tokenizer.decode(model.generate(**model_input, max_new_tokens=300)[0], skip_special_tokens=True))

Training Details

Training Data

The model was trained on the b-mc2/sql-create-context dataset, which contains question and answer pairs for SQL generation tasks.

Training Hyperparameters

•	Training regime: bf16 mixed precision
•	Batch size: 16 
•	Gradient accumulation steps: 4
•	Warmup steps: 50
•	Number of epochs: 2
•	Learning rate: 2e-4
•	Weight decay: 0.01
•	Optimizer: AdamW
•	Learning rate scheduler: Linear

Hardware

Hardware Type: NVIDIA A100
GPU RAM: 40 GB

Framework versions

PEFT 0.12.0

Downloads last month: 13

Model tree for Lei-bw/text-to-sql-fm

Base model

google/gemma-2b

Adapter

(23555)

this model

Lei-bw
/

text-to-sql-fm