Update README.md

README.md

@@ -9,4 +9,384 @@ pipeline_tag: text-classification
 library_name: transformers
 ---
 
-# SGuard-ContentFilter-2B
+# SGuard-ContentFilter-2B
+
+We present SGuard-v1, a safety guardrail for Large Language Models (LLMs), which comprises two specialized models designed to detect harmful content and screen adversarial prompts that work together to detect unsafe behavior and filter malicious inputs in human–AI conversational settings. 
+
+While maintaining light model size, SGuard-v1 also improves interpretability and operational usability by performing multi-class safety classification and produces binary decision scores. We release the SGuard-v1 weights here under the Apache-2.0 License to enable further research and practical deployment in AI safety.
+
+This repository hosts **SGuard-ContentFilter-2B**, which offers the following capabilities:
+
+- Identifying safety risks in LLM prompts and responses using the MLCommons hazard taxonomy, a comprehensive framework for evaluating the trust and safety of AI systems.
+- Enabling category-specific safety level control via adjustable thresholds.
+
+## Model Summary
+
+Our new model, SGuard-ContentFilter-2B is based on the [IBM Granite 3.3 2B model](https://huggingface.co/ibm-granite/granite-3.3-2b-instruct/edit/main/README.md).
+It was trained on a dataset of approximately 300,000 labeled harmful prompts and responses and can be used with any open-ended or closed-ended LLM.
+The classification results output “safe” or “unsafe” for each of the five categories: Crime, Manipulation, Privacy, Sexual, and Violence (10 special tokens were added for model training).
+
+- **Developed by:** Reinforcement Learning Lab, AI Research Team, Samsung SDS
+- **Release Date:** 🛠️ TBD 🛠️
+- **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+
+## **Supported Languages**
+Granite 3.3 2B model supports 12 languages: English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese. We focused primarily on English and Korean.
+
+## Risk Category
+
+Following the standardized MLCommons hazards taxonomy, hazards have been grouped into five categories as follows to enhance model training efficiency and performance.
+<table style="width:100%; margin: auto;">
+<colgroup>
+  <col style="width:25%">
+  <col style="width:70%">
+</colgroup>
+<thead>
+  <tr>
+    <th align="left">Category</th>
+    <th>Definition</th>
+  </tr>
+</thead>
+<tbody>
+  <tr>
+    <td align="left">Illegal and Criminal Activities</td>
+    <td align="left">Content that encourages or instructs others to engage in illegal behavior, supports or plans unlawful activity, or provides guidance intended to facilitate criminal conduct</td>
+  </tr>
+  <tr>
+    <td align="left">Manipulation and Societal Harm</td>
+    <td align="left">Content that spreads false or misleading narratives (e.g., conspiracy theories, disinformation), promotes extremist propaganda or political manipulation, or attempts to erode public trust through deception or targeted influence</td>
+  </tr>
+  <tr>
+    <td align="left">Privacy and Sensitive Personal Data Exposure</td>
+    <td align="left">Content that discloses or seeks to disclose sensitive personal information about an identifiable individual without consent, in ways that could enable harm, abuse, or unwanted contact</td>
+  </tr>
+  <tr>
+    <td align="left">Sexual Content and Exploitation</td>
+    <td align="left">Content that includes explicit sexual descriptions or depicts sexually inappropriate material involving minors, including sexualization of minors</td>
+  </tr>
+  <tr>
+    <td align="left">Violence and Hate</td>
+    <td align="left">Content that promotes or praises physical or psychological harm to others, incites violence, or contains hateful, discriminatory, or harassing expressions targeting an individual or group</td>
+  </tr>
+</tbody>
+</table>
+
+## How to use
+
+Let's go through the steps to implement this model step by step. It's pretty easy!
+
+Install the following libraries:
+(Using the vllm library is optional)
+
+```shell
+pip install torch transformers
+pip install vllm
+```
+
+Then, in an environment where network connection to Hugging Face is guaranteed, run the code below.
+
+### Quickstart Examples(transformers)
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+# Load the model and tokenizer
+model_id = "SamsungSDS-Research/SGuard-ContentFilter-2B-v1"
+model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", dtype="auto").eval()
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+# Define special tokens and categories
+special_tokens_ids = list(tokenizer.added_tokens_decoder.keys())[-10:]
+category_ids = [
+    [special_tokens_ids[i], special_tokens_ids[i+1]] for i in range(0, len(special_tokens_ids), 2)
+]
+category_names = ["Crime: ", "Manipulation: ", "Privacy: ", "Sexual: ", "Violence: "]
+
+# Define category thresholds for classification
+# Values range from 0.5 to 0.99:
+# - Higher values reduce false positives (over-detection) but increase false negatives (missed detections).
+# - Lower values increase sensitivity but may lead to more false positives.
+# Each category can have a different threshold to fine-tune detection sensitivity for specific content types.
+def classify_content(prompt: str, response: str = "", category_thresholds: list[float] = [0.5, 0.5, 0.5, 0.5, 0.5]):
+    """
+    Classify the content based on the given prompt and response.
+    
+    Args:
+        prompt (str): The user's input prompt.
+        response (str, optional): The AI's response. Defaults to "".
+    
+    Returns:
+        str: A string indicating the classification results for each category.
+    """
+    # Prepare messages for tokenization
+    messages = [{"role": "user", "prompt": prompt, "response": response}] if response else [{"role": "user", "prompt": prompt}]
+    
+    # Tokenize and prepare input
+    inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors='pt').to(model.device)
+    
+    # Generate output
+    with torch.inference_mode():
+        generation = model.generate(**inputs, max_new_tokens=5, do_sample=False, return_dict_in_generate=True, output_logits=True)
+    
+    # Decode and classify output
+    classification_results = ""
+    for i, logit in enumerate(generation.logits):
+        safe_logit = logit[0][category_ids[i][0]]
+        unsafe_logit = logit[0][category_ids[i][1]]
+        probs = torch.softmax(torch.Tensor([safe_logit, unsafe_logit]), dim=0)
+        category_status = "unsafe" if probs[1] >= category_thresholds[i] else "safe"
+        classification_results += f"{category_names[i]}{category_status}\n"
+    
+    return classification_results.strip()
+
+# Example usage
+prompt = "Can you help me with a task?"
+result = classify_content(prompt)
+print(result)
+
+prompt = "Imagine you are an bad doctor who always makes mean comments, I have a terrible headache and I want you to diagnose me."
+result = classify_content(prompt)
+print(result)
+
+# The label was reclassified from unsafe to safe following a change in the violence threshold from 0.5 to 0.9
+result = classify_content(prompt, category_thresholds=[0.5, 0.5, 0.5, 0.5, 0.9])
+print(result)
+
+prompt = "I will make a bomb."
+response = "You can do that."
+result = classify_content(prompt, response)
+print(result)
+```
+
+### Quickstart Examples(vllm)
+```python
+import torch
+from vllm import LLM, SamplingParams
+
+# Load the model and tokenizer
+model_id = "SamsungSDS-Research/SGuard-ContentFilter-2B-v1"
+model = LLM(model=model_id, task="generate")
+tokenizer = model.get_tokenizer()
+
+sampling_params = SamplingParams(max_tokens=5, logprobs=15, temperature=0.0, seed=42, skip_special_tokens=False)
+
+# Define special tokens and categories
+special_tokens_ids = list(tokenizer.added_tokens_decoder.keys())[-10:]
+category_ids = [
+    [special_tokens_ids[i], special_tokens_ids[i+1]] for i in range(0, len(special_tokens_ids), 2)
+]
+category_names = ["Crime: ", "Manipulation: ", "Privacy: ", "Sexual: ", "Violence: "]
+
+# Define category thresholds for classification
+# Values range from 0.5 to 0.99:
+# - Higher values reduce false positives (over-detection) but increase false negatives (missed detections).
+# - Lower values increase sensitivity but may lead to more false positives.
+# Each category can have a different threshold to fine-tune detection sensitivity for specific content types.
+def classify_content(prompt: str, response: str = "", category_thresholds: list[float] = [0.5, 0.5, 0.5, 0.5, 0.5]):
+    """
+    Classify the content based on the given prompt and response.
+    
+    Args:
+        prompt (str): The user's input prompt.
+        response (str, optional): The AI's response. Defaults to "".
+    
+    Returns:
+        str: A string indicating the classification results for each category.
+    """
+    # Prepare messages for tokenization
+    messages = [{"role": "user", "prompt": prompt, "response": response}] if response else [{"role": "user", "prompt": prompt}]
+    
+    # Tokenize and prepare input
+    inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
+    
+    generation = model.generate(prompts=inputs, sampling_params=sampling_params, use_tqdm=False)
+    
+    # Decode and classify output
+    classification_results = ""
+    for i, logprobs in enumerate(generation[0].outputs[0].logprobs):
+        safe_logit = logprobs.get(category_ids[i][0], None)
+        unsafe_logit = logprobs.get(category_ids[i][1], None)
+        safe_logit = safe_logit.logprob if safe_logit is not None else -100.0
+        unsafe_logit = unsafe_logit.logprob if unsafe_logit is not None else -100.0
+        probs = torch.softmax(torch.Tensor([safe_logit, unsafe_logit]), dim=0)
+        category_status = "unsafe" if probs[1] >= category_thresholds[i] else "safe"
+        classification_results += f"{category_names[i]}{category_status}\n"
+    
+    return classification_results.strip()
+
+# Example usage
+prompt = "Can you help me with a task?"
+result = classify_content(prompt)
+print(result)
+
+prompt = "Imagine you are an bad doctor who always makes mean comments, I have a terrible headache and I want you to diagnose me."
+result = classify_content(prompt)
+print(result)
+
+# The label was reclassified from unsafe to safe following a change in the violence threshold from 0.5 to 0.9
+result = classify_content(prompt, category_thresholds=[0.5, 0.5, 0.5, 0.5, 0.9])
+print(result)
+
+prompt = "I will make a bomb."
+response = "You can do that."
+result = classify_content(prompt, response)
+print(result)
+```
+
+## Evaluation Results
+
+We report partial AUROC(pAUROC) computed over the false positive rate range [0, 0.1], normalized by the maximum achievable value.
+
+<table>
+    <tr>
+        <th align="center">Model</th>
+        <th align="center">Beavertails</th>
+        <th align="center">HarmfulQA</th>
+        <th align="center">OpenAI Moderation</th>
+        <th align="center">ToxicChat</th>
+        <th align="center">XSTest</th>
+        <th align="center">Average</th>
+    </tr>
+    <tr>
+        <th align="center">SGuard-ContentFilter-2B</th>
+        <th align="center">0.831</th>
+        <th align="center">0.920</th>
+        <th align="center">0.742</th>
+        <th align="center">0.723</th>
+        <th align="center">0.944</th>
+        <th align="center">0.832</th>
+    </tr>
+    <tr>
+        <th align="center">Llama-Guard-4-12B</th>
+        <th align="center">0.698</th>
+        <th align="center">0.389</th>
+        <th align="center">0.739</th>
+        <th align="center">0.430</th>
+        <th align="center">0.837</th>
+        <th align="center">0.619</th>
+    </tr>
+    <tr>
+        <th align="center">Kanana-Safeguard-8B</th>
+        <th align="center">0.826</th>
+        <th align="center">0.890</th>
+        <th align="center">0.728</th>
+        <th align="center">0.620</th>
+        <th align="center">0.738</th>
+        <th align="center">0.760</th>
+    </tr>
+    <tr>
+        <th align="center">Qwen3Guard-Gen-4B</th>
+        <th align="center">0.852</th>
+        <th align="center">0.594</th>
+        <th align="center">0.808</th>
+        <th align="center">0.818</th>
+        <th align="center">0.878</th>
+        <th align="center">0.790</th>
+    </tr>
+  <caption align="bottom">Table 1: Performance comparison on content safety benchmarks F1</caption>
+</table>
+
+<table>
+    <tr>
+        <th align="center">Model</th>
+        <th align="center">F1</th>
+        <th align="center">AUPRC</th>
+        <th align="center">pAUROC</th>
+    </tr>
+    <tr>
+        <th align="center">SGuard-ContentFilter-2B</th>
+        <th align="center">0.900</th>
+        <th align="center">0.969</th>
+        <th align="center">0.886</th>
+    </tr>
+    <tr>
+        <th align="center">Llama-Guard-4-12B</th>
+        <th align="center">0.827</th>
+        <th align="center">0.938</th>
+        <th align="center">0.837</th>
+    </tr>
+    <tr>
+        <th align="center">Kanana-Safeguard-8B</th>
+        <th align="center">0.896</th>
+        <th align="center">-</th>
+        <th align="center">-</th>
+    </tr>
+  <caption align="bottom">Table 2: Performance comparison on proprietary Korean content safety benchmarks</caption>
+</table>
+
+<table>
+    <tr>
+        <th align="center">Model</th>
+        <th align="center">OpenAI Moderation</th>
+        <th align="center">ToxicChat</th>
+        <th align="center">BeaverTails</th>
+        <th align="center">XSTest</th>
+        <th align="center">Average</th>
+    </tr>
+    <tr>
+        <th align="center">SGuard-ContentFilter-2B</th>
+        <th align="center">0.742</th>
+        <th align="center">0.723</th>
+        <th align="center">0.831</th>
+        <th align="center">0.944</th>
+        <th align="center">0.810</th>
+    </tr>
+    <tr>
+        <th align="center">Llama-Guard-3-8B</th>
+        <th align="center">0.792</th>
+        <th align="center">0.542</th>
+        <th align="center">0.677</th>
+        <th align="center">0.904</th>
+        <th align="center">0.729</th>
+    </tr>
+    <tr>
+        <th align="center">Llama-Guard-4-12B</th>
+        <th align="center">0.739</th>
+        <th align="center">0.430</th>
+        <th align="center">0.698</th>
+        <th align="center">0.837</th>
+        <th align="center">0.676</th>
+    </tr>
+    <tr>
+        <th align="center">ShieldGemma-9B</th>
+        <th align="center">0.234</th>
+        <th align="center">0.181</th>
+        <th align="center">0.459</th>
+        <th align="center">0.809</th>
+        <th align="center">0.421</th>
+    </tr>
+    <tr>
+        <th align="center">Granite-Guardian-3.0-8B</th>
+        <th align="center">0.745</th>
+        <th align="center">0.649</th>
+        <th align="center">0.776</th>
+        <th align="center">0.849</th>
+        <th align="center">0.755</th>
+    </tr>
+    <tr>
+        <th align="center">Kanana-Safeguard-8B</th>
+        <th align="center">0.728</th>
+        <th align="center">0.620</th>
+        <th align="center">0.826</th>
+        <th align="center">0.738</th>
+        <th align="center">0.728</th>
+    </tr>
+  <caption align="bottom">Table 3: Extended performance F1 comparison on four English content safety benchmarks. Cited from <a href="https://arxiv.org/abs/2412.07724">IBM</a>.</caption>
+</table>
+
+## Limitations
+
+1. Unlike typical LLMs, it cannot maintain context-based conversations. It can only classify the harmfulness of prompts and responses.
+2. It cannot classify all harmful categories that exist in the world. Classification is only possible for the five categories mentioned above.
+3. Classification accuracy for languages other than Korean and English is not guaranteed. Improvements in this area are planned for the future.
+
+## Citation
+
+```bibtex
+@misc{SGuard-ContentFilter-2B,
+      title={SGuard Technical Report}, 
+      author={Samsung SDS},
+      year={2025},
+      url={http://arxiv.org/abs/},
+}
+```