Upload folder using huggingface_hub

README.md

@@ -27,7 +27,7 @@ Solidity-Code-LLM is a fine tuned large language model designed to understand, g
 - **License:** MIT License
 - **Finetuned from model:** Salesforce/codegen-2B-multi
 
-![Training Pipeline](logo.png)
+![ChainGPT Logo](logo.png)
 
 
 # Model Details
@@ -45,28 +45,34 @@ Solidity-Code-LLM is a specialized language model trained in two stages: pre-tra
 - **Dtype**: bfloat16
 
 ### Model Sources
-For more details, please refer to,
-- **Paper [optional]:** {{ paper | default("[More Information Needed]", true)}}
+Model is deployed on huggingface space for inference
 - **Demo:** [Demo On Hugging Face Space](https://huggingface.co/spaces/Chain-GPT/SolidityLLMDemo)
 
-
 # Model Comparison
 We have compared our model with the following models
-- Qwen/CodeQwen1.5-7B
-- deepseek-ai/deepseek-coder-1.3b-base
-- codellama/CodeLlama-7b-hf
+- GPT 4.5 Preview
 - GPT 4o mini
+- [Qwen 2.5-Coder-7B](https://huggingface.co/Qwen/Qwen2.5-Coder-7B)
+- [DeepSeek-Coder-7B-Instruct-v1.5](https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5)
 
 On the following parameters
-- Compilation(%)--Percentage of generated contracts that compile successfully without modification.
-- OpenZeppelin Compliance(%)--Adherence to OpenZeppelin library usage and standards.
-- Gas Efficiency(%)--Degree of gas optimization based on Slither’s suggestions.
-- Security(%)--Percentage of code free from common vulnerabilities detected by Slither.
+- **Compilation(%)** - Percentage of generated contracts that compile successfully without modification.
+- **OpenZeppelin Compliance(%)** - Adherence to OpenZeppelin library usage and standards.
+- **Gas Efficiency(%)** - Degree of gas optimization based on Slither’s suggestions.
+- **Security(%)** - Percentage of code free from common vulnerabilities detected by Slither.
+- **Average Lines of Code** - Average number of non-empty, commented-included lines in generated contracts, indicating verbosity or conciseness
 
 ## Benchmark
-The figure below presents a detailed comparison of the models across all evaluation criteria
+Below is a figure summarizing the performance of each model across the four evaluation metrics.
 ![Benchmark](Benchmark.png)
 
+Following obvservation were made regarding Solidity LLM.
+- Highest Compilation Success Rate (~83%), demonstrating strong Solidity syntax and structure generation.
+- Good OpenZeppelin Compliance (~65%), indicating frequent use of standard libraries and contract patterns. While GPT-4.5, being a much larger model, naturally exhibits stronger adherence to OpenZeppelin standards due to its broader training data, Solidity LLM achieves commendable compliance given its smaller size.
+- Top Gas Efficiency (~72%), producing optimized code as evaluated by tools like Slither.
+- Moderate Security Score (~58%), showing acceptable security posture but room for improvement. GPT-4.5 benefits from its scale in handling more security cases.
+- Concise Code (~70% LOC score), generating relatively compact and efficient smart contracts.
+
 
 # Uses
 ### Direct Use
@@ -104,7 +110,7 @@ Use the code below to get started with the model.
 ```Python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-modelpath = "ChainGPT/SolidityLLM"
+modelpath = "Chain-GPT/Solidity-LLM"
 
 tokenizer = AutoTokenizer.from_pretrained(modelpath)
 model = AutoModelForCausalLM.from_pretrained(modelpath)
@@ -129,7 +135,7 @@ from threading import Thread
 from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer
 
 model = AutoModelForCausalLM.from_pretrained(
-    "Chain-GPT/Solidity-LLM",
+    "ChainGPT/SolidityLLM",
     torch_dtype=torch.bfloat16,
     device_map="cuda"
 )
@@ -272,7 +278,7 @@ contract DecentralizedLibrary is Ownable(msg.sender) {
 # Evaluation Matrics
 To evaluate the performance of our fine-tuned LLM specialized in Solidity smart contract generation, we used **[Slither](https://github.com/crytic/slither)**, a static analysis framework widely used for analyzing Solidity code.
 
-We focused on four key evaluation criteria:
+We focused on following key evaluation criteria:
 
 - **Compilation Success Rate**
 We measured the percentage of generated smart contracts that compile successfully without modification. This helps assess the syntactic and structural correctness of the model outputs.
@@ -286,8 +292,17 @@ Using Slither’s gas optimization analysis, we identified areas in the generate
 - **Security Vulnerabilities**
 We analyzed each contract for known security vulnerabilities using Slither’s built-in detectors. We recorded the number and severity of the vulnerabilities detected, providing a measure of the security quality of the model’s outputs.
 
-These evaluation metrics help quantify the practical usability and reliability of the generated smart contracts in real-world scenarios.
+- **Average Lines of Code (LOC)**
+Captures the average number of lines per generated contract, excluding blank lines but including comments. This metric reflects code verbosity or conciseness, and helps gauge implementation completeness versus potential redundancy.
+
+These metrics collectively provide a multi-dimensional view of the model’s effectiveness, spanning correctness, efficiency, security, and usability. They are designed to reflect both automated benchmarks and real-world developer expectations.
 
 
 # Summary
-Model shows improved understanding and generation capabilities in Solidity when compared to baseline LLMs not trained on Solidity data.
+Solidity LLM, despite its compact 2B parameter size, delivers standout performance in generating Solidity smart contracts. It achieved the highest compilation success rate (83%), showcasing robust syntactic and structural understanding. Its strong OpenZeppelin compliance (65%), though slightly behind very large models like GPT-4.5, is impressive given the scale difference, reflecting reliable use of industry-standard patterns and libraries.
+
+Further, Solidity LLM ranked highest in gas efficiency (72%), producing optimized code suitable for cost-sensitive deployments. While the security score (58%) indicates room for improvement, the model consistently generated secure-enough contracts for practical use. Its concise output (70% LOC score) also suggests an efficient coding style, balancing brevity with completeness.
+
+Overall, Solidity LLM proves to be a resource-efficient, reliable, and well-balanced model for Solidity code generation.
+
+Looking ahead, future releases will focus on improving support for newer versions of the Solidity language and OpenZeppelin libraries, enhancing user interaction by enabling contract modifications, expanding compatibility to other languages like Rust, and developing larger models capable of handling longer context windows.