Hugging Face's logo

JetBrains
/
Mellum-4b-base

Text Generation
Transformers
Safetensors
llama
code
Eval Results
text-generation-inference
Model card Files
xet
Community
7
dustalov commited on
Commit
c516a59
·
verified ·
1 Parent(s): 55fe2ca

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +16 -12
README.md CHANGED
@@ -17,8 +17,8 @@ model-index:
17
  type: tianyang/repobench_python_v1.1
18
  name: RepoBench 1.1 (Python, 2k)
19
  metrics:
20
- - name: pass@1
21
- type: pass@1
22
  value: 0.2820
23
  verified: false
24
  - task:
@@ -27,8 +27,8 @@ model-index:
27
  type: tianyang/repobench_python_v1.1
28
  name: RepoBench 1.1 (Python, 4k)
29
  metrics:
30
- - name: pass@1
31
- type: pass@1
32
  value: 0.2795
33
  verified: false
34
  - task:
@@ -37,8 +37,8 @@ model-index:
37
  type: tianyang/repobench_python_v1.1
38
  name: RepoBench 1.1 (Python, 8k)
39
  metrics:
40
- - name: pass@1
41
- type: pass@1
42
  value: 0.2777
43
  verified: false
44
  - task:
@@ -47,8 +47,8 @@ model-index:
47
  type: tianyang/repobench_python_v1.1
48
  name: RepoBench 1.1 (Python, 12k)
49
  metrics:
50
- - name: pass@1
51
- type: pass@1
52
  value: 0.2453
53
  verified: false
54
  - task:
@@ -57,8 +57,8 @@ model-index:
57
  type: tianyang/repobench_python_v1.1
58
  name: RepoBench 1.1 (Python, 16k)
59
  metrics:
60
- - name: pass@1
61
- type: pass@1
62
  value: 0.2110
63
  verified: false
64
  - task:
@@ -67,10 +67,14 @@ model-index:
67
  type: tianyang/repobench_python_v1.1
68
  name: RepoBench 1.1 (Python)
69
  metrics:
70
- - name: pass@1
71
- type: pass@1
72
  value: 0.2591
73
  verified: false
 
 
 
 
74
  - task:
75
  type: text-generation
76
  dataset:
 
17
  type: tianyang/repobench_python_v1.1
18
  name: RepoBench 1.1 (Python, 2k)
19
  metrics:
20
+ - name: EM
21
+ type: exact_match
22
  value: 0.2820
23
  verified: false
24
  - task:
 
27
  type: tianyang/repobench_python_v1.1
28
  name: RepoBench 1.1 (Python, 4k)
29
  metrics:
30
+ - name: EM
31
+ type: exact_match
32
  value: 0.2795
33
  verified: false
34
  - task:
 
37
  type: tianyang/repobench_python_v1.1
38
  name: RepoBench 1.1 (Python, 8k)
39
  metrics:
40
+ - name: EM
41
+ type: exact_match
42
  value: 0.2777
43
  verified: false
44
  - task:
 
47
  type: tianyang/repobench_python_v1.1
48
  name: RepoBench 1.1 (Python, 12k)
49
  metrics:
50
+ - name: EM
51
+ type: exact_match
52
  value: 0.2453
53
  verified: false
54
  - task:
 
57
  type: tianyang/repobench_python_v1.1
58
  name: RepoBench 1.1 (Python, 16k)
59
  metrics:
60
+ - name: EM
61
+ type: exact_match
62
  value: 0.2110
63
  verified: false
64
  - task:
 
67
  type: tianyang/repobench_python_v1.1
68
  name: RepoBench 1.1 (Python)
69
  metrics:
70
+ - name: EM
71
+ type: exact_match
72
  value: 0.2591
73
  verified: false
74
+ - name: EM ≤ 8k
75
+ type: exact_match
76
+ value: 0.2797
77
+ verified: false
78
  - task:
79
  type: text-generation
80
  dataset: