cgg507/Valkyrie-v1-gptq-channel

Results

Quant Performance

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.8135	±	0.0107
		strict-match	5	exact_match	↑	0.8059	±	0.0109
hellaswag	1	none	0	acc	↑	0.6308	±	0.0048
		none	0	acc_norm	↑	0.8256	±	0.0038
humaneval	1	create_test	0	pass@1		0.4146	±	0.0386
mbpp	1	none	3	pass_at_1	↑	0.5860	±	0.0220
piqa	1	none	0	acc	↑	0.8194	±	0.0090
		none	0	acc_norm	↑	0.8303	±	0.0088
rte	1	none	0	acc	↑	0.7870	±	0.0246

Base Performance

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.9242	±	0.0073
		strict-match	5	exact_match	↑	0.9219	±	0.0074
hellaswag	1	none	0	acc	↑	0.6548	±	0.0047
		none	0	acc_norm	↑	0.8496	±	0.0036
humaneval	1	create_test	0	pass@1		0.4329	±	0.0388
mbpp	1	none	3	pass_at_1	↑	0.6800	±	0.0209
piqa	1	none	0	acc	↑	0.8270	±	0.0088
		none	0	acc_norm	↑	0.8400	±	0.0086
rte	1	none	0	acc	↑	0.7220	±	0.0270

Differences

Task	Metric	Quant Value	Base Value	Difference (Base - Quant)
gsm8k	exact_match	0.8135	0.9242	+0.1107
gsm8k	exact_match (strict)	0.8059	0.9219	+0.1160
hellaswag	acc	0.6308	0.6548	+0.0240
hellaswag	acc_norm	0.8256	0.8496	+0.0240
humaneval	pass@1	0.4146	0.4329	+0.0183
mbpp	pass_at_1	0.5860	0.6800	+0.0940
piqa	acc	0.8194	0.8270	+0.0076
piqa	acc_norm	0.8303	0.8400	+0.0097
rte	acc	0.7870	0.7220	-0.0650