base_model: []
library_name: transformers
tags:
- mergekit
- merge
This is a merge of Bytedance Seed-OSS-36B Base and Instruct, using the karcher-means method in mergekit, with the idea being to get Bytedance Instruct to 'feel' and write more like a raw continuation model.
Karcher was tested because this and SLERP are seemingly the only viable ways to merge and instruct and base model.
Quantized, it gets an MMLU score (via the exllamav3 eval script) of 1853/ 14042 = 84.41% correct, ( 80.41% prob.)
For reference, ByteDance's instruct model (with the exact same quantization settings) gets 11680/ 14042 = 83.18% correct, ( 80.96% prob.)
These are the raw bf16 weights of the merge.
Merge Details
Merge Method
This model was merged using the Karcher Mean merge method using /home/alpha/Models/Raw/ByteDance-Seed_Seed-OSS-36B-Instruct as a base.
Models Merged
The following models were included in the merge:
- /home/alpha/Models/Raw/ByteDance-Seed_Seed-OSS-36B-Base
Configuration
The following YAML configuration was used to produce this model:
models:
# - model: /home/alpha/Models/Raw/Qwen_Qwen2.5-14B
# No parameters necessary for base model
- model: /home/alpha/Models/Raw/ByteDance-Seed_Seed-OSS-36B-Base
- model: /home/alpha/Models/Raw/ByteDance-Seed_Seed-OSS-36B-Instruct
merge_method: karcher
tokenizer:
source: "base"
base_model: /home/alpha/Models/Raw/ByteDance-Seed_Seed-OSS-36B-Instruct
parameters:
int8_mask: true
dtype: bfloat16