The doom is still buried within Command-A for sure.

The doom is still buried within Command-A for sure.

Only another 38 days to go:

A step 601 preview - all with temperature = 0:

https://pastebin.com/GASKaHTk

https://pastebin.com/CRT81QLb

• It's still messing up some end of lines, but I can live with that if it works... Likely can be fixed later using the new class 0 random data if a problem.
• The Grimdark story was noticeably (much!) better compared to the inverse.
• The Battlestar Galactica story showed that even though Q8_0, F16 and BF16 all diverge slightly from F32; it's not clearly making them any worse (I actually liked the Q8_0 story best!).

It still has a way to go before it starts to converge, but I would think by step 1000 it will be pretty close:

566 responses in previous thread! In the future we may be the reason for hf staff to implement multi-page view of discussions.

This was posted on Hacker News today:

https://outsidetext.substack.com/p/how-does-a-blind-model-see-the-earth?selection=5413dcae-b9f4-4adb-8826-d48e3908de2a#:~:text=Wow%2C%20best%20rendition%20of%20the%20Global%20West%20so%20far

Absolutely fascinating!

This was posted on Hacker News today:

https://outsidetext.substack.com/p/how-does-a-blind-model-see-the-earth?selection=5413dcae-b9f4-4adb-8826-d48e3908de2a#:~:text=Wow%2C%20best%20rendition%20of%20the%20Global%20West%20so%20far

Absolutely fascinating!

That was really cool. Thanks for sharing!

This was posted on Hacker News today:

https://outsidetext.substack.com/p/how-does-a-blind-model-see-the-earth?selection=5413dcae-b9f4-4adb-8826-d48e3908de2a#:~:text=Wow%2C%20best%20rendition%20of%20the%20Global%20West%20so%20far

Absolutely fascinating!

That was really cool. Thanks for sharing!

Yeah, and llama-3.1:405b doing so well was quite a surprise too (and makes you a bit sad everything seems to be moving away from large dense models ).

Gemma4 is awesome for writing, but... it doesn't know characters from anywhere. It even messes up the characters from the most popular series and franchises. You can point it out to them with explicit nudging (e.g. it messed up the questions about the creative output of Phillip Pullman with Phillip Dick and it answered only than I pointed out that Pullman was the author of His Dark Materials, and it then started speaking about the daemons and stuff). If you think about it, the character namings are in Wikipedia, and you could learn them just from reading Wikipedia multiple times. Looks like Google did a lot effort of cleaning copyright out of the training set or including it, but obfuscating as much as possible, or - if it is a 100% distillation of Gemini - simply not including it at all. That's very bad if you were to write a fanfic or to roleplay a pre-existing person, otherwise it's good. Still sad because of the copyright exclusion fact :(

In your opinion, what model has the best knowledge of existing IP (anime, ln, games, books,..) while not being a GLM/Kimi/DS monster? (<100b params)

Many models have medium-to-good writing/RP capability, but absolutely no grasping of fandoms. Can you recommend one? (Tunes/merges are perfectly acceptable) I like to use L3.3 Cu-Mai for fanfics (grabbed it from the UGI leaderboard sorted by pop culture knowledge column), but it is still quite limited

Gotta be gemma for pop-culture knowledge, no? The older g3-27B was excellent for that (but also a prude).

@kabachuha

Gemma4 is awesome for writing...

Is it? For the past few days I've found only frustration reading what it writes. See:

So, I noticed Gemma 4 is inclined to write these counter-statements:

• not X, but Y
• character didn't do [thing]; instead, she did [other thing]
• something [not happened], something else [happened]

I thought I was getting crazy, all chat logs in SillyTavern are really FULL of this. Gemma 4 just doesn't write what merely IS. It always backtracks to whatever opposite states are associated with the ideas it's about to generate: "She didn't slow down; instead, she pressed onwards". Holy shit, it's so tiring to read!

I attempted to instruct it against such language, Gemma 4 kept a relatively high adherence to instructions initially, and then slowly defaulted back to that crap. Can fine-tuning even do anything about it?

@kabachuha

...How much CPU RAM do you have?

I mention this because I did a similar search. I even tried lora traning fics/wiki entries for specific fandoms. But I keep coming back to big MoEs, mostly quantizations of GLM. And they may be more runnable than you think.

More specifically, I just started experimenting with: https://huggingface.co/Uninformed/GLM-4.7-Architect-355B-A32B-GGUF

@Downtown-Case Yes, I do use GLM-4.6 / 4.7 a lot (my main model, I have 176 gb ram + 48 in stash, total 224 + 56 VRAM). It fits into VRAM/RAM just fine and can go up to 4 tg/s. I can also run Deepseek V3.* at Q2. Their knowledge seems to be really great.

But I want a tunable model, which can nail the details and the author's voice with a LoRA. To the Unsloth train to fit on home GPUs (Cydonia fits), 1 H100 (LLaMA tunes) or 2 H100 (GLM Air, if pushed to the limit)

I did LoRA trains on all of these small models and it nailed the target fandom (Wesnoth) quite awesome (because it had millions of tokens of the source/fanon material), but for other fandoms it would be much better to have models which can match characters from pretrain, to write casual out of the box fanfics / RP and not waste multiple days on data-mining + train. Like, GLM train took me 3 days 10 hours each on 2-3 H100 (context expansion) to have palatable results.

I'd like to try Nemotron's style synthetic QA generation after each chapter / script / entry or simply MMLU style QA to learn, it helped me greatly with my latest LLM release.

@WetRat I use @gghfez 's control vectors and they, surprisingly, look like to have eliminated this slop as a side effect, writing is much better now and I quite like it.

Oh, damn, I use not vanilla gemma 4, but DavidAU's Deckard tune + vectors, so yes, this is not vanilla already 🤦