qwen-2.5-0.5b-block-transformer-hiba-wikitext2
tahamajs · Kit
Swaps 18 of the 24 attention layers in Qwen2.5-0.5B for a hierarchical block attention operator, then distills it on WikiText-2. A research checkpoint for stud…
New
Kits of AI agent skills, rules and prompts
Categories and filters
Listings
qwen-2.5-0.5b-bam-umoe-wikitext2
tahamajs · Kit
Swaps 18 of the 24 attention layers in Qwen2.5-0.5B for a mixture-of-experts attention block, then distills it on WikiText-2. A research checkpoint for studyin…
New