qwen-2.5-0.5b-block-transformer-hiba-wikitext2
tahamajs · Kit
Swaps 18 of the 24 attention layers in Qwen2.5-0.5B for a hierarchical block attention operator, then distills it on WikiText-2. A research checkpoint for stud…
New
tahamajs · Kit
Swaps 18 of the 24 attention layers in Qwen2.5-0.5B for a hierarchical block attention operator, then distills it on WikiText-2. A research checkpoint for stud…