svd-linear-attention-roberta-base-adasplash

svd-linear-attention-roberta-base-adasplash

Classifies English text, such as SST-2 sentiment, with a RoBERTa Base model whose attention is made sparse by AdaSplash, trading some accuracy for lower latency and memory use.

Made by Hooshaai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This model starts from a RoBERTa Base that was tuned for the GLUE SST-2 task. Its softmax attention is replaced by AdaSplash attention from the NeuralOps toolkit, which uses a Tsallis 1.5-entmax projection, so 40% of attention weights are exactly zero.

Small rank-16 LoRA adapters sit in the projections and add 0.94% to the parameter count, which comes to 125.8M. A distillation step then recovers part of the accuracy lost in the swap.

Who it suits: Machine learning researchers and engineers who want to see what sparse attention does to a small text classifier. Students of attention mechanisms can follow the card's worked numbers.

Industry | Who uses it | How they use it | Example
Machine learning | Researchers | Study sparse entmax attention in an encoder model | Compare the 40% sparse model with dense RoBERTa
Software | Engineers watching GPU memory | Try a classifier with lower peak memory | Check 712 MB against the 1,063 MB baseline
Education | Students | Learn how entmax replaces softmax in attention | Read the card's projection steps

People who need the baseline's accuracy can skip it, since the distilled model scores about 12 points lower on SST-2.

Where to use it:

• Python with the transformers library, as the quickstart shows
• A machine with an NVIDIA GPU, since the card's timings come from an RTX 4070
• Windows, macOS or Linux with Python, for downloading the files

Use case | Who | What they get back
Predict sentiment of English text | Developers | A class prediction from a sparse attention RoBERTa
Measure sparse attention trade-offs | Researchers | Accuracy, latency and memory figures to compare with the baseline
Add AdaSplash to another RoBERTa | ML engineers | A model with AdaSplash attention injected through replace_attention

How to set it up: You need a working Python setup with PyTorch and transformers.

• Python, PyTorch and the transformers library
• The neuralops package, since the quickstart imports it and the card cites version 1.4.3
• An NVIDIA GPU if you want to repeat the card's speed figures

1. Install the huggingface_hub package.
2. Download the model files with the hf command.
3. Load the tokenizer from roberta-base and the classifier with transformers.

pip install -U huggingface_hub
hf download Hooshaai/svd-linear-attention-roberta-base-adasplash

How to use it: The quickstart loads the tokenizer from roberta-base and the classifier from the Hub. It then shows how to inject AdaSplash into any RoBERTa with one call and how to start a sentiment prediction. Note that its code names a repository called roberta-base-adasplash, so check that name on the model page before you run it. The card's code is cut off after the prediction setup.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from neuralops import replace_attention

# 1. Load tokenizer and model from Hugging Face Hub
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModelForSequenceClassification.from_pretrained("Hooshaai/roberta-base-adasplash")

# 2. Or surgically inject AdaSplash into any RoBERTa in 1 line
base_model = AutoModelForSequenceClassification.from_pretrained("roberta-base")
adasplash_model = replace_attention(base_model, "adasplash_attention")

# 3. Predict sentiment
inputs = tokenizer("The sparse entmax projection preserves salient features with extreme precision!", return_tensors="pt")

Tool | What it does
AutoTokenizer.from_pretrained | Loads the roberta-base tokenizer
AutoModelForSequenceClassification.from_pretrained | Loads a RoBERTa classifier from the Hub
replace_attention | Injects AdaSplash attention into a RoBERTa model

Example: You tokenize a short sentence such as "The film was a delight from start to finish." and pass it to the model without gradients to predict its sentiment. On the card's SST-2 test the distilled model reached 82.10% accuracy, against 94.38% for the dense baseline.

For newcomers and experts: Think of this as a RoBERTa that looks at fewer words at once. Instead of giving every word a little weight, it sets some weights to exactly zero. A good first try is loading it and comparing a few sentences against roberta-base.

Underneath, 1.5-entmax projects each row of attention scores onto the simplex boundary, which forces exact zeros. The projection is found by a bisection search. The encoder keeps its full receptive field, and frozen pre-trained weights get additive LoRA paths. The trade-off is lower accuracy for less latency and memory.

Start from RoBERTa Base on SST-2
Replace softmax with 1.5-entmax attention
Add rank-16 LoRA adapters
Distill to win back accuracy
Classify text with sparse attention

Good to know: The figures below come from the card's own test on the SST-2 validation split, with sequence length 128 and batch size 32 on one RTX 4070.

Title: SST-2 accuracy
Dense RoBERTa baseline: 94.38%
Zero-shot AdaSplash: 75.46%
After distillation: 82.10%

Title: Peak GPU memory
Dense RoBERTa baseline: 1063 MB
AdaSplash model: 712 MB

• Speed: inference step time falls from 25.36 ms to 16.68 ms, and the Triton kernel from 4.10 ms to 1.54 ms
• Sparsity: 40% of attention entries are exact zeros in every encoder block
• License: MIT
• Cost to run: no API key or paid service is named
• Data it reaches: the model files and the roberta-base tokenizer you load from Hugging Face, plus the text you classify
• Not for: the card lists no restrictions, but its accuracy is about 12 points below the dense baseline on SST-2

On Hugging Face, 13 November 2026

• Task: text classification
• Library: transformers
• Downloads in the last 30 days: 36
• Likes: 2
• License: mit
• Languages: en
• Tags: transformers, roberta, fill-mask, neuralops, adasplash, entmax, sparse-attention, subquadratic-attention, sst2, text-classification, text-embeddings-inference

Links

• Model: https://huggingface.co/Hooshaai/svd-linear-attention-roberta-base-adasplash
• Files: https://huggingface.co/Hooshaai/svd-linear-attention-roberta-base-adasplash/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download Hooshaai/svd-linear-attention-roberta-base-adasplash

Then follow the usage in its model card. Nothing in this kit runs by itself.