roberta-base-adasplash

roberta-base-adasplash

A RoBERTa Base sentiment classifier whose attention is swapped for a sparse AdaSplash version, which zeroes out 40% of attention weights to cut time and memory. For developers testing efficient attention.

Made by Hooshaai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This is RoBERTa Base, fine-tuned on the GLUE SST-2 sentiment task, with its attention layers replaced by AdaSplash attention from the neuralops library. AdaSplash uses a sparse projection that sets a share of attention weights to exactly zero.

The card reports 40% zero attention weights in all 12 encoder blocks. It adds about 0.94% more parameters through small LoRA adapters, bringing the model to roughly 125.8 million parameters.

Who it suits: Developers and researchers who want to see how sparse attention changes speed, memory and accuracy in a familiar English text classifier. It is an experiment more than a drop-in upgrade.

Industry | Who uses it | How they use it | Example
Research | ML researchers | Study sparse attention in an encoder model | Comparing sparse and dense attention on one task
Software | Inference engineers | Test memory and latency savings | Measuring peak memory for a classifier
Education | Students | Learn how entmax attention differs from softmax | Reading the architecture notes and running the sample
Product teams | NLP developers | Try an efficient sentiment classifier | Scoring short English reviews as positive or negative

Anyone who needs the highest sentiment accuracy can skip it. The card shows the dense baseline scoring higher on its test.

Where to use it:

• Python with Transformers, PyTorch and the neuralops library.
• Windows, macOS or Linux with Python, for downloading the files.
• A GPU, since the card's measurements used an NVIDIA RTX 4070 with Triton kernels.

Use case | Who | What they get back
Sentiment classification | Developers | A positive or negative prediction for English text
Efficiency testing | Engineers | Lower attention time and memory than the dense model
Attention swapping | Researchers | An existing RoBERTa converted to AdaSplash in one call

How to set it up: You need Python and a few libraries.

• Python with torch and transformers
• The neuralops library, version 1.4.3 per the card
• English text to classify
• A GPU, which is optional for trying the sample

1. Install the Hugging Face tool.
2. Download the model files.
3. Install neuralops and run the quickstart.

pip install -U huggingface_hub
hf download Hooshaai/roberta-base-adasplash

How to use it: Load the tokenizer and the model, tokenize a sentence, and take the highest logit as the prediction. You can also convert any RoBERTa to AdaSplash in one call.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from neuralops import replace_attention

# 1. Load tokenizer and model from Hugging Face Hub
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModelForSequenceClassification.from_pretrained("Hooshaai/roberta-base-adasplash")

# 2. Or surgically inject AdaSplash into any RoBERTa in 1 line
base_model = AutoModelForSequenceClassification.from_pretrained("roberta-base")
adasplash_model = replace_attention(base_model, "adasplash_attention")

# 3. Predict sentiment
inputs = tokenizer("The sparse entmax projection preserves salient features with extreme precision!", return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
    prediction = logits.argmax(-1).item()

Tool | What it does
AutoTokenizer | Turns a sentence into model inputs
AutoModelForSequenceClassification | Loads the classifier and returns logits
replace_attention | Swaps a model's attention for AdaSplash attention

Example: Give the model a short English sentence that expresses an opinion, as the sample does. It returns logits, and the index of the largest one is the predicted sentiment class.

For newcomers and experts: If you are new, a sentiment classifier reads a sentence and says whether it sounds positive or negative. Run the quickstart on one sentence first. The rest of the card is about how its attention was made more efficient.

For experts, the model uses Tsallis 1.5-entmax in place of softmax, so attention weights can be exactly zero. The query, key, value and output projections are rank-16 LoRA adapters over frozen pretrained weights. The card reports 94.38% accuracy for the dense baseline, 75.46% for zero-shot AdaSplash and 82.10% after distillation.

Start from RoBERTa Base fine-tuned on SST-2
Swap in AdaSplash attention with entmax
Add rank-16 LoRA adapters
Distill to recover accuracy
Classify text with 40% sparse attention

Title: SST-2 accuracy from the card
Dense baseline: 94.38%
Zero-shot AdaSplash: 75.46%
After distillation: 82.10%

Good to know: The points below cover limits and costs.

• Accuracy trade-off: the distilled model scores lower than the dense baseline, though the card reports about 1.52 times faster inference steps and 33% less peak memory on its test machine.
• License: MIT.
• Cost to run: no API key or paid service is named.
• Data it reads: only the text you pass in. Downloads come from Hugging Face.
• Not for: uses that need the dense model's accuracy, or languages other than English.

On Hugging Face, 13 November 2026

• Task: text classification
• Library: transformers
• Downloads in the last 30 days: 0
• Likes: 2
• License: mit
• Languages: en
• Tags: transformers, neuralops, adasplash, entmax, sparse-attention, subquadratic-attention, roberta, sst2, text-classification

Links

• Model: https://huggingface.co/Hooshaai/roberta-base-adasplash
• Files: https://huggingface.co/Hooshaai/roberta-base-adasplash/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download Hooshaai/roberta-base-adasplash

Then follow the usage in its model card. Nothing in this kit runs by itself.