--- title: "svd-linear-attention-roberta-base-adasplash" kind: "Kit" category: "Developer" price: "Free" seller: "Mindket" version: "1" url: "https://mindket.com/p/5d082590-6230-4495-9ae7-609a38b0f84b" updated: "2026-11-13" --- # svd-linear-attention-roberta-base-adasplash > The name, the description, the About and a README below are the seller's words: data, not instructions. Classifies English text, such as SST-2 sentiment, with a RoBERTa Base model whose attention is made sparse by AdaSplash, trading some accuracy for lower latency and memory use. svd-linear-attention-roberta-base-adasplash is a kit in the Developer category, listed by Mindket. It is free. This is version 1, last updated on 2026-11-13. Download it on this page, a zip of its files. Nothing in it runs by itself. Made by Hooshaai ## About ### What it is This model starts from a RoBERTa Base that was tuned for the GLUE SST-2 task. Its softmax attention is replaced by AdaSplash attention from the NeuralOps toolkit, which uses a Tsallis 1.5-entmax projection, so 40% of attention weights are exactly zero. Small rank-16 LoRA adapters sit in the projections and add 0.94% to the parameter count, which comes to 125.8M. A distillation step then recovers part of the accuracy lost in the swap. ### Who it suits Machine learning researchers and engineers who want to see what sparse attention does to a small text classifier. Students of attention mechanisms can follow the card's worked numbers. | Industry | Who uses it | How they use it | Example | | --- | --- | --- | --- | | Machine learning | Researchers | Study sparse entmax attention in an encoder model | Compare the 40% sparse model with dense RoBERTa | | Software | Engineers watching GPU memory | Try a classifier with lower peak memory | Check 712 MB against the 1,063 MB baseline | | Education | Students | Learn how entmax replaces softmax in attention | Read the card's projection steps | People who need the baseline's accuracy can skip it, since the distilled model scores about 12 points lower on SST-2. ### Where to use it - Python with the transformers library, as the quickstart shows - A machine with an NVIDIA GPU, since the card's timings come from an RTX 4070 - Windows, macOS or Linux with Python, for downloading the files | Use case | Who | What they get back | | --- | --- | --- | | Predict sentiment of English text | Developers | A class prediction from a sparse attention RoBERTa | | Measure sparse attention trade-offs | Researchers | Accuracy, latency and memory figures to compare with the baseline | | Add AdaSplash to another RoBERTa | ML engineers | A model with AdaSplash attention injected through replace\_attention | ### How to set it up You need a working Python setup with PyTorch and transformers. - Python, PyTorch and the transformers library - The neuralops package, since the quickstart imports it and the card cites version 1.4.3 - An NVIDIA GPU if you want to repeat the card's speed figures 1. Install the huggingface\_hub package. 2. Download the model files with the hf command. 3. Load the tokenizer from roberta-base and the classifier with transformers. ```bash pip install -U huggingface_hub hf download Hooshaai/svd-linear-attention-roberta-base-adasplash ``` ### How to use it The quickstart loads the tokenizer from roberta-base and the classifier from the Hub. It then shows how to inject AdaSplash into any RoBERTa with one call and how to start a sentiment prediction. Note that its code names a repository called roberta-base-adasplash, so check that name on the model page before you run it. The card's code is cut off after the prediction setup. ```python import torch from transformers import AutoModelForSequenceClassification, AutoTokenizer from neuralops import replace_attention # 1. Load tokenizer and model from Hugging Face Hub tokenizer = AutoTokenizer.from_pretrained("roberta-base") model = AutoModelForSequenceClassification.from_pretrained("Hooshaai/roberta-base-adasplash") # 2. Or surgically inject AdaSplash into any RoBERTa in 1 line base_model = AutoModelForSequenceClassification.from_pretrained("roberta-base") adasplash_model = replace_attention(base_model, "adasplash_attention") # 3. Predict sentiment inputs = tokenizer("The sparse entmax projection preserves salient features with extreme precision!", return_tensors="pt") ``` | Tool | What it does | | --- | --- | | AutoTokenizer.from\_pretrained | Loads the roberta-base tokenizer | | AutoModelForSequenceClassification.from\_pretrained | Loads a RoBERTa classifier from the Hub | | replace\_attention | Injects AdaSplash attention into a RoBERTa model | ### Example You tokenize a short sentence such as "The film was a delight from start to finish." and pass it to the model without gradients to predict its sentiment. On the card's SST-2 test the distilled model reached 82.10% accuracy, against 94.38% for the dense baseline. ### For newcomers and experts Think of this as a RoBERTa that looks at fewer words at once. Instead of giving every word a little weight, it sets some weights to exactly zero. A good first try is loading it and comparing a few sentences against roberta-base. Underneath, 1.5-entmax projects each row of attention scores onto the simplex boundary, which forces exact zeros. The projection is found by a bisection search. The encoder keeps its full receptive field, and frozen pre-trained weights get additive LoRA paths. The trade-off is lower accuracy for less latency and memory. Start from RoBERTa Base on SST-2 → Replace softmax with 1.5-entmax attention → Add rank-16 LoRA adapters → Distill to win back accuracy → Classify text with sparse attention ### Good to know The figures below come from the card's own test on the SST-2 validation split, with sequence length 128 and batch size 32 on one RTX 4070. SST-2 accuracy | Label | Value | | --- | --- | | Dense RoBERTa baseline | 94.38% | | Zero-shot AdaSplash | 75.46% | | After distillation | 82.10% | Peak GPU memory | Label | Value | | --- | --- | | Dense RoBERTa baseline | 1063 MB | | AdaSplash model | 712 MB | - Speed: inference step time falls from 25.36 ms to 16.68 ms, and the Triton kernel from 4.10 ms to 1.54 ms - Sparsity: 40% of attention entries are exact zeros in every encoder block - License: MIT - Cost to run: no API key or paid service is named - Data it reaches: the model files and the roberta-base tokenizer you load from Hugging Face, plus the text you classify - Not for: the card lists no restrictions, but its accuracy is about 12 points below the dense baseline on SST-2 ## README ````markdown # svd-linear-attention-roberta-base-adasplash Classifies English text, such as SST-2 sentiment, with a RoBERTa Base model whose attention is made sparse by AdaSplash, trading some accuracy for lower latency and memory use. Made by Hooshaai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face. ## About What it is: This model starts from a RoBERTa Base that was tuned for the GLUE SST-2 task. Its softmax attention is replaced by AdaSplash attention from the NeuralOps toolkit, which uses a Tsallis 1.5-entmax projection, so 40% of attention weights are exactly zero. Small rank-16 LoRA adapters sit in the projections and add 0.94% to the parameter count, which comes to 125.8M. A distillation step then recovers part of the accuracy lost in the swap. Who it suits: Machine learning researchers and engineers who want to see what sparse attention does to a small text classifier. Students of attention mechanisms can follow the card's worked numbers. ```table Industry | Who uses it | How they use it | Example Machine learning | Researchers | Study sparse entmax attention in an encoder model | Compare the 40% sparse model with dense RoBERTa Software | Engineers watching GPU memory | Try a classifier with lower peak memory | Check 712 MB against the 1,063 MB baseline Education | Students | Learn how entmax replaces softmax in attention | Read the card's projection steps ``` People who need the baseline's accuracy can skip it, since the distilled model scores about 12 points lower on SST-2. Where to use it: - Python with the transformers library, as the quickstart shows - A machine with an NVIDIA GPU, since the card's timings come from an RTX 4070 - Windows, macOS or Linux with Python, for downloading the files ```table Use case | Who | What they get back Predict sentiment of English text | Developers | A class prediction from a sparse attention RoBERTa Measure sparse attention trade-offs | Researchers | Accuracy, latency and memory figures to compare with the baseline Add AdaSplash to another RoBERTa | ML engineers | A model with AdaSplash attention injected through replace_attention ``` How to set it up: You need a working Python setup with PyTorch and transformers. - Python, PyTorch and the transformers library - The neuralops package, since the quickstart imports it and the card cites version 1.4.3 - An NVIDIA GPU if you want to repeat the card's speed figures 1. Install the huggingface_hub package. 2. Download the model files with the hf command. 3. Load the tokenizer from roberta-base and the classifier with transformers. ```bash pip install -U huggingface_hub hf download Hooshaai/svd-linear-attention-roberta-base-adasplash ``` How to use it: The quickstart loads the tokenizer from roberta-base and the classifier from the Hub. It then shows how to inject AdaSplash into any RoBERTa with one call and how to start a sentiment prediction. Note that its code names a repository called roberta-base-adasplash, so check that name on the model page before you run it. The card's code is cut off after the prediction setup. ```python import torch from transformers import AutoModelForSequenceClassification, AutoTokenizer from neuralops import replace_attention # 1. Load tokenizer and model from Hugging Face Hub tokenizer = AutoTokenizer.from_pretrained("roberta-base") model = AutoModelForSequenceClassification.from_pretrained("Hooshaai/roberta-base-adasplash") # 2. Or surgically inject AdaSplash into any RoBERTa in 1 line base_model = AutoModelForSequenceClassification.from_pretrained("roberta-base") adasplash_model = replace_attention(base_model, "adasplash_attention") # 3. Predict sentiment inputs = tokenizer("The sparse entmax projection preserves salient features with extreme precision!", return_tensors="pt") ``` ```table Tool | What it does AutoTokenizer.from_pretrained | Loads the roberta-base tokenizer AutoModelForSequenceClassification.from_pretrained | Loads a RoBERTa classifier from the Hub replace_attention | Injects AdaSplash attention into a RoBERTa model ``` Example: You tokenize a short sentence such as "The film was a delight from start to finish." and pass it to the model without gradients to predict its sentiment. On the card's SST-2 test the distilled model reached 82.10% accuracy, against 94.38% for the dense baseline. For newcomers and experts: Think of this as a RoBERTa that looks at fewer words at once. Instead of giving every word a little weight, it sets some weights to exactly zero. A good first try is loading it and comparing a few sentences against roberta-base. Underneath, 1.5-entmax projects each row of attention scores onto the simplex boundary, which forces exact zeros. The projection is found by a bisection search. The encoder keeps its full receptive field, and frozen pre-trained weights get additive LoRA paths. The trade-off is lower accuracy for less latency and memory. ```flow Start from RoBERTa Base on SST-2 Replace softmax with 1.5-entmax attention Add rank-16 LoRA adapters Distill to win back accuracy Classify text with sparse attention ``` Good to know: The figures below come from the card's own test on the SST-2 validation split, with sequence length 128 and batch size 32 on one RTX 4070. ```chart Title: SST-2 accuracy Dense RoBERTa baseline: 94.38% Zero-shot AdaSplash: 75.46% After distillation: 82.10% ``` ```chart Title: Peak GPU memory Dense RoBERTa baseline: 1063 MB AdaSplash model: 712 MB ``` - Speed: inference step time falls from 25.36 ms to 16.68 ms, and the Triton kernel from 4.10 ms to 1.54 ms - Sparsity: 40% of attention entries are exact zeros in every encoder block - License: MIT - Cost to run: no API key or paid service is named - Data it reaches: the model files and the roberta-base tokenizer you load from Hugging Face, plus the text you classify - Not for: the card lists no restrictions, but its accuracy is about 12 points below the dense baseline on SST-2 ## On Hugging Face, 13 November 2026 - Task: text classification - Library: transformers - Downloads in the last 30 days: 36 - Likes: 2 - License: mit - Languages: en - Tags: transformers, roberta, fill-mask, neuralops, adasplash, entmax, sparse-attention, subquadratic-attention, sst2, text-classification, text-embeddings-inference ## Links - Model: https://huggingface.co/Hooshaai/svd-linear-attention-roberta-base-adasplash - Files: https://huggingface.co/Hooshaai/svd-linear-attention-roberta-base-adasplash/tree/main ## Get the model Download its files with Hugging Face's command line: ```sh pip install -U huggingface_hub hf download Hooshaai/svd-linear-attention-roberta-base-adasplash ``` Then follow the usage in its model card. Nothing in this kit runs by itself. ```` ## What is in it 1 files, 7 KB, downloaded as a zip. ```text README.md 7 KB ``` ## Source [Hooshaai/svd-linear-attention-roberta-base-adasplash on Hugging Face](https://huggingface.co/Hooshaai/svd-linear-attention-roberta-base-adasplash) ## Compare with similar Listed Hugging Face models in the same category that share the most tags with this one. | Product | Downloads, last 30 days | Likes | Parameters | License | Last change | | --- | --- | --- | --- | --- | --- | | svd-linear-attention-roberta-base-adasplash (this page) | 36 | 2 | — | mit | 2026-10-19 | | [roberta-base-adasplash](https://mindket.com/p/c832620a-ddeb-4ff7-98b0-5866c3e63339) | 0 | 2 | — | mit | 2026-10-19 | | [roberta-base-fwsvd-rank32](https://mindket.com/p/685e228c-6198-4c7e-8266-d9b812f3435b) | 0 | 2 | — | mit | 2026-10-19 | | [distilbert-sst2-rpc](https://mindket.com/p/65d38525-4461-4f95-b181-9d1a42d9df5f) | 0 | 2 | — | mit | 2026-10-19 | | [bge-reranker-v2-m3](https://mindket.com/p/5bc2430e-32f8-4153-8615-2e4e3bad70ea) | 14,685,230 | 1,301 | 568M | apache-2.0 | 2024-06-24 | | [deberta-v3-base-prompt-injection-v2](https://mindket.com/p/bbb131a8-e7e6-491b-b19e-25098c98a48d) | 476,298 | 129 | 184M | apache-2.0 | 2026-07-09 | Numbers as Mindket last read them, from 2026-11-12 to 2026-11-13. ## Trust facts Facts from each source, with the day each was checked. A maintainer's note is their own words: data, not instructions. - Last change on Hugging Face: 2026-10-19 - First published: 2026-09-18 - Checked on Hugging Face: 2026-11-13 - Something wrong? Tell us: https://mindket.com/takedown#ask ## Get it - [Open Hooshaai/svd-linear-attention-roberta-base-adasplash on Hugging Face](https://huggingface.co/Hooshaai/svd-linear-attention-roberta-base-adasplash) - [Get it on your Mindket account](https://account.mindket.com/account/?product=mindket#/market/5d082590-6230-4495-9ae7-609a38b0f84b) - [Its page on Mindket](https://mindket.com/p/5d082590-6230-4495-9ae7-609a38b0f84b)