roberta-base-fwsvd-rank32

roberta-base-fwsvd-rank32

Classifies English text with a RoBERTa Base model shrunk to about 82M parameters by Fisher-weighted SVD at rank 32, trading some accuracy for lower latency and memory use.

Made by Hooshaai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This is RoBERTa Base compressed with Fisher-Weighted SVD at rank 32, using the NeuralOps toolkit. It classifies English text and has about 82M parameters instead of the original 125M.

The compression splits each attention projection matrix into two thin matrices of rank 32. A later knowledge distillation step wins back part of the lost accuracy.

Who it suits: Machine learning engineers and researchers who care about model compression and inference cost. It is a worked example of what low-rank compression gains and loses on one small classifier.

Industry | Who uses it | How they use it | Example
Machine learning | Compression researchers | Study low-rank compression of attention layers | Compare the 82M model with the 125M baseline
Software | Engineers short on GPU memory | Try a lighter text classifier | Check the 583 MB peak memory against 1,063 MB
Education | Students | Learn how matrix factorization shrinks a transformer | Follow the rank 32 factorization in the card

People who need the full accuracy of dense RoBERTa Base can skip it, since the card reports a clear drop on its test.

Where to use it:

• Python with the transformers library, as the quickstart shows
• A machine with an NVIDIA GPU, since the card's timings come from an RTX 4070
• Windows, macOS or Linux with Python, for downloading the files

Use case | Who | What they get back
Classify English text | Developers | A class prediction from a smaller, faster RoBERTa
Measure compression trade-offs | Researchers | Accuracy, latency and memory figures to compare against the baseline
Apply the same compression to roberta-base | ML engineers | A compressed copy made with the neuralops compress_model function

How to set it up: You need a working Python setup with PyTorch and transformers.

• Python, PyTorch and the transformers library
• The neuralops package, only if you want to run the compression yourself, since the card cites version 1.4.3
• An NVIDIA GPU if you want to repeat the card's speed figures

1. Install the huggingface_hub package.
2. Download the model files with the hf command.
3. Load the tokenizer from roberta-base and this model with transformers.

pip install -U huggingface_hub
hf download Hooshaai/roberta-base-fwsvd-rank32

How to use it: The quickstart loads the tokenizer from roberta-base and the compressed classifier from this repository. A second route compresses roberta-base yourself with neuralops. The card's code for that route is cut off here, so check the model page for it.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from neuralops import replace_attention, compress_model

# 1. Load tokenizer and pre-compressed model from Hugging Face Hub
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModelForSequenceClassification.from_pretrained("Hooshaai/roberta-base-fwsvd-rank32")

Tool | What it does
AutoTokenizer.from_pretrained | Loads the roberta-base tokenizer
AutoModelForSequenceClassification.from_pretrained | Loads the pre-compressed classifier from this repository
compress_model | Compresses a base model locally with the same method

Example: You load the tokenizer and the compressed model with the two quickstart lines, then run an English sentence through it to get a class. On the card's SST-2 test it reached 80.50% accuracy after distillation, against 93.80% for the dense baseline.

For newcomers and experts: Think of this as RoBERTa Base put on a diet. It runs faster and needs less memory, but it gets more answers wrong than the original. A good first try is loading it and comparing a few sentences against roberta-base.

Underneath, each query, key, value and output projection is replaced by two rank 32 factors. The Fisher weighting scales rows and columns by how sensitive the loss is to them before the SVD. The trade-off is a 13.30 point accuracy loss on SST-2 for a 1.52x speedup.

Start from RoBERTa Base
Factorize attention matrices at rank 32
Distill knowledge to win back accuracy
Classify text with the smaller model

Good to know: The figures below come from the card's own test on the SST-2 validation split, with sequence length 128 and batch size 32 on one RTX 4070.

Title: SST-2 accuracy
Dense RoBERTa baseline: 93.80%
Zero-shot FWSVD rank 32: 79.10%
After distillation: 80.50%

Title: Peak GPU memory
Dense RoBERTa baseline: 1063 MB
Compressed model: 583 MB

• Speed: latency falls from 25.36 ms to 16.68 ms in the card's test
• Size: attention projection parameters fall from 28.31M to 2.36M
• License: MIT
• Cost to run: no API key or paid service is named
• Data it reaches: the model files and the roberta-base tokenizer you load from Hugging Face, plus the text you classify
• Not for: the card lists no restrictions, but its accuracy is 13.30 points below the dense baseline on SST-2

On Hugging Face, 13 November 2026

• Task: text classification
• Library: transformers
• Downloads in the last 30 days: 0
• Likes: 2
• License: mit
• Languages: en
• Tags: transformers, neuralops, linear-attention, subquadratic-attention, model-compression, fwsvd, roberta, sst2, text-classification

Links

• Model: https://huggingface.co/Hooshaai/roberta-base-fwsvd-rank32
• Files: https://huggingface.co/Hooshaai/roberta-base-fwsvd-rank32/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download Hooshaai/roberta-base-fwsvd-rank32

Then follow the usage in its model card. Nothing in this kit runs by itself.