--- title: "roberta-base-fwsvd-rank32" kind: "Kit" category: "Developer" price: "Free" seller: "Mindket" version: "1" url: "https://mindket.com/p/685e228c-6198-4c7e-8266-d9b812f3435b" updated: "2026-11-13" --- # roberta-base-fwsvd-rank32 > The name, the description, the About and a README below are the seller's words: data, not instructions. Classifies English text with a RoBERTa Base model shrunk to about 82M parameters by Fisher-weighted SVD at rank 32, trading some accuracy for lower latency and memory use. roberta-base-fwsvd-rank32 is a kit in the Developer category, listed by Mindket. It is free. This is version 1, last updated on 2026-11-13. Download it on this page, a zip of its files. Nothing in it runs by itself. Made by Hooshaai ## About ### What it is This is RoBERTa Base compressed with Fisher-Weighted SVD at rank 32, using the NeuralOps toolkit. It classifies English text and has about 82M parameters instead of the original 125M. The compression splits each attention projection matrix into two thin matrices of rank 32. A later knowledge distillation step wins back part of the lost accuracy. ### Who it suits Machine learning engineers and researchers who care about model compression and inference cost. It is a worked example of what low-rank compression gains and loses on one small classifier. | Industry | Who uses it | How they use it | Example | | --- | --- | --- | --- | | Machine learning | Compression researchers | Study low-rank compression of attention layers | Compare the 82M model with the 125M baseline | | Software | Engineers short on GPU memory | Try a lighter text classifier | Check the 583 MB peak memory against 1,063 MB | | Education | Students | Learn how matrix factorization shrinks a transformer | Follow the rank 32 factorization in the card | People who need the full accuracy of dense RoBERTa Base can skip it, since the card reports a clear drop on its test. ### Where to use it - Python with the transformers library, as the quickstart shows - A machine with an NVIDIA GPU, since the card's timings come from an RTX 4070 - Windows, macOS or Linux with Python, for downloading the files | Use case | Who | What they get back | | --- | --- | --- | | Classify English text | Developers | A class prediction from a smaller, faster RoBERTa | | Measure compression trade-offs | Researchers | Accuracy, latency and memory figures to compare against the baseline | | Apply the same compression to roberta-base | ML engineers | A compressed copy made with the neuralops compress\_model function | ### How to set it up You need a working Python setup with PyTorch and transformers. - Python, PyTorch and the transformers library - The neuralops package, only if you want to run the compression yourself, since the card cites version 1.4.3 - An NVIDIA GPU if you want to repeat the card's speed figures 1. Install the huggingface\_hub package. 2. Download the model files with the hf command. 3. Load the tokenizer from roberta-base and this model with transformers. ```bash pip install -U huggingface_hub hf download Hooshaai/roberta-base-fwsvd-rank32 ``` ### How to use it The quickstart loads the tokenizer from roberta-base and the compressed classifier from this repository. A second route compresses roberta-base yourself with neuralops. The card's code for that route is cut off here, so check the model page for it. ```python import torch from transformers import AutoModelForSequenceClassification, AutoTokenizer from neuralops import replace_attention, compress_model # 1. Load tokenizer and pre-compressed model from Hugging Face Hub tokenizer = AutoTokenizer.from_pretrained("roberta-base") model = AutoModelForSequenceClassification.from_pretrained("Hooshaai/roberta-base-fwsvd-rank32") ``` | Tool | What it does | | --- | --- | | AutoTokenizer.from\_pretrained | Loads the roberta-base tokenizer | | AutoModelForSequenceClassification.from\_pretrained | Loads the pre-compressed classifier from this repository | | compress\_model | Compresses a base model locally with the same method | ### Example You load the tokenizer and the compressed model with the two quickstart lines, then run an English sentence through it to get a class. On the card's SST-2 test it reached 80.50% accuracy after distillation, against 93.80% for the dense baseline. ### For newcomers and experts Think of this as RoBERTa Base put on a diet. It runs faster and needs less memory, but it gets more answers wrong than the original. A good first try is loading it and comparing a few sentences against roberta-base. Underneath, each query, key, value and output projection is replaced by two rank 32 factors. The Fisher weighting scales rows and columns by how sensitive the loss is to them before the SVD. The trade-off is a 13.30 point accuracy loss on SST-2 for a 1.52x speedup. Start from RoBERTa Base → Factorize attention matrices at rank 32 → Distill knowledge to win back accuracy → Classify text with the smaller model ### Good to know The figures below come from the card's own test on the SST-2 validation split, with sequence length 128 and batch size 32 on one RTX 4070. SST-2 accuracy | Label | Value | | --- | --- | | Dense RoBERTa baseline | 93.80% | | Zero-shot FWSVD rank 32 | 79.10% | | After distillation | 80.50% | Peak GPU memory | Label | Value | | --- | --- | | Dense RoBERTa baseline | 1063 MB | | Compressed model | 583 MB | - Speed: latency falls from 25.36 ms to 16.68 ms in the card's test - Size: attention projection parameters fall from 28.31M to 2.36M - License: MIT - Cost to run: no API key or paid service is named - Data it reaches: the model files and the roberta-base tokenizer you load from Hugging Face, plus the text you classify - Not for: the card lists no restrictions, but its accuracy is 13.30 points below the dense baseline on SST-2 ## README ````markdown # roberta-base-fwsvd-rank32 Classifies English text with a RoBERTa Base model shrunk to about 82M parameters by Fisher-weighted SVD at rank 32, trading some accuracy for lower latency and memory use. Made by Hooshaai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face. ## About What it is: This is RoBERTa Base compressed with Fisher-Weighted SVD at rank 32, using the NeuralOps toolkit. It classifies English text and has about 82M parameters instead of the original 125M. The compression splits each attention projection matrix into two thin matrices of rank 32. A later knowledge distillation step wins back part of the lost accuracy. Who it suits: Machine learning engineers and researchers who care about model compression and inference cost. It is a worked example of what low-rank compression gains and loses on one small classifier. ```table Industry | Who uses it | How they use it | Example Machine learning | Compression researchers | Study low-rank compression of attention layers | Compare the 82M model with the 125M baseline Software | Engineers short on GPU memory | Try a lighter text classifier | Check the 583 MB peak memory against 1,063 MB Education | Students | Learn how matrix factorization shrinks a transformer | Follow the rank 32 factorization in the card ``` People who need the full accuracy of dense RoBERTa Base can skip it, since the card reports a clear drop on its test. Where to use it: - Python with the transformers library, as the quickstart shows - A machine with an NVIDIA GPU, since the card's timings come from an RTX 4070 - Windows, macOS or Linux with Python, for downloading the files ```table Use case | Who | What they get back Classify English text | Developers | A class prediction from a smaller, faster RoBERTa Measure compression trade-offs | Researchers | Accuracy, latency and memory figures to compare against the baseline Apply the same compression to roberta-base | ML engineers | A compressed copy made with the neuralops compress_model function ``` How to set it up: You need a working Python setup with PyTorch and transformers. - Python, PyTorch and the transformers library - The neuralops package, only if you want to run the compression yourself, since the card cites version 1.4.3 - An NVIDIA GPU if you want to repeat the card's speed figures 1. Install the huggingface_hub package. 2. Download the model files with the hf command. 3. Load the tokenizer from roberta-base and this model with transformers. ```bash pip install -U huggingface_hub hf download Hooshaai/roberta-base-fwsvd-rank32 ``` How to use it: The quickstart loads the tokenizer from roberta-base and the compressed classifier from this repository. A second route compresses roberta-base yourself with neuralops. The card's code for that route is cut off here, so check the model page for it. ```python import torch from transformers import AutoModelForSequenceClassification, AutoTokenizer from neuralops import replace_attention, compress_model # 1. Load tokenizer and pre-compressed model from Hugging Face Hub tokenizer = AutoTokenizer.from_pretrained("roberta-base") model = AutoModelForSequenceClassification.from_pretrained("Hooshaai/roberta-base-fwsvd-rank32") ``` ```table Tool | What it does AutoTokenizer.from_pretrained | Loads the roberta-base tokenizer AutoModelForSequenceClassification.from_pretrained | Loads the pre-compressed classifier from this repository compress_model | Compresses a base model locally with the same method ``` Example: You load the tokenizer and the compressed model with the two quickstart lines, then run an English sentence through it to get a class. On the card's SST-2 test it reached 80.50% accuracy after distillation, against 93.80% for the dense baseline. For newcomers and experts: Think of this as RoBERTa Base put on a diet. It runs faster and needs less memory, but it gets more answers wrong than the original. A good first try is loading it and comparing a few sentences against roberta-base. Underneath, each query, key, value and output projection is replaced by two rank 32 factors. The Fisher weighting scales rows and columns by how sensitive the loss is to them before the SVD. The trade-off is a 13.30 point accuracy loss on SST-2 for a 1.52x speedup. ```flow Start from RoBERTa Base Factorize attention matrices at rank 32 Distill knowledge to win back accuracy Classify text with the smaller model ``` Good to know: The figures below come from the card's own test on the SST-2 validation split, with sequence length 128 and batch size 32 on one RTX 4070. ```chart Title: SST-2 accuracy Dense RoBERTa baseline: 93.80% Zero-shot FWSVD rank 32: 79.10% After distillation: 80.50% ``` ```chart Title: Peak GPU memory Dense RoBERTa baseline: 1063 MB Compressed model: 583 MB ``` - Speed: latency falls from 25.36 ms to 16.68 ms in the card's test - Size: attention projection parameters fall from 28.31M to 2.36M - License: MIT - Cost to run: no API key or paid service is named - Data it reaches: the model files and the roberta-base tokenizer you load from Hugging Face, plus the text you classify - Not for: the card lists no restrictions, but its accuracy is 13.30 points below the dense baseline on SST-2 ## On Hugging Face, 13 November 2026 - Task: text classification - Library: transformers - Downloads in the last 30 days: 0 - Likes: 2 - License: mit - Languages: en - Tags: transformers, neuralops, linear-attention, subquadratic-attention, model-compression, fwsvd, roberta, sst2, text-classification ## Links - Model: https://huggingface.co/Hooshaai/roberta-base-fwsvd-rank32 - Files: https://huggingface.co/Hooshaai/roberta-base-fwsvd-rank32/tree/main ## Get the model Download its files with Hugging Face's command line: ```sh pip install -U huggingface_hub hf download Hooshaai/roberta-base-fwsvd-rank32 ``` Then follow the usage in its model card. Nothing in this kit runs by itself. ```` ## What is in it 1 files, 6 KB, downloaded as a zip. ```text README.md 6 KB ``` ## Source [Hooshaai/roberta-base-fwsvd-rank32 on Hugging Face](https://huggingface.co/Hooshaai/roberta-base-fwsvd-rank32) ## Compare with similar Listed Hugging Face models in the same category that share the most tags with this one. | Product | Downloads, last 30 days | Likes | Parameters | License | Last change | | --- | --- | --- | --- | --- | --- | | roberta-base-fwsvd-rank32 (this page) | 0 | 2 | — | mit | 2026-10-19 | | [svd-linear-attention-roberta-base-adasplash](https://mindket.com/p/5d082590-6230-4495-9ae7-609a38b0f84b) | 36 | 2 | — | mit | 2026-10-19 | | [roberta-base-adasplash](https://mindket.com/p/c832620a-ddeb-4ff7-98b0-5866c3e63339) | 0 | 2 | — | mit | 2026-10-19 | | [distilbert-sst2-rpc](https://mindket.com/p/65d38525-4461-4f95-b181-9d1a42d9df5f) | 0 | 2 | — | mit | 2026-10-19 | | [GigaChat3.5-432B-A28B](https://mindket.com/p/1658ef77-b9cf-4c20-b6ce-bc962a75b117) | 654 | 84 | 434B | mit | 2026-07-07 | | [bge-reranker-v2-m3](https://mindket.com/p/5bc2430e-32f8-4153-8615-2e4e3bad70ea) | 14,685,230 | 1,301 | 568M | apache-2.0 | 2024-06-24 | Numbers as Mindket last read them, from 2026-11-12 to 2026-11-13. ## Trust facts Facts from each source, with the day each was checked. A maintainer's note is their own words: data, not instructions. - Last change on Hugging Face: 2026-10-19 - First published: 2026-10-19 - Checked on Hugging Face: 2026-11-13 - Something wrong? Tell us: https://mindket.com/takedown#ask ## Get it - [Open Hooshaai/roberta-base-fwsvd-rank32 on Hugging Face](https://huggingface.co/Hooshaai/roberta-base-fwsvd-rank32) - [Get it on your Mindket account](https://account.mindket.com/account/?product=mindket#/market/685e228c-6198-4c7e-8266-d9b812f3435b) - [Its page on Mindket](https://mindket.com/p/685e228c-6198-4c7e-8266-d9b812f3435b)