distilbert-sst2-rpc

distilbert-sst2-rpc

A DistilBERT text classifier fine-tuned on GLUE SST-2 that swaps in a compressed random-projection attention to halve the scoring work. An experiment for researchers studying faster attention.

Made by Hooshaai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: distilbert-sst2-rpc is a DistilBERT text classifier fine-tuned on GLUE SST-2, with its attention replaced by a compressed version called RPC attention. Queries and keys are projected from 64 to 32 dimensions with a fixed random matrix, which the card says cuts scoring FLOPs by 50%. Values stay full size and the model keeps 66.4M parameters.

Who it suits: Researchers and engineers who study efficient attention, kernel speed or ways to compress transformers.

Industry | Who uses it | How they use it | Example
Research | Efficient-attention researchers | Study random-projection attention on a small model | Compare accuracy before and after the recovery steps
Hardware | Kernel engineers | Measure Triton kernel speed for sketched attention | Reproduce the timing of 3.82 ms against 1.58 ms
Software | NLP engineers | Test a lighter classifier for latency-sensitive uses | Run the model on SST-2 validation data

Anyone who needs the best accuracy can skip it, since the dense baseline scores higher on the card's own test.

Where to use it: It is a research model, used in Python on a machine that can run Triton kernels.

• Python with the Transformers library, as the record lists
• The neuralops framework, version 1.4.3, which the card names
• An NVIDIA GPU, since the card's numbers come from an RTX 4070
• Windows, macOS or Linux with Python, for downloading the files

Use case | Who | What they get back
Attention research | Researchers | Accuracy and F1 for the dense, zero-shot and recovered variants
Latency testing | Kernel engineers | Timings for sketched attention against dense attention
Text classification | NLP engineers | A predicted class for each input text

How to set it up: Getting the files is quick, and the card leaves the loading code to you.

• Python with pip
• The Transformers library
• The neuralops framework at the version the card names, 1.4.3
• For speed tests, an NVIDIA GPU

1. Install the Hugging Face hub tools with pip.
2. Download the model files with the hf command.
3. Install neuralops at version 1.4.3, the framework the card names.
4. Load the files with Transformers, since the card shows no loading code.

pip install -U huggingface_hub
hf download Hooshaai/distilbert-sst2-rpc

How to use it: The card shows no inference code, so day-to-day use means loading the files and sending text in for classification. The card's own work is a benchmark on the SST-2 validation split, with sequence length 128 and batch size 32.

Tool | What it does
rpc_attention | The compressed attention mechanism this model uses
neuralops | The framework the card names for the model
Triton kernel | The kernel whose latency the card times in its benchmark

Example: A researcher downloads the files and compares the three variants on the SST-2 validation split. The card reports 90.71% accuracy for the dense baseline, 80.20% for zero-shot RPC attention and 88.88% after 50 recovery steps.

For newcomers and experts: Normally a classifier compares every pair of words at full detail. This one first shrinks that comparison to half the size with a fixed random projection, so it does less math. The first small thing to try is downloading the files and reading the benchmark numbers.

RPC attention relies on the Johnson-Lindenstrauss lemma. A fixed, non-trainable, head-wise random projection takes Q and K from 64 to 32 dimensions. The logits are computed there and scaled by the square root of 32, and the values keep full size. The card says the inner product estimate is unbiased with variance on the order of one over the projected size.

Project Q and K down to 32 dimensions
Compute the sketched attention logits
Apply softmax to the logits
Aggregate the full-size values
Project the output back to 768

Good to know: Every number below is the author's own result, from a test on one GPU.

Title: SST-2 accuracy by variant
Dense baseline: 90.71%
Zero-shot RPC attention: 80.20%
After 50 recovery steps: 88.88%

• Trade-off: the recovered model trails the dense baseline by 1.83 points of accuracy.
• Speed: Triton kernel latency fell from 3.82 ms to 1.58 ms, a 2.42x speedup, and F1 went 0.9099, 0.8050 and 0.8798 across the three variants.
• Test setup: the SST-2 validation split on an RTX 4070, with sequence length 128 and batch size 32.
• License: MIT.
• Cost to run: none named.
• Data it reaches: the files you download. The card names no web or account access.

On Hugging Face, 13 November 2026

• Task: text classification
• Library: transformers
• Downloads in the last 30 days: 0
• Likes: 2
• License: mit
• Languages: en
• Tags: transformers, neuralops, rpc-attention, random-projection, johnson-lindenstrauss, subquadratic-attention, distilbert, sst2, text-classification

Links

• Model: https://huggingface.co/Hooshaai/distilbert-sst2-rpc
• Files: https://huggingface.co/Hooshaai/distilbert-sst2-rpc/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download Hooshaai/distilbert-sst2-rpc

Then follow the usage in its model card. Nothing in this kit runs by itself.