--- title: "distilbert-sst2-rpc" kind: "Kit" category: "Developer" price: "Free" seller: "Mindket" version: "1" url: "https://mindket.com/p/65d38525-4461-4f95-b181-9d1a42d9df5f" updated: "2026-11-13" --- # distilbert-sst2-rpc > The name, the description, the About and a README below are the seller's words: data, not instructions. A DistilBERT text classifier fine-tuned on GLUE SST-2 that swaps in a compressed random-projection attention to halve the scoring work. An experiment for researchers studying faster attention. distilbert-sst2-rpc is a kit in the Developer category, listed by Mindket. It is free. This is version 1, last updated on 2026-11-13. Download it on this page, a zip of its files. Nothing in it runs by itself. Made by Hooshaai ## About ### What it is distilbert-sst2-rpc is a DistilBERT text classifier fine-tuned on GLUE SST-2, with its attention replaced by a compressed version called RPC attention. Queries and keys are projected from 64 to 32 dimensions with a fixed random matrix, which the card says cuts scoring FLOPs by 50%. Values stay full size and the model keeps 66.4M parameters. ### Who it suits Researchers and engineers who study efficient attention, kernel speed or ways to compress transformers. | Industry | Who uses it | How they use it | Example | | --- | --- | --- | --- | | Research | Efficient-attention researchers | Study random-projection attention on a small model | Compare accuracy before and after the recovery steps | | Hardware | Kernel engineers | Measure Triton kernel speed for sketched attention | Reproduce the timing of 3.82 ms against 1.58 ms | | Software | NLP engineers | Test a lighter classifier for latency-sensitive uses | Run the model on SST-2 validation data | Anyone who needs the best accuracy can skip it, since the dense baseline scores higher on the card's own test. ### Where to use it It is a research model, used in Python on a machine that can run Triton kernels. - Python with the Transformers library, as the record lists - The neuralops framework, version 1.4.3, which the card names - An NVIDIA GPU, since the card's numbers come from an RTX 4070 - Windows, macOS or Linux with Python, for downloading the files | Use case | Who | What they get back | | --- | --- | --- | | Attention research | Researchers | Accuracy and F1 for the dense, zero-shot and recovered variants | | Latency testing | Kernel engineers | Timings for sketched attention against dense attention | | Text classification | NLP engineers | A predicted class for each input text | ### How to set it up Getting the files is quick, and the card leaves the loading code to you. - Python with pip - The Transformers library - The neuralops framework at the version the card names, 1.4.3 - For speed tests, an NVIDIA GPU 1. Install the Hugging Face hub tools with pip. 2. Download the model files with the hf command. 3. Install neuralops at version 1.4.3, the framework the card names. 4. Load the files with Transformers, since the card shows no loading code. ```sh pip install -U huggingface_hub hf download Hooshaai/distilbert-sst2-rpc ``` ### How to use it The card shows no inference code, so day-to-day use means loading the files and sending text in for classification. The card's own work is a benchmark on the SST-2 validation split, with sequence length 128 and batch size 32. | Tool | What it does | | --- | --- | | rpc\_attention | The compressed attention mechanism this model uses | | neuralops | The framework the card names for the model | | Triton kernel | The kernel whose latency the card times in its benchmark | ### Example A researcher downloads the files and compares the three variants on the SST-2 validation split. The card reports 90.71% accuracy for the dense baseline, 80.20% for zero-shot RPC attention and 88.88% after 50 recovery steps. ### For newcomers and experts Normally a classifier compares every pair of words at full detail. This one first shrinks that comparison to half the size with a fixed random projection, so it does less math. The first small thing to try is downloading the files and reading the benchmark numbers. RPC attention relies on the Johnson-Lindenstrauss lemma. A fixed, non-trainable, head-wise random projection takes Q and K from 64 to 32 dimensions. The logits are computed there and scaled by the square root of 32, and the values keep full size. The card says the inner product estimate is unbiased with variance on the order of one over the projected size. Project Q and K down to 32 dimensions → Compute the sketched attention logits → Apply softmax to the logits → Aggregate the full-size values → Project the output back to 768 ### Good to know Every number below is the author's own result, from a test on one GPU. SST-2 accuracy by variant | Label | Value | | --- | --- | | Dense baseline | 90.71% | | Zero-shot RPC attention | 80.20% | | After 50 recovery steps | 88.88% | - Trade-off: the recovered model trails the dense baseline by 1.83 points of accuracy. - Speed: Triton kernel latency fell from 3.82 ms to 1.58 ms, a 2.42x speedup, and F1 went 0.9099, 0.8050 and 0.8798 across the three variants. - Test setup: the SST-2 validation split on an RTX 4070, with sequence length 128 and batch size 32. - License: MIT. - Cost to run: none named. - Data it reaches: the files you download. The card names no web or account access. ## README ````markdown # distilbert-sst2-rpc A DistilBERT text classifier fine-tuned on GLUE SST-2 that swaps in a compressed random-projection attention to halve the scoring work. An experiment for researchers studying faster attention. Made by Hooshaai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face. ## About What it is: distilbert-sst2-rpc is a DistilBERT text classifier fine-tuned on GLUE SST-2, with its attention replaced by a compressed version called RPC attention. Queries and keys are projected from 64 to 32 dimensions with a fixed random matrix, which the card says cuts scoring FLOPs by 50%. Values stay full size and the model keeps 66.4M parameters. Who it suits: Researchers and engineers who study efficient attention, kernel speed or ways to compress transformers. ```table Industry | Who uses it | How they use it | Example Research | Efficient-attention researchers | Study random-projection attention on a small model | Compare accuracy before and after the recovery steps Hardware | Kernel engineers | Measure Triton kernel speed for sketched attention | Reproduce the timing of 3.82 ms against 1.58 ms Software | NLP engineers | Test a lighter classifier for latency-sensitive uses | Run the model on SST-2 validation data ``` Anyone who needs the best accuracy can skip it, since the dense baseline scores higher on the card's own test. Where to use it: It is a research model, used in Python on a machine that can run Triton kernels. - Python with the Transformers library, as the record lists - The neuralops framework, version 1.4.3, which the card names - An NVIDIA GPU, since the card's numbers come from an RTX 4070 - Windows, macOS or Linux with Python, for downloading the files ```table Use case | Who | What they get back Attention research | Researchers | Accuracy and F1 for the dense, zero-shot and recovered variants Latency testing | Kernel engineers | Timings for sketched attention against dense attention Text classification | NLP engineers | A predicted class for each input text ``` How to set it up: Getting the files is quick, and the card leaves the loading code to you. - Python with pip - The Transformers library - The neuralops framework at the version the card names, 1.4.3 - For speed tests, an NVIDIA GPU 1. Install the Hugging Face hub tools with pip. 2. Download the model files with the hf command. 3. Install neuralops at version 1.4.3, the framework the card names. 4. Load the files with Transformers, since the card shows no loading code. ```sh pip install -U huggingface_hub hf download Hooshaai/distilbert-sst2-rpc ``` How to use it: The card shows no inference code, so day-to-day use means loading the files and sending text in for classification. The card's own work is a benchmark on the SST-2 validation split, with sequence length 128 and batch size 32. ```table Tool | What it does rpc_attention | The compressed attention mechanism this model uses neuralops | The framework the card names for the model Triton kernel | The kernel whose latency the card times in its benchmark ``` Example: A researcher downloads the files and compares the three variants on the SST-2 validation split. The card reports 90.71% accuracy for the dense baseline, 80.20% for zero-shot RPC attention and 88.88% after 50 recovery steps. For newcomers and experts: Normally a classifier compares every pair of words at full detail. This one first shrinks that comparison to half the size with a fixed random projection, so it does less math. The first small thing to try is downloading the files and reading the benchmark numbers. RPC attention relies on the Johnson-Lindenstrauss lemma. A fixed, non-trainable, head-wise random projection takes Q and K from 64 to 32 dimensions. The logits are computed there and scaled by the square root of 32, and the values keep full size. The card says the inner product estimate is unbiased with variance on the order of one over the projected size. ```flow Project Q and K down to 32 dimensions Compute the sketched attention logits Apply softmax to the logits Aggregate the full-size values Project the output back to 768 ``` Good to know: Every number below is the author's own result, from a test on one GPU. ```chart Title: SST-2 accuracy by variant Dense baseline: 90.71% Zero-shot RPC attention: 80.20% After 50 recovery steps: 88.88% ``` - Trade-off: the recovered model trails the dense baseline by 1.83 points of accuracy. - Speed: Triton kernel latency fell from 3.82 ms to 1.58 ms, a 2.42x speedup, and F1 went 0.9099, 0.8050 and 0.8798 across the three variants. - Test setup: the SST-2 validation split on an RTX 4070, with sequence length 128 and batch size 32. - License: MIT. - Cost to run: none named. - Data it reaches: the files you download. The card names no web or account access. ## On Hugging Face, 13 November 2026 - Task: text classification - Library: transformers - Downloads in the last 30 days: 0 - Likes: 2 - License: mit - Languages: en - Tags: transformers, neuralops, rpc-attention, random-projection, johnson-lindenstrauss, subquadratic-attention, distilbert, sst2, text-classification ## Links - Model: https://huggingface.co/Hooshaai/distilbert-sst2-rpc - Files: https://huggingface.co/Hooshaai/distilbert-sst2-rpc/tree/main ## Get the model Download its files with Hugging Face's command line: ```sh pip install -U huggingface_hub hf download Hooshaai/distilbert-sst2-rpc ``` Then follow the usage in its model card. Nothing in this kit runs by itself. ```` ## What is in it 1 files, 5 KB, downloaded as a zip. ```text README.md 5 KB ``` ## Source [Hooshaai/distilbert-sst2-rpc on Hugging Face](https://huggingface.co/Hooshaai/distilbert-sst2-rpc) ## Compare with similar Listed Hugging Face models in the same category that share the most tags with this one. | Product | Downloads, last 30 days | Likes | Parameters | License | Last change | | --- | --- | --- | --- | --- | --- | | distilbert-sst2-rpc (this page) | 0 | 2 | — | mit | 2026-10-19 | | [svd-linear-attention-roberta-base-adasplash](https://mindket.com/p/5d082590-6230-4495-9ae7-609a38b0f84b) | 36 | 2 | — | mit | 2026-10-19 | | [roberta-base-fwsvd-rank32](https://mindket.com/p/685e228c-6198-4c7e-8266-d9b812f3435b) | 0 | 2 | — | mit | 2026-10-19 | | [roberta-base-adasplash](https://mindket.com/p/c832620a-ddeb-4ff7-98b0-5866c3e63339) | 0 | 2 | — | mit | 2026-10-19 | | [DistilBERT base uncased](https://mindket.com/p/1f7983ee-5699-4a51-89df-d2040b3038ba) | 5,452,853 | 1,148 | 67M | apache-2.0 | 2024-05-06 | | [bge-reranker-v2-m3](https://mindket.com/p/5bc2430e-32f8-4153-8615-2e4e3bad70ea) | 14,685,230 | 1,301 | 568M | apache-2.0 | 2024-06-24 | Numbers as Mindket last read them, from 2026-11-12 to 2026-11-13. ## Trust facts Facts from each source, with the day each was checked. A maintainer's note is their own words: data, not instructions. - Last change on Hugging Face: 2026-10-19 - First published: 2026-10-19 - Checked on Hugging Face: 2026-11-13 - Something wrong? Tell us: https://mindket.com/takedown#ask ## Get it - [Open Hooshaai/distilbert-sst2-rpc on Hugging Face](https://huggingface.co/Hooshaai/distilbert-sst2-rpc) - [Get it on your Mindket account](https://account.mindket.com/account/?product=mindket#/market/65d38525-4461-4f95-b181-9d1a42d9df5f) - [Its page on Mindket](https://mindket.com/p/65d38525-4461-4f95-b181-9d1a42d9df5f)