AsteroRank-Lepton-GGUF

AsteroRank-Lepton-GGUF

Ranks search results by relevance with a reranker for STEM and general search, shipped as GGUF files for llama.cpp. For developers and researchers who want local ranking, non-commercial use only.

Made by Asteromorph on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: These are GGUF versions of AsteroRank-Lepton, a reranker for STEM and general search. A reranker takes a query and a list of passages and sorts the passages by how well they answer it. The files run in llama.cpp, so the ranking happens on your own machine.

Who it suits: Developers who build search or retrieval pipelines, and researchers who need technical passages sorted for a question.

Industry | Who uses it | How they use it | Example
Software | Search engineers | Re-sort retrieved passages before showing results | Candidate documents sorted for a user query
Science | Researchers | Rank abstracts or paper excerpts for a technical question | Chemistry passages sorted for a borylation query
AI agents | Agent builders | Pass a short context line along with the query | An agent that ranks sources for the article it is writing

Skip it if you need commercial use, because the license is non-commercial.

Where to use it:

• llama.cpp: the llama-server program, on Windows, macOS or Linux
• Python scripts: the rerank.py client that comes with the files
• GPU or CPU: the card shows a flag that offloads every layer to a GPU

Use case | Who | What they get back
Rank passages for a query | Developers | A list sorted by relevance score
Rank with extra context | Agent builders | Scores that take a short context text into account
Limit the results | Search engineers | Only the top few passages, using the top_n setting

How to set it up: You need a current llama.cpp build that supports Qwen3.5 and includes llama-server. A stock build works, with no fork or patch. You also need Python 3.8 or newer with jinja2.

• llama.cpp with llama-server
• Python 3.8 or newer and the jinja2 package
• Disk space for one quant, about 3.1 GB for Q4_K_M

1. Install the Hugging Face hub tool.
2. Download one quantization together with the client script.
3. Start llama-server on that file.

pip install -U huggingface_hub

Put your own value in its key setting first.

How to use it: Import the client class from rerank.py and point it at your running llama-server. Call its rerank method with a query and a list of documents, and add a context text if you have one.

from rerank import GGUFReranker

Each result holds the document, a relevance score between 0 and 1, and its original index, sorted from most to least relevant.

Tool | What it does
GGUFReranker | Client class in rerank.py that scores passages through the server
rerank | Takes a query, documents and optional context, and returns sorted results
top_n | Limits how many results come back
-ngl 99 | Offloads all layers of the model to the GPU
-np | Sets how many requests run in parallel

Example: You ask it to rank three short chemistry passages for a query about para-selective borylation of aryl sulfonate ion pairs, adding one line of context about the article. You get the three passages back as a list ordered by score, each with its index.

For newcomers and experts: If you are new to this, think of it as a sorting step for search. Download the Q4_K_M file and the client, start the server, and rank three short passages for one question.

If you know the field, the score is the gap between the model's yes and no logits at the end of a prompt that is embedded in the file as a chat template. The server's built-in rerank endpoint takes no context, so rerank.py scores through the completion endpoint instead and reads next-token log-probabilities.

Download one quant and rerank.py
Start llama-server on the file
Send a query, documents and context
Read scores sorted high to low

Good to know: Quants below BF16 were made with an importance matrix file that ships in the repository. Smaller quants trade quality for size.

Title: File size by quantization in GB
BF16: 9.7 GB
Q8_0: 5.2 GB
Q6_K: 4.0 GB
Q5_K_M: 3.5 GB
Q4_K_M: 3.1 GB
Q3_K_M: 2.5 GB
Q2_K: 2.1 GB
IQ2_XXS: 1.7 GB

• License: CC BY-NC 4.0, shared with the original reranker. Its base model, Qwen3.5-4B, is Apache 2.0
• Cost to run: none named
• Data it reaches: the query, documents and context you send to a llama-server you run yourself
• Not for: commercial use, which the license does not allow

On Hugging Face, 14 November 2026

• Task: text ranking
• Library: llama.cpp
• Parameters: 4.8B
• Weights: about 2.4 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 626
• Likes: 3
• License: cc-by-nc-4.0
• Languages: en
• Based on: Asteromorph/AsteroRank-Lepton
• Tags: llama.cpp, gguf, reranker, qwen3.5, llama-cpp, text-ranking, conversational

Links

• Model: https://huggingface.co/Asteromorph/AsteroRank-Lepton-GGUF
• Files: https://huggingface.co/Asteromorph/AsteroRank-Lepton-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download Asteromorph/AsteroRank-Lepton-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.