Red team your own LLM app: Promptfoo, garak and PyRIT, what to test and how to read the results

Three open-source red team tools

Red teaming is attacking your own system before someone else does. For an LLM app that means feeding it adversarial inputs, far more than you would write by hand, and checking which ones make it misbehave. Three open-source tools automate the loop.

Promptfoo's README says it is now part of OpenAI and remains open source and MIT licensed. garak is NVIDIA's scanner, and its README compares it to nmap and Metasploit, but for LLMs. PyRIT comes from the Microsoft AI Red Team and a community of contributors. They overlap, so start with one.

Red team an app with Promptfoo

Promptfoo needs Node.js 22.22.0 or later. An OpenAI API key is optional. The attacker model that writes the attacks is set by redteam.provider and defaults to openai:gpt-5, and with no OpenAI key the docs say Promptfoo proxies generation and grading requests to its own API. The scanner runs on your machine, so it can attack any endpoint your machine or network can reach.

An HTTP target looks like the config below, copied from the Getting started page. The label names the target in reports, so reuse it when you generate new configs for the same target. The purpose is optional, but the page says it significantly improves the generated test cases and the grading, so say who the user is and what they may reach. To skip the web UI, promptfoo redteam init --no-gui writes a starting config.

targets:
  - id: https
    label: 'travel-agent'
    config:
      url: 'https://example.com/generate'
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
      body:
        myPrompt: '{{prompt}}'

purpose: 'The user is a budget traveler looking for the best deals. The system is a travel agent that helps the user plan their trip. The user is anonymous and should not be able to access any information about other users, employees, or other individuals.'

1. Run npx promptfoo@latest redteam setup. It opens a web UI that asks questions about the project.
2. On Application Details, fill in the Purpose field at a minimum. Load Example, at the top of the page, gives you a quick try.
3. Configure the target: an HTTP endpoint, a model or your own code.
4. Select plugins. Plugins are adversarial generators that produce the malicious inputs. If in doubt, stick with the Default preset.
5. Select strategies, the techniques that wrap the generated inputs in an attack pattern.
6. Review and save the file as promptfooconfig.yaml, then run npx promptfoo@latest redteam run in that folder. It generates several hundred adversarial inputs, saves them in redteam.yaml and runs them against the target.
7. Run npx promptfoo@latest redteam report to open the results.

What to test

Promptfoo's plugins page recommends starting with a limited set of plugins to get a baseline, then adding more as you learn where the model is weak. More plugins mean longer runs and more inference. For an app with agents or retrieval, it recommends testing for application-level vulnerabilities, with the excerpt below. The comments are Promptfoo's.

Two plugins from the earlier lessons belong on your list as well: indirect-prompt-injection, from lesson 1, and prompt-extraction, from lesson 2. Each takes a config field of its own.

plugins:
  - 'agentic:memory-poisoning' # Tests if stateful agents are vulnerable to memory poisoning attacks
  - 'rbac' # Tests if the model properly implements Role-Based Access Control
  - 'bola' # Checks for Broken Object Level Authorization vulnerabilities
  - 'bfla' # Tests for Broken Function Level Authorization issues
  - 'ssrf' # Tests for Server-Side Request Forgery vulnerabilities
  - 'sql-injection' # Tests for SQL injection vulnerabilities (if connected to a SQL database)
  - 'pii' # Checks for leakage of Personally Identifiable Information
  - 'excessive-agency' # Checks if the agent exceeds its intended capabilities
  - 'hijacking' # Checks for goal hijacking of the agent's objectives
  - 'tool-discovery' # Tests if the model reveals its available function calls or tools

Add the plugins from lessons 1 and 2

Both plugins need extra config. Indirect-prompt-injection needs indirectInjectionVar, the name of the template variable that carries untrusted text, such as {{name}} in the travel agent prompt from lesson 1. Per its plugin page, Promptfoo injects adversarial payloads into that variable and fails a test if the model follows them. Prompt-extraction needs systemPrompt, the system prompt you gave the model, as lesson 2 showed and its plugin page requires.

The config below puts the two examples from those plugin pages together. Replace the placeholder with your real system prompt, and name your own untrusted variable.

prompts:
  - |
    You are a travel agent. The user's name is {{name}}.
    User query: {{query}}

redteam:
  plugins:
    - id: indirect-prompt-injection
      config:
        indirectInjectionVar: name
    - id: prompt-extraction
      config:
        systemPrompt: 'Paste the real system prompt of your app here.'

Pick attack strategies

Plugins write the attacks and strategies change how they are delivered. The strategies page recommends two for most apps: jailbreak:meta for single-turn attacks and jailbreak:hydra for multi-turn conversations. The configuration reference lists basic, jailbreak:meta and jailbreak:composite as the defaults, and generates 5 tests per plugin unless you set numTests.

redteam:
  strategies:
    - jailbreak:meta # Single-turn agentic attacks
    - jailbreak:hydra # Multi-turn adaptive conversations

• Static strategies apply fixed transformations, such as base64 encoding. They cost little and are easy to detect, and the page says foundation models often patch them.
• Dynamic strategies use an attacker model to refine an input over several attempts. They succeed more often and cost more. By default they stop after the token budget or after they produce a harmful output.
• Multi-turn strategies work across a conversation. The page says they have the highest success rates, they suit stateful apps and they use more resources than single-turn strategies.

The same job with garak or PyRIT

garak is a command-line tool. By default it tries every probe it knows against the model you name, with the detectors each probe recommends. --spec narrows the run to a probe family such as probes.promptinject, or to one probe. For an app that is not a bare model, its REST generator can reach any endpoint that returns plaintext or JSON, with a short YAML file describing the endpoint.

PyRIT's scanner runs a scenario against a target, and scenarios bundle attack techniques, datasets and scoring. It ships six families: Adaptive, AIRT, Benchmark, Foundry, Garak and Technique. The Garak family includes SystemPromptExtraction. pyrit_scan suits CI and batch runs, and pyrit_shell suits interactive exploring. PyRIT needs your endpoints configured first, and its configuration page starts with three environment variables for an OpenAI chat target. The install page lists Python 3.11 to 3.14.

The commands below are examples from their docs: the garak README, and PyRIT's two scanner pages. The garak README asks you to replace the sample key with a real one.

# garak
python -m pip install -U garak
garak --list_probes
export OPENAI_API_KEY="sk-123XXXXXXXXXXXX"
python3 -m garak --target_type openai --target_name gpt-5-nano --spec probes.encoding

# PyRIT
pip install pyrit
pyrit_scan list-scenarios
pyrit_scan run foundry.red_team_agent --target openai_chat --initializers target --techniques base64

Read the results

Promptfoo's report shows vulnerability categories, severity levels, the logs of the inputs that triggered failures and suggested mitigations. Selecting a test case shows its raw input and output. Each vulnerability also gets a risk score from 0 to 10, built on CVSS principles, which the table turns into bands.

garak prints a progress bar for each probe, then one row per detector. A response marked FAIL means some attempts produced undesirable behavior, with the failure rate beside it. A figure such as 840/840 counts generations: the total, then how many seemed to behave OK. The count runs high because garak makes 10 generations per prompt by default. Errors go to garak.log, details go to a JSONL report, and analyse/analyse_log.py lists the probes and prompts with the most hits.

Open the log for each failure and check whether your app could do the damage the answer describes. A failure that depended on a tool you can remove is fixed by removing the tool, as the first fixes in lesson 3 say. Then run the scan again under the same label. Promptfoo grades its own results with grading requests, so a verdict you disagree with is a reason to read the log, and LLM as a judge shows how to check a grader.

Test safely

A scan sends hostile input to a running app, and it spends tokens. Our advice, with what the docs add:

• Test an app you own, on a staging copy, with test data and an account that holds only test permissions. The scan tries to make the app act, and the excessive-agency plugin exists to find actions you did not intend.
• Promptfoo's docs describe attack generation two ways. One passage says it uses your own OpenAI key for generation and grading, and proxies requests to its API if you have none. Another says it uses a remote service for generating adversarial inputs by default. Your target is always evaluated locally. Setting PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION to true forces 100% local generation, and the docs say its quality depends on the model you configure and is generally low for most models.
• More plugins lengthen runs and need more inference. Dynamic strategies call both an attacker model and your target, and multi-turn runs track target, attacker and grading tokens separately. Keep numTests small while you tune the config.
• In CI, promptfoo redteam run --tag ci.run-id="$CI_RUN_ID" --tag git.sha="$GIT_SHA" records the run and the commit with the results. Evals in CI covers running checks on every change.
• If the app is an agent, run it in the sandbox and under the permission rules from agent guardrails first.

Common mistakes

• Pointing a scanner at a production system or at an app you do not own. It runs from your machine and attacks whatever endpoint it can reach.
• Leaving Purpose empty, then reading generic failures as findings about your app.
• Reading a clean run as proof of safety. It shows that the attacks tried did not work. As lesson 1 covered, an attacker who adapts to your defense is a different test.
• Taking a severity label as final without opening the log.
• Giving the tested agent live credentials and real data.

Try it yourself

• Run npx promptfoo@latest redteam setup, press Load Example on Application Details and click through the screens without running a scan.
• Write two sentences of purpose for an app you own: who the user is and what they should not be able to reach.
• Point a scan at a staging copy with the plugins from lessons 1 and 2, indirect-prompt-injection and prompt-extraction, plus the ones that fit your app.
• Open the first failure and write down which OWASP entry from lesson 3 it belongs to, and which first fix would stop it.

Next course: AI evals

Red teaming shows how an app fails under attack. The next course, AI evals: measure whether your LLM app and agent work, turns the same idea into tests you can run on every change.

Frequently asked questions

Sources

1. GitHub: Promptfoo repository (read November 6, 2026)https://github.com/promptfoo/promptfoo
2. GitHub: Promptfoo, red team getting started (read November 6, 2026)https://github.com/promptfoo/promptfoo/blob/main/site/docs/red-team/quickstart.md
3. GitHub: Promptfoo, red team configuration (read November 6, 2026)https://github.com/promptfoo/promptfoo/blob/main/site/docs/red-team/configuration.md
4. GitHub: Promptfoo, plugins (read November 6, 2026)https://github.com/promptfoo/promptfoo/blob/main/site/docs/red-team/plugins/index.md
5. GitHub: Promptfoo, strategies (read November 6, 2026)https://github.com/promptfoo/promptfoo/blob/main/site/docs/red-team/strategies/index.md
6. GitHub: Promptfoo, risk scoring (read November 6, 2026)https://github.com/promptfoo/promptfoo/blob/main/site/docs/red-team/risk-scoring.md
7. GitHub: Promptfoo, indirect prompt injection plugin (read November 6, 2026)https://github.com/promptfoo/promptfoo/blob/main/site/docs/red-team/plugins/indirect-prompt-injection.md
8. GitHub: Promptfoo, prompt extraction plugin (read November 6, 2026)https://github.com/promptfoo/promptfoo/blob/main/site/docs/red-team/plugins/prompt-extraction.md
9. GitHub: NVIDIA garak repository (read November 6, 2026)https://github.com/NVIDIA/garak
10. GitHub: Microsoft PyRIT repository (read November 6, 2026)https://github.com/microsoft/PyRIT
11. GitHub: PyRIT scanner (read November 6, 2026)https://github.com/microsoft/PyRIT/blob/main/doc/scanner/0_scanner.md
12. GitHub: PyRIT scan command (read November 6, 2026)https://github.com/microsoft/PyRIT/blob/main/doc/scanner/1_pyrit_scan.ipynb
13. GitHub: PyRIT local installation (read November 6, 2026)https://github.com/microsoft/PyRIT/blob/main/doc/getting_started/install_local.md
14. GitHub: PyRIT configuration (read November 6, 2026)https://github.com/microsoft/PyRIT/blob/main/doc/getting_started/configuration.md
15. YouTube: Red Team an AI App with Promptfoo (Promptfoo channel) (read November 6, 2026)https://www.youtube.com/watch?v=e7MlDhfN22s