Návrh Školení
Module 1 — How AI Apps Break
Lab: none — architecture walkthrough & discussion
A builder’s mental model of the attack surface.
Topics:
- LLM, RAG, and agent architectures from the developer’s side
- The request/response lifecycle of an AI feature
- Prompt flow: system, developer, user, and tool messages
- Where untrusted data enters (and re-enters) the model
- The trust boundaries a developer owns vs. inherits
- Why AI attacks are semantic, not syntactic
- Mapping the OWASP LLM Top 10 to code you write
Key insight: Every place untrusted text reaches the model — or model output reaches your code — is a boundary you own.
Module 2 — Prompt Injection for Builders
Lab: Lab 01 — 01-Prompt-Injection
The “SQL injection moment” for AI — but you can’t fully escape it.
Topics:
- Direct vs. indirect prompt injection
- Hidden instructions in documents, web pages, tool output
- Jailbreaks and role-confusion
- Why instruction/data separation matters
- Defensive prompt design (delimiters, structure, minimal authority)
- Why prevention is partial — design for containment
Hands-on:
- Attack your own chatbot
- Bypass a naive filter
- Restructure the prompt to shrink the blast radius
Module 3 — Treating Model Output as Untrusted
Lab: Lab 02 — 02-Output-Handling
The bug class developers underestimate most.
Topics:
- Model output as untrusted input to the rest of the app
- Insecure output handling (LLM02): XSS, SSRF, command/SQL injection downstream
- Never eval/exec/render raw model output
- Structured outputs and schema validation
- Output encoding and allowlists
- Safe rendering in web/UI contexts
Hands-on:
- Find and fix an insecure-output-handling vulnerability
- Enforce a JSON schema on model responses
Module 4 — RAG Security
Lab: Lab 03 — 03-RAG-Security
One of the biggest new attack surfaces — and it’s yours to build.
Topics:
- Vector DB and retrieval threats
- Ingestion sanitization
- Document provenance and trust scoring
- Retrieval scoping and metadata isolation
- Hidden instructions in retrieved content (indirect injection)
- Data exfiltration via retrieval
Hands-on: - poison a RAG pipeline with a malicious document - add ingestion sanitization and retrieval scoping to defend it
Module 5 — Agent & Tool Safety
Lab: Lab 04 — 04-Agent-Safety
Where a bug becomes an action.
Topics:
- Excessive agency (LLM06) and tool abuse
- Least privilege for agents
- Tool allowlists and argument validation
- Approval gates and human-in-the-loop
- Sandboxing tool execution
- Scoped, short-lived credentials for agents
- Limits on autonomous loops and chaining
Hands-on:
- Lock down an over-permissioned agent
- Add an allowlist + approval gate to a dangerous tool
Module 6 — Secrets, Identity & Cost
Lab: Lab 05 — 05-Secrets-and-Cost
The operational mistakes that hurt fastest.
Topics:
- API key and secret management (never in prompts, code, or logs)
- Per-user authentication and authorization for AI features
- Propagating user identity to tools and retrieval
- Denial-of-wallet: unbounded token/cost consumption
- Rate limits, token budgets, and timeouts
- Logging without leaking secrets or PII
Hands-on:
- Get secrets out of the prompt/code path
- Add per-user rate limits and a token/cost budget
Module 7 — Guardrail Libraries
Lab: Lab 06 — 06-Guardrails
Buy vs. build for input/output safety.
Topics:
- What guardrail frameworks do (and don’t)
- Input guardrails: injection/PII/topic classifiers
- Output guardrails: validation, filtering, grounding checks
- When a guardrail is appropriate vs. your own deterministic check
- Layering guardrails with the controls from earlier modules
- Performance, false positives, and failure modes
Hands-on:
- Add an input/output guardrail layer to an AI feature
- Measure what it catches and what it misses
Module 8 — Red-Teaming Your Own App
Lab: Lab 07 — 07-Red-Teaming
Ship it like an attacker already has it.
Topics:
- Building an abuse/test suite for AI features
- Automated prompt-injection and jailbreak tests
- Regression-testing guardrails and policies
- Running AI security checks in CI
- Model and dependency supply chain (provenance, pinning)
- A pre-ship security checklist for AI features
Hands-on:
- Write automated red-team tests for an AI feature
- Wire them into a CI check
Module 9 — Scoring AI Security: The SAIS-100 Framework
Lab: none — scoring exercise (uses the Capstone app)
Turn everything you’ve built into a repeatable score.
Topics:
- The AI Security Hexagon: six questions instead of “is it secure?”
- The six scored categories (Data, Prompt, Agent, Supply Chain, Detection, Governance)
- The 100-point rubric and its weightings
- Verdict bands and the single-category override rule
- The Elephant Scale Secure AI Score (SAIS-100) as a branded, re-runnable framework
- Scoring before/after hardening as a metric
Hands-on:
- Score the Capstone app on the 100-point scale
- Name the single change that most raises the score
Key insight: The three highest-weighted categories map to the trust boundaries a developer owns — so the score measures exactly what this course taught.
Capstone
Students harden a deliberately vulnerable AI application end-to-end.
The starter app contains:
- An injectable prompt
- Insecure output handling
- An unscoped RAG pipeline
- An over-permissioned agent
- Secrets in the prompt path
- No cost limits
Students apply the course:
- Restructure prompts for containment
- Validate and encode model output
- Sanitize and scope retrieval
- Apply least privilege and approval gates to the agent
- Move secrets out and add cost/rate limits
- Add guardrails and automated red-team tests
Deliverable: a hardened app plus a short OWASP LLM Top 10 self-assessment.
Module - Lab map
Labs run in lab order, which follows module order. The course has 9 modules and 7 labs: Module 1 is an architecture walkthrough/discussion and Module 9 is a scoring exercise, so neither has its own lab folder.
- Lab 01 - 01-Prompt-Injection: Attack your chatbot & design for containment (Module 2)
- Lab 02 - 02-Output-Handling: Fix an insecure-output-handling bug (Module 3)
- Lab 03 - 03-RAG-Security: Poison then defend a RAG pipeline (Module 4)
- Lab 04 - 04-Agent-Safety: Lock down an over-permissioned agent (Module 5)
- Lab 05 - 05-Secrets-and-Cost: Secure keys + add cost guardrails (Module 6)
- Lab 06 - 06-Guardrails: Add an input/output guardrail layer (Module 7)
- Lab 07 - 07-Red-Teaming: Automated red-team tests in CI (Module 8)
Module 1 (How AI Apps Break) has no lab — it runs as an architecture walkthrough and discussion. Module 9 (Scoring AI Security) has no lab folder — it runs as a scoring exercise against the Capstone app.
Požadavky
- Úroveň dovedností: Pokročilá.
- Studenti by měli být pohodlně obeznámeni s: tvorbou a využíváním REST API, skriptovacím jazykem (laby jsou v Pythonu), základním ověřováním aplikací, gitem a CLI.
- Není vyžadováno žádné zázemí v machine learningu – toto je kurz bezpečnosti aplikací pro lidi, kteří budují s LLM, nikoliv ty, kteří je trénují.
Cílová skupina
- Softwaroví / backendoví inženýři budující funkce LLM.
- Vývojáři full-stack a API.
- Inženýři aplikací AI/ML.
- Platformoví inženýři nasazující copiloty a agenty.
- Tech leadové a senior inženýři vlastnící funkce AI.
Reference (2)
Měl jsem opravdu velký zájem o poznávání útoků na AI a nástroje dostupné pro zahájení praxe a aktivní používání při bezpečnostním testování. Získal jsem mnoho znalostí, které jsem na začátku neměl, a kurz splnil mé očekávání. Mou nejoblíbenější částí z tréninku byl Comet Browser a byl jsem ohromen tím, co umí. Určitě se tomu chci věnovat více. Celkově byl skvělý kurz a užil jsem si vše, co jsem se naučil o OWASP GenAI Top 10.
Patrick Collins - Optum
Kurz - OWASP GenAI Security
Přeloženo strojem
Profesionální znalosti a způsob, jakým nám je prezentoval
Miroslav Nachev - PUBLIC COURSE
Kurz - Cybersecurity in AI Systems
Přeloženo strojem
