Artificial Intelligence Hacking – Prompt Injection, Jailbreaking and AI Red Teaming

Training process

Training needs analysis

If you have specific requirements regarding the training programme, we will carry out a training needs analysis for you. This will guide us on which aspects of the programme should receive greater emphasis, so that the training programme meets your specific needs.

What will you gain?

Understand AI architecture - You will learn how LLM apps, RAG pipelines, and AI agents are structured, so you can spot trust boundaries, map attack surfaces, and identify where security risk actually emerges.

Spot prompt injection - You will learn to identify direct and indirect prompt injection and test whether a model follows unauthorized instructions hidden in messages, documents, and web content.

Assess jailbreak resistance - You will practice techniques used to bypass model restrictions and learn how to judge which safeguards are effective and which only create a false sense of protection.

Reduce data exposure - You will learn how to test for leaks of sensitive data, system prompts, chat history, and technical secrets, and how to reduce the chance and impact of such disclosures.

Secure RAG systems - You will explore common attacks against documents, knowledge sources, and vector stores, so you can detect data poisoning, broken permissions, and manipulated retrieval results.

Protect AI agents - You will learn to identify excessive privileges, unauthorized tool calls, and risky integrations before an agent performs an expensive, harmful, or otherwise unsafe action.

Run AI red teaming - You will build your own test scenarios for chatbots and LLM applications, classify findings with OWASP categories, and produce results you can use in a real security review.

Apply effective safeguards - You will learn practical defenses such as guardrails, input and output validation, permission controls, and monitoring for incidents, abuse, and unsafe AI-driven behavior.

Training programme

1. Artificial intelligence security – architecture and threats

basics of how LLM language models work,
architecture of applications using generative AI,
foundation models, chatbots, AI agents and RAG systems,
trust boundaries between the user, the model and the application,
the most important threats to AI-based systems,
differences between traditional application hacking and LLM Hacking,
basics of AI Red Teaming and model robustness testing,
OWASP Top 10 for LLM Applications and MITRE ATLAS.

2. Prompt Injection – manipulating the behavior of AI models

the mechanism of Prompt Injection,
Direct Prompt Injection – direct attacks,
Indirect Prompt Injection – attacks through external data sources,
manipulating the hierarchy of the model's instructions,
injecting instructions through documents, messages and websites,
attacks using conversation context,
manipulating chatbot responses,
testing the resistance of models to unauthorized instructions,
detecting and limiting the effects of Prompt Injection.

3. Jailbreaking – bypassing the restrictions of language models

what Jailbreaking is and how it differs from Prompt Injection,
security mechanisms and limitations of LLM models,
manipulating roles and the context of the conversation,
role-playing techniques and changing context,
multi-step attempts to bypass restrictions,
attacks using multilingualism and ambiguity of instructions,
assessment of the effectiveness of model safeguards,
methods of strengthening application resilience to Jailbreaking.

4. Data leaks and information disclosure by AI

Sensitive Information Disclosure – disclosure of sensitive data,
System Prompt Leakage – disclosure of system instructions,
threats related to conversation history and application memory,
security of personal data and company information,
unauthorized access to documents and knowledge sources,
risk of disclosure of API keys and authentication data,
testing data isolation between users,
access control and minimization of transmitted data,
detection and mitigation of information leaks.

5. Security of RAG applications and AI knowledge bases

Retrieval-Augmented Generation architecture,
threats related to retrieving data from external sources,
Prompt Injection in documents used by RAG,
poisoning of data and knowledge sources,
manipulation of document content and search results,
vulnerabilities of vector databases and embedding mechanisms,
control of permissions to documents and data fragments,
testing the correctness of responses generated based on sources,
securing the indexing and information retrieval process.

6. Hacking and Securing AI Agents

AI agent architecture and action-taking mechanisms,
LLM model integrations with tools and external APIs,
threats related to the autonomous execution of operations,
Excessive Agency – excessive permissions of agents,
taking over and manipulating the task flow,
unauthorized calling of functions and tools,
security of agent integrations with databases and files,
threats related to communication between agents,
limiting permissions and applying the principle of least privilege,
human oversight over high-risk operations.

7. Security testing of LLM applications

planning security tests of AI systems,
creating AI Red Teaming scenarios,
preparing Prompt Injection test sets,
testing resistance to information disclosure,
testing the security of responses generated by AI,
detecting dangerous operations initiated by models,
testing authorization and access control mechanisms,
using tools automating LLM tests,
analysis of results and classification of detected vulnerabilities.

8. Mechanisms for securing AI systems

designing a secure architecture for LLM applications,
separating system instructions from untrusted data,
validation of model input and output data,
application of Guardrails mechanisms,
control of user and agent permissions,
securing API integrations and external tools,
filtering and monitoring dangerous operations,
restricting access to confidential data,
managing query limits and model usage costs,
protection against uncontrolled resource consumption,
monitoring AI security incidents.

9. AI Red Teaming – resilience audit of AI systems

methodology for conducting Red Teaming tests for AI,
threat modeling for chatbots and agents,
identification of the attack surface of LLM systems,
preparation of test scenarios,
analysis of the effectiveness of protection mechanisms,
classification of vulnerabilities in accordance with OWASP Top 10 for LLM Applications,
documenting identified weaknesses,
development of security recommendations,
creation of a plan for strengthening the resilience of AI systems,
continuous security testing after updating models and applications.

What are the prerequisites for participating in the training?

Web app fundamentals - You should understand how a typical web application works, including requests, responses, APIs, and user roles, so you can follow the architecture of AI-based systems.

Basic security knowledge - You should know core security concepts such as authentication, authorization, access control, and vulnerabilities, because the course applies them to AI systems.

Familiarity with data and APIs - You should have experience working with data, documents, or APIs, so you can quickly understand risks related to RAG, tool integrations, and information flow.

Technical analysis mindset - You should be comfortable analyzing messages, system logic, and configuration, because the training includes testing and evaluating realistic security scenarios.