data analyst – use of LLM and AI in analysis, insight generation, and business decision-making

1. Introduction to AI and LLM models in the analytical context

LLM models are presented here from the perspective of analytical work: how they operate, where probabilistic AI helps, and where its limits create risk. The section also covers model selection, token and context window management, the changing role of the analyst, and connecting language models with databases and code.

• what LLM models are and how they work, deterministic vs probabilistic AI; strengths and limitations of LLMs in data analysis,
• where AI realistically helps the analyst, and where it harms,
• the evolution of the analyst's role: from „query operator” to „systems orchestrator”,
• token economics and context window management RLM models – selecting the model for the task,
• AI systems architecture: how to connect a language model with a database and code.

2. Prompt Engineering for data analysis – fundamentals

LLM behavior directly shapes how analytical prompts should be structured, contextualized, and constrained. The section covers context engineering, preparing data in formats such as Markdown, JSON, XML, and CSV, defining system prompts, working with SQL, and using RAG, vector, and graph databases as AI memory layers.

• how the LLM model „thinks” and what this means for prompts, the structure of an analytical prompt,
• context engineering, data preparation (Markdown, JSON, XML, CSV),
• system prompt – defining roles, constraints and output formats,
• generation and refactoring of SQL queries,
• translating business requirements into analytical queries,
• analysis of existing queries (readability, optimization, edge cases),
• „memory” of AI systems – RAG, Agentic RAG, vector databases and graph databases.

3. Analytical cases – working with data and insights

From raw results to decision-ready conclusions, AI can support both interpretation and control of analytical quality. Participants work on automatic summaries, product insights, KPI creation and validation, anomaly and gap detection, A/B test interpretation, and spotting potential errors in data or analysis logic.

• automatic summaries of analysis results, generation of product insights and hypotheses, sanity checks: detection of anomalies, gaps, illogical results,
• creation and validation of KPIs,
• interpretation of A/B test results with the help of AI, identification,
• of potential errors in data and analyses.

4. A/B tests and experiments using AI

Designing experiments with AI requires balancing speed of support with statistical discipline. The section covers hypothesis formulation, interpretation of test results, and the use of AI in inference and communication, while also addressing the risks of automated conclusions and the role of synthetic users in experimentation.

• AI support in designing experiments,
• formulating test hypotheses, interpretation of statistical results,
• supporting inference and communication of results,
• risks of automatic interpretations of A/B tests,
• „synthetic users” (AI Personas).

5. Workshops: real team scenarios

Working through real team scenarios, participants refine analytical prompts and test multiple prompt variants against the same task. By comparing outputs and response quality, they assess where AI is genuinely useful in specific use cases and where manual analytical judgment should remain primary.

• working on real examples from the team,
• optimization of existing analytical prompts,
• testing different variants of prompts,
• comparing results and quality of responses,
• assessment of the usefulness of AI in specific use cases.

6. Integration of AI with the analyst's daily work

In day-to-day analysis, AI works best as a copilot that supports decisions rather than replaces the analyst. The focus is on ad-hoc analysis, data exploration, documentation, communication of insights to business stakeholders, automation of repetitive tasks, and practical rules for effective teamwork with AI.

• AI as the analyst's „copilot”, not a replacement,
• use of AI in:
  
  • ad-hoc analyses,
  • data exploration,
  • analysis documentation,
  • communication of insights to the business.
• automation of repetitive analytical tasks,
• best practices of teamwork with AI.

7. Quality, ethics and safety of working with AI

Reliable analytical work with AI depends on validation, traceability, and clear accountability for conclusions. This section addresses hallucination detection, Explainable AI and LLM-as-a-Judge, then moves to safe handling of sensitive data through local models, anonymization, masking, hybrid setups, and organizational practices.

• validation of results generated by AI, the analyst's responsibility for the results, how not to „let through” an incorrect insight,
• methods of detecting hallucinations in numbers and facts,
• auditability and „Explainable AI”, „LLM-as-a-Judge”,
• running open-source models (Llama 3, Mistral, Gemma) using (Ollama, LM Studio),
• analysis of sensitive data without sending it to the cloud (Local-First Analytics),
• data anonymization and masking techniques before sending to external APIs. Hybrid approach,
• good organizational practices.

8. Summary and recommendations for the team

A practical closing view helps the team decide when AI adds value and when classical analytical methods remain the better choice. The section organizes checklists for AI-supported analysis, outlines a recommended analyst workflow, and identifies development areas for improving team capability and governance.

• when to use AI, and when classical methods,
• checklists for analytical work with AI,
• the recommended workflow of an analyst supported by AI,
• areas for the team's further development.