AI Terms Explained: LLM, Diffusion Model, Hallucination & Co.

1. What is Generative AI?

Generative AI refers to systems that create new content — images, text, video, audio, code — rather than just analysing existing content. When you ask ChatGPT to write a cover letter or use Midjourney to generate a photo of a dog in a spacesuit, you are using generative AI.

The key word is generate: these systems do not retrieve or copy content from a database. They synthesise something new based on patterns learned from billions of examples during training. Current systems like GPT-4o (OpenAI), Gemini 2.0 (Google), and Claude 3.5 (Anthropic) show how far this technology has already come.

2. LLM Explained: The Tech Behind ChatGPT, Claude & Co.

A Large Language Model is a neural network trained on enormous amounts of text — large portions of the internet, books, scientific papers — with the goal of predicting which word (or token) comes next.

That sounds simple, but making good predictions requires approximating the statistical patterns of grammar, facts, logic, and context across billions of examples. This is why LLMs can write essays, answer questions, translate languages, and generate code — not because they "understand" in a human sense, but because they have compressed an enormous amount of structure from human-generated text.

The "large" refers to the number of parameters — internal numerical values adjusted during training. GPT-3 had 175 billion. Today's frontier models like GPT-4o or Meta's Llama 3.3 (released late 2024) are estimated to have significantly more.

3. Transformer: The Architecture Behind Modern AI

Virtually every major AI model today — GPT, Claude, Gemini, Stable Diffusion — is built on the Transformer architecture, introduced by Google researchers in 2017. Before Transformers, sequential text processing was slow and struggled with long-range dependencies.

The key innovation is the attention mechanism: instead of processing words one at a time, a Transformer looks at all tokens in a sequence simultaneously and learns which ones are most relevant to each other. This makes training massively parallelisable on modern GPU hardware.

The 2017 paper "Attention Is All You Need" is arguably the most influential research paper in recent AI history. Almost everything you interact with in today's AI landscape — text models, image generators, video models — traces its architecture back to this single design choice.

4. Diffusion Model: How AI Generates Images

Diffusion models are the technology behind most modern AI image generators — Stable Diffusion, DALL-E 3, Midjourney, Flux. The principle is counterintuitive but elegant.

During training, the model learns to gradually remove noise from an image. Start with pure random noise, apply the model iteratively — and a coherent image emerges. A text prompt steers the direction: "dog in a spacesuit" guides the process accordingly.

This is why generating an image takes a few seconds: the model is running many denoising steps. AI images are built up from noise rather than assembled from real photographs — which is why they sometimes have a characteristic look. Test yourself: can you spot the AI-generated image in our paintings quiz or animals quiz?

5. Tokens & Context Window — Why AI "Forgets" After 100 Pages

LLMs do not process text word by word — they work with tokens. A token is roughly 3 to 4 characters, or about three quarters of an average English word. "Hello, how are you?" is around 6 tokens.

The context window is the maximum amount of text a model can process at once — input and output combined. Older models: 4,000 tokens. Today: GPT-4o handles 128,000 tokens, Gemini 1.5 Pro up to 2 million tokens — roughly 20 average novels. Tokens are also the basis for API pricing: the more tokens you send and receive, the more you pay.

6. Embeddings & Vector Databases

When an AI model processes text (or images, audio, etc.), it converts each piece of input into a long list of numbers called an embedding — a point in high-dimensional space. Semantically similar content ends up geometrically close together in this space. "Dog" and "puppy" will have similar embeddings; "dog" and "mortgage" will be far apart.

This property is what makes similarity search possible. A vector database (like Pinecone, Weaviate, or pgvector) stores millions of these embeddings and retrieves the closest matches to any given query — in milliseconds, at scale.

Vector databases are the backbone of most RAG systems (see section 8) and semantic search engines. They are what allows an AI to find "the three most relevant paragraphs from 10,000 internal documents" without reading all of them each time.

7. Hallucination: Why AI Lies With Confidence

Hallucination is the term for when an AI confidently states something that is completely false — inventing a scientist who does not exist, citing a paper with a made-up DOI, giving the wrong date for a historical event.

It happens because base LLMs are not databases — they generate the most statistically plausible continuation of your prompt rather than retrieving verified facts. Sometimes the most plausible-sounding answer is simply wrong.

It is worth noting: modern AI products like ChatGPT or Perplexity are often augmented with web search and tools, which reduces (but does not eliminate) hallucinations. A bare model without these additions is significantly more prone to confabulation. Approaches like RAG (see next section) address this, but the underlying problem remains an active area of research.

8. RAG: How Companies Use Their Own Data Safely With AI

Retrieval-Augmented Generation is the most important technique for reducing hallucinations and connecting LLMs to company-specific knowledge. The idea: before the model responds, a system searches a knowledge base (internal documents, PDFs, manuals) for relevant passages and adds them as context to the prompt.

Instead of answering from memory, the model can reference real sources. This is why companies use RAG to build chatbots on their own data — without retraining the model and without sending sensitive data to the provider.

9. Prompt Engineering: How to Get Better Results From AI

A prompt is the text input you give an AI model. Prompt engineering is the practice of crafting these inputs to get more useful, accurate, or consistent results.

Techniques include: clear role instructions ("You are an experienced lawyer"), providing examples in the prompt (few-shot prompting), breaking tasks into sub-steps (chain-of-thought), or explicitly asking the model to reason before answering.

Well-structured prompts can noticeably improve output quality without any model changes. That said, the impact varies greatly depending on the task and model — modern frontier models have become substantially better at inferring intent, which reduces how much careful prompting is needed compared to earlier systems. For a deeper dive, Prompting Guide is a solid reference.

10. Training Phases: Pre-training, Fine-tuning & RLHF

Modern AI models are not created in one step — they go through distinct training phases, each serving a different purpose.

Pre-training is the expensive foundation: the model learns from billions of documents, building up broad world knowledge and language patterns. Training a frontier model from scratch costs tens of millions of dollars and months of compute.

Fine-tuning takes a pre-trained model and trains it further on a smaller, specialised dataset — without starting from scratch. A company might fine-tune an LLM on its own customer support transcripts, or fine-tune an image model on product photos to generate consistent brand visuals.

RLHF (Reinforcement Learning from Human Feedback) is the phase that turned raw LLMs into the helpful assistants we use today. Human raters evaluate model outputs, score them, and the model is trained to produce more highly-rated responses. This is how ChatGPT learned to be polite, decline harmful requests, and actually answer what was asked — rather than producing statistically plausible but unhelpful text.

11. AI Agents: Autonomous Multi-step Systems

Standard LLMs answer a question and wait. AI agents go further: they execute multiple steps autonomously to reach a goal — using tools like browsers, code interpreters, or external APIs along the way.

Instead of "write me an email", an agent could check your calendar, find a free slot, draft the email, and send it. OpenAI's Operator, Anthropic's Computer Use, and Google's Project Astra are current examples of this direction.

Agents represent a meaningful expansion of what AI can do autonomously — but the field is still early, with reliability and error-handling being active challenges. The core question of how much autonomy to grant AI systems, and under what oversight, is one of the defining design problems of the current era.

12. Deepfake: When AI Fakes Reality

Deepfake originally referred to AI-generated face-swap videos — placing one person's face onto another's body. The term now covers any convincing AI-generated fake of a real person: their voice, their likeness, video of them saying things they never said.

What has fundamentally changed is the barrier to entry. Creating a convincing deepfake once required significant technical skill. Today, consumer tools can do it in minutes. This is exactly why recognising AI-generated content is becoming an essential skill — try the real vs. AI video quiz to see how well you can spot them.

13. Open Source vs. Closed Source: Who Controls the AI?

A model's weights are the billions of numerical parameters that define its behaviour — the result of training. When a company releases their weights publicly, the model can be run by anyone, anywhere, without API dependency.

The open vs. closed debate is one of the defining tensions in the current AI landscape, touching on safety, competition, and societal control over the technology. Most open-weight models are available to download and test on Hugging Face.

14. Multimodal Models: Text, Image, Audio in One System

Early AI models were single-modal: they handled one type of data — either text or images, not both. Multimodal models can process and generate across different data types within a single system.

GPT-4o, Gemini 2.0, and Claude 3.5 are multimodal: you can show them an image and ask questions about it, or upload a chart and ask them to explain it. The trend is toward models that handle text, images, audio, and video interchangeably.