How AI Image Generators Work, Explained Simply

You don't need to understand diffusion models to use an AI image generator, but a rough picture of what happens between pressing Generate and seeing an image explains a lot of otherwise puzzling behaviour: why the same prompt gives different results, why negative prompts work well on some models and barely at all on others, why hands go wrong, and why full sentences often beat keyword lists. This guide gives you that picture, without equations, and then explains how IMAGEKA puts several models to work.

Step zero: what the model learned in training

Every modern image generator starts with a training dataset of images paired with text: alt text, captions, or, increasingly, detailed descriptions written by another AI model. The model never stores these images as a collection it can look up. Training adjusts billions of numerical weights so that the network gets good at one narrow task, described below. What survives training is statistical knowledge: how fur catches rim light, what "isometric" does to perspective, which colours tend to appear with "autumn".

That narrow task is denoising. Take a training image, add a random amount of noise to it (anything from a light speckle to so much that the image is completely destroyed), show the noisy version to the network together with the caption, and ask it to predict what noise was added. Compare its guess with the truth, nudge the weights, and repeat this hundreds of millions of times. Eventually the network can look at almost any noisy image and make a good estimate of which parts are noise and which are picture, using the caption as a clue.

Many newer models use a variant of this idea called flow matching (rectified flow is one well-known form). Instead of predicting the noise itself, the network predicts the direction that leads from noise towards a clean image, and training encourages that path to be close to a straight line. Straighter paths make it easier to generate in fewer steps. For a beginner, "it learned to turn noise back into pictures" is an accurate summary of both approaches.

Generating: from static to picture

To generate a brand-new image, you run the learned skill on an image that is nothing but noise. The model estimates which part of this static is noise, given your prompt, and removes some of it. The result is still very noisy, but it now leans slightly towards something that matches the prompt. Repeat: estimate, remove, estimate, remove. Each pass is called an inference step (or sampling step).

Two properties of this process are worth knowing:

• Composition is decided early. In the first steps, when the canvas is mostly noise, the model commits to large structures: where the horizon sits, where the bright and dark areas fall, how big the subject is. Later steps mostly refine texture and detail. So the words that control layout and lighting have their biggest effect at the start, and once a composition is set, the later steps refine it rather than replace it.
• The model never "understands" objects as separate things. It is predicting patterns of pixels (strictly, latent values) that look plausible together. This is why it can blend two objects into one, or give a person a hand that looks fine locally but has six fingers. Each region is locally convincing, but nothing in the process counts fingers.

Latent space and the VAE

Running a network with billions of parameters over every pixel of a 1024×1024 image at every step would be slow. So almost all current open image models work in a latent space: a compressed representation of the image.

A separate, much smaller network called a variational autoencoder (VAE) was trained to compress images and decompress them again with as little visible loss as possible. A typical VAE shrinks each side of the image by a factor of 8 and describes each small patch with a handful of numbers (4 in older designs, 16 in several newer ones) instead of red, green and blue. A 1024×1024 image is about 3.1 million values (1024 × 1024 × 3). With a 16-channel VAE its latent is a 128 × 128 grid of about 262,000 values, roughly twelve times fewer; a 4-channel latent is smaller still. All the denoising steps happen on this small grid. Only at the very end does the VAE decoder expand the finished latent into the image you download.

This has visible consequences. Very fine, high-contrast detail, such as tiny text, distant faces or thin repeating patterns, has to survive this round trip through 8×8-pixel patches. It is one reason small details come out mushy while large ones look sharp. Newer VAEs with more channels keep noticeably more fine detail than older ones.

The denoiser: U-Net or transformer

The network that does the actual denoising is the largest part of the system, and it comes in two main designs.

• U-Net. Used by the original Stable Diffusion models and their many fine-tuned descendants. It squeezes the latent down to a coarse summary and expands it back up, with shortcuts between matching levels so fine detail isn't lost. Convolutions handle local texture; attention layers at several levels connect the image to your prompt. U-Net models are mature, comparatively light to run, and have a huge ecosystem of community fine-tunes specialised for one look, such as photographic portraits or anime.
• Transformer (often called a diffusion transformer). Used by many newer models. The latent is cut into small patches that are treated like tokens in a language model, and every patch can attend to every other patch and to every word of the prompt. Transformers scale well with size and tend to be better at global layout, counting, spatial relationships and rendering short text, at the cost of more computation.

Neither design is simply "better", and architecture matters less than training. A model fine-tuned for one look can beat a larger general model on that style, while the general model handles a wider range of unusual requests. That specialist-versus-generalist trade-off is the reason IMAGEKA uses more than one model (see below).

Text encoders: how the model reads your prompt

The denoiser can't read English. Your prompt is first split into tokens (whole words or word pieces) and passed through one or more text encoders, which turn it into lists of numbers called embeddings. The denoiser consults these at every step through a mechanism called attention: in effect, each region of the image asks which parts of the prompt are relevant to it.

There are two broad families of text encoder, and the difference explains a lot about how you should write:

Stable Diffusion 1.5 relied on a single CLIP encoder, which is where the habit of comma-separated tag lists came from: CLIP responded to strongly associated words and largely ignored sentence structure. Models that add a large language-model encoder get much of their detailed understanding from it, so a clear sentence such as a ceramic mug on a windowsill, rain on the glass behind it carries real information in "on" and "behind it". CLIP-based models still benefit from clear descriptions, but they lean more on distinctive keywords, and models fine-tuned on tagged images (common for anime) respond especially well to short comma-separated tags. The prompt-writing guide builds on this.

Steps, guidance and distilled fast models

Steps. A classic diffusion model typically needs 20 to 50 denoising steps for a clean image. Each step is a full pass through the denoiser, so steps are the main driver of generation time. Beyond a model's sweet spot, extra steps add little.

Guidance. Early text-to-image models had a problem: left to themselves, they followed the prompt only loosely. The standard fix is classifier-free guidance (CFG). At each step the model makes two predictions, one with your prompt and one with an empty prompt, and then pushes the result further in the direction that separates them. A guidance scale (often called CFG scale) controls how far. On Stable Diffusion-family models, values of roughly 5–8 are typical. Higher values follow the prompt more literally but look harsher and over-saturated; lower values look more natural but drift from the prompt.

The negative prompt piggybacks on this mechanism. It replaces the empty prompt in the second prediction, so "push away from nothing" becomes "push away from blurry, extra fingers". That is why negative prompts are effective on models that run CFG.

Distilled fast models. CFG has a cost: two predictions per step double the work, on top of the many steps. To speed things up, a slower, higher-quality model can act as a teacher while a "student" is trained to reach a similar result in a handful of steps, sometimes without running CFG at all. This is called distillation. Distilled models are dramatically faster, at the price of some fine detail and variety. When a distilled model skips CFG, its guidance is effectively set to 1, which in the CFG formula means "use the prompted prediction as it is": the second prediction, and with it the negative prompt, cancels out. So on such models a negative prompt does little or nothing.

How IMAGEKA uses several models

No single open model is best at everything. A general-purpose model that handles posters, landscapes and odd concept ideas well may produce plasticky skin; a model fine-tuned for photographic portraits may be poor at lettering; an anime model is great at cel shading and little else. So IMAGEKA runs several open-source models, all under open licences that allow commercial use, and picks one for each request based on your prompt and style preset. You can't choose the model manually; the routing is automatic.

A few practical consequences follow from this:

• Presets and keywords steer the model choice. Picking a style preset, or using words like "photograph", "anime" or "watercolour", effectively decides which model renders your image. If a result looks like the wrong kind of model made it, change the preset or the style words rather than piling on adjectives.
• Write for the model you are steering towards. Clear sentences suit the general model; the anime model also responds well to short comma-separated tags.
• Put text in quotes. Words in quotation marks route the request to the general model, which handles lettering best. Keep it to a few words.
• Specialists fall back automatically. If a specialist model fails on a request, IMAGEKA retries it with the general model, so you still get an image.
• Small sizes are rendered larger. The specialist models look poor below 768 pixels, so a 512×512 request that goes to one of them is generated at 768×768.

Seeds and randomness

The starting noise is generated by a pseudo-random number generator, and the number that initialises it is the seed. Same model + same prompt + same settings + same seed gives (almost exactly) the same image. Change only the seed and you get a different composition of the same idea, because the early steps commit to whatever large structures the particular noise pattern suggests.

IMAGEKA picks a fresh random seed for every image and doesn't expose seed control. In practice:

• You can't reproduce an earlier image exactly, so download good results immediately.
• When you test a prompt change, generate 2 or 4 images. Otherwise you can't tell whether the change or the seed caused the difference.
• If a prompt usually gives a good layout but occasionally a strange one, that is seed variation, not a problem with your prompt. Run it again.

Why hands and text go wrong

Both failures come from the same root: the model generates what looks locally plausible, and it has no rule that says "a hand has five fingers" or "this word is spelled B-A-K-E-R-Y".

Hands are hard because they take an enormous variety of shapes, overlap themselves, are often partly hidden or blurred in training photos, and usually occupy a small part of the frame. At a few latent cells wide, a hand gets little room to resolve. Recent models are much better than early ones here but not perfect. Bigger hands in the frame, simple poses ("hands resting on a table") and fewer interacting objects all help, and on the specialist models the default "bad hands" negative helps a little too. The troubleshooting guide has specific rewrites.

Text is hard because the model draws letter shapes; it doesn't type characters. It also has to learn how a quoted word in the prompt maps to specific glyphs, which it does reliably only for short, common words. Language-model text encoders, transformer denoisers and more text-aware training captions have made short quoted text far more reliable than it was on Stable Diffusion 1.5. On IMAGEKA, quoted text goes to the general model, which is the strongest of its models at lettering. Long phrases and small type still drift into near-words.

Resolution and aspect ratio

Resolution sets the size of the latent grid: a 768×768 image is denoised on a 96×96 grid, a 1024×1024 image on a 128×128 grid. Most current models were trained largely on images of around one megapixel, so 1024×1024, 1024×768 and 768×1024 are close to their comfort zone. At 512×512 there is a quarter of the space for detail. Faces in medium shots and small objects suffer first, so treat 512 as a draft size (and remember the specialist models render it at 768×768 anyway).

Aspect ratio changes what gets composed, not just how big it is. The early steps fill whatever canvas they are given, so a tall 768×1024 frame tends to produce full-length figures, doorways and towers, and a wide 1024×768 frame invites horizons, rooms and groups side by side. If a portrait prompt keeps giving you a figure that is too small, switching to the portrait shape often fixes it without any change to the words. The colour and composition guide shows these layouts as diagrams.

What this means for your prompts

• Write sentences with relationships. Language-model encoders read prepositions and word order, so "a lamp to the left of the sofa" means something to them.
• Lead with what matters. The subject and the overall scene belong early in the prompt, and very long prompts get cut off at the end.
• Choose the preset deliberately. It adds style words and steers which model renders the image. If your prompt already describes a style, pick Default so the preset doesn't fight it.
• Name the light and the frame. They drive the early, composition-setting steps. See the lighting and camera guide for the vocabulary.
• Use negatives where they work. They help on photo, anime and painterly images; for everything else, say what you want instead of what you don't.
• Skip the quality-tag soup. "Masterpiece, 8K, best quality" was a habit from CLIP-only models trained on scraped captions. Concrete descriptive words do more on every model.
• Pick the canvas shape first. Aspect ratio shapes composition from the very first step.
• Judge prompts across several seeds. One image is a sample, not a verdict.
• Keep text short and quoted, and make hands large and simple. These are the two areas where the process is weakest.

For definitions of the terms used here and many others, see the AI image generation glossary.

Frequently asked questions