How to Get Consistent Results from Local AI Models
Local models feel unpredictable when the same prompt gives a different shape each time: sometimes bullets, sometimes an essay, sometimes an apology plus the essay. Most of that is settings and missing constraints, not a broken install. You can make summaries, extraction, and small code tasks repeatable enough to use in a daily workflow.
This page assumes you already write a clear prompt. If the wording is still loose, read How to Write Better Prompts for Local AI Models first. The methods that survive testing are summarized in What Actually Works.
Why the answer changes
Sampling picks the next token with some randomness. Higher temperature means more randomness. A long chat means earlier rules are farther from the latest question. An open prompt (“write something helpful”) has many valid completions, so the sampler has room to wander. You reduce variation by shrinking that room: lower temperature, a seed when you need a true rerun, a format with little room, a short context, and a model that is not being asked to do five jobs.
1. Lower the temperature before you rewrite the prompt
For a task you want to repeat, set temperature between 0.1 and 0.3. Leave top-p at the app default unless you know why you are changing it. Then run the identical prompt three times.
Same prompt, three intentions:
- Temperature 0.9: you may get 3, 5, or 4 rules, plus a friendly intro on some runs. The wording jumps around. Do not judge the prompt yet.
- Temperature 0.2: you should get 4 rules most runs. Wording still changes a little. That is normal.
- Temperature 0: the most repeatable, and sometimes the most bland. Use it for extraction. For writing, 0.2 is usually enough.
You changed too much, and you left the setting that causes the shuffle. Freeze the prompt, drop temperature, and only then edit one line if the format is still wrong.
2. Use a seed when you are comparing prompts
Ollama, LM Studio, and some other runners let you set a seed. The same model, same prompt, same seed, and same sampling settings should come back nearly identical. That is how you tell whether a wording change helped.
A practical comparison:
- Set temperature to 0.2 and seed to 42.
- Save the output of prompt A.
- Change one line to make prompt B. Keep the seed.
- Compare A and B. If B is better, keep it and then try seed 43 once to see if the format still holds.
If your app has no seed control, you cannot get bit-for-bit repeats. You can still demand a format and a low temperature, which is what you need for daily use. A seed is for testing, not for users reading a summary.
3. Make the format hard to wander from
The tighter the shape, the less the sampler can do. Labels are better than “organize this.”
If the model still adds a recommended reply, put the ban on its own last line: “Do not write a reply.” Last-line rules are followed more often than rules buried in the middle.
4. Add a one-line check for brittle formats
Use this for JSON, tables, and exact counts. Do not use it for ordinary prose.
Read the output. If you see a paragraph that starts with “I checked the JSON,” the model is showing the check. Add “Output the JSON only” or remove the check and rely on the key list. Some small models cannot do a silent check. In that case the key list alone is the better prompt.
There is nothing to check. You will get a stiff message and sometimes a paragraph about quality. Cap the words instead.
5. Keep the context short
Rules from ten messages ago are easy for a small model to drop. For a repeatable job, do not keep chatting. Start a new session, or paste the rules again at the top of the latest message.
A pattern that stays stable:
- One system or first message with the rules.
- One user message with the source text.
- If you need a revision, a new chat that contains the rules plus the previous output, not the entire history.
It often does not know. Paste the format again. “Same as before” is not a format.
6. Match the model size to the job
A smaller model with a narrow prompt is often more consistent than a larger general model with a sloppy one, but it is not magic. Use a small model for rewrite, extract, and short summaries of text you provide. Use a larger local model when the task needs longer code or multi-step reasoning. Do not ask a 3B model to review a 400-line file and expect a stable review.
If the format holds on a 7B model and falls apart on a 3B model, keep the prompt and switch models. Do not keep adding rules until the small model is buried.
A repeatable checklist
- Write the format as labels, a count, or JSON keys.
- Put the source text after the rules, behind a clear label.
- Set temperature to 0.2 for anything you want to trust.
- Set a max token limit slightly above the answer you asked for.
- Run twice. If the shape differs, tighten the last line before you add new rules.
- If you are comparing two wordings, set a seed.
- If the chat is long, start over and paste the rules again.
What consistency is not
- It is not identical sentences every time, unless temperature is near 0 and the prompt is very tight.
- It does not make a model cite real sources it cannot see. Paste the source.
- It does not remove the need to read the output. A stable wrong answer is still wrong. For extraction, check two fields against the original text.
- It is not improved by pasting a huge “you are an expert who never makes mistakes” preface. That line does not change sampling.
When the prompt is stable, save it. The Prompt Library is a set of starting points you can store next to your own. If you are unsure a saved prompt is too long for the context window, estimate it with the Token Counter before you attach a large file.