Local AI model vs. cloud: how our own fine-tuned model beat it

Why we tried it at all

Our product Inovai Atesty runs in production on cloud AI and works well. But some of the companies we talk to - nuclear energy, pressure equipment, strict IT departments - have one hard condition: documents must not leave the company. For them the cloud is a dead end, not a preference.

So the question was: can a smaller open model, fine-tuned on our own confirmed documents, do the same job as cloud AI - just locally? We measured quality on 99 test certificates the model never saw during training. Without that, the number would lie: the model would have “memorised” the test documents.

What didn’t work: brute force

The first instinct of anyone tuning AI is “take a bigger model”. We tried it - and it turned out the opposite of what intuition says:

• a 4× larger model ended up 11 percentage points worse than the small one,
• full retraining of the whole model (instead of fine-tuning a small part) cost 8 points,
• more copies of the training data cost 7 points,
• artificial variations of documents brought only noise.

Every “brute force” attempt made accuracy worse. Had we measured only one of them, we might have concluded that “a local model can’t do this”. It could - the problem just wasn’t the model.

What made the difference: detective work on the training data

The breakthrough came when we stopped tuning the model and started examining the data it was learning from. We found two systematic errors:

1) The model learned to delete correct data. Dozens of training documents contained values that the operator had deliberately deleted during review - they were useless for her work. A correct business decision, but the model learned from those examples that these values “should be discarded”. Fixing this brought the biggest accuracy jump of the whole project.

2) Shifted chemistry columns. In some documents, chemical composition values had been shifted under the wrong elements during manual transcription. A recalculation exposed it: the carbon equivalent (CEV) can be calculated from the individual elements, and when the result doesn’t match the printed value, the error is certain. Maths, no guesswork.

The confidence layer: every document is read twice

Accuracy is one thing. The other - in operation perhaps more important - is knowing which values to trust. So the final solution reads every document twice, independently, and compares the results:

• where both readings agree, the value is correct in 99.7% of cases,
• where they disagree, the operator gets a warning - and typically checks one flagged value per document instead of all ~28.

On top of that run deterministic safeguards: recalculation of check values and physical sanity checks (elongation below 2% is an absurd value for structural steel, and the system flags it on its own). Where the maths doesn’t add up, the result is discarded - never guessed.

What this means for companies with sensitive data

Until this experience, we offered on-prem deployment with a quiet caveat: “expect lower accuracy than in the cloud”. That caveat is gone: on a narrow, well-trained task, a local model can match or even beat cloud AI - and the data never leaves the company. It also keeps learning from operation: every confirmed document is a new training example.

How we think about choosing between the cloud, a private environment and a local model is described on the Local models page and in the article On-prem AI model: when it makes sense and when it doesn’t.

An honest disclaimer to finish

So this doesn’t come across as “the cloud is dead”: it isn’t. The local model won on one narrow task, for which it was fine-tuned on thousands of confirmed documents from real operation. On general tasks - understanding an unusual document, more complex reasoning - large cloud models remain clearly better. And fine-tuning your own model has entry conditions: enough confirmed data, a strictly separated test set and infrastructure for evaluation. Without that, it’s a lottery.

The practical takeaway for decision-making: start with the cloud or a private environment (whatever your rules allow), collect confirmed data - and deploy a local model once you have something to train on and a reason to (security, volume, independence). In exactly that order it makes both economic and technical sense.