Formal verification is a mathematical proof that a program or system has a given property: for example, that it never goes beyond the bounds of an array or never deletes files outside its working folder. Testing checks a finite number of cases, while a proof covers all the cases described by the model. For agents built on artificial intelligence (AI), formal checking is applied not to the neural network itself but to its actions: each tool call is checked against a rule by a satisfiability solver such as Z3 (developed by Microsoft Research).
Formal verification proves a property rather than hunting for bugs on examples. A satisfiability solver checks whether a logical formula is satisfiable with respect to theories such as arithmetic, arrays and functions, and Z3 is one such program, an open-source solver from Microsoft Research. Proving that a model “will not think anything bad” is impossible, because it runs into Rice’s theorem, but proving that a specific action does not break a rule is possible, because an action has a finite set of parameters. The solver either proves that a violation is impossible, and the action is admitted, or finds values under which the rule is broken, and the agent receives a refusal with a counterexample. In MITR such a decision takes 1.08 milliseconds at the median with 20 contracts (measured on 24 October 2026).
What formal verification is
It is a set of methods that prove programs correct with the tools of mathematical logic. The property is written as a formula, the program or its model is written as a formula too, and then it is proven that the first follows from the second.
The classic example is the seL4 microkernel. In 2009, a team led by Gerwin Klein at NICTA (Australia) proved the functional correctness of its implementation, about 8,700 lines of C, in the Isabelle/HOL (Higher-Order Logic) proof system, and the proof took about 200,000 lines. This level is expensive, so for a long time full verification was used only in critical systems: operating system (OS) kernels, avionics, cryptography.
There are lighter forms too: model checking, static analysis that proves individual properties, and checking the satisfiability of formulas with SMT solvers. The last one is the one that fits checking the actions of AI agents.
What an SMT solver and Z3 are
SMT (Satisfiability Modulo Theories) is the problem of whether formulas are satisfiable with respect to theories: integers and reals, arrays, bit vectors, uninterpreted functions. An SMT solver receives a set of assertions and gives one of three answers.
- sat: the assertions are consistent, and the solver presents example values (a model);
- unsat: the assertions are inconsistent for any values;
- unknown: it could not decide within the resources it was given.
Z3 is an SMT solver from Microsoft Research, described by Leonardo de Moura and Nikolaj Bjørner at the TACAS conference in 2008. Its code is open on GitHub under the license of the Massachusetts Institute of Technology (MIT), and the latest release as of 16 October 2026 is z3-5.3.1 of 15 October 2026. Z3 has bindings for Python, C, C++, Java and other languages, and in Python it is installed as the z3-solver package.
Why checking an action is easier than checking the “thought” of a model
Rice’s theorem (1953) states that every non-trivial property of the behavior of programs is algorithmically undecidable, so no checker can be written that will tell, for any program, whether it will ever do something bad. A language model with its planning is in effect such a program, and proving that the agent will never want to delete the database is impossible in the general case.
But a tool call is not behavior; it is one specific object: the name of the tool, a path, an amount, a recipient. Checking whether it satisfies a rule is a problem of a different class. If the rule is written in a decidable fragment of logic, for example quantifier-free, with linear integer arithmetic and uninterpreted functions (QF_UFLIA), the solver is guaranteed to give an answer.
Hence the principle: do not try to verify the model; verify each of its actions at the boundary with the outside world.
Admission is a proof, refusal is a counterexample
The rule is written as a safety condition, and the facts as the parameters of the call. The solver receives the negation of the condition together with the facts.
- unsat: the condition cannot be violated with these facts. This is a proof of admissibility, and the action is admitted;
- sat: there are values under which the condition is violated. This is a counterexample: it shows exactly what is wrong and suggests how to fix the action.
A simple Python example: an agent is allowed payments no larger than a limit.
from z3 import Int, Solver, Not
amount, limit = Int('amount'), Int('limit')
s = Solver()
s.add(limit == 100000) # rule: the payment limit
s.add(amount == 120000) # fact: the amount in the agent's call
s.add(Not(amount <= limit)) # negation of the safety condition
print(s.check()) # sat: there is a violation, the call is refused
For an amount of 90,000 the same code prints unsat: a violation is impossible, and the call is admitted. If the call lacks a fact the rule needs, admissibility is not proven, and that is a refusal too.
How this works in MITR
In the institute’s MITR, admission rules are data files called contracts; as of 24 October 2026 there are 20 of them (19 on 14 October 2026), and the logic of the solver is QF_UFLIA. Admission is a proof (unsat), refusal is a counterexample with the conditions for a fix, and if a class of actions has no rule, the solver refuses, with no language model taking part in the decision. A refusal is carried out in enforce mode, and since 24 October 2026 all 20 contracts have worked in this mode, while on 14 October 2026 it covered one class, file deletion. The proof holds relative to the contract, and the correctness of the contract itself remains with the person who approved it.
How such a check differs from text filters is explained on the page about guardrails, and the architecture of MITR is described on the page MITR: architecture and rationale (in Russian).
How much time a check takes
As of 24 October 2026, a decision of MITR under a rule with 20 contracts takes 1.08 milliseconds at the median (1.96 milliseconds at the 95th percentile), and a call together with an explanation takes 5.22 milliseconds; on 14 October 2026, with 19 contracts, the figures were 0.95 and 4.26 milliseconds. A network call to an external service usually takes tens of milliseconds, so the check does not become a bottleneck.
The time depends on the complexity of the rule and the logic. In decidable quantifier-free fragments the solver answers quickly, while rules with quantifiers or non-linear arithmetic may take far longer or return “unknown”. That is why rules for actions are written in a simple fragment.
| Verification operation | Logical domain / artifact | Measured latency or replay result | Hardware profile |
|---|---|---|---|
| Warm single-contract check | Satisfiability Modulo Theories (SMT) proof in the Microsoft Research theorem prover (Z3) over quantifier-free linear integer arithmetic | 1.08 ms median · 1.96 ms p95 · 3.61 ms p99 | 4-core x86_64 processor, 8 gigabytes of memory, zero graphics accelerators |
| Refusal with Unsat Core witness | Exact violated policy clause returned to calling agent runtime | 5.22 ms median full round-trip | 4-core x86_64 processor, synchronous enforce |
| Cold solver process initialization | Process start and initial SMT context load | 30.7 ms median | Standard Linux container |
| Adversarial & routine benchmark (internal stand) | 114 adversarial attacks + 43 authorized operations prior to external accredited laboratory trials | 157 / 157 exact verdicts (0 misses, 0 false refusals) | Deterministic contract suite (soundness within formalized axioms) |
| Standalone hash-chain replay | 256-bit cryptographic hash ledger: historical epoch 33 anchored in Bitcoin block 973,024 via OpenTimestamps; live total 5,373 records across 55 epochs as of 15.11.2026 (pair with accredited qualified timestamps for banking/CII) | 5,079 / 5,079 historical epoch-33 decisions reproduced identically | Offline Python 3 standard library |
Frequently asked questions
What is formal verification in simple terms?
It is a mathematical proof that a program has the required property for all possible inputs, not only on the examples that were checked. For example, a proof that a function will never write data beyond the bounds of a buffer.
How does formal verification differ from testing?
A test checks a finite number of cases and may miss a bug in a case that was not checked. Formal verification proves the property for all the cases described by the model, but it requires a precise statement of the property and of the model.
What is Z3?
An open-source SMT solver from Microsoft Research under the MIT license. It checks the satisfiability of logical formulas with respect to arithmetic, arrays, functions and other theories, and it is available from Python (the z3-solver package), C, C++ and Java.
Can the safety of a neural network be formally proven?
Not completely: by Rice’s theorem, non-trivial properties of the behavior of programs are undecidable, and neural networks are probabilistic as well. Individual properties of small networks are proven with special methods, but in practice, for agents, it is not the model that is checked but each of its actions against a rule.
What does “admission is a proof, refusal is a counterexample” mean?
The solver receives the negation of the rule together with the parameters of the action. If the formula is unsatisfiable (unsat), a violation is impossible, and the action is admitted. If it is satisfiable (sat), the solver presents a counterexample, the values under which the rule is violated, and the action is refused.
Sources
- Z3 Theorem Prover, GitHub (release z3-5.3.1 of 15 October 2026, MIT license): https://github.com/Z3Prover/z3
- L. de Moura, N. Bjørner, “Z3: An Efficient SMT Solver”, Tools and Algorithms for the Construction and Analysis of Systems (TACAS), 2008
- G. Klein et al., “seL4: Formal Verification of an OS Kernel”, Symposium on Operating Systems Principles (SOSP), 2009
- H. G. Rice, “Classes of Recursively Enumerable Sets and Their Decision Problems”, Transactions of the American Mathematical Society, 1953
- Measurements of MITR made by the institute on 14 and 24 October 2026
Checked: 16 October 2026; MITR measurements of 24 October 2026.