Two synthetic case studies of arithmetic constraints and source provenance
Prepared for Timothy Hauptrief’s Research portfolio · October 10, 2026 · Version 0.4 · Preliminary Technical Research Note · Not peer reviewed
Abstract
This note examines a narrow failure boundary: a proposal can satisfy arithmetic constraints while relying on an incorrect configured assumption. In the published record, a $1,600 proposal passed validation against an intentionally incorrect $2,000 budget, although the designated reference fixture specified $1,200. A separate evidence gate held the conflicting structured claims and treated agreement under different declared lineages as consistent but unverified. The contribution is an inspectable case study of the distinction between internal consistency and evidence authority. The tests were author-designed, and this analysis does not independently reproduce their execution. Source authenticity and performance on unseen inputs remain unresolved. [1][2]
1. Research question and context
What does a successful validation result establish when the information being validated may be wrong? The question matters whenever software evaluates a proposal against configured values or combines claims from documents. A correct calculation answers whether values fit a rule. It cannot, by itself, establish whether those values accurately describe the world.
NIST’s AI Risk Management Framework treats validity and reliability as characteristics that require evaluation in the intended context of use. That provides a useful reason to keep a narrow test result separate from broader claims of trustworthiness. This note does not claim compliance with the framework. [5]
The W3C PROV data model represents provenance through entities, activities, agents, and relationships such as derivation. Such records can help describe where a claim came from. Our interpretation is that recording a lineage assertion and authenticating that assertion remain separate tasks. [6]
What this case study adds
The contribution is a documented comparison of two kinds of acceptance: arithmetic consistency under a configured budget, and consistency among submitted source claims. The wrong-budget case makes the distinction concrete: a $1,600 proposal is acceptable under the supplied $2,000 pool while exceeding the designated $1,200 reference by $400. The separate gate’s conflict response shows that the source disagreement remained detectable in structured inputs even when the calculation passed. This comparison is the contribution; no improvement over another system was measured. [1][2]
For practitioners reviewing automated recommendations, the case supplies an example of why a receipt should identify its assumptions and the scope of its checks. Its scientific scope is a case study, without a claim of algorithmic novelty or a new general theory. The experimental subjects were deterministic checks. Their relevance to AI-assisted workflows lies in the checks around a recommendation, rather than measured behavior of an AI model.
2. Materials and method
The evidence comprises publicly released plans, synthetic fixtures, redacted machine-readable result records, and an editorial summary of a documented review. Internal document identifiers have been replaced with stable synthetic labels; numerical values and reported outcomes are preserved. The review summary omits private workspace references, implementation specifics, and future proposals. The repository snapshot examined was commit b52116bd655a9b9447c4d7619237f8cbd2350151. Links below are pinned to that snapshot so readers can inspect the same materials. Both result records are dated October 9, 2026. This analysis checks the relationship between the reported inputs, expected outcomes, actual outcomes, and stated limitations; it does not authenticate the original execution environment. [1][2][3][4]
All budget amounts and source descriptions were invented test data. No real household finances or spending decisions are represented. The experiment materials describe predictions recorded before execution, but their timing has not been independently certified in this analysis. The same test author designed the fixtures, expected outcomes, and gate. Consequently, matching predictions measures agreement with the authored specification, rather than accuracy on unseen cases. [3][4]
Experiment 06 exercised a structured validator with a synthetic $1,200 budget, a $900 equipment item, and a $700 activity item. Experiment 07 used four structured source fixtures to test disagreement, repeated lineage, separately declared lineage, duplicate sources, and a simulated hash mismatch. Its evidence gate was a separate test harness, rather than an installed upgrade to the validator. Both records report that real execution remained disabled and trusted records were unchanged. [1][2]
3. Recorded observations and their limits
The observations below are reported outcomes in the public records, not results of an independent replication. The same author designed the cases and expected responses. They establish behavior on those cases only. Different declared lineages were not authenticated, and the validator and evidence gate were tested separately. [1][2][4]
| Condition | Reported outcome | Supported reading of this case |
|---|---|---|
| $900 proposal against $1,200 pool | Constraints valid | The structured proposal fits the configured constraints. |
| $1,600 proposal against $1,200 pool | Hold: reserve floor violation | The overspend was rejected under that configuration. |
| $1,600 proposal against intentionally wrong $2,000 pool | Constraints valid | The recorded validator response accepted this proposal under the intentionally wrong assumption. |
| Sources assert $1,200 and $2,000 | Hold: conflicting values | The recorded gate response held this disagreement in the specified structured field. |
| Agreeing sources share declared lineage | Hold: correlated sources | Agreement is insufficient under the gate’s lineage rule. |
| Agreeing sources declare different lineages | Review only: consistent, unverified | Agreement permits review; it does not establish truth. |
Table 1 summarizes selected cases from the raw result records. The records report six of six and ten of ten authored predictions matched. These counts describe agreement with the test specification; they are not accuracy estimates on unseen inputs. Each case can be inspected in the linked JSON records, including its inputs, prediction, and reported outcome. [1][2]
The most informative case is the structurally valid proposal using the wrong $2,000 pool. Its arithmetic is coherent: $2,000 minus $1,600 leaves $400. The recorded result illustrates the validator’s dependence on its configured assumption. Comparing that result with a separate conflict hold demonstrates a difference between two checks; it does not demonstrate that they operated together as one enforced pipeline. [1][2]
The evidence gate also ignored an embedded instruction in ordinary source prose. The record attributes this to a whitelist parser accepting specified structured fields. This was not a test of an LLM resisting an indirect prompt injection. Research by Greshake and colleagues examines how externally supplied content can carry instructions into LLM-integrated applications; that threat is relevant background, but it is broader than the parser behavior evaluated here. [2][7]
4. Interpretation and practical implications
Three questions should remain distinct: Is the calculation internally consistent? Do the submitted sources agree under defined comparison rules? Has the asserted information been independently established? The reported experiments directly exercise the first two questions within a small synthetic setting. They leave the third unresolved.
A label such as “current,” a recent timestamp, or a different lineage name is itself a claim when supplied by an unauthenticated source. In these fixtures, the gate did not allow self-described freshness to resolve a conflicting value. Likewise, two agreeing values with different declared origins received a review-only result. This is a useful conservative behavior in the tested cases, but it should not be interpreted as authenticated independence. [2]
The practical implication is a reporting discipline: a validation receipt should state precisely what was checked. “Constraints valid” should not silently become “source verified.” An evidence hold should remain visible instead of being collapsed into a generic success indicator. These are interpretations of the observed boundary, rather than demonstrated improvements in real-world decision quality.
5. Limitations
The evaluation is small, author-designed, and non-blind. It includes no held-out corpus, independently adjudicated ground truth, statistical estimate of error, or deployment outcome. Structured-field comparison does not demonstrate general natural-language contradiction detection. Declared lineages were fictional and unauthenticated. The hash-mismatch check deliberately supplied an invalid expected hash while leaving the source documents unchanged; it does not establish an independently anchored tamper-detection process. The public review is a documented critique, not external academic peer review. [2][4][8]
Because implementation code is not included in the public package, readers can inspect inputs, predictions, and recorded outputs, but cannot fully reproduce the original execution from the released materials alone. This note makes no claim of a novel algorithm, production readiness, comprehensive security, or independently authenticated multi-agent behavior. [8]
6. Verification, reproducibility, and stronger evidence
What can a reader verify now?
A reader can inspect the pinned input values and reported outcomes, recalculate the budget example, compare the recorded predictions with their results, and examine the limitations acknowledged in the review. For example, $900 + $700 = $1,600; against $2,000 this leaves $400, while against the designated $1,200 reference it exceeds the pool by $400. This independently checkable arithmetic supports the interpretation of the recorded case. It does not authenticate that the original software executed as reported. [1][3][8]
There are three different levels of reproducibility here. The arithmetic can be reproduced from the printed values. The consistency of the published record can be audited from the linked files. The original software execution cannot be fully reproduced from this public package because implementation code and its execution environment are absent. These levels should not be conflated.
Why are these checks useful if the outcomes were expected?
Designed checks are useful for documenting whether a particular implementation followed its intended rules. Their expected outcomes also make them weak evidence for generalization. In this study, the useful result is the visible boundary between the two reported statuses, including the deliberately wrong assumption accepted by the validator. The perfect prediction-match counts provide no additional basis for claiming reliability on unfamiliar cases.
What evidence would change the interpretation?
The central empirical description would need correction if the released record did not contain the wrong-budget acceptance, if the validator authenticated and enforced the designated reference before accepting it, or if the compared responses came from materially different inputs than those described. A rerun that failed to reproduce the recorded responses would weaken confidence in the implementation-specific findings. Even a successful rerun would leave real-world source truth and generalization unresolved.
What would justify stronger claims?
A small executable example using synthetic data could demonstrate the same arithmetic and source-comparison distinction without disclosing a larger system. If newly written, it would need to be labeled a separate demonstrator, rather than a reproduction of the original implementation. Reproducing the original experiment would require an executable release of the relevant implementation, dependencies, configuration, and fixture bindings.
Claims of broader performance would require independently authored held-out cases, explicit outcome criteria, and reporting of failures as well as successes. Claims of independent provenance would require evidence beyond labels within the fixtures. Claims that evidence checks improve decisions would require an integrated evaluation comparing outcomes with and without those checks. None of these stronger evaluations is supplied by this note.
7. Conclusion
The published synthetic records support a modest, concrete finding: arithmetic validity and evidence authority are different properties. A valid calculation can inherit a wrong assumption, and source agreement can remain unverified. The contribution is a documented comparison of the two recorded statuses and an explanation of which parts readers can audit. This supports precise reporting of validation scope; it does not establish improved decision quality.
Data availability and preparation disclosure
The cited experiment package is publicly available at the pinned repository snapshot. The analysis is limited to this public package. Draft preparation and citation checking used AI assistance. The paper is an analysis of recorded experiments, not a new independent replication.
Questions about output validation and evidence
Does a passed validation mean the evidence is trustworthy?
No. A correct calculation shows that supplied values fit a rule, not that the values are true. In this synthetic record, a $1,600 proposal passed against an intentionally wrong $2,000 budget, even though the designated reference was $1,200. Passing the arithmetic check did not authenticate the budget.
What can a reader verify now?
Readers can inspect the pinned inputs and reported outcomes, recalculate the arithmetic, and compare the recorded predictions with the results. The released materials do not include the implementation and environment needed to fully reproduce the original software execution.
Why are these checks useful if the outcomes were expected?
They document whether the implementation followed its authored rules on these cases and make the boundary between arithmetic validity and evidence authority visible. Matching expected outcomes does not establish accuracy on unfamiliar inputs.
How does this relate to LLM output validation?
LLM output validation may check whether a model’s recommendation fits a format or configured rule. This note illustrates why passing such a check should not be treated as source verification. Its experiments tested deterministic checks around synthetic recommendations; they did not measure an LLM’s behavior.
What does validation versus verification mean in this note?
The distinction here is narrow: internal consistency, agreement among submitted source claims, and independently established evidence are different properties. This is not a general definition of those terms for every engineering discipline. Source provenance records where information is said to come from; the record still needs authentication.
Related research
- When Agreement Isn’t Verification
- Preserving Disagreement
- What Does It Mean to Test an AI System?
- The Problem with AI Agents That Agree with Each Other
- Explore the research notebook
References
- Experiment 06 raw results (2026).
- Experiment 07 raw results (2026).
- Experiment 06 recorded test plan (2026).
- Experiment 07 recorded test plan (2026).
- NIST (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, section 3.1.
- Moreau, L., and Missier, P., editors (2013). PROV-DM: The PROV Data Model. W3C Recommendation, 30 April 2013.
- Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., and Fritz, M. (2023). Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173.
- Public editorial summary of the documented Experiment 07 review (2026). Self-reported reviewer identity; not external peer review.