# Building autopilot

> Canonical page: https://briefcase.so/blog/building-autopilot
> Briefcase content index: https://briefcase.so/llms.txt

Probabilistic AI agents paired with deterministic formal verification create the perfect combination for high trust domains like finance and accounting.

[All Posts](https://briefcase.so/blog)

![Building autopilot](https://briefcase.so/marketing/blog/building-autopilot.png)

Engineering

9 November 202515 min read

01

## Introduction

Probabilistic AI agents paired with deterministic formal verification create the perfect combination for high trust domains like finance and accounting. At Briefcase, we run 12 AI agents in production, but to truly automate critical accounting workflows, we built Autopilot - our deterministic verification system that evaluates AI results in real-time and learns from human corrections. After 6 weeks in production, here's how we built it, why determinism matters, and the impact we've seen.

02

## Invoice Processing AI Agent

Briefcase automates the complete invoice capture and bookkeeping workflow:

1.  Input processing: Unstructured data (emails, images, PDFs) containing invoices/receipts
2.  Data extraction: Core transaction information
3.  Line item split: Individual item separation and processing
4.  Classification: Transaction categorisation
5.  Tax determination: VAT rate calculation per line item

![Example trace of cost transaction processing](https://briefcase.so/marketing/blog/building-autopilot-inline-1.png)

Example trace of our cost transaction processing AI agent combining LLM and business logic chains

The agent outputs fully processed transactions ready for the general ledger. Autopilot then determines whether to auto-publish or require human review based on three criteria:

1.  Document legibility
2.  Processing completeness
3.  Historical consistency

03

## Document Legibility

Uploaded documents can vary widely in quality so checking legibility is essential to ensure reliable downstream processing.

Visual transformers tokenize images into patches (16×16 or 32×32 pixels), flatten them into 1D vectors, add positional embeddings for spatial awareness, stack them into 2D matrices, and feed them through attention mechanisms. To transformers, images are just collections of vectors, they can't inherently distinguish between blurry and legible documents.

To get around this we run extraction multiple times. Identical results across runs indicate high document legibility confidence.

04

## Processing Completeness

This straightforward guardrail verifies:

-   Is all necessary data extracted?
-   Is transaction from an existing supplier?
-   Is document type postable to general ledger (invoice/receipt/credit note)?

All checks must pass for this guardrail to succeed.

05

## Historical Consistency

Now onto the fun stuff. Historical consistency enables pattern recognition and learns from corrections to determine auto-publishing safety. Our design constraints:

1.  Deterministic: Consistent results regardless of run count
2.  Generalisable: Serves diverse businesses (B2B SaaS, retail, hospitality)
3.  Business-isolated: Patterns from Business X never affect Business Y
4.  Self-improving: Adjusts confidence based on human corrections
5.  Recency-weighted: Recent transactions matter more than older ones

06

## Why Not LLM as a Judge?

LLMs exhibit U-shaped confidence distributions rather than normal distributions and perform poorly on out-of-distribution examples (adversarial examples). Not to mention that it's quite costly. LLM as a judge as it stands today has its place in data labelling, but less so in critical evaluation step determining if human should be in the loop or not.

07

## Our Approach: Bayesian Statistical Model

We evaluated two options:

1.  Simple statistical model
2.  Individual ML models per business

Option 2's complexity made it unsuitable for rapid iteration and baseline establishment. We chose a Bayesian approach because it's a much simpler starting point and can be as effective.

The algorithm:

1.  Retrieve semantically similar published transactions
2.  Calculate evidence
3.  Calculate prior
4.  Perform Bayesian update
5.  Make autopilot decision

08

## Implementation Details

### Similar Transactions Retrieval

We use RAG with reranking, maintaining isolated namespaces per business to ensure pattern learning remains business-specific.

More detailed overview about our RAG approach will be covered in future posts and is out of scope of this blog.

### Evidence Calculation

Field weights reflect relative importance:

fieldWeights.ts

```
1export const FIELD_WEIGHTS = {2  supplier: 0.2,3  amountRange: 0.05,4  lineItemCount: 0.15,5  category: 0.25,6  taxRate: 0.25,7  quantity: 0.05,8  description: 0.05,9}
```

Recency decay ensures recent patterns take precedence:

```
1const decay = 0.5 ** ((months - gracePeriod) / halfLife)
```

With example grace period = 3 months and half-life = 12 months:

-   3-month-old transaction: weight = 1.0
-   12-month-old transaction: weight = 0.5
-   24-month-old transaction: weight = 0.25

We normalise weights using softmax to create a proper distribution:

softmax.ts

```
1export const softmax = ({2  weights,3  gamma,4}: {5  weights: number[]6  gamma: number7}): number[] => {8  const exponents = weights.map((weight) => Math.exp(gamma * weight))9  const sum = exponents.reduce((a, b) => a + b, 0)10 11  return exponents.map((exponent) => exponent / (sum || 1))12}
```

Field matching uses tiered scoring:

```
1// supplier match calculation2let score: number3if (matchRate >= 0.85) {4  score = 1.05} else if (matchRate >= 0.7) {6  score = 0.87} else if (matchRate >= 0.55) {8  score = 0.49} else {10  score = 011}
```

Our internal debugging tool visualises individual decisions:

![Internal debugging tool visualization](https://briefcase.so/marketing/blog/building-autopilot-inline-2.png)

Internal debugging tool visualization of individual decisions

09

## Prior Calculation

Constants control learning behaviour:

constants.ts

```
1export const BASE_PRIOR = 0.72export const MIN_DAYS_SINCE_AUTO_PUBLISH = 73export const POSITIVE_ADJUSTMENT_FACTOR = 0.03 // No correction → increase confidence4export const NEGATIVE_ADJUSTMENT_FACTOR = 0.05 // Correction → decrease confidence
```

Human corrections directly influence future decisions:

```
1let adjustedPrior = basePrior2 3for (const transactions of autoPublishedTransactions) {4  if (transaction.humanCorrected) {5    adjustment = -negativeAdjustmentFactor * recencyWeight6  } else {7    adjustment = positiveAdjustmentFactor * recencyWeight8  }9  adjustedPrior += adjustment10}
```

Based on these adjustments we calculate new prior:

```
1const adjustedPrior = Math.max(0, Math.min(1, adjustedPrior))
```

10

## Bayesian Probability Calculation

bayesian.ts

```
1export const calculatePosteriorProbability = ({2  prior,3  evidence,4  epsilon = 1e-6,5}: {6  prior: number7  evidence: number8  epsilon?: number9}): number => {10  const p = Math.min(1 - epsilon, Math.max(epsilon, prior))11  const e = Math.min(1 - epsilon, Math.max(epsilon, evidence))12  const numerator = e * p13  const denominator = numerator + (1 - e) * (1 - p)14 15  return numerator / denominator16}
```

The epsilon prevents division by zero while maintaining numerical stability.

![Bayesian probability visualization](https://briefcase.so/marketing/blog/building-autopilot-inline-3.png)

### Autopilot Decision

Now that we have posterior we need to establish autopilot threshold. This should be tuned empirically and in following example it's 0.8. Converging to a good threshold requires lots of tuning and analysis but usual rule of thumb is to start more conservative first and then relax it as you learn more about how it actually performs in production.

![Autopilot decision threshold](https://briefcase.so/marketing/blog/building-autopilot-inline-4.png)

11

## Result

Using Briefcase for our own bookkeeping, Autopilot reduced manual effort by over 80%, leaving only edge cases for review.

![Auto-published transactions](https://briefcase.so/marketing/blog/building-autopilot-inline-5.png)

Magic wand icons indicate auto-published transactions

![Decision explainability](https://briefcase.so/marketing/blog/building-autopilot-inline-6.png)

Every decision includes full explainability to build user trust

12

## Key Takeaways

Deterministic heuristics complement probabilistic AI agents exceptionally well, enabling true end-to-end automation in high trust domains. Our approach successfully:

-   Maintains determinism through Bayesian statistics
-   Generalises across business types
-   Isolates learning per business
-   Improves through human feedback
-   Weights recent data appropriately

We're not stopping here and are gonna push major improvements (including completely new paradigms) to our AI agents and formal verification mechanisms in 2026. Join us to build robust, scalable, AI-native platform that transforms entire industry.

[

PreviousIntroducing AI Autopilot

](https://briefcase.so/blog/introducing-ai-autopilot)[

NextWhy build what you can buy?

](https://briefcase.so/blog/why-build-what-you-can-buy)

01

## Introduction

Probabilistic AI agents paired with deterministic formal verification create the perfect combination for high trust domains like finance and accounting. At Briefcase, we run 12 AI agents in production, but to truly automate critical accounting workflows, we built Autopilot - our deterministic verification system that evaluates AI results in real-time and learns from human corrections. After 6 weeks in production, here's how we built it, why determinism matters, and the impact we've seen.

02

## Invoice Processing AI Agent

Briefcase automates the complete invoice capture and bookkeeping workflow:

1.  Input processing: Unstructured data (emails, images, PDFs) containing invoices/receipts
2.  Data extraction: Core transaction information
3.  Line item split: Individual item separation and processing
4.  Classification: Transaction categorisation
5.  Tax determination: VAT rate calculation per line item

![Example trace of cost transaction processing](https://briefcase.so/marketing/blog/building-autopilot-inline-1.png)

Example trace of our cost transaction processing AI agent combining LLM and business logic chains

The agent outputs fully processed transactions ready for the general ledger. Autopilot then determines whether to auto-publish or require human review based on three criteria:

1.  Document legibility
2.  Processing completeness
3.  Historical consistency

03

## Document Legibility

Uploaded documents can vary widely in quality so checking legibility is essential to ensure reliable downstream processing.

Visual transformers tokenize images into patches (16×16 or 32×32 pixels), flatten them into 1D vectors, add positional embeddings for spatial awareness, stack them into 2D matrices, and feed them through attention mechanisms. To transformers, images are just collections of vectors, they can't inherently distinguish between blurry and legible documents.

To get around this we run extraction multiple times. Identical results across runs indicate high document legibility confidence.

04

## Processing Completeness

This straightforward guardrail verifies:

-   Is all necessary data extracted?
-   Is transaction from an existing supplier?
-   Is document type postable to general ledger (invoice/receipt/credit note)?

All checks must pass for this guardrail to succeed.

05

## Historical Consistency

Now onto the fun stuff. Historical consistency enables pattern recognition and learns from corrections to determine auto-publishing safety. Our design constraints:

1.  Deterministic: Consistent results regardless of run count
2.  Generalisable: Serves diverse businesses (B2B SaaS, retail, hospitality)
3.  Business-isolated: Patterns from Business X never affect Business Y
4.  Self-improving: Adjusts confidence based on human corrections
5.  Recency-weighted: Recent transactions matter more than older ones

06

## Why Not LLM as a Judge?

LLMs exhibit U-shaped confidence distributions rather than normal distributions and perform poorly on out-of-distribution examples (adversarial examples). Not to mention that it's quite costly. LLM as a judge as it stands today has its place in data labelling, but less so in critical evaluation step determining if human should be in the loop or not.

07

## Our Approach: Bayesian Statistical Model

We evaluated two options:

1.  Simple statistical model
2.  Individual ML models per business

Option 2's complexity made it unsuitable for rapid iteration and baseline establishment. We chose a Bayesian approach because it's a much simpler starting point and can be as effective.

The algorithm:

1.  Retrieve semantically similar published transactions
2.  Calculate evidence
3.  Calculate prior
4.  Perform Bayesian update
5.  Make autopilot decision

08

## Implementation Details

### Similar Transactions Retrieval

We use RAG with reranking, maintaining isolated namespaces per business to ensure pattern learning remains business-specific.

More detailed overview about our RAG approach will be covered in future posts and is out of scope of this blog.

### Evidence Calculation

Field weights reflect relative importance:

fieldWeights.ts

```
1export const FIELD_WEIGHTS = {2  supplier: 0.2,3  amountRange: 0.05,4  lineItemCount: 0.15,5  category: 0.25,6  taxRate: 0.25,7  quantity: 0.05,8  description: 0.05,9}
```

Recency decay ensures recent patterns take precedence:

```
1const decay = 0.5 ** ((months - gracePeriod) / halfLife)
```

With example grace period = 3 months and half-life = 12 months:

-   3-month-old transaction: weight = 1.0
-   12-month-old transaction: weight = 0.5
-   24-month-old transaction: weight = 0.25

We normalise weights using softmax to create a proper distribution:

softmax.ts

```
1export const softmax = ({2  weights,3  gamma,4}: {5  weights: number[]6  gamma: number7}): number[] => {8  const exponents = weights.map((weight) => Math.exp(gamma * weight))9  const sum = exponents.reduce((a, b) => a + b, 0)10 11  return exponents.map((exponent) => exponent / (sum || 1))12}
```

Field matching uses tiered scoring:

```
1// supplier match calculation2let score: number3if (matchRate >= 0.85) {4  score = 1.05} else if (matchRate >= 0.7) {6  score = 0.87} else if (matchRate >= 0.55) {8  score = 0.49} else {10  score = 011}
```

Our internal debugging tool visualises individual decisions:

![Internal debugging tool visualization](https://briefcase.so/marketing/blog/building-autopilot-inline-2.png)

Internal debugging tool visualization of individual decisions

09

## Prior Calculation

Constants control learning behaviour:

constants.ts

```
1export const BASE_PRIOR = 0.72export const MIN_DAYS_SINCE_AUTO_PUBLISH = 73export const POSITIVE_ADJUSTMENT_FACTOR = 0.03 // No correction → increase confidence4export const NEGATIVE_ADJUSTMENT_FACTOR = 0.05 // Correction → decrease confidence
```

Human corrections directly influence future decisions:

```
1let adjustedPrior = basePrior2 3for (const transactions of autoPublishedTransactions) {4  if (transaction.humanCorrected) {5    adjustment = -negativeAdjustmentFactor * recencyWeight6  } else {7    adjustment = positiveAdjustmentFactor * recencyWeight8  }9  adjustedPrior += adjustment10}
```

Based on these adjustments we calculate new prior:

```
1const adjustedPrior = Math.max(0, Math.min(1, adjustedPrior))
```

10

## Bayesian Probability Calculation

bayesian.ts

```
1export const calculatePosteriorProbability = ({2  prior,3  evidence,4  epsilon = 1e-6,5}: {6  prior: number7  evidence: number8  epsilon?: number9}): number => {10  const p = Math.min(1 - epsilon, Math.max(epsilon, prior))11  const e = Math.min(1 - epsilon, Math.max(epsilon, evidence))12  const numerator = e * p13  const denominator = numerator + (1 - e) * (1 - p)14 15  return numerator / denominator16}
```

The epsilon prevents division by zero while maintaining numerical stability.

![Bayesian probability visualization](https://briefcase.so/marketing/blog/building-autopilot-inline-3.png)

### Autopilot Decision

Now that we have posterior we need to establish autopilot threshold. This should be tuned empirically and in following example it's 0.8. Converging to a good threshold requires lots of tuning and analysis but usual rule of thumb is to start more conservative first and then relax it as you learn more about how it actually performs in production.

![Autopilot decision threshold](https://briefcase.so/marketing/blog/building-autopilot-inline-4.png)

11

## Result

Using Briefcase for our own bookkeeping, Autopilot reduced manual effort by over 80%, leaving only edge cases for review.

![Auto-published transactions](https://briefcase.so/marketing/blog/building-autopilot-inline-5.png)

Magic wand icons indicate auto-published transactions

![Decision explainability](https://briefcase.so/marketing/blog/building-autopilot-inline-6.png)

Every decision includes full explainability to build user trust

12

## Key Takeaways

Deterministic heuristics complement probabilistic AI agents exceptionally well, enabling true end-to-end automation in high trust domains. Our approach successfully:

-   Maintains determinism through Bayesian statistics
-   Generalises across business types
-   Isolates learning per business
-   Improves through human feedback
-   Weights recent data appropriately

We're not stopping here and are gonna push major improvements (including completely new paradigms) to our AI agents and formal verification mechanisms in 2026. Join us to build robust, scalable, AI-native platform that transforms entire industry.

[

PreviousIntroducing AI Autopilot

](https://briefcase.so/blog/introducing-ai-autopilot)[

NextWhy build what you can buy?

](https://briefcase.so/blog/why-build-what-you-can-buy)