Testing 104: Property-based testing with Hypothesis

This is the fourth post in our testing series. The first laid out the map, the second put unit and end-to-end tests in the coding agent's loop, and the third asked whether those tests would notice a bug. This one is about the inputs nobody wrote down, and the outputs that are never the same twice.

Every test so far in this series picked its own input. line_total(200, ...), line_total(10, ...), and after mutation testing found the gap, line_total(100, ...). Someone chose those numbers. Real input isn't chosen. It arrives.

A supplier sends a CSV where twelve hundred and a half is written 1.200,50. An ERP export has a unit field that says EA on Monday, ea on Tuesday and nothing at all on Wednesday. An EDI message carries a line with quantity zero because someone cancelled it without deleting it. A supplier email confirms "2 pallets" against an order placed in cases. None of that is a bug in your code yet. It becomes one the moment your code assumes something the data never promised.

The data makes no promises. So stop trying to list every input, and write down what must be true of the output whatever the input was. That's a property, and the tool that checks it, property-based testing, generates hundreds of inputs trying to prove you wrong.

It matters more now than it did a few years ago, for two reasons. A coding agent writes the parser, and it writes the example tests from the same understanding, so both share the same blind spots. And an AI agent increasingly sits in the production path itself, reading those supplier emails and writing what it finds into the ERP. Its output is never byte-for-byte the same twice, so you can't check it with ==. What you can check is that it confirmed the lines that were ordered, in the units they were ordered in, with dates after the order was placed. The answer changes from run to run. The traits don't.

Examples check answers. Properties check traits.

An example-based test fixes an input and asserts an output. It's precise, and it's limited to the inputs you thought of. A property-based test describes a whole class of inputs and asserts something that must hold for every one of them. The framework then searches that class for a counterexample, and when it finds one, shrinks it to the smallest input that still fails. The idea comes from QuickCheck for Haskell;^[1] in Python, the mature implementation is Hypothesis.^[2] It's no longer a research curiosity, either: an interview study of engineers at Jane Street looks at how it's used in day-to-day work there.^[7]

Here is the shape of it on a unit conversion from the purchase-to-delivery system we've used all series. Orders come in eaches, cases of 12 or pallets of 480, and everything is stored in eaches.

from decimal import Decimal
from hypothesis import given, strategies as st

PACK = {"EA": 1, "CS": 12, "PAL": 480}

def to_each(qty: Decimal, uom: str) -> Decimal:
    return qty * PACK[uom.strip().upper()]

def from_each(qty: Decimal, uom: str) -> Decimal:
    return qty / PACK[uom.strip().upper()]

quantities = st.decimals(min_value=0, max_value=100_000, places=2)
units = st.sampled_from(list(PACK)).flatmap(
    lambda u: st.sampled_from([u, u.lower(), f" {u} "])
)

@given(qty=quantities, uom=units)
def test_converting_to_eaches_and_back_changes_nothing(qty, uom):
    assert from_each(to_each(qty, uom), uom) == qty

Two things are worth noticing. The assertion doesn't contain a single expected number; it states a rule, a round-trip, that has to hold for all of them. And the units strategy is where you write down what the data might do: upper case, lower case, stray whitespace. The strategy is the honest description of your input. The property is the honest description of your contract.

The traits worth writing down

The hard part of property-based testing is never the tool. It's deciding what must always be true. In operational data, the useful properties come in a small number of recurring shapes, and most of them are easy to explain to someone who has never written a test:

The last row deserves its own warning, because it's the cheapest property to write and one of the most expensive to skip. In 1999 NASA lost the Mars Climate Orbiter because one piece of ground software produced impulse in pound-force seconds and the next one read it as newton seconds.^[3] Every number was well formed. Nothing crashed. The unit was simply never checked at the seam.

Type and bounds: a clean value or an explicit rejection, never a third thing

The most useful property for inbound data is also the bluntest. Throw the worst values you can think of at the normaliser, and require that it either returns a line downstream code can trust or raises the one error you've decided means "send this to a person". Not a KeyError, not a float that slipped through, not a negative quantity.

from decimal import Decimal
from hypothesis import given, strategies as st

raw_quantities = st.one_of(
    st.integers(min_value=-1_000, max_value=1_000_000),
    st.floats(allow_nan=True, allow_infinity=True),
    st.decimals(allow_nan=True),
    st.sampled_from(["1,200.50", "1.200,50", "12 pcs", "", "  "]),
    st.text(max_size=12),
    st.none(),
)
raw_units = st.one_of(
    st.sampled_from(["EA", "ea", " CS", "PAL", "PALLET", "Stk", ""]),
    st.text(max_size=6),
    st.none(),
)

@given(qty=raw_quantities, uom=raw_units)
def test_normalise_returns_a_clean_line_or_rejects_it(qty, uom):
    try:
        line = normalise_line({"sku": "SKU-42", "qty": qty, "uom": uom})
    except RejectedLine:
        return

    assert isinstance(line.qty, Decimal)
    assert line.qty.is_finite() and line.qty >= 0
    assert line.uom in {"EA", "CS", "PAL"}
    assert normalise_line(line.as_raw()) == line  # idempotent

The last assertion is an idempotence check, and it's there for a practical reason: in a real pipeline the same record often passes through normalisation more than once, after a retry, a replay or a re-import. If normalising a clean line changes it, you have a bug that only shows up on the second pass.

Always growing: properties over a sequence of events

Some traits only make sense over time. Total quantity received for an order should only ever go up. Stock on hand should never go below zero, whatever order the receipts, shipments and corrections arrive in. A single function call can't show you that. Hypothesis's stateful testing can: you describe the operations, and it generates sequences of them, checking invariants after every step.

from decimal import Decimal
from hypothesis import strategies as st
from hypothesis.stateful import RuleBasedStateMachine, invariant, rule

amounts = st.decimals(min_value=0, max_value=500, places=0)

class ReceivingLedger(RuleBasedStateMachine):
    def __init__(self):
        super().__init__()
        self.ledger = Ledger(sku="SKU-42")
        self.received_so_far = Decimal(0)

    @rule(qty=amounts)
    def receive(self, qty):
        self.ledger.receive(qty)

    @rule(qty=amounts)
    def ship(self, qty):
        try:
            self.ledger.ship(qty)
        except InsufficientStock:
            pass

    @rule(qty=amounts)
    def correct_a_miscount(self, qty):
        self.ledger.correct(on_hand=qty)

    @rule(event_id=st.sampled_from(["gr-1", "gr-2", "gr-3"]))
    def replay_receipt_webhook(self, event_id):
        self.ledger.apply_receipt(event_id, qty=Decimal(10))

    @invariant()
    def total_received_only_grows(self):
        assert self.ledger.total_received >= self.received_so_far
        self.received_so_far = self.ledger.total_received

    @invariant()
    def stock_is_never_negative(self):
        assert self.ledger.on_hand >= 0

TestReceivingLedger = ReceivingLedger.TestCase

This is the kind of test that finds the bugs nobody writes examples for. A stock correction implemented as a negative receipt, so total received goes down. A webhook that's applied twice when the carrier retries it. A shipment that's allowed to go through because the correction that preceded it was applied in the wrong order. Each failure comes back as a short, shrunk sequence of steps you can read like a bug report.

Shape: as much comes out as went in

The simplest structural property is also one of the most frequently broken: a function that takes a list of things should usually return a list of the same length, in the same order, with each result still referring to its input. A function that drops a line it can't allocate, or merges two lines for the same SKU, will pass most example tests and quietly corrupt an order.

from hypothesis import given, strategies as st

SKUS = ["SKU-1", "SKU-2", "SKU-42"]

order_lines = st.builds(
    OrderLine,
    line_id=st.uuids().map(str),
    sku=st.sampled_from(SKUS),
    qty=st.decimals(min_value=1, max_value=1_000, places=0),
    uom=st.sampled_from(["EA", "CS", "PAL"]),
)
stock_levels = st.dictionaries(
    st.sampled_from(SKUS), st.decimals(min_value=0, max_value=2_000, places=0)
)

@given(lines=st.lists(order_lines, max_size=30), stock=stock_levels)
def test_one_allocation_per_line_in_the_same_order(lines, stock):
    allocations = allocate(lines, stock)

    assert [a.line_id for a in allocations] == [l.line_id for l in lines]
    for a, l in zip(allocations, lines):
        assert a.uom == l.uom
        assert 0 <= a.qty <= l.qty

Note what the test doesn't say: it doesn't say how much each line should get. That's the allocation policy, and it belongs in example tests and behavioral scenarios. The property only says what any correct policy must respect, which is exactly why it survives a change of policy without being rewritten.

Hypothesis and pytest

Hypothesis has been the standard way to do this in Python for a decade, and the reason it fits into existing suites so easily is that it isn't a separate test runner. It ships its own pytest plugin. A function decorated with @given is an ordinary pytest test: it's collected, selected with -k, marked, parametrised and reported like any other. There is nothing to configure to get started beyond installing the package.^[8]

A few parts of that integration are worth knowing on day one:

• Profiles. How hard Hypothesis searches is a setting, and you want different answers locally and in CI. Register profiles once in conftest.py and pick one per run.
• The example database. Every failing input is saved under .hypothesis/ and replayed first on the next run, so a bug found once is checked every time until it's fixed. Cache that directory in CI. When a failure matters enough to keep forever, pin it with @example(...).
• Statistics. --hypothesis-show-statistics tells you how many examples ran, how many were discarded and where the time went, which is how you notice a strategy that's quietly filtering away most of what it generates.
• Fixtures. Fixtures work with @given, but a function-scoped fixture is set up once per test, not once per generated example. Hypothesis warns you with a health check when that's likely to be a mistake; take the warning seriously, because shared state between examples makes failures impossible to reproduce.
• Reproducing. A failure prints the shrunk falsifying example. --hypothesis-seed lets you rerun a whole run exactly, and print_blob=True prints a blob you can paste into @reproduce_failure to replay one failing case from a CI log.
• Running in parallel. Property tests are slower than example tests by design, so they're the first place a serial suite starts to hurt. rstest, our open-source parallel runner, runs Hypothesis tests unchanged: @given, profiles, --hypothesis-profile and --hypothesis-seed all work as they do under pytest, with each worker generating its own examples. The one thing to watch is the shared .hypothesis/ database. Hypothesis handles concurrent access itself, but rstest hasn't verified it above eight workers. If you see contention, give each worker its own database in your profile with DirectoryBasedExampleDatabase(f".hypothesis/{os.environ.get('PYTEST_XDIST_WORKER', 'master')}"), or set database=None in CI.

# conftest.py
import os
from hypothesis import HealthCheck, settings

settings.register_profile("dev", max_examples=50)
settings.register_profile("ci", max_examples=1_000, print_blob=True)
settings.register_profile(
    "agent",
    max_examples=25,
    deadline=None,
    suppress_health_check=[HealthCheck.too_slow],
)
settings.load_profile(os.getenv("HYPOTHESIS_PROFILE", "dev"))

pytest tests/properties --hypothesis-profile=ci --hypothesis-show-statistics

Hypothesis also helps you get going: hypothesis write reads a function's signature and writes a first property test for it, and st.builds and st.from_type infer strategies from type hints and dataclasses. Treat both as a starting point. The generated strategies describe the types, not your data; it's your job to make them as messy as reality.

Agent outputs are never the same twice. Their traits are.

Now point the same idea at an agent. Take the one we've used as an example all series: it reads a supplier's reply to a purchase order and extracts what was confirmed. Run it three times on the same email and you can get three different results at the byte level: notes worded differently, lines in a different order, a date written as 2026-10-14 once and resolved from "Wednesday next week" the next. An exact-match assertion would fail on every run and tell you nothing.

But the things that matter don't vary. Whatever the wording, the output must refer only to lines that exist on the order. It must not drop a line silently, or confirm the same line twice. A line ordered in cases must be confirmed in cases, not quietly converted to eaches or, worse, to pallets. A quantity must still be a quantity and not a price that got read from the wrong column. A delivery date can't come before the order date. Those are properties, and you write them exactly like the ones above:

from decimal import Decimal

OVERDELIVERY_TOLERANCE = Decimal("1.10")

def check_confirmation(po: PurchaseOrder, result: Confirmation) -> None:
    ordered = {line.line_id: line for line in po.lines}
    confirmed = [c.line_id for c in result.lines]

    # shape: nothing invented, nothing doubled, nothing dropped silently
    assert len(confirmed) == len(set(confirmed))
    assert set(confirmed) <= ordered.keys()
    assert set(confirmed) | set(result.unconfirmed) == ordered.keys()

    for c in result.lines:
        line = ordered[c.line_id]
        # unit of measure: confirmed in the unit it was ordered in
        assert c.uom == line.uom
        # dimension: a non-negative quantity, within tolerance of what was ordered
        assert isinstance(c.qty, Decimal) and c.qty >= 0
        assert c.qty <= line.qty * OVERDELIVERY_TOLERANCE
        # time: a delivery can't be confirmed for before the order existed
        assert c.delivery_date >= po.order_date

The same reasoning applies to anything an agent or a model produces in bulk. A demand-forecasting step asked for a 12-week horizon must return exactly 12 values, not 11 because it ran out of tokens or 13 because it added a summary row. A classification step over 40 invoice lines must return 40 labels. An embedding must have the dimension the vector index was built for. None of those checks needs to know what the right answer is. They only need to know what any right answer looks like.

@given(
    history=st.lists(st.decimals(min_value=0, max_value=10_000, places=0), min_size=12, max_size=104),
    horizon=st.integers(min_value=1, max_value=26),
)
def test_forecast_has_one_value_per_week_in_the_horizon(history, horizon):
    forecast = forecast_step.run(history=history, horizon_weeks=horizon)

    assert len(forecast.values) == horizon
    assert forecast.uom == "EA"
    assert all(v >= 0 for v in forecast.values)

Running Hypothesis against something non-deterministic

There is one real complication, and it's worth being precise about. Hypothesis assumes that the code under test is deterministic: when it finds a failure, it replays and shrinks it, many times, expecting the same input to fail the same way. A live model doesn't. Left on default settings, you'd spend most of your model calls on shrinking, and a failure that doesn't reproduce on replay is reported as flaky rather than as a clean counterexample.

So for tests that call a live agent, change three things. Generate fewer, richer inputs. Skip the shrink phase, because shrinking a non-deterministic failure isn't meaningful. And run each generated input more than once, because a trait has to hold on every sample, not on average.

from hypothesis import Phase, given, settings, strategies as st

@settings(
    max_examples=25,
    deadline=None,
    phases=[Phase.explicit, Phase.reuse, Phase.generate],
)
@given(po=purchase_orders(), style=st.sampled_from(EMAIL_STYLES))
def test_agent_confirmations_keep_their_traits(po, style):
    email = render_supplier_reply(po, style=style)
    for _ in range(3):
        check_confirmation(po, extraction_agent.run(email=email, po=po))

The generated input here isn't raw text. Hypothesis generates the purchase order, a structured object it's good at, and render_supplier_reply turns it into an email in one of several styles: terse, forwarded thread, a pasted table, German, quantities written out in words. Because the test knows the order it started from, it has an independent oracle for every trait, without anyone having labelled a single email by hand.

If a trait breaks on one of the three samples and holds on the replay, Hypothesis will report it as flaky. Treat that as a failure, not as noise. A unit of measure that's right two times out of three is a unit of measure that's wrong in production once a day.

You can push this further with metamorphic properties, which compare two runs instead of checking one.^[4] Renaming the supplier in the email must not change the confirmed dates. Reordering the lines in a pasted table must not change which lines were confirmed. It's the invariance idea from the CheckList work we mentioned in the first post,^[5] expressed as Hypothesis tests, and it catches a model that's latching onto the wrong part of the input.

Write the property once. Run it in three places.

The most useful thing about check_confirmation is that it's an ordinary function. It doesn't know whether it's being called from a test. That means the same definition of "a sane confirmation" can do three jobs:

• In the test suite, against inputs Hypothesis generates, on the agent profile in a scheduled job, and against recorded agent responses in the fast suite so the deterministic code around the agent is property-tested on every commit.
• Over recorded traces, as a regression check: every confirmation the agent produced last week, rechecked against today's properties. When you add a property after an incident, this tells you how often it had already been broken.
• In production, as a guardrail. Before the agent's output is written into the ERP, the same check runs on it. If it fails, nothing is written and the case goes to a person. (In production, raise a real exception rather than relying on assert, which Python drops under -O.) In a test, a property finds bugs. In production, it keeps an agent inside the bounds you set for it.

That last point is where property-based thinking stops being a testing technique and becomes part of how you run agents. An agent that's allowed to act on its own needs a definition of "obviously wrong" that's precise enough to execute. Properties are that definition.

Where properties come from when the domain is messy

The first post promised an answer to the question people actually get stuck on: what are the properties? In operational systems, they're usually already written down somewhere, just not as tests.

• The rules people say out loud. "We never ship more than was ordered." "A cancelled line stays cancelled." Every one of those sentences is a property.
• The reconciliations finance already runs. Goods received against goods invoiced, stock counted against stock on record. A reconciliation is a conservation property that someone checks by hand once a month.
• Incident reviews. Almost every incident is a property nobody wrote. Write it after the fix, and run it over historical data to see how long it had been broken.
• The type signature. If the function says it returns Decimal, check that it does for every input, not just the three in the example tests.
• The old implementation. When you replace a slow allocator with a fast one, or a rules engine with an agent, the strongest property you have is "gives the same answer as the old one" on every input where the old one was right.
• Surviving mutants. When fermut reports a survivor on a boundary or a unit conversion, that's usually a property asking to be written. One good property tends to kill a whole cluster of mutants that would each need their own example test.

This is also work coding agents are good at, provided the split is right. Writing strategies that generate realistic messy data is tedious for people and cheap for an agent, and it's been shown to work at scale: an agent writing Hypothesis tests across 100 popular Python packages produced bug reports of which 56% were judged to be valid bugs, including real ones in NumPy and cloud SDKs.^[6] What the agent shouldn't do on its own is decide what the properties are. Those come from the rules above, and they should be short enough that the person who owns the rule can read them in review.

What properties won't do

• They don't replace examples. "As many allocations as lines" holds for a function that allocates zero to everything. Keep a few example tests that pin the actual answer, and use mutation testing to find out whether the pair of them is really watching.
• They're only as good as the generator. If your strategy never produces PALLET spelled out, you'll never learn what happens when a supplier writes it. Read the statistics, use event() to label what was generated, and feed real values from production into your strategies.
• They cost real money against a live model. Twenty-five inputs times three samples is 75 calls per test. That belongs in a scheduled job on the agent profile, not in the inner loop. The inner loop runs the same properties against recorded responses.
• They can't tell you the policy is right. A property says what must never happen. Whether the approval limit should be €10,000 or €15,000 is a business decision, and it belongs in the behavioral scenarios the business owns.

A practical starting point

• Pick the one function where messy input arrives, the normaliser, the parser, the importer, and write the "clean value or explicit rejection" property first.
• Add the cheap structural traits everywhere: type, bounds, shape, unit. They take minutes and catch a surprising share of real bugs.
• Model the ledger. Anything with receipts, shipments, corrections and retries deserves a stateful test with "always growing" and "never negative" invariants.
• Register profiles in conftest.py and cache .hypothesis/ in CI so every failure found once stays found.
• Write agent traits as plain functions, run them against recorded outputs on every commit and live outputs on a schedule, and turn on shrinking only for deterministic code.
• Put the same check in front of every write an agent makes to a system of record.

What's next in this series

We now have the pieces an agent can't talk its way around: a mutation score that says whether the tests are watching, and properties that say what must hold for every input. The next post puts them together. It's about independent oracles: how to set up an agent's loop so it can't grade its own homework, why the tests and the code shouldn't come from the same prompt, and how to review the tests an agent writes without reading every line of them.

If you've found a property that caught something nothing else would have, tell us. We'd like to hear what it was.

The testing series

1. Testing 101 for the Age of Coding Agents
2. Testing 102: Fast unit tests, honest end-to-end tests
3. Testing 103: Mutation testing with fermut
4. Testing 104: Property-based testing with Hypothesis (this post)

References

1. K. Claessen, J. Hughes, "QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs," ICFP 2000. doi.org/10.1145/351240.351266
2. D. R. MacIver, Z. Hatfield-Dodds, et al., "Hypothesis: A new approach to property-based testing," Journal of Open Source Software 4(43), 2019. doi.org/10.21105/joss.01891
3. Mars Climate Orbiter Mishap Investigation Board, "Phase I Report," NASA, 1999. llis.nasa.gov/llis_lib/pdf/1009464main1_0641-mr.pdf
4. T. Y. Chen, S. C. Cheung, S. M. Yiu, "Metamorphic Testing: A New Approach for Generating Next Test Cases," Technical Report HKUST-CS98-01, 1998. arxiv.org/abs/2002.12543
5. M. T. Ribeiro, T. Wu, C. Guestrin, S. Singh, "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList," ACL 2020. aclanthology.org/2020.acl-main.442
6. M. Maaz, L. DeVoe, Z. Hatfield-Dodds, N. Carlini, "Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem," 2025. arxiv.org/abs/2510.09907
7. H. Goldstein, J. W. Cutler, D. Dickstein, B. C. Pierce, A. Head, "Property-Based Testing in Practice," ICSE 2024. doi.org/10.1145/3597503.3639581
8. Hypothesis documentation. hypothesis.readthedocs.io