Batching the Billing Ledger: Small Trade-Offs with High Impact

Platforms like Netlify have more than 10 million developers.¹ That is more than NYC’s 2020 population. Like Manhattan, there’s always something going on in systems at this scale. Every second

Metering usage at scale makes ingestion a system with real cost and risk. It is a classic event-driven architecture problem. At 10,000 events a second, writing one S3 object per event creates 864 million data-object PUTs a day. Over 30 days, that is 25.92 billion PUTs and a modeled $129,600 in request charges.²

A straightforward trade-off is to batch incoming events before further processing. Use five-second windows based on when each event is ingested, then split a window early when it reaches 10,000 events. At 10,000 events a second, the count cap creates a full batch each second. When traffic is quiet, the five-second window closes the batch.

Gross request savings by event rate

Batching reduces request cost from the first steady event per second. Whether it pays for its own engineering and compute depends on traffic and the skills available to build and operate it. This table shows the gross S3 request saving before those costs.

An illustrative usage event in a billing ledger

I am building Netlify’s billing ledger and the team around it: an event-sourced ledger where every cent is explainable.

Usage-based products need a small record that can move from a request to a meter without losing its identity, model, token count, or time. This is what that record can look like.

syntax = "proto3";

import "google/protobuf/timestamp.proto";

message UsageEvent {
  string event_id = 1;
  string team_id = 2;
  string model = 3;
  uint64 input_tokens = 4;
  uint64 output_tokens = 5;
  bool billable = 6;
  google.protobuf.Timestamp occurred_at = 7; // UTC
}

The writer decodes the fields into DuckDB columns, then writes one compressed, queryable Parquet object. Parquet is an open, columnar format built for efficient analytical storage and selective retrieval. It is not a streaming transport: readers need its schema, structure, and types before they can find the column chunks they need.³ That makes it a useful shared format.

At this volume, it is more important than ever to preserve units and precision from emission through ingestion. Lose either and a quiet data error becomes a billing error, with money and customer trust at stake. AWS’s July 2026 incident was a unit-pricing failure in its estimated-billing computation system.

Customers saw false estimates in the billions. The failure was not in Parquet or an ingestion batch, but its warning still applies: validate values and units at every boundary.

The trade is small: accept up to one second of freshness delay and turn 10,000 paid S3 writes into one. DuckDB turns that one write into a durable Parquet file that downstream systems can use.