# ZIO Blocks

> Type-safe, modular building blocks for Scala

This file contains all documentation content in a single document following the llmstxt.org standard.

## ADR-001: SQL Data Migration Architecture


## Status

Accepted

## Context

The `zio-blocks` library needed an online data migration system that allows evolving database schemas from application code. The system builds on two existing foundations:

- `zio.blocks.schema.migration.Migration[A, B]` for typed schema evolution
- `zio.blocks.sql.Repo[E, ID]` for database access

Key requirements:

- No hand-written SQL in migrations
- No XML or Liquibase-style migration files
- Support for both PostgreSQL and SQLite
- Safe cutover for large tables with zero data loss

The module is Scala 3 only.

## Decisions

### Decision 1: Three Execution Models

**Decision:** Provide three execution tiers, each suited to a different scale of change:

- **A-Tiny** (`TinyMigrator`): Simple startup DDL via `Transactor.transact`. For schema-only changes that are fast and non-blocking.
- **A-Small** (`SmallMigrator`): Queue-based batch worker for moderate tables. Lifecycle: `init()` then loop `processBatch()` then `complete()`.
- **B-Large** (`LargeMigrator`): Incremental worker with safe completion protocol. Lifecycle: `init()` then `fence()` then `drain()` then `complete()`.

**Rationale:** Different migration scales need different guarantees. Tiny is lightweight and runs at startup. Small is transactional and batch-oriented. Large has distributed coordination and a safe cutover protocol. A single execution model would either be too heavy for simple DDL changes or too weak for large-table migrations.

### Decision 2: Coalesced Dirty-Key Queue

**Decision:** The queue table is a single shared table with columns `(id, op, payload)` — `id` is the source primary key (TEXT, PRIMARY KEY so duplicate keys coalesce via upsert), `op` records the capturing operation (`I`/`U`/`D`), and `payload` optionally stores the row JSON on deletes. There is no per-migration queue table or `migration_id` column; the source table remains authoritative. Keys enter the queue either manually (`QueueTable.enqueue`) or automatically via opt-in capture triggers (`captureTriggers = true` installs them in `init()`); trigger installation is idempotent and requires PG 14+ on PostgreSQL.

**Rationale:** Bounded queue storage. A single keyed table keeps the worker protocol simple and lets manual enqueue and trigger capture share one mechanism. The source row is always reread at processing time for `I`/`U` entries, avoiding stale data; `D` entries carry the payload needed to resolve the target key. Coalescing on the primary key means multiple updates to the same row between processing cycles collapse into a single queue entry. Manual enqueue keeps the default behavior dependency-free; triggers let producers write normally without application-code changes, but must not be enabled when the migrator writes to the same physical table it captures from (self-requeue loop).

### Decision 3: Source-Authoritative Replay

**Decision:** Workers claim a dirty key and, in one transaction: reread the source row (without taking a source-row lock), apply the migration, write the target, and remove the key. A missing source row at processing time means delete the target row.

**Rationale:** Prevents deadlock with concurrent source writers. The worker holds only the dirty-key claim, not the source row lock. A source writer can proceed without waiting for the worker to finish reading. The source table is the single source of truth, so rereading at processing time guarantees the worker acts on the latest state.

### Decision 4: Target Strategy Orthogonality

**Decision:** `TargetStrategy` is a thin enum (`InPlace` | `ShadowTable(suffix)`) orthogonal to the execution model. ShadowTable uses `CREATE TABLE IF NOT EXISTS ... LIKE ... INCLUDING ALL` and a DDL swap at completion.

**Rationale:** Decouples "how to write" from "when to write." The same worker code works for both strategies. InPlace updates rows directly in the source table. ShadowTable writes to a separate table and swaps it in at cutover. Callers choose the strategy based on whether the schema change is backward compatible.

### Decision 5: Safe Completion Protocol (B-Large)

**Decision:** State machine: `Initialized -> Fenced -> Drained -> Completed`. Fenced state prevents new source writes. Drain processes remaining queue items. Complete swaps the shadow table.

**Rationale:** Guarantees zero data loss during cutover. Queue persistence across restarts enables crash recovery. State is checked at each transition with `require()` guards. The protocol ensures every row modified between `init()` and `complete()` is migrated, even if the application crashes and restarts mid-migration.

### Decision 6: PostgreSQL-Specific Features

**Decision:** `QueueTable.dequeue` uses `SELECT ... FOR UPDATE SKIP LOCKED` for concurrent workers on PostgreSQL. Shadow table uses `CREATE TABLE ... LIKE ... INCLUDING ALL`. DDL swap is atomic.

**Rationale:** PostgreSQL's row-level locking and DDL transactionality enable safe multi-worker migrations. SQLite uses `dequeueSQLite` without `SKIP LOCKED` since SQLite serializes writes via `BEGIN IMMEDIATE` (single-writer model). The dialect split is necessary because the two databases have fundamentally different concurrency models.

### Decision 7: ID Type Parameters with Equality Constraint

**Decision:** `LargeMigrator` and `SmallMigrator` accept separate `ID1` and `ID2` type parameters, but construction requires evidence that `ID1 =:= ID2`. The queue stores V1 IDs.

**Rationale:** The two type parameters document the roles (source-repo ID vs target-repo ID) and keep repo signatures precise, while the `ID1 =:= ID2` constraint makes delete propagation type-safe: when a source row disappears between dequeue and read, the worker deletes the corresponding target row using the same key, reinterpreted through the evidence (`ev.substituteCo`) rather than a cast. Genuinely different key types (for example, `Int` to `Long`) are not supported by the built-in migrators; callers needing that must provide a custom conversion path outside this module.

### Decision 8: Pre-Commit State Enforcement

**Decision:** `complete()` requires `Drained` state. `drain()` requires `Fenced` state. `fence()` requires `Initialized` state. `drain()` only transitions to `Drained` if not paused.

**Rationale:** Prevents misordered lifecycle calls. The pause guard prevents `complete()` from being called when the queue is not empty (paused mid-drain). The stall valve turns an unbounded hot loop (empty batches with pending items, e.g. a stuck concurrent worker holding row locks) into a clear failure after 100 consecutive rounds; `SmallMigrator.complete()` likewise refuses while items are pending. State transitions are enforced at runtime with `require()` checks, making invalid sequences fail fast rather than producing subtle data inconsistencies.

### Decision 9: Failure Policy

**Decision:** On worker transaction failure: rollback, retain the dirty key, return failure to caller. No automatic retry, no dead-letter queue.

**Rationale:** Simple, predictable semantics. The caller decides retry policy. No silent data loss. No configuration surface for retry counts, backoff strategies, or dead-letter routing. The dirty key remains in the queue, so the row can be retried by calling `processBatch()` or `drain()` again.

### Decision 10: Scope Exclusions

**Decision:** The following are explicitly out of scope:

- No schema-diff or automatic migration derivation (Derivation.scala removed per review)
- No CLI (MigrationCLI removed per review)
- No JSON or full-row outbox payloads
- No external brokers or LISTEN/NOTIFY
- No composite keys, configurable retry, configuration files, third-party dialects, or automatic target-schema DDL generation

**Rationale:** Keeps scope narrow. `Migration[A, B]` provides the typed transformation. `Repo` provides the database surface. The data migration module is a thin orchestration layer. Callers supply the migration logic; the module handles queueing, worker coordination, and cutover.

## Consequences

### Positive

- Clean separation of concerns: `Migration[A, B]` (what) vs execution model (how) vs target strategy (where)
- Queue-based dirty-key tracking is storage-bounded and crash-recoverable
- Safe completion protocol prevents data loss during cutover
- ID type flexibility supports realistic migration scenarios
- No hand-written SQL, no XML, no external dependencies beyond the database

### Tradeoffs

- Missing-source-row delete relies on ID type compatibility (`ID1 =:= ID2`). For genuinely different ID types, a conversion function would be needed.
- No automatic retry requires caller-side error handling. Callers must implement their own retry logic if desired.
- PostgreSQL-only `SKIP LOCKED` limits full multi-worker support. SQLite is single-worker due to its write serialization model.
- State is in-memory (not persisted). Restart resets the state machine, though the queue table persists across restarts for crash recovery.
- Per-key writer blocking during worker processing is accepted. A source writer waits while the worker holds the dirty-key claim. No tuning configuration is provided for this.

## References

- [Data Migration Reference Documentation](../reference/data-migration.md)
- PR #1534 (implementation)

---

## Schema in a Nutshell

`Schema` describes the shape of a Scala data type: its fields, their types and its cases. You derive it once per type, and every format block reads that description to encode and decode your data. This is the central idea of ZIO Blocks, and many other blocks build on it.

## The Problem It Solves

In most Scala projects, each data format needs its own codec for each type: one for JSON, another for Avro, another for the database. The codecs repeat the same information, the shape of the type, and they drift apart when the type changes.

With Schema, the shape lives in one place. A change to a case class updates every format at once, because every format is derived from the same schema.

## Derive a Schema

We derive a schema by putting an implicit `Schema` in the companion object of the type. `Schema.derived` reads the fields at compile time:

```scala

final case class Customer(name: String, email: String, vip: Boolean)

object Customer {
  implicit val schema: Schema[Customer] = Schema.derived
}
```

The same line works for sealed traits, nested case classes and collections; on Scala 3 you can also write `derives Schema` on the case class.

## Read and Write JSON

JSON support is part of the Schema block itself. With the schema in scope, a value encodes with `toJsonString`, and a string decodes with `fromJson`:

```scala
val alice = Customer("Alice", "alice@example.com", vip = true)
// alice: Customer = Customer(
//   name = "Alice",
//   email = "alice@example.com",
//   vip = true
// )

println(alice.toJsonString)
// {"name":"Alice","email":"alice@example.com","vip":true}

"""{"name":"Bob","email":"bob@example.com","vip":false}""".fromJson[Customer]
// res1: Either[SchemaError, Customer] = Right(
//   Customer(name = "Bob", email = "bob@example.com", vip = false)
// )
```

Decoding never throws for bad input. It returns an `Either`, and the message of the error names the field that failed:

```scala
"""{"name":"Carol","email":"carol@example.com","vip":"yes"}""".fromJson[Customer].left.map(_.getMessage)
// res2: Either[String, Customer] = Left("illegal boolean at: .vip")
```

## One Schema, Many Formats

Other formats are separate blocks that read the same schema. For example, after adding `zio-blocks-schema-yaml` to the build, we derive a YAML codec from the schema we already have, with no new code in the `Customer` type:

```scala

val yamlCodec = Customer.schema.derive(YamlFormat)
```

The codec encodes `alice` to YAML bytes:

```scala
println(new String(yamlCodec.encode(alice), "UTF-8"))
// name: Alice
// email: alice@example.com
// vip: true
```

Avro, BSON, CSV, MessagePack, Thrift, TOON and XML work the same way; [Built-in Formats and Codecs](../../reference/schema/built-in-codecs/index.md) lists them all.

## What Else a Schema Gives You

A schema is a full description of the type, so it powers more than codecs:

- [Validated types](../../guides/validated-newtypes.md) wrap a field in a rule with `Schema#transform`, so a value that breaks the rule fails to decode.
- [Optics](../../reference/schema/optics.md) read and update deeply nested fields without hand-written `copy` chains.
- [Patches](../../reference/schema/patch.md) compute the difference between two values and apply it elsewhere.
- [Schema evolution](../../reference/schema/schema-evolution/index.md) converts data between an old and a new version of a type.
- JSON Schema documents come from `Schema#toJsonSchema`, for APIs and payload validation.

Other blocks use the schema too: the [SQL](../../reference/sql/index.md) block maps case classes to table rows with it, and the [Config](../../reference/config/index.md) block decodes configuration files into case classes with it.

## Learn More

The [Your First Schema](../../guides/your-first-schema.md) tutorial walks through a complete round trip and reading decode errors, and the [Schema reference](../../reference/schema/index.md) documents every part of the block. The next concept in this section is [Scope](./scope.md).

---

## Scope in a Nutshell

`Scope` manages resources: files, network connections, database pools and anything else that must be closed after use. You open resources inside a scope, and the scope closes all of them when it ends, in reverse order, even when the code inside fails. The compiler checks that no resource escapes the scope that owns it.

## The Problem It Solves

Plain Scala closes resources with `try`/`finally`, one level of nesting per resource. With three resources the closing logic outgrows the real work, and it is easy to forget one `close` or to return a reader from a method after its file has been closed.

A scope replaces that nesting with one region of code. Every resource opened in it is registered with the scope, and closing happens in one place, automatically.

## Open a Resource in a Scope

A `Resource` is a recipe: it says how to acquire a value and how to release it, but does nothing until a scope allocates it. Here we describe a connection with `Resource.acquireRelease`, printing a message in both steps so we can see when they run:

```scala

final class Connection(val name: String) {
  def query(sql: String): String = s"$name answered: $sql"
}

def connection(name: String): Resource[Connection] =
  Resource.acquireRelease {
    println(s"Opening $name")
    new Connection(name)
  } { conn =>
    println(s"Closing ${conn.name}")
  }
```

`Scope.global.scoped` opens a scope for the block of code we pass it. Inside, `allocate` runs a recipe and registers the release step, and `$(value)(...)` is how we call methods on a scoped value:

```scala
Scope.global.scoped { scope =>

  val users  = allocate(connection("users-db"))
  val orders = allocate(connection("orders-db"))

  println($(users)(_.query("SELECT name FROM users")))
  println($(orders)(_.query("SELECT id FROM orders")))
}
// Opening users-db
// Opening orders-db
// users-db answered: SELECT name FROM users
// orders-db answered: SELECT id FROM orders
// Closing orders-db
// Closing users-db
```

Both connections close when the block ends, the last one opened first. Classes that implement `AutoCloseable` need no release step at all: `Resource(new FileInputStream(file))` closes the stream for you.

## Why the Compiler Can Check It

Inside a scope, `allocate` returns a value of type `$[Connection]`, not a plain `Connection`. The `$` type is tied to the scope that owns the resource, so the compiler rejects a block that tries to return the resource. Plain data, such as the `String` that `query` returns, leaves the scope freely.

In practice this turns the classic bug, handing out a reader whose file is about to close, into a compile error. Code that reaches a resource some other way after its scope has closed, for example from a callback that runs later, gets an `IllegalStateException` that says the scope is closed, instead of silently reading from a closed file.

## What Else Scope Gives You

The block covers more than one flat region:

- Child scopes nest inside a parent and close before it, for per-request or per-task resources.
- `Resource.shared` lets several services use one resource, which closes when the last user is done.
- [Wire](./wire-and-context.md) builds whole applications of services and their resources from constructors.

## Learn More

The [Compile-Time Resource Safety with Scope](../../guides/compile-time-resource-safety-with-scope.md) tutorial goes through the block step by step, and the [Resource Management reference](../../reference/resource-management/index.md) documents `Scope`, `Resource` and the related types. The next concept in this section is [Wire and Context](./wire-and-context.md).

---

## Streams in a Nutshell

A `Stream[E, A]` describes a sequence of `A` values that arrives piece by piece, and that may fail with an error of type `E`. You build a pipeline by transforming the stream, then run it to get a result. Because a stream pulls one element at a time, it processes files, sockets and query results that do not fit in memory.

## The Problem It Solves

Reading a large file into a `List` works until the file is larger than the heap. Iterators avoid that, but they have no way to report a typed error, no way to close the file when processing stops early, and no clear point where the work happens.

A stream fixes all three. Its error type is part of its type, it can own the resource it reads from, and nothing happens until you run it.

## Describe, Then Run

We describe a stream of orders and a pipeline over it. These lines do no work yet; they only describe what should happen:

```scala

final case class Order(id: Int, amountCents: Long, paid: Boolean)

val orders: Stream[Nothing, Order] = Stream(
  Order(1, 2500L, paid = true),
  Order(2, 1200L, paid = false),
  Order(3, 4800L, paid = true),
  Order(4, 900L, paid = true)
)

val paidAmounts: Stream[Nothing, Long] = orders.filter(_.paid).map(_.amountCents)
```

The type `Stream[Nothing, Order]` says that this stream cannot fail. Running it does the work and returns a plain `Either`, with the error on the left and the result on the right:

```scala
paidAmounts.runCollect
// res0: Either[Nothing, Chunk[Long]] = Right(IndexedSeq(2500L, 4800L, 900L))

paidAmounts.runFold(0L)((total, cents) => total + cents)
// res1: Either[Nothing, Long] = Right(8200L)
```

The same `paidAmounts` value can run again, and each run starts from the beginning.

## Errors Are Part of the Type

A step that can fail turns a `Stream[Nothing, A]` into a stream with a real error type. Here each text line becomes an order, or fails with a `BadLine` error:

```scala
final case class BadLine(line: String)

def parse(line: String): Stream[BadLine, Order] =
  line.split(",") match {
    case Array(id, cents, paid) if id.toIntOption.isDefined && cents.toLongOption.isDefined =>
      Stream.succeed(Order(id.toInt, cents.toLong, paid == "paid"))
    case _ =>
      Stream.fail(BadLine(line))
  }

val parsed: Stream[BadLine, Order] =
  Stream("1,2500,paid", "2,oops,unpaid", "3,4800,paid").flatMap(parse)
```

Running the stream stops at the first error and returns it as a value:

```scala
parsed.runCollect
// res2: Either[BadLine, Chunk[Order]] = Left(BadLine("2,oops,unpaid"))
```

`Stream#catchAll` replaces a failure with another stream. Applied to the parsing step of each line, it turns a bad line into an empty stream, so the pipeline skips that line and keeps going, and the error type becomes `Nothing` again:

```scala
Stream("1,2500,paid", "2,oops,unpaid", "3,4800,paid")
  .flatMap(line => parse(line).catchAll(_ => Stream.empty))
  .map(_.id)
  .runCollect
// res3: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 3))
```

Applied to the whole `parsed` stream instead, `catchAll` would take over at the first error and end the run there, without order 3.

## What Else Streams Give You

The block is built for real input and output:

- `Stream.fromAcquireRelease` opens a file or a connection when the stream starts and closes it when the stream ends, even if it fails or stops early.
- A `Sink` describes how to consume a stream, such as counting, folding or writing to a file, and `Stream#run` connects the two.
- Elements of type `Int`, `Long`, `Double` and the other primitives stay unboxed as they flow through a pipeline.
- The run methods on this page block the calling thread and exist on the JVM. On Scala.js, or whenever a thread must not block, use the `Async` variants such as `Stream#runCollectAsync`.

## Learn More

The [Your First Streaming Pipeline](../../guides/your-first-streaming-pipeline.md) tutorial builds a complete pipeline over a log file, and the [Streams reference](../../reference/streams/index.md) documents every operator. The next page shows how to [use ZIO Blocks with ZIO, cats-effect or plain Scala](../use-with-effect-systems.md).

---

## Wire and Context in a Nutshell

An application is a graph of services: a web handler needs a user service, which needs a database, which needs its configuration. `Wire` builds that graph for you from ordinary constructors, checks it at compile time, and hands every resource in it to a [scope](./scope.md) so that it closes in the right order. `Context` is the small data type underneath: a set of values looked up by their type.

## The Problem It Solves

Without a dependency injection tool, `main` creates every service by hand, in the right order, and passes each one to the constructors that need it. That code grows with every service, and the order of closing resources has to be kept in sync by hand.

Runtime injection frameworks remove that code but report a missing service only when the application starts. Wire does the same job with a macro, so a broken graph is a compile error, and there is no framework and no annotations: your classes stay plain classes with constructor parameters.

## Build a Graph from Constructors

We start with three plain classes. `Database` is a resource, so it implements `AutoCloseable`; `UserService` needs a `Database`; `Database` needs a `DbConfig`:

```scala

final case class DbConfig(url: String)

final class Database(config: DbConfig) extends AutoCloseable {
  println(s"Connecting to ${config.url}")
  def query(sql: String): String = s"rows for: $sql"
  def close(): Unit = println("Database closed")
}

final class UserService(db: Database) {
  def userName(id: Int): String = db.query(s"SELECT name FROM users WHERE id = $id")
}
```

`Resource.from[UserService]` reads the constructors at compile time and builds the whole graph. We pass a `Wire` only for what it cannot build on its own, here the configuration value:

```scala
def userService: Resource[UserService] =
  Resource.from[UserService](
    Wire(DbConfig("jdbc:postgresql://localhost/app"))
  )
```

Allocating the resource in a scope creates the database, then the service; when the scope ends, the database closes:

```scala
Scope.global.scoped { scope =>

  val service = allocate(userService)
  println($(service)(_.userName(42)))
}
// Connecting to jdbc:postgresql://localhost/app
// rows for: SELECT name FROM users WHERE id = 42
// Database closed
```

Without the `DbConfig` wire, the macro would try to build `DbConfig` from its constructor, find that it needs a `String` it has no way to invent, and stop with a compile error that names the missing type and the chain of services that needed it.

## Shared and Unique Services

Every service that the macro builds on its own is created once and shared by everything that needs it. To change that for one type, pass `Wire.unique[T]`, which creates a fresh instance for each service that depends on `T`; `Wire.shared[T]` states the default explicitly. Both wires are derived from the constructor of `T`, like the rest of the graph.

## Context: Values Looked Up by Type

`Context` holds a few values of different types and returns each one by its type. The type of the context lists what it contains, here `Context[RequestId & UserId]` (written `RequestId with UserId` on Scala 2), so asking for a type it does not contain is a compile error:

```scala

final case class RequestId(value: String)
final case class UserId(value: Long)

val ctx = Context(RequestId("req-17"), UserId(42L))
```

Each lookup returns the value with its own type, here a `UserId`:

```scala
ctx.get[UserId]
// res1: UserId = UserId(42L)
```

Wire uses `Context` to pass each constructor its dependencies, and you can use it directly to carry request-level values, such as a request id, through your own code.

## Learn More

The [Wire reference](../../reference/resource-management/wire.md) covers manual wires and sharing in detail, the [Context reference](../../reference/context.md) covers the full `Context` API, and [How to Fix Common Scope and Wire Compile Errors](../../guides/fix-scope-and-wire-compile-errors.md) explains each error the macro can report. The next concept in this section is [Streams](./streams.md).

---

## Your First Five Minutes with ZIO Blocks

In five minutes, we write one small program that uses three blocks together. It reads temperature readings from a file in JSON Lines format (one JSON object per line), decodes each line into a case class, skips the line that is broken, and prints a summary as JSON.

Each block does one job:

- [Schema](./concepts/schema.md) turns JSON into case classes and back.
- [Scope](./concepts/scope.md) opens the file and guarantees that it is closed.
- [Chunk](../reference/chunk.md) holds the decoded readings and computes the summary.

No block needs a runtime, an effect type or any setup: every call on this page is an ordinary method call.

## Add Two Dependencies

The program needs two artifacts; Chunk comes along with Schema. On Scala 2.13 and Scala 3.3 to 3.8, add them to `build.sbt`:

```scala
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-schema" % "0.0.56",
  "dev.zio" %% "zio-blocks-scope"  % "0.0.56"
)
```

On Scala 3.9 and newer, the artifact names start with `zio-blocks-next-` and need `-experimental`; [Installation](./installation.md) covers that case, scala-cli and Scala.js.

## Describe the Data with Schema

We start with two case classes: `Reading` is one line of the file, and `Summary` is the result we print. Giving each one a derived `Schema` is all it takes to read and write it as JSON:

```scala

final case class Reading(sensor: String, celsius: Double)

object Reading {
  implicit val schema: Schema[Reading] = Schema.derived
}

final case class Summary(readings: Int, averageCelsius: Double, hottestSensor: String)

object Summary {
  implicit val schema: Schema[Summary] = Schema.derived
}
```

With the schema in scope, every `Reading` has a `toJsonString` method, and every `String` has a `fromJson` method that returns either an error or a value:

```scala
println(Reading("kitchen", 21.5).toJsonString)
// {"sensor":"kitchen","celsius":21.5}

"""{"sensor":"garage","celsius":12.0}""".fromJson[Reading]
// res1: Either[SchemaError, Reading] = Right(
//   Reading(sensor = "garage", celsius = 12.0)
// )
```

## Read the File with Scope

Now we need a file to read. This block writes four readings to a temporary file; the third one is broken on purpose, because its temperature is a string:

```scala

val path: Path = Files.createTempFile("readings", ".jsonl")

Files.writeString(
  path,
  """{"sensor":"kitchen","celsius":21.5}
    |{"sensor":"garage","celsius":12.0}
    |{"sensor":"attic","celsius":"hot"}
    |{"sensor":"office","celsius":23.5}
    |""".stripMargin
)
```

A file reader must be closed after use, even when reading fails. `Scope.global.scoped` opens a region of code; `allocate` registers the reader with that region, and the region closes the reader when it ends. The `$(reader)(...)` call is how we use the reader inside the region: Scope checks at compile time that the reader itself never escapes it, while the list of lines does:

```scala

def readLines(path: Path): List[String] =
  Scope.global.scoped { scope =>

    val reader: $[BufferedReader] = allocate(
      Resource.acquireRelease {
        println("Opened the readings file")
        Files.newBufferedReader(path)
      } { reader =>
        reader.close()
        println("Closed the readings file")
      }
    )
    $(reader)(_.lines().iterator().asScala.toList)
  }
```

When we call it, the file opens, the lines come back, and the file is already closed by the time we hold the result:

```scala
val lines = readLines(path)
// Opened the readings file
// Closed the readings file
// lines: List[String] = List(
//   "{\"sensor\":\"kitchen\",\"celsius\":21.5}",
//   "{\"sensor\":\"garage\",\"celsius\":12.0}",
//   "{\"sensor\":\"attic\",\"celsius\":\"hot\"}",
//   "{\"sensor\":\"office\",\"celsius\":23.5}"
// )
```

## Decode Each Line

Each line decodes to an `Either`. Splitting the results with `List#partitionMap` gives us the errors on one side and the readings on the other:

```scala
val (errors, parsed) = lines.map(_.fromJson[Reading]).partitionMap(identity)
```

Three lines decode, and the broken one becomes an error:

```scala
parsed
// res3: List[Reading] = List(
//   Reading(sensor = "kitchen", celsius = 21.5),
//   Reading(sensor = "garage", celsius = 12.0),
//   Reading(sensor = "office", celsius = 23.5)
// )

errors.foreach(error => println(s"Skipped a line: ${error.getMessage}"))
// Skipped a line: illegal number at: .celsius
```

The error message names the field that failed and the reason, so a bad line in a large file is easy to find.

## Summarize with Chunk

`Chunk` is the sequence type that all blocks share. It works like a `Vector`, and it stores primitives such as `Double` without boxing them. We compute the summary from a `Chunk` of readings, then print it as JSON with the schema we derived for `Summary`:

```scala

def summarize(readings: Chunk[Reading]): Summary =
  Summary(
    readings = readings.size,
    averageCelsius = readings.map(_.celsius).sum / readings.size,
    hottestSensor = readings.maxBy(_.celsius).sensor
  )

println(summarize(Chunk.fromIterable(parsed)).toJsonString)
// {"readings":3,"averageCelsius":19.0,"hottestSensor":"office"}
```

## The Whole Program

Here is the complete program as one file. Run it from the `zio-blocks-examples` directory of the repository with `sbt "gettingStarted/runMain gettingstarted.FirstFiveMinutes"`:

```scala title="zio-blocks-examples/getting-started/src/main/scala/gettingstarted/FirstFiveMinutes.scala"
package gettingstarted

/**
 * Getting Started — Your First Five Minutes
 *
 * Reads temperature readings from a JSON Lines file, decodes them with Schema,
 * summarizes them in a Chunk, and prints the summary as JSON. Scope opens the
 * file and closes it as soon as the lines are read.
 *
 * Run with: sbt "gettingStarted/runMain gettingstarted.FirstFiveMinutes"
 */
final case class Reading(sensor: String, celsius: Double)

object Reading {
  implicit val schema: Schema[Reading] = Schema.derived
}

final case class Summary(readings: Int, averageCelsius: Double, hottestSensor: String)

object Summary {
  implicit val schema: Schema[Summary] = Schema.derived
}

object FirstFiveMinutes {

  def readLines(path: Path): List[String] =
    Scope.global.scoped { scope =>

      val reader: $[BufferedReader] = allocate(
        Resource.acquireRelease {
          println("Opened the readings file")
          Files.newBufferedReader(path)
        } { reader =>
          reader.close()
          println("Closed the readings file")
        }
      )
      $(reader)(_.lines().iterator().asScala.toList)
    }

  def summarize(readings: Chunk[Reading]): Summary =
    Summary(
      readings = readings.size,
      averageCelsius = readings.map(_.celsius).sum / readings.size,
      hottestSensor = readings.maxBy(_.celsius).sensor
    )

  def main(args: Array[String]): Unit = {
    val path = Files.createTempFile("readings", ".jsonl")
    Files.writeString(
      path,
      """{"sensor":"kitchen","celsius":21.5}
        |{"sensor":"garage","celsius":12.0}
        |{"sensor":"attic","celsius":"hot"}
        |{"sensor":"office","celsius":23.5}
        |""".stripMargin
    )

    val lines            = readLines(path)
    val (errors, parsed) = lines.map(_.fromJson[Reading]).partitionMap(identity)

    errors.foreach(error => println(s"Skipped a line: ${error.getMessage}"))

    val readings = Chunk.fromIterable(parsed)

    println(summarize(readings).toJsonString)

    Files.delete(path)
  }
}
```

## What You Just Used

The program uses three blocks, and each one has a page in this section or in the reference:

| Block  | What it does here                                   | Learn more                                       |
| ------ | --------------------------------------------------- | ------------------------------------------------ |
| Schema | Decodes each JSON line and encodes the summary      | [Schema in a Nutshell](./concepts/schema.md)     |
| Scope  | Opens the file and closes it after reading          | [Scope in a Nutshell](./concepts/scope.md)       |
| Chunk  | Holds the readings and computes the summary         | [Chunk reference](../reference/chunk.md)         |

Next, [How ZIO Blocks Thinks](./how-zio-blocks-thinks.md) explains why the blocks fit together this easily.

---

## How ZIO Blocks Thinks

Every block in ZIO Blocks follows the same design. Once you know it, each new block feels familiar: you already know how to add it, how to call it and how it reports errors. This page explains that design in six ideas, then shows how the blocks build on each other.

## A Block Is a Library, Not a Framework

You call a block; it never calls you. There is no application base class to extend, no runtime to start and no main loop to hand control to. A block is a set of types and methods, and your code decides when to use them.

This is why ZIO Blocks fits into an existing project one block at a time. You can replace a JSON library with [Schema](./concepts/schema.md) in one module and leave the rest of the code base untouched.

## Plain Values Instead of an Effect Type

Effect systems such as ZIO and cats-effect describe work as values that run later. ZIO Blocks does not require one: its methods do the work when you call them and return ordinary Scala values. Decoding JSON returns an `Either`, running a [stream](./concepts/streams.md) on the JVM returns an `Either` with the collected elements, and a database query returns its rows.

Expected failures are values too. A malformed JSON document comes back as the `Left` side of an `Either`, with a message that says what went wrong and where, and a stream declares the type of its errors in its own type, `Stream[E, A]`.

When you do want to run work without blocking a thread, the [Async](../reference/async.md) block provides a small asynchronous type, and blocks such as Streams offer `Async` variants of their run methods. If your application already has an effect system, [Use ZIO Blocks with ZIO, cats-effect or Plain Scala](./use-with-effect-systems.md) shows where these calls go.

## One Artifact per Block

Each block is published as its own artifact under the `dev.zio` organization, and blocks depend on each other only where they share a real need. Adding `zio-blocks-scope` to a build does not pull in JSON codecs, and adding `zio-blocks-schema` does not pull in a streaming library.

The blocks form layers. Small foundation blocks at the bottom hold the shared data types; three hub blocks in the middle describe data, own resources and move data; the application blocks at the top combine them. Each arrow means "builds on":

```text
 Application blocks   sql      config    endpoint    html     telemetry
                        │         │          │          │          │
                        ▼         ▼          ▼          ▼          │
 Hub blocks          schema ◄──────────── describes your data      │
                     streams ◄─────────── moves data in pieces     │
                     scope ◄───────────── owns resources           │
                        │                                          │
                        ▼                                          ▼
 Foundation blocks   chunk    maybe    typeid    context    async  combinators
```

For example, the SQL block uses Schema to map case classes to rows, Streams to read large results and Scope to close connections; the Config block decodes your configuration into case classes through Schema.

## Mistakes Become Compile Errors

Many problems that other libraries report at run time are compile errors in ZIO Blocks, because macros check them while your code compiles:

- **Codecs.** `Schema.derived` inspects your case class or sealed trait and builds its schema at compile time, so there is no reflection at run time and a type that cannot be described fails to compile.
- **Resource lifetimes.** [Scope](./concepts/scope.md) tags every resource with the region that owns it, so returning an open file or connection from that region does not compile.
- **Dependency graphs.** [Wire](./concepts/wire-and-context.md) reads your constructors and assembles the services your application needs; a dependency that it cannot build is a compile error that names the type.

## Fast Without Extra Work

The blocks avoid work that slows down common Scala code. [Chunk](../reference/chunk.md) stores `Int`, `Long`, `Double` and the other primitives in primitive arrays instead of boxing each element, and Streams keeps primitive elements unboxed as they flow through a pipeline. You get this by using the default APIs; there is no separate fast mode to turn on.

## The Same Code on Every Platform

Most blocks build for Scala 2.13 and Scala 3, on the JVM and on Scala.js, from the same source code. You can move a project from Scala 2.13 to Scala 3 without changing how you call a block. The few blocks that are narrower, such as SQL, which needs Scala 3, say so in the catalog on the [overview](../index.md).

## Putting the Ideas Together

These ideas explain why the program in [Your First Five Minutes](./first-five-minutes.md) needs no setup: Schema, Scope and Chunk are plain libraries, they return plain values, they share the `Chunk` type, and the compiler checks that the file reader never leaves the scope that owns it.

The next four pages introduce the blocks most applications start with, beginning with [Schema](./concepts/schema.md).

---

## What Is ZIO Blocks?

ZIO Blocks is a family of small, independent Scala libraries. Each one solves one common problem: describing data and turning it into JSON, closing resources, wiring services together, streaming data, loading configuration, talking to a database, and more. You add the blocks you need to your build and call them from ordinary Scala code.

This Getting Started section is a short tour. It teaches the core ideas, not every method: each page takes a few minutes, and links to the reference when you want the full details.

## Why ZIO Blocks Exists

Most Scala libraries for these jobs come tied to an ecosystem. A JSON library expects you to use its effect type for streaming, a resource library expects a runtime, a database library expects a particular effect system. Picking one library often means adopting a whole stack.

ZIO Blocks takes the opposite approach. A block needs no effect system and no runtime: its methods are plain Scala functions that return plain values. That is why the same block works in a ZIO application, a cats-effect application, an Akka or Ox application, or a plain `main` method.

## The Ideas Behind It

Every block follows the same few principles:

- **No lock-in.** No block depends on ZIO, cats-effect or any other effect system.
- **Modular.** Each block is its own artifact, so your build contains only what you use.
- **Cross-platform.** Most blocks build for Scala 2.13 and Scala 3, on the JVM and on Scala.js, from the same source.
- **Checked at compile time.** Macros derive codecs, check resource lifetimes and build dependency graphs while your code compiles, so mistakes show up as compile errors.
- **Fast.** Implementations avoid boxing and needless allocation on hot paths.

[How ZIO Blocks Thinks](./how-zio-blocks-thinks.md) explains these ideas in more detail and shows how the blocks build on each other.

## Which Block Do I Need?

Start from the problem you have. This table lists the most common problems and the block that solves each one; the [overview](../index.md) has the full catalog of every block:

| You need to...                                         | Use this block                                                  | Artifact                        |
| ------------------------------------------------------ | --------------------------------------------------------------- | ------------------------------- |
| Turn a case class into JSON (or Avro, YAML, CSV...)    | [Schema](../reference/schema/index.md)                          | `zio-blocks-schema`             |
| Open files or connections and always close them        | [Scope](../reference/resource-management/index.md)              | `zio-blocks-scope`              |
| Build services and their dependencies                  | [Wire](../reference/resource-management/wire.md)                | `zio-blocks-scope`              |
| Process data that does not fit in memory               | [Streams](../reference/streams/index.md)                        | `zio-blocks-streams`            |
| Load typed configuration from files and environment    | [Config](../reference/config/index.md)                          | `zio-blocks-config`             |
| Query a database with type-safe SQL                    | [SQL](../reference/sql/index.md)                                | `zio-blocks-sql`                |
| Describe HTTP endpoints as data, for OpenAPI and more  | [Endpoint](../reference/endpoint/index.md)                      | `zio-blocks-endpoint`           |
| Render HTML safely                                     | [HTML](../reference/html.md)                                    | `zio-blocks-html`               |
| Record logs, traces and metrics                        | [Telemetry](../reference/telemetry/index.md)                    | `zio-blocks-telemetry`          |
| Store sequences of primitives without boxing           | [Chunk](../reference/chunk.md)                                  | `zio-blocks-chunk`              |
| Run asynchronous code in direct style                  | [Async](../reference/async.md)                                  | `zio-blocks-async`              |

## How to Read This Section

The pages build on each other, so read them in order:

1. [Installation](./installation.md) adds a block to your sbt or scala-cli build.
2. [Your First Five Minutes](./first-five-minutes.md) runs one small program that uses three blocks together.
3. [How ZIO Blocks Thinks](./how-zio-blocks-thinks.md) explains the design that all blocks share.
4. The core concept pages introduce the four blocks most applications start with: [Schema](./concepts/schema.md), [Scope](./concepts/scope.md), [Wire and Context](./concepts/wire-and-context.md) and [Streams](./concepts/streams.md).
5. [Use ZIO Blocks with ZIO, cats-effect or Plain Scala](./use-with-effect-systems.md) shows where ZIO Blocks calls go in an application that already has an effect system.
6. [Where Next](./where-next.md) points you to the tutorials, how-to guides and reference.

If you are only evaluating the library, the first three pages are enough to decide whether it fits your project.

---

## How to Add ZIO Blocks to Your Build

This guide gets ZIO Blocks onto your classpath. You'll pick the modules you need, write the dependency line for your Scala version, and check that it compiles. Each section is a copy-paste recipe.

## The Artifact Name Depends on Your Scala Version

ZIO Blocks publishes every module under two names. The artifacts built with Scala 2.13 and Scala 3.3 LTS are called `zio-blocks-<module>`. The artifacts built with the latest Scala 3 release are called `zio-blocks-next-<module>`:

| Your Scala version | Artifact name                  | Built with |
|--------------------|--------------------------------|------------|
| 2.13.x             | `zio-blocks-<module>`          | 2.13.18    |
| 3.3.x to 3.8.x     | `zio-blocks-<module>`          | 3.3.7 LTS  |
| 3.9.x and newer    | `zio-blocks-next-<module>`     | 3.9.0      |

A Scala 3.9 build can still use `zio-blocks-<module>`, but `zio-blocks-next-<module>` is the build made for it. The reverse does not work: a Scala 3.3 to 3.8 compiler cannot read code compiled by Scala 3.9, so `zio-blocks-next-<module>` fails there with a "Forward incompatible TASTy file" error.

## Prerequisites

- **sbt 1.x** or **scala-cli**.
- **JDK 11+** for Scala 2.13 and Scala 3.3 to 3.8; **JDK 17+** for the `zio-blocks-next-*` artifacts.
- **JDK 21+** on the JVM for `streams`, `telemetry`, and `telemetry-otel`, and for every module built on `streams`: `http-model`, `http-model-schema`, `endpoint`, `datastar`, `http-htmx`, `sql`, `sql-zio`, `data-migration`, and `projection`.
- For Scala.js, **Scala.js 1.22.0+** (the `sbt-scalajs` plugin in `project/plugins.sbt`). Older linkers reject the artifacts with `IRVersionNotSupportedException`.

## Pick Your Modules

Every module lives under the `dev.zio` organization. Replace `<module>` in the artifact name with a value from the first column:

| `<module>`           | What you get                                                        | Platforms | Scala      |
|----------------------|---------------------------------------------------------------------|-----------|------------|
| `schema`             | `Schema` derivation, optics, validation, and the built-in JSON codec | JVM · JS  | 2.13 · 3.x |
| `schema-avro`        | Apache Avro codec                                                   | JVM       | 2.13 · 3.x |
| `schema-bson`        | MongoDB BSON codec                                                  | JVM       | 2.13 · 3.x |
| `schema-csv`         | CSV codec                                                           | JVM · JS  | 2.13 · 3.x |
| `schema-messagepack` | MessagePack codec                                                   | JVM · JS  | 2.13 · 3.x |
| `schema-thrift`      | Apache Thrift codec                                                 | JVM       | 2.13 · 3.x |
| `schema-toon`        | TOON codec                                                          | JVM · JS  | 2.13 · 3.x |
| `schema-xml`         | XML codec                                                           | JVM · JS  | 2.13 · 3.x |
| `schema-yaml`        | YAML codec                                                          | JVM · JS  | 2.13 · 3.x |
| `typeid`             | Compile-time type identity                                          | JVM · JS  | 2.13 · 3.x |
| `chunk`              | Immutable indexed sequences without boxing                          | JVM · JS  | 2.13 · 3.x |
| `maybe`              | Low-allocation optional values                                      | JVM · JS  | 2.13 · 3.x |
| `combinators`        | Tuple, `Either`, and union composition                              | JVM · JS  | 2.13 · 3.x |
| `context`            | Type-indexed heterogeneous collections                              | JVM · JS  | 2.13 · 3.x |
| `scope`              | Compile-time safe resource management and `Wire`                   | JVM · JS  | 2.13 · 3.x |
| `async`              | Asynchronous effect type with direct-style `await`                  | JVM · JS  | 2.13 · 3.x |
| `streams`            | Pull-based streams                                                  | JVM · JS  | 2.13 · 3.x |
| `mux`                | Stream multiplexer for HTTP/2, QUIC, and WebSocket-style protocols  | JVM · JS  | 2.13 · 3.x |
| `ringbuffer`         | Lock-free bounded ring buffers                                      | JVM · JS  | 2.13 · 3.x |
| `config`             | Typed configuration loading                                         | JVM · JS  | 2.13 · 3.x |
| `config-yaml`        | YAML source for `config`                                            | JVM · JS  | 2.13 · 3.x |
| `config-json`        | JSON source for `config`                                            | JVM · JS  | 2.13 · 3.x |
| `config-hocon`       | HOCON source for `config`                                           | JVM · JS  | 2.13 · 3.x |
| `mediatype`          | IANA media types                                                    | JVM · JS  | 2.13 · 3.x |
| `http-model`         | HTTP data model: URLs, headers, requests, responses                 | JVM · JS  | 2.13 · 3.x |
| `http-model-schema`  | Schema-based typed access to the HTTP model                         | JVM · JS  | 2.13 · 3.x |
| `endpoint`           | Typed HTTP endpoint descriptions                                    | JVM · JS  | 2.13 · 3.x |
| `openapi`            | OpenAPI 3.1 generation                                              | JVM · JS  | 2.13 · 3.x |
| `html`               | Type-safe HTML templates                                            | JVM · JS  | 2.13 · 3.x |
| `datastar`           | Datastar attributes, signals, and SSE events                        | JVM · JS  | 3.x        |
| `http-htmx`          | Typed HTMX attributes                                               | JVM · JS  | 3.x        |
| `jwt`                | JWT signing and verification                                        | JVM · JS  | 2.13 · 3.x |
| `markdown`           | GitHub Flavored Markdown parsing and rendering                      | JVM · JS  | 2.13 · 3.x |
| `telemetry`          | Tracing, logging, and metrics                                       | JVM · JS  | 2.13 · 3.x |
| `telemetry-otel`     | OTLP exporters for `telemetry`                                      | JVM       | 2.13 · 3.x |
| `sql`                | Schema-driven SQL and a JDBC transactor                             | JVM · JS  | 3.x        |
| `sql-zio`            | ZIO integration for `sql`                                           | JVM       | 3.x        |
| `data-migration`     | Typed database schema migrations                                    | JVM · JS  | 3.x        |
| `projection`         | Event-sourced projections                                           | JVM       | 3.x        |
| `codegen`            | Scala code generation IR and emitter                                | JVM       | 2.13 · 3.x |
| `smithy`             | Smithy IDL parser and AST                                           | JVM       | 2.13 · 3.x |

Modules pull in the ZIO Blocks modules they build on, so you only list the ones you use directly.

## Add a Dependency With sbt

On Scala 2.13 or Scala 3.3 to 3.8, use the `zio-blocks-` prefix:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"
```

On Scala 3.9 or newer, use the `zio-blocks-next-` prefix and enable experimental features. The `zio-blocks-next-*` artifacts are compiled with `-experimental`, so code that calls them must be too. Without the flag, the compiler reports that a definition "is marked @experimental":

```scala
scalaVersion := "3.9.0"

libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56"

scalacOptions += "-experimental"
```

For Scala.js, write `%%%` instead of `%%`:

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-schema" % "0.0.56"
```

## Cross-Build for Several Scala Versions

When one build compiles for several Scala versions, let sbt pick the prefix and the `-experimental` flag per version. Put this in `build.sbt`:

```scala
crossScalaVersions := Seq("2.13.18", "3.3.7", "3.9.0")
scalaVersion       := crossScalaVersions.value.head

val zioBlocksVersion = "0.0.56"

// Scala 3.9+ uses the zio-blocks-next-* artifacts; older versions use zio-blocks-*.
def zioBlocks(module: String) = Def.setting {
  val prefix = CrossVersion.partialVersion(scalaVersion.value) match {
    case Some((3, minor)) if minor >= 9 => "zio-blocks-next"
    case _                              => "zio-blocks"
  }
  "dev.zio" %% s"$prefix-$module" % zioBlocksVersion
}

libraryDependencies ++= Seq(
  zioBlocks("schema").value,
  zioBlocks("streams").value
)

// The zio-blocks-next-* artifacts require -experimental.
scalacOptions ++= {
  CrossVersion.partialVersion(scalaVersion.value) match {
    case Some((3, minor)) if minor >= 9 => Seq("-experimental")
    case _                              => Seq.empty
  }
}
```

In a Scala.js or `crossProject` build, change `%%` to `%%%` inside `zioBlocks`.

## Add a Dependency With scala-cli

:::warning
scala-cli may report that a module "is outdated, update to 0.017". Ignore the hint: `0.017` is a mistyped February release that Maven ranks above the current versions, and following it downgrades you.
:::

On Scala 2.13 or Scala 3.3 to 3.8:

```scala
//> using scala 3.3.7
//> using dep dev.zio::zio-blocks-schema:0.0.56
```

On Scala 3.9 or newer:

```scala
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-next-schema:0.0.56
//> using option -experimental
```

For Scala.js, add the platform directive and put `::` before the version as well, the scala-cli equivalent of `%%%`:

```scala
//> using platform scala-js
//> using dep dev.zio::zio-blocks-schema::0.0.56
```

## Check That It Compiles

Paste this into your project and run it. If it prints JSON, the dependency is on the classpath:

```scala

case class Book(title: String, author: String)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}
```

Then encode a book to JSON:

```scala
Book("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi").toJsonString
// res0: String = "{\"title\":\"Zionomicon\",\"author\":\"John A. De Goes, Adam Fraser and Milad Khajavi\"}"
```

## Things to Know

- **Scala 3 only.** `sql`, `sql-zio`, `data-migration`, `projection`, `datastar`, and `http-htmx` have no Scala 2.13 artifacts.
- **JVM only.** `schema-avro`, `schema-bson`, `schema-thrift`, `codegen`, `smithy`, `telemetry-otel`, `sql-zio`, and `projection` have no Scala.js artifacts. The `sql` module's JDBC classes are also JVM-only.
- **Third-party dependencies.** Most modules depend only on the Scala standard library. The exceptions are:

  | Module                        | Pulls in                                                       |
  |-------------------------------|----------------------------------------------------------------|
  | `schema-avro`                 | `org.apache.avro:avro`                                         |
  | `schema-bson`                 | `org.mongodb:bson`                                             |
  | `schema-thrift`               | `org.apache.thrift:libthrift`, `jakarta.annotation-api`        |
  | `schema` on Scala.js          | `scala-java-time`, `scala-java-time-tzdb`                      |
  | `async` on Scala 3            | `dotty-cps-async`; also reaches you through `streams` and every module built on it |
  | `sql-zio`                     | `zio`                                                          |
  | `projection`                  | `zio`, `zio-streams`                                           |

- **Scala.js and `async` on Scala 3.9.** The `zio-blocks-next-async` artifact uses native JavaScript `async`/`await`. If the Scala.js linker reports that a feature requires ES2017, set the target in your Scala.js project:

  ```scala
  scalaJSLinkerConfig ~= (_.withESFeatures(_.withESVersion(org.scalajs.linker.interface.ESVersion.ES2017)))
  ```

## Going Further

- [Your First Five Minutes](./first-five-minutes.md) is the next page of Getting Started: one small program that uses three blocks together.
- [Your First Schema](../guides/your-first-schema.md) is the next step once `zio-blocks-schema` is on the classpath.
- The [reference](../reference/index.md) has a page for every module in the table above.
- Each `*-examples/` directory in the [repository](https://github.com/zio/zio-blocks) is a standalone sbt build that depends on the published artifacts; `schema-examples/build.sbt` is a working copy-paste template.

---

## Use ZIO Blocks with ZIO, cats-effect or Plain Scala

ZIO Blocks needs no effect system, so the question for most projects is not whether a block fits, but where its calls go. This page answers it in one table for each kind of call. [How to Use ZIO Blocks in a ZIO or cats-effect Application](../guides/use-zio-blocks-in-zio-or-cats-effect.md) shows every case with complete, runnable code.

## In Plain Scala

In a plain Scala program, there is nothing to arrange: call a block where you need it. Decoding returns an `Either`, a scope closes its resources when its block ends, and a stream runs when you call one of its run methods. Every example in this Getting Started section is plain Scala.

The one choice you still make is about threads. Some calls block the calling thread until they finish, such as a database query or `Stream#runCollect` on the JVM. In a command-line tool or a batch job that is fine; in a server, run them on a thread pool meant for blocking work, or use the `Async` variants that the blocks provide.

## In a ZIO or cats-effect Application

In an application that already runs on ZIO or cats-effect, each kind of ZIO Blocks call has one place where it belongs. Pure calls stay as they are, blocking calls move to the effect system's blocking pool, and resources and streams are wrapped once so that interruption and cleanup work across the boundary:

| Kind of call                                       | Example                                         | In ZIO                                     | In cats-effect                             |
| -------------------------------------------------- | ----------------------------------------------- | ------------------------------------------ | ------------------------------------------ |
| Pure and fast                                      | `toJsonString`, `fromJson`, `Chunk` operations  | Call it directly, or `ZIO.succeed`         | Call it directly, or `IO.pure`             |
| Returns an `Either`                                | `fromJson`, a decode with any format codec      | `ZIO.fromEither`                           | `IO.fromEither`                            |
| Blocks a thread                                    | SQL queries, `Stream#runCollect` on the JVM     | `ZIO.attemptBlocking`                      | `IO.blocking`                              |
| Owns a resource                                    | A `Resource` from Scope                         | Wrap it in `ZIO.acquireRelease`            | Wrap it in `Resource.make`                 |
| Runs asynchronously                                | An `Async` value                                | Bridge it with `ZIO.asyncInterrupt`        | Bridge it with `IO.async`                  |
| Produces a stream                                  | A ZIO Blocks `Stream`                           | Turn it into a `ZStream`                   | Turn it into an fs2 `Stream`               |

ZIO has library help for one case: the `sql-zio` module returns ZIO effects that already run on the blocking pool and provides the database as a `ZLayer` ([How to Use the sql Module from a ZIO Application](../guides/sql-from-zio.md)). Every other bridge is a few lines of your own code, and the full guide gives you those lines for both effect systems.

## Why It Works This Way

Because a block does its work when you call it, it cannot know whether the calling thread may block, or how your effect system cancels work. Keeping that decision in your code means the same block works in ZIO, cats-effect, Kyo, Ox, Akka or a plain `main` method, and you never adapt your application to a runtime that ZIO Blocks would otherwise bring along.

## Learn More

[How to Use ZIO Blocks in a ZIO or cats-effect Application](../guides/use-zio-blocks-in-zio-or-cats-effect.md) builds a small orders service that covers every row of the table, with cancellation and cleanup tested in both effect systems. If you are moving an existing code base, the [migration guides](../guides/index.md) map common Scala libraries to their ZIO Blocks counterparts. Finally, [Where Next](./where-next.md) suggests what to read after this section.

---

## Where Next

You now know what ZIO Blocks is, how its blocks are designed, and the four blocks most applications start with. The rest of the documentation has four kinds of pages, each for a different moment.

## How the Documentation Is Organized

Each kind of page answers a different question:

| Kind of page                                  | Read it when you want to...                          |
| --------------------------------------------- | ---------------------------------------------------- |
| [Tutorials](/guides/tutorials)                | Learn a block step by step, from nothing             |
| [How-to guides](/guides/how-to)               | Solve one specific task in a project                 |
| [Migration guides](/guides/migration)         | Move existing code from another Scala library        |
| [Reference](../reference/index.md)            | Look up every type and method of a block             |

## Pick a Tutorial by Goal

The tutorials are the natural next step. Each one builds a small program from scratch, and they are listed here by what you want to do:

| You want to...                                 | Start with                                                                                                           |
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| Read and write JSON                            | [Your First Schema: Round-Trip a Type Through JSON](../guides/your-first-schema.md)                                  |
| Change a data type without breaking old data   | [Evolve Your Data](../guides/evolve-your-data.md)                                                                    |
| Close files and connections safely             | [Compile-Time Resource Safety with Scope](../guides/compile-time-resource-safety-with-scope.md)                      |
| Load configuration into case classes           | [Build a Typed Configuration Loader](../guides/typed-config-loader.md)                                               |
| Run asynchronous code without blocking threads | [Getting Started with Async](../guides/async-getting-started.md)                                                     |
| Process a large file piece by piece            | [Your First Streaming Pipeline](../guides/your-first-streaming-pipeline.md)                                          |
| Store case classes in a database               | [Build Your First Database Repository on SQLite](../guides/your-first-sqlite-repository.md)                          |
| Render a web page                              | [Render Your First Server-Side HTML Page](../guides/your-first-html-page.md)                                         |
| Add logs, traces and metrics                   | [Instrument a Program with Logs, Traces and Metrics](../guides/instrument-a-program-with-logs-traces-and-metrics.md) |
| Combine several blocks into one small service  | [Build a Small JSON Backend from Blocks](../guides/small-json-backend-from-blocks.md)                                |

## Get Help

If something is unclear or does not work as described, ask on the [ZIO Discord](https://discord.gg/2ccFBr4) or open an issue in the [GitHub repository](https://github.com/zio/zio-blocks). The [API documentation](pathname:///api/) lists every public type of every block.

---

## Add Interactivity to a Page with HTMX

Welcome! This tutorial is for Scala developers who can already build an HTML page with the ZIO Blocks HTML module and now want parts of that page to update without a reload. You don't need any JavaScript, and you don't need prior experience with htmx. If you haven't built a page with the HTML module yet, start with [Render Your First Server-Side HTML Page](./your-first-html-page.md); this tutorial picks up where it ends.

## 1. Introduction

By the end of this tutorial, you will understand:

- How htmx turns an ordinary HTML element into one that asks the server for an HTML fragment and swaps it into the page
- How the typed `hx-*` attributes (`hxGet`, `hxTarget`, `hxTrigger`, `hxInclude`, `hxPushUrl`) attach to elements built with the HTML DSL
- How to debounce a search box with `HxTrigger` so the server is asked only when the reader pauses and the text has changed
- How one URL answers with a full page or a fragment, depending on the typed `HxRequest` header

We learn these concepts by turning a static list of Scala books into a live search page, one step at a time:

1. Serve a page that loads htmx
2. Make your first htmx request
3. Search as the reader types
4. Filter by category
5. Answer one URL with a full page or a fragment

At the end, we run the finished page and try it in a browser. We recommend reading from top to bottom — each section builds on the previous one.

The HTMX module is for Scala 3 only. To follow along, add it to your `build.sbt`:

```scala
// Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-http-htmx" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-http-htmx" % "0.0.56"
```

The HTMX module depends on the HTML module and the http-model module, so `Dom`, `Request` and `Response` come with it. The http-model module is compiled for Java 21, so the programs in this tutorial need JDK 21 or newer to run.

## 2. Background: The Server Sends HTML, htmx Swaps It

In a classic server-rendered site, every click loads a whole new page. A JavaScript single-page app avoids that, but then the browser has to fetch JSON, turn it into HTML itself and keep a second copy of the page's logic in JavaScript.

htmx takes a third path. It is a small JavaScript library that reads extra attributes on ordinary HTML elements. An attribute such as `hx-get="/search"` tells htmx to send a `GET /search` request when something happens to that element, and `hx-target="#results"` tells it where to put the answer. The server answers with a piece of HTML (a fragment), and htmx swaps it into the page. All of the HTML is still built on the server, by the same Scala code that builds the full page:

```text
  Browser                                       Server
  ───────                                       ──────
  <input hx-get="/search" hx-target="#results">
        │  the reader types "zio"
        │
        ├──── GET /search?q=zio ───────────────►  findBooks("zio")
        │     HX-Request: true                     │
        │                                          ▼
        ◄──── Zionomicon … ────  resultList(...).render
        │
        ▼
  htmx replaces the content of 
```

The ZIO Blocks HTMX module gives each `hx-*` attribute a typed key, such as `hxGet` and `hxTarget`, for the HTML DSL. Each key accepts only values that make sense for it, so a typo in a trigger or a target is a compile error instead of a page that silently does nothing. The module writes the attributes; the htmx library in the browser reads them. The attributes follow htmx 2, so this tutorial loads htmx 2.0.11. htmx 4 doesn't read several attributes the module offers, such as `hx-params` and `hx-disinherit`.

## 3. Serve a Page That Loads htmx

Let's start with the data: a `Book` with a title, an author and a category, and eight real Scala books. `findBooks` keeps the books whose title or author contains the search text and whose category matches, and `resultList` turns the matches into a ``, or into a short message when nothing matches:

```scala

final case class Book(title: String, author: String, category: String)

val books: List[Book] = List(
  Book("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", "Effects"),
  Book("Essential Effects", "Adam Rosien", "Effects"),
  Book("Practical FP in Scala", "Gabriel Volpe", "Effects"),
  Book("Functional Programming in Scala", "Paul Chiusano and Rúnar Bjarnason", "Functional Programming"),
  Book("Scala with Cats", "Noel Welsh and Dave Gurnell", "Functional Programming"),
  Book("Programming in Scala", "Martin Odersky, Lex Spoon and Bill Venners", "Language"),
  Book("Scala for the Impatient", "Cay S. Horstmann", "Language"),
  Book("Hands-on Scala Programming", "Li Haoyi", "Language")
)

val categories: List[String] = List("Effects", "Functional Programming", "Language")

def findBooks(query: String, category: String): List[Book] = {
  val needle = query.trim.toLowerCase
  books.filter { book =>
    val matchesText =
      needle.isEmpty || book.title.toLowerCase.contains(needle) || book.author.toLowerCase.contains(needle)
    val matchesCategory = category.isEmpty || book.category == category
    matchesText && matchesCategory
  }
}

def resultList(found: List[Book]): Dom.Element =
  if (found.isEmpty) p("No books match your search.")
  else ul(found.map(book => li(strong(book.title), s" by ${book.author}")))
```

The code above sets up everything the later sections reuse:

- `import zio.blocks.html._` brings in the HTML DSL, and `import zio.http._` brings in the http-model types `Request`, `Response`, `Body` and friends.
- `findBooks` ignores case and treats an empty query or an empty category as "match everything". Its name isn't `search` because the HTML DSL already has a `search` value for the `<search>` tag.
- `resultList` is the piece of HTML that changes as the reader searches. We render it inside the full page now, and later we send it on its own as a fragment.

Now let's build the page. It looks like the page from the HTML tutorial, with one addition: a `<script>` element in the head that loads htmx from a CDN:

```scala
val htmxScript: Dom.Element =
  script(src := "https://cdn.jsdelivr.net/npm/htmx.org@2.0.11/dist/htmx.min.js")

def staticPage: Dom.Element =
  html(lang := "en")(
    head(
      meta(charset := "utf-8"),
      title("Scala Bookshelf"),
      htmxScript
    ),
    body(
      h1("Scala Bookshelf"),
      div(id := "results", resultList(books))
    )
  )
```

The code above builds a normal page whose head loads htmx. `htmxScript` loads htmx 2.0.11 from the jsDelivr CDN, and pinning an exact version means the page keeps working the same way when a new htmx release comes out. `div(id := "results", ...)` wraps the result list in an element with an id. Nothing uses the id yet, but it is the spot htmx updates from the next section on.

Let's render the page to check it:

```scala
println(staticPage.render(indent = 2))
// <html lang="en">
//   <head>
//     <meta charset="utf-8"/>
//     <title>Scala Bookshelf</title>
//     <script src="https://cdn.jsdelivr.net/npm/htmx.org@2.0.11/dist/htmx.min.js"></script>
//   </head>
//   <body>
//     Scala Bookshelf
//     
//       
//         
//           Zionomicon
//            by John A. De Goes, Adam Fraser and Milad Khajavi
//         
//         
//           Essential Effects
//            by Adam Rosien
//         
//         
//           Practical FP in Scala
//            by Gabriel Volpe
//         
//         
//           Functional Programming in Scala
//            by Paul Chiusano and Rúnar Bjarnason
//         
//         
//           Scala with Cats
//            by Noel Welsh and Dave Gurnell
//         
//         
//           Programming in Scala
//            by Martin Odersky, Lex Spoon and Bill Venners
//         
//         
//           Scala for the Impatient
//            by Cay S. Horstmann
//         
//         
//           Hands-on Scala Programming
//            by Li Haoyi
//         
//       
//     
//   </body>
// </html>
```

The page is still static: loading htmx changes nothing until an element carries an `hx-*` attribute.

To serve the page, we write the application as a plain function from `Request` to `Response`. The guide [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md) explains this shape and provides a small adapter, `JdkHttp`, that runs such a function on the HTTP server built into the JDK. The companion examples use that adapter unchanged:

```scala
def htmlResponse(html: String): Response =
  Response.ok(Body.fromArray(html.getBytes("UTF-8"), ContentType.`text/html`))

def notFound(request: Request): Response =
  Response.text(Status.NotFound, s"No route for ${request.method.name} ${request.path.encode}")

def staticApp(request: Request): Response =
  (request.method, request.path.segments.toList) match {
    case (Method.GET, Nil) => htmlResponse(doctype.render + staticPage.render)
    case _                 => notFound(request)
  }
```

The code above:

- `htmlResponse` wraps rendered HTML in a `200 OK` response with `Content-Type: text/html`. `Body.fromString` would label the body `text/plain`, so we build it from bytes with an explicit content type.
- `notFound` answers any other request with a `404` and a short message naming the method and path.
- `staticApp` matches on the method and the path segments. `GET /` returns the page with a `<!DOCTYPE html>` line in front; every other request goes to `notFound`.

Because the application is just a function, we can call it without starting a server. This small helper builds a `GET` request for a target, passes it to an application and prints the status and the body. We use it in every section to see exactly what htmx would receive:

```scala
def call(app: Request => Response, target: String, headers: Header*): Unit = {
  val url      = URL.parse(target).fold(error => throw new IllegalArgumentException(error), identity)
  val request  = headers.foldLeft(Request.get(url))((request, header) => request.addHeader(header))
  val response = app(request)
  println(s"GET $target -> ${response.status.code}")
  println(response.body.asString())
}
```

The code above parses the target into a `URL`, builds the request with `Request.get` and adds any typed headers with `Request#addHeader`. The body is printed with `Body#asString`.

## 4. Make Your First htmx Request

Let's make the page fetch something. A button gets two attributes: `hxGet` names the URL htmx requests when the button is clicked, and `hxTarget` names the element that receives the answer. The server needs a new route, `GET /books`, that answers with the result list only, not a whole page:

```scala

val showAllButton: Dom.Element =
  button(hxGet := "/books", hxTarget := HxTarget.css("#results"))("Show all books")

def buttonPage: Dom.Element =
  html(lang := "en")(
    head(meta(charset := "utf-8"), title("Scala Bookshelf"), htmxScript),
    body(
      h1("Scala Bookshelf"),
      showAllButton,
      div(id := "results", p("Click the button to load the books."))
    )
  )

def buttonApp(request: Request): Response =
  (request.method, request.path.segments.toList) match {
    case (Method.GET, Nil)           => htmlResponse(doctype.render + buttonPage.render)
    case (Method.GET, List("books")) => htmlResponse(resultList(books).render)
    case _                           => notFound(request)
  }
```

The code above:

- `import zio.http.htmx._` brings in every typed attribute key, such as `hxGet` and `hxTarget`, and the value types they accept, such as `HxTarget`.
- `hxGet := "/books"` becomes `hx-get="/books"`. A button triggers its request on a click, which is htmx's default for buttons.
- `hxTarget := HxTarget.css("#results")` becomes `hx-target="#results"`. `hxTarget` only accepts an `HxTarget`, so a plain string such as `"#results"` doesn't compile. `HxTarget` also has `closest`, `find`, `next` and `previous` for targets relative to the element.
- The `/books` route renders `resultList(books)` without `<html>`, `<head>` or `<body>`. That is the fragment.

Let's look at the button the page now contains:

```scala
println(showAllButton.render)
// <button hx-get="/books" hx-target="#results">Show all books</button>
```

When the reader clicks it, htmx sends `GET /books` and puts the answer inside ``, replacing the "Click the button" paragraph. Let's send that same request ourselves:

```scala
call(buttonApp, "/books")
// GET /books -> 200
// Zionomicon by John A. De Goes, Adam Fraser and Milad KhajaviEssential Effects by Adam RosienPractical FP in Scala by Gabriel VolpeFunctional Programming in Scala by Paul Chiusano and Rúnar BjarnasonScala with Cats by Noel Welsh and Dave GurnellProgramming in Scala by Martin Odersky, Lex Spoon and Bill VennersScala for the Impatient by Cay S. HorstmannHands-on Scala Programming by Li Haoyi
```

This is the whole idea of htmx: the server answers with plain HTML, and the attributes decide when to ask and where the answer goes. By default htmx replaces the target's content (`innerHTML`); the `hxSwap` attribute takes an `HxSwap` value to choose another strategy, such as replacing the target itself with `HxSwap.OuterHTML`.

## 5. Search as the Reader Types

A button is a good start, but readers expect results to update as they type. A text input can carry the same `hxGet` and `hxTarget` attributes, and htmx sends the input's value along with the request, using the input's `name` as the query parameter. What we still need to choose is when the request fires. That is the job of `hxTrigger`.

If htmx sent a request on every key press, typing "scala" would send five requests, one per letter. A common fix is debouncing: wait until the reader pauses, then send one request. `HxTrigger` builds the trigger from the event name plus modifiers:

```scala

val debounced: HxTrigger = HxTrigger("keyup").changed.delay(300.millis)
```

The code above builds the trigger value in three parts:

- `HxTrigger("keyup")` fires on the `keyup` event, after each key press. `HxTrigger` has ready-made values for common events, such as `HxTrigger.click`, `HxTrigger.change` and `HxTrigger.input`, and takes any other event name as a string.
- `HxTrigger#changed` skips the request when the value hasn't changed since the last one, so arrow keys and Shift don't trigger a search.
- `HxTrigger#delay` with `300.millis` waits 300 milliseconds after the last `keyup` before sending. Each new key press restarts the wait, so a reader who types "scala" without pausing causes one request, not five.

To see the `hx-trigger` value this produces, we print `HxTrigger#render`:

```scala
println(debounced.render)
// keyup changed delay:300ms
```

Notice that the modifiers render in the order we call them, and that the duration is written as `300ms`. Let's put the trigger on a search box and add a `/search` route that reads the `q` parameter:

```scala
val liveSearchBox: Dom.Element.Void =
  input(
    `type`      := "search",
    name        := "q",
    placeholder := "Search by title or author",
    hxGet       := "/search",
    hxTrigger   := debounced,
    hxTarget    := HxTarget.css("#results")
  )

def liveSearchApp(request: Request): Response =
  (request.method, request.path.segments.toList) match {
    case (Method.GET, List("search")) =>
      val query = request.queryParams.getFirst("q").getOrElse("")
      htmlResponse(resultList(findBooks(query, "")).render)
    case _ => notFound(request)
  }
```

The code above:

- `name := "q"` is what htmx uses as the parameter name, so typing "zio" sends `GET /search?q=zio`.
- `input(...)` returns a `Dom.Element.Void`: `<input>` is a void element, which takes attributes but never children.
- `request.queryParams.getFirst("q")` reads the first value of the `q` parameter, or `None` when the request has none. An empty search shows every book.

Let's check the rendered search box and send two searches:

```scala
println(liveSearchBox.render)
// <input type="search" name="q" placeholder="Search by title or author" hx-get="/search" hx-trigger="keyup changed delay:300ms" hx-target="#results"/>

call(liveSearchApp, "/search?q=zio")
// GET /search?q=zio -> 200
// Zionomicon by John A. De Goes, Adam Fraser and Milad Khajavi
call(liveSearchApp, "/search?q=haskell")
// GET /search?q=haskell -> 200
// No books match your search.
```

The search for "zio" finds Zionomicon, and the search for "haskell" returns the "no match" paragraph. That paragraph is a fragment too, so the reader sees it right where the list was.

:::tip
htmx encodes parameter values with JavaScript's `encodeURIComponent`, so a space arrives as `%20` and `QueryParams` decodes it correctly. A plain HTML form submitted with `GET` encodes a space as `+` instead, which `QueryParams` keeps as a plus sign. If your page also has such a form, read [Values with Spaces and Plus Signs](./validate-query-parameters-and-headers.md#values-with-spaces-and-plus-signs).
:::

## 6. Filter by Category

Let's add a drop-down list of categories next to the search box. A `<select>` can carry `hxGet` and `hxTarget` too, and htmx sends its request on the browser's change event by default, so it needs no `hxTrigger`. There is one catch: each element sends only its own value. A search from the text box would forget the chosen category, and a change of category would forget the search text.

The `hxInclude` attribute fixes that. It names an element whose form fields htmx adds to the request. We put both controls in a `` and point `hxInclude` at it from both, so every request carries `q` and `category`:

```scala
val filterSearchBox: Dom.Element.Void =
  input(
    `type`      := "search",
    name        := "q",
    placeholder := "Search by title or author",
    hxGet       := "/search",
    hxTrigger   := debounced,
    hxTarget    := HxTarget.css("#results"),
    hxInclude   := HxTarget.css("#filters")
  )

val categoryFilter: Dom.Element =
  select(
    name      := "category",
    hxGet     := "/search",
    hxTarget  := HxTarget.css("#results"),
    hxInclude := HxTarget.css("#filters")
  )(
    option(value := "")("All categories") :: categories.map(category => option(value := category)(category))
  )

val filters: Dom.Element = div(id := "filters", filterSearchBox, categoryFilter)

def filterApp(request: Request): Response =
  (request.method, request.path.segments.toList) match {
    case (Method.GET, List("search")) =>
      val query    = request.queryParams.getFirst("q").getOrElse("")
      val category = request.queryParams.getFirst("category").getOrElse("")
      htmlResponse(resultList(findBooks(query, category)).render)
    case _ => notFound(request)
  }
```

The code above:

- `hxInclude := HxTarget.css("#filters")` becomes `hx-include="#filters"` on both controls. htmx collects every named field inside that ``, so the search box sends the category and the drop-down sends the search text.
- `option(value := "")("All categories")` is the first choice. Its empty value means "every category", which is exactly what `findBooks` does with an empty category.
- `categories.map(...)` adds one `<option>` per category, and the `::` puts "All categories" in front.
- `filterApp` reads both parameters and passes them to `findBooks`.

Let's check the rendered controls:

```scala
println(filters.render(indent = 2))
// 
//   <input type="search" name="q" placeholder="Search by title or author" hx-get="/search" hx-trigger="keyup changed delay:300ms" hx-target="#results" hx-include="#filters"/>
//   <select name="category" hx-get="/search" hx-target="#results" hx-include="#filters">
//     <option value="">All categories</option>
//     <option value="Effects">Effects</option>
//     <option value="Functional Programming">Functional Programming</option>
//     <option value="Language">Language</option>
//   </select>
// 
```

Notice that the controls sit in a ``, not a `<form>`. Pressing Enter inside a form submits it the old-fashioned way and reloads the page, which we don't want here. Now let's send the two requests the browser would send, first after choosing "Effects" with an empty search box, then after typing "scala" with "Language" selected:

```scala
call(filterApp, "/search?q=&category=Effects")
// GET /search?q=&category=Effects -> 200
// Zionomicon by John A. De Goes, Adam Fraser and Milad KhajaviEssential Effects by Adam RosienPractical FP in Scala by Gabriel Volpe
call(filterApp, "/search?q=scala&category=Language")
// GET /search?q=scala&category=Language -> 200
// Programming in Scala by Martin Odersky, Lex Spoon and Bill VennersScala for the Impatient by Cay S. HorstmannHands-on Scala Programming by Li Haoyi
```

The first request lists the three books about effects. The second finds three of the six books with "scala" in their title, the ones in the "Language" category.

## 7. One URL, a Full Page or a Fragment

The page works, but the address bar never changes. A reader can't bookmark a search or share it, and the browser's Back button leaves the page instead of going back to the previous search. The `hxPushUrl` attribute fixes that: after each request, htmx pushes the request's URL, such as `/search?q=zio&category=Effects`, into the address bar and the browser history.

That raises a new question. When the reader reloads the page or opens a bookmark, the browser itself requests `/search?q=zio&category=Effects`, and the `/search` route answers with a bare ``. The browser would show an unstyled list with no heading and no search box. The route must answer with a full page when the browser asks and with a fragment when htmx asks.

The server can tell them apart: htmx adds the header `HX-Request: true` to every request it sends. The HTMX module reads it as a typed header, `HxRequest`, from the `zio.http.htmx.headers` package:

```scala

def wantsFragment(request: Request): Boolean =
  request.header(HxRequest).exists(_.enabled) &&
    !request.header(HxHistoryRestoreRequest).exists(_.enabled)
```

The code above reads the header in two steps. `request.header(HxRequest)` parses the `HX-Request` header into an `HxRequest` value, or returns `None` when the header is missing, and `exists(_.enabled)` is `true` only for `HX-Request: true`. `HxHistoryRestoreRequest` handles one special case: when the reader goes back to a search that htmx no longer has in its history cache, htmx 2 requests that URL with `HX-Request: true` and `HX-History-Restore-Request: true`, and it puts the answer in place of the whole `<body>`. Answering with a fragment there would replace the heading and the search box with a bare list, so we answer with the full page.

This check is enough for this page because it has no `hxBoost` links; a page that uses `hxBoost` must also check `HxBoosted`, since a boosted link needs the full page too, as [How to Return HTMX Fragments, Full Pages and Response Headers](return-htmx-fragments-and-response-headers.md) shows.

Now the page itself takes the current search, so a full page loaded from `/search?q=zio&category=Effects` shows "zio" in the box, "Effects" selected and the matching books:

```scala
def searchBox(query: String): Dom.Element.Void =
  input(
    `type`      := "search",
    name        := "q",
    value       := query,
    placeholder := "Search by title or author",
    hxGet       := "/search",
    hxTrigger   := debounced,
    hxTarget    := HxTarget.css("#results"),
    hxInclude   := HxTarget.css("#filters"),
    hxPushUrl   := HxUrlUpdate(true)
  )

def categorySelect(current: String): Dom.Element =
  select(
    name      := "category",
    hxGet     := "/search",
    hxTarget  := HxTarget.css("#results"),
    hxInclude := HxTarget.css("#filters"),
    hxPushUrl := HxUrlUpdate(true)
  )(
    option(value := "")("All categories") ::
      categories.map(category => option(value := category)(category).when(category == current)(selected))
  )

def searchPage(query: String, category: String): Dom.Element =
  html(lang := "en")(
    head(meta(charset := "utf-8"), title("Scala Bookshelf"), htmxScript),
    body(
      h1("Scala Bookshelf"),
      div(id := "filters", searchBox(query), categorySelect(category)),
      div(id := "results", resultList(findBooks(query, category)))
    )
  )

def searchApp(request: Request): Response =
  (request.method, request.path.segments.toList) match {
    case (Method.GET, Nil) | (Method.GET, List("search")) =>
      val query    = request.queryParams.getFirst("q").getOrElse("")
      val category = request.queryParams.getFirst("category").getOrElse("")
      val html =
        if (wantsFragment(request)) resultList(findBooks(query, category)).render
        else doctype.render + searchPage(query, category).render
      htmlResponse(html).addHeader("vary", "hx-request")
    case _ => notFound(request)
  }
```

The code above:

- `hxPushUrl := HxUrlUpdate(true)` becomes `hx-push-url="true"` on both controls, so every search lands in the address bar.
- `value := query` fills the search box. The HTML module escapes attribute values, so a search for `"><script>` stays text inside `value="..."`, just like the user input in the HTML tutorial.
- `option(...).when(category == current)(selected)` adds the `selected` attribute only to the current category. `Dom.Element#when` adds modifiers to an element when a condition holds.
- `searchApp` answers `/` and `/search` with the same code. The only difference between the two answers is what `wantsFragment` says, so the full page and the fragment always show the same results.
- `addHeader("vary", "hx-request")` tells browsers and caches that the answer depends on the `HX-Request` header. Without it, a browser could show a cached fragment when the reader reloads the page.

Let's build the three kinds of requests the route can receive for the same URL: a reload by the browser, a search sent by htmx and a history restore sent by htmx:

```scala
val target = URL.parse("/search?q=zio&category=Effects").toOption.get

val reload         = Request.get(target)
val htmxSearch     = Request.get(target).addHeader(HxRequest(true))
val historyRestore = Request.get(target).addHeader(HxRequest(true)).addHeader(HxHistoryRestoreRequest(true))
```

Now let's ask `wantsFragment` about each of them:

```scala
println(wantsFragment(reload))
// false
println(wantsFragment(htmxSearch))
// true
println(wantsFragment(historyRestore))
// false
```

Only the htmx search gets the fragment. Let's send it, and then the same URL as a reload:

```scala
call(searchApp, "/search?q=zio&category=Effects", HxRequest(true))
// GET /search?q=zio&category=Effects -> 200
// Zionomicon by John A. De Goes, Adam Fraser and Milad Khajavi
call(searchApp, "/search?q=zio&category=Effects")
// GET /search?q=zio&category=Effects -> 200
// <!DOCTYPE html><html lang="en"><head><meta charset="utf-8"/><title>Scala Bookshelf</title><script src="https://cdn.jsdelivr.net/npm/htmx.org@2.0.11/dist/htmx.min.js"></script></head><body>Scala Bookshelf<input type="search" name="q" value="zio" placeholder="Search by title or author" hx-get="/search" hx-trigger="keyup changed delay:300ms" hx-target="#results" hx-include="#filters" hx-push-url="true"/><select name="category" hx-get="/search" hx-target="#results" hx-include="#filters" hx-push-url="true"><option value="">All categories</option><option value="Effects" selected>Effects</option><option value="Functional Programming">Functional Programming</option><option value="Language">Language</option></select>Zionomicon by John A. De Goes, Adam Fraser and Milad Khajavi</body></html>
```

The same URL gave two answers. The htmx request got the one-item list, and the reload got the whole page with `value="zio"` in the search box and `selected` on the "Effects" option. That is the pattern for every htmx page: build the page from the request's parameters, and let one header decide how much of it to send.

## 8. Putting It Together

This example combines all the concepts we've learned into one file. It serves the search page with the JDK HTTP adapter, and it keeps running until we stop it, so we can try it in a browser:

```scala title="zio-blocks-htmx-examples/src/main/scala/addinteractivitywithhtmx/CompleteExample.scala"
package addinteractivitywithhtmx

/**
 * Add Interactivity to a Page with HTMX — Complete Example
 *
 * A book search page served by the JDK HTTP server. The search box updates the
 * results 300 milliseconds after the reader stops typing, the category
 * drop-down filters them, and each search is pushed into the address bar. The
 * same `/search` URL answers an htmx request with the result list only and any
 * other request with the full page.
 *
 * Run with: cd zio-blocks-htmx-examples && sbt "runMain
 * addinteractivitywithhtmx.CompleteExample", then open http://localhost:8080/
 */
object CompleteExample {
  final case class Book(title: String, author: String, category: String)

  val books: List[Book] = List(
    Book("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", "Effects"),
    Book("Essential Effects", "Adam Rosien", "Effects"),
    Book("Practical FP in Scala", "Gabriel Volpe", "Effects"),
    Book("Functional Programming in Scala", "Paul Chiusano and Rúnar Bjarnason", "Functional Programming"),
    Book("Scala with Cats", "Noel Welsh and Dave Gurnell", "Functional Programming"),
    Book("Programming in Scala", "Martin Odersky, Lex Spoon and Bill Venners", "Language"),
    Book("Scala for the Impatient", "Cay S. Horstmann", "Language"),
    Book("Hands-on Scala Programming", "Li Haoyi", "Language")
  )

  val categories: List[String] = List("Effects", "Functional Programming", "Language")

  def findBooks(query: String, category: String): List[Book] = {
    val needle = query.trim.toLowerCase
    books.filter { book =>
      val matchesText =
        needle.isEmpty || book.title.toLowerCase.contains(needle) || book.author.toLowerCase.contains(needle)
      val matchesCategory = category.isEmpty || book.category == category
      matchesText && matchesCategory
    }
  }

  // The fragment htmx swaps into #results.
  def resultList(found: List[Book]): Dom.Element =
    if (found.isEmpty) p("No books match your search.")
    else ul(found.map(book => li(strong(book.title), s" by ${book.author}")))

  // Both controls send GET /search with every field inside #filters, swap the
  // answer into #results and push the request URL into the address bar.
  def searchBox(query: String): Dom.Element.Void =
    input(
      `type`      := "search",
      name        := "q",
      value       := query,
      placeholder := "Search by title or author",
      hxGet       := "/search",
      hxTrigger   := HxTrigger("keyup").changed.delay(300.millis),
      hxTarget    := HxTarget.css("#results"),
      hxInclude   := HxTarget.css("#filters"),
      hxPushUrl   := HxUrlUpdate(true)
    )

  def categoryFilter(current: String): Dom.Element =
    select(
      name      := "category",
      hxGet     := "/search",
      hxTarget  := HxTarget.css("#results"),
      hxInclude := HxTarget.css("#filters"),
      hxPushUrl := HxUrlUpdate(true)
    )(
      option(value := "")("All categories") ::
        categories.map(category => option(value := category)(category).when(category == current)(selected))
    )

  def page(query: String, category: String): Dom.Element =
    html(lang := "en")(
      head(
        meta(charset := "utf-8"),
        title("Scala Bookshelf"),
        script(src := "https://cdn.jsdelivr.net/npm/htmx.org@2.0.11/dist/htmx.min.js")
      ),
      body(
        h1("Scala Bookshelf"),
        div(id := "filters", searchBox(query), categoryFilter(category)),
        div(id := "results", resultList(findBooks(query, category)))
      )
    )

  // htmx 2 also sends HX-Request on a history cache miss, and then it needs the whole page.
  def wantsFragment(request: Request): Boolean =
    request.header(HxRequest).exists(_.enabled) &&
      !request.header(HxHistoryRestoreRequest).exists(_.enabled)

  def htmlResponse(html: String): Response =
    Response.ok(Body.fromArray(html.getBytes("UTF-8"), ContentType.`text/html`))

  def searchPage(request: Request): Response = {
    val query    = request.queryParams.getFirst("q").getOrElse("")
    val category = request.queryParams.getFirst("category").getOrElse("")
    val html     =
      if (wantsFragment(request)) resultList(findBooks(query, category)).render
      else doctype.render + page(query, category).render
    htmlResponse(html).addHeader("vary", "hx-request")
  }

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil) | (Method.GET, List("search")) => searchPage(request)
      case _                                                =>
        Response.text(Status.NotFound, s"No route for ${request.method.name} ${request.path.encode}")
    }

  def main(args: Array[String]): Unit = {
    val port   = args.headOption.flatMap(_.toIntOption).getOrElse(8080)
    val server = JdkHttp.start(app, port)
    sys.addShutdownHook(server.stop(0))
    println(s"Open http://localhost:$port/ in a browser. Press Ctrl+C to stop.")
  }
}
```

The pieces are the ones from the previous sections: `findBooks` and `resultList` from section 3, the debounced search box from section 5, the category filter with `hxInclude` from section 6, and `hxPushUrl` with the `wantsFragment` check from section 7. The `main` method starts the server on port 8080, or on the port given as the first argument, and stops it on Ctrl+C.

Run it and open `http://localhost:8080/` in a browser. Type "scala" in the search box: the list updates once you pause, and the address bar changes to `/search?q=scala&category=`. Pick "Language": the list narrows to three books. Reload the page: the search box still says "scala" and "Language" is still selected. Press the Back button: the previous search comes back. Open the browser's developer tools on the Network tab to watch htmx send one request per pause, each with the `HX-Request: true` header.

## 9. Running the Examples

All examples in this tutorial have corresponding runnable Scala files in the `zio-blocks-htmx-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/zio-blocks-htmx-examples
```

Each step starts the server on port 8080, prints what the server answers to the requests from that section, and keeps running so you can open `http://localhost:8080/` in a browser. Press Ctrl+C to stop it before running the next step. To use another port, pass it as an argument, for example `sbt "runMain addinteractivitywithhtmx.Step1StaticPageExample 9090"`. The steps share the book list and helpers in `Bookshelf.scala`, and the server adapter in `JdkHttp.scala` and the client in `Client.scala` are copied from the guide [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md).

### Share the Book List and Helpers

Every step below imports the book model, the search function and the response helpers from this file. It has no `main`, so there is nothing to run; read it first.

<details>
  <summary>zio-blocks-htmx-examples/src/main/scala/addinteractivitywithhtmx/Bookshelf.scala</summary>

```scala title="zio-blocks-htmx-examples/src/main/scala/addinteractivitywithhtmx/Bookshelf.scala" showLineNumbers
package addinteractivitywithhtmx

/**
 * Add Interactivity to a Page with HTMX — the bookshelf model
 *
 * Eight Scala books, a search function, the HTML fragment that lists search
 * results, the htmx script tag and a helper that wraps rendered HTML in a
 * `text/html` response. Every step of the tutorial reuses these.
 */
final case class Book(title: String, author: String, category: String)

object Bookshelf {
  val books: List[Book] = List(
    Book("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", "Effects"),
    Book("Essential Effects", "Adam Rosien", "Effects"),
    Book("Practical FP in Scala", "Gabriel Volpe", "Effects"),
    Book("Functional Programming in Scala", "Paul Chiusano and Rúnar Bjarnason", "Functional Programming"),
    Book("Scala with Cats", "Noel Welsh and Dave Gurnell", "Functional Programming"),
    Book("Programming in Scala", "Martin Odersky, Lex Spoon and Bill Venners", "Language"),
    Book("Scala for the Impatient", "Cay S. Horstmann", "Language"),
    Book("Hands-on Scala Programming", "Li Haoyi", "Language")
  )

  val categories: List[String] = List("Effects", "Functional Programming", "Language")

  /**
   * Keeps the books whose title or author contains `query` (ignoring case) and
   * whose category is `category`. An empty query or category matches every
   * book.
   */
  def findBooks(query: String, category: String): List[Book] = {
    val needle = query.trim.toLowerCase
    books.filter { book =>
      val matchesText =
        needle.isEmpty || book.title.toLowerCase.contains(needle) || book.author.toLowerCase.contains(needle)
      val matchesCategory = category.isEmpty || book.category == category
      matchesText && matchesCategory
    }
  }

  /** The HTML fragment that lists the books, or says that nothing matched. */
  def resultList(found: List[Book]): Dom.Element =
    if (found.isEmpty) p("No books match your search.")
    else ul(found.map(book => li(strong(book.title), s" by ${book.author}")))

  /** Loads htmx 2.0.11 from the jsDelivr CDN. */
  val htmxScript: Dom.Element = script(src := "https://cdn.jsdelivr.net/npm/htmx.org@2.0.11/dist/htmx.min.js")

  /**
   * Wraps rendered HTML in a 200 OK response with `Content-Type: text/html`.
   */
  def htmlResponse(html: String): Response =
    Response.ok(Body.fromArray(html.getBytes("UTF-8"), ContentType.`text/html`))

  /** Answers 404 for any route the step does not handle. */
  def notFound(request: Request): Response =
    Response.text(Status.NotFound, s"No route for ${request.method.name} ${request.path.encode}")

  /**
   * Starts the server on the port given as the first argument (8080 by default)
   * and says where to open it. The server keeps running until the program is
   * stopped with Ctrl+C.
   */
  def serve(app: Request => Response, args: Array[String]): Int = {
    val port   = args.headOption.flatMap(_.toIntOption).getOrElse(8080)
    val server = JdkHttp.start(app, port)
    sys.addShutdownHook(server.stop(0))
    port
  }

  def openInBrowser(port: Int): Unit =
    println(s"Open http://localhost:$port/ in a browser. Press Ctrl+C to stop.")
}
```

</details>

### Serve a Page That Loads htmx

This example serves the static book list with the htmx script in its head.

<details>
  <summary>zio-blocks-htmx-examples/src/main/scala/addinteractivitywithhtmx/Step1StaticPageExample.scala</summary>

```scala title="zio-blocks-htmx-examples/src/main/scala/addinteractivitywithhtmx/Step1StaticPageExample.scala" showLineNumbers
package addinteractivitywithhtmx

/**
 * Add Interactivity to a Page with HTMX — Step 1: A Static Page That Loads htmx
 *
 * Serves the bookshelf as an ordinary server-rendered page whose head loads the
 * htmx script. Nothing on the page is interactive yet.
 *
 * Run with: cd zio-blocks-htmx-examples && sbt "runMain
 * addinteractivitywithhtmx.Step1StaticPageExample"
 */
object Step1StaticPageExample {
  def page: Dom.Element =
    html(lang := "en")(
      head(
        meta(charset := "utf-8"),
        title("Scala Bookshelf"),
        htmxScript
      ),
      body(
        h1("Scala Bookshelf"),
        div(id := "results", resultList(books))
      )
    )

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil) => htmlResponse(doctype.render + page.render)
      case _                 => notFound(request)
    }

  def main(args: Array[String]): Unit = {
    val port = serve(app, args)
    Client.call(port, "GET", "/")
    openInBrowser(port)
  }
}
```

</details>

Run it from the `zio-blocks-htmx-examples` directory:

```bash
sbt "runMain addinteractivitywithhtmx.Step1StaticPageExample"
```

### Make Your First htmx Request

This example adds a button that loads the book list from `GET /books` into `#results`.

<details>
  <summary>zio-blocks-htmx-examples/src/main/scala/addinteractivitywithhtmx/Step2FirstHtmxRequestExample.scala</summary>

```scala title="zio-blocks-htmx-examples/src/main/scala/addinteractivitywithhtmx/Step2FirstHtmxRequestExample.scala" showLineNumbers
package addinteractivitywithhtmx

/**
 * Add Interactivity to a Page with HTMX — Step 2: Your First htmx Request
 *
 * A button with `hx-get` and `hx-target` asks the server for the `/books`
 * fragment and swaps it into the `#results` element, without reloading the
 * page.
 *
 * Run with: cd zio-blocks-htmx-examples && sbt "runMain
 * addinteractivitywithhtmx.Step2FirstHtmxRequestExample"
 */
object Step2FirstHtmxRequestExample {
  val showAllButton: Dom.Element =
    button(hxGet := "/books", hxTarget := HxTarget.css("#results"))("Show all books")

  def page: Dom.Element =
    html(lang := "en")(
      head(
        meta(charset := "utf-8"),
        title("Scala Bookshelf"),
        htmxScript
      ),
      body(
        h1("Scala Bookshelf"),
        showAllButton,
        div(id := "results", p("Click the button to load the books."))
      )
    )

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil)           => htmlResponse(doctype.render + page.render)
      case (Method.GET, List("books")) => htmlResponse(resultList(books).render)
      case _                           => notFound(request)
    }

  def main(args: Array[String]): Unit = {
    val port = serve(app, args)
    Client.call(port, "GET", "/books")
    openInBrowser(port)
  }
}
```

</details>

Run it from the `zio-blocks-htmx-examples` directory:

```bash
sbt "runMain addinteractivitywithhtmx.Step2FirstHtmxRequestExample"
```

### Search as the Reader Types

This example replaces the button with a debounced search box backed by `GET /search?q=...`.

<details>
  <summary>zio-blocks-htmx-examples/src/main/scala/addinteractivitywithhtmx/Step3LiveSearchExample.scala</summary>

```scala title="zio-blocks-htmx-examples/src/main/scala/addinteractivitywithhtmx/Step3LiveSearchExample.scala" showLineNumbers
package addinteractivitywithhtmx

/**
 * Add Interactivity to a Page with HTMX — Step 3: A Debounced Live Search
 *
 * A search box sends `GET /search?q=...` 300 milliseconds after the reader
 * stops typing, and only when the text has changed. The server answers with the
 * matching books as a fragment.
 *
 * Run with: cd zio-blocks-htmx-examples && sbt "runMain
 * addinteractivitywithhtmx.Step3LiveSearchExample"
 */
object Step3LiveSearchExample {
  val searchBox: Dom.Element.Void =
    input(
      `type`      := "search",
      name        := "q",
      placeholder := "Search by title or author",
      hxGet       := "/search",
      hxTrigger   := HxTrigger("keyup").changed.delay(300.millis),
      hxTarget    := HxTarget.css("#results")
    )

  def page: Dom.Element =
    html(lang := "en")(
      head(
        meta(charset := "utf-8"),
        title("Scala Bookshelf"),
        htmxScript
      ),
      body(
        h1("Scala Bookshelf"),
        searchBox,
        div(id := "results", resultList(books))
      )
    )

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil)            => htmlResponse(doctype.render + page.render)
      case (Method.GET, List("search")) =>
        val query = request.queryParams.getFirst("q").getOrElse("")
        htmlResponse(resultList(findBooks(query, "")).render)
      case _ => notFound(request)
    }

  def main(args: Array[String]): Unit = {
    val port = serve(app, args)
    Client.call(port, "GET", "/search?q=zio")
    Client.call(port, "GET", "/search?q=haskell")
    openInBrowser(port)
  }
}
```

</details>

Run it from the `zio-blocks-htmx-examples` directory:

```bash
sbt "runMain addinteractivitywithhtmx.Step3LiveSearchExample"
```

### Filter by Category

This example adds the category drop-down and uses `hxInclude` so both controls send both values.

<details>
  <summary>zio-blocks-htmx-examples/src/main/scala/addinteractivitywithhtmx/Step4CategoryFilterExample.scala</summary>

```scala title="zio-blocks-htmx-examples/src/main/scala/addinteractivitywithhtmx/Step4CategoryFilterExample.scala" showLineNumbers
package addinteractivitywithhtmx

/**
 * Add Interactivity to a Page with HTMX — Step 4: A Category Filter
 *
 * Adds a category drop-down next to the search box. Both controls send
 * `GET /search` with `hx-include` pointing at their shared `#filters`
 * container, so every request carries the search text and the category.
 *
 * Run with: cd zio-blocks-htmx-examples && sbt "runMain
 * addinteractivitywithhtmx.Step4CategoryFilterExample"
 */
object Step4CategoryFilterExample {
  val searchBox: Dom.Element.Void =
    input(
      `type`      := "search",
      name        := "q",
      placeholder := "Search by title or author",
      hxGet       := "/search",
      hxTrigger   := HxTrigger("keyup").changed.delay(300.millis),
      hxTarget    := HxTarget.css("#results"),
      hxInclude   := HxTarget.css("#filters")
    )

  val categoryFilter: Dom.Element =
    select(
      name      := "category",
      hxGet     := "/search",
      hxTarget  := HxTarget.css("#results"),
      hxInclude := HxTarget.css("#filters")
    )(
      option(value := "")("All categories") :: categories.map(category => option(value := category)(category))
    )

  def page: Dom.Element =
    html(lang := "en")(
      head(
        meta(charset := "utf-8"),
        title("Scala Bookshelf"),
        htmxScript
      ),
      body(
        h1("Scala Bookshelf"),
        div(id := "filters", searchBox, categoryFilter),
        div(id := "results", resultList(books))
      )
    )

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil)            => htmlResponse(doctype.render + page.render)
      case (Method.GET, List("search")) =>
        val query    = request.queryParams.getFirst("q").getOrElse("")
        val category = request.queryParams.getFirst("category").getOrElse("")
        htmlResponse(resultList(findBooks(query, category)).render)
      case _ => notFound(request)
    }

  def main(args: Array[String]): Unit = {
    val port = serve(app, args)
    Client.call(port, "GET", "/search?q=&category=Effects")
    Client.call(port, "GET", "/search?q=scala&category=Language")
    openInBrowser(port)
  }
}
```

</details>

Run it from the `zio-blocks-htmx-examples` directory:

```bash
sbt "runMain addinteractivitywithhtmx.Step4CategoryFilterExample"
```

### One URL, a Full Page or a Fragment

This example pushes each search into the address bar and answers `/search` with a full page or a fragment, depending on the `HX-Request` header.

<details>
  <summary>zio-blocks-htmx-examples/src/main/scala/addinteractivitywithhtmx/Step5FullPageOrFragmentExample.scala</summary>

```scala title="zio-blocks-htmx-examples/src/main/scala/addinteractivitywithhtmx/Step5FullPageOrFragmentExample.scala" showLineNumbers
package addinteractivitywithhtmx

/**
 * Add Interactivity to a Page with HTMX — Step 5: One URL, a Full Page or a
 * Fragment
 *
 * The controls push each search into the address bar with `hx-push-url`, so
 * `/search?q=...` must also work when the browser loads it directly. The server
 * checks the `HX-Request` header: an htmx request gets the result list only,
 * and any other request gets the whole page with the controls filled in.
 *
 * Run with: cd zio-blocks-htmx-examples && sbt "runMain
 * addinteractivitywithhtmx.Step5FullPageOrFragmentExample"
 */
object Step5FullPageOrFragmentExample {
  def searchBox(query: String): Dom.Element.Void =
    input(
      `type`      := "search",
      name        := "q",
      value       := query,
      placeholder := "Search by title or author",
      hxGet       := "/search",
      hxTrigger   := HxTrigger("keyup").changed.delay(300.millis),
      hxTarget    := HxTarget.css("#results"),
      hxInclude   := HxTarget.css("#filters"),
      hxPushUrl   := HxUrlUpdate(true)
    )

  def categoryFilter(current: String): Dom.Element =
    select(
      name      := "category",
      hxGet     := "/search",
      hxTarget  := HxTarget.css("#results"),
      hxInclude := HxTarget.css("#filters"),
      hxPushUrl := HxUrlUpdate(true)
    )(
      option(value := "")("All categories") ::
        categories.map(category => option(value := category)(category).when(category == current)(selected))
    )

  def page(query: String, category: String): Dom.Element =
    html(lang := "en")(
      head(
        meta(charset := "utf-8"),
        title("Scala Bookshelf"),
        htmxScript
      ),
      body(
        h1("Scala Bookshelf"),
        div(id := "filters", searchBox(query), categoryFilter(category)),
        div(id := "results", resultList(findBooks(query, category)))
      )
    )

  /**
   * True when htmx sent the request to swap part of the page. On a history
   * cache miss htmx 2 sends `HX-Request: true` as well, together with
   * `HX-History-Restore-Request: true`, and then it needs the whole page.
   */
  def wantsFragment(request: Request): Boolean =
    request.header(HxRequest).exists(_.enabled) &&
      !request.header(HxHistoryRestoreRequest).exists(_.enabled)

  def searchPage(request: Request): Response = {
    val query    = request.queryParams.getFirst("q").getOrElse("")
    val category = request.queryParams.getFirst("category").getOrElse("")
    val html     =
      if (wantsFragment(request)) resultList(findBooks(query, category)).render
      else doctype.render + page(query, category).render
    htmlResponse(html).addHeader("vary", "hx-request")
  }

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil) | (Method.GET, List("search")) => searchPage(request)
      case _                                                => notFound(request)
    }

  def main(args: Array[String]): Unit = {
    val port = serve(app, args)
    Client.call(port, "GET", "/search?q=zio&category=Effects")
    val htmxRequest = Request.get(URL.parse("/search?q=zio&category=Effects").toOption.get).addHeader(HxRequest(true))
    println("htmx request for /search?q=zio&category=Effects ->")
    println(app(htmxRequest).body.asString())
    openInBrowser(port)
  }
}
```

</details>

Run it from the `zio-blocks-htmx-examples` directory:

```bash
sbt "runMain addinteractivitywithhtmx.Step5FullPageOrFragmentExample"
```

### The Complete Example

The complete example from "Putting It Together" is a single file with everything in it:

```bash
sbt "runMain addinteractivitywithhtmx.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## 10. What You've Learned

In this tutorial, you learned:

- That htmx reads `hx-*` attributes on ordinary HTML elements, sends a request when something happens to the element, and swaps the server's HTML fragment into the page
- How the typed keys `hxGet`, `hxTarget`, `hxTrigger`, `hxInclude` and `hxPushUrl` attach to elements built with the HTML DSL, and how values such as `HxTarget` and `HxUrlUpdate` keep them correct at compile time
- How `HxTrigger("keyup").changed.delay(300.millis)` debounces a search box so the server is asked only when the reader pauses and the text has changed
- How `hxInclude` makes several controls send their values together
- How one route answers with a full page or a fragment by reading the typed `HxRequest` header, and why a history restore needs the full page

You now have a solid foundation in building interactive pages with htmx and ZIO Blocks. The next step is to explore the other `hx-*` attributes and the response headers that let the server steer the browser.

## 11. Where to Go Next

- **Want every attribute and value type?** Read the [HTMX](../reference/htmx/index.md) reference page, then the pages for [`HxTrigger`](../reference/htmx/hx-trigger.md), [`HxTarget`](../reference/htmx/hx-target.md) and [`HxSwap`](../reference/htmx/hx-swap.md).
- **Want the server to steer the browser?** [Request and Response Headers](../reference/htmx/response-headers.md) covers the typed request headers and the response headers that redirect, retarget or refresh the page.
- **Want the server to send full pages, out-of-band swaps and response headers?** [How to Return HTMX Fragments, Full Pages and Response Headers](./return-htmx-fragments-and-response-headers.md) shows one URL answering with a page or a fragment, and the headers that trigger events, redirect and retarget after a form post.
- **Need stricter query parameters?** [How to Validate Query Parameters and Headers and Answer 400](./validate-query-parameters-and-headers.md) decodes typed parameters and reports every mistake at once.
- **Want more on serving?** [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md) explains the adapter the examples use and its limits.
- **Prefer reactive signals?** [Build a Live Counter and Dashboard with Datastar](./build-a-live-dashboard-with-datastar.md) builds a counter and a streamed dashboard with typed `data-*` attributes and server-sent events; the [Datastar](../reference/datastar/index.md) reference covers every attribute.

---

## Getting Started with Async

Welcome! This tutorial introduces `Async[A]`, a zero-allocation effect type from ZIO Blocks that unifies ready values, failures, and genuinely suspended computations under a single type and combinator set. If you know basic Scala syntax and have a sense of what an effect type is, you have everything you need to follow along.

## 1. Introduction

By the end of this tutorial, you will be able to:

- Create ready-value effects with `Async.succeed` and transform them with `Async#map` and `Async#block`.
- Handle errors with `Async.fail`, `Async#catchAll`, and `Async#either`.
- Write sequential async code in direct style using `Async.async { … .await … }`.
- Bridge throw-based code and callback-based APIs with `Async.attempt` and `Async.promise` / `Completer`.
- Fork background computations with `Async.start` and cancel them with `Async.Running#cancel`.

To add the async module to your project, include this dependency in your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-async" % "0.0.56"
```

Then bring the full DSL into scope at the top of each file you use it from:

```scala

```

We recommend reading from top to bottom — each section builds directly on the one before it.

## 2. Background: What Is `Async[A]`?

`Async[A]` was designed to solve a specific problem: code often juggles three different kinds of values — results that are already available, computations that need to wait for I/O or a callback, and failures. Treating these differently in different places creates friction. `Async[A]` is one abstraction that covers all three.

The design's most important property is its **happy-path allocation budget: zero**. When you chain `Async.succeed`, `Async#map`, and another `map` call together, every step is a plain function call. No wrapper objects accumulate on the heap. Only a computation that truly suspends — waiting for a callback, a timer, or a thread — leaves a pending object behind. This is the zero-allocation promise: you pay for suspension only when you actually suspend.

We will explore `Async[A]` through an order-processing scenario: looking up a user via a callback API, parsing an order ID, checking stock availability in the background, and recovering gracefully from any failure — one concept at a time.

## 3. Ready Values: `Async.succeed`, `map`, and `block`

The simplest async computation is one that already has its answer. `Async.succeed(value)` wraps an available value into an `Async[A]` so it can take part in async chains without allocating anything. Once you have an `Async`, you transform it with `Async#map` and drive it to its final result with `Async#block`.

Here we wrap the integer `42`, double it with `map`, then extract the result:

```scala

val result: Int = Async.succeed(42).map(_ * 2).block
println(s"Ready mapped: $result")
```

Expected output:

```text
Ready mapped: 84
```

- `Async.succeed(42)` wraps `42` as a ready-value `Async[Int]` with zero allocation.
- Calling `map` with `_ * 2` applies the function; because the input is already ready, the entire chain is just a function call — no suspension object is created.
- The `block` call is the eager driver — it polls the async until it completes and returns the final value. It is safe to call on the JVM; on Scala.js it throws if the computation is still suspended.

Try changing `42` to a different number and watch the output change accordingly.

<details>
  <summary>Runnable version: <code>Concept1ReadyValuesExample.scala</code></summary>

```scala title="async-examples/src/main/scala/zio/blocks/async/gettingstarted/Concept1ReadyValuesExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-async:0.0.56
// Run from this folder:     scala-cli run Concept1ReadyValuesExample.scala
// Or from async-examples:   sbt "runMain zio.blocks.async.gettingstarted.Concept1ReadyValuesExample"

package zio.blocks.async.gettingstarted

/**
 * Getting Started with Async — Ready Values: `Async.succeed`, `map`, and
 * `block`
 *
 * Wraps an available value with `Async.succeed`, transforms it with
 * `Async#map`, and drives it to its result with `Async#block`.
 */
object Concept1ReadyValuesExample {
  def main(args: Array[String]): Unit = {
    val result: Int = Async.succeed(42).map(_ * 2).block
    println(s"Ready mapped: $result")
  }
}
```

</details>

## 4. Error Handling: `Async.fail`, `catchAll`, and `either`

Not every computation succeeds. `Async.fail(throwable)` creates a failed async that short-circuits all downstream `map` calls without invoking their functions. `Async#catchAll` recovers by applying a function that returns a new `Async`.

Here we create a failed computation and recover from it with `catchAll`:

```scala

val recovered: String = Async
  .fail(new Exception("oops"))
  .catchAll(_ => Async.succeed("default"))
  .block
println(s"Recovered: $recovered")
```

Expected output:

```text
Recovered: default
```

- `Async.fail(new Exception("oops"))` creates a computation that carries the exception as its failure.
- Calling `catchAll` with a recovery function intercepts the failure; the lambda ignores the specific error and returns a ready-value replacement.
- The `block` call drives the recovered chain to its result, `"default"`.

Sometimes you want to observe both the success and failure branches as plain data rather than handle them immediately. `Async#either` reifies both outcomes as `Either[Throwable, A]` so you can pattern-match on them:

```scala

val observed: Either[Throwable, Int] = Async
  .fail(new Exception("error"))
  .either
  .block
println(s"Observed: $observed")
```

Expected output:

```text
Observed: Left(java.lang.Exception: error)
```

- Calling `either` wraps the failure in `Left`; a successful result would appear in `Right`, turning the async's outcome into an ordinary Scala value.
- The `block` call materialises that `Either` so the learner can inspect it.

<details>
  <summary>Runnable version, with both blocks of this section: <code>Concept2ErrorHandlingExample.scala</code></summary>

```scala title="async-examples/src/main/scala/zio/blocks/async/gettingstarted/Concept2ErrorHandlingExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-async:0.0.56
// Run from this folder:     scala-cli run Concept2ErrorHandlingExample.scala
// Or from async-examples:   sbt "runMain zio.blocks.async.gettingstarted.Concept2ErrorHandlingExample"

package zio.blocks.async.gettingstarted

/**
 * Getting Started with Async — Error Handling: `Async.fail`, `catchAll`, and
 * `either`
 *
 * Recovers a failed `Async` with `Async#catchAll`, then observes a failure as
 * data with `Async#either`.
 */
object Concept2ErrorHandlingExample {
  def main(args: Array[String]): Unit = {
    val recovered: String = Async
      .fail(new Exception("oops"))
      .catchAll(_ => Async.succeed("default"))
      .block
    println(s"Recovered: $recovered")

    val observed: Either[Throwable, Int] = Async
      .fail(new Exception("error"))
      .either
      .block
    println(s"Observed: $observed")
  }
}
```

</details>

## 5. Direct Style: `Async.async` and `await`

Writing nested `Async#flatMap` chains is precise but becomes hard to read when many steps depend on each other. `Async.async { … }` lets you write that same sequencing in direct style: inside the block, call `Async#await` on any `Async` to extract its value and bind it to a local variable, as if you were writing straight-line code. The compiler rewrites every `await` call into a `flatMap` chain at compile time, so the runtime behaviour is identical.

Here we compose a user name and an order ID without a single explicit `flatMap`:

```scala

val summary: String = Async.async {
  val user  = Async.succeed("Ada").await
  val order = Async.succeed(9001).await
  s"${user}'s order ${order}"
}.block
println(s"Summary: $summary")
```

Expected output:

```text
Summary: Ada's order 9001
```

- `Async.async { … }` opens a macro-powered block; the entire expression produces an `Async[String]`.
- Calling `await` on `Async.succeed("Ada")` extracts `"Ada"` and binds it to `user`; this is not a blocking call — the macro rewrites it into a `flatMap` continuation.
- Calling `await` on `Async.succeed(9001)` similarly binds the order number to `order`.
- The final string expression becomes the block's result value.
- The `block` call drives the whole composed async to completion.

:::caution[`await` Is Only Valid Inside `Async.async { … }`]
The `await` method is enforced by the compiler to be used only inside an `Async.async { … }` block. On Scala 3 the inline expansion fails with:

```
".await may only be used directly inside an Async.async { ... } block."
```

On Scala 2, the `@compileTimeOnly` annotation fires at the same point. Try deleting the `Async.async { … }` wrapper — the compiler will tell you immediately.
:::

<details>
  <summary>Runnable version: <code>Concept3DirectStyleExample.scala</code></summary>

```scala title="async-examples/src/main/scala/zio/blocks/async/gettingstarted/Concept3DirectStyleExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-async:0.0.56
// Run from this folder:     scala-cli run Concept3DirectStyleExample.scala
// Or from async-examples:   sbt "runMain zio.blocks.async.gettingstarted.Concept3DirectStyleExample"

package zio.blocks.async.gettingstarted

/**
 * Getting Started with Async — Direct Style: `Async.async` and `await`
 *
 * Sequences two `Async` values in direct style: each `.await` inside
 * `Async.async { ... }` is rewritten into a `flatMap` at compile time.
 */
object Concept3DirectStyleExample {
  def main(args: Array[String]): Unit = {
    val summary: String = Async.async {
      val user  = Async.succeed("Ada").await
      val order = Async.succeed(9001).await
      s"${user}'s order ${order}"
    }.block
    println(s"Summary: $summary")
  }
}
```

</details>

## 6. Bridging Exceptions: `Async.attempt`

Scala code often signals failure by throwing exceptions rather than returning error values. `Async.attempt(body)` evaluates a block that may throw and captures any exception as an async failure, turning it into a value that `catchAll` can recover. A block that succeeds produces a ready-value `Async`; one that throws produces a failed `Async`.

The example parses a well-formed string and then a malformed one:

```scala

val good: Int = Async.attempt("42".toInt).block
println(s"Parsed: $good")

val bad: Either[Throwable, Int] = Async.attempt("oops".toInt).either.block
println(s"Failed parse: $bad")
```

Expected output:

```text
Parsed: 42
Failed parse: Left(java.lang.NumberFormatException: For input string: "oops")
```

- `Async.attempt("42".toInt)` evaluates `"42".toInt`; because it succeeds, the result `42` becomes a ready-value `Async[Int]`.
- `Async.attempt("oops".toInt)` evaluates `"oops".toInt`; the `NumberFormatException` is caught and becomes a failed `Async`.
- Chaining `either` and `block` drives the failed async and reifies its outcome as `Left(…)`.

<details>
  <summary>Runnable version: <code>Concept4BridgingExceptionsExample.scala</code></summary>

```scala title="async-examples/src/main/scala/zio/blocks/async/gettingstarted/Concept4BridgingExceptionsExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-async:0.0.56
// Run from this folder:     scala-cli run Concept4BridgingExceptionsExample.scala
// Or from async-examples:   sbt "runMain zio.blocks.async.gettingstarted.Concept4BridgingExceptionsExample"

package zio.blocks.async.gettingstarted

/**
 * Getting Started with Async — Bridging Exceptions: `Async.attempt`
 *
 * Turns a throwing block into an `Async`: a well-formed string parses to a
 * ready value, a malformed one becomes a failure that `Async#either` reifies.
 */
object Concept4BridgingExceptionsExample {
  def main(args: Array[String]): Unit = {
    val good: Int = Async.attempt("42".toInt).block
    println(s"Parsed: $good")

    val bad: Either[Throwable, Int] = Async.attempt("oops".toInt).either.block
    println(s"Failed parse: $bad")
  }
}
```

</details>

## 7. Callback Bridging: `Async.promise` and `Completer`

Many real-world APIs — database drivers, network libraries, timers — signal completion by calling a callback rather than returning a value. `Async.promise` lets you lift these APIs into `Async` without rewriting them. It suspends the computation and provides a `Completer[A]` — a thread-safe, one-shot handle — that you can pass to the callback. Calling `completer.succeed(value)` from any thread resolves the async and wakes up any awaiter.

The example starts a background thread that completes the promise after a short delay:

<Tabs groupId="scala-version" defaultValue="scala2">
<TabItem value="scala2" label="Scala 2">

```scala

val result: String = Async
  .promise[String] { c =>
    new Thread {
      override def run(): Unit = {
        Thread.sleep(10)
        c.succeed("hello from callback")
      }
    }.start()
  }
  .block
println(s"Promise resolved: $result")
```

</TabItem>
<TabItem value="scala3" label="Scala 3">

```scala

val result: String = Async
  .promise[String] {
    val c = summon[Completer[String]]
    new Thread {
      override def run(): Unit = {
        Thread.sleep(10)
        c.succeed("hello from callback")
      }
    }.start()
  }
  .block
println(s"Promise resolved: $result")
```

</TabItem>
</Tabs>

Expected output:

```text
Promise resolved: hello from callback
```

- `Async.promise[String] { … }` opens the promise body; on Scala 2 the `Completer[String]` arrives as an explicit parameter `c`; on Scala 3 it arrives as a context function argument retrieved with `summon[Completer[String]]`.
- We capture the completer in `c` so it can be referenced from the Thread's `run()` method — implicits and givens do not propagate across thread boundaries, so we capture explicitly.
- Calling `c.succeed("hello from callback")` completes the promise; the first call wins and subsequent calls are no-ops.
- The `block` call waits until the completer fires and returns the resolved value.

:::warning[Scala 3: do not annotate the binder]
On Scala 3, `Async.promise[String] { (c: Completer[String]) => ... }` compiles, but the statements inside the lambda never run, so the completer is never completed and `block` waits forever. Use the Scala 3 form above, `summon[Completer[String]]` or `{ c ?=> ... }`. Compiler warning E190 at the call site is the only hint.
:::

<details>
  <summary>Runnable version, the Scala 3 tab: <code>Concept5CallbackBridgingExample.scala</code></summary>

```scala title="async-examples/src/main/scala/zio/blocks/async/gettingstarted/Concept5CallbackBridgingExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-async:0.0.56
// Run from this folder:     scala-cli run Concept5CallbackBridgingExample.scala
// Or from async-examples:   sbt "runMain zio.blocks.async.gettingstarted.Concept5CallbackBridgingExample"

package zio.blocks.async.gettingstarted

/**
 * Getting Started with Async — Callback Bridging: `Async.promise` and
 * `Completer`
 *
 * The Scala 3 version of the section's code: `summon[Completer[String]]`
 * retrieves the completer, which a background thread completes after 10 ms.
 */
object Concept5CallbackBridgingExample {
  def main(args: Array[String]): Unit = {
    val result: String = Async
      .promise[String] {
        val c = summon[Completer[String]]
        new Thread {
          override def run(): Unit = {
            Thread.sleep(10)
            c.succeed("hello from callback")
          }
        }.start()
      }
      .block
    println(s"Promise resolved: $result")
  }
}
```

</details>

## 8. Forking and Cancellation: `start` and `Async.Running`

By default, async chains run eagerly on the calling thread until the first suspension or completion. To evaluate a block on a **background worker** (a separate JVM thread or a Scala.js microtask) instead, pass it to `Async.start`. This returns an `Async.Running[A]` handle — itself an `Async[A]` — representing the in-flight computation. You can join it by calling `block` on the handle, or stop waiting for it with `cancel`.

The first block forks a computation and joins it, printing the name of the thread each step runs on:

```scala

val running: Async.Running[Int] = Async.start {
  println(s"Computing on ${Thread.currentThread.getName}")
  42 * 2
}
val joined: Int = running.block
println(s"Joined on ${Thread.currentThread.getName}: $joined")
```

Expected output:

```text
Computing on zio-blocks-async-eval
Joined on main: 84
```

- `Async.start { … }` hands the block to a background worker and returns the `Running` handle immediately; the first line shows the block running on the worker thread, `zio-blocks-async-eval`.
- `running.block` blocks the calling thread until the background computation finishes and returns the result `84`, back on `main`.

The extension method `Async#start` drives an `Async` you have already built. Notice what happens when that `Async` is a chain of ready values:

```scala

val ready: Async.Running[Int] = Async
  .succeed(42)
  .map { x => println(s"map runs on ${Thread.currentThread.getName}"); x * 2 }
  .start
println(s"Joined: ${ready.block}")
```

Expected output:

```text
map runs on main
Joined: 84
```

- The `map` function runs on `main`, while the chain is being built and before `start` is called: its input is already available, so there is nothing to wait for.
- Calling `start` on the result finds a ready value and settles at once, without a background worker. This is the zero-allocation promise from section 2 at work — a chain leaves the calling thread only when it genuinely suspends, such as waiting on a `Completer`.

The last block demonstrates cancellation:

```scala

val running2: Async.Running[Int] = Async.start {
  Thread.sleep(100)
  println("Body finished after cancel")
  99
}
running2.cancel()
println("Cancelled running2")
Thread.sleep(200) // give the body time to finish
```

Expected output:

```text
Cancelled running2
Body finished after cancel
```

- `Async.start { … }` forks the block onto a background worker; the call returns the `Running` handle immediately.
- Calling `cancel` stops the driver, so no result is ever published to an awaiter. The call is idempotent.
- The second line shows that `cancel` does not interrupt the block itself: it finishes on its own thread. Cancelling an `Async.start` means you have stopped waiting for the result, not that the work has stopped.

:::caution
A cancelled run never settles: it neither succeeds nor fails. Calling `block` on a cancelled handle waits forever on the JVM and throws `IllegalStateException` on Scala.js, so cancel only a handle whose every consumer you own. The [`Async.Running`](../reference/async.md#asyncrunning) reference covers the details.
:::

<details>
  <summary>Runnable version, with all three blocks of this section: <code>Concept6ForkingAndCancellationExample.scala</code></summary>

```scala title="async-examples/src/main/scala/zio/blocks/async/gettingstarted/Concept6ForkingAndCancellationExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-async:0.0.56
// Run from this folder:     scala-cli run Concept6ForkingAndCancellationExample.scala
// Or from async-examples:   sbt "runMain zio.blocks.async.gettingstarted.Concept6ForkingAndCancellationExample"

package zio.blocks.async.gettingstarted

/**
 * Getting Started with Async — Forking and Cancellation: `start` and
 * `Async.Running`
 *
 * Forks a block onto a background worker with `Async.start` and joins it, shows
 * that `Async#start` on a ready chain settles on the calling thread, and
 * cancels a running handle whose body still runs to completion.
 */
object Concept6ForkingAndCancellationExample {
  def main(args: Array[String]): Unit = {
    val running: Async.Running[Int] = Async.start {
      println(s"Computing on ${Thread.currentThread.getName}")
      42 * 2
    }
    val joined: Int = running.block
    println(s"Joined on ${Thread.currentThread.getName}: $joined")

    val ready: Async.Running[Int] = Async
      .succeed(42)
      .map { x => println(s"map runs on ${Thread.currentThread.getName}"); x * 2 }
      .start
    println(s"Joined: ${ready.block}")

    val running2: Async.Running[Int] = Async.start {
      Thread.sleep(100)
      println("Body finished after cancel")
      99
    }
    running2.cancel()
    println("Cancelled running2")
    Thread.sleep(200) // give the body time to finish
  }
}
```

</details>

## 9. Putting It Together

Let's combine all six concepts into a single order-processing pipeline. The program bridges a callback-based user-lookup API with `Async.promise`, safely parses an order ID with `Async.attempt`, runs a stock check in the background with `Async.start`, sequences everything in direct style inside `Async.async`, and recovers from any failure with `catchAll`. Each step reuses a section's code with order-processing values: the callback delivers the user name `"Ada"` instead of a greeting, `Async.attempt` parses the order ID `"9001"`, the background block is a stock check that sleeps 5 ms and returns `true`, and the recovery handler reports the failure's message instead of a fixed default:

```scala title="async-examples/src/main/scala/zio/blocks/async/gettingstarted/CompleteExample.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-async:0.0.56
// Run from this folder:     scala-cli run CompleteExample.scala
// Or from async-examples:   sbt "runMain zio.blocks.async.gettingstarted.CompleteExample"

package zio.blocks.async.gettingstarted

/**
 * Getting Started with Async — Putting It Together
 *
 * An order-processing pipeline assembled from the sections: a callback lookup
 * bridged with `Async.promise`, an order ID parsed with `Async.attempt`, a
 * stock check forked with `Async.start`, all sequenced in `Async.async` and
 * recovered with `Async#catchAll`.
 */
object CompleteExample {
  def main(args: Array[String]): Unit = {

    // Step 1: Bridge a callback-based user-lookup API with promise + Completer
    val userLookup: Async[String] = Async.promise[String] {
      val c = summon[Completer[String]]
      new Thread {
        override def run(): Unit = {
          Thread.sleep(10)
          c.succeed("Ada")
        }
      }.start()
    }

    // Step 2: Parse an order ID that might be malformed
    val orderId: Async[Int] = Async.attempt("9001".toInt)

    // Step 3: Start a stock-availability check on a background worker
    val stockCheck: Async.Running[Boolean] = Async.start { Thread.sleep(5); true }

    // Step 4: Compose all steps in direct style and recover from any failure
    val report: String = Async.async {
      val user    = userLookup.await
      val id      = orderId.await
      val inStock = stockCheck.await
      s"Order $id for $user: ${if (inStock) "in stock" else "out of stock"}"
    }.catchAll { err =>
      Async.succeed(s"Order failed: ${err.getMessage}")
    }.block

    println(s"Report: $report")
  }
}
```

Expected output:

```text
Report: Order 9001 for Ada: in stock
```

## 10. Running the Examples

All runnable code from this tutorial is in [`async-examples/src/main/scala/zio/blocks/async/gettingstarted/`](https://github.com/zio/zio-blocks/tree/main/async-examples/src/main/scala/zio/blocks/async/gettingstarted).
Run any file from that folder with `scala-cli run <File>.scala`, or from `async-examples` with
`sbt "runMain zio.blocks.async.gettingstarted.<ObjectName>"`.

## 11. What You've Learned

By completing this tutorial, you can now:

- Create ready-value effects with `Async.succeed`, transform them with `map`, and drive them to a result with `block`.
- Handle failures with `Async.fail`, recover them with `catchAll`, and observe both outcomes as data with `either`.
- Write sequential async pipelines in direct style using `Async.async { … .await … }` — the compiler threads the `flatMap` calls for you.
- Lift throw-based Scala code safely into the async error channel with `Async.attempt`.
- Bridge a callback-based API with `Async.promise` and `Completer`, fork background work with `Async.start`, and hold a cancellable `Async.Running` handle.

## 12. Where to Go Next

If your code already calls `Future`-based, Java `CompletionStage`-based, or callback-based APIs, the next step is [How to Wrap Callback and Future-Based APIs in Async](./wrap-callback-and-future-apis.md), which brings each of them into `Async` and shows how to cancel the underlying call.

The [Async reference page](../reference/async.md) documents every method and combinator with full signatures — it is the natural next stop once you are comfortable with the basics in this tutorial.

If your application manages resources that need deterministic cleanup — database connections, file handles, or connection pools — read the [Compile-Time Resource Safety with Scope](./compile-time-resource-safety-with-scope.md) tutorial, which shows how to tie resource lifetimes to lexical scopes and compose them without try/finally boilerplate.

---

## How to Avoid Allocation on Hot Paths

In this guide, we take a price-tick handler that runs for every trade on a market data feed and remove the allocations from its hot loop, one at a time. By the end, you will know how to keep a ZIO Blocks `Stream` of `Long` prices on its primitive lane, how to read and build a `Chunk` of prices without boxing them, how to keep per-tick state in a `Maybe` instead of an `Option`, and how to record a metric with fixed labels without building an `Attributes` value on every call. Every claim comes with a number we measured, and the last section shows how to measure your own code the same way.

Each section is independent: pick the one that matches the allocation you have. This guide covers:

- Measuring the bytes a loop allocates per element
- Keeping stream elements on a primitive lane, and reading them in a custom `Sink`
- Wrapping, reading and building `Chunk`s without boxing
- Replacing `Option` with `Maybe` for state written on every tick
- Pre-binding metric labels with `Counter#bind`
- Checking the results with JMH and its GC profiler

## The Problem

A loop that runs millions of times per second turns every small allocation into garbage-collector work. On a 64-bit JVM with compressed pointers, a boxed `java.lang.Long` takes 24 bytes, a `Some` 16, and a tuple key for a metrics map another 24; at a million ticks per second that is tens of megabytes of short-lived objects per second, which the JVM pays for in young-generation collections, cache misses and lost throughput.

Generic Scala code allocates this way without saying so. A tick handler written with the standard library often looks like this:

```scala
final case class Tick(symbol: String, priceMicros: Long, size: Int)

final class TickHandler {
  var lastTick: Option[Tick] = None
  val counts                 = scala.collection.mutable.HashMap.empty[(String, String), Long]

  // Called for every trade
  def onTick(tick: Tick): Unit = {
    lastTick = Some(tick)
    val key = (tick.symbol, "XNAS")
    counts.update(key, counts.getOrElse(key, 0L) + 1L)
  }

  // Called for every batch of prices, in micro-dollars
  def turnoverCents(prices: Array[Long]): Long =
    prices.iterator.map(_ / 10000L).filter(_ > 18750L).foldLeft(0L)(_ + _)
}
```

Nothing here looks expensive, yet on this machine `TickHandler#onTick` allocates 40 to 64 bytes per tick and `turnoverCents` about 62 bytes per price. The tick handler pays for the `Some`, the boxed count and, unless the JIT removes it, the tuple key; the turnover pays for a boxed `Long` at every step of `Iterator#map`, `Iterator#filter` and `Iterator#foldLeft`. None of it is visible in the source.

ZIO Blocks has a tool for each of these allocations. In this guide, we replace them one by one and measure what each change saves.

## Prerequisites

We use four modules: `streams`, `chunk`, `maybe` and `telemetry`. On Scala 2.13 and Scala 3.3 to 3.8, add:

```scala
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-streams"   % "0.0.56",
  "dev.zio" %% "zio-blocks-chunk"     % "0.0.56",
  "dev.zio" %% "zio-blocks-maybe"     % "0.0.56",
  "dev.zio" %% "zio-blocks-telemetry" % "0.0.56"
)
```

Scala 3.9 and newer use the `zio-blocks-next-*` artifact names and need `scalacOptions += "-experimental"`; [How to Add ZIO Blocks to Your Build](../getting-started/installation.md) has the details. The `streams` and `telemetry` modules need JDK 21 or newer on the JVM. We assume you know what boxing is and have read [Your First Streaming Pipeline](./your-first-streaming-pipeline.md) or written a `Stream` before.

The examples use these imports:

```scala

```

## Measure Allocation Per Tick

Before changing anything, we need a number. The JVM counts the bytes each thread allocates, and `com.sun.management.ThreadMXBean` exposes that counter, so a few lines give us bytes per run of any block. The block must run a few thousand times first, because the JIT compiler removes some allocations once it has compiled the code, and a cold measurement reports allocations that a hot loop never makes:

```scala

object Allocation {
  private val threads =
    ManagementFactory.getThreadMXBean.asInstanceOf[com.sun.management.ThreadMXBean]

  @volatile private var sink: Int = 0

  // Runs `body` until the JIT has compiled it, then returns the average number
  // of bytes one run allocates on the current thread.
  def bytesPerRun(body: => Any): Long = {
    var i = 0
    while (i < 2000) { sink ^= System.identityHashCode(body); i += 1 }
    val runs   = 200
    val before = threads.getCurrentThreadAllocatedBytes
    i = 0
    while (i < runs) { sink ^= System.identityHashCode(body); i += 1 }
    (threads.getCurrentThreadAllocatedBytes - before) / runs
  }
}
```

The `sink` field keeps the JIT from discarding a result nobody reads. The counter is per thread, so the block must run on the calling thread, which every example in this guide does.

The running example is a batch of 10,000 trades for one symbol. Prices are fixed-point `Long`s in micro-dollars, and every one of them lies far outside the small-value cache that `java.lang.Long.valueOf` keeps for -128 to 127, so boxing one always allocates:

```scala
final case class Tick(symbol: String, priceMicros: Long, size: Int)

val tickCount           = 10000
val prices: Array[Long] = Array.tabulate(tickCount)(i => 187000000L + (i % 997) * 1300L)
val sizes: Array[Int]   = Array.tabulate(tickCount)(i => 100 + i % 50)
val ticks: Array[Tick]  = Array.tabulate(tickCount)(i => Tick("ACME", prices(i), sizes(i)))
```

Dividing `Allocation.bytesPerRun` by `tickCount` gives bytes per tick. The figures in this guide come from the companion examples and from the JMH benchmark in [Check the Numbers With JMH](#check-the-numbers-with-jmh), run on one machine with JDK 25; treat them as rough sizes, not guarantees.

## Keep a Stream on a Primitive Lane

A ZIO Blocks `Stream` carries each element on a lane: one of the primitive JVM types, or `AnyRef` for objects. On a primitive lane a `Long` moves from the source through `Stream#map`, `Stream#filter` and the sink as a raw `long`, so the pipeline allocates nothing per element. The lane is picked at compile time from the element type each operator produces, so the same code that boxed with an `Iterator` stops boxing when it is a `Stream`:

```scala
val iteratorBytes = Allocation.bytesPerRun {
  prices.iterator.map(_ / 10000L).filter(_ > 18750L).foldLeft(0L)(_ + _)
}

val streamBytes = Allocation.bytesPerRun {
  Stream.fromArray(prices).map(_ / 10000L).filter(_ > 18750L).runFold(0L)(_ + _)
}

println(s"Iterator: ${iteratorBytes.toDouble / tickCount} bytes per tick")
println(s"Stream:   ${streamBytes.toDouble / tickCount} bytes per tick")
```

On this machine the `Iterator` version allocates about 62 bytes per tick and the `Stream` version about 0.1: some 700 to 1,000 bytes per run of 10,000 ticks, for the stream's own nodes and the `Either` that `Stream#runFold` returns. Both compute the same turnover:

```scala
val viaIterator = prices.iterator.map(_ / 10000L).filter(_ > 18750L).foldLeft(0L)(_ + _)
// viaIterator: Long = 113490340L
val viaStream   = Stream.fromArray(prices).map(_ / 10000L).filter(_ > 18750L).runFold(0L)(_ + _)
// viaStream: Either[Nothing, Long] = Right(113490340L)
```

`Stream.fromArray` reads straight from the array. `Stream.fromIterable` also picks the `Long` lane for a `List[Long]`, but the list already holds one boxed `Long` per element, and the stream unboxes them as it reads; build hot-path data in an array or a `Chunk` instead.

### See Which Lane a Stream Is On

`Stream` has no method that reports its lane, but the reader a sink receives does: `Reader#jvmType` names the lane the elements arrive on. A one-line sink built with `Sink.create` (JVM only) turns that into a diagnostic:

```scala
def lane[A]: Sink[Nothing, A, JvmType] = Sink.create[Nothing, A, JvmType](reader => reader.jvmType)
```

Running a few pipelines into it shows which operators keep the `Long` lane and which leave it:

```scala
Stream.fromArray(prices).map(_ / 10000L).filter(_ > 18750L).run(lane)
// res2: Either[Nothing, JvmType] = Right(Long)
Stream.fromArray(prices).map(_.toDouble / 1e6).run(lane)
// res3: Either[Nothing, JvmType] = Right(Double)
Stream.fromArray(prices).scan(0L)(_ + _).run(lane)
// res4: Either[Nothing, JvmType] = Right(Long)
(Stream.fromArray(prices) && Stream.fromArray(sizes)).run(lane)
// res5: Either[Nothing, JvmType] = Right(AnyRef)
Stream.fromArray(prices).grouped(100).run(lane)
// res6: Either[Nothing, JvmType] = Right(AnyRef)
```

The rules behind these results:

| Operators                                                                                       | Lane of the result                                               |
| ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| `Stream#filter`, `Stream#take`, `Stream#drop`, `Stream#takeWhile`, `Stream#tapEach`             | The input's lane                                                 |
| `Stream#map`, `Stream#collect`, `Stream#scan`, `Stream#mapAccum`, `Stream#flatMap`, `Stream#++` | The lane of the result type (`Long => Double` moves to `Double`) |
| `Stream#&&` (zip), `Stream#grouped`, `Stream#chunked`, `Stream#sliding`                         | `AnyRef`: their elements are tuples or `Chunk`s                  |
| Any operator whose result type is a reference or `Any`                                          | `AnyRef`                                                         |

None of these changes is reported at compile time. A `Stream#map` whose function returns `Any` (for example, an `if` with a `Long` in one branch and a `String` in the other) quietly moves the stream to the `AnyRef` lane and boxes every element from there on.

### Replace a Zip With an Index

Zipping two primitive streams is the most common way to leave the lane by accident: `&&` pairs prices with sizes as tuples, so every element becomes an object. When both sides are arrays of the same length, a stream of indices does the same work on the `Long` lane:

```scala
val zippedBytes = Allocation.bytesPerRun {
  (Stream.fromArray(prices) && Stream.fromArray(sizes)).map { case (p, s) => p * s }.runFold(0L)(_ + _)
}

val indexedBytes = Allocation.bytesPerRun {
  Stream.range(0, tickCount).map(i => prices(i) * sizes(i)).runFold(0L)(_ + _)
}

println(s"zipped:  ${zippedBytes.toDouble / tickCount} bytes per tick")
println(s"indexed: ${indexedBytes.toDouble / tickCount} bytes per tick")
```

The zipped version allocates about 55 bytes per tick on this machine, the indexed one under 0.05. Both compute the same notional value:

```scala
(Stream.fromArray(prices) && Stream.fromArray(sizes)).map { case (p, s) => p * s }.runFold(0L)(_ + _)
// res8: Either[Nothing, Long] = Right(233618271696500L)
Stream.range(0, tickCount).map(i => prices(i) * sizes(i)).runFold(0L)(_ + _)
// res9: Either[Nothing, Long] = Right(233618271696500L)
```

### Read Primitives in a Custom Sink

`Stream#runFold` and `Sink.foldLeft` with a `Long` start value read the lane and keep the running total unboxed. The other built-in aggregates also read the lane without boxing, but in 0.0.56 they box their running total once per element, a known bug: `Sink.sumLong`, `Sink.sumInt`, `Sink.sumDouble`, `Sink.count`, and `Stream#runFold` or `Sink.foldLeft` with a `Double` start value. Measured side by side:

```scala
val sumSinkBytes   = Allocation.bytesPerRun(Stream.fromArray(prices).run(Sink.sumLong))
val countSinkBytes = Allocation.bytesPerRun(Stream.fromArray(prices).run(Sink.count))
val runFoldBytes   = Allocation.bytesPerRun(Stream.fromArray(prices).runFold(0L)(_ + _))

println(s"Sink.sumLong: ${sumSinkBytes.toDouble / tickCount} bytes per tick")
println(s"Sink.count:   ${countSinkBytes.toDouble / tickCount} bytes per tick")
println(s"runFold(0L):  ${runFoldBytes.toDouble / tickCount} bytes per tick")
```

On this machine `Sink.sumLong` and `Sink.count` allocate about 48 bytes per tick and `Stream#runFold` with `0L` nothing per tick, so on a hot path we sum and count with a `Long` fold instead (`runFold(0L)((n, _) => n + 1)` counts). When we need a sink of our own, we pull from the reader with its primitive reads instead of `Reader#read`, which returns a boxed element.

For a `Long` stream, the read that never boxes and never mistakes a price for the end of the stream is `Reader#readLongs` with a one-element array: it copies the next element into the array and returns `1`, or returns `-1` once the stream has ended. A sink that tracks the highest price looks like this:

```scala
val highestPrice: Sink[Nothing, Long, Long] = Sink.create[Nothing, Long, Long] { reader =>
  val one     = new Array[Long](1)
  var highest = Long.MinValue
  var n       = reader.readLongs(one, 0, 1)
  while (n >= 0) {
    if (one(0) > highest) highest = one(0)
    n = reader.readLongs(one, 0, 1)
  }
  highest
}
```

It reads every price on the `Long` lane:

```scala
Stream.fromArray(prices).run(highestPrice)
// res11: Either[Nothing, Long] = Right(188294800L)
```

Measured the same way as before, a run over 10,000 ticks allocates about 224 bytes in total, all of it per run (the stream, the reader and the one-element array), none of it per tick.

`Sink.create` exists only on the JVM. On Scala.js, or to support both blocking and asynchronous runs, build the sink with `Sink.createBoth`, which takes one function for a `Reader.SyncReader` and one for a `Reader.AsyncReader`; both reader kinds have the same primitive reads. The [Sink reference](../reference/streams/core/sink.md) covers both factories.

### Choose an End-of-Stream Signal That Cannot Collide

Each primitive lane has a read that widens the element into a larger type, so a sentinel outside the element's range can mark the end of the stream and no real element can be mistaken for it. `Reader#readInt(sentinel)` returns a `Long`, and the convention is to pass `Long.MinValue`; `Reader#readFloat(sentinel)` returns a `Double`, and the convention is to pass `Double.MaxValue`; `Reader#readByte()` returns `0` to `255`, or `-1` at the end.

`Long` and `Double` have no wider type to escape into, so `Reader#readLong` and `Reader#readDouble` return the element type itself, and the sentinel we pass in can equal a real element. A real element equal to that marker ends the loop early, and nothing tells us it happened. For example, a sink that sums price changes and picks `0L` as its marker stops at the first unchanged price:

```scala
val priceChanges = Array(5L, -3L, 0L, 7L)

val sumUntilMarker: Sink[Nothing, Long, Long] = Sink.create[Nothing, Long, Long] { reader =>
  var sum   = 0L
  var value = reader.readLong(0L)
  while (value != 0L) {
    sum += value
    value = reader.readLong(0L)
  }
  sum
}
```

It returns `2`, while the built-in `Sink.sumLong`, which uses the count-returning read, returns the full `9`:

```scala
Stream.fromArray(priceChanges).run(sumUntilMarker)
// res12: Either[Nothing, Long] = Right(2L)
Stream.fromArray(priceChanges).run(Sink.sumLong)
// res13: Either[Nothing, Long] = Right(9L)
```

No marker value is safe for every feed, so read `Long` and `Double` lanes with `Reader#readLongs` or `Reader#readDoubles` and a one-element array, as `highestPrice` does. The library's own sinks do the same, which is why `Sink.sumLong` keeps the zero. Do not add a check against the marker inside the loop to work around a collision: the count-returning read costs nothing extra per element and never collides. The [zero-boxing reference](../reference/streams/execution-and-compatibility/zero-boxing.md) lists the end-of-stream signal of every lane.

## Read and Build Chunks Without Boxing

A `Chunk` built from a primitive array stores its elements in that array, unboxed. Three habits decide whether our code keeps that advantage: how we create the chunk, how we read it, and how we build a new one.

### Wrap an Array Instead of Copying It

`Chunk.fromArray` wraps the array we already have; `Chunk.from` (like `Chunk.fromIterable`) copies it first:

```scala
val copyBytes = Allocation.bytesPerRun(Chunk.from(prices))
val wrapBytes = Allocation.bytesPerRun(Chunk.fromArray(prices))

println(s"Chunk.from:      $copyBytes bytes per batch")
println(s"Chunk.fromArray: $wrapBytes bytes per batch")
```

On this machine `Chunk.from` allocates about 80,000 bytes per batch of 10,000 prices (8 bytes per price for the copy), and `Chunk.fromArray` allocates 32 bytes, the wrapper object alone.

:::warning
A chunk from `Chunk.fromArray` shares the array with the caller. Writing to the array afterwards changes the chunk, which breaks the promise that a `Chunk` is immutable. Wrap only arrays that nothing writes to again, such as a buffer we have just filled and handed over.
:::

### Read Elements With a Primitive Accessor

`Chunk#foldLeft` is generic: its accumulator and each element pass through `(S, A) => S` as objects, so summing 10,000 prices allocates 24 to 48 bytes per price on this machine (one or two boxed `Long`s, depending on how much of the fold the JIT inlines). An index loop with `Chunk#long` reads each `Long` straight from the backing array:

```scala
val priceChunk = Chunk.fromArray(prices)

def totalMicros(chunk: Chunk[Long]): Long = {
  var sum = 0L
  var i   = 0
  while (i < chunk.length) {
    sum += chunk.long(i)
    i += 1
  }
  sum
}
```

Both give the same total, but the loop allocates nothing per price:

```scala
priceChunk.foldLeft(0L)(_ + _)
// res15: Long = 1876455143500L
totalMicros(priceChunk)
// res16: Long = 1876455143500L
```

The same accessor exists for every primitive: `Chunk#int`, `Chunk#double`, `Chunk#byte`, `Chunk#float`, `Chunk#short`, `Chunk#char` and `Chunk#boolean`. They read the array directly only when the chunk is backed by a primitive array, which is what `Chunk.fromArray`, `Chunk#map` on such a chunk, and the primitive builders below produce. A chunk made by `Chunk#++`, `Chunk#drop` or `Chunk#slice` is a view over other chunks, and its accessors go through `Chunk#apply`, which returns a boxed element; call `Chunk#materialize` once before the loop to get an array-backed copy. On an array-backed chunk, `Chunk#long` never boxes, whatever the JIT does.

### Build a Chunk With a Primitive Builder

`Chunk.newBuilder[Long]` is generic, so it boxes each price before storing it. `ChunkBuilder.Long` stores raw `long`s, and `sizeHint` sizes its array once instead of doubling it as it grows:

```scala
def pricesAbove(limitMicros: Long): Chunk[Long] = {
  val builder = new ChunkBuilder.Long
  builder.sizeHint(prices.length)
  var i = 0
  while (i < prices.length) {
    if (prices(i) > limitMicros) builder.addOne(prices(i))
    i += 1
  }
  builder.result()
}
```

The result is a chunk on a primitive array:

```scala
pricesAbove(188000000L).length
// res17: Int = 2270
```

On this machine, adding all 10,000 prices costs about 58 bytes per price through `Chunk.newBuilder[Long]` and 16 through a sized `ChunkBuilder.Long`: 8 bytes for the builder's array and 8 for the copy that `ChunkBuilder#result` makes. `ChunkBuilder.Int`, `ChunkBuilder.Double`, `ChunkBuilder.Byte` and the other primitive builders work the same way.

## Replace Option With Maybe for Per-Tick State

State that changes on every tick, such as the last trade seen, is a field written in the hot loop. With `Option`, every write allocates a new `Some`, and because the `Some` is stored in a field, the JIT cannot remove it. On Scala 3, `Maybe` stores a present reference as the reference itself, so writing it allocates nothing:

```scala
final class LastTrade {
  var tick: Maybe[Tick] = Maybe.absent

  def onTick(t: Tick): Unit = tick = Maybe(t)

  def lastPriceMicros: Long = tick.fold(0L)(_.priceMicros)
}
```

We feed it a batch of ticks:

```scala
val lastTrade = new LastTrade
ticks.foreach(lastTrade.onTick)
```

The state now holds the last tick's price:

```scala
lastTrade.lastPriceMicros
// res19: Long = 187037700L
```

To compare it with the `Option` version, we write every tick of the batch into each kind of field:

```scala
final class OptionLastTrade { var tick: Option[Tick] = None }

val optionTrade = new OptionLastTrade
val maybeTrade  = new LastTrade

val optionBytes = Allocation.bytesPerRun {
  var i = 0
  while (i < tickCount) { optionTrade.tick = Some(ticks(i)); i += 1 }
  optionTrade
}

val maybeBytes = Allocation.bytesPerRun {
  var i = 0
  while (i < tickCount) { maybeTrade.onTick(ticks(i)); i += 1 }
  maybeTrade
}

println(s"Option field: ${optionBytes.toDouble / tickCount} bytes per tick")
println(s"Maybe field:  ${maybeBytes.toDouble / tickCount} bytes per tick")
```

On this machine, writing `Some(tick)` into an `Option[Tick]` field costs 16 bytes per tick, and writing `Maybe(tick)` into a `Maybe[Tick]` field costs nothing.

Absence is the `Absent` singleton, not `null`: `Maybe.absent` returns it, and `Maybe.apply` turns a `null` argument into it. `Maybe.present(null)` instead returns a `Present(null)` wrapper, the one case besides nested `Maybe`s where a flat value allocates.

Three details decide whether `Maybe` saves anything:

- Read it with `Maybe#fold`, `Maybe#getOrElse`, `Maybe#isPresent` and `Maybe#get`, or `Maybe#foreach`. On Scala 3, `case Present(t)` calls `Present.unapply`, which returns an `Option` and so allocates a `Some` per match unless the JIT removes it.
- `Maybe[Long]`, `Maybe[Double]` and other primitive types still box the value, because `Maybe[A]` erases to `Object`. The companion example measures 24 bytes per tick for a `Maybe[Long]` price field against 40 for `Option[Long]`. For a primitive, keep the value in a primitive field, or keep the object it belongs to in a `Maybe`, as `LastTrade` does.
- On Scala 2.13, `Maybe` is a sealed trait, and `Maybe.present` allocates a `MaybeValue.Present` for every present value, so it saves nothing over `Option` there.

The [Maybe reference](../reference/maybe.md) covers the full API.

<details>
  <summary>Runnable version, with the <code>Maybe[Long]</code> price-field measurement: <code>Step6MaybeExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/avoidallocation/Step6MaybeExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-maybe:0.0.56
//> using options -experimental
//> using file Allocation.scala
// Run from this folder:      scala-cli run Step6MaybeExample.scala
// Or from schema-examples:   sbt "runMain avoidallocation.Step6MaybeExample"

package avoidallocation

/**
 * How to Avoid Allocation on Hot Paths — Replace Option With Maybe for Per-Tick
 * State
 *
 * Writes the last trade of every tick into an `Option` field and a `Maybe`
 * field, and shows that a `Maybe[Long]` still boxes its value.
 */
object Step6MaybeExample {

  final case class Tick(symbol: String, priceMicros: Long, size: Int)

  val tickCount           = 10000
  val prices: Array[Long] = Array.tabulate(tickCount)(i => 187000000L + (i % 997) * 1300L)
  val sizes: Array[Int]   = Array.tabulate(tickCount)(i => 100 + i % 50)
  val ticks: Array[Tick]  = Array.tabulate(tickCount)(i => Tick("ACME", prices(i), sizes(i)))

  final class LastTrade {
    var tick: Maybe[Tick] = Maybe.absent

    def onTick(t: Tick): Unit = tick = Maybe(t)

    def lastPriceMicros: Long = tick.fold(0L)(_.priceMicros)
  }

  final class OptionLastTrade { var tick: Option[Tick] = None }

  // The primitive case: the price itself in each kind of field
  final class OptionLastPrice { var price: Option[Long] = None        }
  final class MaybeLastPrice  { var price: Maybe[Long] = Maybe.absent }

  def main(args: Array[String]): Unit = {
    val lastTrade = new LastTrade
    ticks.foreach(lastTrade.onTick)
    println(s"last price: ${lastTrade.lastPriceMicros}")

    val optionTrade = new OptionLastTrade
    val maybeTrade  = new LastTrade

    val optionBytes = Allocation.bytesPerRun {
      var i = 0
      while (i < tickCount) { optionTrade.tick = Some(ticks(i)); i += 1 }
      optionTrade
    }

    val maybeBytes = Allocation.bytesPerRun {
      var i = 0
      while (i < tickCount) { maybeTrade.onTick(ticks(i)); i += 1 }
      maybeTrade
    }

    println(s"Option field: ${optionBytes.toDouble / tickCount} bytes per tick")
    println(s"Maybe field:  ${maybeBytes.toDouble / tickCount} bytes per tick")

    val optionPrice = new OptionLastPrice
    val maybePrice  = new MaybeLastPrice

    val optionPriceBytes = Allocation.bytesPerRun {
      var i = 0
      while (i < tickCount) { optionPrice.price = Some(prices(i)); i += 1 }
      optionPrice
    }

    val maybePriceBytes = Allocation.bytesPerRun {
      var i = 0
      while (i < tickCount) { maybePrice.price = Maybe(prices(i)); i += 1 }
      maybePrice
    }

    println(s"Option[Long] field: ${optionPriceBytes.toDouble / tickCount} bytes per tick")
    println(s"Maybe[Long] field:  ${maybePriceBytes.toDouble / tickCount} bytes per tick")
  }
}
```

</details>

## Pre-Bind Metric Labels Outside the Loop

Counting ticks per symbol and venue with a telemetry `Counter` is the natural way to export the feed's rate. Passing the labels on every call makes the counter build an `Attributes` value and look it up in its table each time:

```scala
val meter        = MeterProvider.builder.build().get("feed")
val ticksCounter = meter.counterBuilder("ticks").build()
```

When the label values are known before the loop, `Counter#bind` builds the `Attributes` once and returns a `BoundCounter` that adds straight to the counter's accumulator for those labels:

```scala
val acmeOnXnas: BoundCounter =
  ticksCounter.bind(Attributes.builder.put("symbol", "ACME").put("venue", "XNAS").build)
```

A call with label pairs and a call on the bound counter write to the same data point, which `Counter#collect` reads back:

```scala
ticksCounter.add(1L, "symbol" -> "ACME", "venue" -> "XNAS")
acmeOnXnas.add(1L)

ticksCounter.collect() match {
  case MetricData.SumData(points) => points.map(_.value)
  case _                          => Nil
}
// res23: List[Long] = List(2L)
```

The difference is what each call allocates. We count every tick of the batch both ways:

```scala
val pairBytes = Allocation.bytesPerRun {
  var i = 0
  while (i < tickCount) {
    ticksCounter.add(1L, "symbol" -> "ACME", "venue" -> "XNAS")
    i += 1
  }
}

val boundBytes = Allocation.bytesPerRun {
  var i = 0
  while (i < tickCount) {
    acmeOnXnas.add(1L)
    i += 1
  }
}

println(s"labels per call: ${pairBytes.toDouble / tickCount} bytes per tick")
println(s"bound counter:   ${boundBytes.toDouble / tickCount} bytes per tick")
```

On this machine the label pairs cost about 232 bytes per tick, and the bound counter 0.

The same holds for `Histogram#bind` and `Gauge#bind`, and for the labeled instruments. `Meter#labeledCounter` declares the label names once, so each call passes only the values, but `LabeledCounter#add` still builds a fresh `Attributes` on every call. On a hot path, call `LabeledCounter#bind` once and keep the handle:

```scala
val labeledTicks = meter.labeledCounter("ticks.labeled", "symbol", "venue")
val acmeHandle   = labeledTicks.bind("ACME", "XNAS")
```

Measured the same way, `LabeledCounter#add` costs about 400 bytes per tick on this machine, more than the label pairs, and the handle costs nothing:

```scala
val labeledBytes = Allocation.bytesPerRun {
  var i = 0
  while (i < tickCount) {
    labeledTicks.add(1L, "ACME", "XNAS")
    i += 1
  }
}

val handleBytes = Allocation.bytesPerRun {
  var i = 0
  while (i < tickCount) {
    acmeHandle.add(1L)
    i += 1
  }
}

println(s"LabeledCounter#add: ${labeledBytes.toDouble / tickCount} bytes per tick")
println(s"bound handle:       ${handleBytes.toDouble / tickCount} bytes per tick")
```

When label values change per tick but come from a small, known set (the symbols the feed carries), bind each combination once at startup and pick the handle in the loop. The [labeled instruments reference](../reference/telemetry/metrics/labeled-instruments.md) describes `Counter#bind`, `Histogram#bind`, `Gauge#bind` and `LabeledCounter#bind`.

## Check the Numbers With JMH

`Allocation.bytesPerRun` is enough to compare two versions of a loop, but a JMH benchmark with the GC profiler gives steadier numbers: it warms each benchmark in a fresh JVM and reports `gc.alloc.rate.norm`, the bytes allocated per call. JMH is not on the classpath of the docs or the example builds, so this benchmark runs with scala-cli. JMH rejects classes in the default package, hence the `package` line. Save it as `HotPathBench.scala`:

```scala
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-streams:0.0.56
//> using dep dev.zio::zio-blocks-next-maybe:0.0.56
//> using dep dev.zio::zio-blocks-next-telemetry:0.0.56
package hotpath

final case class Tick(symbol: String, priceMicros: Long, size: Int)

final class OptionState { var lastTick: Option[Tick] = None }
final class MaybeState  { var lastTick: Maybe[Tick]  = Maybe.absent }

@State(Scope.Thread)
class HotPathBench {
  val n                   = 10000
  val prices: Array[Long] = Array.tabulate(n)(i => 187000000L + (i % 997) * 1300L)
  val sizes: Array[Int]   = Array.tabulate(n)(i => 100 + i % 50)
  val ticks: Array[Tick]  = Array.tabulate(n)(i => Tick("ACME", prices(i), sizes(i)))
  val chunk: Chunk[Long]  = Chunk.fromArray(prices)

  val optionState = new OptionState
  val maybeState  = new MaybeState

  val ticksCounter = MeterProvider.builder.build().get("feed").counterBuilder("ticks").build()
  val acmeOnXnas   = ticksCounter.bind(Attributes.builder.put("symbol", "ACME").put("venue", "XNAS").build)

  // Streams
  @Benchmark def iteratorPipeline(): Long =
    prices.iterator.map(_ / 10000L).filter(_ > 18750L).foldLeft(0L)(_ + _)

  @Benchmark def primitiveStream(): Long =
    Stream.fromArray(prices).map(_ / 10000L).filter(_ > 18750L).runFold(0L)(_ + _).getOrElse(0L)

  @Benchmark def zippedStream(): Long =
    (Stream.fromArray(prices) && Stream.fromArray(sizes)).map { case (p, s) => p * s }.runFold(0L)(_ + _).getOrElse(0L)

  @Benchmark def indexedStream(): Long =
    Stream.range(0, n).map(i => prices(i) * sizes(i)).runFold(0L)(_ + _).getOrElse(0L)

  // Chunk
  @Benchmark def chunkCopy(): Chunk[Long] = Chunk.from(prices)
  @Benchmark def chunkWrap(): Chunk[Long] = Chunk.fromArray(prices)

  @Benchmark def chunkFoldLeft(): Long = chunk.foldLeft(0L)(_ + _)

  @Benchmark def chunkIndexLoop(): Long = {
    var sum = 0L
    var i   = 0
    while (i < chunk.length) { sum += chunk.long(i); i += 1 }
    sum
  }

  @Benchmark def genericBuilder(): Chunk[Long] = {
    val builder = Chunk.newBuilder[Long]
    var i       = 0
    while (i < n) { builder.addOne(prices(i)); i += 1 }
    builder.result()
  }

  @Benchmark def longBuilder(): Chunk[Long] = {
    val builder = new ChunkBuilder.Long
    builder.sizeHint(n)
    var i       = 0
    while (i < n) { builder.addOne(prices(i)); i += 1 }
    builder.result()
  }

  // Maybe
  @Benchmark def optionField(): OptionState = {
    var i = 0
    while (i < n) { optionState.lastTick = Some(ticks(i)); i += 1 }
    optionState
  }

  @Benchmark def maybeField(): MaybeState = {
    var i = 0
    while (i < n) { maybeState.lastTick = Maybe(ticks(i)); i += 1 }
    maybeState
  }

  // Telemetry
  @Benchmark def labelsPerCall(): Unit = {
    var i = 0
    while (i < n) { ticksCounter.add(1L, "symbol" -> "ACME", "venue" -> "XNAS"); i += 1 }
  }

  @Benchmark def boundCounter(): Unit = {
    var i = 0
    while (i < n) { acmeOnXnas.add(1L); i += 1 }
  }
}
```

Run it with the GC profiler (`-prof gc`); the short warm-up and measurement settings keep the run to a few minutes:

```bash
scala-cli --power run --jmh HotPathBench.scala -- -wi 3 -i 3 -f 1 -w 2s -r 2s -prof gc
```

Each benchmark handles 10,000 ticks per call, so dividing `gc.alloc.rate.norm` by 10,000 gives bytes per tick. On this machine, with JDK 25, the results were roughly:

| Hot-path code                                        | Before (bytes per tick) | After (bytes per tick)     |
| ---------------------------------------------------- | ----------------------- | -------------------------- |
| `Iterator` pipeline → `Stream` on the `Long` lane    | 62                      | 0.07                       |
| `&&` zip → `Stream.range` index                      | 55                      | 0.01                       |
| `Chunk.from` copy → `Chunk.fromArray` wrap           | 8                       | 0.003 (32 bytes per batch) |
| `Chunk#foldLeft` → `Chunk#long` index loop           | 24                      | 0                          |
| `Chunk.newBuilder[Long]` → sized `ChunkBuilder.Long` | 58                      | 16                         |
| `Option` field → `Maybe` field                       | 16                      | 0                          |
| Label pairs per call → `BoundCounter`                | 232                     | 0                          |

Throughput moves with allocation but varies more between runs, so compare `gc.alloc.rate.norm` first. A benchmark is the only way to know: the JIT removes some allocations on its own (the plain `chunk(i)` loop above) and keeps others that look free, so measure before and after every change. [How to Tune JSON Encoding and Decoding for Throughput](./tune-json-throughput.md) shows how to run the repository's own benchmarks.

## Running the Examples

The runnable code from this guide lives in three example builds, one per set of modules:

- [`streams-examples/src/main/scala/avoidallocation/`](https://github.com/zio/zio-blocks/tree/main/streams-examples/src/main/scala/avoidallocation): the baseline handler, streams and `Chunk`
- [`schema-examples/src/main/scala/avoidallocation/`](https://github.com/zio/zio-blocks/tree/main/schema-examples/src/main/scala/avoidallocation): `Maybe`
- [`telemetry-examples/src/main/scala/avoidallocation/`](https://github.com/zio/zio-blocks/tree/main/telemetry-examples/src/main/scala/avoidallocation): bound counters

Run any file from its folder with `scala-cli run <File>.scala`, or from the build's directory with `sbt "runMain avoidallocation.<ObjectName>"`. Each example prints the bytes it allocates per tick on your machine.

## Going Further

- [Zero-Boxing Optimization](../reference/streams/execution-and-compatibility/zero-boxing.md) explains how lanes work inside the stream interpreter.
- The [Reader reference](../reference/streams/primitives/reader.md) lists every primitive read and its end-of-stream signal.
- The [Chunk reference](../reference/chunk.md) covers the other `Chunk` constructors and builders.
- [Instrument a Program with Logs, Traces and Metrics](./instrument-a-program-with-logs-traces-and-metrics.md) shows where metrics fit in a whole program.

---

## Build a Live Counter and Dashboard with Datastar

Welcome! This tutorial is for Scala developers who can already build an HTML page with the ZIO Blocks HTML module and now want that page to react to clicks and to server updates without writing any JavaScript. You don't need prior experience with Datastar. If you haven't built a page with the HTML module yet, start with [Render Your First Server-Side HTML Page](./your-first-html-page.md); this tutorial picks up where it ends.

## 1. Introduction

By the end of this tutorial, you will understand:

- What a Datastar signal is, and how `dataSignals` and `dataText` put one on a page
- How a `dataOn.click` attribute makes the browser send the signals to the server, and how the server answers with a `DatastarEvent.patchSignals` event
- How to read the signals the browser sends into a case class with the schema JSON codec
- How a `DatastarEvent.patchElements` event swaps a piece of HTML into the page
- How one long-lived event stream turns a table into a live dashboard, and what the server sees when the browser goes away

We learn these concepts by building a small weather station, one step at a time:

1. A page with a signal
2. A button that posts
3. Reading the signals
4. Swapping a fragment
5. A live dashboard

At the end, we run the finished page and try it in a browser. We recommend reading from top to bottom — each section builds on the previous one.

The datastar module is for Scala 3 only: there is no Scala 2.13 artifact. Add it to your `build.sbt`:

```scala
// Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-datastar" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-datastar" % "0.0.56"
```

The datastar module depends on the HTML module and the http-model module. The http-model module is compiled for Java 21, so the programs in this tutorial need JDK 21 or newer to run. The module's package is `zio.http.datastar`, even though the artifact is called `zio-blocks-datastar`.

The module gives you the `data-*` attributes and the events. It does not ship a server and it does not ship the Datastar JavaScript, so this tutorial serves its pages with the streaming adapter from [How to Send Server-Sent Events with http-model](./send-server-sent-events.md), a small piece of code that connects the JDK's built-in HTTP server to a `Request => Response` function. We reuse it and don't explain it again.

## 2. Background: Signals in the Browser, Patches from the Server

Datastar is a small JavaScript library that reads `data-*` attributes on ordinary HTML elements. The page keeps named values called *signals*. An attribute such as `data-text="$readings"` shows a signal, and an attribute such as `data-on:click="@post('/reading')"` says what to do on a click: send a `POST` request to the server.

Every request carries the signals. A `POST` sends them as a JSON body, and a `GET` sends the same JSON in a query parameter called `datastar`. The server answers with a `text/event-stream` response, a stream of Server-Sent Events (SSE), where each event is a *patch*: a `datastar-patch-signals` event sets new signal values, and a `datastar-patch-elements` event swaps new HTML into the page. The exchange for one click looks like this:

```text
  Browser                                          Server
  ───────                                          ──────
  <button data-on:click="@post('/reading')">
        │  the reader clicks
        │
        ├──── POST /reading  {"readings":0} ─────►  decode the signals
        │                                            │
        │◄──── text/event-stream ───────────────────┤
        │      event: datastar-patch-signals         │
        │      data: signals {"readings":1}          │
        │      event: datastar-patch-elements        │
        │      data: elements …   │
        ▼
  signals update, the page changes
```

All of the HTML and all of the logic stay on the server, in Scala. The `datastar` module is the typed Scala side of this protocol: it renders the attributes and builds the events. This tutorial targets Datastar 1.0.4, the version the module's event names and attribute syntax match.

## 3. A Page with a Signal

Let's start with the page itself. A signal is a named, typed handle: `Signal[Int]("readings")` names a signal `readings` that holds an `Int`. `dataSignals` declares it with its first value, and `dataText` shows it. Here is the weather station's first page, which counts the readings taken:

```scala

val readings = Signal[Int]("readings")

val counter = div(
  dataSignals(readings := 0),
  p("Readings taken: ", span(dataText := readings))
)
```

The code above:

- `Signal[Int]("readings")` creates the typed handle; the compiler checks the name, which must be a dot-separated path of JavaScript identifiers and must not contain `__`
- `readings := 0` pairs the signal with its first value, encoded as JSON with the type's schema
- `dataSignals(...)` renders the `data-signals` attribute, which declares the signal when the page loads
- `dataText := readings` renders `data-text="$readings"`, which keeps the span's text equal to the signal

Let's look at the HTML this renders:

```scala
println(counter.renderMinified)
// Readings taken: 
```

Notice that the attribute value for `data-signals` is a JSON object, with its quotes escaped as `&quot;` because it sits inside an HTML attribute, and that `dataText := readings` became `$readings`: the `$` prefix is how a Datastar expression refers to a signal.

Datastar has to be loaded by the page, and here the library stops: it has no helper for the script tag. We write the `<script type="module">` tag by hand. This is glue that the module doesn't provide yet. We pin an exact release from the jsDelivr CDN, so that the page keeps working the way this tutorial describes:

```scala
val datastarVersion = "1.0.4"

val datastarScript: Dom.Element =
  script(
    `type` := "module",
    src    := s"https://cdn.jsdelivr.net/gh/starfederation/datastar@v$datastarVersion/bundles/datastar.js"
  )

def page(heading: String)(content: Dom.Element*): Dom.Element =
  html(lang := "en")(
    head(meta(charset := "utf-8"), title(heading), datastarScript),
    body(h1(heading), content)
  )
```

The code above:

- `datastarScript` is a `script` element with `type="module"` that loads the pinned Datastar bundle from the CDN
- `page` wraps any content in an `html` document whose `head` holds a `meta` charset, the `title` and the script
- `body(h1(heading), content)` puts the heading above the content we pass in

Serve `page("Weather Station")(counter)` as `text/html` and the browser shows `Readings taken: 0`. Nothing can change the signal yet; that is the next step.

## 4. A Button That Posts

An event attribute turns a click into a request. `dataOn.click := js"@post('/reading')"` renders `data-on:click="@post('/reading')"`, where `@post` is a Datastar action that sends the signals to the server. The `js"..."` interpolator marks the text as a Datastar expression; the module rejects a plain `String` in that position, so a typo cannot turn into a literal. To add the button to the counter:

```scala
val withButton = div(
  dataSignals(readings := 0),
  p("Readings taken: ", span(dataText := readings)),
  button(dataOn.click := js"@post('/reading')", "Take a reading")
)
```

The code above:

- `dataSignals(readings := 0)` declares the signal with its first value
- `dataText := readings` shows the signal in the `span`
- `dataOn.click := js"@post('/reading')"` posts the signals to `/reading` on each click

To see the attribute the button renders:

```scala
println(withButton.renderMinified)
// Readings taken: <button data-on:click="@post(&#x27;/reading&#x27;)">Take a reading</button>
```

When the reader clicks, Datastar sends `POST /reading` with the JSON body `{"readings":0}`, the `Content-Type: application/json` header and a `Datastar-Request: true` header. The server answers with a patch. `DatastarEvent.patchSignals` takes the same `signal := value` pairs as `dataSignals`, and `DatastarEvent#renderSSE` renders the event to its wire text:

```scala
println(DatastarEvent.patchSignals(readings := 1).renderSSE)
// event: datastar-patch-signals
// data: signals {"readings":1}
// 
//
```

The event has a name, `datastar-patch-signals`, and one `data:` line with the key `signals` followed by the JSON. The blank line after it ends the event. The browser merges the JSON into its signals, and `data-text` updates the span.

An event is only text; the response that carries it must have the content type `text/event-stream`. The media type registry has no constant for it, so we build the `ContentType` ourselves, and a small helper turns a stream of rendered events into a `Response`. This is the second piece of glue:

```scala

val eventStream: ContentType = ContentType(MediaType("text", "event-stream"))

def eventResponse(events: Stream[Nothing, String]): Response = {
  val bytes = events.flatMap(event => Stream.fromChunk(Chunk.fromArray(event.getBytes(StandardCharsets.UTF_8))))
  Response.ok(Body.fromStream(bytes, eventStream)).addHeader(Header.CacheControl.NoCache)
}

def eventResponse(first: String, rest: String*): Response =
  eventResponse(Stream.fromIterable(first +: rest))
```

The code above:

- `eventResponse(events)` renders nothing itself: each string is the `DatastarEvent#renderSSE` text of one event, and the helper encodes it as UTF-8, the only encoding SSE allows
- `Body.fromStream` makes a streaming body, and `Cache-Control: no-cache` keeps proxies from caching it
- The second overload sends a few events and ends, which is all a button click needs

The first server counts the clicks in memory, which is enough to see the loop work. `Step2CounterExample` serves the page and answers `POST /reading` with `eventResponse(DatastarEvent.patchSignals(readings := taken.incrementAndGet()).renderSSE)`. Its test client sends the click twice, and the output is:

```text
POST /reading -> 200 [text/event-stream]
event: datastar-patch-signals
data: signals {"readings":1}

POST /reading -> 200 [text/event-stream]
event: datastar-patch-signals
data: signals {"readings":2}
```

The server's counter works, but it is a second copy of a value the browser already has. In the next step the server reads the signals instead.

## 5. Reading the Signals

The browser sends all of its signals with every request, so the server can be stateless: it reads the current value, computes the next one and sends it back. This is where the library has a gap: it has no `readSignals`. We decode the JSON ourselves with the schema JSON codec into a case class whose fields are the signal names:

```scala

final case class Signals(readings: Int, city: String)

object Signals {
  implicit val schema: Schema[Signals] = Schema.derived
}
```

The code above gives `Signals` one field per signal the server reads, with the same names as the page's signals, and `Schema.derived` builds the schema from the case class. `schema.jsonCodec` is the JSON codec for it. A `POST` carries the JSON in the body, so decoding is one call:

```scala
println(Signals.schema.jsonCodec.decode("""{"readings":4,"city":"Oslo"}"""))
// Right(Signals(4,Oslo))
```

The browser sends every signal on the page, not only the ones the server cares about, and the codec ignores fields the case class doesn't have:

```scala
println(Signals.schema.jsonCodec.decode("""{"readings":4,"city":"Oslo","extra":true}"""))
// Right(Signals(4,Oslo))
```

A wrong type is an error, and the codec reports where:

```scala
println(Signals.schema.jsonCodec.decode("""{"readings":"four"}"""))
// Left(zio.blocks.schema.SchemaError: illegal number at: .readings)
```

A `GET` request carries the same JSON in the `datastar` query parameter, so one helper reads both. This is the third piece of glue:

```scala
def readSignals[A](request: Request)(implicit schema: Schema[A]): Either[String, A] = {
  val json =
    if (request.method == Method.GET) request.url.queryParams.getFirst("datastar").getOrElse("")
    else request.body.asString()
  schema.jsonCodec.decode(json).left.map(_.toString)
}
```

The code above:

- A `GET` takes the JSON from the `datastar` query parameter and every other method reads the request body
- `schema.jsonCodec.decode` returns the decoded value or an error
- `.left.map(_.toString)` turns the error into text, which the route turns into a `400 Bad Request`

Here is the helper reading a `GET` request whose query parameter holds the URL-encoded JSON:

```scala
val getRequest =
  Request.get(URL.parse("/dashboard?datastar=%7B%22readings%22%3A7%2C%22city%22%3A%22Lisbon%22%7D").toOption.get)
```

The code above builds a `GET` request for `/dashboard` whose `datastar` parameter holds the URL-encoded JSON `{"readings":7,"city":"Lisbon"}`. To call the helper on that request:

```scala
println(readSignals[Signals](getRequest))
// Right(Signals(7,Lisbon))
```

The route for the button is now stateless. It decodes `Signals`, adds one to `readings` and sends the patch back, and it answers `400` with the error text when the JSON doesn't decode. `Step3DecodeSignalsExample` serves it and adds a city picker: `dataBind(city)` on a `select` keeps the `city` signal equal to the chosen option. Its test client sends a good click, a click with an extra field, a click with a wrong type and a `GET`, and the output is:

```text
POST /reading -> 200 [text/event-stream]
event: datastar-patch-signals
data: signals {"readings":5}

POST /reading -> 200 [text/event-stream]
event: datastar-patch-signals
data: signals {"readings":5}

POST /reading -> 400 [text/plain; charset=UTF-8]
zio.blocks.schema.SchemaError: illegal number at: .readings
Right(Signals(7,Lisbon))
```

## 6. Swapping a Fragment

A signal patch changes values, and `data-text` turns them into text. To change the structure of the page, the server sends HTML. `DatastarEvent.patchElements` takes an element and, by default, replaces the element on the page that has the same `id`:

```scala
final case class Reading(city: String, tenthsCelsius: Int, windKmh: Int, humidity: Int) {
  def celsius: String = String.format(java.util.Locale.ROOT, "%.1f", Double.box(tenthsCelsius / 10.0))
}

def latest(reading: Reading): Dom.Element =
  p(
    id := "latest",
    s"${reading.city}: ${reading.celsius} °C, wind ${reading.windKmh} km/h, humidity ${reading.humidity}%"
  )
```

The code above:

- `Reading` holds one city's numbers, with the temperature in tenths of a degree so that it stays an `Int`
- `Reading#celsius` formats the tenths as text such as `3.6`, always with a dot
- `latest` builds a paragraph with `id := "latest"`, the id that the next patch replaces

The paragraph keeps its `id`, so the next patch finds it again. Here is the event for a reading in Oslo:

```scala
println(DatastarEvent.patchElements(latest(Reading("Oslo", 36, 12, 73))).renderSSE)
// event: datastar-patch-elements
// data: elements Oslo: 3.6 °C, wind 12 km/h, humidity 73%
// 
//
```

The data line has the key `elements` followed by the minified HTML. When the target isn't the element's own id, `PatchElementsBuilder#selector` names it, and `PatchElementsBuilder#mode` says where the content lands, for example `ElementPatchMode.Append` to add to a list. Each of them adds its own `data:` line before the `elements` line, and the default mode, `ElementPatchMode.Outer`, adds none. Here the paragraph has no id, so we name the target with `PatchElementsBuilder#selector` and ask for `ElementPatchMode.Inner`, which replaces the target's content:

```scala
println(
  DatastarEvent
    .patchElements(p("Oslo: 3.6 °C"))
    .selector(CssSelector.id("latest"))
    .mode(ElementPatchMode.Inner)
    .renderSSE
)
// event: datastar-patch-elements
// data: selector #latest
// data: mode inner
// data: elements Oslo: 3.6 °C
// 
//
```

One response can carry several events. The click now answers with a counter patch and a fragment patch, one after the other, and `eventResponse(first, rest*)` sends them both:

```scala
val answer = eventResponse(
  DatastarEvent.patchSignals(readings := 5).renderSSE,
  DatastarEvent.patchElements(latest(Reading("Oslo", 36, 12, 73))).renderSSE
)
```

The code above:

- Each argument is the `DatastarEvent#renderSSE` text of one event
- `eventResponse(first, rest*)` sends the two events in order and ends the response

To read the response body of the two events:

```scala
print(answer.body.asString())
// event: datastar-patch-signals
// data: signals {"readings":5}
// 
// event: datastar-patch-elements
// data: elements Oslo: 3.6 °C, wind 12 km/h, humidity 73%
//
```

`Step4PatchElementsExample` serves this route, and the weather model behind it gives every city a reading for each tick (see `Weather.scala` in the examples). Its client posts `{"readings":4,"city":"Oslo"}` and prints the same two events.

:::caution
`DatastarEvent.patchElements` renders the HTML on one line, but a line break inside the HTML, such as the text of a `<pre>` element, is split across several `data:` lines, and only the first of them carries the `elements` key. In a real browser the rest of the text is dropped. This is a known bug in 0.0.56, so keep line breaks out of the elements you patch.
:::

## 7. A Live Dashboard

The click-driven patches come from requests the reader makes. A dashboard needs the opposite: the server pushes new numbers without being asked. We build it with one open stream: the page sends one request when it loads, and the server keeps that response open, writing a new event whenever it has one. A stream is one request for the whole session, and the server decides when to send, so the page never repeats a request and never resends its signals just to ask for news. The price of the stream is that the server holds a connection, and in our JDK adapter a thread, for as long as the page is open.

The page starts the stream with `dataInit`, which runs its expression once when Datastar first sees the element, and it contains the first version of the dashboard:

```scala
def dashboard(rows: Seq[Reading]): Dom.Element = {
  val header = tr(th("City"), th("Temperature"), th("Wind"), th("Humidity"))
  val body   = rows.map(r => tr(td(r.city), td(s"${r.celsius} °C"), td(s"${r.windKmh} km/h"), td(s"${r.humidity}%")))
  table(id := "dashboard", (header +: body)*)
}

val live = div(dataInit := js"@get('/dashboard/stream')", dashboard(Seq(Reading("Oslo", 40, 14, 71))))
```

The code above:

- `dashboard` builds a `table` with `id := "dashboard"`, one header row and one row per reading
- `dataInit := js"@get('/dashboard/stream')"` runs the request once when Datastar first sees the `div`
- `live` wraps the first version of the table in that `div`

To render the element and see the attribute and the first table:

```scala
println(live.renderMinified)
// CityTemperatureWindHumidityOslo4.0 °C14 km/h71%
```

The table has an `id`, so each `DatastarEvent.patchElements` event replaces it whole. The `dashboard` function takes the rows to show; `Weather.readings(tick)` from the examples supplies a row per city for every tick.

The stream itself is a `Stream` of rendered events, one per second. The approach is the one from [How to Send Server-Sent Events with http-model](./send-server-sent-events.md): `Stream#tapEach` runs a side effect for each element before the element moves on, so a sleep inside it makes the server write an event only when the pause is over. Here it is with no pause, cut to two events so that we can read it:

```scala
def ticks(intervalMillis: Long): Stream[Nothing, String] =
  Stream
    .fromIterable(LazyList.from(1))
    .tapEach { _ =>
      Thread.sleep(intervalMillis)
      ()
    }
    .map(tick => DatastarEvent.patchElements(dashboard(Seq(Reading("Oslo", 40 + tick, 14, 71)))).renderSSE)

val firstTwoTicks = eventResponse(ticks(0).take(2))
```

The code above:

- `Stream#tapEach` runs its function for each element, and the `Thread.sleep(intervalMillis)` inside that function makes each element wait
- `Stream#map` turns each tick into a `DatastarEvent.patchElements` event for the table
- `Stream#take` with `2` ends the otherwise endless stream so that we can print it

To read the first two events from the response body:

```scala
print(firstTwoTicks.body.asString())
// event: datastar-patch-elements
// data: elements CityTemperatureWindHumidityOslo4.1 °C14 km/h71%
// 
// event: datastar-patch-elements
// data: elements CityTemperatureWindHumidityOslo4.2 °C14 km/h71%
//
```

The stream has no end: `LazyList.from(1)` counts up for as long as somebody reads it, and the response stays open until the client leaves.

When the browser goes away, a closed tab doesn't tell the server anything. The server finds out the next time it writes: the write fails with an `IOException`, `Stream#run` stops pulling the stream and closes it, and the streaming adapter closes the exchange. A finalizer registered with `Stream#ensuring` runs at that moment, which is the place to release anything the dashboard holds. `Step5LiveDashboardExample` registers one that prints how many events were sent, and its client reads two events and then disconnects:

```text
event: datastar-patch-elements
data: elements CityTemperatureWindHumidityOslo4.6 °C17 km/h68%Lisbon19.1 °C16 km/h66%Nairobi23.2 °C12 km/h54%

event: datastar-patch-elements
data: elements CityTemperatureWindHumidityOslo3.8 °C13 km/h72%Lisbon20.5 °C23 km/h59%Nairobi22.4 °C8 km/h58%

client: disconnected
server: stream closed after 4 events
```

The server writes two more events than the client reads before it notices the disconnect: those are the events in flight when the client closes the connection.

## 8. Putting It Together

This example combines everything we've learned into one file. It serves one page with the city picker, the counter button, the "latest reading" paragraph and the live dashboard, and it keeps running until we stop it:

```scala title="datastar-examples/src/main/scala/buildalivedashboardwithdatastar/CompleteExample.scala"
package buildalivedashboardwithdatastar

/**
 * Build a Live Counter and Dashboard with Datastar — the complete example
 *
 * One page with a "Take a reading" button and a live dashboard. The button
 * `POST`s the signals; the server decodes them and answers with a
 * `patchSignals` and a `patchElements` event. The dashboard opens one
 * `GET /dashboard/stream` when the page loads and receives a new table every
 * second.
 *
 * Run with: cd datastar-examples && sbt "runMain
 * buildalivedashboardwithdatastar.CompleteExample", then open
 * http://localhost:8080/
 */
object CompleteExample {
  val readings = Signal[Int]("readings")
  val city     = Signal[String]("city")

  final case class Signals(readings: Int, city: String)
  object Signals {
    implicit val schema: Schema[Signals] = Schema.derived
  }

  def latest(reading: Reading): Dom.Element =
    p(
      id := "latest",
      s"${reading.city}: ${reading.celsius} °C, wind ${reading.windKmh} km/h, humidity ${reading.humidity}%"
    )

  def dashboard(tick: Int): Dom.Element = {
    val header = tr(th("City"), th("Temperature"), th("Wind"), th("Humidity"))
    val rows   = Weather.readings(tick).map { reading =>
      tr(td(reading.city), td(s"${reading.celsius} °C"), td(s"${reading.windKmh} km/h"), td(s"${reading.humidity}%"))
    }
    table(id := "dashboard", (header +: rows)*)
  }

  def page: Dom.Element =
    Glue.page("Weather Station")(
      div(
        dataSignals(readings := 0, city := "Oslo"),
        select(dataBind(city), option("Oslo"), option("Lisbon"), option("Nairobi")),
        p("Readings taken: ", span(dataText := readings)),
        button(dataOn.click := js"@post('/reading')", "Take a reading"),
        p(id                := "latest", "No reading yet.")
      ),
      div(dataInit := js"@get('/dashboard/stream')", dashboard(0))
    )

  def ticks(intervalMillis: Long): Stream[Nothing, String] =
    Stream
      .fromIterable(LazyList.from(1))
      .tapEach { _ =>
        Thread.sleep(intervalMillis)
        ()
      }
      .map(tick => DatastarEvent.patchElements(dashboard(tick)).renderSSE)

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil)              => htmlResponse(doctype.render + page.render)
      case (Method.POST, List("reading")) =>
        readSignals[Signals](request) match {
          case Right(signals) =>
            val taken = signals.readings + 1
            eventResponse(
              DatastarEvent.patchSignals(readings := taken).renderSSE,
              DatastarEvent.patchElements(latest(Weather.reading(signals.city, taken))).renderSSE
            )
          case Left(error) => Response.text(Status.BadRequest, error)
        }
      case (Method.GET, List("dashboard", "stream")) => eventResponse(ticks(1000))
      case _                                         => notFound(request)
    }

  def main(args: Array[String]): Unit =
    openInBrowser(serve(app, args))
}
```

The pieces are the ones from the previous sections: the signals and the button from sections 3 and 4, the `Signals` decoding from section 5, the `latest` fragment from section 6, and the `dashboard` table and the `ticks` stream from section 7. It differs from the sections in two ways: the page holds the counter and the dashboard together, and `main` always starts the server, on port 8080 or the port given as the first argument.

Run it and open `http://localhost:8080/` in a browser. The dashboard fills in right away and its numbers change every second. Pick "Lisbon" and press "Take a reading": the counter goes up and the paragraph shows Lisbon's reading, while the dashboard keeps updating and the picker still says "Lisbon". Open the browser's developer tools on the Network tab: you see one `POST /reading` per click, with the body `{"readings":0,"city":"Oslo"}` and the header `Datastar-Request: true`, and one `GET /dashboard/stream?datastar=...` that never finishes.

## 9. Running the Examples

All examples in this tutorial have corresponding runnable Scala files in the `datastar-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/datastar-examples
```

Every step serves its page on the port you pass as the first argument, for example `sbt "runMain buildalivedashboardwithdatastar.Step1StaticPageExample 8080"`, and keeps running until you press Ctrl+C, so you can open it in a browser. Without an argument a step starts a server on a free port, sends the requests from its section with a small test client and prints what the server answers. The complete example always serves, on port 8080 unless you pass another. The steps share the weather model in `Weather.scala`, the glue in `Glue.scala` and the test client in `Client.scala`. The server adapters in `JdkHttp.scala` and `StreamingJdkHttp.scala` are copied from [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md) and [How to Send Server-Sent Events with http-model](./send-server-sent-events.md).

### Share the Weather Model and the Glue

Every step imports these files. They have no `main`, so there is nothing to run; read them first.

<details>
  <summary>datastar-examples/src/main/scala/buildalivedashboardwithdatastar/Weather.scala</summary>

```scala title="datastar-examples/src/main/scala/buildalivedashboardwithdatastar/Weather.scala" showLineNumbers
package buildalivedashboardwithdatastar

/**
 * Build a Live Counter and Dashboard with Datastar — the weather station model
 *
 * Three cities and a function that gives each city a reading for every tick.
 * The readings drift up and down in a fixed pattern, so every run of the
 * tutorial prints the same numbers. Every step reuses these.
 */
final case class Reading(city: String, tenthsCelsius: Int, windKmh: Int, humidity: Int) {

  /** The temperature as text with one decimal, such as `4.0`. */
  def celsius: String = String.format(Locale.ROOT, "%.1f", Double.box(tenthsCelsius / 10.0))
}

object Weather {
  val cities: Vector[String] = Vector("Oslo", "Lisbon", "Nairobi")

  private val typical: Map[String, Reading] = Map(
    "Oslo"    -> Reading("Oslo", 40, 14, 71),
    "Lisbon"  -> Reading("Lisbon", 195, 18, 64),
    "Nairobi" -> Reading("Nairobi", 230, 11, 55)
  )

  /**
   * The reading of `city` at the given tick: its typical weather plus a drift.
   */
  def reading(city: String, tick: Int): Reading = {
    val base  = typical(city)
    val drift = ((tick * 7 + city.length * 3) % 11) - 5
    Reading(
      city,
      base.tenthsCelsius + drift * 2,
      math.max(0, base.windKmh + drift),
      math.min(100, math.max(0, base.humidity - drift))
    )
  }

  /** One reading per city at the given tick. */
  def readings(tick: Int): Vector[Reading] = cities.map(reading(_, tick))
}
```

</details>

<details>
  <summary>datastar-examples/src/main/scala/buildalivedashboardwithdatastar/Glue.scala</summary>

```scala title="datastar-examples/src/main/scala/buildalivedashboardwithdatastar/Glue.scala" showLineNumbers
package buildalivedashboardwithdatastar

/**
 * Build a Live Counter and Dashboard with Datastar — the glue
 *
 * Three things the datastar module does not provide yet: the `<script
 * type="module">` tag that loads Datastar, a way to read the signals the
 * browser sends, and a `Response` that streams Datastar events. Every step
 * reuses them.
 */
object Glue {

  /** The Datastar release the tutorial was checked against. */
  val datastarVersion: String = "1.0.4"

  /** Loads Datastar from the jsDelivr CDN as a JavaScript module. */
  val datastarScript: Dom.Element =
    script(
      `type` := "module",
      src    := s"https://cdn.jsdelivr.net/gh/starfederation/datastar@v$datastarVersion/bundles/datastar.js"
    )

  /** A page that loads Datastar, with a heading and the given content. */
  def page(heading: String)(content: Dom.Element*): Dom.Element =
    html(lang := "en")(
      head(meta(charset := "utf-8"), title(heading), datastarScript),
      body(h1(heading), content)
    )

  /** A 200 OK response with `Content-Type: text/html`. */
  def htmlResponse(markup: String): Response =
    Response.ok(Body.fromArray(markup.getBytes(StandardCharsets.UTF_8), ContentType.`text/html`))

  /** A 404 for any route the step does not handle. */
  def notFound(request: Request): Response =
    Response.text(Status.NotFound, s"No route for ${request.method.name} ${request.path.encode}")

  /** `text/event-stream`; the media type registry has no constant for it. */
  val eventStream: ContentType = ContentType(MediaType("text", "event-stream"))

  /**
   * A 200 OK whose body streams the already rendered events as they are
   * produced. Each string is the `renderSSE` text of one Datastar event.
   */
  def eventResponse(events: Stream[Nothing, String]): Response = {
    val bytes = events.flatMap(event => Stream.fromChunk(Chunk.fromArray(event.getBytes(StandardCharsets.UTF_8))))
    Response.ok(Body.fromStream(bytes, eventStream)).addHeader(Header.CacheControl.NoCache)
  }

  /** A response that sends the given rendered events and then ends. */
  def eventResponse(first: String, rest: String*): Response =
    eventResponse(Stream.fromIterable(first +: rest))

  /**
   * Reads the signals Datastar sent with a request: the JSON body of a `POST`,
   * or the `datastar` query parameter of a `GET`. Returns the decoding error as
   * text when the JSON does not match `A`.
   */
  def readSignals[A](request: Request)(implicit schema: Schema[A]): Either[String, A] = {
    val json =
      if (request.method == Method.GET) request.url.queryParams.getFirst("datastar").getOrElse("")
      else request.body.asString()
    schema.jsonCodec.decode(json).left.map(_.toString)
  }

  /**
   * Starts the server on the port given as the first argument (8080 by default)
   * and keeps it running until the program is stopped with Ctrl+C.
   */
  def serve(app: Request => Response, args: Array[String]): Int = {
    val port   = args.headOption.flatMap(_.toIntOption).getOrElse(8080)
    val server = StreamingJdkHttp.start(app, port)
    sys.addShutdownHook(server.stop(0))
    port
  }

  def openInBrowser(port: Int): Unit =
    println(s"Open http://localhost:$port/ in a browser. Press Ctrl+C to stop.")
}
```

</details>

### A Page with a Signal

This example serves the page from section 3 and prints the page the server returns.

<details>
  <summary>datastar-examples/src/main/scala/buildalivedashboardwithdatastar/Step1StaticPageExample.scala</summary>

```scala title="datastar-examples/src/main/scala/buildalivedashboardwithdatastar/Step1StaticPageExample.scala" showLineNumbers
package buildalivedashboardwithdatastar

/**
 * Build a Live Counter and Dashboard with Datastar — Step 1: A Page with
 * Signals
 *
 * Serves a page that loads Datastar, declares one signal and shows it with
 * `dataText`. Nothing on the page can change the signal yet.
 *
 * Run with: cd datastar-examples && sbt "runMain
 * buildalivedashboardwithdatastar.Step1StaticPageExample"
 */
object Step1StaticPageExample {
  val readings = Signal[Int]("readings")

  def page: Dom.Element =
    Glue.page("Weather Station")(
      div(
        dataSignals(readings := 0),
        p("Readings taken: ", span(dataText := readings))
      )
    )

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil) => htmlResponse(doctype.render + page.render)
      case _                 => notFound(request)
    }

  def main(args: Array[String]): Unit =
    if (args.nonEmpty) openInBrowser(serve(app, args))
    else StreamingJdkHttp.withServer(app)(port => Client.call(port, "GET", "/"))
}
```

</details>

Run it from the `datastar-examples` directory:

```bash
sbt "runMain buildalivedashboardwithdatastar.Step1StaticPageExample"
```

### A Button That Posts

This example adds the button and a server that counts the clicks.

<details>
  <summary>datastar-examples/src/main/scala/buildalivedashboardwithdatastar/Step2CounterExample.scala</summary>

```scala title="datastar-examples/src/main/scala/buildalivedashboardwithdatastar/Step2CounterExample.scala" showLineNumbers
package buildalivedashboardwithdatastar

/**
 * Build a Live Counter and Dashboard with Datastar — Step 2: A Button That
 * Posts
 *
 * The page gets a button. Clicking it makes Datastar `POST /reading`; the
 * server counts the click in memory and answers with a `patchSignals` event
 * that sets the signal.
 *
 * Run with: cd datastar-examples && sbt "runMain
 * buildalivedashboardwithdatastar.Step2CounterExample"
 */
object Step2CounterExample {
  val readings = Signal[Int]("readings")

  def page: Dom.Element =
    Glue.page("Weather Station")(
      div(
        dataSignals(readings := 0),
        p("Readings taken: ", span(dataText := readings)),
        button(dataOn.click := js"@post('/reading')", "Take a reading")
      )
    )

  def newApp(): Request => Response = {
    val taken = new AtomicInteger(0)
    request =>
      (request.method, request.path.segments.toList) match {
        case (Method.GET, Nil)              => htmlResponse(doctype.render + page.render)
        case (Method.POST, List("reading")) =>
          eventResponse(DatastarEvent.patchSignals(readings := taken.incrementAndGet()).renderSSE)
        case _ => notFound(request)
      }
  }

  def main(args: Array[String]): Unit =
    if (args.nonEmpty) openInBrowser(serve(newApp(), args))
    else
      StreamingJdkHttp.withServer(newApp()) { port =>
        Client.call(port, "POST", "/reading", """{"readings":0}""")
        Client.call(port, "POST", "/reading", """{"readings":0}""")
      }
}
```

</details>

Run it from the `datastar-examples` directory:

```bash
sbt "runMain buildalivedashboardwithdatastar.Step2CounterExample"
```

### Reading the Signals

This example decodes the signals into `Signals` and answers with the next value.

<details>
  <summary>datastar-examples/src/main/scala/buildalivedashboardwithdatastar/Step3DecodeSignalsExample.scala</summary>

```scala title="datastar-examples/src/main/scala/buildalivedashboardwithdatastar/Step3DecodeSignalsExample.scala" showLineNumbers
package buildalivedashboardwithdatastar

/**
 * Build a Live Counter and Dashboard with Datastar — Step 3: Reading the
 * Signals
 *
 * The server stops counting by itself. It decodes the signals the browser sends
 * into a case class with the schema JSON codec, and answers with the next
 * value. A `GET` carries the same JSON in the `datastar` query parameter.
 *
 * Run with: cd datastar-examples && sbt "runMain
 * buildalivedashboardwithdatastar.Step3DecodeSignalsExample"
 */
object Step3DecodeSignalsExample {
  val readings = Signal[Int]("readings")
  val city     = Signal[String]("city")

  final case class Signals(readings: Int, city: String)
  object Signals {
    implicit val schema: Schema[Signals] = Schema.derived
  }

  def page: Dom.Element =
    Glue.page("Weather Station")(
      div(
        dataSignals(readings := 0, city := "Oslo"),
        select(dataBind(city), option("Oslo"), option("Lisbon"), option("Nairobi")),
        p("Readings taken: ", span(dataText := readings)),
        button(dataOn.click := js"@post('/reading')", "Take a reading")
      )
    )

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil)              => htmlResponse(doctype.render + page.render)
      case (Method.POST, List("reading")) =>
        readSignals[Signals](request) match {
          case Right(signals) => eventResponse(DatastarEvent.patchSignals(readings := signals.readings + 1).renderSSE)
          case Left(error)    => Response.text(Status.BadRequest, error)
        }
      case _ => notFound(request)
    }

  def main(args: Array[String]): Unit =
    if (args.nonEmpty) openInBrowser(serve(app, args))
    else
      StreamingJdkHttp.withServer(app) { port =>
        Client.call(port, "POST", "/reading", """{"readings":4,"city":"Oslo"}""")
        Client.call(port, "POST", "/reading", """{"readings":4,"city":"Oslo","extra":true}""")
        Client.call(port, "POST", "/reading", """{"readings":"four"}""")
        println(
          readSignals[Signals](
            Request.get(
              URL
                .parse(
                  "/reading?datastar=" + URLEncoder.encode("""{"readings":7,"city":"Lisbon"}""", StandardCharsets.UTF_8)
                )
                .toOption
                .get
            )
          )
        )
      }
}
```

</details>

Run it from the `datastar-examples` directory:

```bash
sbt "runMain buildalivedashboardwithdatastar.Step3DecodeSignalsExample"
```

### Swapping a Fragment

This example answers a click with a `DatastarEvent.patchSignals` event and a `DatastarEvent.patchElements` event.

<details>
  <summary>datastar-examples/src/main/scala/buildalivedashboardwithdatastar/Step4PatchElementsExample.scala</summary>

```scala title="datastar-examples/src/main/scala/buildalivedashboardwithdatastar/Step4PatchElementsExample.scala" showLineNumbers
package buildalivedashboardwithdatastar

/**
 * Build a Live Counter and Dashboard with Datastar — Step 4: Swapping a
 * Fragment
 *
 * The answer to a click now has two events: a `patchSignals` that sets the
 * counter and a `patchElements` that replaces the `#latest` paragraph with the
 * reading for the selected city.
 *
 * Run with: cd datastar-examples && sbt "runMain
 * buildalivedashboardwithdatastar.Step4PatchElementsExample"
 */
object Step4PatchElementsExample {
  val readings = Signal[Int]("readings")
  val city     = Signal[String]("city")

  final case class Signals(readings: Int, city: String)
  object Signals {
    implicit val schema: Schema[Signals] = Schema.derived
  }

  /**
   * The paragraph Datastar swaps in; it keeps the id, so the next patch finds
   * it.
   */
  def latest(reading: Reading): Dom.Element =
    p(
      id := "latest",
      s"${reading.city}: ${reading.celsius} °C, wind ${reading.windKmh} km/h, humidity ${reading.humidity}%"
    )

  def page: Dom.Element =
    Glue.page("Weather Station")(
      div(
        dataSignals(readings := 0, city := "Oslo"),
        select(dataBind(city), option("Oslo"), option("Lisbon"), option("Nairobi")),
        p("Readings taken: ", span(dataText := readings)),
        button(dataOn.click := js"@post('/reading')", "Take a reading"),
        p(id                := "latest", "No reading yet.")
      )
    )

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil)              => htmlResponse(doctype.render + page.render)
      case (Method.POST, List("reading")) =>
        readSignals[Signals](request) match {
          case Right(signals) =>
            val taken = signals.readings + 1
            eventResponse(
              DatastarEvent.patchSignals(readings := taken).renderSSE,
              DatastarEvent.patchElements(latest(Weather.reading(signals.city, taken))).renderSSE
            )
          case Left(error) => Response.text(Status.BadRequest, error)
        }
      case _ => notFound(request)
    }

  def main(args: Array[String]): Unit =
    if (args.nonEmpty) openInBrowser(serve(app, args))
    else
      StreamingJdkHttp.withServer(app) { port =>
        Client.call(port, "POST", "/reading", """{"readings":4,"city":"Oslo"}""")
      }
}
```

</details>

Run it from the `datastar-examples` directory:

```bash
sbt "runMain buildalivedashboardwithdatastar.Step4PatchElementsExample"
```

### A Live Dashboard

This example streams the dashboard and shows the server noticing a disconnect.

<details>
  <summary>datastar-examples/src/main/scala/buildalivedashboardwithdatastar/Step5LiveDashboardExample.scala</summary>

```scala title="datastar-examples/src/main/scala/buildalivedashboardwithdatastar/Step5LiveDashboardExample.scala" showLineNumbers
package buildalivedashboardwithdatastar

/**
 * Build a Live Counter and Dashboard with Datastar — Step 5: A Live Dashboard
 *
 * The page opens one `GET /dashboard/stream` when it loads, and the server
 * keeps that response open, sending a new `patchElements` event with the whole
 * table every second. The program also shows what the server sees when the
 * client goes away.
 *
 * Run with: cd datastar-examples && sbt "runMain
 * buildalivedashboardwithdatastar.Step5LiveDashboardExample"
 */
object Step5LiveDashboardExample {

  def dashboard(tick: Int): Dom.Element = {
    val header = tr(th("City"), th("Temperature"), th("Wind"), th("Humidity"))
    val rows   = Weather.readings(tick).map { reading =>
      tr(td(reading.city), td(s"${reading.celsius} °C"), td(s"${reading.windKmh} km/h"), td(s"${reading.humidity}%"))
    }
    table(id := "dashboard", (header +: rows)*)
  }

  def page: Dom.Element =
    Glue.page("Weather Station")(
      div(dataInit := js"@get('/dashboard/stream')", dashboard(0))
    )

  /**
   * One `patchElements` event per tick, `intervalMillis` apart, until the
   * client leaves.
   */
  def ticks(intervalMillis: Long, sent: AtomicInteger): Stream[Nothing, String] =
    Stream
      .fromIterable(LazyList.from(1))
      .tapEach { _ =>
        Thread.sleep(intervalMillis)
        sent.incrementAndGet()
        ()
      }
      .map(tick => DatastarEvent.patchElements(dashboard(tick)).renderSSE)

  def newApp(intervalMillis: Long = 1000, sent: AtomicInteger = new AtomicInteger(0)): Request => Response =
    request =>
      (request.method, request.path.segments.toList) match {
        case (Method.GET, Nil)                         => htmlResponse(doctype.render + page.render)
        case (Method.GET, List("dashboard", "stream")) =>
          eventResponse(
            ticks(intervalMillis, sent).ensuring(println(s"server: stream closed after ${sent.get} events"))
          )
        case _ => notFound(request)
      }

  def main(args: Array[String]): Unit =
    if (args.nonEmpty) openInBrowser(serve(newApp(), args))
    else
      StreamingJdkHttp.withServer(newApp(intervalMillis = 300)) { port =>
        Client.watch(port, "/dashboard/stream", maxLines = 6)
        println("client: disconnected")
        Thread.sleep(1500)
      }
}
```

</details>

Run it from the `datastar-examples` directory:

```bash
sbt "runMain buildalivedashboardwithdatastar.Step5LiveDashboardExample"
```

### The Complete Example

The complete example from "Putting It Together" is a single file with everything in it:

```bash
sbt "runMain buildalivedashboardwithdatastar.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## 10. What You've Learned

In this tutorial, you learned:

- That Datastar keeps named signals in the browser, and that `dataSignals` and `dataText` declare and show a signal on a page built with the HTML DSL
- How `dataOn.click := js"@post('/reading')"` sends the signals to the server, and how `DatastarEvent.patchSignals` answers with new values over a `text/event-stream` response
- How to decode the signals in a `POST` body, or in the `datastar` query parameter of a `GET`, into a case class with the schema JSON codec
- How `DatastarEvent.patchElements` replaces the element with the same `id`, and how `PatchElementsBuilder#selector` and `PatchElementsBuilder#mode` change where the content lands
- How one open `GET` stream pushes a new table every second, and how `Stream#ensuring` tells you when the browser has gone away

You also saw the three pieces of glue the library doesn't provide yet: the `<script type="module">` tag, a `readSignals` helper, and a `Response` that streams Datastar events. You now have a solid foundation in building reactive pages with Datastar and ZIO Blocks.

## 11. Where to Go Next

- **Want every attribute and event?** Read the [Datastar](../reference/datastar/index.md) reference, then [Signals](../reference/datastar/signals.md), [Attributes](../reference/datastar/attributes.md), [Event Handlers](../reference/datastar/events.md) and [Server-Sent Events](../reference/datastar/sse.md).
- **Want to see the other approach to server-driven pages?** [Add Interactivity to a Page with HTMX](./add-interactivity-with-htmx.md) builds a live search with htmx, where the server sends HTML fragments only.
- **Want more on streaming events?** [How to Send Server-Sent Events with http-model](./send-server-sent-events.md) covers event ids, resuming with `Last-Event-ID` and disconnects.
- **Want to learn streams first?** [Your First Streaming Pipeline](./your-first-streaming-pipeline.md) covers creating and running a `Stream`.
- **Want more on serving?** [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md) explains the adapter the examples use and its limits.

---

## How to Build a Query from Optional Search Filters

In this guide, we build the query behind a support desk's ticket search screen. The screen has optional fields: a text box, a customer, a set of status checkboxes and a date range. The user fills in some of them and leaves the rest empty. By the end, you will have one function that turns whatever the user filled in into one SQL query whose `WHERE` clause holds only those conditions, joined with `AND`, followed by a sort order and a page of results, with every value the user typed bound as a parameter.

We build it one field at a time with [`Frag`](../reference/sql/frag.md), the `sql` module's piece of SQL text with its parameters kept separate:

- One condition per filled-in field, and nothing for an empty one
- A small helper that joins the conditions with `AND`
- A `LIKE` search on part of a text, a date range and a multi-select with `IN`
- A sort order chosen from a fixed list, and paging with `LIMIT` and `OFFSET`
- The finished search, with the SQL and parameters it sends
- The same search written with the `SqlQuery` builder, and when to choose each

This guide builds on [Build Your First Database Repository on SQLite](./your-first-sqlite-repository.md). We use `JdbcTransactor`, `Table`, `Repo` and the `sql"..."` interpolator the way that tutorial introduces them and don't explain them again.

## The Problem

A search form with five optional fields can be filled in 32 different ways, and each way needs a different `WHERE` clause. Writing 32 queries is out of the question, so the query has to be assembled at run time from the fields that have a value.

Assembling SQL at run time is where bugs and security holes come from. With plain JDBC, the usual approach keeps a `StringBuilder` for the SQL and a list for the parameters, and appends to both in step:

```scala

def searchTickets(
  conn: Connection,
  text: Option[String],
  customer: Option[String],
  statuses: Set[String],
  openedFrom: Option[LocalDate]
): List[String] = {
  // "1 = 1" lets every condition start with " AND "
  val sql    = new StringBuilder("SELECT subject FROM ticket WHERE 1 = 1")
  val params = ListBuffer.empty[Any]
  text.foreach { t => sql.append(" AND subject LIKE ?"); params += s"%$t%" }
  customer.foreach { c => sql.append(" AND customer = ?"); params += c }
  if (statuses.nonEmpty) {
    sql.append(statuses.map(_ => "?").mkString(" AND status IN (", ", ", ")"))
    params ++= statuses
  }
  openedFrom.foreach { d => sql.append(" AND opened_on >= ?"); params += Date.valueOf(d) }
  sql.append(" ORDER BY opened_on DESC")

  val statement = conn.prepareStatement(sql.toString)
  params.zipWithIndex.foreach { case (p, i) => statement.setObject(i + 1, p) }
  val rows     = statement.executeQuery()
  val subjects = ListBuffer.empty[String]
  while (rows.next()) subjects += rows.getString("subject")
  subjects.toList
}
```

Every `?` in the text must line up with exactly one entry in the `params` list, and nothing checks that they do: add a condition to one and forget the other, and the query binds the wrong values or fails. The `%` and `_` characters a user types are wildcards inside `LIKE`, so a search for `50%` matches more than the user asked for. The shortcut that skips the parameter list, `sql.append(s" AND customer = '$c'")`, lets a user who types a quote rewrite the query: that is SQL injection.

In this guide, each condition is a `Frag` that carries its own SQL text and its own parameters, so the two can't drift apart, and the user's values never become part of the SQL text.

## Prerequisites

Add the `sql` module and the SQLite JDBC driver to your `build.sbt`:

```scala
// Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-sql" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-sql" % "0.0.56"
scalacOptions += "-experimental"

// The SQLite JDBC driver, for every Scala version
libraryDependencies += "org.xerial" % "sqlite-jdbc" % "3.53.4.0"
```

The `sql` module is Scala 3 only, so the code on this page uses Scala 3 syntax.

## The Ticket Table and the Search Form

Our support desk stores one row per ticket: a subject line, the customer who opened it, a status (`open`, `pending` or `closed`), a priority from 1 (urgent) to 3 (low) and the date it was opened. We create the table in a SQLite file and add eight tickets:

```scala

case class Ticket(id: Long, subject: String, customer: String, status: String, priority: Int, openedOn: LocalDate)

object Ticket {
  given Schema[Ticket] = Schema.derived
}

val ticketTable = Table.derived[Ticket]
val tickets     = Repo.derived[Ticket, Long]

val dbFile     = java.nio.file.Files.createTempFile("helpdesk", ".db")
val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

transactor.connect {
  ticketTable.createTable(SqlDialect.SQLite).update
  List(
    Ticket(1L, "Cannot log in after password reset", "Acme", "open", 1, LocalDate.of(2026, 9, 28)),
    Ticket(2L, "Invoice shows the 50% discount twice", "Globex", "closed", 2, LocalDate.of(2026, 10, 2)),
    Ticket(3L, "Login page loads slowly", "Acme", "pending", 2, LocalDate.of(2026, 10, 6)),
    Ticket(4L, "Export to CSV drops the last row", "Initech", "open", 3, LocalDate.of(2026, 10, 9)),
    Ticket(5L, "Two-factor code never arrives", "Globex", "open", 1, LocalDate.of(2026, 10, 14)),
    Ticket(6L, "Typo on the pricing page", "Acme", "closed", 3, LocalDate.of(2026, 10, 17)),
    Ticket(7L, "API returns 500 on large uploads", "Initech", "pending", 1, LocalDate.of(2026, 10, 21)),
    Ticket(8L, "Cannot change the billing email", "Acme", "open", 2, LocalDate.of(2026, 10, 24))
  ).foreach(tickets.insert)
}
```

We import `SqlQuery` and `SortOrder` from `zio.blocks.sql.query` now, for the last section, and keep `import zio.blocks.sql.*` after every other `zio` import, as the tutorial explains.

The search form arrives as a case class. Every filter is optional: an `Option` is `None` when the user left the field empty, and an empty `Set` means no status box is ticked. The sort order is a choice from a fixed list, and the page number and page size have defaults:

```scala
enum SortBy {
  case Newest, Oldest, MostUrgent
}

case class TicketSearch(
  text: Option[String] = None,
  customer: Option[String] = None,
  statuses: Set[String] = Set.empty,
  openedFrom: Option[LocalDate] = None,
  openedTo: Option[LocalDate] = None,
  sortBy: SortBy = SortBy.Newest,
  page: Int = 1,
  pageSize: Int = 20
)
```

`TicketSearch()` is the empty form, which should list every ticket.

<details>
  <summary>Runnable version, which prints the empty form and the number of tickets: <code>Step3TicketTableExample.scala</code></summary>

```scala title="sql-examples/src/main/scala/buildaqueryfromoptionalsearchfilters/Step3TicketTableExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-sql:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
//> using file Helpdesk.scala
// Run from this folder: scala-cli run Step3TicketTableExample.scala
// Or from sql-examples: sbt "runMain buildaqueryfromoptionalsearchfilters.Step3TicketTableExample"

package buildaqueryfromoptionalsearchfilters

/**
 * How to Build a Query from Optional Search Filters — The Ticket Table and the
 * Search Form
 *
 * Creates the ticket table in a SQLite file, adds eight tickets and shows the
 * empty search form.
 */
object Step3TicketTableExample {
  def main(args: Array[String]): Unit = {
    val dbFile     = Files.createTempFile("helpdesk", ".db")
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    seedTickets(transactor)

    println(s"the empty form: ${TicketSearch()}")
    transactor.connect {
      println(s"tickets in the table: ${tickets.count}")
    }

    Files.deleteIfExists(dbFile)
  }
}
```

</details>

<details>
  <summary>The model and seed data, shared by every example file: <code>Helpdesk.scala</code></summary>

```scala title="sql-examples/src/main/scala/buildaqueryfromoptionalsearchfilters/Helpdesk.scala" showLineNumbers
package buildaqueryfromoptionalsearchfilters

/**
 * How to Build a Query from Optional Search Filters — Shared Model
 *
 * The ticket row and its table, the search form and its sort choices, and the
 * seed that creates the ticket table and adds eight tickets (section "The
 * Ticket Table and the Search Form"). It has no main method.
 */
case class Ticket(id: Long, subject: String, customer: String, status: String, priority: Int, openedOn: LocalDate)

object Ticket {
  given Schema[Ticket] = Schema.derived
}

val ticketTable = Table.derived[Ticket]
val tickets     = Repo.derived[Ticket, Long]

def seedTickets(transactor: JdbcTransactor): Unit =
  transactor.connect {
    ticketTable.createTable(SqlDialect.SQLite).update
    List(
      Ticket(1L, "Cannot log in after password reset", "Acme", "open", 1, LocalDate.of(2026, 9, 28)),
      Ticket(2L, "Invoice shows the 50% discount twice", "Globex", "closed", 2, LocalDate.of(2026, 10, 2)),
      Ticket(3L, "Login page loads slowly", "Acme", "pending", 2, LocalDate.of(2026, 10, 6)),
      Ticket(4L, "Export to CSV drops the last row", "Initech", "open", 3, LocalDate.of(2026, 10, 9)),
      Ticket(5L, "Two-factor code never arrives", "Globex", "open", 1, LocalDate.of(2026, 10, 14)),
      Ticket(6L, "Typo on the pricing page", "Acme", "closed", 3, LocalDate.of(2026, 10, 17)),
      Ticket(7L, "API returns 500 on large uploads", "Initech", "pending", 1, LocalDate.of(2026, 10, 21)),
      Ticket(8L, "Cannot change the billing email", "Acme", "open", 2, LocalDate.of(2026, 10, 24))
    ).foreach(tickets.insert)
  }

enum SortBy {
  case Newest, Oldest, MostUrgent
}

case class TicketSearch(
  text: Option[String] = None,
  customer: Option[String] = None,
  statuses: Set[String] = Set.empty,
  openedFrom: Option[LocalDate] = None,
  openedTo: Option[LocalDate] = None,
  sortBy: SortBy = SortBy.Newest,
  page: Int = 1,
  pageSize: Int = 20
)
```

</details>

## Turn a Filled-In Field into a Condition

Each field becomes an `Option[Frag]`: `Some` condition when the user filled it in, `None` when they didn't. `Option#map` does exactly that, so the customer field takes one line. A browser sends an empty text box as an empty string, not as a missing field, so we first treat blank text as empty:

```scala
def filled(field: Option[String]): Option[String] =
  field.map(_.trim).filter(_.nonEmpty)

def customerIs(search: TicketSearch): Option[Frag] =
  filled(search.customer).map(customer => sql"customer = $customer")
```

To see what a condition holds, we print its SQL text and its parameters. `Frag#sql` renders the text for a dialect, and `Frag#params` holds the values that are bound to the `?` placeholders:

```scala
def describe(frag: Frag): Unit = {
  println(frag.sql(SqlDialect.SQLite))
  println(frag.params)
}

customerIs(TicketSearch(customer = Some("Acme"))).foreach(describe)
// customer = ?
// Vector(DbString(Acme))
customerIs(TicketSearch(customer = Some("  ")))
// res3: Option[Frag] = None
customerIs(TicketSearch())
// res4: Option[Frag] = None
```

The customer name is a `?` in the text and a `DbString` in the parameters. A blank or missing customer gives `None`, so the query gets no condition for that field.

:::warning[Skip the condition, don't bind `None`]
The interpolator accepts an `Option` value and binds `None` as SQL `NULL`. So `sql"customer = ${search.customer}"` compiles, but for an empty field it sends `customer = NULL`, which is never true in SQL, and the search returns no rows at all.

A query with a fixed text can still handle an empty field: `sql"(${search.customer} IS NULL OR customer = ${search.customer})"` binds the value twice and lets the database skip the comparison when it is `NULL`. That works for an exact match, but not for the `LIKE` pattern or the `IN` list below, so this guide leaves out each empty field's condition instead.
:::

## Join the Conditions with AND

With one `Option[Frag]` per field, we need a `WHERE` clause that holds only the conditions that are present, joined with `AND`, and nothing at all when the form is empty. `Frag` has the parts for it: `Frag#++` joins two fragments, `Frag.literal` makes a fragment from fixed SQL text, and `Frag.empty` adds nothing. In zio-blocks 0.0.56 there is no helper that joins fragments with a separator, such as doobie's `whereAndOpt`, so we write one:

```scala
def whereAnd(conditions: Option[Frag]*): Frag = {
  val present = conditions.flatten
  if (present.isEmpty) Frag.empty
  else Frag.literal(" WHERE ") ++ present.map(c => sql"($c)").reduce((a, b) => a ++ Frag.literal(" AND ") ++ b)
}
```

The helper is our own code, not part of the library. `conditions.flatten` drops the `None`s. When nothing is left, the helper returns `Frag.empty`, so an empty form gets no `WHERE` at all. Otherwise, it wraps each condition in parentheses and puts `AND` between them. Interpolating a `Frag` into `sql"..."` splices its text and its parameters in place; only other values become `?` parameters. The parentheses matter when a condition contains `OR`: without them, `a AND b OR c` means `(a AND b) OR c` in SQL.

We keep the fixed `SELECT` in a value, since every search starts with it:

```scala
val selectTickets = Frag.literal("SELECT id, subject, customer, status, priority, opened_on FROM ticket")
```

We put the clause after it, first with no conditions and then with a customer:

```scala
describe(selectTickets ++ whereAnd())
// SELECT id, subject, customer, status, priority, opened_on FROM ticket
// Vector()
describe(selectTickets ++ whereAnd(customerIs(TicketSearch(customer = Some("Acme")))))
// SELECT id, subject, customer, status, priority, opened_on FROM ticket WHERE (customer = ?)
// Vector(DbString(Acme))
```

The query with a condition runs like any other `Frag`:

```scala
transactor.connect {
  (selectTickets ++ whereAnd(customerIs(TicketSearch(customer = Some("Acme"))))).query[Ticket].foreach(println)
}
// Ticket(1,Cannot log in after password reset,Acme,open,1,2026-09-28)
// Ticket(3,Login page loads slowly,Acme,pending,2,2026-10-06)
// Ticket(6,Typo on the pricing page,Acme,closed,3,2026-10-17)
// Ticket(8,Cannot change the billing email,Acme,open,2,2026-10-24)
```

Acme's four tickets come back. Every following filter is one more `Option[Frag]` passed to `whereAnd`.

<details>
  <summary>Runnable version, with the setup from section 3 and the functions from earlier sections: <code>Step5JoinWithAndExample.scala</code></summary>

```scala title="sql-examples/src/main/scala/buildaqueryfromoptionalsearchfilters/Step5JoinWithAndExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-sql:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
//> using file Helpdesk.scala SearchConditions.scala
// Run from this folder: scala-cli run Step5JoinWithAndExample.scala
// Or from sql-examples: sbt "runMain buildaqueryfromoptionalsearchfilters.Step5JoinWithAndExample"

package buildaqueryfromoptionalsearchfilters

/**
 * How to Build a Query from Optional Search Filters — Join the Conditions with
 * AND
 *
 * Puts the WHERE clause after the SELECT, with no condition and with a
 * customer, and runs the second query. The shared functions come from sections
 * 3 to 5; the SQLite table is set up as in section 3.
 */
object Step5JoinWithAndExample {
  def main(args: Array[String]): Unit = {
    // Setup from section 3: table created, eight tickets inserted
    val dbFile     = Files.createTempFile("helpdesk", ".db")
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
    seedTickets(transactor)

    println("no conditions:")
    describe(selectTickets ++ whereAnd())
    println("customer Acme:")
    describe(selectTickets ++ whereAnd(customerIs(TicketSearch(customer = Some("Acme")))))

    println("Acme's tickets:")
    transactor.connect {
      (selectTickets ++ whereAnd(customerIs(TicketSearch(customer = Some("Acme"))))).query[Ticket].foreach(println)
    }

    Files.deleteIfExists(dbFile)
  }
}
```

</details>

## Match Part of the Subject with LIKE

The text box searches the subject line. `LIKE` with `%` on both sides finds the text anywhere in the subject, but `%` and `_` in the user's own text are wildcards too. We escape them with an escape character of our choosing, `!`, and name it in the `ESCAPE` clause:

```scala
def likePattern(text: String): String =
  "%" + text.replace("!", "!!").replace("%", "!%").replace("_", "!_") + "%"

def subjectContains(search: TicketSearch): Option[Frag] =
  filled(search.text).map(text => sql"subject LIKE ${likePattern(text)} ESCAPE '!'")
```

`likePattern` escapes `!` first, so the `!` characters it adds afterwards aren't escaped a second time. The pattern itself is a bound parameter, like any other value.

To compare, we search for `50%`, once without escaping and once with `subjectContains`:

```scala
transactor.connect {
  println(sql"SELECT subject FROM ticket WHERE subject LIKE ${"%50%%"}".query[String])
  println((Frag.literal("SELECT subject FROM ticket") ++ whereAnd(subjectContains(TicketSearch(text = Some("50%"))))).query[String])
}
// List(Invoice shows the 50% discount twice, API returns 500 on large uploads)
// List(Invoice shows the 50% discount twice)
```

Without escaping, the `%` the user typed is a wildcard, so the API ticket's "500" matches as well. With the escaped pattern, only the invoice ticket matches.

We use `!` rather than the more common backslash because the `sql"..."` interpolator does not process escape sequences: `ESCAPE '\\'` inside it sends two backslashes, and SQLite rejects that with "ESCAPE expression must be a single character". Also note that SQLite's `LIKE` ignores the case of ASCII letters, while PostgreSQL's `LIKE` is case-sensitive and its `ILIKE` is not.

<details>
  <summary>Runnable version, with the setup from section 3 and the functions from earlier sections: <code>Step6LikeSearchExample.scala</code></summary>

```scala title="sql-examples/src/main/scala/buildaqueryfromoptionalsearchfilters/Step6LikeSearchExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-sql:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
//> using file Helpdesk.scala SearchConditions.scala
// Run from this folder: scala-cli run Step6LikeSearchExample.scala
// Or from sql-examples: sbt "runMain buildaqueryfromoptionalsearchfilters.Step6LikeSearchExample"

package buildaqueryfromoptionalsearchfilters

/**
 * How to Build a Query from Optional Search Filters — Match Part of the Subject
 * with LIKE
 *
 * Searches for `50%` once with the raw pattern and once with the escaped
 * pattern. The shared functions come from sections 3 to 6; the SQLite table is
 * set up as in section 3.
 */
object Step6LikeSearchExample {
  def main(args: Array[String]): Unit = {
    // Setup from section 3: table created, eight tickets inserted
    val dbFile     = Files.createTempFile("helpdesk", ".db")
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
    seedTickets(transactor)

    transactor.connect {
      print("raw pattern matches: ")
      println(sql"SELECT subject FROM ticket WHERE subject LIKE ${"%50%%"}".query[String])
      print("escaped pattern matches: ")
      println(
        (Frag.literal("SELECT subject FROM ticket") ++ whereAnd(subjectContains(TicketSearch(text = Some("50%")))))
          .query[String]
      )
    }

    Files.deleteIfExists(dbFile)
  }
}
```

</details>

## Filter a Date Range

The date range is two separate fields, so it gives two conditions, either of which may be missing. A `LocalDate` binds like any other value:

```scala
def openedOnOrAfter(search: TicketSearch): Option[Frag] =
  search.openedFrom.map(from => sql"opened_on >= $from")

def openedOnOrBefore(search: TicketSearch): Option[Frag] =
  search.openedTo.map(to => sql"opened_on <= $to")
```

To see both ends of the range, we ask for the tickets opened in the first half of October, and then for those opened from October 20 on, with no end date:

```scala
transactor.connect {
  val firstHalf = TicketSearch(openedFrom = Some(LocalDate.of(2026, 10, 1)), openedTo = Some(LocalDate.of(2026, 10, 15)))
  val fromOct20 = TicketSearch(openedFrom = Some(LocalDate.of(2026, 10, 20)))

  println((Frag.literal("SELECT id FROM ticket") ++ whereAnd(openedOnOrAfter(firstHalf), openedOnOrBefore(firstHalf))).query[Long])
  println((Frag.literal("SELECT id FROM ticket") ++ whereAnd(openedOnOrAfter(fromOct20), openedOnOrBefore(fromOct20))).query[Long])
}
// List(2, 3, 4, 5)
// List(7, 8)
```

Both ends of the range are inclusive. On SQLite, the table stores a `LocalDate` as ISO text such as `2026-10-14`, which sorts in date order, so `>=` and `<=` compare dates correctly; PostgreSQL has a native `DATE` type.

<details>
  <summary>Runnable version, with the setup from section 3 and the functions from earlier sections: <code>Step7DateRangeExample.scala</code></summary>

```scala title="sql-examples/src/main/scala/buildaqueryfromoptionalsearchfilters/Step7DateRangeExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-sql:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
//> using file Helpdesk.scala SearchConditions.scala
// Run from this folder: scala-cli run Step7DateRangeExample.scala
// Or from sql-examples: sbt "runMain buildaqueryfromoptionalsearchfilters.Step7DateRangeExample"

package buildaqueryfromoptionalsearchfilters

/**
 * How to Build a Query from Optional Search Filters — Filter a Date Range
 *
 * Asks for the tickets opened in the first half of October, and then for those
 * opened from October 20 on. The shared functions come from sections 3 to 7;
 * the SQLite table is set up as in section 3.
 */
object Step7DateRangeExample {
  def main(args: Array[String]): Unit = {
    // Setup from section 3: table created, eight tickets inserted
    val dbFile     = Files.createTempFile("helpdesk", ".db")
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
    seedTickets(transactor)

    transactor.connect {
      val firstHalf =
        TicketSearch(openedFrom = Some(LocalDate.of(2026, 10, 1)), openedTo = Some(LocalDate.of(2026, 10, 15)))
      val fromOct20 = TicketSearch(openedFrom = Some(LocalDate.of(2026, 10, 20)))

      print("first half of October: ")
      println(
        (Frag.literal("SELECT id FROM ticket") ++ whereAnd(openedOnOrAfter(firstHalf), openedOnOrBefore(firstHalf)))
          .query[Long]
      )
      print("from October 20: ")
      println(
        (Frag.literal("SELECT id FROM ticket") ++ whereAnd(openedOnOrAfter(fromOct20), openedOnOrBefore(fromOct20)))
          .query[Long]
      )
    }

    Files.deleteIfExists(dbFile)
  }
}
```

</details>

## Match Any of Several Statuses with IN

The status checkboxes allow several values at once, which is an `IN (...)` list with one `?` per value. The interpolator binds one value per `$`, and a `List` or `Set` there still compiles: it binds as one parameter that holds the whole collection as JSON text. So the obvious version renders `status IN (?)` and matches no row:

```scala
describe(sql"status IN (${List("open", "pending")})")
// status IN (?)
// Vector(DbString(["open","pending"]))

transactor.connect {
  println((Frag.literal("SELECT id FROM ticket") ++ whereAnd(Some(sql"status IN (${List("open", "pending")})"))).query[Long])
}
// List()
```

The query runs without an error and returns an empty list: the database compares each status with the text `["open","pending"]`, which no ticket has.

We build the list from one `sql"$v"` fragment per value instead, joined with commas:

```scala
def commaSeparated(values: Seq[String]): Frag =
  values.map(v => sql"$v").reduce((a, b) => a ++ Frag.literal(", ") ++ b)

def statusIn(search: TicketSearch): Option[Frag] =
  if (search.statuses.isEmpty) None
  else Some(sql"status IN (${commaSeparated(search.statuses.toSeq.sorted)})")
```

No ticked box means no condition, so the search shows every status. Sorting the values keeps the SQL and parameter order the same for the same set of boxes, whatever order the `Set` holds them in.

We tick two boxes:

```scala
val openOrPending = TicketSearch(statuses = Set("pending", "open"))
```

Then we look at the condition and run it:

```scala
statusIn(openOrPending).foreach(describe)
// status IN (?, ?)
// Vector(DbString(open), DbString(pending))

transactor.connect {
  println((Frag.literal("SELECT id FROM ticket") ++ whereAnd(statusIn(openOrPending))).query[Long])
}
// List(1, 3, 4, 5, 7, 8)
```

The SQL has two placeholders, and the six open or pending tickets match.

<details>
  <summary>Runnable version, with the setup from section 3 and the functions from earlier sections: <code>Step8StatusInExample.scala</code></summary>

```scala title="sql-examples/src/main/scala/buildaqueryfromoptionalsearchfilters/Step8StatusInExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-sql:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
//> using file Helpdesk.scala SearchConditions.scala
// Run from this folder: scala-cli run Step8StatusInExample.scala
// Or from sql-examples: sbt "runMain buildaqueryfromoptionalsearchfilters.Step8StatusInExample"

package buildaqueryfromoptionalsearchfilters

/**
 * How to Build a Query from Optional Search Filters — Match Any of Several
 * Statuses with IN
 *
 * Ticks the `open` and `pending` boxes, prints the condition and runs it. The
 * shared functions come from sections 3 to 8; the SQLite table is set up as in
 * section 3.
 */
object Step8StatusInExample {
  def main(args: Array[String]): Unit = {
    // Setup from section 3: table created, eight tickets inserted
    val dbFile     = Files.createTempFile("helpdesk", ".db")
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
    seedTickets(transactor)

    println("a List interpolated as one value:")
    describe(sql"status IN (${List("open", "pending")})")

    transactor.connect {
      print("matching ids: ")
      println(
        (Frag.literal("SELECT id FROM ticket") ++ whereAnd(Some(sql"status IN (${List("open", "pending")})")))
          .query[Long]
      )
    }

    val openOrPending = TicketSearch(statuses = Set("pending", "open"))

    println("open or pending:")
    statusIn(openOrPending).foreach(describe)

    transactor.connect {
      print("matching ids: ")
      println((Frag.literal("SELECT id FROM ticket") ++ whereAnd(statusIn(openOrPending))).query[Long])
    }

    Files.deleteIfExists(dbFile)
  }
}
```

</details>

## Sort by an Allowed Column

A sort order can't be a parameter: `ORDER BY ?` sorts by a constant value, not by the column named in it. So the column names must be part of the SQL text, which is exactly where user input must never go. The form therefore sends a choice from a fixed list, our `SortBy` enum, and each choice maps to a fixed piece of SQL:

```scala
def orderBy(sortBy: SortBy): Frag = sortBy match {
  case SortBy.Newest     => Frag.literal(" ORDER BY opened_on DESC, id DESC")
  case SortBy.Oldest     => Frag.literal(" ORDER BY opened_on ASC, id ASC")
  case SortBy.MostUrgent => Frag.literal(" ORDER BY priority ASC, opened_on DESC, id DESC")
}
```

Each order ends with `id`, so tickets with the same date or priority always come back in the same order, and a ticket can't move from one page to another between two requests.

A web request carries the choice as a string. We turn it into a `SortBy` with a default for anything we don't recognize, rather than with `SortBy.valueOf`, which throws:

```scala
def parseSortBy(param: String): SortBy =
  SortBy.values.find(_.toString == param).getOrElse(SortBy.Newest)

parseSortBy("MostUrgent")
// res14: SortBy = MostUrgent
parseSortBy("subject; DROP TABLE ticket")
// res15: SortBy = Newest
```

The unknown value falls back to `Newest`, and the text of the request never reaches the SQL.

:::danger
Never build the `ORDER BY` from the request's text with `Frag.literal` or `SqlLiteral`. Both splice their text into the SQL as it is, without parameters or checks, so they turn a sort field into an injection point.
:::

<details>
  <summary>Runnable version, which also prints the <code>ORDER BY</code> text of each choice: <code>Step9SortByExample.scala</code></summary>

```scala title="sql-examples/src/main/scala/buildaqueryfromoptionalsearchfilters/Step9SortByExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-sql:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
//> using file Helpdesk.scala SearchConditions.scala
// Run from this folder: scala-cli run Step9SortByExample.scala
// Or from sql-examples: sbt "runMain buildaqueryfromoptionalsearchfilters.Step9SortByExample"

package buildaqueryfromoptionalsearchfilters

/**
 * How to Build a Query from Optional Search Filters — Sort by an Allowed Column
 *
 * Turns the sort choice of a web request into a `SortBy`, with a default for
 * anything unknown, and prints the ORDER BY text of each choice. The shared
 * `orderBy` comes from section 9.
 */
object Step9SortByExample {
  def parseSortBy(param: String): SortBy =
    SortBy.values.find(_.toString == param).getOrElse(SortBy.Newest)

  def main(args: Array[String]): Unit = {
    println(s"MostUrgent parses to: ${parseSortBy("MostUrgent")}")
    println(s"an injection attempt parses to: ${parseSortBy("subject; DROP TABLE ticket")}")
    SortBy.values.foreach(sortBy => println(s"$sortBy:${orderBy(sortBy).sql(SqlDialect.SQLite)}"))
  }
}
```

</details>

## Return One Page of Results

A search screen shows one page at a time. `LIMIT` takes the page size and `OFFSET` skips the earlier pages. Both are numbers that come from the request, so we keep them in a sane range and bind them as parameters:

```scala
def pageSizeOf(search: TicketSearch): Int = search.pageSize.max(1).min(100)

def offsetOf(search: TicketSearch): Int = (search.page.max(1) - 1) * pageSizeOf(search)

def pageOf(search: TicketSearch): Frag =
  sql" LIMIT ${pageSizeOf(search)} OFFSET ${offsetOf(search)}"
```

A page number below 1 becomes 1, and the page size stays between 1 and 100, so a request can't ask for a million rows at once.

The screen also shows how many tickets match, to draw the page links. The count uses the same `WHERE` clause with a different `SELECT` and no sort order or paging. We ask for the second page of three tickets, newest first:

```scala
transactor.connect {
  val secondPage = TicketSearch(sortBy = SortBy.Newest, page = 2, pageSize = 3)
  val where = whereAnd()
  println((sql"SELECT COUNT(*) FROM ticket" ++ where).queryOne[Long])
  println((Frag.literal("SELECT id FROM ticket") ++ where ++ orderBy(secondPage.sortBy) ++ pageOf(secondPage)).query[Long])
}
// 8
// List(5, 4, 3)
```

All eight tickets match, and the second page holds the fourth to the sixth newest.

:::tip
`OFFSET` makes the database read and throw away every row before the page, so deep pages get slower as the table grows, and a row inserted between two requests shifts every later page by one. For long result lists, page with a cursor instead: [How to Upsert Rows and Paginate with Keyset Cursors](./upsert-and-paginate-with-keyset-cursors.md) shows `Frag.keysetAfter`, which writes its own `WHERE`, so a filtered search goes into a subquery that it pages through.
:::

<details>
  <summary>Runnable version, with the setup from section 3 and the functions from earlier sections: <code>Step10PagingExample.scala</code></summary>

```scala title="sql-examples/src/main/scala/buildaqueryfromoptionalsearchfilters/Step10PagingExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-sql:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
//> using file Helpdesk.scala SearchConditions.scala
// Run from this folder: scala-cli run Step10PagingExample.scala
// Or from sql-examples: sbt "runMain buildaqueryfromoptionalsearchfilters.Step10PagingExample"

package buildaqueryfromoptionalsearchfilters

/**
 * How to Build a Query from Optional Search Filters — Return One Page of
 * Results
 *
 * Counts all tickets and asks for the second page of three, newest first. The
 * shared functions come from sections 3 to 10; the SQLite table is set up as in
 * section 3.
 */
object Step10PagingExample {
  def main(args: Array[String]): Unit = {
    // Setup from section 3: table created, eight tickets inserted
    val dbFile     = Files.createTempFile("helpdesk", ".db")
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
    seedTickets(transactor)

    transactor.connect {
      val secondPage = TicketSearch(sortBy = SortBy.Newest, page = 2, pageSize = 3)
      val where      = whereAnd()
      print("matching tickets: ")
      println((sql"SELECT COUNT(*) FROM ticket" ++ where).queryOne[Long])
      print("second page of three, newest first: ")
      println(
        (Frag.literal("SELECT id FROM ticket") ++ where ++ orderBy(secondPage.sortBy) ++ pageOf(secondPage)).query[Long]
      )
    }

    Files.deleteIfExists(dbFile)
  }
}
```

</details>

## Run the Whole Search

Every field now has a function that gives its condition. We collect them in one list, and the search query is the `SELECT`, the `WHERE` clause, the sort order and the page, joined with `Frag#++`:

```scala
def conditions(search: TicketSearch): Seq[Option[Frag]] = Seq(
  subjectContains(search),
  customerIs(search),
  statusIn(search),
  openedOnOrAfter(search),
  openedOnOrBefore(search)
)

def searchQuery(search: TicketSearch): Frag =
  selectTickets ++ whereAnd(conditions(search)*) ++ orderBy(search.sortBy) ++ pageOf(search)

def countQuery(search: TicketSearch): Frag =
  sql"SELECT COUNT(*) FROM ticket" ++ whereAnd(conditions(search)*)
```

A new filter is one more function and one more line in `conditions`. We fill in three of the fields, the way a support agent looking for Acme's unresolved login problems would:

```scala
val loginProblems = TicketSearch(text = Some("log"), customer = Some("Acme"), statuses = Set("open", "pending"), sortBy = SortBy.MostUrgent)
```

Next, we print the query, count the matching tickets and run it:

```scala
describe(searchQuery(loginProblems))
// SELECT id, subject, customer, status, priority, opened_on FROM ticket WHERE (subject LIKE ? ESCAPE '!') AND (customer = ?) AND (status IN (?, ?)) ORDER BY priority ASC, opened_on DESC, id DESC LIMIT ? OFFSET ?
// Vector(DbString(%log%), DbString(Acme), DbString(open), DbString(pending), DbInt(20), DbInt(0))

transactor.connect {
  println(s"matching tickets: ${countQuery(loginProblems).queryOne[Long]}")
  searchQuery(loginProblems).query[Ticket].foreach(println)
}
// matching tickets: 2
// Ticket(1,Cannot log in after password reset,Acme,open,1,2026-09-28)
// Ticket(3,Login page loads slowly,Acme,pending,2,2026-10-06)
```

The `WHERE` clause holds the three filled-in conditions and nothing for the two empty date fields. Every value is a `?` in the SQL text and an entry in the parameters, in the same order.

That also holds for hostile input. We type a classic injection attempt into the text box:

```scala
val hostile = TicketSearch(text = Some("'; DROP TABLE ticket; --"))
```

Then we print the query and run it:

```scala
describe(searchQuery(hostile))
// SELECT id, subject, customer, status, priority, opened_on FROM ticket WHERE (subject LIKE ? ESCAPE '!') ORDER BY opened_on DESC, id DESC LIMIT ? OFFSET ?
// Vector(DbString(%'; DROP TABLE ticket; --%), DbInt(20), DbInt(0))

transactor.connect {
  println(searchQuery(hostile).query[Ticket])
  println(s"tickets in the table: ${tickets.count}")
}
// List()
// tickets in the table: 8
```

The input is a parameter like any other: it is a pattern no subject matches, so the search finds nothing and the table still holds all eight tickets.

<details>
  <summary>Every condition function from sections 4 to 11, in the shared file the examples use: <code>SearchConditions.scala</code></summary>

```scala title="sql-examples/src/main/scala/buildaqueryfromoptionalsearchfilters/SearchConditions.scala" showLineNumbers
package buildaqueryfromoptionalsearchfilters

/**
 * How to Build a Query from Optional Search Filters — Shared Condition
 * Functions
 *
 * The condition, sort and paging functions that sections 4 to 10 define and
 * later sections reuse, each labelled with its section. It has no main method.
 */

// Section "Turn a Filled-In Field into a Condition"
def filled(field: Option[String]): Option[String] =
  field.map(_.trim).filter(_.nonEmpty)

def customerIs(search: TicketSearch): Option[Frag] =
  filled(search.customer).map(customer => sql"customer = $customer")

def describe(frag: Frag): Unit = {
  println(frag.sql(SqlDialect.SQLite))
  println(frag.params)
}

// Section "Join the Conditions with AND"
def whereAnd(conditions: Option[Frag]*): Frag = {
  val present = conditions.flatten
  if (present.isEmpty) Frag.empty
  else Frag.literal(" WHERE ") ++ present.map(c => sql"($c)").reduce((a, b) => a ++ Frag.literal(" AND ") ++ b)
}

val selectTickets = Frag.literal("SELECT id, subject, customer, status, priority, opened_on FROM ticket")

// Section "Match Part of the Subject with LIKE"
def likePattern(text: String): String =
  "%" + text.replace("!", "!!").replace("%", "!%").replace("_", "!_") + "%"

def subjectContains(search: TicketSearch): Option[Frag] =
  filled(search.text).map(text => sql"subject LIKE ${likePattern(text)} ESCAPE '!'")

// Section "Filter a Date Range"
def openedOnOrAfter(search: TicketSearch): Option[Frag] =
  search.openedFrom.map(from => sql"opened_on >= $from")

def openedOnOrBefore(search: TicketSearch): Option[Frag] =
  search.openedTo.map(to => sql"opened_on <= $to")

// Section "Match Any of Several Statuses with IN"
def commaSeparated(values: Seq[String]): Frag =
  values.map(v => sql"$v").reduce((a, b) => a ++ Frag.literal(", ") ++ b)

def statusIn(search: TicketSearch): Option[Frag] =
  if (search.statuses.isEmpty) None
  else Some(sql"status IN (${commaSeparated(search.statuses.toSeq.sorted)})")

// Section "Sort by an Allowed Column"
def orderBy(sortBy: SortBy): Frag = sortBy match {
  case SortBy.Newest     => Frag.literal(" ORDER BY opened_on DESC, id DESC")
  case SortBy.Oldest     => Frag.literal(" ORDER BY opened_on ASC, id ASC")
  case SortBy.MostUrgent => Frag.literal(" ORDER BY priority ASC, opened_on DESC, id DESC")
}

// Section "Return One Page of Results"
def pageSizeOf(search: TicketSearch): Int = search.pageSize.max(1).min(100)

def offsetOf(search: TicketSearch): Int = (search.page.max(1) - 1) * pageSizeOf(search)

def pageOf(search: TicketSearch): Frag =
  sql" LIMIT ${pageSizeOf(search)} OFFSET ${offsetOf(search)}"

// Section "Run the Whole Search" (the list of conditions and the support agent's search; the queries are in Step11)
def conditions(search: TicketSearch): Seq[Option[Frag]] = Seq(
  subjectContains(search),
  customerIs(search),
  statusIn(search),
  openedOnOrAfter(search),
  openedOnOrBefore(search)
)

val loginProblems = TicketSearch(
  text = Some("log"),
  customer = Some("Acme"),
  statuses = Set("open", "pending"),
  sortBy = SortBy.MostUrgent
)
```

</details>

<details>
  <summary>Runnable version, with the setup from section 3 and the functions from earlier sections: <code>Step11WholeSearchExample.scala</code></summary>

```scala title="sql-examples/src/main/scala/buildaqueryfromoptionalsearchfilters/Step11WholeSearchExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-sql:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
//> using file Helpdesk.scala SearchConditions.scala
// Run from this folder: scala-cli run Step11WholeSearchExample.scala
// Or from sql-examples: sbt "runMain buildaqueryfromoptionalsearchfilters.Step11WholeSearchExample"

package buildaqueryfromoptionalsearchfilters

/**
 * How to Build a Query from Optional Search Filters — Run the Whole Search
 *
 * Joins the SELECT, the WHERE clause, the sort order and the page, runs the
 * support agent's search with its count, and then a hostile input. The shared
 * functions come from sections 3 to 11; the SQLite table is set up as in
 * section 3.
 */
object Step11WholeSearchExample {
  def searchQuery(search: TicketSearch): Frag =
    selectTickets ++ whereAnd(conditions(search)*) ++ orderBy(search.sortBy) ++ pageOf(search)

  def countQuery(search: TicketSearch): Frag =
    sql"SELECT COUNT(*) FROM ticket" ++ whereAnd(conditions(search)*)

  def main(args: Array[String]): Unit = {
    // Setup from section 3: table created, eight tickets inserted
    val dbFile     = Files.createTempFile("helpdesk", ".db")
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
    seedTickets(transactor)

    println("support agent's search:")
    describe(searchQuery(loginProblems))

    transactor.connect {
      println(s"matching tickets: ${countQuery(loginProblems).queryOne[Long]}")
      searchQuery(loginProblems).query[Ticket].foreach(println)
    }

    val hostile = TicketSearch(text = Some("'; DROP TABLE ticket; --"))

    println("hostile input:")
    describe(searchQuery(hostile))

    transactor.connect {
      print("hostile search finds: ")
      println(searchQuery(hostile).query[Ticket])
      println(s"tickets in the table: ${tickets.count}")
    }

    Files.deleteIfExists(dbFile)
  }
}
```

</details>

## The Same Search with the Query Builder

The `sql` module also has a query builder, [`SqlQuery`](../reference/sql/sql-query.md). It writes the `SELECT` list from the table's schema, joins related tables, and takes `WHERE` conditions as `Frag`s. `SqlQuery#where` adds one condition per call and the builder joins them with `AND`, so optional filters fold into it without a helper. We reuse the condition functions from the earlier sections:

```scala
def searchWithBuilder(search: TicketSearch): SqlQuery[Ticket] = {
  val filtered = conditions(search).flatten.foldLeft(SqlQuery.from(ticketTable)) { (query, condition) =>
    query.where(sql"($condition)")
  }
  val sorted = search.sortBy match {
    case SortBy.Newest     => filtered.orderBy("opened_on", SortOrder.Desc).orderBy("id", SortOrder.Desc)
    case SortBy.Oldest     => filtered.orderBy("opened_on").orderBy("id")
    case SortBy.MostUrgent =>
      filtered.orderBy("priority").orderBy("opened_on", SortOrder.Desc).orderBy("id", SortOrder.Desc)
  }
  sorted.limit(pageSizeOf(search)).offset(offsetOf(search))
}
```

`conditions(search).flatten` keeps only the filled-in conditions, and `Seq#foldLeft` adds them to the query one by one. An empty form adds none, and the builder then writes no `WHERE`. `SqlQuery#orderBy` checks that each name is a plain SQL identifier, and `SqlQuery#limit` and `SqlQuery#offset` write their numbers into the SQL text.

:::warning
`SqlQuery#where` joins its conditions with `AND` without parentheses, a known bug in 0.0.56. A condition that contains `OR` then mixes with the others: `query.where(a).where(b OR c)` means `(a AND b) OR c`, which returns rows that fail `a`. That's why the fold wraps each condition in `sql"($condition)"`. For the same reason, never pass `Frag.empty` to `SqlQuery#where`: it renders `WHERE` with nothing after it.
:::

We build the support agent's search with it:

```scala
val builderQuery = searchWithBuilder(loginProblems)
```

`SqlQuery#explain` prints the SQL with numbered placeholders and the type of each parameter. `SqlQuery#toFrag` renders the query as a `Frag`, which runs like the others:

```scala
println(builderQuery.explain(SqlDialect.SQLite))
// SELECT t0."id", t0."subject", t0."customer", t0."status", t0."priority", t0."opened_on" FROM "ticket" AS t0 WHERE (subject LIKE ?1 ESCAPE '!') AND (customer = ?2) AND (status IN (?3, ?4)) ORDER BY t0."priority" ASC, t0."opened_on" DESC, t0."id" DESC LIMIT 20 OFFSET 0
// -- params: 1:String, 2:String, 3:String, 4:String

transactor.connect {
  builderQuery.toFrag(SqlDialect.SQLite).query[Ticket].foreach(println)
}
// Ticket(1,Cannot log in after password reset,Acme,open,1,2026-09-28)
// Ticket(3,Login page loads slowly,Acme,pending,2,2026-10-06)
```

The builder returns the same tickets in the same order as the `Frag` version. Here is how the two approaches compare:

| Concern                     | `Frag` and `whereAnd`                                       | `SqlQuery`                                           |
| --------------------------- | ----------------------------------------------------------- | ---------------------------------------------------- |
| `SELECT` list               | Written by hand                                             | Generated from the table's schema                    |
| Joins                       | Written by hand                                             | Typed relations; see the joins guide                 |
| Joining conditions with AND | Our `whereAnd` helper                                       | Built in: one `SqlQuery#where` call per condition    |
| A condition with `OR`       | `whereAnd` adds the parentheses                             | Add the parentheses yourself (a known bug in 0.0.56) |
| Sort order                  | Fixed `Frag.literal` per choice                             | `SqlQuery#orderBy`, which checks identifiers         |
| `LIMIT` and `OFFSET`        | Bound parameters                                            | Numbers written into the SQL text                    |
| Other SQL                   | Anything SQL allows, such as `COUNT(*)` on the same `WHERE` | Only what the builder models                         |

Choose `SqlQuery` when the search reads whole rows from one or more joined tables, so the builder can write the column list and the joins for you; [How to Write Joins and Inspect the Generated SQL](./write-joins-and-inspect-generated-sql.md) shows how. Choose `Frag` when the query needs SQL the builder doesn't model, or when you want to reuse the same `WHERE` clause for a count, as `countQuery` does. Either way, the conditions are the same `Option[Frag]` values.

<details>
  <summary>Runnable version, with the setup from section 3 and the functions from earlier sections: <code>Step12QueryBuilderExample.scala</code></summary>

```scala title="sql-examples/src/main/scala/buildaqueryfromoptionalsearchfilters/Step12QueryBuilderExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-sql:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
//> using file Helpdesk.scala SearchConditions.scala
// Run from this folder: scala-cli run Step12QueryBuilderExample.scala
// Or from sql-examples: sbt "runMain buildaqueryfromoptionalsearchfilters.Step12QueryBuilderExample"

package buildaqueryfromoptionalsearchfilters

/**
 * How to Build a Query from Optional Search Filters — The Same Search with the
 * Query Builder
 *
 * Builds the support agent's search with `SqlQuery`, prints its SQL and runs
 * it. The shared functions come from sections 3 to 11; the SQLite table is set
 * up as in section 3.
 */
object Step12QueryBuilderExample {
  def searchWithBuilder(search: TicketSearch): SqlQuery[Ticket] = {
    val filtered = conditions(search).flatten.foldLeft(SqlQuery.from(ticketTable)) { (query, condition) =>
      query.where(sql"($condition)")
    }
    val sorted = search.sortBy match {
      case SortBy.Newest     => filtered.orderBy("opened_on", SortOrder.Desc).orderBy("id", SortOrder.Desc)
      case SortBy.Oldest     => filtered.orderBy("opened_on").orderBy("id")
      case SortBy.MostUrgent =>
        filtered.orderBy("priority").orderBy("opened_on", SortOrder.Desc).orderBy("id", SortOrder.Desc)
    }
    sorted.limit(pageSizeOf(search)).offset(offsetOf(search))
  }

  def main(args: Array[String]): Unit = {
    // Setup from section 3: table created, eight tickets inserted
    val dbFile     = Files.createTempFile("helpdesk", ".db")
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
    seedTickets(transactor)

    val builderQuery = searchWithBuilder(loginProblems)

    println(builderQuery.explain(SqlDialect.SQLite))

    println("builder results:")
    transactor.connect {
      builderQuery.toFrag(SqlDialect.SQLite).query[Ticket].foreach(println)
    }

    Files.deleteIfExists(dbFile)
  }
}
```

</details>

## Putting It Together

The complete example runs the whole search from one program. It creates and fills the ticket table, defines the search form and every condition function from the sections above, runs the support agent's search with its count, the second page of all tickets, and the hostile input, and then runs the same search through `SqlQuery`. It labels each part of its output and deletes its SQLite file when it finishes:

```scala title="sql-examples/src/main/scala/buildaqueryfromoptionalsearchfilters/CompleteExample.scala"
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-sql:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
// Run from this folder: scala-cli run CompleteExample.scala
// Or from sql-examples: sbt "runMain buildaqueryfromoptionalsearchfilters.CompleteExample"

package buildaqueryfromoptionalsearchfilters

/**
 * How to Build a Query from Optional Search Filters — Complete Example
 *
 * Creates and fills the ticket table, defines the search form and every
 * condition function, runs the support agent's search with its count, a second
 * page and a hostile input, and then runs the same search through `SqlQuery`.
 * It deletes its SQLite file at the end.
 */
object CompleteExample {
  case class Ticket(id: Long, subject: String, customer: String, status: String, priority: Int, openedOn: LocalDate)

  object Ticket {
    given Schema[Ticket] = Schema.derived
  }

  val ticketTable = Table.derived[Ticket]
  val tickets     = Repo.derived[Ticket, Long]

  enum SortBy {
    case Newest, Oldest, MostUrgent
  }

  case class TicketSearch(
    text: Option[String] = None,
    customer: Option[String] = None,
    statuses: Set[String] = Set.empty,
    openedFrom: Option[LocalDate] = None,
    openedTo: Option[LocalDate] = None,
    sortBy: SortBy = SortBy.Newest,
    page: Int = 1,
    pageSize: Int = 20
  )

  def filled(field: Option[String]): Option[String] =
    field.map(_.trim).filter(_.nonEmpty)

  def customerIs(search: TicketSearch): Option[Frag] =
    filled(search.customer).map(customer => sql"customer = $customer")

  def describe(frag: Frag): Unit = {
    println(frag.sql(SqlDialect.SQLite))
    println(frag.params)
  }

  def whereAnd(conditions: Option[Frag]*): Frag = {
    val present = conditions.flatten
    if (present.isEmpty) Frag.empty
    else Frag.literal(" WHERE ") ++ present.map(c => sql"($c)").reduce((a, b) => a ++ Frag.literal(" AND ") ++ b)
  }

  val selectTickets = Frag.literal("SELECT id, subject, customer, status, priority, opened_on FROM ticket")

  def likePattern(text: String): String =
    "%" + text.replace("!", "!!").replace("%", "!%").replace("_", "!_") + "%"

  def subjectContains(search: TicketSearch): Option[Frag] =
    filled(search.text).map(text => sql"subject LIKE ${likePattern(text)} ESCAPE '!'")

  def openedOnOrAfter(search: TicketSearch): Option[Frag] =
    search.openedFrom.map(from => sql"opened_on >= $from")

  def openedOnOrBefore(search: TicketSearch): Option[Frag] =
    search.openedTo.map(to => sql"opened_on <= $to")

  def commaSeparated(values: Seq[String]): Frag =
    values.map(v => sql"$v").reduce((a, b) => a ++ Frag.literal(", ") ++ b)

  def statusIn(search: TicketSearch): Option[Frag] =
    if (search.statuses.isEmpty) None
    else Some(sql"status IN (${commaSeparated(search.statuses.toSeq.sorted)})")

  def orderBy(sortBy: SortBy): Frag = sortBy match {
    case SortBy.Newest     => Frag.literal(" ORDER BY opened_on DESC, id DESC")
    case SortBy.Oldest     => Frag.literal(" ORDER BY opened_on ASC, id ASC")
    case SortBy.MostUrgent => Frag.literal(" ORDER BY priority ASC, opened_on DESC, id DESC")
  }

  def pageSizeOf(search: TicketSearch): Int = search.pageSize.max(1).min(100)

  def offsetOf(search: TicketSearch): Int = (search.page.max(1) - 1) * pageSizeOf(search)

  def pageOf(search: TicketSearch): Frag =
    sql" LIMIT ${pageSizeOf(search)} OFFSET ${offsetOf(search)}"

  def conditions(search: TicketSearch): Seq[Option[Frag]] = Seq(
    subjectContains(search),
    customerIs(search),
    statusIn(search),
    openedOnOrAfter(search),
    openedOnOrBefore(search)
  )

  def searchQuery(search: TicketSearch): Frag =
    selectTickets ++ whereAnd(conditions(search)*) ++ orderBy(search.sortBy) ++ pageOf(search)

  def countQuery(search: TicketSearch): Frag =
    sql"SELECT COUNT(*) FROM ticket" ++ whereAnd(conditions(search)*)

  def searchWithBuilder(search: TicketSearch): SqlQuery[Ticket] = {
    val filtered = conditions(search).flatten.foldLeft(SqlQuery.from(ticketTable)) { (query, condition) =>
      query.where(sql"($condition)")
    }
    val sorted = search.sortBy match {
      case SortBy.Newest     => filtered.orderBy("opened_on", SortOrder.Desc).orderBy("id", SortOrder.Desc)
      case SortBy.Oldest     => filtered.orderBy("opened_on").orderBy("id")
      case SortBy.MostUrgent =>
        filtered.orderBy("priority").orderBy("opened_on", SortOrder.Desc).orderBy("id", SortOrder.Desc)
    }
    sorted.limit(pageSizeOf(search)).offset(offsetOf(search))
  }

  def main(args: Array[String]): Unit = {
    val dbFile     = Files.createTempFile("helpdesk", ".db")
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.connect {
      ticketTable.createTable(SqlDialect.SQLite).update
      List(
        Ticket(1L, "Cannot log in after password reset", "Acme", "open", 1, LocalDate.of(2026, 9, 28)),
        Ticket(2L, "Invoice shows the 50% discount twice", "Globex", "closed", 2, LocalDate.of(2026, 10, 2)),
        Ticket(3L, "Login page loads slowly", "Acme", "pending", 2, LocalDate.of(2026, 10, 6)),
        Ticket(4L, "Export to CSV drops the last row", "Initech", "open", 3, LocalDate.of(2026, 10, 9)),
        Ticket(5L, "Two-factor code never arrives", "Globex", "open", 1, LocalDate.of(2026, 10, 14)),
        Ticket(6L, "Typo on the pricing page", "Acme", "closed", 3, LocalDate.of(2026, 10, 17)),
        Ticket(7L, "API returns 500 on large uploads", "Initech", "pending", 1, LocalDate.of(2026, 10, 21)),
        Ticket(8L, "Cannot change the billing email", "Acme", "open", 2, LocalDate.of(2026, 10, 24))
      ).foreach(tickets.insert)
    }

    val loginProblems = TicketSearch(
      text = Some("log"),
      customer = Some("Acme"),
      statuses = Set("open", "pending"),
      sortBy = SortBy.MostUrgent
    )

    println("support agent's search:")
    describe(searchQuery(loginProblems))

    transactor.connect {
      println(s"matching tickets: ${countQuery(loginProblems).queryOne[Long]}")
      searchQuery(loginProblems).query[Ticket].foreach(println)
    }

    transactor.connect {
      val secondPage = TicketSearch(sortBy = SortBy.Newest, page = 2, pageSize = 3)
      val where      = whereAnd()
      print("all tickets: ")
      println((sql"SELECT COUNT(*) FROM ticket" ++ where).queryOne[Long])
      print("second page of three, newest first: ")
      println(
        (Frag.literal("SELECT id FROM ticket") ++ where ++ orderBy(secondPage.sortBy) ++ pageOf(secondPage)).query[Long]
      )
    }

    val hostile = TicketSearch(text = Some("'; DROP TABLE ticket; --"))

    println("hostile input:")
    describe(searchQuery(hostile))

    transactor.connect {
      print("hostile search finds: ")
      println(searchQuery(hostile).query[Ticket])
      println(s"tickets in the table: ${tickets.count}")
    }

    val builderQuery = searchWithBuilder(loginProblems)

    println("same search with SqlQuery:")
    println(builderQuery.explain(SqlDialect.SQLite))

    transactor.connect {
      builderQuery.toFrag(SqlDialect.SQLite).query[Ticket].foreach(println)
    }

    Files.deleteIfExists(dbFile)
  }
}
```

Running it prints:

```text
support agent's search:
SELECT id, subject, customer, status, priority, opened_on FROM ticket WHERE (subject LIKE ? ESCAPE '!') AND (customer = ?) AND (status IN (?, ?)) ORDER BY priority ASC, opened_on DESC, id DESC LIMIT ? OFFSET ?
Vector(DbString(%log%), DbString(Acme), DbString(open), DbString(pending), DbInt(20), DbInt(0))
matching tickets: 2
Ticket(1,Cannot log in after password reset,Acme,open,1,2026-09-28)
Ticket(3,Login page loads slowly,Acme,pending,2,2026-10-06)
all tickets: 8
second page of three, newest first: List(5, 4, 3)
hostile input:
SELECT id, subject, customer, status, priority, opened_on FROM ticket WHERE (subject LIKE ? ESCAPE '!') ORDER BY opened_on DESC, id DESC LIMIT ? OFFSET ?
Vector(DbString(%'; DROP TABLE ticket; --%), DbInt(20), DbInt(0))
hostile search finds: List()
tickets in the table: 8
same search with SqlQuery:
SELECT t0."id", t0."subject", t0."customer", t0."status", t0."priority", t0."opened_on" FROM "ticket" AS t0 WHERE (subject LIKE ?1 ESCAPE '!') AND (customer = ?2) AND (status IN (?3, ?4)) ORDER BY t0."priority" ASC, t0."opened_on" DESC, t0."id" DESC LIMIT 20 OFFSET 0
-- params: 1:String, 2:String, 3:String, 4:String
Ticket(1,Cannot log in after password reset,Acme,open,1,2026-09-28)
Ticket(3,Login page loads slowly,Acme,pending,2,2026-10-06)
```

The output matches the sections: the same two tickets from both the `Frag` search and `SqlQuery`, and the hostile input bound as a parameter.

## Running the Examples

All runnable code from this guide is in [`sql-examples/src/main/scala/buildaqueryfromoptionalsearchfilters/`](https://github.com/zio/zio-blocks/tree/main/sql-examples/src/main/scala/buildaqueryfromoptionalsearchfilters). Run any file from that folder with `scala-cli run <File>.scala`, or from `sql-examples` with `sbt "runMain buildaqueryfromoptionalsearchfilters.<ObjectName>"`.

## Going Further

- The [Frag reference](../reference/sql/frag.md) lists every way to build and run a fragment, including `Frag.sequence` and `Frag.values`.
- The [SqlQuery reference](../reference/sql/sql-query.md) covers grouping, `HAVING` and inspecting a built query.
- [sql(table) Checked Interpolation](./sql-checked-interpolation.md) checks table and column names in `sql"..."` at compile time, and explains when `SqlLiteral` is safe to use.
- [How to Upsert Rows and Paginate with Keyset Cursors](./upsert-and-paginate-with-keyset-cursors.md) pages through long results without `OFFSET`.

---

## How to Build a Schema Registry with DynamicSchema

In this guide, we will build a small schema registry for a bookshop's `BookPublished` events, using [`DynamicSchema`](../reference/schema/dynamic-schema.md) and [`BindingResolver`](../reference/schema/binding-resolver.md) from ZIO Blocks Schema. By the end, you will have a registry that stores every version of an event's shape in a directory, a consumer that loads a stored shape and checks incoming events against it without the event's class, a consumer that turns a stored shape back into a typed `Schema` to decode events, and a small comparison that tells you what changed between version 1 and version 2.

Services that exchange data through a message broker, an event store or an API see the shape of that data change over time. A schema registry is a central store of those shapes: it answers "what does version 3 of `BookPublished` look like?", so producers and consumers can check data against it. We start from a `Schema` derived for a case class and build the registry step by step:

- Turn a `Schema[A]` into a `DynamicSchema`, a description of the shape that is plain data.
- Store it in the registry under a subject and a version number.
- Load it in another program that has no compiled class for the event.
- Check incoming events against it and report what doesn't fit.
- Rebind it to a Scala type with a `BindingResolver` to get typed values back.
- Register version 2 and compare it with version 1.

## The Problem

A registry needs the event's shape as data that it can store, send and read back. A `Schema[A]` doesn't qualify, because it holds the Scala functions that build and take apart an `A`, and functions can't be written to a file. So teams often describe each event a second time by hand, next to the case class, and write a checker for that description.

For example, consider version 1 of `BookPublished` with an ISBN, a title and a price. A hand-written description and checker look like this:

```scala
// A hand-written description of version 1, kept next to the case class
final case class FieldSpec(name: String, typeName: String)

val bookPublishedV1: List[FieldSpec] = List(
  FieldSpec("isbn", "String"),
  FieldSpec("title", "String"),
  FieldSpec("price", "BigDecimal")
)

// A hand-written checker for events that arrive as maps
def check(event: Map[String, Any], spec: List[FieldSpec]): List[String] = {
  val missing = spec.filterNot(f => event.contains(f.name)).map(f => s"missing field ${f.name}")
  val unknown = event.keySet.diff(spec.map(_.name).toSet).toList.map(name => s"unknown field $name")
  val wrongType = spec.flatMap { f =>
    event.get(f.name).toList.collect {
      case v if f.typeName == "String" && !v.isInstanceOf[String]         => s"${f.name} is not a String"
      case v if f.typeName == "BigDecimal" && !v.isInstanceOf[BigDecimal] => s"${f.name} is not a BigDecimal"
    }
  }
  missing ++ unknown ++ wrongType
}
```

This approach breaks down as soon as the event grows. Nothing ties `bookPublishedV1` to the case class, so a field added to one is easily forgotten in the other. Every new field type needs another case in the checker, and lists, optional fields, nested records and sealed traits each need their own code. Storing the description and reading it back in another service is more code again.

In this guide, we solve this by letting the `Schema` describe itself: `Schema#toDynamicSchema` turns it into a `DynamicSchema`, which keeps the full shape (type names, fields, cases, element types, validations) and drops the functions. `DynamicSchema` already knows how to check a value, how to be stored, and how to become a typed `Schema` again.

## Prerequisites

To follow along, add the schema module and its Avro module to your build. The registry stores schemas in Avro, for a reason explained in [Choose a Storage Format](#choose-a-storage-format):

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-schema"      % "0.0.56",
  "dev.zio" %% "zio-blocks-schema-avro" % "0.0.56"
)

// Scala 3.9 and newer: the same modules with the zio-blocks-next- prefix
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-schema"      % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema-avro" % "0.0.56"
)
scalacOptions += "-experimental"
```

For scala-cli, Scala.js, or cross-building, see [How to Add ZIO Blocks to Your Build](../getting-started/installation.md). The code on this page compiles on Scala 2.13 and Scala 3, and the registry runs on the JVM only: `zio-blocks-schema-avro` is published for the JVM, and the registry writes files with `java.nio.file`. `DynamicSchema` and `BindingResolver` themselves also work on Scala.js, except `BindingResolver.reflection`, which finds classes only on the JVM.

We assume you can derive a `Schema` for a case class, as shown in [Your First Schema](./your-first-schema.md), and that you know `DynamicValue`, the untyped value tree described on the [DynamicValue reference page](../reference/schema/dynamic-value.md). The [DynamicSchema reference page](../reference/schema/dynamic-schema.md) and the [BindingResolver reference page](../reference/schema/binding-resolver.md) cover the two main types in more depth than this guide needs.

## The Events

Every example on this page uses two versions of one event. Version 1 has an ISBN, a title and a price. Version 2 adds a list of authors and an optional edition number. In a real code base there is one `BookPublished` class whose fields change between releases; here the two versions live in objects named `v1` and `v2`, so one page can show both.

We import the schema package, the binding package for `BindingResolver`, the JSON package and the Avro format, and define both versions:

```scala

object v1 {
  final case class BookPublished(isbn: String, title: String, price: BigDecimal)

  object BookPublished {
    implicit val schema: Schema[BookPublished] = Schema.derived
  }
}

object v2 {
  final case class BookPublished(
    isbn: String,
    title: String,
    price: BigDecimal,
    authors: List[String],
    edition: Option[Int]
  )

  object BookPublished {
    implicit val schema: Schema[BookPublished] = Schema.derived
  }
}
```

The object a class lives in is part of its identity: the stored schema of `v1.BookPublished` names the type `BookPublished` with the owner `v1`. That matters when we rebind a stored schema in [Rebind to a Scala Type](#rebind-to-a-scala-type).

## Choose a Storage Format

`Schema#toDynamicSchema` gives us the shape as a `DynamicSchema`, and the `DynamicSchema` companion has a `Schema[DynamicSchema]`, so any ZIO Blocks format can write it. The format matters, though: in 0.0.56 a `DynamicSchema` written as JSON, YAML, TOON or MessagePack reads back as an accept-anything schema, so the registry stores Avro bytes.

Seeing the failure once makes the choice clear. We take version 1's schema, an event that is wrong in four ways (a number for the ISBN, no title, a text price and an extra field), and a codec for `DynamicSchema` in each of the two formats:

```scala
val v1Schema: DynamicSchema = Schema[v1.BookPublished].toDynamicSchema

val wrongEvent: DynamicValue =
  Json.parseUnsafe("""{"isbn": 9781000000011, "price": "free", "pages": 300}""").toDynamicValue

val schemaAsJson = Schema[DynamicSchema].derive(JsonFormat)
val schemaAsAvro = Schema[DynamicSchema].derive(AvroFormat)

val fromJson = schemaAsJson.decode(schemaAsJson.encode(v1Schema))
val fromAvro = schemaAsAvro.decode(schemaAsAvro.encode(v1Schema))
```

The schema we started with rejects the event. The copy that went through JSON is no longer equal to it, has become a `Dynamic` schema, and accepts the event:

```scala
println(v1Schema.check(wrongEvent))
// Some(zio.blocks.schema.SchemaError: Expected String, got Long at: .isbn)
println(fromJson.map(_ == v1Schema))
// Right(false)
println(fromJson.map(_.reflect.isDynamic))
// Right(true)
println(fromJson.map(_.check(wrongEvent)))
// Right(None)
```

The copy that went through Avro is equal to the original and rejects the event just as the original does:

```scala
println(fromAvro.map(_ == v1Schema))
// Right(true)
println(fromAvro.map(_.check(wrongEvent)))
// Right(Some(zio.blocks.schema.SchemaError: Expected String, got Long at: .isbn))
```

The JSON codec loses two kinds of information that the stored schema depends on. It writes a tagged value such as "this node is a `Record`" as an object with one key, `{"Record": {...}}`, and reads that object back as a plain record. It also reads every JSON number back as an `Int`, a `Long` or a `BigDecimal`, whatever it was before. When `DynamicSchema` reads a shape it doesn't recognise, it falls back to `Dynamic` without reporting an error. This is a known bug in 0.0.56. Avro keeps each tag and each number's type, so it round-trips.

## Build the Registry

Our registry keeps each subject in its own directory, with one file per version: `v1.avro`, `v2.avro`, and so on. A subject is the name the registry files schemas under, such as `book-published`. Next to each Avro file it writes the same schema as indented JSON, for people to read. A map in memory caches the schemas it has already read.

The registry has four operations. `SchemaRegistry#register` stores a schema and returns its version number; registering a schema that is already stored returns the existing version instead of adding a new one. `SchemaRegistry#versions` lists the stored versions of a subject, `SchemaRegistry#load` reads one version, and `SchemaRegistry#latest` reads the newest one:

```scala
final class SchemaRegistry(root: Path) {
  private val codec = Schema[DynamicSchema].derive(AvroFormat)
  private val cache = scala.collection.mutable.Map.empty[(String, Int), DynamicSchema]

  def versions(subject: String): List[Int] = {
    val names = Option(root.resolve(subject).toFile.list()).fold(List.empty[String])(_.toList)
    names.collect { case name if name.matches("v[0-9]+\\.avro") => name.drop(1).dropRight(5).toInt }.sorted
  }

  def register(subject: String, schema: DynamicSchema): Int =
    versions(subject).find(version => load(subject, version) == schema) match {
      case Some(existing) => existing
      case None           =>
        val version = versions(subject).lastOption.fold(1)(_ + 1)
        val dir     = Files.createDirectories(root.resolve(subject))
        Files.write(dir.resolve(s"v$version.avro"), codec.encode(schema))
        Files.write(dir.resolve(s"v$version.json"), jsonView(schema).getBytes(UTF_8))
        cache((subject, version)) = schema
        version
    }

  def load(subject: String, version: Int): DynamicSchema =
    cache.getOrElseUpdate((subject, version), read(subject, version))

  def latest(subject: String): Option[(Int, DynamicSchema)] =
    versions(subject).lastOption.map(version => (version, load(subject, version)))

  private def read(subject: String, version: Int): DynamicSchema = {
    val bytes  = Files.readAllBytes(root.resolve(subject).resolve(s"v$version.avro"))
    val schema = codec.decode(bytes).fold(error => throw error, identity)
    if (schema.reflect.isDynamic)
      throw new IllegalStateException(s"$subject v$version was read back as an accept-anything schema")
    schema
  }

  private def jsonView(schema: DynamicSchema): String =
    Json.fromDynamicValue(DynamicSchema.toDynamicValue(schema)).print(WriterConfig.withIndentionStep(2))
}
```

Here is what each part does:

- `Schema[DynamicSchema].derive(AvroFormat)` builds the Avro codec that `SchemaRegistry#register` writes with and `SchemaRegistry#read` reads with.
- `SchemaRegistry#register` compares the new schema with every stored version using `==`, which compares the two shapes field by field.
- `SchemaRegistry#read` throws when a schema reads back as `Dynamic`. Avro never does this, but the check costs two lines and turns a silent accept-anything registry into an error if someone later swaps the format.
- `SchemaRegistry#jsonView` turns the schema into a `DynamicValue` with `DynamicSchema.toDynamicValue` and prints it as JSON.

:::warning
The `.json` file is for people only. In 0.0.56 neither `Schema[DynamicSchema]`'s JSON codec nor `DynamicSchema.fromDynamicValue` reads it back correctly, so the registry never loads it. Load schemas from the `.avro` files.
:::

## Register Version 1

A producer, the service that publishes `BookPublished` events, registers the event's schema when it starts. We create a registry in a temporary directory and register version 1 twice, as a producer does each time it restarts:

```scala
val root: Path               = Files.createTempDirectory("schema-registry")
val registry: SchemaRegistry = new SchemaRegistry(root)
```

The first call stores the schema as version 1. The second call finds the same schema already stored and returns 1 again, so the directory still holds one version:

```scala
println(registry.register("book-published", Schema[v1.BookPublished].toDynamicSchema))
// 1
println(registry.register("book-published", Schema[v1.BookPublished].toDynamicSchema))
// 1
println(registry.versions("book-published"))
// List(1)
println(root.resolve("book-published").toFile.list().toList.sorted)
// List(v1.avro, v1.json)
```

The JSON view shows what the registry stored. We read it back as text, along with the sizes of both files:

```scala
val v1Dir: Path  = root.resolve("book-published")
val view: String = new String(Files.readAllBytes(v1Dir.resolve("v1.json")), UTF_8)
```

The first lines name the type and its owner. On this page the owner starts with the objects that mdoc wraps around the code (`repl`, `MdocSession` and `MdocApp`), followed by `v1`:

```scala
println(view.linesIterator.take(22).mkString("\n"))
// {
//   "Record": {
//     "typeId": {
//       "name": "BookPublished",
//       "owner": [
//         {
//           "type": "Package",
//           "name": "repl"
//         },
//         {
//           "type": "Term",
//           "name": "MdocSession"
//         },
//         {
//           "type": "Type",
//           "name": "MdocApp"
//         },
//         {
//           "type": "Term",
//           "name": "v1"
//         }
//       ],
println(s"v1.json: ${Files.size(v1Dir.resolve("v1.json"))} bytes, v1.avro: ${Files.size(v1Dir.resolve("v1.avro"))} bytes")
// v1.json: 16330 bytes, v1.avro: 3348 bytes
```

Most of that size is type identity: every type in the schema, down to `String` and `BigDecimal`, is stored with its full name, owner and kind, so a reader can tell a `java.lang.String` from any other type called `String`.

:::warning
In 0.0.56, a schema for a recursive type, such as a `Chapter` that holds a `List[Chapter]`, can't be stored: `DynamicSchema.toDynamicValue` and the codecs follow the recursion forever and fail with a `StackOverflowError`. This is a known bug. The non-recursive shapes that events usually have (records, lists, maps, options, sealed traits) store fine.
:::

## Load the Schema Without the Class

A consumer, a service that reads `BookPublished` events, might be a gateway that forwards events and has never compiled `BookPublished`. It loads the shape from the registry and works with it as data.

We open the same directory with a new registry, so nothing comes from the producer's cache, and load version 1:

```scala
val consumerRegistry: SchemaRegistry = new SchemaRegistry(root)
val storedV1: DynamicSchema          = consumerRegistry.load("book-published", 1)
```

The stored schema can tell us its fields without any compiled class. The `describe` helper reads them from `DynamicSchema#reflect`, the shape's tree: `Reflect#asRecord` returns the record node when the shape is a record, and each field has a name and the `TypeId` of its value:

```scala
def describe(schema: DynamicSchema): String =
  schema.reflect.asRecord match {
    case Some(record) =>
      record.fields.map(field => s"${field.name}: ${field.value.typeId}").mkString(s"${schema.typeId.name}(", ", ", ")")
    case None => schema.typeId.toString
  }

println(describe(storedV1))
// BookPublished(isbn: String, title: String, price: BigDecimal)
```

The loaded schema is also equal to the one the producer derived, which is how a service that does have the class can confirm that its class matches a registered version:

```scala
println(storedV1 == Schema[v1.BookPublished].toDynamicSchema)
// true
println(storedV1 == Schema[v2.BookPublished].toDynamicSchema)
// false
```

## Check Incoming Events

`DynamicSchema#check` takes a `DynamicValue` and returns `None` when the value fits the schema, or `Some(SchemaError)` that says what doesn't fit and where. `DynamicSchema#conforms` returns the same answer as a `Boolean`.

The producer turns each event into a `DynamicValue` with `Schema#toDynamicValue` and sends it. Avro serves for the events too: `Schema[DynamicValue].derive(AvroFormat)` writes a `DynamicValue` with every tag and number type intact, so the consumer reads back exactly what the producer sent:

```scala
val eventCodec = Schema[DynamicValue].derive(AvroFormat)

val zionomicon = v1.BookPublished("978-1-00-000001-1", "Zionomicon", BigDecimal("49.99"))
val sent: Array[Byte] = eventCodec.encode(Schema[v1.BookPublished].toDynamicValue(zionomicon))

val received: DynamicValue = eventCodec.decode(sent).fold(error => throw error, identity)
```

The event the producer sent fits version 1:

```scala
println(storedV1.check(received))
// None
println(storedV1.conforms(received))
// true
```

Events from other sources may not fit. We parse four wrong ones as JSON:

```scala
val textPrice   = Json.parseUnsafe("""{"isbn": "978-1-00-000001-1", "title": "Zionomicon", "price": "49.99"}""")
val noTitle     = Json.parseUnsafe("""{"isbn": "978-1-00-000001-1", "price": 49.99}""")
val extraField  = Json.parseUnsafe("""{"isbn": "978-1-00-000001-1", "title": "Zionomicon", "price": 49.99, "pages": 312}""")
val manyMistakes = Json.parseUnsafe("""{"isbn": 9781000000011, "title": 42}""")
```

`Json#toDynamicValue` turns each parsed document into a `DynamicValue`, which we check against version 1:

```scala
println(storedV1.check(textPrice.toDynamicValue))
// Some(zio.blocks.schema.SchemaError: Expected BigDecimal, got String at: .price)
println(storedV1.check(noTitle.toDynamicValue))
// Some(zio.blocks.schema.SchemaError: Missing field 'title' at: .)
println(storedV1.check(extraField.toDynamicValue))
// Some(zio.blocks.schema.SchemaError: Unknown field 'pages' in Record at: .)
println(storedV1.check(manyMistakes.toDynamicValue))
// Some(zio.blocks.schema.SchemaError: Expected String, got Long at: .isbn)
```

Each error names the problem and the path where it occurs: `.price` is the `price` field, and `.` is the record itself. The last event has three mistakes, but `DynamicSchema#check` stops at the first one it finds and reports only that, so a consumer that wants to list every mistake needs another pass after the first one is fixed.

`DynamicSchema#check` compares primitive types exactly, and that matters for JSON. `Json#toDynamicValue` reads a whole number such as `50` as an `Int`, so a JSON price of `50` fails a `BigDecimal` field, even though the typed JSON codec accepts it:

```scala
val wholePrice = Json.parseUnsafe("""{"isbn": "978-1-00-000001-1", "title": "Zionomicon", "price": 50}""")
```

The schema rejects the price, and the codec of the compiled class reads it as `50`:

```scala
println(storedV1.check(wholePrice.toDynamicValue))
// Some(zio.blocks.schema.SchemaError: Expected BigDecimal, got Int at: .price)
println(v1.BookPublished.schema.jsonCodec.decode(wholePrice.print))
// Right(BookPublished(978-1-00-000001-1,Zionomicon,50))
```

:::warning
Check JSON events with `DynamicSchema#check` only when their fields are strings, booleans, `Int`s, lists and records. `Json#toDynamicValue` reads a whole number as an `Int`, or as a `Long` when it is too big for an `Int`, and every other number as a `BigDecimal`. So `Double`, `Float`, `Short` and `Byte` fields always fail, a `Long` field fails for small values, and a `BigDecimal` field fails for whole numbers. An `Option` field fails too: the schema expects a tagged `Some` or `None`, and JSON has a value or `null`. For JSON events with such fields, rebind the schema and decode with its JSON codec, as the next section shows. `DynamicSchema#check` also doesn't run the checks of a newtype built with `Schema#transform`, such as a smart constructor that rejects an invalid ISBN; see [How to Define Validated Newtypes and Collect Every Validation Error](./validated-newtypes.md).
:::

## Rebind to a Scala Type

A consumer that has compiled `BookPublished` wants typed values, not a `DynamicValue`. `DynamicSchema#rebind` turns a stored schema back into a `Schema[A]` by attaching a binding (the functions that build and take apart a value) to each node of the shape. It gets the bindings from a `BindingResolver`, which looks them up by each node's `TypeId`.

We build a resolver from two parts joined with `BindingResolver#++`, which tries the left side first. `BindingResolver.empty.bind(Binding.of[v1.BookPublished])` supplies the binding for the event's record, and `BindingResolver.defaults` supplies the bindings for primitives such as `String` and `BigDecimal`, the `java.time` types, and the standard collections:

```scala
val v1Resolver: BindingResolver =
  BindingResolver.empty.bind(Binding.of[v1.BookPublished]) ++ BindingResolver.defaults

val typedV1: Schema[v1.BookPublished] = storedV1.rebind[v1.BookPublished](v1Resolver)
```

The rebound schema works like a derived one. Its JSON codec decodes events into `v1.BookPublished` values, accepts the whole-number price that `DynamicSchema#check` rejected, and reports a wrong field with its path:

```scala
println(typedV1.jsonCodec.decode(wholePrice.print))
// Right(BookPublished(978-1-00-000001-1,Zionomicon,50))
println(typedV1.jsonCodec.decode(textPrice.print))
// Left(zio.blocks.schema.SchemaError: illegal number at: .price)
```

When the resolver has no binding for a type that the stored schema names, `DynamicSchema#rebind` throws a `RebindException` at once. That is what happens when a consumer's class has been renamed or moved since the schema was registered: the stored schema still names the old type, and the consumer's binding is for a different one. Rebinding version 1 with only `v2.BookPublished` bound fails like this:

```scala
val renamedResolver: BindingResolver =
  BindingResolver.empty.bind(Binding.of[v2.BookPublished]) ++ BindingResolver.defaults
```

The call throws before it returns a schema, and the exception's message names what is missing:

```scala
try storedV1.rebind[v2.BookPublished](renamedResolver)
catch { case e: RebindException => println(e.getMessage) }
// Missing Record binding for type repl.MdocSession.MdocApp.v1.BookPublished at path: .
```

The message names the missing type with its full name, here with the wrapper objects that mdoc puts around this page's code, and the path `.` says that the root of the schema is missing its binding. To keep rebinding working, keep the event class's name and package stable across versions, and change its fields instead.

:::tip
On the JVM, `BindingResolver.reflection` finds the binding for any case class by its name, so `BindingResolver.reflection ++ BindingResolver.defaults` rebinds a record schema without a `Binding.of` line per class. On Scala.js it finds nothing, so bind each record with `Binding.of` there.
:::

## Register Version 2 and Compare It With Version 1

The producer's next release adds `authors` and `edition`. It registers the new shape, and the registry stores it as version 2:

```scala
println(registry.register("book-published", Schema[v2.BookPublished].toDynamicSchema))
// 2
println(registry.versions("book-published"))
// List(1, 2)
```

ZIO Blocks has no built-in compatibility check between two versions, like the backward and forward rules of an Avro registry. A small comparison of the record fields is enough for this event, and it is ours to write. The `compare` function below lists each field that was removed, added or changed type, and marks the changes that break old data, meaning that a reader on the new version can't decode an event written with the old one:

```scala
final case class Change(description: String, breaksOldData: Boolean)

def compare(older: DynamicSchema, newer: DynamicSchema): List[Change] = {
  def fields(schema: DynamicSchema): Map[String, Reflect.Unbound[_]] =
    schema.reflect.asRecord.fold(Map.empty[String, Reflect.Unbound[_]])(_.fields.map(f => f.name -> f.value).toMap)

  val before = fields(older)
  val after  = fields(newer)

  val removed = (before.keySet -- after.keySet).toList.sorted.map { name =>
    Change(s"removed $name", breaksOldData = false)
  }
  val added = (after.keySet -- before.keySet).toList.sorted.map { name =>
    val field  = after(name)
    val filled = field.isOption || field.isCollection
    Change(s"added $name: ${field.typeId}", breaksOldData = !filled)
  }
  val changed = (before.keySet intersect after.keySet).toList.sorted.collect {
    case name if before(name).typeId != after(name).typeId =>
      Change(s"changed $name: ${before(name).typeId} -> ${after(name).typeId}", breaksOldData = true)
  }
  removed ++ added ++ changed
}
```

The rules come from how the JSON codec reads an event that lacks a field. It fills a missing `Option` field with `None` and a missing collection with an empty one, so adding such a field doesn't break old data. Adding any other field does, and so does changing a field's type. Removing a field doesn't, because the codec skips fields it doesn't know. The stored schema of a case class doesn't record a field's default value in 0.0.56, so `compare` can't tell that a field with a default is safe and reports it as breaking.

Comparing the two stored versions gives two additions, neither of which breaks old data:

```scala
val changes: List[Change] = compare(registry.load("book-published", 1), registry.load("book-published", 2))
```

Each change prints on its own line, followed by whether any of them breaks old data:

```scala
changes.foreach(println)
// Change(added authors: List[String],false)
// Change(added edition: Option[Int],false)
println(changes.exists(_.breaksOldData))
// false
```

A consumer on version 2 rebinds the new schema. `v2.BookPublished` has an `Option[Int]` field, and `BindingResolver.defaults` has no binding for `Option`, so the resolver needs three more: the `Option[Int]` variant itself, its `Some` case for an `Int` (`Binding.Record.someInt`), and its `None` case (`Binding.Record.none`). Without them, `DynamicSchema#rebind` fails with `Missing Record binding for type scala.Some at path: .edition<Some>`:

```scala
val v2Resolver: BindingResolver =
  BindingResolver.empty
    .bind(Binding.of[v2.BookPublished])
    .bind(Binding.of[Option[Int]])
    .bind(Binding.Record.someInt)
    .bind(Binding.Record.none) ++ BindingResolver.defaults

val typedV2: Schema[v2.BookPublished] = consumerRegistry.load("book-published", 2).rebind[v2.BookPublished](v2Resolver)
```

The version 2 consumer decodes new events, and also old events written with version 1, as `compare` predicted:

```scala
val fpInScala: String =
  """{"isbn": "978-1-00-000002-8", "title": "Functional Programming in Scala", "price": 44.99,
    | "authors": ["Michael Pilquist", "Rúnar Bjarnason", "Paul Chiusano"], "edition": 2}""".stripMargin
```

We decode the new event, then the two version 1 events from earlier, the one without a title and the one with a whole-number price:

```scala
println(typedV2.jsonCodec.decode(fpInScala))
// Right(BookPublished(978-1-00-000002-8,Functional Programming in Scala,44.99,List(Michael Pilquist, Rúnar Bjarnason, Paul Chiusano),Some(2)))
println(typedV2.jsonCodec.decode(noTitle.print))
// Left(zio.blocks.schema.SchemaError: missing required field "title" at: .)
println(typedV2.jsonCodec.decode(wholePrice.print))
// Right(BookPublished(978-1-00-000001-1,Zionomicon,50,List(),None))
```

The last line decodes a version 1 event into a version 2 value, with an empty author list and no edition. The middle line fails because that event lacks a title, which no version of the event allows.

:::note
Decode old events with the JSON codec, as above, not with `Schema#fromDynamicValue`. In 0.0.56, `Schema#fromDynamicValue` reports a missing `Option` or collection field as `Missing field` instead of filling it, so it rejects version 1 events that the JSON codec accepts. This is a known bug.
:::

## Putting It Together

The complete example runs the whole life of a subject in one program: the producer registers version 1, a gateway loads it without the class and checks events, a typed consumer rebinds it, and the producer's next release registers version 2, which the registry compares with version 1 before the consumer upgrades:

```scala title="schema-examples/src/main/scala/buildaschemaregistry/CompleteExample.scala"
package buildaschemaregistry

/**
 * How to Build a Schema Registry with DynamicSchema — The Complete Example
 *
 * Runs the whole life of a subject in one program: the producer registers
 * version 1, a gateway loads it without the class and checks events, a typed
 * consumer rebinds it, and the producer's next release registers version 2,
 * which the registry compares with version 1 before the consumer upgrades.
 *
 * Run with: cd schema-examples && sbt "runMain
 * buildaschemaregistry.CompleteExample"
 */
object CompleteExample {

  // The events: two versions of one class, in objects named v1 and v2

  object v1 {
    final case class BookPublished(isbn: String, title: String, price: BigDecimal)

    object BookPublished {
      implicit val schema: Schema[BookPublished] = Schema.derived
    }
  }

  object v2 {
    final case class BookPublished(
      isbn: String,
      title: String,
      price: BigDecimal,
      authors: List[String],
      edition: Option[Int]
    )

    object BookPublished {
      implicit val schema: Schema[BookPublished] = Schema.derived
    }
  }

  // The registry

  final class SchemaRegistry(root: Path) {
    private val codec = Schema[DynamicSchema].derive(AvroFormat)
    private val cache = scala.collection.mutable.Map.empty[(String, Int), DynamicSchema]

    def versions(subject: String): List[Int] = {
      val names = Option(root.resolve(subject).toFile.list()).fold(List.empty[String])(_.toList)
      names.collect { case name if name.matches("v[0-9]+\\.avro") => name.drop(1).dropRight(5).toInt }.sorted
    }

    def register(subject: String, schema: DynamicSchema): Int =
      versions(subject).find(version => load(subject, version) == schema) match {
        case Some(existing) => existing
        case None           =>
          val version = versions(subject).lastOption.fold(1)(_ + 1)
          val dir     = Files.createDirectories(root.resolve(subject))
          Files.write(dir.resolve(s"v$version.avro"), codec.encode(schema))
          Files.write(dir.resolve(s"v$version.json"), jsonView(schema).getBytes(UTF_8))
          cache((subject, version)) = schema
          version
      }

    def load(subject: String, version: Int): DynamicSchema =
      cache.getOrElseUpdate((subject, version), read(subject, version))

    def latest(subject: String): Option[(Int, DynamicSchema)] =
      versions(subject).lastOption.map(version => (version, load(subject, version)))

    private def read(subject: String, version: Int): DynamicSchema = {
      val bytes  = Files.readAllBytes(root.resolve(subject).resolve(s"v$version.avro"))
      val schema = codec.decode(bytes).fold(error => throw error, identity)
      if (schema.reflect.isDynamic)
        throw new IllegalStateException(s"$subject v$version was read back as an accept-anything schema")
      schema
    }

    private def jsonView(schema: DynamicSchema): String =
      Json.fromDynamicValue(DynamicSchema.toDynamicValue(schema)).print(WriterConfig.withIndentionStep(2))
  }

  def describe(schema: DynamicSchema): String =
    schema.reflect.asRecord match {
      case Some(record) =>
        record.fields
          .map(field => s"${field.name}: ${field.value.typeId}")
          .mkString(s"${schema.typeId.name}(", ", ", ")")
      case None => schema.typeId.toString
    }

  // The comparison

  final case class Change(description: String, breaksOldData: Boolean)

  def compare(older: DynamicSchema, newer: DynamicSchema): List[Change] = {
    def fields(schema: DynamicSchema): Map[String, Reflect.Unbound[_]] =
      schema.reflect.asRecord.fold(Map.empty[String, Reflect.Unbound[_]])(_.fields.map(f => f.name -> f.value).toMap)

    val before = fields(older)
    val after  = fields(newer)

    val removed = (before.keySet -- after.keySet).toList.sorted.map { name =>
      Change(s"removed $name", breaksOldData = false)
    }
    val added = (after.keySet -- before.keySet).toList.sorted.map { name =>
      val field  = after(name)
      val filled = field.isOption || field.isCollection
      Change(s"added $name: ${field.typeId}", breaksOldData = !filled)
    }
    val changed = (before.keySet intersect after.keySet).toList.sorted.collect {
      case name if before(name).typeId != after(name).typeId =>
        Change(s"changed $name: ${before(name).typeId} -> ${after(name).typeId}", breaksOldData = true)
    }
    removed ++ added ++ changed
  }

  def main(args: Array[String]): Unit = {
    val subject = "book-published"
    val root    = Files.createTempDirectory("schema-registry")

    // 1. The producer registers version 1, twice, as it does on every restart
    val producerRegistry = new SchemaRegistry(root)
    println(producerRegistry.register(subject, Schema[v1.BookPublished].toDynamicSchema))
    println(producerRegistry.register(subject, Schema[v1.BookPublished].toDynamicSchema))
    println(producerRegistry.versions(subject))
    println(root.resolve(subject).toFile.list().toList.sorted)

    // 2. A gateway without the class loads version 1 in a fresh registry
    val gatewayRegistry = new SchemaRegistry(root)
    val storedV1        = gatewayRegistry.load(subject, 1)
    println(describe(storedV1))
    println(storedV1 == Schema[v1.BookPublished].toDynamicSchema)

    // 3. The gateway checks one good event and one bad one
    val eventCodec = Schema[DynamicValue].derive(AvroFormat)
    val zionomicon = v1.BookPublished("978-1-00-000001-1", "Zionomicon", BigDecimal("49.99"))
    val sent       = eventCodec.encode(Schema[v1.BookPublished].toDynamicValue(zionomicon))
    val received   = eventCodec.decode(sent).fold(error => throw error, identity)
    println(storedV1.check(received))

    val textPrice  = Json.parseUnsafe("""{"isbn": "978-1-00-000001-1", "title": "Zionomicon", "price": "49.99"}""")
    val noTitle    = Json.parseUnsafe("""{"isbn": "978-1-00-000001-1", "price": 49.99}""")
    val wholePrice = Json.parseUnsafe("""{"isbn": "978-1-00-000001-1", "title": "Zionomicon", "price": 50}""")
    println(storedV1.check(textPrice.toDynamicValue))
    println(storedV1.check(wholePrice.toDynamicValue))

    // 4. A typed consumer rebinds version 1 and decodes JSON events
    val v1Resolver = BindingResolver.empty.bind(Binding.of[v1.BookPublished]) ++ BindingResolver.defaults
    val typedV1    = storedV1.rebind[v1.BookPublished](v1Resolver)
    println(typedV1.jsonCodec.decode(wholePrice.print))
    println(typedV1.jsonCodec.decode(textPrice.print))

    // 5. The producer's next release registers version 2
    println(producerRegistry.register(subject, Schema[v2.BookPublished].toDynamicSchema))
    println(producerRegistry.versions(subject))

    // 6. The registry compares version 1 with version 2
    val changes = compare(producerRegistry.load(subject, 1), producerRegistry.load(subject, 2))
    changes.foreach(println)
    println(s"breaks old data: ${changes.exists(_.breaksOldData)}")

    // 7. A version 2 consumer rebinds with Option bindings and decodes new and old events
    val v2Resolver =
      BindingResolver.empty
        .bind(Binding.of[v2.BookPublished])
        .bind(Binding.of[Option[Int]])
        .bind(Binding.Record.someInt)
        .bind(Binding.Record.none) ++ BindingResolver.defaults
    val typedV2 = gatewayRegistry.load(subject, 2).rebind[v2.BookPublished](v2Resolver)

    val fpInScala =
      """{"isbn": "978-1-00-000002-8", "title": "Functional Programming in Scala", "price": 44.99,
        | "authors": ["Michael Pilquist", "Rúnar Bjarnason", "Paul Chiusano"], "edition": 2}""".stripMargin
    val zionomiconV1 =
      """{"isbn": "978-1-00-000001-1", "title": "Zionomicon", "price": 49.99}"""

    println(typedV2.jsonCodec.decode(fpInScala))
    println(typedV2.jsonCodec.decode(zionomiconV1))
  }
}
```

## Running the Examples

All examples in this guide have runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

The step examples share two files that sit next to them: `BookEvents.scala` holds both versions of the event, and `SchemaRegistry.scala` holds the registry. The complete example has its own copies and doesn't need them.

<details>
  <summary>schema-examples/src/main/scala/buildaschemaregistry/BookEvents.scala</summary>

```scala title="schema-examples/src/main/scala/buildaschemaregistry/BookEvents.scala" showLineNumbers
package buildaschemaregistry

/**
 * How to Build a Schema Registry with DynamicSchema — The Events
 *
 * Both versions of the `BookPublished` event, shared by the step examples. In a
 * real code base there is one `BookPublished` class whose fields change between
 * releases; here the two versions live in objects named `v1` and `v2`.
 */
object v1 {
  final case class BookPublished(isbn: String, title: String, price: BigDecimal)

  object BookPublished {
    implicit val schema: Schema[BookPublished] = Schema.derived
  }
}

object v2 {
  final case class BookPublished(
    isbn: String,
    title: String,
    price: BigDecimal,
    authors: List[String],
    edition: Option[Int]
  )

  object BookPublished {
    implicit val schema: Schema[BookPublished] = Schema.derived
  }
}
```

</details>

<details>
  <summary>schema-examples/src/main/scala/buildaschemaregistry/SchemaRegistry.scala</summary>

```scala title="schema-examples/src/main/scala/buildaschemaregistry/SchemaRegistry.scala" showLineNumbers
package buildaschemaregistry

/**
 * How to Build a Schema Registry with DynamicSchema — The Registry
 *
 * A directory-backed registry that stores each version of a subject's schema as
 * Avro bytes, with an indented JSON view next to it for people to read. Shared
 * by the step examples.
 */
final class SchemaRegistry(root: Path) {
  private val codec = Schema[DynamicSchema].derive(AvroFormat)
  private val cache = scala.collection.mutable.Map.empty[(String, Int), DynamicSchema]

  def versions(subject: String): List[Int] = {
    val names = Option(root.resolve(subject).toFile.list()).fold(List.empty[String])(_.toList)
    names.collect { case name if name.matches("v[0-9]+\\.avro") => name.drop(1).dropRight(5).toInt }.sorted
  }

  def register(subject: String, schema: DynamicSchema): Int =
    versions(subject).find(version => load(subject, version) == schema) match {
      case Some(existing) => existing
      case None           =>
        val version = versions(subject).lastOption.fold(1)(_ + 1)
        val dir     = Files.createDirectories(root.resolve(subject))
        Files.write(dir.resolve(s"v$version.avro"), codec.encode(schema))
        Files.write(dir.resolve(s"v$version.json"), jsonView(schema).getBytes(UTF_8))
        cache((subject, version)) = schema
        version
    }

  def load(subject: String, version: Int): DynamicSchema =
    cache.getOrElseUpdate((subject, version), read(subject, version))

  def latest(subject: String): Option[(Int, DynamicSchema)] =
    versions(subject).lastOption.map(version => (version, load(subject, version)))

  private def read(subject: String, version: Int): DynamicSchema = {
    val bytes  = Files.readAllBytes(root.resolve(subject).resolve(s"v$version.avro"))
    val schema = codec.decode(bytes).fold(error => throw error, identity)
    if (schema.reflect.isDynamic)
      throw new IllegalStateException(s"$subject v$version was read back as an accept-anything schema")
    schema
  }

  private def jsonView(schema: DynamicSchema): String =
    Json.fromDynamicValue(DynamicSchema.toDynamicValue(schema)).print(WriterConfig.withIndentionStep(2))
}

object SchemaRegistry {

  /** Reads a stored schema's fields from `DynamicSchema#reflect`. */
  def describe(schema: DynamicSchema): String =
    schema.reflect.asRecord match {
      case Some(record) =>
        record.fields
          .map(field => s"${field.name}: ${field.value.typeId}")
          .mkString(s"${schema.typeId.name}(", ", ", ")")
      case None => schema.typeId.toString
    }
}
```

</details>

### Choose a Storage Format

This example round-trips version 1's schema through JSON and through Avro, and checks a wrong event against each copy.

<details>
  <summary>schema-examples/src/main/scala/buildaschemaregistry/Step1StorageFormatExample.scala</summary>

```scala title="schema-examples/src/main/scala/buildaschemaregistry/Step1StorageFormatExample.scala" showLineNumbers
package buildaschemaregistry

/**
 * How to Build a Schema Registry with DynamicSchema — Choose a Storage Format
 *
 * Round-trips version 1's schema through JSON and through Avro, and checks a
 * wrong event against each copy. The JSON copy reads back as an accept-anything
 * schema; the Avro copy is equal to the original.
 *
 * Run with: cd schema-examples && sbt "runMain
 * buildaschemaregistry.Step1StorageFormatExample"
 */
object Step1StorageFormatExample {
  def main(args: Array[String]): Unit = {
    val v1Schema: DynamicSchema = Schema[v1.BookPublished].toDynamicSchema

    val wrongEvent: DynamicValue =
      Json.parseUnsafe("""{"isbn": 9781000000011, "price": "free", "pages": 300}""").toDynamicValue

    val schemaAsJson = Schema[DynamicSchema].derive(JsonFormat)
    val schemaAsAvro = Schema[DynamicSchema].derive(AvroFormat)

    val fromJson = schemaAsJson.decode(schemaAsJson.encode(v1Schema))
    val fromAvro = schemaAsAvro.decode(schemaAsAvro.encode(v1Schema))

    println(v1Schema.check(wrongEvent))
    println(fromJson.map(_ == v1Schema))
    println(fromJson.map(_.reflect.isDynamic))
    println(fromJson.map(_.check(wrongEvent)))

    println(fromAvro.map(_ == v1Schema))
    println(fromAvro.map(_.check(wrongEvent)))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain buildaschemaregistry.Step1StorageFormatExample"
```

### Register Version 1

This example registers version 1 twice, lists the stored files, and prints the start of the JSON view and the sizes of both files.

<details>
  <summary>schema-examples/src/main/scala/buildaschemaregistry/Step2RegisterExample.scala</summary>

```scala title="schema-examples/src/main/scala/buildaschemaregistry/Step2RegisterExample.scala" showLineNumbers
package buildaschemaregistry

/**
 * How to Build a Schema Registry with DynamicSchema — Register Version 1
 *
 * Registers version 1 twice, as a producer does each time it restarts, lists
 * the stored files, and prints the start of the JSON view and the sizes of both
 * files.
 *
 * Run with: cd schema-examples && sbt "runMain
 * buildaschemaregistry.Step2RegisterExample"
 */
object Step2RegisterExample {
  def main(args: Array[String]): Unit = {
    val root: Path               = Files.createTempDirectory("schema-registry")
    val registry: SchemaRegistry = new SchemaRegistry(root)

    println(registry.register("book-published", Schema[v1.BookPublished].toDynamicSchema))
    println(registry.register("book-published", Schema[v1.BookPublished].toDynamicSchema))
    println(registry.versions("book-published"))
    println(root.resolve("book-published").toFile.list().toList.sorted)

    val v1Dir: Path  = root.resolve("book-published")
    val view: String = new String(Files.readAllBytes(v1Dir.resolve("v1.json")), UTF_8)

    println(view.linesIterator.take(12).mkString("\n"))
    println(
      s"v1.json: ${Files.size(v1Dir.resolve("v1.json"))} bytes, v1.avro: ${Files.size(v1Dir.resolve("v1.avro"))} bytes"
    )
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain buildaschemaregistry.Step2RegisterExample"
```

### Load the Schema Without the Class

This example loads version 1 in a fresh registry, prints its fields, and compares it with the schemas of both event classes.

<details>
  <summary>schema-examples/src/main/scala/buildaschemaregistry/Step3LoadWithoutClassExample.scala</summary>

```scala title="schema-examples/src/main/scala/buildaschemaregistry/Step3LoadWithoutClassExample.scala" showLineNumbers
package buildaschemaregistry

/**
 * How to Build a Schema Registry with DynamicSchema — Load the Schema Without
 * the Class
 *
 * A producer registers version 1; a second registry on the same directory loads
 * it, prints its fields, and compares it with the schemas of both event
 * classes.
 *
 * Run with: cd schema-examples && sbt "runMain
 * buildaschemaregistry.Step3LoadWithoutClassExample"
 */
object Step3LoadWithoutClassExample {

  def main(args: Array[String]): Unit = {
    val root: Path               = Files.createTempDirectory("schema-registry")
    val registry: SchemaRegistry = new SchemaRegistry(root)
    registry.register("book-published", Schema[v1.BookPublished].toDynamicSchema)

    val consumerRegistry: SchemaRegistry = new SchemaRegistry(root)
    val storedV1: DynamicSchema          = consumerRegistry.load("book-published", 1)

    println(describe(storedV1))

    println(storedV1 == Schema[v1.BookPublished].toDynamicSchema)
    println(storedV1 == Schema[v2.BookPublished].toDynamicSchema)
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain buildaschemaregistry.Step3LoadWithoutClassExample"
```

### Check Incoming Events

This example sends an event as Avro bytes, checks it against version 1, and checks four wrong events and one JSON event with a whole-number price.

<details>
  <summary>schema-examples/src/main/scala/buildaschemaregistry/Step4CheckEventsExample.scala</summary>

```scala title="schema-examples/src/main/scala/buildaschemaregistry/Step4CheckEventsExample.scala" showLineNumbers
package buildaschemaregistry

/**
 * How to Build a Schema Registry with DynamicSchema — Check Incoming Events
 *
 * Sends an event as Avro bytes and checks it against version 1, then checks
 * four wrong events and one JSON event with a whole-number price.
 *
 * Run with: cd schema-examples && sbt "runMain
 * buildaschemaregistry.Step4CheckEventsExample"
 */
object Step4CheckEventsExample {
  def main(args: Array[String]): Unit = {
    val root: Path               = Files.createTempDirectory("schema-registry")
    val registry: SchemaRegistry = new SchemaRegistry(root)
    registry.register("book-published", Schema[v1.BookPublished].toDynamicSchema)

    val consumerRegistry: SchemaRegistry = new SchemaRegistry(root)
    val storedV1: DynamicSchema          = consumerRegistry.load("book-published", 1)

    val eventCodec = Schema[DynamicValue].derive(AvroFormat)

    val zionomicon        = v1.BookPublished("978-1-00-000001-1", "Zionomicon", BigDecimal("49.99"))
    val sent: Array[Byte] = eventCodec.encode(Schema[v1.BookPublished].toDynamicValue(zionomicon))

    val received: DynamicValue = eventCodec.decode(sent).fold(error => throw error, identity)

    println(storedV1.check(received))
    println(storedV1.conforms(received))

    val textPrice  = Json.parseUnsafe("""{"isbn": "978-1-00-000001-1", "title": "Zionomicon", "price": "49.99"}""")
    val noTitle    = Json.parseUnsafe("""{"isbn": "978-1-00-000001-1", "price": 49.99}""")
    val extraField =
      Json.parseUnsafe("""{"isbn": "978-1-00-000001-1", "title": "Zionomicon", "price": 49.99, "pages": 312}""")
    val manyMistakes = Json.parseUnsafe("""{"isbn": 9781000000011, "title": 42}""")

    println(storedV1.check(textPrice.toDynamicValue))
    println(storedV1.check(noTitle.toDynamicValue))
    println(storedV1.check(extraField.toDynamicValue))
    println(storedV1.check(manyMistakes.toDynamicValue))

    val wholePrice = Json.parseUnsafe("""{"isbn": "978-1-00-000001-1", "title": "Zionomicon", "price": 50}""")

    println(storedV1.check(wholePrice.toDynamicValue))
    println(v1.BookPublished.schema.jsonCodec.decode(wholePrice.print))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain buildaschemaregistry.Step4CheckEventsExample"
```

### Rebind to a Scala Type

This example rebinds version 1 to `v1.BookPublished`, decodes JSON events with the rebound schema, and shows the `RebindException` for a type the resolver doesn't know.

<details>
  <summary>schema-examples/src/main/scala/buildaschemaregistry/Step5RebindExample.scala</summary>

```scala title="schema-examples/src/main/scala/buildaschemaregistry/Step5RebindExample.scala" showLineNumbers
package buildaschemaregistry

/**
 * How to Build a Schema Registry with DynamicSchema — Rebind to a Scala Type
 *
 * Rebinds version 1 to `v1.BookPublished`, decodes JSON events with the rebound
 * schema, and shows the `RebindException` for a type the resolver doesn't know.
 *
 * Run with: cd schema-examples && sbt "runMain
 * buildaschemaregistry.Step5RebindExample"
 */
object Step5RebindExample {
  def main(args: Array[String]): Unit = {
    val root: Path               = Files.createTempDirectory("schema-registry")
    val registry: SchemaRegistry = new SchemaRegistry(root)
    registry.register("book-published", Schema[v1.BookPublished].toDynamicSchema)

    val consumerRegistry: SchemaRegistry = new SchemaRegistry(root)
    val storedV1: DynamicSchema          = consumerRegistry.load("book-published", 1)

    val textPrice  = Json.parseUnsafe("""{"isbn": "978-1-00-000001-1", "title": "Zionomicon", "price": "49.99"}""")
    val wholePrice = Json.parseUnsafe("""{"isbn": "978-1-00-000001-1", "title": "Zionomicon", "price": 50}""")

    val v1Resolver: BindingResolver =
      BindingResolver.empty.bind(Binding.of[v1.BookPublished]) ++ BindingResolver.defaults

    val typedV1: Schema[v1.BookPublished] = storedV1.rebind[v1.BookPublished](v1Resolver)

    println(typedV1.jsonCodec.decode(wholePrice.print))
    println(typedV1.jsonCodec.decode(textPrice.print))

    val renamedResolver: BindingResolver =
      BindingResolver.empty.bind(Binding.of[v2.BookPublished]) ++ BindingResolver.defaults

    try storedV1.rebind[v2.BookPublished](renamedResolver)
    catch { case e: RebindException => println(e.getMessage) }
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain buildaschemaregistry.Step5RebindExample"
```

### Register Version 2 and Compare It With Version 1

This example registers version 2, compares it with version 1, rebinds it with the `Option` bindings, and decodes a new event and an old one.

<details>
  <summary>schema-examples/src/main/scala/buildaschemaregistry/Step6CompareVersionsExample.scala</summary>

```scala title="schema-examples/src/main/scala/buildaschemaregistry/Step6CompareVersionsExample.scala" showLineNumbers
package buildaschemaregistry

/**
 * How to Build a Schema Registry with DynamicSchema — Register Version 2 and
 * Compare It With Version 1
 *
 * Registers version 2, compares it with version 1, rebinds it with the `Option`
 * bindings, and decodes a new event and old ones.
 *
 * Run with: cd schema-examples && sbt "runMain
 * buildaschemaregistry.Step6CompareVersionsExample"
 */
object Step6CompareVersionsExample {
  final case class Change(description: String, breaksOldData: Boolean)

  def compare(older: DynamicSchema, newer: DynamicSchema): List[Change] = {
    def fields(schema: DynamicSchema): Map[String, Reflect.Unbound[_]] =
      schema.reflect.asRecord.fold(Map.empty[String, Reflect.Unbound[_]])(_.fields.map(f => f.name -> f.value).toMap)

    val before = fields(older)
    val after  = fields(newer)

    val removed = (before.keySet -- after.keySet).toList.sorted.map { name =>
      Change(s"removed $name", breaksOldData = false)
    }
    val added = (after.keySet -- before.keySet).toList.sorted.map { name =>
      val field  = after(name)
      val filled = field.isOption || field.isCollection
      Change(s"added $name: ${field.typeId}", breaksOldData = !filled)
    }
    val changed = (before.keySet intersect after.keySet).toList.sorted.collect {
      case name if before(name).typeId != after(name).typeId =>
        Change(s"changed $name: ${before(name).typeId} -> ${after(name).typeId}", breaksOldData = true)
    }
    removed ++ added ++ changed
  }

  def main(args: Array[String]): Unit = {
    val root: Path               = Files.createTempDirectory("schema-registry")
    val registry: SchemaRegistry = new SchemaRegistry(root)
    registry.register("book-published", Schema[v1.BookPublished].toDynamicSchema)

    val consumerRegistry: SchemaRegistry = new SchemaRegistry(root)

    println(registry.register("book-published", Schema[v2.BookPublished].toDynamicSchema))
    println(registry.versions("book-published"))

    val changes = compare(registry.load("book-published", 1), registry.load("book-published", 2))

    changes.foreach(println)
    println(changes.exists(_.breaksOldData))

    val v2Resolver: BindingResolver =
      BindingResolver.empty
        .bind(Binding.of[v2.BookPublished])
        .bind(Binding.of[Option[Int]])
        .bind(Binding.Record.someInt)
        .bind(Binding.Record.none) ++ BindingResolver.defaults

    val typedV2: Schema[v2.BookPublished] =
      consumerRegistry.load("book-published", 2).rebind[v2.BookPublished](v2Resolver)

    val noTitle    = Json.parseUnsafe("""{"isbn": "978-1-00-000001-1", "price": 49.99}""")
    val wholePrice = Json.parseUnsafe("""{"isbn": "978-1-00-000001-1", "title": "Zionomicon", "price": 50}""")

    val fpInScala =
      """{"isbn": "978-1-00-000002-8", "title": "Functional Programming in Scala", "price": 44.99,
        | "authors": ["Michael Pilquist", "Rúnar Bjarnason", "Paul Chiusano"], "edition": 2}""".stripMargin

    println(typedV2.jsonCodec.decode(fpInScala))
    println(typedV2.jsonCodec.decode(noTitle.print))
    println(typedV2.jsonCodec.decode(wholePrice.print))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain buildaschemaregistry.Step6CompareVersionsExample"
```

### The Complete Example

This example runs the whole guide in one program:

```bash
sbt "runMain buildaschemaregistry.CompleteExample"
```

### Compile All Examples

To compile every example without running one, run this from the `schema-examples` directory:

```bash
sbt compile
```

## Going Further

- The [DynamicSchema reference page](../reference/schema/dynamic-schema.md) covers the rest of the type: reading docs, modifiers and examples from a stored schema, navigating it with a `DynamicOptic`, and `DynamicSchema#toSchema`.
- The [BindingResolver reference page](../reference/schema/binding-resolver.md) shows how to bind sequence and map types, wrappers and variants, and how resolvers combine.
- A JSON Schema document describes the same kind of shape, but for JSON only and for any tool that speaks JSON Schema. [How to Generate JSON Schema from a Type and Validate Payloads Against It](./generate-json-schema-and-validate-payloads.md) shows that route. A `DynamicSchema` registry keeps the full ZIO Blocks shape, including sealed traits and type names, and turns back into a typed `Schema` that works with every format.
- To send the same events as JSON, Avro, MessagePack and more from one schema, see [One Type, Many Formats](./one-type-many-formats.md).
- To work with JSON events whose shape you don't model at all, see [How to Read and Modify JSON Without Case Classes](./json-without-case-classes.md).
- To record what changed between two versions of a value, rather than of its schema, see [How to Compute, Store and Replay Changes with Diff and Patch](./diff-and-patch.md).

---

## Compile-Time Resource Safety with Scope

Welcome to ZIO Blocks Scope—a library that makes resource management safe, composable, and verifiable at compile time. If you've ever struggled with `try/finally` chains, wondered when to close a database connection, or worried about resources outliving their owners, this tutorial is for you. You don't need any prior experience with Scope or advanced type system concepts to follow along.

## Learning Objectives

By the end of this tutorial, you will be able to:

- **Allocate resources safely** and trust the compiler to prevent use-after-close bugs
- **Understand the `$[A]` type and `scoped { }` block** — the two core mechanisms that make resource safety compile-time verifiable
- **Manage complex resource lifetimes** including cleanup with `defer`, nested scopes with `lower`, and reference counting with `Resource.shared`
- **Wire applications** using `Wire` and `Resource.from` for compile-time verified dependency injection

We'll learn through step-by-step examples that build from simple (allocating a single resource) to advanced patterns (nested scopes, shared resources, and dependency injection). Each section builds on the previous one, so we recommend reading from top to bottom.

## 1. The Problem: Why Resource Management Is Hard

Managing resources safely is deceptively difficult. When you open a database connection, read a file, or create a network socket, you must eventually close it—but only once, and only after you're done using it. Forget to close it, and you leak a resource. Close it too early, and you get a crash. Close it twice, and you get an error.

Nested resources make this worse. If you're reading a config file and then opening a database connection, you need nested `try/finally` blocks. If the inner resource throws an exception, the outer one may not close. Callbacks and closures can capture resources that outlive their intended scope, creating subtle use-after-free bugs.

Consider a typical `try/finally` pattern in Scala:

```scala
trait Connection extends AutoCloseable {
  def createStatement(): Statement
}
trait Statement extends AutoCloseable {
  def executeQuery(sql: String): ResultSet
}
trait ResultSet
def openConnection(): Connection = ???
def process(result: ResultSet): Unit = ???
def handleError(e: Exception): Unit = ???
val sql = ""

try {
  val connection = openConnection()
  try {
    val statement = connection.createStatement()
    try {
      val result = statement.executeQuery(sql)
      process(result)
    } finally {
      statement.close()
    }
  } finally {
    connection.close()
  }
} catch {
  case e: Exception => handleError(e)
}
```

This is hard to read, easy to get wrong, and doesn't compose—every additional resource adds another level of nesting. If an exception happens during cleanup, subsequent finalizers may not run. And if you pass a resource to another function, there's no compile-time guarantee that function won't use it after your scope ends.

Scope eliminates these problems by making resource ownership explicit and enforcing it at compile time.

## 2. Your First Scope

Let's start with the simplest possible example: allocating a single resource, using it, and letting it close automatically.

Scope builds on the concept of `Scope.global`—the root scope that outlives your entire program. You enter a scoped region using `scoped { }`, and inside that block, you can allocate resources. When the block exits, all resources close in reverse order (last allocated, first closed).

Here's a database connection that prints messages when opening and closing:

```scala

class Database extends AutoCloseable {
  def connect(): Unit = println("Database: connecting")
  def query(sql: String): String = s"Results of: $sql"
  override def close(): Unit = println("Database: closing")
}

Scope.global.scoped { scope =>

  val db: $[Database] = allocate(Resource {
    val database = new Database()
    database.connect()
    database
  })

  $(db) { database =>
    val result = database.query("SELECT * FROM users")
    println(s"Query result: $result")
  }
}
```

Let's break down what happens:

- `Scope.global.scoped { scope => ... }` — Creates a scoped region. When the block exits, all allocated resources are closed.
- `import scope._` — Imports scope operations: `$`, `allocate`, and `defer`.
- `allocate(Resource { ... })` — Allocates a resource. Since `Database` extends `AutoCloseable`, its `close()` method is automatically registered as a finalizer.
- `$[Database]` — A scoped value of type `Database`. It can only be used within the scope where it was allocated.
- `$(db) { database => ... }` — Unwraps the scoped value and passes it to the block. This is the only way to access a resource.

When the scope exits, `database.close()` runs automatically, printing `"Database: closing"`.

With multiple resources, finalizers run in LIFO order (last allocated, first closed):

```scala

class Connection extends AutoCloseable {
  def name: String = this.getClass.getSimpleName
  override def close(): Unit = println(s"$name: closing")
}

Scope.global.scoped { scope =>

  val conn1 = allocate(Resource(new Connection() { override def name = "Connection-1" }))
  val conn2 = allocate(Resource(new Connection() { override def name = "Connection-2" }))
  val conn3 = allocate(Resource(new Connection() { override def name = "Connection-3" }))

  println("All connections allocated")
  println("Exiting scope - connections will close in reverse order (3, 2, 1)")
}
```

:::note
The `$[A]` type is central to Scope's compile-time safety: each scope instance has a unique `$` type that cannot be mixed with other scopes. Resources allocated in one scope literally cannot be used in another, even at the type level. This prevents entire classes of resource-lifetime bugs at compile time.
:::

## 3. The `$[A]` Type and the `$` Operator

To understand Scope, you need to understand `$[A]`. It is a marker type that says "this value is owned by a specific scope." At runtime, `$[A]` is erased to `A` (zero overhead), but at compile time, it enforces a fundamental rule: **you can only use a resource via the operator defined by the scope it belongs to**.

Every scope instance has its own unique `$` type. Two different scopes have incompatible `$` types, so you cannot accidentally mix them at the type level. For example, if you allocate a resource in an outer scope and try to use it directly in an inner scope without `lower`, you get a compile error because the types are incompatible.

To use a parent-scoped resource in a child scope safely, you must use the `lower` operator, which we'll cover in Section 7.

To use a resource, apply the `$(value)` operator (it's a macro) with a single-argument block. The parameter must be used as the receiver of all operations:

```scala

class Logger extends AutoCloseable {
  def log(msg: String): Unit = println(msg)
  override def close(): Unit = ()
}

Scope.global.scoped { scope =>

  val logger = allocate(Resource(new Logger()))

  // Correct: parameter used as receiver
  $(logger) { log =>
    log.log("Message 1")
    log.log("Message 2")
  }
}
```

The following patterns will not compile:
- `$(logger) { log => logger.log("X") }` — cannot use `logger` outside the `$()` operator
- `$(logger) { log => val x = log; x }` — result is `$[Logger]`, which is not `Unscoped`
- `$(logger) { log => (log, "data") }` — tuples containing `$[Logger]` are not `Unscoped`

The `$` operator automatically unwraps the result if it is an `Unscoped[B]` type. We'll cover `Unscoped` in detail in Section 5, but for now, know that primitives like `Int`, `String`, and `Unit` are always `Unscoped`:

```scala

class Calculator extends AutoCloseable {
  def add(a: Int, b: Int): Int = a + b
  override def close(): Unit = ()
}

Scope.global.scoped { scope =>

  val calc = allocate(Resource(new Calculator()))

  // Result is Int, which is Unscoped, so it unwraps automatically
  val sum = $(calc)(_.add(3, 4))
  assert(sum == 7)
}
```

:::note
The `$` operator is not a regular method—it's a compile-time macro that inspects what you do with its parameter. This macro enforcement is what makes the rule "parameter must be receiver" checkable at compile time, not at runtime.
:::

## 4. Resources: Describing Acquisition and Cleanup

A `Resource[A]` is a lazy description of how to acquire a value and register any cleanup needed when the scope closes. Creating a resource does not acquire it—that only happens when you pass it to `scope.allocate()`.

There are several ways to construct a `Resource`:

**`Resource(value: => A)`** — The simplest form. Wraps a by-name value. If the value is `AutoCloseable`, its `close()` method is automatically registered:

```scala

class Connection extends AutoCloseable {
  override def close(): Unit = println("Connection closed")
}

Scope.global.scoped { scope =>

  // By-name: creates connection on demand, closes automatically
  val conn = allocate(Resource(new Connection()))

  $(conn) { c =>
    println("Using connection")
  }
}
```

**`Resource.acquireRelease(acquire)(release)`** — Explicit lifecycle control. Useful when cleanup is not a simple method call:

```scala

case class Connection(id: Int) {
  def query(sql: String): String = s"[$id] $sql"
}

Scope.global.scoped { scope =>

  // Explicit acquire and release
  val conn = allocate(Resource.acquireRelease {
    println("Opening connection...")
    Connection(42)
  } { c =>
    println(s"Closing connection $c")
  })

  $(conn) { c =>
    println(c.query("SELECT 1"))
  }
}
```

**`Resource.fromAutoCloseable(thunk)`** — Explicit wrapper for `AutoCloseable` subtypes. Type-safe and clear:

```scala

Scope.global.scoped { scope =>

  val file = allocate(Resource.fromAutoCloseable(new FileInputStream("/etc/hostname")))

  $(file) { f =>
    val bytes = Array.ofDim[Byte](100)
    val n = f.read(bytes)
    println(s"Read $n bytes")
  }
}
```

Resources compose: you can transform them with `map`, combine them with `zip`, or sequence them with `flatMap`:

```scala

case class Database(url: String) extends AutoCloseable {
  def getConnection(name: String): Connection =
    Connection(s"$url/$name")
  override def close(): Unit = println(s"Database closed: $url")
}

case class Connection(name: String) extends AutoCloseable {
  def query(sql: String): String = s"[$name] $sql"
  override def close(): Unit = println(s"Connection closed: $name")
}

Scope.global.scoped { scope =>

  val dbResource = Resource(new Database("localhost:5432"))

  // flatMap sequences: database opens first, then connection opens from it
  val connResource = dbResource.flatMap { db =>
    Resource(db.getConnection("myapp"))
  }

  val conn = allocate(connResource)

  $(conn) { c =>
    println(c.query("SELECT 1"))
  }
}
```

When you use `flatMap` to open a connection from an already-open database, the database stays open until the scope closes, ensuring the connection is always valid.

## 5. Returning Values: The `Unscoped[A]` Typeclass

When you exit a `scoped { }` block, the scope closes and all resources are finalized. But what can you return from a `scoped` block? A scoped value `$[A]` cannot escape—it would be used after the scope closes. That's where `Unscoped[A]` comes in.

`Unscoped[A]` is a typeclass that marks types as safe to return from a scoped block. It means "this type contains no scope-bound resources; it is pure data." The type system only allows returning a value if it has an `Unscoped` instance:

```scala

case class Config(host: String, port: Int)

Scope.global.scoped { scope =>

  // Config is a case class: on Scala 3 it has Unscoped automatically
  Config("localhost", 5432)
}
```

Built-in `Unscoped` instances include primitives (`Int`, `String`, `Boolean`), collections (`List[A]`, `Map[K, V]`), and common library types (`UUID`, `java.time.LocalDate`). On Scala 3, a case class whose fields are all `Unscoped` gets an `Unscoped` instance automatically:

```scala

case class Result(count: Int, message: String)
case class ServerConfig(host: String, port: Int, timeout: Long)

Scope.global.scoped { scope =>

  // Both can be returned—they have implicit Unscoped instances
  Result(42, "success") -> ServerConfig("0.0.0.0", 8080, 30000)
}
```

On Scala 2, a custom class needs an `Unscoped` instance before you can return it from `scoped`: derive one with `Unscoped.derived` or write it by hand. On Scala 3 you can also say so in the declaration with `derives`:

```scala

case class CustomData(x: Int, y: String) derives Unscoped

Scope.global.scoped { scope =>

  CustomData(10, "hello")
}
```

Alternatively, provide an instance explicitly using a `given`:

```scala

case class CustomData(x: Int, y: String)

given Unscoped[CustomData] = Unscoped.derived

Scope.global.scoped { scope =>

  CustomData(10, "hello")
}
```

If you try to return a scoped value without an `Unscoped` instance, you get a compile error:

```scala

class Connection extends AutoCloseable {
  override def close(): Unit = ()
}

// This does not compile because Connection has no Unscoped instance:
// val conn = Scope.global.scoped { scope =>
//   import scope._
//   allocate(Resource(new Connection()))
// }
```

This compile-time barrier prevents entire classes of resource-lifetime bugs—you cannot accidentally return a resource reference from a scoped block.

:::warning[Automatic derivation can let a closed resource escape]
A known bug in 0.0.56: the Scala 3 automatic derivation does not check whether a case class extends `AutoCloseable`. A class such as `case class Conn(url: String) extends AutoCloseable` is `Unscoped`, so `Resource.fromAutoCloseable(Conn("x")).use(c => c)` compiles and returns a closed resource. Return computed values only. See [`Unscoped`](../reference/resource-management/unscoped.md).
:::

## 6. Finalizers and Error Handling

Sometimes you need to register cleanup that is not a simple resource `close()`. The `defer` operator lets you register arbitrary cleanup actions:

```scala

case class Transaction(id: Int) {
  def begin(): Unit = println(s"Transaction $id: begin")
  def commit(): Unit = println(s"Transaction $id: commit")
  def rollback(): Unit = println(s"Transaction $id: rollback")
}

Scope.global.scoped { scope =>

  val txn = Transaction(1)
  txn.begin()

  // Register rollback as cleanup
  scope.defer {
    txn.rollback()
  }

  // The rollback still runs when the scope exits, even after commit
  txn.commit()
}
```

`scope.defer()` returns a `DeferHandle` that lets you cancel the finalizer before it runs:

```scala

case class Transaction(id: Int) {
  def begin(): Unit = println(s"Transaction $id: begin")
  def commit(): Unit = println(s"Transaction $id: commit")
  def rollback(): Unit = println(s"Transaction $id: rollback")
}

Scope.global.scoped { scope =>

  val txn = Transaction(1)
  txn.begin()

  // Register rollback, but keep the handle so we can cancel it
  val rollbackHandle = scope.defer {
    txn.rollback()
  }

  // On success, cancel the rollback finalizer
  txn.commit()
  rollbackHandle.cancel()

  println("Scope exiting - rollback will NOT run because we cancelled it")
}
```

Finalizers run in **LIFO order** (last registered, first executed) and are guaranteed to run even if the scoped block throws an exception. If multiple finalizers throw, they are collected:

```scala

Scope.global.scoped { scope =>

  scope.defer { println("Finalizer 1") }
  scope.defer { println("Finalizer 2") }
  scope.defer { println("Finalizer 3") }

  println("Block executing")
}
```

When this code runs, the finalizers execute in reverse order of registration:

```
Block executing
Finalizer 3
Finalizer 2
Finalizer 1
```

:::note
The `DeferHandle.cancel()` operation is O(1)—it marks the finalizer as cancelled without traversing the entire registry. This makes it safe to use in performance-sensitive code, like transaction commits in tight loops.
:::

## 7. Nested Scopes and `lower`

Scopes form a tree: each scope can create child scopes via `scope.scoped { }`. Children are guaranteed to close before their parent, which is the foundation of hierarchical resource management.

But child scopes have a different `$[A]` type than their parent, so a parent-scoped value cannot be directly used in a child. That's where `lower` comes in. `Scope#lower` re-tags a parent-scoped value into a child scope, which is safe because the parent always outlives the child:

```scala

class Database(name: String) extends AutoCloseable {
  def query(sql: String): String = s"[$name] $sql"
  override def close(): Unit = println(s"Database [$name] closed")
}

class Connection(db: String, id: Int) extends AutoCloseable {
  def query(sql: String): String = s"[$db/$id] $sql"
  override def close(): Unit = println(s"Connection [$db/$id] closed")
}

Scope.global.scoped { parentScope =>

  // Open database in parent scope
  val db = allocate(Resource(new Database("maindb")))

  // Use database in parent scope
  $(db) { database =>
    println(s"Parent: ${database.query("SELECT 1")}")
  }

  // Create a child scope (e.g., for a request)
  parentScope.scoped { childScope =>

    // Lower the parent-scoped database into the child scope
    val dbInChild = childScope.lower(db)

    // Now we can use the database in the child scope
    val conn = allocate(Resource(new Connection("maindb", 1)))

    $(dbInChild) { database =>
      $(conn) { connection =>
        println(s"Child: ${connection.query("SELECT 2")}")
      }
    }

    println("Child scope exiting - connection closes first")
  }

  println("Parent scope exiting - database closes after children")
}
```

When the child scope exits, all resources allocated in it close first. Then the parent scope's finalizers run. This ensures that if a child holds a reference to a parent's resource, that resource is not closed until all children have finished.

## 8. Explicit Lifetime Management with `open()`

The `scoped { }` syntax ties resource lifetime to a lexical block. But sometimes you need explicit, decoupled lifetime management—for example, a request handler that opens a connection when processing begins and closes it when processing ends, independent of any fixed scope nesting.

Child scopes created via `scoped { }` are owned by the thread that creates them and must close within the creating thread. But `Scope.global.open()` creates an unowned scope that can be closed from any thread. This is useful for bridging structured scope-based resource management with callbacks or cross-thread communication:

```scala

class ConnectionPool extends AutoCloseable {
  def acquire(): String = "conn-001"
  override def close(): Unit = println("Pool closed")
}

// Simulate a request handler in an async framework
case class RequestContext(id: Int) {
  var connection: String = ""

  def setConnection(c: String): Unit = {
    connection = c
    println(s"Processing request ${id}, connection: $connection")
  }
}

// Using open() to manage lifetime explicitly, decoupled from lexical scope
val request = RequestContext(1)

Scope.global.scoped { scope =>

  // open() creates an unowned scope that can be closed explicitly
  $(open()) { handle =>
    val requestScope = handle.scope

    try {
      // Use the scope to allocate and manage the resource
      requestScope.scoped { innerScope =>

        val pool = allocate(Resource(new ConnectionPool()))

        $(pool) { p =>
          request.setConnection(p.acquire())
        }
      }
    } finally {
      // Close the open scope explicitly and handle any finalizer failures
      handle.close().orThrow()
    }
  }
}
```

The standard pattern for managing resource lifetimes in your application is to use `scoped { }` with careful nesting. The `open()` method is reserved for low-level integration points (like application startup/shutdown boundaries) and is not typically needed in application code.

:::note
Thread ownership is checked when `scoped { }` runs on a child scope, and never for unowned scopes from `open()`. This difference allows Scope to prevent thread-related bugs in structured code while still supporting integration with callback-driven or asynchronous frameworks that require explicit lifetime management.
:::

## 9. Shared Resources and Reference Counting

When multiple parts of your application need the same heavyweight resource (like a database connection pool), you want to create it once and destroy it only when the last user is done. `Resource.shared` provides reference-counted sharing:

```scala

class ConnectionPool(id: Int) extends AutoCloseable {
  def getConnection(): String = s"conn-from-pool-$id"
  override def close(): Unit = println(s"Pool $id closed")
}

case class UserService(poolId: String)
case class OrderService(poolId: String)

Scope.global.scoped { scope =>

  // Create a shared resource: only one pool instance, reference-counted
  val sharedPool = Resource.shared[ConnectionPool] { _ =>
    println("Creating shared pool...")
    new ConnectionPool(42)
  }

  // Both services allocate from the same shared resource
  // The pool is created once, destroyed after both services release it
  val pool1 = allocate(sharedPool)
  val pool2 = allocate(sharedPool)

  $(pool1) { p1 =>
    $(pool2) { p2 =>
      // Both p1 and p2 point to the same pool instance
      println(s"Service 1 got: ${p1.getConnection()}")
      println(s"Service 2 got: ${p2.getConnection()}")
      println("Both services are using the same shared pool instance")
    }
  }

  println("Scope exiting - shared pool closed (once)")
}
```

`Resource.shared` is memoized: the first `allocate` creates the pool, and subsequent `allocate` calls get the same instance. The finalizer runs only after all allocations have released their reference (implicitly when the scope closes).

This pattern is essential for applications with a shared database connection pool, cache, or logging infrastructure.

## 10. Dependency Injection with Wire

Applications often have many services with interdependencies. Manual wiring is error-prone: forget a dependency, pass the wrong type, create a cycle, or accidentally duplicate an instance where sharing was intended.

`Wire` and `Resource.from` provide compile-time dependency injection. Wires are builders that describe how to construct instances, and `Resource.from` resolves the entire dependency graph:

```scala

case class DbConfig(url: String)
case class Database(config: DbConfig) extends AutoCloseable {
  override def close(): Unit = println("Database closed")
}

case class CacheService(db: Database)
case class AuthService(db: Database)
case class AppService(cache: CacheService, auth: AuthService)

// Wire.shared means all dependents get the same instance
val configWire = Wire(DbConfig("localhost"))
val dbWire = Wire.shared[Database]
val cacheWire = Wire.shared[CacheService]
val authWire = Wire.shared[AuthService]
val appWire = Wire.shared[AppService]

// Resource.from resolves all wires and returns the root app service
def appResource = Resource.from[AppService](
  configWire,
  dbWire,
  cacheWire,
  authWire,
  appWire
)

appResource.use { app =>
  println("App service created: " + app.toString())
}
```

The compiler verifies that:
- Every dependency can be satisfied.
- No dependency is missing or provided twice. (A circular dependency is not reported yet: in 0.0.56 it crashes the macro, a known bug, #1767.)
- Types match correctly.

If you violate any of these rules, you get a clear compile error before runtime.

## 11. Thread Ownership

On the JVM, each `Scope.Child` (any scope created with `scoped { }` or as a child of another scope) is owned by the thread that created it. Opening a child with `scoped` from another thread is rejected, which stops a scope reference that escaped to another thread from creating nested scopes there. The check covers `scoped` only; see the note below.

You can check ownership with `Scope#isOwner`:

```scala

Scope.global.scoped { scope =>
  println(s"Global scope owned by current thread: ${scope.isOwner}")

  scope.scoped { childScope =>
    println(s"Child scope owned by current thread: ${childScope.isOwner}")
  }
}
```

Only `scoped` checks the owner thread. Calling `scoped` on a child scope from a different thread throws an `IllegalStateException`. `allocate`, `open`, `defer` and `$` do not check the owner, so keeping them on the owning thread is a rule you must follow yourself:

```scala

class Database extends AutoCloseable {
  override def close(): Unit = ()
}

Scope.global.scoped { scope =>

  val db = allocate(Resource(new Database()))

  // Calling scoped on this scope from another thread throws:
  // val thread = new Thread(() => {
  //   scope.scoped { _ => }  // throws IllegalStateException: current thread does not own this scope
  // })
  // thread.start()
}
```

To pass a resource to another thread safely, use `Scope.global.open()` to create an unowned scope, or redesign to keep all operations on the creating thread. Unowned scopes are not checked either way, so the choice is yours to get right.

For platforms like Scala.js (single-threaded), thread ownership checks are disabled—ownership is always considered valid.

:::note
Thread ownership enforcement is not about thread safety in the traditional sense—it's about structured concurrency. It prevents subtle bugs where a scope escapes to another thread and resources are closed on a different thread than they were allocated.
:::

## 12. Common Errors and Troubleshooting

This section lists the most common runtime and compile errors, explains what caused them, and how to fix them. For every error the Scope and Wire macros report, see [How to Fix Common Scope and Wire Compile Errors](fix-scope-and-wire-compile-errors.md).

### Runtime Errors

**`IllegalStateException`** when calling `allocate`, `$`, or `open` on a closed scope. The message is one of `Cannot allocate resource: scope is already closed.`, `Cannot access scoped value: scope is already closed.` or `Cannot open child scope: scope is already closed.`. `defer` does not throw: on a closed scope it returns a no-op handle and the finalizer never runs. The most common way to reach a closed scope is to let a resource escape the scoped block and use it afterwards:

```scala

class Database extends AutoCloseable {
  override def close(): Unit = ()
}

var db: Option[Database] = None  // Wrong: trying to escape scoped value

Scope.global.scoped { scope =>

  // db = Some(allocate(Resource.fromAutoCloseable(new Database())))  // Error: can't assign scope.$[Database] to Option[Database]
}

// scope is now closed; this throws IllegalStateException:
// db.foreach(_.close())
```

**Fix:** Ensure all resource usage happens before the scoped block exits. If you need to return a resource reference, return only the underlying value (wrapped in an `Unscoped` type).

**`IllegalStateException: Cannot create child scope: current thread '...' does not own this scope`** when calling `scoped` on a child scope from a different thread:

```scala

var scope: Option[Scope] = None

Scope.global.scoped { s =>
  scope = Some(s)
}

Future {
  // This throws: the scope was created on the main thread, not this one
  // scope.foreach(_.scoped { _ => })
}
```

**Fix:** Use `Scope.global.open()` to create an unowned scope that can be shared across threads, or keep all operations on the creating thread.

### Compile Errors

**No `Unscoped` instance for type `T`** when trying to return a value from `scoped`:

```scala

class Connection extends AutoCloseable {
  override def close(): Unit = ()
}

// This does not compile:
// val conn = Scope.global.scoped { scope =>
//   import scope._
//   allocate(Resource(new Connection()))  // ERROR: $[Connection] has no Unscoped
// }
```

**Fix:** Only return types with an `Unscoped` instance (primitives, case classes, collections). If you need to return a resource reference, extract its underlying value first, or use `Wire` to manage the resource's lifetime.

**Cannot call method directly on `$[T]`**:

```scala

class Logger extends AutoCloseable {
  def log(msg: String): Unit = println(msg)
  override def close(): Unit = ()
}

// This does not compile:
// Scope.global.scoped { scope =>
//   import scope._
//   val logger = allocate(Resource(new Logger()))
//   logger.log("test")  // ERROR: method log not visible on $[Logger]
// }
```

**Fix:** Use the `$` operator to unwrap: `$(logger)(_.log("test"))`.

**`Cannot auto-create java.lang.String`** when a constructor argument has no wire:

```scala

case class Database(url: String) extends AutoCloseable {
  override def close(): Unit = ()
}

// Wire.shared[Database] compiles, even though no wire provides the String url.
// The error appears later, when the Resource is built:
val dbWire = Wire.shared[Database]
// Resource.from[Database](dbWire)  // ERROR: Cannot auto-create java.lang.String
```

The error names the type that has no wire (`java.lang.String`), and it is reported when the `Resource` is built, not when the `Wire` is declared.

**Fix:** Provide a wire for every required dependency:

```scala

case class Database(url: String) extends AutoCloseable {
  override def close(): Unit = ()
}

// Correct: provide the String
val urlWire = Wire("localhost")
val dbWire = Wire.shared[Database]

Scope.global.scoped { scope =>

  val db = allocate(Resource.from[Database](urlWire, dbWire))
  $(db) { d => println("Connected to " + d.toString()) }
}
```

For every error the Scope and Wire macros report, with the real compiler output and a fix for each, see [How to Fix Common Scope and Wire Compile Errors](./fix-scope-and-wire-compile-errors.md).

---

## Putting It Together

Now let's combine everything we've learned into a single, realistic example. This example demonstrates:

- Multiple allocated resources (database and logger)
- Wire-based dependency injection with shared and unique wires
- Automatic cleanup in reverse allocation order
- The complete interaction of all concepts

This example combines core concepts — allocation, cleanup, resource composition, and dependency injection:

```scala

class Database extends AutoCloseable {
  def query(sql: String): String = s"Results: $sql"
  override def close(): Unit = println("Closing database")
}

class Logger extends AutoCloseable {
  def log(msg: String): Unit = println(s"[LOG] $msg")
  override def close(): Unit = println("Closing logger")
}

class Application(database: Database, logger: Logger) extends AutoCloseable {
  def run(): String = {
    logger.log("Starting application")
    val result = database.query("SELECT * FROM users")
    logger.log(s"Query executed: $result")
    result
  }
  override def close(): Unit = logger.log("Shutting down application")
}

// Create an app using Scope with wire-based dependency injection
Scope.global.scoped { scope =>

  // Wire.shared means all dependents receive the same Logger instance
  // Wire.shared[Database] means all dependents receive the same Database
  // Resource.from[Application] constructs Application by resolving dependencies
  val app = allocate(
    Resource.from[Application](
      Wire.shared[Database],
      Wire.shared[Logger]
    )
  )

  // Use the application
  $(app) { a =>
    val result = a.run()
    println(s"Result: $result")

    // Register additional cleanup operations
    defer {
      println("Application cleanup complete")
    }
  }

  // All resources are closed in LIFO order: Application, Logger, then Database
  ()
}

println("Program finished")
```

This example demonstrates:
- **Wire-based DI**: `Wire.shared[T]` automatically derives how to construct `T` from its dependencies
- **Resource.from**: The macro analyzes the dependency graph and automatically allocates in correct order
- **Composition**: Multiple resources (Database, Logger, Application) with declared dependencies
- **Cleanup**: All resources close in reverse order when the scope exits (LIFO)

---

## What You've Learned

In this tutorial, you learned:

- What Scope is and why compile-time resource safety matters
- How to allocate, use, and manage resources with the `scoped { }` block
- The `$[A]` type and how it enforces resource ownership at the type level
- How to construct resources with `Resource` and its variants
- The `Unscoped[A]` typeclass and why returning resources is forbidden
- How to register cleanup with `defer` and manage finalizer order
- Nested scopes and the `lower` operator for hierarchical resource management
- Advanced patterns like `open()` for decoupled lifetime management, `Resource.shared` for reference counting, and `Wire` for dependency injection
- Common errors and how to fix them

You now have a solid foundation in Scope. The next step is to see how to apply these concepts in practice with realistic scenarios.

---

## Running the Examples

The code examples in this tutorial are embedded directly in the documentation and compile with mdoc. To run them locally:

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Compile and run the tutorial examples:**

All examples from the tutorial sections are compile-checked using mdoc. To verify they compile:

```bash
sbt "docs/mdoc --in docs/guides/compile-time-resource-safety-with-scope.md"
```

**3. Run standalone examples from the scope-examples module:**

The repository also includes additional companion examples in `scope-examples/`. For example:

```bash
cd scope-examples && sbt "runMain runDatabaseExample"
cd scope-examples && sbt "runMain runCachingExample"
cd scope-examples && sbt "runMain runThreadOwnershipExample"
```

To compile all examples:

```bash
cd scope-examples && sbt "compile"
```

---

## Where to Go Next

- **Ready to use this in practice?** Check out the how-to guides (coming soon) which walk through real-world examples.
- **Want to dive deeper into the API?** Read the [Scope Reference](../reference/resource-management/scope.md) for comprehensive API documentation.
- **Interested in related concepts?** Explore dependency injection with [Wire](../reference/resource-management/wire.md) or resource composition patterns.

---

## Summary

You now understand Scope's core concepts:

- **`$[A]`** — a type-level owner tag that prevents resources from escaping their scope.
- **`scoped { }`** — the syntax for entering and exiting a scope.
- **`allocate` and `defer`** — operations to register resources and cleanup.
- **`Resource`** — lazy descriptions of acquisition and cleanup.
- **`Unscoped`** — a compile-time guarantee that a type is safe to return from a scope.
- **Nesting and `lower`** — hierarchical resource management with compile-time parent-child relationships.
- **Shared resources** — reference counting for multiply-used resources.
- **`Wire` and dependency injection** — compile-time-verified wiring of complex applications.
- **Thread ownership** — JVM enforcement of structured concurrency.

For complete API documentation, see the [Scope Reference](../reference/resource-management/scope.md).

## See Also

- [Coming from cats-effect Resource, ZIO Scope/ZLayer, macwire or Using](./resource-and-di-migration.md) — migrate existing resource handling and dependency wiring to Scope
- [Telemetry Reference](../reference/telemetry/index.md) — `TracerProvider`, `LoggerProvider`, and `MeterProvider` are `AutoCloseable`, so the ownership rules here carry over to their lifetimes
- [Telemetry Guide](./telemetry-guide.md) — Provider startup and shutdown ordering, done with plain `AutoCloseable` shutdown rather than `Scope`
- [Async Reference](../reference/async.md) — `Async.Running` extends `AutoCloseable` and integrates with `scala.util.Using` for scoped cancellation; the same resource-ownership mental model used by Scope applies to in-flight async computations

---

## Coming from PureConfig, Typesafe Config or zio-config

In this guide, we move a Scala bookstore service that loads its settings with PureConfig, Typesafe Config or zio-config over to the ZIO Blocks `config` module. By the end, the service reads the `application.conf` it already ships with `config-hocon`, ZIO Blocks' own HOCON parser, takes overrides from environment variables, reports every configuration error at once, and no longer depends on Typesafe Config at all.

We take an incremental approach: first we read the unchanged file, then we map its kebab-case keys to our camelCase fields, replace its `${?VAR}` environment substitutions with an environment layer, and check which other HOCON features carry over. Last, we compare error reports and keep the database password out of the logs.

This is a migration recipe, not an introduction. If you have never loaded configuration with ZIO Blocks, work through [Build a Typed Configuration Loader](./typed-config-loader.md) first: it teaches sources, layering, defaults and provenance with the same bookstore, and this guide builds on it.

**What we will cover:**

- Which ZIO Blocks call replaces each PureConfig, Typesafe Config and zio-config call
- Reading an existing `application.conf` with `config-hocon`, from a string or from a file with includes
- Keeping kebab-case keys such as `featured-title` with `KeyFormat.KebabCase`
- Replacing `${?DB_PASSWORD}` with an `EnvSource` layer, and the two environment pitfalls to avoid
- The HOCON features that behave differently, and what to change in the file
- Reading every error at once, and keeping secrets out of logs
- What works differently: no effect system, no reload, and Scala.js support

## The Problem

A configuration loader written against PureConfig, Typesafe Config or zio-config ties the service to Typesafe Config, the Java library that parses HOCON underneath all three. Typesafe Config runs only on the JVM, and its README describes it as feature complete, with changes now rare. The Scala code around it has its own costs:

- **Typesafe Config** reads one key at a time through getters such as `getString` and `getInt`. Every key is spelled by hand, and the first problem throws a `ConfigException`, so a deployment with three mistakes fails three times.
- **PureConfig** derives readers, but its Scala 3 derivation has changed between releases: `derives ConfigReader` needed an extra import until 0.17.7, and field-name hints need the separate `pureconfig-generic-scala3` module. Teams moving to Scala 3 have found the documentation out of step with the code, as in [this Scala Users thread](https://users.scala-lang.org/t/scala-3-migration-pureconfig/12096).
- **zio-config** loads configuration as a ZIO effect, so reading a file needs a ZIO runtime, and its derivation (`zio-config-magnolia`) and HOCON (`zio-config-typesafe`) modules are published for the JVM only.

For example, this is the hand-written Typesafe Config loader for our bookstore (old code, not compiled on this page):

```scala
// Before: Typesafe Config 1.4.9

def load(root: Config): BookstoreConfig = {
  val c  = root.getConfig("bookstore")
  val db = c.getConfig("db")
  BookstoreConfig(
    name            = c.getString("name"),
    port            = c.getInt("port"),
    featuredTitle   = c.getString("featured-title"),
    db              = DbConfig(db.getString("host"), db.getInt("port"), db.getString("user"), db.getString("password")),
    tags            = c.getStringList("tags").asScala.toList,
    discountPercent = if (c.hasPath("discount-percent")) Some(c.getInt("discount-percent")) else None
  )
}

val config = load(ConfigFactory.load()) // throws ConfigException.Missing on the first missing key
```

Every field is read by name, a renamed field compiles but fails at startup, and the optional discount needs its own `hasPath` check. ZIO Blocks replaces all of this with one call that walks a case class with a derived `Schema`, runs the same way on Scala 2.13, Scala 3 and Scala.js, and parses HOCON without Typesafe Config.

## Prerequisites

### Dependency Changes

Remove the old configuration dependencies:

```scala
// Before: pick the lines your project has
libraryDependencies += "com.github.pureconfig" %% "pureconfig-core"     % "0.17.11" // Scala 3
libraryDependencies += "com.github.pureconfig" %% "pureconfig"          % "0.17.11" // Scala 2.13
libraryDependencies += "com.typesafe"           % "config"              % "1.4.9"
libraryDependencies += "dev.zio"               %% "zio-config"          % "4.1.0"
libraryDependencies += "dev.zio"               %% "zio-config-magnolia" % "4.1.0"
libraryDependencies += "dev.zio"               %% "zio-config-typesafe" % "4.1.0"
```

Add the ZIO Blocks config module and its HOCON adapter:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-config"       % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-config-hocon" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-config"       % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-next-config-hocon" % "0.0.56"
```

`zio-blocks-config-hocon` depends only on `zio-blocks-config`. Neither module pulls in Typesafe Config or any other third-party library, so once the old lines are gone, `com.typesafe:config` leaves the classpath unless something else still needs it.

### Imports

Three imports cover this guide:

```scala

```

`zio.blocks.config._` brings in `Config`, `ConfigSource`, `EnvSource`, `KeyFormat`, `Secret` and `ConfigError`. `zio.blocks.config.hocon._` adds `ConfigSource.fromHocon`, and on the JVM `ConfigSource.fromFile`.

## Mapping Old Calls to ZIO Blocks

Each row is one task the old loader did, and the ZIO Blocks call that does it now:

| Task                             | PureConfig                                        | Typesafe Config                          | zio-config                                  | ZIO Blocks                                                           |
|----------------------------------|---------------------------------------------------|------------------------------------------|---------------------------------------------|----------------------------------------------------------------------|
| Describe the configuration       | `derives ConfigReader`, or `generic.auto._`       | Hand-written getters                     | `deriveConfig[A]`                           | `Schema.derived` for the case class                                  |
| Read `application.conf`          | `ConfigSource.default`                            | `ConfigFactory.load()`                   | `ConfigProvider.fromResourcePath()`         | `ConfigSource.fromFile(path)` (JVM) or `ConfigSource.fromHocon(text, name)` |
| Select the `bookstore` block     | `.at("bookstore")`                                | `getConfig("bookstore")`                 | `.nested("bookstore")`                      | `ConfigSource#prefix("bookstore")`                                   |
| Map `featuredTitle` to `featured-title` | Default field mapping                      | Key spelled in each getter               | `ConfigProvider#kebabCase`                  | `ConfigSource#keyFormat(KeyFormat.KebabCase)`                        |
| Override from the environment    | `${?VAR}` in the file                             | `${?VAR}` in the file                    | `ConfigProvider.envProvider`                | `EnvSource` layered with `ConfigSource#orElse`                       |
| Add a fallback                   | `withFallback`                                    | `withFallback`                           | `orElse`                                    | `ConfigSource#orElse`                                                |
| Load                             | `load[A]`: `Either[ConfigReaderFailures, A]`      | Getters that throw                       | `ZIO.config(config)`: `IO[Config.Error, A]` | `Config.load[A](source)`: `Either[::[ConfigError], A]`               |
| Load or stop                     | `loadOrThrow[A]`                                  | Getters that throw                       | Fail the effect                             | `Config.loadOrThrow[A](source)`                                      |
| Hide a password                  | Your own wrapper type                             | Your own wrapper type                    | `Config.Secret`                             | `Secret`                                                             |
| See where a value came from      | Not available                                     | `ConfigValue#origin`                     | Not available                               | `Config.loadWithProvenance` and `ProvenanceMap#dump`                 |

Two differences run through the whole table. ZIO Blocks never reads `application.conf` from the classpath on its own: we name the file or pass its text. And the order of layers is explicit: `env.orElse(file)` asks `env` first, the way a stack of `withFallback` calls reads in reverse.

## The Application We Are Migrating

The bookstore keeps its settings in `application.conf`, under a `bookstore` block with kebab-case keys. The database password comes from the `DB_PASSWORD` environment variable through an optional substitution:

```text
bookstore {
  name = "Scala Bookstore"
  port = 8080
  featured-title = "Zionomicon"
  discount-percent = 10

  db {
    host = "localhost"
    port = 5432
    user = "bookstore"
    password = ${?DB_PASSWORD}
  }

  tags = ["scala", "functional-programming", "zio"]
}
```

With PureConfig, the loader is a derived reader and one call (old code, not compiled on this page):

```scala
// Before: PureConfig 0.17.11, Scala 3

final case class DbConfig(host: String, port: Int = 5432, user: String, password: String) derives ConfigReader
final case class BookstoreConfig(
  name: String,
  port: Int = 8080,
  featuredTitle: String,
  db: DbConfig,
  tags: List[String],
  discountPercent: Option[Int]
) derives ConfigReader

val result: ConfigReader.Result[BookstoreConfig] = ConfigSource.default.at("bookstore").load[BookstoreConfig]
```

With zio-config, the same load is an effect that runs inside a ZIO application (old code, not compiled on this page):

```scala
// Before: zio-config 4.1.0

object Main extends ZIOAppDefault {
  override val bootstrap = Runtime.setConfigProvider(ConfigProvider.fromResourcePath().kebabCase)

  def run = ZIO.config(deriveConfig[BookstoreConfig].nested("bookstore")).flatMap(Console.printLine(_))
}
```

On the ZIO Blocks side, the case classes stay the same as in [Build a Typed Configuration Loader](./typed-config-loader.md). Only two things change: the password becomes a `Secret`, and the reader derivation becomes a `Schema`:

```scala
case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

case class BookstoreConfig(
  name: String,
  port: Int = 8080,
  featuredTitle: String,
  db: DbConfig,
  tags: List[String],
  discountPercent: Option[Int]
)

object BookstoreConfig {
  implicit val schema: Schema[BookstoreConfig] = Schema.derived
}
```

`implicit val schema` with `Schema.derived` compiles on Scala 2.13 and Scala 3 alike, so a cross-built service keeps one definition. `Schema.derived` also derives the schemas for `DbConfig`, `List[String]` and `Option[Int]`, and `Secret` ships with its own.

We also write a small helper that prints either the loaded configuration or the message of every error:

```scala
def report[A](result: Either[::[ConfigError], A]): Unit =
  result match {
    case Right(config) => println(config)
    case Left(errors)  => errors.foreach(error => println(error.message))
  }
```

## Reading Your Existing application.conf

`ConfigSource.fromHocon` parses HOCON text with `config-hocon`'s own parser and flattens it into dot-separated keys such as `bookstore.db.host`, the shape every ZIO Blocks source shares. It takes the text and a name for the source, and returns `Either[ConfigError, ConfigSource]`, because the text might not be valid HOCON. A documentation page can't ship a file, so we paste the unchanged `application.conf` into a string:

```scala
val applicationConf =
  """bookstore {
    |  name = "Scala Bookstore"
    |  port = 8080
    |  featured-title = "Zionomicon"
    |  discount-percent = 10
    |
    |  db {
    |    host = "localhost"
    |    port = 5432
    |    user = "bookstore"
    |    password = ${?DB_PASSWORD}
    |  }
    |
    |  tags = ["scala", "functional-programming", "zio"]
    |}
    |""".stripMargin

val hocon: ConfigSource = ConfigSource.fromHocon(applicationConf, "application.conf") match {
  case Right(source) => source
  case Left(error)   => throw error
}
```

PureConfig's `.at("bookstore")` becomes `ConfigSource#prefix("bookstore")`, which puts `bookstore.` in front of every key the loader asks for. Load the file as it is:

```scala
report(Config.load[BookstoreConfig](hocon.prefix("bookstore")))
// Missing required key 'featuredTitle' in source 'application.conf'
// Missing required key 'db.password' in source 'application.conf'
```

The file parses, but the load reports two missing keys, and each one is a migration step of its own:

- `featuredTitle` is missing because the file spells it `featured-title`. ZIO Blocks looks up field names exactly as written in the case class; the next section maps them.
- `db.password` is missing because `${?DB_PASSWORD}` found nothing. We replace it with an environment layer two sections below.

The report leaves out one key: `discount-percent` is missing for the same reason as `featured-title`, but `discountPercent` is an `Option[Int]`, and a missing key for an optional field is `None`, not an error. Until the keys are mapped, every kebab-case key behind an optional field or a field with a default is silently ignored.

## Keeping Your Kebab-Case Keys

PureConfig maps the camelCase field `featuredTitle` to the kebab-case key `featured-title` by default, and zio-config does the same with `ConfigProvider#kebabCase`. In ZIO Blocks, `ConfigSource#keyFormat` wraps a source and rewrites each key the loader asks for into the format the file uses. `KeyFormat.KebabCase` turns `featuredTitle` into `featured-title` and leaves dots alone, so nested keys such as `db.host` keep working:

```scala
val application: ConfigSource = hocon.keyFormat(KeyFormat.KebabCase).prefix("bookstore")
```

Load again:

```scala
report(Config.load[BookstoreConfig](application))
// Missing required key 'db.password' in source 'application.conf'
```

Only the password is missing now. `featuredTitle` and `discountPercent` both read their kebab-case keys.

The file keeps its kebab-case keys, so it stays readable by the old loader while both run side by side. `ConfigSource#keyFormat` also accepts `KeyFormat.SnakeCase`, `KeyFormat.UpperSnakeCase` and `KeyFormat.CamelCase`, and `ConfigSource#keyMapper` takes a custom `KeyMapper` for any other convention.

## Replacing `${?VAR}` With an Environment Layer

In Typesafe Config, `password = ${?DB_PASSWORD}` first looks for a `DB_PASSWORD` key in the document and then falls back to the environment variable. `config-hocon` resolves substitutions inside the document only: it never reads environment variables, so `${?DB_PASSWORD}` finds nothing and the key disappears. Delete these lines from the file and add the environment as a layer of its own.

`ConfigSource#orElse` stacks two sources: `primary.orElse(fallback)` looks up each key in `primary` and asks `fallback` only for the keys `primary` doesn't have. This page can't set environment variables, so, as in [Build a Typed Configuration Loader](./typed-config-loader.md), a map source named `env` stands in for them:

```scala
val env = ConfigSource.fromMap(Map("db.password" -> "s3cr3t-from-env", "port" -> "9090"), "env")

val layered = env.orElse(application)
```

Load the layered source:

```scala
report(Config.load[BookstoreConfig](layered))
// BookstoreConfig(Scala Bookstore,9090,Zionomicon,DbConfig(localhost,5432,bookstore,<secret>),List(scala, functional-programming, zio),Some(10))
```

The password and the port `9090` come from `env`, and every other value from `application.conf`.

In a real service, `EnvSource` replaces the stand-in, and two details decide which variables it reads:

```scala
val realEnv: ConfigSource = EnvSource.keyFormat(KeyFormat.UpperSnakeCase).prefix("bookstore")

val production = realEnv.orElse(application)
```

`EnvSource` turns a key into a variable name by replacing dots with underscores and upper-casing the result. With `ConfigSource#prefix("bookstore")`, `db.password` is read from `BOOKSTORE_DB_PASSWORD` and `port` from `BOOKSTORE_PORT`. `KeyFormat.UpperSnakeCase` splits camelCase field names first, so `featuredTitle` is read from `BOOKSTORE_FEATURED_TITLE`.

:::warning
Without `KeyFormat.UpperSnakeCase`, `EnvSource` upper-cases a camelCase field name without splitting it: `featuredTitle` is read from `BOOKSTORE_FEATUREDTITLE`, and an exported `BOOKSTORE_FEATURED_TITLE` is silently ignored. Dotted keys such as `db.password` map correctly either way, so the mistake shows up only for multi-word fields.
:::

:::warning
Always give `EnvSource` a prefix. A bare `EnvSource` answers for every variable in the process, so an unrelated `PORT` or `HOST` set for another program becomes configuration. `ProvenanceMap#dump` over a bare `EnvSource` also prints the **entire process environment**, and it hides only the values whose names contain a sensitive word such as `password`, `secret` or `token`. A `DATABASE_URL` with an embedded password is printed in clear text.
:::

The prefix renames the variables: `DB_PASSWORD` becomes `BOOKSTORE_DB_PASSWORD`. If the deployment can't change its variable names yet, read the few old names into a map source instead of opening up the whole environment:

```scala
val legacyEnv = ConfigSource.fromMap(sys.env.get("DB_PASSWORD").map(password => "db.password" -> password).toMap, "env")
```

`legacyEnv.orElse(application)` then reads the password from `DB_PASSWORD` and nothing else from the environment.

## Loading the File on the JVM, Includes and All

On the JVM, `ConfigSource.fromFile` reads the file itself and resolves its `include "file"` directives relative to the including file's directory, the way Typesafe Config does for files. It returns `Either[ConfigError, ConfigSource]`, and a missing file is a `Left`, never an exception.

To show an include at work, we write a two-file configuration into a temporary directory. `application.conf` includes `db.conf` inside its `bookstore` block:

```scala

val confDir = Files.createTempDirectory("bookstore")

Files.writeString(
  confDir.resolve("application.conf"),
  """bookstore {
    |  name = "Scala Bookstore"
    |  featured-title = "Zionomicon"
    |  tags = ["scala", "zio"]
    |  include "db.conf"
    |}
    |""".stripMargin
)

Files.writeString(
  confDir.resolve("db.conf"),
  """db {
    |  host = "localhost"
    |  user = "bookstore"
    |  password = "dev-password"
    |}
    |""".stripMargin
)
```

Now we load `application.conf` from disk:

```scala
ConfigSource.fromFile(confDir.resolve("application.conf").toString) match {
  case Right(file) =>
    println(file.sourceId)
    report(Config.load[BookstoreConfig](file.keyFormat(KeyFormat.KebabCase).prefix("bookstore")))
  case Left(error) =>
    println(error.message)
}
// hocon:application.conf
// BookstoreConfig(Scala Bookstore,8080,Zionomicon,DbConfig(localhost,5432,bookstore,<secret>),List(scala, zio),None)
```

The `db` block from `db.conf` lands under `bookstore`, where the include sits, and the source is named `hocon:` followed by the file name.

`ConfigSource.fromFile` also takes `allowedBase`, a directory that the file and its includes must stay inside, and `maxIncludeDepth`, which defaults to `10`. `ConfigSource.fromFileWithLimits` adds `maxFileBytes`, which caps the size of the file and of each include and defaults to 1 MiB. Configuration files are usually small, so the defaults rarely need changing.

:::note
`ConfigFactory.load()` and PureConfig's `ConfigSource.default` find `application.conf` on the classpath. `config-hocon` has no classpath lookup: pass `fromFile` a path on disk, or read a bundled resource yourself, for example with `scala.io.Source.fromResource("application.conf").mkString`, and pass its text to `ConfigSource.fromHocon`. Text passed to `fromHocon` has no directory to resolve includes from, so its `include` lines are dropped. `fromFile` is JVM-only; on Scala.js, `fromHocon` is the entry point.
:::

## Checking Which HOCON Features Carry Over

Most of an `application.conf` parses unchanged: objects with `=`, `:` or no separator, dotted keys such as `a.b.c = 1`, duplicate keys where the last one wins, objects merged key by key, `#` and `//` comments, quoted, unquoted and triple-quoted strings, `${path}` and `${?path}` substitutions that point inside the document, string concatenation such as `${base}"/data"`, `+=` to append to an array, and JSON. A few features behave differently, and a migration needs to check for each of them:

| HOCON in the file                                   | Typesafe Config                                   | `config-hocon`                                                                 | What to change                                                  |
|-----------------------------------------------------|---------------------------------------------------|--------------------------------------------------------------------------------|-----------------------------------------------------------------|
| `${?VAR}` naming an environment variable            | Falls back to the environment                     | Looks inside the document only; the key disappears                             | Delete it and layer `EnvSource`                                 |
| `${VAR}` naming an environment variable             | Falls back to the environment                     | Parse error: unresolved substitution                                           | Delete it and layer `EnvSource`                                 |
| `include "file"`                                    | Supported                                         | Supported by `fromFile`; dropped by `fromHocon`                                | Load the file with `fromFile`                                   |
| `include file(...)`, `classpath(...)`, `url(...)`, `required(...)` | Supported                          | Parse error                                                                    | Use `include "file"`, or load the file and layer it with `orElse` |
| `reference.conf` merged under `application.conf`    | Automatic                                         | Not read                                                                       | Load it and add it with `orElse`, or use case-class defaults    |
| Self-reference such as `path = ${path}":/extra"`    | Supported                                         | Parse error: circular substitution                                             | Write the full value                                            |
| Array concatenation such as `[a] [b]`               | Supported                                         | The value becomes an empty string                                              | Write one array, or use `+=`                                    |
| Durations such as `10s` and sizes such as `512M`    | Parsed by `getDuration` and `getBytes`            | Kept as strings; `java.time.Duration` fields need ISO-8601 (`PT10S`)            | Rewrite durations as ISO-8601; read sizes as numbers            |

Four of these show up on small documents. The helper `keys` parses a document with `fromHocon` and prints the keys it produced, or the parse error:

```scala
def keys(document: String): Unit =
  ConfigSource.fromHocon(document) match {
    case Right(source) => println(source.all("").map { case (key, value) => s"$key = '${value.value}'" }.toList.sorted)
    case Left(error)   => println(error.message)
  }
```

Now we pass it an environment substitution, a classpath include, an include in a string and an array concatenation:

```scala
keys("home = ${?HOME}")
// List()
keys("""include classpath("reference.conf")""")
// Parse error for key '' (expected valid HOCON) in source 'hocon:string'
keys("""db { include "db.conf" }, name = "Scala Bookstore"""")
// List(name = 'Scala Bookstore')
keys("tags = [scala] [zio]")
// List(tags = '')
```

The code above:

- `home = ${?HOME}` produces no keys, even though `HOME` is set in almost every environment.
- `include classpath(...)` fails the whole document with a parse error.
- The `include` inside a string is dropped, and so is the `db` block it was the only content of.
- `[scala] [zio]` produces a single key whose value is an empty string, not a two-element list.

The parse error names only the source. To find the line, parse the file with the old loader once, or look for the constructs in the table above.

Durations need the most care, because the file still parses and only the load fails. `config-hocon` keeps `10s` as the string `10s`, and a `java.time.Duration` field is read with `java.time.Duration.parse`, which accepts ISO-8601 only:

```scala
case class Timeouts(requestTimeout: java.time.Duration)

object Timeouts {
  implicit val schema: Schema[Timeouts] = Schema.derived
}
```

Load it from both spellings:

```scala
report(Config.load[Timeouts](ConfigSource.fromHocon("requestTimeout = 10s").toOption.get))
// Invalid value '10s' for key 'requestTimeout' (expected Duration) in source 'hocon:string': Text cannot be parsed to a Duration
report(Config.load[Timeouts](ConfigSource.fromHocon("requestTimeout = PT10S").toOption.get))
// Timeouts(PT10S)
```

`10s` is an invalid value, and `PT10S` loads as a ten-second duration.

## Reading Every Error at Once

PureConfig and zio-config already collect errors from every field before they fail, and Typesafe Config getters throw on the first one. `Config.load` collects them too: it checks every field and returns a non-empty list `::[ConfigError]`, one entry per problem.

Break the file in three ways: a word for the port, a word for the discount, and no featured title. The environment stand-in is empty, so the password is missing too:

```scala
val brokenConf =
  """bookstore {
    |  name = "Scala Bookstore"
    |  port = eighty
    |  discount-percent = "ten"
    |  db { host = "localhost", user = "bookstore" }
    |  tags = ["scala"]
    |}
    |""".stripMargin

val broken = ConfigSource.fromHocon(brokenConf, "application.conf") match {
  case Right(source) => source.keyFormat(KeyFormat.KebabCase).prefix("bookstore")
  case Left(error)   => throw error
}

val emptyEnv = ConfigSource.fromMap(Map.empty, "env")
```

Now we load the broken layers:

```scala
report(Config.load[BookstoreConfig](emptyEnv.orElse(broken)))
// Invalid value 'eighty' for key 'port' (expected Int) in source 'env|application.conf': For input string: "eighty"
// Missing required key 'featuredTitle' in source 'env|application.conf'
// Missing required key 'db.password' in source 'env|application.conf'
// Invalid value 'ten' for key 'discountPercent' (expected Int) in source 'env|application.conf': For input string: "ten"
```

The code above reports all four problems from one load:

- Each message names the key in the case class's spelling, such as `discountPercent`, not the file's `discount-percent`.
- An invalid value ends with the reason the conversion failed, here the JDK's `For input string` message.
- The source is the whole layered stack, `env|application.conf`, because the loader asked the stack as a whole. To find the layer that supplied a value that did load, use `Config.loadWithProvenance` from the next section.

The port has a default of `8080`, and it is still reported. A default replaces a missing key, never a broken one, so a typo can't fall back silently.

`Config.loadOrThrow` replaces PureConfig's `loadOrThrow`: it returns the configuration or throws a `ConfigLoadException` whose message is a report of the errors, for services that should stop at startup.

## Keeping Secrets Out of Logs

PureConfig and Typesafe Config leave secret handling to us, usually with a wrapper type whose `toString` hides the value, and zio-config ships `Config.Secret`. ZIO Blocks ships `Secret`: a wrapper around a `String` whose `Secret#toString` is `<secret>`, so printing or logging the configuration never shows the password. `Secret.unwrap` is the only way to read the value.

`Config.loadWithProvenance` loads the configuration and keeps the source stack, so `ProvenanceMap#dump` can print every key, its value and the layer that supplied it:

```scala
Config.loadWithProvenance[BookstoreConfig](layered) match {
  case Right(provenance) =>
    println(provenance.value)
    println(provenance.dump())
    println(Secret.unwrap(provenance.value.db.password).length)
  case Left(errors) =>
    errors.foreach(error => println(error.message))
}
// BookstoreConfig(Scala Bookstore,9090,Zionomicon,DbConfig(localhost,5432,bookstore,<secret>),List(scala, functional-programming, zio),Some(10))
// ┌─────────────────┬────────────────────────┬──────────────────┐
// │ Key             │ Value                  │ Source           │
// ├─────────────────┼────────────────────────┼──────────────────┤
// │ db.host         │ localhost              │ application.conf │
// │ db.password     │ <secret>               │ env              │
// │ db.port         │ 5432                   │ application.conf │
// │ db.user         │ bookstore              │ application.conf │
// │ discountPercent │ 10                     │ application.conf │
// │ featuredTitle   │ Zionomicon             │ application.conf │
// │ name            │ Scala Bookstore        │ application.conf │
// │ port            │ 9090                   │ env              │
// │ tags.0          │ scala                  │ application.conf │
// │ tags.1          │ functional-programming │ application.conf │
// │ tags.2          │ zio                    │ application.conf │
// └─────────────────┴────────────────────────┴──────────────────┘
// 15
```

The code above:

- prints the configuration, with the password shown as `<secret>` by `Secret#toString`.
- prints the provenance table. The rows use the case class's key names, each value names its layer, and `db.password` shows `<secret>`.
- prints the length of the real password from `Secret.unwrap`, a value the program can use without ever logging it.

`ProvenanceMap#dump` hides a value because its **key name** contains a sensitive word such as `password`, `secret` or `token`, not because the field is a `Secret`. Give every secret setting a name that says what it is. The [Secrets](../reference/config/config-source.md#secrets) section of the `ConfigSource` reference lists the words.

:::note
When `EnvSource` is wrapped in a key format, as in `production` above, `ProvenanceMap#dump` lists an environment override under the variable's own words: `BOOKSTORE_FEATURED_TITLE` appears as `featured.title`, and the overridden `featuredTitle` row from the file stays in the table. The loaded value is correct. To check one key, ask `ProvenanceMap#provenanceOf("featuredTitle")`, which names the variable that supplied it.
:::

## What Works Differently

A few habits from the old libraries don't carry over:

- **No effect system.** `Config.load` is a plain function call that returns an `Either`. There is no ZIO runtime to set up and no `bootstrap` layer; a ZIO application wraps the call in `ZIO.fromEither` where it needs an effect.
- **No reload or watch.** A load reads the sources once. To pick up a changed file, load again, as in the example below. Runtime switches that change while the service runs are covered by [Flags](../reference/config/flags.md), not by `Config`.
- **Every Scala version and Scala.js.** `zio-blocks-config` and `zio-blocks-config-hocon` are built for Scala 2.13, Scala 3.3 LTS and Scala 3.9, on the JVM and Scala.js. `ConfigSource.fromFile` is JVM-only, and on Scala.js `SysPropSource` is always empty; `ConfigSource.fromHocon` and `EnvSource` work on both platforms.
- **Nothing is read implicitly.** `ConfigFactory.load()` merges system properties, `application.conf` and `reference.conf` on its own. In ZIO Blocks, each layer is a source we name, and the order of `orElse` calls is the order of precedence. `SysPropSource` reads JVM system properties by their dotted names when a `-D` override layer is wanted.

Loading is an ordinary call, so reloading is a function we call again whenever the service wants fresh values:

```scala
def loadBookstore(): Either[::[ConfigError], BookstoreConfig] = Config.load[BookstoreConfig](layered)

println(loadBookstore().map(_.featuredTitle))
// Right(Zionomicon)
```

## Putting It Together

This example loads the migrated `application.conf` from disk with `ConfigSource.fromFile`, maps its kebab-case keys, layers `BOOKSTORE_*` environment variables over it, and prints the configuration and where each value came from, or every error at once. The file is the PureConfig file without its `${?DB_PASSWORD}` line:

```scala title="config-examples/src/main/scala/configlibrarymigration/CompleteExample.scala"
package configlibrarymigration

/**
 * Coming from PureConfig, Typesafe Config or zio-config — Putting It Together
 *
 * Loads the migrated src/main/resources/configlibrarymigration/application.conf
 * with config-hocon, maps its kebab-case keys to camelCase fields, layers
 * BOOKSTORE_* environment variables over it, and prints the configuration and
 * where each value came from, or every error at once.
 *
 * Run with: cd config-examples && sbt "runMain
 * configlibrarymigration.CompleteExample"
 *
 * Override settings with: BOOKSTORE_DB_PASSWORD=s3cr3t
 * BOOKSTORE_FEATURED_TITLE="Programming in Scala" sbt "runMain
 * configlibrarymigration.CompleteExample"
 */
object CompleteExample extends App {

  case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

  case class BookstoreConfig(
    name: String,
    port: Int = 8080,
    featuredTitle: String,
    db: DbConfig,
    tags: List[String],
    discountPercent: Option[Int]
  )

  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  // Same file the PureConfig application read, minus the `${?DB_PASSWORD}` line.
  val application: Either[ConfigError, ConfigSource] =
    ConfigSource
      .fromFile("src/main/resources/configlibrarymigration/application.conf")
      .map(_.keyFormat(KeyFormat.KebabCase).prefix("bookstore"))

  // BOOKSTORE_FEATURED_TITLE, BOOKSTORE_DB_PASSWORD, ... override the file.
  val env: ConfigSource = EnvSource.keyFormat(KeyFormat.UpperSnakeCase).prefix("bookstore")

  application match {
    case Left(error) =>
      println(error.message)
    case Right(file) =>
      Config.loadWithProvenance[BookstoreConfig](env.orElse(file)) match {
        case Right(provenance) =>
          println(provenance.value)
          println(provenance.provenanceOf("featuredTitle"))
          println(provenance.dump())
        case Left(errors) =>
          errors.foreach(error => println(error.message))
      }
  }
}
```

## Running the Examples

All examples in this guide have runnable Scala files in the `config-examples` project, which reads its HOCON files from `config-examples/src/main/resources/configlibrarymigration/`. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/config-examples
```

Each example is self-contained: it declares the bookstore case classes and the helpers it uses, so any file can be copied on its own. Run them in order to follow the guide.

### Reading Your Existing application.conf

This example parses the unchanged file with `ConfigSource.fromHocon` and loads it, reporting the kebab-case key and the missing password.

<details>
  <summary>config-examples/src/main/scala/configlibrarymigration/Step1ReadApplicationConfExample.scala</summary>

```scala title="config-examples/src/main/scala/configlibrarymigration/Step1ReadApplicationConfExample.scala" showLineNumbers
package configlibrarymigration

/**
 * Coming from PureConfig, Typesafe Config or zio-config — Step 1: Read Your
 * Existing application.conf
 *
 * Parses the unchanged PureConfig application.conf with config-hocon, scopes it
 * to the `bookstore` block with ConfigSource#prefix, and loads it. The load
 * reports the kebab-case `featured-title` key and the `${?DB_PASSWORD}`
 * substitution, which the next steps fix.
 *
 * Run with: cd config-examples && sbt "runMain
 * configlibrarymigration.Step1ReadApplicationConfExample"
 */
object Step1ReadApplicationConfExample extends App {

  case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

  case class BookstoreConfig(
    name: String,
    port: Int = 8080,
    featuredTitle: String,
    db: DbConfig,
    tags: List[String],
    discountPercent: Option[Int]
  )

  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  // The application.conf a PureConfig application already ships, unchanged.
  val applicationConf: String =
    """bookstore {
      |  name = "Scala Bookstore"
      |  port = 8080
      |  featured-title = "Zionomicon"
      |  discount-percent = 10
      |
      |  db {
      |    host = "localhost"
      |    port = 5432
      |    user = "bookstore"
      |    password = ${?DB_PASSWORD}
      |  }
      |
      |  tags = ["scala", "functional-programming", "zio"]
      |}
      |""".stripMargin

  def report[A](result: Either[::[ConfigError], A]): Unit =
    result match {
      case Right(config) => println(config)
      case Left(errors)  => errors.foreach(error => println(error.message))
    }

  def orThrow(parsed: Either[ConfigError, ConfigSource]): ConfigSource =
    parsed match {
      case Right(source) => source
      case Left(error)   => throw error
    }

  val hocon: ConfigSource = orThrow(ConfigSource.fromHocon(applicationConf, "application.conf"))

  // `bookstore.featured-title` is looked up as `bookstore.featuredTitle`, so it is missing.
  // `discount-percent` is missing too, but `discountPercent` is an Option, so it silently
  // becomes None instead of an error.
  report(Config.load[BookstoreConfig](hocon.prefix("bookstore")))
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain configlibrarymigration.Step1ReadApplicationConfExample"
```

### Keeping Your Kebab-Case Keys

This example wraps the source with `KeyFormat.KebabCase`, leaving only the password missing.

<details>
  <summary>config-examples/src/main/scala/configlibrarymigration/Step2KebabCaseKeysExample.scala</summary>

```scala title="config-examples/src/main/scala/configlibrarymigration/Step2KebabCaseKeysExample.scala" showLineNumbers
package configlibrarymigration

/**
 * Coming from PureConfig, Typesafe Config or zio-config — Step 2: Keep Your
 * Kebab-Case Keys
 *
 * Wraps the HOCON source with ConfigSource#keyFormat(KeyFormat.KebabCase), so
 * the camelCase field `featuredTitle` reads the `featured-title` key, the way
 * PureConfig maps field names by default.
 *
 * Run with: cd config-examples && sbt "runMain
 * configlibrarymigration.Step2KebabCaseKeysExample"
 */
object Step2KebabCaseKeysExample extends App {

  case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

  case class BookstoreConfig(
    name: String,
    port: Int = 8080,
    featuredTitle: String,
    db: DbConfig,
    tags: List[String],
    discountPercent: Option[Int]
  )

  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  // The application.conf a PureConfig application already ships, unchanged.
  val applicationConf: String =
    """bookstore {
      |  name = "Scala Bookstore"
      |  port = 8080
      |  featured-title = "Zionomicon"
      |  discount-percent = 10
      |
      |  db {
      |    host = "localhost"
      |    port = 5432
      |    user = "bookstore"
      |    password = ${?DB_PASSWORD}
      |  }
      |
      |  tags = ["scala", "functional-programming", "zio"]
      |}
      |""".stripMargin

  def report[A](result: Either[::[ConfigError], A]): Unit =
    result match {
      case Right(config) => println(config)
      case Left(errors)  => errors.foreach(error => println(error.message))
    }

  def orThrow(parsed: Either[ConfigError, ConfigSource]): ConfigSource =
    parsed match {
      case Right(source) => source
      case Left(error)   => throw error
    }

  val hocon: ConfigSource = orThrow(ConfigSource.fromHocon(applicationConf, "application.conf"))

  val application: ConfigSource = hocon.keyFormat(KeyFormat.KebabCase).prefix("bookstore")

  // Only the password is still missing: `${?DB_PASSWORD}` never reads the environment.
  report(Config.load[BookstoreConfig](application))
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain configlibrarymigration.Step2KebabCaseKeysExample"
```

### Replacing `${?VAR}` With an Environment Layer

This example layers the real environment over the file with `EnvSource`, scoped to `BOOKSTORE_*` variables and split into upper snake case.

<details>
  <summary>config-examples/src/main/scala/configlibrarymigration/Step3EnvironmentLayerExample.scala</summary>

```scala title="config-examples/src/main/scala/configlibrarymigration/Step3EnvironmentLayerExample.scala" showLineNumbers
package configlibrarymigration

/**
 * Coming from PureConfig, Typesafe Config or zio-config — Step 3: Replace
 * `${?VAR}` Substitutions With an Environment Layer
 *
 * Layers the real environment over the HOCON file with ConfigSource#orElse.
 * EnvSource is scoped to BOOKSTORE_* variables with ConfigSource#prefix, and
 * KeyFormat.UpperSnakeCase splits camelCase field names, so `featuredTitle` is
 * read from BOOKSTORE_FEATURED_TITLE and `db.password` from
 * BOOKSTORE_DB_PASSWORD.
 *
 * Run with: cd config-examples && BOOKSTORE_DB_PASSWORD=s3cr3t sbt "runMain
 * configlibrarymigration.Step3EnvironmentLayerExample"
 */
object Step3EnvironmentLayerExample extends App {

  case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

  case class BookstoreConfig(
    name: String,
    port: Int = 8080,
    featuredTitle: String,
    db: DbConfig,
    tags: List[String],
    discountPercent: Option[Int]
  )

  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  // The application.conf a PureConfig application already ships, unchanged.
  val applicationConf: String =
    """bookstore {
      |  name = "Scala Bookstore"
      |  port = 8080
      |  featured-title = "Zionomicon"
      |  discount-percent = 10
      |
      |  db {
      |    host = "localhost"
      |    port = 5432
      |    user = "bookstore"
      |    password = ${?DB_PASSWORD}
      |  }
      |
      |  tags = ["scala", "functional-programming", "zio"]
      |}
      |""".stripMargin

  def report[A](result: Either[::[ConfigError], A]): Unit =
    result match {
      case Right(config) => println(config)
      case Left(errors)  => errors.foreach(error => println(error.message))
    }

  def orThrow(parsed: Either[ConfigError, ConfigSource]): ConfigSource =
    parsed match {
      case Right(source) => source
      case Left(error)   => throw error
    }

  val application: ConfigSource =
    orThrow(ConfigSource.fromHocon(applicationConf, "application.conf"))
      .keyFormat(KeyFormat.KebabCase)
      .prefix("bookstore")

  val env: ConfigSource = EnvSource.keyFormat(KeyFormat.UpperSnakeCase).prefix("bookstore")

  report(Config.load[BookstoreConfig](env.orElse(application)))
}
```

</details>

Run it from the `config-examples` directory, with the password and a featured title in the environment:

```bash
BOOKSTORE_DB_PASSWORD=s3cr3t BOOKSTORE_FEATURED_TITLE="Programming in Scala" sbt "runMain configlibrarymigration.Step3EnvironmentLayerExample"
```

### Loading the File on the JVM, Includes and All

This example reads `split/application.conf` with `ConfigSource.fromFile`, which includes `split/db.conf`, and shows the error for a missing file.

<details>
  <summary>config-examples/src/main/scala/configlibrarymigration/Step4LoadFileWithIncludesExample.scala</summary>

```scala title="config-examples/src/main/scala/configlibrarymigration/Step4LoadFileWithIncludesExample.scala" showLineNumbers
package configlibrarymigration

/**
 * Coming from PureConfig, Typesafe Config or zio-config — Step 4: Load the File
 * on the JVM, Includes and All
 *
 * Reads src/main/resources/configlibrarymigration/split/application.conf with
 * the JVM-only ConfigSource.fromFile, which resolves its `include "db.conf"`
 * relative to the including file, then loads it with the kebab-case mapping.
 *
 * Run with: cd config-examples && sbt "runMain
 * configlibrarymigration.Step4LoadFileWithIncludesExample"
 */
object Step4LoadFileWithIncludesExample extends App {

  case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

  case class BookstoreConfig(
    name: String,
    port: Int = 8080,
    featuredTitle: String,
    db: DbConfig,
    tags: List[String],
    discountPercent: Option[Int]
  )

  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  def report[A](result: Either[::[ConfigError], A]): Unit =
    result match {
      case Right(config) => println(config)
      case Left(errors)  => errors.foreach(error => println(error.message))
    }

  def orThrow(parsed: Either[ConfigError, ConfigSource]): ConfigSource =
    parsed match {
      case Right(source) => source
      case Left(error)   => throw error
    }

  val file: ConfigSource = orThrow(
    ConfigSource.fromFile("src/main/resources/configlibrarymigration/split/application.conf")
  )

  println(file.sourceId)

  report(Config.load[BookstoreConfig](file.keyFormat(KeyFormat.KebabCase).prefix("bookstore")))

  // A file that does not exist is a Left, not an exception.
  ConfigSource.fromFile("src/main/resources/configlibrarymigration/missing.conf") match {
    case Right(_)    => println("unexpected: the file exists")
    case Left(error) => println(error.message)
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain configlibrarymigration.Step4LoadFileWithIncludesExample"
```

### Checking Which HOCON Features Carry Over

This example parses the HOCON constructs that behave differently from Typesafe Config and loads a duration in both spellings.

<details>
  <summary>config-examples/src/main/scala/configlibrarymigration/Step5HoconCaveatsExample.scala</summary>

```scala title="config-examples/src/main/scala/configlibrarymigration/Step5HoconCaveatsExample.scala" showLineNumbers
package configlibrarymigration

/**
 * Coming from PureConfig, Typesafe Config or zio-config — Step 5: Check Which
 * HOCON Features Carry Over
 *
 * Shows the HOCON constructs that behave differently in config-hocon than in
 * Typesafe Config: `${?VAR}` never reads the environment, includes other than
 * `include "file"` are parse errors, `fromHocon` drops includes, array
 * concatenation becomes an empty string, and durations must be ISO-8601.
 *
 * Run with: cd config-examples && sbt "runMain
 * configlibrarymigration.Step5HoconCaveatsExample"
 */
object Step5HoconCaveatsExample extends App {

  def report[A](result: Either[::[ConfigError], A]): Unit =
    result match {
      case Right(config) => println(config)
      case Left(errors)  => errors.foreach(error => println(error.message))
    }

  def orThrow(parsed: Either[ConfigError, ConfigSource]): ConfigSource =
    parsed match {
      case Right(source) => source
      case Left(error)   => throw error
    }

  def keys(hocon: String): Unit =
    ConfigSource.fromHocon(hocon) match {
      case Right(source) =>
        println(source.all("").map { case (key, value) => s"$key = '${value.value}'" }.toList.sorted)
      case Left(error) => println(error.message)
    }

  // HOME is set in almost every environment, but `${?HOME}` only looks inside the document.
  keys("home = ${?HOME}")

  // Only the quoted `include "file"` form parses.
  keys("""include classpath("reference.conf")""")

  // From a string there is nowhere to read an include from, so it is silently dropped.
  keys("""db { include "db.conf" }, name = "Scala Bookstore"""")

  // Array concatenation is not supported: the value becomes an empty string.
  keys("tags = [scala] [zio]")

  // `+=` appends to an array, and substitutions inside the document work.
  keys("""tags = [scala], tags += zio, base = "/srv", data = ${base}"/data"""")

  case class Timeouts(requestTimeout: java.time.Duration)

  object Timeouts {
    implicit val schema: Schema[Timeouts] = Schema.derived
  }

  // Durations are parsed with java.time.Duration.parse, so `10s` must become `PT10S`.
  report(Config.load[Timeouts](orThrow(ConfigSource.fromHocon("requestTimeout = 10s"))))
  report(Config.load[Timeouts](orThrow(ConfigSource.fromHocon("requestTimeout = PT10S"))))
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain configlibrarymigration.Step5HoconCaveatsExample"
```

### Reading Every Error at Once

This example loads a broken file and prints every error from one load.

<details>
  <summary>config-examples/src/main/scala/configlibrarymigration/Step6ErrorsExample.scala</summary>

```scala title="config-examples/src/main/scala/configlibrarymigration/Step6ErrorsExample.scala" showLineNumbers
package configlibrarymigration

/**
 * Coming from PureConfig, Typesafe Config or zio-config — Step 6: Read Every
 * Error at Once
 *
 * Loads a broken application.conf, layered under an environment stand-in, and
 * prints every error the load collected: a missing key and two invalid values,
 * each named with the whole layered source.
 *
 * Run with: cd config-examples && sbt "runMain
 * configlibrarymigration.Step6ErrorsExample"
 */
object Step6ErrorsExample extends App {

  case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

  case class BookstoreConfig(
    name: String,
    port: Int = 8080,
    featuredTitle: String,
    db: DbConfig,
    tags: List[String],
    discountPercent: Option[Int]
  )

  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  def report[A](result: Either[::[ConfigError], A]): Unit =
    result match {
      case Right(config) => println(config)
      case Left(errors)  => errors.foreach(error => println(error.message))
    }

  def orThrow(parsed: Either[ConfigError, ConfigSource]): ConfigSource =
    parsed match {
      case Right(source) => source
      case Left(error)   => throw error
    }

  val brokenConf =
    """bookstore {
      |  name = "Scala Bookstore"
      |  port = eighty
      |  discount-percent = "ten"
      |  db { host = "localhost", user = "bookstore" }
      |  tags = ["scala"]
      |}
      |""".stripMargin

  val broken: ConfigSource =
    orThrow(ConfigSource.fromHocon(brokenConf, "application.conf")).keyFormat(KeyFormat.KebabCase).prefix("bookstore")

  val env: ConfigSource = ConfigSource.fromMap(Map.empty, "env")

  report(Config.load[BookstoreConfig](env.orElse(broken)))
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain configlibrarymigration.Step6ErrorsExample"
```

### Keeping Secrets Out of Logs

This example prints the configuration and its provenance table with the password redacted, and reads the password with `Secret.unwrap`.

<details>
  <summary>config-examples/src/main/scala/configlibrarymigration/Step7SecretsExample.scala</summary>

```scala title="config-examples/src/main/scala/configlibrarymigration/Step7SecretsExample.scala" showLineNumbers
package configlibrarymigration

/**
 * Coming from PureConfig, Typesafe Config or zio-config — Step 7: Keep Secrets
 * Out of Logs
 *
 * Loads the layered configuration with Config.loadWithProvenance and prints the
 * configuration and its provenance table. The password is a Secret, so the
 * configuration prints it as <secret>, and ProvenanceMap#dump hides it because
 * its key name contains "password". Secret.unwrap reads the real value.
 *
 * Run with: cd config-examples && sbt "runMain
 * configlibrarymigration.Step7SecretsExample"
 */
object Step7SecretsExample extends App {

  case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

  case class BookstoreConfig(
    name: String,
    port: Int = 8080,
    featuredTitle: String,
    db: DbConfig,
    tags: List[String],
    discountPercent: Option[Int]
  )

  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  // The application.conf a PureConfig application already ships, unchanged.
  val applicationConf: String =
    """bookstore {
      |  name = "Scala Bookstore"
      |  port = 8080
      |  featured-title = "Zionomicon"
      |  discount-percent = 10
      |
      |  db {
      |    host = "localhost"
      |    port = 5432
      |    user = "bookstore"
      |    password = ${?DB_PASSWORD}
      |  }
      |
      |  tags = ["scala", "functional-programming", "zio"]
      |}
      |""".stripMargin

  def orThrow(parsed: Either[ConfigError, ConfigSource]): ConfigSource =
    parsed match {
      case Right(source) => source
      case Left(error)   => throw error
    }

  val application: ConfigSource =
    orThrow(ConfigSource.fromHocon(applicationConf, "application.conf"))
      .keyFormat(KeyFormat.KebabCase)
      .prefix("bookstore")

  // Stands in for EnvSource so the output does not depend on the machine.
  val env: ConfigSource = ConfigSource.fromMap(Map("db.password" -> "s3cr3t-from-env", "port" -> "9090"), "env")

  Config.loadWithProvenance[BookstoreConfig](env.orElse(application)) match {
    case Right(provenance) =>
      println(provenance.value)
      println(provenance.dump())
      println(Secret.unwrap(provenance.value.db.password).length)
    case Left(errors) =>
      errors.foreach(error => println(error.message))
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain configlibrarymigration.Step7SecretsExample"
```

### The Complete Example

The complete example from "Putting It Together" reads the real environment, so the password must be exported. Any other `BOOKSTORE_*` variable overrides the file:

```bash
BOOKSTORE_DB_PASSWORD=s3cr3t sbt "runMain configlibrarymigration.CompleteExample"
BOOKSTORE_DB_PASSWORD=s3cr3t BOOKSTORE_FEATURED_TITLE="Programming in Scala" sbt "runMain configlibrarymigration.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## Going Further

- [Build a Typed Configuration Loader](./typed-config-loader.md) teaches the loader from scratch with the same bookstore: defaults, optional fields, layering and provenance.
- [How to Load Configuration Files on the JVM and in the Browser](./load-configuration-files.md) goes deeper on includes, safe loading and running the same settings in the browser.
- [File Formats](../reference/config/formats.md) documents the HOCON adapter's parameters and failures, next to the YAML and JSON adapters.
- [ConfigSource](../reference/config/config-source.md) covers every source, `ConfigSource#prefix`, key formats and secret redaction.
- [ConfigDecoder](../reference/config/config-decoder.md) lists how fields map to keys for nested records, sealed traits, lists and maps.
- [Configuration Errors](../reference/config/errors.md) describes every `ConfigError` case and how errors combine.
- [Flags](../reference/config/flags.md) and [Rollout](../reference/config/rollout.md) cover feature flags and percentage rollouts, built on the same sources.
- The [Configuration & Feature Flags](../reference/config/index.md) overview shows how the pieces fit, including `Config.wire` for dependency graphs.

---

## How to Connect to PostgreSQL with a Connection Pool

In this guide, we connect the ZIO Blocks `sql` module to a PostgreSQL server through a HikariCP connection pool. By the end, you will have a [`JdbcTransactor`](../reference/sql/transactor.md) that borrows connections from a pool configured from environment variables, a health check, a bookstore repository running on PostgreSQL, and a pool that closes when your program is done with it.

We start from an empty build and add one piece at a time:

- Add the PostgreSQL driver and HikariCP to the build, and start a local server
- Configure the pool from the environment
- Build a `JdbcTransactor` on the pool and check that the database answers
- Run a bookstore repository on PostgreSQL
- Size the pool for many concurrent callers, including virtual threads
- Pick an isolation level and read-only mode for a report
- Close the pool with a `Scope`
- Recognise the common pool and connection errors

This guide assumes you know the basics of `Table`, `Repo` and the `sql"..."` interpolator from [Build Your First Database Repository on SQLite](./your-first-sqlite-repository.md); we don't explain them again here.

## The Problem

Opening a PostgreSQL connection is expensive: the driver opens a TCP connection, negotiates TLS if it is enabled, and authenticates, and the server starts a new backend process for it. A program that opens a fresh connection for every unit of work pays that cost every time, and under load it can open more connections than the server accepts — PostgreSQL refuses new connections once it reaches its `max_connections` setting.

For example, this is plain JDBC code that counts the books in a table:

```scala

def countBooks(url: String, user: String, password: String): Long = {
  val connection = DriverManager.getConnection(url, user, password)
  try {
    val statement = connection.createStatement()
    try {
      val rows = statement.executeQuery("SELECT count(*) FROM book")
      rows.next()
      rows.getLong(1)
    } finally statement.close()
  } finally connection.close()
}
```

Every call to `countBooks` opens and closes a physical connection. A web server that calls it from 200 requests at once asks PostgreSQL for 200 connections. `JdbcTransactor.fromUrl` behaves the same way: it calls `DriverManager.getConnection` for every `JdbcTransactor#connect` or `JdbcTransactor#transact` and does no pooling.

A connection pool solves this. It opens a fixed number of connections up front and lends them out: "closing" a pooled connection returns it to the pool instead of disconnecting. In this guide, we let HikariCP own the connections and give ZIO Blocks a `JdbcTransactor` that borrows from it.

## Prerequisites

Add the `sql` module, the PostgreSQL JDBC driver and HikariCP to your `build.sbt`:

```scala
// Scala 3.3 LTS
libraryDependencies += "dev.zio" %% "zio-blocks-sql" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-sql" % "0.0.56"

// The PostgreSQL JDBC driver and the HikariCP pool, for every Scala version
libraryDependencies += "org.postgresql" % "postgresql" % "42.7.14"
libraryDependencies += "com.zaxxer"     % "HikariCP"   % "7.1.0"
```

The `sql` module is published for Scala 3 only, and `JdbcTransactor` runs on the JVM. HikariCP 7 needs Java 11 or newer. HikariCP logs through SLF4J, so add an SLF4J provider such as `slf4j-simple` or Logback if you want to see its pool messages; without one, SLF4J prints a warning at startup and drops the log lines.

To follow along, you need a PostgreSQL server. If you have Docker, this command starts PostgreSQL 16 with a `bookstore` database, user and password:

```bash
docker run --rm -d --name bookstore-db -p 5432:5432 \
  -e POSTGRES_USER=bookstore -e POSTGRES_PASSWORD=bookstore -e POSTGRES_DB=bookstore \
  postgres:16
```

The code in this guide reads the password from the `BOOKSTORE_DB_PASSWORD` environment variable, so set it in the shell that runs your program:

```bash
export BOOKSTORE_DB_PASSWORD=bookstore
```

## Configure the Pool from the Environment

A `HikariConfig` holds the pool's settings: where the database is, who we log in as, and how the pool behaves. We fill it from environment variables, so the password never appears in the source code or in version control:

```scala

def bookstorePoolConfig(): HikariConfig = {
  val config = new HikariConfig()
  config.setJdbcUrl(sys.env.getOrElse("BOOKSTORE_DB_URL", "jdbc:postgresql://localhost:5432/bookstore"))
  config.setUsername(sys.env.getOrElse("BOOKSTORE_DB_USER", "bookstore"))
  config.setPassword(sys.env("BOOKSTORE_DB_PASSWORD"))
  config.setPoolName("bookstore")
  config.setMaximumPoolSize(10)
  config.setConnectionTimeout(5000)
  config
}
```

The code above sets the JDBC URL and user name from `BOOKSTORE_DB_URL` and `BOOKSTORE_DB_USER`, with defaults that match the Docker command. The password has no default: `sys.env("BOOKSTORE_DB_PASSWORD")` throws a `NoSuchElementException` naming the variable when it is missing, so a misconfigured deployment fails at startup. `HikariConfig#setPoolName` names the pool in log lines and error messages. `HikariConfig#setMaximumPoolSize` caps the pool at 10 connections, which is also HikariCP's default; we size it properly in [Size the Pool for Many Concurrent Callers](#size-the-pool-for-many-concurrent-callers). `HikariConfig#setConnectionTimeout` makes a caller wait at most 5 seconds for a free connection instead of the default 30 seconds.

:::tip
If you load your settings with [ZIO Blocks Config](../reference/config/index.md), as in [Build a Typed Configuration Loader](./typed-config-loader.md), read the password as a [`Secret`](../reference/config/config-source.md#secrets). `Secret#toString` is always `<secret>`, so the password can't leak into logs, and `Secret.unwrap` gives you the `String` to pass to `HikariConfig#setPassword`.
:::

Creating a `HikariDataSource` from the config starts the pool:

```scala
def openBookstorePool(): HikariDataSource =
  new HikariDataSource(bookstorePoolConfig())
```

HikariCP checks the database right away: by default, the constructor opens one connection and throws if it can't, so a wrong URL or password stops your program at startup instead of at the first query. The `HikariDataSource` is a `javax.sql.DataSource`, which is all ZIO Blocks needs from it.

## Build the Transactor on the Pool

`JdbcTransactor.postgres` creates a transactor that takes its connections from a `DataSource` and speaks the PostgreSQL dialect:

```scala

def bookstoreTransactor(pool: DataSource): JdbcTransactor =
  JdbcTransactor.postgres(pool)
```

:::note
The `Schema` and `scope` imports, used later on this page, come before `import zio.blocks.sql.*` on purpose. When the `sql-zio` module is also on the classpath, `zio.blocks.sql.*` brings its subpackage `zio.blocks.sql.zio` into scope under the name `zio`, and every `import zio.blocks...` after it fails to compile. Keeping `import zio.blocks.sql.*` last avoids this.
:::

`JdbcTransactor.postgres(pool)` is shorthand for `JdbcTransactor.fromDataSource(pool, SqlDialect.PostgreSQL)` and does nothing else: it runs no setup statements on the connections and changes none of their settings. Each `JdbcTransactor#connect` or `JdbcTransactor#transact` call borrows one connection with `DataSource#getConnection` and closes it at the end of the block, which returns it to the pool.

The `JdbcTransactor` factories differ in where the connections come from:

| Factory                                                  | Connections come from                      | Use it when                                                        |
|----------------------------------------------------------|--------------------------------------------|--------------------------------------------------------------------|
| `JdbcTransactor.postgres(dataSource)`                    | The `DataSource`, so the pool              | You run on PostgreSQL with a pool; the usual production choice     |
| `JdbcTransactor.fromDataSource(dataSource, dialect)`     | The `DataSource`, so the pool              | You choose the `SqlDialect` at runtime                             |
| `JdbcTransactor.fromUrl(url, dialect)`                   | `DriverManager`, a new connection per call | Scripts, tests and one-off tools, where the URL carries everything |
| `JdbcTransactor.fromUrl(url, user, password, dialect)`   | `DriverManager`, a new connection per call | The same, with the credentials passed separately                   |

:::note
A `JdbcTransactor` doesn't own its `DataSource` and isn't `AutoCloseable`, so there is nothing to close on it. The pool stays open until you close it yourself, as we do in [Close the Pool with a Scope](#close-the-pool-with-a-scope).
:::

## Check That the Database Answers

A health check runs the cheapest possible query and reports whether it worked. `SELECT 1` needs no table, and `Frag#queryOne[Int]` returns the single value as a `Maybe[Int]`:

```scala

def isHealthy(transactor: JdbcTransactor): Boolean =
  try transactor.connect(sql"SELECT 1".queryOne[Int].contains(1))
  catch { case _: SQLException => false }

def serverVersion()(using DbCon): String =
  sql"SELECT version()".queryOne[String].getOrElse("unknown")
```

The code above defines two functions. `isHealthy` borrows a connection, runs `SELECT 1` and checks the answer with `Maybe#contains`. When the database is down, or the pool can't hand out a connection within the connection timeout, the JDBC call throws an `SQLException` and `isHealthy` returns `false`. `serverVersion` asks PostgreSQL for its version string, which is a quick way to confirm which server the URL points to. It takes a `DbCon` from its caller instead of a transactor, like `findByTitle` in the SQLite tutorial, so we run it with `transactor.connect(serverVersion())`.

From here on, we write the database work as functions with a `using DbCon` parameter and leave the choice of `JdbcTransactor#connect` or `JdbcTransactor#transact`, and of the transaction settings, to the caller. This shape also lets us use the work with a transactor that a `Scope` manages, as we see in [Close the Pool with a Scope](#close-the-pool-with-a-scope).

## Run the Bookstore Repository on PostgreSQL

The repository code from the SQLite tutorial works unchanged on PostgreSQL. We use the same `Book` shape:

```scala
case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String])

object Book {
  given Schema[Book] = Schema.derived
}

val bookTable = Table.derived[Book]
val books     = Repo.derived[Book, Long]
```

The only thing that changes is the dialect we pass to `Table#createTable`. With `SqlDialect.PostgreSQL`, the column types are PostgreSQL types:

```scala
println(bookTable.createTable(SqlDialect.PostgreSQL).sql(SqlDialect.PostgreSQL))
// CREATE TABLE IF NOT EXISTS book (
//   id BIGINT NOT NULL,
//   title TEXT NOT NULL,
//   author TEXT NOT NULL,
//   year INTEGER NOT NULL,
//   isbn TEXT
// )
```

`Long` becomes `BIGINT`, `Int` becomes `INTEGER` and `String` becomes `TEXT`; on SQLite, both `Long` and `Int` map to `INTEGER`. To create the table and put three books on the shelf:

```scala
def stockShelf()(using DbCon): List[Book] = {
  bookTable.createTable(SqlDialect.PostgreSQL).update
  books.clear()
  books.insert(Book(1L, "Zionomicon", "John A. De Goes", 2024, None))
  books.insert(Book(2L, "Programming in Scala", "Martin Odersky", 2008, Some("9780981531601")))
  books.insert(Book(3L, "Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657")))
  books.all
}

def restockShelf(transactor: JdbcTransactor): List[Book] =
  transactor.transact(stockShelf())
```

The code above defines two functions. `stockShelf` runs the `CREATE TABLE IF NOT EXISTS` statement, empties the table with `Repo#clear` so it gives the same result every time we run it, inserts three books and reads them back with `Repo#all`. `restockShelf` runs `stockShelf` in one transaction. PostgreSQL can run DDL inside a transaction, so if any insert fails, the whole block is rolled back, the `CREATE TABLE` included.

:::caution
The generated `CREATE TABLE` declares no primary key, so PostgreSQL won't stop two rows from sharing an `id`. That is fine for a demo; in production, create your tables with a migration tool and declare the keys there. See [Table](../reference/sql/table.md) for what `Table#createTable` generates.
:::

## Size the Pool for Many Concurrent Callers

`JdbcTransactor` has no thread pool of its own. Every JDBC call blocks the thread that makes it, until PostgreSQL answers. Which threads call it is up to you, and the connection pool decides how many of them can use the database at once: with `maximumPoolSize` set to 10, the eleventh concurrent caller waits for a connection to come back, for at most the connection timeout.

To see this, we run many lookups at once, each holding its connection for 0.2 seconds. The function takes any `ExecutorService`, so it works with platform threads and virtual threads alike:

```scala

def runLookups(transactor: JdbcTransactor, executor: ExecutorService, callers: Int): Long = {
  val started = System.nanoTime()
  val lookups = (1 to callers).map { _ =>
    executor.submit[Int](() => transactor.connect(sql"SELECT 1 FROM pg_sleep(0.2)".queryOne[Int].getOrElse(0)))
  }
  lookups.foreach(_.get())
  (System.nanoTime() - started) / 1000000
}
```

The code above submits one task per caller, waits for all of them and returns the elapsed milliseconds. With 50 callers and 10 connections, the lookups run in five waves of ten, so the call takes a little over one second however many threads the executor has.

Virtual threads, final since JDK 21, make this style cheap: a virtual thread that blocks on JDBC gives its carrier thread back to the JVM, so thousands of waiting callers cost little memory. On JDK 21 or newer, give each caller its own virtual thread:

```scala
val executor = java.util.concurrent.Executors.newVirtualThreadPerTaskExecutor()
val elapsed  = runLookups(transactor, executor, callers = 50)
```

Before JDK 24, a virtual thread that blocked inside a `synchronized` block stayed pinned to its carrier thread. [JEP 491](https://openjdk.org/jeps/491) removed that limitation in JDK 24, and the PostgreSQL driver replaced its `synchronized` blocks with `ReentrantLock` in version 42.6.0, so blocking JDBC calls don't pin on any JDK that has virtual threads.

With virtual threads, threads are no longer the scarce resource; connections are. Size the pool with these rules:

- **Stay under the server limit.** The `maximumPoolSize` of every pool, across every instance of every application that uses the database, must add up to less than PostgreSQL's `max_connections` (100 by default), minus the connections it reserves for superusers. Three instances with a pool of 30 already use 90.
- **Start small.** A database server can only do as much work in parallel as its CPU cores and disks allow, so a bigger pool doesn't make it faster. HikariCP's [About Pool Sizing](https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing) page suggests `(core_count * 2) + effective_spindle_count` of the database server as a starting point, and recommends "a small pool, saturated with threads waiting for connections". Callers that wait briefly in the pool are cheaper than queries that compete for the same CPUs.
- **Keep connections short.** Every `JdbcTransactor#connect` or `JdbcTransactor#transact` block holds its connection until the block ends, so don't make HTTP calls or wait for user input inside one.

## Pick an Isolation Level and Read-Only Mode

The one-argument `JdbcTransactor#transact` keeps the driver's defaults, which on PostgreSQL is the `READ COMMITTED` isolation level with writes allowed. For a report that runs several queries, we want them all to see the same data, and we want the database to refuse accidental writes. The two-argument overload sets both:

```scala
def shelfReport()(using DbCon): (Long, List[Book]) =
  (books.count, books.all)

def readShelfReport(transactor: JdbcTransactor): (Long, List[Book]) =
  transactor.transact(TransactionIsolation.RepeatableRead, readOnly = true)(shelfReport())
```

The code above:

- `TransactionIsolation.RepeatableRead` gives the transaction one snapshot of the database, so `Repo#count` and `Repo#all` agree even when other transactions insert books in between.
- `readOnly = true` makes the PostgreSQL driver start the transaction with `BEGIN READ ONLY`, so an `INSERT`, `UPDATE` or `DELETE` in the block throws an `SQLException` such as `ERROR: cannot execute INSERT in a read-only transaction` instead of changing data.
- `JdbcTransactor#transact` restores the connection's previous isolation level and read-only flag before returning it to the pool, so the next borrower gets a clean connection.

Choose `TransactionIsolation.Serializable` when a transaction must behave as if it ran alone. PostgreSQL then aborts one of two conflicting transactions with a serialization failure (SQLSTATE `40001`), which your code must retry. [SQL Transactions](./sql-transactions.md) covers every isolation level on PostgreSQL and SQLite, and nested transactions with savepoints.

## Close the Pool with a Scope

A `HikariDataSource` holds open connections and a background housekeeping thread until we call `HikariDataSource#close`. A [`Scope`](../reference/resource-management/scope.md) ties that call to a block of code: `Resource.fromAutoCloseable` turns the pool into a resource that the scope closes when the block ends, whether it ends normally or with an exception:

```scala
def stockShelfWithScopedPool(): List[String] =
  Scope.global.scoped { scope =>

    val transactor: $[JdbcTransactor] =
      Resource.fromAutoCloseable(openBookstorePool())
        .map(JdbcTransactor.postgres)
        .allocate
    $(transactor)(_.transact(stockShelf()).map(_.title))
  }
```

The code above:

- `Resource.fromAutoCloseable(openBookstorePool())` describes how to open the pool and registers `HikariDataSource#close` as its finalizer.
- `Resource#map(JdbcTransactor.postgres)` builds the transactor on the pool, so the scope hands us a transactor whose pool it closes for us.
- `Resource#allocate` opens the pool in this scope and returns a scoped `$[JdbcTransactor]`, which can't escape the block.
- `$(transactor)(_.transact(stockShelf()).map(_.title))` runs `stockShelf` in a transaction and returns the titles as a plain `List[String]`. When `Scope#scoped` returns, the scope closes the pool.

Inside `$(transactor)(...)`, the transactor may only be the receiver of a method call. Passing it to a function, as in `$(transactor)(t => restockShelf(t))`, doesn't compile: the compiler reports that the parameter "cannot be passed as an argument to a function or method". That is why `stockShelf` takes a `DbCon` instead of a transactor: we call `JdbcTransactor#transact` on the scoped transactor and pass the work as the block.

In a long-running service, wrap the whole application in one `Scope#scoped` block, so the pool lives exactly as long as the service. [Compile-Time Resource Safety with Scope](./compile-time-resource-safety-with-scope.md) explains how scopes nest and why a scoped value can't leak out of its block.

## Troubleshooting

Most problems show up as an exception from `HikariDataSource`'s constructor or from the first `JdbcTransactor#connect`:

- **The pool doesn't start.** The `HikariDataSource` constructor throws `HikariPool.PoolInitializationException` when it can't open its first connection, and the message names the real problem. `Failed to initialize pool: Connection to localhost:5432 refused` means nothing listens at the host and port in the URL; `Failed to initialize pool: FATAL: password authentication failed for user "bookstore"` means the credentials are wrong.
- **`Connection is not available, request timed out`.** Every connection stays in use for the whole connection timeout, and HikariCP throws an `SQLTransientConnectionException` whose message ends with the pool's state, such as `(total=1, active=1, idle=0, waiting=0)`. Either the pool is too small for the load, or connections are held too long. A common cause is calling `JdbcTransactor#connect` or `JdbcTransactor#transact` inside another such block: the inner call borrows a second connection, and with a small pool, every thread can end up holding one connection while it waits for another. Inside a transaction, use the `DbTx` you already have, as [SQL Transactions](./sql-transactions.md) shows. HikariCP's `leakDetectionThreshold` setting logs a warning with a stack trace for any connection that stays out of the pool longer than the threshold.
- **The connection must be encrypted.** The PostgreSQL driver's default `sslmode` is `prefer`, which encrypts only when the server offers TLS and never checks the server's certificate. To require TLS and verify both the certificate and the host name, add the parameters to the URL, for example `jdbc:postgresql://db.example.com:5432/bookstore?sslmode=verify-full&sslrootcert=/etc/ssl/certs/bookstore-root.crt`. `sslmode=require` encrypts without verifying the certificate. The [PostgreSQL JDBC SSL documentation](https://jdbc.postgresql.org/documentation/ssl/) lists every mode.

To make HikariCP log a warning for any connection held for more than 10 seconds, set the leak detection threshold on the config:

```scala
def leakDetectingPoolConfig(): HikariConfig = {
  val config = bookstorePoolConfig()
  config.setLeakDetectionThreshold(10000)
  config
}
```

## Putting It Together

The complete example opens the pool in a scope, prints the server version, stocks the shelf and prints a read-only report:

```scala title="sql-examples/src/main/scala/postgrespool/CompleteExample.scala"
package postgrespool

/**
 * How to Connect to PostgreSQL with a Connection Pool: Complete Example
 *
 * Opens the pool in a Scope, prints the server version, stocks the shelf and
 * prints a read-only report. The scope closes the pool when the block ends.
 *
 * Run with: cd sql-examples && sbt "runMain postgrespool.CompleteExample"
 */
object CompleteExample extends App {

  Scope.global.scoped { scope =>

    val transactor: $[JdbcTransactor] =
      Resource
        .fromAutoCloseable(openBookstorePool())
        .map(JdbcTransactor.postgres)
        .allocate

    val version = $(transactor)(_.connect(serverVersion()))
    println(s"Server version: $version")

    val stocked = $(transactor)(_.transact(stockShelf()).map(_.title))
    println(s"Stocked: ${stocked.mkString(", ")}")

    val report = $(transactor)(_.transact(TransactionIsolation.RepeatableRead, readOnly = true) {
      val (count, shelf) = shelfReport()
      s"$count books, newest from ${shelf.map(_.year).max}"
    })
    println(s"Read-only report: $report")
  }
}
```

## Running the Examples

All examples in this guide have runnable Scala files in the `sql-examples` project. They need a running PostgreSQL server, so start one with the Docker command from [Prerequisites](#prerequisites) and set `BOOKSTORE_DB_PASSWORD` first. Then clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/sql-examples
```

The examples share the pool settings and the `Book` repository, which live in one file:

<details>
  <summary>sql-examples/src/main/scala/postgrespool/Bookstore.scala</summary>

```scala title="sql-examples/src/main/scala/postgrespool/Bookstore.scala" showLineNumbers
package postgrespool

/**
 * How to Connect to PostgreSQL with a Connection Pool: shared setup
 *
 * The pool settings read from the environment, the health check, the Book case
 * class with its Schema, the Table and Repo for it, and the database work that
 * the examples in this package share.
 */
def bookstorePoolConfig(): HikariConfig = {
  val config = new HikariConfig()
  config.setJdbcUrl(sys.env.getOrElse("BOOKSTORE_DB_URL", "jdbc:postgresql://localhost:5432/bookstore"))
  config.setUsername(sys.env.getOrElse("BOOKSTORE_DB_USER", "bookstore"))
  config.setPassword(sys.env("BOOKSTORE_DB_PASSWORD"))
  config.setPoolName("bookstore")
  config.setMaximumPoolSize(10)
  config.setConnectionTimeout(5000)
  config
}

def openBookstorePool(): HikariDataSource =
  new HikariDataSource(bookstorePoolConfig())

def isHealthy(transactor: JdbcTransactor): Boolean =
  try transactor.connect(sql"SELECT 1".queryOne[Int].contains(1))
  catch { case _: SQLException => false }

def serverVersion()(using DbCon): String =
  sql"SELECT version()".queryOne[String].getOrElse("unknown")

case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String])

object Book {
  given Schema[Book] = Schema.derived
}

val bookTable = Table.derived[Book]
val books     = Repo.derived[Book, Long]

def stockShelf()(using DbCon): List[Book] = {
  bookTable.createTable(SqlDialect.PostgreSQL).update
  books.clear()
  books.insert(Book(1L, "Zionomicon", "John A. De Goes", 2024, None))
  books.insert(Book(2L, "Programming in Scala", "Martin Odersky", 2008, Some("9780981531601")))
  books.insert(Book(3L, "Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657")))
  books.all
}

def shelfReport()(using DbCon): (Long, List[Book]) =
  (books.count, books.all)
```

</details>

### Connect and Check the Database

This example opens the pool, builds the transactor, runs the health check and prints the server version.

<details>
  <summary>sql-examples/src/main/scala/postgrespool/ConnectExample.scala</summary>

```scala title="sql-examples/src/main/scala/postgrespool/ConnectExample.scala" showLineNumbers
package postgrespool

/**
 * How to Connect to PostgreSQL with a Connection Pool: Connect and Check the
 * Database
 *
 * Opens the pool, builds a JdbcTransactor on it, runs the health check, prints
 * the server version and closes the pool.
 *
 * Run with: cd sql-examples && sbt "runMain postgrespool.ConnectExample"
 */
object ConnectExample extends App {

  val pool = openBookstorePool()
  try {
    val transactor = JdbcTransactor.postgres(pool)
    println(s"Healthy: ${isHealthy(transactor)}")
    println(s"Server version: ${transactor.connect(serverVersion())}")
  } finally pool.close()
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "runMain postgrespool.ConnectExample"
```

### Run the Repository

This example creates the `book` table on PostgreSQL, stocks the shelf and lists the books.

<details>
  <summary>sql-examples/src/main/scala/postgrespool/RepositoryExample.scala</summary>

```scala title="sql-examples/src/main/scala/postgrespool/RepositoryExample.scala" showLineNumbers
package postgrespool

/**
 * How to Connect to PostgreSQL with a Connection Pool: Run the Repository
 *
 * Prints the CREATE TABLE statement for PostgreSQL, creates the book table,
 * stocks the shelf and lists the books.
 *
 * Run with: cd sql-examples && sbt "runMain postgrespool.RepositoryExample"
 */
object RepositoryExample extends App {

  println(bookTable.createTable(SqlDialect.PostgreSQL).sql(SqlDialect.PostgreSQL))

  val pool = openBookstorePool()
  try {
    val transactor = JdbcTransactor.postgres(pool)
    transactor.transact(stockShelf()).foreach(println)
  } finally pool.close()
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "runMain postgrespool.RepositoryExample"
```

### Many Concurrent Callers

This example runs 50 lookups at once on a pool of 10 connections and prints how long they took.

<details>
  <summary>sql-examples/src/main/scala/postgrespool/ConcurrentCallersExample.scala</summary>

```scala title="sql-examples/src/main/scala/postgrespool/ConcurrentCallersExample.scala" showLineNumbers
package postgrespool

/**
 * How to Connect to PostgreSQL with a Connection Pool: Many Concurrent Callers
 *
 * Runs 50 lookups at once on a pool of 10 connections, each holding its
 * connection for 0.2 seconds, and prints how long they took.
 *
 * Run with: cd sql-examples && sbt "runMain
 * postgrespool.ConcurrentCallersExample"
 */
object ConcurrentCallersExample {

  def runLookups(transactor: JdbcTransactor, executor: ExecutorService, callers: Int): Long = {
    val started = System.nanoTime()
    val lookups = (1 to callers).map { _ =>
      executor.submit[Int](() => transactor.connect(sql"SELECT 1 FROM pg_sleep(0.2)".queryOne[Int].getOrElse(0)))
    }
    lookups.foreach(_.get())
    (System.nanoTime() - started) / 1000000
  }

  def main(args: Array[String]): Unit = {
    val pool = openBookstorePool()
    // On JDK 21 or newer, Executors.newVirtualThreadPerTaskExecutor() gives each caller a virtual thread.
    // This example uses platform threads, so it also compiles on JDK 17.
    val executor = Executors.newFixedThreadPool(50)
    try {
      val transactor = JdbcTransactor.postgres(pool)
      val elapsed    = runLookups(transactor, executor, callers = 50)
      println(s"50 callers took $elapsed ms")
      println(s"Pool size: ${pool.getMaximumPoolSize} connections")
    } finally {
      executor.shutdown()
      pool.close()
    }
  }
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "runMain postgrespool.ConcurrentCallersExample"
```

### Read-Only Report

This example prints a report from a `RepeatableRead`, read-only transaction, then shows the error PostgreSQL raises for a write inside one.

<details>
  <summary>sql-examples/src/main/scala/postgrespool/ReadOnlyReportExample.scala</summary>

```scala title="sql-examples/src/main/scala/postgrespool/ReadOnlyReportExample.scala" showLineNumbers
package postgrespool

/**
 * How to Connect to PostgreSQL with a Connection Pool: Read-Only Report
 *
 * Prints a report from a RepeatableRead, read-only transaction, then shows the
 * error PostgreSQL raises for a write inside a read-only transaction.
 *
 * Run with: cd sql-examples && sbt "runMain postgrespool.ReadOnlyReportExample"
 */
object ReadOnlyReportExample extends App {

  val pool = openBookstorePool()
  try {
    val transactor = JdbcTransactor.postgres(pool)
    transactor.transact(stockShelf())

    val (count, shelf) = transactor.transact(TransactionIsolation.RepeatableRead, readOnly = true)(shelfReport())
    println(s"The shelf holds $count books:")
    shelf.foreach(println)

    try
      transactor.transact(TransactionIsolation.ReadCommitted, readOnly = true) {
        books.insert(Book(4L, "Scala with Cats", "Noel Welsh", 2020, None))
      }
    catch {
      case e: SQLException => println(s"Write rejected: ${e.getMessage}")
    }
  } finally pool.close()
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "runMain postgrespool.ReadOnlyReportExample"
```

### Pool Exhaustion

This example uses a pool of one connection and borrows a second connection inside a transaction, so it prints the timeout error from [Troubleshooting](#troubleshooting).

<details>
  <summary>sql-examples/src/main/scala/postgrespool/PoolExhaustionExample.scala</summary>

```scala title="sql-examples/src/main/scala/postgrespool/PoolExhaustionExample.scala" showLineNumbers
package postgrespool

/**
 * How to Connect to PostgreSQL with a Connection Pool: Pool Exhaustion
 *
 * Uses a pool of one connection and borrows a second connection inside a
 * transaction, so the second borrow times out.
 *
 * Run with: cd sql-examples && sbt "runMain postgrespool.PoolExhaustionExample"
 */
object PoolExhaustionExample extends App {

  val config = bookstorePoolConfig()
  config.setMaximumPoolSize(1)
  config.setConnectionTimeout(1000)

  val pool = new HikariDataSource(config)
  try {
    val transactor = JdbcTransactor.postgres(pool)
    try
      transactor.transact {
        transactor.connect(sql"SELECT 1".queryOne[Int].getOrElse(0))
      }
    catch {
      case e: SQLException => println(s"${e.getClass.getSimpleName}: ${e.getMessage}")
    }
  } finally pool.close()
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "runMain postgrespool.PoolExhaustionExample"
```

### The Complete Example

The complete example from "Putting It Together" runs everything from this guide in one program:

```bash
sbt "runMain postgrespool.CompleteExample"
```

To compile every example without running them, which needs no database:

```bash
sbt compile
```

When you are done, stop the database container:

```bash
docker stop bookstore-db
```

## Going Further

- [Transactor](../reference/sql/transactor.md) lists every `JdbcTransactor` factory and `JdbcTransactor#transact` overload.
- [SQL Transactions](./sql-transactions.md) covers isolation levels, read-only transactions and nested transactions with savepoints.
- [SqlDialect](../reference/sql/sql-dialect.md) shows how the PostgreSQL dialect maps each column type.
- [TransactorZIO](../reference/sql/transactor-zio.md) runs the same work as ZIO effects, if your application uses ZIO.
- The [SQL Module](../reference/sql/index.md) overview links to every `sql` reference page.

---

## How to Correlate Logs and Traces Across Services

One request to a shop crosses two services: the checkout service takes the order and calls the payment service over HTTP. In this guide, we connect the logs and spans of both services, so that every log line written while handling one order carries the same trace id, whichever service wrote it. We use the `telemetry` module for logs and spans, the `otel` module's propagators to carry the trace context in HTTP headers, the `http-model` module's `Request` as the HTTP message, and the `async` module to run work on a background thread.

We take it in five moves:

- Share one context storage between the tracer and the logger, so log lines carry the active span's ids.
- Write the trace context into the checkout service's outgoing request as a W3C `traceparent` header.
- Read it back in the payment service and continue the same trace.
- Carry the span context to work that runs on another thread, with `OtelContext`.
- Switch the header format to B3, and handle requests that arrive without a usable header.

## The Problem

A log line from the payment service says "Card charged", and a log line from the checkout service says "Order received". When a customer reports a failed order, you need every line that belongs to that one order, from both services, and nothing else. Timestamps don't do it: under load, hundreds of orders interleave in the same second.

The usual home-grown fix is a request id that every function passes along by hand and every service copies into a custom header:

```scala

object Checkout {
  def placeOrder(orderId: String): Unit = {
    val requestId = UUID.randomUUID().toString
    println(s"[$requestId] checkout: order $orderId received")
    Payment.charge(orderId, Map("X-Request-Id" -> requestId))
    println(s"[$requestId] checkout: payment answered")
  }
}

object Payment {
  def charge(orderId: String, headers: Map[String, String]): Unit = {
    val requestId = headers.getOrElse("X-Request-Id", UUID.randomUUID().toString)
    println(s"[$requestId] payment: card charged for $orderId")
    Fraud.check(orderId, requestId)
  }
}

object Fraud {
  def check(orderId: String, requestId: String): Unit =
    println(s"[$requestId] fraud: order $orderId looks fine")
}
```

Every method signature now carries `requestId`, and every log call has to remember to print it. One forgotten parameter and that code's lines drop out of the search. The id says nothing about which step ran inside which, or how long each step took, and `X-Request-Id` is a private convention: a service written by another team, in another language, won't read it.

Tracing already solves the "which steps belong together" half: a trace is one id shared by all the spans of one request. What remains is to make the log lines carry the active span's ids, and to make the trace id cross the HTTP boundary in a header that every tracing system understands.

## Prerequisites

Add the `telemetry`, `telemetry-otel`, `http-model` and `async` modules to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-telemetry"      % "0.0.56",
  "dev.zio" %% "zio-blocks-telemetry-otel" % "0.0.56",
  "dev.zio" %% "zio-blocks-http-model"     % "0.0.56",
  "dev.zio" %% "zio-blocks-async"          % "0.0.56"
)

// Scala 3.9 and newer
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-telemetry"      % "0.0.56",
  "dev.zio" %% "zio-blocks-next-telemetry-otel" % "0.0.56",
  "dev.zio" %% "zio-blocks-next-http-model"     % "0.0.56",
  "dev.zio" %% "zio-blocks-next-async"          % "0.0.56"
)
scalacOptions += "-experimental"
```

`telemetry-otel` exists only for the JVM, and the `telemetry` module needs JDK 21 or newer. [How to Add ZIO Blocks to Your Build](../getting-started/installation.md) explains the two artifact names. The code in this guide is Scala 3.

:::caution
In version 0.0.56, the `log.*` calls don't compile on Scala 2.13 outside the library's own package. This is a known bug in that version. The propagators, `OtelContext` and spans work on Scala 2.13.
:::

We assume you know how to write log lines, open spans with `trace.span`, and switch the log output to JSON, as shown in [Instrument a Program with Logs, Traces and Metrics](./instrument-a-program-with-logs-traces-and-metrics.md). The code below uses these imports:

```scala

```

## Share One Context Storage Between Logs and Spans

A log line picks up trace and span ids from a `ContextStorage`: while a span's block runs, the tracer stores the span's context there, and the logger reads it each time it writes a line. The global `log` and `trace` objects already share a built-in storage, but that storage is private to the library. Continuing a trace that arrives in a request means putting the caller's context into the storage yourself, so create your own and hand the same instance to both the tracer and the logger:

```scala
object Telemetry {
  val storage: ContextStorage[Option[SpanContext]] =
    ContextStorage.create[Option[SpanContext]](None)

  val tracer: Tracer =
    TracerProvider.builder.setContextStorage(storage).build().get("shop")

  def installLogger(): Unit = {
    log.install(LoggerProvider.builder.setContextStorage(storage).build().get("shop"))
    log.writer(JsonLogFormatter, StdoutWriter)
  }
}
```

The `Telemetry` object holds the three pieces both services use:

- `ContextStorage.create[Option[SpanContext]](None)` creates the storage. It starts empty: `None` means no span is active.
- `TracerProviderBuilder#setContextStorage` makes the tracer store each span's context in it, and `TracerProvider#get` returns a `Tracer` named `"shop"`.
- `LoggerProviderBuilder#setContextStorage` makes the logger read from the same storage, and `log.install` makes that logger the one behind the `log.*` calls.
- `log.writer(JsonLogFormatter, StdoutWriter)` writes each line as JSON. The default text format leaves the ids out, so a format that keeps them, such as JSON, is part of the setup.

We keep the tracer in a value and call `Tracer#span` on it. If your code calls the global `trace.span` instead, pass the same provider to `trace.install`; the global `trace` object then uses your storage too.

Install the logger and write a log line inside a span:

```scala
Telemetry.installLogger()

Telemetry.tracer.span("checkout.placeOrder") { _ =>
  log.info("Order received", "order.id" -> "order-1042")
}
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Order received"},"attributes":[{"key":"code.filepath","value":{"stringValue":"correlate-logs-and-traces.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp"}},{"key":"code.function","value":{"stringValue":"<unknown>"}},{"key":"code.lineno","value":{"intValue":"126"}},{"key":"order.id","value":{"stringValue":"order-1042"}}],"traceId":"<trace-id-1>","spanId":""}
```

The line ends with `traceId` and `spanId`: the ids of the `checkout.placeOrder` span, read from the shared storage. We didn't pass them to `log.info`. (Real ids are random hexadecimal strings; this page shows them as numbered placeholders, and placeholders with the same number stand for the same id.)

:::warning
The tracer and the logger must get the same `ContextStorage` instance. If you give a storage only to the tracer, the logger keeps reading the built-in one, which never holds your spans, and every log line comes out without `traceId` and `spanId`, with no error. Also call `log.install` before `log.writer`: in version 0.0.56, `log.install` silently drops the writers added before it.
:::

## Write the Trace Context into an Outgoing Request

Inside the checkout service, the shared storage is enough. The payment service runs in another process, though, and nothing in memory crosses that boundary: the trace context has to travel in the request. A `Propagator` writes a span's context into a carrier, such as a request, and reads it back on the other side. `W3CTraceContextPropagator` uses the W3C `traceparent` header, which OpenTelemetry and most tracing backends expect.

A propagator works with any carrier type, so it takes two functions to read and write headers. These two values are the glue between the `otel` module and the `http-model` module's `Request`:

```scala
object HttpGlue {
  val getHeader: (Request, String) => Option[String] =
    (request, name) => request.headers.rawGet(name)

  val setHeader: (Request, String, String) => Request =
    (request, name, value) => request.setHeader(name, value)
}
```

`Headers#rawGet` reads a header's raw text, ignoring the case of its name, and `Request#setHeader` returns a copy of the request with the header set. The request itself is immutable, so `Request#setHeader` returns the new request rather than changing the old one.

The checkout service sends the order to the payment service as a POST request:

```scala
object PaymentApi {
  val chargesUrl: URL = URL.parse("http://payment.internal/charges").toOption.get

  def chargeRequest(orderId: String, amountCents: Long): Request =
    Request.post(chargesUrl, Body.fromString(s"""{"orderId":"$orderId","amountCents":$amountCents}"""))
}
```

To send the trace along, open a client span for the call and inject its context into the request:

```scala
Telemetry.tracer.span("checkout.callPayment", SpanKind.Client) { span =>
  val request = W3CTraceContextPropagator.inject(
    span.spanContext,
    PaymentApi.chargeRequest("order-1042", 5990),
    HttpGlue.setHeader
  )
  println(request.headers)
  log.info("Calling the payment service")
}
// Headers(traceparent: 00-<trace-id-2>--01)
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Calling the payment service"},"attributes":[{"key":"code.filepath","value":{"stringValue":"correlate-logs-and-traces.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp"}},{"key":"code.function","value":{"stringValue":"<unknown>"}},{"key":"code.lineno","value":{"intValue":"162"}}],"traceId":"<trace-id-2>","spanId":""}
```

`W3CTraceContextPropagator.inject` takes the span's `SpanContext`, the request, and the setter, and returns the request with one new header. The `traceparent` value has four parts separated by dashes: the format version `00`, the trace id, the id of the span that makes the call, and the flags, where `01` means the trace is sampled. The ids in the header are the same as the ids in the log line. When a span context carries vendor data in its `traceState`, `W3CTraceContextPropagator.inject` also writes it as a `tracestate` header.

Using `SpanKind.Client` marks this span as the caller's side of a remote call; the receiving side uses `SpanKind.Server`.

## Continue the Trace in the Payment Service

The payment service reads the header back with `W3CTraceContextPropagator.extract`, which returns the caller's `SpanContext`, or `None` when there is no usable header. To make it the parent of the payment service's span, run the span inside `ContextStorage#scoped`, the same storage the tracer reads:

```scala
object PaymentService {
  def handle(request: Request): Response = {
    val parent = W3CTraceContextPropagator.extract(request, HttpGlue.getHeader)
    Telemetry.storage.scoped(parent) {
      Telemetry.tracer.span("payment.authorize", SpanKind.Server) { _ =>
        log.annotated("service.name" -> "payment") {
          log.info("Card charged")
        }
        Response.text("charged")
      }
    }
  }
}
```

`Telemetry.storage.scoped(parent) { ... }` makes `parent` the active context while the block runs and restores the previous one afterward. `Tracer#span` takes its parent from the storage, so `payment.authorize` reuses the caller's trace id, gets a new span id of its own, and records the caller's span as its parent. `log.annotated("service.name" -> "payment")` adds a `service.name` field to the lines written inside it, so we can tell the two services' lines apart. `OtelContext.withSpan(span, storage)` is not an alternative here: it takes a `Span`, not the `SpanContext` that `W3CTraceContextPropagator.extract` returns, and a span you start yourself with `Tracer#spanBuilder` is local only and never reaches the span processors, so `ContextStorage#scoped` around `Tracer#span` is the way to continue a remote trace.

To run both services in one program, we need a stand-in for the network. A real call crosses a process boundary, and the receiving service gets nothing from the caller's memory. Running the payment handler on another thread gives the same guarantee: the storage is bound to the thread that runs a block, so the other thread starts with an empty storage, and the request is the only thing that crosses over. `Async.start` from the [`async`](../reference/async.md) module runs a block on a background worker thread, and `Async#block` waits for its result, the way a client waits for a response:

```scala
// Stand-in for the network: the payment service runs on a background worker
// thread, so the only thing it receives from checkout is the request.
object Network {
  def send(request: Request): Response =
    Async.start(PaymentService.handle(request)).block
}
```

We chose this stand-in over a real HTTP server because the propagation code is the same either way: inject on the way out, extract on the way in. [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md) shows how to put an http-model handler such as `PaymentService.handle` behind the JDK's HTTP server.

The checkout service now injects the header into the request it sends:

```scala
object CheckoutService {
  def placeOrder(orderId: String, amountCents: Long): Unit =
    log.annotated("service.name" -> "checkout") {
      Telemetry.tracer.span("checkout.placeOrder") { _ =>
        log.info("Order received", "order.id" -> orderId)
        Telemetry.tracer.span("checkout.callPayment", SpanKind.Client) { span =>
          val request = W3CTraceContextPropagator.inject(
            span.spanContext,
            PaymentApi.chargeRequest(orderId, amountCents),
            HttpGlue.setHeader
          )
          val response = Network.send(request)
          log.info("Payment answered", "status" -> response.status.code)
        }
      }
    }
}
```

Place an order:

```scala
CheckoutService.placeOrder("order-1042", 5990)
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Order received"},"attributes":[{"key":"code.filepath","value":{"stringValue":"correlate-logs-and-traces.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp.CheckoutService$"}},{"key":"code.function","value":{"stringValue":"placeOrder"}},{"key":"code.lineno","value":{"intValue":"198"}},{"key":"service.name","value":{"stringValue":"checkout"}},{"key":"order.id","value":{"stringValue":"order-1042"}}],"traceId":"<trace-id-3>","spanId":""}
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Card charged"},"attributes":[{"key":"code.filepath","value":{"stringValue":"correlate-logs-and-traces.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp.PaymentService$"}},{"key":"code.function","value":{"stringValue":"handle"}},{"key":"code.lineno","value":{"intValue":"175"}},{"key":"service.name","value":{"stringValue":"payment"}}],"traceId":"<trace-id-3>","spanId":""}
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Payment answered"},"attributes":[{"key":"code.filepath","value":{"stringValue":"correlate-logs-and-traces.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp.CheckoutService$"}},{"key":"code.function","value":{"stringValue":"placeOrder"}},{"key":"code.lineno","value":{"intValue":"206"}},{"key":"service.name","value":{"stringValue":"checkout"}},{"key":"status","value":{"intValue":"200"}}],"traceId":"<trace-id-3>","spanId":""}
```

All three lines carry the same `traceId`, although the middle one was written by the payment service on another thread. A log search for that trace id returns the whole order, from both services. The `spanId` values show where each line was written: "Order received" in `checkout.placeOrder`, "Card charged" in `payment.authorize`, and "Payment answered" in `checkout.callPayment`.

<details>
  <summary>Runnable version, with the setup and services from the earlier sections: <code>Step3ContinueTraceExample.scala</code></summary>

```scala title="telemetry-examples/src/main/scala/correlatelogsandtraces/Step3ContinueTraceExample.scala" showLineNumbers
package correlatelogsandtraces

/**
 * How to Correlate Logs and Traces Across Services: Continue the Trace in the
 * Payment Service.
 *
 * The checkout service sends the charge request through the network stand-in;
 * the payment service extracts the `traceparent` header and continues the same
 * trace, so both services' log lines carry the same trace id.
 */
// Run from telemetry-examples:  sbt "runMain correlatelogsandtraces.Step3ContinueTraceExample"
object Step3ContinueTraceExample {
  object CheckoutService {
    def placeOrder(orderId: String, amountCents: Long): Unit =
      log.annotated("service.name" -> "checkout") {
        Telemetry.tracer.span("checkout.placeOrder") { _ =>
          log.info("Order received", "order.id" -> orderId)
          Telemetry.tracer.span("checkout.callPayment", SpanKind.Client) { span =>
            val request = W3CTraceContextPropagator.inject(
              span.spanContext,
              PaymentApi.chargeRequest(orderId, amountCents),
              HttpGlue.setHeader
            )
            val response = Network.send(request)
            log.info("Payment answered", "status" -> response.status.code)
          }
        }
      }
  }

  def main(args: Array[String]): Unit = {
    // Setup from the first section: tracer and logger share one storage
    Telemetry.installLogger()

    CheckoutService.placeOrder("order-1042", 5990)
  }
}
```

</details>

## Carry the Span Context to Another Thread

The storage follows the call stack of one thread. That is what kept the payment service's context separate in the previous section, and it is also what loses the context when your own code hands work to another thread, with `Async.start` or a thread pool: the work running there sees an empty storage. `OtelContext` is a snapshot of the storage's current value that you can pass along like any other parameter, and restore with `ContextStorage#scoped` where the work runs.

Here the checkout service sends the confirmation email on a background worker with `Async.start`, once without the context and once with an `OtelContext` snapshot:

```scala
object Emails {
  def sendWithoutContext(orderId: String): Unit =
    Async.start(log.info("Confirmation email sent", "order.id" -> orderId)).block

  def send(orderId: String, context: OtelContext): Unit =
    Async.start {
      Telemetry.storage.scoped(context.spanContext) {
        log.info("Confirmation email sent", "order.id" -> orderId)
      }
    }.block
}
```

`OtelContext.current(storage)` takes the snapshot on the calling thread, while the span is still active:

```scala
Telemetry.tracer.span("checkout.placeOrder") { _ =>
  log.info("Order received", "order.id" -> "order-1042")
  Emails.sendWithoutContext("order-1042")
  Emails.send("order-1042", OtelContext.current(Telemetry.storage))
}
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Order received"},"attributes":[{"key":"code.filepath","value":{"stringValue":"correlate-logs-and-traces.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp"}},{"key":"code.function","value":{"stringValue":"<unknown>"}},{"key":"code.lineno","value":{"intValue":"241"}},{"key":"order.id","value":{"stringValue":"order-1042"}}],"traceId":"<trace-id-4>","spanId":""}
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Confirmation email sent"},"attributes":[{"key":"code.filepath","value":{"stringValue":"correlate-logs-and-traces.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp.Emails$"}},{"key":"code.function","value":{"stringValue":"sendWithoutContext"}},{"key":"code.lineno","value":{"intValue":"226"}},{"key":"order.id","value":{"stringValue":"order-1042"}}]}
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Confirmation email sent"},"attributes":[{"key":"code.filepath","value":{"stringValue":"correlate-logs-and-traces.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp.Emails$"}},{"key":"code.function","value":{"stringValue":"send"}},{"key":"code.lineno","value":{"intValue":"231"}},{"key":"order.id","value":{"stringValue":"order-1042"}}],"traceId":"<trace-id-4>","spanId":""}
```

The first email line has no `traceId` or `spanId`, so a search by trace id never finds it. The second one carries the ids of `checkout.placeOrder`, because `Emails.send` restored the snapshot before logging. Only the span context travels this way; fields added with `log.annotated` are bound to the calling thread too, so add them again on the worker when you need them there.

When you hold the `Span` itself, for example inside the block of `Tracer#span`, `OtelContext.withSpan(span, Telemetry.storage) { ... }` restores its context in one call. `OtelContext` also works as a service in a [`Context`](../reference/context.md), so code that already threads a `Context` through its calls can carry the span context with it.

<details>
  <summary>Runnable version, with the setup from the first section: <code>Step4CarrySpanContextExample.scala</code></summary>

```scala title="telemetry-examples/src/main/scala/correlatelogsandtraces/Step4CarrySpanContextExample.scala" showLineNumbers
package correlatelogsandtraces

/**
 * How to Correlate Logs and Traces Across Services: Carry the Span Context to
 * Another Thread.
 *
 * Hands the confirmation email to a worker thread twice: once without the span
 * context, whose log line has no ids, and once with an `OtelContext` snapshot,
 * whose log line keeps the order's trace and span ids.
 */
// Run from telemetry-examples:  sbt "runMain correlatelogsandtraces.Step4CarrySpanContextExample"
object Step4CarrySpanContextExample {
  object Emails {
    def sendWithoutContext(orderId: String): Unit =
      Async.start(log.info("Confirmation email sent", "order.id" -> orderId)).block

    def send(orderId: String, context: OtelContext): Unit =
      Async.start {
        Telemetry.storage.scoped(context.spanContext) {
          log.info("Confirmation email sent", "order.id" -> orderId)
        }
      }.block
  }

  def main(args: Array[String]): Unit = {
    // Setup from the first section: tracer and logger share one storage
    Telemetry.installLogger()

    Telemetry.tracer.span("checkout.placeOrder") { _ =>
      log.info("Order received", "order.id" -> "order-1042")
      Emails.sendWithoutContext("order-1042")
      Emails.send("order-1042", OtelContext.current(Telemetry.storage))
    }
  }
}
```

</details>

## Switch to B3 Headers

Services instrumented with Zipkin often expect B3 headers instead of `traceparent`. `B3Propagator` has two variants with the same `Propagator` methods: `B3Propagator.single` writes one `b3` header, and `B3Propagator.multi` writes one header per field. To see what each writes for the same span:

```scala
Telemetry.tracer.span("checkout.callPayment", SpanKind.Client) { span =>
  val request = PaymentApi.chargeRequest("order-1042", 5990)
  println(B3Propagator.single.inject(span.spanContext, request, HttpGlue.setHeader).headers)
  println(B3Propagator.multi.inject(span.spanContext, request, HttpGlue.setHeader).headers)
}
// Headers(b3: <trace-id-5>--1)
// Headers(x-b3-traceid: <trace-id-5>, x-b3-spanid: , x-b3-sampled: 1)
```

The single header holds the trace id, the span id and the sampling decision (`1` for sampled) separated by dashes. The multi-header variant writes `X-B3-TraceId`, `X-B3-SpanId` and `X-B3-Sampled`, which http-model stores in lower case. `Propagator#fields` lists the header names a propagator reads, which is what a proxy or a CORS allow-list must let through.

Both sides must agree on the format: `W3CTraceContextPropagator.extract` finds nothing in a request that carries only B3 headers, and the receiver starts a new trace. A receiver that serves callers of both kinds can try the formats in turn:

```scala
def extractParent(request: Request): Option[SpanContext] =
  W3CTraceContextPropagator
    .extract(request, HttpGlue.getHeader)
    .orElse(B3Propagator.multi.extract(request, HttpGlue.getHeader))
```

With the fallback, the payment service finds the caller's trace id in a B3 request:

```scala
Telemetry.tracer.span("checkout.callPayment", SpanKind.Client) { span =>
  val request = B3Propagator.multi.inject(span.spanContext, PaymentApi.chargeRequest("order-1042", 5990), HttpGlue.setHeader)
  println(s"W3C only: ${W3CTraceContextPropagator.extract(request, HttpGlue.getHeader).map(_.traceIdHex)}")
  println(s"with B3 fallback: ${extractParent(request).map(_.traceIdHex)}")
  println(s"checkout span: ${span.spanContext.traceIdHex}")
}
// W3C only: None
// with B3 fallback: Some(<trace-id-6>)
// checkout span: <trace-id-6>
```

To make the payment service accept both formats, call `extractParent(request)` in `PaymentService.handle` where it calls `W3CTraceContextPropagator.extract`.

## Handle Missing or Invalid Headers

A request can arrive with no trace header at all, from a client that isn't instrumented, or with a malformed one. `W3CTraceContextPropagator.extract` returns `None` in both cases, and it rejects the values the W3C specification calls invalid, such as an all-zero trace id or the forbidden version `ff`. Starting from a charge request with no trace header:

```scala
val untraced = PaymentApi.chargeRequest("order-1043", 1250)
```

To try `W3CTraceContextPropagator.extract` on that request and on three broken `traceparent` values:

```scala
List(
  "no header"         -> untraced,
  "all-zero trace id" -> untraced.setHeader("traceparent", "00-00000000000000000000000000000000-00f067aa0ba902b7-01"),
  "version ff"        -> untraced.setHeader("traceparent", "ff-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"),
  "cut short"         -> untraced.setHeader("traceparent", "00-4bf92f3577b34da6a3ce929d0e0e4736-01")
).foreach { case (label, request) =>
  println(s"$label: ${W3CTraceContextPropagator.extract(request, HttpGlue.getHeader)}")
}
// no header: None
// all-zero trace id: None
// version ff: None
// cut short: None
```

`PaymentService.handle` needs no special case for this: `scoped(None)` leaves the storage empty, and `Tracer#span` starts a new trace with a fresh trace id. To see it, send the request without injecting any header:

```scala
Telemetry.tracer.span("checkout.callPayment", SpanKind.Client) { _ =>
  log.info("Calling the payment service without trace headers")
  val response = Network.send(untraced)
  log.info("Payment answered", "status" -> response.status.code)
}
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Calling the payment service without trace headers"},"attributes":[{"key":"code.filepath","value":{"stringValue":"correlate-logs-and-traces.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp"}},{"key":"code.function","value":{"stringValue":"<unknown>"}},{"key":"code.lineno","value":{"intValue":"305"}}],"traceId":"<trace-id-7>","spanId":""}
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Card charged"},"attributes":[{"key":"code.filepath","value":{"stringValue":"correlate-logs-and-traces.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp.PaymentService$"}},{"key":"code.function","value":{"stringValue":"handle"}},{"key":"code.lineno","value":{"intValue":"175"}},{"key":"service.name","value":{"stringValue":"payment"}}],"traceId":"<trace-id-8>","spanId":""}
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Payment answered"},"attributes":[{"key":"code.filepath","value":{"stringValue":"correlate-logs-and-traces.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp"}},{"key":"code.function","value":{"stringValue":"<unknown>"}},{"key":"code.lineno","value":{"intValue":"307"}},{"key":"status","value":{"intValue":"200"}}],"traceId":"<trace-id-7>","spanId":""}
```

The payment service's line has a different trace id from the two checkout lines: its work is still traced and its lines still correlate with each other, but they no longer join the caller's trace. That is the right outcome for a header you can't trust.

:::tip
A tracer built with the default sampler records every span, even when the caller's flags say the trace isn't sampled (`-00` at the end of `traceparent`). To follow the caller's decision, build the provider with `TracerProviderBuilder#setSampler(ParentBasedSampler(AlwaysOnSampler))`. Log lines written in an unsampled span still carry its trace id.
:::

:::caution
In version 0.0.56, `W3CTraceContextPropagator.extract` is more lenient than the W3C specification in a few places: it accepts upper-case hex digits and extra characters after a version `00` header, and it rejects headers with a future version such as `01` instead of reading their first four fields. `B3Propagator.single` returns a context with zero ids, rather than `None`, for a `b3` header that holds only a sampling decision such as `0`. These are known bugs in that version.
:::

## Putting It Together

The complete example places two orders. For each one, the checkout service calls the payment service with a `traceparent` header, the payment service continues the trace, and the confirmation email is logged on a worker thread from an `OtelContext` snapshot. It adds nothing the sections don't show; the services, `Telemetry`, `HttpGlue`, `PaymentApi` and `Network` live in the other files of the same package (`Network` sits in `PaymentService.scala`):

```scala title="telemetry-examples/src/main/scala/correlatelogsandtraces/CompleteExample.scala"
package correlatelogsandtraces

/**
 * How to Correlate Logs and Traces Across Services: Complete Example.
 *
 * Places two orders. For each one, the checkout service calls the payment
 * service with a `traceparent` header, the payment service continues the trace,
 * and the confirmation email is logged on a worker thread with an `OtelContext`
 * snapshot. Every line of one order carries that order's trace id.
 */
// Run from telemetry-examples:  sbt "runMain correlatelogsandtraces.CompleteExample"
object CompleteExample {
  object Emails {
    def send(orderId: String, context: OtelContext): Unit =
      Async.start {
        Telemetry.storage.scoped(context.spanContext) {
          log.info("Confirmation email sent", "order.id" -> orderId)
        }
      }.block
  }

  object CheckoutService {
    def placeOrder(orderId: String, amountCents: Long): Unit =
      log.annotated("service.name" -> "checkout") {
        Telemetry.tracer.span("checkout.placeOrder") { _ =>
          log.info("Order received", "order.id" -> orderId)
          Telemetry.tracer.span("checkout.callPayment", SpanKind.Client) { span =>
            val request = W3CTraceContextPropagator.inject(
              span.spanContext,
              PaymentApi.chargeRequest(orderId, amountCents),
              HttpGlue.setHeader
            )
            val response = Network.send(request)
            log.info("Payment answered", "status" -> response.status.code)
          }
          Emails.send(orderId, OtelContext.current(Telemetry.storage))
        }
      }
  }

  def main(args: Array[String]): Unit = {
    Telemetry.installLogger()

    CheckoutService.placeOrder("order-1042", 5990)
    CheckoutService.placeOrder("order-1043", 1250)
  }
}
```

Each order has its own trace id, and all four lines of an order carry it, from both services and from the worker thread (output from one run, with the times, file paths and ids replaced by placeholders):

```text
{"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Order received"},"attributes":[{"key":"code.filepath","value":{"stringValue":"<path>/CompleteExample.scala"}},{"key":"code.namespace","value":{"stringValue":"correlatelogsandtraces.CompleteExample$.CheckoutService$"}},{"key":"code.function","value":{"stringValue":"placeOrder"}},{"key":"code.lineno","value":{"intValue":"30"}},{"key":"service.name","value":{"stringValue":"checkout"}},{"key":"order.id","value":{"stringValue":"order-1042"}}],"traceId":"<trace-id-1>","spanId":""}
{"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Card charged"},"attributes":[{"key":"code.filepath","value":{"stringValue":"<path>/PaymentService.scala"}},{"key":"code.namespace","value":{"stringValue":"correlatelogsandtraces.PaymentService$"}},{"key":"code.function","value":{"stringValue":"handle"}},{"key":"code.lineno","value":{"intValue":"19"}},{"key":"service.name","value":{"stringValue":"payment"}}],"traceId":"<trace-id-1>","spanId":""}
{"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Payment answered"},"attributes":[{"key":"code.filepath","value":{"stringValue":"<path>/CompleteExample.scala"}},{"key":"code.namespace","value":{"stringValue":"correlatelogsandtraces.CompleteExample$.CheckoutService$"}},{"key":"code.function","value":{"stringValue":"placeOrder"}},{"key":"code.lineno","value":{"intValue":"38"}},{"key":"service.name","value":{"stringValue":"checkout"}},{"key":"status","value":{"intValue":"200"}}],"traceId":"<trace-id-1>","spanId":""}
{"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Confirmation email sent"},"attributes":[{"key":"code.filepath","value":{"stringValue":"<path>/CompleteExample.scala"}},{"key":"code.namespace","value":{"stringValue":"correlatelogsandtraces.CompleteExample$.Emails$"}},{"key":"code.function","value":{"stringValue":"send"}},{"key":"code.lineno","value":{"intValue":"21"}},{"key":"order.id","value":{"stringValue":"order-1042"}}],"traceId":"<trace-id-1>","spanId":""}
{"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Order received"},"attributes":[{"key":"code.filepath","value":{"stringValue":"<path>/CompleteExample.scala"}},{"key":"code.namespace","value":{"stringValue":"correlatelogsandtraces.CompleteExample$.CheckoutService$"}},{"key":"code.function","value":{"stringValue":"placeOrder"}},{"key":"code.lineno","value":{"intValue":"30"}},{"key":"service.name","value":{"stringValue":"checkout"}},{"key":"order.id","value":{"stringValue":"order-1043"}}],"traceId":"<trace-id-2>","spanId":""}
{"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Card charged"},"attributes":[{"key":"code.filepath","value":{"stringValue":"<path>/PaymentService.scala"}},{"key":"code.namespace","value":{"stringValue":"correlatelogsandtraces.PaymentService$"}},{"key":"code.function","value":{"stringValue":"handle"}},{"key":"code.lineno","value":{"intValue":"19"}},{"key":"service.name","value":{"stringValue":"payment"}}],"traceId":"<trace-id-2>","spanId":""}
{"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Payment answered"},"attributes":[{"key":"code.filepath","value":{"stringValue":"<path>/CompleteExample.scala"}},{"key":"code.namespace","value":{"stringValue":"correlatelogsandtraces.CompleteExample$.CheckoutService$"}},{"key":"code.function","value":{"stringValue":"placeOrder"}},{"key":"code.lineno","value":{"intValue":"38"}},{"key":"service.name","value":{"stringValue":"checkout"}},{"key":"status","value":{"intValue":"200"}}],"traceId":"<trace-id-2>","spanId":""}
{"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Confirmation email sent"},"attributes":[{"key":"code.filepath","value":{"stringValue":"<path>/CompleteExample.scala"}},{"key":"code.namespace","value":{"stringValue":"correlatelogsandtraces.CompleteExample$.Emails$"}},{"key":"code.function","value":{"stringValue":"send"}},{"key":"code.lineno","value":{"intValue":"21"}},{"key":"order.id","value":{"stringValue":"order-1043"}}],"traceId":"<trace-id-2>","spanId":""}
```

## Running the Examples

All runnable code from this guide is in [`telemetry-examples/src/main/scala/correlatelogsandtraces/`](https://github.com/zio/zio-blocks/tree/main/telemetry-examples/src/main/scala/correlatelogsandtraces). The examples need JDK 21 or newer. Run any of them from `telemetry-examples` with `sbt "runMain correlatelogsandtraces.<ObjectName>"`, for example `sbt "runMain correlatelogsandtraces.CompleteExample"`.

## Going Further

- The [OTLP Export](../reference/telemetry/otel/index.md) reference page describes the `Propagator` trait and `OtelContext`, and [SpanContext](../reference/telemetry/tracing/span-context.md) describes the ids and flags a propagator carries.
- [How to Test Code That Logs, Traces and Records Metrics](./test-telemetry.md) shows how to assert that a log record carries the ids of the span it was written in.
- [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md) runs an http-model handler on a real HTTP server.
- [Telemetry: Architecture, Patterns, and Real-World Usage](./telemetry-guide.md) explains how sampling, context storage and the log macros work inside the module.

---

## How to Customize JSON Field Names, ADT Encoding and Optional Fields

In this guide, we change the JSON that a derived `Schema` reads and writes until it matches an API contract we don't control. By the end, you will know how to rename fields to `snake_case`, rename one field while still accepting its old name, tag sealed trait cases with a `"type"` field or with no tag at all, decide whether `None` is written as `null`, leave fields out of the JSON, and reject unknown fields.

Each section is a short recipe: the problem, the code, and the JSON it produces. We use two tools throughout — [`Modifier`](../reference/schema/modifier.md) annotations, which change the schema of one type, and `JsonCodecDeriver` settings, which change every type a codec touches — and finish with a comparison to help you choose between them.

This guide covers:

- Renaming every field with a naming strategy
- Renaming one field and accepting old names
- Tagging sealed trait cases with a type field
- Encoding sealed traits without a tag
- Encoding case-object enums as strings
- Controlling optional fields, empty collections and default values
- Rejecting unknown fields
- Leaving a field out of the JSON
- Choosing between global settings and per-type modifiers

## The Problem

The default JSON format of ZIO Blocks uses Scala field names as they are, wraps each sealed trait case in an object keyed by the case name, and leaves out `None` values and empty lists. That's a good default for data we write and read ourselves, but the services we talk to often expect something else: `snake_case` keys, a `"type"` field next to the case's own fields, or an explicit `null` for every missing value.

When the format is fixed by someone else, the usual fallback is to write the JSON by hand. For example, to send a book to a service that expects `snake_case` keys and no empty arrays, we might write:

```scala
case class Author(fullName: String)
case class Book(title: String, publishedYear: Int, isbn13: Option[String], author: Author, tags: List[String])

def quote(s: String): String = "\"" + s.replace("\\", "\\\\").replace("\"", "\\\"") + "\""

def authorToJson(author: Author): String =
  s"""{"full_name":${quote(author.fullName)}}"""

def bookToJson(book: Book): String = {
  val fields = List(
    Some(s""""title":${quote(book.title)}"""),
    Some(s""""published_year":${book.publishedYear}"""),
    book.isbn13.map(isbn => s""""isbn13":${quote(isbn)}"""),
    Some(s""""author":${authorToJson(book.author)}"""),
    if (book.tags.isEmpty) None else Some(s""""tags":${book.tags.map(quote).mkString("[", ",", "]")}""")
  )
  fields.flatten.mkString("{", ",", "}")
}
```

This approach repeats every field name a second time, has no decoder at all, and silently drifts from the case class the next time someone adds a field. Each new type, and each sealed trait with its own tagging rule, needs another hand-written function.

In this guide, we keep `Schema.derived` and change the format instead, with one annotation or one deriver setting per rule.

## Prerequisites

We use the same schema module as the [Your First Schema](./your-first-schema.md) tutorial:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56"
```

This guide assumes you can derive a schema with `Schema.derived` and round-trip a value with `toJsonString` and `fromJson`, as shown in [Your First Schema](./your-first-schema.md). Every example uses two imports: `zio.blocks.schema._` brings in `Schema`, `Modifier` and `NameMapper`, and `zio.blocks.schema.json._` brings in `JsonCodecDeriver`, `JsonCodec` and `DiscriminatorKind`.

## The Book Catalog Model

Our running example is a small catalog of Scala books. A `Book` has a nested `Author`, an optional ISBN, a list of tags, an optional edition number and a `language` field with a default value:

```scala

case class Author(fullName: String)
object Author {
  implicit val schema: Schema[Author] = Schema.derived
}

case class Book(
  title: String,
  publishedYear: Int,
  isbn13: Option[String],
  author: Author,
  tags: List[String],
  edition: Option[Int],
  language: String = "en"
)
object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

val fpInScala = Book(
  "Functional Programming in Scala",
  2014,
  Some("9781617290657"),
  Author("Paul Chiusano"),
  List("scala", "functional-programming"),
  Some(1)
)
```

With no customization, `toJsonString` writes every field under its Scala name and leaves out `language` because it holds the default value:

```scala
println(fpInScala.toJsonString)
// {"title":"Functional Programming in Scala","publishedYear":2014,"isbn13":"9781617290657","author":{"fullName":"Paul Chiusano"},"tags":["scala","functional-programming"],"edition":1}
```

This is the shape we change in the rest of the guide.

## Rename Every Field With a Naming Strategy

The most common request is a different naming convention, such as `snake_case` for a Python or Ruby service. A `NameMapper` converts each Scala field name; the built-in strategies are `NameMapper.SnakeCase`, `NameMapper.KebabCase`, `NameMapper.CamelCase`, `NameMapper.PascalCase`, `NameMapper.Identity` (the default) and `NameMapper.Custom(f)`, which applies any `String => String` function.

To apply a strategy to every type in a codec, we derive a codec from the schema with `JsonCodecDeriver#withFieldNameMapper`. `Schema#derive` returns a `JsonCodec[Book]`:

```scala
val snakeCaseCodec: JsonCodec[Book] =
  Book.schema.derive(JsonCodecDeriver.withFieldNameMapper(NameMapper.SnakeCase))
```

We call `JsonCodec#encodeToString` and `JsonCodec#decode` on the codec directly:

```scala
println(snakeCaseCodec.encodeToString(fpInScala))
// {"title":"Functional Programming in Scala","published_year":2014,"isbn13":"9781617290657","author":{"full_name":"Paul Chiusano"},"tags":["scala","functional-programming"],"edition":1}
snakeCaseCodec.decode(snakeCaseCodec.encodeToString(fpInScala)) == Right(fpInScala)
// res3: Boolean = true
```

The mapper reaches the nested `Author` too, so `fullName` becomes `full_name`. Digits don't start a new word, so `isbn13` stays as it is.

:::warning
`toJsonString` and `fromJson` always use the default codec of the schema. A codec built with `Schema#derive` applies its settings only when you call it yourself, so keep it in one place — for example, as a `val` in the companion object — and use it everywhere you read or write that JSON.
:::

When only one type needs the convention, annotate it with `Modifier.fieldNaming` instead. The annotation becomes part of the schema, so the plain `toJsonString` picks it up. It accepts `"snake_case"`, `"kebab-case"`, `"camelCase"`, `"PascalCase"` and `"identity"`:

```scala

case class Author(fullName: String)
object Author {
  implicit val schema: Schema[Author] = Schema.derived
}

@Modifier.fieldNaming("snake_case")
case class Book(title: String, publishedYear: Int, isbn13: Option[String], author: Author)
object Book {
  implicit val schema: Schema[Book] = Schema.derived
}
```

Now we encode a book with the plain extension method:

```scala
println(Book("Programming in Scala", 2008, Some("9780981531601"), Author("Martin Odersky")).toJsonString)
// {"title":"Programming in Scala","published_year":2008,"isbn13":"9780981531601","author":{"fullName":"Martin Odersky"}}
```

The annotation renames only the fields declared directly in `Book`. The nested `Author` keeps `fullName`, because its own schema has no annotation. A strategy name the library doesn't know, such as `"SCREAMING_SNAKE"`, fails with an `IllegalArgumentException` when the codec is derived, not while decoding.

## Rename One Field and Accept Its Old Name

Sometimes a single field needs a different name — the API calls the ISBN `isbn`, not `isbn13` — and older clients still send a previous name. `Modifier.rename` sets the name used for both encoding and decoding, and `Modifier.alias` adds an extra name that the decoder accepts but the encoder never writes:

```scala

case class Book(
  title: String,
  @Modifier.rename("isbn")
  @Modifier.alias("isbn_13")
  isbn13: Option[String]
)
object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

val pis = Book("Programming in Scala", Some("9780981531601"))
```

The encoder writes the renamed key:

```scala
println(pis.toJsonString)
// {"title":"Programming in Scala","isbn":"9780981531601"}
```

The decoder accepts both the new name and the alias:

```scala
"""{"title":"Programming in Scala","isbn":"9780981531601"}""".fromJson[Book]
// res8: Either[SchemaError, Book] = Right(
//   Book(title = "Programming in Scala", isbn13 = Some("9780981531601"))
// )
"""{"title":"Programming in Scala","isbn_13":"9780981531601"}""".fromJson[Book]
// res9: Either[SchemaError, Book] = Right(
//   Book(title = "Programming in Scala", isbn13 = Some("9780981531601"))
// )
```

The original Scala name is no longer one of the accepted names. Because `isbn13` is optional and unknown fields are ignored by default, input that still uses it decodes without an error, but the value is lost:

```scala
"""{"title":"Programming in Scala","isbn13":"9780981531601"}""".fromJson[Book]
// res10: Either[SchemaError, Book] = Right(
//   Book(title = "Programming in Scala", isbn13 = None)
// )
```

If clients may still send the Scala name, add it as an alias too, for example `@Modifier.alias("isbn13")`. `Modifier.rename` also wins over any naming strategy from the previous section, so it is the way to make one exception to a `snake_case` rule.

## Tag Sealed Trait Cases With a Type Field

A sealed trait needs a tag in the JSON so the decoder knows which case to build. Model the formats a book is sold in:

```scala

sealed trait Format
object Format {
  case class Paperback(pages: Int)    extends Format
  case class Ebook(sizeMb: Double)    extends Format
  case class Audiobook(minutes: Int)  extends Format

  implicit val schema: Schema[Format] = Schema.derived
}

val paperback: Format = Format.Paperback(320)
```

By default, the case name is the key of a wrapper object around the case's fields. This is `DiscriminatorKind.Key`:

```scala
println(paperback.toJsonString)
// {"Paperback":{"pages":320}}
```

Many APIs expect a flat object with a `"type"` field instead. `JsonCodecDeriver#withDiscriminatorKind` with `DiscriminatorKind.Field` produces that shape, and `JsonCodecDeriver#withCaseNameMapper` changes how case names are written:

```scala
val typeFieldCodec: JsonCodec[Format] =
  Format.schema.derive(
    JsonCodecDeriver
      .withDiscriminatorKind(DiscriminatorKind.Field("type"))
      .withCaseNameMapper(NameMapper.SnakeCase)
  )
```

The new codec writes the case name into a `"type"` field and reads it back:

```scala
println(typeFieldCodec.encodeToString(paperback))
// {"type":"paperback","pages":320}
typeFieldCodec.decode("""{"type":"audiobook","minutes":600}""")
// res14: Either[SchemaError, Format] = Right(Audiobook(600))
```

The decoder looks up the `"type"` field wherever it appears in the object, so it doesn't need to come first.

The same choice can live on the sealed trait itself. `Modifier.discriminator` selects a field discriminator, `Modifier.caseNaming` applies a naming strategy to the case names, and `Modifier.rename` on a case overrides one name:

```scala

@Modifier.discriminator("type")
@Modifier.caseNaming("snake_case")
sealed trait Format
object Format {
  case class Paperback(pages: Int)   extends Format
  case class Ebook(sizeMb: Double)   extends Format
  @Modifier.rename("audio")
  case class Audiobook(minutes: Int) extends Format

  implicit val schema: Schema[Format] = Schema.derived
}

val formats: List[Format] = List(Format.Paperback(320), Format.Ebook(4.5), Format.Audiobook(600))
```

Because the annotations are part of the schema, `toJsonString` uses them:

```scala
formats.foreach(format => println(format.toJsonString))
// {"type":"paperback","pages":320}
// {"type":"ebook","sizeMb":4.5}
// {"type":"audio","minutes":600}
```

## Encode Sealed Traits Without a Tag

Some APIs send a case's fields with no tag at all and expect the reader to work out the case from the fields that are present. `DiscriminatorKind.None` writes each case as a plain object and, when decoding, tries the cases in declaration order until one succeeds:

```scala

sealed trait Format
object Format {
  case class Paperback(pages: Int)   extends Format
  case class Ebook(sizeMb: Double)   extends Format
  case class Audiobook(minutes: Int) extends Format

  implicit val schema: Schema[Format] = Schema.derived
}

val untaggedCodec: JsonCodec[Format] =
  Format.schema.derive(JsonCodecDeriver.withDiscriminatorKind(DiscriminatorKind.None))
```

Encoding drops the tag, and decoding picks the first case whose fields fit:

```scala
println(untaggedCodec.encodeToString(Format.Paperback(320)))
// {"pages":320}
untaggedCodec.decode("""{"minutes":600}""")
// res19: Either[SchemaError, Format] = Right(Audiobook(600))
```

`{"minutes":600}` is not a valid `Paperback` or `Ebook`, because their required fields are missing, so the decoder moves on and builds an `Audiobook`. Input that fits no case fails with one generic message:

```scala
untaggedCodec.decode("""{"chapters":12}""") match {
  case Left(error) => println(error.message)
  case Right(format) => println(s"Decoded: $format")
}
// expected a variant value at: .
```

:::warning
Without a tag, the order of the cases decides the result. A case whose fields are all optional, or a case whose fields are a subset of a later case's fields, matches input meant for the later case, and the decoder never reaches it. Use `DiscriminatorKind.None` only when every case has a required field that no earlier case accepts, and prefer a type field whenever you control the format.
:::

## Encode Case-Object Enums as Strings

A sealed trait whose cases are all case objects is an enumeration, and the JSON codec writes each value as a plain string by default:

```scala

sealed trait Availability
object Availability {
  case object InStock    extends Availability
  case object OutOfPrint extends Availability
  case object Preorder   extends Availability

  implicit val schema: Schema[Availability] = Schema.derived
}

val outOfPrint: Availability = Availability.OutOfPrint
```

The plain encoder writes the case name as a string, a case name mapper changes its spelling, and `JsonCodecDeriver#withEnumValuesAsStrings` set to `false` switches back to the wrapper-object form used for other sealed traits:

```scala
println(outOfPrint.toJsonString)
// "OutOfPrint"
println(Availability.schema.derive(JsonCodecDeriver.withCaseNameMapper(NameMapper.SnakeCase)).encodeToString(outOfPrint))
// "out_of_print"
println(Availability.schema.derive(JsonCodecDeriver.withEnumValuesAsStrings(false)).encodeToString(outOfPrint))
// {"OutOfPrint":{}}
```

`@Modifier.caseNaming("snake_case")` on the sealed trait gives the same `snake_case` strings through `toJsonString`.

## Control Optional Fields, Empty Collections and Defaults

By default, the encoder leaves out three kinds of fields, and the decoder fills them back in when they are missing:

| Field value                        | Encoder setting (default)                                       | Decoder setting (default)                                               |
| ---------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------------- |
| `None` in an `Option` field        | `JsonCodecDeriver#withTransientNone(true)`: left out            | `JsonCodecDeriver#withRequireOptionFields(false)`: may be missing       |
| An empty collection                | `JsonCodecDeriver#withTransientEmptyCollection(true)`: left out | `JsonCodecDeriver#withRequireCollectionFields(false)`: may be missing   |
| A value equal to the field default | `JsonCodecDeriver#withTransientDefaultValue(true)`: left out    | `JsonCodecDeriver#withRequireDefaultValueFields(false)`: may be missing |

Return to the book model from the start of the guide and add a book whose ISBN and edition are unknown and which has no tags yet:

```scala

case class Author(fullName: String)
object Author {
  implicit val schema: Schema[Author] = Schema.derived
}

case class Book(
  title: String,
  publishedYear: Int,
  isbn13: Option[String],
  author: Author,
  tags: List[String],
  edition: Option[Int],
  language: String = "en"
)
object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

val zionomicon = Book("Zionomicon", 2024, None, Author("John A. De Goes"), Nil, None)
```

The default codec leaves out `isbn13`, `tags`, `edition` and `language`:

```scala
println(zionomicon.toJsonString)
// {"title":"Zionomicon","publishedYear":2024,"author":{"fullName":"John A. De Goes"}}
```

If the receiving service expects every field to be present, turn the three encoder settings off. `None` is then written as `null`, an empty list as `[]`, and the default value as itself:

```scala
val explicitCodec: JsonCodec[Book] =
  Book.schema.derive(
    JsonCodecDeriver
      .withTransientNone(false)
      .withTransientEmptyCollection(false)
      .withTransientDefaultValue(false)
  )
```

The same book now comes out with every field, and it still decodes back to the original value:

```scala
println(explicitCodec.encodeToString(zionomicon))
// {"title":"Zionomicon","publishedYear":2024,"isbn13":null,"author":{"fullName":"John A. De Goes"},"tags":[],"edition":null,"language":"en"}
explicitCodec.decode(explicitCodec.encodeToString(zionomicon)) == Right(zionomicon)
// res28: Boolean = true
```

The decoder settings work the other way: they make decoding strict. With `JsonCodecDeriver#withRequireOptionFields` set to `true`, an `Option` field must appear in the input, either with a value or as `null`:

```scala
val strictCodec: JsonCodec[Book] =
  Book.schema.derive(JsonCodecDeriver.withRequireOptionFields(true))
```

Decoding the default output of the previous step now fails, because `isbn13` and `edition` are missing:

```scala
strictCodec.decode("""{"title":"Zionomicon","publishedYear":2024,"author":{"fullName":"John A. De Goes"}}""") match {
  case Left(error) => println(error.message)
  case Right(book) => println(s"Decoded: $book")
}
// missing required field "isbn13" at: .
```

`JsonCodecDeriver#withRequireCollectionFields` does the same for collections, and `JsonCodecDeriver#withRequireDefaultValueFields` ignores field defaults when decoding, so a field such as `language` must be present even though it has a default. Both settings also change the encoder: with `withRequireCollectionFields(true)` the encoder writes empty collections, and with `withRequireDefaultValueFields(true)` it writes default values, instead of leaving them out. Set them only when you want that output too.

## Reject Unknown Fields

By default, the decoder skips JSON fields that don't belong to the type. That keeps old readers working when a writer adds fields, but it also hides typos in hand-written input. `Modifier.noExtraFields` makes the decoder of one type reject unknown fields:

```scala

@Modifier.noExtraFields()
case class Author(fullName: String)
object Author {
  implicit val schema: Schema[Author] = Schema.derived
}

case class Book(title: String, author: Author)
object Book {
  implicit val schema: Schema[Book] = Schema.derived
}
```

`Book` still skips an unknown `subtitle`, while `Author` rejects a misspelled field and reports where it is:

```scala
"""{"title":"Zionomicon","subtitle":"A Guide","author":{"fullName":"John A. De Goes"}}""".fromJson[Book]
// res31: Either[SchemaError, Book] = Right(
//   Book(title = "Zionomicon", author = Author("John A. De Goes"))
// )

"""{"title":"Zionomicon","author":{"fullname":"John A. De Goes"}}""".fromJson[Book] match {
  case Left(error) => println(error.message)
  case Right(book) => println(s"Decoded: $book")
}
// unexpected field "fullname" at: .author
```

To reject unknown fields in every type a codec handles, use `JsonCodecDeriver#withRejectExtraFields` instead:

```scala
Book.schema
  .derive(JsonCodecDeriver.withRejectExtraFields(true))
  .decode("""{"title":"Zionomicon","subtitle":"A Guide","author":{"fullName":"John A. De Goes"}}""") match {
  case Left(error) => println(error.message)
  case Right(book) => println(s"Decoded: $book")
}
// unexpected field "subtitle" at: .
```

## Leave a Field Out of the JSON

Some fields belong to the Scala value but not to the JSON: a relevance score computed in memory, or an old identifier that we still accept from clients but no longer send. `Modifier.transient` leaves a field out of the JSON in both directions, and `Modifier.encodeTransient` leaves it out only when encoding:

```scala

case class Book(
  title: String,
  @Modifier.transient() searchScore: Double = 0.0,
  @Modifier.encodeTransient() legacyId: Option[Long]
)
object Book {
  implicit val schema: Schema[Book] = Schema.derived
}
```

The encoder writes neither field:

```scala
println(Book("Zionomicon", 0.93, Some(42L)).toJsonString)
// {"title":"Zionomicon"}
```

When decoding, a transient field ignores the input and takes its default value, while an encode-transient field is still read:

```scala
"""{"title":"Zionomicon","searchScore":0.5,"legacyId":42}""".fromJson[Book]
// res36: Either[SchemaError, Book] = Right(
//   Book(title = "Zionomicon", searchScore = 0.0, legacyId = Some(42L))
// )
```

Because the decoder has to produce a value for a field it never reads, a transient or encode-transient field needs a default value unless it is an `Option` or a collection, which fall back to `None` or an empty collection. Without one, `Schema.derived` fails at compile time with a `Missing default value for transient or encodeTransient field` error that names the field and its type, so the mistake never reaches production.

## Choose Between Global Settings and Per-Type Modifiers

Most rules in this guide can be applied in two ways. The table below sums up where each one lives:

| Rule                               | Per-type modifier                                      | Global deriver setting                                                 |
| ---------------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------- |
| Field naming strategy              | `@Modifier.fieldNaming(...)`                           | `JsonCodecDeriver#withFieldNameMapper(...)`                            |
| One field or case name             | `@Modifier.rename(...)`                                | —                                                                      |
| Extra accepted names               | `@Modifier.alias(...)`                                 | —                                                                      |
| Type field discriminator           | `@Modifier.discriminator(...)`                         | `JsonCodecDeriver#withDiscriminatorKind(DiscriminatorKind.Field(...))` |
| No discriminator                   | —                                                      | `JsonCodecDeriver#withDiscriminatorKind(DiscriminatorKind.None)`       |
| Case naming strategy               | `@Modifier.caseNaming(...)`                            | `JsonCodecDeriver#withCaseNameMapper(...)`                             |
| Enums as wrapper objects           | —                                                      | `JsonCodecDeriver#withEnumValuesAsStrings(false)`                      |
| Optional, empty and default fields | —                                                      | `JsonCodecDeriver#withTransient*` and `JsonCodecDeriver#withRequire*`  |
| Leave a field out                  | `@Modifier.transient()`, `@Modifier.encodeTransient()` | —                                                                      |
| Reject unknown fields              | `@Modifier.noExtraFields()`                            | `JsonCodecDeriver#withRejectExtraFields(true)`                         |

Use a modifier when the rule describes the type itself — "this API always calls the ISBN `isbn`". The modifier travels with the schema, so `toJsonString`, `fromJson` and every codec derived from the schema agree on it, and nobody can forget to pass a setting.

Use a deriver setting when the rule describes the service you are talking to — "this partner wants `snake_case` and `null` everywhere". One codec applies the rule to every nested type without annotating each one, and a different codec for a different service can use different settings for the same types.

When both are present, the modifier wins: a type annotated with `@Modifier.fieldNaming("snake_case")` keeps `snake_case` names even in a codec built with `JsonCodecDeriver#withFieldNameMapper(NameMapper.KebabCase)`, and a `@Modifier.discriminator` on a sealed trait overrides `JsonCodecDeriver#withDiscriminatorKind`. `@Modifier.noExtraFields()` can only turn rejection on, never off.

To see the precedence in action, annotate a type with one strategy and build a codec with another:

```scala

@Modifier.fieldNaming("snake_case")
case class Book(title: String, publishedYear: Int)
object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

val kebabCodec: JsonCodec[Book] =
  Book.schema.derive(JsonCodecDeriver.withFieldNameMapper(NameMapper.KebabCase))
```

The codec still writes `snake_case` names, because the annotation on `Book` takes precedence over the deriver setting:

```scala
println(kebabCodec.encodeToString(Book("Zionomicon", 2024)))
// {"title":"Zionomicon","published_year":2024}
```

## Putting It Together

This example combines the recipes into one API format: `snake_case` field names, a renamed ISBN with an alias, a `"type"` field for book formats, explicit `null` values and strict decoding of unknown fields:

```scala title="schema-examples/src/main/scala/customizejsonencoding/CompleteExample.scala"
package customizejsonencoding

/**
 * Customize JSON: Putting It Together — Complete Example
 *
 * One API format combining snake_case field names, a renamed ISBN with an
 * alias, a "type" field for book formats, explicit null values and strict
 * decoding of unknown fields.
 *
 * Run with: cd schema-examples && sbt "runMain
 * customizejsonencoding.CompleteExample"
 */
object CompleteExample extends App {

  @Modifier.discriminator("type")
  @Modifier.caseNaming("snake_case")
  sealed trait Format
  object Format {
    case class Paperback(pages: Int)   extends Format
    case class Ebook(sizeMb: Double)   extends Format
    case class Audiobook(minutes: Int) extends Format

    implicit val schema: Schema[Format] = Schema.derived
  }

  case class Author(fullName: String)
  object Author {
    implicit val schema: Schema[Author] = Schema.derived
  }

  case class Book(
    title: String,
    publishedYear: Int,
    @Modifier.rename("isbn")
    @Modifier.alias("isbn_13")
    isbn13: Option[String],
    author: Author,
    tags: List[String],
    edition: Option[Int],
    formats: List[Format]
  )
  object Book {
    implicit val schema: Schema[Book] = Schema.derived

    // Keep the codec in one place and use it everywhere this JSON is read or written
    val apiCodec: JsonCodec[Book] =
      schema.derive(
        JsonCodecDeriver
          .withFieldNameMapper(NameMapper.SnakeCase)
          .withTransientNone(false)
          .withRejectExtraFields(true)
      )
  }

  val zionomicon = Book("Zionomicon", 2024, None, Author("John A. De Goes"), Nil, None, Nil)

  val fpInScala = Book(
    "Functional Programming in Scala",
    2014,
    Some("9781617290657"),
    Author("Paul Chiusano"),
    List("scala", "functional-programming"),
    Some(1),
    List(Format.Paperback(320))
  )

  // Encode both books
  println(Book.apiCodec.encodeToString(zionomicon))
  println(Book.apiCodec.encodeToString(fpInScala))

  // Decode a payload that still uses the alias
  println(
    Book.apiCodec.decode(
      """{"title":"Programming in Scala","published_year":2008,"isbn_13":"9780981531601","author":{"full_name":"Martin Odersky"},"edition":null}"""
    )
  )

  // Reject a payload with an unknown field
  Book.apiCodec.decode(
    """{"title":"Zionomicon","published_year":2024,"isbn":null,"author":{"full_name":"John A. De Goes"},"edition":null,"subtitle":"A Guide"}"""
  ) match {
    case Left(error) => println(error.message)
    case Right(book) => println(s"Decoded: $book")
  }

  // Check the round trip
  println(Book.apiCodec.decode(Book.apiCodec.encodeToString(fpInScala)) == Right(fpInScala))
}
```

## Running the Examples

All examples in this guide have corresponding runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

Each example runs on its own, so you can jump straight to the recipe you need.

### Rename Every Field With a Naming Strategy

This example applies `snake_case` names globally with `JsonCodecDeriver#withFieldNameMapper` and to one type with `@Modifier.fieldNaming`:

<details>
  <summary>schema-examples/src/main/scala/customizejsonencoding/FieldNamingExample.scala</summary>

```scala title="schema-examples/src/main/scala/customizejsonencoding/FieldNamingExample.scala" showLineNumbers
package customizejsonencoding

/**
 * Customize JSON: Rename Every Field With a Naming Strategy
 *
 * Applies snake_case names to every type with JsonCodecDeriver, and to one type
 * with Modifier.fieldNaming.
 *
 * Run with: cd schema-examples && sbt "runMain
 * customizejsonencoding.FieldNamingExample"
 */
object FieldNamingExample extends App {

  case class Author(fullName: String)
  object Author {
    implicit val schema: Schema[Author] = Schema.derived
  }

  case class Book(
    title: String,
    publishedYear: Int,
    isbn13: Option[String],
    author: Author,
    tags: List[String],
    edition: Option[Int],
    language: String = "en"
  )
  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  val fpInScala = Book(
    "Functional Programming in Scala",
    2014,
    Some("9781617290657"),
    Author("Paul Chiusano"),
    List("scala", "functional-programming"),
    Some(1)
  )

  // With no customization, every field is written under its Scala name
  println(fpInScala.toJsonString)

  // A codec derived with a field name mapper applies to every type it touches
  val snakeCaseCodec: JsonCodec[Book] =
    Book.schema.derive(JsonCodecDeriver.withFieldNameMapper(NameMapper.SnakeCase))

  println(snakeCaseCodec.encodeToString(fpInScala))
  println(snakeCaseCodec.decode(snakeCaseCodec.encodeToString(fpInScala)) == Right(fpInScala))

  // Modifier.fieldNaming applies to one type and travels with the schema
  @Modifier.fieldNaming("snake_case")
  case class AnnotatedBook(title: String, publishedYear: Int, isbn13: Option[String], author: Author)
  object AnnotatedBook {
    implicit val schema: Schema[AnnotatedBook] = Schema.derived
  }

  println(
    AnnotatedBook("Programming in Scala", 2008, Some("9780981531601"), Author("Martin Odersky")).toJsonString
  )
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain customizejsonencoding.FieldNamingExample"
```

### Rename One Field and Accept Its Old Name

This example renames one field with `@Modifier.rename`, accepts an old name with `@Modifier.alias`, and shows that the original Scala name is no longer read:

<details>
  <summary>schema-examples/src/main/scala/customizejsonencoding/RenameAndAliasExample.scala</summary>

```scala title="schema-examples/src/main/scala/customizejsonencoding/RenameAndAliasExample.scala" showLineNumbers
package customizejsonencoding

/**
 * Customize JSON: Rename One Field and Accept Its Old Name
 *
 * Renames one field with Modifier.rename, accepts an old name with
 * Modifier.alias, and shows the Scala name is no longer read.
 *
 * Run with: cd schema-examples && sbt "runMain
 * customizejsonencoding.RenameAndAliasExample"
 */
object RenameAndAliasExample extends App {

  case class Book(
    title: String,
    @Modifier.rename("isbn")
    @Modifier.alias("isbn_13")
    isbn13: Option[String]
  )
  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  val pis = Book("Programming in Scala", Some("9780981531601"))

  // The encoder writes the renamed key
  println(pis.toJsonString)

  // The decoder accepts both the new name and the alias
  println("""{"title":"Programming in Scala","isbn":"9780981531601"}""".fromJson[Book])
  println("""{"title":"Programming in Scala","isbn_13":"9780981531601"}""".fromJson[Book])

  // The original Scala name is no longer accepted, so the value is lost
  println("""{"title":"Programming in Scala","isbn13":"9780981531601"}""".fromJson[Book])
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain customizejsonencoding.RenameAndAliasExample"
```

### Tag Sealed Trait Cases With a Type Field

This example compares the default wrapper-key encoding with a `"type"` field, set first through `JsonCodecDeriver` and then through modifiers on the sealed trait:

<details>
  <summary>schema-examples/src/main/scala/customizejsonencoding/DiscriminatorFieldExample.scala</summary>

```scala title="schema-examples/src/main/scala/customizejsonencoding/DiscriminatorFieldExample.scala" showLineNumbers
package customizejsonencoding

/**
 * Customize JSON: Tag Sealed Trait Cases With a Type Field
 *
 * Compares the default wrapper-key encoding with a "type" field, set through
 * JsonCodecDeriver and then through modifiers on the sealed trait.
 *
 * Run with: cd schema-examples && sbt "runMain
 * customizejsonencoding.DiscriminatorFieldExample"
 */
object DiscriminatorFieldExample extends App {

  sealed trait Format
  object Format {
    case class Paperback(pages: Int)   extends Format
    case class Ebook(sizeMb: Double)   extends Format
    case class Audiobook(minutes: Int) extends Format

    implicit val schema: Schema[Format] = Schema.derived
  }

  val paperback: Format = Format.Paperback(320)

  // By default, the case name is the key of a wrapper object
  println(paperback.toJsonString)

  // A deriver setting produces a flat object with a "type" field
  val typeFieldCodec: JsonCodec[Format] =
    Format.schema.derive(
      JsonCodecDeriver
        .withDiscriminatorKind(DiscriminatorKind.Field("type"))
        .withCaseNameMapper(NameMapper.SnakeCase)
    )

  println(typeFieldCodec.encodeToString(paperback))
  println(typeFieldCodec.decode("""{"type":"audiobook","minutes":600}"""))

  // The same choice can live on the sealed trait itself
  @Modifier.discriminator("type")
  @Modifier.caseNaming("snake_case")
  sealed trait AnnotatedFormat
  object AnnotatedFormat {
    case class Paperback(pages: Int) extends AnnotatedFormat
    case class Ebook(sizeMb: Double) extends AnnotatedFormat
    @Modifier.rename("audio")
    case class Audiobook(minutes: Int) extends AnnotatedFormat

    implicit val schema: Schema[AnnotatedFormat] = Schema.derived
  }

  val formats: List[AnnotatedFormat] =
    List(AnnotatedFormat.Paperback(320), AnnotatedFormat.Ebook(4.5), AnnotatedFormat.Audiobook(600))

  formats.foreach(format => println(format.toJsonString))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain customizejsonencoding.DiscriminatorFieldExample"
```

### Encode Sealed Traits Without a Tag

This example encodes sealed trait cases without a tag using `DiscriminatorKind.None` and decodes them by trying each case in order:

<details>
  <summary>schema-examples/src/main/scala/customizejsonencoding/NoDiscriminatorExample.scala</summary>

```scala title="schema-examples/src/main/scala/customizejsonencoding/NoDiscriminatorExample.scala" showLineNumbers
package customizejsonencoding

/**
 * Customize JSON: Encode Sealed Traits Without a Tag
 *
 * Writes each case as a plain object with DiscriminatorKind.None and decodes by
 * trying the cases in declaration order.
 *
 * Run with: cd schema-examples && sbt "runMain
 * customizejsonencoding.NoDiscriminatorExample"
 */
object NoDiscriminatorExample extends App {

  sealed trait Format
  object Format {
    case class Paperback(pages: Int)   extends Format
    case class Ebook(sizeMb: Double)   extends Format
    case class Audiobook(minutes: Int) extends Format

    implicit val schema: Schema[Format] = Schema.derived
  }

  val untaggedCodec: JsonCodec[Format] =
    Format.schema.derive(JsonCodecDeriver.withDiscriminatorKind(DiscriminatorKind.None))

  // Encoding drops the tag, and decoding picks the first case whose fields fit
  println(untaggedCodec.encodeToString(Format.Paperback(320)))
  println(untaggedCodec.decode("""{"minutes":600}"""))

  // Input that fits no case fails with one generic message
  untaggedCodec.decode("""{"chapters":12}""") match {
    case Left(error)   => println(error.message)
    case Right(format) => println(s"Decoded: $format")
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain customizejsonencoding.NoDiscriminatorExample"
```

### Encode Case-Object Enums as Strings

This example writes a case-object enumeration as a string, with a case name mapper, and as a wrapper object:

<details>
  <summary>schema-examples/src/main/scala/customizejsonencoding/EnumStringsExample.scala</summary>

```scala title="schema-examples/src/main/scala/customizejsonencoding/EnumStringsExample.scala" showLineNumbers
package customizejsonencoding

/**
 * Customize JSON: Encode Case-Object Enums as Strings
 *
 * Writes a case-object enumeration as a string, with a case name mapper, and as
 * a wrapper object.
 *
 * Run with: cd schema-examples && sbt "runMain
 * customizejsonencoding.EnumStringsExample"
 */
object EnumStringsExample extends App {

  sealed trait Availability
  object Availability {
    case object InStock    extends Availability
    case object OutOfPrint extends Availability
    case object Preorder   extends Availability

    implicit val schema: Schema[Availability] = Schema.derived
  }

  val outOfPrint: Availability = Availability.OutOfPrint

  println(outOfPrint.toJsonString)
  println(
    Availability.schema.derive(JsonCodecDeriver.withCaseNameMapper(NameMapper.SnakeCase)).encodeToString(outOfPrint)
  )
  println(Availability.schema.derive(JsonCodecDeriver.withEnumValuesAsStrings(false)).encodeToString(outOfPrint))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain customizejsonencoding.EnumStringsExample"
```

### Control Optional Fields, Empty Collections and Defaults

This example writes `None`, empty lists and default values explicitly, and makes decoding require every `Option` field:

<details>
  <summary>schema-examples/src/main/scala/customizejsonencoding/OptionalFieldsExample.scala</summary>

```scala title="schema-examples/src/main/scala/customizejsonencoding/OptionalFieldsExample.scala" showLineNumbers
package customizejsonencoding

/**
 * Customize JSON: Control Optional Fields, Empty Collections and Defaults
 *
 * Writes None, empty lists and default values explicitly, and makes decoding
 * require every Option field.
 *
 * Run with: cd schema-examples && sbt "runMain
 * customizejsonencoding.OptionalFieldsExample"
 */
object OptionalFieldsExample extends App {

  case class Author(fullName: String)
  object Author {
    implicit val schema: Schema[Author] = Schema.derived
  }

  case class Book(
    title: String,
    publishedYear: Int,
    isbn13: Option[String],
    author: Author,
    tags: List[String],
    edition: Option[Int],
    language: String = "en"
  )
  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  val zionomicon = Book("Zionomicon", 2024, None, Author("John A. De Goes"), Nil, None)

  // The default codec leaves out isbn13, tags, edition and language
  println(zionomicon.toJsonString)

  // Turning the three encoder settings off writes every field
  val explicitCodec: JsonCodec[Book] =
    Book.schema.derive(
      JsonCodecDeriver
        .withTransientNone(false)
        .withTransientEmptyCollection(false)
        .withTransientDefaultValue(false)
    )

  println(explicitCodec.encodeToString(zionomicon))
  println(explicitCodec.decode(explicitCodec.encodeToString(zionomicon)) == Right(zionomicon))

  // The decoder settings make decoding strict
  val strictCodec: JsonCodec[Book] =
    Book.schema.derive(JsonCodecDeriver.withRequireOptionFields(true))

  strictCodec.decode("""{"title":"Zionomicon","publishedYear":2024,"author":{"fullName":"John A. De Goes"}}""") match {
    case Left(error) => println(error.message)
    case Right(book) => println(s"Decoded: $book")
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain customizejsonencoding.OptionalFieldsExample"
```

### Reject Unknown Fields

This example rejects unknown fields for one type with `@Modifier.noExtraFields` and for every type with `JsonCodecDeriver#withRejectExtraFields`:

<details>
  <summary>schema-examples/src/main/scala/customizejsonencoding/RejectExtraFieldsExample.scala</summary>

```scala title="schema-examples/src/main/scala/customizejsonencoding/RejectExtraFieldsExample.scala" showLineNumbers
package customizejsonencoding

/**
 * Customize JSON: Reject Unknown Fields
 *
 * Rejects unknown fields for one type with Modifier.noExtraFields and for every
 * type with JsonCodecDeriver#withRejectExtraFields.
 *
 * Run with: cd schema-examples && sbt "runMain
 * customizejsonencoding.RejectExtraFieldsExample"
 */
object RejectExtraFieldsExample extends App {

  @Modifier.noExtraFields()
  case class Author(fullName: String)
  object Author {
    implicit val schema: Schema[Author] = Schema.derived
  }

  case class Book(title: String, author: Author)
  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  // Book still skips an unknown subtitle
  println("""{"title":"Zionomicon","subtitle":"A Guide","author":{"fullName":"John A. De Goes"}}""".fromJson[Book])

  // Author rejects a misspelled field and reports where it is
  """{"title":"Zionomicon","author":{"fullname":"John A. De Goes"}}""".fromJson[Book] match {
    case Left(error) => println(error.message)
    case Right(book) => println(s"Decoded: $book")
  }

  // A deriver setting rejects unknown fields in every type the codec handles
  Book.schema
    .derive(JsonCodecDeriver.withRejectExtraFields(true))
    .decode("""{"title":"Zionomicon","subtitle":"A Guide","author":{"fullName":"John A. De Goes"}}""") match {
    case Left(error) => println(error.message)
    case Right(book) => println(s"Decoded: $book")
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain customizejsonencoding.RejectExtraFieldsExample"
```

### Leave a Field Out of the JSON

This example leaves one field out in both directions with `@Modifier.transient` and another only when encoding with `@Modifier.encodeTransient`:

<details>
  <summary>schema-examples/src/main/scala/customizejsonencoding/TransientFieldsExample.scala</summary>

```scala title="schema-examples/src/main/scala/customizejsonencoding/TransientFieldsExample.scala" showLineNumbers
package customizejsonencoding

/**
 * Customize JSON: Leave a Field Out of the JSON
 *
 * Leaves one field out in both directions with Modifier.transient and another
 * only when encoding with Modifier.encodeTransient.
 *
 * Run with: cd schema-examples && sbt "runMain
 * customizejsonencoding.TransientFieldsExample"
 */
object TransientFieldsExample extends App {

  case class Book(
    title: String,
    @Modifier.transient() searchScore: Double = 0.0,
    @Modifier.encodeTransient() legacyId: Option[Long]
  )
  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  // The encoder writes neither field
  println(Book("Zionomicon", 0.93, Some(42L)).toJsonString)

  // When decoding, a transient field ignores the input and takes its default,
  // while an encode-transient field is still read
  println("""{"title":"Zionomicon","searchScore":0.5,"legacyId":42}""".fromJson[Book])

  // A transient field without a default value (and that is not an Option or a
  // collection) fails at compile time, so that case is not shown here.
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain customizejsonencoding.TransientFieldsExample"
```

### The Complete Example

The complete example from "Putting It Together" combines every recipe into one API format:

```bash
sbt "runMain customizejsonencoding.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## Going Further

- [Modifier](../reference/schema/modifier.md) documents every modifier, including the JSON modifiers used in this guide.
- [JSON Configuration](../reference/schema/built-in-codecs/json/json-config.md) covers `WriterConfig` for pretty-printing, `ReaderConfig` and `NameMapper`.
- [How to Tune JSON Encoding and Decoding for Throughput](./tune-json-throughput.md) shows how to decode from bytes and streams, reuse codecs and size buffers for speed.
- [Your First Schema](./your-first-schema.md) is the tutorial this guide builds on.
- To change how one type is encoded everywhere, such as a custom `LocalDate` format, see the instance overrides in [Type-Class Derivation](../reference/schema/type-class-derivation.md).

---

## How to Derive Your Own Type Class from a Schema

In this guide, we write `Redact[A]`, a type class that renders a value for a log line with its sensitive fields replaced by `***`, and we derive it for a small Scala bookstore from each type's `Schema`. By the end, any case class or sealed trait with a schema gets a redacting renderer for free, sensitive fields are marked with one annotation, and the type class is packaged so its users write `derives Schema, Redact`.

The [Type-Class Derivation](../reference/schema/type-class-derivation.md) reference already builds a `Show` and a `Gen` type class and walks through every `Deriver` method in depth; that is the place to learn each method's parameters. This guide is the short recipe for your own type class, and it adds the parts a real library needs: reading modifiers during derivation, overriding one type, and packaging the result.

We cover:

- Defining the type class
- Handling primitives, records and variants
- Handling collections, wrappers and dynamic values
- Deriving the type class and printing the result
- Masking fields through modifiers
- Overriding the instance for one type
- Packaging the type class for reuse

## The Problem

Logs must not contain payment tokens, card numbers or email addresses, but a case class's `toString` prints every field. The usual fix is a hand-written log renderer for each type, which must mask the right fields and recurse into every nested type.

Every new type needs another renderer, and every new field needs an edit to an existing one. Nothing fails when someone adds a sensitive field and forgets to mask it; the value goes straight into the logs. Here is that approach written by hand:

```scala
final case class LoggedBook(title: String, firstAuthor: String, year: Int)
final case class LoggedCustomer(name: String, email: String)
final case class LoggedOrder(id: Long, customer: LoggedCustomer, books: List[LoggedBook], paymentToken: String)

def bookForLog(b: LoggedBook): String =
  s"""LoggedBook(title = "${b.title}", firstAuthor = "${b.firstAuthor}", year = ${b.year})"""

def customerForLog(c: LoggedCustomer): String =
  s"""LoggedCustomer(name = "${c.name}", email = ***)"""

def orderForLog(o: LoggedOrder): String =
  s"""LoggedOrder(id = ${o.id}, customer = ${customerForLog(o.customer)}, books = ${o.books
      .map(bookForLog)
      .mkString("[", ", ", "]")}, paymentToken = ***)"""
```

This doesn't scale past a handful of types. The masking rules live in string templates far from the fields they protect, and a library cannot ship such renderers for its users' types at all.

In ZIO Blocks, a `Schema[A]` describes the structure of `A`, and a `Deriver[TC]` turns that structure into a `TC[A]` for any type. We write one `Deriver[Redact]`, mark the sensitive fields in the case class, and derive every renderer from the schema.

## Prerequisites

This guide builds on [Your First Schema](./your-first-schema.md): you should know how to derive a `Schema` for a case class. Add the schema module to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56"
```

We use these imports throughout:

```scala

```

## The Bookstore Model

Our bookstore sells books in several formats, and an `Order` holds a customer, the books bought, and a payment token. The token is the field that must never reach a log, so we mark it with `Modifier.config("log.redact", "true")`; the deriver reads that marker later. `Email` is a wrapper type with its own schema, built with `transform`:

```scala
sealed trait Format
object Format {
  case object Hardcover                  extends Format
  case object Paperback                  extends Format
  final case class Ebook(sizeMb: Double) extends Format

  implicit val schema: Schema[Format] = Schema.derived[Format]
}

final case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String], format: Format)
object Book {
  implicit val schema: Schema[Book] = Schema.derived[Book]
}

final case class Email(value: String)
object Email {
  implicit val schema: Schema[Email] = Schema[String].transform(Email(_), _.value)
}

final case class Customer(name: String, email: Email)
object Customer {
  implicit val schema: Schema[Customer] = Schema.derived[Customer]
}

final case class Order(
  id: Long,
  customer: Customer,
  books: List[Book],
  @Modifier.config("log.redact", "true") paymentToken: String
)
object Order {
  implicit val schema: Schema[Order] = Schema.derived[Order]
}
```

We use three real Scala books as data. `Book` has a single author field, which holds the first author:

```scala
val zionomicon         = Book("Zionomicon", "John A. De Goes", 2024, None, Format.Ebook(12.5))
val programmingInScala = Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"), Format.Hardcover)
val fpInScala          = Book("Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"), Format.Paperback)

val order = Order(1001L, Customer("Ada", Email("ada@example.com")), List(zionomicon), "tok_4242")
```

## Defining the Type Class

A type class for derivation is an ordinary trait with one type parameter. `Redact[A]` turns an `A` into a log-safe string. The companion holds the mask text and a `derived` method that builds an instance from a schema with the deriver we write next:

```scala
trait Redact[A] {
  def redact(value: A): String
}

object Redact {
  val Masked = "***"

  def derived[A](implicit schema: Schema[A]): Redact[A] = schema.derive(RedactDeriver)
}
```

`Schema#derive` walks the schema's structure and calls one `Deriver` method per node. We implement those methods in the next two sections, then return to `derived` in [Packaging the Type Class for Reuse](#packaging-the-type-class-for-reuse).

## Handling Primitives, Records and Variants

A `Deriver[Redact]` has one method for each kind of schema node. Primitives, records and variants carry the shape of a domain model, so we put them in a trait of their own:

```scala
trait RedactStructure extends Deriver[Redact] {

  protected def isSensitive(modifiers: Seq[Modifier]): Boolean =
    modifiers.exists {
      case Modifier.config("log.redact", "true") => true
      case _                                     => false
    }

  override def derivePrimitive[A](
    primitiveType: PrimitiveType[A],
    typeId: TypeId[A],
    binding: Binding.Primitive[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  ): Lazy[Redact[A]] =
    Lazy {
      primitiveType match {
        case _: PrimitiveType.String => (value: A) => "\"" + value + "\""
        case _                       => (value: A) => String.valueOf(value)
      }
    }

  override def deriveRecord[F[_, _], A](
    fields: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Record[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[A]] = {
    val names     = fields.map(_.name)
    val sensitive = fields.map(field => isSensitive(field.modifiers))
    val reflect   = new Reflect.Record[Binding, A](
      fields.asInstanceOf[IndexedSeq[Term[Binding, A, ?]]],
      typeId,
      binding,
      doc,
      modifiers
    )
    Lazy {
      new Redact[A] {
        private lazy val children: IndexedSeq[Redact[Any]] =
          fields.map(field => D.instance(field.value.metadata).force.asInstanceOf[Redact[Any]])

        def redact(value: A): String = {
          val registers = Registers(reflect.usedRegisters)
          binding.deconstructor.deconstruct(registers, RegisterOffset.Zero, value)
          val rendered = fields.indices.map { i =>
            val shown =
              if (sensitive(i)) Redact.Masked
              else children(i).redact(reflect.registers(i).get(registers, RegisterOffset.Zero))
            s"${names(i)} = $shown"
          }
          if (fields.isEmpty) typeId.name else rendered.mkString(s"${typeId.name}(", ", ", ")")
        }
      }
    }
  }

  override def deriveVariant[F[_, _], A](
    cases: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Variant[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[A]] =
    Lazy {
      new Redact[A] {
        private lazy val children: IndexedSeq[Redact[Any]] =
          cases.map(c => D.instance(c.value.metadata).force.asInstanceOf[Redact[Any]])

        def redact(value: A): String = {
          val i = binding.discriminator.discriminate(value)
          children(i).redact(binding.matchers(i).downcastOrNull(value))
        }
      }
    }
}
```

Each method returns a `Lazy[Redact[A]]`, and the instances for child nodes come from `D.instance(...)`, also as `Lazy` values:

- **`derivePrimitive`** quotes strings and prints every other primitive with `String.valueOf`.
- **`deriveRecord`** reads each field's name and modifiers once, at derivation time. At run time it copies the field values out of the record with the binding's deconstructor and renders each one with its child instance, or prints `***` for a sensitive field. The temporary `Reflect.Record` gives us the register positions of the fields; [Record Derivation](../reference/schema/type-class-derivation.md#record-derivation) in the reference explains this pattern line by line.
- **`deriveVariant`** asks the binding's discriminator which case a value belongs to, downcasts it, and delegates to that case's instance.

Both `deriveRecord` and `deriveVariant` force their child instances inside a `private lazy val`, at the first `redact` call, not during derivation. A recursive type refers back to an instance that is still being built, and forcing it early would loop forever. See [Lazy](../reference/schema/lazy.md) for how `Lazy` breaks such cycles.

## Handling Collections, Wrappers and Dynamic Values

The remaining four methods cover containers. We implement them on the object our users will pass to `derive`:

```scala
object RedactDeriver extends RedactStructure {

  override def deriveSequence[F[_, _], C[_], A](
    element: Reflect[F, A],
    typeId: TypeId[C[A]],
    binding: Binding.Seq[C, A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[C[A]],
    examples: Seq[C[A]]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[C[A]]] =
    D.instance(element.metadata).map { elem => (value: C[A]) =>
      binding.deconstructor.deconstruct(value).map(elem.redact).mkString("[", ", ", "]")
    }

  override def deriveMap[F[_, _], M[_, _], K, V](
    key: Reflect[F, K],
    value: Reflect[F, V],
    typeId: TypeId[M[K, V]],
    binding: Binding.Map[M, K, V],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[M[K, V]],
    examples: Seq[M[K, V]]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[M[K, V]]] =
    D.instance(key.metadata).zip(D.instance(value.metadata)).map { case (k, v) => (m: M[K, V]) =>
      val d = binding.deconstructor
      d.deconstruct(m).map(kv => s"${k.redact(d.getKey(kv))} -> ${v.redact(d.getValue(kv))}").mkString("{", ", ", "}")
    }

  override def deriveWrapper[F[_, _], A, B](
    wrapped: Reflect[F, B],
    typeId: TypeId[A],
    binding: Binding.Wrapper[A, B],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[A]] =
    D.instance(wrapped.metadata).map(inner => (value: A) => inner.redact(binding.unwrap(value)))

  override def deriveDynamic[F[_, _]](
    binding: Binding.Dynamic,
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[DynamicValue],
    examples: Seq[DynamicValue]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[DynamicValue]] =
    Lazy((_: DynamicValue) => Redact.Masked)
}
```

Sequences, maps and wrappers are not recursive on their own, so they combine their child instances with `Lazy#map` and `Lazy#zip` instead of a `lazy val`. A wrapper unwraps its value and renders the inner one, so `Email("ada@example.com")` prints as a plain string.

A `DynamicValue` has no schema and so no modifiers; the deriver cannot tell which of its parts are sensitive. For a log redactor the safe choice is to mask the whole value. A type class with no sensible behaviour for a node kind can fail instead, by returning `Lazy.fail(new UnsupportedOperationException("..."))`, which throws when the instance is forced.

## Deriving the Type Class and Printing the Result

`Schema#derive` takes the deriver and returns the instance. We derive `Redact[Book]` and render our three books:

```scala
val redactBook: Redact[Book] = Book.schema.derive(RedactDeriver)
```

```scala
List(zionomicon, programmingInScala, fpInScala).foreach(book => println(redactBook.redact(book)))
// Book(title = "Zionomicon", firstAuthor = "John A. De Goes", year = 2024, isbn = None, format = Ebook(sizeMb = 12.5))
// Book(title = "Programming in Scala", firstAuthor = "Martin Odersky", year = 2008, isbn = Some(value = "9780981531601"), format = Hardcover)
// Book(title = "Functional Programming in Scala", firstAuthor = "Paul Chiusano", year = 2014, isbn = Some(value = "9781617290657"), format = Paperback)
```

The `format` field is a sealed trait, so `deriveVariant` picks the case and the case's record instance renders it; a case object is a record with no fields and prints as its name. `Option` is a variant too, with a `Some` record holding one `value` field and a field-less `None`.

Now the whole order. The nested `Customer`, the list of books and the `Email` wrapper all come from the same deriver:

```scala
val redactOrder: Redact[Order] = Order.schema.derive(RedactDeriver)
```

```scala
println(redactOrder.redact(order))
// Order(id = 1001, customer = Customer(name = "Ada", email = "ada@example.com"), books = [Book(title = "Zionomicon", firstAuthor = "John A. De Goes", year = 2024, isbn = None, format = Ebook(sizeMb = 12.5))], paymentToken = ***)
```

The payment token is masked. The customer's email address is still in plain text; the next two sections fix that.

## Masking Fields Through Modifiers

The `@Modifier.config("log.redact", "true")` annotation on `paymentToken` reaches `deriveRecord` through the field's `Term#modifiers`, which is where `isSensitive` looks. `Modifier.config` takes any key and value, so a type class can define its own markers without new annotation types; [`config`](../reference/schema/modifier.md#config) in the Modifier reference describes the key convention. The annotation works on case class fields in Scala 2.13 and Scala 3.

When you cannot annotate the case class, for example because it comes from another library, attach the same modifier from outside with `Deriver#withModifier`. It takes the owning type's `TypeId`, the field name and a `Modifier.Term`, and returns a new deriver. Here we mask the customer's name as well:

```scala
val strictDeriver: Deriver[Redact] =
  RedactDeriver.withModifier(TypeId.of[Customer], "name", Modifier.config("log.redact", "true"))
```

```scala
println(Order.schema.derive(strictDeriver).redact(order))
// Order(id = 1001, customer = Customer(name = ***, email = "ada@example.com"), books = [Book(title = "Zionomicon", firstAuthor = "John A. De Goes", year = 2024, isbn = None, format = Ebook(sizeMb = 12.5))], paymentToken = ***)
```

:::warning
`withModifier` silently ignores a field name that does not exist in the type. Check the output after renaming a field.
:::

## Overriding the Instance for One Type

Sometimes the rule belongs to a type, not a field: every `Email`, wherever it appears, should show only its first letter and its domain. Write that instance by hand and register it once on the deriver with `Deriver#withInstance`. Every derivation through the returned deriver uses it instead of calling `deriveWrapper` for `Email`:

```scala
val partialEmail: Redact[Email] = (email: Email) => {
  val at = email.value.indexOf('@')
  email.value.take(1) + Redact.Masked + email.value.drop(at)
}

val logDeriver: Deriver[Redact] = RedactDeriver.withInstance[Email](partialEmail)
```

```scala
println(Order.schema.derive(logDeriver).redact(order))
// Order(id = 1001, customer = Customer(name = "Ada", email = a***@example.com), books = [Book(title = "Zionomicon", firstAuthor = "John A. De Goes", year = 2024, isbn = None, format = Ebook(sizeMb = 12.5))], paymentToken = ***)
```

Derivation never searches implicit scope, so declaring `implicit val partialEmail` next to `Email` has no effect on the derived instances. Register the instance on the deriver, as above.

For an override that applies to one derivation only, use `Schema#deriving` and `DerivationBuilder#instance` instead:

```scala
println(Order.schema.deriving(RedactDeriver).instance(TypeId.of[Email], partialEmail).derive.redact(order))
// Order(id = 1001, customer = Customer(name = "Ada", email = a***@example.com), books = [Book(title = "Zionomicon", firstAuthor = "John A. De Goes", year = 2024, isbn = None, format = Ebook(sizeMb = 12.5))], paymentToken = ***)
```

:::warning
When the deriver and the builder both override the same type, the deriver's `withInstance` wins and the builder's `instance` is silently ignored. Override a type in one place only.
:::

[How to Override the Codec for One Type Everywhere](./override-codec-for-one-type.md) applies the same tools to JSON codecs and compares every override scope.

## Packaging the Type Class for Reuse

Users of our type class should not need to know about `RedactDeriver`. The `derived` method on the `Redact` companion takes the schema as an implicit parameter and derives the instance, and in Scala 3 that is all a `derives` clause needs. A recursive category tree shows that the `lazy val` pattern from `deriveRecord` handles recursion:

```scala
final case class Category(name: String, subcategories: List[Category]) derives Schema, Redact

val programming = Category("Programming", List(Category("Scala", List(Category("Functional Programming", Nil)))))
```

```scala
println(summon[Redact[Category]].redact(programming))
// Category(name = "Programming", subcategories = [Category(name = "Scala", subcategories = [Category(name = "Functional Programming", subcategories = [])])])
```

`derives Schema, Redact` generates a `Schema[Category]` and then a `Redact[Category]` from it, both in the companion object. Scala 2.13 has no `derives` clause, so declare the instances in the companion yourself:

```scala
// Scala 2.13
final case class Category(name: String, subcategories: List[Category])
object Category {
  implicit val schema: Schema[Category] = Schema.derived[Category]
  implicit val redact: Redact[Category] = Redact.derived[Category]
}
```

The deriver code itself is the same in both Scala versions, apart from wildcard syntax: Scala 2.13 writes `Term[F, A, _]` where Scala 3 writes `Term[F, A, ?]`.

Store the derived instance in a `val`, as both forms do. Each `derive` call walks the whole schema again.

## Putting It Together

This example puts the whole type class, the deriver, the bookstore model, a field marked through `withModifier` and the `Email` override into one program:

```scala title="schema-examples/src/main/scala/deriveyourowntypeclass/CompleteExample.scala"
package deriveyourowntypeclass

/**
 * How to Derive Your Own Type Class from a Schema — The Complete Example
 *
 * The whole type class, deriver, bookstore model, a field masked through
 * withModifier and the Email override in one program.
 *
 * Run with: cd schema-examples && sbt "runMain
 * deriveyourowntypeclass.CompleteExample"
 */
object CompleteExample {

  sealed trait Format
  object Format {
    case object Hardcover                  extends Format
    case object Paperback                  extends Format
    final case class Ebook(sizeMb: Double) extends Format

    implicit val schema: Schema[Format] = Schema.derived[Format]
  }

  final case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String], format: Format)
  object Book {
    implicit val schema: Schema[Book] = Schema.derived[Book]
  }

  final case class Email(value: String)
  object Email {
    implicit val schema: Schema[Email] = Schema[String].transform(Email(_), _.value)
  }

  final case class Customer(name: String, email: Email)
  object Customer {
    implicit val schema: Schema[Customer] = Schema.derived[Customer]
  }

  final case class Order(
    id: Long,
    customer: Customer,
    books: List[Book],
    @Modifier.config("log.redact", "true") paymentToken: String
  )
  object Order {
    implicit val schema: Schema[Order] = Schema.derived[Order]
  }

  val zionomicon         = Book("Zionomicon", "John A. De Goes", 2024, None, Format.Ebook(12.5))
  val programmingInScala = Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"), Format.Hardcover)
  val fpInScala          =
    Book("Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"), Format.Paperback)

  val order = Order(1001L, Customer("Ada", Email("ada@example.com")), List(zionomicon), "tok_4242")

  trait Redact[A] {
    def redact(value: A): String
  }

  object Redact {
    val Masked = "***"

    def derived[A](implicit schema: Schema[A]): Redact[A] = schema.derive(RedactDeriver)
  }

  trait RedactStructure extends Deriver[Redact] {

    protected def isSensitive(modifiers: Seq[Modifier]): Boolean =
      modifiers.exists {
        case Modifier.config("log.redact", "true") => true
        case _                                     => false
      }

    override def derivePrimitive[A](
      primitiveType: PrimitiveType[A],
      typeId: TypeId[A],
      binding: Binding.Primitive[A],
      doc: Doc,
      modifiers: Seq[Modifier.Reflect],
      defaultValue: Option[A],
      examples: Seq[A]
    ): Lazy[Redact[A]] =
      Lazy {
        primitiveType match {
          case _: PrimitiveType.String => (value: A) => "\"" + value + "\""
          case _                       => (value: A) => String.valueOf(value)
        }
      }

    override def deriveRecord[F[_, _], A](
      fields: IndexedSeq[Term[F, A, ?]],
      typeId: TypeId[A],
      binding: Binding.Record[A],
      doc: Doc,
      modifiers: Seq[Modifier.Reflect],
      defaultValue: Option[A],
      examples: Seq[A]
    )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[A]] = {
      val names     = fields.map(_.name)
      val sensitive = fields.map(field => isSensitive(field.modifiers))
      val reflect   = new Reflect.Record[Binding, A](
        fields.asInstanceOf[IndexedSeq[Term[Binding, A, ?]]],
        typeId,
        binding,
        doc,
        modifiers
      )
      Lazy {
        new Redact[A] {
          private lazy val children: IndexedSeq[Redact[Any]] =
            fields.map(field => D.instance(field.value.metadata).force.asInstanceOf[Redact[Any]])

          def redact(value: A): String = {
            val registers = Registers(reflect.usedRegisters)
            binding.deconstructor.deconstruct(registers, RegisterOffset.Zero, value)
            val rendered = fields.indices.map { i =>
              val shown =
                if (sensitive(i)) Redact.Masked
                else children(i).redact(reflect.registers(i).get(registers, RegisterOffset.Zero))
              s"${names(i)} = $shown"
            }
            if (fields.isEmpty) typeId.name else rendered.mkString(s"${typeId.name}(", ", ", ")")
          }
        }
      }
    }

    override def deriveVariant[F[_, _], A](
      cases: IndexedSeq[Term[F, A, ?]],
      typeId: TypeId[A],
      binding: Binding.Variant[A],
      doc: Doc,
      modifiers: Seq[Modifier.Reflect],
      defaultValue: Option[A],
      examples: Seq[A]
    )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[A]] =
      Lazy {
        new Redact[A] {
          private lazy val children: IndexedSeq[Redact[Any]] =
            cases.map(c => D.instance(c.value.metadata).force.asInstanceOf[Redact[Any]])

          def redact(value: A): String = {
            val i = binding.discriminator.discriminate(value)
            children(i).redact(binding.matchers(i).downcastOrNull(value))
          }
        }
      }
  }

  object RedactDeriver extends RedactStructure {

    override def deriveSequence[F[_, _], C[_], A](
      element: Reflect[F, A],
      typeId: TypeId[C[A]],
      binding: Binding.Seq[C, A],
      doc: Doc,
      modifiers: Seq[Modifier.Reflect],
      defaultValue: Option[C[A]],
      examples: Seq[C[A]]
    )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[C[A]]] =
      D.instance(element.metadata).map { elem => (value: C[A]) =>
        binding.deconstructor.deconstruct(value).map(elem.redact).mkString("[", ", ", "]")
      }

    override def deriveMap[F[_, _], M[_, _], K, V](
      key: Reflect[F, K],
      value: Reflect[F, V],
      typeId: TypeId[M[K, V]],
      binding: Binding.Map[M, K, V],
      doc: Doc,
      modifiers: Seq[Modifier.Reflect],
      defaultValue: Option[M[K, V]],
      examples: Seq[M[K, V]]
    )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[M[K, V]]] =
      D.instance(key.metadata).zip(D.instance(value.metadata)).map { case (k, v) =>
        (m: M[K, V]) =>
          val d = binding.deconstructor
          d.deconstruct(m)
            .map(kv => s"${k.redact(d.getKey(kv))} -> ${v.redact(d.getValue(kv))}")
            .mkString("{", ", ", "}")
      }

    override def deriveWrapper[F[_, _], A, B](
      wrapped: Reflect[F, B],
      typeId: TypeId[A],
      binding: Binding.Wrapper[A, B],
      doc: Doc,
      modifiers: Seq[Modifier.Reflect],
      defaultValue: Option[A],
      examples: Seq[A]
    )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[A]] =
      D.instance(wrapped.metadata).map(inner => (value: A) => inner.redact(binding.unwrap(value)))

    override def deriveDynamic[F[_, _]](
      binding: Binding.Dynamic,
      doc: Doc,
      modifiers: Seq[Modifier.Reflect],
      defaultValue: Option[DynamicValue],
      examples: Seq[DynamicValue]
    )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[DynamicValue]] =
      Lazy((_: DynamicValue) => Redact.Masked)
  }

  def main(args: Array[String]): Unit = {
    val redactBook: Redact[Book] = Book.schema.derive(RedactDeriver)
    List(zionomicon, programmingInScala, fpInScala).foreach(book => println(redactBook.redact(book)))

    println(Order.schema.derive(RedactDeriver).redact(order))

    val strictDeriver: Deriver[Redact] =
      RedactDeriver.withModifier(TypeId.of[Customer], "name", Modifier.config("log.redact", "true"))
    println(Order.schema.derive(strictDeriver).redact(order))

    val partialEmail: Redact[Email] = (email: Email) => {
      val at = email.value.indexOf('@')
      email.value.take(1) + Redact.Masked + email.value.drop(at)
    }
    val logDeriver: Deriver[Redact] = strictDeriver.withInstance[Email](partialEmail)
    println(Order.schema.derive(logDeriver).redact(order))
  }
}
```

## Running the Examples

All examples in this guide have corresponding runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

The step examples share two files: `Redact.scala` holds the type class and its deriver, and `Bookstore.scala` holds the domain model and sample data.

<details>
  <summary>schema-examples/src/main/scala/deriveyourowntypeclass/Redact.scala</summary>

```scala title="schema-examples/src/main/scala/deriveyourowntypeclass/Redact.scala" showLineNumbers
package deriveyourowntypeclass

// The Redact type class, its structure deriver and RedactDeriver (library file, not runnable).
trait Redact[A] {
  def redact(value: A): String
}

object Redact {
  val Masked = "***"

  def derived[A](implicit schema: Schema[A]): Redact[A] = schema.derive(RedactDeriver)
}

trait RedactStructure extends Deriver[Redact] {

  protected def isSensitive(modifiers: Seq[Modifier]): Boolean =
    modifiers.exists {
      case Modifier.config("log.redact", "true") => true
      case _                                     => false
    }

  override def derivePrimitive[A](
    primitiveType: PrimitiveType[A],
    typeId: TypeId[A],
    binding: Binding.Primitive[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  ): Lazy[Redact[A]] =
    Lazy {
      primitiveType match {
        case _: PrimitiveType.String => (value: A) => "\"" + value + "\""
        case _                       => (value: A) => String.valueOf(value)
      }
    }

  override def deriveRecord[F[_, _], A](
    fields: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Record[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[A]] = {
    val names     = fields.map(_.name)
    val sensitive = fields.map(field => isSensitive(field.modifiers))
    val reflect   = new Reflect.Record[Binding, A](
      fields.asInstanceOf[IndexedSeq[Term[Binding, A, ?]]],
      typeId,
      binding,
      doc,
      modifiers
    )
    Lazy {
      new Redact[A] {
        private lazy val children: IndexedSeq[Redact[Any]] =
          fields.map(field => D.instance(field.value.metadata).force.asInstanceOf[Redact[Any]])

        def redact(value: A): String = {
          val registers = Registers(reflect.usedRegisters)
          binding.deconstructor.deconstruct(registers, RegisterOffset.Zero, value)
          val rendered = fields.indices.map { i =>
            val shown =
              if (sensitive(i)) Redact.Masked
              else children(i).redact(reflect.registers(i).get(registers, RegisterOffset.Zero))
            s"${names(i)} = $shown"
          }
          if (fields.isEmpty) typeId.name else rendered.mkString(s"${typeId.name}(", ", ", ")")
        }
      }
    }
  }

  override def deriveVariant[F[_, _], A](
    cases: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Variant[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[A]] =
    Lazy {
      new Redact[A] {
        private lazy val children: IndexedSeq[Redact[Any]] =
          cases.map(c => D.instance(c.value.metadata).force.asInstanceOf[Redact[Any]])

        def redact(value: A): String = {
          val i = binding.discriminator.discriminate(value)
          children(i).redact(binding.matchers(i).downcastOrNull(value))
        }
      }
    }
}

object RedactDeriver extends RedactStructure {

  override def deriveSequence[F[_, _], C[_], A](
    element: Reflect[F, A],
    typeId: TypeId[C[A]],
    binding: Binding.Seq[C, A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[C[A]],
    examples: Seq[C[A]]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[C[A]]] =
    D.instance(element.metadata).map { elem => (value: C[A]) =>
      binding.deconstructor.deconstruct(value).map(elem.redact).mkString("[", ", ", "]")
    }

  override def deriveMap[F[_, _], M[_, _], K, V](
    key: Reflect[F, K],
    value: Reflect[F, V],
    typeId: TypeId[M[K, V]],
    binding: Binding.Map[M, K, V],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[M[K, V]],
    examples: Seq[M[K, V]]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[M[K, V]]] =
    D.instance(key.metadata).zip(D.instance(value.metadata)).map { case (k, v) =>
      (m: M[K, V]) =>
        val d = binding.deconstructor
        d.deconstruct(m).map(kv => s"${k.redact(d.getKey(kv))} -> ${v.redact(d.getValue(kv))}").mkString("{", ", ", "}")
    }

  override def deriveWrapper[F[_, _], A, B](
    wrapped: Reflect[F, B],
    typeId: TypeId[A],
    binding: Binding.Wrapper[A, B],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[A]] =
    D.instance(wrapped.metadata).map(inner => (value: A) => inner.redact(binding.unwrap(value)))

  override def deriveDynamic[F[_, _]](
    binding: Binding.Dynamic,
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[DynamicValue],
    examples: Seq[DynamicValue]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[Redact[DynamicValue]] =
    Lazy((_: DynamicValue) => Redact.Masked)
}
```

</details>

<details>
  <summary>schema-examples/src/main/scala/deriveyourowntypeclass/Bookstore.scala</summary>

```scala title="schema-examples/src/main/scala/deriveyourowntypeclass/Bookstore.scala" showLineNumbers
package deriveyourowntypeclass

sealed trait Format
object Format {
  case object Hardcover                  extends Format
  case object Paperback                  extends Format
  final case class Ebook(sizeMb: Double) extends Format

  implicit val schema: Schema[Format] = Schema.derived[Format]
}

final case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String], format: Format)
object Book {
  implicit val schema: Schema[Book] = Schema.derived[Book]
}

final case class Email(value: String)
object Email {
  implicit val schema: Schema[Email] = Schema[String].transform(Email(_), _.value)
}

final case class Customer(name: String, email: Email)
object Customer {
  implicit val schema: Schema[Customer] = Schema.derived[Customer]
}

final case class Order(
  id: Long,
  customer: Customer,
  books: List[Book],
  @Modifier.config("log.redact", "true") paymentToken: String
)
object Order {
  implicit val schema: Schema[Order] = Schema.derived[Order]
}

object Bookstore {
  val zionomicon         = Book("Zionomicon", "John A. De Goes", 2024, None, Format.Ebook(12.5))
  val programmingInScala = Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"), Format.Hardcover)
  val fpInScala          =
    Book("Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"), Format.Paperback)

  val order = Order(1001L, Customer("Ada", Email("ada@example.com")), List(zionomicon), "tok_4242")
}
```

</details>

Run the steps in order to follow the guide.

### Deriving the Type Class

This example derives `Redact` for `Book` and `Order` and prints the three books and the order.

<details>
  <summary>schema-examples/src/main/scala/deriveyourowntypeclass/Step1DeriveExample.scala</summary>

```scala title="schema-examples/src/main/scala/deriveyourowntypeclass/Step1DeriveExample.scala" showLineNumbers
package deriveyourowntypeclass

/**
 * How to Derive Your Own Type Class from a Schema — Deriving the Type Class and
 * Printing the Result
 *
 * Derives Redact for Book and Order from their schemas and prints the three
 * books and the order.
 *
 * Run with: cd schema-examples && sbt "runMain
 * deriveyourowntypeclass.Step1DeriveExample"
 */
object Step1DeriveExample {
  def main(args: Array[String]): Unit = {
    val redactBook: Redact[Book] = Book.schema.derive(RedactDeriver)
    List(zionomicon, programmingInScala, fpInScala).foreach(book => println(redactBook.redact(book)))

    val redactOrder: Redact[Order] = Order.schema.derive(RedactDeriver)
    println(redactOrder.redact(order))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain deriveyourowntypeclass.Step1DeriveExample"
```

### Masking Fields Through Modifiers

This example masks the customer's name with `withModifier`, next to the annotated payment token.

<details>
  <summary>schema-examples/src/main/scala/deriveyourowntypeclass/Step2ModifiersExample.scala</summary>

```scala title="schema-examples/src/main/scala/deriveyourowntypeclass/Step2ModifiersExample.scala" showLineNumbers
package deriveyourowntypeclass

/**
 * How to Derive Your Own Type Class from a Schema — Masking Fields Through
 * Modifiers
 *
 * Masks the customer's name with Deriver#withModifier, next to the annotated
 * payment token.
 *
 * Run with: cd schema-examples && sbt "runMain
 * deriveyourowntypeclass.Step2ModifiersExample"
 */
object Step2ModifiersExample {
  def main(args: Array[String]): Unit = {
    println(Order.schema.derive(RedactDeriver).redact(order))

    val strictDeriver: Deriver[Redact] =
      RedactDeriver.withModifier(TypeId.of[Customer], "name", Modifier.config("log.redact", "true"))

    println(Order.schema.derive(strictDeriver).redact(order))
  }
}
```

</details>

```bash
sbt "runMain deriveyourowntypeclass.Step2ModifiersExample"
```

### Overriding the Instance for One Type

This example masks every `Email` partially, first on the deriver with `withInstance`, then for one derivation with `DerivationBuilder#instance`.

<details>
  <summary>schema-examples/src/main/scala/deriveyourowntypeclass/Step3OverrideExample.scala</summary>

```scala title="schema-examples/src/main/scala/deriveyourowntypeclass/Step3OverrideExample.scala" showLineNumbers
package deriveyourowntypeclass

/**
 * How to Derive Your Own Type Class from a Schema — Overriding the Instance for
 * One Type
 *
 * Masks every Email partially, first on the deriver with withInstance, then for
 * one derivation with DerivationBuilder#instance.
 *
 * Run with: cd schema-examples && sbt "runMain
 * deriveyourowntypeclass.Step3OverrideExample"
 */
object Step3OverrideExample {
  def main(args: Array[String]): Unit = {
    val partialEmail: Redact[Email] = (email: Email) => {
      val at = email.value.indexOf('@')
      email.value.take(1) + Redact.Masked + email.value.drop(at)
    }

    val logDeriver: Deriver[Redact] = RedactDeriver.withInstance[Email](partialEmail)

    println(Order.schema.derive(logDeriver).redact(order))

    println(Order.schema.deriving(RedactDeriver).instance(TypeId.of[Email], partialEmail).derive.redact(order))
  }
}
```

</details>

```bash
sbt "runMain deriveyourowntypeclass.Step3OverrideExample"
```

### Packaging the Type Class for Reuse

This example derives `Redact` for a recursive `Category` with a `derives Schema, Redact` clause.

<details>
  <summary>schema-examples/src/main/scala/deriveyourowntypeclass/Step4PackagingExample.scala</summary>

```scala title="schema-examples/src/main/scala/deriveyourowntypeclass/Step4PackagingExample.scala" showLineNumbers
package deriveyourowntypeclass

/**
 * How to Derive Your Own Type Class from a Schema — Packaging the Type Class
 * for Reuse
 *
 * Derives Redact for a recursive Category with a derives Schema, Redact clause.
 *
 * Run with: cd schema-examples && sbt "runMain
 * deriveyourowntypeclass.Step4PackagingExample"
 */
final case class Category(name: String, subcategories: List[Category]) derives Schema, Redact

object Step4PackagingExample {
  def main(args: Array[String]): Unit = {
    val programming = Category("Programming", List(Category("Scala", List(Category("Functional Programming", Nil)))))

    println(summon[Redact[Category]].redact(programming))
  }
}
```

</details>

```bash
sbt "runMain deriveyourowntypeclass.Step4PackagingExample"
```

### The Complete Example

```bash
sbt "runMain deriveyourowntypeclass.CompleteExample"
```

## Going Further

- [Type-Class Derivation](../reference/schema/type-class-derivation.md) builds `Show` and `Gen` instances and explains every `Deriver` method, including the record-register and recursion patterns used here.
- [Customizing Derivation with Instance and Modifier Overrides](../reference/schema/type-class-derivation.md#customizing-derivation-with-instance-and-modifier-overrides) covers every `withInstance` and `withModifier` overload and how overrides chain.
- [Modifier](../reference/schema/modifier.md) lists the built-in modifiers and how to read them from a schema.
- [Lazy](../reference/schema/lazy.md) documents the `Lazy` operations a deriver uses to compose child instances.
- [How to Write a Codec for a New Wire Format](./write-a-codec-for-a-new-wire-format.md) applies the same `Deriver` to a binary codec: it builds records back from their fields, decodes collections, and reports decode errors with a path.

---

## How to Compute, Store and Replay Changes with Diff and Patch

In this guide, we track the changes made to a book catalog: who lowered the price of *Zionomicon*, from what to what, and what the catalog looked like before each edit. Instead of storing a full copy of every version, we compute a *patch*, a small value that describes only what changed, with `Schema#diff`. By the end, you will have an edit log that stores each change as JSON, replays it from a base version, undoes the last edit, and sends changes to other systems as JSON patches.

We take an incremental approach:

- Diff two versions of a `Book` and read the patch
- Turn a patch into audit-log lines
- Apply a patch, and compose two patches into one
- Store a patch as JSON text and load it back
- Choose a `PatchMode` for when the data changed after the patch was made
- Keep an edit history, replay it and undo the last edit
- Exchange JSON patches with other systems, including the standard RFC 6902 format

## The Problem

Many programs need to know *what changed* between two versions of their data, not just the new version. An audit log answers "who changed this price, and from what?". A client that syncs with a server wants to send only the fields that changed. An editor needs undo, and a collaboration feature stores a stream of small edits and replays them.

Without help, we compare the two values by hand, one field at a time. Every new field needs another line, lists and maps need their own comparison code, and nothing turns the result back into an update we can apply later.

For example, an audit function for a `Book` often looks like this:

```scala
final case class Book(id: String, title: String, price: BigDecimal, tags: List[String], edition: Int)

def changes(before: Book, after: Book): List[String] = {
  val title   = if (before.title != after.title) List(s"title: ${before.title} -> ${after.title}") else Nil
  val price   = if (before.price != after.price) List(s"price: ${before.price} -> ${after.price}") else Nil
  val added   = after.tags.filterNot(before.tags.contains).map(t => s"tag added: $t")
  val removed = before.tags.filterNot(after.tags.contains).map(t => s"tag removed: $t")
  val edition = if (before.edition != after.edition) List(s"edition: ${before.edition} -> ${after.edition}") else Nil
  title ++ price ++ added ++ removed ++ edition
}
```

The function forgets new fields silently, describes changes only as text, and gives us nothing to *apply*: to replay or undo the edit, we would write a second function that reads these strings back.

In this guide, we let the schema do this work. `Schema#diff` compares any two values of a type that has a `Schema` and returns a `Patch`, which we can print, store, send and apply.

## Prerequisites

This guide builds on [Your First Schema](./your-first-schema.md): you should know how to derive a `Schema` for a case class. Add the schema module to your `build.sbt` (see [How to Add ZIO Blocks to Your Build](../getting-started/installation.md) for the full setup):

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56"
scalacOptions += "-experimental"
```

Everything in this guide works the same on Scala 2.13 and Scala 3. We use these imports throughout:

```scala

```

## The Book Model

Our catalog holds books. Each `Book` has an id, a title, a price, a list of tags and an edition number, which covers the field kinds a diff treats differently: strings, numbers and lists. The `Schema` is all `Schema#diff` needs:

```scala
final case class Book(id: String, title: String, price: BigDecimal, tags: List[String], edition: Int)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

val zionomicon = Book("zionomicon", "Zionomicon", BigDecimal("44.99"), List("zio", "scala"), 1)
```

## Diff Two Versions of a Book

Suppose an editor publishes the second edition of *Zionomicon*: the price drops by five dollars, the book gets an `effects` tag and the edition goes up. We pass the old and the new version to `Schema#diff`:

```scala
val secondEdition = zionomicon.copy(price = BigDecimal("39.99"), tags = List("zio", "scala", "effects"), edition = 2)

val pricePatch: Patch[Book] = Book.schema.diff(zionomicon, secondEdition)
```

A `Patch` prints as a readable list of operations, one per changed field:

```scala
println(pricePatch)
// DynamicPatch {
//   .price -= 5.00
//   .tags:
//     + "effects"
//   .edition += 1
// }
```

The patch lists only the three fields that changed. It does not store the new values: numbers change by a *delta* (`-= 5.00`, `+= 1`), and the list gets an append operation (`+ "effects"`). Unchanged fields such as `title` do not appear at all.

A text field changes by an edit at a position, not by replacing the whole string. Adding a subtitle inserts text at index 10:

```scala
println(Book.schema.diff(secondEdition, secondEdition.copy(title = "Zionomicon (2nd Edition)")))
// DynamicPatch {
//   .title:
//     + [10: " (2nd Edition)"]
// }
```

When nothing changed, the diff is empty, which `Patch#isEmpty` reports:

```scala
Book.schema.diff(zionomicon, zionomicon).isEmpty
// res3: Boolean = true
```

## Turn a Patch into Audit Lines

An audit log wants lines like "price: 44.99 -> 39.99", with the old and the new value. The patch knows *where* each change happened: `Patch#dynamicPatch` exposes the untyped `DynamicPatch`, whose operations (`DynamicPatch#ops`) each carry a `path`, a `DynamicOptic` that prints as `.price` for the price field, and an `operation`. We look up each path in the old and the new value, converted to a `DynamicValue` with `Schema#toDynamicValue`, and print both sides as JSON:

```scala
def auditLines[A](schema: Schema[A], before: A, after: A, patch: Patch[A]): List[String] = {
  val old = schema.toDynamicValue(before)
  val now = schema.toDynamicValue(after)
  def show(value: DynamicValue, path: DynamicOptic): String =
    value.get(path).one.fold(_ => "(none)", v => Json.fromDynamicValue(v).print)
  patch.dynamicPatch.ops.toList.map(op => s"${op.path}: ${show(old, op.path)} -> ${show(now, op.path)}")
}
```

The function works for any type with a `Schema`, so new fields show up in the audit without changes to it:

```scala
auditLines(Book.schema, zionomicon, secondEdition, pricePatch).foreach(println)
// .price: 44.99 -> 39.99
// .tags: ["zio","scala"] -> ["zio","scala","effects"]
// .edition: 1 -> 2
```

## Apply a Patch

`Schema#patch` applies a patch to a value and returns `Either[SchemaError, A]`. Applying the patch to the version it was computed from gives back the new version, which is the basic law of diff and patch:

```scala
Book.schema.patch(zionomicon, pricePatch)
// res5: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon",
//     price = 39.99,
//     tags = List("zio", "scala", "effects"),
//     edition = 2
//   )
// )

Book.schema.patch(zionomicon, pricePatch) == Right(secondEdition)
// res6: Boolean = true
```

`Schema#patch` always uses `PatchMode.Strict`. To choose the mode, call `Patch#apply(value, mode)` instead; we look at the modes in [Choose a PatchMode When the Data Changed](#choose-a-patchmode-when-the-data-changed).

:::warning
`Patch#apply(value)`, with no mode, returns a plain `Book`, not an `Either`. It applies the patch in `PatchMode.Lenient`, which skips the operations that fail, and returns the original value if the result cannot be built. A failed patch then looks the same as a patch that did nothing. Prefer `Patch#apply(value, mode)` or `Schema#patch`, which report the error.
:::

## Compose Patches into One

Two patches compose with `Patch#++`: the result applies the left patch first, then the right one. Here, a second edit adds the subtitle:

```scala
val withSubtitle  = secondEdition.copy(title = "Zionomicon (2nd Edition)")
val subtitlePatch = Book.schema.diff(secondEdition, withSubtitle)
```

Applying the composed patch to the first edition gives the book after both edits:

```scala
(pricePatch ++ subtitlePatch)(zionomicon, PatchMode.Strict)
// res7: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon (2nd Edition)",
//     price = 39.99,
//     tags = List("zio", "scala", "effects"),
//     edition = 2
//   )
// )
```

## Store a Patch as JSON and Load It Back

To keep a patch in a database, write it to a file or send it over the network, we store its `DynamicPatch`. `DynamicPatch` has its own `Schema`, so it has a JSON codec like any other type:

```scala
val patchCodec = Schema[DynamicPatch].jsonCodec

val storedText: String = patchCodec.encodeToString(pricePatch.dynamicPatch)
```

The JSON names each operation and its path:

```scala
println(storedText)
// {"ops":[{"path":{"nodes":[{"Field":{"name":"price"}}]},"operation":{"PrimitiveDelta":{"op":{"BigDecimalDelta":{"delta":-5.00}}}}},{"path":{"nodes":[{"Field":{"name":"tags"}}]},"operation":{"SequenceEdit":{"ops":[{"Append":{"values":["effects"]}}]}}},{"path":{"nodes":[{"Field":{"name":"edition"}}]},"operation":{"PrimitiveDelta":{"op":{"IntDelta":{"delta":1}}}}}]}
```

Decoding the text gives back an equal `DynamicPatch`:

```scala
patchCodec.decode(storedText).map(_ == pricePatch.dynamicPatch)
// res9: Either[SchemaError, Boolean] = Right(true)
```

A decoded `DynamicPatch` is not a `Patch[Book]`, and `Patch` has no public way to wrap one again. Instead, we apply it to the book's `DynamicValue` and convert the result back with `Schema#fromDynamicValue`. A small helper does both steps:

```scala
def applyStored[A](schema: Schema[A], value: A, patch: DynamicPatch, mode: PatchMode = PatchMode.Strict): Either[SchemaError, A] =
  patch(schema.toDynamicValue(value), mode).flatMap(schema.fromDynamicValue)
```

Loading the stored text and applying it to the first edition gives the second edition:

```scala
patchCodec.decode(storedText).flatMap(loaded => applyStored(Book.schema, zionomicon, loaded))
// res10: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon",
//     price = 39.99,
//     tags = List("zio", "scala", "effects"),
//     edition = 2
//   )
// )
```

The JSON refers to fields by name, so a stored patch keeps working when we add fields to `Book`, as long as the fields it names still exist.

## Choose a PatchMode When the Data Changed

A stored or sent patch is often applied to a value that changed after the patch was made: two editors work on the same catalog, or a client syncs a change made while offline. `PatchMode` decides what happens when an operation no longer fits:

| Mode      | When an operation does not fit                                                                                     |
| --------- | ------------------------------------------------------------------------------------------------------------------ |
| `Strict`  | Stops and returns `Left` with the first error                                                                      |
| `Lenient` | Skips that operation and applies the rest                                                                          |
| `Clobber` | Like `Lenient`, but overwrites a map key that already exists and clamps list inserts and deletes to the list's end |

To see the modes at work, we keep the books in a `Catalog`, a map from book id to `Book`:

```scala
final case class Catalog(books: Map[String, Book])

object Catalog {
  implicit val schema: Schema[Catalog] = Schema.derived
}

val programmingInScala = Book("programming-in-scala", "Programming in Scala", BigDecimal("59.99"), List("scala"), 5)
val fpInScala          = Book("fp-in-scala", "Functional Programming in Scala", BigDecimal("49.99"), List("scala", "fp"), 2)

val monday  = Catalog(Map(zionomicon.id -> zionomicon, programmingInScala.id -> programmingInScala))
val tuesday = Catalog(Map(zionomicon.id -> secondEdition, fpInScala.id -> fpInScala))

val catalogPatch = Catalog.schema.diff(monday, tuesday)
```

The patch modifies the *Zionomicon* entry, adds *Functional Programming in Scala* and removes *Programming in Scala*. The added book's fields print without indentation, a known bug in 0.0.56:

```scala
println(catalogPatch)
// DynamicPatch {
//   .books:
//     ~ {"zionomicon"}:
//       .price -= 5.00
//       .tags:
//         + "effects"
//       .edition += 1
//     + {"fp-in-scala": {
//   id: "fp-in-scala",
//   title: "Functional Programming in Scala",
//   price: 49.99,
//   tags: [
//     "scala",
//     "fp"
//   ],
//   edition: 2
// }}
//     - {"programming-in-scala"}
// }
```

A helper applies the patch in all three modes, so we can compare them:

```scala
def applyInEveryMode(target: Catalog): Unit =
  List(PatchMode.Strict, PatchMode.Lenient, PatchMode.Clobber).foreach { mode =>
    println(s"$mode: ${catalogPatch(target, mode).map(_.books.values.map(b => s"${b.id} ${b.price}").toList.sorted)}")
  }
```

Now suppose another editor added *Functional Programming in Scala* first, at a price of 45.00. `Strict` refuses the patch, `Lenient` keeps the other editor's book, and `Clobber` overwrites it with ours:

```scala
applyInEveryMode(monday.copy(books = monday.books + (fpInScala.id -> fpInScala.copy(price = BigDecimal("45.00")))))
// Strict: Left(zio.blocks.schema.SchemaError: Key already exists in map at: .books)
// Lenient: Right(List(fp-in-scala 45.00, zionomicon 39.99))
// Clobber: Right(List(fp-in-scala 49.99, zionomicon 39.99))
```

If the other editor removed *Programming in Scala* already, `Strict` fails because the key is gone, and the other two modes skip the removal:

```scala
applyInEveryMode(monday.copy(books = monday.books - programmingInScala.id))
// Strict: Left(zio.blocks.schema.SchemaError: Key not found in map at: .books)
// Lenient: Right(List(fp-in-scala 49.99, zionomicon 39.99))
// Clobber: Right(List(fp-in-scala 49.99, zionomicon 39.99))
```

Use `Strict` when a conflict must reach a person or a retry loop, `Lenient` when the latest writer should not undo other people's additions, and `Clobber` when our patch should win.

## Strict Mode Does Not Detect Every Change

`Strict` only checks that each operation *fits* the value: a map key exists, an index is inside the list, an insert position is inside the string. It does not check that the value still holds what it held when the patch was made. Numeric deltas, string edits and list operations by index all apply to whatever value is there now.

For example, the *Zionomicon* price patch says "minus 5.00", not "set to 39.99". If someone set the price to 42.00 in the meantime, the patch still applies, and the result is 37.00:

```scala
pricePatch(zionomicon.copy(price = BigDecimal("42.00")), PatchMode.Strict)
// res14: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon",
//     price = 37.00,
//     tags = List("zio", "scala", "effects"),
//     edition = 2
//   )
// )
```

Lists are riskier, because a diff removes list elements by index. This patch removes the `scala` tag, which sits at index 1:

```scala
val tagged         = zionomicon.copy(tags = List("zio", "scala", "effects"))
val removeScalaTag = Book.schema.diff(tagged, tagged.copy(tags = List("zio", "effects")))
```

The patch stores only the position:

```scala
println(removeScalaTag)
// DynamicPatch {
//   .tags:
//     - [1]
// }
```

:::warning
If someone already removed the `scala` tag, index 1 now holds `effects`, and `Strict` removes it without an error. The book loses a tag that nobody asked to remove:

```scala
removeScalaTag(tagged.copy(tags = List("zio", "effects")), PatchMode.Strict)
// res16: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon",
//     price = 44.99,
//     tags = List("zio"),
//     edition = 1
//   )
// )
```
:::

When the result must be exact, check that the value is still the one the patch was computed from before applying it. If it changed, reject the edit or compute a fresh diff against the current value. Comparing the whole value works for any type; a version number stored next to the value does the same job with less data to send:

```scala
def applyIfUnchanged(current: Book, base: Book, patch: Patch[Book]): Either[String, Book] =
  if (current != base) Left(s"${current.id} changed since this patch was made; diff again against the current version")
  else patch(current, PatchMode.Strict).left.map(_.message)
```

The same patch is now rejected for the changed book, and still applies to the original:

```scala
applyIfUnchanged(tagged.copy(tags = List("zio", "effects")), tagged, removeScalaTag)
// res17: Either[String, Book] = Left(
//   "zionomicon changed since this patch was made; diff again against the current version"
// )

applyIfUnchanged(tagged, tagged, removeScalaTag)
// res18: Either[String, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon",
//     price = 44.99,
//     tags = List("zio", "effects"),
//     edition = 1
//   )
// )
```

## Keep an Edit History, Replay It and Undo

An edit history is a list of patches plus whatever we want to know about each one. We record two patches per edit: the *forward* patch from the old to the new version, and the *backward* patch from the new to the old version, which undoes the edit. Both are `DynamicPatch` values, so the case class gets a `Schema` and a JSON codec like any other:

```scala
final case class Edit(author: String, forward: DynamicPatch, backward: DynamicPatch)

object Edit {
  implicit val schema: Schema[Edit] = Schema.derived
}

def record(author: String, before: Book, after: Book): Edit =
  Edit(author, Book.schema.diff(before, after).dynamicPatch, Book.schema.diff(after, before).dynamicPatch)

val history: List[Edit] = List(
  record("alice", zionomicon, secondEdition),
  record("bob", secondEdition, withSubtitle)
)
```

The whole history is a `List[Edit]`, so one codec stores it as JSON text, in the same shape as the stored patch above:

```scala
val historyCodec = Schema[List[Edit]].jsonCodec

val historyText: String = historyCodec.encodeToString(history)
```

Decoding the text gives back the same history:

```scala
historyCodec.decode(historyText) == Right(history)
// res19: Boolean = true
```

Replaying the history is a fold over the edits, starting from the base version. Each step applies one forward patch with the `applyStored` helper, and the first failure stops the replay:

```scala
def replay(base: Book, edits: List[Edit]): Either[SchemaError, Book] =
  edits.foldLeft[Either[SchemaError, Book]](Right(base))((book, edit) => book.flatMap(applyStored(Book.schema, _, edit.forward)))
```

Replaying every edit gives the latest version, and replaying a prefix gives any earlier version:

```scala
replay(zionomicon, history)
// res20: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon (2nd Edition)",
//     price = 39.99,
//     tags = List("zio", "scala", "effects"),
//     edition = 2
//   )
// )

replay(zionomicon, history.take(1))
// res21: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon",
//     price = 39.99,
//     tags = List("zio", "scala", "effects"),
//     edition = 2
//   )
// )
```

To undo the last edit, we apply its backward patch to the current version:

```scala
applyStored(Book.schema, withSubtitle, history.last.backward)
// res22: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon",
//     price = 39.99,
//     tags = List("zio", "scala", "effects"),
//     edition = 2
//   )
// )
```

A history only replays from the base it was recorded against. Started from a different book, Bob's insert at index 10 no longer fits a three-letter title, and the replay stops with an error:

```scala
replay(zionomicon.copy(title = "ZIO"), history)
// res23: Either[SchemaError, Book] = Left(
//   SchemaError(
//     List(
//       ExpectationMismatch(
//         source = DynamicOptic(ArraySeq(Field("title"))),
//         expectation = "String insert index 10 out of bounds for string of length 3"
//       )
//     )
//   )
// )
```

To store one patch instead of many, for example to squash old history, `DynamicPatch#++` composes the forward patches in order:

```scala
println(history.map(_.forward).foldLeft(DynamicPatch.empty)(_ ++ _))
// DynamicPatch {
//   .price -= 5.00
//   .tags:
//     + "effects"
//   .edition += 1
//   .title:
//     + [10: " (2nd Edition)"]
// }
```

## Exchange JSON Patches with Other Systems

When the data is JSON rather than a Scala type, for example a document from another service, `Json#diff` compares two `Json` values and returns a `JsonPatch`, and `Json#patch` applies one. `Json.from` turns a `Book` into `Json` through its schema:

```scala
val mondayJson  = Json.from(zionomicon)
val tuesdayJson = Json.from(withSubtitle)

val jsonPatch: JsonPatch = mondayJson.diff(tuesdayJson)
```

The `JsonPatch` holds the same kinds of operations as a typed patch, nested by object key:

```scala
println(jsonPatch)
// JsonPatch {
//   root:
//     ~ {"title"}:
//       root:
//         + [10: " (2nd Edition)"]
//     ~ {"price"}:
//       root -= 5.00
//     ~ {"tags"}:
//       root:
//         + "effects"
//     ~ {"edition"}:
//       root += 1
// }

mondayJson.patch(jsonPatch).map(_.print)
// res26: Either[SchemaError, String] = Right(
//   "{\"id\":\"zionomicon\",\"title\":\"Zionomicon (2nd Edition)\",\"price\":39.99,\"tags\":[\"zio\",\"scala\",\"effects\"],\"edition\":2}"
// )
```

A typed patch's `DynamicPatch` converts to a `JsonPatch` with `JsonPatch.fromDynamicPatch`, which returns `Either[SchemaError, JsonPatch]` because temporal deltas and non-string map keys have no JSON form. `JsonPatch` has a `Schema`, so another ZIO Blocks service can receive it with `JsonPatch.schema.jsonCodec`.

The ZIO Blocks `JsonPatch` is its own format, not the [RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902) "JSON Patch" standard that HTTP `PATCH` APIs and many JSON libraries understand. RFC 6902 has no number deltas or string edits: it sends operations such as `add`, `remove` and `replace` (it also defines `move`, `copy` and `test`) with [JSON Pointer](https://datatracker.ietf.org/doc/html/rfc6901) paths such as `/tags/1`. The [JsonPatch reference](../reference/schema/built-in-codecs/json/json-patch.md) lists every operation and compares the two formats.

## Convert a JsonPatch to RFC 6902

To send a change to a system that expects RFC 6902, we convert the `JsonPatch` ourselves. The converter below is our own code, not a library API. It walks the operations in order and keeps the current JSON value next to them, applying each operation with `JsonPatch.root`, so that every index and every computed value refers to the document as it is at that point:

```scala
object Rfc6902 {
  def fromJsonPatch(source: Json, patch: JsonPatch): Either[SchemaError, Json] =
    patchOps(source, "", patch).map(ops => Json.Array(Chunk.from(ops)))

  private def patchOps(value: Json, path: String, patch: JsonPatch): Either[SchemaError, Vector[Json]] =
    steps(value, patch.ops.toVector)(op => JsonPatch(Chunk.single(op))) { (current, op) =>
      if (op.path.nodes.nonEmpty) Left(SchemaError(s"only root paths are supported, got ${op.path}"))
      else opOps(current, path, op.operation)
    }

  private def opOps(value: Json, path: String, op: JsonPatch.Op): Either[SchemaError, Vector[Json]] = op match {
    case JsonPatch.Op.Set(v)            => Right(Vector(rfc("replace", path, Some(v))))
    case JsonPatch.Op.PrimitiveDelta(_) => JsonPatch.root(op)(value).map(v => Vector(rfc("replace", path, Some(v))))
    case JsonPatch.Op.Nested(p)         => patchOps(value, path, p)
    case JsonPatch.Op.ArrayEdit(edits)  =>
      steps(value, edits.toVector)(e => JsonPatch.root(JsonPatch.Op.ArrayEdit(Chunk.single(e)))) {
        case (_, JsonPatch.ArrayOp.Insert(i, vs)) =>
          Right(vs.toVector.zipWithIndex.map { case (v, k) => rfc("add", s"$path/${i + k}", Some(v)) })
        case (_, JsonPatch.ArrayOp.Append(vs))     => Right(vs.toVector.map(v => rfc("add", s"$path/-", Some(v))))
        case (_, JsonPatch.ArrayOp.Delete(i, n))   => Right(Vector.fill(n)(rfc("remove", s"$path/$i", None)))
        case (cur, JsonPatch.ArrayOp.Modify(i, o)) =>
          cur.elements.lift(i).toRight(SchemaError(s"no index $i at $path")).flatMap(opOps(_, s"$path/$i", o))
      }
    case JsonPatch.Op.ObjectEdit(edits) =>
      steps(value, edits.toVector)(e => JsonPatch.root(JsonPatch.Op.ObjectEdit(Chunk.single(e)))) {
        case (_, JsonPatch.ObjectOp.Add(k, v))      => Right(Vector(rfc("add", s"$path/${escape(k)}", Some(v))))
        case (_, JsonPatch.ObjectOp.Remove(k))      => Right(Vector(rfc("remove", s"$path/${escape(k)}", None)))
        case (cur, JsonPatch.ObjectOp.Modify(k, p)) =>
          cur.fields.collectFirst { case (`k`, v) => v }.toRight(SchemaError(s"no key $k at $path"))
            .flatMap(patchOps(_, s"$path/${escape(k)}", p))
      }
  }

  // Emits the RFC 6902 operations for each edit, then applies the edit, so later edits see the updated value.
  private def steps[E](value: Json, edits: Vector[E])(asPatch: E => JsonPatch)(
    emit: (Json, E) => Either[SchemaError, Vector[Json]]
  ): Either[SchemaError, Vector[Json]] =
    edits
      .foldLeft[Either[SchemaError, (Json, Vector[Json])]](Right((value, Vector.empty))) { (acc, e) =>
        acc.flatMap { case (current, out) =>
          for {
            ops  <- emit(current, e)
            next <- asPatch(e)(current)
          } yield (next, out ++ ops)
        }
      }
      .map(_._2)

  private def escape(key: String): String = key.replace("~", "~0").replace("/", "~1")

  private def rfc(op: String, path: String, value: Option[Json]): Json =
    Json.Object(Chunk.from(Vector("op" -> Json.String(op), "path" -> Json.String(path)) ++ value.map("value" -> _)))
}
```

Converting the *Zionomicon* patch gives standard RFC 6902 operations:

```scala
Rfc6902.fromJsonPatch(mondayJson, jsonPatch).map(_.print).foreach(println)
// [{"op":"replace","path":"/title","value":"Zionomicon (2nd Edition)"},{"op":"replace","path":"/price","value":39.99},{"op":"add","path":"/tags/-","value":"effects"},{"op":"replace","path":"/edition","value":2}]
```

List edits become `add` and `remove` operations by index. Here, we remove two tags and append one:

```scala
val retagged = Json.from(zionomicon.copy(tags = List("zio", "scala", "effects", "fibers")))
val trimmed  = Json.from(zionomicon.copy(tags = List("zio", "fibers", "streams")))
```

The result has two `remove` operations at index 1, because the first removal shifts the next tag into its place:

```scala
Rfc6902.fromJsonPatch(retagged, retagged.diff(trimmed)).map(_.print).foreach(println)
// [{"op":"remove","path":"/tags/1"},{"op":"remove","path":"/tags/1"},{"op":"add","path":"/tags/-","value":"streams"}]
```

The conversion loses some detail, and it needs the source document. A number delta becomes a `replace` with the new number, which the converter computes from the source, and a string edit becomes a `replace` of the whole string. The result says "set the price to 39.99", not "lower it by 5.00". The converter accepts the patches that `Json#diff` and `JsonPatch.diff` produce, whose operations all sit at the root path, and returns `Left` for a hand-built patch with a deeper path. Going the other way, reading an RFC 6902 patch that another system sends us, is out of scope for this guide.

## Putting It Together

The complete example defines the catalog, records two edits, stores the history as JSON, replays and undoes it, shows `PatchMode` on a changed catalog, and prints the RFC 6902 form of a change:

```scala title="schema-examples/src/main/scala/diffandpatch/CompleteExample.scala"
package diffandpatch

/**
 * Diff and Patch: Putting It Together, Complete Example
 *
 * Records two edits to Zionomicon in an edit log, prints the audit lines,
 * stores the log as JSON and loads it back, replays and undoes it, applies a
 * catalog patch to a catalog another editor changed, rejects a stale patch, and
 * prints the RFC 6902 form of a change.
 *
 * Run with: cd schema-examples && sbt "runMain diffandpatch.CompleteExample"
 */
object CompleteExample extends App {

  final case class Book(id: String, title: String, price: BigDecimal, tags: List[String], edition: Int)

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  final case class Catalog(books: Map[String, Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  final case class Edit(author: String, forward: DynamicPatch, backward: DynamicPatch)

  object Edit {
    implicit val schema: Schema[Edit] = Schema.derived
  }

  def auditLines[A](schema: Schema[A], before: A, after: A, patch: Patch[A]): List[String] = {
    val old                                                   = schema.toDynamicValue(before)
    val now                                                   = schema.toDynamicValue(after)
    def show(value: DynamicValue, path: DynamicOptic): String =
      value.get(path).one.fold(_ => "(none)", v => Json.fromDynamicValue(v).print)
    patch.dynamicPatch.ops.toList.map(op => s"${op.path}: ${show(old, op.path)} -> ${show(now, op.path)}")
  }

  def applyStored[A](schema: Schema[A], value: A, patch: DynamicPatch): Either[SchemaError, A] =
    patch(schema.toDynamicValue(value), PatchMode.Strict).flatMap(schema.fromDynamicValue)

  def record(author: String, before: Book, after: Book): Edit =
    Edit(author, Book.schema.diff(before, after).dynamicPatch, Book.schema.diff(after, before).dynamicPatch)

  def replay(base: Book, edits: List[Edit]): Either[SchemaError, Book] =
    edits.foldLeft[Either[SchemaError, Book]](Right(base))((book, edit) =>
      book.flatMap(applyStored(Book.schema, _, edit.forward))
    )

  def applyIfUnchanged(current: Book, base: Book, patch: Patch[Book]): Either[String, Book] =
    if (current != base)
      Left(s"${current.id} changed since this patch was made; diff again against the current version")
    else patch(current, PatchMode.Strict).left.map(_.message)

  object Rfc6902 {
    def fromJsonPatch(source: Json, patch: JsonPatch): Either[SchemaError, Json] =
      patchOps(source, "", patch).map(ops => Json.Array(Chunk.from(ops)))

    private def patchOps(value: Json, path: String, patch: JsonPatch): Either[SchemaError, Vector[Json]] =
      steps(value, patch.ops.toVector)(op => JsonPatch(Chunk.single(op))) { (current, op) =>
        if (op.path.nodes.nonEmpty) Left(SchemaError(s"only root paths are supported, got ${op.path}"))
        else opOps(current, path, op.operation)
      }

    private def opOps(value: Json, path: String, op: JsonPatch.Op): Either[SchemaError, Vector[Json]] = op match {
      case JsonPatch.Op.Set(v)            => Right(Vector(rfc("replace", path, Some(v))))
      case JsonPatch.Op.PrimitiveDelta(_) => JsonPatch.root(op)(value).map(v => Vector(rfc("replace", path, Some(v))))
      case JsonPatch.Op.Nested(p)         => patchOps(value, path, p)
      case JsonPatch.Op.ArrayEdit(edits)  =>
        steps(value, edits.toVector)(e => JsonPatch.root(JsonPatch.Op.ArrayEdit(Chunk.single(e)))) {
          case (_, JsonPatch.ArrayOp.Insert(i, vs)) =>
            Right(vs.toVector.zipWithIndex.map { case (v, k) => rfc("add", s"$path/${i + k}", Some(v)) })
          case (_, JsonPatch.ArrayOp.Append(vs))     => Right(vs.toVector.map(v => rfc("add", s"$path/-", Some(v))))
          case (_, JsonPatch.ArrayOp.Delete(i, n))   => Right(Vector.fill(n)(rfc("remove", s"$path/$i", None)))
          case (cur, JsonPatch.ArrayOp.Modify(i, o)) =>
            cur.elements.lift(i).toRight(SchemaError(s"no index $i at $path")).flatMap(opOps(_, s"$path/$i", o))
        }
      case JsonPatch.Op.ObjectEdit(edits) =>
        steps(value, edits.toVector)(e => JsonPatch.root(JsonPatch.Op.ObjectEdit(Chunk.single(e)))) {
          case (_, JsonPatch.ObjectOp.Add(k, v))      => Right(Vector(rfc("add", s"$path/${escape(k)}", Some(v))))
          case (_, JsonPatch.ObjectOp.Remove(k))      => Right(Vector(rfc("remove", s"$path/${escape(k)}", None)))
          case (cur, JsonPatch.ObjectOp.Modify(k, p)) =>
            cur.fields.collectFirst { case (`k`, v) => v }
              .toRight(SchemaError(s"no key $k at $path"))
              .flatMap(patchOps(_, s"$path/${escape(k)}", p))
        }
    }

    // Emits the RFC 6902 operations for each edit, then applies the edit, so later edits see the updated value.
    private def steps[E](value: Json, edits: Vector[E])(asPatch: E => JsonPatch)(
      emit: (Json, E) => Either[SchemaError, Vector[Json]]
    ): Either[SchemaError, Vector[Json]] =
      edits
        .foldLeft[Either[SchemaError, (Json, Vector[Json])]](Right((value, Vector.empty))) { (acc, e) =>
          acc.flatMap { case (current, out) =>
            for {
              ops  <- emit(current, e)
              next <- asPatch(e)(current)
            } yield (next, out ++ ops)
          }
        }
        .map(_._2)

    private def escape(key: String): String = key.replace("~", "~0").replace("/", "~1")

    private def rfc(op: String, path: String, value: Option[Json]): Json =
      Json.Object(Chunk.from(Vector("op" -> Json.String(op), "path" -> Json.String(path)) ++ value.map("value" -> _)))
  }

  val zionomicon    = Book("zionomicon", "Zionomicon", BigDecimal("44.99"), List("zio", "scala"), 1)
  val secondEdition = zionomicon.copy(price = BigDecimal("39.99"), tags = List("zio", "scala", "effects"), edition = 2)
  val withSubtitle  = secondEdition.copy(title = "Zionomicon (2nd Edition)")

  // Record two edits and print what alice changed.
  val history = List(record("alice", zionomicon, secondEdition), record("bob", secondEdition, withSubtitle))
  auditLines(Book.schema, zionomicon, secondEdition, Book.schema.diff(zionomicon, secondEdition))
    .foreach(line => println(s"alice changed $line"))

  // Store the log as JSON, load it back, replay it and undo the last edit.
  val historyCodec = Schema[List[Edit]].jsonCodec
  val loaded       = historyCodec.decode(historyCodec.encodeToString(history))
  println(loaded.flatMap(replay(zionomicon, _)))
  println(loaded.flatMap(edits => applyStored(Book.schema, withSubtitle, edits.last.backward)))

  // Apply a catalog patch to a catalog that another editor changed.
  val programmingInScala = Book("programming-in-scala", "Programming in Scala", BigDecimal("59.99"), List("scala"), 5)
  val fpInScala          = Book("fp-in-scala", "Functional Programming in Scala", BigDecimal("49.99"), List("scala", "fp"), 2)
  val monday             = Catalog(Map(zionomicon.id -> zionomicon, programmingInScala.id -> programmingInScala))
  val tuesday            = Catalog(Map(zionomicon.id -> secondEdition, fpInScala.id -> fpInScala))
  val catalogPatch       = Catalog.schema.diff(monday, tuesday)
  val changedMonday      = Catalog(monday.books - programmingInScala.id)
  List(PatchMode.Strict, PatchMode.Lenient).foreach { mode =>
    println(s"$mode: ${catalogPatch(changedMonday, mode).map(_.books.keys.toList.sorted)}")
  }

  // Reject a patch whose base changed, instead of removing the wrong tag.
  val tagged         = zionomicon.copy(tags = List("zio", "scala", "effects"))
  val removeScalaTag = Book.schema.diff(tagged, tagged.copy(tags = List("zio", "effects")))
  println(applyIfUnchanged(tagged.copy(tags = List("zio", "effects")), tagged, removeScalaTag))

  // Send the change to a system that speaks RFC 6902.
  val mondayJson = Json.from(zionomicon)
  println(Rfc6902.fromJsonPatch(mondayJson, mondayJson.diff(Json.from(withSubtitle))).map(_.print))
}
```

## Running the Examples

All examples in this guide have corresponding runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

The step examples share the model in `BookModel.scala`:

<details>
  <summary>schema-examples/src/main/scala/diffandpatch/BookModel.scala</summary>

```scala title="schema-examples/src/main/scala/diffandpatch/BookModel.scala" showLineNumbers
package diffandpatch

/**
 * Diff and Patch: the shared model
 *
 * The book, the catalog and the edit-log entry used by every step example, the
 * sample books, and two helpers from the guide: `auditLines` and `applyStored`.
 */
final case class Book(id: String, title: String, price: BigDecimal, tags: List[String], edition: Int)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

final case class Catalog(books: Map[String, Book])

object Catalog {
  implicit val schema: Schema[Catalog] = Schema.derived
}

final case class Edit(author: String, forward: DynamicPatch, backward: DynamicPatch)

object Edit {
  implicit val schema: Schema[Edit] = Schema.derived
}

object Books {
  val zionomicon: Book = Book("zionomicon", "Zionomicon", BigDecimal("44.99"), List("zio", "scala"), 1)

  val secondEdition: Book =
    zionomicon.copy(price = BigDecimal("39.99"), tags = List("zio", "scala", "effects"), edition = 2)

  val withSubtitle: Book = secondEdition.copy(title = "Zionomicon (2nd Edition)")

  val programmingInScala: Book =
    Book("programming-in-scala", "Programming in Scala", BigDecimal("59.99"), List("scala"), 5)

  val fpInScala: Book =
    Book("fp-in-scala", "Functional Programming in Scala", BigDecimal("49.99"), List("scala", "fp"), 2)
}

object PatchHelpers {

  /**
   * One line per changed path: the path, the old value and the new value as
   * JSON.
   */
  def auditLines[A](schema: Schema[A], before: A, after: A, patch: Patch[A]): List[String] = {
    val old                                                   = schema.toDynamicValue(before)
    val now                                                   = schema.toDynamicValue(after)
    def show(value: DynamicValue, path: DynamicOptic): String =
      value.get(path).one.fold(_ => "(none)", v => Json.fromDynamicValue(v).print)
    patch.dynamicPatch.ops.toList.map(op => s"${op.path}: ${show(old, op.path)} -> ${show(now, op.path)}")
  }

  /** Applies a stored `DynamicPatch` to a typed value. */
  def applyStored[A](
    schema: Schema[A],
    value: A,
    patch: DynamicPatch,
    mode: PatchMode = PatchMode.Strict
  ): Either[SchemaError, A] =
    patch(schema.toDynamicValue(value), mode).flatMap(schema.fromDynamicValue)
}
```

</details>

Run them in order to follow the guide step by step.

### Diff and Audit

This example diffs the first and second edition of *Zionomicon*, prints the patch, and turns it into audit lines.

<details>
  <summary>schema-examples/src/main/scala/diffandpatch/DiffAndAuditExample.scala</summary>

```scala title="schema-examples/src/main/scala/diffandpatch/DiffAndAuditExample.scala" showLineNumbers
package diffandpatch

/**
 * Diff and Patch: Diff Two Versions and Turn the Patch into Audit Lines
 *
 * Diffs the first and second edition of Zionomicon, prints the patch, a string
 * edit and an empty diff, then prints one audit line per changed field.
 *
 * Run with: cd schema-examples && sbt "runMain
 * diffandpatch.DiffAndAuditExample"
 */
object DiffAndAuditExample extends App {

  val pricePatch = Book.schema.diff(zionomicon, secondEdition)
  println(pricePatch)

  println(Book.schema.diff(secondEdition, withSubtitle))

  println(Book.schema.diff(zionomicon, zionomicon).isEmpty)

  PatchHelpers.auditLines(Book.schema, zionomicon, secondEdition, pricePatch).foreach(println)
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain diffandpatch.DiffAndAuditExample"
```

### Apply and Compose Patches

This example applies a patch with `Schema#patch`, checks the round trip, and composes two patches with `Patch#++`.

<details>
  <summary>schema-examples/src/main/scala/diffandpatch/ApplyAndComposeExample.scala</summary>

```scala title="schema-examples/src/main/scala/diffandpatch/ApplyAndComposeExample.scala" showLineNumbers
package diffandpatch

/**
 * Diff and Patch: Apply a Patch and Compose Patches
 *
 * Applies a patch with `Schema#patch`, checks that diff then patch gives back
 * the new version, and composes two patches with `Patch#++`.
 *
 * Run with: cd schema-examples && sbt "runMain
 * diffandpatch.ApplyAndComposeExample"
 */
object ApplyAndComposeExample extends App {

  val pricePatch    = Book.schema.diff(zionomicon, secondEdition)
  val subtitlePatch = Book.schema.diff(secondEdition, withSubtitle)

  println(Book.schema.patch(zionomicon, pricePatch))
  println(Book.schema.patch(zionomicon, pricePatch) == Right(secondEdition))

  println((pricePatch ++ subtitlePatch)(zionomicon, PatchMode.Strict))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain diffandpatch.ApplyAndComposeExample"
```

### Store a Patch as JSON

This example encodes a `DynamicPatch` as JSON text, decodes it, and applies the loaded patch.

<details>
  <summary>schema-examples/src/main/scala/diffandpatch/StorePatchExample.scala</summary>

```scala title="schema-examples/src/main/scala/diffandpatch/StorePatchExample.scala" showLineNumbers
package diffandpatch

/**
 * Diff and Patch: Store a Patch as JSON and Load It Back
 *
 * Encodes the `DynamicPatch` of the Zionomicon price change as JSON text,
 * decodes it, and applies the loaded patch to the first edition.
 *
 * Run with: cd schema-examples && sbt "runMain diffandpatch.StorePatchExample"
 */
object StorePatchExample extends App {

  val pricePatch = Book.schema.diff(zionomicon, secondEdition)
  val patchCodec = Schema[DynamicPatch].jsonCodec

  val storedText: String = patchCodec.encodeToString(pricePatch.dynamicPatch)
  println(storedText)

  println(patchCodec.decode(storedText).map(_ == pricePatch.dynamicPatch))

  println(patchCodec.decode(storedText).flatMap(loaded => PatchHelpers.applyStored(Book.schema, zionomicon, loaded)))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain diffandpatch.StorePatchExample"
```

### Patch Modes

This example applies a catalog patch in every `PatchMode` to two changed catalogs, shows how `Strict` misses a changed price and a changed tag list, and rejects a stale patch with a base check.

<details>
  <summary>schema-examples/src/main/scala/diffandpatch/PatchModeExample.scala</summary>

```scala title="schema-examples/src/main/scala/diffandpatch/PatchModeExample.scala" showLineNumbers
package diffandpatch

/**
 * Diff and Patch: Choose a PatchMode When the Data Changed
 *
 * Applies a catalog patch in Strict, Lenient and Clobber mode to two catalogs
 * that another editor changed, shows that Strict mode misses a changed price
 * and a changed tag list, and rejects a stale patch by checking its base.
 *
 * Run with: cd schema-examples && sbt "runMain diffandpatch.PatchModeExample"
 */
object PatchModeExample extends App {

  val monday  = Catalog(Map(zionomicon.id -> zionomicon, programmingInScala.id -> programmingInScala))
  val tuesday = Catalog(Map(zionomicon.id -> secondEdition, fpInScala.id -> fpInScala))

  val catalogPatch = Catalog.schema.diff(monday, tuesday)
  println(catalogPatch)

  def applyInEveryMode(target: Catalog): Unit =
    List(PatchMode.Strict, PatchMode.Lenient, PatchMode.Clobber).foreach { mode =>
      println(s"$mode: ${catalogPatch(target, mode).map(_.books.values.map(b => s"${b.id} ${b.price}").toList.sorted)}")
    }

  // Another editor added Functional Programming in Scala first, at 45.00.
  applyInEveryMode(monday.copy(books = monday.books + (fpInScala.id -> fpInScala.copy(price = BigDecimal("45.00")))))

  // Another editor removed Programming in Scala already.
  applyInEveryMode(monday.copy(books = monday.books - programmingInScala.id))

  // A numeric delta applies to whatever price is there now.
  val pricePatch = Book.schema.diff(zionomicon, secondEdition)
  println(pricePatch(zionomicon.copy(price = BigDecimal("42.00")), PatchMode.Strict))

  // A list removal by index removes the wrong tag when the list changed.
  val tagged         = zionomicon.copy(tags = List("zio", "scala", "effects"))
  val removeScalaTag = Book.schema.diff(tagged, tagged.copy(tags = List("zio", "effects")))
  println(removeScalaTag)
  println(removeScalaTag(tagged.copy(tags = List("zio", "effects")), PatchMode.Strict))

  def applyIfUnchanged(current: Book, base: Book, patch: Patch[Book]): Either[String, Book] =
    if (current != base)
      Left(s"${current.id} changed since this patch was made; diff again against the current version")
    else patch(current, PatchMode.Strict).left.map(_.message)

  println(applyIfUnchanged(tagged.copy(tags = List("zio", "effects")), tagged, removeScalaTag))
  println(applyIfUnchanged(tagged, tagged, removeScalaTag))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain diffandpatch.PatchModeExample"
```

### Edit History

This example records two edits, stores the history as JSON, replays it, undoes the last edit, and squashes the history into one patch.

<details>
  <summary>schema-examples/src/main/scala/diffandpatch/EditHistoryExample.scala</summary>

```scala title="schema-examples/src/main/scala/diffandpatch/EditHistoryExample.scala" showLineNumbers
package diffandpatch

/**
 * Diff and Patch: Keep an Edit History, Replay It and Undo
 *
 * Records two edits with a forward and a backward patch each, stores the
 * history as JSON, replays it from the first edition, undoes the last edit, and
 * squashes the history into one patch.
 *
 * Run with: cd schema-examples && sbt "runMain diffandpatch.EditHistoryExample"
 */
object EditHistoryExample extends App {

  def record(author: String, before: Book, after: Book): Edit =
    Edit(author, Book.schema.diff(before, after).dynamicPatch, Book.schema.diff(after, before).dynamicPatch)

  val history: List[Edit] = List(
    record("alice", zionomicon, secondEdition),
    record("bob", secondEdition, withSubtitle)
  )

  val historyCodec        = Schema[List[Edit]].jsonCodec
  val historyText: String = historyCodec.encodeToString(history)
  println(historyText)
  println(historyCodec.decode(historyText) == Right(history))

  def replay(base: Book, edits: List[Edit]): Either[SchemaError, Book] =
    edits.foldLeft[Either[SchemaError, Book]](Right(base)) { (book, edit) =>
      book.flatMap(PatchHelpers.applyStored(Book.schema, _, edit.forward))
    }

  println(replay(zionomicon, history))
  println(replay(zionomicon, history.take(1)))

  // Undo the last edit.
  println(PatchHelpers.applyStored(Book.schema, withSubtitle, history.last.backward))

  // A history replays only from the base it was recorded against.
  println(replay(zionomicon.copy(title = "ZIO"), history))

  // Squash the history into one patch.
  println(history.map(_.forward).foldLeft(DynamicPatch.empty)(_ ++ _))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain diffandpatch.EditHistoryExample"
```

### JSON Patches and RFC 6902

This example diffs two `Json` documents, applies the `JsonPatch`, and converts three edits to RFC 6902, checking each result against the expected operations.

<details>
  <summary>schema-examples/src/main/scala/diffandpatch/JsonPatchExample.scala</summary>

```scala title="schema-examples/src/main/scala/diffandpatch/JsonPatchExample.scala" showLineNumbers
package diffandpatch

/**
 * Converts a ZIO Blocks `JsonPatch` produced by `Json#diff` into an RFC 6902
 * JSON Patch (an array of `add`, `remove` and `replace` operations).
 *
 * This is user code, not a library API. Number deltas and string edits become a
 * `replace` with the new value, computed from `source`.
 */
object Rfc6902 {
  def fromJsonPatch(source: Json, patch: JsonPatch): Either[SchemaError, Json] =
    patchOps(source, "", patch).map(ops => Json.Array(Chunk.from(ops)))

  private def patchOps(value: Json, path: String, patch: JsonPatch): Either[SchemaError, Vector[Json]] =
    steps(value, patch.ops.toVector)(op => JsonPatch(Chunk.single(op))) { (current, op) =>
      if (op.path.nodes.nonEmpty) Left(SchemaError(s"only root paths are supported, got ${op.path}"))
      else opOps(current, path, op.operation)
    }

  private def opOps(value: Json, path: String, op: JsonPatch.Op): Either[SchemaError, Vector[Json]] = op match {
    case JsonPatch.Op.Set(v)            => Right(Vector(rfc("replace", path, Some(v))))
    case JsonPatch.Op.PrimitiveDelta(_) => JsonPatch.root(op)(value).map(v => Vector(rfc("replace", path, Some(v))))
    case JsonPatch.Op.Nested(p)         => patchOps(value, path, p)
    case JsonPatch.Op.ArrayEdit(edits)  =>
      steps(value, edits.toVector)(e => JsonPatch.root(JsonPatch.Op.ArrayEdit(Chunk.single(e)))) {
        case (_, JsonPatch.ArrayOp.Insert(i, vs)) =>
          Right(vs.toVector.zipWithIndex.map { case (v, k) => rfc("add", s"$path/${i + k}", Some(v)) })
        case (_, JsonPatch.ArrayOp.Append(vs))     => Right(vs.toVector.map(v => rfc("add", s"$path/-", Some(v))))
        case (_, JsonPatch.ArrayOp.Delete(i, n))   => Right(Vector.fill(n)(rfc("remove", s"$path/$i", None)))
        case (cur, JsonPatch.ArrayOp.Modify(i, o)) =>
          cur.elements.lift(i).toRight(SchemaError(s"no index $i at $path")).flatMap(opOps(_, s"$path/$i", o))
      }
    case JsonPatch.Op.ObjectEdit(edits) =>
      steps(value, edits.toVector)(e => JsonPatch.root(JsonPatch.Op.ObjectEdit(Chunk.single(e)))) {
        case (_, JsonPatch.ObjectOp.Add(k, v))      => Right(Vector(rfc("add", s"$path/${escape(k)}", Some(v))))
        case (_, JsonPatch.ObjectOp.Remove(k))      => Right(Vector(rfc("remove", s"$path/${escape(k)}", None)))
        case (cur, JsonPatch.ObjectOp.Modify(k, p)) =>
          cur.fields.collectFirst { case (`k`, v) => v }
            .toRight(SchemaError(s"no key $k at $path"))
            .flatMap(patchOps(_, s"$path/${escape(k)}", p))
      }
  }

  // Emits the RFC 6902 operations for each edit, then applies the edit, so later edits see the updated value.
  private def steps[E](value: Json, edits: Vector[E])(asPatch: E => JsonPatch)(
    emit: (Json, E) => Either[SchemaError, Vector[Json]]
  ): Either[SchemaError, Vector[Json]] =
    edits
      .foldLeft[Either[SchemaError, (Json, Vector[Json])]](Right((value, Vector.empty))) { (acc, e) =>
        acc.flatMap { case (current, out) =>
          for {
            ops  <- emit(current, e)
            next <- asPatch(e)(current)
          } yield (next, out ++ ops)
        }
      }
      .map(_._2)

  private def escape(key: String): String = key.replace("~", "~0").replace("/", "~1")

  private def rfc(op: String, path: String, value: Option[Json]): Json =
    Json.Object(Chunk.from(Vector("op" -> Json.String(op), "path" -> Json.String(path)) ++ value.map("value" -> _)))
}

/**
 * Diff and Patch: Exchange JSON Patches with Other Systems
 *
 * Diffs two `Json` documents, applies the `JsonPatch`, converts a typed patch
 * with `JsonPatch.fromDynamicPatch`, and converts three edits to RFC 6902,
 * checking each result against the operations we expect.
 *
 * Run with: cd schema-examples && sbt "runMain diffandpatch.JsonPatchExample"
 */
object JsonPatchExample extends App {

  val mondayJson  = Json.from(zionomicon)
  val tuesdayJson = Json.from(withSubtitle)

  val jsonPatch: JsonPatch = mondayJson.diff(tuesdayJson)
  println(jsonPatch)
  println(mondayJson.patch(jsonPatch).map(_.print))

  println(JsonPatch.fromDynamicPatch(Book.schema.diff(zionomicon, secondEdition).dynamicPatch))

  def check(label: String, source: Json, target: Json, expected: String): Unit = {
    val actual = Rfc6902.fromJsonPatch(source, source.diff(target)).map(_.print)
    println(s"$label: ${actual.fold(_.message, identity)}")
    assert(actual == Right(expected), s"$label: expected $expected")
    println(s"$label: OK")
  }

  check(
    "book",
    mondayJson,
    tuesdayJson,
    """[{"op":"replace","path":"/title","value":"Zionomicon (2nd Edition)"},""" +
      """{"op":"replace","path":"/price","value":39.99},""" +
      """{"op":"add","path":"/tags/-","value":"effects"},""" +
      """{"op":"replace","path":"/edition","value":2}]"""
  )

  check(
    "tags",
    Json.from(zionomicon.copy(tags = List("zio", "scala", "effects", "fibers"))),
    Json.from(zionomicon.copy(tags = List("zio", "fibers", "streams"))),
    """[{"op":"remove","path":"/tags/1"},""" +
      """{"op":"remove","path":"/tags/1"},""" +
      """{"op":"add","path":"/tags/-","value":"streams"}]"""
  )

  check(
    "catalog",
    Json.from(Catalog(Map(zionomicon.id -> zionomicon, programmingInScala.id -> programmingInScala))),
    Json.from(Catalog(Map(zionomicon.id -> secondEdition, fpInScala.id -> fpInScala))),
    """[{"op":"replace","path":"/books/zionomicon/price","value":39.99},""" +
      """{"op":"add","path":"/books/zionomicon/tags/-","value":"effects"},""" +
      """{"op":"replace","path":"/books/zionomicon/edition","value":2},""" +
      """{"op":"add","path":"/books/fp-in-scala","value":""" +
      """{"id":"fp-in-scala","title":"Functional Programming in Scala","price":49.99,"tags":["scala","fp"],"edition":2}},""" +
      """{"op":"remove","path":"/books/programming-in-scala"}]"""
  )
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain diffandpatch.JsonPatchExample"
```

### The Complete Example

The complete example runs every step in one program:

```bash
sbt "runMain diffandpatch.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## Going Further

- [Patch](../reference/schema/patch.md) documents every patch operation, including building patches by hand with optics, such as `Patch.set` and `Patch.increment`.
- [JsonPatch](../reference/schema/built-in-codecs/json/json-patch.md) and [JsonDiffer](../reference/schema/built-in-codecs/json/json-differ.md) cover patches on raw JSON in detail.
- [DynamicValue](../reference/schema/dynamic-value.md) explains the schema-less representation that patches work on.
- [Navigate and Update Nested Data with Optics](./nested-data-with-optics.md) shows the optics that hand-built patches use.
- [Migrating from ZIO Schema to ZIO Blocks Schema](./zio-schema-migration.md) maps ZIO Schema's diff and patch API to this one.

---

## How to Encode and Decode Records with Avro

In this guide, we encode order events as Avro records with `AvroFormat` from ZIO Blocks Schema. By the end, you will have one codec, derived from a sealed trait, that writes the `.avsc` schema file another team needs, turns each event into compact bytes and back, writes and reads a stream of events, and produces bytes that Apache Avro's own Java library reads. A typical place for these bytes is a Kafka message body; the same bytes work in a file, a queue or an HTTP body, so nothing in this guide depends on a broker.

We start from the event types and build up in this order:

- Derive the codec and write the `.avsc` schema file.
- Encode and decode one event, then a stream of events.
- See what `AvroCodec#decode` does with bad input.
- Check how each Scala type maps to Avro, which a reader in another language must know.
- Read and write the same bytes with Apache Avro.
- Learn what the module leaves to you: schema evolution, mismatched schemas, container files and registry framing.

## The Problem

Avro bytes hold only values. The writer and the reader must agree on a schema, a JSON document usually kept in an `.avsc` file, that says which field each value belongs to. When you write that schema by hand, it describes your case class a second time, and nothing keeps the two in step.

For example, consider the `OrderShipped` event. With Apache Avro alone, you write the schema as a JSON string and copy every field into a generic record by name:

```scala
// The schema repeats the case class by hand
val schemaJson =
  """{"type":"record","name":"OrderShipped","fields":[
    |  {"name":"orderId","type":"long"},
    |  {"name":"shippedOn","type":{"type":"record","name":"LocalDate","fields":[
    |    {"name":"year","type":"int"},{"name":"month","type":"int"},{"name":"day","type":"int"}]}},
    |  {"name":"carrier","type":"string"}]}""".stripMargin
val schema = new org.apache.avro.Schema.Parser().parse(schemaJson)

// Every field is copied by name: a misspelled name compiles, and fails only when this line runs
val shippedOn = new org.apache.avro.generic.GenericData.Record(schema.getField("shippedOn").schema())
shippedOn.put("year", 2024)
shippedOn.put("month", 3)
shippedOn.put("day", 9)

val record = new org.apache.avro.generic.GenericData.Record(schema)
record.put("orderId", 7L)
record.put("shippedOn", shippedOn)
record.put("carrier", "DHL")

// The writer and the encoder are set up by hand too
val out     = new java.io.ByteArrayOutputStream()
val encoder = org.apache.avro.io.EncoderFactory.get.binaryEncoder(out, null)
new org.apache.avro.generic.GenericDatumWriter[AnyRef](schema).write(record, encoder)
encoder.flush()
```

Add a field to the case class and you must remember to change the JSON, the `GenericData.Record#put` calls and the reading code in every service that consumes the event. A mistake shows up as a decoding failure in another service, often long after the change. In this guide, the case class is the single source: its `Schema` derives the Avro schema, the encoder and the decoder, so they cannot drift apart.

## Prerequisites

Add the Avro module to your build. It runs on the JVM only; there is no Scala.js artifact:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema-avro" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema-avro" % "0.0.56"
```

The module depends on Apache Avro (`org.apache.avro:avro`), so the Apache Avro classes used later in this guide are already on the classpath. We assume you can derive a `Schema` for a case class, as shown in [Your First Schema](./your-first-schema.md).

Every code block in this guide uses these imports:

```scala

```

## The Order Events

The running example is the life of an order: placed, shipped or cancelled. `OrderEvent` is a sealed trait and each event is a case class, so one codec covers all three:

```scala
sealed trait OrderEvent

object OrderEvent {
  implicit val schema: Schema[OrderEvent] = Schema.derived
}

case class OrderPlaced(
  orderId: Long,
  customer: String,
  amount: BigDecimal,
  placedAt: Instant,
  note: Option[String],
  items: List[LineItem]
) extends OrderEvent

case class OrderShipped(orderId: Long, shippedOn: LocalDate, carrier: String) extends OrderEvent

case class OrderCancelled(orderId: Long, reason: String) extends OrderEvent

case class LineItem(sku: String, quantity: Int, unitPrice: BigDecimal)
```

The fields cover what readers of Avro data usually ask about: ids, money, timestamps, an optional note and a list of line items.

<details>
  <summary>The same model as a file, shared by every example: <code>Model.scala</code></summary>

```scala title="schema-examples/src/main/scala/avrorecords/Model.scala" showLineNumbers
package avrorecords

sealed trait OrderEvent

object OrderEvent {
  implicit val schema: Schema[OrderEvent] = Schema.derived
}

case class OrderPlaced(
  orderId: Long,
  customer: String,
  amount: BigDecimal,
  placedAt: Instant,
  note: Option[String],
  items: List[LineItem]
) extends OrderEvent

case class OrderShipped(orderId: Long, shippedOn: LocalDate, carrier: String) extends OrderEvent

case class OrderCancelled(orderId: Long, reason: String) extends OrderEvent

case class LineItem(sku: String, quantity: Int, unitPrice: BigDecimal)
```

</details>

## Write the .avsc Schema File

An Avro reader needs the schema that the writer encodes with. Derive the codec once from the schema of the sealed trait; the result is an `AvroCodec[OrderEvent]`, which carries that schema:

```scala
val codec = OrderEvent.schema.derive(AvroFormat)
```

`AvroCodec#avroSchema` returns the schema as an Apache Avro `Schema`, and Apache Avro's `SchemaFormatter` turns it into the JSON text of an `.avsc` file. To print the schema of the smallest event, `OrderCancelled`:

```scala

println(SchemaFormatter.format("json/pretty", Schema.derived[OrderCancelled].derive(AvroFormat).avroSchema))
// {
//   "type" : "record",
//   "name" : "OrderCancelled",
//   "namespace" : "repl.MdocSession.MdocApp",
//   "fields" : [ {
//     "name" : "orderId",
//     "type" : "long"
//   }, {
//     "name" : "reason",
//     "type" : "string"
//   } ]
// }
```

The namespace above is `repl.MdocSession.MdocApp` because the documentation build compiles this page inside its own object. In your project, the namespace is the package of the case class (`avrorecords` in the example files), so moving a case class to another package changes the schema you share.

For the whole sealed trait, `SchemaFormatter.format("json/pretty", codec.avroSchema)` prints a union: a JSON array with one record per event, in the order the events are declared. Write that text to `order-event.avsc` with `java.nio.file.Files#writeString` and hand the file to the team that reads your events; the example file for this section does exactly that. `SchemaFormatter` is the printer since Apache Avro 1.12; the older `Schema#toString(true)` is deprecated.

<details>
  <summary>Runnable version, which also writes <code>order-event.avsc</code>: <code>Step1WriteAvscSchemaExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/avrorecords/Step1WriteAvscSchemaExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-avro:0.0.56
//> using options -experimental
//> using file Model.scala
// Run from this folder:  scala-cli run Step1WriteAvscSchemaExample.scala
// Or from schema-examples:  sbt "runMain avrorecords.Step1WriteAvscSchemaExample"

package avrorecords

/**
 * How to Encode and Decode Records with Avro — Write the .avsc Schema File
 *
 * Derives the codec, prints the schema of `OrderCancelled`, and writes the
 * schema of the whole sealed trait to `order-event.avsc`.
 */
object Step1WriteAvscSchemaExample {
  def main(args: Array[String]): Unit = {
    val codec = OrderEvent.schema.derive(AvroFormat)

    println(SchemaFormatter.format("json/pretty", Schema.derived[OrderCancelled].derive(AvroFormat).avroSchema))

    // The whole sealed trait, written to order-event.avsc in the current directory
    val avscText = SchemaFormatter.format("json/pretty", codec.avroSchema)
    val avscFile = Paths.get("order-event.avsc")
    Files.writeString(avscFile, avscText)
    println(
      s"wrote $avscFile: ${Files.readAllLines(avscFile).size} lines, starting with ${Files.readAllLines(avscFile).get(0)}"
    )
  }
}
```

</details>

## Encode and Decode One Event

`AvroCodec#encode` turns a value into a byte array, and `AvroCodec#decode` turns the bytes back into a value. To encode a placed order:

```scala
val event: OrderEvent = OrderPlaced(
  orderId = 42L,
  customer = "Ada",
  amount = BigDecimal("19.99"),
  placedAt = Instant.ofEpochSecond(1700000000L, 123000000),
  note = Some("gift"),
  items = List(LineItem("sku-1", 2, BigDecimal("9.995")))
)

val bytes = codec.encode(event)
```

To print the bytes in hexadecimal and decode them back:

```scala
println(s"${bytes.length} bytes: ${bytes.map(b => f"$b%02x").mkString(" ")}")
// 42 bytes: 00 54 06 41 64 61 04 07 cf 04 44 0c 80 c4 9f d5 0c 80 d3 a6 75 02 08 67 69 66 74 02 0a 73 6b 75 2d 31 04 04 27 0b 06 44 0c 00

println(codec.decode(bytes))
// Right(OrderPlaced(42,Ada,19.99,2023-11-14T22:13:20.123Z,Some(gift),List(LineItem(sku-1,2,9.995))))
```

The bytes hold values only; the field names live in the schema. The first byte, `00`, is the index of the `OrderPlaced` branch of the union. Then come the order id `54` (42 as a zig-zag variable-length number) and the customer `06 41 64 61` (a length of 3, then the letters `Ada`). `AvroCodec#decode` returns an `Either[SchemaError, OrderEvent]`, so a failure is a value to handle, not an exception.

<details>
  <summary>Runnable version, with the codec from “Write the .avsc Schema File”: <code>Step2EncodeDecodeExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/avrorecords/Step2EncodeDecodeExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-avro:0.0.56
//> using options -experimental
//> using file Model.scala
// Run from this folder:  scala-cli run Step2EncodeDecodeExample.scala
// Or from schema-examples:  sbt "runMain avrorecords.Step2EncodeDecodeExample"

package avrorecords

/**
 * How to Encode and Decode Records with Avro — Encode and Decode One Event
 *
 * Encodes one `OrderPlaced` event, prints the bytes in hexadecimal and decodes
 * them back.
 */
object Step2EncodeDecodeExample {
  def main(args: Array[String]): Unit = {
    // Setup from "Write the .avsc Schema File": the codec
    val codec = OrderEvent.schema.derive(AvroFormat)

    val event: OrderEvent = OrderPlaced(
      orderId = 42L,
      customer = "Ada",
      amount = BigDecimal("19.99"),
      placedAt = Instant.ofEpochSecond(1700000000L, 123000000),
      note = Some("gift"),
      items = List(LineItem("sku-1", 2, BigDecimal("9.995")))
    )

    val bytes = codec.encode(event)

    println(s"${bytes.length} bytes: ${bytes.map(b => f"$b%02x").mkString(" ")}")

    println(codec.decode(bytes))
  }
}
```

</details>

## Write Several Events to One Stream

`AvroCodec#encode(value, OutputStream)` writes one record to any `OutputStream`, such as a file, a socket or a response body. To write three shipped orders to one buffer:

```scala
val shipped: List[OrderEvent] = List(
  OrderShipped(1L, LocalDate.of(2024, 1, 1), "DHL"),
  OrderShipped(2L, LocalDate.of(2024, 1, 2), "UPS"),
  OrderShipped(3L, LocalDate.of(2024, 1, 3), "FedEx")
)

val buffer = new ByteArrayOutputStream()

shipped.foreach(e => codec.encode(e, buffer))

val input = new ByteArrayInputStream(buffer.toByteArray)
```

`AvroCodec#decode(InputStream)` reads exactly one record and leaves the rest of the stream unread, so each call continues where the previous one stopped. To read the three events back:

```scala
println(s"stream holds ${buffer.size()} bytes")
// stream holds 32 bytes

shipped.foreach(_ => println(codec.decode(input)))
// Right(OrderShipped(1,2024-01-01,DHL))
// Right(OrderShipped(2,2024-01-02,UPS))
// Right(OrderShipped(3,2024-01-03,FedEx))

println(s"bytes left in the stream: ${input.available()}")
// bytes left in the stream: 0
```

The 32 bytes are the three records back to back, with no length and no separator between them. The stream carries no count and no end marker either, so the reader must know when to stop; here we read as many events as we wrote.

<details>
  <summary>Runnable version, with the codec from “Write the .avsc Schema File”: <code>Step3StreamEventsExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/avrorecords/Step3StreamEventsExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-avro:0.0.56
//> using options -experimental
//> using file Model.scala
// Run from this folder:  scala-cli run Step3StreamEventsExample.scala
// Or from schema-examples:  sbt "runMain avrorecords.Step3StreamEventsExample"

package avrorecords

/**
 * How to Encode and Decode Records with Avro — Write Several Events to One
 * Stream
 *
 * Writes three `OrderShipped` events to one stream, back to back, and reads
 * them back one record at a time.
 */
object Step3StreamEventsExample {
  def main(args: Array[String]): Unit = {
    // Setup from "Write the .avsc Schema File": the codec
    val codec = OrderEvent.schema.derive(AvroFormat)

    val shipped: List[OrderEvent] = List(
      OrderShipped(1L, LocalDate.of(2024, 1, 1), "DHL"),
      OrderShipped(2L, LocalDate.of(2024, 1, 2), "UPS"),
      OrderShipped(3L, LocalDate.of(2024, 1, 3), "FedEx")
    )

    val buffer = new ByteArrayOutputStream()

    shipped.foreach(e => codec.encode(e, buffer))

    val input = new ByteArrayInputStream(buffer.toByteArray)

    println(s"stream holds ${buffer.size()} bytes")

    shipped.foreach(_ => println(codec.decode(input)))

    println(s"bytes left in the stream: ${input.available()}")
  }
}
```

</details>

## Handle Bad Input

`AvroCodec#decode` reports malformed bytes as a `SchemaError` that names the path of the field it was reading. When the bytes end too early, the error points at the field where the input ran out:

```scala
println(codec.decode(bytes.take(10)))
// Left(zio.blocks.schema.SchemaError: Unexpected end of input at: .when[OrderPlaced].amount)
```

Extra bytes after a complete record are not an error. `AvroCodec#decode` returns the record and ignores the rest:

```scala
println(codec.decode(bytes ++ Array[Byte](1, 2, 3)))
// Right(OrderPlaced(42,Ada,19.99,2023-11-14T22:13:20.123Z,Some(gift),List(LineItem(sku-1,2,9.995))))
```

:::warning
Two records in one byte array decode as the first record, with no error. Give every message its own byte array, or frame the messages yourself, for example with a length prefix.
:::

A collection whose length prefix is impossibly large fails before any element is read. The limit is `AvroCodec.maxCollectionSize`, 2147483639 elements. To build such a prefix with Apache Avro's encoder and decode it as a list:

```scala

val hugeList = new ByteArrayOutputStream()
val prefixEncoder = EncoderFactory.get.binaryEncoder(hugeList, null)
prefixEncoder.writeLong(AvroCodec.maxCollectionSize.toLong + 1L)
prefixEncoder.writeLong(0L)
prefixEncoder.flush()
```

The decoder reads the length prefix and stops there:

```scala
println(Schema[List[Int]].derive(AvroFormat).decode(hugeList.toByteArray))
// Left(zio.blocks.schema.SchemaError: Expected collection size not greater than 2147483639, got 2147483640 at: .)
```

<details>
  <summary>Runnable version, with the event and bytes from the earlier sections: <code>Step4BadInputExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/avrorecords/Step4BadInputExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-avro:0.0.56
//> using options -experimental
//> using file Model.scala
// Run from this folder:  scala-cli run Step4BadInputExample.scala
// Or from schema-examples:  sbt "runMain avrorecords.Step4BadInputExample"

package avrorecords

/**
 * How to Encode and Decode Records with Avro — Handle Bad Input
 *
 * Decodes truncated bytes, bytes with extra data at the end, and a list whose
 * length prefix is above `AvroCodec.maxCollectionSize`.
 */
object Step4BadInputExample {
  def main(args: Array[String]): Unit = {
    // Setup from "Write the .avsc Schema File" and "Encode and Decode One Event": the codec and the bytes of one event
    val codec             = OrderEvent.schema.derive(AvroFormat)
    val event: OrderEvent = OrderPlaced(
      orderId = 42L,
      customer = "Ada",
      amount = BigDecimal("19.99"),
      placedAt = Instant.ofEpochSecond(1700000000L, 123000000),
      note = Some("gift"),
      items = List(LineItem("sku-1", 2, BigDecimal("9.995")))
    )
    val bytes = codec.encode(event)

    println(codec.decode(bytes.take(10)))

    println(codec.decode(bytes ++ Array[Byte](1, 2, 3)))

    val hugeList      = new ByteArrayOutputStream()
    val prefixEncoder = EncoderFactory.get.binaryEncoder(hugeList, null)
    prefixEncoder.writeLong(AvroCodec.maxCollectionSize.toLong + 1L)
    prefixEncoder.writeLong(0L)
    prefixEncoder.flush()

    println(Schema[List[Int]].derive(AvroFormat).decode(hugeList.toByteArray))
  }
}
```

</details>

## Check How Scala Types Map to Avro

A reader in Java, Python or another Avro library works from the `.avsc` file, so it sees each Scala type as the Avro type that the codec derives. To print that Avro type for the common Scala types, we define a small helper and a few sample types, a sealed trait of case objects, a one-field case class, and an id built with `Schema#transform`:

```scala

def avsc[A](implicit schema: Schema[A]): String = schema.derive(AvroFormat).avroSchema.toString

sealed trait Status
case object Active extends Status
case object Closed extends Status

case class CustomerId(value: Long)

case class Sku(value: String)

implicit val statusSchema: Schema[Status]         = Schema.derived
implicit val customerIdSchema: Schema[CustomerId] = Schema.derived
implicit val skuSchema: Schema[Sku]               = Schema[String].transform(Sku(_), _.value)
implicit val bytesSchema: Schema[Array[Byte]]     = Schema.derived
implicit val chunkSchema: Schema[Chunk[Byte]]     = Schema.derived
```

To print the Avro type of each one:

```scala
List(
  "Int"                 -> avsc[Int],
  "Long"                -> avsc[Long],
  "Short"               -> avsc[Short],
  "Byte"                -> avsc[Byte],
  "Char"                -> avsc[Char],
  "Float"               -> avsc[Float],
  "Double"              -> avsc[Double],
  "Boolean"             -> avsc[Boolean],
  "String"              -> avsc[String],
  "Unit"                -> avsc[Unit],
  "BigInt"              -> avsc[BigInt],
  "BigDecimal"          -> avsc[BigDecimal],
  "Array[Byte]"         -> avsc[Array[Byte]],
  "Chunk[Byte]"         -> avsc[Chunk[Byte]],
  "List[Int]"           -> avsc[List[Int]],
  "Map[String, Int]"    -> avsc[Map[String, Int]],
  "Instant"             -> avsc[Instant],
  "LocalDate"           -> avsc[LocalDate],
  "UUID"                -> avsc[UUID],
  "Option[Int]"         -> avsc[Option[Int]],
  "Either[String, Int]" -> avsc[Either[String, Int]],
  "Status"              -> avsc[Status],
  "CustomerId"          -> avsc[CustomerId],
  "Sku"                 -> avsc[Sku]
).foreach { case (scalaType, avroType) => println(s"$scalaType: $avroType") }
// Int: "int"
// Long: "long"
// Short: "int"
// Byte: "int"
// Char: "int"
// Float: "float"
// Double: "double"
// Boolean: "boolean"
// String: "string"
// Unit: "null"
// BigInt: "bytes"
// BigDecimal: {"type":"record","name":"BigDecimal","namespace":"scala","fields":[{"name":"mantissa","type":"bytes"},{"name":"scale","type":"int"},{"name":"precision","type":"int"},{"name":"roundingMode","type":"int"}]}
// Array[Byte]: {"type":"array","items":"int"}
// Chunk[Byte]: {"type":"array","items":"int"}
// List[Int]: {"type":"array","items":"int"}
// Map[String, Int]: {"type":"map","values":"int"}
// Instant: {"type":"record","name":"Instant","namespace":"java.time","fields":[{"name":"epochSecond","type":"long"},{"name":"nano","type":"int"}]}
// LocalDate: {"type":"record","name":"LocalDate","namespace":"java.time","fields":[{"name":"year","type":"int"},{"name":"month","type":"int"},{"name":"day","type":"int"}]}
// UUID: {"type":"fixed","name":"UUID","namespace":"java.util","size":16}
// Option[Int]: [{"type":"record","name":"None","namespace":"scala","fields":[]},{"type":"record","name":"Some","namespace":"scala","fields":[{"name":"value","type":"int"}]}]
// Either[String, Int]: [{"type":"record","name":"Left","namespace":"scala.util","fields":[{"name":"value","type":"string"}]},{"type":"record","name":"Right","namespace":"scala.util","fields":[{"name":"value","type":"int"}]}]
// Status: [{"type":"record","name":"Active","namespace":"repl.MdocSession.MdocApp","fields":[]},{"type":"record","name":"Closed","namespace":"repl.MdocSession.MdocApp","fields":[]}]
// CustomerId: {"type":"record","name":"CustomerId","namespace":"repl.MdocSession.MdocApp","fields":[{"name":"value","type":"long"}]}
// Sku: "string"
```

The table sums up the output and what a reader in another language should know about each type:

| Scala type                            | Avro type                                                 | What a reader in another language sees                                                        |
| ------------------------------------- | --------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| `Int`, `Short`, `Byte`                | `int`                                                     | A 32-bit number; `Short` and `Byte` are widened                                               |
| `Long`                                | `long`                                                    |                                                                                               |
| `Float`, `Double`                     | `float`, `double`                                         |                                                                                               |
| `Boolean`                             | `boolean`                                                 |                                                                                               |
| `String`                              | `string`                                                  |                                                                                               |
| `Char`                                | `int`                                                     | The character's code, not a one-letter string                                                 |
| `Unit`                                | `null`                                                    |                                                                                               |
| `BigInt`                              | `bytes`                                                   | Two's-complement big-endian bytes                                                             |
| `BigDecimal`                          | record `scala.BigDecimal`                                 | `mantissa` bytes plus `scale`, `precision` and `roundingMode`; not the `decimal` logical type |
| `Array[Byte]`, `Chunk[Byte]`          | array of `int`                                            | One number per byte, not `bytes`                                                              |
| `List[A]`                             | array of `A`                                              |                                                                                               |
| `Map[String, A]`                      | map of `A`                                                |                                                                                               |
| `Instant`                             | record `java.time.Instant`                                | `epochSecond` and `nano`; not the `timestamp-millis` logical type                             |
| `LocalDate`                           | record `java.time.LocalDate`                              | `year`, `month` and `day`; not the `date` logical type                                        |
| `UUID`                                | `fixed` of 16 bytes named `java.util.UUID`                | Not the `uuid` logical type                                                                   |
| `Option[A]`                           | union of records `scala.None` and `scala.Some`            | Not `["null", A]`; `None` is branch 0 and `Some` holds `value`                                |
| `Either[A, B]`                        | union of records `scala.util.Left` and `scala.util.Right` | Each side holds `value`                                                                       |
| Sealed trait                          | union of one record per case, in declaration order        | A case object is a record with no fields                                                      |
| Case class                            | record named after the class, namespace from its package  | A one-field class such as `CustomerId` stays a record                                         |
| Newtype built with `Schema#transform` | the wrapped type                                          | `Sku` is a plain `string`                                                                     |

Four rows surprise readers in other languages. `Option` is a union of two records, not the `["null", T]` union that most Avro libraries write for an optional field, so a reader that expects `null` for a missing value does not find one. `Array[Byte]` is an array of `int` rather than `bytes`, so a byte payload comes out as a list of numbers. `BigDecimal`, `Instant` and `LocalDate` are records and `UUID` is a 16-byte `fixed`, not the `decimal`, `timestamp-millis`, `date` and `uuid` logical types. And `Char` is a number, not a one-letter string. zio-blocks does not write those other forms, so if another team expects them, agree on the layout with them using the `.avsc` file.

<details>
  <summary>Runnable version, as one program: <code>Step5TypeMappingExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/avrorecords/Step5TypeMappingExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-avro:0.0.56
//> using options -experimental
// Run from this folder:  scala-cli run Step5TypeMappingExample.scala
// Or from schema-examples:  sbt "runMain avrorecords.Step5TypeMappingExample"

package avrorecords

/**
 * How to Encode and Decode Records with Avro — Check How Scala Types Map to
 * Avro
 *
 * Prints the Avro schema that the codec derives for common Scala types.
 */
object Step5TypeMappingExample {

  def avsc[A](implicit schema: Schema[A]): String = schema.derive(AvroFormat).avroSchema.toString

  sealed trait Status
  case object Active extends Status
  case object Closed extends Status

  case class CustomerId(value: Long)

  case class Sku(value: String)

  implicit val statusSchema: Schema[Status]         = Schema.derived
  implicit val customerIdSchema: Schema[CustomerId] = Schema.derived
  implicit val skuSchema: Schema[Sku]               = Schema[String].transform(Sku(_), _.value)
  implicit val bytesSchema: Schema[Array[Byte]]     = Schema.derived
  implicit val chunkSchema: Schema[Chunk[Byte]]     = Schema.derived

  def main(args: Array[String]): Unit =
    List(
      "Int"                 -> avsc[Int],
      "Long"                -> avsc[Long],
      "Short"               -> avsc[Short],
      "Byte"                -> avsc[Byte],
      "Char"                -> avsc[Char],
      "Float"               -> avsc[Float],
      "Double"              -> avsc[Double],
      "Boolean"             -> avsc[Boolean],
      "String"              -> avsc[String],
      "Unit"                -> avsc[Unit],
      "BigInt"              -> avsc[BigInt],
      "BigDecimal"          -> avsc[BigDecimal],
      "Array[Byte]"         -> avsc[Array[Byte]],
      "Chunk[Byte]"         -> avsc[Chunk[Byte]],
      "List[Int]"           -> avsc[List[Int]],
      "Map[String, Int]"    -> avsc[Map[String, Int]],
      "Instant"             -> avsc[Instant],
      "LocalDate"           -> avsc[LocalDate],
      "UUID"                -> avsc[UUID],
      "Option[Int]"         -> avsc[Option[Int]],
      "Either[String, Int]" -> avsc[Either[String, Int]],
      "Status"              -> avsc[Status],
      "CustomerId"          -> avsc[CustomerId],
      "Sku"                 -> avsc[Sku]
    ).foreach { case (scalaType, avroType) => println(s"$scalaType: $avroType") }
}
```

</details>

## Read the Bytes with Apache Avro

The proof that the bytes are standard Avro is to read them with another Avro library. Apache Avro's `GenericDatumReader` reads them with the schema that zio-blocks emits, parsed by Apache Avro's own parser:

```scala

val avroSchema = new Parser().parse(codec.avroSchema.toString)

val reader = new GenericDatumReader[AnyRef](avroSchema)

def readWithApacheAvro(bytes: Array[Byte]): AnyRef =
  reader.read(null, DecoderFactory.get.binaryDecoder(bytes, null))
```

To read a shipped order and the placed order from the earlier sections:

```scala
println(readWithApacheAvro(codec.encode(shipped.head)))
// {"orderId": 1, "shippedOn": {"year": 2024, "month": 1, "day": 1}, "carrier": "DHL"}

println(readWithApacheAvro(bytes))
// {"orderId": 42, "customer": "Ada", "amount": {"mantissa": "\u0007Ï", "scale": 2, "precision": 34, "roundingMode": 6}, "placedAt": {"epochSecond": 1700000000, "nano": 123000000}, "note": {"value": "gift"}, "items": [{"sku": "sku-1", "quantity": 2, "unitPrice": {"mantissa": "'\u000B", "scale": 3, "precision": 34, "roundingMode": 6}}]}
```

Apache Avro returns a `GenericRecord` for each event. The placed order shows how the types from the previous section look to a Java reader: `amount` is a record of raw `mantissa` bytes, `scale`, `precision` and `roundingMode`, `placedAt` is a record of seconds and nanoseconds, and `note` is the `Some` record with its `value`.

<details>
  <summary>Runnable version, with the codec and events from the earlier sections: <code>Step6ReadWithApacheAvroExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/avrorecords/Step6ReadWithApacheAvroExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-avro:0.0.56
//> using options -experimental
//> using file Model.scala
// Run from this folder:  scala-cli run Step6ReadWithApacheAvroExample.scala
// Or from schema-examples:  sbt "runMain avrorecords.Step6ReadWithApacheAvroExample"

package avrorecords

/**
 * How to Encode and Decode Records with Avro — Read the Bytes with Apache Avro
 *
 * Reads bytes written by zio-blocks with Apache Avro's `GenericDatumReader`,
 * using the schema that zio-blocks emits.
 */
object Step6ReadWithApacheAvroExample {
  def main(args: Array[String]): Unit = {
    // Setup from the earlier sections: the codec, a shipped order and the bytes of a placed order
    val codec                     = OrderEvent.schema.derive(AvroFormat)
    val shipped: List[OrderEvent] = List(OrderShipped(1L, LocalDate.of(2024, 1, 1), "DHL"))
    val event: OrderEvent         = OrderPlaced(
      orderId = 42L,
      customer = "Ada",
      amount = BigDecimal("19.99"),
      placedAt = Instant.ofEpochSecond(1700000000L, 123000000),
      note = Some("gift"),
      items = List(LineItem("sku-1", 2, BigDecimal("9.995")))
    )
    val bytes = codec.encode(event)

    val avroSchema = new Parser().parse(codec.avroSchema.toString)

    val reader = new GenericDatumReader[AnyRef](avroSchema)

    def readWithApacheAvro(bytes: Array[Byte]): AnyRef =
      reader.read(null, DecoderFactory.get.binaryDecoder(bytes, null))

    println(readWithApacheAvro(codec.encode(shipped.head)))

    println(readWithApacheAvro(bytes))
  }
}
```

</details>

## Write Bytes with Apache Avro

The other direction is to write with Apache Avro and read with zio-blocks. The second branch of the union is the `OrderShipped` record, so we fill a generic record of that branch and write it with `GenericDatumWriter`:

```scala
val shippedSchema = avroSchema.getTypes.get(1)

val shippedOn = new GenericData.Record(shippedSchema.getField("shippedOn").schema())
shippedOn.put("year", 2024)
shippedOn.put("month", 3)
shippedOn.put("day", 9)

val fromJava = new GenericData.Record(shippedSchema)
fromJava.put("orderId", 7L)
fromJava.put("shippedOn", shippedOn)
fromJava.put("carrier", "DHL")

val javaBytes = new ByteArrayOutputStream()
val javaEncoder = EncoderFactory.get.binaryEncoder(javaBytes, null)
new GenericDatumWriter[AnyRef](avroSchema).write(fromJava, javaEncoder)
javaEncoder.flush()
```

To decode the bytes that Apache Avro wrote:

```scala
println(codec.decode(javaBytes.toByteArray))
// Right(OrderShipped(7,2024-03-09,DHL))
```

<details>
  <summary>Runnable version, with the codec and the parsed schema from the earlier sections: <code>Step7WriteWithApacheAvroExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/avrorecords/Step7WriteWithApacheAvroExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-avro:0.0.56
//> using options -experimental
//> using file Model.scala
// Run from this folder:  scala-cli run Step7WriteWithApacheAvroExample.scala
// Or from schema-examples:  sbt "runMain avrorecords.Step7WriteWithApacheAvroExample"

package avrorecords

/**
 * How to Encode and Decode Records with Avro — Write Bytes with Apache Avro
 *
 * Writes an `OrderShipped` record with Apache Avro's `GenericDatumWriter` and
 * decodes the bytes with zio-blocks.
 */
object Step7WriteWithApacheAvroExample {
  def main(args: Array[String]): Unit = {
    // Setup from the earlier sections: the codec and its schema parsed by Apache Avro
    val codec      = OrderEvent.schema.derive(AvroFormat)
    val avroSchema = new Parser().parse(codec.avroSchema.toString)

    val shippedSchema = avroSchema.getTypes.get(1)

    val shippedOn = new GenericData.Record(shippedSchema.getField("shippedOn").schema())
    shippedOn.put("year", 2024)
    shippedOn.put("month", 3)
    shippedOn.put("day", 9)

    val fromJava = new GenericData.Record(shippedSchema)
    fromJava.put("orderId", 7L)
    fromJava.put("shippedOn", shippedOn)
    fromJava.put("carrier", "DHL")

    val javaBytes   = new ByteArrayOutputStream()
    val javaEncoder = EncoderFactory.get.binaryEncoder(javaBytes, null)
    new GenericDatumWriter[AnyRef](avroSchema).write(fromJava, javaEncoder)
    javaEncoder.flush()

    println(codec.decode(javaBytes.toByteArray))
  }
}
```

</details>

## What Avro Support Does Not Cover

The module encodes and decodes one value with one schema, and several things that Avro users expect from other libraries are left to you. The codec writes the bytes of a single value. An Avro container file, the `.avro` file that Apache Avro's `DataFileWriter` writes with its schema embedded, is not supported, so a reader of a container file needs Apache Avro itself. There is also no registry framing: a schema registry stores your `.avsc` files under version ids and expects each message to start with the id of its schema, which you add yourself. [Build a Schema Registry](./build-a-schema-registry.md) shows how to store schemas as data in ZIO Blocks. The two cases below need more care, because the codec does not report them.

### Old and New Versions of a Type

There is no reader and writer schema resolution: the codec reads bytes with its own schema only, and a default value in the new type is not used to fill a field that the old bytes lack. To see both directions, we define two versions of an order, the second with a new field that has a default:

```scala
case class OrderV1(orderId: Long, customer: String)

object OrderV1 {
  implicit val schema: Schema[OrderV1] = Schema.derived
}

case class OrderV2(orderId: Long, customer: String, priority: Int = 0)

object OrderV2 {
  implicit val schema: Schema[OrderV2] = Schema.derived
}

val v1Bytes = OrderV1.schema.derive(AvroFormat).encode(OrderV1(9L, "Bob"))
val v2Bytes = OrderV2.schema.derive(AvroFormat).encode(OrderV2(9L, "Bob", 3))
```

To read old bytes with the new type, and new bytes with the old type:

```scala
println(OrderV2.schema.derive(AvroFormat).decode(v1Bytes))
// Left(zio.blocks.schema.SchemaError: Unexpected end of input at: .priority)

println(OrderV1.schema.derive(AvroFormat).decode(v2Bytes))
// Right(OrderV1(9,Bob))
```

Old bytes fail with an error at the new field, and new bytes read by the old type lose the new field without an error. To change a type safely, convert between versions in Scala with `Into`, `As` or `Migration`, as shown in [Evolve Your Data](./evolve-your-data.md).

### Bytes Written by a Different Type

Avro bytes carry no schema id and no fingerprint, so nothing stops a reader from decoding bytes with the wrong schema. To decode the bytes of a shipped order as an `OrderV1`:

```scala
println(OrderV1.schema.derive(AvroFormat).decode(codec.encode(shipped.head)))
// Right(OrderV1(1,�))
```

The decode succeeds and returns nonsense. The union index becomes the `orderId`, the real order id becomes the length of `customer`, and `customer` holds one byte of the date; the remaining bytes are ignored, as in the previous section. Keep track of which schema wrote the bytes, for example with one topic or file per event type, a version field, or a schema registry.

<details>
  <summary>Runnable version, with the codec and events from the earlier sections: <code>Step8AvroLimitsExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/avrorecords/Step8AvroLimitsExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-avro:0.0.56
//> using options -experimental
//> using file Model.scala
// Run from this folder:  scala-cli run Step8AvroLimitsExample.scala
// Or from schema-examples:  sbt "runMain avrorecords.Step8AvroLimitsExample"

package avrorecords

/**
 * How to Encode and Decode Records with Avro — What Avro Support Does Not Cover
 *
 * Reads old bytes with a new version of a type and new bytes with the old
 * version, then decodes the bytes of one type as another type.
 */
object Step8AvroLimitsExample {

  case class OrderV1(orderId: Long, customer: String)

  object OrderV1 {
    implicit val schema: Schema[OrderV1] = Schema.derived
  }

  case class OrderV2(orderId: Long, customer: String, priority: Int = 0)

  object OrderV2 {
    implicit val schema: Schema[OrderV2] = Schema.derived
  }

  def main(args: Array[String]): Unit = {
    // Setup from the earlier sections: the codec and a shipped order
    val codec                     = OrderEvent.schema.derive(AvroFormat)
    val shipped: List[OrderEvent] = List(OrderShipped(1L, LocalDate.of(2024, 1, 1), "DHL"))

    val v1Bytes = OrderV1.schema.derive(AvroFormat).encode(OrderV1(9L, "Bob"))
    val v2Bytes = OrderV2.schema.derive(AvroFormat).encode(OrderV2(9L, "Bob", 3))

    println(OrderV2.schema.derive(AvroFormat).decode(v1Bytes))

    println(OrderV1.schema.derive(AvroFormat).decode(v2Bytes))

    println(OrderV1.schema.derive(AvroFormat).decode(codec.encode(shipped.head)))
  }
}
```

</details>

## Putting It Together

The complete program derives the codec, prints the `.avsc` schema of the whole union, writes a stream of three events to a buffer, reads them back, and checks the round trip. It differs from the sections in three ways: the stream mixes an `OrderPlaced`, an `OrderShipped` and an `OrderCancelled` event, where the stream section wrote only shipped orders; it prints the schema of the whole union rather than of `OrderCancelled`; and it ends by comparing the decoded events with the originals:

```scala title="schema-examples/src/main/scala/avrorecords/CompleteExample.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-avro:0.0.56
//> using options -experimental
//> using file Model.scala
// Run from this folder:  scala-cli run CompleteExample.scala
// Or from schema-examples:  sbt "runMain avrorecords.CompleteExample"

package avrorecords

/**
 * How to Encode and Decode Records with Avro — Putting It Together
 *
 * Derives the codec, prints the `.avsc` schema of the whole sealed trait,
 * writes three different events to one stream and reads them back.
 */
object CompleteExample {
  def main(args: Array[String]): Unit = {
    val codec = OrderEvent.schema.derive(AvroFormat)

    println(SchemaFormatter.format("json/pretty", codec.avroSchema))

    val events: List[OrderEvent] = List(
      OrderPlaced(
        orderId = 42L,
        customer = "Ada",
        amount = BigDecimal("19.99"),
        placedAt = Instant.ofEpochSecond(1700000000L, 123000000),
        note = Some("gift"),
        items = List(LineItem("sku-1", 2, BigDecimal("9.995")))
      ),
      OrderShipped(42L, LocalDate.of(2024, 1, 3), "DHL"),
      OrderCancelled(43L, "customer request")
    )

    val buffer = new ByteArrayOutputStream()

    events.foreach(e => codec.encode(e, buffer))

    val input = new ByteArrayInputStream(buffer.toByteArray)

    println(s"stream holds ${buffer.size()} bytes")

    val decoded = events.map(_ => codec.decode(input))

    decoded.foreach(println)

    println(s"decoded events equal the originals: ${decoded == events.map(Right(_))}")
  }
}
```

Running it prints the following; the `.avsc` text, 120 lines in full, is cut after its first field, where the `...` line stands:

```text
[ {
  "type" : "record",
  "name" : "OrderPlaced",
  "namespace" : "avrorecords",
  "fields" : [ {
    "name" : "orderId",
    "type" : "long"
...
stream holds 71 bytes
Right(OrderPlaced(42,Ada,19.99,2023-11-14T22:13:20.123Z,Some(gift),List(LineItem(sku-1,2,9.995))))
Right(OrderShipped(42,2024-01-03,DHL))
Right(OrderCancelled(43,customer request))
decoded events equal the originals: true
```

## Running the Examples

All runnable code from this guide is in [`schema-examples/src/main/scala/avrorecords/`](https://github.com/zio/zio-blocks/tree/main/schema-examples/src/main/scala/avrorecords). Run any file from that folder with `scala-cli run <File>.scala`, or from `schema-examples` with `sbt "runMain avrorecords.<ObjectName>"`.

## Going Further

- The [Avro codec reference](../reference/schema/built-in-codecs/avro.md) covers every method of `AvroCodec`, the error model and recursive types.
- [One Type, Many Formats](./one-type-many-formats.md) derives JSON, MessagePack, Avro and other codecs from one schema and compares their sizes.
- [Build a Schema Registry](./build-a-schema-registry.md) stores versioned event schemas as data, the usual next step once several services exchange Avro messages.
- [Evolve Your Data](./evolve-your-data.md) changes a type between versions with `Into`, `As` and `Migration`.
- The `AvroListOfRecordsBenchmark` in the repository's `benchmarks` module compares this codec with avro4s and zio-schema-avro.

---

## Evolve Your Data: Change Your Types with Into, As and Migration

Welcome! This tutorial is for Scala developers who have finished [Your First Schema](./your-first-schema.md), or who already know how to write a case class, derive a `Schema` for it and read an `Either`. You don't need any prior experience with data conversion libraries or schema migrations to follow along.

## 1. Introduction

By the end of this tutorial, you will understand:

- How `Into` converts one type into another and validates the values on the way
- Why `Into` reports every bad field at once instead of stopping at the first one
- How `As` converts between two versions of a type in both directions
- How a `Migration` describes a change such as a field rename, and how the compiler checks that it is complete
- When to reach for each of the three tools

We learn these concepts by evolving a small book catalog, one step at a time:

1. Convert incoming data with `Into`
2. Collect every error at once
3. Convert both ways with `As`
4. Rename a field with `Migration`
5. Let the compiler check a migration
6. Run a migration backwards and look inside it

We recommend reading from top to bottom — each section builds on the previous one.

To follow along, add the schema module to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56"
scalacOptions += "-experimental"
```

The code in this tutorial is compiled with Scala 3. For scala-cli, Scala.js, or cross-building, see [How to Add ZIO Blocks to Your Build](../getting-started/installation.md).

## 2. Background: Data Outlives Its Types

A case class describes the shape of your data today. The data itself lives much longer: it arrives from other services in their own shapes, it sits in databases and message queues, and it is read by older and newer versions of your own program at the same time. Every time a shape changes, something has to turn the old shape into the new one.

ZIO Blocks gives you three tools for that job, and each one adds a guarantee to the one before it. `Into[A, B]` converts an `A` into a `B` in one direction and checks every value on the way. `As[A, B]` does the same in both directions, so two versions of a type can be converted back and forth. `Migration[A, B]` describes a change between two versions as data — rename this field, add that one — and the compiler refuses to build a migration that forgets a field. In this tutorial we use all three on the same book catalog.

## 3. Convert Incoming Data with Into

Our catalog receives books from a partner's feed. The feed is written in JavaScript, so every number arrives as a `Double`. Our own `BookV1` type uses `Int` for the publication year and the edition number. An `Into[BookDto, BookV1]` is a value that converts one into the other.

Let's define both types and derive the conversion:

```scala

case class BookDto(title: String, author: String, year: Double, edition: Double)

case class BookV1(title: String, author: String, year: Int, edition: Int)

val dtoToBook: Into[BookDto, BookV1] = Into.derived[BookDto, BookV1]
```

The code above:

- `BookDto` is the shape the feed sends, with `Double` numbers.
- `BookV1` is our domain type, with `Int` numbers.
- `Into.derived[BookDto, BookV1]` generates the conversion at compile time. It matches the fields by name and, for `year` and `edition`, uses the built-in `Into[Double, Int]` conversion that ZIO Blocks provides.

Converting can fail, because not every `Double` is a whole number. That's why `Into#into` returns an `Either[SchemaError, BookV1]`. Let's convert a book from the feed:

```scala
val zionomiconDto = BookDto("Zionomicon", "John A. De Goes", 2024.0, 1.0)
// zionomiconDto: BookDto = BookDto(
//   title = "Zionomicon",
//   author = "John A. De Goes",
//   year = 2024.0,
//   edition = 1.0
// )

println(dtoToBook.into(zionomiconDto))
// Right(BookV1(Zionomicon,John A. De Goes,2024,1))
```

The code above:

- `BookDto("Zionomicon", ...)` builds a value in the feed's shape.
- `dtoToBook.into(zionomiconDto)` converts it and returns a `Right` holding a `BookV1`.

Notice that `2024.0` became the `Int` `2024`. We never wrote a conversion function: the field names told `Into.derived` which fields belong together, and the field types told it which conversion to use.

## 4. Collect Every Error at Once

A feed can also send values that make no sense for our type. Let's send a book whose year and edition are not whole numbers:

```scala
val brokenDto = BookDto("Programming in Scala", "Martin Odersky", 2008.5, 1.5)
// brokenDto: BookDto = BookDto(
//   title = "Programming in Scala",
//   author = "Martin Odersky",
//   year = 2008.5,
//   edition = 1.5
// )

dtoToBook.into(brokenDto) match {
  case Left(error) => println(error.message)
  case Right(book) => println(s"Converted: $book")
}
// converting field BookDto.edition to BookV1.edition failed
//   Caused by: Value 1.5 cannot be precisely converted to Int
// converting field BookDto.year to BookV1.year failed
//   Caused by: Value 2008.5 cannot be precisely converted to Int
```

The code above:

- `BookDto("Programming in Scala", ...)` carries two bad values: `2008.5` and `1.5`.
- `dtoToBook.into(brokenDto)` returns a `Left` holding a `SchemaError`.
- `case Left(error) => println(error.message)` prints the error as readable text.

Notice that the message lists **both** problems. `Into` doesn't stop at the first bad field; it converts every field, collects every failure and returns them together in one `SchemaError`. Each entry names the source field, the target field and the reason, so whoever runs the feed can fix all of the bad values in one go instead of discovering them one by one.

:::tip
Try changing `2008.5` back to `2008.0` and running the example again. The message then lists only the `edition` problem.
:::

## 5. Convert Both Ways with As

Our catalog grows: the next version of the service also stores the book's ISBN. Not every book has a known ISBN, so the new field is an `Option`:

```scala
case class BookV2(title: String, author: String, year: Int, edition: Int, isbn: Option[String])
```

During a rolling deploy, some servers run the old code and some run the new code, and they exchange books. The new servers need to read `BookV1` values, and the old servers need to read `BookV2` values. An `As[BookV1, BookV2]` is an `Into` that also knows the way back:

```scala
val v1AndV2: As[BookV1, BookV2] = As.derived[BookV1, BookV2]

val v2ToV1: As[BookV2, BookV1] = v1AndV2.reverse
```

The code above:

- `As.derived[BookV1, BookV2]` derives both directions at compile time: `As#into` turns a `BookV1` into a `BookV2`, and `As#from` turns a `BookV2` back into a `BookV1`.
- `v1AndV2.reverse` is the same pair of conversions with the directions swapped, so `v2ToV1.into` goes from `BookV2` to `BookV1`.

Let's take a `BookV1` forward to `BookV2`:

```scala
val programmingInScalaV1 = BookV1("Programming in Scala", "Martin Odersky", 2008, 1)
// programmingInScalaV1: BookV1 = BookV1(
//   title = "Programming in Scala",
//   author = "Martin Odersky",
//   year = 2008,
//   edition = 1
// )

println(v1AndV2.into(programmingInScalaV1))
// Right(BookV2(Programming in Scala,Martin Odersky,2008,1,None))
```

The code above converts the book forward. The `BookV1` has no ISBN, so `isbn` becomes `None`. Now let's take a book that a new server already filled in, with its ISBN, and send it back to an old server:

```scala
val programmingInScalaV2 = BookV2("Programming in Scala", "Martin Odersky", 2008, 1, Some("978-0981531601"))
// programmingInScalaV2: BookV2 = BookV2(
//   title = "Programming in Scala",
//   author = "Martin Odersky",
//   year = 2008,
//   edition = 1,
//   isbn = Some("978-0981531601")
// )

println(v2ToV1.into(programmingInScalaV2))
// Right(BookV1(Programming in Scala,Martin Odersky,2008,1))
```

The code above converts the `BookV2` back into a `BookV1`. The old shape has nowhere to keep the ISBN, so the result leaves it out. Finally, let's check the round trip that a rolling deploy relies on — forward to `BookV2` and back again:

```scala
println(v1AndV2.into(programmingInScalaV1).flatMap(v1AndV2.from))
// Right(BookV1(Programming in Scala,Martin Odersky,2008,1))
```

The code above chains `into` and `from` with `flatMap`, so the result is `Right` only if both steps succeed. Notice that we get back exactly the `BookV1` we started with.

:::note
`As` checks at compile time that every field can travel in both directions. A field that exists only in the newer type must be an `Option`, so the conversion back has nothing to invent. `As` pairs fields by name, so a renamed field looks like one field removed and a new one added — and when the new one is not an `Option`, such as an `Int`, `As.derived` refuses to compile.
:::

## 6. Rename a Field with Migration

The catalog's next version renames `year` to `publishedYear`, because the team wants to add other years later. As we just saw, `As` pairs fields by name, so a rename is exactly the change it can't describe. This is a job for `Migration`.

A `Migration[A, B]` is built step by step from a list of changes. To build one, the library needs a `Schema` for both versions, because it applies the changes to a generic view of the data:

```scala

case class BookV3(title: String, author: String, publishedYear: Int, edition: Int, isbn: Option[String])

implicit val bookV2Schema: Schema[BookV2] = Schema.derived
implicit val bookV3Schema: Schema[BookV3] = Schema.derived

val v2ToV3: Migration[BookV2, BookV3] =
  Migration
    .newBuilder[BookV2, BookV3]
    .renameField(_.year, _.publishedYear)
    .build
```

The code above:

- `import zio.blocks.schema.migration.Migration` brings `Migration` into scope; it lives in its own package.
- `BookV3` is the new shape: the same fields as `BookV2`, with `year` renamed to `publishedYear`.
- The two `implicit val`s derive the schemas that `Migration.newBuilder` asks for.
- `Migration.newBuilder[BookV2, BookV3]` starts an empty builder.
- `.renameField(_.year, _.publishedYear)` records one change. The two arguments are selectors: `_.year` points at the field in `BookV2` and `_.publishedYear` at the field in `BookV3`, so a typo in either name is a compile error.
- `.build` turns the recorded changes into a `Migration[BookV2, BookV3]`. Fields that keep their name and type, such as `title` and `isbn`, are carried over without a step of their own.

Like `Into` and `As`, `Migration#apply` returns an `Either[SchemaError, BookV3]`. Let's migrate a stored book:

```scala
val fpInScalaV2 = BookV2("Functional Programming in Scala", "Paul Chiusano", 2014, 1, Some("978-1617290657"))
// fpInScalaV2: BookV2 = BookV2(
//   title = "Functional Programming in Scala",
//   author = "Paul Chiusano",
//   year = 2014,
//   edition = 1,
//   isbn = Some("978-1617290657")
// )

println(v2ToV3(fpInScalaV2))
// Right(BookV3(Functional Programming in Scala,Paul Chiusano,2014,1,Some(978-1617290657)))
```

The code above migrates a `BookV2` to a `BookV3`. Notice that the value of `year` now sits in `publishedYear`, and every other field kept its value.

## 7. Let the Compiler Check a Migration

A migration is only useful if it accounts for every field. Let's forget the rename and try to build the same migration with no steps at all:

```scala
Migration.newBuilder[BookV2, BookV3].build
// error: 
// Migration incomplete:
// 
//   Target fields not provided: publishedYear
//   Use addField, renameField, transformField, or migrateField to provide these fields.
// 
//   Source fields not handled: year
//   Use dropField, renameField, transformField, or migrateField to handle these fields.
//
```

The code above does not compile, and the output shows the compiler error. `build` compares the fields of `BookV2` with the fields of `BookV3`. Notice that the message names both sides of the forgotten rename: `publishedYear` is a target field that no step provides, and `year` is a source field that no step handles. The message also lists the builder methods that can fix each side.

This check runs every time the code compiles. When someone later adds a field to `BookV3` and forgets to update the migration, the build fails before any stored data is touched.

## 8. Run a Migration Backwards and Look Inside It

A migration is a description of changes, not a function hidden inside a closure. That gives us two things for free. The first is `Migration#reverse`, which undoes each step in the opposite order:

```scala
val v3ToV2: Migration[BookV3, BookV2] = v2ToV3.reverse

val fpInScalaV3 = BookV3("Functional Programming in Scala", "Paul Chiusano", 2014, 1, Some("978-1617290657"))
```

The code above:

- `v2ToV3.reverse` builds a `Migration[BookV3, BookV2]` that renames `publishedYear` back to `year`.
- `fpInScalaV3` is the book we migrated in section 6, now in its `BookV3` shape.

Let's apply the reversed migration:

```scala
println(v3ToV2(fpInScalaV3))
// Right(BookV2(Functional Programming in Scala,Paul Chiusano,2014,1,Some(978-1617290657)))
```

The code above returns a `Right` holding the original `BookV2`, with the publication year back in `year`.

The second is that we can print the steps themselves with `Migration#actions`:

```scala
println(v2ToV3.actions)
// Chunk(RenameField(.year,publishedYear))
```

The code above prints the actions inside `v2ToV3`. Notice that there is one `RenameField` action holding the path of the field and its new name — plain data, with no Scala function inside. Underneath every `Migration` sits this untyped `DynamicMigration`, which is why a migration can be inspected, reversed and combined with another migration.

## 9. Putting It Together

This example combines all the concepts we've learned — validating feed data with `Into`, collecting several errors at once, converting between two versions with `As`, renaming a field with `Migration` and running the migration backwards:

```scala title="schema-examples/src/main/scala/evolveyourdata/CompleteExample.scala"
package evolveyourdata

/**
 * Evolve Your Data: Change Your Types with Into, As and Migration — Complete
 * Example
 *
 * Validates feed data with `Into`, collects several errors at once, converts
 * between two versions with `As`, renames a field with `Migration` and runs the
 * migration backwards.
 *
 * Run with: cd schema-examples && sbt "runMain evolveyourdata.CompleteExample"
 */
object CompleteExample extends App {

  case class BookDto(title: String, author: String, year: Double, edition: Double)

  case class BookV1(title: String, author: String, year: Int, edition: Int)

  case class BookV2(title: String, author: String, year: Int, edition: Int, isbn: Option[String])

  case class BookV3(title: String, author: String, publishedYear: Int, edition: Int, isbn: Option[String])

  // Convert incoming data with Into
  val dtoToBook: Into[BookDto, BookV1] = Into.derived[BookDto, BookV1]

  val zionomiconDto = BookDto("Zionomicon", "John A. De Goes", 2024.0, 1.0)

  println(dtoToBook.into(zionomiconDto))

  // Collect every error at once
  val brokenDto = BookDto("Programming in Scala", "Martin Odersky", 2008.5, 1.5)

  dtoToBook.into(brokenDto) match {
    case Left(error) => println(error.message)
    case Right(book) => println(s"Converted: $book")
  }

  // Convert both ways with As
  val v1AndV2: As[BookV1, BookV2] = As.derived[BookV1, BookV2]

  val v2ToV1: As[BookV2, BookV1] = v1AndV2.reverse

  val programmingInScalaV1 = BookV1("Programming in Scala", "Martin Odersky", 2008, 1)

  println(v1AndV2.into(programmingInScalaV1))

  val programmingInScalaV2 = BookV2("Programming in Scala", "Martin Odersky", 2008, 1, Some("978-0981531601"))

  println(v2ToV1.into(programmingInScalaV2))

  println(v1AndV2.into(programmingInScalaV1).flatMap(v1AndV2.from))

  // Rename a field with Migration
  implicit val bookV2Schema: Schema[BookV2] = Schema.derived
  implicit val bookV3Schema: Schema[BookV3] = Schema.derived

  val v2ToV3: Migration[BookV2, BookV3] =
    Migration
      .newBuilder[BookV2, BookV3]
      .renameField(_.year, _.publishedYear)
      .build

  val fpInScalaV2 = BookV2("Functional Programming in Scala", "Paul Chiusano", 2014, 1, Some("978-1617290657"))

  println(v2ToV3(fpInScalaV2))

  // Run the migration backwards and look inside it
  val v3ToV2: Migration[BookV3, BookV2] = v2ToV3.reverse

  val fpInScalaV3 = BookV3("Functional Programming in Scala", "Paul Chiusano", 2014, 1, Some("978-1617290657"))

  println(v3ToV2(fpInScalaV3))

  println(v2ToV3.actions)
}
```

## 10. Running the Examples

All examples in this tutorial have corresponding runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

Run them in order to follow the tutorial step by step.

### Convert Incoming Data with Into

This example derives an `Into[BookDto, BookV1]` and converts a book from the feed.

<details>
  <summary>schema-examples/src/main/scala/evolveyourdata/Concept1IntoExample.scala</summary>

```scala title="schema-examples/src/main/scala/evolveyourdata/Concept1IntoExample.scala" showLineNumbers
package evolveyourdata

/**
 * Evolve Your Data: Convert Incoming Data with Into
 *
 * Derives an `Into[BookDto, BookV1]` and converts a book from the feed.
 *
 * Run with: cd schema-examples && sbt "runMain
 * evolveyourdata.Concept1IntoExample"
 */
object Concept1IntoExample extends App {

  case class BookDto(title: String, author: String, year: Double, edition: Double)

  case class BookV1(title: String, author: String, year: Int, edition: Int)

  val dtoToBook: Into[BookDto, BookV1] = Into.derived[BookDto, BookV1]

  val zionomiconDto = BookDto("Zionomicon", "John A. De Goes", 2024.0, 1.0)

  println(dtoToBook.into(zionomiconDto))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain evolveyourdata.Concept1IntoExample"
```

### Collect Every Error at Once

This example converts a feed record with two bad fields and prints the single `SchemaError` that lists both.

<details>
  <summary>schema-examples/src/main/scala/evolveyourdata/Concept2ErrorAccumulationExample.scala</summary>

```scala title="schema-examples/src/main/scala/evolveyourdata/Concept2ErrorAccumulationExample.scala" showLineNumbers
package evolveyourdata

/**
 * Evolve Your Data: Collect Every Error at Once
 *
 * Converts a feed record with two bad fields and prints the single
 * `SchemaError` that lists both.
 *
 * Run with: cd schema-examples && sbt "runMain
 * evolveyourdata.Concept2ErrorAccumulationExample"
 */
object Concept2ErrorAccumulationExample extends App {

  case class BookDto(title: String, author: String, year: Double, edition: Double)

  case class BookV1(title: String, author: String, year: Int, edition: Int)

  val dtoToBook: Into[BookDto, BookV1] = Into.derived[BookDto, BookV1]

  val brokenDto = BookDto("Programming in Scala", "Martin Odersky", 2008.5, 1.5)

  dtoToBook.into(brokenDto) match {
    case Left(error) => println(error.message)
    case Right(book) => println(s"Converted: $book")
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain evolveyourdata.Concept2ErrorAccumulationExample"
```

### Convert Both Ways with As

This example derives an `As[BookV1, BookV2]`, converts in both directions and checks the round trip.

<details>
  <summary>schema-examples/src/main/scala/evolveyourdata/Concept3AsExample.scala</summary>

```scala title="schema-examples/src/main/scala/evolveyourdata/Concept3AsExample.scala" showLineNumbers
package evolveyourdata

/**
 * Evolve Your Data: Convert Both Ways with As
 *
 * Derives an `As[BookV1, BookV2]`, converts in both directions and checks the
 * round trip.
 *
 * Run with: cd schema-examples && sbt "runMain
 * evolveyourdata.Concept3AsExample"
 */
object Concept3AsExample extends App {

  case class BookV1(title: String, author: String, year: Int, edition: Int)

  case class BookV2(title: String, author: String, year: Int, edition: Int, isbn: Option[String])

  val v1AndV2: As[BookV1, BookV2] = As.derived[BookV1, BookV2]

  val v2ToV1: As[BookV2, BookV1] = v1AndV2.reverse

  val programmingInScalaV1 = BookV1("Programming in Scala", "Martin Odersky", 2008, 1)

  println(v1AndV2.into(programmingInScalaV1))

  val programmingInScalaV2 = BookV2("Programming in Scala", "Martin Odersky", 2008, 1, Some("978-0981531601"))

  println(v2ToV1.into(programmingInScalaV2))

  println(v1AndV2.into(programmingInScalaV1).flatMap(v1AndV2.from))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain evolveyourdata.Concept3AsExample"
```

### Rename a Field with Migration

This example builds a `Migration[BookV2, BookV3]` that renames `year` to `publishedYear` and applies it.

<details>
  <summary>schema-examples/src/main/scala/evolveyourdata/Concept4MigrationExample.scala</summary>

```scala title="schema-examples/src/main/scala/evolveyourdata/Concept4MigrationExample.scala" showLineNumbers
package evolveyourdata

/**
 * Evolve Your Data: Rename a Field with Migration
 *
 * Builds a `Migration[BookV2, BookV3]` that renames `year` to `publishedYear`
 * and applies it.
 *
 * Run with: cd schema-examples && sbt "runMain
 * evolveyourdata.Concept4MigrationExample"
 */
object Concept4MigrationExample extends App {

  case class BookV2(title: String, author: String, year: Int, edition: Int, isbn: Option[String])

  case class BookV3(title: String, author: String, publishedYear: Int, edition: Int, isbn: Option[String])

  implicit val bookV2Schema: Schema[BookV2] = Schema.derived
  implicit val bookV3Schema: Schema[BookV3] = Schema.derived

  val v2ToV3: Migration[BookV2, BookV3] =
    Migration
      .newBuilder[BookV2, BookV3]
      .renameField(_.year, _.publishedYear)
      .build

  val fpInScalaV2 = BookV2("Functional Programming in Scala", "Paul Chiusano", 2014, 1, Some("978-1617290657"))

  println(v2ToV3(fpInScalaV2))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain evolveyourdata.Concept4MigrationExample"
```

The compile-time check from "Let the Compiler Check a Migration" has no runnable example, because code that fails to compile can't run. To see it, remove the `.renameField(...)` line from this file and run `sbt compile`.

### Run a Migration Backwards and Look Inside It

This example reverses the migration, applies it to a `BookV3` and prints the migration's actions.

<details>
  <summary>schema-examples/src/main/scala/evolveyourdata/Concept5ReverseMigrationExample.scala</summary>

```scala title="schema-examples/src/main/scala/evolveyourdata/Concept5ReverseMigrationExample.scala" showLineNumbers
package evolveyourdata

/**
 * Evolve Your Data: Run a Migration Backwards and Look Inside It
 *
 * Reverses the migration, applies it to a `BookV3` and prints the migration's
 * actions.
 *
 * Run with: cd schema-examples && sbt "runMain
 * evolveyourdata.Concept5ReverseMigrationExample"
 */
object Concept5ReverseMigrationExample extends App {

  case class BookV2(title: String, author: String, year: Int, edition: Int, isbn: Option[String])

  case class BookV3(title: String, author: String, publishedYear: Int, edition: Int, isbn: Option[String])

  implicit val bookV2Schema: Schema[BookV2] = Schema.derived
  implicit val bookV3Schema: Schema[BookV3] = Schema.derived

  val v2ToV3: Migration[BookV2, BookV3] =
    Migration
      .newBuilder[BookV2, BookV3]
      .renameField(_.year, _.publishedYear)
      .build

  val v3ToV2: Migration[BookV3, BookV2] = v2ToV3.reverse

  val fpInScalaV3 = BookV3("Functional Programming in Scala", "Paul Chiusano", 2014, 1, Some("978-1617290657"))

  println(v3ToV2(fpInScalaV3))

  println(v2ToV3.actions)
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain evolveyourdata.Concept5ReverseMigrationExample"
```

### The Complete Example

The complete example from "Putting It Together" runs everything from the tutorial in one program:

```bash
sbt "runMain evolveyourdata.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## 11. What You've Learned

In this tutorial, you learned:

- How `Into.derived` converts one type into another, pairing fields by name and validating values such as a `Double` that must become an `Int`
- How `Into` collects every failing field into one `SchemaError` instead of stopping at the first
- How `As.derived` gives you both directions of a conversion, and why fields that exist only in the newer type must be `Option`s
- How `Migration.newBuilder` records changes such as `renameField`, and how `build` refuses to compile a migration that misses a field
- How a migration can be reversed and inspected, because it is data

When your data changes shape, pick the tool by the guarantee you need. Reach for `Into` when data flows one way and must be validated, such as input from another service. Reach for `As` when two versions must be converted back and forth and every change is name-preserving. Reach for `Migration` when you evolve stored data between versions and want the compiler to check that every field is accounted for.

## 12. Where to Go Next

- **Want every conversion rule?** Read the [Into](../reference/schema/schema-evolution/into.md) and [As](../reference/schema/schema-evolution/as.md) reference pages, which cover numeric widening and narrowing, collections, `Option`, custom conversions and more. The [Schema Evolution](../reference/schema/schema-evolution/index.md) overview shows how the two fit together.
- **Need more migration steps?** The [Migration](../reference/schema/migration.md) reference page documents every builder method, including `addField`, `dropField`, `transformField` and `renameCase`, and how migrations are composed.
- **Want to write default values and transformations?** Read [SchemaExpr](../reference/schema/schema-expr.md), the expression type that builder methods such as `addField` take.
- **Want to understand errors in depth?** Read [SchemaError](../reference/schema/schema-error.md) for every error case and how errors are combined.
- **More runnable examples:** the `schema-examples` project has more `Into` examples in [`into/`](https://github.com/zio/zio-blocks/tree/main/schema-examples/src/main/scala/into) and `As` examples in [`as/`](https://github.com/zio/zio-blocks/tree/main/schema-examples/src/main/scala/as).
- **Need to migrate stored rows while the application keeps running?** Follow [How to Migrate Table Rows to a New Version Online](./migrate-table-rows-online.md), which runs a `Migration` like the ones in this tutorial over every row of a live SQL table.
- **Moving from ZIO Schema?** That's a different kind of migration: see [Migrating from ZIO Schema to ZIO Blocks Schema](./zio-schema-migration.md).

---

## How to Feed Configuration into a Wire Dependency Graph

In this guide, we build a small bookstore backend from three services, `Db`, `InventoryRepo` and `Catalog`, and let each one receive its own typed settings through the dependency graph. By the end, you will have one `ConfigSource` that layers environment variables over a YAML file over defaults, a graph that hands each service the settings section it asks for, a shutdown that closes the database last, a startup that stops with a full report when a setting is wrong, and tests that need no files or environment variables.

We read one settings section first, then build the whole graph, look at the shutdown order, break a setting on purpose, and finish with tests:

- [Reading One Settings Section](#reading-one-settings-section)
- [Building the Whole Graph](#building-the-whole-graph)
- [Closing Services in Reverse Order](#closing-services-in-reverse-order)
- [Stopping at Startup on a Bad Setting](#stopping-at-startup-on-a-bad-setting)
- [Testing Without Files or Environment Variables](#testing-without-files-or-environment-variables)

## The Problem

A service graph needs every service built after its dependencies, with the right settings, and closed in the opposite order. Writing that by hand mixes three jobs in `main`: reading raw strings, constructing objects in the right order, and closing them in reverse.

Without a configuration library or a dependency-injection library, the bookstore's `main` usually looks like this:

```scala
final case class DbSettings(url: String, user: String, password: String, poolSize: Int)
final case class CatalogSettings(featuredTitle: String, pageSize: Int)

class Db(settings: DbSettings) extends AutoCloseable {
  def close(): Unit = ()
}
class InventoryRepo(db: Db)
class Catalog(repo: InventoryRepo, settings: CatalogSettings) extends AutoCloseable {
  def close(): Unit = ()
}

object Main {
  private def setting(name: String, default: => String): String =
    sys.env.getOrElse(name, default)

  def main(args: Array[String]): Unit = {
    val db = new Db(
      DbSettings(
        setting("DB_URL", "jdbc:postgresql://localhost:5432/bookstore"),
        setting("DB_USER", "bookstore"),
        setting("DB_PASSWORD", sys.error("DB_PASSWORD is not set")),
        setting("DB_POOL_SIZE", "10").toInt
      )
    )
    try {
      val repo    = new InventoryRepo(db)
      val catalog = new Catalog(
        repo,
        CatalogSettings(
          setting("CATALOG_FEATURED_TITLE", "Zionomicon"),
          setting("CATALOG_PAGE_SIZE", "3").toInt
        )
      )
      try println("serving") // the application runs here
      finally catalog.close()
    } finally db.close()
  }
}
```

Each new service adds another hand-placed `try`/`finally`, and moving a constructor call means rechecking the closing order. A `CATALOG_PAGE_SIZE=ten` throws `NumberFormatException: For input string: "ten"`, which names neither the key nor the source, and it throws only after the database is already open, so you find out about the next bad setting on the next restart. Testing `Catalog` means exporting environment variables.

In this guide, the config module turns each settings section into a typed case class with every error reported at once, and the scope module builds the graph from the constructors and closes it in reverse order.

## Prerequisites

Add the config module and its YAML format to your `build.sbt`. The config module depends on the scope and context modules, so `Scope`, `Resource`, `Wire` and `Context` come with it:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-config"      % "0.0.56",
  "dev.zio" %% "zio-blocks-config-yaml" % "0.0.56"
)

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-config"      % "0.0.56",
  "dev.zio" %% "zio-blocks-next-config-yaml" % "0.0.56"
)
```

This guide assumes you have loaded a case class from a layered `ConfigSource`, as in [Build a Typed Configuration Loader](./typed-config-loader.md), and that you have built a `Resource` and used it inside `Scope.global.scoped`, as in [Compile-Time Resource Safety with Scope](./compile-time-resource-safety-with-scope.md). The snippets below share these imports:

```scala

```

## The Bookstore's Services and Their Settings

The bookstore has a database pool, a repository on top of it, and a catalog that renders the front page. `Db` and `Catalog` each take a settings case class as a constructor parameter, and each settings class derives a `Schema`, which is all the config module needs to decode it:

```scala
final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
object DbConfig {
  implicit val schema: Schema[DbConfig] = Schema.derived
}

final case class CatalogConfig(featuredTitle: String, pageSize: Int)
object CatalogConfig {
  implicit val schema: Schema[CatalogConfig] = Schema.derived
}

class Db(config: DbConfig) extends AutoCloseable {
  println(s"Db: opening ${config.poolSize} connections to ${config.url} as ${config.user}, password ${config.password}")

  def titles(): List[String] =
    List("Zionomicon", "Programming in Scala", "Functional Programming in Scala")

  def close(): Unit = println("Db: closing the connection pool")
}

class InventoryRepo(db: Db) {
  println("InventoryRepo: ready")

  def inStock(): List[String] = db.titles()
}

class Catalog(repo: InventoryRepo, config: CatalogConfig) extends AutoCloseable {
  println(s"Catalog: featuring ${config.featuredTitle}")

  def frontPage(): String =
    s"Featured: ${config.featuredTitle} | In stock: ${repo.inStock().take(config.pageSize).mkString(", ")}"

  def close(): Unit = println("Catalog: flushing the page cache")
}
```

The services print when they start and stop, so the output below shows the order. The password is a `Secret`, so printing it shows `<secret>` instead of the value.

:::tip[One settings type per service]
The graph finds a constructor's arguments by type. Give each service its own settings case class instead of loose `String` or `Int` parameters: two `String` parameters for a URL and a user name can't be told apart by type. [How to Fix Common Scope and Wire Compile Errors](./fix-scope-and-wire-compile-errors.md) shows what the macro reports for such a constructor.
:::

## Reading One Settings Section

`Config.wire[A](prefix)` turns one settings section into a wire, the scope module's recipe for building a value. Its type, `Wire.Shared[ConfigSource, DbConfig]`, says that it needs a `ConfigSource` and produces a `DbConfig`; `Shared` means the graph builds it once and hands the same value to every service that asks for it. The prefix `"db"` makes the wire read `db.url`, `db.user` and so on. A wire is only a recipe, so we run it by hand with `Wire.Shared#make`, passing a scope and a `Context` that holds the source. The source holds the `db` section:

```scala
val sectionSource = ConfigSource.fromMap(
  Map(
    "db.url"      -> "jdbc:postgresql://localhost:5432/bookstore",
    "db.user"     -> "bookstore",
    "db.password" -> "change-me",
    "db.poolSize" -> "10"
  ),
  "sections"
)
```

Running the wire prints the decoded section, with the password hidden:

```scala
Scope.global.scoped { scope =>
  val dbConfig = Config.wire[DbConfig]("db").make(scope, Context[ConfigSource](sectionSource))
  println(dbConfig)
}
// DbConfig(jdbc:postgresql://localhost:5432/bookstore,bookstore,<secret>,10)
```

`Config.wire[A]` without a prefix reads the fields from the top level of the source. Inside a graph, you never call `Wire.Shared#make` yourself: `Resource.from` does it, as the next section shows.

## Building the Whole Graph

A real service reads its settings from several places, so we layer them as [Build a Typed Configuration Loader](./typed-config-loader.md) does: environment variables first, then a YAML file, then defaults. In production, the environment layer is `EnvSource`, keyed to the `BOOKSTORE_` prefix and converted to upper snake case, so `catalog.featuredTitle` is read from `BOOKSTORE_CATALOG_FEATURED_TITLE`:

```scala
val productionEnv: ConfigSource = EnvSource.keyFormat(KeyFormat.UpperSnakeCase).prefix("bookstore")
```

This page must give the same output on every machine, so it stands in a map for the environment, holding one exported variable, `BOOKSTORE_DB_PASSWORD`. The YAML file and the defaults supply the rest:

```scala
val env = ConfigSource.fromMap(Map("db.password" -> "change-me"), "env")

val bookstoreYaml =
  """db:
    |  url: jdbc:postgresql://localhost:5432/bookstore
    |  user: bookstore
    |catalog:
    |  featuredTitle: Zionomicon
    |""".stripMargin

val yaml = ConfigSource.fromYaml(bookstoreYaml, "bookstore.yaml") match {
  case Right(source) => source
  case Left(error)   => throw error
}

val defaults = ConfigSource.fromMap(Map("db.poolSize" -> "10", "catalog.pageSize" -> "3"), "defaults")

val settings = env.orElse(yaml).orElse(defaults)
```

`Resource.from[Catalog]` builds the whole graph from `Catalog`'s constructor. The wires passed to it supply what the constructors can't: `Wire(source)` hands the graph the `ConfigSource` value, and the two `Config.wire` calls turn it into `DbConfig` and `CatalogConfig`. The macro derives a wire for `Db` and `InventoryRepo` from their constructors:

```scala
def catalogResource(source: ConfigSource): Resource[Catalog] =
  Resource.from[Catalog](
    Wire(source),
    Config.wire[DbConfig]("db"),
    Config.wire[CatalogConfig]("catalog")
  )
```

Allocating the resource in a scope builds every service, and `$` calls the catalog inside the scope:

```scala
val frontPage = Scope.global.scoped { scope =>

  val catalog = allocate(catalogResource(settings))
  $(catalog)(_.frontPage())
}
// Db: opening 10 connections to jdbc:postgresql://localhost:5432/bookstore as bookstore, password <secret>
// InventoryRepo: ready
// Catalog: featuring Zionomicon
// Catalog: flushing the page cache
// Db: closing the connection pool
// frontPage: String = "Featured: Zionomicon | In stock: Zionomicon, Programming in Scala, Functional Programming in Scala"

println(frontPage)
// Featured: Zionomicon | In stock: Zionomicon, Programming in Scala, Functional Programming in Scala
```

The password came from the environment stand-in, the URL, user and featured title from YAML, and the pool and page sizes from the defaults. Neither `Db` nor `Catalog` knows where its settings came from.

:::warning[Make the graph a `def`, not a `val`]
A `Resource` built by `Resource.from` is shared: the first allocation builds the services, and closing the last scope that uses them closes them for good. A `val` holding it works once, and a second `Scope.global.scoped` that allocates the same `val` throws `IllegalStateException: Cannot allocate from a destroyed shared resource`. A `def`, as above, builds a fresh resource for each scope.
:::

## Closing Services in Reverse Order

The scope closes what it opened when the `scoped` block ends, in the reverse order of opening. The graph opens `Db` first because everything else depends on it, so `Db` closes last, after `Catalog` has flushed its cache. The output above shows the order; here it is again with the work in the middle:

```scala
Scope.global.scoped { scope =>

  val catalog = allocate(catalogResource(settings))
  println("Serving: " + $(catalog)(_.frontPage()))
}
// Db: opening 10 connections to jdbc:postgresql://localhost:5432/bookstore as bookstore, password <secret>
// InventoryRepo: ready
// Catalog: featuring Zionomicon
// Serving: Featured: Zionomicon | In stock: Zionomicon, Programming in Scala, Functional Programming in Scala
// Catalog: flushing the page cache
// Db: closing the connection pool
```

You write no `close()` calls. The macro registers `close()` for every class that extends `AutoCloseable` and is built from its constructor, as `Db` and `Catalog` are here. `InventoryRepo` holds no resources of its own, so it has nothing to close.

:::warning[Let the macro build anything you need to close]
Build `Db` from its constructor, as above, not with `Wire.Shared.fromFunction`. That function runs your code and registers nothing, so a `Db` built there is never closed. `Config.wire` is built that way, which is fine because a settings case class holds no resources. Also keep `Db` and `Catalog` inside `scoped`: return plain data such as the front-page `String`, never the service, which is closed once the block ends.
:::

## Stopping at Startup on a Bad Setting

`Config.wire` decodes its section while the graph is built, and when anything is missing or malformed it throws `ConfigLoadException` with every error in that section. Here the catalog section has a page size of `two` and no featured title, layered over `sectionSource`, the complete `db` section from [Reading One Settings Section](#reading-one-settings-section):

```scala
val badCatalog = ConfigSource.fromMap(Map("catalog.pageSize" -> "two"), "bad-catalog").orElse(sectionSource)
```

The scope opens `Db` and `InventoryRepo`, stops at `CatalogConfig`, and closes what it had opened before the exception leaves `scoped`:

```scala
try {
  Scope.global.scoped { scope =>

    val catalog = allocate(catalogResource(badCatalog))
    $(catalog)(_.frontPage())
  }
} catch {
  case e: ConfigLoadException => println(e.getMessage)
}
// Db: opening 10 connections to jdbc:postgresql://localhost:5432/bookstore as bookstore, password <secret>
// InventoryRepo: ready
// Db: closing the connection pool
// Configuration loading failed with 1 error(s):
//   - Missing required key 'featuredTitle' in source 'bad-catalog|sections'
// Invalid value 'two' for key 'pageSize' (expected Int) in source 'bad-catalog|sections': For input string: "two"
//
```

The application never serves a request with half its settings. When the bad setting is in the `db` section, the graph stops before it opens `Db`, so nothing needs closing.

Two details of the report trip readers up. The keys are relative to the section's prefix: `pageSize` means `catalog.pageSize`, and the report doesn't name the section. The count in the first line also reads `1 error(s)` even when more lines follow, because the errors of one case class are collected into a single composite error; read the lines, not the count.

## Testing Without Files or Environment Variables

The graph depends only on the `ConfigSource` passed to `catalogResource`, so a test passes a `ConfigSource.fromMap` instead of the production layers. A test usually changes one or two settings, so it puts a small map in front of the defaults and leaves the rest alone, as `frontPageWith` does:

```scala
def frontPageWith(overrides: Map[String, String]): String = {
  val testSettings = ConfigSource.fromMap(overrides, "test").orElse(settings)
  Scope.global.scoped { scope =>

    val catalog = allocate(catalogResource(testSettings))
    $(catalog)(_.frontPage())
  }
}
```

Each call builds and closes its own graph, so tests don't share services:

```scala
val page = frontPageWith(Map("catalog.featuredTitle" -> "Functional Programming in Scala", "catalog.pageSize" -> "1"))
// Db: opening 10 connections to jdbc:postgresql://localhost:5432/bookstore as bookstore, password <secret>
// InventoryRepo: ready
// Catalog: featuring Functional Programming in Scala
// Catalog: flushing the page cache
// Db: closing the connection pool
// page: String = "Featured: Functional Programming in Scala | In stock: Zionomicon"

println(page == "Featured: Functional Programming in Scala | In stock: Zionomicon")
// true
```

In a test suite, `settings` would be a map with every key, so the test reads no file and no environment variable at all.

## Putting It Together

The complete example layers the real environment, a YAML document and defaults, builds the bookstore graph, prints the front page and the shutdown order, and shows the report for a bad setting:

```scala title="config-examples/src/main/scala/feedconfigintoawiregraph/CompleteExample.scala"
package feedconfigintoawiregraph

/**
 * How to Feed Configuration into a Wire Dependency Graph — Putting It Together
 *
 * Layers the real environment over YAML over defaults, builds the graph, and
 * shows the report for a bad catalog setting.
 *
 * Run with: cd config-examples && sbt "runMain
 * feedconfigintoawiregraph.CompleteExample"
 */
object CompleteExample {

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredTitle: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  class Db(config: DbConfig) extends AutoCloseable {
    println(
      s"Db: opening ${config.poolSize} connections to ${config.url} as ${config.user}, password ${config.password}"
    )

    def titles(): List[String] =
      List("Zionomicon", "Programming in Scala", "Functional Programming in Scala")

    def close(): Unit = println("Db: closing the connection pool")
  }

  class InventoryRepo(db: Db) {
    println("InventoryRepo: ready")

    def inStock(): List[String] = db.titles()
  }

  class Catalog(repo: InventoryRepo, config: CatalogConfig) extends AutoCloseable {
    println(s"Catalog: featuring ${config.featuredTitle}")

    def frontPage(): String =
      s"Featured: ${config.featuredTitle} | In stock: ${repo.inStock().take(config.pageSize).mkString(", ")}"

    def close(): Unit = println("Catalog: flushing the page cache")
  }

  val productionEnv: ConfigSource = EnvSource.keyFormat(KeyFormat.UpperSnakeCase).prefix("bookstore")

  val bookstoreYaml =
    """db:
      |  url: jdbc:postgresql://localhost:5432/bookstore
      |  user: bookstore
      |catalog:
      |  featuredTitle: Zionomicon
      |""".stripMargin

  val yaml = ConfigSource.fromYaml(bookstoreYaml, "bookstore.yaml") match {
    case Right(source) => source
    case Left(error)   => throw error
  }

  // The password default is for local runs only; production exports BOOKSTORE_DB_PASSWORD.
  val defaults = ConfigSource.fromMap(
    Map("db.password" -> "change-me", "db.poolSize" -> "10", "catalog.pageSize" -> "3"),
    "defaults"
  )

  val settings = productionEnv.orElse(yaml).orElse(defaults)

  def catalogResource(source: ConfigSource): Resource[Catalog] =
    Resource.from[Catalog](
      Wire(source),
      Config.wire[DbConfig]("db"),
      Config.wire[CatalogConfig]("catalog")
    )

  def main(args: Array[String]): Unit = {
    // 1. Build the graph, serve the front page, and close in reverse order.
    val frontPage = Scope.global.scoped { scope =>

      val catalog = allocate(catalogResource(settings))
      $(catalog)(_.frontPage())
    }
    println(frontPage)

    // 2. A bad catalog setting stops startup with a full report.
    val dbSection = ConfigSource.fromMap(
      Map(
        "db.url"      -> "jdbc:postgresql://localhost:5432/bookstore",
        "db.user"     -> "bookstore",
        "db.password" -> "change-me",
        "db.poolSize" -> "10"
      ),
      "db-section"
    )
    val badCatalog = ConfigSource.fromMap(Map("catalog.pageSize" -> "two"), "bad-catalog").orElse(dbSection)
    try {
      Scope.global.scoped { scope =>

        val catalog = allocate(catalogResource(badCatalog))
        $(catalog)(_.frontPage())
      }
    } catch {
      case e: ConfigLoadException => println(e.getMessage)
    }
  }
}
```

## Running the Examples

All examples in this guide have corresponding runnable Scala files in the `config-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/config-examples
```

Each file below matches one section of this guide.

### Reading One Settings Section

<details>
  <summary>config-examples/src/main/scala/feedconfigintoawiregraph/Step1ReadOneSection.scala</summary>

```scala title="config-examples/src/main/scala/feedconfigintoawiregraph/Step1ReadOneSection.scala" showLineNumbers
package feedconfigintoawiregraph

/**
 * How to Feed Configuration into a Wire Dependency Graph — Reading One Settings
 * Section
 *
 * Runs one Config.wire by hand against a map source.
 *
 * Run with: cd config-examples && sbt "runMain
 * feedconfigintoawiregraph.Step1ReadOneSection"
 */
object Step1ReadOneSection {

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  val sectionSource = ConfigSource.fromMap(
    Map(
      "db.url"      -> "jdbc:postgresql://localhost:5432/bookstore",
      "db.user"     -> "bookstore",
      "db.password" -> "change-me",
      "db.poolSize" -> "10"
    ),
    "sections"
  )

  def main(args: Array[String]): Unit =
    Scope.global.scoped { scope =>
      val dbConfig = Config.wire[DbConfig]("db").make(scope, Context[ConfigSource](sectionSource))
      println(dbConfig)
    }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain feedconfigintoawiregraph.Step1ReadOneSection"
```

### Building the Whole Graph

<details>
  <summary>config-examples/src/main/scala/feedconfigintoawiregraph/Step2BuildTheGraph.scala</summary>

```scala title="config-examples/src/main/scala/feedconfigintoawiregraph/Step2BuildTheGraph.scala" showLineNumbers
package feedconfigintoawiregraph

/**
 * How to Feed Configuration into a Wire Dependency Graph — Building the Whole
 * Graph
 *
 * Layers env over YAML over defaults and builds the bookstore graph.
 *
 * Run with: cd config-examples && sbt "runMain
 * feedconfigintoawiregraph.Step2BuildTheGraph"
 */
object Step2BuildTheGraph {

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredTitle: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  class Db(config: DbConfig) extends AutoCloseable {
    println(
      s"Db: opening ${config.poolSize} connections to ${config.url} as ${config.user}, password ${config.password}"
    )

    def titles(): List[String] =
      List("Zionomicon", "Programming in Scala", "Functional Programming in Scala")

    def close(): Unit = println("Db: closing the connection pool")
  }

  class InventoryRepo(db: Db) {
    println("InventoryRepo: ready")

    def inStock(): List[String] = db.titles()
  }

  class Catalog(repo: InventoryRepo, config: CatalogConfig) extends AutoCloseable {
    println(s"Catalog: featuring ${config.featuredTitle}")

    def frontPage(): String =
      s"Featured: ${config.featuredTitle} | In stock: ${repo.inStock().take(config.pageSize).mkString(", ")}"

    def close(): Unit = println("Catalog: flushing the page cache")
  }

  val env = ConfigSource.fromMap(Map("db.password" -> "change-me"), "env")

  val bookstoreYaml =
    """db:
      |  url: jdbc:postgresql://localhost:5432/bookstore
      |  user: bookstore
      |catalog:
      |  featuredTitle: Zionomicon
      |""".stripMargin

  val yaml = ConfigSource.fromYaml(bookstoreYaml, "bookstore.yaml") match {
    case Right(source) => source
    case Left(error)   => throw error
  }

  val defaults = ConfigSource.fromMap(Map("db.poolSize" -> "10", "catalog.pageSize" -> "3"), "defaults")

  val settings = env.orElse(yaml).orElse(defaults)

  def catalogResource(source: ConfigSource): Resource[Catalog] =
    Resource.from[Catalog](
      Wire(source),
      Config.wire[DbConfig]("db"),
      Config.wire[CatalogConfig]("catalog")
    )

  def main(args: Array[String]): Unit = {
    val frontPage = Scope.global.scoped { scope =>

      val catalog = allocate(catalogResource(settings))
      $(catalog)(_.frontPage())
    }

    println(frontPage)
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain feedconfigintoawiregraph.Step2BuildTheGraph"
```

### Closing Services in Reverse Order

<details>
  <summary>config-examples/src/main/scala/feedconfigintoawiregraph/Step3ShutdownOrder.scala</summary>

```scala title="config-examples/src/main/scala/feedconfigintoawiregraph/Step3ShutdownOrder.scala" showLineNumbers
package feedconfigintoawiregraph

/**
 * How to Feed Configuration into a Wire Dependency Graph — Closing Services in
 * Reverse Order
 *
 * Shows Catalog closing before Db when the scope ends.
 *
 * Run with: cd config-examples && sbt "runMain
 * feedconfigintoawiregraph.Step3ShutdownOrder"
 */
object Step3ShutdownOrder {

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredTitle: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  class Db(config: DbConfig) extends AutoCloseable {
    println(
      s"Db: opening ${config.poolSize} connections to ${config.url} as ${config.user}, password ${config.password}"
    )

    def titles(): List[String] =
      List("Zionomicon", "Programming in Scala", "Functional Programming in Scala")

    def close(): Unit = println("Db: closing the connection pool")
  }

  class InventoryRepo(db: Db) {
    println("InventoryRepo: ready")

    def inStock(): List[String] = db.titles()
  }

  class Catalog(repo: InventoryRepo, config: CatalogConfig) extends AutoCloseable {
    println(s"Catalog: featuring ${config.featuredTitle}")

    def frontPage(): String =
      s"Featured: ${config.featuredTitle} | In stock: ${repo.inStock().take(config.pageSize).mkString(", ")}"

    def close(): Unit = println("Catalog: flushing the page cache")
  }

  val env = ConfigSource.fromMap(Map("db.password" -> "change-me"), "env")

  val bookstoreYaml =
    """db:
      |  url: jdbc:postgresql://localhost:5432/bookstore
      |  user: bookstore
      |catalog:
      |  featuredTitle: Zionomicon
      |""".stripMargin

  val yaml = ConfigSource.fromYaml(bookstoreYaml, "bookstore.yaml") match {
    case Right(source) => source
    case Left(error)   => throw error
  }

  val defaults = ConfigSource.fromMap(Map("db.poolSize" -> "10", "catalog.pageSize" -> "3"), "defaults")

  val settings = env.orElse(yaml).orElse(defaults)

  def catalogResource(source: ConfigSource): Resource[Catalog] =
    Resource.from[Catalog](
      Wire(source),
      Config.wire[DbConfig]("db"),
      Config.wire[CatalogConfig]("catalog")
    )

  def main(args: Array[String]): Unit =
    Scope.global.scoped { scope =>

      val catalog = allocate(catalogResource(settings))
      println("Serving: " + $(catalog)(_.frontPage()))
    }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain feedconfigintoawiregraph.Step3ShutdownOrder"
```

### Stopping at Startup on a Bad Setting

<details>
  <summary>config-examples/src/main/scala/feedconfigintoawiregraph/Step4BadSetting.scala</summary>

```scala title="config-examples/src/main/scala/feedconfigintoawiregraph/Step4BadSetting.scala" showLineNumbers
package feedconfigintoawiregraph

/**
 * How to Feed Configuration into a Wire Dependency Graph — Stopping at Startup
 * on a Bad Setting
 *
 * A malformed catalog section stops the graph with a full report.
 *
 * Run with: cd config-examples && sbt "runMain
 * feedconfigintoawiregraph.Step4BadSetting"
 */
object Step4BadSetting {

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredTitle: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  class Db(config: DbConfig) extends AutoCloseable {
    println(
      s"Db: opening ${config.poolSize} connections to ${config.url} as ${config.user}, password ${config.password}"
    )

    def titles(): List[String] =
      List("Zionomicon", "Programming in Scala", "Functional Programming in Scala")

    def close(): Unit = println("Db: closing the connection pool")
  }

  class InventoryRepo(db: Db) {
    println("InventoryRepo: ready")

    def inStock(): List[String] = db.titles()
  }

  class Catalog(repo: InventoryRepo, config: CatalogConfig) extends AutoCloseable {
    println(s"Catalog: featuring ${config.featuredTitle}")

    def frontPage(): String =
      s"Featured: ${config.featuredTitle} | In stock: ${repo.inStock().take(config.pageSize).mkString(", ")}"

    def close(): Unit = println("Catalog: flushing the page cache")
  }

  val sectionSource = ConfigSource.fromMap(
    Map(
      "db.url"      -> "jdbc:postgresql://localhost:5432/bookstore",
      "db.user"     -> "bookstore",
      "db.password" -> "change-me",
      "db.poolSize" -> "10"
    ),
    "sections"
  )

  def catalogResource(source: ConfigSource): Resource[Catalog] =
    Resource.from[Catalog](
      Wire(source),
      Config.wire[DbConfig]("db"),
      Config.wire[CatalogConfig]("catalog")
    )

  val badCatalog = ConfigSource.fromMap(Map("catalog.pageSize" -> "two"), "bad-catalog").orElse(sectionSource)

  def main(args: Array[String]): Unit =
    try {
      Scope.global.scoped { scope =>

        val catalog = allocate(catalogResource(badCatalog))
        $(catalog)(_.frontPage())
      }
    } catch {
      case e: ConfigLoadException => println(e.getMessage)
    }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain feedconfigintoawiregraph.Step4BadSetting"
```

### Testing Without Files or Environment Variables

<details>
  <summary>config-examples/src/main/scala/feedconfigintoawiregraph/Step5Testing.scala</summary>

```scala title="config-examples/src/main/scala/feedconfigintoawiregraph/Step5Testing.scala" showLineNumbers
package feedconfigintoawiregraph

/**
 * How to Feed Configuration into a Wire Dependency Graph — Testing Without
 * Files or Environment Variables
 *
 * Overrides a few settings with a map and checks the front page.
 *
 * Run with: cd config-examples && sbt "runMain
 * feedconfigintoawiregraph.Step5Testing"
 */
object Step5Testing {

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredTitle: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  class Db(config: DbConfig) extends AutoCloseable {
    println(
      s"Db: opening ${config.poolSize} connections to ${config.url} as ${config.user}, password ${config.password}"
    )

    def titles(): List[String] =
      List("Zionomicon", "Programming in Scala", "Functional Programming in Scala")

    def close(): Unit = println("Db: closing the connection pool")
  }

  class InventoryRepo(db: Db) {
    println("InventoryRepo: ready")

    def inStock(): List[String] = db.titles()
  }

  class Catalog(repo: InventoryRepo, config: CatalogConfig) extends AutoCloseable {
    println(s"Catalog: featuring ${config.featuredTitle}")

    def frontPage(): String =
      s"Featured: ${config.featuredTitle} | In stock: ${repo.inStock().take(config.pageSize).mkString(", ")}"

    def close(): Unit = println("Catalog: flushing the page cache")
  }

  val env = ConfigSource.fromMap(Map("db.password" -> "change-me"), "env")

  val bookstoreYaml =
    """db:
      |  url: jdbc:postgresql://localhost:5432/bookstore
      |  user: bookstore
      |catalog:
      |  featuredTitle: Zionomicon
      |""".stripMargin

  val yaml = ConfigSource.fromYaml(bookstoreYaml, "bookstore.yaml") match {
    case Right(source) => source
    case Left(error)   => throw error
  }

  val defaults = ConfigSource.fromMap(Map("db.poolSize" -> "10", "catalog.pageSize" -> "3"), "defaults")

  val settings = env.orElse(yaml).orElse(defaults)

  def catalogResource(source: ConfigSource): Resource[Catalog] =
    Resource.from[Catalog](
      Wire(source),
      Config.wire[DbConfig]("db"),
      Config.wire[CatalogConfig]("catalog")
    )

  def frontPageWith(overrides: Map[String, String]): String = {
    val testSettings = ConfigSource.fromMap(overrides, "test").orElse(settings)
    Scope.global.scoped { scope =>

      val catalog = allocate(catalogResource(testSettings))
      $(catalog)(_.frontPage())
    }
  }

  def main(args: Array[String]): Unit = {
    val page =
      frontPageWith(Map("catalog.featuredTitle" -> "Functional Programming in Scala", "catalog.pageSize" -> "1"))

    println(page == "Featured: Functional Programming in Scala | In stock: Zionomicon")
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain feedconfigintoawiregraph.Step5Testing"
```

### The Complete Example

<details>
  <summary>config-examples/src/main/scala/feedconfigintoawiregraph/CompleteExample.scala</summary>

```scala title="config-examples/src/main/scala/feedconfigintoawiregraph/CompleteExample.scala" showLineNumbers
package feedconfigintoawiregraph

/**
 * How to Feed Configuration into a Wire Dependency Graph — Putting It Together
 *
 * Layers the real environment over YAML over defaults, builds the graph, and
 * shows the report for a bad catalog setting.
 *
 * Run with: cd config-examples && sbt "runMain
 * feedconfigintoawiregraph.CompleteExample"
 */
object CompleteExample {

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredTitle: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  class Db(config: DbConfig) extends AutoCloseable {
    println(
      s"Db: opening ${config.poolSize} connections to ${config.url} as ${config.user}, password ${config.password}"
    )

    def titles(): List[String] =
      List("Zionomicon", "Programming in Scala", "Functional Programming in Scala")

    def close(): Unit = println("Db: closing the connection pool")
  }

  class InventoryRepo(db: Db) {
    println("InventoryRepo: ready")

    def inStock(): List[String] = db.titles()
  }

  class Catalog(repo: InventoryRepo, config: CatalogConfig) extends AutoCloseable {
    println(s"Catalog: featuring ${config.featuredTitle}")

    def frontPage(): String =
      s"Featured: ${config.featuredTitle} | In stock: ${repo.inStock().take(config.pageSize).mkString(", ")}"

    def close(): Unit = println("Catalog: flushing the page cache")
  }

  val productionEnv: ConfigSource = EnvSource.keyFormat(KeyFormat.UpperSnakeCase).prefix("bookstore")

  val bookstoreYaml =
    """db:
      |  url: jdbc:postgresql://localhost:5432/bookstore
      |  user: bookstore
      |catalog:
      |  featuredTitle: Zionomicon
      |""".stripMargin

  val yaml = ConfigSource.fromYaml(bookstoreYaml, "bookstore.yaml") match {
    case Right(source) => source
    case Left(error)   => throw error
  }

  // The password default is for local runs only; production exports BOOKSTORE_DB_PASSWORD.
  val defaults = ConfigSource.fromMap(
    Map("db.password" -> "change-me", "db.poolSize" -> "10", "catalog.pageSize" -> "3"),
    "defaults"
  )

  val settings = productionEnv.orElse(yaml).orElse(defaults)

  def catalogResource(source: ConfigSource): Resource[Catalog] =
    Resource.from[Catalog](
      Wire(source),
      Config.wire[DbConfig]("db"),
      Config.wire[CatalogConfig]("catalog")
    )

  def main(args: Array[String]): Unit = {
    // 1. Build the graph, serve the front page, and close in reverse order.
    val frontPage = Scope.global.scoped { scope =>

      val catalog = allocate(catalogResource(settings))
      $(catalog)(_.frontPage())
    }
    println(frontPage)

    // 2. A bad catalog setting stops startup with a full report.
    val dbSection = ConfigSource.fromMap(
      Map(
        "db.url"      -> "jdbc:postgresql://localhost:5432/bookstore",
        "db.user"     -> "bookstore",
        "db.password" -> "change-me",
        "db.poolSize" -> "10"
      ),
      "db-section"
    )
    val badCatalog = ConfigSource.fromMap(Map("catalog.pageSize" -> "two"), "bad-catalog").orElse(dbSection)
    try {
      Scope.global.scoped { scope =>

        val catalog = allocate(catalogResource(badCatalog))
        $(catalog)(_.frontPage())
      }
    } catch {
      case e: ConfigLoadException => println(e.getMessage)
    }
  }
}
```

</details>

Run it from the `config-examples` directory, optionally overriding a setting through the environment:

```bash
BOOKSTORE_CATALOG_FEATURED_TITLE="Programming in Scala" sbt "runMain feedconfigintoawiregraph.CompleteExample"
```

To check that every example compiles without running it, use:

```bash
sbt compile
```

## Going Further

- [Configuration & Feature Flags](../reference/config/index.md#wiring-config-into-a-dependency-graph) covers `Config.wire` alongside `Config.load` and `Config.loadOrThrow`.
- [Wire](../reference/resource-management/wire.md) covers `Wire.shared`, `Wire.unique`, `Wire.Shared.fromFunction` and how `Resource.from` resolves a graph.
- [How to Fix Common Scope and Wire Compile Errors](./fix-scope-and-wire-compile-errors.md) explains each error the graph macro can report, such as a missing dependency or two parameters of the same type.
- [Coming from cats-effect Resource, ZIO Scope/ZLayer, macwire or Using](./resource-and-di-migration.md) maps those libraries onto `Resource`, `Scope` and `Wire`.
- [How to Roll Out a Feature Gradually with Flags](./roll-out-feature-gradually.md) adds settings that change at runtime, which a `Config.wire` value read once at startup can't do.
- [Build a Small JSON Backend from Blocks](./small-json-backend-from-blocks.md) is the capstone tutorial that uses a wire graph like this one inside a small JSON backend.

---

## How to Fix Common Scope and Wire Compile Errors

This guide is a lookup table for the errors that `Scope`, `Resource.from` and `Wire` report. Find your error message in the [Quick Index](#quick-index), jump to its entry, and leave with a fix. Each entry shows the smallest code that triggers the error, the real compiler output (rendered by mdoc, not pasted), one or two sentences on why it happens, and the corrected code.

The entries cover:

- the errors you will most likely hit first: escaping `$[A]` values, the `$` receiver rule, dependency cycles (which crash the macro in 0.0.56), duplicate providers, and dependencies the macro cannot build;
- every remaining code from E1 to E17, in code order;
- the leak warning (W1) and the closed-scope runtime errors (RE1 to RE3);
- the differences between the Scala 2 and Scala 3 macros.

## The Problem

`Scope` checks resource safety at compile time, and `Resource.from` resolves a whole dependency graph at compile time. That moves mistakes from production into the compiler, but it also means the first thing you see is a macro error. A message such as `No given instance of type Unscoped[Scope.Child[Scope.global.type]#$[Db]]` names the problem precisely, yet it doesn't tell you what to change. A large, unfamiliar error is the point where many people give up on a library.

Every Scope error has a known cause and a short fix. This page maps each message to both.

## Prerequisites

Add the module to your build:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-scope" % "0.0.56"
```

Every example on this page starts with this import:

```scala

```

You should know what `scoped`, `allocate`, `$` and `Resource.from` do. If you don't, read the [Compile-Time Resource Safety with Scope](./compile-time-resource-safety-with-scope.md) tutorial first; this page doesn't re-explain those concepts.

## Reading a Scope Error

Most macro errors come in a box that starts with `── Scope Error ──` and ends with a rule line. The first line inside the box says what went wrong, and the `Hint:` or `Fix:` line says what to do.

The codes on this page (E1, E10, RE2 and so on) come from the library source, `ErrorMessages.scala`. **The compiler output doesn't print them**, so match on the first line of the message, not on a code.

Three more things to know before you search:

- **Colors.** In a terminal the boxes are colored. Set the `NO_COLOR` environment variable to any non-empty value, or set the system property `-Dsbt.log.noformat=true`, to get plain text. The check runs inside the compiler's JVM.
- **Type names in this page.** mdoc compiles each snippet inside a generated class, so types appear as `MdocApp3.this.Catalog`. In your project you will see your own package, such as `com.example.Catalog`.
- **Some errors have no code.** Escaping a scope and breaking the `$` receiver rule come from the compiler's implicit search and from the `$` macro, not from `ErrorMessages`. They are listed first because they are the most common.

## Quick Index

| Code | The message starts with | Usual fix |
|------|-------------------------|-----------|
| none | `No given instance of type zio.blocks.scope.Unscoped[...]` | [Return plain data, not the scoped value](#escaping-scoped-value) |
| none | `Parameter 1 ('d') must only be used as a method receiver` | [Call a method on the parameter](#dollar-receiver-rule) |
| none | `value ... is not a member of scope.$[...]` | [Go through `$`](#method-on-scoped-value) |
| E10 | *(today: `Exception occurred while executing macro expansion` with `key not found`)* | [Break the cycle](#e10-dependency-cycle) |
| E11 | `Multiple providers for ...` | [Pass exactly one wire per type](#e11-multiple-providers) |
| E8 | `Cannot auto-create ...` / `This type is abstract` | [Wire a concrete implementation](#e8-abstract-type) |
| E7 | `Cannot auto-create ...` / `This type (primitive, collection, or function)` | [Provide the value with `Wire(value)`](#e7-unmakeable-type) |
| E9 | `Cannot auto-create ...` / `This type has no primary constructor` | [Provide an explicit wire](#e9-no-constructor) |
| E4 | `Resource.from[...] cannot be derived.` | [Pass wires, or call `Resource.from[T]()`](#e4-has-dependencies) |
| E1 | `Cannot derive Wire for ...: not a class.` | [Derive the concrete class](#e1-not-a-class) |
| E2 | `... has no primary constructor.` | [Write the wire by hand](#e2-no-primary-constructor) |
| E3 | `Dependency type conflict in ...` | [Wrap one of the types](#e3-subtype-conflict) |
| E5 | `Unsupported implicit parameter in ...` | [Make the parameter explicit](#e5-unsupported-implicit) |
| E6 | `Cannot extract types from wire expression.` | [Pass a real wire](#e6-cannot-extract-wire-types) |
| E12 | `Constructor of ... has multiple parameters of type ...` | [Wrap one of the parameters](#e12-duplicate-parameter-type) |
| E13 | `Expected varargs of Wire[?, ?] expressions.` | [List the wires inline](#e13-invalid-varargs) |
| E14-E17 | `Cannot derive Unscoped for ...` (Scala 2 only) | [Fix the type or write the instance](#e14-e17-unscoped-derivation) |
| W1 | `── Scope Warning ──` / `... is being leaked from scope` | [Stay inside `$`, or add `Unscoped`](#w1-leak-warning) |
| RE1 | `IllegalStateException`: `Cannot allocate resource: scope is already closed.` | [Allocate while the scope is open](#re1-allocate-on-closed-scope) |
| RE2 | `IllegalStateException`: `Cannot open child scope: scope is already closed.` | [Open while the parent is open](#re2-open-on-closed-scope) |
| RE3 | `IllegalStateException`: `Cannot access scoped value: scope is already closed.` | [Use `$` inside the owning scope](#re3-use-on-closed-scope) |

## The Most Common Errors

### Returning a Scoped Value from `scoped` {#escaping-scoped-value}

A `scoped` block may return a value only if its type has an `Unscoped` instance. A resource you allocated has type `$[Db]`, which has none:

```scala

class Db extends AutoCloseable {
  def countBooks(): Int = 42
  def close(): Unit     = ()
}
```

```scala
val db = Scope.global.scoped { scope =>

  allocate(Resource(new Db))
}
// error:
// No given instance of type zio.blocks.scope.Unscoped[
//   zio.blocks.scope.Scope.Child[zio.blocks.scope.Scope.global.type]#$[
//     MdocApp0.this.Db]
// ] was found for parameter ev of method scoped in trait ScopeVersionSpecific.
// I found:
// 
//     zio.blocks.scope.Unscoped.derived[
//       zio.blocks.scope.Scope.Child[zio.blocks.scope.Scope.global.type]#$[
//         MdocApp0.this.Db]
//     ](
//       /* missing */
//         summon[
//           scala.deriving.Mirror.Of[
//             zio.blocks.scope.Scope.Child[zio.blocks.scope.Scope.global.type]#$[
//               MdocApp0.this.Db]
//           ]
//         ]
//     )
// 
// But Failed to synthesize an instance of type scala.deriving.Mirror.Of[
//   zio.blocks.scope.Scope.Child[zio.blocks.scope.Scope.global.type]#$[
//     MdocApp0.this.Db]
// ]:
// 	* class Any is not a generic product because it is not a case class
// 	* class Any is not a generic sum because it is not a sealed class.
// }
// ^
```

The block's resources are closed when it exits, so returning `db` would hand out a closed connection. The `Mirror` lines in the message are the compiler explaining that it also tried, and failed, to derive an instance.

Return plain data computed from the resource instead:

```scala
val count: Int = Scope.global.scoped { scope =>

  val db = allocate(Resource(new Db))
  $(db)(_.countBooks())
}
// count: Int = 42
println(count)
// 42
```

The same error appears when you return your own type and it has no `Unscoped` instance:

```scala

class Book(val title: String)
```

```scala
val book = Scope.global.scoped { _ => new Book("Zionomicon") }
// error:
// No given instance of type zio.blocks.scope.Unscoped[MdocApp1.this.Book] was found for parameter ev of method scoped in trait ScopeVersionSpecific.
// I found:
// 
//     zio.blocks.scope.Unscoped.derived[MdocApp1.this.Book](
//       /* missing */summon[scala.deriving.Mirror.Of[MdocApp1.this.Book]])
// 
// But Failed to synthesize an instance of type scala.deriving.Mirror.Of[MdocApp1.this.Book]:
// 	* class Book is not a generic product because it is not a case class
// 	* class Book is not a generic sum because it is not a sealed class.
// val book = Scope.global.scoped { _ => new Book("Zionomicon") }
//                                                              ^
```

`Unscoped` instances exist for primitives, `String`, collections, `Option`, `Either` and tuples of up to four elements, as long as every element type is itself `Unscoped`. For your own data, declare an instance. In Scala 3, case classes, sealed traits and enums whose fields are all `Unscoped` get one automatically. In Scala 2 you derive it explicitly:

<Tabs groupId="scala-version" defaultValue="scala2">
<TabItem value="scala2" label="Scala 2">

```scala

final case class Book(title: String)
object Book {
  implicit val unscoped: Unscoped[Book] = Unscoped.derived[Book]
}

val book: Book = Scope.global.scoped { _ => Book("Zionomicon") }
```

</TabItem>
<TabItem value="scala3" label="Scala 3">

```scala

final case class Book(title: String) // Unscoped is derived automatically

val book: Book = Scope.global.scoped { _ => Book("Zionomicon") }
```

</TabItem>
</Tabs>

For a class that isn't a case class, write the instance by hand: `given Unscoped[Book] = new Unscoped[Book] {}` in Scala 3, or `implicit val unscopedBook: Unscoped[Book] = new Unscoped[Book] {}` in Scala 2. Only do this for pure data, never for a type that holds a resource.

If you return a case class that has a resource field, the error names the field's type, not the case class:

```scala

class Db extends AutoCloseable { def close(): Unit = () }
case class Receipt(db: Db)
```

```scala
val receipt = Scope.global.scoped { _ => Receipt(new Db) }
// error: 
// No given instance of type zio.blocks.scope.Unscoped[MdocApp3.this.Db] was found.
// I found:
// 
//     zio.blocks.scope.Unscoped.derived[MdocApp3.this.Db](
//       /* missing */summon[scala.deriving.Mirror.Of[MdocApp3.this.Db]])
// 
// But Failed to synthesize an instance of type scala.deriving.Mirror.Of[MdocApp3.this.Db]:
// 	* class Db is not a generic product because it is not a case class
// 	* class Db is not a generic sum because it is not a sealed class.
```

Derivation needs an `Unscoped` instance for every field, and `Db` has none. Keep the resource out of the returned type: put only plain data in `Receipt`.

### Using a `$` Parameter as a Value {#dollar-receiver-rule}

Inside `$(db)(d => ...)`, the parameter `d` may only be the receiver of a method call. Returning it, storing it, passing it to a function or capturing it in a closure is rejected:

```scala

class Db extends AutoCloseable {
  def countBooks(): Int = 42
  def close(): Unit     = ()
}

def describe(db: Db): String = s"${db.countBooks()} books"
```

```scala
Scope.global.scoped { scope =>

  val db = allocate(Resource(new Db))
  $(db)(d => d)
  ()
}
// error:
// Parameter 1 ('d') must only be used as a method receiver. It cannot be returned, stored, passed as an argument, or captured. Scoped values may only be used as a method receiver (e.g., d.method()).
//   $(db)(d => d)
//              ^
```

```scala
Scope.global.scoped { scope =>

  val db = allocate(Resource(new Db))
  $(db)(describe)
}
// error: 
// Parameter 1 ('db') cannot be passed as an argument to a function or method. Scoped values may only be used as a method receiver (e.g., db.method()).
```

Any of those uses would let the raw resource outlive the `$` call, and with it the scope. Passing a method reference such as `describe` counts as passing the parameter itself as an argument.

Call methods on the parameter, and pass their results along instead:

```scala
def describeCount(count: Int): String = s"$count books"

Scope.global.scoped { scope =>

  val db = allocate(Resource(new Db))
  $(db)(d => describeCount(d.countBooks()))
}
```

The [Scope reference](../reference/resource-management/scope.md#compile-errors) lists every form the `$` macro rejects.

### Calling a Method Directly on `$[A]` {#method-on-scoped-value}

A value of type `$[Db]` has none of `Db`'s methods:

```scala
Scope.global.scoped { scope =>

  val db = allocate(Resource(new Db))
  db.countBooks()
  ()
}
// error: 
// value countBooks is not a member of scope.$[MdocApp4.this.Db]
```

`$[A]` is an opaque wrapper, so the compiler can see when you use the resource. Go through `$`:

```scala
Scope.global.scoped { scope =>

  val db = allocate(Resource(new Db))
  $(db)(_.countBooks())
}
```

### E10: Dependency Cycle Detected {#e10-dependency-cycle}

Two services that need each other can't both be built first:

```scala

class InventoryRepo(prices: PriceService)
class PriceService(inventory: InventoryRepo)
class Catalog(inventory: InventoryRepo)
```

```scala
Resource.from[Catalog]()
```

`ErrorMessages` defines an E10 message with a cycle diagram, but in the current release `Resource.from` never reaches it. A cycle crashes the macro with `Exception occurred while executing macro expansion.` followed by `java.util.NoSuchElementException: key not found:` and the name of a type in the cycle, plus a long stack trace ([zio-blocks#1767](https://github.com/zio/zio-blocks/issues/1767)). If you see that exception from `Resource.from`, look for a cycle.

Break the cycle by restructuring the dependencies. Here both services only needed shared data, so they now depend on a `Db` instead of on each other:

```scala

class Db
class PriceService(db: Db)
class InventoryRepo(db: Db, prices: PriceService)
class Catalog(inventory: InventoryRepo)
```

```scala
Resource.from[Catalog]()
```

You can also break a cycle by putting a trait between the two classes and wiring an implementation that doesn't depend back on the first one.

### E11: Multiple Providers for a Type {#e11-multiple-providers}

`Resource.from` matches each dependency against the wires you pass, subtypes included. Two wires that both provide `PriceService` are ambiguous:

```scala

trait PriceService
class FixedPrices    extends PriceService
class DiscountPrices extends PriceService
class Catalog(prices: PriceService)
```

```scala
Resource.from[Catalog](Wire.shared[FixedPrices], Wire.shared[DiscountPrices])
// error: 
// ── Scope Error ─────────────────────────────────────────────────────────────────
// 
//   Multiple providers for MdocApp7.this.PriceService
// 
//   Conflicting wires:
//     1. MdocApp7.this.FixedPrices
//     2. MdocApp7.this.DiscountPrices
// 
//   Hint: Remove duplicate wires or use distinct wrapper types.
// 
// ────────────────────────────────────────────────────────────────────────────────
```

The macro can't guess which implementation `Catalog` should get. Pass exactly one wire for each dependency type:

```scala
Resource.from[Catalog](Wire.shared[FixedPrices])
```

If you really need both implementations in one graph, give them distinct types, for example wrapper case classes, and make the constructors ask for those types.

### E8: Abstract Type Cannot Be Auto-Created {#e8-abstract-type}

`Resource.from` builds every dependency you don't provide a wire for, but it can't instantiate a trait or an abstract class:

```scala

trait PriceService
class FixedPrices extends PriceService
class Catalog(prices: PriceService)
```

```scala
Resource.from[Catalog]()
// error: 
// ── Scope Error ─────────────────────────────────────────────────────────────────
// 
//   Cannot auto-create MdocApp8.this.PriceService
// 
//   This type is abstract (trait or abstract class).
// 
//   Required by:
//   └── MdocApp8.this.Catalog
// 
//   Fix: Provide a wire for a concrete implementation:
// 
//     Resource.from[...](
//       Wire.shared[ConcreteImpl],  // provides MdocApp8.this.PriceService
//       ...
//     )
// 
// ────────────────────────────────────────────────────────────────────────────────
```

The `Required by:` tree shows which constructor asked for the abstract type. Wire a concrete implementation:

```scala
Resource.from[Catalog](Wire.shared[FixedPrices])
```

### E7: Primitive, Collection, or Function Cannot Be Auto-Created {#e7-unmakeable-type}

The macro never invents values for `String`, the numeric types, `Boolean`, `Unit`, `List`, `Seq`, `Set`, `Map`, `Option` or function types:

```scala

class Db(url: String)
class Catalog(db: Db)
```

```scala
Resource.from[Catalog]()
// error: 
// ── Scope Error ─────────────────────────────────────────────────────────────────
// 
//   Cannot auto-create java.lang.String
// 
//   This type (primitive, collection, or function) cannot be auto-created.
// 
//   Required by:
//   ├── MdocApp9.this.Db
//     └── MdocApp9.this.Catalog
// 
//   Fix: Provide Wire(value) with the desired value:
// 
//     Resource.from[...](
//       Wire(...),  // provide a value for java.lang.String
//       ...
//     )
// 
// ────────────────────────────────────────────────────────────────────────────────
```

There is no sensible default for a database URL. Provide the value with `Wire(value)`:

```scala
Resource.from[Catalog](Wire("jdbc:h2:mem:bookstore"))
```

If a graph needs two values of the same primitive type, wrap each in its own case class (for example `final case class DbUrl(value: String)`), because the graph has one slot per type.

### E9: No Constructor for Auto-Create {#e9-no-constructor}

This message means a dependency is a concrete class with no primary constructor. Every Scala class has one, so this guide found no way to trigger E9 from Scala code. If you do see it, provide an explicit wire for that type, as the message's `Fix:` line shows.

### E4: `Resource.from[T]` Has Dependencies {#e4-has-dependencies}

`Resource.from[T]` without an argument list only works for types whose constructor takes nothing, or takes only a `Finalizer` or `Scope`:

```scala

class Db extends AutoCloseable { def close(): Unit = () }
class Catalog(db: Db)
```

```scala
Resource.from[Catalog]
// error: 
// ── Scope Error ─────────────────────────────────────────────────────────────────
// 
//   Resource.from[MdocApp10.this.Catalog] cannot be derived.
// 
//   MdocApp10.this.Catalog has dependencies that must be provided:
//     • MdocApp10.this.Db
// 
//   Hint: Use Resource.from[MdocApp10.this.Catalog](wire1, wire2, ...)
//         to provide wires for all dependencies.
// 
// ────────────────────────────────────────────────────────────────────────────────
```

That form doesn't resolve a dependency graph. Use the wires form: pass wires for the dependencies, or an empty argument list to let the macro build every dependency it can:

```scala
def explicitDb = Resource.from[Catalog](Wire.shared[Db])
val autoDb     = Resource.from[Catalog]()
```

Note the difference between `Resource.from[Catalog]` and `Resource.from[Catalog]()`: only the second one resolves dependencies.

## The Remaining Compile Errors

### E1: Not a Class {#e1-not-a-class}

`Wire.shared[T]`, `Wire.unique[T]` and `Resource.from[T]` call `T`'s constructor, so `T` must be a concrete class:

```scala

trait PriceService
class FixedPrices extends PriceService
```

```scala
Wire.shared[PriceService]
// error: 
// ── Scope Error ─────────────────────────────────────────────────────────────────
// 
//   Cannot derive Wire for MdocApp11.this.PriceService: not a class.
// 
//   Hint: Use Wire.Shared / Wire.Unique directly.
// 
// ────────────────────────────────────────────────────────────────────────────────
```

A trait or an abstract class has no constructor to call. Derive the concrete class; the wire still provides `PriceService` to anything that needs one:

```scala
val prices = Wire.shared[FixedPrices]
```

If you need construction logic the macro can't derive, build the wire by hand with `Wire.Shared` or `Wire.Unique`, as the hint says.

### E2: No Primary Constructor {#e2-no-primary-constructor}

Like E9, this message needs a class without a primary constructor, and every Scala class has one, so this guide found no Scala trigger for it. The fix is to write the wire by hand with `Wire.Shared` or `Wire.Unique`.

### E3: Subtype Conflict Between Dependencies {#e3-subtype-conflict}

A constructor can't take two dependencies where one type is a subtype of the other:

```scala

class InventoryRepo
class SqlInventoryRepo extends InventoryRepo
class Catalog(inventory: InventoryRepo, sql: SqlInventoryRepo)
```

```scala
Wire.shared[Catalog]
// error: 
// ── Scope Error ─────────────────────────────────────────────────────────────────
// 
//   Dependency type conflict in MdocApp12.this.Catalog
// 
//   MdocApp12.this.SqlInventoryRepo is a subtype of MdocApp12.this.InventoryRepo.
// 
//   When both types are dependencies, Context cannot reliably distinguish
//   them. The more specific type may be retrieved when the more general
//   type is requested.
// 
//   To fix this, wrap one or both types in a distinct wrapper:
// 
//     case class WrappedMdocApp12.this.InventoryRepo(value: MdocApp12.this.InventoryRepo)
//     or
//     opaque type WrappedMdocApp12.this.InventoryRepo = MdocApp12.this.InventoryRepo
// 
// ────────────────────────────────────────────────────────────────────────────────
```

The dependency graph is indexed by type, so a request for `InventoryRepo` could be answered with the `SqlInventoryRepo`. Wrap one of the two types so they're unrelated. The code in the message's hint isn't valid Scala when the type has a package prefix, so write the wrapper yourself:

```scala

class InventoryRepo
class SqlInventoryRepo extends InventoryRepo
final case class ReadReplica(repo: SqlInventoryRepo)
class Catalog(inventory: InventoryRepo, replica: ReadReplica)
```

```scala
val catalog = Wire.shared[Catalog]
```

### E5: Unsupported Implicit Parameter {#e5-unsupported-implicit}

This message targets a `Resource.from[T]` constructor with an implicit parameter other than `Finalizer` or `Scope`. Both macros check for dependencies (E4) first, and an implicit `Clock` parameter is reported as an E4 dependency, so E5 isn't reachable from user code. Follow the E4 fix: make the parameter explicit and provide it with a wire.

### E6: Cannot Extract Wire Types {#e6-cannot-extract-wire-types}

Every real wire has the type `Wire[In, Out]`, so `Resource.from` reports E6 only when an argument has some other static type, such as `???` (type `Nothing`). Pass real `Wire` values.

### E12: Duplicate Parameter Type {#e12-duplicate-parameter-type}

This message is for a constructor with two parameters of the same type. In Scala 3 it isn't reported: the code compiles, and both parameters receive the same instance:

```scala

class Db
class Catalog(val primary: Db, val replica: Db)
```

```scala
val sameDb: Boolean = Scope.global.scoped { scope =>

  val catalog = allocate(Resource.from[Catalog]())
  $(catalog)(c => c.primary.eq(c.replica))
}
// sameDb: Boolean = true
println(sameDb)
// true
```

The graph holds one value per type, so it can't supply two different `Db` values. If the two parameters must differ, give each a distinct type:

```scala

class Db
final case class PrimaryDb(db: Db)
final case class ReplicaDb(db: Db)
class Catalog(primary: PrimaryDb, replica: ReplicaDb)
```

```scala
Resource.from[Catalog](
  Wire(PrimaryDb(new Db)),
  Wire(ReplicaDb(new Db))
)
```

### E13: Invalid Varargs {#e13-invalid-varargs}

`Resource.from[T](wires*)` reads its arguments at compile time, so it needs them written out at the call site:

```scala

class Db
class Catalog(db: Db)
```

```scala
val wires = Seq(Wire.shared[Db])
Resource.from[Catalog](wires*)
// error:
// ── Scope Error ─────────────────────────────────────────────────────────────────
// 
//   Expected varargs of Wire[?, ?] expressions.
// 
//   Got: MdocApp16.this.wires: _*
// 
//   Hint: Use Resource.from[T](wire1, wire2, ...)
// 
// ────────────────────────────────────────────────────────────────────────────────
// Resource.from[Catalog](wires*)
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

A `Seq` only exists at run time, so the macro can't see which wires it holds. List the wires inline:

```scala
Resource.from[Catalog](Wire.shared[Db])
```

E13 is Scala 3 only.

### E14 to E17: `Unscoped` Derivation Errors (Scala 2 Only) {#e14-e17-unscoped-derivation}

These four errors come from Scala 2's `Unscoped.derived[A]` macro, which you call yourself (see the Scala 2 tab in [Returning a Scoped Value](#escaping-scoped-value)). All of them start with `Cannot derive Unscoped for`:

- **E14, type parameter not deducible.** A subtype of a sealed generic type has a type parameter the macro can't work out from the parent's type arguments. Give the subtype concrete type arguments, or simplify the hierarchy.
- **E15, sealed type with no known subclasses.** Define at least one subclass in the same file as the sealed type.
- **E16, no `Unscoped` instance for a field or subtype.** For example, `Unscoped.derived[Receipt]` where `Receipt` has a `Db` field. Remove the resource from the type, or give the field's type its own instance if it's pure data.
- **E17, no primary constructor.** Write the instance by hand: `implicit val unscoped: Unscoped[A] = new Unscoped[A] {}`.

Scala 3 doesn't use these codes. Its `Unscoped` derivation is a `given` built on `Mirror`, so the same mistakes produce the compiler's standard `No given instance of type ...Unscoped[...]` message, as in [Returning a Scoped Value](#escaping-scoped-value).

## Warnings

### W1: Value Leaked from Its Scope {#w1-leak-warning}

`leak` unwraps a `$[A]` into a plain `A`, which turns off the compile-time checks for that value. The macro warns every time:

```scala

class Db extends AutoCloseable {
  def countBooks(): Int = 42
  def close(): Unit     = ()
}
```

```scala
Scope.global.scoped { scope =>

  val db      = allocate(Resource(new Db))
  val raw: Db = leak(db)
  ()
}
// warn:
// ── Scope Warning ───────────────────────────────────────────────────────────────
// 
//   leak(db)
//        ^
//        |
// 
//   Warning: db is being leaked from scope zio.blocks.scope.Scope.Child[zio.blocks.scope.Scope.global].
//   This may result in undefined behavior.
// 
//   Hint:
//      If you know this data type is not resourceful, then add an Unscoped
//      instance for it so you do not need to leak it.
// 
// ────────────────────────────────────────────────────────────────────────────────
//   val raw: Db = leak(db)
//                      ^^
```

After the block exits, `raw` points at a closed resource. Keep the work inside `$`:

```scala
Scope.global.scoped { scope =>

  val db = allocate(Resource(new Db))
  $(db)(_.countBooks())
}
```

Use `leak` only to hand a resource to a library that can't accept `$[A]`, and keep that library call inside the block. If the leaked type is pure data, give it an `Unscoped` instance instead: `$` then returns it unwrapped and nothing has to leak.

## Runtime Errors

These errors are `IllegalStateException`s, thrown when code uses a scope after it has closed. The compiler can't catch them when a scope reference outlives its block, for example through `open()` and `close()`, a field, or another thread.

### RE1: Allocate on a Closed Scope {#re1-allocate-on-closed-scope}

```scala
val shop1   = Scope.global.open()
val closed1 = shop1.scope
shop1.close()
closed1.allocate(Resource(new Db))
// java.lang.IllegalStateException: ── Scope Error ─────────────────────────────────────────────────────────────────
// 
//   Cannot allocate resource: scope is already closed.
// 
//   Scope: Scope.Child
// 
//   What happened:
//     A call to allocate was made on a scope whose finalizers have
//     already run. The resource was never acquired.
// 
//   Common causes:
//     • A scope reference escaped a scoped { } block (e.g. stored in a
//       field, captured in a Future or passed to another thread).
//     • close() was called on an OpenScope before all
//       allocations inside it completed.
// 
//   Fix:
//     Call allocate only inside a live scoped { } block, or before
//     calling close() on an OpenScope.
// 
//     // Correct usage:
//     Scope.global.scoped { scope =>
//       import scope.*
//       val db = allocate(Resource(new Database))
//       $(db)(_.query("SELECT 1"))
//     }
// 
// ────────────────────────────────────────────────────────────────────────────────
// 	at zio.blocks.scope.Scope.allocate(Scope.scala:229)
// 	at repl.MdocSession$MdocApp43.$init$$$anonfun$8(fix-scope-and-wire-compile-errors.md:553)
```

`close()` already ran the scope's finalizers, so a new resource would never be released. Allocate before you close the scope.

### RE2: Open a Child of a Closed Scope {#re2-open-on-closed-scope}

```scala
val shop2   = Scope.global.open()
val closed2 = shop2.scope
shop2.close()
closed2.open()
// java.lang.IllegalStateException: ── Scope Error ─────────────────────────────────────────────────────────────────
// 
//   Cannot open child scope: scope is already closed.
// 
//   Scope: Scope.Child
// 
//   What happened:
//     A call to open() was made on a scope whose finalizers have
//     already run. No child scope was created.
// 
//   Common causes:
//     • A scope reference escaped a scoped { } block and open()
//       was called after the block exited.
//     • close() was called on the parent OpenScope before
//       open() was called on it.
// 
//   Fix:
//     Call open() only on a live (not yet closed) scope.
// 
//     // Correct usage:
//     Scope.global.scoped { scope =>
//       import scope.*
//       val child = open()
//       $(child)(_.scope.allocate(Resource(new Database)))
//     }
// 
// ────────────────────────────────────────────────────────────────────────────────
// 	at zio.blocks.scope.Scope.open(Scope.scala:313)
// 	at repl.MdocSession$MdocApp43.$init$$$anonfun$9(fix-scope-and-wire-compile-errors.md:565)
```

A child scope is linked to its parent's finalizers, and a closed parent has none left to link to. Open children while the parent is still open.

### RE3: Use `$` on a Closed Scope {#re3-use-on-closed-scope}

```scala
val shop3   = Scope.global.open()
val closed3 = shop3.scope
val db3     = closed3.allocate(Resource(new Db))
shop3.close()
(closed3 $ db3)(_.countBooks())
// java.lang.IllegalStateException: ── Scope Error ─────────────────────────────────────────────────────────────────
// 
//   Cannot access scoped value: scope is already closed.
// 
//   Scope: Scope.Child
// 
//   What happened:
//     The $ operator was called on a scope whose finalizers have
//     already run. The underlying resource may have been released.
//     Accessing it would be undefined behavior.
// 
//   Common causes:
//     • A $[A] value or its owning scope escaped a scoped { }
//       block (e.g. captured in a Future, stored in a field, or
//       passed to another thread).
//     • close() was called on an OpenScope that still has
//       live $[A] values being accessed.
// 
//   Fix:
//     Ensure all $ calls occur strictly within the scoped { }
//     block that owns the value, and that the scope has not been closed.
// 
//     // Correct usage:
//     Scope.global.scoped { scope =>
//       import scope.*
//       val db = allocate(Resource(new Database))
//       $(db)(_.query("SELECT 1"))  // $ used inside the block
//     }
// 
// ────────────────────────────────────────────────────────────────────────────────
// 	at repl.MdocSession$MdocApp43.$init$$$anonfun$10(fix-scope-and-wire-compile-errors.md:578)
```

The value behind `db` has already been released. Make every `$` call before the owning scope closes.

`defer` on a closed scope is not an error: the finalizer is ignored and you get a no-op handle back. Calling `scoped` from a thread that doesn't own the scope throws a different `IllegalStateException`, `Cannot create child scope: current thread '...' does not own this scope`. The tutorial's [Thread Ownership](./compile-time-resource-safety-with-scope.md#11-thread-ownership) section covers that case.

## Scala 2 and Scala 3 Differences

Both macro implementations render their errors with the same `ErrorMessages` functions, so the boxes read the same in both versions. The differences:

- **E13** is reported only by Scala 3.
- **E14 to E17** are reported only by Scala 2, by `Unscoped.derived`. Scala 3 reports the same mistakes with the compiler's standard `No given instance` message.
- **Escaping a scope** is a missing implicit in both versions, but Scala 2 uses its own wording for a missing implicit.
- **Type names** are printed with `toString` in Scala 2 and with `show` in Scala 3, so some types are formatted slightly differently.

mdoc compiles this page with Scala 3, so every rendered message above is Scala 3 output.

## Putting It Together

This bookstore graph compiles cleanly. It provides the database URL with `Wire(value)` (E7), wires a concrete `PriceService` (E8), avoids a cycle between `InventoryRepo` and `PriceService` (E10), and returns a case class instead of a scoped value:

```scala title="scope-examples/src/main/scala/scope/examples/compileerrors/BookstoreWiringExample.scala"
package scope.examples.compileerrors

/**
 * A bookstore service graph that compiles cleanly, built from the fixes in the
 * "How to Fix Common Scope and Wire Compile Errors" guide:
 *
 *   - `Db` needs a `String`, which cannot be auto-created, so the graph
 *     provides it with `Wire(value)`.
 *   - `PriceService` is a trait, so the graph names a concrete implementation
 *     with `Wire.shared[FixedPrices]`.
 *   - `InventoryRepo` and `PriceService` both depend on `Db` instead of on each
 *     other, so there is no dependency cycle.
 *   - The `scoped` block returns a `Quote` case class, which has an `Unscoped`
 *     instance, instead of the scoped `Catalog` itself.
 */

final class Db(url: String) extends AutoCloseable {
  println(s"[Db] connected to $url")

  private val stock = Map("Zionomicon" -> 3, "Programming in Scala" -> 0)
  private val price = Map("Zionomicon" -> BigDecimal("49.00"), "Programming in Scala" -> BigDecimal("59.95"))

  def stockOf(title: String): Int        = stock.getOrElse(title, 0)
  def priceOf(title: String): BigDecimal = price.getOrElse(title, BigDecimal(0))

  def close(): Unit = println("[Db] closed")
}

trait PriceService {
  def priceOf(title: String): BigDecimal
}

final class FixedPrices(db: Db) extends PriceService {
  def priceOf(title: String): BigDecimal = db.priceOf(title)
}

final class InventoryRepo(db: Db) {
  def stockOf(title: String): Int = db.stockOf(title)
}

final class Catalog(inventory: InventoryRepo, prices: PriceService) {
  def quote(title: String): Quote = Quote(title, prices.priceOf(title), inventory.stockOf(title))
}

final case class Quote(title: String, price: BigDecimal, inStock: Int)

@main def bookstoreWiringExample(): Unit = {
  val quote: Quote = Scope.global.scoped { scope =>

    val catalog = allocate(
      Resource.from[Catalog](
        Wire("jdbc:h2:mem:bookstore"), // String cannot be auto-created: provide the value
        Wire.shared[FixedPrices]       // PriceService is a trait: pick an implementation
      )
    )

    $(catalog)(_.quote("Zionomicon"))
  }

  println(quote)
}
```

## Running the Examples

Clone the repository and move into the examples module:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/scope-examples
```

<details>
<summary>A bookstore graph with every fix applied</summary>

```scala title="scope-examples/src/main/scala/scope/examples/compileerrors/BookstoreWiringExample.scala" showLineNumbers
package scope.examples.compileerrors

/**
 * A bookstore service graph that compiles cleanly, built from the fixes in the
 * "How to Fix Common Scope and Wire Compile Errors" guide:
 *
 *   - `Db` needs a `String`, which cannot be auto-created, so the graph
 *     provides it with `Wire(value)`.
 *   - `PriceService` is a trait, so the graph names a concrete implementation
 *     with `Wire.shared[FixedPrices]`.
 *   - `InventoryRepo` and `PriceService` both depend on `Db` instead of on each
 *     other, so there is no dependency cycle.
 *   - The `scoped` block returns a `Quote` case class, which has an `Unscoped`
 *     instance, instead of the scoped `Catalog` itself.
 */

final class Db(url: String) extends AutoCloseable {
  println(s"[Db] connected to $url")

  private val stock = Map("Zionomicon" -> 3, "Programming in Scala" -> 0)
  private val price = Map("Zionomicon" -> BigDecimal("49.00"), "Programming in Scala" -> BigDecimal("59.95"))

  def stockOf(title: String): Int        = stock.getOrElse(title, 0)
  def priceOf(title: String): BigDecimal = price.getOrElse(title, BigDecimal(0))

  def close(): Unit = println("[Db] closed")
}

trait PriceService {
  def priceOf(title: String): BigDecimal
}

final class FixedPrices(db: Db) extends PriceService {
  def priceOf(title: String): BigDecimal = db.priceOf(title)
}

final class InventoryRepo(db: Db) {
  def stockOf(title: String): Int = db.stockOf(title)
}

final class Catalog(inventory: InventoryRepo, prices: PriceService) {
  def quote(title: String): Quote = Quote(title, prices.priceOf(title), inventory.stockOf(title))
}

final case class Quote(title: String, price: BigDecimal, inStock: Int)

@main def bookstoreWiringExample(): Unit = {
  val quote: Quote = Scope.global.scoped { scope =>

    val catalog = allocate(
      Resource.from[Catalog](
        Wire("jdbc:h2:mem:bookstore"), // String cannot be auto-created: provide the value
        Wire.shared[FixedPrices]       // PriceService is a trait: pick an implementation
      )
    )

    $(catalog)(_.quote("Zionomicon"))
  }

  println(quote)
}
```

Run it with:

```bash
sbt "runMain scope.examples.compileerrors.bookstoreWiringExample"
```

</details>

To compile the example without running it:

```bash
sbt compile
```

## See Also

- [Compile-Time Resource Safety with Scope](./compile-time-resource-safety-with-scope.md): the tutorial that introduces every concept this page assumes.
- [Scope reference](../reference/resource-management/scope.md): `scoped`, `allocate`, `$`, `open` and `leak` in detail.
- [Wire reference](../reference/resource-management/wire.md): `Wire.shared`, `Wire.unique`, `Wire(value)` and macro derivation.
- [Resource reference](../reference/resource-management/resource.md): `Resource.from[T]` and `Resource.from[T](wires*)`.
- [Unscoped reference](../reference/resource-management/unscoped.md): the built-in instances and how to declare your own.

---

## How to Generate JSON Schema from a Type and Validate Payloads Against It

In this guide, we build the API contract for a book-catalog service that accepts new book submissions. We derive a JSON Schema document (draft 2020-12) from the `BookSubmission` case class, publish it as a file that other teams can read, validate raw JSON payloads against it, and load a stricter schema from another team. By the end, you will have one program that generates the contract from the type and rejects bad payloads with readable error messages.

We take an incremental approach:

- Generate the JSON Schema contract from the case class with `Schema#toJsonSchema`
- Publish the contract as a JSON file and read it back
- Validate untyped JSON payloads with `JsonSchema#check`
- Load an external JSON Schema with `JsonSchema.parse` and decode payloads through `Schema.fromJsonSchema`
- Choose between typed decoding and JSON Schema validation

## The Problem

A service that accepts JSON from other teams needs a written contract — a JSON Schema document that says which fields exist, which are required, and what type each one has. Writing that document by hand means describing the same type twice: once as a Scala case class and once as JSON.

The two descriptions drift apart silently. Rename a field, make one optional, or add a list, and the case class still compiles while the contract keeps promising the old shape. Clients that validate against the stale contract send payloads the service rejects, and nothing fails until production.

For example, a hand-written contract for a book submission sits next to the case class it describes:

```scala
case class BookSubmission(title: String, firstAuthor: String, year: Int, isbn: Option[String], tags: List[String])

// Kept in sync with BookSubmission by hand, and shipped to other teams.
val bookSubmissionContract: String =
  """{
    |  "type": "object",
    |  "properties": {
    |    "title": { "type": "string" },
    |    "firstAuthor": { "type": "string" },
    |    "year": { "type": "integer" },
    |    "isbn": { "type": "string" },
    |    "tags": { "type": "array", "items": { "type": "string" } }
    |  },
    |  "required": ["title", "firstAuthor", "year"]
    |}""".stripMargin
```

Nothing connects the string to the case class. When `firstAuthor` becomes `authors: List[String]`, the compiler stays quiet, and checking an incoming payload against the string still needs a separate JSON Schema library.

In this guide, we derive the contract from the case class itself, so it changes whenever the type changes, and we validate payloads with the same library that generates it.

## Prerequisites

Add the schema module to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56"
```

This guide assumes you can derive a `Schema` for a case class and decode JSON with it, as shown in the [Your First Schema](./your-first-schema.md) tutorial. The JSON Schema types, such as `JsonSchema` and `Json`, live in the `zio.blocks.schema.json` package.

## The Contract Type

Our service accepts one kind of payload: a book submission with a title, the first author's name, the publication year, an optional ISBN, and a list of tags. We derive its schema in the companion object and mark the record with `@Modifier.noExtraFields()`, because a contract for incoming data should reject fields it does not know:

```scala

@Modifier.noExtraFields()
case class BookSubmission(
  title: String,
  firstAuthor: String,
  year: Int,
  isbn: Option[String],
  tags: List[String]
)

object BookSubmission {
  implicit val schema: Schema[BookSubmission] = Schema.derived
}
```

## Generate the JSON Schema Contract

`Schema#toJsonSchema` derives a `JsonSchema` from the same description that the JSON codec uses, so the contract and the decoder agree on every field. We derive the contract once and keep it as a value:

```scala
val contract: JsonSchema = BookSubmission.schema.toJsonSchema
```

`JsonSchema#toString` pretty-prints the document, so printing the contract shows exactly what other teams receive:

```scala
println(contract)
// {
//   "properties": {
//     "title": {
//       "type": "string"
//     },
//     "firstAuthor": {
//       "type": "string"
//     },
//     "year": {
//       "type": "integer",
//       "maximum": 2147483647,
//       "minimum": -2147483648
//     },
//     "isbn": {
//       "type": [
//         "null",
//         "string"
//       ]
//     },
//     "tags": {
//       "items": {
//         "type": "string"
//       },
//       "type": [
//         "array",
//         "null"
//       ],
//       "title": "List[String]"
//     }
//   },
//   "additionalProperties": false,
//   "type": "object",
//   "required": [
//     "title",
//     "firstAuthor",
//     "year"
//   ],
//   "title": "BookSubmission"
// }
```

The contract that `Schema#toJsonSchema` derives follows the codec's rules for each field:

- `"required"` lists `title`, `firstAuthor` and `year`. The `Option` field `isbn` and the `List` field `tags` are left out, because the decoder accepts a payload without them — fields with a default value are left out too.
- `isbn` accepts `null` as well as a string (`"type": ["null", "string"]`), and `tags` accepts `null` as well as an array.
- `year` carries `minimum` and `maximum` bounds, because a JSON number outside the `Int` range cannot become an `Int`.
- `"additionalProperties": false` comes from `@Modifier.noExtraFields()`. Without the annotation, the key is absent and extra properties are allowed.
- `"title"` names the type that each part of the contract derives from.

The output contains no `$schema` keyword; the document follows draft 2020-12, the version that `JsonSchema` implements.

:::note
Constraints that you add to a schema with `Validation` (see [Validation](../reference/schema/validation.md)), such as a minimum length or a pattern, do not appear in the contract that `Schema#toJsonSchema` derives. The only bounds in the output come from the field types, like the `Int` range on `year`.
:::

## Publish the Contract as a JSON File

Other teams need the contract as a file, not as a Scala value. `JsonSchema#toJson` turns the contract into a `Json` value, and `Json#print` renders it as text — compact by default, or indented when we pass a `WriterConfig`. We write the indented text to a file with the JDK's `Files.writeString`:

```scala

val contractText: String = contract.toJson.print(WriterConfig.withIndentionStep2)

val contractFile: Path = Files.createTempFile("book-submission", ".schema.json")
Files.writeString(contractFile, contractText)
```

The indented text is the same document that `println(contract)` shows. The compact form fits on one line, which suits logs and HTTP responses:

```scala
println(contract.toJson.print)
// {"properties":{"title":{"type":"string"},"firstAuthor":{"type":"string"},"year":{"type":"integer","maximum":2147483647,"minimum":-2147483648},"isbn":{"type":["null","string"]},"tags":{"items":{"type":"string"},"type":["array","null"],"title":"List[String]"}},"additionalProperties":false,"type":"object","required":["title","firstAuthor","year"],"title":"BookSubmission"}
```

Reading the file back with `JsonSchema.parse` gives a contract equal to the one we write:

```scala
println(JsonSchema.parse(Files.readString(contractFile)) == Right(contract))
// true
```

For the full parsing and serialization API, see [Parsing and Serialization](../reference/schema/built-in-codecs/json/json-schema.md#parsing-and-serialization) in the JSON Schema reference.

## Validate an Untyped Payload

A gateway, a message consumer or a test fixture often holds raw JSON text rather than a Scala value. We parse the text with `Json.parse` and validate the result with `JsonSchema#check`, which returns `None` when the payload conforms and `Some(SchemaError)` when it does not. A small helper prints either outcome:

```scala
def validate(schema: JsonSchema, payload: String): Unit =
  Json.parse(payload) match {
    case Left(parseError) => println(s"not JSON: ${parseError.message}")
    case Right(json)      =>
      schema.check(json) match {
        case None        => println("valid")
        case Some(error) => println(error.message)
      }
  }
```

A complete submission for Programming in Scala passes:

```scala
validate(
  contract,
  """{"title":"Programming in Scala","firstAuthor":"Martin Odersky","year":2008,"isbn":"9780981531601","tags":["scala","fundamentals"]}"""
)
// valid
```

A submission for Zionomicon without an ISBN or tags passes too, because the contract does not require them:

```scala
validate(contract, """{"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024}""")
// valid
```

A submission with an unknown `publisher` field, the year sent as a string and no author fails, and `JsonSchema#check` reports every problem at once, one per line:

```scala
validate(contract, """{"title":"Zionomicon","year":"2024","publisher":"Ziverge"}""")
// Schema rejects all values at: .
// Expected type integer at: .year
// Missing required property 'firstAuthor' at: .
```

Each line ends with the path to the failing value. The first line comes from `"additionalProperties": false`: it reports the extra `publisher` field, but its path points at the enclosing object (`.`) rather than at the field itself.

A year that does not fit in an `Int` breaks the `maximum` bound that the contract derives from the field type:

```scala
validate(
  contract,
  """{"title":"Functional Programming in Scala","firstAuthor":"Paul Chiusano","year":20140000000}"""
)
// Value 20140000000 is greater than maximum 2147483647 at: .year
```

When you only need a yes-or-no answer, `JsonSchema#conforms` returns a `Boolean` instead. For more on the validation API, see [Validation](../reference/schema/built-in-codecs/json/json-schema.md#validation) in the JSON Schema reference.

## Load a Schema from Another Source

Sometimes the contract comes from somewhere else — a partner team, a schema registry or a configuration file read at startup. Suppose the catalog team publishes a stricter schema for the same payload: titles and authors must be non-empty, the year must be 1450 or later, an ISBN must be an ISBN-13 without hyphens, and tags must not repeat:

```scala
val catalogContractText: String =
  """{
    |  "$schema": "https://json-schema.org/draft/2020-12/schema",
    |  "type": "object",
    |  "properties": {
    |    "title": { "type": "string", "minLength": 1 },
    |    "firstAuthor": { "type": "string", "minLength": 1 },
    |    "year": { "type": "integer", "minimum": 1450 },
    |    "isbn": { "type": "string", "pattern": "^97[89][0-9]{10}$" },
    |    "tags": { "type": "array", "items": { "type": "string" }, "uniqueItems": true }
    |  },
    |  "required": ["title", "firstAuthor", "year"],
    |  "additionalProperties": false
    |}""".stripMargin
```

We can use this document in two ways: check `Json` values against it directly, or turn it into a `Schema[Json]` that validates while it decodes.

### Check Payloads Against the Parsed Schema

`JsonSchema.parse` turns the text into a `JsonSchema`, or returns a `SchemaError` when the text is not a valid schema. We fail fast at startup when the document is broken, then reuse the `validate` helper from the previous section:

```scala
val catalogContract: JsonSchema =
  JsonSchema.parse(catalogContractText).fold(error => throw error, identity)
```

The Zionomicon submission passes the stricter schema:

```scala
validate(catalogContract, """{"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024}""")
// valid
```

A submission that breaks the catalog team's rules — an empty title, a hyphenated ISBN and a repeated tag — fails with one message per broken rule:

```scala
validate(
  catalogContract,
  """{"title":"","firstAuthor":"Paul Chiusano","year":2014,"isbn":"978-1617290657","tags":["fp","fp"]}"""
)
// Array items are not unique at: .tags
// String does not match pattern '^97[89][0-9]{10}$' at: .isbn
// String length 0 is less than minLength 1 at: .title
```

`JsonSchema#check` also validates the `format` keyword (`email`, `uuid`, `date-time` and others) by default; see [Format Validation](../reference/schema/built-in-codecs/json/json-schema.md#format-validation) for the supported formats and how to turn it off.

### Decode Through a Validating Schema

`Schema.fromJsonSchema` turns a `JsonSchema` into a `Schema[Json]`, which we can hand to any code that expects a `Schema`:

```scala
val catalogSchema: Schema[Json] = Schema.fromJsonSchema(catalogContract)
```

Decoding with its `Schema#jsonCodec` runs in two stages. First the codec decodes the JSON text into the structure that the schema describes; then, as it builds the resulting `Json` value, it runs `JsonSchema#check` and fails the decode if the value does not conform. A successful decode returns the payload:

```scala
println(
  catalogSchema.jsonCodec.decode(
    """{"title":"Programming in Scala","firstAuthor":"Martin Odersky","year":2008,"isbn":"9780981531601","tags":["scala"]}"""
  )
)
// Right({
//   "title": "Programming in Scala",
//   "firstAuthor": "Martin Odersky",
//   "year": 2008,
//   "isbn": "9780981531601",
//   "tags": [
//     "scala"
//   ]
// })
```

A payload that breaks the catalog rules fails with the same messages that `JsonSchema#check` reports, followed by `at: .wrapped`, the path of the wrapper that runs the check:

```scala
catalogSchema.jsonCodec.decode(
  """{"title":"","firstAuthor":"Paul Chiusano","year":2014,"isbn":"978-1617290657","tags":["fp","fp"]}"""
) match {
  case Left(error) => println(error.message)
  case Right(json) => println(s"decoded: $json")
}
// Array items are not unique at: .tags
// String does not match pattern '^97[89][0-9]{10}$' at: .isbn
// String length 0 is less than minLength 1 at: .title at: .wrapped
```

:::warning
The first stage treats every property listed in `"properties"` as required, even when `"required"` leaves it out. The Zionomicon submission, which conforms to the catalog schema, fails to decode because it has no `isbn`:

```scala
println(
  catalogSchema.jsonCodec.decode("""{"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024}""")
)
// Left(zio.blocks.schema.SchemaError: missing required field "isbn" at: .wrapped)
```

When payloads may leave out optional properties, parse them with `Json.parse` and validate them with `JsonSchema#check`, as in the previous section.
:::

## Typed Decoding or JSON Schema Validation?

When the payload is meant for our own Scala code, we can skip the JSON Schema and decode straight into the case class with `fromJson`. Decoding the bad Zionomicon submission from earlier this way gives a typed result, but only the first problem the decoder meets:

```scala
"""{"title":"Zionomicon","year":"2024","publisher":"Ziverge"}""".fromJson[BookSubmission] match {
  case Left(error) => println(error.message)
  case Right(book) => println(s"decoded: $book")
}
// illegal number at: .year
```

The two approaches answer different questions:

| Approach                                      | Result                                | Errors reported                         | Needs a Scala type |
| --------------------------------------------- | ------------------------------------- | --------------------------------------- | ------------------ |
| `fromJson[BookSubmission]`                    | `Either[SchemaError, BookSubmission]` | First error only                        | Yes                |
| `Json.parse` then `JsonSchema#check`          | `Option[SchemaError]`                 | All errors at once                      | No                 |
| `Schema.fromJsonSchema(...).jsonCodec.decode` | `Either[SchemaError, Json]`           | First structural error, then all others | No                 |

Decode straight into the case class when the payload is for your own service and you want a `BookSubmission` to work with: the derived contract and the typed decoder accept the same payloads, so validating first adds nothing.

Validate with JSON Schema first when:

- The contract is shared with services written in other languages, and the JSON Schema document is the source of truth that every team checks against.
- The schema arrives at runtime — from a registry, a partner or a configuration file — and there is no case class for it.
- The caller needs every problem in one response, for example in the body of an HTTP 400 reply.
- The code forwards or stores the JSON as it is, such as a gateway or an event log, and never needs a typed value.

## Putting It Together

The complete example derives the contract, writes it to a file, validates payloads against it, loads the catalog team's schema and compares typed decoding with JSON Schema validation:

```scala title="schema-examples/src/main/scala/generatejsonschemaandvalidatepayloads/CompleteExample.scala"
package generatejsonschemaandvalidatepayloads

/**
 * How to Generate JSON Schema from a Type and Validate Payloads Against It —
 * Complete Example
 *
 * Derives the contract, writes it to a file, validates payloads against it,
 * loads the catalog team's schema and compares typed decoding with JSON Schema
 * validation.
 *
 * Run with: cd schema-examples && sbt "runMain
 * generatejsonschemaandvalidatepayloads.CompleteExample"
 */
object CompleteExample extends App {

  @Modifier.noExtraFields()
  case class BookSubmission(
    title: String,
    firstAuthor: String,
    year: Int,
    isbn: Option[String],
    tags: List[String]
  )

  object BookSubmission {
    implicit val schema: Schema[BookSubmission] = Schema.derived
  }

  // Generate and publish the contract
  println("=== Generate and publish the contract ===")
  val contract: JsonSchema = BookSubmission.schema.toJsonSchema

  println(contract)
  println(contract.toJson.print)

  val contractText: String = contract.toJson.print(WriterConfig.withIndentionStep2)

  val contractFile: Path = Files.createTempFile("book-submission", ".schema.json")
  Files.writeString(contractFile, contractText)
  println(s"Wrote contract to $contractFile")

  println(JsonSchema.parse(Files.readString(contractFile)) == Right(contract))

  // Validate untyped payloads
  println("=== Validate untyped payloads ===")
  def validate(schema: JsonSchema, payload: String): Unit =
    Json.parse(payload) match {
      case Left(parseError) => println(s"not JSON: ${parseError.message}")
      case Right(json)      =>
        schema.check(json) match {
          case None        => println("valid")
          case Some(error) => println(error.message)
        }
    }

  // A complete submission passes
  validate(
    contract,
    """{"title":"Programming in Scala","firstAuthor":"Martin Odersky","year":2008,"isbn":"9780981531601","tags":["scala","fundamentals"]}"""
  )

  // A submission without isbn or tags passes too
  validate(contract, """{"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024}""")

  // An unknown field, a string year and a missing author: every problem is reported
  validate(contract, """{"title":"Zionomicon","year":"2024","publisher":"Ziverge"}""")

  // A year outside the Int range
  validate(
    contract,
    """{"title":"Functional Programming in Scala","firstAuthor":"Paul Chiusano","year":20140000000}"""
  )

  // A yes-or-no answer
  println(
    contract.conforms(
      Json
        .parse("""{"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024}""")
        .fold(error => throw error, identity)
    )
  )

  // Load an external schema
  println("=== Load an external schema ===")
  val catalogContractText: String =
    """{
      |  "$schema": "https://json-schema.org/draft/2020-12/schema",
      |  "type": "object",
      |  "properties": {
      |    "title": { "type": "string", "minLength": 1 },
      |    "firstAuthor": { "type": "string", "minLength": 1 },
      |    "year": { "type": "integer", "minimum": 1450 },
      |    "isbn": { "type": "string", "pattern": "^97[89][0-9]{10}$" },
      |    "tags": { "type": "array", "items": { "type": "string" }, "uniqueItems": true }
      |  },
      |  "required": ["title", "firstAuthor", "year"],
      |  "additionalProperties": false
      |}""".stripMargin

  val catalogContract: JsonSchema =
    JsonSchema.parse(catalogContractText).fold(error => throw error, identity)

  validate(catalogContract, """{"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024}""")

  validate(
    catalogContract,
    """{"title":"","firstAuthor":"Paul Chiusano","year":2014,"isbn":"978-1617290657","tags":["fp","fp"]}"""
  )

  val catalogSchema: Schema[Json] = Schema.fromJsonSchema(catalogContract)

  println(
    catalogSchema.jsonCodec.decode(
      """{"title":"Programming in Scala","firstAuthor":"Martin Odersky","year":2008,"isbn":"9780981531601","tags":["scala"]}"""
    )
  )

  catalogSchema.jsonCodec.decode(
    """{"title":"","firstAuthor":"Paul Chiusano","year":2014,"isbn":"978-1617290657","tags":["fp","fp"]}"""
  ) match {
    case Left(error) => println(error.message)
    case Right(json) => println(s"decoded: $json")
  }

  // The decoder treats every listed property as required, so this conforming payload fails without an isbn
  println(
    catalogSchema.jsonCodec.decode("""{"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024}""")
  )

  // Typed decoding versus JSON Schema validation
  println("=== Typed decoding versus JSON Schema validation ===")
  val badPayload = """{"title":"Zionomicon","year":"2024","publisher":"Ziverge"}"""

  println("Typed decoding (first error only):")
  badPayload.fromJson[BookSubmission] match {
    case Left(error) => println(error.message)
    case Right(book) => println(s"decoded: $book")
  }

  println("JSON Schema validation (all errors):")
  Json.parse(badPayload) match {
    case Left(parseError) => println(s"not JSON: ${parseError.message}")
    case Right(json)      =>
      contract.check(json) match {
        case None        => println("valid")
        case Some(error) => println(error.message)
      }
  }

}
```

## Running the Examples

All examples in this guide have corresponding runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

Run them in order to follow the guide step by step.

### Generate and Publish the Contract

This example derives the JSON Schema contract for `BookSubmission`, prints it pretty and compact, writes it to a temporary file and parses the file back.

<details>
  <summary>schema-examples/src/main/scala/generatejsonschemaandvalidatepayloads/Step1GenerateContractExample.scala</summary>

```scala title="schema-examples/src/main/scala/generatejsonschemaandvalidatepayloads/Step1GenerateContractExample.scala" showLineNumbers
package generatejsonschemaandvalidatepayloads

/**
 * How to Generate JSON Schema from a Type and Validate Payloads Against It —
 * Generate and Publish the Contract
 *
 * Derives the JSON Schema contract for BookSubmission, prints it pretty and
 * compact, writes it to a temporary file and parses the file back.
 *
 * Run with: cd schema-examples && sbt "runMain
 * generatejsonschemaandvalidatepayloads.Step1GenerateContractExample"
 */
object Step1GenerateContractExample extends App {

  @Modifier.noExtraFields()
  case class BookSubmission(
    title: String,
    firstAuthor: String,
    year: Int,
    isbn: Option[String],
    tags: List[String]
  )

  object BookSubmission {
    implicit val schema: Schema[BookSubmission] = Schema.derived
  }

  val contract: JsonSchema = BookSubmission.schema.toJsonSchema

  println(contract)
  println(contract.toJson.print)

  val contractText: String = contract.toJson.print(WriterConfig.withIndentionStep2)

  val contractFile: Path = Files.createTempFile("book-submission", ".schema.json")
  Files.writeString(contractFile, contractText)
  println(s"Wrote contract to $contractFile")

  println(JsonSchema.parse(Files.readString(contractFile)) == Right(contract))

}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain generatejsonschemaandvalidatepayloads.Step1GenerateContractExample"
```

### Validate Untyped Payloads

This example parses JSON text with `Json.parse` and validates it against the contract that `Schema#toJsonSchema` derives, using `JsonSchema#check`, printing the result for valid submissions and for a missing field, a wrong type, an unknown field and an out-of-range year.

<details>
  <summary>schema-examples/src/main/scala/generatejsonschemaandvalidatepayloads/Step2ValidatePayloadsExample.scala</summary>

```scala title="schema-examples/src/main/scala/generatejsonschemaandvalidatepayloads/Step2ValidatePayloadsExample.scala" showLineNumbers
package generatejsonschemaandvalidatepayloads

/**
 * How to Generate JSON Schema from a Type and Validate Payloads Against It —
 * Validate Untyped Payloads
 *
 * Parses JSON text with Json.parse and validates it against the generated
 * contract with JsonSchema#check for valid and invalid submissions.
 *
 * Run with: cd schema-examples && sbt "runMain
 * generatejsonschemaandvalidatepayloads.Step2ValidatePayloadsExample"
 */
object Step2ValidatePayloadsExample extends App {

  @Modifier.noExtraFields()
  case class BookSubmission(
    title: String,
    firstAuthor: String,
    year: Int,
    isbn: Option[String],
    tags: List[String]
  )

  object BookSubmission {
    implicit val schema: Schema[BookSubmission] = Schema.derived
  }

  val contract: JsonSchema = BookSubmission.schema.toJsonSchema

  def validate(schema: JsonSchema, payload: String): Unit =
    Json.parse(payload) match {
      case Left(parseError) => println(s"not JSON: ${parseError.message}")
      case Right(json)      =>
        schema.check(json) match {
          case None        => println("valid")
          case Some(error) => println(error.message)
        }
    }

  // A complete submission passes
  validate(
    contract,
    """{"title":"Programming in Scala","firstAuthor":"Martin Odersky","year":2008,"isbn":"9780981531601","tags":["scala","fundamentals"]}"""
  )

  // A submission without isbn or tags passes too
  validate(contract, """{"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024}""")

  // An unknown field, a string year and a missing author: every problem is reported
  validate(contract, """{"title":"Zionomicon","year":"2024","publisher":"Ziverge"}""")

  // A year outside the Int range
  validate(
    contract,
    """{"title":"Functional Programming in Scala","firstAuthor":"Paul Chiusano","year":20140000000}"""
  )

  // A yes-or-no answer
  println(
    contract.conforms(
      Json
        .parse("""{"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024}""")
        .fold(error => throw error, identity)
    )
  )

}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain generatejsonschemaandvalidatepayloads.Step2ValidatePayloadsExample"
```

### Load an External Schema

This example parses the catalog team's JSON Schema with `JsonSchema.parse`, checks payloads against it, and decodes payloads through `Schema.fromJsonSchema`, including the payload without an `isbn` that the decoder rejects.

<details>
  <summary>schema-examples/src/main/scala/generatejsonschemaandvalidatepayloads/Step3ExternalSchemaExample.scala</summary>

```scala title="schema-examples/src/main/scala/generatejsonschemaandvalidatepayloads/Step3ExternalSchemaExample.scala" showLineNumbers
package generatejsonschemaandvalidatepayloads

/**
 * How to Generate JSON Schema from a Type and Validate Payloads Against It —
 * Load an External Schema
 *
 * Parses the catalog team's JSON Schema, checks payloads against it, and
 * decodes payloads through Schema.fromJsonSchema.
 *
 * Run with: cd schema-examples && sbt "runMain
 * generatejsonschemaandvalidatepayloads.Step3ExternalSchemaExample"
 */
object Step3ExternalSchemaExample extends App {

  @Modifier.noExtraFields()
  case class BookSubmission(
    title: String,
    firstAuthor: String,
    year: Int,
    isbn: Option[String],
    tags: List[String]
  )

  object BookSubmission {
    implicit val schema: Schema[BookSubmission] = Schema.derived
  }

  def validate(schema: JsonSchema, payload: String): Unit =
    Json.parse(payload) match {
      case Left(parseError) => println(s"not JSON: ${parseError.message}")
      case Right(json)      =>
        schema.check(json) match {
          case None        => println("valid")
          case Some(error) => println(error.message)
        }
    }

  val catalogContractText: String =
    """{
      |  "$schema": "https://json-schema.org/draft/2020-12/schema",
      |  "type": "object",
      |  "properties": {
      |    "title": { "type": "string", "minLength": 1 },
      |    "firstAuthor": { "type": "string", "minLength": 1 },
      |    "year": { "type": "integer", "minimum": 1450 },
      |    "isbn": { "type": "string", "pattern": "^97[89][0-9]{10}$" },
      |    "tags": { "type": "array", "items": { "type": "string" }, "uniqueItems": true }
      |  },
      |  "required": ["title", "firstAuthor", "year"],
      |  "additionalProperties": false
      |}""".stripMargin

  val catalogContract: JsonSchema =
    JsonSchema.parse(catalogContractText).fold(error => throw error, identity)

  validate(catalogContract, """{"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024}""")

  validate(
    catalogContract,
    """{"title":"","firstAuthor":"Paul Chiusano","year":2014,"isbn":"978-1617290657","tags":["fp","fp"]}"""
  )

  val catalogSchema: Schema[Json] = Schema.fromJsonSchema(catalogContract)

  println(
    catalogSchema.jsonCodec.decode(
      """{"title":"Programming in Scala","firstAuthor":"Martin Odersky","year":2008,"isbn":"9780981531601","tags":["scala"]}"""
    )
  )

  catalogSchema.jsonCodec.decode(
    """{"title":"","firstAuthor":"Paul Chiusano","year":2014,"isbn":"978-1617290657","tags":["fp","fp"]}"""
  ) match {
    case Left(error) => println(error.message)
    case Right(json) => println(s"decoded: $json")
  }

  // The decoder treats every listed property as required, so this conforming payload fails without an isbn
  println(
    catalogSchema.jsonCodec.decode("""{"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024}""")
  )

}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain generatejsonschemaandvalidatepayloads.Step3ExternalSchemaExample"
```

### Compare Typed Decoding with Validation

This example decodes the same bad payload with `fromJson[BookSubmission]` and with `JsonSchema#check`, showing the first error from the typed decoder next to every error from the JSON Schema.

<details>
  <summary>schema-examples/src/main/scala/generatejsonschemaandvalidatepayloads/Step4TypedVersusUntypedExample.scala</summary>

```scala title="schema-examples/src/main/scala/generatejsonschemaandvalidatepayloads/Step4TypedVersusUntypedExample.scala" showLineNumbers
package generatejsonschemaandvalidatepayloads

/**
 * How to Generate JSON Schema from a Type and Validate Payloads Against It —
 * Compare Typed Decoding with Validation
 *
 * Decodes one bad payload with fromJson[BookSubmission] and validates it with
 * JsonSchema#check.
 *
 * Run with: cd schema-examples && sbt "runMain
 * generatejsonschemaandvalidatepayloads.Step4TypedVersusUntypedExample"
 */
object Step4TypedVersusUntypedExample extends App {

  @Modifier.noExtraFields()
  case class BookSubmission(
    title: String,
    firstAuthor: String,
    year: Int,
    isbn: Option[String],
    tags: List[String]
  )

  object BookSubmission {
    implicit val schema: Schema[BookSubmission] = Schema.derived
  }

  val contract: JsonSchema = BookSubmission.schema.toJsonSchema

  val badPayload = """{"title":"Zionomicon","year":"2024","publisher":"Ziverge"}"""

  println("Typed decoding (first error only):")
  badPayload.fromJson[BookSubmission] match {
    case Left(error) => println(error.message)
    case Right(book) => println(s"decoded: $book")
  }

  println("JSON Schema validation (all errors):")
  Json.parse(badPayload) match {
    case Left(parseError) => println(s"not JSON: ${parseError.message}")
    case Right(json)      =>
      contract.check(json) match {
        case None        => println("valid")
        case Some(error) => println(error.message)
      }
  }

}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain generatejsonschemaandvalidatepayloads.Step4TypedVersusUntypedExample"
```

### The Complete Example

The complete example from "Putting It Together" runs everything from the guide in one program:

```bash
sbt "runMain generatejsonschemaandvalidatepayloads.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## Going Further

- [JSON Schema](../reference/schema/built-in-codecs/json/json-schema.md) covers building schemas by hand, the combinators, every supported keyword and the known limitations.
- [Dynamic Schema](../reference/schema/dynamic-schema.md) validates `DynamicValue`s against a schema with `DynamicSchema#check` and `DynamicSchema#conforms`; `schema-examples/src/main/scala/dynamicschema/DynamicSchemaValidationExample.scala` shows it end to end.
- [Your First Schema](./your-first-schema.md) introduces `Schema`, `fromJson` and `SchemaError` from the beginning.

---

## Generate Scala Code from a Smithy Model

Welcome! This tutorial is for Scala developers who know case classes, sealed traits and pattern matching, and who have derived a `Schema` before (if you haven't, read [Your First Schema](./your-first-schema.md) first). You don't need any prior experience with Smithy or with code generation to follow along.

## 1. Introduction

By the end of this tutorial, you will understand:

- How to parse a Smithy model into a `SmithyModel` and walk its shapes
- How Smithy structures, unions, enums, lists, maps and optional members map to Scala types
- How the codegen module describes Scala source as data, and how `ScalaEmitter` turns that data into Scala 3 or Scala 2 code
- How to make the generated types carry a `Schema`, so they encode to JSON and back
- How to run the generator from sbt, so the generated code is compiled with the rest of your build

We learn these concepts by writing a small generator for an order service's data contract, one step at a time:

1. Parse the model
2. Map Smithy types to Scala types
3. Generate your first case class
4. Turn unions and enums into sealed types
5. Emit a whole file for Scala 3 and Scala 2
6. Add schema derivation
7. Put the generator together
8. Compile the generated code with sbt

We recommend reading from top to bottom — each section builds on the previous one.

To follow along, add the smithy and codegen modules to your `build.sbt`. The code we generate also needs the schema module:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-smithy"  % "0.0.56",
  "dev.zio" %% "zio-blocks-codegen" % "0.0.56",
  "dev.zio" %% "zio-blocks-schema"  % "0.0.56"
)

// Scala 3.9 and newer
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-smithy"  % "0.0.56",
  "dev.zio" %% "zio-blocks-next-codegen" % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema"  % "0.0.56"
)
scalacOptions += "-experimental"
```

The smithy and codegen modules are published for the JVM only. The code in this tutorial is compiled with Scala 3; for scala-cli or cross-building, see [How to Add ZIO Blocks to Your Build](../getting-started/installation.md).

## 2. Background: One Contract, Many Programs

Our order service and the teams that call it agree on the shape of an order: an id, a list of line items, a status and an optional note. That agreement is a data contract. If each team writes its own Scala case classes for it, the copies drift apart: one team adds a field, another renames one, and the mistake only shows up when a message fails to decode in production. Smithy is a language for writing such contracts once, in a `.smithy` file, and code generation turns that file into types, so every program compiles against the same definition.

ZIO Blocks has two modules for this, and they don't know about each other. The [smithy module](../reference/smithy.md) parses a `.smithy` file into a `SmithyModel`, a plain Scala value that lists every shape. The [codegen module](../reference/codegen/index.md) describes Scala source as data — a `ScalaFile` holding `CaseClass`, `SealedTrait` and `Enum` values — and its `ScalaEmitter` turns that data into source text. The glue between them, which decides that a Smithy structure becomes a case class, is code we write ourselves in this tutorial:

```
orders.smithy ──SmithyModel.parse──▶ SmithyModel ──our mapping──▶ ScalaFile ──ScalaEmitter.emit──▶ Orders.scala
   (text)                            (shapes)                     (Scala as data)                 (text)
```

If you need a complete Smithy toolchain, [smithy4s](https://disneystreaming.github.io/smithy4s/) generates data types and services, clients and servers too. This tutorial builds something smaller: a generator you own and can read in one sitting, covering data shapes only, whose output carries ZIO Blocks schemas.

Next step: [How to Generate Service Traits from Smithy Operations or OpenAPI Paths](./generate-service-traits-from-smithy-or-openapi.md) maps each operation of the service to a method of a Scala trait.

## 3. Parse the Model

Let's start with the contract itself. A Smithy file begins with a `$version` line and a `namespace`, followed by shapes: a `structure` is a record with named members, a `list` and a `map` are collections, a `union` holds exactly one of several members, and an `enum` is a fixed set of names. Members are optional unless they carry the `@required` trait, and a `///` line is a documentation comment.

We keep the model in a string so the tutorial runs anywhere; in section 9 our generator reads the same text from an `orders.smithy` file:

```scala

val ordersSmithy: String =
  """$version: "2"
    |
    |namespace com.acme.orders
    |
    |/// An order placed by a customer.
    |structure Order {
    |    @required
    |    id: String
    |
    |    @required
    |    items: LineItems
    |
    |    @required
    |    status: OrderStatus
    |
    |    @required
    |    priority: Priority
    |
    |    /// Free-text instructions from the customer.
    |    note: String
    |
    |    attributes: Attributes
    |}
    |
    |list LineItems {
    |    member: LineItem
    |}
    |
    |map Attributes {
    |    key: String
    |    value: String
    |}
    |
    |/// One product line in an order.
    |structure LineItem {
    |    @required
    |    sku: String
    |
    |    @required
    |    quantity: Integer
    |
    |    @required
    |    unitPrice: BigDecimal
    |}
    |
    |/// Where an order is in its life cycle.
    |union OrderStatus {
    |    placed: Unit
    |    shipped: Shipment
    |    cancelled: Cancellation
    |}
    |
    |structure Shipment {
    |    @required
    |    trackingNumber: String
    |
    |    @required
    |    shippedAt: Timestamp
    |}
    |
    |structure Cancellation {
    |    @required
    |    reason: String
    |}
    |
    |/// How fast the order ships.
    |enum Priority {
    |    STANDARD
    |    EXPRESS
    |}
    |""".stripMargin
```

The three imports bring in the Smithy parser and model (`zio.blocks.smithy`), the codegen data types (`zio.blocks.codegen.ir`) and the emitter (`zio.blocks.codegen.emit`); we use all of them in later sections.

Parsing can fail, so `SmithyModel.parse` returns an `Either[SmithyError, SmithyModel]`. A generator can't do anything useful with a broken contract, so we stop with the parser's message:

```scala
val model: SmithyModel = SmithyModel.parse(ordersSmithy) match {
  case Right(parsed) => parsed
  case Left(error)   => throw new IllegalArgumentException(error.formatMessage)
}
```

The `SmithyError#formatMessage` method includes the line and column of the problem. For example, a file that forgets its `$version` line fails like this:

```scala
SmithyModel.parse("namespace com.acme.orders").left.map(_.formatMessage)
// res0: Either[String, SmithyModel] = Left(
//   "Parse error at line 1, column 1: Expected '$version' declaration"
// )
```

A parsed model is a list of `ShapeDefinition` values, each holding a shape's name and the shape itself. Let's print every shape and its kind:

```scala
model.shapes.foreach { definition =>
  println(s"${definition.name}: ${definition.shape.getClass.getSimpleName}")
}
// Order: StructureShape
// LineItems: ListShape
// Attributes: MapShape
// LineItem: StructureShape
// OrderStatus: UnionShape
// Shipment: StructureShape
// Cancellation: StructureShape
// Priority: EnumShape
```

Each Smithy shape type has its own Scala class: `StructureShape`, `ListShape`, `MapShape`, `UnionShape`, `EnumShape` and one class per simple type. Now let's look inside `Order`. First we pick its `StructureShape` out of the model:

```scala
val orderShape: StructureShape =
  model.shapes.collectFirst { case ShapeDefinition("Order", shape: StructureShape) => shape }.get
```

Then we print each member's target — the shape its value has — and whether it is required:

```scala
orderShape.members.foreach { member =>
  val required = member.traits.contains(TraitApplication.required)
  println(s"${member.name}: target=${member.target}, required=$required")
}
// id: target=#String, required=true
// items: target=#LineItems, required=true
// status: target=#OrderStatus, required=true
// priority: target=#Priority, required=true
// note: target=#String, required=false
// attributes: target=#Attributes, required=false
```

The code above:

- `model.shapes.collectFirst { ... }` finds the definition named `Order` and keeps its `StructureShape`.
- `member.target` is a `ShapeId`, a namespace and a name.
- `TraitApplication.required` is the parsed form of `@required`, so `contains` tells us whether a member carries it.

Notice the targets: they print as `#String` and `#LineItems`, with nothing before the `#`. The parser keeps a name exactly as the file spells it, so an unqualified name like `String` gets an empty namespace, whether it means a built-in Smithy type or a shape from our own file. Our generator therefore looks a target up by its name: first among the model's shapes, then among Smithy's built-in types.

## 4. Map Smithy Types to Scala Types

Every member needs a Scala type. In the codegen module a type is a `TypeRef`: a name plus type arguments, so `TypeRef.list(TypeRef.String)` stands for `List[String]`. Smithy's built-in simple types map to these Scala types:

| Smithy type  | Scala type                        |
| ------------ | --------------------------------- |
| `String`     | `String`                          |
| `Boolean`    | `Boolean`                         |
| `Byte`       | `Byte`                            |
| `Short`      | `Short`                           |
| `Integer`    | `Int`                             |
| `Long`       | `Long`                            |
| `Float`      | `Float`                           |
| `Double`     | `Double`                          |
| `BigInteger` | `BigInt`                          |
| `BigDecimal` | `BigDecimal`                      |
| `Timestamp`  | `java.time.Instant`               |
| `Blob`       | `zio.blocks.chunk.Chunk[Byte]`    |
| `Document`   | `zio.blocks.schema.DynamicValue`  |

Every type in the table has a built-in `Schema`, which matters in section 8. We write the last three with their full package names, so the generated file needs no extra imports.

The table becomes a function that returns `None` for a name it doesn't know:

```scala
def simpleTypeRef(name: String): Option[TypeRef] = name match {
  case "String"     => Some(TypeRef.String)
  case "Boolean"    => Some(TypeRef.Boolean)
  case "Byte"       => Some(TypeRef.Byte)
  case "Short"      => Some(TypeRef.Short)
  case "Integer"    => Some(TypeRef.Int)
  case "Long"       => Some(TypeRef.Long)
  case "Float"      => Some(TypeRef.Float)
  case "Double"     => Some(TypeRef.Double)
  case "BigInteger" => Some(TypeRef.BigInt)
  case "BigDecimal" => Some(TypeRef.BigDecimal)
  case "Timestamp"  => Some(TypeRef("java.time.Instant"))
  case "Blob"       => Some(TypeRef.of("zio.blocks.chunk.Chunk", TypeRef.Byte))
  case "Document"   => Some(TypeRef("zio.blocks.schema.DynamicValue"))
  case _            => None
}
```

Shapes defined in our file need more thought. A list or a map gets no Scala type of its own: a member that targets `LineItems` becomes a `List[LineItem]` directly. Structures, unions and enums become named Scala types, so a member that targets one refers to it by name. We turn any other shape into an error, so the generator never writes code it can't stand behind:

```scala
def typeRefFor(model: SmithyModel, target: ShapeId): TypeRef =
  model.findShape(target.name).map(_.shape) match {
    case Some(list: ListShape) =>
      TypeRef.list(typeRefFor(model, list.member.target))
    case Some(map: MapShape) =>
      TypeRef.map(typeRefFor(model, map.key.target), typeRefFor(model, map.value.target))
    case Some(_: StructureShape) | Some(_: UnionShape) | Some(_: EnumShape) =>
      TypeRef(target.name)
    case Some(other) =>
      throw new IllegalArgumentException(s"Unsupported shape: ${other.name}")
    case None =>
      simpleTypeRef(target.name).getOrElse(throw new IllegalArgumentException(s"Unknown shape: $target"))
  }
```

The code above:

- `SmithyModel#findShape` looks up a shape of our own file by name, and returns `None` for built-in types such as `String`.
- `ListShape#member` and `MapShape#key`/`MapShape#value` are members too, so `typeRefFor` calls itself on their targets.
- `TypeRef.list` and `TypeRef.map` build `List[...]` and `Map[..., ...]`.
- The last case falls back to the table, and fails loudly on a name it can't map.

To check the mapping, let's render the type of each `Order` member with `ScalaEmitter.emitTypeRef`, which turns a single `TypeRef` into source text:

```scala
orderShape.members.foreach { member =>
  println(s"${member.name}: ${ScalaEmitter.emitTypeRef(typeRefFor(model, member.target))}")
}
// id: String
// items: List[LineItem]
// status: OrderStatus
// priority: Priority
// note: String
// attributes: Map[String, String]
```

The list and the map have disappeared into `List[LineItem]` and `Map[String, String]`. The optional members don't show `Option` yet; that is the job of the next section.

<details>
  <summary>Runnable version, with setup from section 3: <code>Concept4MapTypesExample.scala</code></summary>

```scala title="smithy-examples/src/main/scala/generatescalafromsmithy/Concept4MapTypesExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-smithy:0.0.56
//> using dep dev.zio::zio-blocks-codegen:0.0.56
//> using options -experimental
//> using file OrdersModel.scala
// Run from this folder:  scala-cli run Concept4MapTypesExample.scala OrdersModel.scala
// Or from smithy-examples:  sbt "runMain generatescalafromsmithy.Concept4MapTypesExample"

package generatescalafromsmithy

/**
 * Generate Scala Code from a Smithy Model — 4. Map Smithy Types to Scala Types
 *
 * Maps Smithy shapes to codegen `TypeRef` values and renders the type of each
 * `Order` member.
 */
object Concept4MapTypesExample {
  def main(args: Array[String]): Unit = {
    // Setup from section 3: the parsed model and the Order structure

    val model: SmithyModel = SmithyModel.parse(ordersSmithy) match {
      case Right(parsed) => parsed
      case Left(error)   => throw new IllegalArgumentException(error.formatMessage)
    }

    val orderShape: StructureShape =
      model.shapes.collectFirst { case ShapeDefinition("Order", shape: StructureShape) => shape }.get

    def simpleTypeRef(name: String): Option[TypeRef] = name match {
      case "String"     => Some(TypeRef.String)
      case "Boolean"    => Some(TypeRef.Boolean)
      case "Byte"       => Some(TypeRef.Byte)
      case "Short"      => Some(TypeRef.Short)
      case "Integer"    => Some(TypeRef.Int)
      case "Long"       => Some(TypeRef.Long)
      case "Float"      => Some(TypeRef.Float)
      case "Double"     => Some(TypeRef.Double)
      case "BigInteger" => Some(TypeRef.BigInt)
      case "BigDecimal" => Some(TypeRef.BigDecimal)
      case "Timestamp"  => Some(TypeRef("java.time.Instant"))
      case "Blob"       => Some(TypeRef.of("zio.blocks.chunk.Chunk", TypeRef.Byte))
      case "Document"   => Some(TypeRef("zio.blocks.schema.DynamicValue"))
      case _            => None
    }

    def typeRefFor(model: SmithyModel, target: ShapeId): TypeRef =
      model.findShape(target.name).map(_.shape) match {
        case Some(list: ListShape) =>
          TypeRef.list(typeRefFor(model, list.member.target))
        case Some(map: MapShape) =>
          TypeRef.map(typeRefFor(model, map.key.target), typeRefFor(model, map.value.target))
        case Some(_: StructureShape) | Some(_: UnionShape) | Some(_: EnumShape) =>
          TypeRef(target.name)
        case Some(other) =>
          throw new IllegalArgumentException(s"Unsupported shape: ${other.name}")
        case None =>
          simpleTypeRef(target.name).getOrElse(throw new IllegalArgumentException(s"Unknown shape: $target"))
      }

    orderShape.members.foreach { member =>
      println(s"${member.name}: ${ScalaEmitter.emitTypeRef(typeRefFor(model, member.target))}")
    }
  }
}
```

</details>

## 5. Generate Your First Case Class

A Smithy structure becomes a case class. In the codegen module a case class is a `CaseClass` value with a name and a list of `Field`s, and each `Field` has a name, a `TypeRef` and an optional default value. A required member becomes a plain field. Any other member may be missing, so it becomes an `Option` with a default of `None`, which lets callers leave it out.

Let's write the mapping, one small function per idea:

```scala
def docOf(traits: List[TraitApplication]): Option[String] =
  traits.collectFirst {
    case TraitApplication(ShapeId("smithy.api", "documentation"), Some(NodeValue.String(text))) => text
  }

def fieldFor(model: SmithyModel, member: MemberDefinition): Field = {
  val fieldType = typeRefFor(model, member.target)
  if (member.traits.contains(TraitApplication.required)) Field(member.name, fieldType)
  else Field(member.name, TypeRef.optional(fieldType), defaultValue = Some("None"))
}

def caseClassFor(model: SmithyModel, name: String, shape: StructureShape): CaseClass =
  CaseClass(name, shape.members.map(fieldFor(model, _)), doc = docOf(shape.traits))
```

The code above:

- `docOf` finds the `@documentation` trait. The parser turns every `///` comment into this trait, with the comment's text as a `NodeValue.String`.
- `fieldFor` keeps a required member's type and wraps any other in `TypeRef.optional`, with the default value `None` written as source text.
- `caseClassFor` turns every member into a field and carries the structure's documentation over as a Scaladoc comment.

A `CaseClass` is only data. To see source, we wrap it in a `ScalaFile` with a package and emit it:

```scala
println(ScalaEmitter.emit(ScalaFile(PackageDecl(model.namespace), types = List(caseClassFor(model, "Order", orderShape)))))
// package com.acme.orders
// 
// /** An order placed by a customer. */
// case class Order(
//   id: String,
//   items: List[LineItem],
//   status: OrderStatus,
//   priority: Priority,
//   note: Option[String] = None,
//   attributes: Option[Map[String, String]] = None,
// )
//
```

That's our first generated type. `ScalaEmitter.emit` takes the `ScalaFile` and an `EmitterConfig`; we leave the config out, so it uses `EmitterConfig.default`, which writes Scala 3 code (the trailing comma after the last field is Scala 3 style). The package name comes straight from the Smithy namespace.

:::note
The `///` comment on the `note` member doesn't reach the generated code: in this release `ScalaEmitter` doesn't render documentation on fields, only on types, so `fieldFor` doesn't copy it.
:::

<details>
  <summary>Runnable version, with setup from sections 3–4: <code>Concept5CaseClassExample.scala</code></summary>

```scala title="smithy-examples/src/main/scala/generatescalafromsmithy/Concept5CaseClassExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-smithy:0.0.56
//> using dep dev.zio::zio-blocks-codegen:0.0.56
//> using options -experimental
//> using file OrdersModel.scala
// Run from this folder:  scala-cli run Concept5CaseClassExample.scala OrdersModel.scala
// Or from smithy-examples:  sbt "runMain generatescalafromsmithy.Concept5CaseClassExample"

package generatescalafromsmithy

/**
 * Generate Scala Code from a Smithy Model — 5. Generate Your First Case Class
 *
 * Turns the `Order` structure into a `CaseClass` and emits it as Scala 3
 * source.
 */
object Concept5CaseClassExample {
  def main(args: Array[String]): Unit = {
    // Setup from sections 3–4: the parsed model, the Order structure and the type mapping

    val model: SmithyModel = SmithyModel.parse(ordersSmithy) match {
      case Right(parsed) => parsed
      case Left(error)   => throw new IllegalArgumentException(error.formatMessage)
    }

    val orderShape: StructureShape =
      model.shapes.collectFirst { case ShapeDefinition("Order", shape: StructureShape) => shape }.get

    def simpleTypeRef(name: String): Option[TypeRef] = name match {
      case "String"     => Some(TypeRef.String)
      case "Boolean"    => Some(TypeRef.Boolean)
      case "Byte"       => Some(TypeRef.Byte)
      case "Short"      => Some(TypeRef.Short)
      case "Integer"    => Some(TypeRef.Int)
      case "Long"       => Some(TypeRef.Long)
      case "Float"      => Some(TypeRef.Float)
      case "Double"     => Some(TypeRef.Double)
      case "BigInteger" => Some(TypeRef.BigInt)
      case "BigDecimal" => Some(TypeRef.BigDecimal)
      case "Timestamp"  => Some(TypeRef("java.time.Instant"))
      case "Blob"       => Some(TypeRef.of("zio.blocks.chunk.Chunk", TypeRef.Byte))
      case "Document"   => Some(TypeRef("zio.blocks.schema.DynamicValue"))
      case _            => None
    }

    def typeRefFor(model: SmithyModel, target: ShapeId): TypeRef =
      model.findShape(target.name).map(_.shape) match {
        case Some(list: ListShape) =>
          TypeRef.list(typeRefFor(model, list.member.target))
        case Some(map: MapShape) =>
          TypeRef.map(typeRefFor(model, map.key.target), typeRefFor(model, map.value.target))
        case Some(_: StructureShape) | Some(_: UnionShape) | Some(_: EnumShape) =>
          TypeRef(target.name)
        case Some(other) =>
          throw new IllegalArgumentException(s"Unsupported shape: ${other.name}")
        case None =>
          simpleTypeRef(target.name).getOrElse(throw new IllegalArgumentException(s"Unknown shape: $target"))
      }

    def docOf(traits: List[TraitApplication]): Option[String] =
      traits.collectFirst {
        case TraitApplication(ShapeId("smithy.api", "documentation"), Some(NodeValue.String(text))) => text
      }

    def fieldFor(model: SmithyModel, member: MemberDefinition): Field = {
      val fieldType = typeRefFor(model, member.target)
      if (member.traits.contains(TraitApplication.required)) Field(member.name, fieldType)
      else Field(member.name, TypeRef.optional(fieldType), defaultValue = Some("None"))
    }

    def caseClassFor(model: SmithyModel, name: String, shape: StructureShape): CaseClass =
      CaseClass(name, shape.members.map(fieldFor(model, _)), doc = docOf(shape.traits))

    println(
      ScalaEmitter.emit(ScalaFile(PackageDecl(model.namespace), types = List(caseClassFor(model, "Order", orderShape))))
    )
  }
}
```

</details>

## 6. Turn Unions and Enums into Sealed Types

A Smithy union holds exactly one of its members, which is what a Scala sealed trait expresses. Each union member becomes one case of the sealed trait: a member that targets `Unit` carries no data and becomes a case object, and any other member becomes a case class with one `value` field. A Smithy enum is a fixed set of names, so it becomes the codegen `Enum` type.

Let's write both mappings:

```scala
def sealedTraitFor(model: SmithyModel, name: String, shape: UnionShape): SealedTrait = {
  val cases = shape.members.map { member =>
    val caseName = member.name.capitalize
    if (member.target.name == "Unit") SealedTraitCase.CaseObjectCase(caseName)
    else SealedTraitCase.CaseClassCase(CaseClass(caseName, List(Field("value", typeRefFor(model, member.target)))))
  }
  SealedTrait(name, cases = cases, doc = docOf(shape.traits))
}

def enumFor(name: String, shape: EnumShape): Enum =
  Enum(name, shape.members.map(member => EnumCase.SimpleCase(member.name)), doc = docOf(shape.traits))
```

The code above:

- `member.name.capitalize` turns the Smithy member `shipped` into the Scala case `Shipped`.
- `SealedTraitCase.CaseObjectCase` and `SealedTraitCase.CaseClassCase` are the two kinds of case a `SealedTrait` can have; we don't add `extends OrderStatus` ourselves, because the emitter adds it to every case.
- `EnumCase.SimpleCase` is an enum case without fields, one per Smithy enum member.

Next, we generate both types from the model:

```scala
val statusTrait: SealedTrait =
  model.shapes.collectFirst { case ShapeDefinition(name, shape: UnionShape) => sealedTraitFor(model, name, shape) }.get

val priorityEnum: Enum =
  model.shapes.collectFirst { case ShapeDefinition(name, shape: EnumShape) => enumFor(name, shape) }.get
```

Then we emit them together in one file:

```scala
println(ScalaEmitter.emit(ScalaFile(PackageDecl(model.namespace), types = List(statusTrait, priorityEnum))))
// package com.acme.orders
// 
// /** Where an order is in its life cycle. */
// sealed trait OrderStatus
// 
// object OrderStatus {
//   case object Placed extends OrderStatus
//   case class Shipped(
//     value: Shipment,
//   ) extends OrderStatus
//   case class Cancelled(
//     value: Cancellation,
//   ) extends OrderStatus
// }
// 
// /** How fast the order ships. */
// enum Priority {
//   case STANDARD, EXPRESS
// }
//
```

The emitter places the cases inside the companion object `OrderStatus`, so code that uses them writes `OrderStatus.Shipped(...)`. `Priority` comes out as a Scala 3 `enum`, because we are still emitting with the default, Scala 3 configuration.

<details>
  <summary>Runnable version, with setup from sections 3–5: <code>Concept6SealedTypesExample.scala</code></summary>

```scala title="smithy-examples/src/main/scala/generatescalafromsmithy/Concept6SealedTypesExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-smithy:0.0.56
//> using dep dev.zio::zio-blocks-codegen:0.0.56
//> using options -experimental
//> using file OrdersModel.scala
// Run from this folder:  scala-cli run Concept6SealedTypesExample.scala OrdersModel.scala
// Or from smithy-examples:  sbt "runMain generatescalafromsmithy.Concept6SealedTypesExample"

package generatescalafromsmithy

/**
 * Generate Scala Code from a Smithy Model — 6. Turn Unions and Enums into
 * Sealed Types
 *
 * Turns the `OrderStatus` union into a `SealedTrait` and the `Priority` enum
 * into an `Enum`, and emits both.
 */
object Concept6SealedTypesExample {
  def main(args: Array[String]): Unit = {
    // Setup from sections 3–5: the parsed model, the type mapping and docOf

    val model: SmithyModel = SmithyModel.parse(ordersSmithy) match {
      case Right(parsed) => parsed
      case Left(error)   => throw new IllegalArgumentException(error.formatMessage)
    }

    val orderShape: StructureShape =
      model.shapes.collectFirst { case ShapeDefinition("Order", shape: StructureShape) => shape }.get

    def simpleTypeRef(name: String): Option[TypeRef] = name match {
      case "String"     => Some(TypeRef.String)
      case "Boolean"    => Some(TypeRef.Boolean)
      case "Byte"       => Some(TypeRef.Byte)
      case "Short"      => Some(TypeRef.Short)
      case "Integer"    => Some(TypeRef.Int)
      case "Long"       => Some(TypeRef.Long)
      case "Float"      => Some(TypeRef.Float)
      case "Double"     => Some(TypeRef.Double)
      case "BigInteger" => Some(TypeRef.BigInt)
      case "BigDecimal" => Some(TypeRef.BigDecimal)
      case "Timestamp"  => Some(TypeRef("java.time.Instant"))
      case "Blob"       => Some(TypeRef.of("zio.blocks.chunk.Chunk", TypeRef.Byte))
      case "Document"   => Some(TypeRef("zio.blocks.schema.DynamicValue"))
      case _            => None
    }

    def typeRefFor(model: SmithyModel, target: ShapeId): TypeRef =
      model.findShape(target.name).map(_.shape) match {
        case Some(list: ListShape) =>
          TypeRef.list(typeRefFor(model, list.member.target))
        case Some(map: MapShape) =>
          TypeRef.map(typeRefFor(model, map.key.target), typeRefFor(model, map.value.target))
        case Some(_: StructureShape) | Some(_: UnionShape) | Some(_: EnumShape) =>
          TypeRef(target.name)
        case Some(other) =>
          throw new IllegalArgumentException(s"Unsupported shape: ${other.name}")
        case None =>
          simpleTypeRef(target.name).getOrElse(throw new IllegalArgumentException(s"Unknown shape: $target"))
      }

    def docOf(traits: List[TraitApplication]): Option[String] =
      traits.collectFirst {
        case TraitApplication(ShapeId("smithy.api", "documentation"), Some(NodeValue.String(text))) => text
      }

    def fieldFor(model: SmithyModel, member: MemberDefinition): Field = {
      val fieldType = typeRefFor(model, member.target)
      if (member.traits.contains(TraitApplication.required)) Field(member.name, fieldType)
      else Field(member.name, TypeRef.optional(fieldType), defaultValue = Some("None"))
    }

    def caseClassFor(model: SmithyModel, name: String, shape: StructureShape): CaseClass =
      CaseClass(name, shape.members.map(fieldFor(model, _)), doc = docOf(shape.traits))

    def sealedTraitFor(model: SmithyModel, name: String, shape: UnionShape): SealedTrait = {
      val cases = shape.members.map { member =>
        val caseName = member.name.capitalize
        if (member.target.name == "Unit") SealedTraitCase.CaseObjectCase(caseName)
        else SealedTraitCase.CaseClassCase(CaseClass(caseName, List(Field("value", typeRefFor(model, member.target)))))
      }
      SealedTrait(name, cases = cases, doc = docOf(shape.traits))
    }

    def enumFor(name: String, shape: EnumShape): Enum =
      Enum(name, shape.members.map(member => EnumCase.SimpleCase(member.name)), doc = docOf(shape.traits))

    val statusTrait: SealedTrait =
      model.shapes.collectFirst { case ShapeDefinition(name, shape: UnionShape) =>
        sealedTraitFor(model, name, shape)
      }.get

    val priorityEnum: Enum =
      model.shapes.collectFirst { case ShapeDefinition(name, shape: EnumShape) => enumFor(name, shape) }.get

    println(ScalaEmitter.emit(ScalaFile(PackageDecl(model.namespace), types = List(statusTrait, priorityEnum))))
  }
}
```

</details>

## 7. Emit a Whole File for Scala 3 and Scala 2

We now have a mapping for every kind of shape in our contract. One more function walks the whole model and collects the generated types into a `ScalaFile`; lists and maps produce no type of their own, so the `collect` skips them:

```scala
def scalaFileFor(model: SmithyModel): ScalaFile = {
  val types = model.shapes.collect {
    case ShapeDefinition(name, shape: StructureShape) => caseClassFor(model, name, shape)
    case ShapeDefinition(name, shape: UnionShape)     => sealedTraitFor(model, name, shape)
    case ShapeDefinition(name, shape: EnumShape)      => enumFor(name, shape)
  }
  ScalaFile(PackageDecl(model.namespace), types = types)
}

val ordersFile: ScalaFile = scalaFileFor(model)
```

The code above:

- `model.shapes.collect { ... }` maps each structure, union and enum definition to its codegen type, using the functions from sections 5 and 6.
- Lists and maps match no case, so `collect` drops them.
- `ScalaFile(PackageDecl(model.namespace), types = types)` wraps the types in a file whose package comes from the Smithy namespace.
- `ordersFile` keeps the result, so we can emit it more than once.

Because `ordersFile` describes the code rather than spelling it out, we can emit it more than once. `EmitterConfig.default` writes Scala 3:

```scala
println(ScalaEmitter.emit(ordersFile, EmitterConfig.default))
// package com.acme.orders
// 
// /** An order placed by a customer. */
// case class Order(
//   id: String,
//   items: List[LineItem],
//   status: OrderStatus,
//   priority: Priority,
//   note: Option[String] = None,
//   attributes: Option[Map[String, String]] = None,
// )
// 
// /** One product line in an order. */
// case class LineItem(
//   sku: String,
//   quantity: Int,
//   unitPrice: BigDecimal,
// )
// 
// /** Where an order is in its life cycle. */
// sealed trait OrderStatus
// 
// object OrderStatus {
//   case object Placed extends OrderStatus
//   case class Shipped(
//     value: Shipment,
//   ) extends OrderStatus
//   case class Cancelled(
//     value: Cancellation,
//   ) extends OrderStatus
// }
// 
// case class Shipment(
//   trackingNumber: String,
//   shippedAt: java.time.Instant,
// )
// 
// case class Cancellation(
//   reason: String,
// )
// 
// /** How fast the order ships. */
// enum Priority {
//   case STANDARD, EXPRESS
// }
//
```

`EmitterConfig.scala2` writes the same types in Scala 2 syntax:

```scala
println(ScalaEmitter.emit(ordersFile, EmitterConfig.scala2))
// package com.acme.orders
// 
// /** An order placed by a customer. */
// case class Order(
//   id: String,
//   items: List[LineItem],
//   status: OrderStatus,
//   priority: Priority,
//   note: Option[String] = None,
//   attributes: Option[Map[String, String]] = None
// )
// 
// /** One product line in an order. */
// case class LineItem(
//   sku: String,
//   quantity: Int,
//   unitPrice: BigDecimal
// )
// 
// /** Where an order is in its life cycle. */
// sealed trait OrderStatus
// 
// object OrderStatus {
//   case object Placed extends OrderStatus
//   case class Shipped(
//     value: Shipment
//   ) extends OrderStatus
//   case class Cancelled(
//     value: Cancellation
//   ) extends OrderStatus
// }
// 
// case class Shipment(
//   trackingNumber: String,
//   shippedAt: java.time.Instant
// )
// 
// case class Cancellation(
//   reason: String
// )
// 
// /** How fast the order ships. */
// sealed trait Priority
// 
// object Priority {
//   case object STANDARD extends Priority
//   case object EXPRESS extends Priority
// }
//
```

Compare the two outputs. The Scala 2 version drops the trailing commas, and `Priority` becomes a sealed trait with one case object per value, because Scala 2 has no `enum`. We didn't change a single line of our mapping: the dialect is a choice we make when we emit, not when we describe the types. Both files compile, with Scala 3.9 and Scala 2.13 respectively.

<details>
  <summary>Runnable version, with setup from sections 3–6: <code>Concept7EmitBothDialectsExample.scala</code></summary>

```scala title="smithy-examples/src/main/scala/generatescalafromsmithy/Concept7EmitBothDialectsExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-smithy:0.0.56
//> using dep dev.zio::zio-blocks-codegen:0.0.56
//> using options -experimental
//> using file OrdersModel.scala
// Run from this folder:  scala-cli run Concept7EmitBothDialectsExample.scala OrdersModel.scala
// Or from smithy-examples:  sbt "runMain generatescalafromsmithy.Concept7EmitBothDialectsExample"

package generatescalafromsmithy

/**
 * Generate Scala Code from a Smithy Model — 7. Emit a Whole File for Scala 3
 * and Scala 2
 *
 * Collects every generated type into one `ScalaFile` and emits it with the
 * Scala 3 and the Scala 2 configuration.
 */
object Concept7EmitBothDialectsExample {
  def main(args: Array[String]): Unit = {
    // Setup from sections 3–6: the parsed model and the mappings for case classes, sealed traits and enums

    val model: SmithyModel = SmithyModel.parse(ordersSmithy) match {
      case Right(parsed) => parsed
      case Left(error)   => throw new IllegalArgumentException(error.formatMessage)
    }

    val orderShape: StructureShape =
      model.shapes.collectFirst { case ShapeDefinition("Order", shape: StructureShape) => shape }.get

    def simpleTypeRef(name: String): Option[TypeRef] = name match {
      case "String"     => Some(TypeRef.String)
      case "Boolean"    => Some(TypeRef.Boolean)
      case "Byte"       => Some(TypeRef.Byte)
      case "Short"      => Some(TypeRef.Short)
      case "Integer"    => Some(TypeRef.Int)
      case "Long"       => Some(TypeRef.Long)
      case "Float"      => Some(TypeRef.Float)
      case "Double"     => Some(TypeRef.Double)
      case "BigInteger" => Some(TypeRef.BigInt)
      case "BigDecimal" => Some(TypeRef.BigDecimal)
      case "Timestamp"  => Some(TypeRef("java.time.Instant"))
      case "Blob"       => Some(TypeRef.of("zio.blocks.chunk.Chunk", TypeRef.Byte))
      case "Document"   => Some(TypeRef("zio.blocks.schema.DynamicValue"))
      case _            => None
    }

    def typeRefFor(model: SmithyModel, target: ShapeId): TypeRef =
      model.findShape(target.name).map(_.shape) match {
        case Some(list: ListShape) =>
          TypeRef.list(typeRefFor(model, list.member.target))
        case Some(map: MapShape) =>
          TypeRef.map(typeRefFor(model, map.key.target), typeRefFor(model, map.value.target))
        case Some(_: StructureShape) | Some(_: UnionShape) | Some(_: EnumShape) =>
          TypeRef(target.name)
        case Some(other) =>
          throw new IllegalArgumentException(s"Unsupported shape: ${other.name}")
        case None =>
          simpleTypeRef(target.name).getOrElse(throw new IllegalArgumentException(s"Unknown shape: $target"))
      }

    def docOf(traits: List[TraitApplication]): Option[String] =
      traits.collectFirst {
        case TraitApplication(ShapeId("smithy.api", "documentation"), Some(NodeValue.String(text))) => text
      }

    def fieldFor(model: SmithyModel, member: MemberDefinition): Field = {
      val fieldType = typeRefFor(model, member.target)
      if (member.traits.contains(TraitApplication.required)) Field(member.name, fieldType)
      else Field(member.name, TypeRef.optional(fieldType), defaultValue = Some("None"))
    }

    def caseClassFor(model: SmithyModel, name: String, shape: StructureShape): CaseClass =
      CaseClass(name, shape.members.map(fieldFor(model, _)), doc = docOf(shape.traits))

    def sealedTraitFor(model: SmithyModel, name: String, shape: UnionShape): SealedTrait = {
      val cases = shape.members.map { member =>
        val caseName = member.name.capitalize
        if (member.target.name == "Unit") SealedTraitCase.CaseObjectCase(caseName)
        else SealedTraitCase.CaseClassCase(CaseClass(caseName, List(Field("value", typeRefFor(model, member.target)))))
      }
      SealedTrait(name, cases = cases, doc = docOf(shape.traits))
    }

    def enumFor(name: String, shape: EnumShape): Enum =
      Enum(name, shape.members.map(member => EnumCase.SimpleCase(member.name)), doc = docOf(shape.traits))

    def scalaFileFor(model: SmithyModel): ScalaFile = {
      val types = model.shapes.collect {
        case ShapeDefinition(name, shape: StructureShape) => caseClassFor(model, name, shape)
        case ShapeDefinition(name, shape: UnionShape)     => sealedTraitFor(model, name, shape)
        case ShapeDefinition(name, shape: EnumShape)      => enumFor(name, shape)
      }
      ScalaFile(PackageDecl(model.namespace), types = types)
    }

    val ordersFile: ScalaFile = scalaFileFor(model)

    println(ScalaEmitter.emit(ordersFile, EmitterConfig.default))

    println(ScalaEmitter.emit(ordersFile, EmitterConfig.scala2))
  }
}
```

</details>

## 8. Add Schema Derivation

The generated types compile, but our services also need to encode them to JSON and back. In ZIO Blocks that takes a `Schema` for each type, and the compiler can derive one. In Scala 3, a case class asks for it with `derives Schema`. In Scala 2, it gets a companion object holding `implicit val schema: Schema[Order] = Schema.derived`.

Both forms are codegen data too. A `CaseClass` has a `derives` list, which the emitter renders in Scala 3 only, and an optional `CompanionObject` whose members the emitter renders in either dialect. We add the schema after the mapping, as a separate step over the finished `ScalaFile`:

```scala
def withSchemas(file: ScalaFile, scala3: Boolean): ScalaFile = {
  val types = file.types.map {
    case caseClass: CaseClass if scala3 =>
      caseClass.copy(derives = List("Schema"))
    case caseClass: CaseClass =>
      val schema = ObjectMember.ValMember(
        "schema",
        TypeRef.of("Schema", TypeRef(caseClass.name)),
        "Schema.derived",
        isImplicit = true
      )
      caseClass.copy(companion = Some(CompanionObject(List(schema))))
    case other => other
  }
  file.copy(imports = Import.WildcardImport("zio.blocks.schema") :: file.imports, types = types)
}
```

The code above:

- `caseClass.copy(derives = List("Schema"))` asks the emitter to write `derives Schema` after the class.
- `ObjectMember.ValMember(..., isImplicit = true)` describes the companion's `implicit val`, and its value is the source text `Schema.derived`.
- `Import.WildcardImport("zio.blocks.schema")` adds the import that brings `Schema` into scope.
- Sealed traits and enums keep their shape (the `case other => other` line).

Only case classes get a schema of their own. `Schema.derived` derives schemas for the types it meets inside a class, so deriving the schema for `Order` also covers its `OrderStatus` field and its `Priority` field, and `Order` is the type our services send.

Let's print the generated `Order` in both dialects, using `ScalaEmitter.emitTypeDefinition`, which emits a single type:

```scala
println(ScalaEmitter.emitTypeDefinition(withSchemas(ordersFile, scala3 = true).types.head, EmitterConfig.default))
// /** An order placed by a customer. */
// case class Order(
//   id: String,
//   items: List[LineItem],
//   status: OrderStatus,
//   priority: Priority,
//   note: Option[String] = None,
//   attributes: Option[Map[String, String]] = None,
// ) derives Schema
println()
// 
println(ScalaEmitter.emitTypeDefinition(withSchemas(ordersFile, scala3 = false).types.head, EmitterConfig.scala2))
// /** An order placed by a customer. */
// case class Order(
//   id: String,
//   items: List[LineItem],
//   status: OrderStatus,
//   priority: Priority,
//   note: Option[String] = None,
//   attributes: Option[Map[String, String]] = None
// )
// 
// object Order {
//   implicit val schema: Schema[Order] = Schema.derived
// }
```

The Scala 3 class ends with `derives Schema`, and the Scala 2 class has a companion object with the `implicit val`. The full Scala 2 file also starts with `import zio.blocks.schema._`, while the Scala 3 file uses `import zio.blocks.schema.*`.

:::caution
In this release, only `CaseClass` is a safe place for the schema. The emitter renders `derives` on case classes only, so it ignores the `derives` list of a `SealedTrait` or an `Enum`. Two known bugs in 0.0.56 rule out the companion route for the other types and for Scala 3: the emitter drops the companion object of an `Enum`, and under `EmitterConfig.default` it renders an implicit companion value as `given val`, which Scala 3 rejects. Putting the schema on case classes only, as above, avoids all of them.
:::

<details>
  <summary>Runnable version, with setup from sections 3–7: <code>Concept8SchemaExample.scala</code></summary>

```scala title="smithy-examples/src/main/scala/generatescalafromsmithy/Concept8SchemaExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-smithy:0.0.56
//> using dep dev.zio::zio-blocks-codegen:0.0.56
//> using options -experimental
//> using file OrdersModel.scala
// Run from this folder:  scala-cli run Concept8SchemaExample.scala OrdersModel.scala
// Or from smithy-examples:  sbt "runMain generatescalafromsmithy.Concept8SchemaExample"

package generatescalafromsmithy

/**
 * Generate Scala Code from a Smithy Model — 8. Add Schema Derivation
 *
 * Adds `derives Schema` or a companion `implicit val schema` to the generated
 * case classes and prints `Order` in both dialects.
 */
object Concept8SchemaExample {
  def main(args: Array[String]): Unit = {
    // Setup from sections 3–7: the parsed model, the mappings and the whole ScalaFile

    val model: SmithyModel = SmithyModel.parse(ordersSmithy) match {
      case Right(parsed) => parsed
      case Left(error)   => throw new IllegalArgumentException(error.formatMessage)
    }

    val orderShape: StructureShape =
      model.shapes.collectFirst { case ShapeDefinition("Order", shape: StructureShape) => shape }.get

    def simpleTypeRef(name: String): Option[TypeRef] = name match {
      case "String"     => Some(TypeRef.String)
      case "Boolean"    => Some(TypeRef.Boolean)
      case "Byte"       => Some(TypeRef.Byte)
      case "Short"      => Some(TypeRef.Short)
      case "Integer"    => Some(TypeRef.Int)
      case "Long"       => Some(TypeRef.Long)
      case "Float"      => Some(TypeRef.Float)
      case "Double"     => Some(TypeRef.Double)
      case "BigInteger" => Some(TypeRef.BigInt)
      case "BigDecimal" => Some(TypeRef.BigDecimal)
      case "Timestamp"  => Some(TypeRef("java.time.Instant"))
      case "Blob"       => Some(TypeRef.of("zio.blocks.chunk.Chunk", TypeRef.Byte))
      case "Document"   => Some(TypeRef("zio.blocks.schema.DynamicValue"))
      case _            => None
    }

    def typeRefFor(model: SmithyModel, target: ShapeId): TypeRef =
      model.findShape(target.name).map(_.shape) match {
        case Some(list: ListShape) =>
          TypeRef.list(typeRefFor(model, list.member.target))
        case Some(map: MapShape) =>
          TypeRef.map(typeRefFor(model, map.key.target), typeRefFor(model, map.value.target))
        case Some(_: StructureShape) | Some(_: UnionShape) | Some(_: EnumShape) =>
          TypeRef(target.name)
        case Some(other) =>
          throw new IllegalArgumentException(s"Unsupported shape: ${other.name}")
        case None =>
          simpleTypeRef(target.name).getOrElse(throw new IllegalArgumentException(s"Unknown shape: $target"))
      }

    def docOf(traits: List[TraitApplication]): Option[String] =
      traits.collectFirst {
        case TraitApplication(ShapeId("smithy.api", "documentation"), Some(NodeValue.String(text))) => text
      }

    def fieldFor(model: SmithyModel, member: MemberDefinition): Field = {
      val fieldType = typeRefFor(model, member.target)
      if (member.traits.contains(TraitApplication.required)) Field(member.name, fieldType)
      else Field(member.name, TypeRef.optional(fieldType), defaultValue = Some("None"))
    }

    def caseClassFor(model: SmithyModel, name: String, shape: StructureShape): CaseClass =
      CaseClass(name, shape.members.map(fieldFor(model, _)), doc = docOf(shape.traits))

    def sealedTraitFor(model: SmithyModel, name: String, shape: UnionShape): SealedTrait = {
      val cases = shape.members.map { member =>
        val caseName = member.name.capitalize
        if (member.target.name == "Unit") SealedTraitCase.CaseObjectCase(caseName)
        else SealedTraitCase.CaseClassCase(CaseClass(caseName, List(Field("value", typeRefFor(model, member.target)))))
      }
      SealedTrait(name, cases = cases, doc = docOf(shape.traits))
    }

    def enumFor(name: String, shape: EnumShape): Enum =
      Enum(name, shape.members.map(member => EnumCase.SimpleCase(member.name)), doc = docOf(shape.traits))

    def scalaFileFor(model: SmithyModel): ScalaFile = {
      val types = model.shapes.collect {
        case ShapeDefinition(name, shape: StructureShape) => caseClassFor(model, name, shape)
        case ShapeDefinition(name, shape: UnionShape)     => sealedTraitFor(model, name, shape)
        case ShapeDefinition(name, shape: EnumShape)      => enumFor(name, shape)
      }
      ScalaFile(PackageDecl(model.namespace), types = types)
    }

    val ordersFile: ScalaFile = scalaFileFor(model)

    def withSchemas(file: ScalaFile, scala3: Boolean): ScalaFile = {
      val types = file.types.map {
        case caseClass: CaseClass if scala3 =>
          caseClass.copy(derives = List("Schema"))
        case caseClass: CaseClass =>
          val schema = ObjectMember.ValMember(
            "schema",
            TypeRef.of("Schema", TypeRef(caseClass.name)),
            "Schema.derived",
            isImplicit = true
          )
          caseClass.copy(companion = Some(CompanionObject(List(schema))))
        case other => other
      }
      file.copy(imports = Import.WildcardImport("zio.blocks.schema") :: file.imports, types = types)
    }

    println(ScalaEmitter.emitTypeDefinition(withSchemas(ordersFile, scala3 = true).types.head, EmitterConfig.default))
    println()
    println(ScalaEmitter.emitTypeDefinition(withSchemas(ordersFile, scala3 = false).types.head, EmitterConfig.scala2))
  }
}
```

</details>

## 9. Put the Generator Together

So far the model comes from a string and the output goes to the console. A real generator reads a `.smithy` file and writes a `.scala` file. This example combines everything from sections 3 to 8 into one program, with two differences: it reads the model from the path in its first argument (by default `src/main/smithy/orders.smithy`), and it writes the Scala 3 version, with schemas, to `Orders.scala` in the directory named by its second argument (by default `target/generated-orders`):

```scala title="smithy-examples/src/main/scala/generatescalafromsmithy/CompleteExample.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-smithy:0.0.56
//> using dep dev.zio::zio-blocks-codegen:0.0.56
//> using options -experimental
// Run from this folder:  scala-cli run CompleteExample.scala
// Or from smithy-examples:  sbt "runMain generatescalafromsmithy.CompleteExample"

package generatescalafromsmithy

/**
 * Generate Scala Code from a Smithy Model — Put the Generator Together
 *
 * Reads a Smithy model from a file, maps it to a `ScalaFile`, adds schema
 * derivation and writes the Scala 3 source of the generated types to
 * `Orders.scala`.
 */
object CompleteExample {
  def main(args: Array[String]): Unit = {
    val modelPath: String    = if (args.length > 0) args(0) else "src/main/smithy/orders.smithy"
    val outputDir: String    = if (args.length > 1) args(1) else "target/generated-orders"
    val ordersSmithy: String = new String(Files.readAllBytes(Paths.get(modelPath)), StandardCharsets.UTF_8)

    val model: SmithyModel = SmithyModel.parse(ordersSmithy) match {
      case Right(parsed) => parsed
      case Left(error)   => throw new IllegalArgumentException(error.formatMessage)
    }

    def simpleTypeRef(name: String): Option[TypeRef] = name match {
      case "String"     => Some(TypeRef.String)
      case "Boolean"    => Some(TypeRef.Boolean)
      case "Byte"       => Some(TypeRef.Byte)
      case "Short"      => Some(TypeRef.Short)
      case "Integer"    => Some(TypeRef.Int)
      case "Long"       => Some(TypeRef.Long)
      case "Float"      => Some(TypeRef.Float)
      case "Double"     => Some(TypeRef.Double)
      case "BigInteger" => Some(TypeRef.BigInt)
      case "BigDecimal" => Some(TypeRef.BigDecimal)
      case "Timestamp"  => Some(TypeRef("java.time.Instant"))
      case "Blob"       => Some(TypeRef.of("zio.blocks.chunk.Chunk", TypeRef.Byte))
      case "Document"   => Some(TypeRef("zio.blocks.schema.DynamicValue"))
      case _            => None
    }

    def typeRefFor(model: SmithyModel, target: ShapeId): TypeRef =
      model.findShape(target.name).map(_.shape) match {
        case Some(list: ListShape) =>
          TypeRef.list(typeRefFor(model, list.member.target))
        case Some(map: MapShape) =>
          TypeRef.map(typeRefFor(model, map.key.target), typeRefFor(model, map.value.target))
        case Some(_: StructureShape) | Some(_: UnionShape) | Some(_: EnumShape) =>
          TypeRef(target.name)
        case Some(other) =>
          throw new IllegalArgumentException(s"Unsupported shape: ${other.name}")
        case None =>
          simpleTypeRef(target.name).getOrElse(throw new IllegalArgumentException(s"Unknown shape: $target"))
      }

    def docOf(traits: List[TraitApplication]): Option[String] =
      traits.collectFirst {
        case TraitApplication(ShapeId("smithy.api", "documentation"), Some(NodeValue.String(text))) => text
      }

    def fieldFor(model: SmithyModel, member: MemberDefinition): Field = {
      val fieldType = typeRefFor(model, member.target)
      if (member.traits.contains(TraitApplication.required)) Field(member.name, fieldType)
      else Field(member.name, TypeRef.optional(fieldType), defaultValue = Some("None"))
    }

    def caseClassFor(model: SmithyModel, name: String, shape: StructureShape): CaseClass =
      CaseClass(name, shape.members.map(fieldFor(model, _)), doc = docOf(shape.traits))

    def sealedTraitFor(model: SmithyModel, name: String, shape: UnionShape): SealedTrait = {
      val cases = shape.members.map { member =>
        val caseName = member.name.capitalize
        if (member.target.name == "Unit") SealedTraitCase.CaseObjectCase(caseName)
        else SealedTraitCase.CaseClassCase(CaseClass(caseName, List(Field("value", typeRefFor(model, member.target)))))
      }
      SealedTrait(name, cases = cases, doc = docOf(shape.traits))
    }

    def enumFor(name: String, shape: EnumShape): Enum =
      Enum(name, shape.members.map(member => EnumCase.SimpleCase(member.name)), doc = docOf(shape.traits))

    def scalaFileFor(model: SmithyModel): ScalaFile = {
      val types = model.shapes.collect {
        case ShapeDefinition(name, shape: StructureShape) => caseClassFor(model, name, shape)
        case ShapeDefinition(name, shape: UnionShape)     => sealedTraitFor(model, name, shape)
        case ShapeDefinition(name, shape: EnumShape)      => enumFor(name, shape)
      }
      ScalaFile(PackageDecl(model.namespace), types = types)
    }

    def withSchemas(file: ScalaFile, scala3: Boolean): ScalaFile = {
      val types = file.types.map {
        case caseClass: CaseClass if scala3 =>
          caseClass.copy(derives = List("Schema"))
        case caseClass: CaseClass =>
          val schema = ObjectMember.ValMember(
            "schema",
            TypeRef.of("Schema", TypeRef(caseClass.name)),
            "Schema.derived",
            isImplicit = true
          )
          caseClass.copy(companion = Some(CompanionObject(List(schema))))
        case other => other
      }
      file.copy(imports = Import.WildcardImport("zio.blocks.schema") :: file.imports, types = types)
    }

    val ordersFile: ScalaFile = scalaFileFor(model)
    val outputFile            = Paths.get(outputDir, "Orders.scala")
    Files.createDirectories(Paths.get(outputDir))
    Files.write(
      outputFile,
      ScalaEmitter.emit(withSchemas(ordersFile, scala3 = true), EmitterConfig.default).getBytes(StandardCharsets.UTF_8)
    )
    println(s"Generated ${ordersFile.types.map(_.name).mkString(", ")} into $outputFile")
  }
}
```

Running it from the `smithy-examples` folder with `sbt "runMain generatescalafromsmithy.CompleteExample"` prints:

```text
Generated Order, LineItem, OrderStatus, Shipment, Cancellation, Priority into target/generated-orders/Orders.scala
```

## 10. Compile the Generated Code with sbt

The last step is to make the generator part of the build, so the generated code is compiled — and kept up to date — every time the build compiles. sbt calls this a source generator: a task that writes source files before compilation and returns them, so sbt compiles them with everything else. Their usual home is the `src_managed` folder under `target/`, so the generated code never ends up in version control.

sbt 1 runs its own build code on Scala 2.12, and the smithy and codegen modules aren't published for Scala 2.12, so the generator can't run inside the build definition itself. Instead it stays an ordinary program in one sbt project, and a second project runs it as its source generator. Here is the part of `build.sbt` that does it, adapted from our examples project (which writes its dependencies through a small helper); the root project `smithy-examples` holds `CompleteExample` and depends on the smithy and codegen modules, and the new `orders-app` project compiles the result:

```scala
// The project that holds the generator, referred to by its id
val generator = LocalProject("smithy-examples")

lazy val ordersApp = (project in file("orders-app"))
  .settings(
    run / fork := true,
    libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56",
    Compile / sourceGenerators += Def.task {
      val model     = (generator / baseDirectory).value / "src" / "main" / "smithy" / "orders.smithy"
      val outputDir = (Compile / sourceManaged).value / "smithy"
      val classpath = (generator / Compile / fullClasspath).value.files
      (generator / Compile / runner).value
        .run(
          "generatescalafromsmithy.CompleteExample",
          classpath,
          Seq(model.getPath, outputDir.getPath),
          streams.value.log
        )
        .get
      (outputDir ** "*.scala").get
    }.taskValue
  )
```

The code above:

- `LocalProject("smithy-examples")` refers to the generator's project by its id; a direct reference to its `lazy val` would make the two project definitions depend on each other.
- `(generator / Compile / fullClasspath)` is the generator's compiled code plus the smithy and codegen jars, and `(generator / Compile / runner)` runs a main class on that classpath.
- The task passes the model's path and `orders-app`'s `src_managed/main/smithy` folder to `CompleteExample`, then returns every `.scala` file the generator wrote there.
- `orders-app` itself depends only on the schema module: it never sees the generator, only the code it writes.

With that in place, `orders-app` can use the generated types like hand-written ones. This program builds an `Order`, encodes it to JSON with `toJsonString` and decodes it back with `fromJson`:

```scala title="smithy-examples/orders-app/src/main/scala/generatescalafromsmithy/app/OrdersApp.scala"
// Run from smithy-examples:  sbt ordersApp/run
// The generated types come from `smithy-examples/src/main/smithy/orders.smithy`; sbt generates them before it compiles.

package generatescalafromsmithy.app

/**
 * Generate Scala Code from a Smithy Model — Compile the Generated Code with sbt
 *
 * Builds an `Order` from the generated types, encodes it to JSON with
 * `toJsonString` and decodes it back with `fromJson`.
 */
object OrdersApp {
  def main(args: Array[String]): Unit = {
    val order = Order(
      id = "order-1001",
      items = List(LineItem("MUG-RED", 2, BigDecimal("12.50"))),
      status = OrderStatus.Shipped(Shipment("1Z999", java.time.Instant.parse("2026-10-27T09:30:00Z"))),
      priority = Priority.EXPRESS,
      note = Some("Leave at the door")
    )
    val json = order.toJsonString
    println(json)
    println(s"round trip: ${json.fromJson[Order] == Right(order)}")
  }
}
```

The code above:

- `import com.acme.orders._` brings in the generated `Order`, `LineItem`, `OrderStatus`, `Shipment` and `Priority`, which come from the Smithy namespace.
- `Order(...)` builds a value with the generated case class. The `note` field takes `Some(...)` and `attributes` is left at its `None` default.
- `order.toJsonString` and `json.fromJson[Order]` use the derived `Schema` that the generator added.
- The last line compares the decoded value with the original.

Running `sbt ordersApp/run` from the `smithy-examples` folder generates `Orders.scala`, compiles it together with `OrdersApp` and prints:

```text
{"id":"order-1001","items":[{"sku":"MUG-RED","quantity":2,"unitPrice":12.50}],"status":{"Shipped":{"value":{"trackingNumber":"1Z999","shippedAt":"2026-10-27T09:30:00Z"}}},"priority":"EXPRESS","note":"Leave at the door"}
round trip: true
```

The JSON follows the generated types: the union member becomes an object keyed by its case name, the enum value becomes its name, and the missing `attributes` member is left out. Try adding a `@required` member to `Order` in `orders.smithy` and compiling again: the new field appears in the generated class, and every place that builds an `Order` without it fails to compile until you update it.

## 11. Running the Examples

All runnable code from this tutorial is in [`smithy-examples/src/main/scala/generatescalafromsmithy/`](https://github.com/zio/zio-blocks/tree/main/smithy-examples/src/main/scala/generatescalafromsmithy), and the app that compiles the generated code is in [`smithy-examples/orders-app/`](https://github.com/zio/zio-blocks/tree/main/smithy-examples/orders-app). Run any file from the `smithy-examples` folder with `sbt "runMain generatescalafromsmithy.<ObjectName>"`, and the app with `sbt ordersApp/run`.

## 12. What You've Learned

In this tutorial, you learned:

- How to parse a Smithy model with `SmithyModel.parse` and walk its shapes, and why an unqualified target such as `String` has an empty namespace
- How structures, unions, enums, lists, maps, simple types and optional members map to `CaseClass`, `SealedTrait`, `Enum` and `TypeRef` values
- How `ScalaEmitter.emit` turns one `ScalaFile` into Scala 3 or Scala 2 source, depending on the `EmitterConfig`
- How to add `derives Schema` or a companion `implicit val schema` as a separate step, so the generated types encode to JSON and back
- How a source generator in a second sbt project runs the generator and compiles its output

You now own a small, readable code generator. The next step is to extend it for your own contracts: more Smithy traits, more shapes, or a different output style.

## 13. Where to Go Next

- **Want to know everything the parser reads?** The [smithy module reference](../reference/smithy.md) lists every shape and trait type, and shows how to build models in code.
- **Want to generate more than data types?** The [codegen module reference](../reference/codegen/index.md) covers the whole IR, including traits, objects, methods and opaque types, and the [`ScalaEmitter`](../reference/codegen/scala-emitter.md) and [`EmitterConfig`](../reference/codegen/emitter-config.md) pages describe every output option.
- **New to schemas?** [Your First Schema](./your-first-schema.md) explains what the derived `Schema` gives the generated types, from JSON codecs to JSON Schema.
- **Using OpenAPI instead of Smithy?** [How to Generate Scala Types from an OpenAPI Spec at Build Time](./generate-scala-types-from-openapi.md) builds the same kind of generator for the component schemas of an OpenAPI document.

---

## How to Generate Scala Types from an OpenAPI Spec at Build Time

In this guide, we will write a small generator that reads an OpenAPI 3.1 document for a bookstore API and turns its `components.schemas` into Scala case classes, enums and sealed traits, with a derived `Schema` so they decode the JSON the API sends. By the end, an sbt project regenerates those types from the spec on every compile and decodes the spec's own examples into them, so the types can never drift from the spec.

We'll parse the spec with the [openapi module](../reference/openapi.md), map each schema to the [codegen module](../reference/codegen/index.md)'s description of Scala source, emit that source for Scala 3 and Scala 2, and finally run the generator as an sbt source generator. The guide covers:

- Parsing an OpenAPI document into the typed `OpenAPI` model
- Reading component schemas as typed JSON Schema values
- Mapping JSON Schema types and formats to Scala types
- Turning objects, string enums and `oneOf` with a discriminator into case classes, enums and sealed traits
- Emitting the file for Scala 3 and Scala 2, with schema derivation
- Running the generator on every build
- Reading a YAML spec, and what the generator doesn't cover

This guide follows the same pattern as [Generate Scala Code from a Smithy Model](./generate-scala-code-from-smithy.md), which explains the codegen IR step by step. If you haven't used `CaseClass`, `TypeRef` or `ScalaEmitter` before, read that tutorial first; here we focus on what is specific to OpenAPI.

## The Problem

A partner sends us `bookstore.json`, an OpenAPI 3.1 document describing their bookstore API, and we need Scala types for the JSON it returns. The usual first step is to read the spec and copy each schema into a case class by hand. That works on the day we write it, and then the spec changes: a field becomes required, a payment method is added, an enum gains a value. Nothing in our build notices. The mismatch shows up at runtime, as a decoding error in production.

For example, a hand-written copy of the order schema might look like this:

```scala

// Copied by hand from bookstore.json. Keep in sync!
final case class OrderLine(bookId: String, quantity: Int, unitPrice: BigDecimal)

sealed trait Payment
final case class CardPayment(last4: String)    extends Payment
final case class GiftCardPayment(code: String) extends Payment

final case class Order(
  id: UUID,
  customerId: Long,
  placedAt: Instant,
  lines: List[OrderLine],
  payment: Payment,
  status: String,              // the spec says: one of Placed, Shipped, Delivered, Cancelled
  giftMessage: Option[String]  // optional in the spec
)
```

Every line of this is a decision someone made while reading JSON: which properties are required, what `format: int64` means in Scala, how the `method` property chooses between card and gift-card payments. Multiply it by every schema in a real API, and every release of the spec, and the copy becomes a maintenance job that nobody owns.

In this guide, we make those decisions once, in code. The generator reads the spec, applies the same rules to every schema, and writes the Scala source as part of the build. When the spec changes, the next compile regenerates the types, and every place that uses them either still compiles or points at what changed.

Nothing in ZIO Blocks connects the openapi module to the codegen module: the first parses specs, the second writes Scala source, and the mapping between them is ours to write. That mapping is the generator we build here, and it is short enough to read in one sitting:

```
bookstore.json ──openAPICodec──▶ OpenAPI ──our mapping──▶ ScalaFile ──ScalaEmitter.emit──▶ Bookstore.scala
    (text)                     (schemas)                  (Scala as data)                    (text)
```

The generator handles data types only. It writes no HTTP clients or servers; if you need those, full-featured tools such as [guardrail](https://guardrail.dev/) and [OpenAPI Generator](https://openapi-generator.tech/) generate them from the same document.

## Prerequisites

To follow along, add the openapi and codegen modules to your `build.sbt`. The code we generate also needs the schema module:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-openapi" % "0.0.56",
  "dev.zio" %% "zio-blocks-codegen" % "0.0.56",
  "dev.zio" %% "zio-blocks-schema"  % "0.0.56"
)

// Scala 3.9 and newer
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-openapi" % "0.0.56",
  "dev.zio" %% "zio-blocks-next-codegen" % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema"  % "0.0.56"
)
scalacOptions += "-experimental"
```

The openapi and schema modules are published for the JVM and Scala.js; codegen is published for the JVM only. That is no limit for a build-time generator, which always runs on the JVM inside the build, and the code it writes depends only on the schema module. For scala-cli or cross-building, see [How to Add ZIO Blocks to Your Build](../getting-started/installation.md).

We also assume you know what a derived `Schema` gives a type; if not, [Your First Schema](./your-first-schema.md) explains it.

## The Bookstore Spec

Our spec describes two kinds of data: a catalog of books and the orders customers place. It is small, but it uses the features most real specs rely on: objects with required and optional properties, arrays, a map, string enums, `$ref`s between schemas, formats such as `int64`, `date-time` and `uuid`, a nullable property, and a `oneOf` with a discriminator that tells card payments from gift-card payments. The `Book` and `Order` schemas also carry an `example` keyword, a sample payload that `SchemaObject#example` exposes and that we decode at the end.

We keep the spec in a string here so every block on this page can use it; the generator in [Putting It Together](#putting-it-together) reads the same text from a `bookstore.json` file:

```scala
val bookstoreJson: String =
  """{
    |  "openapi": "3.1.0",
    |  "info": { "title": "Bookstore", "version": "1.0.0" },
    |  "paths": {
    |    "/orders/{orderId}": {
    |      "get": {
    |        "operationId": "getOrder",
    |        "parameters": [
    |          { "name": "orderId", "in": "path", "required": true, "schema": { "type": "string", "format": "uuid" } }
    |        ],
    |        "responses": {
    |          "200": {
    |            "description": "The order",
    |            "content": { "application/json": { "schema": { "$ref": "#/components/schemas/Order" } } }
    |          }
    |        }
    |      }
    |    }
    |  },
    |  "components": {
    |    "schemas": {
    |      "Book": {
    |        "type": "object",
    |        "description": "A book in the catalog.",
    |        "required": ["id", "title", "authors", "format", "price"],
    |        "properties": {
    |          "id": { "type": "string" },
    |          "title": { "type": "string" },
    |          "authors": { "type": "array", "items": { "$ref": "#/components/schemas/Author" } },
    |          "format": { "$ref": "#/components/schemas/BookFormat" },
    |          "price": { "type": "number" },
    |          "pages": { "type": "integer", "format": "int32" },
    |          "publishedOn": { "type": "string", "format": "date" },
    |          "subtitle": { "type": ["string", "null"] },
    |          "tags": { "type": "array", "items": { "type": "string" } }
    |        },
    |        "example": {
    |          "id": "zionomicon",
    |          "title": "Zionomicon",
    |          "authors": [{ "name": "John A. De Goes" }, { "name": "Adam Fraser" }, { "name": "Milad Khajavi" }],
    |          "format": "Ebook",
    |          "price": 59.99,
    |          "tags": ["zio", "functional-programming"]
    |        }
    |      },
    |      "Author": {
    |        "type": "object",
    |        "required": ["name"],
    |        "properties": {
    |          "name": { "type": "string" },
    |          "website": { "type": "string", "format": "uri" }
    |        }
    |      },
    |      "BookFormat": {
    |        "type": "string",
    |        "description": "How the book is delivered.",
    |        "enum": ["Hardcover", "Paperback", "Ebook"]
    |      },
    |      "Order": {
    |        "type": "object",
    |        "description": "A customer's order.",
    |        "required": ["id", "customerId", "placedAt", "lines", "payment", "status"],
    |        "properties": {
    |          "id": { "type": "string", "format": "uuid" },
    |          "customerId": { "type": "integer", "format": "int64" },
    |          "placedAt": { "type": "string", "format": "date-time" },
    |          "lines": { "type": "array", "items": { "$ref": "#/components/schemas/OrderLine" } },
    |          "payment": { "$ref": "#/components/schemas/Payment" },
    |          "status": { "$ref": "#/components/schemas/OrderStatus" },
    |          "giftMessage": { "type": "string" },
    |          "metadata": { "type": "object", "additionalProperties": { "type": "string" } }
    |        },
    |        "example": {
    |          "id": "3f0c1c5e-8a59-4d55-9b8e-2b2f0b6a4c11",
    |          "customerId": 1001,
    |          "placedAt": "2026-10-27T09:30:00Z",
    |          "lines": [
    |            { "bookId": "zionomicon", "quantity": 1, "unitPrice": 59.99 },
    |            { "bookId": "programming-in-scala-5", "quantity": 2, "unitPrice": 44.95 }
    |          ],
    |          "payment": { "method": "card", "last4": "4242" },
    |          "status": "Placed",
    |          "metadata": { "channel": "web" }
    |        }
    |      },
    |      "OrderLine": {
    |        "type": "object",
    |        "required": ["bookId", "quantity", "unitPrice"],
    |        "properties": {
    |          "bookId": { "type": "string" },
    |          "quantity": { "type": "integer", "format": "int32" },
    |          "unitPrice": { "type": "number" }
    |        }
    |      },
    |      "OrderStatus": {
    |        "type": "string",
    |        "enum": ["Placed", "Shipped", "Delivered", "Cancelled"]
    |      },
    |      "Payment": {
    |        "description": "How the customer paid.",
    |        "oneOf": [
    |          { "$ref": "#/components/schemas/CardPayment" },
    |          { "$ref": "#/components/schemas/GiftCardPayment" }
    |        ],
    |        "discriminator": {
    |          "propertyName": "method",
    |          "mapping": {
    |            "card": "#/components/schemas/CardPayment",
    |            "gift_card": "#/components/schemas/GiftCardPayment"
    |          }
    |        }
    |      },
    |      "CardPayment": {
    |        "type": "object",
    |        "required": ["method", "last4"],
    |        "properties": {
    |          "method": { "type": "string" },
    |          "last4": { "type": "string" }
    |        }
    |      },
    |      "GiftCardPayment": {
    |        "type": "object",
    |        "required": ["method", "code"],
    |        "properties": {
    |          "method": { "type": "string" },
    |          "code": { "type": "string" }
    |        }
    |      }
    |    }
    |  }
    |}""".stripMargin
```

## Parse the Spec

The openapi module models a whole OpenAPI 3.1 document as plain Scala values: `OpenAPI` at the root, with `Info`, `Paths`, `Components` and the rest below it. `OpenAPICodec.openAPICodec` is the codec that reads a document into that model. It works on the `Json` tree, so we parse the text with `Json.parse` first and then decode the tree:

```scala

def parseSpec(text: String): OpenAPI =
  Json.parse(text).flatMap(json => OpenAPICodec.openAPICodec.decode(json)) match {
    case Right(api)  => api
    case Left(error) => throw new IllegalArgumentException(s"Not an OpenAPI document: ${error.getMessage}")
  }

val api: OpenAPI = parseSpec(bookstoreJson)
```

Both steps return an `Either`, so a malformed file and a document that isn't OpenAPI fail the same way. A generator runs inside the build, so `parseSpec` turns the failure into an exception, which fails the build with the message. The parsed document gives us the API's name and the names of its component schemas:

```scala
println(s"${api.info.title} ${api.info.version}")
// Bookstore 1.0.0
println(api.components.toList.flatMap(_.schemas.toList).map(_._1).mkString(", "))
// Book, Author, BookFormat, Order, OrderLine, OrderStatus, Payment, CardPayment, GiftCardPayment
```

A document that misses a required field, such as `info`, is rejected with the field's name:

```scala
println(Json.parse("""{"openapi": "3.1.0"}""").flatMap(json => OpenAPICodec.openAPICodec.decode(json)))
// Left(zio.blocks.schema.SchemaError: Missing required field: info at: .)
```

:::warning
Decode the `Json` tree, not the text. In 0.0.56, a known bug makes `OpenAPICodec.openAPICodec.decode` on a `String` return a `Left` ("an implementation is missing") and `OpenAPICodec.openAPICodec.encodeToString` throw, because the codec implements only the `Json` tree methods.
:::

## Read the Component Schemas

The types we want live in `api.components.schemas`, a map from each schema's name to a `ReferenceOr[SchemaObject]`. A component is almost always a `ReferenceOr.Value` holding the schema itself; a `ReferenceOr.Ref` would mean the component is only an alias for another one, which our generator doesn't support.

A `SchemaObject` keeps the schema in two parts. The OpenAPI-only keywords get their own fields, such as `SchemaObject#discriminator` and `SchemaObject#example`, and everything else, the JSON Schema 2020-12 part, stays in `SchemaObject#jsonSchema` as raw JSON. `SchemaObject#toJsonSchema` parses that part into the typed `JsonSchema` from the schema module, so we can read `properties`, `required`, `format`, `enum` and `oneOf` as fields instead of digging through JSON.

We collect each component's name, its typed schema and its discriminator into one small case class:

```scala
final case class Component(name: String, schema: JsonSchema.Object, discriminator: Option[Discriminator])

def componentsOf(api: OpenAPI): List[Component] =
  api.components.toList.flatMap(_.schemas.toList).map {
    case (name, ReferenceOr.Value(schemaObject)) =>
      schemaObject.toJsonSchema match {
        case Right(schema: JsonSchema.Object) => Component(name, schema, schemaObject.discriminator)
        case _                                => throw new IllegalArgumentException(s"$name: unsupported schema")
      }
    case (name, ReferenceOr.Ref(_)) =>
      throw new IllegalArgumentException(s"$name: a component that is only a $$ref is not supported")
  }

val components: List[Component] = componentsOf(api)
```

`JsonSchema.Object` is the case of `JsonSchema` that holds keywords; the other two cases, `JsonSchema.True` and `JsonSchema.False`, are the boolean schemas that accept everything or nothing, and never describe a type. Printing each component shows what the generator has to work with:

```scala
components.foreach { component =>
  val schema = component.schema
  println(
    s"${component.name}: type=${schema.`type`.map(_.toJson.print).getOrElse("-")}, " +
      s"properties=${schema.properties.map(_.toList.map(_._1).mkString("[", ", ", "]")).getOrElse("-")}, " +
      s"required=${schema.required.map(_.toList.sorted.mkString("[", ", ", "]")).getOrElse("-")}"
  )
}
// Book: type="object", properties=[id, title, authors, format, price, pages, publishedOn, subtitle, tags], required=[authors, format, id, price, title]
// Author: type="object", properties=[name, website], required=[name]
// BookFormat: type="string", properties=-, required=-
// Order: type="object", properties=[id, customerId, placedAt, lines, payment, status, giftMessage, metadata], required=[customerId, id, lines, payment, placedAt, status]
// OrderLine: type="object", properties=[bookId, quantity, unitPrice], required=[bookId, quantity, unitPrice]
// OrderStatus: type="string", properties=-, required=-
// Payment: type=-, properties=-, required=-
// CardPayment: type="object", properties=[method, last4], required=[last4, method]
// GiftCardPayment: type="object", properties=[method, code], required=[code, method]
println(components.find(_.name == "Payment").flatMap(_.discriminator))
// Some(Discriminator(method,ChunkMap(card -> #/components/schemas/CardPayment, gift_card -> #/components/schemas/GiftCardPayment)))
```

Three things in this output shape the rest of the generator. The properties come out in the order the spec lists them, so the generated fields keep that order too. `required` is a set of names, separate from `properties`, so each property's optionality is a lookup. And `Payment` has no `type` at all: it is a `oneOf`, and its discriminator names the `method` property and maps each value to a schema.

<details>
  <summary>Runnable version, with setup from the previous section: <code>Step2ComponentSchemasExample.scala</code></summary>

```scala title="openapi-examples/src/main/scala/generatescalatypesfromopenapi/Step2ComponentSchemasExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-next-openapi:0.0.56
//> using options -experimental
//> using file BookstoreSpec.scala
// Run from this folder:  scala-cli run Step2ComponentSchemasExample.scala BookstoreSpec.scala
// Or from openapi-examples:  sbt "runMain generatescalatypesfromopenapi.Step2ComponentSchemasExample"

package generatescalatypesfromopenapi

/**
 * Generate Scala Types from an OpenAPI Spec — Read the Component Schemas
 *
 * Reads every component schema as a typed `JsonSchema.Object` and prints its
 * type, properties and required properties, and the discriminator of `Payment`.
 */
object Step2ComponentSchemasExample {
  // Setup from "Parse the Spec": the parsed spec.
  def parseSpec(text: String): OpenAPI =
    Json.parse(text).flatMap(json => OpenAPICodec.openAPICodec.decode(json)) match {
      case Right(api)  => api
      case Left(error) => throw new IllegalArgumentException(s"Not an OpenAPI document: ${error.getMessage}")
    }

  val api: OpenAPI = parseSpec(bookstoreJson)

  // Read the Component Schemas
  final case class Component(name: String, schema: JsonSchema.Object, discriminator: Option[Discriminator])

  def componentsOf(api: OpenAPI): List[Component] =
    api.components.toList.flatMap(_.schemas.toList).map {
      case (name, ReferenceOr.Value(schemaObject)) =>
        schemaObject.toJsonSchema match {
          case Right(schema: JsonSchema.Object) => Component(name, schema, schemaObject.discriminator)
          case _                                => throw new IllegalArgumentException(s"$name: unsupported schema")
        }
      case (name, ReferenceOr.Ref(_)) =>
        throw new IllegalArgumentException(s"$name: a component that is only a $$ref is not supported")
    }

  val components: List[Component] = componentsOf(api)

  def main(args: Array[String]): Unit = {
    components.foreach { component =>
      val schema = component.schema
      println(
        s"${component.name}: type=${schema.`type`.map(_.toJson.print).getOrElse("-")}, " +
          s"properties=${schema.properties.map(_.toList.map(_._1).mkString("[", ", ", "]")).getOrElse("-")}, " +
          s"required=${schema.required.map(_.toList.sorted.mkString("[", ", ", "]")).getOrElse("-")}"
      )
    }
    println(components.find(_.name == "Payment").flatMap(_.discriminator))
  }
}
```

</details>

## Map Schema Types to Scala Types

Next, we decide which Scala type each property gets. A property is either a `$ref` to another component, which becomes that component's type by name, or an inline schema with a `type` and an optional `format`. `TypeRef` is codegen's description of a type, such as `TypeRef.Long` or `TypeRef.list(TypeRef("Author"))`.

Our mapping follows the formats the OpenAPI specification defines, and picks a type with a built-in `Schema` for each:

| JSON Schema                               | Scala type                     |
| ----------------------------------------- | ------------------------------ |
| `string`                                  | `String`                       |
| `string`, `format: date`                  | `java.time.LocalDate`          |
| `string`, `format: date-time`             | `java.time.Instant`            |
| `string`, `format: uuid`                  | `java.util.UUID`               |
| `integer`, `format: int32`                | `Int`                          |
| `integer`, `format: int64` or no format   | `Long`                         |
| `number`, `format: float`                 | `Float`                        |
| `number`, `format: double`                | `Double`                       |
| `number` with no format                   | `BigDecimal`                   |
| `boolean`                                 | `Boolean`                      |
| `array` with `items: A`                   | `List[A]`                      |
| `object` with only `additionalProperties` | `Map[String, A]`               |
| `$ref: '#/components/schemas/X'`          | `X`                            |
| `type: [A, "null"]` (OpenAPI 3.1)         | `Option[A]`                    |
| `nullable: true` (OpenAPI 3.0)            | `Option[A]`                    |

Every other format, such as `uri`, `email`, `byte` or `binary`, stays a `String`, which is what the JSON carries.

The two functions call each other, because an array's items and a map's values are schemas too:

```scala

val refPrefix = "#/components/schemas/"

def refName(schema: JsonSchema): Option[String] = schema match {
  case obj: JsonSchema.Object => obj.$ref.map(_.value.stripPrefix(refPrefix))
  case _                      => None
}

def scalarType(jsonType: JsonSchemaType, schema: JsonSchema.Object): TypeRef = jsonType match {
  case JsonSchemaType.String =>
    schema.format match {
      case Some("date")      => TypeRef("java.time.LocalDate")
      case Some("date-time") => TypeRef("java.time.Instant")
      case Some("uuid")      => TypeRef("java.util.UUID")
      case _                 => TypeRef.String
    }
  case JsonSchemaType.Integer =>
    if (schema.format.contains("int32")) TypeRef.Int else TypeRef.Long
  case JsonSchemaType.Number =>
    schema.format match {
      case Some("float")  => TypeRef.Float
      case Some("double") => TypeRef.Double
      case _              => TypeRef.BigDecimal
    }
  case JsonSchemaType.Boolean =>
    TypeRef.Boolean
  case JsonSchemaType.Array if schema.items.isDefined =>
    TypeRef.list(typeRefFor(schema.items.get))
  case JsonSchemaType.Object if schema.properties.isEmpty && schema.additionalProperties.isDefined =>
    TypeRef.map(TypeRef.String, typeRefFor(schema.additionalProperties.get))
  case other =>
    throw new IllegalArgumentException(s"Unsupported inline schema of type ${other.toJsonString}: ${schema.toJson}")
}

def typeRefFor(schema: JsonSchema): TypeRef = schema match {
  case obj: JsonSchema.Object if obj.$ref.isDefined =>
    TypeRef(refName(obj).get)
  case obj: JsonSchema.Object =>
    val nullable30 = obj.extensions.get("nullable").contains(Json.True)
    obj.`type` match {
      case Some(SchemaType.Single(jsonType)) if nullable30 =>
        TypeRef.optional(scalarType(jsonType, obj))
      case Some(SchemaType.Single(jsonType)) =>
        scalarType(jsonType, obj)
      case Some(SchemaType.Union(types)) if types.length == 2 && types.contains(JsonSchemaType.Null) =>
        TypeRef.optional(scalarType(types.find(_ != JsonSchemaType.Null).get, obj))
      case _ =>
        throw new IllegalArgumentException(s"Unsupported inline schema: ${obj.toJson}")
    }
  case other =>
    throw new IllegalArgumentException(s"Unsupported inline schema: ${other.toJson}")
}
```

The code above handles nullability in both spec versions. OpenAPI 3.1 writes a nullable string as `type: ["string", "null"]`, which the typed schema reads as a `SchemaType.Union`. OpenAPI 3.0 writes it as `nullable: true`, a keyword JSON Schema 2020-12 doesn't know, so `JsonSchema` keeps it in `extensions` and `typeRefFor` looks it up there. Any schema the table doesn't cover, such as an inline object with its own properties, stops the generator with the schema in the message, rather than guessing.

To check the mapping, we map the properties of `Book` and `Order` and render each `TypeRef` as source text with `ScalaEmitter.emitTypeRef`:

```scala
for {
  component           <- components.filter(c => c.name == "Book" || c.name == "Order")
  (property, schema) <- component.schema.properties.toList.flatMap(_.toList)
} println(s"${component.name}.$property: ${ScalaEmitter.emitTypeRef(typeRefFor(schema))}")
// Book.id: String
// Book.title: String
// Book.authors: List[Author]
// Book.format: BookFormat
// Book.price: BigDecimal
// Book.pages: Int
// Book.publishedOn: java.time.LocalDate
// Book.subtitle: Option[String]
// Book.tags: List[String]
// Order.id: java.util.UUID
// Order.customerId: Long
// Order.placedAt: java.time.Instant
// Order.lines: List[OrderLine]
// Order.payment: Payment
// Order.status: OrderStatus
// Order.giftMessage: String
// Order.metadata: Map[String, String]
```

The same function reads an OpenAPI 3.0 nullable property:

```scala
println(ScalaEmitter.emitTypeRef(typeRefFor(JsonSchema.parse("""{"type": "string", "nullable": true}""").toOption.get)))
// Option[String]
```

:::note
`date-time` maps to `Instant`, not `OffsetDateTime`, because `Instant` always writes its seconds: in 0.0.56 the `OffsetDateTime` codec writes `2026-10-27T09:30:00+03:30` as `"2026-10-27T09:30+03:30"`, which strict RFC 3339 parsers reject. `Instant` decodes a value with any offset, but keeps only the instant, not the offset.
:::

<details>
  <summary>Runnable version, with setup from the previous sections: <code>Step3MapTypesExample.scala</code></summary>

```scala title="openapi-examples/src/main/scala/generatescalatypesfromopenapi/Step3MapTypesExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-next-openapi:0.0.56
//> using dep dev.zio::zio-blocks-next-codegen:0.0.56
//> using options -experimental
//> using file BookstoreSpec.scala
// Run from this folder:  scala-cli run Step3MapTypesExample.scala BookstoreSpec.scala
// Or from openapi-examples:  sbt "runMain generatescalatypesfromopenapi.Step3MapTypesExample"

package generatescalatypesfromopenapi

/**
 * Generate Scala Types from an OpenAPI Spec — Map Schema Types to Scala Types
 *
 * Maps the properties of `Book` and `Order`, and an OpenAPI 3.0 nullable
 * property, to codegen `TypeRef`s and prints them as Scala source.
 */
object Step3MapTypesExample {
  // Setup from "Parse the Spec" and "Read the Component Schemas": the parsed spec and its components.
  def parseSpec(text: String): OpenAPI =
    Json.parse(text).flatMap(json => OpenAPICodec.openAPICodec.decode(json)) match {
      case Right(api)  => api
      case Left(error) => throw new IllegalArgumentException(s"Not an OpenAPI document: ${error.getMessage}")
    }

  val api: OpenAPI = parseSpec(bookstoreJson)

  final case class Component(name: String, schema: JsonSchema.Object, discriminator: Option[Discriminator])

  def componentsOf(api: OpenAPI): List[Component] =
    api.components.toList.flatMap(_.schemas.toList).map {
      case (name, ReferenceOr.Value(schemaObject)) =>
        schemaObject.toJsonSchema match {
          case Right(schema: JsonSchema.Object) => Component(name, schema, schemaObject.discriminator)
          case _                                => throw new IllegalArgumentException(s"$name: unsupported schema")
        }
      case (name, ReferenceOr.Ref(_)) =>
        throw new IllegalArgumentException(s"$name: a component that is only a $$ref is not supported")
    }

  val components: List[Component] = componentsOf(api)

  // Map Schema Types to Scala Types
  val refPrefix = "#/components/schemas/"

  def refName(schema: JsonSchema): Option[String] = schema match {
    case obj: JsonSchema.Object => obj.$ref.map(_.value.stripPrefix(refPrefix))
    case _                      => None
  }

  def scalarType(jsonType: JsonSchemaType, schema: JsonSchema.Object): TypeRef = jsonType match {
    case JsonSchemaType.String =>
      schema.format match {
        case Some("date")      => TypeRef("java.time.LocalDate")
        case Some("date-time") => TypeRef("java.time.Instant")
        case Some("uuid")      => TypeRef("java.util.UUID")
        case _                 => TypeRef.String
      }
    case JsonSchemaType.Integer =>
      if (schema.format.contains("int32")) TypeRef.Int else TypeRef.Long
    case JsonSchemaType.Number =>
      schema.format match {
        case Some("float")  => TypeRef.Float
        case Some("double") => TypeRef.Double
        case _              => TypeRef.BigDecimal
      }
    case JsonSchemaType.Boolean =>
      TypeRef.Boolean
    case JsonSchemaType.Array if schema.items.isDefined =>
      TypeRef.list(typeRefFor(schema.items.get))
    case JsonSchemaType.Object if schema.properties.isEmpty && schema.additionalProperties.isDefined =>
      TypeRef.map(TypeRef.String, typeRefFor(schema.additionalProperties.get))
    case other =>
      throw new IllegalArgumentException(s"Unsupported inline schema of type ${other.toJsonString}: ${schema.toJson}")
  }

  def typeRefFor(schema: JsonSchema): TypeRef = schema match {
    case obj: JsonSchema.Object if obj.$ref.isDefined =>
      TypeRef(refName(obj).get)
    case obj: JsonSchema.Object =>
      val nullable30 = obj.extensions.get("nullable").contains(Json.True)
      obj.`type` match {
        case Some(SchemaType.Single(jsonType)) if nullable30 =>
          TypeRef.optional(scalarType(jsonType, obj))
        case Some(SchemaType.Single(jsonType)) =>
          scalarType(jsonType, obj)
        case Some(SchemaType.Union(types)) if types.length == 2 && types.contains(JsonSchemaType.Null) =>
          TypeRef.optional(scalarType(types.find(_ != JsonSchemaType.Null).get, obj))
        case _ =>
          throw new IllegalArgumentException(s"Unsupported inline schema: ${obj.toJson}")
      }
    case other =>
      throw new IllegalArgumentException(s"Unsupported inline schema: ${other.toJson}")
  }

  def main(args: Array[String]): Unit = {
    for {
      component          <- components.filter(c => c.name == "Book" || c.name == "Order")
      (property, schema) <- component.schema.properties.toList.flatMap(_.toList)
    } println(s"${component.name}.$property: ${ScalaEmitter.emitTypeRef(typeRefFor(schema))}")

    println(
      ScalaEmitter.emitTypeRef(typeRefFor(JsonSchema.parse("""{"type": "string", "nullable": true}""").toOption.get))
    )
  }
}
```

</details>

## Turn Objects, Enums and oneOf into Scala Types

With every property mapped, each component becomes one Scala type. There are three kinds in our spec, and codegen has a type definition for each: an object becomes a `CaseClass`, a string `enum` becomes an `Enum`, and a `oneOf` with a discriminator becomes a `SealedTrait`.

An object's properties become `Field`s. A required property keeps its type. Any other property may be missing from the JSON, so it becomes an `Option` with a default of `None`; when the type is already an `Option` (a nullable property), we don't wrap it twice:

```scala
def fieldFor(name: String, schema: JsonSchema, required: Boolean): Field = {
  val fieldType = typeRefFor(schema)
  if (required) Field(name, fieldType)
  else if (fieldType.name == "Option") Field(name, fieldType, defaultValue = Some("None"))
  else Field(name, TypeRef.optional(fieldType), defaultValue = Some("None"))
}

def caseClassFor(name: String, schema: JsonSchema.Object, skip: Set[String] = Set.empty): CaseClass = {
  val required   = schema.required.getOrElse(Set.empty[String])
  val properties = schema.properties.map(_.toList).getOrElse(Nil)
  val fields     = properties.collect {
    case (property, propertySchema) if !skip.contains(property) =>
      fieldFor(property, propertySchema, required.contains(property))
  }
  CaseClass(name, fields, doc = schema.description)
}

def enumFor(name: String, schema: JsonSchema.Object): Enum = {
  val values = schema.`enum`.toList.flatMap(_.toList).map {
    case Json.String(value) => EnumCase.SimpleCase(value)
    case other              => throw new IllegalArgumentException(s"$name: non-string enum value $other")
  }
  Enum(name, values, doc = schema.description)
}
```

The schema's `description` becomes the type's Scaladoc. An enum value becomes a case name as it is, so `"Ebook"` becomes `Ebook`, and the derived JSON codec writes the case back as the string `"Ebook"`.

The `oneOf` needs more care, because its JSON has a shape of its own. The spec says a payment is one of `CardPayment` or `GiftCardPayment`, and that the `method` property tells them apart: `{"method": "card", "last4": "4242"}` is a card payment. ZIO Blocks' JSON codec writes a sealed trait as `{"CardPayment": {...}}` by default, so the generator has to annotate the generated types to match the spec's shape:

- `@Modifier.discriminator(name = "method")` on the sealed trait makes the codec write and read the case name in a `method` property.
- `@Modifier.rename(name = "card")` on a case makes the codec use the discriminator's mapping key, `card`, instead of the class name. A case with no mapping entry keeps its schema name, which is also what OpenAPI expects.
- The `method` property is left out of each case class: the codec owns that property now, and a field with the same name would clash with it.

Codegen's `Annotation` takes the annotation's name and its arguments as source text, so a string argument needs its quotes:

```scala
def stringLiteral(value: String): String = "\"" + value + "\""

def sealedTraitFor(component: Component, byName: Map[String, Component]): SealedTrait = {
  val discriminator = component.discriminator.get
  val wireNames     = discriminator.mapping.toList.map { case (value, ref) => ref.stripPrefix(refPrefix) -> value }.toMap
  val cases         = component.schema.oneOf.get.toList.flatMap(refName).map { caseName =>
    val caseClass = caseClassFor(caseName, byName(caseName).schema, skip = Set(discriminator.propertyName))
    val renamed   = wireNames.get(caseName) match {
      case Some(wireName) => caseClass.copy(annotations = List(Annotation("Modifier.rename", List("name" -> stringLiteral(wireName)))))
      case None           => caseClass
    }
    SealedTraitCase.CaseClassCase(renamed)
  }
  SealedTrait(
    component.name,
    cases = cases,
    annotations = List(Annotation("Modifier.discriminator", List("name" -> stringLiteral(discriminator.propertyName)))),
    doc = component.schema.description
  )
}

val byName: Map[String, Component] = components.map(component => component.name -> component).toMap
```

To see one type of each kind as source, we emit them one at a time. `ScalaEmitter.emitTypeDefinition` renders a single type definition with an `EmitterConfig`; `EmitterConfig.default` writes Scala 3:

```scala
println(ScalaEmitter.emitTypeDefinition(caseClassFor("Book", byName("Book").schema), EmitterConfig.default))
// /** A book in the catalog. */
// case class Book(
//   id: String,
//   title: String,
//   authors: List[Author],
//   format: BookFormat,
//   price: BigDecimal,
//   pages: Option[Int] = None,
//   publishedOn: Option[java.time.LocalDate] = None,
//   subtitle: Option[String] = None,
//   tags: Option[List[String]] = None,
// )
println(ScalaEmitter.emitTypeDefinition(enumFor("BookFormat", byName("BookFormat").schema), EmitterConfig.default))
// /** How the book is delivered. */
// enum BookFormat {
//   case Hardcover, Paperback, Ebook
// }
println(ScalaEmitter.emitTypeDefinition(sealedTraitFor(byName("Payment"), byName), EmitterConfig.default))
// /** How the customer paid. */
// @Modifier.discriminator(name = "method")
// sealed trait Payment
// 
// object Payment {
//   @Modifier.rename(name = "card")
//   case class CardPayment(
//     last4: String,
//   ) extends Payment
//   @Modifier.rename(name = "gift_card")
//   case class GiftCardPayment(
//     code: String,
//   ) extends Payment
// }
```

`Book` has its five required properties as plain fields and the other four as `Option`s, with `subtitle` wrapped once even though it is both optional and nullable. The trailing comma after the last field is Scala 3 style.

<details>
  <summary>Runnable version, with setup from the previous sections: <code>Step4TypeDefinitionsExample.scala</code></summary>

```scala title="openapi-examples/src/main/scala/generatescalatypesfromopenapi/Step4TypeDefinitionsExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-next-openapi:0.0.56
//> using dep dev.zio::zio-blocks-next-codegen:0.0.56
//> using options -experimental
//> using file BookstoreSpec.scala
// Run from this folder:  scala-cli run Step4TypeDefinitionsExample.scala BookstoreSpec.scala
// Or from openapi-examples:  sbt "runMain generatescalatypesfromopenapi.Step4TypeDefinitionsExample"

package generatescalatypesfromopenapi

/**
 * Generate Scala Types from an OpenAPI Spec — Turn Objects, Enums and oneOf
 * into Scala Types
 *
 * Turns `Book` into a case class, `BookFormat` into an enum and `Payment` into
 * a sealed trait with discriminator annotations, and prints each one.
 */
object Step4TypeDefinitionsExample {
  // Setup from the previous sections: the parsed spec, its components and the type mapping.
  def parseSpec(text: String): OpenAPI =
    Json.parse(text).flatMap(json => OpenAPICodec.openAPICodec.decode(json)) match {
      case Right(api)  => api
      case Left(error) => throw new IllegalArgumentException(s"Not an OpenAPI document: ${error.getMessage}")
    }

  val api: OpenAPI = parseSpec(bookstoreJson)

  final case class Component(name: String, schema: JsonSchema.Object, discriminator: Option[Discriminator])

  def componentsOf(api: OpenAPI): List[Component] =
    api.components.toList.flatMap(_.schemas.toList).map {
      case (name, ReferenceOr.Value(schemaObject)) =>
        schemaObject.toJsonSchema match {
          case Right(schema: JsonSchema.Object) => Component(name, schema, schemaObject.discriminator)
          case _                                => throw new IllegalArgumentException(s"$name: unsupported schema")
        }
      case (name, ReferenceOr.Ref(_)) =>
        throw new IllegalArgumentException(s"$name: a component that is only a $$ref is not supported")
    }

  val components: List[Component] = componentsOf(api)

  val refPrefix = "#/components/schemas/"

  def refName(schema: JsonSchema): Option[String] = schema match {
    case obj: JsonSchema.Object => obj.$ref.map(_.value.stripPrefix(refPrefix))
    case _                      => None
  }

  def scalarType(jsonType: JsonSchemaType, schema: JsonSchema.Object): TypeRef = jsonType match {
    case JsonSchemaType.String =>
      schema.format match {
        case Some("date")      => TypeRef("java.time.LocalDate")
        case Some("date-time") => TypeRef("java.time.Instant")
        case Some("uuid")      => TypeRef("java.util.UUID")
        case _                 => TypeRef.String
      }
    case JsonSchemaType.Integer =>
      if (schema.format.contains("int32")) TypeRef.Int else TypeRef.Long
    case JsonSchemaType.Number =>
      schema.format match {
        case Some("float")  => TypeRef.Float
        case Some("double") => TypeRef.Double
        case _              => TypeRef.BigDecimal
      }
    case JsonSchemaType.Boolean =>
      TypeRef.Boolean
    case JsonSchemaType.Array if schema.items.isDefined =>
      TypeRef.list(typeRefFor(schema.items.get))
    case JsonSchemaType.Object if schema.properties.isEmpty && schema.additionalProperties.isDefined =>
      TypeRef.map(TypeRef.String, typeRefFor(schema.additionalProperties.get))
    case other =>
      throw new IllegalArgumentException(s"Unsupported inline schema of type ${other.toJsonString}: ${schema.toJson}")
  }

  def typeRefFor(schema: JsonSchema): TypeRef = schema match {
    case obj: JsonSchema.Object if obj.$ref.isDefined =>
      TypeRef(refName(obj).get)
    case obj: JsonSchema.Object =>
      val nullable30 = obj.extensions.get("nullable").contains(Json.True)
      obj.`type` match {
        case Some(SchemaType.Single(jsonType)) if nullable30 =>
          TypeRef.optional(scalarType(jsonType, obj))
        case Some(SchemaType.Single(jsonType)) =>
          scalarType(jsonType, obj)
        case Some(SchemaType.Union(types)) if types.length == 2 && types.contains(JsonSchemaType.Null) =>
          TypeRef.optional(scalarType(types.find(_ != JsonSchemaType.Null).get, obj))
        case _ =>
          throw new IllegalArgumentException(s"Unsupported inline schema: ${obj.toJson}")
      }
    case other =>
      throw new IllegalArgumentException(s"Unsupported inline schema: ${other.toJson}")
  }

  // Turn Objects, Enums and oneOf into Scala Types
  def fieldFor(name: String, schema: JsonSchema, required: Boolean): Field = {
    val fieldType = typeRefFor(schema)
    if (required) Field(name, fieldType)
    else if (fieldType.name == "Option") Field(name, fieldType, defaultValue = Some("None"))
    else Field(name, TypeRef.optional(fieldType), defaultValue = Some("None"))
  }

  def caseClassFor(name: String, schema: JsonSchema.Object, skip: Set[String] = Set.empty): CaseClass = {
    val required   = schema.required.getOrElse(Set.empty[String])
    val properties = schema.properties.map(_.toList).getOrElse(Nil)
    val fields     = properties.collect {
      case (property, propertySchema) if !skip.contains(property) =>
        fieldFor(property, propertySchema, required.contains(property))
    }
    CaseClass(name, fields, doc = schema.description)
  }

  def enumFor(name: String, schema: JsonSchema.Object): Enum = {
    val values = schema.`enum`.toList.flatMap(_.toList).map {
      case Json.String(value) => EnumCase.SimpleCase(value)
      case other              => throw new IllegalArgumentException(s"$name: non-string enum value $other")
    }
    Enum(name, values, doc = schema.description)
  }

  def stringLiteral(value: String): String = "\"" + value + "\""

  def sealedTraitFor(component: Component, byName: Map[String, Component]): SealedTrait = {
    val discriminator = component.discriminator.get
    val wireNames     = discriminator.mapping.toList.map { case (value, ref) => ref.stripPrefix(refPrefix) -> value }.toMap
    val cases         = component.schema.oneOf.get.toList.flatMap(refName).map { caseName =>
      val caseClass = caseClassFor(caseName, byName(caseName).schema, skip = Set(discriminator.propertyName))
      val renamed   = wireNames.get(caseName) match {
        case Some(wireName) =>
          caseClass.copy(annotations = List(Annotation("Modifier.rename", List("name" -> stringLiteral(wireName)))))
        case None => caseClass
      }
      SealedTraitCase.CaseClassCase(renamed)
    }
    SealedTrait(
      component.name,
      cases = cases,
      annotations =
        List(Annotation("Modifier.discriminator", List("name" -> stringLiteral(discriminator.propertyName)))),
      doc = component.schema.description
    )
  }

  val byName: Map[String, Component] = components.map(component => component.name -> component).toMap

  def main(args: Array[String]): Unit = {
    println(ScalaEmitter.emitTypeDefinition(caseClassFor("Book", byName("Book").schema), EmitterConfig.default))
    println(ScalaEmitter.emitTypeDefinition(enumFor("BookFormat", byName("BookFormat").schema), EmitterConfig.default))
    println(ScalaEmitter.emitTypeDefinition(sealedTraitFor(byName("Payment"), byName), EmitterConfig.default))
  }
}
```

</details>

## Emit the File for Scala 3 and Scala 2

A `ScalaFile` holds a package declaration, imports and type definitions, and `ScalaEmitter.emit` turns the whole file into source text. OpenAPI has no package names, so the generator takes one as a parameter.

The schemas that a `oneOf` lists are already cases of their sealed trait, so `scalaFileFor` leaves them out of the top level; every other component goes through the function for its kind:

```scala
def isSealedTrait(component: Component): Boolean =
  component.schema.oneOf.isDefined && component.discriminator.isDefined

def scalaFileFor(api: OpenAPI, packageName: String): ScalaFile = {
  val components = componentsOf(api)
  val byName     = components.map(component => component.name -> component).toMap
  val caseNames  = components.filter(isSealedTrait).flatMap(_.schema.oneOf.get.toList.flatMap(refName)).toSet
  val types      = components.filterNot(component => caseNames.contains(component.name)).map {
    case component if isSealedTrait(component)                     => sealedTraitFor(component, byName)
    case Component(name, schema, _) if schema.`enum`.isDefined     => enumFor(name, schema)
    case Component(name, schema, _) if schema.properties.isDefined => caseClassFor(name, schema)
    case Component(name, schema, _) =>
      throw new IllegalArgumentException(s"$name: unsupported component schema ${schema.toJson}")
  }
  ScalaFile(PackageDecl(packageName), types = types)
}

val bookstoreFile: ScalaFile = scalaFileFor(api, "com.example.bookstore")
```

The same `ScalaFile` emits Scala 3 source with `EmitterConfig.default`:

```scala
println(ScalaEmitter.emit(bookstoreFile, EmitterConfig.default))
// package com.example.bookstore
// 
// /** A book in the catalog. */
// case class Book(
//   id: String,
//   title: String,
//   authors: List[Author],
//   format: BookFormat,
//   price: BigDecimal,
//   pages: Option[Int] = None,
//   publishedOn: Option[java.time.LocalDate] = None,
//   subtitle: Option[String] = None,
//   tags: Option[List[String]] = None,
// )
// 
// case class Author(
//   name: String,
//   website: Option[String] = None,
// )
// 
// /** How the book is delivered. */
// enum BookFormat {
//   case Hardcover, Paperback, Ebook
// }
// 
// /** A customer's order. */
// case class Order(
//   id: java.util.UUID,
//   customerId: Long,
//   placedAt: java.time.Instant,
//   lines: List[OrderLine],
//   payment: Payment,
//   status: OrderStatus,
//   giftMessage: Option[String] = None,
//   metadata: Option[Map[String, String]] = None,
// )
// 
// case class OrderLine(
//   bookId: String,
//   quantity: Int,
//   unitPrice: BigDecimal,
// )
// 
// enum OrderStatus {
//   case Placed, Shipped, Delivered, Cancelled
// }
// 
// /** How the customer paid. */
// @Modifier.discriminator(name = "method")
// sealed trait Payment
// 
// object Payment {
//   @Modifier.rename(name = "card")
//   case class CardPayment(
//     last4: String,
//   ) extends Payment
//   @Modifier.rename(name = "gift_card")
//   case class GiftCardPayment(
//     code: String,
//   ) extends Payment
// }
//
```

And Scala 2 source with `EmitterConfig.scala2`:

```scala
println(ScalaEmitter.emit(bookstoreFile, EmitterConfig.scala2))
// package com.example.bookstore
// 
// /** A book in the catalog. */
// case class Book(
//   id: String,
//   title: String,
//   authors: List[Author],
//   format: BookFormat,
//   price: BigDecimal,
//   pages: Option[Int] = None,
//   publishedOn: Option[java.time.LocalDate] = None,
//   subtitle: Option[String] = None,
//   tags: Option[List[String]] = None
// )
// 
// case class Author(
//   name: String,
//   website: Option[String] = None
// )
// 
// /** How the book is delivered. */
// sealed trait BookFormat
// 
// object BookFormat {
//   case object Hardcover extends BookFormat
//   case object Paperback extends BookFormat
//   case object Ebook extends BookFormat
// }
// 
// /** A customer's order. */
// case class Order(
//   id: java.util.UUID,
//   customerId: Long,
//   placedAt: java.time.Instant,
//   lines: List[OrderLine],
//   payment: Payment,
//   status: OrderStatus,
//   giftMessage: Option[String] = None,
//   metadata: Option[Map[String, String]] = None
// )
// 
// case class OrderLine(
//   bookId: String,
//   quantity: Int,
//   unitPrice: BigDecimal
// )
// 
// sealed trait OrderStatus
// 
// object OrderStatus {
//   case object Placed extends OrderStatus
//   case object Shipped extends OrderStatus
//   case object Delivered extends OrderStatus
//   case object Cancelled extends OrderStatus
// }
// 
// /** How the customer paid. */
// @Modifier.discriminator(name = "method")
// sealed trait Payment
// 
// object Payment {
//   @Modifier.rename(name = "card")
//   case class CardPayment(
//     last4: String
//   ) extends Payment
//   @Modifier.rename(name = "gift_card")
//   case class GiftCardPayment(
//     code: String
//   ) extends Payment
// }
//
```

The generator doesn't change between the two: only the config does. The Scala 2 version writes each `Enum` as a sealed trait with case objects, and drops the trailing commas.

<details>
  <summary>Runnable version, with setup from the previous sections: <code>Step5EmitBothDialectsExample.scala</code></summary>

```scala title="openapi-examples/src/main/scala/generatescalatypesfromopenapi/Step5EmitBothDialectsExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-next-openapi:0.0.56
//> using dep dev.zio::zio-blocks-next-codegen:0.0.56
//> using options -experimental
//> using file BookstoreSpec.scala
// Run from this folder:  scala-cli run Step5EmitBothDialectsExample.scala BookstoreSpec.scala
// Or from openapi-examples:  sbt "runMain generatescalatypesfromopenapi.Step5EmitBothDialectsExample"

package generatescalatypesfromopenapi

/**
 * Generate Scala Types from an OpenAPI Spec — Emit the File for Scala 3 and
 * Scala 2
 *
 * Builds one `ScalaFile` from the spec and emits it as Scala 3 and as Scala 2
 * source.
 */
object Step5EmitBothDialectsExample {
  // Setup from the previous sections: the parsed spec, the type mapping and the type definitions.
  def parseSpec(text: String): OpenAPI =
    Json.parse(text).flatMap(json => OpenAPICodec.openAPICodec.decode(json)) match {
      case Right(api)  => api
      case Left(error) => throw new IllegalArgumentException(s"Not an OpenAPI document: ${error.getMessage}")
    }

  val api: OpenAPI = parseSpec(bookstoreJson)

  final case class Component(name: String, schema: JsonSchema.Object, discriminator: Option[Discriminator])

  def componentsOf(api: OpenAPI): List[Component] =
    api.components.toList.flatMap(_.schemas.toList).map {
      case (name, ReferenceOr.Value(schemaObject)) =>
        schemaObject.toJsonSchema match {
          case Right(schema: JsonSchema.Object) => Component(name, schema, schemaObject.discriminator)
          case _                                => throw new IllegalArgumentException(s"$name: unsupported schema")
        }
      case (name, ReferenceOr.Ref(_)) =>
        throw new IllegalArgumentException(s"$name: a component that is only a $$ref is not supported")
    }

  val components: List[Component] = componentsOf(api)

  val refPrefix = "#/components/schemas/"

  def refName(schema: JsonSchema): Option[String] = schema match {
    case obj: JsonSchema.Object => obj.$ref.map(_.value.stripPrefix(refPrefix))
    case _                      => None
  }

  def scalarType(jsonType: JsonSchemaType, schema: JsonSchema.Object): TypeRef = jsonType match {
    case JsonSchemaType.String =>
      schema.format match {
        case Some("date")      => TypeRef("java.time.LocalDate")
        case Some("date-time") => TypeRef("java.time.Instant")
        case Some("uuid")      => TypeRef("java.util.UUID")
        case _                 => TypeRef.String
      }
    case JsonSchemaType.Integer =>
      if (schema.format.contains("int32")) TypeRef.Int else TypeRef.Long
    case JsonSchemaType.Number =>
      schema.format match {
        case Some("float")  => TypeRef.Float
        case Some("double") => TypeRef.Double
        case _              => TypeRef.BigDecimal
      }
    case JsonSchemaType.Boolean =>
      TypeRef.Boolean
    case JsonSchemaType.Array if schema.items.isDefined =>
      TypeRef.list(typeRefFor(schema.items.get))
    case JsonSchemaType.Object if schema.properties.isEmpty && schema.additionalProperties.isDefined =>
      TypeRef.map(TypeRef.String, typeRefFor(schema.additionalProperties.get))
    case other =>
      throw new IllegalArgumentException(s"Unsupported inline schema of type ${other.toJsonString}: ${schema.toJson}")
  }

  def typeRefFor(schema: JsonSchema): TypeRef = schema match {
    case obj: JsonSchema.Object if obj.$ref.isDefined =>
      TypeRef(refName(obj).get)
    case obj: JsonSchema.Object =>
      val nullable30 = obj.extensions.get("nullable").contains(Json.True)
      obj.`type` match {
        case Some(SchemaType.Single(jsonType)) if nullable30 =>
          TypeRef.optional(scalarType(jsonType, obj))
        case Some(SchemaType.Single(jsonType)) =>
          scalarType(jsonType, obj)
        case Some(SchemaType.Union(types)) if types.length == 2 && types.contains(JsonSchemaType.Null) =>
          TypeRef.optional(scalarType(types.find(_ != JsonSchemaType.Null).get, obj))
        case _ =>
          throw new IllegalArgumentException(s"Unsupported inline schema: ${obj.toJson}")
      }
    case other =>
      throw new IllegalArgumentException(s"Unsupported inline schema: ${other.toJson}")
  }

  def fieldFor(name: String, schema: JsonSchema, required: Boolean): Field = {
    val fieldType = typeRefFor(schema)
    if (required) Field(name, fieldType)
    else if (fieldType.name == "Option") Field(name, fieldType, defaultValue = Some("None"))
    else Field(name, TypeRef.optional(fieldType), defaultValue = Some("None"))
  }

  def caseClassFor(name: String, schema: JsonSchema.Object, skip: Set[String] = Set.empty): CaseClass = {
    val required   = schema.required.getOrElse(Set.empty[String])
    val properties = schema.properties.map(_.toList).getOrElse(Nil)
    val fields     = properties.collect {
      case (property, propertySchema) if !skip.contains(property) =>
        fieldFor(property, propertySchema, required.contains(property))
    }
    CaseClass(name, fields, doc = schema.description)
  }

  def enumFor(name: String, schema: JsonSchema.Object): Enum = {
    val values = schema.`enum`.toList.flatMap(_.toList).map {
      case Json.String(value) => EnumCase.SimpleCase(value)
      case other              => throw new IllegalArgumentException(s"$name: non-string enum value $other")
    }
    Enum(name, values, doc = schema.description)
  }

  def stringLiteral(value: String): String = "\"" + value + "\""

  def sealedTraitFor(component: Component, byName: Map[String, Component]): SealedTrait = {
    val discriminator = component.discriminator.get
    val wireNames     = discriminator.mapping.toList.map { case (value, ref) => ref.stripPrefix(refPrefix) -> value }.toMap
    val cases         = component.schema.oneOf.get.toList.flatMap(refName).map { caseName =>
      val caseClass = caseClassFor(caseName, byName(caseName).schema, skip = Set(discriminator.propertyName))
      val renamed   = wireNames.get(caseName) match {
        case Some(wireName) =>
          caseClass.copy(annotations = List(Annotation("Modifier.rename", List("name" -> stringLiteral(wireName)))))
        case None => caseClass
      }
      SealedTraitCase.CaseClassCase(renamed)
    }
    SealedTrait(
      component.name,
      cases = cases,
      annotations =
        List(Annotation("Modifier.discriminator", List("name" -> stringLiteral(discriminator.propertyName)))),
      doc = component.schema.description
    )
  }

  val byName: Map[String, Component] = components.map(component => component.name -> component).toMap

  // Emit the File for Scala 3 and Scala 2
  def isSealedTrait(component: Component): Boolean =
    component.schema.oneOf.isDefined && component.discriminator.isDefined

  def scalaFileFor(api: OpenAPI, packageName: String): ScalaFile = {
    val components = componentsOf(api)
    val byName     = components.map(component => component.name -> component).toMap
    val caseNames  = components.filter(isSealedTrait).flatMap(_.schema.oneOf.get.toList.flatMap(refName)).toSet
    val types      = components.filterNot(component => caseNames.contains(component.name)).map {
      case component if isSealedTrait(component)                     => sealedTraitFor(component, byName)
      case Component(name, schema, _) if schema.`enum`.isDefined     => enumFor(name, schema)
      case Component(name, schema, _) if schema.properties.isDefined => caseClassFor(name, schema)
      case Component(name, schema, _)                                =>
        throw new IllegalArgumentException(s"$name: unsupported component schema ${schema.toJson}")
    }
    ScalaFile(PackageDecl(packageName), types = types)
  }

  val bookstoreFile: ScalaFile = scalaFileFor(api, "com.example.bookstore")

  def main(args: Array[String]): Unit = {
    println(ScalaEmitter.emit(bookstoreFile, EmitterConfig.default))

    println(ScalaEmitter.emit(bookstoreFile, EmitterConfig.scala2))
  }
}
```

</details>

## Add Schema Derivation

The generated types compile, but they can't read JSON yet. To decode the API's payloads, each type needs a `Schema`, which gives it a JSON codec. We add the derivation as a separate step over the finished `ScalaFile`, so the mapping stays free of it and you can swap in another type class.

On Scala 3, a case class gets `derives Schema`. On Scala 2, it gets a companion object with an `implicit val schema` that calls `Schema.derived`. Both need `import zio.blocks.schema._`, which also brings in the `Modifier` annotations on `Payment`:

```scala
def withSchemas(file: ScalaFile, scala3: Boolean): ScalaFile = {
  val types = file.types.map {
    case caseClass: CaseClass if scala3 =>
      caseClass.copy(derives = List("Schema"))
    case caseClass: CaseClass =>
      val schema = ObjectMember.ValMember(
        "schema",
        TypeRef.of("Schema", TypeRef(caseClass.name)),
        "Schema.derived",
        isImplicit = true
      )
      caseClass.copy(companion = Some(CompanionObject(List(schema))))
    case other => other
  }
  file.copy(imports = Import.WildcardImport("zio.blocks.schema") :: file.imports, types = types)
}
```

Only the case classes get a `Schema` of their own. Deriving the schema of `Order` also derives the schemas of the types it contains, `Payment` and `OrderStatus` included, so the codec of every top-level payload covers the whole tree. The Scala 3 version of the file is now:

```scala
println(ScalaEmitter.emit(withSchemas(bookstoreFile, scala3 = true), EmitterConfig.default))
// package com.example.bookstore
// 
// import zio.blocks.schema.*
// 
// /** A book in the catalog. */
// case class Book(
//   id: String,
//   title: String,
//   authors: List[Author],
//   format: BookFormat,
//   price: BigDecimal,
//   pages: Option[Int] = None,
//   publishedOn: Option[java.time.LocalDate] = None,
//   subtitle: Option[String] = None,
//   tags: Option[List[String]] = None,
// ) derives Schema
// 
// case class Author(
//   name: String,
//   website: Option[String] = None,
// ) derives Schema
// 
// /** How the book is delivered. */
// enum BookFormat {
//   case Hardcover, Paperback, Ebook
// }
// 
// /** A customer's order. */
// case class Order(
//   id: java.util.UUID,
//   customerId: Long,
//   placedAt: java.time.Instant,
//   lines: List[OrderLine],
//   payment: Payment,
//   status: OrderStatus,
//   giftMessage: Option[String] = None,
//   metadata: Option[Map[String, String]] = None,
// ) derives Schema
// 
// case class OrderLine(
//   bookId: String,
//   quantity: Int,
//   unitPrice: BigDecimal,
// ) derives Schema
// 
// enum OrderStatus {
//   case Placed, Shipped, Delivered, Cancelled
// }
// 
// /** How the customer paid. */
// @Modifier.discriminator(name = "method")
// sealed trait Payment
// 
// object Payment {
//   @Modifier.rename(name = "card")
//   case class CardPayment(
//     last4: String,
//   ) extends Payment
//   @Modifier.rename(name = "gift_card")
//   case class GiftCardPayment(
//     code: String,
//   ) extends Payment
// }
//
```

On Scala 2, the same step gives `Book` a companion object:

```scala
println(ScalaEmitter.emitTypeDefinition(withSchemas(bookstoreFile, scala3 = false).types.head, EmitterConfig.scala2))
// /** A book in the catalog. */
// case class Book(
//   id: String,
//   title: String,
//   authors: List[Author],
//   format: BookFormat,
//   price: BigDecimal,
//   pages: Option[Int] = None,
//   publishedOn: Option[java.time.LocalDate] = None,
//   subtitle: Option[String] = None,
//   tags: Option[List[String]] = None
// )
// 
// object Book {
//   implicit val schema: Schema[Book] = Schema.derived
// }
```

:::warning
Use `EmitterConfig.scala2` for the companion `implicit val`. With `EmitterConfig.default`, the codegen module in 0.0.56 writes it as `given val schema`, which Scala 3 rejects. The generator also can't put `derives` on a sealed trait or an enum yet, because the emitter drops it; that's why `withSchemas` changes the case classes only.
:::

We'll decode the spec's examples with these schemas once sbt compiles the generated file, in [Run the Generator on Every Build](#run-the-generator-on-every-build).

<details>
  <summary>Runnable version, with setup from the previous sections: <code>Step6SchemaDerivationExample.scala</code></summary>

```scala title="openapi-examples/src/main/scala/generatescalatypesfromopenapi/Step6SchemaDerivationExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-next-openapi:0.0.56
//> using dep dev.zio::zio-blocks-next-codegen:0.0.56
//> using options -experimental
//> using file BookstoreSpec.scala
// Run from this folder:  scala-cli run Step6SchemaDerivationExample.scala BookstoreSpec.scala
// Or from openapi-examples:  sbt "runMain generatescalatypesfromopenapi.Step6SchemaDerivationExample"

package generatescalatypesfromopenapi

/**
 * Generate Scala Types from an OpenAPI Spec — Add Schema Derivation
 *
 * Adds `derives Schema` (Scala 3) or a companion `implicit val schema` (Scala
 * 2) to the generated case classes and prints the result.
 */
object Step6SchemaDerivationExample {
  // Setup from the previous sections: the generator up to the emitted `ScalaFile`.
  def parseSpec(text: String): OpenAPI =
    Json.parse(text).flatMap(json => OpenAPICodec.openAPICodec.decode(json)) match {
      case Right(api)  => api
      case Left(error) => throw new IllegalArgumentException(s"Not an OpenAPI document: ${error.getMessage}")
    }

  val api: OpenAPI = parseSpec(bookstoreJson)

  final case class Component(name: String, schema: JsonSchema.Object, discriminator: Option[Discriminator])

  def componentsOf(api: OpenAPI): List[Component] =
    api.components.toList.flatMap(_.schemas.toList).map {
      case (name, ReferenceOr.Value(schemaObject)) =>
        schemaObject.toJsonSchema match {
          case Right(schema: JsonSchema.Object) => Component(name, schema, schemaObject.discriminator)
          case _                                => throw new IllegalArgumentException(s"$name: unsupported schema")
        }
      case (name, ReferenceOr.Ref(_)) =>
        throw new IllegalArgumentException(s"$name: a component that is only a $$ref is not supported")
    }

  val components: List[Component] = componentsOf(api)

  val refPrefix = "#/components/schemas/"

  def refName(schema: JsonSchema): Option[String] = schema match {
    case obj: JsonSchema.Object => obj.$ref.map(_.value.stripPrefix(refPrefix))
    case _                      => None
  }

  def scalarType(jsonType: JsonSchemaType, schema: JsonSchema.Object): TypeRef = jsonType match {
    case JsonSchemaType.String =>
      schema.format match {
        case Some("date")      => TypeRef("java.time.LocalDate")
        case Some("date-time") => TypeRef("java.time.Instant")
        case Some("uuid")      => TypeRef("java.util.UUID")
        case _                 => TypeRef.String
      }
    case JsonSchemaType.Integer =>
      if (schema.format.contains("int32")) TypeRef.Int else TypeRef.Long
    case JsonSchemaType.Number =>
      schema.format match {
        case Some("float")  => TypeRef.Float
        case Some("double") => TypeRef.Double
        case _              => TypeRef.BigDecimal
      }
    case JsonSchemaType.Boolean =>
      TypeRef.Boolean
    case JsonSchemaType.Array if schema.items.isDefined =>
      TypeRef.list(typeRefFor(schema.items.get))
    case JsonSchemaType.Object if schema.properties.isEmpty && schema.additionalProperties.isDefined =>
      TypeRef.map(TypeRef.String, typeRefFor(schema.additionalProperties.get))
    case other =>
      throw new IllegalArgumentException(s"Unsupported inline schema of type ${other.toJsonString}: ${schema.toJson}")
  }

  def typeRefFor(schema: JsonSchema): TypeRef = schema match {
    case obj: JsonSchema.Object if obj.$ref.isDefined =>
      TypeRef(refName(obj).get)
    case obj: JsonSchema.Object =>
      val nullable30 = obj.extensions.get("nullable").contains(Json.True)
      obj.`type` match {
        case Some(SchemaType.Single(jsonType)) if nullable30 =>
          TypeRef.optional(scalarType(jsonType, obj))
        case Some(SchemaType.Single(jsonType)) =>
          scalarType(jsonType, obj)
        case Some(SchemaType.Union(types)) if types.length == 2 && types.contains(JsonSchemaType.Null) =>
          TypeRef.optional(scalarType(types.find(_ != JsonSchemaType.Null).get, obj))
        case _ =>
          throw new IllegalArgumentException(s"Unsupported inline schema: ${obj.toJson}")
      }
    case other =>
      throw new IllegalArgumentException(s"Unsupported inline schema: ${other.toJson}")
  }

  def fieldFor(name: String, schema: JsonSchema, required: Boolean): Field = {
    val fieldType = typeRefFor(schema)
    if (required) Field(name, fieldType)
    else if (fieldType.name == "Option") Field(name, fieldType, defaultValue = Some("None"))
    else Field(name, TypeRef.optional(fieldType), defaultValue = Some("None"))
  }

  def caseClassFor(name: String, schema: JsonSchema.Object, skip: Set[String] = Set.empty): CaseClass = {
    val required   = schema.required.getOrElse(Set.empty[String])
    val properties = schema.properties.map(_.toList).getOrElse(Nil)
    val fields     = properties.collect {
      case (property, propertySchema) if !skip.contains(property) =>
        fieldFor(property, propertySchema, required.contains(property))
    }
    CaseClass(name, fields, doc = schema.description)
  }

  def enumFor(name: String, schema: JsonSchema.Object): Enum = {
    val values = schema.`enum`.toList.flatMap(_.toList).map {
      case Json.String(value) => EnumCase.SimpleCase(value)
      case other              => throw new IllegalArgumentException(s"$name: non-string enum value $other")
    }
    Enum(name, values, doc = schema.description)
  }

  def stringLiteral(value: String): String = "\"" + value + "\""

  def sealedTraitFor(component: Component, byName: Map[String, Component]): SealedTrait = {
    val discriminator = component.discriminator.get
    val wireNames     = discriminator.mapping.toList.map { case (value, ref) => ref.stripPrefix(refPrefix) -> value }.toMap
    val cases         = component.schema.oneOf.get.toList.flatMap(refName).map { caseName =>
      val caseClass = caseClassFor(caseName, byName(caseName).schema, skip = Set(discriminator.propertyName))
      val renamed   = wireNames.get(caseName) match {
        case Some(wireName) =>
          caseClass.copy(annotations = List(Annotation("Modifier.rename", List("name" -> stringLiteral(wireName)))))
        case None => caseClass
      }
      SealedTraitCase.CaseClassCase(renamed)
    }
    SealedTrait(
      component.name,
      cases = cases,
      annotations =
        List(Annotation("Modifier.discriminator", List("name" -> stringLiteral(discriminator.propertyName)))),
      doc = component.schema.description
    )
  }

  val byName: Map[String, Component] = components.map(component => component.name -> component).toMap

  def isSealedTrait(component: Component): Boolean =
    component.schema.oneOf.isDefined && component.discriminator.isDefined

  def scalaFileFor(api: OpenAPI, packageName: String): ScalaFile = {
    val components = componentsOf(api)
    val byName     = components.map(component => component.name -> component).toMap
    val caseNames  = components.filter(isSealedTrait).flatMap(_.schema.oneOf.get.toList.flatMap(refName)).toSet
    val types      = components.filterNot(component => caseNames.contains(component.name)).map {
      case component if isSealedTrait(component)                     => sealedTraitFor(component, byName)
      case Component(name, schema, _) if schema.`enum`.isDefined     => enumFor(name, schema)
      case Component(name, schema, _) if schema.properties.isDefined => caseClassFor(name, schema)
      case Component(name, schema, _)                                =>
        throw new IllegalArgumentException(s"$name: unsupported component schema ${schema.toJson}")
    }
    ScalaFile(PackageDecl(packageName), types = types)
  }

  val bookstoreFile: ScalaFile = scalaFileFor(api, "com.example.bookstore")

  // Add Schema Derivation
  def withSchemas(file: ScalaFile, scala3: Boolean): ScalaFile = {
    val types = file.types.map {
      case caseClass: CaseClass if scala3 =>
        caseClass.copy(derives = List("Schema"))
      case caseClass: CaseClass =>
        val schema = ObjectMember.ValMember(
          "schema",
          TypeRef.of("Schema", TypeRef(caseClass.name)),
          "Schema.derived",
          isImplicit = true
        )
        caseClass.copy(companion = Some(CompanionObject(List(schema))))
      case other => other
    }
    file.copy(imports = Import.WildcardImport("zio.blocks.schema") :: file.imports, types = types)
  }

  def main(args: Array[String]): Unit = {
    println(ScalaEmitter.emit(withSchemas(bookstoreFile, scala3 = true), EmitterConfig.default))

    println(
      ScalaEmitter.emitTypeDefinition(withSchemas(bookstoreFile, scala3 = false).types.head, EmitterConfig.scala2)
    )
  }
}
```

</details>

## Putting It Together

So far the spec comes from a string and the output goes to the console. A real generator reads a spec file and writes a `.scala` file. `CompleteExample` combines the functions from the previous sections into one program, with two differences: it reads the spec from the path in its first argument (by default `src/main/openapi/bookstore.json`), and it writes the Scala 3 source, with schemas, to `Bookstore.scala` in the directory named by its second argument (by default `target/generated-bookstore`):

```scala title="openapi-examples/src/main/scala/generatescalatypesfromopenapi/CompleteExample.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-next-openapi:0.0.56
//> using dep dev.zio::zio-blocks-next-codegen:0.0.56
//> using options -experimental
// Run from this folder:  scala-cli run CompleteExample.scala -- ../../openapi/bookstore.json
// Or from openapi-examples:  sbt "runMain generatescalatypesfromopenapi.CompleteExample"

package generatescalatypesfromopenapi

/**
 * Generate Scala Types from an OpenAPI Spec — Putting It Together
 *
 * Reads an OpenAPI document from a file, maps its component schemas to a
 * `ScalaFile`, adds schema derivation and writes the Scala 3 source of the
 * generated types to `Bookstore.scala`.
 */
object CompleteExample {
  final case class Component(name: String, schema: JsonSchema.Object, discriminator: Option[Discriminator])

  val refPrefix = "#/components/schemas/"

  def main(args: Array[String]): Unit = {
    val specPath  = if (args.length > 0) args(0) else "src/main/openapi/bookstore.json"
    val outputDir = if (args.length > 1) args(1) else "target/generated-bookstore"
    val specText  = new String(Files.readAllBytes(Paths.get(specPath)), StandardCharsets.UTF_8)

    def parseSpec(text: String): OpenAPI =
      Json.parse(text).flatMap(json => OpenAPICodec.openAPICodec.decode(json)) match {
        case Right(api)  => api
        case Left(error) => throw new IllegalArgumentException(s"Not an OpenAPI document: ${error.getMessage}")
      }

    def componentsOf(api: OpenAPI): List[Component] =
      api.components.toList.flatMap(_.schemas.toList).map {
        case (name, ReferenceOr.Value(schemaObject)) =>
          schemaObject.toJsonSchema match {
            case Right(schema: JsonSchema.Object) => Component(name, schema, schemaObject.discriminator)
            case _                                => throw new IllegalArgumentException(s"$name: unsupported schema")
          }
        case (name, ReferenceOr.Ref(_)) =>
          throw new IllegalArgumentException(s"$name: a component that is only a $$ref is not supported")
      }

    def refName(schema: JsonSchema): Option[String] = schema match {
      case obj: JsonSchema.Object => obj.$ref.map(_.value.stripPrefix(refPrefix))
      case _                      => None
    }

    def scalarType(jsonType: JsonSchemaType, schema: JsonSchema.Object): TypeRef = jsonType match {
      case JsonSchemaType.String =>
        schema.format match {
          case Some("date")      => TypeRef("java.time.LocalDate")
          case Some("date-time") => TypeRef("java.time.Instant")
          case Some("uuid")      => TypeRef("java.util.UUID")
          case _                 => TypeRef.String
        }
      case JsonSchemaType.Integer =>
        if (schema.format.contains("int32")) TypeRef.Int else TypeRef.Long
      case JsonSchemaType.Number =>
        schema.format match {
          case Some("float")  => TypeRef.Float
          case Some("double") => TypeRef.Double
          case _              => TypeRef.BigDecimal
        }
      case JsonSchemaType.Boolean =>
        TypeRef.Boolean
      case JsonSchemaType.Array if schema.items.isDefined =>
        TypeRef.list(typeRefFor(schema.items.get))
      case JsonSchemaType.Object if schema.properties.isEmpty && schema.additionalProperties.isDefined =>
        TypeRef.map(TypeRef.String, typeRefFor(schema.additionalProperties.get))
      case other =>
        throw new IllegalArgumentException(s"Unsupported inline schema of type ${other.toJsonString}: ${schema.toJson}")
    }

    def typeRefFor(schema: JsonSchema): TypeRef = schema match {
      case obj: JsonSchema.Object if obj.$ref.isDefined =>
        TypeRef(refName(obj).get)
      case obj: JsonSchema.Object =>
        val nullable30 = obj.extensions.get("nullable").contains(Json.True)
        obj.`type` match {
          case Some(SchemaType.Single(jsonType)) if nullable30 =>
            TypeRef.optional(scalarType(jsonType, obj))
          case Some(SchemaType.Single(jsonType)) =>
            scalarType(jsonType, obj)
          case Some(SchemaType.Union(types)) if types.length == 2 && types.contains(JsonSchemaType.Null) =>
            TypeRef.optional(scalarType(types.find(_ != JsonSchemaType.Null).get, obj))
          case _ =>
            throw new IllegalArgumentException(s"Unsupported inline schema: ${obj.toJson}")
        }
      case other =>
        throw new IllegalArgumentException(s"Unsupported inline schema: ${other.toJson}")
    }

    def fieldFor(name: String, schema: JsonSchema, required: Boolean): Field = {
      val fieldType = typeRefFor(schema)
      if (required) Field(name, fieldType)
      else if (fieldType.name == "Option") Field(name, fieldType, defaultValue = Some("None"))
      else Field(name, TypeRef.optional(fieldType), defaultValue = Some("None"))
    }

    def caseClassFor(name: String, schema: JsonSchema.Object, skip: Set[String] = Set.empty): CaseClass = {
      val required   = schema.required.getOrElse(Set.empty[String])
      val properties = schema.properties.map(_.toList).getOrElse(Nil)
      val fields     = properties.collect {
        case (property, propertySchema) if !skip.contains(property) =>
          fieldFor(property, propertySchema, required.contains(property))
      }
      CaseClass(name, fields, doc = schema.description)
    }

    def enumFor(name: String, schema: JsonSchema.Object): Enum = {
      val values = schema.`enum`.toList.flatMap(_.toList).map {
        case Json.String(value) => EnumCase.SimpleCase(value)
        case other              => throw new IllegalArgumentException(s"$name: non-string enum value $other")
      }
      Enum(name, values, doc = schema.description)
    }

    def stringLiteral(value: String): String = "\"" + value + "\""

    def sealedTraitFor(component: Component, byName: Map[String, Component]): SealedTrait = {
      val discriminator = component.discriminator.get
      val wireNames     = discriminator.mapping.toList.map { case (value, ref) =>
        ref.stripPrefix(refPrefix) -> value
      }.toMap
      val cases = component.schema.oneOf.get.toList.flatMap(refName).map { caseName =>
        val caseClass = caseClassFor(caseName, byName(caseName).schema, skip = Set(discriminator.propertyName))
        val renamed   = wireNames.get(caseName) match {
          case Some(wireName) =>
            caseClass.copy(annotations = List(Annotation("Modifier.rename", List("name" -> stringLiteral(wireName)))))
          case None => caseClass
        }
        SealedTraitCase.CaseClassCase(renamed)
      }
      SealedTrait(
        component.name,
        cases = cases,
        annotations =
          List(Annotation("Modifier.discriminator", List("name" -> stringLiteral(discriminator.propertyName)))),
        doc = component.schema.description
      )
    }

    def isSealedTrait(component: Component): Boolean =
      component.schema.oneOf.isDefined && component.discriminator.isDefined

    def scalaFileFor(api: OpenAPI, packageName: String): ScalaFile = {
      val components = componentsOf(api)
      val byName     = components.map(component => component.name -> component).toMap
      val caseNames  = components.filter(isSealedTrait).flatMap(_.schema.oneOf.get.toList.flatMap(refName)).toSet
      val types      = components.filterNot(component => caseNames.contains(component.name)).map {
        case component if isSealedTrait(component)                     => sealedTraitFor(component, byName)
        case Component(name, schema, _) if schema.`enum`.isDefined     => enumFor(name, schema)
        case Component(name, schema, _) if schema.properties.isDefined => caseClassFor(name, schema)
        case Component(name, schema, _)                                =>
          throw new IllegalArgumentException(s"$name: unsupported component schema ${schema.toJson}")
      }
      ScalaFile(PackageDecl(packageName), types = types)
    }

    def withSchemas(file: ScalaFile, scala3: Boolean): ScalaFile = {
      val types = file.types.map {
        case caseClass: CaseClass if scala3 =>
          caseClass.copy(derives = List("Schema"))
        case caseClass: CaseClass =>
          val schema = ObjectMember.ValMember(
            "schema",
            TypeRef.of("Schema", TypeRef(caseClass.name)),
            "Schema.derived",
            isImplicit = true
          )
          caseClass.copy(companion = Some(CompanionObject(List(schema))))
        case other => other
      }
      file.copy(imports = Import.WildcardImport("zio.blocks.schema") :: file.imports, types = types)
    }

    val bookstoreFile = scalaFileFor(parseSpec(specText), "com.example.bookstore")
    val outputFile    = Paths.get(outputDir, "Bookstore.scala")
    Files.createDirectories(Paths.get(outputDir))
    Files.write(
      outputFile,
      ScalaEmitter
        .emit(withSchemas(bookstoreFile, scala3 = true), EmitterConfig.default)
        .getBytes(StandardCharsets.UTF_8)
    )
    println(s"Generated ${bookstoreFile.types.map(_.name).mkString(", ")} into $outputFile")
  }
}
```

Running it from the `openapi-examples` folder with `sbt "runMain generatescalatypesfromopenapi.CompleteExample"` prints:

```text
Generated Book, Author, BookFormat, Order, OrderLine, OrderStatus, Payment into target/generated-bookstore/Bookstore.scala
```

## Run the Generator on Every Build

Now we make the generator part of the build, so the types are regenerated, and recompiled, whenever the project compiles. sbt calls this a source generator: a task that writes source files before compilation and returns them, so sbt compiles them with everything else. They go in the `src_managed` folder under `target/`, so the generated code never ends up in version control.

sbt runs its build definition on Scala 2.12, and the openapi and codegen modules aren't published for Scala 2.12, so the generator can't run inside `build.sbt` itself. As in the Smithy tutorial, it stays an ordinary program in one sbt project, and a second project runs it as its source generator. Here is the part of `build.sbt` that does it, adapted from our examples project (which writes its dependencies through a small helper); the root project `openapi-examples` holds `CompleteExample` and depends on the openapi and codegen modules, and the new `bookstore-app` project compiles the result:

```scala
// The project that holds the generator, referred to by its id
val generator = LocalProject("openapi-examples")

lazy val bookstoreApp = (project in file("bookstore-app"))
  .settings(
    run / fork := true,
    libraryDependencies ++= Seq(
      "dev.zio" %% "zio-blocks-next-schema"  % "0.0.56",
      "dev.zio" %% "zio-blocks-next-openapi" % "0.0.56"
    ),
    // The app reads the spec's examples at run time, so the spec is also on its classpath.
    Compile / unmanagedResourceDirectories += (generator / baseDirectory).value / "src" / "main" / "openapi",
    Compile / sourceGenerators += Def.task {
      val spec      = (generator / baseDirectory).value / "src" / "main" / "openapi" / "bookstore.json"
      val outputDir = (Compile / sourceManaged).value / "openapi"
      val classpath = (generator / Compile / fullClasspath).value.files
      (generator / Compile / runner).value
        .run(
          "generatescalatypesfromopenapi.CompleteExample",
          classpath,
          Seq(spec.getPath, outputDir.getPath),
          streams.value.log
        )
        .get
      (outputDir ** "*.scala").get
    }.taskValue
  )
```

The code above:

- `LocalProject("openapi-examples")` refers to the generator's project by its id; a direct reference to its `lazy val` would make the two project definitions depend on each other.
- `(generator / Compile / fullClasspath)` is the generator's compiled code plus the openapi and codegen jars, and `(generator / Compile / runner)` runs a main class on that classpath.
- The task passes the spec's path and `bookstore-app`'s `src_managed/main/openapi` folder to `CompleteExample`, then returns every `.scala` file the generator wrote there.
- `bookstore-app` depends on the schema module for the generated code, and on the openapi module only because this program reads the spec's examples; it never sees the generator itself.

With that in place, `bookstore-app` can use the generated types like hand-written ones. This program closes the loop: it reads the `example` of `Book` and of `Order` from the spec, decodes each into the generated type, and encodes the order back to JSON:

```scala title="openapi-examples/bookstore-app/src/main/scala/generatescalatypesfromopenapi/app/BookstoreApp.scala"
// Run from openapi-examples:  sbt bookstoreApp/run
// The generated types come from `openapi-examples/src/main/openapi/bookstore.json`; sbt generates them before it compiles.

package generatescalatypesfromopenapi.app

/**
 * Generate Scala Types from an OpenAPI Spec — Run the Generator on Every Build
 *
 * Reads the examples of `Book` and `Order` from the spec on the classpath,
 * decodes them into the generated types, and encodes the order back to JSON.
 */
object BookstoreApp {
  def main(args: Array[String]): Unit = {
    val specText = new String(getClass.getResourceAsStream("/bookstore.json").readAllBytes(), StandardCharsets.UTF_8)
    val api      =
      Json.parse(specText).flatMap(json => OpenAPICodec.openAPICodec.decode(json)).fold(error => throw error, identity)

    def example(name: String): Json =
      api.components.flatMap(_.schemas.get(name)) match {
        case Some(ReferenceOr.Value(schemaObject)) if schemaObject.example.isDefined => schemaObject.example.get
        case _                                                                       => throw new IllegalArgumentException(s"No example for $name")
      }

    val book  = Schema[Book].jsonCodec.decode(example("Book"))
    val order = Schema[Order].jsonCodec.decode(example("Order"))
    println(book)
    println(order)
    println(order.map(_.toJsonString))
  }
}
```

Running `sbt bookstoreApp/run` from the `openapi-examples` folder generates `Bookstore.scala`, compiles it together with `BookstoreApp` and prints:

```text
Right(Book(zionomicon,Zionomicon,List(Author(John A. De Goes,None), Author(Adam Fraser,None), Author(Milad Khajavi,None)),Ebook,59.99,None,None,None,Some(List(zio, functional-programming))))
Right(Order(3f0c1c5e-8a59-4d55-9b8e-2b2f0b6a4c11,1001,2026-10-27T09:30:00Z,List(OrderLine(zionomicon,1,59.99), OrderLine(programming-in-scala-5,2,44.95)),CardPayment(4242),Placed,None,Some(Map(channel -> web))))
Right({"id":"3f0c1c5e-8a59-4d55-9b8e-2b2f0b6a4c11","customerId":1001,"placedAt":"2026-10-27T09:30:00Z","lines":[{"bookId":"zionomicon","quantity":1,"unitPrice":59.99},{"bookId":"programming-in-scala-5","quantity":2,"unitPrice":44.95}],"payment":{"method":"card","last4":"4242"},"status":"Placed","metadata":{"channel":"web"}})
```

Both examples decode. The optional properties the book example leaves out are `None`, the `uuid` and `date-time` strings are a `UUID` and an `Instant`, and `{"method": "card", ...}` is a `CardPayment`. Encoding the order gives back the example's JSON, with the payment in the spec's shape. Decoding the examples this way is a cheap contract test: if the spec's authors change a schema without updating its example, or our generator maps a type wrongly, this program fails.

Try making `giftMessage` required in `bookstore.json` and compiling again: the field loses its `Option`, and every place that builds an `Order` without it stops compiling until you update it.

## Read a YAML Spec

Many specs are written in YAML instead of JSON. The openapi module reads JSON only, but the schema-yaml module reads YAML text into a `Yaml` value, and `Yaml#toJson` converts it into the same `Json` tree that `OpenAPICodec.openAPICodec` decodes. Add `zio-blocks-schema-yaml` (`zio-blocks-next-schema-yaml` on Scala 3.9 and newer) to the generator's dependencies.

Only the parsing step changes. Here is the start of our spec in YAML:

```scala

val bookstoreYaml: String =
  """openapi: 3.1.0
    |info:
    |  title: Bookstore
    |  version: 1.0.0
    |components:
    |  schemas:
    |    BookFormat:
    |      type: string
    |      description: How the book is delivered.
    |      enum:
    |        - Hardcover
    |        - Paperback
    |        - Ebook
    |    Author:
    |      type: object
    |      required:
    |        - name
    |      properties:
    |        name:
    |          type: string
    |        website:
    |          type: string
    |          format: uri
    |""".stripMargin

def parseYamlSpec(text: String): OpenAPI =
  OpenAPICodec.openAPICodec.decode(YamlReader.read(text).toJson) match {
    case Right(api)  => api
    case Left(error) => throw new IllegalArgumentException(s"Not an OpenAPI document: ${error.getMessage}")
  }
```

The rest of the generator works on the parsed `OpenAPI` exactly as before:

```scala
println(ScalaEmitter.emit(scalaFileFor(parseYamlSpec(bookstoreYaml), "com.example.bookstore"), EmitterConfig.default))
// package com.example.bookstore
// 
// /** How the book is delivered. */
// enum BookFormat {
//   case Hardcover, Paperback, Ebook
// }
// 
// case class Author(
//   name: String,
//   website: Option[String] = None,
// )
//
```

:::warning
Indent the `- ` items of a list under their key, as above. In 0.0.56, a known bug in `YamlReader` reads a list written at the same indentation as its key (`required:` followed by `- name` on the next line, a style many YAML tools write) as `null`, without an error. The generator then sees, for example, no required properties and makes every field optional.
:::

<details>
  <summary>Runnable version, with the generator from the previous sections: <code>Step8YamlSpecExample.scala</code></summary>

```scala title="openapi-examples/src/main/scala/generatescalatypesfromopenapi/Step8YamlSpecExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-next-openapi:0.0.56
//> using dep dev.zio::zio-blocks-next-codegen:0.0.56
//> using dep dev.zio::zio-blocks-next-schema-yaml:0.0.56
//> using options -experimental
//> using file BookstoreSpec.scala
// Run from this folder:  scala-cli run Step8YamlSpecExample.scala BookstoreSpec.scala
// Or from openapi-examples:  sbt "runMain generatescalatypesfromopenapi.Step8YamlSpecExample"

package generatescalatypesfromopenapi

/**
 * Generate Scala Types from an OpenAPI Spec — Read a YAML Spec
 *
 * Reads a YAML version of part of the spec through `YamlReader` and runs the
 * generator on it.
 */
object Step8YamlSpecExample {
  // The generator from the previous sections, plus the YAML parsing step.
  def parseSpec(text: String): OpenAPI =
    Json.parse(text).flatMap(json => OpenAPICodec.openAPICodec.decode(json)) match {
      case Right(api)  => api
      case Left(error) => throw new IllegalArgumentException(s"Not an OpenAPI document: ${error.getMessage}")
    }

  val api: OpenAPI = parseSpec(bookstoreJson)

  final case class Component(name: String, schema: JsonSchema.Object, discriminator: Option[Discriminator])

  def componentsOf(api: OpenAPI): List[Component] =
    api.components.toList.flatMap(_.schemas.toList).map {
      case (name, ReferenceOr.Value(schemaObject)) =>
        schemaObject.toJsonSchema match {
          case Right(schema: JsonSchema.Object) => Component(name, schema, schemaObject.discriminator)
          case _                                => throw new IllegalArgumentException(s"$name: unsupported schema")
        }
      case (name, ReferenceOr.Ref(_)) =>
        throw new IllegalArgumentException(s"$name: a component that is only a $$ref is not supported")
    }

  val components: List[Component] = componentsOf(api)

  val refPrefix = "#/components/schemas/"

  def refName(schema: JsonSchema): Option[String] = schema match {
    case obj: JsonSchema.Object => obj.$ref.map(_.value.stripPrefix(refPrefix))
    case _                      => None
  }

  def scalarType(jsonType: JsonSchemaType, schema: JsonSchema.Object): TypeRef = jsonType match {
    case JsonSchemaType.String =>
      schema.format match {
        case Some("date")      => TypeRef("java.time.LocalDate")
        case Some("date-time") => TypeRef("java.time.Instant")
        case Some("uuid")      => TypeRef("java.util.UUID")
        case _                 => TypeRef.String
      }
    case JsonSchemaType.Integer =>
      if (schema.format.contains("int32")) TypeRef.Int else TypeRef.Long
    case JsonSchemaType.Number =>
      schema.format match {
        case Some("float")  => TypeRef.Float
        case Some("double") => TypeRef.Double
        case _              => TypeRef.BigDecimal
      }
    case JsonSchemaType.Boolean =>
      TypeRef.Boolean
    case JsonSchemaType.Array if schema.items.isDefined =>
      TypeRef.list(typeRefFor(schema.items.get))
    case JsonSchemaType.Object if schema.properties.isEmpty && schema.additionalProperties.isDefined =>
      TypeRef.map(TypeRef.String, typeRefFor(schema.additionalProperties.get))
    case other =>
      throw new IllegalArgumentException(s"Unsupported inline schema of type ${other.toJsonString}: ${schema.toJson}")
  }

  def typeRefFor(schema: JsonSchema): TypeRef = schema match {
    case obj: JsonSchema.Object if obj.$ref.isDefined =>
      TypeRef(refName(obj).get)
    case obj: JsonSchema.Object =>
      val nullable30 = obj.extensions.get("nullable").contains(Json.True)
      obj.`type` match {
        case Some(SchemaType.Single(jsonType)) if nullable30 =>
          TypeRef.optional(scalarType(jsonType, obj))
        case Some(SchemaType.Single(jsonType)) =>
          scalarType(jsonType, obj)
        case Some(SchemaType.Union(types)) if types.length == 2 && types.contains(JsonSchemaType.Null) =>
          TypeRef.optional(scalarType(types.find(_ != JsonSchemaType.Null).get, obj))
        case _ =>
          throw new IllegalArgumentException(s"Unsupported inline schema: ${obj.toJson}")
      }
    case other =>
      throw new IllegalArgumentException(s"Unsupported inline schema: ${other.toJson}")
  }

  def fieldFor(name: String, schema: JsonSchema, required: Boolean): Field = {
    val fieldType = typeRefFor(schema)
    if (required) Field(name, fieldType)
    else if (fieldType.name == "Option") Field(name, fieldType, defaultValue = Some("None"))
    else Field(name, TypeRef.optional(fieldType), defaultValue = Some("None"))
  }

  def caseClassFor(name: String, schema: JsonSchema.Object, skip: Set[String] = Set.empty): CaseClass = {
    val required   = schema.required.getOrElse(Set.empty[String])
    val properties = schema.properties.map(_.toList).getOrElse(Nil)
    val fields     = properties.collect {
      case (property, propertySchema) if !skip.contains(property) =>
        fieldFor(property, propertySchema, required.contains(property))
    }
    CaseClass(name, fields, doc = schema.description)
  }

  def enumFor(name: String, schema: JsonSchema.Object): Enum = {
    val values = schema.`enum`.toList.flatMap(_.toList).map {
      case Json.String(value) => EnumCase.SimpleCase(value)
      case other              => throw new IllegalArgumentException(s"$name: non-string enum value $other")
    }
    Enum(name, values, doc = schema.description)
  }

  def stringLiteral(value: String): String = "\"" + value + "\""

  def sealedTraitFor(component: Component, byName: Map[String, Component]): SealedTrait = {
    val discriminator = component.discriminator.get
    val wireNames     = discriminator.mapping.toList.map { case (value, ref) => ref.stripPrefix(refPrefix) -> value }.toMap
    val cases         = component.schema.oneOf.get.toList.flatMap(refName).map { caseName =>
      val caseClass = caseClassFor(caseName, byName(caseName).schema, skip = Set(discriminator.propertyName))
      val renamed   = wireNames.get(caseName) match {
        case Some(wireName) =>
          caseClass.copy(annotations = List(Annotation("Modifier.rename", List("name" -> stringLiteral(wireName)))))
        case None => caseClass
      }
      SealedTraitCase.CaseClassCase(renamed)
    }
    SealedTrait(
      component.name,
      cases = cases,
      annotations =
        List(Annotation("Modifier.discriminator", List("name" -> stringLiteral(discriminator.propertyName)))),
      doc = component.schema.description
    )
  }

  val byName: Map[String, Component] = components.map(component => component.name -> component).toMap

  def isSealedTrait(component: Component): Boolean =
    component.schema.oneOf.isDefined && component.discriminator.isDefined

  def scalaFileFor(api: OpenAPI, packageName: String): ScalaFile = {
    val components = componentsOf(api)
    val byName     = components.map(component => component.name -> component).toMap
    val caseNames  = components.filter(isSealedTrait).flatMap(_.schema.oneOf.get.toList.flatMap(refName)).toSet
    val types      = components.filterNot(component => caseNames.contains(component.name)).map {
      case component if isSealedTrait(component)                     => sealedTraitFor(component, byName)
      case Component(name, schema, _) if schema.`enum`.isDefined     => enumFor(name, schema)
      case Component(name, schema, _) if schema.properties.isDefined => caseClassFor(name, schema)
      case Component(name, schema, _)                                =>
        throw new IllegalArgumentException(s"$name: unsupported component schema ${schema.toJson}")
    }
    ScalaFile(PackageDecl(packageName), types = types)
  }

  val bookstoreFile: ScalaFile = scalaFileFor(api, "com.example.bookstore")

  // Read a YAML Spec
  val bookstoreYaml: String =
    """openapi: 3.1.0
      |info:
      |  title: Bookstore
      |  version: 1.0.0
      |components:
      |  schemas:
      |    BookFormat:
      |      type: string
      |      description: How the book is delivered.
      |      enum:
      |        - Hardcover
      |        - Paperback
      |        - Ebook
      |    Author:
      |      type: object
      |      required:
      |        - name
      |      properties:
      |        name:
      |          type: string
      |        website:
      |          type: string
      |          format: uri
      |""".stripMargin

  def parseYamlSpec(text: String): OpenAPI =
    OpenAPICodec.openAPICodec.decode(YamlReader.read(text).toJson) match {
      case Right(api)  => api
      case Left(error) => throw new IllegalArgumentException(s"Not an OpenAPI document: ${error.getMessage}")
    }

  def main(args: Array[String]): Unit =
    println(
      ScalaEmitter.emit(scalaFileFor(parseYamlSpec(bookstoreYaml), "com.example.bookstore"), EmitterConfig.default)
    )
}
```

</details>

## What the Generator Doesn't Cover

The openapi module parses much more of a spec than our generator turns into types. The parser also decodes two large public specs: GitHub's REST API description (OpenAPI 3.1, 975 component schemas) and Stripe's (OpenAPI 3.0, 1538 component schemas), and every component schema in both parses into a typed `JsonSchema`. Our generator, on the other hand, stops at the first schema it doesn't support, which in both specs is an inline object or an `anyOf`.

This table lists what each part handles, so you know where to extend the generator for your own spec:

| Spec feature                                                     | openapi module (parsing)             | Our generator                                   |
| ---------------------------------------------------------------- | ------------------------------------ | ----------------------------------------------- |
| `object` with `properties` and `required`                        | Yes                                  | Case class; optional properties are `Option`    |
| String `enum`                                                    | Yes                                  | `Enum`                                          |
| `oneOf` with a `discriminator` and `mapping`                     | Yes                                  | Sealed trait with `Modifier` annotations        |
| `oneOf` or `anyOf` without a discriminator                       | Yes                                  | Not supported: stops with an error              |
| `allOf` (composition and inheritance)                            | Yes                                  | Not supported: stops with an error              |
| Inline object as a property's schema                             | Yes                                  | Not supported: name it under `components`       |
| A component that is an array, a primitive or a `$ref`            | Yes                                  | Not supported: stops with an error              |
| `$ref` into another file (`other.json#/...`)                     | Kept as the reference text           | Not supported: only `#/components/schemas/`     |
| `type: [A, "null"]` (3.1) and `nullable: true` (3.0)             | Yes; `nullable` in `extensions`      | `Option[A]`                                     |
| Validation keywords (`minLength`, `pattern`, `minimum`, ...)     | Yes                                  | Ignored: types only                             |
| OpenAPI 3.0 boolean `exclusiveMinimum` and `exclusiveMaximum`    | Silently dropped                     | Ignored                                         |
| `default` values                                                 | Yes                                  | Ignored                                         |
| Property `description`                                           | Yes                                  | Dropped: the emitter doesn't render field docs  |
| Paths, operations, parameters and responses                      | Yes                                  | Not generated: data types only                  |
| YAML documents                                                   | Through schema-yaml, see above       | Same as JSON                                    |

Next step: [How to Generate Service Traits from Smithy Operations or OpenAPI Paths](./generate-service-traits-from-smithy-or-openapi.md) maps each path and operation to a method of a Scala service trait.

Note that the model is OpenAPI 3.1, and the parser doesn't check the `openapi` version field: a 3.0 document decodes, and its schemas are read as JSON Schema 2020-12. The two differ in the keywords above, so a 3.0 spec needs the `nullable` handling from [Map Schema Types to Scala Types](#map-schema-types-to-scala-types), and its exclusive bounds are lost.

## Running the Examples

All runnable code from this guide is in [`openapi-examples/src/main/scala/generatescalatypesfromopenapi/`](https://github.com/zio/zio-blocks/tree/main/openapi-examples/src/main/scala/generatescalatypesfromopenapi), the spec is in [`openapi-examples/src/main/openapi/`](https://github.com/zio/zio-blocks/tree/main/openapi-examples/src/main/openapi), and the app that compiles the generated code is in [`openapi-examples/bookstore-app/`](https://github.com/zio/zio-blocks/tree/main/openapi-examples/bookstore-app). Run any file from the `openapi-examples` folder with `sbt "runMain generatescalatypesfromopenapi.<ObjectName>"`, and the app with `sbt bookstoreApp/run`.

## Going Further

- **Want to go the other way, from Scala types to a spec?** [How to Write an OpenAPI Document and Export It as JSON or YAML](./write-and-export-openapi.md) builds the spec for the same bookstore from `Schema`s.
- **Want to know everything the parser reads?** The [openapi module reference](../reference/openapi.md) lists every type of the `OpenAPI` model.
- **Want to generate more than data types?** The [codegen module reference](../reference/codegen/index.md) covers the whole IR, including objects, methods, opaque types and newtypes, and the [`ScalaEmitter`](../reference/codegen/scala-emitter.md) and [`EmitterConfig`](../reference/codegen/emitter-config.md) pages describe every output option.
- **Want to shape the JSON further?** [How to Customize JSON Field Names, ADT Encoding and Optional Fields](./customize-json-encoding.md) explains the `Modifier` annotations the generator writes, and the others you can add the same way, such as field renames.
- **Generating from Smithy instead?** [Generate Scala Code from a Smithy Model](./generate-scala-code-from-smithy.md) builds the same kind of generator for a Smithy model.

---

## How to Generate Service Traits from Smithy Operations or OpenAPI Paths

In this guide, we generate a Scala service trait from a Smithy service and from an OpenAPI document. Each operation becomes one abstract method, each set of errors becomes a sealed type, and the generator writes the result as Scala 3 source. The Smithy walkthrough comes first in full, and the OpenAPI version follows with the places where it differs.

The generators in [Generate Scala Code from a Smithy Model](./generate-scala-code-from-smithy.md) and [Generate Scala Types from an OpenAPI Spec at Build Time](./generate-scala-types-from-openapi.md) stop at data types. This guide adds the step after them: we map operations to methods, and the codegen IR writes the trait.

## The Problem

A service contract lists its operations, the input each one takes, the output it returns and the errors it can fail with. Your Scala code has to repeat that list as a trait, and the trait has to change whenever the contract does.

Written by hand, the trait for two operations of an order service looks like this:

```scala
final case class GetOrderInput(orderId: String)
final case class GetOrderOutput(orderId: String, state: String, trackingNumber: Option[String])
final case class CancelOrderInput(orderId: String, reason: Option[String])
final case class CancelOrderOutput(orderId: String, state: String)

sealed trait OrderError
final case class OrderNotFound(orderId: String) extends OrderError
final case class OrderAlreadyShipped(orderId: String) extends OrderError

trait OrderService {
  def getOrder(input: GetOrderInput): Either[OrderError, GetOrderOutput]
  def cancelOrder(input: CancelOrderInput): Either[OrderError, CancelOrderOutput]
}
```

When the model gains an operation or an error, nothing tells you the trait is stale: the compiler accepts the old version, and the gap shows up later, when a client calls the operation that is missing. In this guide, the model is the only place the operations are written, and the trait is generated from it on every build.

## Prerequisites

Add the codegen and schema artifacts to your build. The generated code uses `derives Schema`, so the output needs the schema module as well as the generator modules:

```scala
// Scala 3.9 and newer
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-smithy"  % "0.0.56",
  "dev.zio" %% "zio-blocks-next-codegen" % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema"  % "0.0.56"
)
scalacOptions += "-experimental"
```

The generator is ordinary Scala that runs on the JVM at build time, so it needs no service runtime. The examples in this guide are Scala 3 only, because the generated code uses `derives`.

The base imports for the Smithy part go in the first block of the page, and every block after it shares them:

```scala

```

## The Service Model

A Smithy service is a shape that lists its operations. Each `operation` names an input structure, an output structure and the errors it can return, and an `@error` structure is the shape of one such error. The model below is the same file the examples read, and it is kept as a string here so every block on this page can use it:

```scala
val orderServiceSmithy: String =
  """$version: "2"

namespace com.acme.orders

/// The operations a client can call on the order service.
service OrderService {
    operations: [GetOrder, CancelOrder]
}

/// Look up one order by its id.
operation GetOrder {
    input: GetOrderInput
    output: GetOrderOutput
    errors: [OrderNotFound]
}

/// Cancel an order that has not shipped yet.
operation CancelOrder {
    input: CancelOrderInput
    output: CancelOrderOutput
    errors: [OrderNotFound, OrderAlreadyShipped]
}

structure GetOrderInput {
    @required
    orderId: String
}

structure GetOrderOutput {
    @required
    orderId: String

    @required
    state: OrderState

    trackingNumber: String
}

structure CancelOrderInput {
    @required
    orderId: String

    reason: String
}

structure CancelOrderOutput {
    @required
    orderId: String

    @required
    state: OrderState
}

/// The order does not exist.
@error("client")
structure OrderNotFound {
    @required
    orderId: String
}

/// The order has shipped, so it can no longer be cancelled.
@error("client")
structure OrderAlreadyShipped {
    @required
    orderId: String
}

/// Where an order is in its life cycle, as the service reports it.
enum OrderState {
    PLACED
    SHIPPED
    CANCELLED
}"""

val model: SmithyModel = SmithyModel.parse(orderServiceSmithy) match {
  case Right(parsed) => parsed
  case Left(error)   => throw new IllegalArgumentException(error.formatMessage)
}
```

`SmithyModel.parse` returns the whole model as a value, so the generator can look up any shape by name. The `service` shape is the entry point, and the rest of this guide walks from it to the operations, their errors and the types they use.

## Read the Operations

The service shape lists its operations by identifier, so the generator first resolves each identifier to an `OperationShape`. A `ServiceShape` holds `operations: List[ShapeId]`, and `SmithyModel#findShape` gives the shape behind a name:

```scala
def service(model: SmithyModel): ServiceShape =
  model.shapes.collectFirst { case ShapeDefinition(_, s: ServiceShape) => s }.getOrElse {
    throw new IllegalArgumentException("the model has no service shape")
  }

def operation(model: SmithyModel, id: ShapeId): OperationShape =
  model.findShape(id.name).map(_.shape) match {
    case Some(op: OperationShape) => op
    case _                        => throw new IllegalArgumentException(s"Unknown operation: $id")
  }

def operations(model: SmithyModel): List[OperationShape] =
  service(model).operations.map(operation(model, _))
```

Printing each operation shows the three things the generator needs from it: its input, its output and its errors:

```scala
operations(model).foreach { op =>
  val input  = op.input.map(_.name).getOrElse("none")
  val output = op.output.map(_.name).getOrElse("none")
  val errors = op.errors.map(_.name).mkString(", ")
  println(s"${op.name}: input=$input, output=$output, errors=[$errors]")
}
// GetOrder: input=GetOrderInput, output=GetOrderOutput, errors=[OrderNotFound]
// CancelOrder: input=CancelOrderInput, output=CancelOrderOutput, errors=[OrderNotFound, OrderAlreadyShipped]
```

The `OrderNotFound` error appears under both operations. That is why the errors are collected once for the whole service later, rather than per operation.

## Map Smithy Types to Scala Types

Each operation refers to its input and output by name, and those names are structures or enums in the model. We map a member's Smithy target to a `TypeRef`, the codegen description of a Scala type, and a member that is not `@required` becomes an `Option` with a `None` default:

```scala
def docOf(traits: List[TraitApplication]): Option[String] =
  traits.collectFirst {
    case TraitApplication(ShapeId("smithy.api", "documentation"), Some(NodeValue.String(text))) => text
  }

def typeRefFor(model: SmithyModel, target: ShapeId): TypeRef =
  model.findShape(target.name).map(_.shape) match {
    case Some(_: StructureShape) | Some(_: EnumShape) | Some(_: UnionShape) => TypeRef(target.name)
    case Some(other) =>
      throw new IllegalArgumentException(s"Unsupported shape: ${other.name}")
    case None =>
      target.name match {
        case "String"    => TypeRef.String
        case "Integer"   => TypeRef.Int
        case "Long"      => TypeRef.Long
        case "Boolean"   => TypeRef.Boolean
        case "Timestamp" => TypeRef("java.time.Instant")
        case other       => throw new IllegalArgumentException(s"Unknown shape: $other")
      }
  }

def fieldFor(model: SmithyModel, member: MemberDefinition): Field = {
  val fieldType = typeRefFor(model, member.target)
  if (member.traits.contains(TraitApplication.required)) Field(member.name, fieldType)
  else Field(member.name, TypeRef.optional(fieldType), defaultValue = Some("None"))
}

def caseClassFor(model: SmithyModel, name: String, shape: StructureShape, extendsTypes: List[TypeRef]): CaseClass =
  CaseClass(
    name,
    shape.members.map(fieldFor(model, _)),
    extendsTypes = extendsTypes,
    derives = List("Schema"),
    doc = docOf(shape.traits)
  )

def enumFor(name: String, shape: EnumShape): Enum =
  Enum(name, shape.members.map(member => EnumCase.SimpleCase(member.name)), doc = docOf(shape.traits))
```

The `derives = List("Schema")` argument asks the emitter for a `derives Schema` clause, so each generated type gets its ZIO Blocks schema. The output of one structure shows the result:

```scala
println(
  ScalaEmitter.emit(
    ScalaFile(
      PackageDecl(model.namespace),
      imports = List(Import.WildcardImport("zio.blocks.schema")),
      types = model.shapes.collect {
        case ShapeDefinition(name, shape: StructureShape) if name == "GetOrderOutput" =>
          caseClassFor(model, name, shape, Nil)
      }
    ),
    EmitterConfig.default
  )
)
// package com.acme.orders
// 
// import zio.blocks.schema.*
// 
// case class GetOrderOutput(
//   orderId: String,
//   state: OrderState,
//   trackingNumber: Option[String] = None,
// ) derives Schema
//
```

Optional members come out as `Option` fields with a default, and the `@documentation` text becomes the Scaladoc of the type.

## Map Each Operation to a Method

An operation becomes a method of the service trait. The method takes the operation's input structure as its single parameter, so a new member in the input never changes the signature. Its return type is the output structure, wrapped in `Either` when the operation has errors, and an operation without an output returns `Unit`:

```scala
def errorTypeName(svc: ServiceShape): String = svc.name.stripSuffix("Service") + "Error"

def methodFor(model: SmithyModel, op: OperationShape): Method = {
  val errorType = errorTypeName(service(model))
  val params    = op.input.toList.map(in => ParamList(List(MethodParam("input", TypeRef(in.name)))))
  val output    = op.output.map(o => TypeRef(o.name)).getOrElse(TypeRef.Unit)
  val returnType =
    if (op.errors.isEmpty) output
    else TypeRef.of("Either", TypeRef(errorType), output)
  Method(
    name = op.name.take(1).toLowerCase + op.name.drop(1),
    params = params,
    returnType = returnType,
    doc = docOf(op.traits)
  )
}

def serviceTrait(model: SmithyModel): Trait = {
  val svc = service(model)
  Trait(
    svc.name,
    members = operations(model).map(op => ObjectMember.DefMember(methodFor(model, op))),
    doc = docOf(svc.traits)
  )
}
```

A `Trait` holds `ObjectMember.DefMember` values, and a `Method` with no body is abstract, so the emitter writes a declaration with no implementation. The service shape's name is the trait's name, and its `@documentation` is the trait's Scaladoc:

```scala
println(ScalaEmitter.emit(ScalaFile(PackageDecl(model.namespace), types = List(serviceTrait(model))), EmitterConfig.default))
// package com.acme.orders
// 
// /** The operations a client can call on the order service. */
// trait OrderService {
//   /** Look up one order by its id. */
//   def getOrder(input: GetOrderInput): Either[OrderError, GetOrderOutput]
//   /** Cancel an order that has not shipped yet. */
//   def cancelOrder(input: CancelOrderInput): Either[OrderError, CancelOrderOutput]
// }
//
```

The trait is the contract, and the implementation is yours. An application writes a class that extends `OrderService`, and the compiler checks every method against the generated signature, so a missing operation fails the build instead of a request.

## Collect the Errors into One Sealed Type

Smithy errors belong to the service, not to one operation, and an error such as `OrderNotFound` can be shared by several operations. The generator therefore collects every error the service's operations name, and writes one sealed type for them. Each error becomes a case class that extends it:

```scala
def errorNames(model: SmithyModel): List[String] =
  operations(model).flatMap(_.errors.map(_.name)).distinct

def errorTypes(model: SmithyModel): List[TypeDefinition] = {
  val errorType  = errorTypeName(service(model))
  val sealedType = SealedTrait(errorType, doc = Some("Every error an operation of the service can return."))
  val cases = errorNames(model).map { name =>
    model.findShape(name).map(_.shape) match {
      case Some(shape: StructureShape) =>
        caseClassFor(model, name, shape, List(TypeRef(errorType)))
      case _ => throw new IllegalArgumentException(s"Error is not a structure: $name")
    }
  }
  sealedType :: cases
}
```

`errorNames` uses `distinct`, so `OrderNotFound` gets one case class even though two operations name it. The result is the sealed trait and its cases:

```scala
println(
  ScalaEmitter.emit(
    ScalaFile(
      PackageDecl(model.namespace),
      imports = List(Import.WildcardImport("zio.blocks.schema")),
      types = errorTypes(model)
    ),
    EmitterConfig.default
  )
)
// package com.acme.orders
// 
// import zio.blocks.schema.*
// 
// /** Every error an operation of the service can return. */
// sealed trait OrderError
// 
// /** The order does not exist. */
// case class OrderNotFound(
//   orderId: String,
// ) extends OrderError derives Schema
// 
// /** The order has shipped, so it can no longer be cancelled. */
// case class OrderAlreadyShipped(
//   orderId: String,
// ) extends OrderError derives Schema
//
```

An operation can return only the errors its own signature names, and a single type for the whole service keeps the trait short. The price is that a client matching on `OrderError` also has to handle `OrderAlreadyShipped` for `getOrder`, which can never return it. If you need that precision, write one sealed type per operation, which is what the OpenAPI version below does.

## Generate One File

The data types, the error type and the service trait belong in one source file, so the generator puts them in one `ScalaFile`. The data types are the structures and enums that are not errors:

```scala
def dataTypes(model: SmithyModel): List[TypeDefinition] = {
  val errors = errorNames(model)
  model.shapes.flatMap {
    case ShapeDefinition(name, _) if errors.contains(name) => None
    case ShapeDefinition(name, shape: StructureShape)      => Some(caseClassFor(model, name, shape, Nil))
    case ShapeDefinition(name, shape: EnumShape)           => Some(enumFor(name, shape))
    case _                                                 => None
  }
}

def scalaFile(model: SmithyModel): ScalaFile =
  ScalaFile(
    PackageDecl(model.namespace),
    imports = List(Import.WildcardImport("zio.blocks.schema")),
    types = (dataTypes(model) ++ errorTypes(model)) :+ serviceTrait(model)
  )
```

Printing the whole file shows the three parts in order: the data types, the error type with its cases, and the service trait last:

```scala
println(ScalaEmitter.emit(scalaFile(model), EmitterConfig.default))
// package com.acme.orders
// 
// import zio.blocks.schema.*
// 
// case class GetOrderInput(
//   orderId: String,
// ) derives Schema
// 
// case class GetOrderOutput(
//   orderId: String,
//   state: OrderState,
//   trackingNumber: Option[String] = None,
// ) derives Schema
// 
// case class CancelOrderInput(
//   orderId: String,
//   reason: Option[String] = None,
// ) derives Schema
// 
// case class CancelOrderOutput(
//   orderId: String,
//   state: OrderState,
// ) derives Schema
// 
// /** Where an order is in its life cycle, as the service reports it. */
// enum OrderState {
//   case PLACED, SHIPPED, CANCELLED
// }
// 
// /** Every error an operation of the service can return. */
// sealed trait OrderError
// 
// /** The order does not exist. */
// case class OrderNotFound(
//   orderId: String,
// ) extends OrderError derives Schema
// 
// /** The order has shipped, so it can no longer be cancelled. */
// case class OrderAlreadyShipped(
//   orderId: String,
// ) extends OrderError derives Schema
// 
// /** The operations a client can call on the order service. */
// trait OrderService {
//   /** Look up one order by its id. */
//   def getOrder(input: GetOrderInput): Either[OrderError, GetOrderOutput]
//   /** Cancel an order that has not shipped yet. */
//   def cancelOrder(input: CancelOrderInput): Either[OrderError, CancelOrderOutput]
// }
//
```

## Putting It Together

The generator reads a model file, maps it with the functions above and writes one file. The complete program below does that, and the only differences from the sections are that it reads the model from `smithy-examples/src/main/smithy/order-service.smithy` and writes the output to a file instead of printing it:

```scala title="smithy-examples/src/main/scala/generateservicetraits/CompleteExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-next-smithy:0.0.56
//> using dep dev.zio::zio-blocks-next-codegen:0.0.56
//> using options -experimental
//> using file ServiceMapping.scala
// Run from this folder:  scala-cli run CompleteExample.scala
// Or from smithy-examples:  sbt "runMain generateservicetraits.CompleteExample"

package generateservicetraits

/**
 * Generate Scala Service Traits from a Smithy Model — Putting It Together
 *
 * Reads the order service model, maps it with `ServiceMapping` and writes the
 * Scala 3 source of the data types, the error type and the service trait to
 * `OrderService.scala`.
 */
object CompleteExample {
  def main(args: Array[String]): Unit = {
    val modelPath: String = if (args.length > 0) args(0) else "src/main/smithy/order-service.smithy"
    val outputDir: String = if (args.length > 1) args(1) else "target/generated-order-service"
    val text              = new String(Files.readAllBytes(Paths.get(modelPath)), StandardCharsets.UTF_8)

    val model: SmithyModel = SmithyModel.parse(text) match {
      case Right(parsed) => parsed
      case Left(error)   => throw new IllegalArgumentException(error.formatMessage)
    }

    val outputFile = Paths.get(outputDir, "OrderService.scala")
    Files.createDirectories(Paths.get(outputDir))
    Files.write(
      outputFile,
      ScalaEmitter.emit(ServiceMapping.scalaFile(model), EmitterConfig.default).getBytes(StandardCharsets.UTF_8)
    )
    println(s"Generated ${ServiceMapping.service(model).name} into $outputFile")
  }
}
```

Running it writes `OrderService.scala`, and that file has the same contents as the output of the last block.

## Running the Examples

All runnable code from the Smithy part is in [`smithy-examples/src/main/scala/generateservicetraits/`](https://github.com/zio/zio-blocks/tree/main/smithy-examples/src/main/scala/generateservicetraits), and the model is in [`smithy-examples/src/main/smithy/`](https://github.com/zio/zio-blocks/tree/main/smithy-examples/src/main/smithy). Run any file from the `smithy-examples` folder with `sbt "runMain generateservicetraits.<ObjectName>"`, for example `sbt "runMain generateservicetraits.CompleteExample"`.

## Generate Service Traits from OpenAPI Paths

OpenAPI has no service or operation shapes. Each path item holds up to eight operations, one per HTTP method, and each operation has an `operationId`, which names it. The generator reads `get`, `put`, `post`, `delete` and `patch`. Its parameters, its request body and its responses sit inside the operation, and each response is keyed by its status code. The mapping follows the same shape as the Smithy one, with three differences: the path parameters become method parameters, the request body becomes a `body` parameter, and the non-2xx responses become the error type of that one operation.

The spec below is the bookstore's paths, with the schemas reduced to their names. The trait depends only on the paths and the names of the schemas they refer to, so the output is the same as for the full spec in `openapi-examples/src/main/openapi/bookstore-service.json`:

```scala

val bookstoreServiceJson: String =
  """{
  "openapi": "3.1.0",
  "info": {
    "title": "Bookstore",
    "version": "1.0.0"
  },
  "paths": {
    "/orders/{orderId}": {
      "get": {
        "operationId": "getOrder",
        "summary": "Look up one order by its id.",
        "parameters": [
          {
            "name": "orderId",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string",
              "format": "uuid"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "The order",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/Order"
                }
              }
            }
          },
          "404": {
            "description": "No order has this id",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/ApiError"
                }
              }
            }
          }
        }
      }
    },
    "/orders/{orderId}/cancel": {
      "post": {
        "operationId": "cancelOrder",
        "summary": "Cancel an order that has not shipped yet.",
        "parameters": [
          {
            "name": "orderId",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string",
              "format": "uuid"
            }
          }
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/CancelRequest"
              }
            }
          }
        },
        "responses": {
          "200": {
            "description": "The cancelled order",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/Order"
                }
              }
            }
          },
          "404": {
            "description": "No order has this id",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/ApiError"
                }
              }
            }
          },
          "409": {
            "description": "The order has already shipped",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/ApiError"
                }
              }
            }
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "Order": {
        "type": "object"
      },
      "ApiError": {
        "type": "object"
      },
      "CancelRequest": {
        "type": "object"
      }
    }
  }
}"""

val api: OpenAPI = Json.parse(bookstoreServiceJson).flatMap(json => OpenAPICodec.openAPICodec.decode(json)) match {
  case Right(parsed) => parsed
  case Left(error)   => throw new IllegalArgumentException(s"Not an OpenAPI document: ${error.getMessage}")
}
```

## Read the Paths

The generator walks the paths and collects every operation, paired with the path item it belongs to, because the path item can hold parameters that apply to all its operations:

```scala
val refPrefix = "#/components/schemas/"

def docOf(text: Option[Doc]): Option[String] = text.map(_.toString.trim)

final case class Endpoint(method: String, path: String, operationId: String, operation: Operation, item: PathItem)

def endpoints(api: OpenAPI): List[Endpoint] =
  api.paths.toList.flatMap(_.paths.toList).flatMap { case (path, item) =>
    List(
      "get"    -> item.get,
      "put"    -> item.put,
      "post"   -> item.post,
      "delete" -> item.delete,
      "patch"  -> item.patch
    ).collect { case (method, Some(op)) =>
      val id = op.operationId.getOrElse(
        throw new IllegalArgumentException(s"$method $path has no operationId")
      )
      Endpoint(method, path, id, op, item)
    }
  }
```

Each `operationId` becomes the method name, so an operation without one cannot be mapped, and the generator stops with an error. The next functions read a method's parameters and its success type from the spec:

```scala
def componentName(schema: ReferenceOr[SchemaObject]): String = schema match {
  case ReferenceOr.Ref(reference) => reference.$ref.stripPrefix(refPrefix)
  case ReferenceOr.Value(_)       => throw new IllegalArgumentException("inline schemas are not supported: name the schema in components")
}

def jsonSchema(content: Iterable[(String, MediaType)]): Option[ReferenceOr[SchemaObject]] =
  content.collectFirst { case ("application/json", mediaType) => mediaType.schema }.flatten

def pathParams(e: Endpoint): List[MethodParam] =
  (e.item.parameters.toList ++ e.operation.parameters.toList).collect {
    case ReferenceOr.Value(p) if p.in == ParameterLocation.Path => MethodParam(p.name, TypeRef.String)
  }

def bodyParam(e: Endpoint): List[MethodParam] =
  e.operation.requestBody.toList.collect {
    case ReferenceOr.Value(body) =>
      jsonSchema(body.content.toList).map(s => MethodParam("body", TypeRef(componentName(s))))
  }.flatten

def successType(e: Endpoint): TypeRef =
  e.operation.responses.responses.toList.collectFirst {
    case (code, ReferenceOr.Value(response)) if code.startsWith("2") =>
      jsonSchema(response.content.toList).map(s => TypeRef(componentName(s))).getOrElse(TypeRef.Unit)
  }.getOrElse(TypeRef.Unit)
```

Printing the endpoints shows the method, the path, the parameters and the success type each one gets:

```scala
endpoints(api).foreach { e =>
  val params = pathParams(e).map(p => s"${p.name}: ${p.typeRef.name}")
  val body   = bodyParam(e).map(p => s"${p.name}: ${p.typeRef.name}")
  println(s"${e.method.toUpperCase} ${e.path} -> ${e.operationId}(${(params ++ body).mkString(", ")}): ${successType(e).name}")
}
// GET /orders/{orderId} -> getOrder(orderId: String): Order
// POST /orders/{orderId}/cancel -> cancelOrder(orderId: String, body: CancelRequest): Order
```

Every path parameter maps to `String` here. The spec's `schema` for `orderId` is a UUID string, and the generator does not read formats for parameters, so a parameter declared as an integer would also come out as `String`. The success type is the schema of the first 2xx response, and an operation whose only response is a non-2xx code gets `Unit`.

## Map Status Codes to a Sealed Error Type per Operation

Each non-2xx response becomes a case of an error type that belongs to its operation. The case is named after the status code, and it carries the response's body when the response has one. Because each operation gets its own type, `getOrder` cannot return the `409` case that only `cancelOrder` declares:

```scala
def errorType(e: Endpoint): Option[SealedTrait] = {
  val name = e.operationId.take(1).toUpperCase + e.operationId.drop(1) + "Error"
  val cases = e.operation.responses.responses.toList.collect {
    case (code, ReferenceOr.Value(response)) if !code.startsWith("2") =>
      val fields = jsonSchema(response.content.toList)
        .map(s => List(Field("body", TypeRef(componentName(s)))))
        .getOrElse(Nil)
      SealedTraitCase.CaseClassCase(
        CaseClass(s"Status$code", fields, derives = List("Schema"), doc = Some(response.description.toString.trim))
      )
  }
  if (cases.isEmpty) None
  else Some(SealedTrait(name, cases = cases, doc = Some(s"Responses of ${e.operationId} other than success.")))
}
```

A `SealedTrait` with cases emits its cases inside a companion object, so `getOrder`'s cases are written as `GetOrderError.Status404`. The generator returns `None` for an operation with no error responses, and then the method returns its success type without `Either`.

## Generate the Service Trait for the Bookstore

The last functions turn each endpoint into a method and collect the error types and the trait into one file:

```scala
def method(e: Endpoint): Method = {
  val params     = List(ParamList(pathParams(e) ++ bodyParam(e)))
  val success    = successType(e)
  val returnType = errorType(e) match {
    case Some(err) => TypeRef.of("Either", TypeRef(err.name), success)
    case None      => success
  }
  Method(e.operationId, params = params.filter(_.params.nonEmpty), returnType = returnType, doc = docOf(e.operation.summary))
}

def serviceTrait(api: OpenAPI, packageName: String): ScalaFile = {
  val eps    = endpoints(api)
  val name   = api.info.title.filter(_.isLetterOrDigit) + "Service"
  val errors = eps.flatMap(errorType)
  val service = Trait(
    name,
    members = eps.map(e => ObjectMember.DefMember(method(e))),
    doc = Some(s"Operations of the ${api.info.title} API, one method per operation.")
  )
  ScalaFile(
    PackageDecl(packageName),
    imports = List(Import.WildcardImport("zio.blocks.schema")),
    types = errors :+ service
  )
}
```

The package is passed in, because the trait belongs next to the data types that another generator writes, and it must be the same package for their names to resolve. Printing the file shows the bookstore's trait:

```scala
println(ScalaEmitter.emit(serviceTrait(api, "com.example.bookstore"), EmitterConfig.default))
// package com.example.bookstore
// 
// import zio.blocks.schema.*
// 
// /** Responses of getOrder other than success. */
// sealed trait GetOrderError
// 
// object GetOrderError {
//   /** No order has this id */
//   case class Status404(
//     body: ApiError,
//   ) extends GetOrderError derives Schema
// }
// 
// /** Responses of cancelOrder other than success. */
// sealed trait CancelOrderError
// 
// object CancelOrderError {
//   /** No order has this id */
//   case class Status404(
//     body: ApiError,
//   ) extends CancelOrderError derives Schema
//   /** The order has already shipped */
//   case class Status409(
//     body: ApiError,
//   ) extends CancelOrderError derives Schema
// }
// 
// /** Operations of the Bookstore API, one method per operation. */
// trait BookstoreService {
//   /** Look up one order by its id. */
//   def getOrder(orderId: String): Either[GetOrderError, Order]
//   /** Cancel an order that has not shipped yet. */
//   def cancelOrder(orderId: String, body: CancelRequest): Either[CancelOrderError, Order]
// }
//
```

The trait refers to `Order`, `CancelRequest` and `ApiError`, which come from the data types of the same spec. Generate those with the [OpenAPI data types generator](./generate-scala-types-from-openapi.md) from the same file. Its output derives `Schema` for each case class, which the `Status` cases need, and both outputs compile together in one package.

## What the OpenAPI Mapping Leaves Out

The table lists what the generator reads from a spec, and what it does not.

| Spec element                           | Handled | Behavior                                                    |
| -------------------------------------- | ------- | ----------------------------------------------------------- |
| Path and operation with an `operationId` | Yes   | One method per operation                                    |
| Path parameters                        | Yes     | Each becomes a `String` parameter, whatever its schema says |
| JSON request body with a `$ref`        | Yes     | A `body` parameter of the referenced type                   |
| 2xx response with a JSON `$ref`        | Yes     | The success type, or `Unit` without a body                  |
| Non-2xx responses                      | Yes     | A case per status code in an error type of the operation    |
| Inline schemas in bodies or responses  | No      | Fails the build: name the schema in `components`            |
| `default` response                     | No      | Ignored                                                     |
| `options`, `head` and `trace` operations | No    | Ignored                                                     |
| Query, header and cookie parameters    | No      | Ignored                                                     |
| Security requirements, callbacks       | No      | Ignored                                                     |

## Running the OpenAPI Example

The OpenAPI files are in [`openapi-examples/src/main/scala/generateservicetraits/`](https://github.com/zio/zio-blocks/tree/main/openapi-examples/src/main/scala/generateservicetraits), and the spec is in [`openapi-examples/src/main/openapi/bookstore-service.json`](https://github.com/zio/zio-blocks/tree/main/openapi-examples/src/main/openapi/bookstore-service.json). Run any file from the `openapi-examples` folder with `sbt "runMain generateservicetraits.<ObjectName>"`, for example `sbt "runMain generateservicetraits.CompleteExample"`.

## Going Further

- **Want the data types for the same spec?** [Generate Scala Types from an OpenAPI Spec at Build Time](./generate-scala-types-from-openapi.md) generates the `Order`, `CancelRequest` and `ApiError` types that the trait above refers to.
- **Want the data types for a Smithy model?** [Generate Scala Code from a Smithy Model](./generate-scala-code-from-smithy.md) generates the structures, unions and enums of a Smithy file.
- **Want to know everything the OpenAPI parser reads?** The [openapi module reference](../reference/openapi.md) lists every type of the `OpenAPI` model.
- **Want to know every IR node the generator writes?** The [codegen module reference](../reference/codegen/index.md) covers `Trait`, `Method`, `SealedTrait` and the other types used here.

---

## Getting Started with Mux

Welcome to Getting Started with Mux! This tutorial is for developers who want to understand how to manage concurrent, request-response style communication where multiple independent conversations happen over a single channel. You don't need any prior experience with multiplexing or the Mux library to follow along — we'll build your understanding step by step.

## Learning Objectives

By the end of this tutorial, you will understand:

- What a `Mux` is and the problem it solves
- How to create a mux and open streams within it
- The relationship between a protocol and your application code
- How `send()` and `receive()` work for application messages
- How `takeOutbound()` and `offerInbound()` work for protocol code
- The stream lifecycle: OPEN → HALF_CLOSED → CLOSED
- How to perform graceful shutdown (halfClose/signalRemoteClose) vs immediate closure (close)
- How to work with multiple independent streams
- When and how to handle errors from capacity limits
- Why thread safety matters and what it means in practice

We'll learn these concepts through:

1. **The Big Picture** — What Mux does and why it's useful
2. **Creating a Mux** — Your first mux and opening a stream
3. **Understanding Streams and Message Queues** — Two-way communication
4. **The Stream Lifecycle** — Open, half-close, and close
5. **Working with Multiple Streams** — Independence and isolation
6. **Managing Capacity** — When limits matter and how to handle them
7. **Thread Safety** — Who can call what, and from where
8. **Putting It Together** — A complete, runnable example
9. **Running the Examples** — Step-by-step commands

We recommend reading from top to bottom — each section builds on the previous one.

:::info[Scala 2 and Scala 3]
The code examples in this tutorial use **Scala 3 syntax**. In Scala 2.13, the API returns `Either[MuxError, T]` instead of union types, and pattern matching uses `case Right(value)` and `case Left(error)` instead of `case value` and `case error`. For complete API signatures with both syntaxes, see the [Mux Reference](../reference/mux.mdx).

**Example: Scala 2 equivalent:**
```scala
// Scala 3
stream.send("Hello") match {
  case () => println("Success")
  case error: MuxError => println(s"Error: $error")
}

// Scala 2
stream.send("Hello") match {
  case Right(()) => println("Success")
  case Left(error) => println(s"Error: $error")
}
```
:::

## The Big Picture

Imagine you're building a communication protocol: a client sends requests to a server over a network connection, and the server sends responses back. Both directions use the same channel, but they're independent of each other. The client doesn't need to wait for one response before sending the next request — it can send five requests in a row, get three responses, send one more request, then get two more responses.

This is multiplexing: packing multiple independent conversations into a single bidirectional channel.

A `Mux` is a registry of these conversations. It manages:

- **Streams**: Each stream is one independent conversation, identified by a numeric ID.
- **Two message queues per stream**: One for messages going out (outbound), one for messages coming in (inbound).
- **Capacity limits**: A per-stream queue capacity and a per-mux overall capacity to prevent memory exhaustion.
- **Lifecycle management**: Each stream transitions through well-defined states (OPEN, HALF_CLOSED, CLOSED) to ensure orderly shutdown.

The key insight: application code calls `send()` and `receive()` to exchange messages with its peer. Protocol code (your networking layer) calls `takeOutbound()` and `offerInbound()` to move messages in and out of the mux. The mux itself never touches the network — it just manages queues and state.

## 1. Creating a Mux

The simplest starting point is to create a mux and open your first stream. A `Mux` is parameterized by three types:

- The **stream ID type** — typically `Int` or `Long`
- The **outbound message type** — what your application sends out
- The **inbound message type** — what your application receives

Let's create a mux that carries `String` messages in both directions, with stream IDs as integers:

```scala

// Create a mux that carries String messages in both directions
val mux = Mux[Int, String, String](100)
```

The code above:
- `Mux[Int, String, String]` — creates a mux where streams are identified by `Int`, outbound messages are `String`, and inbound messages are also `String`
- `(100)` — the mux can have up to 100 concurrent streams. Each stream has its own message queues with a fixed capacity of 256 messages per direction (inbound and outbound)

Now let's open a stream and pattern-match on the result (because opening can fail if capacity is exceeded):

```scala

val mux = Mux[Int, String, String](100)

val streamOrError = mux.open(1)  // Try to open stream 1
streamOrError match {
  case stream: MuxStream[Int, String, String] =>
    println(s"Opened stream with ID 1: $stream")
  case error: MuxError =>
    println(s"Failed to open stream: $error")
}
```

The code above:
- `mux.open(1)` — attempts to open a new stream with ID `1`
- The result is either a `MuxStream` (success) or a `MuxError` (failure — e.g., capacity exceeded)
- We pattern-match to handle both cases

:::note[Why Pattern-Matching?]
Opening a stream can fail. If the mux has reached its capacity limit, `open()` returns a `CapacityExceeded` error instead of a stream. By pattern-matching, we ensure our code handles both outcomes.
:::

## 2. Understanding Streams and Message Queues

Once you've opened a stream, you have access to a two-way communication channel. The key insight is that there are two different perspectives:

- **Application perspective**: You call `send()` to send outbound messages and `receive()` to get inbound messages.
- **Protocol perspective**: The protocol calls `takeOutbound()` to extract messages your application sent, and `offerInbound()` to deliver messages it received from the peer.

Let's see how this works:

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case e: MuxError => sys.error(s"Failed to open stream: $e")
}

// Application code: send a message
stream.send("Hello from app") match {
  case () => println("✓ Message sent")
  case error: MuxError => println(s"✗ Send failed: $error")
}

// Protocol code: extract the message to send over network
val outbound = stream.takeOutbound() match {
  case Some(msg) => msg
  case None => "no message available"
  case error: MuxError => s"error: $error"
}
println(s"Protocol will send: $outbound")

// Protocol code: deliver a response from the peer
stream.offerInbound("Hello from peer") match {
  case () => println("✓ Response delivered")
  case error: MuxError => println(s"✗ Offer failed: $error")
}

// Application code: receive the response
val inbound = stream.receive() match {
  case Some(msg) => msg
  case None => "no message available"
  case error: MuxError => s"error: $error"
}
println(s"Application received: $inbound")
```

The code above demonstrates the `MuxStream` API:
- `MuxStream#send(msg)` returns `Unit | MuxError` (success or error); we pattern-match to handle both cases
- `MuxStream#takeOutbound()` returns `Option[String] | MuxError` (Some/None/error); we match all three
- `MuxStream#offerInbound(msg)` returns `Unit | MuxError` (success or error); we pattern-match to handle both
- `MuxStream#receive()` returns `Option[String] | MuxError` (Some/None/error); we match all three cases

:::tip[Two Queues, Two Roles]
Think of it as a pipeline: your application uses `send()` and `receive()`, the protocol uses `takeOutbound()` and `offerInbound()`. The mux sits in the middle, managing two FIFO queues per stream. This separation is intentional — it lets you reason about sending independently from receiving.
:::

## 3. The Stream Lifecycle

Every stream has a well-defined lifecycle with three states: OPEN, HALF_CLOSED, and CLOSED. This ensures graceful shutdown when both sides need to agree that communication is complete.

The state transitions look like this:

```
OPEN
├─ (local side calls halfClose()) ──→ HALF_CLOSED_LOCAL
│                                       └─ (remote side calls signalRemoteClose()) ──→ CLOSED
│
└─ (remote side calls signalRemoteClose()) ──→ HALF_CLOSED_REMOTE
                                                └─ (local side calls halfClose()) ──→ CLOSED
```

Let's walk through a complete lifecycle:

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case e: MuxError => sys.error(s"Failed to open stream: $e")
}

// Stream starts in OPEN state
println(s"Is stream closed? ${stream.isClosed}")
println(s"Is stream half-closed? ${stream.isHalfClosed}")

// Application is done sending, signals half-close
stream.halfClose()
println(s"After halfClose(), is half-closed? ${stream.isHalfClosed}")

// Protocol receives a close signal from the peer
stream.signalRemoteClose()
println(s"After signalRemoteClose(), is closed? ${stream.isClosed}")
```

The code above:
- `stream.isClosed` — checks if the stream is in CLOSED state
- `stream.isHalfClosed` — checks if the stream is in HALF_CLOSED state (one side done)
- `stream.halfClose()` — signals that this side is done sending (but can still receive)
- `stream.signalRemoteClose()` — signals that the peer is done sending
- Once you call both `halfClose()` and `signalRemoteClose()`, the stream becomes fully closed

However, sometimes you don't need a graceful handshake—you just want to close the stream immediately. The `close()` method transitions directly from any state to CLOSED without the half-close protocol:

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case e: MuxError => sys.error(s"Failed to open stream: $e")
}

// Application decides to close immediately (e.g., due to an error)
println(s"Before close: isClosed = ${stream.isClosed}")

stream.close()
println(s"After close: isClosed = ${stream.isClosed}")

// The close was synchronous and immediate
// Capacity is released, no handshake needed
```

The code above:
- `stream.close()` — transitions the stream from OPEN (or any state) immediately to CLOSED
- The `close()` method also enqueues a terminal error (`StreamClosed`) that `receive()` returns after all buffered messages are drained, allowing applications to detect closure
- This bypasses the graceful half-close handshake and releases capacity immediately
- Use `close()` when you need abrupt termination (error conditions, cleanup)
- Use `halfClose()` + `signalRemoteClose()` for orderly shutdown where both sides need to agree

:::tip[When to Use Each Closure Method]
- **Graceful (halfClose + signalRemoteClose):** Normal shutdown where both sides need to drain pending messages and agree to close
- **Immediate (close):** When the stream should terminate right now (error recovery, cleanup, timeout)
- **External (mux.cancel(id, reason)):** When you need to close a stream from outside its context (thread-safe operation)
:::

### External Stream Cancellation: `mux.cancel(id, reason)`

While `halfClose()` and `close()` are stream-level operations that modify the stream's state directly, **`mux.cancel(id, reason)` is a thread-safe mux-level operation** that cancels a stream from any thread without needing to access the stream object itself. This is useful for cleanup handlers, timeouts, and other scenarios where you can't guarantee single-threaded access.

Here's how to use `mux.cancel()` to externally cancel a stream:

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case e: MuxError                       => sys.error(s"Failed to open stream: $e")
}

// From any thread, you can cancel a stream by ID
// This is thread-safe and doesn't require access to the stream object
mux.cancel(1, MuxError.Cancelled(1, "Cleanup: application shutting down"))
println("✓ Stream 1 cancelled successfully")

// After cancellation, the stream is closed
println(s"Stream isClosed: ${stream.isClosed}")
```

The code above:
- `mux.cancel(streamId, reason)` — cancels the stream with the given ID and enqueues a terminal error on its inbound queue
- This operation is **thread-safe** and you can call it from any thread
- Unlike `close()`, which requires direct access to the stream, `cancel()` works with just the stream ID
- The `reason` parameter becomes the terminal error that `receive()` returns

:::note[Important: `offerInbound()` After `signalRemoteClose()`]
When you call `signalRemoteClose()`, the stream transitions to `HALF_CLOSED_REMOTE` state. In this state, subsequent `offerInbound()` calls fail with `StreamClosed` error because the remote side signals it will not send any more messages. This prevents the protocol layer from enqueuing messages after the peer declares it's done sending.
:::

:::caution[Closing Is Coordinated]
Calling `halfClose()` when the stream is already in HALF_CLOSED_LOCAL or beyond is idempotent—the second call has no effect. Similarly, calling `signalRemoteClose()` when the stream is already in HALF_CLOSED_REMOTE or CLOSED has no effect. Calling `close()` at any time immediately transitions to CLOSED, bypassing any handshake.
:::

## 4. Working with Multiple Streams

The real power of mux comes from managing multiple independent streams simultaneously. Each stream is isolated: messages in stream 1 never appear in stream 2, and capacity is tracked separately.

Let's open three streams and exchange messages on each one:

```scala

val mux = Mux[Int, String, String](100)

// Open three streams
val stream1 = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case e: MuxError                       => sys.error(s"Failed to open stream 1: $e")
}
val stream2 = mux.open(2) match {
  case s: MuxStream[Int, String, String] => s
  case e: MuxError                       => sys.error(s"Failed to open stream 2: $e")
}
val stream3 = mux.open(3) match {
  case s: MuxStream[Int, String, String] => s
  case e: MuxError                       => sys.error(s"Failed to open stream 3: $e")
}

// Send different messages on each stream
stream1.send("Message for stream 1") match { case () => (); case e: MuxError => println(s"Stream 1 send error: $e") }
stream2.send("Message for stream 2") match { case () => (); case e: MuxError => println(s"Stream 2 send error: $e") }
stream3.send("Message for stream 3") match { case () => (); case e: MuxError => println(s"Stream 3 send error: $e") }

// Extract messages — they stay on their own stream
val out1 = stream1.takeOutbound() match { case Some(msg) => msg; case None => "(empty)"; case e: MuxError => s"error: $e" }
val out2 = stream2.takeOutbound() match { case Some(msg) => msg; case None => "(empty)"; case e: MuxError => s"error: $e" }
val out3 = stream3.takeOutbound() match { case Some(msg) => msg; case None => "(empty)"; case e: MuxError => s"error: $e" }
println(s"Stream 1 outbound: $out1")
println(s"Stream 2 outbound: $out2")
println(s"Stream 3 outbound: $out3")

// Deliver responses to different streams
stream1.offerInbound("Response for stream 1") match { case () => (); case e: MuxError => println(s"Stream 1 offer error: $e") }
stream2.offerInbound("Response for stream 2") match { case () => (); case e: MuxError => println(s"Stream 2 offer error: $e") }
stream3.offerInbound("Response for stream 3") match { case () => (); case e: MuxError => println(s"Stream 3 offer error: $e") }

// Receive on each stream — no crosstalk
val in1 = stream1.receive() match { case Some(msg) => msg; case None => "(empty)"; case e: MuxError => s"error: $e" }
val in2 = stream2.receive() match { case Some(msg) => msg; case None => "(empty)"; case e: MuxError => s"error: $e" }
val in3 = stream3.receive() match { case Some(msg) => msg; case None => "(empty)"; case e: MuxError => s"error: $e" }
println(s"Stream 1 inbound: $in1")
println(s"Stream 2 inbound: $in2")
println(s"Stream 3 inbound: $in3")
```

The code above:
- We open three streams with IDs 1, 2, and 3
- Each stream has its own outbound and inbound queues
- Sending on stream 1 doesn't affect stream 2 or stream 3
- Messages stay on their originating stream — there is no crosstalk

:::tip[Streams Are Independent]
Each stream is a completely independent message channel. If stream 1 fills up its queue, stream 2 is unaffected. If you close stream 1, streams 2 and 3 continue operating normally. This isolation is the core benefit of multiplexing.
:::

## 5. Managing Capacity

The mux enforces two separate capacity limits: **mux-level** (controlling how many streams can exist concurrently) and **per-stream** (controlling how many messages each stream's queues can hold).

- **Mux-level capacity** (set at creation): The total number of concurrent streams. When exceeded, `open()` fails with `CapacityExceeded`.
- **Per-stream capacity** (fixed at 256 messages per direction): Each stream's outbound and inbound queues. When exceeded, `send()` and `offerInbound()` fail with `QueueFull`.

:::note[Null Message Rejection]
Both `send()` and `offerInbound()` reject `null` messages with `MuxError.ProtocolError("null message")`. This ensures the protocol layer never enqueues null values, which helps catch bugs early.
:::

Let's see what happens when we hit these limits:

```scala

// Create a tiny mux with a small capacity
val mux = Mux[Int, String, String](10)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case e: MuxError => sys.error(s"Failed to open stream: $e")
}

// Try to send many messages until the queue fills (per-stream capacity is 256 per direction)
for (i <- 1 to 300) {
  val result = stream.send(s"Message $i")
  result match {
    case () => if (i <= 5 || i > 255) println(s"✓ Sent message $i")
    case error: MuxError => println(s"✗ Message $i failed: $error")
  }
}
```

The code above:
- We create a mux with total capacity 10 (very small)
- Each `send()` either succeeds (returns `Unit`) or fails with a `MuxError`
- When the per-stream queue is full, further sends return `MuxError.QueueFull`
- Each stream has its own per-stream message queues with a fixed capacity of 256 messages per direction

The standard recovery pattern is to drain messages from the outbound queue (by calling `takeOutbound()` repeatedly) until space is available:

```scala

val mux = Mux[Int, String, String](10)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case e: MuxError                       => sys.error(s"Failed to open stream: $e")
}

// Fill the queue
for (i <- 1 to 5) {
  stream.send(s"Message $i") match {
    case ()          => ()
    case e: MuxError => sys.error(s"send failed: $e")
  }
}

// Drain one message from the outbound queue
println(s"Drained: ${stream.takeOutbound()}")

// Now we have space to send again
val result = stream.send("Message 6")
println(s"Sent after draining: $result is Unit")
```

The code above:
- We fill the queue with 5 messages
- We call `takeOutbound()` once to simulate the protocol extracting and sending a message
- This frees space in the queue, and the next `send()` succeeds

:::caution[Handling QueueFull in Real Code]
In a real application, `QueueFull` means the protocol is not keeping up with the application. You should either:
- Wait for the protocol to drain messages (using a poll loop or a callback mechanism)
- Close the stream if you decide recovery is impossible
- Implement a backpressure mechanism that slows the application
Never ignore `QueueFull` — it indicates a broken contract between layers.
:::

## 6. Thread Safety

The mux implementation is **thread-safe** for certain operations and **single-threaded** for others. Understanding this boundary is essential.

**Mux-level thread-safe operations** (call from any thread on the mux object):
- `mux.open(id)` — open a new stream
- `mux.get(id)` — retrieve a stream by ID
- `mux.cancel(id, reason)` — externally cancel a stream (thread-safe alternative to `close()`)
- `mux.closeAll(reason)` — close all streams

**Stream-level thread-safe operations** (call from multiple threads on a stream):
- `stream.send(msg)` — queue a message to send (uses lock-free ring buffer on JVM)
- `stream.offerInbound(msg)` — deliver a received message (uses lock-free ring buffer on JVM)

**Stream-level single-threaded operations** (call from the same dedicated thread only):
- `stream.receive()` — dequeue an inbound message
- `stream.takeOutbound()` — dequeue an outbound message
- `stream.halfClose()` — signal local side is done sending
- `stream.signalRemoteClose()` — signal remote side is done sending
- `stream.close()` — immediately close the stream

**Why?** The outbound and inbound queues are lock-free (on JVM) or single-threaded (on JS), which requires that the consumer threads remain stable. State transitions (`halfClose()`, `signalRemoteClose()`, `close()`) are not locked and should be coordinated from a single thread or with external synchronization.

Here's a correct usage pattern:

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case e: MuxError                       => sys.error(s"Failed to open stream: $e")
}

// Thread A: application code (single thread)
// Always call send() from the same thread
stream.send("Message 1") match { case () => println("✓ Sent 1"); case e: MuxError => println(s"✗ Error: $e") }
stream.send("Message 2") match { case () => println("✓ Sent 2"); case e: MuxError => println(s"✗ Error: $e") }

// Thread B: protocol code (single thread)
// Always call takeOutbound() from the same thread
stream.takeOutbound() match { case Some(msg) => println(s"Extracted: $msg"); case None => println("(empty)"); case e: MuxError => println(s"✗ Error: $e") }
stream.takeOutbound() match { case Some(msg) => println(s"Extracted: $msg"); case None => println("(empty)"); case e: MuxError => println(s"✗ Error: $e") }

// Thread C, D, E, ... : multiple threads can safely deliver inbound messages
// This is thread-safe
for (_ <- 1 to 3) {
  stream.offerInbound("Response from peer") match { case () => (); case e: MuxError => println(s"✗ Offer error: $e") }
}

// Thread A: back on the application thread
// Always receive() on the same thread
stream.receive() match { case Some(msg) => println(s"Received: $msg"); case None => println("(empty)"); case e: MuxError => println(s"✗ Error: $e") }
stream.receive() match { case Some(msg) => println(s"Received: $msg"); case None => println("(empty)"); case e: MuxError => println(s"✗ Error: $e") }
stream.receive() match { case Some(msg) => println(s"Received: $msg"); case None => println("(empty)"); case e: MuxError => println(s"✗ Error: $e") }
```

The code above:
- Call `send()` and `receive()` each from a single dedicated thread, but they **can be different threads**
- Call `takeOutbound()` from a single dedicated thread (the protocol/reading thread)
- Multiple threads can safely call `offerInbound()` to deliver inbound messages
- Violating the single-threaded requirement for `receive()` and `takeOutbound()` corrupts the ring buffer

:::tip[Comparing Closure Methods]
There are three ways to close a stream, each with different semantics and thread safety properties:

| Method | Thread-Safe | Semantics | When to Use |
|--------|:---:|-----------|------------|
| **halfClose() + signalRemoteClose()** | ❌ Single-threaded | Graceful shutdown: both sides agree to close, pending messages drained | Normal orderly shutdown |
| **close()** | ❌ Single-threaded | Immediate closure: transitions to CLOSED right now, capacity released | Error recovery, cleanup, timeouts |
| **mux.cancel(id, reason)** | ✅ Thread-safe | External cancellation: close from any thread with an error reason | Cleanup from outside the stream, concurrent scenarios |

For example, use `mux.cancel(streamId, error)` when you're shutting down the entire application and need to close streams from a cleanup handler (which might run on a different thread).
:::

:::danger[Thread Safety Violation]
Do **not** call `receive()` or `takeOutbound()` from multiple threads on the same stream. These methods operate on lock-free ring buffers (JVM) or mutable arrays (JS) that are not thread-safe for concurrent access. Violating this will cause data corruption or panics.
:::

## 7. Putting It Together

Now let's write a complete example that demonstrates all the concepts: creating a mux, opening streams, exchanging messages, handling errors, managing capacity, and closing gracefully:

```scala

object MuxExample {
  def main(args: Array[String]): Unit = {
    // Create a mux: stream IDs are Int, messages are String in both directions
    val mux = Mux[Int, String, String](100)

    println("=== Opening Streams ===")
    // Open three streams, handling potential capacity errors
    val streams = (1 to 3).map { id =>
      mux.open(id) match {
        case stream: MuxStream[Int, String, String] =>
          println(s"Opened stream $id")
          stream
        case error: MuxError =>
          println(s"Failed to open stream $id: $error")
          throw new RuntimeException(s"Cannot proceed without stream $id")
      }
    }.toList

    println("\n=== Application Sends Messages ===")
    // Application sends messages on each stream
    streams.zipWithIndex.foreach { case (stream, idx) =>
      val msg = s"Hello from stream ${idx + 1}"
      stream.send(msg) match {
        case () => println(s"Stream ${idx + 1} sent: $msg")
        case error: MuxError => println(s"Stream ${idx + 1} send failed: $error")
      }
    }

    println("\n=== Protocol Extracts Outbound Messages ===")
    // Protocol extracts messages to send over the network
    streams.zipWithIndex.foreach { case (stream, idx) =>
      stream.takeOutbound() match {
        case Some(msg) => println(s"Protocol will send on stream ${idx + 1}: $msg")
        case None => println(s"Stream ${idx + 1} has no outbound messages")
        case error: MuxError => println(s"Stream ${idx + 1} takeOutbound failed: $error")
      }
    }

    println("\n=== Protocol Delivers Inbound Messages ===")
    // Protocol delivers responses received from the peer
    streams.zipWithIndex.foreach { case (stream, idx) =>
      val response = s"Response from peer on stream ${idx + 1}"
      stream.offerInbound(response) match {
        case () => println(s"Protocol delivered on stream ${idx + 1}: $response")
        case error: MuxError => println(s"Stream ${idx + 1} offerInbound failed: $error")
      }
    }

    println("\n=== Application Receives Messages ===")
    // Application receives the responses
    streams.zipWithIndex.foreach { case (stream, idx) =>
      stream.receive() match {
        case Some(msg) => println(s"Stream ${idx + 1} received: $msg")
        case None => println(s"Stream ${idx + 1} has no inbound messages")
        case error: MuxError => println(s"Stream ${idx + 1} receive failed: $error")
      }
    }

    println("\n=== Graceful Shutdown ===")
    // Streams close gracefully
    val stream1 = streams(0)
    stream1.halfClose()
    println("Stream 1: halfClose() called")

    stream1.signalRemoteClose()
    println(s"Stream 1: signalRemoteClose() called, isClosed = ${stream1.isClosed}")

    // Verify the stream is closed
    assert(stream1.isClosed, "Stream 1 should be closed")

    println("\n=== Verify Independence ===")
    // Streams 2 and 3 are unaffected by stream 1 closing
    println(s"Stream 2 is open: ${!streams(1).isClosed}")
    println(s"Stream 3 is open: ${!streams(2).isClosed}")

    println("\nExample complete!")
  }
}
```

This example demonstrates:
- Creating a mux with a specific capacity
- Opening multiple streams and handling errors
- Application code sending messages and receiving responses
- Protocol code extracting and delivering messages
- Graceful stream closure using halfClose and signalRemoteClose
- Verifying that streams remain independent

## What You've Learned

In this tutorial, you learned:

- What a `Mux` is: a registry that manages multiple independent, multiplexed message streams
- How to create a mux and open streams within it, handling capacity errors
- The two-role pattern: application code uses `send()` and `receive()`, protocol code uses `takeOutbound()` and `offerInbound()`
- How streams transition through a well-defined lifecycle (OPEN → HALF_CLOSED → CLOSED)
- How to perform graceful shutdown (halfClose/signalRemoteClose) vs immediate closure (close)
- Why multiple streams are completely independent with no crosstalk
- How capacity limits work and the standard recovery pattern (drain to free space)
- The thread-safety contract: only `send()`, `offerInbound()`, and `cancel()` are multi-threaded safe; state transitions and message dequeuing are single-threaded
- How to build a complete request-response system using mux

You now have a solid foundation in how mux simplifies multiplexed communication. The next step is to see how to build production-grade protocols using mux as the foundation.

## Running the Examples

All examples in this tutorial have corresponding runnable Scala files in the `mux-examples` module. Run them in order to progressively build your understanding in practice.

### Creating a Mux

This example creates a mux and opens your first stream, demonstrating the basic API for stream initialization and handling both success and failure cases when capacity is exceeded.

<details>
  <summary>mux-examples/src/main/scala/mux/Example1CreatingAMux.scala</summary>

```scala title="mux-examples/src/main/scala/mux/Example1CreatingAMux.scala" showLineNumbers
package mux

/**
 * Title: Creating a Mux Description: Create a mux that carries String messages
 * in both directions with integer stream IDs, then open a stream and handle
 * both success and failure cases. Run: sbt "mux-examples/runMain
 * mux.example1CreatingAMux"
 */
@main def example1CreatingAMux(): Unit = {
  println("=== Creating a Mux ===")

  // Create a mux that carries String messages in both directions
  val mux = Mux[Int, String, String](100)
  println("Created mux with capacity for 100 concurrent streams")

  println("\n=== Opening a Stream ===")

  // Try to open stream 1
  val streamOrError = mux.open(1)
  streamOrError match {
    case stream: MuxStream[Int, String, String] =>
      println(s"✓ Opened stream with ID 1: $stream")
      println(s"  Stream is closed: ${stream.isClosed}")
      println(s"  Stream is half-closed: ${stream.isHalfClosed}")
    case error: MuxError =>
      println(s"✗ Failed to open stream: $error")
  }
}
```

</details>

Observe the output: the stream is successfully created and its initial state (not closed, not half-closed) is displayed.

```bash
cd mux-examples && sbt "runMain mux.example1CreatingAMux"
```

### Understanding Streams and Message Queues

This example demonstrates the two-perspective communication model where application code uses `send()`/`receive()` while protocol code uses `takeOutbound()`/`offerInbound()`, showing how messages flow through the queue system.

<details>
  <summary>mux-examples/src/main/scala/mux/Example2UnderstandingStreamsAndMessageQueues.scala</summary>

```scala title="mux-examples/src/main/scala/mux/Example2UnderstandingStreamsAndMessageQueues.scala" showLineNumbers
package mux

/**
 * Title: Understanding Streams and Message Queues
 *
 * Description: Demonstrate the two-way communication pattern: application uses
 * send()/receive(), protocol uses takeOutbound()/offerInbound().
 *
 * Run: sbt "mux-examples/runMain
 * mux.example2UnderstandingStreamsAndMessageQueues"
 */
@main def example2UnderstandingStreamsAndMessageQueues(): Unit = {
  println("=== Streams and Message Queues ===\n")

  val mux    = Mux[Int, String, String](100)
  val stream = mux.open(1) match {
    case s: MuxStream[Int, String, String] => s
    case e: MuxError                       => throw new RuntimeException(s"Failed to open stream: $e")
  }

  println("--- Application sends a message ---")
  // Application code: send a message
  stream.send("Hello from app") match {
    case ()              => println("✓ Message sent successfully")
    case error: MuxError => println(s"✗ Send failed: $error")
  }

  println("\n--- Protocol extracts the message ---")
  // Protocol code: extract the message to send over network
  val outbound = stream.takeOutbound() match {
    case Some(msg)       => msg
    case None            => "(no message available)"
    case error: MuxError => s"error: $error"
  }
  println(s"Protocol will send: '$outbound'")

  println("\n--- Protocol delivers a response ---")
  // Protocol code: deliver a response from the peer
  stream.offerInbound("Hello from peer") match {
    case ()              => println("✓ Response delivered successfully")
    case error: MuxError => println(s"✗ Offer failed: $error")
  }

  println("\n--- Application receives the response ---")
  // Application code: receive the response
  val inbound = stream.receive() match {
    case Some(msg)       => msg
    case None            => "(no message available)"
    case error: MuxError => s"error: $error"
  }
  println(s"Application received: '$inbound'")

  println("\n=== Summary ===")
  println("Application uses send() and receive()")
  println("Protocol uses takeOutbound() and offerInbound()")
  println("Messages stay in their FIFO queues per stream")
}
```

</details>

Observe the output: messages sent by the application are extracted by the protocol, and responses delivered by the protocol are received by the application.

```bash
cd mux-examples && sbt "runMain mux.example2UnderstandingStreamsAndMessageQueues"
```

### The Stream Lifecycle

This example shows all three ways to close a stream: graceful two-phase closure (halfClose + signalRemoteClose), immediate closure (close), and external thread-safe cancellation (mux.cancel).

<details>
  <summary>mux-examples/src/main/scala/mux/Example3StreamLifecycle.scala</summary>

```scala title="mux-examples/src/main/scala/mux/Example3StreamLifecycle.scala" showLineNumbers
package mux

/**
 * Title: The Stream Lifecycle
 *
 * Description: Demonstrate state transitions through OPEN → HALF_CLOSED →
 * CLOSED using graceful shutdown, immediate closure, and external cancellation.
 *
 * Run: sbt "mux-examples/runMain mux.example3StreamLifecycle"
 */
@main def example3StreamLifecycle(): Unit = {
  println("=== Graceful Stream Lifecycle (halfClose + signalRemoteClose) ===\n")

  val mux     = Mux[Int, String, String](100)
  val stream1 = mux.open(1) match {
    case s: MuxStream[Int, String, String] => s
    case e: MuxError                       => throw new RuntimeException(s"Failed to open stream 1: $e")
  }

  // Stream starts in OPEN state
  println(s"Initial state:")
  println(s"  isClosed: ${stream1.isClosed}")
  println(s"  isHalfClosed: ${stream1.isHalfClosed}")

  // Application is done sending, signals half-close
  println(s"\nCalling halfClose()...")
  stream1.halfClose()
  println(s"After halfClose():")
  println(s"  isClosed: ${stream1.isClosed}")
  println(s"  isHalfClosed: ${stream1.isHalfClosed}")

  // Protocol receives a close signal from the peer
  println(s"\nCalling signalRemoteClose()...")
  stream1.signalRemoteClose()
  println(s"After signalRemoteClose():")
  println(s"  isClosed: ${stream1.isClosed}")
  println(s"  isHalfClosed: ${stream1.isHalfClosed}")

  println("\n=== Immediate Stream Closure (close) ===\n")

  val stream2 = mux.open(2) match {
    case s: MuxStream[Int, String, String] => s
    case e: MuxError                       => throw new RuntimeException(s"Failed to open stream 2: $e")
  }

  println(s"Before close():")
  println(s"  isClosed: ${stream2.isClosed}")

  // Application decides to close immediately (e.g., due to an error)
  println(s"\nCalling close()...")
  stream2.close()

  println(s"After close():")
  println(s"  isClosed: ${stream2.isClosed}")

  println("\n=== External Stream Cancellation (mux.cancel) ===\n")

  val stream3 = mux.open(3) match {
    case s: MuxStream[Int, String, String] => s
    case e: MuxError                       => throw new RuntimeException(s"Failed to open stream 3: $e")
  }

  println(s"Before cancel():")
  println(s"  isClosed: ${stream3.isClosed}")

  // From any thread, cancel the stream by ID (thread-safe)
  println(s"\nCalling mux.cancel(3, ...)...")
  mux.cancel(3, MuxError.Cancelled(3, "Cleanup: application shutting down"))

  println(s"After mux.cancel():")
  println(s"  isClosed: ${stream3.isClosed}")

  println("\n=== Summary ===")
  println("Graceful shutdown: halfClose() + signalRemoteClose() (single-threaded)")
  println("Immediate closure: close() (single-threaded)")
  println("External cancellation: mux.cancel(id, reason) (thread-safe)")
}
```

</details>

Observe the output: state transitions from OPEN → HALF_CLOSED → CLOSED in graceful shutdown, instant transition in immediate closure, and the final closed state after external cancellation.

```bash
cd mux-examples && sbt "runMain mux.example3StreamLifecycle"
```

### Working with Multiple Streams

This example opens three independent streams and exchanges messages on each one, demonstrating that streams are isolated with no message crosstalk and each has its own inbound and outbound queues.

<details>
  <summary>mux-examples/src/main/scala/mux/Example4WorkingWithMultipleStreams.scala</summary>

```scala title="mux-examples/src/main/scala/mux/Example4WorkingWithMultipleStreams.scala" showLineNumbers
package mux

/**
 * Title: Working with Multiple Streams Description: Demonstrate that multiple
 * streams are independent: each has its own queues with no crosstalk, and
 * messages stay on their originating stream. Run: sbt "mux-examples/runMain
 * mux.example4WorkingWithMultipleStreams"
 */
@main def example4WorkingWithMultipleStreams(): Unit = {
  println("=== Working with Multiple Streams ===\n")

  val mux = Mux[Int, String, String](100)

  println("--- Opening three streams ---")
  // Open three streams
  val stream1 = mux.open(1) match {
    case s: MuxStream[Int, String, String] => s
    case e: MuxError                       => throw new RuntimeException(s"Failed to open stream 1: $e")
  }
  val stream2 = mux.open(2) match {
    case s: MuxStream[Int, String, String] => s
    case e: MuxError                       => throw new RuntimeException(s"Failed to open stream 2: $e")
  }
  val stream3 = mux.open(3) match {
    case s: MuxStream[Int, String, String] => s
    case e: MuxError                       => throw new RuntimeException(s"Failed to open stream 3: $e")
  }
  println("✓ Opened streams 1, 2, 3")

  println("\n--- Application sends messages on each stream ---")
  // Send different messages on each stream
  stream1.send("Message for stream 1") match { case () => (); case e: MuxError => println(s"Stream 1 send error: $e") }
  stream2.send("Message for stream 2") match { case () => (); case e: MuxError => println(s"Stream 2 send error: $e") }
  stream3.send("Message for stream 3") match { case () => (); case e: MuxError => println(s"Stream 3 send error: $e") }
  println("✓ Sent outbound messages on all streams")

  println("\n--- Protocol extracts messages (no crosstalk) ---")
  // Extract messages — they stay on their own stream
  val out1 = stream1.takeOutbound() match {
    case Some(msg)   => msg
    case None        => "(empty)"
    case e: MuxError => s"error: $e"
  }
  val out2 = stream2.takeOutbound() match {
    case Some(msg)   => msg
    case None        => "(empty)"
    case e: MuxError => s"error: $e"
  }
  val out3 = stream3.takeOutbound() match {
    case Some(msg)   => msg
    case None        => "(empty)"
    case e: MuxError => s"error: $e"
  }
  println(s"Stream 1 outbound: '$out1'")
  println(s"Stream 2 outbound: '$out2'")
  println(s"Stream 3 outbound: '$out3'")

  println("\n--- Protocol delivers responses to different streams ---")
  // Deliver responses to different streams
  stream1.offerInbound("Response for stream 1") match {
    case () => (); case e: MuxError => println(s"Stream 1 offer error: $e")
  }
  stream2.offerInbound("Response for stream 2") match {
    case () => (); case e: MuxError => println(s"Stream 2 offer error: $e")
  }
  stream3.offerInbound("Response for stream 3") match {
    case () => (); case e: MuxError => println(s"Stream 3 offer error: $e")
  }
  println("✓ Delivered inbound responses on all streams")

  println("\n--- Application receives on each stream (no crosstalk) ---")
  // Receive on each stream — no crosstalk
  val in1 = stream1.receive() match {
    case Some(msg)   => msg
    case None        => "(empty)"
    case e: MuxError => s"error: $e"
  }
  val in2 = stream2.receive() match {
    case Some(msg)   => msg
    case None        => "(empty)"
    case e: MuxError => s"error: $e"
  }
  val in3 = stream3.receive() match {
    case Some(msg)   => msg
    case None        => "(empty)"
    case e: MuxError => s"error: $e"
  }
  println(s"Stream 1 inbound: '$in1'")
  println(s"Stream 2 inbound: '$in2'")
  println(s"Stream 3 inbound: '$in3'")

  println("\n=== Summary ===")
  println("Each stream is independent:")
  println("  - Own outbound and inbound queues")
  println("  - No message crosstalk between streams")
  println("  - Closing one stream doesn't affect others")
}
```

</details>

Observe the output: messages sent on stream 1 appear only in stream 1's outbound queue, never in streams 2 or 3, proving complete independence.

```bash
cd mux-examples && sbt "runMain mux.example4WorkingWithMultipleStreams"
```

### Managing Capacity

This example demonstrates both mux-level capacity (controlling concurrent streams) and per-stream capacity (controlling message queue depth), showing how to handle QueueFull errors and the recovery pattern of draining to free space.

<details>
  <summary>mux-examples/src/main/scala/mux/Example5ManagingCapacity.scala</summary>

```scala title="mux-examples/src/main/scala/mux/Example5ManagingCapacity.scala" showLineNumbers
package mux

/**
 * Title: Managing Capacity Description: Demonstrate mux-level and per-stream
 * capacity limits, handling QueueFull errors, and the recovery pattern of
 * draining outbound messages to free space. Run: sbt "mux-examples/runMain
 * mux.example5ManagingCapacity"
 */
@main def example5ManagingCapacity(): Unit = {
  println("=== Managing Capacity ===\n")

  println("--- Mux-level capacity (maximum concurrent streams) ---")
  // Create a tiny mux with a small capacity (for demo)
  val mux = Mux[Int, String, String](3)
  println("Created mux with capacity for 3 concurrent streams")

  // Open streams until capacity is exceeded
  println("\nOpening streams:")
  for (i <- 1 to 5) {
    mux.open(i) match {
      case _: MuxStream[Int, String, String] =>
        println(s"  ✓ Opened stream $i")
      case error: MuxError =>
        println(s"  ✗ Failed to open stream $i: $error")
    }
  }

  println("\n--- Per-stream capacity (message queue limits) ---")
  // Create a fresh mux for this demo
  val mux2   = Mux[Int, String, String](100)
  val stream = mux2.open(10) match {
    case s: MuxStream[Int, String, String] => s
    case error: MuxError                   =>
      println(s"Failed to open stream: $error")
      throw new RuntimeException("Cannot proceed")
  }

  println("Sending messages until queue fills (per-stream capacity is 256):")
  var successCount = 0
  var failureCount = 0

  for (i <- 1 to 300) {
    val result = stream.send(s"Message $i")
    result match {
      case () =>
        successCount += 1
        if (i <= 5 || i == 256) println(s"  ✓ Message $i sent")
      case error: MuxError =>
        failureCount += 1
        if (failureCount <= 3) println(s"  ✗ Message $i failed: $error")
    }
  }
  println(s"Summary: $successCount succeeded, $failureCount failed")

  println("\n--- Recovery pattern: draining to free space ---")
  // Fresh mux for recovery demo
  val mux3    = Mux[Int, String, String](100)
  val stream2 = mux3.open(11) match {
    case s: MuxStream[Int, String, String] => s
    case e: MuxError                       => throw new RuntimeException(s"Failed to open stream 11: $e")
  }

  // Fill the queue to capacity (256 messages per direction)
  println("Filling queue to capacity (256 messages):")
  for (i <- 1 to 256) {
    stream2.send(s"Message $i") match {
      case ()          => if (i == 1 || i == 256) println(s"  ✓ Message $i sent")
      case e: MuxError => println(s"  ✗ Unexpected failure at $i: $e")
    }
  }

  // Queue is now full — the next send returns QueueFull
  println("\nAttempting send when queue is full:")
  stream2.send("overflow") match {
    case ()          => println("  (unexpected: send succeeded)")
    case e: MuxError => println(s"  ✗ Expected error: $e")
  }

  // Protocol drains one message to free space
  println("\nProtocol drains one message from outbound queue:")
  stream2.takeOutbound() match {
    case Some(msg)   => println(s"  Extracted: '$msg'")
    case None        => println("  (queue empty)")
    case e: MuxError => println(s"  Error: $e")
  }

  // Retry after draining — now succeeds
  println("Retrying send after draining:")
  stream2.send("overflow") match {
    case ()          => println("✓ Send succeeded after draining (recovery pattern works)")
    case e: MuxError => println(s"✗ Still failed: $e")
  }

  println("\n=== Summary ===")
  println("Capacity limits prevent memory exhaustion:")
  println("  - Mux-level: controls number of concurrent streams")
  println("  - Per-stream: limits messages in each queue (256 per direction)")
  println("Recovery pattern: drain outbound queue to free space")
}
```

</details>

Observe the output: the mux rejects stream 4 and 5 with CapacityExceeded, messages 257+ fail with QueueFull, and after draining one message, sending resumes successfully.

```bash
cd mux-examples && sbt "runMain mux.example5ManagingCapacity"
```

### Thread Safety

This example documents which operations are thread-safe (mux-level and send/offerInbound) versus single-threaded (state transitions and message dequeuing), with a correct usage pattern showing how threads interact safely.

<details>
  <summary>mux-examples/src/main/scala/mux/Example6ThreadSafety.scala</summary>

```scala title="mux-examples/src/main/scala/mux/Example6ThreadSafety.scala"
package mux

/**
 * Title: Thread Safety Description: Demonstrate thread-safe vs single-threaded
 * operations, correct usage patterns, and what operations can be called from
 * which context. Run: sbt "mux-examples/runMain mux.example6ThreadSafety"
 */
@main def example6ThreadSafety(): Unit = {
  println("=== Thread Safety: Operations and Constraints ===\n")

  val mux = Mux[Int, String, String](100)

  println("--- Mux-level thread-safe operations (call from any thread) ---")
  println("✓ mux.open(id) — thread-safe")
  println("✓ mux.get(id) — thread-safe")
  println("✓ mux.cancel(id, reason) — thread-safe")
  println("✓ mux.closeAll(reason) — thread-safe")

  println("\n--- Stream-level thread-safe operations (multiple threads OK) ---")
  println("✓ stream.send(msg) — thread-safe (uses lock-free ring buffer on JVM)")
  println("✓ stream.offerInbound(msg) — thread-safe (uses lock-free ring buffer on JVM)")

  println("\n--- Stream-level SINGLE-THREADED operations (ONE thread only) ---")
  println("✗ stream.receive() — single-threaded (corrupts if called concurrently)")
  println("✗ stream.takeOutbound() — single-threaded (corrupts if called concurrently)")
  println("✗ stream.halfClose() — single-threaded (state transition)")
  println("✗ stream.signalRemoteClose() — single-threaded (state transition)")
  println("✗ stream.close() — single-threaded (state transition)")

  println("\n--- Correct usage pattern ---\n")

  // Reset
  val stream2 = mux.open(2) match {
    case s: MuxStream[Int, String, String] => s
    case e: MuxError                       => throw new RuntimeException(s"Failed to open stream 2: $e")
  }

  println("Thread A (Application thread): calls send() consistently")
  stream2.send("Message 1") match {
    case ()          => println("  ✓ Sent message 1 from thread A")
    case e: MuxError => println(s"  ✗ Error: $e")
  }
  stream2.send("Message 2") match {
    case ()          => println("  ✓ Sent message 2 from thread A")
    case e: MuxError => println(s"  ✗ Error: $e")
  }

  println("\nThread B (Protocol/Read thread): calls takeOutbound() consistently")
  stream2.takeOutbound() match {
    case Some(msg)   => println(s"  ✓ Extracted from thread B: '$msg'")
    case None        => println("  (queue empty)")
    case e: MuxError => println(s"  ✗ Error: $e")
  }
  stream2.takeOutbound() match {
    case Some(msg)   => println(s"  ✓ Extracted from thread B: '$msg'")
    case None        => println("  (queue empty)")
    case e: MuxError => println(s"  ✗ Error: $e")
  }

  println("\nThreads C, D, E, ... (Multiple inbound delivery threads): can safely call offerInbound()")
  for (threadId <- 1 to 3) {
    stream2.offerInbound(s"Response from thread $threadId") match {
      case ()          => println(s"  ✓ Offered inbound from thread $threadId (thread-safe)")
      case e: MuxError => println(s"  ✗ Error: $e")
    }
  }

  println("\nThread A (back on Application thread): calls receive() consistently")
  stream2.receive() match {
    case Some(msg)   => println(s"  ✓ Received from thread A: '$msg'")
    case None        => println("  (queue empty)")
    case e: MuxError => println(s"  ✗ Error: $e")
  }
  stream2.receive() match {
    case Some(msg)   => println(s"  ✓ Received from thread A: '$msg'")
    case None        => println("  (queue empty)")
    case e: MuxError => println(s"  ✗ Error: $e")
  }
  stream2.receive() match {
    case Some(msg)   => println(s"  ✓ Received from thread A: '$msg'")
    case None        => println("  (queue empty)")
    case e: MuxError => println(s"  ✗ Error: $e")
  }

  println("\n--- Closing streams: thread safety comparison ---\n")

  println("Single-threaded close (from dedicated thread):")
  println("  stream.halfClose() + stream.signalRemoteClose()")
  println("  OR stream.close()")

  println("\nThread-safe external cancel (from any thread):")
  println("  mux.cancel(id, reason)")
  println("  → Call this when closing from cleanup handler or different thread")
  mux.cancel(2, MuxError.Cancelled(2, "Example cleanup"))
  println("  ✓ Cancelled stream 2 from main thread (would work from any thread)")

  println("\n=== Summary ===")
  println("✓ send() and offerInbound() are thread-safe (use lock-free queues)")
  println("✓ mux operations (open, cancel, closeAll) are thread-safe")
  println("✗ receive(), takeOutbound(), and state transitions must use ONE thread")
  println("✓ Use mux.cancel(id, reason) for thread-safe external cancellation")
}
```

</details>

Observe the output: mux.open and mux.cancel are declared thread-safe, while receive() and takeOutbound() must use single threads, and the pattern shows multiple threads safely offering inbound messages.

```bash
cd mux-examples && sbt "runMain mux.example6ThreadSafety"
```

### Putting It All Together

This comprehensive example demonstrates a complete request-response system that ties together all concepts: creating a mux, opening multiple streams, exchanging messages, handling errors, managing lifecycle transitions, and verifying independence.

<details>
  <summary>mux-examples/src/main/scala/mux/CompleteExample.scala</summary>

```scala title="mux-examples/src/main/scala/mux/CompleteExample.scala" showLineNumbers
package mux

/**
 * Title: Putting It Together Description: Complete, runnable example that
 * demonstrates all mux concepts: creating mux, opening streams, exchanging
 * messages, handling errors, managing capacity, and closing gracefully. This is
 * the comprehensive end-to-end demonstration showing real request-response
 * communication. Run: sbt "mux-examples/runMain mux.completeExample"
 */
@main def completeExample(): Unit = {
  println("=" * 70)
  println("COMPLETE MUX EXAMPLE: Request-Response Communication")
  println("=" * 70)

  // Create a mux: stream IDs are Int, messages are String in both directions
  val mux = Mux[Int, String, String](100)
  println("\n✓ Created mux with capacity for 100 concurrent streams\n")

  println("=" * 70)
  println("OPENING STREAMS")
  println("=" * 70)

  // Open three streams, handling potential capacity errors
  val streams = (1 to 3).map { id =>
    mux.open(id) match {
      case stream: MuxStream[Int, String, String] =>
        println(s"✓ Opened stream $id")
        stream
      case error: MuxError =>
        println(s"✗ Failed to open stream $id: $error")
        throw new RuntimeException(s"Cannot proceed without stream $id")
    }
  }.toList

  println("\n" + "=" * 70)
  println("APPLICATION SENDS MESSAGES")
  println("=" * 70)

  // Application sends messages on each stream
  println()
  streams.zipWithIndex.foreach { case (stream, idx) =>
    val msg = s"Hello from stream ${idx + 1}"
    stream.send(msg) match {
      case ()          => println(s"✓ Stream ${idx + 1} sent: '$msg'")
      case _: MuxError => println(s"✗ Stream ${idx + 1} send failed")
    }
  }

  println("\n" + "=" * 70)
  println("PROTOCOL EXTRACTS OUTBOUND MESSAGES")
  println("=" * 70)

  // Protocol extracts messages to send over the network
  println()
  streams.zipWithIndex.foreach { case (stream, idx) =>
    stream.takeOutbound() match {
      case Some(msg) =>
        println(s"✓ Protocol will send on stream ${idx + 1}: '$msg'")
      case None =>
        println(s"✗ Stream ${idx + 1} has no outbound messages")
      case _: MuxError =>
        println(s"✗ Stream ${idx + 1} takeOutbound failed")
    }
  }

  println("\n" + "=" * 70)
  println("PROTOCOL DELIVERS INBOUND MESSAGES")
  println("=" * 70)

  // Protocol delivers responses received from the peer
  println()
  streams.zipWithIndex.foreach { case (stream, idx) =>
    val response = s"Response from peer on stream ${idx + 1}"
    stream.offerInbound(response) match {
      case ()          => println(s"✓ Protocol delivered on stream ${idx + 1}: '$response'")
      case _: MuxError => println(s"✗ Stream ${idx + 1} offerInbound failed")
    }
  }

  println("\n" + "=" * 70)
  println("APPLICATION RECEIVES MESSAGES")
  println("=" * 70)

  // Application receives the responses
  println()
  streams.zipWithIndex.foreach { case (stream, idx) =>
    stream.receive() match {
      case Some(msg) =>
        println(s"✓ Stream ${idx + 1} received: '$msg'")
      case None =>
        println(s"✗ Stream ${idx + 1} has no inbound messages")
      case _: MuxError =>
        println(s"✗ Stream ${idx + 1} receive failed")
    }
  }

  println("\n" + "=" * 70)
  println("GRACEFUL SHUTDOWN")
  println("=" * 70)

  // Streams close gracefully
  println()
  val stream1 = streams(0)
  println("Stream 1: signalling local half-close...")
  stream1.halfClose()
  println(s"  isClosed: ${stream1.isClosed}, isHalfClosed: ${stream1.isHalfClosed}")

  println("Stream 1: signalling remote close...")
  stream1.signalRemoteClose()
  println(s"  isClosed: ${stream1.isClosed}, isHalfClosed: ${stream1.isHalfClosed}")

  // Verify the stream is closed
  assert(stream1.isClosed, "Stream 1 should be closed")
  println("✓ Stream 1 is fully closed")

  println("\n" + "=" * 70)
  println("VERIFY INDEPENDENCE")
  println("=" * 70)

  // Streams 2 and 3 are unaffected by stream 1 closing
  println()
  println(s"Stream 2 is open: ${!streams(1).isClosed}")
  println(s"Stream 3 is open: ${!streams(2).isClosed}")

  println("\n" + "=" * 70)
  println("IMMEDIATE CLOSURE")
  println("=" * 70)

  // Demonstrate immediate closure on stream 2
  println()
  val stream2 = streams(1)
  println("Stream 2: performing immediate close()...")
  stream2.close()
  println(s"  isClosed: ${stream2.isClosed}")
  println("✓ Stream 2 is immediately closed")

  println("\n" + "=" * 70)
  println("EXTERNAL CANCELLATION")
  println("=" * 70)

  // Demonstrate external cancellation on stream 3
  println()
  println("Stream 3: performing external mux.cancel()...")
  mux.cancel(3, MuxError.Cancelled(3, "Application shutdown"))
  val stream3 = streams(2)
  println(s"  isClosed: ${stream3.isClosed}")
  println("✓ Stream 3 is cancelled")

  println("\n" + "=" * 70)
  println("EXAMPLE COMPLETE")
  println("=" * 70)
  println("\nYou've seen:")
  println("  • Creating a mux with capacity control")
  println("  • Opening multiple independent streams")
  println("  • Application sending messages (send/receive)")
  println("  • Protocol exchanging messages (takeOutbound/offerInbound)")
  println("  • Graceful shutdown (halfClose + signalRemoteClose)")
  println("  • Immediate closure (close())")
  println("  • External cancellation (mux.cancel())")
  println("  • Stream independence and isolation")
  println("\n✓ All concepts demonstrated successfully!")
}
```

</details>

Observe the output: three streams are opened, messages flow through send/receive and takeOutbound/offerInbound, graceful shutdown completes with halfClose/signalRemoteClose, and other streams remain unaffected.

```bash
cd mux-examples && sbt "runMain mux.completeExample"
```

## Where to Go Next

- **Ready to put a mux on a real connection?** Follow [How to Run a Mux over a Real Socket](./run-a-mux-over-a-socket.md) to carry several streams over one TCP connection, with a frame format, reader and writer loops, half-close, back-pressure and shutdown.
- **Want to dive deeper into the API?** Read the reference page for [`Mux`](../reference/mux.mdx) for complete method signatures and advanced patterns.
- **Ready to explore related concepts?** Check out the [Ring Buffer](../reference/ringbuffer/index.mdx) reference for understanding lock-free data structures used internally.
- **Interested in other concurrency and stream management?** Explore other libraries in ZIO Blocks for managing complex async workflows.

---

## Guides

Each guide works through one task with ZIO Blocks from start to finish. New to ZIO Blocks? Read [Getting Started](../getting-started/index.md) first, then the [tutorials](/guides/tutorials), in order: they take you from your first schema to a small working backend. When you need the full description of a data type along the way, the [reference](../reference/index.md) has a page for it.

<SidebarCards />

---

## Instrument a Program with Logs, Traces and Metrics

Welcome! This tutorial is for Scala 3 developers who are new to the ZIO Blocks `telemetry` module. You need to know how to write an `object` with a `def main` and how to add a dependency to `build.sbt`. You don't need any prior experience with logging libraries, tracing or OpenTelemetry to follow along.

## 1. Introduction

By the end of this tutorial, you will understand:

- How the global `log`, `trace` and `metric` objects let you record telemetry with zero setup
- How to attach structured fields to log lines, and how to switch the output to JSON on standard error
- How spans measure the steps of a job, how they nest, and how every log line written inside a span carries the span's ids
- How a counter and a histogram summarize what happened across many runs
- How to read recorded spans back with `trace.collectedSpans` and assert on them, so instrumentation can't silently break

We learn these concepts by instrumenting a bookstore's nightly restock job, one step at a time. The job restocks three books: Zionomicon, Programming in Scala, and Functional Programming in Scala (the counts are made up).

1. Your first log line
2. Add structured fields
3. Switch to JSON logs on standard error
4. Trace the job with spans
5. Count restocked books
6. Time each step
7. Test the instrumentation

We recommend reading from top to bottom — each section builds on the previous one.

To follow along, add the `telemetry` module to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-telemetry" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-telemetry" % "0.0.56"
scalacOptions += "-experimental"
```

On the JVM, the `telemetry` module needs **JDK 21 or newer**. It is compiled for Java 21, so on JDK 17 the compiler fails with `java.lang.UnsupportedClassVersionError` as soon as it expands a `log.*` call (the log macros run inside the compiler), and code without log calls compiles but fails the same way when it runs. JDK 21 needs no extra flags such as `--enable-preview`. [How to Add ZIO Blocks to Your Build](../getting-started/installation.md) explains the two artifact names and the JDK each module needs. The code in this tutorial is Scala 3.

:::caution
In version 0.0.56, the `log.*` calls don't compile on Scala 2.13 outside the library's own package: the compiler reports `object GlobalLogState in package telemetry cannot be accessed`. This is a known bug in that version. Spans and metrics are not affected.
:::

## 2. Background: Three Signals

When a program runs unattended — a nightly job, a server, a queue consumer — you can't watch it with a debugger. You find out what it did from the records it leaves behind. Observability tools split those records into three kinds of signal, each answering a different question:

- A **log** is a timestamped line that says *what happened*: "restock started", "title restocked".
- A **trace** says *where the time went*. It is a tree of **spans**: one span per unit of work, with a start time, an end time and a parent. A span for the whole job contains one span for each of its steps.
- A **metric** says *how many* or *how much*, summed over time: books restocked per title, or how long each step usually takes.

The ZIO Blocks `telemetry` module follows the OpenTelemetry data model for all three, and gives you one global entry point per signal: `log`, `trace` and `metric`. They work as soon as the module is on the classpath. Logs go to standard output, and spans and metrics are kept in memory, where your program and your tests can read them. The [Telemetry Guide](./telemetry-guide.md) explains how the pieces fit together; this tutorial focuses on using them.

## 3. Your First Log Line

Let's start the restock job with a single log line. The `log` object has one method per severity level — `trace`, `debug`, `info`, `warn`, `error` and `fatal` — and each one takes the message as a `String`:

```scala

object RestockJob {
  def run(): Unit =
    log.info("Nightly restock started")
}

RestockJob.run()
// <timestamp> INFO  [(repl.MdocSession$.MdocApp4.RestockJob$,27,38).run:72] Nightly restock started
```

The code above:

- `import zio.blocks.telemetry._` brings `log`, `trace` and `metric` into scope, along with every type this tutorial uses.
- `log.info("Nightly restock started")` writes one line at the `INFO` level. There is no logger to create and nothing to configure first.

The line goes to standard output, shown above as a comment under the call. Each line has four parts: the time in UTC (shown here as `<timestamp>`, because it changes on every run), the severity padded to five characters, the source location in square brackets, and the message. The `log.*` methods are macros, so the source location is filled in at compile time — the class, the method and the line number of the call — at no runtime cost.

:::caution
The brackets should hold the short class name, the method and the line, such as `[RestockJob$.run:5]`. In version 0.0.56, a class inside a package prints its full name followed by two numbers instead, as in `[(bookstore.RestockJob$,10,21).run:5]`. On this page, the class also sits inside the `repl.MdocSession` wrapper that renders the examples. This is a known bug in that version; the message and fields are not affected.
:::

## 4. Add Structured Fields

A message such as "Restocked 12 copies of Zionomicon" is easy for a person to read, but a log search tool has to parse the text to find the title. Structured fields keep the values separate. Pass them after the message as `key -> value` pairs; keys are strings, and values can be a `String`, an `Int`, a `Long`, a `Double` or a `Boolean`:

```scala

object RestockJob {
  val stock = List("Zionomicon" -> 12, "Programming in Scala" -> 5, "Functional Programming in Scala" -> 8)

  def run(): Unit =
    log.annotated("job.id" -> "restock-2026-10-21") {
      log.info("Nightly restock started")
      stock.foreach { case (title, copies) =>
        log.info("Restocked a title", "title" -> title, "copies" -> copies)
      }
    }
}

RestockJob.run()
// <timestamp> INFO  [(repl.MdocSession$.MdocApp6.RestockJob$,27,38).run:96] Nightly restock started {job.id="restock-2026-10-21"}
// <timestamp> INFO  [(repl.MdocSession$.MdocApp6.RestockJob$,27,38).run:98] Restocked a title {job.id="restock-2026-10-21", title="Zionomicon", copies=12}
// <timestamp> INFO  [(repl.MdocSession$.MdocApp6.RestockJob$,27,38).run:98] Restocked a title {job.id="restock-2026-10-21", title="Programming in Scala", copies=5}
// <timestamp> INFO  [(repl.MdocSession$.MdocApp6.RestockJob$,27,38).run:98] Restocked a title {job.id="restock-2026-10-21", title="Functional Programming in Scala", copies=8}
```

The code above:

- `stock` lists each title with the number of copies to add.
- `log.annotated("job.id" -> "restock-2026-10-21") { ... }` attaches the `job.id` field to every log line written while the block runs, including lines written by methods it calls. Annotations take `String` values.
- `log.info("Restocked a title", "title" -> title, "copies" -> copies)` adds two fields to one line. The `Int` value is stored as a whole number, not as text.

Notice that the fields appear in braces after the message: strings in quotes, numbers without them. The annotation comes first, followed by the fields of the call itself. Annotations are the way to tag a whole unit of work, such as a job run or a request, without passing the id to every method.

## 5. Switch to JSON Logs on Standard Error

Text lines are pleasant in a terminal, but log collectors prefer one JSON object per line, and many deployments keep standard output for the program's real output. Two calls switch the global log output to JSON on standard error:

```scala

log.install(LoggerProvider.builder.build().get("bookstore.restock"))
log.writer(JsonLogFormatter, StderrWriter)

object RestockJob {
  def run(): Unit =
    log.info("Nightly restock started", "titles" -> 3)
}

RestockJob.run()
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Nightly restock started"},"attributes":[{"key":"code.filepath","value":{"stringValue":"instrument-a-program-with-logs-traces-and-metrics.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp8.RestockJob$"}},{"key":"code.function","value":{"stringValue":"run"}},{"key":"code.lineno","value":{"intValue":"131"}},{"key":"titles","value":{"intValue":"3"}}]}
```

The code above:

- `LoggerProvider.builder.build().get("bookstore.restock")` creates a logger with no outputs of its own; `"bookstore.restock"` names the part of the program it logs for.
- `log.install(...)` makes that logger the global one. This removes the default text output to standard output.
- `log.writer(JsonLogFormatter, StderrWriter)` adds an output that formats each line as JSON and writes it to standard error.

Each line is now an OpenTelemetry log record in JSON. The time is in `timeUnixNano`, in nanoseconds since 1970 (shown here as `<time>`). The message is under `body`, the severity is under `severityNumber` and `severityText`, and the source location becomes the `code.*` attributes, next to your own `titles` field. Whole numbers are written as strings (`"intValue":"3"`), as the OpenTelemetry JSON encoding requires.

:::note
`log.writer` adds an output; it never replaces one. Without the `log.install` call first, the default text output would stay, and every line would be printed twice: once as text and once as JSON. The setup is global, so the rest of this tutorial logs JSON to standard error.
:::

## 6. Trace the Job with Spans

Logs tell us what happened, but not how long each step took or which step a line belongs to. A **span** answers both. `trace.span(name) { span => ... }` starts a span, runs the block, and ends the span when the block finishes, even if it throws. A span started inside another span's block becomes its child, so the code's nesting becomes the trace's tree.

Let's give the restock job one root span and one child span per step:

```scala

object RestockJob {
  val stock = List("Zionomicon" -> 12, "Programming in Scala" -> 5, "Functional Programming in Scala" -> 8)

  def run(): Unit =
    trace.span("restock") { _ =>
      trace.span("load-inventory") { _ =>
        log.info("Inventory loaded", "titles" -> stock.size)
      }
      trace.span("restock-titles") { span =>
        span.setAttribute("copies", stock.map(_._2).sum.toLong)
      }
      trace.span("write-report") { span =>
        log.info("Report written")
        span.setStatus(SpanStatus.Ok)
      }
    }
}

RestockJob.run()
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Inventory loaded"},"attributes":[{"key":"code.filepath","value":{"stringValue":"instrument-a-program-with-logs-traces-and-metrics.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp12.RestockJob$"}},{"key":"code.function","value":{"stringValue":"run"}},{"key":"code.lineno","value":{"intValue":"156"}},{"key":"titles","value":{"intValue":"3"}}],"traceId":"<trace-id-1>","spanId":""}
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Report written"},"attributes":[{"key":"code.filepath","value":{"stringValue":"instrument-a-program-with-logs-traces-and-metrics.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp12.RestockJob$"}},{"key":"code.function","value":{"stringValue":"run"}},{"key":"code.lineno","value":{"intValue":"162"}}],"traceId":"<trace-id-1>","spanId":""}
```

The code above:

- `trace.span("restock") { _ => ... }` is the root span; it covers the whole job.
- The three inner `trace.span` calls are its children: one for each step.
- `span.setAttribute("copies", ...)` records a value on the span itself; spans take the same kinds of values as log fields.
- `span.setStatus(SpanStatus.Ok)` marks the step as successful. A span you don't mark keeps the status `Unset`, which also counts as "no error"; `SpanStatus.Error("reason")` marks a failure.

Look closely at the two log lines. Both carry the same `traceId`, because they belong to the same run of the job. Each one carries the `spanId` of the span it was written in, and the ids differ because the lines came from different steps. (Real ids are random hexadecimal strings; this page shows them as numbered placeholders, and placeholders with the same number stand for the same id.) We didn't pass any id to `log.info`: the active span is tracked for the current call stack, and the log macros read it. That is what lets a log tool show "all the lines of this run" or "the lines of this step".

Now let's look at the spans themselves. They are not printed anywhere yet: by default, the global tracer keeps finished spans in memory, and `trace.collectedSpans` returns them as `SpanData` values:

```scala
def printSpans(): Unit = {
  val spans  = trace.collectedSpans
  val nameOf = spans.map(s => s.spanContext.spanId -> s.name).toMap

  spans.foreach { s =>
    val parent = nameOf.getOrElse(s.parentSpanContext.spanId, "none")
    val timed  = s.endTimeNanos >= s.startTimeNanos
    println(s"${s.name}: parent=$parent, status=${s.status}, timed=$timed, attributes=${s.attributes.toMap}")
  }
}

printSpans()
// load-inventory: parent=restock, status=Unset, timed=true, attributes=Map()
// restock-titles: parent=restock, status=Unset, timed=true, attributes=Map(copies -> LongValue(25))
// write-report: parent=restock, status=Ok, timed=true, attributes=Map()
// restock: parent=none, status=Unset, timed=true, attributes=Map()
```

The code above:

- `trace.collectedSpans` returns the finished spans, in the order they ended.
- `nameOf` maps each span's id to its name, so we can print a parent's name instead of its id.
- `s.parentSpanContext.spanId` is the id of the parent span. The root span has no parent, so its parent id matches no span and we print `none`.
- `timed` checks that the span recorded both a start and an end time. We don't print the durations themselves, because they differ on every run.

Notice the order: the children come first, because each child ends before its parent does. Every span shares the root's trace id, and each child's parent is `restock`.

## 7. Count Restocked Books

Spans describe one run. Many questions are about all runs together: how many copies of Zionomicon did we restock this month? A **counter** answers that. It is a running total that only goes up, and each `Counter#add` call can carry attributes, so one counter keeps a separate total for each title.

Let's count the restocked copies:

```scala

object RestockJob {
  val stock = List("Zionomicon" -> 12, "Programming in Scala" -> 5, "Functional Programming in Scala" -> 8)

  val booksRestocked: Counter = metric.counter("bookstore.books.restocked")

  def run(): Unit =
    trace.span("restock") { _ =>
      trace.span("load-inventory") { _ =>
        log.info("Inventory loaded", "titles" -> stock.size)
      }
      trace.span("restock-titles") { _ =>
        stock.foreach { case (title, copies) =>
          booksRestocked.add(copies.toLong, "title" -> title)
        }
      }
      trace.span("write-report") { span =>
        log.info("Report written")
        span.setStatus(SpanStatus.Ok)
      }
    }
}

RestockJob.run()
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Inventory loaded"},"attributes":[{"key":"code.filepath","value":{"stringValue":"instrument-a-program-with-logs-traces-and-metrics.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp15.RestockJob$"}},{"key":"code.function","value":{"stringValue":"run"}},{"key":"code.lineno","value":{"intValue":"211"}},{"key":"titles","value":{"intValue":"3"}}],"traceId":"<trace-id-2>","spanId":""}
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Report written"},"attributes":[{"key":"code.filepath","value":{"stringValue":"instrument-a-program-with-logs-traces-and-metrics.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp15.RestockJob$"}},{"key":"code.function","value":{"stringValue":"run"}},{"key":"code.lineno","value":{"intValue":"219"}}],"traceId":"<trace-id-2>","spanId":""}
```

The code above:

- `metric.counter("bookstore.books.restocked")` creates a counter. We create it once, as a `val`, and reuse it: every call to `metric.counter` registers a new instrument, even with a name that already exists.
- `booksRestocked.add(copies.toLong, "title" -> title)` adds the copies to the total for that title. The `title` attribute splits the counter into one series per title.

To read the totals, ask the counter for its current data. A counter's data is a `MetricData.SumData` with one point per distinct set of attributes:

```scala
val titleKey = AttributeKey.string("title")
// titleKey: AttributeKey[String] = StringKey("title")

RestockJob.booksRestocked.collect() match {
  case MetricData.SumData(points) =>
    points.sortBy(_.attributes.get(titleKey)).foreach { p =>
      println(s"${p.attributes.get(titleKey).getOrElse("?")}: ${p.value}")
    }
  case other => println(s"unexpected: $other")
}
// Functional Programming in Scala: 8
// Programming in Scala: 5
// Zionomicon: 12
```

The code above:

- `AttributeKey.string("title")` is a typed key: `attributes.get(titleKey)` returns an `Option[String]`.
- `Counter#collect` returns a snapshot of the counter's totals.
- `points.sortBy(...)` sorts the series by title, so the output has a stable order.

Each title has its own total. Run the job again and the totals grow: a counter keeps adding up for as long as the program runs.

## 8. Time Each Step

Our spans already record how long each step took in one run. To know how long a step *usually* takes — and how often it is slow — we need the durations of many runs summarized. A **histogram** does that: each recorded value falls into a bucket, and the histogram keeps the count, sum, minimum and maximum of the values.

Let's time each step with a small helper that wraps a step in its span and records its duration in milliseconds:

```scala

object RestockJob {
  val stock = List("Zionomicon" -> 12, "Programming in Scala" -> 5, "Functional Programming in Scala" -> 8)

  val booksRestocked: Counter = metric.counter("bookstore.books.restocked")
  val stepDuration: Histogram = metric.histogram("bookstore.restock.step.duration.ms")

  def step[A](name: String)(body: Span => A): A =
    trace.span(name) { span =>
      val start = System.nanoTime()
      try body(span)
      finally stepDuration.record((System.nanoTime() - start) / 1e6, "step" -> name)
    }

  def run(): Unit =
    trace.span("restock") { _ =>
      step("load-inventory") { _ =>
        log.info("Inventory loaded", "titles" -> stock.size)
      }
      step("restock-titles") { _ =>
        stock.foreach { case (title, copies) =>
          booksRestocked.add(copies.toLong, "title" -> title)
        }
      }
      step("write-report") { span =>
        log.info("Report written")
        span.setStatus(SpanStatus.Ok)
      }
    }
}

RestockJob.run()
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Inventory loaded"},"attributes":[{"key":"code.filepath","value":{"stringValue":"instrument-a-program-with-logs-traces-and-metrics.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp18.RestockJob$"}},{"key":"code.function","value":{"stringValue":"run"}},{"key":"code.lineno","value":{"intValue":"274"}},{"key":"titles","value":{"intValue":"3"}}],"traceId":"<trace-id-3>","spanId":""}
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Report written"},"attributes":[{"key":"code.filepath","value":{"stringValue":"instrument-a-program-with-logs-traces-and-metrics.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp18.RestockJob$"}},{"key":"code.function","value":{"stringValue":"run"}},{"key":"code.lineno","value":{"intValue":"282"}}],"traceId":"<trace-id-3>","spanId":""}
RestockJob.run()
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Inventory loaded"},"attributes":[{"key":"code.filepath","value":{"stringValue":"instrument-a-program-with-logs-traces-and-metrics.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp18.RestockJob$"}},{"key":"code.function","value":{"stringValue":"run"}},{"key":"code.lineno","value":{"intValue":"274"}},{"key":"titles","value":{"intValue":"3"}}],"traceId":"<trace-id-4>","spanId":""}
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Report written"},"attributes":[{"key":"code.filepath","value":{"stringValue":"instrument-a-program-with-logs-traces-and-metrics.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp18.RestockJob$"}},{"key":"code.function","value":{"stringValue":"run"}},{"key":"code.lineno","value":{"intValue":"282"}}],"traceId":"<trace-id-4>","spanId":""}
```

The code above:

- `metric.histogram("bookstore.restock.step.duration.ms")` creates a histogram once, like the counter. We put the unit in the name, because `metric.histogram` takes only a name.
- `step(name) { span => ... }` opens a span for the step and measures the time the body takes. The `finally` records the duration even when the body throws.
- `stepDuration.record(ms, "step" -> name)` adds one value to the series for that step.
- We run the job twice, so each step records two durations.

Now let's read the histogram:

```scala
val stepKey = AttributeKey.string("step")
// stepKey: AttributeKey[String] = StringKey("step")

RestockJob.stepDuration.collect() match {
  case MetricData.HistogramData(points) =>
    points.sortBy(_.attributes.get(stepKey)).foreach { p =>
      println(s"${p.attributes.get(stepKey).getOrElse("?")}: count=${p.count}, buckets=${p.boundaries.length + 1}")
    }
  case other => println(s"unexpected: $other")
}
// load-inventory: count=2, buckets=16
// restock-titles: count=2, buckets=16
// write-report: count=2, buckets=16
```

The code above:

- `MetricData.HistogramData(points)` holds one point per step.
- `p.count` is the number of recorded values: two per step, one per run.
- `p.boundaries` are the bucket limits. The defaults suit milliseconds, from 0 up to 10,000; there is one more bucket than there are boundaries, for values above the last limit. We don't print `p.sum`, `p.min` or `p.max` here, because the durations differ on every run.

The histogram doesn't keep the individual durations. It keeps enough to answer "how many runs, how long in total, and how many fell in each range" for any number of runs, in a fixed amount of memory.

## 9. Test the Instrumentation

Instrumentation is code, and code breaks. Someone renames a step or moves a call outside its span, and dashboards go quiet without any error. Because spans are kept in memory, a test can run the job and check the trace's shape.

Let's write that check as a plain function and run it:

```scala
def spanTree(): List[(String, String)] = {
  val spans  = trace.collectedSpans
  val nameOf = spans.map(s => s.spanContext.spanId -> s.name).toMap
  spans.map(s => s.name -> nameOf.getOrElse(s.parentSpanContext.spanId, "none"))
}

val expected = List(
  "load-inventory" -> "restock",
  "restock-titles" -> "restock",
  "write-report"   -> "restock",
  "restock"        -> "none"
)
```

The code above:

- `spanTree()` turns the collected spans into `(name, parent name)` pairs, which leave out everything that changes between runs: ids, timestamps and durations.
- `expected` is the tree we want: three steps, each a child of `restock`, listed in the order they end.

Now let's run the job and check its trace:

```scala
trace.clearSpans()
RestockJob.run()
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Inventory loaded"},"attributes":[{"key":"code.filepath","value":{"stringValue":"instrument-a-program-with-logs-traces-and-metrics.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp18.RestockJob$"}},{"key":"code.function","value":{"stringValue":"run"}},{"key":"code.lineno","value":{"intValue":"274"}},{"key":"titles","value":{"intValue":"3"}}],"traceId":"<trace-id-5>","spanId":""}
// {"timeUnixNano":"<time>","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Report written"},"attributes":[{"key":"code.filepath","value":{"stringValue":"instrument-a-program-with-logs-traces-and-metrics.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp18.RestockJob$"}},{"key":"code.function","value":{"stringValue":"run"}},{"key":"code.lineno","value":{"intValue":"282"}}],"traceId":"<trace-id-5>","spanId":""}

assert(spanTree() == expected, s"unexpected spans: ${spanTree()}")
println(s"Instrumentation check passed: ${expected.size} spans")
// Instrumentation check passed: 4 spans
```

The code above:

- `trace.clearSpans()` empties the in-memory store first, so spans from earlier runs don't leak into the check. Call it at the start of every test that reads `trace.collectedSpans`.
- `assert(...)` compares the whole tree. If a step is renamed, dropped or moved out of `restock`, the check fails and prints the tree it found.

In a ZIO Test spec, the same check is `assertTrue(spanTree() == expected)`.

## 10. Putting It Together

This example combines all the concepts we've learned: a restock job with structured JSON logs on standard error, a span per step, a counter per title, a histogram per step, and a check on the trace's shape.

```scala title="telemetry-examples/src/main/scala/instrumentaprogramwithlogstracesandmetrics/CompleteExample.scala"
package instrumentaprogramwithlogstracesandmetrics

/**
 * Instrument a Program with Logs, Traces and Metrics: Complete example. JSON logs on standard error, a span per step,
 * a counter per title, a histogram per step, and a check on the shape of the trace.
 *
 * Run with: sbt "runMain instrumentaprogramwithlogstracesandmetrics.CompleteExample"
 */
object CompleteExample {
  object RestockJob {
    val stock = List("Zionomicon" -> 12, "Programming in Scala" -> 5, "Functional Programming in Scala" -> 8)

    val booksRestocked: Counter = metric.counter("bookstore.books.restocked")
    val stepDuration: Histogram = metric.histogram("bookstore.restock.step.duration.ms")

    def step[A](name: String)(body: Span => A): A =
      trace.span(name) { span =>
        val start = System.nanoTime()
        try body(span)
        finally stepDuration.record((System.nanoTime() - start) / 1e6, "step" -> name)
      }

    def run(): Unit =
      log.annotated("job.id" -> "restock-2026-10-21") {
        trace.span("restock") { _ =>
          step("load-inventory") { _ =>
            log.info("Inventory loaded", "titles" -> stock.size)
          }
          step("restock-titles") { _ =>
            stock.foreach { case (title, copies) =>
              booksRestocked.add(copies.toLong, "title" -> title)
            }
          }
          step("write-report") { span =>
            log.info("Report written")
            span.setStatus(SpanStatus.Ok)
          }
        }
      }
  }

  def spanTree(): List[(String, String)] = {
    val spans  = trace.collectedSpans
    val nameOf = spans.map(s => s.spanContext.spanId -> s.name).toMap
    spans.map(s => s.name -> nameOf.getOrElse(s.parentSpanContext.spanId, "none"))
  }

  val expected = List(
    "load-inventory" -> "restock",
    "restock-titles" -> "restock",
    "write-report"   -> "restock",
    "restock"        -> "none"
  )

  def main(args: Array[String]): Unit = {
    log.install(LoggerProvider.builder.build().get("bookstore.restock"))
    log.writer(JsonLogFormatter, StderrWriter)

    RestockJob.run()

    val titleKey = AttributeKey.string("title")
    RestockJob.booksRestocked.collect() match {
      case MetricData.SumData(points) =>
        points.sortBy(_.attributes.get(titleKey)).foreach { p =>
          println(s"${p.attributes.get(titleKey).getOrElse("?")}: ${p.value}")
        }
      case other => println(s"unexpected: $other")
    }

    val stepKey = AttributeKey.string("step")
    RestockJob.stepDuration.collect() match {
      case MetricData.HistogramData(points) =>
        points.sortBy(_.attributes.get(stepKey)).foreach { p =>
          println(s"${p.attributes.get(stepKey).getOrElse("?")}: count=${p.count}, buckets=${p.boundaries.length + 1}")
        }
      case other => println(s"unexpected: $other")
    }

    assert(spanTree() == expected, s"unexpected spans: ${spanTree()}")
    println(s"Instrumentation check passed: ${expected.size} spans")
  }
}
```

## 11. Running the Examples

All the code from this tutorial is available as runnable examples in the `telemetry-examples` module. They need JDK 21 or newer.

To run the examples, clone the ZIO Blocks repository and navigate to the examples module:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/telemetry-examples
```

Each example below is self-contained. Expand one to see its code, then run it with the command beneath it.

<details>
<summary>1. Your first log line</summary>

```scala title="telemetry-examples/src/main/scala/instrumentaprogramwithlogstracesandmetrics/Concept1FirstLogExample.scala" showLineNumbers
package instrumentaprogramwithlogstracesandmetrics

/**
 * Instrument a Program with Logs, Traces and Metrics: Your first log line.
 *
 * Writes one INFO line to standard output with the global `log` object, with no setup.
 *
 * Run with: sbt "runMain instrumentaprogramwithlogstracesandmetrics.Concept1FirstLogExample"
 */
object Concept1FirstLogExample {
  object RestockJob {
    def run(): Unit =
      log.info("Nightly restock started")
  }

  def main(args: Array[String]): Unit =
    RestockJob.run()
}
```

</details>

```bash
sbt "runMain instrumentaprogramwithlogstracesandmetrics.Concept1FirstLogExample"
```

<details>
<summary>2. Add structured fields</summary>

```scala title="telemetry-examples/src/main/scala/instrumentaprogramwithlogstracesandmetrics/Concept2StructuredFieldsExample.scala" showLineNumbers
package instrumentaprogramwithlogstracesandmetrics

/**
 * Instrument a Program with Logs, Traces and Metrics: Add structured fields.
 *
 * Attaches `key -> value` fields to log lines and tags a whole unit of work with `log.annotated`.
 *
 * Run with: sbt "runMain instrumentaprogramwithlogstracesandmetrics.Concept2StructuredFieldsExample"
 */
object Concept2StructuredFieldsExample {
  object RestockJob {
    val stock = List("Zionomicon" -> 12, "Programming in Scala" -> 5, "Functional Programming in Scala" -> 8)

    def run(): Unit =
      log.annotated("job.id" -> "restock-2026-10-21") {
        log.info("Nightly restock started")
        stock.foreach { case (title, copies) =>
          log.info("Restocked a title", "title" -> title, "copies" -> copies)
        }
      }
  }

  def main(args: Array[String]): Unit =
    RestockJob.run()
}
```

</details>

```bash
sbt "runMain instrumentaprogramwithlogstracesandmetrics.Concept2StructuredFieldsExample"
```

<details>
<summary>3. Switch to JSON logs on standard error</summary>

```scala title="telemetry-examples/src/main/scala/instrumentaprogramwithlogstracesandmetrics/Concept3JsonLogsExample.scala" showLineNumbers
package instrumentaprogramwithlogstracesandmetrics

/**
 * Instrument a Program with Logs, Traces and Metrics: Switch to JSON logs on standard error.
 *
 * Installs a fresh logger and adds a JSON output on standard error with `log.install` and `log.writer`.
 *
 * Run with: sbt "runMain instrumentaprogramwithlogstracesandmetrics.Concept3JsonLogsExample"
 */
object Concept3JsonLogsExample {
  object RestockJob {
    def run(): Unit =
      log.info("Nightly restock started", "titles" -> 3)
  }

  def main(args: Array[String]): Unit = {
    log.install(LoggerProvider.builder.build().get("bookstore.restock"))
    log.writer(JsonLogFormatter, StderrWriter)

    RestockJob.run()
  }
}
```

</details>

```bash
sbt "runMain instrumentaprogramwithlogstracesandmetrics.Concept3JsonLogsExample"
```

<details>
<summary>4. Trace the job with spans</summary>

```scala title="telemetry-examples/src/main/scala/instrumentaprogramwithlogstracesandmetrics/Concept4SpansExample.scala" showLineNumbers
package instrumentaprogramwithlogstracesandmetrics

/**
 * Instrument a Program with Logs, Traces and Metrics: Trace the job with spans.
 *
 * Wraps the restock job in a root span with one child span per step, then prints the collected spans.
 *
 * Run with: sbt "runMain instrumentaprogramwithlogstracesandmetrics.Concept4SpansExample"
 */
object Concept4SpansExample {
  object RestockJob {
    val stock = List("Zionomicon" -> 12, "Programming in Scala" -> 5, "Functional Programming in Scala" -> 8)

    def run(): Unit =
      trace.span("restock") { _ =>
        trace.span("load-inventory") { _ =>
          log.info("Inventory loaded", "titles" -> stock.size)
        }
        trace.span("restock-titles") { span =>
          span.setAttribute("copies", stock.map(_._2).sum.toLong)
        }
        trace.span("write-report") { span =>
          log.info("Report written")
          span.setStatus(SpanStatus.Ok)
        }
      }
  }

  def printSpans(): Unit = {
    val spans  = trace.collectedSpans
    val nameOf = spans.map(s => s.spanContext.spanId -> s.name).toMap

    spans.foreach { s =>
      val parent = nameOf.getOrElse(s.parentSpanContext.spanId, "none")
      val timed  = s.endTimeNanos >= s.startTimeNanos
      println(s"${s.name}: parent=$parent, status=${s.status}, timed=$timed, attributes=${s.attributes.toMap}")
    }
  }

  def main(args: Array[String]): Unit = {
    log.install(LoggerProvider.builder.build().get("bookstore.restock"))
    log.writer(JsonLogFormatter, StderrWriter)

    RestockJob.run()

    printSpans()
  }
}
```

</details>

```bash
sbt "runMain instrumentaprogramwithlogstracesandmetrics.Concept4SpansExample"
```

<details>
<summary>5. Count restocked books</summary>

```scala title="telemetry-examples/src/main/scala/instrumentaprogramwithlogstracesandmetrics/Concept5CounterExample.scala" showLineNumbers
package instrumentaprogramwithlogstracesandmetrics

/**
 * Instrument a Program with Logs, Traces and Metrics: Count restocked books.
 *
 * Counts restocked copies per title with a counter and reads the totals back.
 *
 * Run with: sbt "runMain instrumentaprogramwithlogstracesandmetrics.Concept5CounterExample"
 */
object Concept5CounterExample {
  object RestockJob {
    val stock = List("Zionomicon" -> 12, "Programming in Scala" -> 5, "Functional Programming in Scala" -> 8)

    val booksRestocked: Counter = metric.counter("bookstore.books.restocked")

    def run(): Unit =
      trace.span("restock") { _ =>
        trace.span("load-inventory") { _ =>
          log.info("Inventory loaded", "titles" -> stock.size)
        }
        trace.span("restock-titles") { _ =>
          stock.foreach { case (title, copies) =>
            booksRestocked.add(copies.toLong, "title" -> title)
          }
        }
        trace.span("write-report") { span =>
          log.info("Report written")
          span.setStatus(SpanStatus.Ok)
        }
      }
  }

  def main(args: Array[String]): Unit = {
    log.install(LoggerProvider.builder.build().get("bookstore.restock"))
    log.writer(JsonLogFormatter, StderrWriter)

    RestockJob.run()

    val titleKey = AttributeKey.string("title")

    RestockJob.booksRestocked.collect() match {
      case MetricData.SumData(points) =>
        points.sortBy(_.attributes.get(titleKey)).foreach { p =>
          println(s"${p.attributes.get(titleKey).getOrElse("?")}: ${p.value}")
        }
      case other => println(s"unexpected: $other")
    }
  }
}
```

</details>

```bash
sbt "runMain instrumentaprogramwithlogstracesandmetrics.Concept5CounterExample"
```

<details>
<summary>6. Time each step</summary>

```scala title="telemetry-examples/src/main/scala/instrumentaprogramwithlogstracesandmetrics/Concept6HistogramExample.scala" showLineNumbers
package instrumentaprogramwithlogstracesandmetrics

/**
 * Instrument a Program with Logs, Traces and Metrics: Time each step.
 *
 * Records each step's duration in a histogram and reads the count and bucket numbers back.
 *
 * Run with: sbt "runMain instrumentaprogramwithlogstracesandmetrics.Concept6HistogramExample"
 */
object Concept6HistogramExample {
  object RestockJob {
    val stock = List("Zionomicon" -> 12, "Programming in Scala" -> 5, "Functional Programming in Scala" -> 8)

    val booksRestocked: Counter = metric.counter("bookstore.books.restocked")
    val stepDuration: Histogram = metric.histogram("bookstore.restock.step.duration.ms")

    def step[A](name: String)(body: Span => A): A =
      trace.span(name) { span =>
        val start = System.nanoTime()
        try body(span)
        finally stepDuration.record((System.nanoTime() - start) / 1e6, "step" -> name)
      }

    def run(): Unit =
      trace.span("restock") { _ =>
        step("load-inventory") { _ =>
          log.info("Inventory loaded", "titles" -> stock.size)
        }
        step("restock-titles") { _ =>
          stock.foreach { case (title, copies) =>
            booksRestocked.add(copies.toLong, "title" -> title)
          }
        }
        step("write-report") { span =>
          log.info("Report written")
          span.setStatus(SpanStatus.Ok)
        }
      }
  }

  def main(args: Array[String]): Unit = {
    log.install(LoggerProvider.builder.build().get("bookstore.restock"))
    log.writer(JsonLogFormatter, StderrWriter)

    RestockJob.run()
    RestockJob.run()

    val stepKey = AttributeKey.string("step")

    RestockJob.stepDuration.collect() match {
      case MetricData.HistogramData(points) =>
        points.sortBy(_.attributes.get(stepKey)).foreach { p =>
          println(s"${p.attributes.get(stepKey).getOrElse("?")}: count=${p.count}, buckets=${p.boundaries.length + 1}")
        }
      case other => println(s"unexpected: $other")
    }
  }
}
```

</details>

```bash
sbt "runMain instrumentaprogramwithlogstracesandmetrics.Concept6HistogramExample"
```

<details>
<summary>7. Test the instrumentation</summary>

```scala title="telemetry-examples/src/main/scala/instrumentaprogramwithlogstracesandmetrics/Concept7TestSpansExample.scala" showLineNumbers
package instrumentaprogramwithlogstracesandmetrics

/**
 * Instrument a Program with Logs, Traces and Metrics: Test the instrumentation.
 *
 * Runs the job and asserts on the shape of the collected span tree.
 *
 * Run with: sbt "runMain instrumentaprogramwithlogstracesandmetrics.Concept7TestSpansExample"
 */
object Concept7TestSpansExample {
  object RestockJob {
    val stock = List("Zionomicon" -> 12, "Programming in Scala" -> 5, "Functional Programming in Scala" -> 8)

    val booksRestocked: Counter = metric.counter("bookstore.books.restocked")
    val stepDuration: Histogram = metric.histogram("bookstore.restock.step.duration.ms")

    def step[A](name: String)(body: Span => A): A =
      trace.span(name) { span =>
        val start = System.nanoTime()
        try body(span)
        finally stepDuration.record((System.nanoTime() - start) / 1e6, "step" -> name)
      }

    def run(): Unit =
      trace.span("restock") { _ =>
        step("load-inventory") { _ =>
          log.info("Inventory loaded", "titles" -> stock.size)
        }
        step("restock-titles") { _ =>
          stock.foreach { case (title, copies) =>
            booksRestocked.add(copies.toLong, "title" -> title)
          }
        }
        step("write-report") { span =>
          log.info("Report written")
          span.setStatus(SpanStatus.Ok)
        }
      }
  }

  def spanTree(): List[(String, String)] = {
    val spans  = trace.collectedSpans
    val nameOf = spans.map(s => s.spanContext.spanId -> s.name).toMap
    spans.map(s => s.name -> nameOf.getOrElse(s.parentSpanContext.spanId, "none"))
  }

  val expected = List(
    "load-inventory" -> "restock",
    "restock-titles" -> "restock",
    "write-report"   -> "restock",
    "restock"        -> "none"
  )

  def main(args: Array[String]): Unit = {
    log.install(LoggerProvider.builder.build().get("bookstore.restock"))
    log.writer(JsonLogFormatter, StderrWriter)

    trace.clearSpans()
    RestockJob.run()

    assert(spanTree() == expected, s"unexpected spans: ${spanTree()}")
    println(s"Instrumentation check passed: ${expected.size} spans")
  }
}
```

</details>

```bash
sbt "runMain instrumentaprogramwithlogstracesandmetrics.Concept7TestSpansExample"
```

To run the complete example:

```bash
sbt "runMain instrumentaprogramwithlogstracesandmetrics.CompleteExample"
```

To check that every example compiles without running any of them:

```bash
sbt compile
```

## 12. What You've Learned

In this tutorial, you learned:

- How the global `log`, `trace` and `metric` objects record telemetry with zero setup
- How to attach structured fields with `key -> value` pairs and tag a whole unit of work with `log.annotated`
- How `log.install` and `log.writer` switch the output to JSON on standard error
- How `trace.span` measures each step, how spans nest into a tree, and how log lines written inside a span carry its trace and span ids
- How a counter keeps a total per attribute, and how a histogram summarizes durations into buckets
- How to read spans back with `trace.collectedSpans`, reset them with `trace.clearSpans`, and assert on the shape of a trace

You now have a solid foundation in recording and inspecting telemetry. To ship it to a backend such as an OpenTelemetry Collector, you write a small exporter of your own: the module's built-in OTLP exporters aren't public yet, but the pieces they are built from are. [Building an Exporter](../reference/telemetry/otel/custom-exporter.md) shows how to encode spans, logs and metrics with `OtlpJsonEncoder` and send them with `HttpSender`.

## 13. Where to Go Next

- **Want to know how it works inside?** The [Telemetry Guide](./telemetry-guide.md) is a deep dive into the module's architecture: how the log macros keep disabled levels nearly free, how sampling decides which spans are recorded, how span context propagates, and what is safe across threads.
- **Want to dive deeper into the API?** Start at the [Telemetry reference](../reference/telemetry/index.md), then read the pages for [logging](../reference/telemetry/logging/index.md), [tracing](../reference/telemetry/tracing/index.md) and [metrics](../reference/telemetry/metrics/index.md).
- **Need labels bound up front?** The [labeled instruments](../reference/telemetry/metrics/labeled-instruments.md) page shows counters and histograms that take attribute values by position.

---

## How to Issue and Verify JWTs

In this guide, we will issue a JSON Web Token (JWT) for a reader of a Scala bookstore and verify it the way an API would. By the end, you will have a small program that signs a token with `Jwt.sign`, checks it with `Jwt.decode`, and turns every kind of bad token into a typed `JwtError`.

The program plays both sides in memory, with no HTTP server and a fixed clock, so that every result below is the same on every run:

- the **bookstore auth service** issues tokens, and
- the **catalog API** verifies them before showing a reader their purchased books.

We start with a shared secret (HMAC), then make the token fail in every way we can think of, and end with the realistic setup: an RSA or EdDSA key pair, where only the auth service can sign.

## The Problem

A service that receives a token has to decide, from the string alone, whether to trust it. It must check that the signature is valid, that the token was signed with the algorithm it expects, that it was issued by the right party for the right audience, and that it has not expired. It must also refuse tokens that are built to waste its memory or CPU.

If you write this by hand, it is easy to get one of these checks wrong, and each mistake is a security hole. Here is what a first attempt with only the JDK often looks like:

```scala

def handRolledVerify(token: String, secret: Array[Byte]): Option[String] = {
  val parts = token.split('.')
  if (parts.length != 3) None
  else {
    val mac = Mac.getInstance("HmacSHA256")
    mac.init(new SecretKeySpec(secret, "HmacSHA256"))
    val signingInput = s"${parts(0)}.${parts(1)}".getBytes(UTF_8)
    val expected     = Base64.getUrlEncoder.withoutPadding.encodeToString(mac.doFinal(signingInput))
    if (expected == parts(2)) Some(new String(Base64.getUrlDecoder.decode(parts(1)), UTF_8))
    else None
  }
}
```

This code compiles and even works for a valid token, but it never reads the header's `alg`, compares signatures with a non-constant-time `==`, accepts tokens of any size, and returns raw JSON that still has to be parsed and checked for `exp`, `iss` and `aud`. Supporting RSA or ECDSA keys would add key parsing and signature format conversion on top.

The ZIO Blocks `jwt` module does these checks for us, in a fixed order, and returns a `JwtError` value that says exactly what was wrong.

## Prerequisites

Add the module to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-jwt" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-jwt" % "0.0.56"
```

The module depends only on ZIO Blocks `chunk`. Its API is synchronous: no effect type, no I/O. You should know what a JWT is at a high level: three Base64URL segments, `header.payload.signature`. The [JWT reference](../reference/jwt.md) covers the full API.

Throughout the guide we use these imports, a fixed clock and a sample secret:

```scala

// 2026-01-01T00:00:00Z. A fixed "now" keeps every result in this guide reproducible.
val now: Long = 1767225600L

// Sample data only. A real HS256 secret is at least 32 random bytes, loaded from a secret store.
val secret: Array[Byte] = "sample-bookstore-secret-32-bytes".getBytes(UTF_8)
```

We also add a tiny helper that prints the outcome of a verification. It is used for every check below:

```scala
def report(result: Either[JwtError, JwtClaims]): Unit = result match {
  case Right(claims) => println(s"accepted: sub=${claims.sub.getOrElse("-")}")
  case Left(error)   => println(s"rejected: ${error.getClass.getSimpleName}: ${error.message}")
}
```

## Issuing a Token

The auth service describes the reader in a `JwtClaims` value. The registered claims have their own fields: `iss` (who issued the token), `sub` (who it is about), `aud` (who it is for), and `iat` and `exp` (issued-at and expiry, in seconds since the epoch). Anything else goes into `extra` as a `JwtValue`. Here we record the reader's purchased titles:

```scala
val claims = JwtClaims(
  iss = Some("bookstore-auth"),
  sub = Some("reader-1024"),
  aud = Some(JwtAudience.single("catalog-api")),
  iat = Some(now),
  exp = Some(now + 3600),
  extra = Map(
    "titles" -> JwtValue.Arr(
      JwtValue.Str("Zionomicon"),
      JwtValue.Str("Programming in Scala"),
      JwtValue.Str("Functional Programming in Scala")
    )
  )
)

val token: String = Jwt.sign(claims, secret, Algorithm.HS256) match {
  case Right(signed) => signed
  case Left(error)   => throw error
}
```

`Jwt.sign` returns `Either[JwtError, String]`. It fails, for example, when the key is too short for the algorithm. `JwtError` extends `Throwable`, so we can throw it at startup if signing is impossible. Here is the token:

```scala
println(token)
// eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJib29rc3RvcmUtYXV0aCIsInN1YiI6InJlYWRlci0xMDI0IiwiYXVkIjoiY2F0YWxvZy1hcGkiLCJleHAiOjE3NjcyMjkyMDAsImlhdCI6MTc2NzIyNTYwMCwidGl0bGVzIjpbIlppb25vbWljb24iLCJQcm9ncmFtbWluZyBpbiBTY2FsYSIsIkZ1bmN0aW9uYWwgUHJvZ3JhbW1pbmcgaW4gU2NhbGEiXX0.FIpruOfYryBN6g8dSziS4htkWaP5q1Ikq-v6TYzOkgE
```

The first two segments are only Base64URL-encoded JSON. `Base64Url.decode` turns them back into text:

```scala
val segments = token.split('.')

def decodeSegment(segment: String): String =
  Base64Url.decode(segment).map(bytes => new String(bytes, UTF_8)).fold(_.message, identity)
```

```scala
println(decodeSegment(segments(0)))
// {"alg":"HS256","typ":"JWT"}
println(decodeSegment(segments(1)))
// {"iss":"bookstore-auth","sub":"reader-1024","aud":"catalog-api","exp":1767229200,"iat":1767225600,"titles":["Zionomicon","Programming in Scala","Functional Programming in Scala"]}
```

:::warning
A JWT is **signed, not encrypted**. Anyone who holds the token can read every claim in it. Never put passwords, card numbers or other secrets in claims.
:::

## Verifying a Token

The catalog API verifies the token with the same secret and algorithm. `JwtDecodeOptions` says what else to check. `issuer` and `expectedAudience` must match the token's `iss` and `aud`. `nowSeconds` replaces the system clock, which we do here so that the expiry check gives the same answer on every run:

```scala
val catalogOptions = JwtDecodeOptions(
  issuer = Some("bookstore-auth"),
  expectedAudience = Some("catalog-api"),
  nowSeconds = Some(now + 60)
)

val verified: Either[JwtError, JwtClaims] =
  Jwt.decode(token, secret, Algorithm.HS256, catalogOptions)
```

On success we get back the claims, and we can read the custom `titles` claim from `extra`:

```scala
println(verified)
// Right(JwtClaims(Some(bookstore-auth),Some(reader-1024),Some(Single(catalog-api)),Some(1767229200),None,Some(1767225600),None,Map(titles -> Arr(Chunk(Str(Zionomicon),Str(Programming in Scala),Str(Functional Programming in Scala))))))
println(verified.map(_.extra.get("titles")))
// Right(Some(Arr(Chunk(Str(Zionomicon),Str(Programming in Scala),Str(Functional Programming in Scala)))))
```

The algorithm argument is the one the **verifier** expects, not the one the token claims. The next sections show why that matters.

:::note
In production, leave `nowSeconds` unset (it defaults to `None`). The check then uses `System.currentTimeMillis() / 1000`.
:::

## Reading the Token From an Authorization Header

Over HTTP, a client usually sends the token in an `Authorization` header with the `Bearer` scheme. The module does not deal with headers, so the job is a string operation: take the header value, check the scheme, and pass what follows to `Jwt.decode`. The scheme name is case-insensitive, so we compare it that way:

```scala
def bearerToken(headerValue: String): Option[String] = {
  val scheme = "Bearer "
  if (headerValue.regionMatches(true, 0, scheme, 0, scheme.length))
    Some(headerValue.substring(scheme.length).trim)
  else None
}

val authorization = s"Bearer $token"
```

```scala
println(s"Authorization: $authorization")
// Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJib29rc3RvcmUtYXV0aCIsInN1YiI6InJlYWRlci0xMDI0IiwiYXVkIjoiY2F0YWxvZy1hcGkiLCJleHAiOjE3NjcyMjkyMDAsImlhdCI6MTc2NzIyNTYwMCwidGl0bGVzIjpbIlppb25vbWljb24iLCJQcm9ncmFtbWluZyBpbiBTY2FsYSIsIkZ1bmN0aW9uYWwgUHJvZ3JhbW1pbmcgaW4gU2NhbGEiXX0.FIpruOfYryBN6g8dSziS4htkWaP5q1Ikq-v6TYzOkgE

bearerToken(authorization) match {
  case Some(bearer) => report(Jwt.decode(bearer, secret, Algorithm.HS256, catalogOptions))
  case None         => println("rejected: no bearer token")
}
// accepted: sub=reader-1024
```

Wiring this into a real server's request pipeline is a separate topic. ZIO Blocks has no HTTP server of its own, so the code above is all the module needs from your framework.

## Seeing Every Failure

Every way a token can be wrong is a small edit to a string or to the decode options. We make them one at a time and look at the error each gives.

First, an attacker rewrites the payload to claim another reader's identity and keeps the original signature:

```scala
val forgedPayload = Base64Url.encode(decodeSegment(segments(1)).replace("reader-1024", "reader-0007").getBytes(UTF_8))
val tampered      = s"${segments(0)}.$forgedPayload.${segments(2)}"
```

```scala
report(Jwt.decode(tampered, secret, Algorithm.HS256, catalogOptions))
// rejected: InvalidSignature: JWT signature verification failed for algorithm HS256
```

The signature covers the header and payload, so any change breaks it. The payload is not even parsed: `Jwt.decode` checks the signature before it reads any claim.

A verifier configured with a different secret also rejects the token, with the same error:

```scala
val otherSecret: Array[Byte] = "another-service-secret-32-bytes!".getBytes(UTF_8)
```

```scala
report(Jwt.decode(token, otherSecret, Algorithm.HS256, catalogOptions))
// rejected: InvalidSignature: JWT signature verification failed for algorithm HS256
```

If the verifier expects a different algorithm from the one in the token's header, the token is rejected before any signature is computed:

```scala
report(Jwt.decode(token, secret, Algorithm.HS512, catalogOptions))
// rejected: AlgorithmMismatch: JWT algorithm mismatch: expected HS512 but found HS256
```

A token issued for another service fails the audience check, and a token with no `aud` at all fails it too, once `expectedAudience` is set:

```scala
val noAudienceToken: String = Jwt.sign(claims.copy(aud = None), secret, Algorithm.HS256).fold(e => throw e, t => t)
```

```scala
report(Jwt.decode(token, secret, Algorithm.HS256, catalogOptions.copy(expectedAudience = Some("orders-api"))))
// rejected: InvalidClaim: Invalid JWT claim 'aud': expected orders-api but found catalog-api
report(Jwt.decode(noAudienceToken, secret, Algorithm.HS256, catalogOptions))
// rejected: MissingClaim: Missing JWT claim: aud
```

The issuer check works the same way:

```scala
report(Jwt.decode(token, secret, Algorithm.HS256, catalogOptions.copy(issuer = Some("partner-auth"))))
// rejected: InvalidClaim: Invalid JWT claim 'iss': expected partner-auth but found bookstore-auth
```

`JwtError` is a sealed trait, so a verifier can match on these cases to decide what to log or return. An HTTP API usually answers all of them with `401 Unauthorized` and keeps the detail in its logs.

:::tip
`issuer` and `expectedAudience` are only checked when you set them. With the default `JwtDecodeOptions`, a token issued by anyone you share the key with, for any audience, is accepted. Set both whenever one key is used by more than one service.
:::

## Testing Expiry Without Waiting

Tests that wait an hour for a token to expire are not tests anyone runs. Because `nowSeconds` replaces the clock, we can move "now" instead. Our token expires at `now + 3600`. Verified at `now + 7200`, it is expired:

```scala
report(Jwt.decode(token, secret, Algorithm.HS256, catalogOptions.copy(nowSeconds = Some(now + 7200))))
// rejected: ExpiredToken: JWT token expired at 1767229200 (now: 1767232800)
```

Clocks on different machines drift, so `clockSkewSeconds` gives a grace period. A token counts as expired only when `now - clockSkewSeconds > exp`. With a 60-second skew, the token is still accepted 30 seconds after `exp` and rejected 61 seconds after:

```scala
val expiresAt   = now + 3600
val withSkew    = catalogOptions.copy(clockSkewSeconds = 60L)
```

```scala
report(Jwt.decode(token, secret, Algorithm.HS256, withSkew.copy(nowSeconds = Some(expiresAt + 30))))
// accepted: sub=reader-1024
report(Jwt.decode(token, secret, Algorithm.HS256, withSkew.copy(nowSeconds = Some(expiresAt + 61))))
// rejected: ExpiredToken: JWT token expired at 1767229200 (now: 1767229261)
```

The skew also applies to `nbf` ("not before"). Suppose the bookstore issues a token for a pre-order that unlocks ten minutes from now. Before `nbf` it is not yet valid, and from `nbf` on it is accepted:

```scala
val preOrderToken: String =
  Jwt.sign(claims.copy(nbf = Some(now + 600)), secret, Algorithm.HS256).fold(e => throw e, t => t)
```

```scala
report(Jwt.decode(preOrderToken, secret, Algorithm.HS256, catalogOptions.copy(nowSeconds = Some(now))))
// rejected: NotYetValid: JWT token is not valid before 1767226200 (now: 1767225600)
report(Jwt.decode(preOrderToken, secret, Algorithm.HS256, catalogOptions.copy(nowSeconds = Some(now + 600))))
// accepted: sub=reader-1024
```

Finally, `exp` and `nbf` are only checked when they are present. A token without `exp` never expires:

```scala
val foreverToken: String = Jwt.sign(claims.copy(exp = None), secret, Algorithm.HS256).fold(e => throw e, t => t)
```

```scala
report(Jwt.decode(foreverToken, secret, Algorithm.HS256, catalogOptions.copy(nowSeconds = Some(now + 10L * 365 * 24 * 3600))))
// accepted: sub=reader-1024
```

:::warning
The module does not require an `exp` claim. If your service must only accept tokens that expire, check `claims.exp.isDefined` yourself after a successful `Jwt.decode`.
:::

## Rejecting Hostile Tokens

A public endpoint receives tokens from anyone, including tokens built to exhaust memory or to trick the verifier. `Jwt.decode` checks sizes before it parses, and parses the header before it verifies the signature, so these checks run even when the attacker has no key. The limits live in `JwtLimits`, and the defaults are `maxTokenChars = 8192`, `maxSegmentChars = 4096`, `maxJsonChars = 8192`, `maxDepth = 32`, `maxFields = 512` and `maxArrayElements = 512`.

A token over `maxTokenChars` is rejected before anything else happens:

```scala
report(Jwt.decode("a" * 10000, secret, Algorithm.HS256, catalogOptions))
// rejected: TokenTooLarge: JWT token too large: 10000 chars exceeds limit 8192
```

A header that nests arrays deeper than `maxDepth` is rejected while the header is parsed, before the signature is checked:

```scala
val deepHeader = """{"alg":"HS256","x":""" + ("[" * 64) + ("]" * 64) + "}"
val deepToken  = s"${Base64Url.encode(deepHeader.getBytes(UTF_8))}.${segments(1)}.${segments(2)}"
```

```scala
report(Jwt.decode(deepToken, secret, Algorithm.HS256, catalogOptions))
// rejected: TooDeep: JWT JSON too deep: depth 33 exceeds limit 32
```

The classic `alg: none` attack sends an unsigned token and hopes the verifier skips the signature check. Both of its forms are rejected, with an empty signature segment and with any other signature segment:

```scala
val noneHeader      = Base64Url.encode("""{"alg":"none","typ":"JWT"}""".getBytes(UTF_8))
val unsignedToken   = s"$noneHeader.${segments(1)}."
val noneWithGarbage = s"$noneHeader.${segments(1)}.${segments(2)}"
```

```scala
report(Jwt.decode(unsignedToken, secret, Algorithm.HS256, catalogOptions))
// rejected: InvalidToken: Invalid JWT token: JWT signature segment must not be empty
report(Jwt.decode(noneWithGarbage, secret, Algorithm.HS256, catalogOptions))
// rejected: UnsupportedAlgorithm: Unsupported JWT algorithm: none
```

`none` is not one of the module's `Algorithm` values, so no backend can accept it.

For a public endpoint, the defaults are generous. Our token is a few hundred characters with a payload two levels deep, so we can tighten every limit and still accept it:

```scala
val publicEndpointLimits = JwtLimits(
  maxTokenChars = 2048,
  maxSegmentChars = 1024,
  maxJsonChars = 1024,
  maxDepth = 4,
  maxFields = 32,
  maxArrayElements = 32
)

val publicOptions = catalogOptions.copy(limits = publicEndpointLimits)
```

```scala
report(Jwt.decode(token, secret, Algorithm.HS256, publicOptions))
// accepted: sub=reader-1024
report(Jwt.decode("a" * 4000, secret, Algorithm.HS256, publicOptions))
// rejected: TokenTooLarge: JWT token too large: 4000 chars exceeds limit 2048
```

## Signing With an Asymmetric Key Pair

With HMAC, every service that verifies tokens holds the secret, so every one of them could also issue tokens. The realistic setup uses a key pair instead: the auth service signs with a **private key** that never leaves it, and the catalog API verifies with the matching **public key**, which can be shared freely.

Asymmetric algorithms need a platform crypto backend. On the JVM, install it once at application startup. HMAC algorithms work without it:

```scala
JvmJwtCryptoBackend.install()
```

We generate a 2048-bit RSA key pair in memory with the JDK's `KeyPairGenerator`. The module takes keys as DER bytes: the private key in **PKCS#8** format and the public key in **X.509** (SubjectPublicKeyInfo) format. That is exactly what the JDK's `getEncoded` returns:

```scala

val rsaGenerator = KeyPairGenerator.getInstance("RSA")
rsaGenerator.initialize(2048)
val authServiceKeys = rsaGenerator.generateKeyPair()

val privateKeyDer: Array[Byte] = authServiceKeys.getPrivate.getEncoded // stays in the auth service
val publicKeyDer: Array[Byte]  = authServiceKeys.getPublic.getEncoded  // shared with the catalog API
```

```scala
println(authServiceKeys.getPrivate.getFormat)
// PKCS#8
println(authServiceKeys.getPublic.getFormat)
// X.509
```

Signing and verifying look the same as before, with `Algorithm.RS256` and a different key on each side:

```scala
val rsaToken: String = Jwt.sign(claims, privateKeyDer, Algorithm.RS256).fold(e => throw e, t => t)

val strangerKeys = rsaGenerator.generateKeyPair()
```

```scala
report(Jwt.decode(rsaToken, publicKeyDer, Algorithm.RS256, catalogOptions))
// accepted: sub=reader-1024
report(Jwt.decode(rsaToken, strangerKeys.getPublic.getEncoded, Algorithm.RS256, catalogOptions))
// rejected: InvalidSignature: JWT signature verification failed for algorithm RS256
```

The second check uses a public key from a different key pair, so the signature does not verify.

The public key is public, which opens the door to **algorithm confusion**: an attacker signs a forged token with HS256, using the public key's bytes as the HMAC secret, and hopes the verifier trusts the token's `alg` header. Because the verifier passes the algorithm it expects, the forged token is rejected before any key is used:

```scala
val confusedToken: String =
  Jwt.sign(claims.copy(sub = Some("reader-0007")), publicKeyDer, Algorithm.HS256).fold(e => throw e, t => t)
```

```scala
report(Jwt.decode(confusedToken, publicKeyDer, Algorithm.RS256, catalogOptions))
// rejected: AlgorithmMismatch: JWT algorithm mismatch: expected RS256 but found HS256
```

The same pattern works for ECDSA (`ES256`, `ES384`, `ES512`) and EdDSA (Ed25519). Only the key generator and the algorithm change:

```scala
val edKeys = KeyPairGenerator.getInstance("Ed25519").generateKeyPair()

val edToken: String = Jwt.sign(claims, edKeys.getPrivate.getEncoded, Algorithm.EdDSA).fold(e => throw e, t => t)
```

```scala
report(Jwt.decode(edToken, edKeys.getPublic.getEncoded, Algorithm.EdDSA, catalogOptions))
// accepted: sub=reader-1024
```

On the JVM, the backend rejects RSA keys shorter than 2048 bits and EC keys on the wrong curve for the algorithm with `InvalidKey`. On Scala.js (Node.js), the backend supports HMAC, RSA and ECDSA but not PS256/384/512 or EdDSA, and it does not check RSA key size or EC curves.

### Loading Keys From PEM Files

Keys usually arrive as PEM files: Base64 text between `-----BEGIN ...-----` and `-----END ...-----` lines. The module takes DER bytes, and PEM is just DER in Base64, so the conversion is two steps: drop the marker lines and Base64-decode the rest.

```scala

def pemToDer(pem: String): Array[Byte] =
  Base64.getMimeDecoder.decode(pem.linesIterator.filterNot(_.startsWith("-----")).mkString)
```

To check it, we write our public key out as PEM, the way it would appear in a file, and read it back:

```scala
val publicKeyPem =
  "-----BEGIN PUBLIC KEY-----\n" +
    Base64.getMimeEncoder(64, "\n".getBytes(UTF_8)).encodeToString(publicKeyDer) +
    "\n-----END PUBLIC KEY-----\n"
```

```scala
report(Jwt.decode(rsaToken, pemToDer(publicKeyPem), Algorithm.RS256, catalogOptions))
// accepted: sub=reader-1024
```

The marker line tells you the format inside, and only two of them match what the module expects:

| PEM marker                       | Format inside                 | Use as is? |
|----------------------------------|-------------------------------|------------|
| `BEGIN PRIVATE KEY`              | PKCS#8                        | Yes, for signing |
| `BEGIN PUBLIC KEY`               | X.509 SubjectPublicKeyInfo    | Yes, for verifying |
| `BEGIN RSA PRIVATE KEY`          | PKCS#1                        | No: convert with `openssl pkcs8 -topk8 -nocrypt -in key.pem` |
| `BEGIN EC PRIVATE KEY`           | SEC1                          | No: convert with `openssl pkcs8 -topk8 -nocrypt -in key.pem` |
| `BEGIN RSA PUBLIC KEY`           | PKCS#1                        | No: convert with `openssl rsa -RSAPublicKey_in -pubout -in key.pem` |
| `BEGIN ENCRYPTED PRIVATE KEY`    | Encrypted PKCS#8              | No: decrypt first |

A key in the wrong format is rejected with `InvalidKey`.

## What the Module Does Not Do

Before you rely on the module, know its limits:

- **No JWK or JWKS.** You cannot point it at an identity provider's key-set URL, and it does not pick a key by the header's `kid`. Fetch the provider's public key yourself, convert it to X.509 DER, and verify with it as shown above.
- **No encryption (JWE).** Tokens are signed only, so their claims are readable by anyone who holds them.
- **No required claims beyond what you configure.** `exp`, `nbf`, `iss` and `aud` are checked only when present or configured, and `typ`, `iat` and `jti` are never checked.
- **A new API.** The module is new and has no binary-compatibility baseline yet, so expect API changes between releases.

`Jwt.decodeWithoutVerification` reads a token's header and claims **without checking its signature**. Use it only to inspect a token, for example while debugging. Its output is whatever the sender wrote, so never use it to make an access decision.

## Putting It Together

This example combines everything from the guide in one program: the auth service issues an RSA-signed token, the catalog API reads it from an `Authorization` header and verifies it with tightened limits and a fixed clock, and every failure from the guide is printed:

```scala title="jwt-examples/src/main/scala/issueandverifyjwts/CompleteExample.scala"
package issueandverifyjwts

/**
 * How to Issue and Verify JWTs: Putting It Together — Complete Example
 *
 * The auth service issues an RSA-signed token, the catalog API reads it from an
 * Authorization header and verifies it with tightened limits and a fixed clock,
 * and every failure from the guide is printed.
 *
 * Run with: cd jwt-examples && sbt "runMain issueandverifyjwts.CompleteExample"
 */
object CompleteExample extends App {

  // 2026-01-01T00:00:00Z. A fixed "now" keeps every result reproducible.
  val now: Long = 1767225600L

  def report(result: Either[JwtError, JwtClaims]): Unit = result match {
    case Right(claims) => println(s"accepted: sub=${claims.sub.getOrElse("-")}")
    case Left(error)   => println(s"rejected: ${error.getClass.getSimpleName}: ${error.message}")
  }

  def signOrThrow(claims: JwtClaims, key: Array[Byte], algorithm: Algorithm): String =
    Jwt.sign(claims, key, algorithm).fold(e => throw e, t => t)

  // Asymmetric algorithms need a platform crypto backend, installed once at startup.
  JvmJwtCryptoBackend.install()

  // The bookstore auth service holds the private key and issues tokens.
  val rsaGenerator = KeyPairGenerator.getInstance("RSA")
  rsaGenerator.initialize(2048)
  val authServiceKeys = rsaGenerator.generateKeyPair()

  val privateKeyDer: Array[Byte] = authServiceKeys.getPrivate.getEncoded // stays in the auth service
  val publicKeyDer: Array[Byte]  = authServiceKeys.getPublic.getEncoded  // shared with the catalog API

  val claims = JwtClaims(
    iss = Some("bookstore-auth"),
    sub = Some("reader-1024"),
    aud = Some(JwtAudience.single("catalog-api")),
    iat = Some(now),
    exp = Some(now + 3600),
    extra = Map(
      "titles" -> JwtValue.Arr(
        JwtValue.Str("Zionomicon"),
        JwtValue.Str("Programming in Scala"),
        JwtValue.Str("Functional Programming in Scala")
      )
    )
  )

  val token: String = signOrThrow(claims, privateKeyDer, Algorithm.RS256)

  // The catalog API reads the token from the Authorization header and verifies it.
  def bearerToken(headerValue: String): Option[String] = {
    val scheme = "Bearer "
    if (headerValue.regionMatches(true, 0, scheme, 0, scheme.length))
      Some(headerValue.substring(scheme.length).trim)
    else None
  }

  val catalogOptions = JwtDecodeOptions(
    issuer = Some("bookstore-auth"),
    expectedAudience = Some("catalog-api"),
    nowSeconds = Some(now + 60),
    limits = JwtLimits(
      maxTokenChars = 2048,
      maxSegmentChars = 1024,
      maxJsonChars = 1024,
      maxDepth = 4,
      maxFields = 32,
      maxArrayElements = 32
    )
  )

  def verify(bearer: String, options: JwtDecodeOptions = catalogOptions): Either[JwtError, JwtClaims] =
    Jwt.decode(bearer, publicKeyDer, Algorithm.RS256, options)

  val authorization = s"Bearer $token"

  println("-- valid token from the Authorization header")
  bearerToken(authorization) match {
    case Some(bearer) =>
      val verified = verify(bearer)
      report(verified)
      println(verified.map(_.extra.get("titles")))
    case None => println("rejected: no bearer token")
  }

  // Every failure from the guide.
  def decodeSegment(segment: String): String =
    Base64Url.decode(segment).map(bytes => new String(bytes, UTF_8)).fold(_.message, identity)

  val segments = token.split('.')

  println("-- tampered payload")
  val forgedPayload = Base64Url.encode(decodeSegment(segments(1)).replace("reader-1024", "reader-0007").getBytes(UTF_8))
  report(verify(s"${segments(0)}.$forgedPayload.${segments(2)}"))

  println("-- signature from a different key pair")
  val strangerKeys = rsaGenerator.generateKeyPair()
  report(Jwt.decode(token, strangerKeys.getPublic.getEncoded, Algorithm.RS256, catalogOptions))

  println("-- verifier expects a different algorithm")
  report(Jwt.decode(token, publicKeyDer, Algorithm.RS512, catalogOptions))

  println("-- algorithm confusion: HS256 signed with the public key bytes")
  report(verify(signOrThrow(claims.copy(sub = Some("reader-0007")), publicKeyDer, Algorithm.HS256)))

  println("-- wrong audience, missing audience, wrong issuer")
  report(verify(token, catalogOptions.copy(expectedAudience = Some("orders-api"))))
  report(verify(signOrThrow(claims.copy(aud = None), privateKeyDer, Algorithm.RS256)))
  report(verify(token, catalogOptions.copy(issuer = Some("partner-auth"))))

  println("-- expired, within clock skew, beyond clock skew")
  report(verify(token, catalogOptions.copy(nowSeconds = Some(now + 7200))))
  val withSkew = catalogOptions.copy(clockSkewSeconds = 60L)
  report(verify(token, withSkew.copy(nowSeconds = Some(now + 3600 + 30))))
  report(verify(token, withSkew.copy(nowSeconds = Some(now + 3600 + 61))))

  println("-- not yet valid, then valid")
  val preOrderToken = signOrThrow(claims.copy(nbf = Some(now + 600)), privateKeyDer, Algorithm.RS256)
  report(verify(preOrderToken, catalogOptions.copy(nowSeconds = Some(now))))
  report(verify(preOrderToken, catalogOptions.copy(nowSeconds = Some(now + 600))))

  println("-- hostile tokens")
  report(verify("a" * 10000))
  val deepHeader = """{"alg":"RS256","x":""" + ("[" * 64) + ("]" * 64) + "}"
  report(verify(s"${Base64Url.encode(deepHeader.getBytes(UTF_8))}.${segments(1)}.${segments(2)}"))
  val noneHeader = Base64Url.encode("""{"alg":"none","typ":"JWT"}""".getBytes(UTF_8))
  report(verify(s"$noneHeader.${segments(1)}."))
  report(verify(s"$noneHeader.${segments(1)}.${segments(2)}"))
}
```

## Running the Examples

All examples in this guide have runnable Scala files in the `jwt-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/jwt-examples
```

### Issue a Token

This example signs the reader's claims with HS256 and decodes the header and payload to show that they are plain JSON.

<details>
  <summary>jwt-examples/src/main/scala/issueandverifyjwts/Step1IssueTokenExample.scala</summary>

```scala title="jwt-examples/src/main/scala/issueandverifyjwts/Step1IssueTokenExample.scala" showLineNumbers
package issueandverifyjwts

/**
 * How to Issue and Verify JWTs: Step 1 — Issuing a Token
 *
 * Signs the reader's claims with HS256 and decodes the header and payload to
 * show that they are plain JSON.
 *
 * Run with: cd jwt-examples && sbt "runMain
 * issueandverifyjwts.Step1IssueTokenExample"
 */
object Step1IssueTokenExample extends App {

  // 2026-01-01T00:00:00Z. A fixed "now" keeps every result reproducible.
  val now: Long = 1767225600L

  // Sample data only. A real HS256 secret is at least 32 random bytes, loaded from a secret store.
  val secret: Array[Byte] = "sample-bookstore-secret-32-bytes".getBytes(UTF_8)

  def report(result: Either[JwtError, JwtClaims]): Unit = result match {
    case Right(claims) => println(s"accepted: sub=${claims.sub.getOrElse("-")}")
    case Left(error)   => println(s"rejected: ${error.getClass.getSimpleName}: ${error.message}")
  }

  val claims = JwtClaims(
    iss = Some("bookstore-auth"),
    sub = Some("reader-1024"),
    aud = Some(JwtAudience.single("catalog-api")),
    iat = Some(now),
    exp = Some(now + 3600),
    extra = Map(
      "titles" -> JwtValue.Arr(
        JwtValue.Str("Zionomicon"),
        JwtValue.Str("Programming in Scala"),
        JwtValue.Str("Functional Programming in Scala")
      )
    )
  )

  val token: String = Jwt.sign(claims, secret, Algorithm.HS256) match {
    case Right(signed) => signed
    case Left(error)   => throw error
  }

  val segments = token.split('.')

  def decodeSegment(segment: String): String =
    Base64Url.decode(segment).map(bytes => new String(bytes, UTF_8)).fold(_.message, identity)

  println(token)
  println(decodeSegment(segments(0)))
  println(decodeSegment(segments(1)))
}
```

</details>

Run it from the `jwt-examples` directory:

```bash
sbt "runMain issueandverifyjwts.Step1IssueTokenExample"
```

### Verify a Token

This example verifies the token with `issuer`, `expectedAudience` and a fixed clock, and reads the custom `titles` claim.

<details>
  <summary>jwt-examples/src/main/scala/issueandverifyjwts/Step2VerifyTokenExample.scala</summary>

```scala title="jwt-examples/src/main/scala/issueandverifyjwts/Step2VerifyTokenExample.scala" showLineNumbers
package issueandverifyjwts

/**
 * How to Issue and Verify JWTs: Step 2 — Verifying a Token
 *
 * Verifies the token with issuer, expectedAudience and a fixed clock, and reads
 * the custom titles claim.
 *
 * Run with: cd jwt-examples && sbt "runMain
 * issueandverifyjwts.Step2VerifyTokenExample"
 */
object Step2VerifyTokenExample extends App {

  // 2026-01-01T00:00:00Z. A fixed "now" keeps every result reproducible.
  val now: Long = 1767225600L

  // Sample data only. A real HS256 secret is at least 32 random bytes, loaded from a secret store.
  val secret: Array[Byte] = "sample-bookstore-secret-32-bytes".getBytes(UTF_8)

  def report(result: Either[JwtError, JwtClaims]): Unit = result match {
    case Right(claims) => println(s"accepted: sub=${claims.sub.getOrElse("-")}")
    case Left(error)   => println(s"rejected: ${error.getClass.getSimpleName}: ${error.message}")
  }

  val claims = JwtClaims(
    iss = Some("bookstore-auth"),
    sub = Some("reader-1024"),
    aud = Some(JwtAudience.single("catalog-api")),
    iat = Some(now),
    exp = Some(now + 3600),
    extra = Map(
      "titles" -> JwtValue.Arr(
        JwtValue.Str("Zionomicon"),
        JwtValue.Str("Programming in Scala"),
        JwtValue.Str("Functional Programming in Scala")
      )
    )
  )

  val token: String = Jwt.sign(claims, secret, Algorithm.HS256) match {
    case Right(signed) => signed
    case Left(error)   => throw error
  }

  val catalogOptions = JwtDecodeOptions(
    issuer = Some("bookstore-auth"),
    expectedAudience = Some("catalog-api"),
    nowSeconds = Some(now + 60)
  )

  val verified: Either[JwtError, JwtClaims] =
    Jwt.decode(token, secret, Algorithm.HS256, catalogOptions)

  println(verified)
  println(verified.map(_.extra.get("titles")))
}
```

</details>

Run it from the `jwt-examples` directory:

```bash
sbt "runMain issueandverifyjwts.Step2VerifyTokenExample"
```

### Read the Token From an Authorization Header

This example extracts the token from a `Bearer` header value and verifies it.

<details>
  <summary>jwt-examples/src/main/scala/issueandverifyjwts/Step3BearerHeaderExample.scala</summary>

```scala title="jwt-examples/src/main/scala/issueandverifyjwts/Step3BearerHeaderExample.scala" showLineNumbers
package issueandverifyjwts

/**
 * How to Issue and Verify JWTs: Step 3 — Reading the Token From an
 * Authorization Header
 *
 * Extracts the token from a Bearer header value and verifies it.
 *
 * Run with: cd jwt-examples && sbt "runMain
 * issueandverifyjwts.Step3BearerHeaderExample"
 */
object Step3BearerHeaderExample extends App {

  // 2026-01-01T00:00:00Z. A fixed "now" keeps every result reproducible.
  val now: Long = 1767225600L

  // Sample data only. A real HS256 secret is at least 32 random bytes, loaded from a secret store.
  val secret: Array[Byte] = "sample-bookstore-secret-32-bytes".getBytes(UTF_8)

  def report(result: Either[JwtError, JwtClaims]): Unit = result match {
    case Right(claims) => println(s"accepted: sub=${claims.sub.getOrElse("-")}")
    case Left(error)   => println(s"rejected: ${error.getClass.getSimpleName}: ${error.message}")
  }

  val claims = JwtClaims(
    iss = Some("bookstore-auth"),
    sub = Some("reader-1024"),
    aud = Some(JwtAudience.single("catalog-api")),
    iat = Some(now),
    exp = Some(now + 3600),
    extra = Map(
      "titles" -> JwtValue.Arr(
        JwtValue.Str("Zionomicon"),
        JwtValue.Str("Programming in Scala"),
        JwtValue.Str("Functional Programming in Scala")
      )
    )
  )

  val token: String = Jwt.sign(claims, secret, Algorithm.HS256) match {
    case Right(signed) => signed
    case Left(error)   => throw error
  }

  val catalogOptions = JwtDecodeOptions(
    issuer = Some("bookstore-auth"),
    expectedAudience = Some("catalog-api"),
    nowSeconds = Some(now + 60)
  )

  def bearerToken(headerValue: String): Option[String] = {
    val scheme = "Bearer "
    if (headerValue.regionMatches(true, 0, scheme, 0, scheme.length))
      Some(headerValue.substring(scheme.length).trim)
    else None
  }

  val authorization = s"Bearer $token"

  println(s"Authorization: $authorization")

  bearerToken(authorization) match {
    case Some(bearer) => report(Jwt.decode(bearer, secret, Algorithm.HS256, catalogOptions))
    case None         => println("rejected: no bearer token")
  }
}
```

</details>

Run it from the `jwt-examples` directory:

```bash
sbt "runMain issueandverifyjwts.Step3BearerHeaderExample"
```

### See Every Failure

This example tampers with the payload and verifies with the wrong secret, algorithm, audience and issuer.

<details>
  <summary>jwt-examples/src/main/scala/issueandverifyjwts/Step4FailuresExample.scala</summary>

```scala title="jwt-examples/src/main/scala/issueandverifyjwts/Step4FailuresExample.scala" showLineNumbers
package issueandverifyjwts

/**
 * How to Issue and Verify JWTs: Step 4 — Seeing Every Failure
 *
 * Tampers with the payload and verifies with the wrong secret, algorithm,
 * audience and issuer.
 *
 * Run with: cd jwt-examples && sbt "runMain
 * issueandverifyjwts.Step4FailuresExample"
 */
object Step4FailuresExample extends App {

  // 2026-01-01T00:00:00Z. A fixed "now" keeps every result reproducible.
  val now: Long = 1767225600L

  // Sample data only. A real HS256 secret is at least 32 random bytes, loaded from a secret store.
  val secret: Array[Byte] = "sample-bookstore-secret-32-bytes".getBytes(UTF_8)

  def report(result: Either[JwtError, JwtClaims]): Unit = result match {
    case Right(claims) => println(s"accepted: sub=${claims.sub.getOrElse("-")}")
    case Left(error)   => println(s"rejected: ${error.getClass.getSimpleName}: ${error.message}")
  }

  val claims = JwtClaims(
    iss = Some("bookstore-auth"),
    sub = Some("reader-1024"),
    aud = Some(JwtAudience.single("catalog-api")),
    iat = Some(now),
    exp = Some(now + 3600),
    extra = Map(
      "titles" -> JwtValue.Arr(
        JwtValue.Str("Zionomicon"),
        JwtValue.Str("Programming in Scala"),
        JwtValue.Str("Functional Programming in Scala")
      )
    )
  )

  val token: String = Jwt.sign(claims, secret, Algorithm.HS256) match {
    case Right(signed) => signed
    case Left(error)   => throw error
  }

  val segments = token.split('.')

  def decodeSegment(segment: String): String =
    Base64Url.decode(segment).map(bytes => new String(bytes, UTF_8)).fold(_.message, identity)

  val catalogOptions = JwtDecodeOptions(
    issuer = Some("bookstore-auth"),
    expectedAudience = Some("catalog-api"),
    nowSeconds = Some(now + 60)
  )

  val forgedPayload = Base64Url.encode(decodeSegment(segments(1)).replace("reader-1024", "reader-0007").getBytes(UTF_8))
  val tampered      = s"${segments(0)}.$forgedPayload.${segments(2)}"

  report(Jwt.decode(tampered, secret, Algorithm.HS256, catalogOptions))

  val otherSecret: Array[Byte] = "another-service-secret-32-bytes!".getBytes(UTF_8)

  report(Jwt.decode(token, otherSecret, Algorithm.HS256, catalogOptions))

  report(Jwt.decode(token, secret, Algorithm.HS512, catalogOptions))

  val noAudienceToken: String = Jwt.sign(claims.copy(aud = None), secret, Algorithm.HS256).fold(e => throw e, t => t)

  report(Jwt.decode(token, secret, Algorithm.HS256, catalogOptions.copy(expectedAudience = Some("orders-api"))))
  report(Jwt.decode(noAudienceToken, secret, Algorithm.HS256, catalogOptions))

  report(Jwt.decode(token, secret, Algorithm.HS256, catalogOptions.copy(issuer = Some("partner-auth"))))
}
```

</details>

Run it from the `jwt-examples` directory:

```bash
sbt "runMain issueandverifyjwts.Step4FailuresExample"
```

### Test Expiry Without Waiting

This example moves the clock to show `ExpiredToken`, clock skew, `NotYetValid`, and a token without `exp`.

<details>
  <summary>jwt-examples/src/main/scala/issueandverifyjwts/Step5ExpiryExample.scala</summary>

```scala title="jwt-examples/src/main/scala/issueandverifyjwts/Step5ExpiryExample.scala" showLineNumbers
package issueandverifyjwts

/**
 * How to Issue and Verify JWTs: Step 5 — Testing Expiry Without Waiting
 *
 * Moves the clock to show ExpiredToken, clock skew, NotYetValid, and a token
 * without exp.
 *
 * Run with: cd jwt-examples && sbt "runMain
 * issueandverifyjwts.Step5ExpiryExample"
 */
object Step5ExpiryExample extends App {

  // 2026-01-01T00:00:00Z. A fixed "now" keeps every result reproducible.
  val now: Long = 1767225600L

  // Sample data only. A real HS256 secret is at least 32 random bytes, loaded from a secret store.
  val secret: Array[Byte] = "sample-bookstore-secret-32-bytes".getBytes(UTF_8)

  def report(result: Either[JwtError, JwtClaims]): Unit = result match {
    case Right(claims) => println(s"accepted: sub=${claims.sub.getOrElse("-")}")
    case Left(error)   => println(s"rejected: ${error.getClass.getSimpleName}: ${error.message}")
  }

  val claims = JwtClaims(
    iss = Some("bookstore-auth"),
    sub = Some("reader-1024"),
    aud = Some(JwtAudience.single("catalog-api")),
    iat = Some(now),
    exp = Some(now + 3600),
    extra = Map(
      "titles" -> JwtValue.Arr(
        JwtValue.Str("Zionomicon"),
        JwtValue.Str("Programming in Scala"),
        JwtValue.Str("Functional Programming in Scala")
      )
    )
  )

  val token: String = Jwt.sign(claims, secret, Algorithm.HS256) match {
    case Right(signed) => signed
    case Left(error)   => throw error
  }

  val catalogOptions = JwtDecodeOptions(
    issuer = Some("bookstore-auth"),
    expectedAudience = Some("catalog-api"),
    nowSeconds = Some(now + 60)
  )

  report(Jwt.decode(token, secret, Algorithm.HS256, catalogOptions.copy(nowSeconds = Some(now + 7200))))

  val expiresAt = now + 3600
  val withSkew  = catalogOptions.copy(clockSkewSeconds = 60L)

  report(Jwt.decode(token, secret, Algorithm.HS256, withSkew.copy(nowSeconds = Some(expiresAt + 30))))
  report(Jwt.decode(token, secret, Algorithm.HS256, withSkew.copy(nowSeconds = Some(expiresAt + 61))))

  val preOrderToken: String =
    Jwt.sign(claims.copy(nbf = Some(now + 600)), secret, Algorithm.HS256).fold(e => throw e, t => t)

  report(Jwt.decode(preOrderToken, secret, Algorithm.HS256, catalogOptions.copy(nowSeconds = Some(now))))
  report(Jwt.decode(preOrderToken, secret, Algorithm.HS256, catalogOptions.copy(nowSeconds = Some(now + 600))))

  val foreverToken: String = Jwt.sign(claims.copy(exp = None), secret, Algorithm.HS256).fold(e => throw e, t => t)

  report(
    Jwt.decode(
      foreverToken,
      secret,
      Algorithm.HS256,
      catalogOptions.copy(nowSeconds = Some(now + 10L * 365 * 24 * 3600))
    )
  )
}
```

</details>

Run it from the `jwt-examples` directory:

```bash
sbt "runMain issueandverifyjwts.Step5ExpiryExample"
```

### Reject Hostile Tokens

This example sends an oversized token, a deeply nested header and `alg: none` tokens, and tightens `JwtLimits`.

<details>
  <summary>jwt-examples/src/main/scala/issueandverifyjwts/Step6HostileTokensExample.scala</summary>

```scala title="jwt-examples/src/main/scala/issueandverifyjwts/Step6HostileTokensExample.scala" showLineNumbers
package issueandverifyjwts

/**
 * How to Issue and Verify JWTs: Step 6 — Rejecting Hostile Tokens
 *
 * Sends an oversized token, a deeply nested header and alg none tokens, and
 * tightens JwtLimits.
 *
 * Run with: cd jwt-examples && sbt "runMain
 * issueandverifyjwts.Step6HostileTokensExample"
 */
object Step6HostileTokensExample extends App {

  // 2026-01-01T00:00:00Z. A fixed "now" keeps every result reproducible.
  val now: Long = 1767225600L

  // Sample data only. A real HS256 secret is at least 32 random bytes, loaded from a secret store.
  val secret: Array[Byte] = "sample-bookstore-secret-32-bytes".getBytes(UTF_8)

  def report(result: Either[JwtError, JwtClaims]): Unit = result match {
    case Right(claims) => println(s"accepted: sub=${claims.sub.getOrElse("-")}")
    case Left(error)   => println(s"rejected: ${error.getClass.getSimpleName}: ${error.message}")
  }

  val claims = JwtClaims(
    iss = Some("bookstore-auth"),
    sub = Some("reader-1024"),
    aud = Some(JwtAudience.single("catalog-api")),
    iat = Some(now),
    exp = Some(now + 3600),
    extra = Map(
      "titles" -> JwtValue.Arr(
        JwtValue.Str("Zionomicon"),
        JwtValue.Str("Programming in Scala"),
        JwtValue.Str("Functional Programming in Scala")
      )
    )
  )

  val token: String = Jwt.sign(claims, secret, Algorithm.HS256) match {
    case Right(signed) => signed
    case Left(error)   => throw error
  }

  val segments = token.split('.')

  def decodeSegment(segment: String): String =
    Base64Url.decode(segment).map(bytes => new String(bytes, UTF_8)).fold(_.message, identity)

  val catalogOptions = JwtDecodeOptions(
    issuer = Some("bookstore-auth"),
    expectedAudience = Some("catalog-api"),
    nowSeconds = Some(now + 60)
  )

  report(Jwt.decode("a" * 10000, secret, Algorithm.HS256, catalogOptions))

  val deepHeader = """{"alg":"HS256","x":""" + ("[" * 64) + ("]" * 64) + "}"
  val deepToken  = s"${Base64Url.encode(deepHeader.getBytes(UTF_8))}.${segments(1)}.${segments(2)}"

  report(Jwt.decode(deepToken, secret, Algorithm.HS256, catalogOptions))

  val noneHeader      = Base64Url.encode("""{"alg":"none","typ":"JWT"}""".getBytes(UTF_8))
  val unsignedToken   = s"$noneHeader.${segments(1)}."
  val noneWithGarbage = s"$noneHeader.${segments(1)}.${segments(2)}"

  report(Jwt.decode(unsignedToken, secret, Algorithm.HS256, catalogOptions))
  report(Jwt.decode(noneWithGarbage, secret, Algorithm.HS256, catalogOptions))

  val publicEndpointLimits = JwtLimits(
    maxTokenChars = 2048,
    maxSegmentChars = 1024,
    maxJsonChars = 1024,
    maxDepth = 4,
    maxFields = 32,
    maxArrayElements = 32
  )

  val publicOptions = catalogOptions.copy(limits = publicEndpointLimits)

  report(Jwt.decode(token, secret, Algorithm.HS256, publicOptions))
  report(Jwt.decode("a" * 4000, secret, Algorithm.HS256, publicOptions))
}
```

</details>

Run it from the `jwt-examples` directory:

```bash
sbt "runMain issueandverifyjwts.Step6HostileTokensExample"
```

### Sign With an Asymmetric Key Pair

This example signs with RSA and Ed25519 private keys, verifies with the public keys, rejects an algorithm-confusion token, and loads a public key from PEM text.

<details>
  <summary>jwt-examples/src/main/scala/issueandverifyjwts/Step7AsymmetricKeysExample.scala</summary>

```scala title="jwt-examples/src/main/scala/issueandverifyjwts/Step7AsymmetricKeysExample.scala" showLineNumbers
package issueandverifyjwts

/**
 * How to Issue and Verify JWTs: Step 7 — Signing With an Asymmetric Key Pair
 *
 * Signs with RSA and Ed25519 private keys, verifies with the public keys,
 * rejects an algorithm-confusion token, and loads a public key from PEM text.
 *
 * Run with: cd jwt-examples && sbt "runMain
 * issueandverifyjwts.Step7AsymmetricKeysExample"
 */
object Step7AsymmetricKeysExample extends App {

  // 2026-01-01T00:00:00Z. A fixed "now" keeps every result reproducible.
  val now: Long = 1767225600L

  def report(result: Either[JwtError, JwtClaims]): Unit = result match {
    case Right(claims) => println(s"accepted: sub=${claims.sub.getOrElse("-")}")
    case Left(error)   => println(s"rejected: ${error.getClass.getSimpleName}: ${error.message}")
  }

  val claims = JwtClaims(
    iss = Some("bookstore-auth"),
    sub = Some("reader-1024"),
    aud = Some(JwtAudience.single("catalog-api")),
    iat = Some(now),
    exp = Some(now + 3600),
    extra = Map(
      "titles" -> JwtValue.Arr(
        JwtValue.Str("Zionomicon"),
        JwtValue.Str("Programming in Scala"),
        JwtValue.Str("Functional Programming in Scala")
      )
    )
  )

  val catalogOptions = JwtDecodeOptions(
    issuer = Some("bookstore-auth"),
    expectedAudience = Some("catalog-api"),
    nowSeconds = Some(now + 60)
  )

  JvmJwtCryptoBackend.install()

  val rsaGenerator = KeyPairGenerator.getInstance("RSA")
  rsaGenerator.initialize(2048)
  val authServiceKeys = rsaGenerator.generateKeyPair()

  val privateKeyDer: Array[Byte] = authServiceKeys.getPrivate.getEncoded // stays in the auth service
  val publicKeyDer: Array[Byte]  = authServiceKeys.getPublic.getEncoded  // shared with the catalog API

  println(authServiceKeys.getPrivate.getFormat)
  println(authServiceKeys.getPublic.getFormat)

  val rsaToken: String = Jwt.sign(claims, privateKeyDer, Algorithm.RS256).fold(e => throw e, t => t)

  val strangerKeys = rsaGenerator.generateKeyPair()

  report(Jwt.decode(rsaToken, publicKeyDer, Algorithm.RS256, catalogOptions))
  report(Jwt.decode(rsaToken, strangerKeys.getPublic.getEncoded, Algorithm.RS256, catalogOptions))

  val confusedToken: String =
    Jwt.sign(claims.copy(sub = Some("reader-0007")), publicKeyDer, Algorithm.HS256).fold(e => throw e, t => t)

  report(Jwt.decode(confusedToken, publicKeyDer, Algorithm.RS256, catalogOptions))

  val edKeys = KeyPairGenerator.getInstance("Ed25519").generateKeyPair()

  val edToken: String = Jwt.sign(claims, edKeys.getPrivate.getEncoded, Algorithm.EdDSA).fold(e => throw e, t => t)

  report(Jwt.decode(edToken, edKeys.getPublic.getEncoded, Algorithm.EdDSA, catalogOptions))

  def pemToDer(pem: String): Array[Byte] =
    Base64.getMimeDecoder.decode(pem.linesIterator.filterNot(_.startsWith("-----")).mkString)

  val publicKeyPem =
    "-----BEGIN PUBLIC KEY-----\n" +
      Base64.getMimeEncoder(64, "\n".getBytes(UTF_8)).encodeToString(publicKeyDer) +
      "\n-----END PUBLIC KEY-----\n"

  report(Jwt.decode(rsaToken, pemToDer(publicKeyPem), Algorithm.RS256, catalogOptions))
}
```

</details>

Run it from the `jwt-examples` directory:

```bash
sbt "runMain issueandverifyjwts.Step7AsymmetricKeysExample"
```

### The Complete Example

The complete example from "Putting It Together" runs everything in one program:

```bash
sbt "runMain issueandverifyjwts.CompleteExample"
```

### Compile All Examples

To compile every example in the project without running any of them, run this from the `jwt-examples` directory:

```bash
sbt compile
```

## Going Further

- The [JWT reference](../reference/jwt.md) lists every algorithm, key size rule, limit and `JwtError` case.
- Use `JwtHeader(alg, kid = Some("..."))` with the `Jwt.sign` overload that takes a header to label which key signed a token. The module writes `kid` but does not use it to choose a key, so the verifier picks the key itself.
- [How to Protect an Endpoint with JWT Bearer Tokens](./protect-endpoints-with-jwt.md) wires the verification into an `http-model` endpoint that reads the `Authorization` header and answers 401 or 403.

---

## Coming from circe, zio-json or jsoniter-scala

This guide moves a small catalog of Scala books from circe, zio-json or jsoniter-scala to ZIO Blocks. By the end, each type has one derived `Schema` instead of a library-specific codec, your naming, discriminator and date-format rules live on a `JsonCodecDeriver` or in `Modifier` annotations, and you know the five places where habits from your old library silently do the wrong thing. Each source library gets its own section with the same three parts: the code you wrote, the code you write now, and what changes. Read the concept map, then jump to the section for the library you are leaving.

- [Concept Map](#concept-map)
- [Coming from circe](#coming-from-circe)
- [Coming from zio-json](#coming-from-zio-json)
- [Coming from jsoniter-scala](#coming-from-jsoniter-scala)
- [Traps for Migrating Code](#traps-for-migrating-code)
- [What You Gain and What Is Different](#what-you-gain-and-what-is-different)
- [Migration Checklist](#migration-checklist)

## The Problem

All three libraries derive a JSON codec from a case class, but the codec is the end of the road: it only does JSON. The same `Book` needs a second derivation for YAML, Avro or a database row, and every naming or discriminator rule is configured again for each one. The configuration also lives in a different place in each library — a `Configuration` value in implicit scope for circe, annotations for zio-json, a `CodecMakerConfig` passed to a macro for jsoniter-scala — so moving between them means relearning where each rule goes.

This is the old code, the same type with a codec for each library (not compiled here):

```scala
// circe

case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String], tags: List[String])
    derives Codec.AsObject

// zio-json

object Book { implicit val codec: JsonCodec[Book] = DeriveJsonCodec.gen[Book] }

// jsoniter-scala

object Book { implicit val codec: JsonValueCodec[Book] = JsonCodecMaker.make }
```

In ZIO Blocks, a type derives a `Schema` once. The JSON codec is derived from that schema, and so is every other format's codec, from the same description and the same annotations.

## Prerequisites

Add the schema module, which includes the JSON codec:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56"
```

For Scala.js, use `%%%`. You can remove the circe, zio-json or jsoniter-scala dependency once the last codec is gone; on the JVM the schema module has no third-party dependencies, and on Scala.js it adds only the `scala-java-time` libraries.

This guide assumes you already know one of the three source libraries. It maps their concepts onto ZIO Blocks rather than teaching `Schema` from scratch, so if `Schema.derived`, `toJsonString` and `fromJson` are new to you, work through [Your First Schema](./your-first-schema.md) first. Every ZIO Blocks snippet below starts from these imports:

```scala

```

`zio.blocks.schema._` brings in `Schema`, `Modifier`, `NameMapper` and the `toJsonString` and `fromJson` extension methods; `zio.blocks.schema.json._` brings in `JsonCodec`, `JsonCodecDeriver` and `DiscriminatorKind`.

## The Book Catalog Model

The running example is a catalog of Scala books. A `Book` records its title, its first author, the year it came out, an ISBN when we know it, and a list of tags. A sealed `Format` says how a copy is sold:

```scala
case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String], tags: List[String])
object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

sealed trait Format
object Format {
  case class Paperback(pages: Int) extends Format
  case class Ebook(sizeMb: Double) extends Format

  implicit val schema: Schema[Format] = Schema.derived
}

val zionomicon = Book("Zionomicon", "John A. De Goes", 2024, None, List("zio", "effects"))
val programmingInScala =
  Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"), List("scala"))
val fpInScala =
  Book("Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"), Nil)
```

The ISBN of Zionomicon is unknown here, so it is `None`, and the page counts and file sizes in later examples are sample data.

## Concept Map

The table maps each task to its spelling in the three source libraries and in ZIO Blocks:

| Task                           | circe 0.14                                    | zio-json 1.x                                     | jsoniter-scala 2                                        | ZIO Blocks                                                                                    |
| ------------------------------ | --------------------------------------------- | ------------------------------------------------ | ------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| Derive a codec                 | `deriveCodec[A]`, `derives Codec.AsObject`    | `DeriveJsonCodec.gen[A]`                         | `JsonCodecMaker.make[A]`                                | `Schema.derived[A]`; the JSON codec comes from the schema                                     |
| Encode to a string             | `a.asJson.noSpaces`                           | `a.toJson`                                       | `writeToString(a)`                                      | `a.toJsonString`                                                                              |
| Decode from a string           | `decode[A](s)`                                | `s.fromJson[A]`                                  | `readFromString[A](s)`                                  | `s.fromJson[A]`                                                                               |
| Decode error                   | `Left(io.circe.Error)`                        | `Left(String)`                                   | throws `JsonReaderException`                            | `Left(SchemaError)`                                                                           |
| JSON AST                       | `io.circe.Json`                               | `zio.json.ast.Json`                              | none                                                    | `zio.blocks.schema.json.Json`, from `a.toJson`                                                |
| Naming strategy for fields     | `Configuration#withSnakeCaseMemberNames`      | `@jsonMemberNames(SnakeCase)`                    | `CodecMakerConfig#withFieldNameMapper`                  | `JsonCodecDeriver#withFieldNameMapper`, `@Modifier.fieldNaming`                               |
| Rename one field               | `@JsonKey` (circe-generic-extras, Scala 2)    | `@jsonField("name")`                             | `@named("name")`                                        | `@Modifier.rename("name")`                                                                    |
| Accept an old field name       | none                                          | `@jsonAliases("old")`                            | none                                                    | `@Modifier.alias("old")`                                                                      |
| Type discriminator field       | `Configuration#withDiscriminator("type")`     | `@jsonDiscriminator("type")`                     | `"type"` by default                                     | `DiscriminatorKind.Field("type")`, `@Modifier.discriminator("type")`                          |
| Case names                     | `Configuration#withTransformConstructorNames` | `@jsonHint`, `@jsonHintNames`                    | `@named`, `CodecMakerConfig#withAdtLeafClassNameMapper` | `@Modifier.rename` on the case, `@Modifier.caseNaming`, `JsonCodecDeriver#withCaseNameMapper` |
| Leave a field out              | none                                          | `@jsonExclude`                                   | `@transient`                                            | `@Modifier.transient()`                                                                       |
| Use defaults for missing input | `Configuration#withDefaults`                  | on by default                                    | on by default                                           | on by default                                                                                 |
| Reject unknown fields          | `Configuration#withStrictDecoding`            | `@jsonNoExtraFields`                             | `CodecMakerConfig#withSkipUnexpectedFields(false)`      | `@Modifier.noExtraFields()`, `JsonCodecDeriver#withRejectExtraFields(true)`                   |
| Custom codec for one type      | a given `Encoder` and `Decoder` in scope      | a given `JsonEncoder` and `JsonDecoder` in scope | a given `JsonValueCodec` in scope                       | `JsonCodecDeriver.withInstance[A](codec)`                                                     |

The last row is the biggest change. In all three libraries, the derived codec searches implicit scope for an encoder and decoder for each field type. ZIO Blocks derivation never does: it reads the field types from the schema and uses its own codecs, unless you register an override on the deriver. The [traps section](#an-implicit-codec-in-scope-is-ignored) shows what happens when you keep the old habit.

## Coming from circe

circe derives an `Encoder` and a `Decoder` (or both, as a `Codec`) per type, and configured derivation reads a `Configuration` from implicit scope. In ZIO Blocks, the type gets a `Schema`, and configuration is a value you pass to the deriver.

### Derived Codecs

The first change replaces the codec with a schema and keeps the encode and decode calls almost as they were.

#### You Wrote This

This is the old code, circe 0.14 on Scala 3 (not compiled here):

```scala

case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String], tags: List[String])
    derives Codec.AsObject

val programmingInScala = Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"), List("scala"))

println(programmingInScala.asJson.noSpaces)
println(decode[Book](programmingInScala.asJson.noSpaces))
```

#### You Write This

The schema in the companion replaces `derives Codec.AsObject`, `toJsonString` replaces `asJson.noSpaces`, and `fromJson` replaces `decode`:

```scala
println(programmingInScala.toJsonString)
// {"title":"Programming in Scala","firstAuthor":"Martin Odersky","year":2008,"isbn":"9780981531601","tags":["scala"]}
println(programmingInScala.toJsonString.fromJson[Book])
// Right(Book(Programming in Scala,Martin Odersky,2008,Some(9780981531601),List(scala)))
```

### Configured Derivation

circe's configured derivation puts the rules in a `Configuration`. The ZIO Blocks equivalent is a `JsonCodecDeriver` with the same rules, from which we derive the codec for each top-level type we send.

#### You Wrote This

This is the old code, with `snake_case` member names, a `"type"` discriminator and default values (not compiled here):

```scala

given Configuration = Configuration.default.withSnakeCaseMemberNames.withDiscriminator("type").withDefaults

case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String], tags: List[String])
    derives ConfiguredCodec

sealed trait Format derives ConfiguredCodec
object Format {
  case class Paperback(pages: Int) extends Format
  case class Ebook(sizeMb: Double) extends Format
}
```

#### You Write This

`JsonCodecDeriver#withFieldNameMapper` sets the naming strategy and `JsonCodecDeriver#withDiscriminatorKind` with `DiscriminatorKind.Field` sets the discriminator. `Schema#derive` builds a codec through that deriver:

```scala
val apiDeriver: JsonCodecDeriver =
  JsonCodecDeriver
    .withFieldNameMapper(NameMapper.SnakeCase)
    .withDiscriminatorKind(DiscriminatorKind.Field("type"))

val bookCodec: JsonCodec[Book]     = Book.schema.derive(apiDeriver)
val formatCodec: JsonCodec[Format] = Format.schema.derive(apiDeriver)
```

We call `JsonCodec#encodeToString` and `JsonCodec#decode` on the derived codecs:

```scala
println(bookCodec.encodeToString(programmingInScala))
// {"title":"Programming in Scala","first_author":"Martin Odersky","year":2008,"isbn":"9780981531601","tags":["scala"]}
println(formatCodec.encodeToString(Format.Paperback(320)))
// {"type":"Paperback","pages":320}
println(formatCodec.decode("""{"type":"Ebook","size_mb":4.5}"""))
// Right(Ebook(4.5))
```

There is no equivalent of `Configuration#withDefaults` to turn on: a field with a default value is filled in when it is missing from the input, and left out of the output when it holds that value. If the rule belongs to one type rather than to the service you talk to, annotate the type with `@Modifier.fieldNaming("snake_case")` or `@Modifier.discriminator("type")` instead, and the plain `toJsonString` picks it up. [How to Customize JSON Field Names, ADT Encoding and Optional Fields](./customize-json-encoding.md) covers every option and when to pick an annotation over a deriver setting.

### Custom Codecs for One Type

In circe, a custom `Encoder[LocalDate]` and `Decoder[LocalDate]` in implicit scope change every derived codec that contains a `LocalDate`. In ZIO Blocks, you write one `JsonCodec[LocalDate]` and register it on the deriver.

#### You Wrote This

This is the old code, writing dates as `dd/MM/yyyy` (not compiled here):

```scala

val partnerDateFormat = DateTimeFormatter.ofPattern("dd/MM/yyyy")

given Encoder[LocalDate] = Encoder.encodeString.contramap(partnerDateFormat.format)
given Decoder[LocalDate] = Decoder.decodeString.emap { s =>
  Try(LocalDate.parse(s, partnerDateFormat)).toEither.left.map(_ => s"expected a dd/MM/yyyy date, got: $s")
}

case class Order(title: String, orderedOn: LocalDate) derives Codec.AsObject
```

#### You Write This

A `JsonCodec[LocalDate]` reads and writes the string through the streaming `JsonReader` and `JsonWriter`. `JsonCodecDeriver#withInstance` registers it, and `Schema#deriving` derives a codec through the resulting deriver:

```scala

val partnerDateFormat = DateTimeFormatter.ofPattern("dd/MM/yyyy")

val partnerDateCodec: JsonCodec[LocalDate] = new JsonCodec[LocalDate] {
  def decodeValue(in: JsonReader): LocalDate = {
    val s = in.readString()
    try LocalDate.parse(s, partnerDateFormat)
    catch { case _: DateTimeParseException => error(s"expected a dd/MM/yyyy date, got: $s") }
  }

  def encodeValue(x: LocalDate, out: JsonWriter): Unit =
    out.writeVal(partnerDateFormat.format(x))
}

case class Order(title: String, orderedOn: LocalDate)
object Order {
  implicit val schema: Schema[Order] = Schema.derived
}

val partnerDeriver = JsonCodecDeriver.withInstance[LocalDate](partnerDateCodec)
val orderCodec: JsonCodec[Order] = Order.schema.deriving(partnerDeriver).derive
```

The derived codec uses the custom format for `orderedOn`, and would do so for a `LocalDate` at any depth:

```scala
println(orderCodec.encodeToString(Order("Zionomicon", LocalDate.of(2025, 3, 14))))
// {"title":"Zionomicon","orderedOn":"14/03/2025"}
println(orderCodec.decode("""{"title":"Zionomicon","orderedOn":"14/03/2025"}"""))
// Right(Order(Zionomicon,2025-03-14))
```

The order date is sample data. [How to Override the Codec for One Type Everywhere](./override-codec-for-one-type.md) covers the full codec, including the methods for the `Json` value type, and the narrower scopes of one field or one derivation.

### Decode Errors

circe returns an `io.circe.Error`, either a `ParsingFailure` or a `DecodingFailure` with a cursor history. ZIO Blocks returns a `SchemaError`, whose `SchemaError#message` names the problem and the path to it:

```scala
"""{"title":"Zionomicon","firstAuthor":"John A. De Goes","year":"2024"}""".fromJson[Book] match {
  case Left(error) => println(error.message)
  case Right(book) => println(s"Decoded: $book")
}
// illegal number at: .year
```

The path after `at:` names the failing field and starts at `.` for the top-level value; here it points at the `year` field of the book. circe's `Decoder[Int]` accepts the string `"2024"`, so circe decodes this input; ZIO Blocks requires a JSON number. `SchemaError` extends `Exception`, so code that converts the result with `Either#toTry` or rethrows the error keeps working.

### What Changes

- **A schema replaces the codec.** `derives Codec.AsObject` or `deriveCodec` becomes `implicit val schema: Schema[A] = Schema.derived` in the companion. The same schema gives you every other format.
- **Configuration is a value, not an implicit.** A `Configuration` in implicit scope becomes a `JsonCodecDeriver` that you pass to `Schema#derive`, or a `Modifier` annotation on the type.
- **Codecs in implicit scope are ignored.** Register a custom codec with `JsonCodecDeriver.withInstance`; see the [traps](#an-implicit-codec-in-scope-is-ignored).
- **`None` and empty lists are left out by default.** circe writes `"isbn":null` and `"tags":[]`; ZIO Blocks omits both. See the [traps](#none-and-empty-collections-are-left-out).
- **Numbers must be JSON numbers.** circe decodes `"year":"2024"` into an `Int`; ZIO Blocks rejects it with `illegal number`.
- **Missing collections decode as empty.** circe fails with `Missing required field` when `tags` is absent and has no default; ZIO Blocks decodes it as `Nil`.
- **The default sealed trait shape is the same.** Without a discriminator, both wrap the case in an object keyed by its name, such as `{"Paperback":{"pages":320}}`. With a discriminator, circe writes the `"type"` field last and ZIO Blocks writes it first; both read it from any position.

## Coming from zio-json

zio-json and ZIO Blocks look alike on purpose: both derive from case classes, both configure with annotations, and both have `toJson` and `fromJson`. Most annotations map one to one, but two method names mean something slightly different.

### Derived Codecs

`DeriveJsonCodec.gen` becomes `Schema.derived`, and the call sites keep their names.

#### You Wrote This

This is the old code, zio-json 1.x (not compiled here):

```scala

case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String], tags: List[String])
object Book {
  implicit val codec: JsonCodec[Book] = DeriveJsonCodec.gen[Book]
}

val json: String = programmingInScala.toJson
val back: Either[String, Book] = json.fromJson[Book]
```

#### You Write This

The schema replaces the codec. `toJson` becomes `toJsonString`, and `fromJson` keeps its name:

```scala
val json: String                    = programmingInScala.toJsonString
val back: Either[SchemaError, Book] = json.fromJson[Book]
```

The string and the decoded value are the same as before:

```scala
println(json)
// {"title":"Programming in Scala","firstAuthor":"Martin Odersky","year":2008,"isbn":"9780981531601","tags":["scala"]}
println(back)
// Right(Book(Programming in Scala,Martin Odersky,2008,Some(9780981531601),List(scala)))
```

In ZIO Blocks, `toJson` exists too, but it returns the JSON AST, a `zio.blocks.schema.json.Json`, rather than a `String`. Its `toString` renders the JSON with indentation:

```scala
val ast: Json = programmingInScala.toJson
```

Printing the AST shows the same fields:

```scala
println(ast)
// {
//   "title": "Programming in Scala",
//   "firstAuthor": "Martin Odersky",
//   "year": 2008,
//   "isbn": "9780981531601",
//   "tags": [
//     "scala"
//   ]
// }
```

The error side of `fromJson` is a `SchemaError` instead of a `String`. Where the old code expected a `String`, map the error to its message with `SchemaError#message`:

```scala
val asString: Either[String, Book] = """{"title":"Zionomicon"}""".fromJson[Book].left.map(_.message)
// asString: Either[String, Book] = Left(
//   "missing required field \"firstAuthor\" at: ."
// )
println(asString)
// Left(missing required field "firstAuthor" at: .)
```

### Annotations

zio-json annotations become `Modifier` annotations. They are part of the schema, so `toJsonString`, `fromJson` and every codec derived from the schema follow them.

#### You Wrote This

This is the old code, zio-json 1.x, with a naming strategy, a renamed field, an accepted old name, an excluded field, a discriminator and `snake_case` case names (not compiled here):

```scala

@jsonMemberNames(SnakeCase)
case class Book(
  title: String,
  firstAuthor: String,
  year: Int,
  @jsonField("isbn_13") @jsonAliases("isbn") isbn: Option[String],
  tags: List[String],
  @jsonExclude searchScore: Double = 0.0
)

@jsonDiscriminator("type")
@jsonHintNames(SnakeCase)
sealed trait Format
object Format {
  case class Paperback(pages: Int) extends Format
  case class Ebook(sizeMb: Double) extends Format
}
```

#### You Write This

Each zio-json annotation has a `Modifier` counterpart, and the naming strategies are written as strings such as `"snake_case"`:

```scala

@Modifier.fieldNaming("snake_case")
case class Book(
  title: String,
  firstAuthor: String,
  year: Int,
  @Modifier.rename("isbn_13") @Modifier.alias("isbn") isbn: Option[String],
  tags: List[String],
  @Modifier.transient() searchScore: Double = 0.0
)
object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

@Modifier.discriminator("type")
@Modifier.caseNaming("snake_case")
sealed trait Format
object Format {
  case class Paperback(pages: Int) extends Format
  case class Ebook(sizeMb: Double) extends Format

  implicit val schema: Schema[Format] = Schema.derived
}

val programmingInScala =
  Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"), List("scala"), 0.93)
```

The plain extension methods follow the annotations. The renamed field and its alias both decode, and the transient field takes its default:

```scala
println(programmingInScala.toJsonString)
// {"title":"Programming in Scala","first_author":"Martin Odersky","year":2008,"isbn_13":"9780981531601","tags":["scala"]}
println((Format.Paperback(320): Format).toJsonString)
// {"type":"paperback","pages":320}
println("""{"title":"Programming in Scala","first_author":"Martin Odersky","year":2008,"isbn":"9780981531601"}""".fromJson[Book])
// Right(Book(Programming in Scala,Martin Odersky,2008,Some(9780981531601),List(),0.0))
```

The full mapping, including the annotations not shown here:

| zio-json                       | ZIO Blocks                                          |
| ------------------------------ | --------------------------------------------------- |
| `@jsonField("name")`           | `@Modifier.rename("name")`                          |
| `@jsonAliases("a", "b")`       | `@Modifier.alias("a") @Modifier.alias("b")`         |
| `@jsonMemberNames(SnakeCase)`  | `@Modifier.fieldNaming("snake_case")`               |
| `@jsonDiscriminator("type")`   | `@Modifier.discriminator("type")`                   |
| `@jsonHint("name")`            | `@Modifier.rename("name")` on the case              |
| `@jsonHintNames(SnakeCase)`    | `@Modifier.caseNaming("snake_case")`                |
| `@jsonExclude`                 | `@Modifier.transient()`                             |
| `@jsonNoExtraFields`           | `@Modifier.noExtraFields()`                         |

### What Changes

- **`toJson` returns a `Json` value.** Use `toJsonString` for the `String` that zio-json's `toJson` returns.
- **`fromJson` returns `Either[SchemaError, A]`.** Use `SchemaError#message` where you need the old `String`.
- **Encoders in implicit scope are ignored.** In zio-json, an implicit `JsonEncoder[LocalDate]` and `JsonDecoder[LocalDate]` change every derived codec; register a `JsonCodec[LocalDate]` with `JsonCodecDeriver.withInstance` instead. See the [traps](#an-implicit-codec-in-scope-is-ignored).
- **`@Modifier.rename` replaces the name.** Like `@jsonField`, the Scala name is no longer accepted; add `@Modifier.alias` with the old name if clients still send it. See the [traps](#modifierrename-drops-the-scala-name).
- **Empty collections are left out.** zio-json writes `"tags":[]`; ZIO Blocks omits the field, and decodes a missing collection as empty where zio-json fails with a missing-field error for `tags`. See the [traps](#none-and-empty-collections-are-left-out). `None` is omitted by both.
- **Numbers must be JSON numbers.** zio-json decodes `"year":"2024"` into an `Int`; ZIO Blocks rejects it.
- **Transient fields need a default.** A `@Modifier.transient()` field must have a default value unless it is an `Option` or a collection; `Schema.derived` reports the missing default at compile time.

## Coming from jsoniter-scala

jsoniter-scala generates a codec at compile time from a `CodecMakerConfig`. In ZIO Blocks, `Schema.derived` describes the type, and a `JsonCodecDeriver` plays the role of the config: its `with*` methods mirror most of the options.

### Derived Codecs

`JsonCodecMaker.make` becomes `Schema.derived`, and `writeToString` and `readFromString` become methods on the codec or extension methods on the value.

#### You Wrote This

This is the old code, jsoniter-scala 2 (not compiled here):

```scala

case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String], tags: List[String])
object Book {
  implicit val codec: JsonValueCodec[Book] =
    JsonCodecMaker.make(CodecMakerConfig.withFieldNameMapper(JsonCodecMaker.enforce_snake_case))
}

val json = writeToString(programmingInScala)
val back: Book = readFromString[Book](json) // throws JsonReaderException on bad input
```

#### You Write This

A `JsonCodecDeriver` takes the place of `CodecMakerConfig`, and `Schema#derive` takes the place of `JsonCodecMaker.make`. Keep the derived codec in the companion, as you kept the jsoniter-scala one:

```scala

case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String], tags: List[String])
object Book {
  implicit val schema: Schema[Book] = Schema.derived
  val codec: JsonCodec[Book]        = schema.derive(JsonCodecDeriver.withFieldNameMapper(NameMapper.SnakeCase))
}

val programmingInScala =
  Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"), List("scala"))
```

`JsonCodec#encodeToString` and `JsonCodec#decode` replace `writeToString` and `readFromString`. Bad input comes back as a `Left` instead of an exception:

```scala
println(Book.codec.encodeToString(programmingInScala))
// {"title":"Programming in Scala","first_author":"Martin Odersky","year":2008,"isbn":"9780981531601","tags":["scala"]}
println(Book.codec.decode(Book.codec.encodeToString(programmingInScala)))
// Right(Book(Programming in Scala,Martin Odersky,2008,Some(9780981531601),List(scala)))
println(Book.codec.decode("""{"title":"Zionomicon","first_author":42}"""))
// Left(zio.blocks.schema.SchemaError: expected '"' at: .firstAuthor)
```

`JsonCodec` also reads and writes `Array[Byte]`, `java.nio.ByteBuffer` and `java.io.InputStream`/`OutputStream`, the counterparts of `readFromArray`, `readFromByteBuffer` and `readFromStream`.

### Mapping `CodecMakerConfig`

Most `CodecMakerConfig` options that shape the JSON have a `JsonCodecDeriver` counterpart. jsoniter-scala's `requireCollectionFields` and `requireDefaultFields` also make the encoder write those fields, and so do their counterparts, so each is a single setting:

| `CodecMakerConfig`                                       | `JsonCodecDeriver`                                       |
| -------------------------------------------------------- | -------------------------------------------------------- |
| `withFieldNameMapper(JsonCodecMaker.enforce_snake_case)` | `withFieldNameMapper(NameMapper.SnakeCase)`              |
| `withAdtLeafClassNameMapper(f)`                          | `withCaseNameMapper(NameMapper.Custom(f))`               |
| `withDiscriminatorFieldName(Some("type"))` (default)     | `withDiscriminatorKind(DiscriminatorKind.Field("type"))` |
| `withDiscriminatorFieldName(None)`                       | `withDiscriminatorKind(DiscriminatorKind.Key)` (default) |
| `withTransientNone(b)`                                   | `withTransientNone(b)`                                   |
| `withTransientEmpty(b)`                                  | `withTransientEmptyCollection(b)`                        |
| `withTransientDefault(b)`                                | `withTransientDefaultValue(b)`                           |
| `withRequireCollectionFields(true)`                      | `withRequireCollectionFields(true)`                      |
| `withRequireDefaultFields(true)`                         | `withRequireDefaultValueFields(true)`                    |
| `withSkipUnexpectedFields(false)`                        | `withRejectExtraFields(true)`                            |
| `withCheckFieldDuplication(true)` (default)              | always on                                                |

Several options have no `JsonCodecDeriver` setting:

- `withIsStringified` and `@stringified`, which read and write numbers and booleans as JSON strings.
- `withMapAsArray`; maps are always JSON objects.
- The input limits: `withBigDecimalPrecision`, `withBigDecimalScaleLimit`, `withBigDecimalDigitsLimit`, `withBigIntDigitsLimit`, `withBitSetValueLimit`, `withMapMaxInsertNumber` and `withSetMaxInsertNumber`.
- `withRequireDiscriminatorFirst`; the ZIO Blocks decoder finds the discriminator field at any position.
- `withInlineOneValueClasses`, `withSkipNestedOptionValues`, `withCirceLikeObjectEncoding`, `withUseScalaEnumValueId`, `withTransientNull`, `withEncodingOnly` and `withDecodingOnly`.

If your API depends on one of the encoding options, register a custom codec for the affected type with `JsonCodecDeriver.withInstance`, as in the [circe section](#custom-codecs-for-one-type). `withAllowRecursiveTypes` has no counterpart because none is needed: `Schema.derived` handles recursive types.

### Performance

jsoniter-scala users usually pick it for speed. The repository's `benchmarks` project has a JMH benchmark, `JsonListOfRecordsBenchmark`, that reads and writes lists of records with jsoniter-scala, zio-json, ZIO Schema and ZIO Blocks, but the repository publishes no results. Run it on your own hardware and payloads before deciding.

### What Changes

- **The default sealed trait shape is different.** jsoniter-scala writes a `"type"` field by default; ZIO Blocks wraps the case in an object keyed by its name. Set `DiscriminatorKind.Field("type")` to keep your current JSON.
- **Bad input returns a `Left`.** `JsonCodec#decode` returns `Either[SchemaError, A]` instead of throwing.
- **The defaults for `None`, empty collections and default values match.** Both libraries leave them out of the output and fill them back in when they are missing from the input.
- **The discriminator can come anywhere.** jsoniter-scala requires the `"type"` field first by default; ZIO Blocks reads it from any position and writes it first.
- **Codecs in implicit scope are ignored.** A hand-written `JsonValueCodec` for a field type is not picked up; register it with `JsonCodecDeriver.withInstance`.

## Traps for Migrating Code

These five habits compile in ZIO Blocks but don't do what they did in your old library. None of them produces an error, so check for each one as you migrate.

### An Implicit Codec in Scope Is Ignored

In circe, zio-json and jsoniter-scala, an encoder and decoder for `LocalDate` in implicit scope change every derived codec. In ZIO Blocks, the deriver picks the codec for each field from the schema and never searches implicit scope, so the implicit below has no effect:

```scala

val partnerDateFormat = DateTimeFormatter.ofPattern("dd/MM/yyyy")

implicit val partnerDateCodec: JsonCodec[LocalDate] = new JsonCodec[LocalDate] {
  def decodeValue(in: JsonReader): LocalDate = LocalDate.parse(in.readString(), partnerDateFormat)
  def encodeValue(x: LocalDate, out: JsonWriter): Unit = out.writeVal(partnerDateFormat.format(x))
}

case class Order(title: String, orderedOn: LocalDate)
object Order {
  implicit val schema: Schema[Order] = Schema.derived
}

val order = Order("Zionomicon", LocalDate.of(2025, 3, 14))
```

The date still comes out in ISO-8601 form:

```scala
println(order.toJsonString)
// {"title":"Zionomicon","orderedOn":"2025-03-14"}
```

Register the codec with `JsonCodecDeriver.withInstance` and derive through that deriver instead:

```scala
println(Order.schema.deriving(JsonCodecDeriver.withInstance[LocalDate](partnerDateCodec)).derive.encodeToString(order))
// {"title":"Zionomicon","orderedOn":"14/03/2025"}
```

### `toJsonString` Ignores a Codec You Derived

`toJsonString`, `toJson` and `fromJson` always use `Schema#jsonCodec`, the codec from the default `JsonCodecDeriver`. A codec built with `Schema#derive` or `Schema#deriving` applies its settings only when you call it yourself:

```scala

case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String], tags: List[String])
object Book {
  implicit val schema: Schema[Book] = Schema.derived
  val snakeCaseCodec: JsonCodec[Book] = schema.derive(JsonCodecDeriver.withFieldNameMapper(NameMapper.SnakeCase))
}

val programmingInScala =
  Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"), List("scala"))
```

The extension method writes `firstAuthor`, and only the codec writes `first_author`:

```scala
println(programmingInScala.toJsonString)
// {"title":"Programming in Scala","firstAuthor":"Martin Odersky","year":2008,"isbn":"9780981531601","tags":["scala"]}
println(Book.snakeCaseCodec.encodeToString(programmingInScala))
// {"title":"Programming in Scala","first_author":"Martin Odersky","year":2008,"isbn":"9780981531601","tags":["scala"]}
```

Keep a derived codec in one place, such as the companion, and call it everywhere you read or write that JSON. If the rule should apply to every caller, put it on the type with a `Modifier` annotation instead, which `toJsonString` does follow.

### `@Modifier.rename` Drops the Scala Name

`@Modifier.rename` replaces the field name for both encoding and decoding, so the Scala name is no longer accepted. Unknown fields are skipped by default, so for an `Option` field, input that still uses the Scala name decodes without an error and the value is lost:

```scala

case class Book(title: String, @Modifier.rename("isbn_13") isbn: Option[String])
object Book {
  implicit val schema: Schema[Book] = Schema.derived
}
```

The ISBN sent under its Scala name becomes `None`:

```scala
println("""{"title":"Programming in Scala","isbn":"9780981531601"}""".fromJson[Book])
// Right(Book(Programming in Scala,None))
```

Add the old name with `@Modifier.alias`, which the decoder accepts but the encoder never writes, the way `@jsonAliases` works in zio-json:

```scala

case class Book(title: String, @Modifier.rename("isbn_13") @Modifier.alias("isbn") isbn: Option[String])
object Book {
  implicit val schema: Schema[Book] = Schema.derived
}
```

Now both names decode:

```scala
println("""{"title":"Programming in Scala","isbn":"9780981531601"}""".fromJson[Book])
// Right(Book(Programming in Scala,Some(9780981531601)))
println("""{"title":"Programming in Scala","isbn_13":"9780981531601"}""".fromJson[Book])
// Right(Book(Programming in Scala,Some(9780981531601)))
```

### An Implicit `NameMapper` Does Nothing

circe users put a `Configuration` in implicit scope, so an implicit `NameMapper` in the companion looks like the natural translation. Nothing reads it:

```scala

case class Book(title: String, firstAuthor: String)
object Book {
  implicit val schema: Schema[Book]   = Schema.derived
  implicit val nameMapper: NameMapper = NameMapper.SnakeCase
}
```

The output keeps the Scala name `firstAuthor`:

```scala
println(Book("Programming in Scala", "Martin Odersky").toJsonString)
// {"title":"Programming in Scala","firstAuthor":"Martin Odersky"}
```

Use `@Modifier.fieldNaming("snake_case")` on the type, or `JsonCodecDeriver#withFieldNameMapper` with `Schema#derive`, as in the [circe section](#configured-derivation).

### `None` and Empty Collections Are Left Out

By default, the encoder leaves out a `None` field and an empty collection, and the decoder fills them back in as `None` and an empty collection when they are missing. circe writes them as `null` and `[]`, so a client that expects every key to be present breaks:

```scala

case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String], tags: List[String])
object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

val zionomicon = Book("Zionomicon", "John A. De Goes", 2024, None, Nil)
```

The default output has no `isbn` and no `tags`:

```scala
println(zionomicon.toJsonString)
// {"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024}
```

To keep circe's shape, turn both off on the deriver:

```scala
val circeShaped: JsonCodec[Book] =
  Book.schema.derive(JsonCodecDeriver.withTransientNone(false).withTransientEmptyCollection(false))
```

`None` is then written as `null` and an empty list as `[]`:

```scala
println(circeShaped.encodeToString(zionomicon))
// {"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024,"isbn":null,"tags":[]}
```

`JsonCodecDeriver#withTransientDefaultValue(false)` does the same for fields that hold their default value. [Control Optional Fields, Empty Collections and Defaults](./customize-json-encoding.md#control-optional-fields-empty-collections-and-defaults) covers the matching decoder settings, which make missing fields an error; `JsonCodecDeriver#withRequireCollectionFields(true)` and `JsonCodecDeriver#withRequireDefaultValueFields(true)` also make the encoder write empty collections and default values.

## What You Gain and What Is Different

You gain:

- **One derivation for every format.** The `Schema` that gives you JSON also gives you YAML, Avro, MessagePack, TOON and the other formats in [One Type, Many Formats](./one-type-many-formats.md), with the same annotations.
- **Configuration you can see.** Rules are values on a `JsonCodecDeriver` or annotations on the type, not instances found through implicit search, so the codec a call site uses is the one it names.
- **Errors with paths, never exceptions.** `fromJson` and `JsonCodec#decode` return `Either[SchemaError, A]`, and the message names the field that failed.
- **No third-party JSON dependencies.** The schema module runs on Scala 2.13 and 3, on the JVM and Scala.js, and needs no JSON library.

What is different:

- **Implicit codecs are not part of derivation.** Overrides go through `JsonCodecDeriver.withInstance`.
- **The extension methods use the default codec.** A configured codec must be called directly.
- **`None` and empty collections are omitted by default**, as in jsoniter-scala and unlike circe.
- **Some jsoniter-scala options have no counterpart**, such as stringified numbers and maps as arrays.

## Migration Checklist

1. Add `zio-blocks-schema` (or `zio-blocks-next-schema` on Scala 3.9 and newer) and `import zio.blocks.schema._`.
2. Replace every `deriveCodec`, `derives Codec.AsObject`, `DeriveJsonCodec.gen` or `JsonCodecMaker.make` with `implicit val schema: Schema[A] = Schema.derived` in the companion.
3. Replace `asJson.noSpaces`, zio-json `toJson` and `writeToString` with `toJsonString`, and `decode`, `fromJson` and `readFromString` with `fromJson` or `JsonCodec#decode`.
4. Replace error handling on `io.circe.Error`, `String` or `JsonReaderException` with `SchemaError`.
5. Turn a circe `Configuration` or a jsoniter-scala `CodecMakerConfig` into a `JsonCodecDeriver`, derive codecs with `Schema#derive`, and call those codecs directly instead of `toJsonString`.
6. Replace zio-json annotations with their `Modifier` counterparts, and add `@Modifier.alias` for any renamed field whose Scala name clients still send.
7. Replace each implicit codec for a field type with a `JsonCodec` registered through `JsonCodecDeriver.withInstance`, and delete the implicit.
8. Decide whether `None` and empty collections may be omitted; if not, set `withTransientNone(false)` and `withTransientEmptyCollection(false)`.
9. If you came from jsoniter-scala, set `DiscriminatorKind.Field("type")` to keep your sealed trait JSON.
10. Round-trip a sample of real payloads through the new codecs and compare the JSON with what the old library writes.

## Putting It Together

This is the catalog fully migrated: `Schema.derived` for every type, one `JsonCodecDeriver` that writes `snake_case` names, a `"type"` field, explicit `null`s and empty lists, and dates as `dd/MM/yyyy`, and a renamed ISBN that still accepts its old name:

```scala title="schema-examples/src/main/scala/jsonlibrarymigration/BookCatalogMigratedExample.scala"
package jsonlibrarymigration

/**
 * Coming from circe, zio-json or jsoniter-scala — Complete Example
 *
 * A book catalog migrated to ZIO Blocks: Schema.derived for every type, one
 * JsonCodecDeriver for snake_case names, a "type" discriminator, explicit nulls
 * and empty lists, and dd/MM/yyyy dates, plus a renamed ISBN that still accepts
 * its Scala name.
 *
 * Run with: cd schema-examples && sbt "runMain
 * jsonlibrarymigration.BookCatalogMigratedExample"
 */
object BookCatalogMigratedExample extends App {

  sealed trait Format
  object Format {
    case class Paperback(pages: Int) extends Format
    case class Ebook(sizeMb: Double) extends Format

    implicit val schema: Schema[Format] = Schema.derived
  }

  case class Book(
    title: String,
    firstAuthor: String,
    year: Int,
    @Modifier.rename("isbn_13")
    @Modifier.alias("isbn")
    isbn: Option[String],
    tags: List[String],
    addedOn: LocalDate,
    formats: List[Format]
  )
  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  // Replaces an implicit Encoder/Decoder (circe), JsonCodec (zio-json) or JsonValueCodec (jsoniter-scala)
  val partnerDateFormat = DateTimeFormatter.ofPattern("dd/MM/yyyy")

  val partnerDateCodec: JsonCodec[LocalDate] = new JsonCodec[LocalDate] {
    private def parse(s: String): LocalDate =
      try LocalDate.parse(s, partnerDateFormat)
      catch { case _: DateTimeParseException => error(s"expected a dd/MM/yyyy date, got: $s") }

    def decodeValue(in: JsonReader): LocalDate =
      parse(in.readString())

    def encodeValue(x: LocalDate, out: JsonWriter): Unit =
      out.writeVal(partnerDateFormat.format(x))

    override def decodeValue(json: Json): LocalDate = json match {
      case s: Json.String => parse(s.value)
      case _              => error("expected a JSON string")
    }

    override def encodeValue(x: LocalDate): Json =
      Json.String(partnerDateFormat.format(x))
  }

  // Replaces a circe Configuration or a jsoniter-scala CodecMakerConfig
  val apiDeriver =
    JsonCodecDeriver
      .withFieldNameMapper(NameMapper.SnakeCase)
      .withDiscriminatorKind(DiscriminatorKind.Field("type"))
      .withTransientNone(false)
      .withTransientEmptyCollection(false)
      .withInstance[LocalDate](partnerDateCodec)

  // Keep the codec in one place: toJsonString and fromJson would use the default codec instead
  val bookCodec: JsonCodec[Book] = Book.schema.deriving(apiDeriver).derive

  // The added-on dates, page counts and file sizes are sample data
  val catalog = List(
    Book("Zionomicon", "John A. De Goes", 2024, None, List("zio", "effects"), LocalDate.of(2025, 3, 14), Nil),
    Book(
      "Programming in Scala",
      "Martin Odersky",
      2008,
      Some("9780981531601"),
      List("scala"),
      LocalDate.of(2025, 4, 2),
      List(Format.Paperback(736))
    ),
    Book(
      "Functional Programming in Scala",
      "Paul Chiusano",
      2014,
      Some("9781617290657"),
      Nil,
      LocalDate.of(2025, 5, 21),
      List(Format.Paperback(320), Format.Ebook(4.5))
    )
  )

  // Encode every book
  catalog.foreach(book => println(bookCodec.encodeToString(book)))

  // Decode a payload that still sends the ISBN under its Scala name
  println(
    bookCodec.decode(
      """{"title":"Programming in Scala","first_author":"Martin Odersky","year":2008,"isbn":"9780981531601","tags":["scala"],"added_on":"02/04/2025","formats":[{"type":"Paperback","pages":736}]}"""
    )
  )

  // Read a decode error with its path
  bookCodec.decode(
    """{"title":"Zionomicon","first_author":"John A. De Goes","year":2024,"tags":[],"added_on":"2025-03-14","formats":[]}"""
  ) match {
    case Left(error) => println(error.message)
    case Right(book) => println(s"Decoded: $book")
  }

  // Check the round trip
  println(catalog.forall(book => bookCodec.decode(bookCodec.encodeToString(book)) == Right(book)))
}
```

## Running the Examples

The complete example is a runnable file in the `schema-examples` project. Clone the repository and change into the project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

<details>
  <summary>schema-examples/src/main/scala/jsonlibrarymigration/BookCatalogMigratedExample.scala</summary>

```scala title="schema-examples/src/main/scala/jsonlibrarymigration/BookCatalogMigratedExample.scala" showLineNumbers
package jsonlibrarymigration

/**
 * Coming from circe, zio-json or jsoniter-scala — Complete Example
 *
 * A book catalog migrated to ZIO Blocks: Schema.derived for every type, one
 * JsonCodecDeriver for snake_case names, a "type" discriminator, explicit nulls
 * and empty lists, and dd/MM/yyyy dates, plus a renamed ISBN that still accepts
 * its Scala name.
 *
 * Run with: cd schema-examples && sbt "runMain
 * jsonlibrarymigration.BookCatalogMigratedExample"
 */
object BookCatalogMigratedExample extends App {

  sealed trait Format
  object Format {
    case class Paperback(pages: Int) extends Format
    case class Ebook(sizeMb: Double) extends Format

    implicit val schema: Schema[Format] = Schema.derived
  }

  case class Book(
    title: String,
    firstAuthor: String,
    year: Int,
    @Modifier.rename("isbn_13")
    @Modifier.alias("isbn")
    isbn: Option[String],
    tags: List[String],
    addedOn: LocalDate,
    formats: List[Format]
  )
  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  // Replaces an implicit Encoder/Decoder (circe), JsonCodec (zio-json) or JsonValueCodec (jsoniter-scala)
  val partnerDateFormat = DateTimeFormatter.ofPattern("dd/MM/yyyy")

  val partnerDateCodec: JsonCodec[LocalDate] = new JsonCodec[LocalDate] {
    private def parse(s: String): LocalDate =
      try LocalDate.parse(s, partnerDateFormat)
      catch { case _: DateTimeParseException => error(s"expected a dd/MM/yyyy date, got: $s") }

    def decodeValue(in: JsonReader): LocalDate =
      parse(in.readString())

    def encodeValue(x: LocalDate, out: JsonWriter): Unit =
      out.writeVal(partnerDateFormat.format(x))

    override def decodeValue(json: Json): LocalDate = json match {
      case s: Json.String => parse(s.value)
      case _              => error("expected a JSON string")
    }

    override def encodeValue(x: LocalDate): Json =
      Json.String(partnerDateFormat.format(x))
  }

  // Replaces a circe Configuration or a jsoniter-scala CodecMakerConfig
  val apiDeriver =
    JsonCodecDeriver
      .withFieldNameMapper(NameMapper.SnakeCase)
      .withDiscriminatorKind(DiscriminatorKind.Field("type"))
      .withTransientNone(false)
      .withTransientEmptyCollection(false)
      .withInstance[LocalDate](partnerDateCodec)

  // Keep the codec in one place: toJsonString and fromJson would use the default codec instead
  val bookCodec: JsonCodec[Book] = Book.schema.deriving(apiDeriver).derive

  // The added-on dates, page counts and file sizes are sample data
  val catalog = List(
    Book("Zionomicon", "John A. De Goes", 2024, None, List("zio", "effects"), LocalDate.of(2025, 3, 14), Nil),
    Book(
      "Programming in Scala",
      "Martin Odersky",
      2008,
      Some("9780981531601"),
      List("scala"),
      LocalDate.of(2025, 4, 2),
      List(Format.Paperback(736))
    ),
    Book(
      "Functional Programming in Scala",
      "Paul Chiusano",
      2014,
      Some("9781617290657"),
      Nil,
      LocalDate.of(2025, 5, 21),
      List(Format.Paperback(320), Format.Ebook(4.5))
    )
  )

  // Encode every book
  catalog.foreach(book => println(bookCodec.encodeToString(book)))

  // Decode a payload that still sends the ISBN under its Scala name
  println(
    bookCodec.decode(
      """{"title":"Programming in Scala","first_author":"Martin Odersky","year":2008,"isbn":"9780981531601","tags":["scala"],"added_on":"02/04/2025","formats":[{"type":"Paperback","pages":736}]}"""
    )
  )

  // Read a decode error with its path
  bookCodec.decode(
    """{"title":"Zionomicon","first_author":"John A. De Goes","year":2024,"tags":[],"added_on":"2025-03-14","formats":[]}"""
  ) match {
    case Left(error) => println(error.message)
    case Right(book) => println(s"Decoded: $book")
  }

  // Check the round trip
  println(catalog.forall(book => bookCodec.decode(bookCodec.encodeToString(book)) == Right(book)))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain jsonlibrarymigration.BookCatalogMigratedExample"
```

To check that the example compiles without running it:

```bash
sbt compile
```

## See Also

- [Your First Schema](./your-first-schema.md): derive a schema, round-trip JSON and read decode errors.
- [How to Customize JSON Field Names, ADT Encoding and Optional Fields](./customize-json-encoding.md): every naming, discriminator and optional-field option.
- [How to Override the Codec for One Type Everywhere](./override-codec-for-one-type.md): custom codecs with `Deriver#withInstance`.
- [How to Tune JSON Encoding and Decoding for Throughput](./tune-json-throughput.md): decode from bytes and streams, reuse codecs and size buffers.
- [JSON](../reference/schema/built-in-codecs/json/index.md): the JSON codec reference.
- [Migrating from ZIO Schema](./zio-schema-migration.md): the migration guide for ZIO Schema 1.x users.

---

## How to Read and Modify JSON Without Case Classes

In this guide, we will read and change a bookstore's JSON document without writing a case class for it, using the `Json` tree of ZIO Blocks Schema. By the end, you will have a handful of small snippets that parse JSON text, read one value deep in the document, collect many values at once, change prices, tags and fields in place, print the result, build new JSON in code, and turn one part of the document into a typed `Book` when you need it.

This is the tool for JSON whose shape you don't own or don't want to model: a webhook payload where you need 2 fields out of 50, a document where you patch one value, a third-party response whose shape keeps changing, or a quick script. We start with JSON text and work on it step by step:

- Parse JSON text into a `Json` tree and handle malformed input.
- Read one value deep in the tree, and handle a missing field or a wrong type.
- Select many values at once with a wildcard path.
- Change values, add fields and remove fields.
- Print the tree back to compact or pretty JSON text.
- Build new JSON in code.
- Decode one part of the tree into a case class.

## The Problem

When your code needs one value from a large JSON document, the usual options are both poor. You can write case classes for the whole document, which means modelling dozens of fields you never read and updating them whenever the sender adds one. Or you can search the text directly, which looks fine until the document contains the same key twice.

For example, consider an order from a bookshop's webhook, where we want the price of the first book. A regular expression finds the first `"price"` key in the text:

```scala
val webhookText =
  """{"shipping": {"method": "express", "price": 4.99}, "books": [{"title": "Zionomicon", "price": 49.99}]}"""
// webhookText: String = "{\"shipping\": {\"method\": \"express\", \"price\": 4.99}, \"books\": [{\"title\": \"Zionomicon\", \"price\": 49.99}]}"

val pricePattern = "\"price\"\\s*:\\s*([0-9.]+)".r
// pricePattern: Regex = "price"\s*:\s*([0-9.]+)

println(pricePattern.findFirstMatchIn(webhookText).map(_.group(1)))
// Some(4.99)
```

The code above returns the shipping price instead of the book's price, because text search has no idea which object a key belongs to. It also breaks when the value is a string with an escaped quote, when the sender reorders the keys, or when the number uses an exponent. Changing a value is worse: a text replacement can hit the wrong key, and nothing checks that the result is still valid JSON.

In this guide, we solve this with `Json`, an immutable tree that mirrors the JSON document exactly. We parse the text once, then read and change values by their path in the tree, such as `.books[0].price`, and print the tree back to text when we are done.

## Prerequisites

To follow along, add the schema module to your build. `Json` lives in it; no extra JSON module is needed:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"

// Scala 3.9 and newer: the same module with the zio-blocks-next- prefix
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56"
scalacOptions += "-experimental"
```

For scala-cli, Scala.js, or cross-building, see [How to Add ZIO Blocks to Your Build](../getting-started/installation.md).

The code on this page compiles on Scala 2.13 and Scala 3, except the `json"..."` interpolator in [Build JSON in Code](#build-json-in-code), which works only on Scala 3 in this release. No knowledge of `Schema` is needed until the last section, [Decode Part of the Tree Into a Case Class](#decode-part-of-the-tree-into-a-case-class), which assumes you can derive a `Schema` as shown in [Your First Schema](./your-first-schema.md).

## The Bookstore Document

Every example on this page works on one document: a small bookstore with three books. It has the things real payloads have: nested objects, arrays of strings, decimal prices, and one book that lacks a field the other two have (`Functional Programming in Scala` has no `stock`).

We import the schema package, which provides the `p"..."` path interpolator, and the `json` package, which provides `Json` itself:

```scala

val bookstoreText: String =
  """{
    |  "store": "Lambda Books",
    |  "currency": "USD",
    |  "books": [
    |    {
    |      "title": "Zionomicon",
    |      "authors": ["John A. De Goes", "Adam Fraser", "Milad Khajavi"],
    |      "price": 49.99,
    |      "tags": ["zio", "effects"],
    |      "stock": 12
    |    },
    |    {
    |      "title": "Programming in Scala",
    |      "authors": ["Martin Odersky", "Lex Spoon", "Bill Venners", "Frank Sommers"],
    |      "price": 59.99,
    |      "tags": ["scala", "language"],
    |      "stock": 0
    |    },
    |    {
    |      "title": "Functional Programming in Scala",
    |      "authors": ["Michael Pilquist", "Rúnar Bjarnason", "Paul Chiusano"],
    |      "price": 44.99,
    |      "tags": ["fp"]
    |    }
    |  ]
    |}""".stripMargin
```

## Parse JSON Text

`Json.parse` turns JSON text into a `Json` tree. It never throws: it returns `Either[SchemaError, Json]`, with `Right` holding the tree and `Left` holding what went wrong. To parse the bookstore and stop with an exception if the text is malformed:

```scala
val bookstore: Json = Json.parse(bookstoreText) match {
  case Right(json)  => json
  case Left(error) => throw error
}
```

`Json` is a sealed trait with one case class per JSON kind: `Json.Object`, `Json.Array`, `Json.String`, `Json.Number`, `Json.Boolean` and `Json.Null`. The tree keeps the document's key order, and every number is held as a `BigDecimal`, so `49.99` stays exactly `49.99`.

When the text is not valid JSON, `Json.parse` returns `Left`. For a trailing comma and for a missing comma between two fields:

```scala
println(Json.parse("""{"title": "Zionomicon", "price": 49.99,}"""))
// Left(zio.blocks.schema.SchemaError: expected '"' at: .)

println(Json.parse("""{"title": "Zionomicon" "price": 49.99}"""))
// Left(zio.blocks.schema.SchemaError: expected '}' or ',' at: .)
```

The message says what the parser expected and, after `at:`, where in the document structure it was (`.` is the root object). It gives no line or column: in this release, a parse error never carries a character offset, so for a large document you have to find the spot by its path in the structure.

`Json.parse` also accepts `Array[Byte]`, `java.nio.ByteBuffer` and `Chunk[Byte]`, so a raw HTTP request body needs no conversion to `String` first. When the input is trusted, such as a resource file shipped with your program, `Json.parseUnsafe` returns the `Json` directly and throws the `SchemaError` on malformed input.

## Read One Value Deep in the Tree

To read a value, we walk to it and decode it. `Json#get` takes a key (for an object) or an index (for an array) and returns a `JsonSelection`: the values found so far, or the error that stopped the walk. `JsonSelection#as[A]` then decodes the selected value into a Scala type.

To read the first book's price one step at a time, use `Json#get` for the `books` key, `JsonSelection#apply` for the index, and `JsonSelection#get` for the `price` key:

```scala
bookstore.get("books")(0).get("price").as[BigDecimal]
// res3: Either[SchemaError, BigDecimal] = Right(49.99)
```

The same walk can be written as one path with the `p"..."` interpolator, which reads like the JSON it walks: `.books` selects a key, `[0]` an array index. `Json#get` also accepts this path:

```scala
bookstore.get(p".books[0].price").as[BigDecimal]
// res4: Either[SchemaError, BigDecimal] = Right(49.99)

bookstore.get(p".books[1].authors[0]").as[String]
// res5: Either[SchemaError, String] = Right("Martin Odersky")
```

The `p"..."` path is checked when your code compiles, and it accepts only literal text. When the key or the index is only known at run time, chain `Json#get` and `JsonSelection#get` calls with the runtime values instead, for example `bookstore.get("books")(index).get(field)`.

:::tip
A key that is not a Scala identifier, such as `first-name`, can be written in `p"..."` only as a quoted key, `p"""{"first-name"}"""`, which `Json#get`, `Json#modify` and `Json#set` follow, but `Json#delete` and `Json#insert` ignore it and return the document unchanged. Build a path that works with every method with `DynamicOptic.root.field("first-name")`, which `Json#get` accepts the same way.
:::

Every read returns an `Either`, so a missing field and a value of the wrong type are ordinary results, not exceptions. When the second book has no `isbn` field, and when we read a title as a number:

```scala
println(bookstore.get(p".books[1].isbn").as[String])
// Left(zio.blocks.schema.SchemaError: Field 'isbn' not found)

println(bookstore.get(p".books[0].title").as[BigDecimal])
// Left(zio.blocks.schema.SchemaError: expected Json.Number at: .)

println(bookstore.get(p".books[5].title").as[String])
// Left(zio.blocks.schema.SchemaError: Index 5 out of bounds)
```

A failed step stops the walk, and the error passes through every later `JsonSelection#get`, so we check for errors once at the end. The path in a decode error (`at: .`) is relative to the selected value, not to the document. A not-found error names only the last key, without the path to it, so a long walk is easier to debug when you keep the path at hand.

For a field that may be missing, turn the `Either` into an `Option`. The third book has no `stock`, so we read it with a default:

```scala
bookstore.get(p".books[2].stock").as[Int].getOrElse(0)
// res9: Int = 0
```

## Select Many Values at Once

A path can select more than one value. `[*]` selects every element of an array, so `.books[*].title` selects the title of every book. `JsonSelection#asAll[A]` decodes all selected values and returns them in a `Chunk`. To list every title and add up every price:

```scala
println(bookstore.get(p".books[*].title").asAll[String])
// Right(Chunk(Zionomicon,Programming in Scala,Functional Programming in Scala))

println(bookstore.get(p".books[*].price").asAll[BigDecimal].map(_.sum))
// Right(154.97)
```

`JsonSelection#filter` keeps the selected values that pass a test, and we can keep walking from the values that remain. To list the titles of the books that cost less than 50:

```scala
println(
  bookstore
    .get(p".books[*]")
    .filter(book => book.get("price").as[BigDecimal].exists(_ < 50))
    .get("title")
    .asAll[String]
)
// Right(Chunk(Zionomicon,Functional Programming in Scala))
```

`JsonSelection#as[A]` expects exactly one value, so `bookstore.get(p".books[*].title").as[String]` fails with `Expected single value but got 3`. Use `JsonSelection#asAll` for a wildcard path, and `JsonSelection#as` for a path that names one value.

:::warning
A wildcard path skips the elements that lack the field. `bookstore.get(p".books[*].stock").asAll[Int]` returns `Chunk(12, 0)`, two values for three books, with no error. It fails only when no element has the field at all. When you need one result per element, walk the elements yourself and give each missing field a default:

```scala
println(bookstore.get("books").one.map(_.elements.map(book => book.get("stock").as[Int].getOrElse(0))))
// Right(Chunk(12,0,0))
```
:::

`JsonSelection#one` returns the single selected value as `Either[SchemaError, Json]`, and `Json#elements` returns an array's elements as a `Chunk[Json]`.

## Change Values in the Tree

`Json` is immutable: every change returns a new tree and leaves the original as it was. Each change method takes a path, which can contain `[*]` to change every element of an array at once.

`Json#modify` replaces each value at the path with the result of a function. To raise every price by 10%, we match on `Json.Number`, which holds a `BigDecimal`, and round the result to cents:

```scala
val raised: Json = bookstore.modify(p".books[*].price") {
  case Json.Number(price) =>
    Json.Number((price * BigDecimal("1.1")).setScale(2, BigDecimal.RoundingMode.HALF_UP))
  case other => other
}
```

We read the prices back to check the change:

```scala
println(raised.get(p".books[*].price").asAll[BigDecimal])
// Right(Chunk(54.99,65.99,49.49))
```

Without `setScale`, the new prices would keep every digit of the multiplication, such as `54.989`, because `BigDecimal` arithmetic never rounds by itself.

The same method changes an array. To add a tag to the first book, we append a `Json.String` to the `Chunk` that `Json.Array` holds:

```scala
val tagged: Json = raised.modify(p".books[0].tags") {
  case Json.Array(tags) => Json.Array(tags :+ Json.String("bestseller"))
  case other            => other
}
```

`Json#set` replaces a value that exists, and `Json#insert` adds a field that doesn't exist yet. To rename the store and give every book an `available` field that says whether its `stock` is above zero, we insert into each book through `Json#modify`:

```scala
val withAvailability: Json = tagged
  .set(p".store", Json.String("Lambda Books Online"))
  .modify(p".books[*]") { book =>
    book.insert(p".available", Json.Boolean(book.get("stock").as[Int].exists(_ > 0)))
  }
```

`Json#delete` removes a field or an array element. To remove the internal `stock` field from every book before publishing the document, we delete it inside each book:

```scala
val published: Json = withAvailability.modify(p".books[*]")(_.delete(p".stock"))
```

`JsonSelection#toChunk` returns the selected values as a `Chunk[Json]`. Each book now has its new fields and no `stock`:

```scala
println(published.get(p".books[*]").toChunk.map(_.print).mkString("\n"))
// {"title":"Zionomicon","authors":["John A. De Goes","Adam Fraser","Milad Khajavi"],"price":54.99,"tags":["zio","effects","bestseller"],"available":true}
// {"title":"Programming in Scala","authors":["Martin Odersky","Lex Spoon","Bill Venners","Frank Sommers"],"price":65.99,"tags":["scala","language"],"available":false}
// {"title":"Functional Programming in Scala","authors":["Michael Pilquist","Rúnar Bjarnason","Paul Chiusano"],"price":49.49,"tags":["fp"],"available":false}
```

:::warning
Do not write `withAvailability.delete(p".books[*].stock")` in this release. A known bug in 0.0.56 removes every book that has no `stock` field, together with the field: here, `Functional Programming in Scala` would disappear from the array. Deleting inside each element with `modify(p".books[*]")(_.delete(p".stock"))`, as above, keeps every book.
:::

A change at a path that doesn't exist does nothing and reports nothing, so a typo in a path changes nothing silently. When the path must exist, use the `OrFail` variant of the method (`Json#modifyOrFail`, `Json#setOrFail`, `Json#deleteOrFail`, `Json#insertOrFail`), which returns `Either[SchemaError, Json]`. With the key `currency` misspelled:

```scala
println(published.set(p".curency", Json.String("EUR")) == published)
// true

println(published.setOrFail(p".curency", Json.String("EUR")))
// Left(zio.blocks.schema.SchemaError: Field 'curency' not found)
```

`Json#modifyOrFail` takes a partial function, and it also fails when the function has no case for a value it finds, so it can turn the `case other => other` fallback above into an error.

## Print JSON Text

`Json#print` writes the tree as compact JSON text, with no spaces or line breaks. `Json#print(config)` takes a `WriterConfig`; `WriterConfig.withIndentionStep(2)` indents nested values by two spaces. To print the first book both ways, we take it out of its selection with `JsonSelection#one`:

```scala
val firstBook: Json = published.get(p".books[0]").one.getOrElse(Json.Null)
```

Then we print it compact and indented:

```scala
println(firstBook.print)
// {"title":"Zionomicon","authors":["John A. De Goes","Adam Fraser","Milad Khajavi"],"price":54.99,"tags":["zio","effects","bestseller"],"available":true}

println(firstBook.print(WriterConfig.withIndentionStep(2)))
// {
//   "title": "Zionomicon",
//   "authors": [
//     "John A. De Goes",
//     "Adam Fraser",
//     "Milad Khajavi"
//   ],
//   "price": 54.99,
//   "tags": [
//     "zio",
//     "effects",
//     "bestseller"
//   ],
//   "available": true
// }
```

`Json#toString` prints the same indented form, so `println(json)` shows readable JSON while you debug. Three things hold for both forms:

- Keys come out in the order they had in the input, and inserted keys come last.
- Numbers keep the text they were parsed from, so `49.99` prints as `49.99` and `1.10` keeps its trailing zero.
- Strings are escaped where JSON requires it, and non-ASCII characters such as the `ú` in `Rúnar` are written as they are. `WriterConfig.withEscapeUnicode(true)` writes them as `\u` escapes instead.

Two trees are equal with `==` when they hold the same values, and the key order of an object does not matter. The order of array elements does:

```scala
Json.parse("""{"title": "Zionomicon", "price": 49.99}""") ==
  Json.parse("""{"price": 49.99, "title": "Zionomicon"}""")
// res20: Boolean = true

Json.parse("""["zio", "effects"]""") == Json.parse("""["effects", "zio"]""")
// res21: Boolean = false
```

This makes `==` a good way to compare an expected document with an actual one in a test, without caring how either was formatted.

## Build JSON in Code

Sometimes the JSON comes from your own code: a request body, a test fixture, or a new element to add to a document. ZIO Blocks has two ways to write it.

### With Constructors

The `Json` case classes build any document, on Scala 2.13 and Scala 3. `Json.Object` and `Json.Array` take their contents as arguments. To build a fourth book and append it to the `books` array:

```scala
val handsOnScala: Json = Json.Object(
  "title"   -> Json.String("Hands-on Scala Programming"),
  "authors" -> Json.Array(Json.String("Li Haoyi")),
  "price"   -> Json.Number(BigDecimal("39.99")),
  "tags"    -> Json.Array(Json.String("scala"), Json.String("tooling"))
)

val fourBooks: Json = published.modify(p".books") {
  case Json.Array(books) => Json.Array(books :+ handsOnScala)
  case other             => other
}
```

The new book is the last element of the array:

```scala
println(fourBooks.get(p".books[*].title").asAll[String])
// Right(Chunk(Zionomicon,Programming in Scala,Functional Programming in Scala,Hands-on Scala Programming))
```

:::warning
Build decimal numbers from a `BigDecimal` or a `String`, not from a `Double`. In this release, a known bug in 0.0.56 makes `Json.Number(49.99)` print as `49.990` and `Json.Number(2.5)` as `2.50`. `Json.Number(BigDecimal("49.99"))` prints `49.99`. `Int` and `Long` values are not affected.
:::

### With the `json"..."` Interpolator (Scala 3)

On Scala 3, the `json"..."` interpolator from `zio.blocks.schema.json` writes the same document as JSON text, with Scala values spliced in. The text is checked when your code compiles: a malformed literal such as `json"""{"title": }"""` fails with `Invalid JSON literal: expected JSON value at: .title`.

To build the same fourth book from text, with its title, price and tags spliced in from Scala values:

```scala
val title = "Hands-on Scala Programming"
val price = BigDecimal("39.99")
val tags  = List("scala", "tooling")

val handsOnScalaFromText: Json =
  json"""{"title": $title, "authors": ["Li Haoyi"], "price": $price, "tags": $tags}"""
```

The result is the same tree that the constructors built:

```scala
println(handsOnScalaFromText.print)
// {"title":"Hands-on Scala Programming","authors":["Li Haoyi"],"price":39.99,"tags":["scala","tooling"]}

handsOnScalaFromText == handsOnScala
// res24: Boolean = true
```

A spliced value is encoded as JSON, not pasted into the text, so a `String` with a quote or a backslash in it is escaped correctly, and a `List` becomes an array. A `Json` value is spliced as it is. Any other type needs an implicit `Schema` for it: a `case class` with a derived `Schema` becomes an object.

On Scala 2.13, `json"..."` does not compile in this release: every call fails with `object JsonInterpolatorRuntime in package json cannot be accessed as a member of package zio.blocks.schema.json`, a known bug in 0.0.56. On Scala 2.13, build JSON with the constructors above, or parse a string with `Json.parse`.

## Decode Part of the Tree Into a Case Class

Once you know the shape of one part of the document, you can decode just that part into a case class and leave the rest as JSON. We model a book with the four fields we care about, and derive its `Schema`:

```scala
case class Book(title: String, authors: List[String], price: BigDecimal, tags: List[String])

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}
```

`JsonSelection#as[Book]` decodes one selected value, and `JsonSelection#asAll[Book]` decodes all of them. To decode the first book, and then every book:

```scala
bookstore.get(p".books[0]").as[Book]
// res25: Either[SchemaError, Book] = Right(
//   Book(
//     title = "Zionomicon",
//     authors = List("John A. De Goes", "Adam Fraser", "Milad Khajavi"),
//     price = 49.99,
//     tags = List("zio", "effects")
//   )
// )

println(bookstore.get(p".books[*]").asAll[Book].map(_.map(_.title)))
// Right(Chunk(Zionomicon,Programming in Scala,Functional Programming in Scala))
```

The decoder ignores fields that `Book` doesn't declare, such as `stock`, so the case class can model only the part of the payload you read. `Json#as[A]` works the same way when you already hold a `Json` value, and a missing `List` field decodes as an empty list.

When the sub-tree doesn't match the case class, the result is a `Left`. Its path is relative to the decoded value: for a book whose price is a string, the error points at `.price`, not at the book's place in a larger document:

```scala
val badBook: Json = Json.parse("""{"title": "Functional Programming in Scala", "price": "44.99"}""").getOrElse(Json.Null)
```

Decoding it with `Json#as` reports the price:

```scala
println(badBook.as[Book])
// Left(zio.blocks.schema.SchemaError: expected Json.Number at: .price)
```

`Json#as` stops at the first error. To report every problem in a payload at once, convert the tree to a `DynamicValue` with `Json#toDynamicValue` and decode it with `Schema#fromDynamicValue`, the approach the guide [How to Define Validated Newtypes and Collect Every Validation Error](./validated-newtypes.md) uses to collect every decode error:

```scala
println(Schema[Book].fromDynamicValue(badBook.toDynamicValue))
// Left(zio.blocks.schema.SchemaError: Expected BigDecimal at: .price
// Missing field 'authors' at: .
// Missing field 'tags' at: .)
```

This route is stricter than `Json#as`, a known difference in 0.0.56: besides the wrong price, it reports the missing `authors` and `tags` lists, which `Json#as` decodes as empty lists. It also rejects a missing `Option` field or a missing field that has a default value, which `Json#as` fills with `None` or the default. Send those fields explicitly when you decode this way.

To go the other way and put a typed value back into a document, `Json.from(value)` encodes it with its `Schema`. To replace the first book with a copy that has a new price:

```scala
val zionomicon: Book = bookstore.get(p".books[0]").as[Book].getOrElse(sys.error("no first book"))

val reprinted: Json = bookstore.set(p".books[0]", Json.from(zionomicon.copy(price = BigDecimal("39.99"))))
```

The first element is now the encoded `Book`:

```scala
println(reprinted.get(p".books[0]").one.map(_.print))
// Right({"title":"Zionomicon","authors":["John A. De Goes","Adam Fraser","Milad Khajavi"],"price":39.99,"tags":["zio","effects"]})
```

The encoded book replaces the whole element, so fields that `Book` doesn't declare, such as `stock`, are gone from it. To change one field and keep the rest, use `Json#set` on that field's path instead.

## Coming From circe or ujson

If you come from circe's cursors and optics, or from ujson, most operations have a direct counterpart. This table covers the tree operations on this page; for moving derived codecs, see [Coming from circe, zio-json or jsoniter-scala](./json-library-migration.md).

| Task                     | circe / ujson                                                 | ZIO Blocks                                              |
| ------------------------ | ------------------------------------------------------------- | ------------------------------------------------------- |
| Parse text               | `io.circe.parser.parse(text)`, `ujson.read(text)`             | `Json.parse(text)`                                      |
| Read one value           | `json.hcursor.downField("books").downN(0).get[BigDecimal]("price")` | `json.get(p".books[0].price").as[BigDecimal]`     |
| Read by runtime index    | `ujson` `json("books")(i)("price").num`                       | `json.get("books")(i).get("price").as[BigDecimal]`      |
| Change every element     | circe-optics `root.books.each.price.bigDecimal.modify(f)(json)` | `json.modify(p".books[*].price")(f)`                  |
| Remove a field           | `json.hcursor.downField("stock").delete`                      | `json.delete(p".stock")`                                |
| Print compact or pretty  | `json.noSpaces`, `json.spaces2`                               | `json.print`, `json.print(WriterConfig.withIndentionStep(2))` |
| Decode a sub-tree        | `cursor.as[Book]`                                             | `json.get(p".books[0]").as[Book]`                       |
| JSON literal             | circe-literal `json"..."`                                     | `json"..."` (Scala 3)                                   |

## Putting It Together

The complete example handles a bookstore webhook end to end: it parses the payload, reads the store name and every title, raises the prices, tags the first book, removes the internal `stock` field, decodes the books into case classes, and prints the document it would publish. It carries its own copy of the document, so it does not need `Bookstore.scala`, the support file that the step examples under [Running the Examples](#running-the-examples) share:

```scala title="schema-examples/src/main/scala/jsonwithoutcaseclasses/CompleteExample.scala"
package jsonwithoutcaseclasses

/**
 * How to Read and Modify JSON Without Case Classes — Putting It Together
 *
 * Handles a bookstore webhook end to end: parses the payload, reads the store
 * name and every title, raises the prices by 10%, tags the first book, removes
 * the internal `stock` field, decodes the books into case classes, and prints
 * the document it would publish.
 *
 * Run with: cd schema-examples && sbt "runMain
 * jsonwithoutcaseclasses.CompleteExample"
 */
object CompleteExample {

  case class Book(title: String, authors: List[String], price: BigDecimal, tags: List[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  val payload: String =
    """{
      |  "store": "Lambda Books",
      |  "currency": "USD",
      |  "books": [
      |    {
      |      "title": "Zionomicon",
      |      "authors": ["John A. De Goes", "Adam Fraser", "Milad Khajavi"],
      |      "price": 49.99,
      |      "tags": ["zio", "effects"],
      |      "stock": 12
      |    },
      |    {
      |      "title": "Programming in Scala",
      |      "authors": ["Martin Odersky", "Lex Spoon", "Bill Venners", "Frank Sommers"],
      |      "price": 59.99,
      |      "tags": ["scala", "language"],
      |      "stock": 0
      |    },
      |    {
      |      "title": "Functional Programming in Scala",
      |      "authors": ["Michael Pilquist", "Rúnar Bjarnason", "Paul Chiusano"],
      |      "price": 44.99,
      |      "tags": ["fp"]
      |    }
      |  ]
      |}""".stripMargin

  def main(args: Array[String]): Unit = {
    val bookstore: Json = Json.parse(payload) match {
      case Right(json) => json
      case Left(error) => throw error
    }

    println(bookstore.get(p".store").as[String])
    println(bookstore.get(p".books[*].title").asAll[String])

    val raised: Json = bookstore.modify(p".books[*].price") {
      case Json.Number(price) =>
        Json.Number((price * BigDecimal("1.1")).setScale(2, BigDecimal.RoundingMode.HALF_UP))
      case other => other
    }

    val tagged: Json = raised.modify(p".books[0].tags") {
      case Json.Array(tags) => Json.Array(tags :+ Json.String("bestseller"))
      case other            => other
    }

    val published: Json = tagged.modify(p".books[*]")(_.delete(p".stock"))

    println(published.get(p".books[*]").asAll[Book])

    println(published.print(WriterConfig.withIndentionStep(2)))
  }
}
```

## Running the Examples

All examples in this guide have runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

The step examples share the bookstore document, which `Bookstore.scala` next to them parses once. The complete example has its own copy and doesn't need it.

<details>
  <summary>schema-examples/src/main/scala/jsonwithoutcaseclasses/Bookstore.scala</summary>

```scala title="schema-examples/src/main/scala/jsonwithoutcaseclasses/Bookstore.scala" showLineNumbers
package jsonwithoutcaseclasses

/**
 * How to Read and Modify JSON Without Case Classes — The Bookstore Document
 *
 * The bookstore document that the step examples share: the JSON text and the
 * parsed `bookstore` tree. This file has no main method.
 */
object Bookstore {

  val bookstoreText: String =
    """{
      |  "store": "Lambda Books",
      |  "currency": "USD",
      |  "books": [
      |    {
      |      "title": "Zionomicon",
      |      "authors": ["John A. De Goes", "Adam Fraser", "Milad Khajavi"],
      |      "price": 49.99,
      |      "tags": ["zio", "effects"],
      |      "stock": 12
      |    },
      |    {
      |      "title": "Programming in Scala",
      |      "authors": ["Martin Odersky", "Lex Spoon", "Bill Venners", "Frank Sommers"],
      |      "price": 59.99,
      |      "tags": ["scala", "language"],
      |      "stock": 0
      |    },
      |    {
      |      "title": "Functional Programming in Scala",
      |      "authors": ["Michael Pilquist", "Rúnar Bjarnason", "Paul Chiusano"],
      |      "price": 44.99,
      |      "tags": ["fp"]
      |    }
      |  ]
      |}""".stripMargin

  val bookstore: Json = Json.parse(bookstoreText) match {
    case Right(json) => json
    case Left(error) => throw error
  }
}
```

</details>

### Parse JSON Text

This example parses the bookstore document and prints the errors for two malformed inputs.

<details>
  <summary>schema-examples/src/main/scala/jsonwithoutcaseclasses/Step1ParseExample.scala</summary>

```scala title="schema-examples/src/main/scala/jsonwithoutcaseclasses/Step1ParseExample.scala" showLineNumbers
package jsonwithoutcaseclasses

/**
 * How to Read and Modify JSON Without Case Classes — Parse JSON Text
 *
 * Parses the bookstore document with Json.parse, and prints the errors for a
 * trailing comma and for a missing comma between two fields.
 *
 * Run with: cd schema-examples && sbt "runMain
 * jsonwithoutcaseclasses.Step1ParseExample"
 */
object Step1ParseExample {
  def main(args: Array[String]): Unit = {
    val bookstore: Json = Json.parse(Bookstore.bookstoreText) match {
      case Right(json) => json
      case Left(error) => throw error
    }

    println(bookstore.print(WriterConfig.withIndentionStep(2)))

    println(Json.parse("""{"title": "Zionomicon", "price": 49.99,}"""))

    println(Json.parse("""{"title": "Zionomicon" "price": 49.99}"""))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain jsonwithoutcaseclasses.Step1ParseExample"
```

### Read One Value Deep in the Tree

This example reads single values by key, by index and by path, and prints the errors for a missing field and a wrong type.

<details>
  <summary>schema-examples/src/main/scala/jsonwithoutcaseclasses/Step2ReadOneValueExample.scala</summary>

```scala title="schema-examples/src/main/scala/jsonwithoutcaseclasses/Step2ReadOneValueExample.scala" showLineNumbers
package jsonwithoutcaseclasses

/**
 * How to Read and Modify JSON Without Case Classes — Read One Value Deep in the
 * Tree
 *
 * Reads single values by key, by index and by path, and prints the errors for a
 * missing field, a wrong type and an index out of range.
 *
 * Run with: cd schema-examples && sbt "runMain
 * jsonwithoutcaseclasses.Step2ReadOneValueExample"
 */
object Step2ReadOneValueExample {
  def main(args: Array[String]): Unit = {
    val bookstore: Json = Bookstore.bookstore

    println(bookstore.get("books")(0).get("price").as[BigDecimal])

    println(bookstore.get(p".books[0].price").as[BigDecimal])

    println(bookstore.get(p".books[1].authors[0]").as[String])

    println(bookstore.get(p".books[1].isbn").as[String])

    println(bookstore.get(p".books[0].title").as[BigDecimal])

    println(bookstore.get(p".books[5].title").as[String])

    println(bookstore.get(p".books[2].stock").as[Int].getOrElse(0))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain jsonwithoutcaseclasses.Step2ReadOneValueExample"
```

### Select Many Values at Once

This example selects every title and price with `[*]`, filters the books by price, and reads a field that one book lacks.

<details>
  <summary>schema-examples/src/main/scala/jsonwithoutcaseclasses/Step3SelectManyExample.scala</summary>

```scala title="schema-examples/src/main/scala/jsonwithoutcaseclasses/Step3SelectManyExample.scala" showLineNumbers
package jsonwithoutcaseclasses

/**
 * How to Read and Modify JSON Without Case Classes — Select Many Values at Once
 *
 * Selects every title and price with the [*] wildcard, filters the books by
 * price, and reads a field that one book lacks.
 *
 * Run with: cd schema-examples && sbt "runMain
 * jsonwithoutcaseclasses.Step3SelectManyExample"
 */
object Step3SelectManyExample {
  def main(args: Array[String]): Unit = {
    val bookstore: Json = Bookstore.bookstore

    println(bookstore.get(p".books[*].title").asAll[String])

    println(bookstore.get(p".books[*].price").asAll[BigDecimal].map(_.sum))

    println(
      bookstore
        .get(p".books[*]")
        .filter(book => book.get("price").as[BigDecimal].exists(_ < 50))
        .get("title")
        .asAll[String]
    )

    println(bookstore.get(p".books[*].title").as[String])

    println(bookstore.get(p".books[*].stock").asAll[Int])

    println(bookstore.get("books").one.map(_.elements.map(book => book.get("stock").as[Int].getOrElse(0))))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain jsonwithoutcaseclasses.Step3SelectManyExample"
```

### Change Values in the Tree

This example raises the prices, adds a tag, sets and inserts fields, removes `stock` from every book, and shows how `Json#setOrFail` catches a misspelled path.

<details>
  <summary>schema-examples/src/main/scala/jsonwithoutcaseclasses/Step4ChangeTreeExample.scala</summary>

```scala title="schema-examples/src/main/scala/jsonwithoutcaseclasses/Step4ChangeTreeExample.scala" showLineNumbers
package jsonwithoutcaseclasses

/**
 * How to Read and Modify JSON Without Case Classes — Change Values in the Tree
 *
 * Raises every price by 10%, adds a tag, sets and inserts fields, removes
 * `stock` from every book, and shows how setOrFail catches a misspelled path.
 *
 * Run with: cd schema-examples && sbt "runMain
 * jsonwithoutcaseclasses.Step4ChangeTreeExample"
 */
object Step4ChangeTreeExample {
  def main(args: Array[String]): Unit = {
    val bookstore: Json = Bookstore.bookstore

    val raised: Json = bookstore.modify(p".books[*].price") {
      case Json.Number(price) =>
        Json.Number((price * BigDecimal("1.1")).setScale(2, BigDecimal.RoundingMode.HALF_UP))
      case other => other
    }

    println(raised.get(p".books[*].price").asAll[BigDecimal])

    val tagged: Json = raised.modify(p".books[0].tags") {
      case Json.Array(tags) => Json.Array(tags :+ Json.String("bestseller"))
      case other            => other
    }

    val withAvailability: Json = tagged
      .set(p".store", Json.String("Lambda Books Online"))
      .modify(p".books[*]") { book =>
        book.insert(p".available", Json.Boolean(book.get("stock").as[Int].exists(_ > 0)))
      }

    val published: Json = withAvailability.modify(p".books[*]")(_.delete(p".stock"))

    println(published.get(p".books[*]").toChunk.map(_.print).mkString("\n"))

    println(published.set(p".curency", Json.String("EUR")) == published)

    println(published.setOrFail(p".curency", Json.String("EUR")))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain jsonwithoutcaseclasses.Step4ChangeTreeExample"
```

### Print JSON Text

This example prints a book as compact and as indented JSON, and compares two documents whose keys come in a different order.

<details>
  <summary>schema-examples/src/main/scala/jsonwithoutcaseclasses/Step5PrintExample.scala</summary>

```scala title="schema-examples/src/main/scala/jsonwithoutcaseclasses/Step5PrintExample.scala" showLineNumbers
package jsonwithoutcaseclasses

/**
 * How to Read and Modify JSON Without Case Classes — Print JSON Text
 *
 * Prints the first book as compact and as indented JSON, and compares two
 * documents whose keys come in a different order.
 *
 * Run with: cd schema-examples && sbt "runMain
 * jsonwithoutcaseclasses.Step5PrintExample"
 */
object Step5PrintExample {
  def main(args: Array[String]): Unit = {
    val bookstore: Json = Bookstore.bookstore

    val firstBook: Json = bookstore.get(p".books[0]").one.getOrElse(Json.Null)

    println(firstBook.print)

    println(firstBook.print(WriterConfig.withIndentionStep(2)))

    println(
      Json.parse("""{"title": "Zionomicon", "price": 49.99}""") ==
        Json.parse("""{"price": 49.99, "title": "Zionomicon"}""")
    )

    println(Json.parse("""["zio", "effects"]""") == Json.parse("""["effects", "zio"]"""))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain jsonwithoutcaseclasses.Step5PrintExample"
```

### Build JSON in Code

This example builds a new book with the `Json` constructors and with the `json"..."` interpolator, and appends it to the document. It uses `json"..."`, so it needs Scala 3, which the `schema-examples` build uses.

<details>
  <summary>schema-examples/src/main/scala/jsonwithoutcaseclasses/Step6BuildJsonExample.scala</summary>

```scala title="schema-examples/src/main/scala/jsonwithoutcaseclasses/Step6BuildJsonExample.scala" showLineNumbers
package jsonwithoutcaseclasses

/**
 * How to Read and Modify JSON Without Case Classes — Build JSON in Code
 *
 * Builds a fourth book with the Json constructors and with the json"..."
 * interpolator, checks that both give the same tree, and appends it to the
 * document. The interpolator works only on Scala 3 in this release: on Scala
 * 2.13 it fails to compile (known bug in 0.0.56), so use the constructors
 * there.
 *
 * Run with: cd schema-examples && sbt "runMain
 * jsonwithoutcaseclasses.Step6BuildJsonExample"
 */
object Step6BuildJsonExample {
  def main(args: Array[String]): Unit = {
    val bookstore: Json = Bookstore.bookstore

    val handsOnScala: Json = Json.Object(
      "title"   -> Json.String("Hands-on Scala Programming"),
      "authors" -> Json.Array(Json.String("Li Haoyi")),
      "price"   -> Json.Number(BigDecimal("39.99")),
      "tags"    -> Json.Array(Json.String("scala"), Json.String("tooling"))
    )

    val fourBooks: Json = bookstore.modify(p".books") {
      case Json.Array(books) => Json.Array(books :+ handsOnScala)
      case other             => other
    }

    println(fourBooks.get(p".books[*].title").asAll[String])

    val title = "Hands-on Scala Programming"
    val price = BigDecimal("39.99")
    val tags  = List("scala", "tooling")

    val handsOnScalaFromText: Json =
      json"""{"title": $title, "authors": ["Li Haoyi"], "price": $price, "tags": $tags}"""

    println(handsOnScalaFromText.print)

    println(handsOnScalaFromText == handsOnScala)
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain jsonwithoutcaseclasses.Step6BuildJsonExample"
```

### Decode Part of the Tree Into a Case Class

This example decodes one book and every book into `Book`, prints the errors for a malformed book from both decoders, and writes an updated book back into the document.

<details>
  <summary>schema-examples/src/main/scala/jsonwithoutcaseclasses/Step7DecodeSubTreeExample.scala</summary>

```scala title="schema-examples/src/main/scala/jsonwithoutcaseclasses/Step7DecodeSubTreeExample.scala" showLineNumbers
package jsonwithoutcaseclasses

/**
 * How to Read and Modify JSON Without Case Classes — Decode Part of the Tree
 * Into a Case Class
 *
 * Decodes one book and every book into Book, prints the errors for a malformed
 * book from both decoders, and writes an updated book back into the document.
 *
 * Run with: cd schema-examples && sbt "runMain
 * jsonwithoutcaseclasses.Step7DecodeSubTreeExample"
 */
object Step7DecodeSubTreeExample {

  case class Book(title: String, authors: List[String], price: BigDecimal, tags: List[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  def main(args: Array[String]): Unit = {
    val bookstore: Json = Bookstore.bookstore

    println(bookstore.get(p".books[0]").as[Book])

    println(bookstore.get(p".books[*]").asAll[Book].map(_.map(_.title)))

    val badBook: Json =
      Json.parse("""{"title": "Functional Programming in Scala", "price": "44.99"}""").getOrElse(Json.Null)

    println(badBook.as[Book])

    println(Schema[Book].fromDynamicValue(badBook.toDynamicValue))

    val zionomicon: Book = bookstore.get(p".books[0]").as[Book].getOrElse(sys.error("no first book"))

    val reprinted: Json = bookstore.set(p".books[0]", Json.from(zionomicon.copy(price = BigDecimal("39.99"))))

    println(reprinted.get(p".books[0]").one.map(_.print))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain jsonwithoutcaseclasses.Step7DecodeSubTreeExample"
```

### The Complete Example

Run the complete example from the `schema-examples` directory:

```bash
sbt "runMain jsonwithoutcaseclasses.CompleteExample"
```

### Compile All Examples

To compile every example in the project without running any of them, run this from the `schema-examples` directory:

```bash
sbt compile
```

## Going Further

- The [Json reference page](../reference/schema/built-in-codecs/json/json.md) covers the rest of the tree API: `transformUp` and `transformKeys` for rewriting a whole document, `prune` and `retain`, `merge` with its strategies, and folds.
- The [JsonSelection reference page](../reference/schema/built-in-codecs/json/json-selection.md) lists every way to filter, combine and extract a selection.
- The [path interpolator reference](../reference/schema/path-interpolator.md) documents the full `p"..."` syntax, including index ranges such as `[0:2]` and several indices such as `[0,2]`.
- To compute the difference between two documents and apply it as a patch, see [JSON Patch](../reference/schema/built-in-codecs/json/json-patch.md).
- When the shape is known and stable, derive a codec for the whole type instead: [How to Customize JSON Encoding](./customize-json-encoding.md) shows how to control field names, optional fields and enums.

---

## How to Send Compact Data to an LLM with TOON and Parse the Reply

In this guide, we will send a list of Scala books to a large language model (LLM) as compact TOON, ask it to recommend one, and decode its reply into a case class. By the end, you will have a prompt builder that encodes records in TOON's tabular layout and a decoder that turns the model's TOON answer into a typed `Recommendation`, or into a clear error when the answer is malformed.

We take an incremental approach:

- Encode a catalog of books as tabular TOON and compare its size with JSON
- See why `ArrayFormat.Tabular` must be set explicitly
- Pick a delimiter that keeps values unquoted
- Build the prompt string
- Decode the model's reply, handle malformed replies and strip the prose around them

No LLM is called anywhere in this guide: the model's reply is a fixed string, so every output you see is deterministic.

## The Problem

LLM APIs bill and limit by tokens, and a prompt that carries data spends most of its tokens on that data. JSON repeats every field name in every record and wraps every string in quotes, so a list of 50 records pays for its field names 50 times. TOON (Token-Oriented Object Notation) sends a uniform list of records as a table instead: the field names appear once in a header, and each record becomes one row of values.

The reply is the other half. Asking the model to answer in a structured format only helps if the answer is checked: a model can rename a field, leave one out, or wrap its answer in friendly prose. Without a decoder, that check is a hand-written parser.

For example, without a codec library, preparing the prompt and reading the reply looks like this:

```scala
case class Book(title: String, firstAuthor: String, year: Int, tags: String)
case class Recommendation(title: String, reason: String)

// Building the table by hand: header, quoting and escaping are all on us
def toTable(books: List[Book]): String =
  books
    .map(b => s"${b.title},${b.firstAuthor},${b.year},${b.tags}") // breaks when a value contains a comma
    .mkString("books{title,firstAuthor,year,tags}:\n  ", "\n  ", "")

// Parsing the reply by hand: every field, every failure case
def parseReply(reply: String): Either[String, Recommendation] = {
  val fields = reply.linesIterator.flatMap { line =>
    line.split(":", 2) match {
      case Array(key, value) => Some(key.trim -> value.trim)
      case _                 => None
    }
  }.toMap
  for {
    title  <- fields.get("title").toRight("missing title")
    reason <- fields.get("reason").toRight("missing reason")
  } yield Recommendation(title, reason)
}
```

This approach doesn't scale: the table writer silently corrupts any value that contains a comma, the parser silently drops lines it doesn't understand, and both must be rewritten whenever a field is added.

In this guide, we derive one TOON codec per type from its `Schema` instead. The codec writes the table, quotes values only when needed, and reports every malformed reply as a `SchemaError`.

## Prerequisites

Add the TOON module to your `build.sbt`. It depends on the schema module, so you don't need to add that separately:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema-toon" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema-toon" % "0.0.56"
```

All code in this guide uses these two imports:

```scala

```

This guide assumes you can derive a `Schema` for a case class and read a `SchemaError`. If not, start with the tutorial [Your First Schema](./your-first-schema.md).

## The Book Catalog

Our catalog holds three books. Each `Book` has only primitive fields, which is what lets TOON lay the list out as a table. The `tags` field is a single comma-separated string rather than a `List[String]` for that reason. A `Catalog` wraps the list in a named field, so the TOON header reads `books[3]{...}:`, a label the model can refer to. To define the catalog:

```scala
case class Book(title: String, firstAuthor: String, year: Int, tags: String)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

case class Catalog(books: List[Book])

object Catalog {
  implicit val schema: Schema[Catalog] = Schema.derived
}

val catalog = Catalog(
  List(
    Book("Zionomicon", "John A. De Goes", 2024, "zio, effects, concurrency"),
    Book("Programming in Scala", "Martin Odersky", 2008, "scala, language, fundamentals"),
    Book("Functional Programming in Scala", "Paul Chiusano", 2014, "fp, scala, exercises")
  )
)
```

:::warning
The tabular layout needs every field of the record to be a primitive value: a string, a number, a boolean and so on. A field that holds a list or a nested record breaks the table. Flatten such fields before encoding, as `tags` does here.
:::

## Encoding the Records for a Prompt

`ToonCodecDeriver` is the deriver behind `ToonFormat`, and its `ToonCodecDeriver#with*` methods return a configured copy. We call `ToonCodecDeriver#withArrayFormat(ArrayFormat.Tabular)` to ask for the tabular layout, derive a codec for `Catalog` with it, and encode the catalog with `ToonCodec#encodeToString`:

```scala
val tabular = ToonCodecDeriver.withArrayFormat(ArrayFormat.Tabular)

val catalogCodec: ToonCodec[Catalog] = Catalog.schema.derive(tabular)

val catalogToon = catalogCodec.encodeToString(catalog)
```

The result is the text we put in the prompt:

```scala
println(catalogToon)
// books[3]{title,firstAuthor,year,tags}:
//   Zionomicon,John A. De Goes,2024,"zio, effects, concurrency"
//   Programming in Scala,Martin Odersky,2008,"scala, language, fundamentals"
//   Functional Programming in Scala,Paul Chiusano,2014,"fp, scala, exercises"
```

The header `books[3]{title,firstAuthor,year,tags}:` names the field, the number of rows and the column names, once. Each line below it is one book. The `tags` values contain commas, so the encoder quotes them to keep them apart from the commas that separate columns.

For comparison, here is the same catalog as JSON, from the `toJsonString` extension method that the schema provides:

```scala
val catalogJson = catalog.toJsonString
```

To print the JSON text and compare its length with the TOON text:

```scala
println(catalogJson)
// {"books":[{"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024,"tags":"zio, effects, concurrency"},{"title":"Programming in Scala","firstAuthor":"Martin Odersky","year":2008,"tags":"scala, language, fundamentals"},{"title":"Functional Programming in Scala","firstAuthor":"Paul Chiusano","year":2014,"tags":"fp, scala, exercises"}]}
println(s"JSON: ${catalogJson.length} characters, TOON: ${catalogToon.length} characters")
// JSON: 344 characters, TOON: 251 characters
```

The TOON text is shorter because it states each field name once instead of once per book, and drops most quotes and braces. Characters are only a rough proxy for cost: the exact token count depends on the model's tokenizer, so measure with your provider's token counter before relying on a number.

The savings grow with the number of rows and shrink with the shape of the data. TOON pays off for long lists of uniform, flat records. For deeply nested or irregular data, where few records share the same fields, it saves little over compact JSON.

## Why `ArrayFormat.Tabular` Must Be Set

`ArrayFormat` has four cases: `Auto`, `Tabular`, `Inline` and `List`. The default deriver, the one `derive(ToonFormat)` uses, is set to `Auto`. In practice, `Auto` writes a list of records in list form, one `- ` item per record with every field name repeated, even when the records are flat and uniform. Only `ArrayFormat.Tabular` produces the table.

To compare the two layouts, encode the same catalog with both settings:

```scala
val autoToon = Catalog.schema.derive(ToonFormat).encodeToString(catalog)
```

To print both encodings and their lengths side by side:

```scala
println("ArrayFormat.Auto:")
// ArrayFormat.Auto:
println(autoToon)
// books[3]:
//   - title: Zionomicon
//     firstAuthor: John A. De Goes
//     year: 2024
//     tags: zio, effects, concurrency
//   - title: Programming in Scala
//     firstAuthor: Martin Odersky
//     year: 2008
//     tags: scala, language, fundamentals
//   - title: Functional Programming in Scala
//     firstAuthor: Paul Chiusano
//     year: 2014
//     tags: fp, scala, exercises
println()
// 
println("ArrayFormat.Tabular:")
// ArrayFormat.Tabular:
println(catalogToon)
// books[3]{title,firstAuthor,year,tags}:
//   Zionomicon,John A. De Goes,2024,"zio, effects, concurrency"
//   Programming in Scala,Martin Odersky,2008,"scala, language, fundamentals"
//   Functional Programming in Scala,Paul Chiusano,2014,"fp, scala, exercises"
println()
// 
println(s"Auto: ${autoToon.length} characters, Tabular: ${catalogToon.length} characters")
// Auto: 354 characters, Tabular: 251 characters
```

The list form is valid TOON and decodes back to the same `Catalog`, but it repeats `title:`, `firstAuthor:`, `year:` and `tags:` for every book. That repetition is exactly what we wanted to avoid, so always set `ArrayFormat.Tabular` when encoding records for a prompt.

## Choosing a Delimiter

With the default comma delimiter, any value that contains a comma must be quoted, as the `tags` column showed. `ToonCodecDeriver#withDelimiter` picks a different column separator: `Delimiter.Comma` (the default), `Delimiter.Tab` or `Delimiter.Pipe`. A fourth case, `Delimiter.None`, exists for the reader's internal use and isn't meant for encoding.

To see how each delimiter changes the encoded table:

```scala
for (delimiter <- List(Delimiter.Comma, Delimiter.Pipe, Delimiter.Tab)) {
  val codec = Catalog.schema.derive(tabular.withDelimiter(delimiter))
  println(s"$delimiter:")
  println(codec.encodeToString(catalog))
  println()
}
// Comma:
// books[3]{title,firstAuthor,year,tags}:
//   Zionomicon,John A. De Goes,2024,"zio, effects, concurrency"
//   Programming in Scala,Martin Odersky,2008,"scala, language, fundamentals"
//   Functional Programming in Scala,Paul Chiusano,2014,"fp, scala, exercises"
// 
// Pipe:
// books[3|]{title|firstAuthor|year|tags}:
//   Zionomicon|John A. De Goes|2024|zio, effects, concurrency
//   Programming in Scala|Martin Odersky|2008|scala, language, fundamentals
//   Functional Programming in Scala|Paul Chiusano|2014|fp, scala, exercises
// 
// Tab:
// books[3	]{title	firstAuthor	year	tags}:
//   Zionomicon	John A. De Goes	2024	zio, effects, concurrency
//   Programming in Scala	Martin Odersky	2008	scala, language, fundamentals
//   Functional Programming in Scala	Paul Chiusano	2014	fp, scala, exercises
//
```

The pipe and tab versions need no quotes, because no value contains a pipe or a tab. The header records the delimiter, as in `books[3|]` or `books[3\t]` (with a real tab character), so a decoder knows how to split the rows. Pick the delimiter that is least likely to appear in your values:

- `Delimiter.Comma` suits numbers, IDs and short words.
- `Delimiter.Pipe` suits free text with commas, such as titles, tags and descriptions, and stays readable.
- `Delimiter.Tab` also avoids quoting free text, but tabs are invisible in logs and easy to lose when text is copied.

The codec reads its own output back without any extra setting:

```scala
val pipeCodec = Catalog.schema.derive(tabular.withDelimiter(Delimiter.Pipe))
```

To check that the codec reads its own output back:

```scala
println(pipeCodec.decode(pipeCodec.encodeToString(catalog)) == Right(catalog))
// true
```

## Building the Prompt

A prompt is a plain string, so we build it with ordinary Scala. It holds three things: a short instruction, the TOON block, and a description of the reply we want. Naming the reply's fields and showing their shape makes it far more likely that the model answers in a form our decoder accepts. To assemble the prompt:

```scala
val catalogPipeToon = pipeCodec.encodeToString(catalog)

def buildPrompt(catalogToon: String, request: String): String =
  List(
    "You recommend books from the catalog below. The catalog is in TOON format.",
    "",
    catalogToon,
    "",
    s"Request: $request",
    "",
    "Reply in TOON with exactly two fields and no other text:",
    "title: ",
    "reason: <one sentence>"
  ).mkString("\n")

val prompt = buildPrompt(catalogPipeToon, "I know Scala and want to learn effect systems.")
```

This is the text we would send to the model:

```scala
println(prompt)
// You recommend books from the catalog below. The catalog is in TOON format.
// 
// books[3|]{title|firstAuthor|year|tags}:
//   Zionomicon|John A. De Goes|2024|zio, effects, concurrency
//   Programming in Scala|Martin Odersky|2008|scala, language, fundamentals
//   Functional Programming in Scala|Paul Chiusano|2014|fp, scala, exercises
// 
// Request: I know Scala and want to learn effect systems.
// 
// Reply in TOON with exactly two fields and no other text:
// title: 
// reason: <one sentence>
```

## Decoding the Model's Reply

The model's answer is described by a case class like any other type. We derive its codec with `ToonCodecDeriver#withRejectExtraFields(true)`, so any key the case class doesn't declare is an error instead of being ignored. The reason for that choice shows up below, when the model adds prose. To define the reply type and its strict codec:

```scala
case class Recommendation(title: String, reason: String)

object Recommendation {
  implicit val schema: Schema[Recommendation] = Schema.derived
}

val replyCodec: ToonCodec[Recommendation] =
  Recommendation.schema.derive(ToonCodecDeriver.withRejectExtraFields(true))
```

Suppose the model answered with this text. `ToonCodec#decode` returns an `Either[SchemaError, Recommendation]`:

```scala
val reply =
  """title: Zionomicon
    |reason: It teaches ZIO, an effect system for Scala, from first steps to concurrency.""".stripMargin
```

To decode the reply:

```scala
println(replyCodec.decode(reply))
// Right(Recommendation(Zionomicon,It teaches ZIO, an effect system for Scala, from first steps to concurrency.))
```

A single record's value runs to the end of its line, so the commas in `reason` need no quotes.

### Reading a Malformed Reply

Models don't always follow instructions. `ToonCodec#decode` turns each kind of mistake into a `Left`, and the error's `message` names the field at fault and its path (`.` is the top-level record):

```scala
val malformedReplies = List(
  "missing field"     -> "title: Zionomicon",
  "renamed field"     -> "book: Zionomicon\nreason: It teaches ZIO.",
  "extra field"       -> "title: Zionomicon\nreason: It teaches ZIO.\nconfidence: 0.9",
  "prose before TOON" -> "Sure! Here is my pick:\ntitle: Zionomicon\nreason: It teaches ZIO."
)
```

To decode each malformed reply and print its error:

```scala
for ((label, text) <- malformedReplies)
  replyCodec.decode(text) match {
    case Left(error)  => println(s"$label: ${error.message}")
    case Right(value) => println(s"$label: decoded $value")
  }
// missing field: Missing required field: reason at: .
// renamed field: Unexpected field: book at: .
// extra field: Unexpected field: confidence at: .
// prose before TOON: Unexpected field: Sure! Here is my pick at: .
```

The last case is the reason for `ToonCodecDeriver#withRejectExtraFields(true)`. The line `Sure! Here is my pick:` ends with a colon, so TOON reads it as a key named `Sure! Here is my pick`. With the default deriver, unknown keys are skipped, and that prose would be accepted without a word:

```scala
val lenientCodec = Recommendation.schema.derive(ToonFormat)
```

To decode the prose-prefixed reply with the lenient codec:

```scala
println(lenientCodec.decode("Sure! Here is my pick:\ntitle: Zionomicon\nreason: It teaches ZIO."))
// Right(Recommendation(Zionomicon,It teaches ZIO.))
```

Rejecting extra fields turns that silent pass into an error you can log, retry or show.

### Stripping Prose and Code Fences

A common reply shape wraps the TOON in a Markdown code fence, with prose before and after it. That text isn't TOON, so `ToonCodec#decode` rejects it:

```scala
val chattyReply =
  """Happy to help! Here is my recommendation:
    |
    |```toon
    |title: Zionomicon
    |reason: It teaches ZIO, an effect system for Scala, from first steps to concurrency.
    |```
    |
    |Enjoy the book!""".stripMargin
```

To decode the chatty reply with the strict codec:

```scala
println(replyCodec.decode(chattyReply).left.map(_.message))
// Left(Unexpected field: Happy to help! Here is my recommendation at: .)
```

Plain string handling is enough to cut out the fenced block before decoding. When there is no fence, the helper returns the trimmed reply unchanged:

```scala
def extractToon(reply: String): String = {
  val fence = "```"
  val open  = reply.indexOf(fence)
  if (open < 0) reply.trim
  else {
    val bodyStart = reply.indexOf('\n', open) + 1
    val close     = reply.indexOf(fence, bodyStart)
    reply.substring(bodyStart, if (close < 0) reply.length else close).trim
  }
}
```

Now the chatty reply decodes:

```scala
println(replyCodec.decode(extractToon(chattyReply)))
// Right(Recommendation(Zionomicon,It teaches ZIO, an effect system for Scala, from first steps to concurrency.))
```

Prose without a fence is left in place on purpose. Guessing where TOON starts in free text is fragile, and the strict codec already reports it as an error.

## Putting It Together

This example combines every step: it encodes the catalog as pipe-delimited tabular TOON, builds the prompt, and decodes a fixed model reply, stripping its code fence and rejecting extra fields:

```scala title="schema-examples/src/main/scala/llmdatawithtoon/CompleteExample.scala"
package llmdatawithtoon

/**
 * How to Send Compact Data to an LLM with TOON and Parse the Reply — Complete
 * Example
 *
 * Encodes the catalog as pipe-delimited tabular TOON, builds the prompt, and
 * decodes a fixed model reply, stripping its code fence and rejecting extra
 * fields.
 *
 * Run with: cd schema-examples && sbt "runMain llmdatawithtoon.CompleteExample"
 */
object CompleteExample extends App {

  case class Book(title: String, firstAuthor: String, year: Int, tags: String)

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalog = Catalog(
    List(
      Book("Zionomicon", "John A. De Goes", 2024, "zio, effects, concurrency"),
      Book("Programming in Scala", "Martin Odersky", 2008, "scala, language, fundamentals"),
      Book("Functional Programming in Scala", "Paul Chiusano", 2014, "fp, scala, exercises")
    )
  )

  val tabular = ToonCodecDeriver.withArrayFormat(ArrayFormat.Tabular)

  val pipeCodec = Catalog.schema.derive(tabular.withDelimiter(Delimiter.Pipe))

  val catalogPipeToon = pipeCodec.encodeToString(catalog)

  def buildPrompt(catalogToon: String, request: String): String =
    List(
      "You recommend books from the catalog below. The catalog is in TOON format.",
      "",
      catalogToon,
      "",
      s"Request: $request",
      "",
      "Reply in TOON with exactly two fields and no other text:",
      "title: ",
      "reason: <one sentence>"
    ).mkString("\n")

  val prompt = buildPrompt(catalogPipeToon, "I know Scala and want to learn effect systems.")

  case class Recommendation(title: String, reason: String)

  object Recommendation {
    implicit val schema: Schema[Recommendation] = Schema.derived
  }

  val replyCodec: ToonCodec[Recommendation] =
    Recommendation.schema.derive(ToonCodecDeriver.withRejectExtraFields(true))

  def extractToon(reply: String): String = {
    val fence = "```"
    val open  = reply.indexOf(fence)
    if (open < 0) reply.trim
    else {
      val bodyStart = reply.indexOf('\n', open) + 1
      val close     = reply.indexOf(fence, bodyStart)
      reply.substring(bodyStart, if (close < 0) reply.length else close).trim
    }
  }

  val chattyReply =
    """Happy to help! Here is my recommendation:
      |
      |```toon
      |title: Zionomicon
      |reason: It teaches ZIO, an effect system for Scala, from first steps to concurrency.
      |```
      |
      |Enjoy the book!""".stripMargin

  println(prompt)

  replyCodec.decode(extractToon(chattyReply)) match {
    case Left(error)           => println(error.message)
    case Right(recommendation) => println(recommendation)
  }
}
```

## Running the Examples

All examples in this guide have corresponding runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

### Encoding the Records for a Prompt

This example encodes the catalog as tabular TOON and compares its length with the JSON encoding.

<details>
  <summary>schema-examples/src/main/scala/llmdatawithtoon/Step1EncodeForPromptExample.scala</summary>

```scala title="schema-examples/src/main/scala/llmdatawithtoon/Step1EncodeForPromptExample.scala" showLineNumbers
package llmdatawithtoon

/**
 * How to Send Compact Data to an LLM with TOON and Parse the Reply — Encoding
 * the Records for a Prompt
 *
 * Encodes the catalog as tabular TOON and compares its length with the JSON
 * encoding.
 *
 * Run with: cd schema-examples && sbt "runMain
 * llmdatawithtoon.Step1EncodeForPromptExample"
 */
object Step1EncodeForPromptExample extends App {

  case class Book(title: String, firstAuthor: String, year: Int, tags: String)

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalog = Catalog(
    List(
      Book("Zionomicon", "John A. De Goes", 2024, "zio, effects, concurrency"),
      Book("Programming in Scala", "Martin Odersky", 2008, "scala, language, fundamentals"),
      Book("Functional Programming in Scala", "Paul Chiusano", 2014, "fp, scala, exercises")
    )
  )

  val tabular = ToonCodecDeriver.withArrayFormat(ArrayFormat.Tabular)

  val catalogCodec: ToonCodec[Catalog] = Catalog.schema.derive(tabular)

  val catalogToon = catalogCodec.encodeToString(catalog)

  println(catalogToon)

  val catalogJson = catalog.toJsonString

  println(catalogJson)
  println(s"JSON: ${catalogJson.length} characters, TOON: ${catalogToon.length} characters")
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain llmdatawithtoon.Step1EncodeForPromptExample"
```

### Why `ArrayFormat.Tabular` Must Be Set

This example prints the catalog with `ArrayFormat.Auto` and with `ArrayFormat.Tabular`, side by side.

<details>
  <summary>schema-examples/src/main/scala/llmdatawithtoon/Step2AutoVsTabularExample.scala</summary>

```scala title="schema-examples/src/main/scala/llmdatawithtoon/Step2AutoVsTabularExample.scala" showLineNumbers
package llmdatawithtoon

/**
 * How to Send Compact Data to an LLM with TOON and Parse the Reply — Why
 * ArrayFormat.Tabular Must Be Set
 *
 * Prints the catalog with ArrayFormat.Auto and with ArrayFormat.Tabular, side
 * by side.
 *
 * Run with: cd schema-examples && sbt "runMain
 * llmdatawithtoon.Step2AutoVsTabularExample"
 */
object Step2AutoVsTabularExample extends App {

  case class Book(title: String, firstAuthor: String, year: Int, tags: String)

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalog = Catalog(
    List(
      Book("Zionomicon", "John A. De Goes", 2024, "zio, effects, concurrency"),
      Book("Programming in Scala", "Martin Odersky", 2008, "scala, language, fundamentals"),
      Book("Functional Programming in Scala", "Paul Chiusano", 2014, "fp, scala, exercises")
    )
  )

  val tabular = ToonCodecDeriver.withArrayFormat(ArrayFormat.Tabular)

  val catalogCodec: ToonCodec[Catalog] = Catalog.schema.derive(tabular)

  val catalogToon = catalogCodec.encodeToString(catalog)

  val autoToon = Catalog.schema.derive(ToonFormat).encodeToString(catalog)

  println("ArrayFormat.Auto:")
  println(autoToon)
  println()
  println("ArrayFormat.Tabular:")
  println(catalogToon)
  println()
  println(s"Auto: ${autoToon.length} characters, Tabular: ${catalogToon.length} characters")
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain llmdatawithtoon.Step2AutoVsTabularExample"
```

### Choosing a Delimiter

This example prints the catalog with the comma, pipe and tab delimiters and decodes the pipe version back.

<details>
  <summary>schema-examples/src/main/scala/llmdatawithtoon/Step3DelimitersExample.scala</summary>

```scala title="schema-examples/src/main/scala/llmdatawithtoon/Step3DelimitersExample.scala" showLineNumbers
package llmdatawithtoon

/**
 * How to Send Compact Data to an LLM with TOON and Parse the Reply — Choosing a
 * Delimiter
 *
 * Prints the catalog with the comma, pipe and tab delimiters and decodes the
 * pipe version back.
 *
 * Run with: cd schema-examples && sbt "runMain
 * llmdatawithtoon.Step3DelimitersExample"
 */
object Step3DelimitersExample extends App {

  case class Book(title: String, firstAuthor: String, year: Int, tags: String)

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalog = Catalog(
    List(
      Book("Zionomicon", "John A. De Goes", 2024, "zio, effects, concurrency"),
      Book("Programming in Scala", "Martin Odersky", 2008, "scala, language, fundamentals"),
      Book("Functional Programming in Scala", "Paul Chiusano", 2014, "fp, scala, exercises")
    )
  )

  val tabular = ToonCodecDeriver.withArrayFormat(ArrayFormat.Tabular)

  for (delimiter <- List(Delimiter.Comma, Delimiter.Pipe, Delimiter.Tab)) {
    val codec = Catalog.schema.derive(tabular.withDelimiter(delimiter))
    println(s"$delimiter:")
    println(codec.encodeToString(catalog))
    println()
  }

  val pipeCodec = Catalog.schema.derive(tabular.withDelimiter(Delimiter.Pipe))

  println(pipeCodec.decode(pipeCodec.encodeToString(catalog)) == Right(catalog))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain llmdatawithtoon.Step3DelimitersExample"
```

### Building the Prompt

This example builds the prompt string around the pipe-delimited catalog and prints it.

<details>
  <summary>schema-examples/src/main/scala/llmdatawithtoon/Step4BuildPromptExample.scala</summary>

```scala title="schema-examples/src/main/scala/llmdatawithtoon/Step4BuildPromptExample.scala" showLineNumbers
package llmdatawithtoon

/**
 * How to Send Compact Data to an LLM with TOON and Parse the Reply — Building
 * the Prompt
 *
 * Builds the prompt string around the pipe-delimited catalog and prints it.
 *
 * Run with: cd schema-examples && sbt "runMain
 * llmdatawithtoon.Step4BuildPromptExample"
 */
object Step4BuildPromptExample extends App {

  case class Book(title: String, firstAuthor: String, year: Int, tags: String)

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalog = Catalog(
    List(
      Book("Zionomicon", "John A. De Goes", 2024, "zio, effects, concurrency"),
      Book("Programming in Scala", "Martin Odersky", 2008, "scala, language, fundamentals"),
      Book("Functional Programming in Scala", "Paul Chiusano", 2014, "fp, scala, exercises")
    )
  )

  val tabular = ToonCodecDeriver.withArrayFormat(ArrayFormat.Tabular)

  val pipeCodec = Catalog.schema.derive(tabular.withDelimiter(Delimiter.Pipe))

  val catalogPipeToon = pipeCodec.encodeToString(catalog)

  def buildPrompt(catalogToon: String, request: String): String =
    List(
      "You recommend books from the catalog below. The catalog is in TOON format.",
      "",
      catalogToon,
      "",
      s"Request: $request",
      "",
      "Reply in TOON with exactly two fields and no other text:",
      "title: ",
      "reason: <one sentence>"
    ).mkString("\n")

  val prompt = buildPrompt(catalogPipeToon, "I know Scala and want to learn effect systems.")

  println(prompt)
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain llmdatawithtoon.Step4BuildPromptExample"
```

### Decoding the Model's Reply

This example decodes a well-formed reply, prints the error for each malformed reply, and strips a code fence before decoding.

<details>
  <summary>schema-examples/src/main/scala/llmdatawithtoon/Step5DecodeReplyExample.scala</summary>

```scala title="schema-examples/src/main/scala/llmdatawithtoon/Step5DecodeReplyExample.scala" showLineNumbers
package llmdatawithtoon

/**
 * How to Send Compact Data to an LLM with TOON and Parse the Reply — Decoding
 * the Model's Reply
 *
 * Decodes a well-formed reply, prints the error for each malformed reply, and
 * strips a code fence before decoding.
 *
 * Run with: cd schema-examples && sbt "runMain
 * llmdatawithtoon.Step5DecodeReplyExample"
 */
object Step5DecodeReplyExample extends App {

  case class Recommendation(title: String, reason: String)

  object Recommendation {
    implicit val schema: Schema[Recommendation] = Schema.derived
  }

  val replyCodec: ToonCodec[Recommendation] =
    Recommendation.schema.derive(ToonCodecDeriver.withRejectExtraFields(true))

  val reply =
    """title: Zionomicon
      |reason: It teaches ZIO, an effect system for Scala, from first steps to concurrency.""".stripMargin

  println(replyCodec.decode(reply))

  val malformedReplies = List(
    "missing field"     -> "title: Zionomicon",
    "renamed field"     -> "book: Zionomicon\nreason: It teaches ZIO.",
    "extra field"       -> "title: Zionomicon\nreason: It teaches ZIO.\nconfidence: 0.9",
    "prose before TOON" -> "Sure! Here is my pick:\ntitle: Zionomicon\nreason: It teaches ZIO."
  )

  for ((label, text) <- malformedReplies)
    replyCodec.decode(text) match {
      case Left(error)  => println(s"$label: ${error.message}")
      case Right(value) => println(s"$label: decoded $value")
    }

  val lenientCodec = Recommendation.schema.derive(ToonFormat)

  println(lenientCodec.decode("Sure! Here is my pick:\ntitle: Zionomicon\nreason: It teaches ZIO."))

  def extractToon(reply: String): String = {
    val fence = "```"
    val open  = reply.indexOf(fence)
    if (open < 0) reply.trim
    else {
      val bodyStart = reply.indexOf('\n', open) + 1
      val close     = reply.indexOf(fence, bodyStart)
      reply.substring(bodyStart, if (close < 0) reply.length else close).trim
    }
  }

  val chattyReply =
    """Happy to help! Here is my recommendation:
      |
      |```toon
      |title: Zionomicon
      |reason: It teaches ZIO, an effect system for Scala, from first steps to concurrency.
      |```
      |
      |Enjoy the book!""".stripMargin

  println(replyCodec.decode(chattyReply).left.map(_.message))

  println(replyCodec.decode(extractToon(chattyReply)))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain llmdatawithtoon.Step5DecodeReplyExample"
```

### Complete Example

Run the complete example from the `schema-examples` directory:

```bash
sbt "runMain llmdatawithtoon.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## Going Further

- The [TOON Codec Module](../reference/schema/built-in-codecs/toon.md) reference documents every `ToonCodecDeriver` option, including `ReaderConfig` and `WriterConfig`, key folding and discriminators.
- The [Built-in Formats and Codecs](../reference/schema/built-in-codecs/index.md) overview shows the other formats you can derive from the same `Schema`.
- The upcoming tutorial "One Type, Many Formats" encodes one type to TOON alongside JSON, YAML, CSV and other formats.
- To ask the model for several picks, decode a record with a list field, such as `case class Picks(picks: List[Recommendation])`. The model can answer with a tabular block like `picks[2]{title,reason}:`.

---

## How to Load Configuration Files on the JVM and in the Browser

In this guide, we load the settings of a Scala bookstore service from configuration files into one typed case class. By the end, you will read an `application.conf` from disk, split it into several files with includes, reuse values with substitutions, keep the loader inside one folder and below a size limit, read the same settings from JSON and YAML files, override file values with environment variables and system properties, and load the configuration where there is no filesystem at all: from the classpath and in the browser.

We start with one HOCON file and add one capability per section:

- [Loading application.conf From a File](#loading-applicationconf-from-a-file)
- [Splitting the File With Includes](#splitting-the-file-with-includes)
- [Reusing Values With Substitutions and Overrides](#reusing-values-with-substitutions-and-overrides)
- [Loading Files Safely](#loading-files-safely)
- [Loading JSON and YAML Files](#loading-json-and-yaml-files)
- [Guarding Against Lenient YAML](#guarding-against-lenient-yaml)
- [Overriding File Values From the Environment](#overriding-file-values-from-the-environment)
- [Loading application.conf From the Classpath](#loading-applicationconf-from-the-classpath)
- [Loading the Same Settings in the Browser](#loading-the-same-settings-in-the-browser)

## The Problem

A service keeps its settings in files so that the same build can run with different settings on a laptop, in CI and in production. Reading those files by hand means parsing the format, converting strings to numbers, merging the files of a split configuration, applying overrides from the environment, and reporting what is wrong, for every setting.

Without a configuration library, the bookstore's loader usually reads a flat `.properties` file:

```scala

final case class BookstoreSettings(
  featuredBook: String,
  pageSize: Int,
  dbUrl: String,
  dbUser: String,
  dbPassword: String
)

object LoadSettings {
  def load(path: String): BookstoreSettings = {
    val properties = new Properties()
    val in         = new FileInputStream(path)
    try properties.load(in)
    finally in.close()

    def setting(key: String): String =
      sys.env
        .get(key.toUpperCase.replace('.', '_'))
        .orElse(Option(properties.getProperty(key)))
        .getOrElse(throw new IllegalStateException(s"Missing setting $key in $path"))

    BookstoreSettings(
      featuredBook = setting("catalog.featuredBook"),
      pageSize = setting("catalog.pageSize").toInt,
      dbUrl = setting("db.url"),
      dbUser = setting("db.user"),
      dbPassword = setting("db.password")
    )
  }
}
```

Every key is spelled out twice, once in the file and once in the code. The format has no nesting, so a large file can't be split into a `db` part and a `catalog` part, and nothing stops two teams from defining the same key. A `catalog.pageSize=twenty` throws `NumberFormatException` without naming the key, the loader stops at the first missing setting, it reads a file of any size from any path it is given, and the password is a plain `String` that ends up in the logs as soon as someone prints the settings. On Scala.js, `FileInputStream` doesn't exist, so the browser build needs a second loader.

In this guide, the config module decodes the settings into a case class from its `Schema`, and its format modules read HOCON, JSON and YAML text into a `ConfigSource`, a flat set of dotted keys such as `db.url`. On the JVM, `config-hocon` also reads files itself, resolving includes and enforcing limits.

## Prerequisites

Add the config module and the three format modules to your `build.sbt`. `config-hocon` depends only on `zio-blocks-config`; `config-json` and `config-yaml` also bring in the schema and schema-yaml modules, so add only the formats you read:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-config"       % "0.0.56",
  "dev.zio" %% "zio-blocks-config-hocon" % "0.0.56",
  "dev.zio" %% "zio-blocks-config-json"  % "0.0.56",
  "dev.zio" %% "zio-blocks-config-yaml"  % "0.0.56"
)

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-config"       % "0.0.56",
  "dev.zio" %% "zio-blocks-next-config-hocon" % "0.0.56",
  "dev.zio" %% "zio-blocks-next-config-json"  % "0.0.56",
  "dev.zio" %% "zio-blocks-next-config-yaml"  % "0.0.56"
)
```

For a Scala.js build, use `%%%` instead of `%%`. [How to Add ZIO Blocks to Your Build](../getting-started/installation.md) explains which artifact names to pick for each Scala version.

This guide assumes you have loaded a case class from a `ConfigSource`, as in [Build a Typed Configuration Loader](./typed-config-loader.md). If you are moving an existing Typesafe Config or PureConfig application, [Coming from PureConfig, Typesafe Config or zio-config](./config-library-migration.md) maps the old calls one by one. The snippets below share these imports:

```scala

```

`zio.blocks.config.hocon._` adds `ConfigSource.fromHocon` and, on the JVM, `ConfigSource.fromFile` and `ConfigSource.fromFileWithLimits`. `zio.blocks.config.json._` adds `ConfigSource.fromJson`, and `zio.blocks.config.yaml._` adds `ConfigSource.fromYaml`.

## The Bookstore's Settings

The bookstore has three sections of settings: the HTTP server, the catalog with its featured book and page size, and the database. Each section is a case class that derives a `Schema`, which is all the config module needs to decode it, and the database password is a `Secret`, so printing the settings never shows it:

```scala
final case class ServerConfig(host: String, port: Int)
object ServerConfig {
  implicit val schema: Schema[ServerConfig] = Schema.derived
}

final case class CatalogConfig(featuredBook: String, pageSize: Int)
object CatalogConfig {
  implicit val schema: Schema[CatalogConfig] = Schema.derived
}

final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
object DbConfig {
  implicit val schema: Schema[DbConfig] = Schema.derived
}

final case class BookstoreConfig(server: ServerConfig, catalog: CatalogConfig, db: DbConfig)
object BookstoreConfig {
  implicit val schema: Schema[BookstoreConfig] = Schema.derived
}
```

A nested case class reads a nested section: `BookstoreConfig#catalog` reads the keys under `catalog`, and `CatalogConfig#featuredBook` reads `catalog.featuredBook`. The keys are the field names as written, in camelCase. Files that use kebab-case keys such as `featured-book` need `KeyFormat.KebabCase`, which [Coming from PureConfig, Typesafe Config or zio-config](./config-library-migration.md#keeping-your-kebab-case-keys) shows.

This page writes its configuration files into a temporary folder, `confDir`, inside a temporary work folder, `workDir`. In your application, `confDir` is the folder you ship your configuration in. Two small helpers keep the snippets short: `write` writes a file under `confDir`, and `report` prints an error with the temporary folder's path replaced by `<work>`, so the output reads the same on every machine:

```scala
def write(name: String, text: String): Path = {
  val file = confDir.resolve(name)
  Files.createDirectories(file.getParent)
  Files.write(file, text.getBytes("UTF-8"))
}

def report(error: ConfigError): Unit =
  println(error.getMessage.replace(workDir.toString, "<work>"))
```

## Loading application.conf From a File

The bookstore starts with one HOCON file, the format of Typesafe Config's `application.conf`:

```scala
val applicationConf = write(
  "application.conf",
  """server {
    |  host = "0.0.0.0"
    |  port = 8080
    |}
    |catalog {
    |  featuredBook = "Zionomicon"
    |  pageSize = 20
    |}
    |db {
    |  url = "jdbc:postgresql://localhost:5432/bookstore"
    |  user = "bookstore"
    |  password = "dev-password"
    |  poolSize = 10
    |}
    |""".stripMargin
)
```

`ConfigSource.fromFile` reads the file and returns `Either[ConfigError, ConfigSource]`. `Config.load` then decodes the case class from the source and returns `Either` with every decoding error on the left. `decode` chains the two steps, and later sections reuse it for every other way of getting a source:

```scala
def decode(source: Either[ConfigError, ConfigSource]): Either[List[ConfigError], BookstoreConfig] =
  source.left.map(List(_)).flatMap(Config.load[BookstoreConfig](_))

def loadBookstore(path: Path): Either[List[ConfigError], BookstoreConfig] =
  decode(ConfigSource.fromFile(path.toString))

println(loadBookstore(applicationConf))
// Right(BookstoreConfig(ServerConfig(0.0.0.0,8080),CatalogConfig(Zionomicon,20),DbConfig(jdbc:postgresql://localhost:5432/bookstore,bookstore,<secret>,10)))
```

The password prints as `<secret>`; `Secret.unwrap` returns the real value where the database driver needs it. The source that `ConfigSource.fromFile` returns is named `hocon:` followed by the file name, and that name appears in every error the source produces.

`ConfigSource.fromFile` never throws. A path that doesn't exist, a typo in the file name here, comes back as a `Left`:

```scala
loadBookstore(confDir.resolve("aplication.conf")).left.foreach(_.foreach(report))
// Parse error for key '<work>/conf/aplication.conf' (expected existing file) in source 'hocon:file'
```

`ConfigSource.fromFile` is JVM-only: Scala.js has no filesystem API to read from, so [Loading the Same Settings in the Browser](#loading-the-same-settings-in-the-browser) uses `ConfigSource.fromHocon`, which parses HOCON text on both platforms.

## Splitting the File With Includes

A large `application.conf` is easier to own when each section lives in its own file. `ConfigSource.fromFile` resolves `include "file"` lines relative to the folder of the file that contains them, the way Typesafe Config does for files, and an include inside a block puts the included keys under that block. Here `application.conf` includes `server.conf` at the top level and `db/db.conf` inside its `db` block, and `db/db.conf` in turn includes `pool.conf` from its own folder, `db/`:

```scala
write(
  "server.conf",
  """server {
    |  host = "0.0.0.0"
    |  port = 8080
    |}
    |""".stripMargin
)

write(
  "db/db.conf",
  """url = "jdbc:postgresql://localhost:5432/bookstore"
    |user = "bookstore"
    |password = "dev-password"
    |include "pool.conf"
    |""".stripMargin
)

write("db/pool.conf", "poolSize = 10\n")

val splitConf = write(
  "split.conf",
  """include "server.conf"
    |catalog {
    |  featuredBook = "Zionomicon"
    |  pageSize = 20
    |}
    |db {
    |  include "db/db.conf"
    |}
    |""".stripMargin
)
```

The split configuration decodes to the same settings as the single file:

```scala
println(loadBookstore(splitConf))
// Right(BookstoreConfig(ServerConfig(0.0.0.0,8080),CatalogConfig(Zionomicon,20),DbConfig(jdbc:postgresql://localhost:5432/bookstore,bookstore,<secret>,10)))
```

Includes are resolved while the file is parsed, so the included keys behave exactly as if they were written in the including file: later keys override earlier ones, and substitutions can refer to keys from any of the files.

:::warning[A missing include is skipped without an error]
Like Typesafe Config, `ConfigSource.fromFile` silently skips an `include` whose file doesn't exist, so a typo such as `include "db/dbb.conf"` loads without the `db` keys. `config-hocon` 0.0.56 doesn't parse `include required("...")`, nor `file(...)`, `classpath(...)` or `url(...)`: each is a parse error, so only the plain `include "file"` form is available. An absolute path such as `include "/etc/bookstore/db.conf"` is also resolved inside the including file's folder, so it is skipped the same way. Keep every required key free of a default in your case classes: then a skipped include shows up as `Missing required key` errors when the configuration loads, not later in production.
:::

Includes work only when the loader reads the files itself. `ConfigSource.fromHocon`, which takes the text of a file, drops every `include` line without an error, a known bug in 0.0.56. [Loading application.conf From the Classpath](#loading-applicationconf-from-the-classpath) shows how to resolve includes when the text doesn't come from `ConfigSource.fromFile`.

## Reusing Values With Substitutions and Overrides

HOCON lets one value refer to another with `${path}`, and lets a later definition of a key override an earlier one. Both work across included files, which makes a shared `defaults.conf` possible: the application file includes it first and then changes only what differs. Here `defaults.conf` holds the database host, port and name once, and builds the JDBC URL from them with substitutions; `layered.conf` includes it and overrides two values:

```scala
write(
  "defaults.conf",
  """server {
    |  host = "0.0.0.0"
    |  port = 8080
    |}
    |catalog {
    |  featuredBook = "Programming in Scala"
    |  pageSize = 20
    |}
    |db {
    |  host = "localhost"
    |  port = 5432
    |  name = "bookstore"
    |  url = "jdbc:postgresql://"${db.host}":"${db.port}"/"${db.name}
    |  user = "bookstore"
    |  password = "dev-password"
    |  poolSize = 10
    |}
    |""".stripMargin
)

val layeredConf = write(
  "layered.conf",
  """include "defaults.conf"
    |catalog.featuredBook = "Zionomicon"
    |db.host = "db.internal"
    |""".stripMargin
)
```

Substitutions are resolved after every file is merged, so `db.url` picks up the overridden `db.host`, and the `catalog` block keeps `pageSize` from the defaults because two objects with the same key merge key by key:

```scala
println(loadBookstore(layeredConf))
// Right(BookstoreConfig(ServerConfig(0.0.0.0,8080),CatalogConfig(Zionomicon,20),DbConfig(jdbc:postgresql://db.internal:5432/bookstore,bookstore,<secret>,10)))
```

`${?path}` is the optional form: when `path` is not defined anywhere in the files, a key whose whole value is `${?path}` is left out instead of failing the parse. Inside a concatenation such as `"jdbc:postgresql://localhost:5432/"${?db.name}`, the missing part becomes the text `null` in 0.0.56, a known bug, so avoid `${?path}` inside concatenations.

:::warning[Two substitutions Typesafe Config users rely on]
In 0.0.56, a substitution never falls back to an environment variable: `${DB_PASSWORD}` fails to parse and `${?DB_PASSWORD}` resolves to nothing, even when the variable is set. This is a known bug. Read environment variables with an `EnvSource` layer instead, as [Overriding File Values From the Environment](#overriding-file-values-from-the-environment) shows. A self-reference such as `path = ${path}":/opt/bin"` is also rejected as a circular substitution; `+=`, which appends to a list, works.
:::

## Loading Files Safely

The defaults of `ConfigSource.fromFile` trust the file it is given and every file that file includes. When the path, or the folder it sits in, can be written by someone other than your team, two limits keep the loader from reading what it shouldn't.

### Keeping Includes Inside the Configuration Folder

Without a limit, an include can reach any file the process can read. Here `escape.conf`, in `confDir`, includes `secrets.conf` from the work folder above it:

```scala
Files.write(workDir.resolve("secrets.conf"), "db.password = \"production-password\"\n".getBytes("UTF-8"))

val escapeConf = write("escape.conf", "include \"../secrets.conf\"\n")
```

By default, `ConfigSource.fromFile` follows the include out of the folder and reads the other file's keys:

```scala
println(ConfigSource.fromFile(escapeConf.toString).map(_.all("").keys.toList))
// Right(List(db.password))
```

`allowedBase` names a folder that the file and every include must stay inside. `ConfigSource.fromFile` resolves each path to its canonical form first, so neither `../` nor a symbolic link that points outside can get around it, and the include above now fails the whole load:

```scala
ConfigSource.fromFile(escapeConf.toString, allowedBase = Some(confDir.toFile)) match {
  case Left(ConfigError.ParseError(_, _, _, Some(cause))) => println(cause.getMessage.replace(workDir.toString, "<work>"))
  case other                                              => println(other)
}
// Include path traversal: ../secrets.conf resolves outside <work>/conf (line 0, column 0)
```

The match prints the error's cause because, for an error inside the document, the error's own message ends at `Parse error for key '' (expected valid HOCON) in source 'hocon:escape.conf'`. It leaves out the reason and the line on purpose, so that a cause quoting a line of the file, which may hold a secret, never reaches a log. Print the cause yourself only where that is safe. A root file outside `allowedBase` fails too, and then the message itself says `expected path inside` followed by the folder.

`maxIncludeDepth` limits how deep includes may nest, and defaults to `10`. Two files that include each other exceed it and fail with `Include depth exceeded (10)` instead of looping forever.

### Capping the File Size

`ConfigSource.fromFile` refuses a file larger than 1 MiB, and checks the same limit for every included file, so a huge or endless file never fills the heap. Here a file of about 2 MB is refused:

```scala
val largeConf = write("large.conf", "# padding\n" * (200 * 1024))
```

The size check runs before the file is parsed:

```scala
ConfigSource.fromFile(largeConf.toString).left.foreach(report)
// Parse error for key '<work>/conf/large.conf' (expected valid HOCON) in source 'hocon:large.conf': File size exceeds limit (1048576 bytes) (line 0, column 0)
```

`ConfigSource.fromFileWithLimits` takes the same parameters as `ConfigSource.fromFile` plus `maxFileBytes`. A service with a large generated configuration raises the limit there:

```scala
println(ConfigSource.fromFileWithLimits(largeConf.toString, allowedBase = Some(confDir.toFile), maxFileBytes = 4L * 1024 * 1024).map(_.sourceId))
// Right(hocon:large.conf)
```

A smaller limit than the default also makes sense when the file comes from an upload or a shared volume.

## Loading JSON and YAML Files

`config-json` and `config-yaml` parse text, not files: `ConfigSource.fromJson` and `ConfigSource.fromYaml` take the document as a `String` and a source name for error messages, and return `Either[ConfigError, ConfigSource]`. On the JVM, we read the file ourselves and pass its text and file name:

```scala
def readConfigFile(path: Path)(parse: (String, String) => Either[ConfigError, ConfigSource]): Either[ConfigError, ConfigSource] =
  parse(new String(Files.readAllBytes(path), "UTF-8"), path.getFileName.toString)

val configJson = write(
  "config.json",
  """{
    |  "server": { "host": "0.0.0.0", "port": 8080 },
    |  "catalog": { "featuredBook": "Zionomicon", "pageSize": 20 },
    |  "db": {
    |    "url": "jdbc:postgresql://localhost:5432/bookstore",
    |    "user": "bookstore",
    |    "password": "dev-password",
    |    "poolSize": 10
    |  }
    |}
    |""".stripMargin
)

val configYaml = write(
  "config.yaml",
  """server:
    |  host: 0.0.0.0
    |  port: 8080
    |catalog:
    |  featuredBook: Zionomicon
    |  pageSize: 20
    |db:
    |  url: jdbc:postgresql://localhost:5432/bookstore
    |  user: bookstore
    |  password: dev-password
    |  poolSize: 10
    |""".stripMargin
)
```

Both files flatten to the same dotted keys as the HOCON file, so the same case class decodes from either:

```scala
println(decode(readConfigFile(configJson)(ConfigSource.fromJson(_, _))))
// Right(BookstoreConfig(ServerConfig(0.0.0.0,8080),CatalogConfig(Zionomicon,20),DbConfig(jdbc:postgresql://localhost:5432/bookstore,bookstore,<secret>,10)))
println(decode(readConfigFile(configYaml)(ConfigSource.fromYaml(_, _))))
// Right(BookstoreConfig(ServerConfig(0.0.0.0,8080),CatalogConfig(Zionomicon,20),DbConfig(jdbc:postgresql://localhost:5432/bookstore,bookstore,<secret>,10)))
```

Reading the file yourself means the limits of `ConfigSource.fromFile` don't apply: `Files.readAllBytes` throws `NoSuchFileException` for a missing file and reads a file of any size. Check `Files.size` first when the path comes from outside your team.

## Guarding Against Lenient YAML

The YAML reader accepts malformed documents without an error: it stops at the first line it can't read and drops that line and everything after it. Here `pageSize` is missing the space after its colon:

```scala
val brokenYaml = write(
  "broken.yaml",
  """server:
    |  host: 0.0.0.0
    |  port: 8080
    |catalog:
    |  featuredBook: Zionomicon
    |  pageSize:20
    |db:
    |  url: jdbc:postgresql://localhost:5432/bookstore
    |  user: bookstore
    |  password: dev-password
    |  poolSize: 10
    |""".stripMargin
)
```

Parsing succeeds, and the source ends at `featuredBook`: `pageSize` and the whole `db` section after it are gone:

```scala
println(readConfigFile(brokenYaml)(ConfigSource.fromYaml(_, _)).map(_.all("").keys.toList.sorted))
// Right(List(catalog.featuredBook, server.host, server.port))
```

The typed load is the guard: every field without a default is required, so the dropped keys come back as errors instead of a half-configured service. Keep defaults out of the case class for settings that must come from the file, and supply them through a defaults layer instead, as the next section does:

```scala
decode(readConfigFile(brokenYaml)(ConfigSource.fromYaml(_, _))).left.foreach(_.foreach(report))
// Missing required key 'catalog.pageSize' in source 'broken.yaml'
// Missing required key 'db.url' in source 'broken.yaml'
// Missing required key 'db.user' in source 'broken.yaml'
// Missing required key 'db.password' in source 'broken.yaml'
// Missing required key 'db.poolSize' in source 'broken.yaml'
```

The errors name the missing keys, not the line that caused them. In 0.0.56, the YAML reader almost never reports a parse error, and a JSON parse error names neither the reason nor the line, so when a file doesn't load as expected, open it in an editor that validates the format.

## Overriding File Values From the Environment

The file holds the defaults that ship with the service, and each deployment overrides a few of them: the database password always, the page size sometimes. `ConfigSource#orElse` layers two sources: the source on the left wins, and the source on the right fills in every key the left one doesn't have.

In production, the environment layer is `EnvSource`. `KeyFormat.UpperSnakeCase` and the `"bookstore"` prefix make it read `catalog.featuredBook` from the variable `BOOKSTORE_CATALOG_FEATURED_BOOK` and `db.password` from `BOOKSTORE_DB_PASSWORD`. `SysPropSource` reads JVM system properties, so `SysPropSource.prefix("bookstore")` reads `catalog.pageSize` from `-Dbookstore.catalog.pageSize=50`:

```scala
val productionEnv: ConfigSource = EnvSource.keyFormat(KeyFormat.UpperSnakeCase).prefix("bookstore")

val systemProperties: ConfigSource = SysPropSource.prefix("bookstore")
```

This page must give the same output on every machine, so it uses a map in place of the environment, holding the two variables a deployment would export. The system property is real:

```scala
val env = ConfigSource.fromMap(
  Map("db.password" -> "production-password", "catalog.featuredBook" -> "Functional Programming in Scala"),
  "env"
)

System.setProperty("bookstore.catalog.pageSize", "50")

val fileSource = ConfigSource.fromFile(applicationConf.toString) match {
  case Right(source) => source
  case Left(error)   => throw error
}

val settings = systemProperties.orElse(env).orElse(fileSource)
```

System properties win over the environment, and the environment wins over the file. `Config.loadWithProvenance` decodes the same case class and also records where each value came from:

```scala
val loaded = Config.loadWithProvenance[BookstoreConfig](settings) match {
  case Right(provenance) => provenance
  case Left(errors)      => throw errors.head
}
```

The page size came from the system property, the featured book from the environment, and the port from the file:

```scala
println(loaded.value)
// BookstoreConfig(ServerConfig(0.0.0.0,8080),CatalogConfig(Functional Programming in Scala,50),DbConfig(jdbc:postgresql://localhost:5432/bookstore,bookstore,<secret>,10))
println(loaded.provenanceOf("catalog.pageSize"))
// Resolved(sysprop,bookstore.catalog.pageSize,50)
println(loaded.provenanceOf("catalog.featuredBook"))
// Resolved(env,catalog.featuredBook,Functional Programming in Scala)
println(loaded.provenanceOf("server.port"))
// Resolved(hocon:application.conf,server.port,8080)
```

Prefix the environment layer, always. An `EnvSource` without a prefix reads every variable of the process, and `ProvenanceMap#dump` then prints all of them, including tokens that have nothing to do with the bookstore.

:::warning[Provenance shows a secret's value]
`ProvenanceMap#provenanceOf` and `ProvenanceMap#dump` in 0.0.56 print the raw text of a `Secret` field: `provenanceOf("db.password")` shows `production-password`, and `dump()` hides a value only when its key name looks sensitive, as `password` does, whatever the field's type. Don't log the provenance of secret fields, and give them names that contain `password`, `secret`, `token` or `credential`.
:::

[Build a Typed Configuration Loader](./typed-config-loader.md) shows more layering with defaults and provenance, and [How to Feed Configuration into a Wire Dependency Graph](./feed-config-into-a-wire-graph.md) hands each section of the layered source to the service that needs it.

## Loading application.conf From the Classpath

Typesafe Config's `ConfigFactory.load()` finds `application.conf` on the classpath. `config-hocon` has no classpath lookup, so a service that ships its configuration inside its jar reads the resource's text itself. When the configuration is one file, `ConfigSource.fromHocon` parses that text:

```scala

def resourceText(name: String): Option[String] =
  Option(getClass.getClassLoader.getResourceAsStream(name)).map { in =>
    try Source.fromInputStream(in, "UTF-8").mkString
    finally in.close()
  }

val fromClasspath: Either[ConfigError, ConfigSource] =
  resourceText("application.conf") match {
    case Some(text) => ConfigSource.fromHocon(text, "classpath:application.conf")
    case None       => Left(ConfigError.ParseError("application.conf", "classpath", "existing resource"))
  }
```

`ConfigSource.fromHocon` drops every `include`, so a split configuration needs `HoconParser.parse`. It takes the text of the root file and a function from an include's name to that file's text, and returns the parsed tree, which `HoconValue.flatten` turns into the flat map that `ConfigSource.fromMap` takes. Passing `resourceText` as that function resolves `include "server.conf"` from the classpath. The function can return the text from anywhere, so the same code also works with files embedded in the program. Here it looks the names up in a map that holds the three files of the split configuration:

```scala
val bundledFiles: Map[String, String] = Map(
  "application.conf" ->
    """include "server.conf"
      |include "catalog.conf"
      |db {
      |  include "db.conf"
      |}
      |""".stripMargin,
  "server.conf"  -> "server { host = \"0.0.0.0\", port = 8080 }\n",
  "catalog.conf" -> "catalog { featuredBook = \"Zionomicon\", pageSize = 20 }\n",
  "db.conf" ->
    """url = "jdbc:postgresql://localhost:5432/bookstore"
      |user = "bookstore"
      |password = "dev-password"
      |poolSize = 10
      |""".stripMargin
)

def parseBundled(files: Map[String, String], root: String): Either[ConfigError, ConfigSource] =
  HoconParser.parse(files(root), files.get) match {
    case Right(tree) => Right(ConfigSource.fromMap(HoconValue.flatten(tree), s"hocon:$root"))
    case Left(error) => Left(ConfigError.ParseError("", s"hocon:$root", "valid HOCON", Some(error)))
  }
```

The bundled files decode to the same settings as the files on disk:

```scala
println(decode(parseBundled(bundledFiles, "application.conf")))
// Right(BookstoreConfig(ServerConfig(0.0.0.0,8080),CatalogConfig(Zionomicon,20),DbConfig(jdbc:postgresql://localhost:5432/bookstore,bookstore,<secret>,10)))
```

The include function returns text, not a file, so the parser has no folder to resolve names against: every name is looked up as written, `allowedBase` and the size limit don't apply, and an include that the function can't find is skipped. In 0.0.56, `HoconParser.parse` resolves only one level of includes this way: an `include` inside an included file is skipped without an error. Keep every include in the root file, as `bundledFiles` does.

## Loading the Same Settings in the Browser

A Scala.js application in the browser has no files, no environment variables and no system properties. `ConfigSource.fromHocon`, `ConfigSource.fromJson`, `ConfigSource.fromYaml`, `HoconParser.parse` and `Config.load` work the same way on Scala.js as on the JVM; `ConfigSource.fromFile` and `ConfigSource.fromFileWithLimits` don't exist there. So the browser build gets its configuration as text, in one of two ways.

The simplest is to compile the configuration into the program: put the HOCON text in a `String` constant, or the split files in a map like `bundledFiles`, and parse it with `ConfigSource.fromHocon` or `parseBundled` exactly as above. The settings are then fixed at build time, which suits values such as the catalog's page size.

Settings that change per deployment, such as the API host, come from the server at startup. The page fetches the text with the browser's `fetch`, and parses it once the text arrives. This snippet needs Scala.js, so the build of this page doesn't compile it; it compiles with Scala.js 1.22.0 on Scala 2.13 and Scala 3.9:

```scala

def loadCatalog(): Unit =
  for {
    response <- js.Dynamic.global.fetch("/config/catalog.conf").asInstanceOf[js.Promise[js.Dynamic]]
    text     <- response.text().asInstanceOf[js.Promise[String]]
  } ConfigSource.fromHocon(text, "catalog.conf").flatMap(Config.load[CatalogConfig](_)) match {
    case Right(catalog) => println(s"Featured: ${catalog.featuredBook}")
    case Left(errors)   => println(errors)
  }
```

The two layers that come from the process don't apply in the browser. `SysPropSource` is always empty on Scala.js, and `EnvSource` reads `process.env` under Node.js and is empty in a browser. To override embedded defaults per deployment, layer the fetched source over the embedded one with `ConfigSource#orElse`, the same way the environment layered over the file.

Never put the database password in a browser configuration: everything the page downloads, the user can read. The browser's settings are the public part of the configuration, and the secrets stay in the server's files and environment.

## Putting It Together

The complete example writes a split `application.conf` and a `defaults.conf` into a temporary folder, loads them with `ConfigSource.fromFile` locked to that folder, layers system properties and `BOOKSTORE_*` environment variables over them, prints the settings and where three of them came from, and then shows the errors for a file outside the folder and for a broken YAML file:

```scala title="config-examples/src/main/scala/loadconfigurationfiles/CompleteExample.scala"
package loadconfigurationfiles

/**
 * How to Load Configuration Files on the JVM and in the Browser — Putting It
 * Together
 *
 * Writes a split application.conf and a defaults.conf into a temporary folder,
 * loads them with fromFile locked to that folder, layers system properties and
 * BOOKSTORE_* environment variables over them, prints the settings and where
 * three of them came from, and shows the errors for an include outside the
 * folder and for a broken YAML file.
 *
 * Run with: cd config-examples && BOOKSTORE_DB_PASSWORD=production-password sbt
 * "runMain loadconfigurationfiles.CompleteExample"
 */
object CompleteExample {

  final case class ServerConfig(host: String, port: Int)
  object ServerConfig {
    implicit val schema: Schema[ServerConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredBook: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class BookstoreConfig(server: ServerConfig, catalog: CatalogConfig, db: DbConfig)
  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  val workDir: Path = Files.createTempDirectory("bookstore").toRealPath()
  val confDir: Path = Files.createDirectories(workDir.resolve("conf"))

  def write(name: String, text: String): Path = {
    val file = confDir.resolve(name)
    Files.createDirectories(file.getParent)
    Files.write(file, text.getBytes("UTF-8"))
  }

  def report(error: ConfigError): Unit =
    println(error.getMessage.replace(workDir.toString, "<work>"))

  def decode(source: Either[ConfigError, ConfigSource]): Either[List[ConfigError], BookstoreConfig] =
    source.left.map(List(_)).flatMap(Config.load[BookstoreConfig](_))

  def loadBookstore(path: Path): Either[List[ConfigError], BookstoreConfig] =
    decode(ConfigSource.fromFile(path.toString))

  val productionEnv: ConfigSource = EnvSource.keyFormat(KeyFormat.UpperSnakeCase).prefix("bookstore")

  val systemProperties: ConfigSource = SysPropSource.prefix("bookstore")

  def main(args: Array[String]): Unit = {
    // 1. Write a split application.conf and a shared defaults.conf.
    write(
      "defaults.conf",
      """server {
        |  host = "0.0.0.0"
        |  port = 8080
        |}
        |catalog {
        |  featuredBook = "Programming in Scala"
        |  pageSize = 20
        |}
        |db {
        |  host = "localhost"
        |  port = 5432
        |  name = "bookstore"
        |  url = "jdbc:postgresql://"${db.host}":"${db.port}"/"${db.name}
        |  user = "bookstore"
        |  password = "dev-password"
        |  poolSize = 10
        |}
        |""".stripMargin
    )
    write("db/pool.conf", "poolSize = 10\n")
    write(
      "db/db.conf",
      """user = "bookstore"
        |password = "dev-password"
        |include "pool.conf"
        |""".stripMargin
    )
    val applicationConf = write(
      "application.conf",
      """include "defaults.conf"
        |catalog.featuredBook = "Zionomicon"
        |db {
        |  host = "db.internal"
        |  include "db/db.conf"
        |}
        |""".stripMargin
    )

    // 2. Load the file locked to the configuration folder, then layer the process settings over it.
    val fileSource = ConfigSource.fromFile(applicationConf.toString, allowedBase = Some(confDir.toFile)) match {
      case Right(source) => source
      case Left(error)   => throw error
    }
    val settings = systemProperties.orElse(productionEnv).orElse(fileSource)

    val loaded = Config.loadWithProvenance[BookstoreConfig](settings) match {
      case Right(provenance) => provenance
      case Left(errors)      => throw errors.head
    }

    // 3. The settings and where three non-secret values came from.
    println(loaded.value)
    println(loaded.provenanceOf("catalog.pageSize"))
    println(loaded.provenanceOf("catalog.featuredBook"))
    println(loaded.provenanceOf("server.port"))

    // 4. An include that leaves the folder fails the whole load.
    Files.write(workDir.resolve("secrets.conf"), "db.password = \"production-password\"\n".getBytes("UTF-8"))
    val escapeConf = write("escape.conf", "include \"../secrets.conf\"\n")
    ConfigSource.fromFile(escapeConf.toString, allowedBase = Some(confDir.toFile)) match {
      case Left(ConfigError.ParseError(_, _, _, Some(cause))) =>
        println(cause.getMessage.replace(workDir.toString, "<work>"))
      case other => println(other.map(_.sourceId))
    }

    // 5. A broken YAML file parses, but the typed load reports every missing key.
    val brokenYaml = write(
      "broken.yaml",
      """server:
        |  host: 0.0.0.0
        |  port: 8080
        |catalog:
        |  featuredBook: Zionomicon
        |  pageSize:20
        |db:
        |  url: jdbc:postgresql://localhost:5432/bookstore
        |  user: bookstore
        |  password: dev-password
        |  poolSize: 10
        |""".stripMargin
    )
    val yamlText = new String(Files.readAllBytes(brokenYaml), "UTF-8")
    decode(ConfigSource.fromYaml(yamlText, brokenYaml.getFileName.toString)).left.foreach(_.foreach(report))
  }
}
```

## Running the Examples

All examples in this guide have corresponding runnable Scala files in the `config-examples` project. The JVM examples write their configuration files into a temporary folder when they start, so they need no setup; the classpath example reads the files in `config-examples/src/main/resources/loadconfigurationfiles/`. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/config-examples
```

Each file below matches one section of this guide. To compile them all without running any, run `sbt compile`.

### Loading application.conf From a File

<details>
  <summary>config-examples/src/main/scala/loadconfigurationfiles/Step1LoadHoconFile.scala</summary>

```scala title="config-examples/src/main/scala/loadconfigurationfiles/Step1LoadHoconFile.scala" showLineNumbers
package loadconfigurationfiles

/**
 * How to Load Configuration Files on the JVM and in the Browser — Loading
 * application.conf From a File
 *
 * Writes an application.conf into a temporary folder, loads it with
 * ConfigSource.fromFile into one typed case class, and shows the error for a
 * path that does not exist.
 *
 * Run with: cd config-examples && sbt "runMain
 * loadconfigurationfiles.Step1LoadHoconFile"
 */
object Step1LoadHoconFile {

  final case class ServerConfig(host: String, port: Int)
  object ServerConfig {
    implicit val schema: Schema[ServerConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredBook: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class BookstoreConfig(server: ServerConfig, catalog: CatalogConfig, db: DbConfig)
  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  val workDir: Path = Files.createTempDirectory("bookstore").toRealPath()
  val confDir: Path = Files.createDirectories(workDir.resolve("conf"))

  def write(name: String, text: String): Path = {
    val file = confDir.resolve(name)
    Files.createDirectories(file.getParent)
    Files.write(file, text.getBytes("UTF-8"))
  }

  def report(error: ConfigError): Unit =
    println(error.getMessage.replace(workDir.toString, "<work>"))

  def decode(source: Either[ConfigError, ConfigSource]): Either[List[ConfigError], BookstoreConfig] =
    source.left.map(List(_)).flatMap(Config.load[BookstoreConfig](_))

  def loadBookstore(path: Path): Either[List[ConfigError], BookstoreConfig] =
    decode(ConfigSource.fromFile(path.toString))

  val applicationConf: Path = write(
    "application.conf",
    """server {
      |  host = "0.0.0.0"
      |  port = 8080
      |}
      |catalog {
      |  featuredBook = "Zionomicon"
      |  pageSize = 20
      |}
      |db {
      |  url = "jdbc:postgresql://localhost:5432/bookstore"
      |  user = "bookstore"
      |  password = "dev-password"
      |  poolSize = 10
      |}
      |""".stripMargin
  )

  def main(args: Array[String]): Unit = {
    println(loadBookstore(applicationConf))
    loadBookstore(confDir.resolve("aplication.conf")).left.foreach(_.foreach(report))
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain loadconfigurationfiles.Step1LoadHoconFile"
```

### Splitting the File With Includes

<details>
  <summary>config-examples/src/main/scala/loadconfigurationfiles/Step2Includes.scala</summary>

```scala title="config-examples/src/main/scala/loadconfigurationfiles/Step2Includes.scala" showLineNumbers
package loadconfigurationfiles

/**
 * How to Load Configuration Files on the JVM and in the Browser — Splitting the
 * File With Includes
 *
 * Splits application.conf into several files with include lines, including an
 * include inside a block and a nested include, and decodes the result.
 *
 * Run with: cd config-examples && sbt "runMain
 * loadconfigurationfiles.Step2Includes"
 */
object Step2Includes {

  final case class ServerConfig(host: String, port: Int)
  object ServerConfig {
    implicit val schema: Schema[ServerConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredBook: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class BookstoreConfig(server: ServerConfig, catalog: CatalogConfig, db: DbConfig)
  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  val workDir: Path = Files.createTempDirectory("bookstore").toRealPath()
  val confDir: Path = Files.createDirectories(workDir.resolve("conf"))

  def write(name: String, text: String): Path = {
    val file = confDir.resolve(name)
    Files.createDirectories(file.getParent)
    Files.write(file, text.getBytes("UTF-8"))
  }

  def report(error: ConfigError): Unit =
    println(error.getMessage.replace(workDir.toString, "<work>"))

  def decode(source: Either[ConfigError, ConfigSource]): Either[List[ConfigError], BookstoreConfig] =
    source.left.map(List(_)).flatMap(Config.load[BookstoreConfig](_))

  def loadBookstore(path: Path): Either[List[ConfigError], BookstoreConfig] =
    decode(ConfigSource.fromFile(path.toString))

  def main(args: Array[String]): Unit = {
    write(
      "server.conf",
      """server {
        |  host = "0.0.0.0"
        |  port = 8080
        |}
        |""".stripMargin
    )

    write(
      "db/db.conf",
      """url = "jdbc:postgresql://localhost:5432/bookstore"
        |user = "bookstore"
        |password = "dev-password"
        |include "pool.conf"
        |""".stripMargin
    )

    write("db/pool.conf", "poolSize = 10\n")

    val splitConf = write(
      "split.conf",
      """include "server.conf"
        |catalog {
        |  featuredBook = "Zionomicon"
        |  pageSize = 20
        |}
        |db {
        |  include "db/db.conf"
        |}
        |""".stripMargin
    )

    println(loadBookstore(splitConf))
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain loadconfigurationfiles.Step2Includes"
```

### Reusing Values With Substitutions and Overrides

<details>
  <summary>config-examples/src/main/scala/loadconfigurationfiles/Step3Substitutions.scala</summary>

```scala title="config-examples/src/main/scala/loadconfigurationfiles/Step3Substitutions.scala" showLineNumbers
package loadconfigurationfiles

/**
 * How to Load Configuration Files on the JVM and in the Browser — Reusing
 * Values With Substitutions and Overrides
 *
 * A shared defaults.conf builds the JDBC URL with substitutions; layered.conf
 * includes it and overrides two values.
 *
 * Run with: cd config-examples && sbt "runMain
 * loadconfigurationfiles.Step3Substitutions"
 */
object Step3Substitutions {

  final case class ServerConfig(host: String, port: Int)
  object ServerConfig {
    implicit val schema: Schema[ServerConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredBook: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class BookstoreConfig(server: ServerConfig, catalog: CatalogConfig, db: DbConfig)
  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  val workDir: Path = Files.createTempDirectory("bookstore").toRealPath()
  val confDir: Path = Files.createDirectories(workDir.resolve("conf"))

  def write(name: String, text: String): Path = {
    val file = confDir.resolve(name)
    Files.createDirectories(file.getParent)
    Files.write(file, text.getBytes("UTF-8"))
  }

  def report(error: ConfigError): Unit =
    println(error.getMessage.replace(workDir.toString, "<work>"))

  def decode(source: Either[ConfigError, ConfigSource]): Either[List[ConfigError], BookstoreConfig] =
    source.left.map(List(_)).flatMap(Config.load[BookstoreConfig](_))

  def loadBookstore(path: Path): Either[List[ConfigError], BookstoreConfig] =
    decode(ConfigSource.fromFile(path.toString))

  def main(args: Array[String]): Unit = {
    write(
      "defaults.conf",
      """server {
        |  host = "0.0.0.0"
        |  port = 8080
        |}
        |catalog {
        |  featuredBook = "Programming in Scala"
        |  pageSize = 20
        |}
        |db {
        |  host = "localhost"
        |  port = 5432
        |  name = "bookstore"
        |  url = "jdbc:postgresql://"${db.host}":"${db.port}"/"${db.name}
        |  user = "bookstore"
        |  password = "dev-password"
        |  poolSize = 10
        |}
        |""".stripMargin
    )

    val layeredConf = write(
      "layered.conf",
      """include "defaults.conf"
        |catalog.featuredBook = "Zionomicon"
        |db.host = "db.internal"
        |""".stripMargin
    )

    println(loadBookstore(layeredConf))
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain loadconfigurationfiles.Step3Substitutions"
```

### Loading Files Safely

<details>
  <summary>config-examples/src/main/scala/loadconfigurationfiles/Step4SafeLoading.scala</summary>

```scala title="config-examples/src/main/scala/loadconfigurationfiles/Step4SafeLoading.scala" showLineNumbers
package loadconfigurationfiles

/**
 * How to Load Configuration Files on the JVM and in the Browser — Loading Files
 * Safely
 *
 * Shows an include escaping the configuration folder (and allowedBase refusing
 * it), then a 2 MB file refused by the default size limit and accepted by
 * fromFileWithLimits.
 *
 * Run with: cd config-examples && sbt "runMain
 * loadconfigurationfiles.Step4SafeLoading"
 */
object Step4SafeLoading {

  final case class ServerConfig(host: String, port: Int)
  object ServerConfig {
    implicit val schema: Schema[ServerConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredBook: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class BookstoreConfig(server: ServerConfig, catalog: CatalogConfig, db: DbConfig)
  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  val workDir: Path = Files.createTempDirectory("bookstore").toRealPath()
  val confDir: Path = Files.createDirectories(workDir.resolve("conf"))

  def write(name: String, text: String): Path = {
    val file = confDir.resolve(name)
    Files.createDirectories(file.getParent)
    Files.write(file, text.getBytes("UTF-8"))
  }

  def report(error: ConfigError): Unit =
    println(error.getMessage.replace(workDir.toString, "<work>"))

  def decode(source: Either[ConfigError, ConfigSource]): Either[List[ConfigError], BookstoreConfig] =
    source.left.map(List(_)).flatMap(Config.load[BookstoreConfig](_))

  def loadBookstore(path: Path): Either[List[ConfigError], BookstoreConfig] =
    decode(ConfigSource.fromFile(path.toString))

  def main(args: Array[String]): Unit = {
    // 1. Keeping includes inside the configuration folder
    Files.write(workDir.resolve("secrets.conf"), "db.password = \"production-password\"\n".getBytes("UTF-8"))

    val escapeConf = write("escape.conf", "include \"../secrets.conf\"\n")

    println(ConfigSource.fromFile(escapeConf.toString).map(_.all("").keys.toList))

    ConfigSource.fromFile(escapeConf.toString, allowedBase = Some(confDir.toFile)) match {
      case Left(ConfigError.ParseError(_, _, _, Some(cause))) =>
        println(cause.getMessage.replace(workDir.toString, "<work>"))
      case other => println(other)
    }

    // 2. Capping the file size
    val largeConf = write("large.conf", "# padding\n" * (200 * 1024))

    ConfigSource.fromFile(largeConf.toString).left.foreach(report)

    println(
      ConfigSource
        .fromFileWithLimits(largeConf.toString, allowedBase = Some(confDir.toFile), maxFileBytes = 4L * 1024 * 1024)
        .map(_.sourceId)
    )
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain loadconfigurationfiles.Step4SafeLoading"
```

### Loading JSON and YAML Files

<details>
  <summary>config-examples/src/main/scala/loadconfigurationfiles/Step5JsonAndYaml.scala</summary>

```scala title="config-examples/src/main/scala/loadconfigurationfiles/Step5JsonAndYaml.scala" showLineNumbers
package loadconfigurationfiles

/**
 * How to Load Configuration Files on the JVM and in the Browser — Loading JSON
 * and YAML Files
 *
 * Reads a JSON and a YAML file with ConfigSource.fromJson and fromYaml, decodes
 * the same case class from both, and shows how the typed load guards against
 * lenient YAML.
 *
 * Run with: cd config-examples && sbt "runMain
 * loadconfigurationfiles.Step5JsonAndYaml"
 */
object Step5JsonAndYaml {

  final case class ServerConfig(host: String, port: Int)
  object ServerConfig {
    implicit val schema: Schema[ServerConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredBook: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class BookstoreConfig(server: ServerConfig, catalog: CatalogConfig, db: DbConfig)
  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  val workDir: Path = Files.createTempDirectory("bookstore").toRealPath()
  val confDir: Path = Files.createDirectories(workDir.resolve("conf"))

  def write(name: String, text: String): Path = {
    val file = confDir.resolve(name)
    Files.createDirectories(file.getParent)
    Files.write(file, text.getBytes("UTF-8"))
  }

  def report(error: ConfigError): Unit =
    println(error.getMessage.replace(workDir.toString, "<work>"))

  def decode(source: Either[ConfigError, ConfigSource]): Either[List[ConfigError], BookstoreConfig] =
    source.left.map(List(_)).flatMap(Config.load[BookstoreConfig](_))

  def loadBookstore(path: Path): Either[List[ConfigError], BookstoreConfig] =
    decode(ConfigSource.fromFile(path.toString))

  def readConfigFile(path: Path)(
    parse: (String, String) => Either[ConfigError, ConfigSource]
  ): Either[ConfigError, ConfigSource] =
    parse(new String(Files.readAllBytes(path), "UTF-8"), path.getFileName.toString)

  def main(args: Array[String]): Unit = {
    val configJson = write(
      "config.json",
      """{
        |  "server": { "host": "0.0.0.0", "port": 8080 },
        |  "catalog": { "featuredBook": "Zionomicon", "pageSize": 20 },
        |  "db": {
        |    "url": "jdbc:postgresql://localhost:5432/bookstore",
        |    "user": "bookstore",
        |    "password": "dev-password",
        |    "poolSize": 10
        |  }
        |}
        |""".stripMargin
    )

    val configYaml = write(
      "config.yaml",
      """server:
        |  host: 0.0.0.0
        |  port: 8080
        |catalog:
        |  featuredBook: Zionomicon
        |  pageSize: 20
        |db:
        |  url: jdbc:postgresql://localhost:5432/bookstore
        |  user: bookstore
        |  password: dev-password
        |  poolSize: 10
        |""".stripMargin
    )

    println(decode(readConfigFile(configJson)(ConfigSource.fromJson(_, _))))
    println(decode(readConfigFile(configYaml)(ConfigSource.fromYaml(_, _))))

    // Guarding against lenient YAML
    val brokenYaml = write(
      "broken.yaml",
      """server:
        |  host: 0.0.0.0
        |  port: 8080
        |catalog:
        |  featuredBook: Zionomicon
        |  pageSize:20
        |db:
        |  url: jdbc:postgresql://localhost:5432/bookstore
        |  user: bookstore
        |  password: dev-password
        |  poolSize: 10
        |""".stripMargin
    )

    println(readConfigFile(brokenYaml)(ConfigSource.fromYaml(_, _)).map(_.all("").keys.toList.sorted))
    decode(readConfigFile(brokenYaml)(ConfigSource.fromYaml(_, _))).left.foreach(_.foreach(report))
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain loadconfigurationfiles.Step5JsonAndYaml"
```

### Overriding File Values From the Environment

<details>
  <summary>config-examples/src/main/scala/loadconfigurationfiles/Step6EnvironmentOverrides.scala</summary>

```scala title="config-examples/src/main/scala/loadconfigurationfiles/Step6EnvironmentOverrides.scala" showLineNumbers
package loadconfigurationfiles

/**
 * How to Load Configuration Files on the JVM and in the Browser — Overriding
 * File Values From the Environment
 *
 * Layers SysPropSource and EnvSource over a HOCON file and prints where each
 * value came from. Provenance of the secret db.password is never printed.
 *
 * Run with: BOOKSTORE_DB_PASSWORD=production-password sbt 'set run /
 * javaOptions += "-Dbookstore.catalog.pageSize=50"' "runMain
 * loadconfigurationfiles.Step6EnvironmentOverrides" (from config-examples)
 */
object Step6EnvironmentOverrides {

  final case class ServerConfig(host: String, port: Int)
  object ServerConfig {
    implicit val schema: Schema[ServerConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredBook: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class BookstoreConfig(server: ServerConfig, catalog: CatalogConfig, db: DbConfig)
  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  val workDir: Path = Files.createTempDirectory("bookstore").toRealPath()
  val confDir: Path = Files.createDirectories(workDir.resolve("conf"))

  def write(name: String, text: String): Path = {
    val file = confDir.resolve(name)
    Files.createDirectories(file.getParent)
    Files.write(file, text.getBytes("UTF-8"))
  }

  def report(error: ConfigError): Unit =
    println(error.getMessage.replace(workDir.toString, "<work>"))

  def decode(source: Either[ConfigError, ConfigSource]): Either[List[ConfigError], BookstoreConfig] =
    source.left.map(List(_)).flatMap(Config.load[BookstoreConfig](_))

  def loadBookstore(path: Path): Either[List[ConfigError], BookstoreConfig] =
    decode(ConfigSource.fromFile(path.toString))

  val productionEnv: ConfigSource = EnvSource.keyFormat(KeyFormat.UpperSnakeCase).prefix("bookstore")

  val systemProperties: ConfigSource = SysPropSource.prefix("bookstore")

  def main(args: Array[String]): Unit = {
    val applicationConf = write(
      "application.conf",
      """server {
        |  host = "0.0.0.0"
        |  port = 8080
        |}
        |catalog {
        |  featuredBook = "Zionomicon"
        |  pageSize = 20
        |}
        |db {
        |  url = "jdbc:postgresql://localhost:5432/bookstore"
        |  user = "bookstore"
        |  password = "dev-password"
        |  poolSize = 10
        |}
        |""".stripMargin
    )

    val fileSource = ConfigSource.fromFile(applicationConf.toString) match {
      case Right(source) => source
      case Left(error)   => throw error
    }

    // System properties win over the environment, and the environment wins over the file.
    val settings = systemProperties.orElse(productionEnv).orElse(fileSource)

    val loaded = Config.loadWithProvenance[BookstoreConfig](settings) match {
      case Right(provenance) => provenance
      case Left(errors)      => throw errors.head
    }

    println(loaded.value)
    println(loaded.provenanceOf("catalog.pageSize"))
    println(loaded.provenanceOf("server.port"))
  }
}
```

</details>

Run it from the `config-examples` directory, with an environment variable and a system property set. The build runs each example in a separate JVM, so the system property goes into `run / javaOptions`; a `-D` flag on the `sbt` command line would reach only sbt itself:

```bash
BOOKSTORE_DB_PASSWORD=production-password sbt 'set run / javaOptions += "-Dbookstore.catalog.pageSize=50"' "runMain loadconfigurationfiles.Step6EnvironmentOverrides"
```

### Loading application.conf From the Classpath

<details>
  <summary>config-examples/src/main/scala/loadconfigurationfiles/Step7Classpath.scala</summary>

```scala title="config-examples/src/main/scala/loadconfigurationfiles/Step7Classpath.scala" showLineNumbers
package loadconfigurationfiles

/**
 * How to Load Configuration Files on the JVM and in the Browser — Loading
 * application.conf From the Classpath
 *
 * Reads a single application.conf from the classpath with fromHocon, then
 * resolves the includes of a split configuration with HoconParser.parse and a
 * resource-reading include function.
 *
 * Run with: cd config-examples && sbt "runMain
 * loadconfigurationfiles.Step7Classpath"
 */
object Step7Classpath {

  final case class ServerConfig(host: String, port: Int)
  object ServerConfig {
    implicit val schema: Schema[ServerConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredBook: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class BookstoreConfig(server: ServerConfig, catalog: CatalogConfig, db: DbConfig)
  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  val workDir: Path = Files.createTempDirectory("bookstore").toRealPath()
  val confDir: Path = Files.createDirectories(workDir.resolve("conf"))

  def write(name: String, text: String): Path = {
    val file = confDir.resolve(name)
    Files.createDirectories(file.getParent)
    Files.write(file, text.getBytes("UTF-8"))
  }

  def report(error: ConfigError): Unit =
    println(error.getMessage.replace(workDir.toString, "<work>"))

  def decode(source: Either[ConfigError, ConfigSource]): Either[List[ConfigError], BookstoreConfig] =
    source.left.map(List(_)).flatMap(Config.load[BookstoreConfig](_))

  def loadBookstore(path: Path): Either[List[ConfigError], BookstoreConfig] =
    decode(ConfigSource.fromFile(path.toString))

  def resourceText(name: String): Option[String] =
    Option(getClass.getClassLoader.getResourceAsStream(name)).map { in =>
      try Source.fromInputStream(in, "UTF-8").mkString
      finally in.close()
    }

  val fromClasspath: Either[ConfigError, ConfigSource] =
    resourceText("loadconfigurationfiles/application.conf") match {
      case Some(text) => ConfigSource.fromHocon(text, "classpath:application.conf")
      case None       => Left(ConfigError.ParseError("application.conf", "classpath", "existing resource"))
    }

  def parseSplit(root: String): Either[ConfigError, ConfigSource] =
    resourceText(s"loadconfigurationfiles/split/$root") match {
      case None       => Left(ConfigError.ParseError("", s"classpath:$root", "existing resource"))
      case Some(text) =>
        HoconParser.parse(text, name => resourceText(s"loadconfigurationfiles/split/$name")) match {
          case Right(tree) => Right(ConfigSource.fromMap(HoconValue.flatten(tree), s"hocon:$root"))
          case Left(error) => Left(ConfigError.ParseError("", s"hocon:$root", "valid HOCON", Some(error)))
        }
    }

  def main(args: Array[String]): Unit = {
    println(decode(fromClasspath))
    println(decode(parseSplit("application.conf")))
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain loadconfigurationfiles.Step7Classpath"
```

### Loading the Same Settings in the Browser

<details>
  <summary>config-examples/src/main/scala/loadconfigurationfiles/Step8EmbeddedConfig.scala</summary>

```scala title="config-examples/src/main/scala/loadconfigurationfiles/Step8EmbeddedConfig.scala" showLineNumbers
package loadconfigurationfiles

/**
 * How to Load Configuration Files on the JVM and in the Browser — Loading the
 * Same Settings in the Browser
 *
 * The platform-independent part of the browser setup: the configuration is
 * compiled into the program as a HOCON String constant, or as a map of split
 * files, and parsed with fromHocon or HoconParser.parse. It runs on the JVM
 * here and behaves the same on Scala.js.
 *
 * Run with: cd config-examples && sbt "runMain
 * loadconfigurationfiles.Step8EmbeddedConfig"
 */
object Step8EmbeddedConfig {

  final case class ServerConfig(host: String, port: Int)
  object ServerConfig {
    implicit val schema: Schema[ServerConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredBook: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class BookstoreConfig(server: ServerConfig, catalog: CatalogConfig, db: DbConfig)
  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  val workDir: Path = Files.createTempDirectory("bookstore").toRealPath()
  val confDir: Path = Files.createDirectories(workDir.resolve("conf"))

  def write(name: String, text: String): Path = {
    val file = confDir.resolve(name)
    Files.createDirectories(file.getParent)
    Files.write(file, text.getBytes("UTF-8"))
  }

  def report(error: ConfigError): Unit =
    println(error.getMessage.replace(workDir.toString, "<work>"))

  def decode(source: Either[ConfigError, ConfigSource]): Either[List[ConfigError], BookstoreConfig] =
    source.left.map(List(_)).flatMap(Config.load[BookstoreConfig](_))

  def loadBookstore(path: Path): Either[List[ConfigError], BookstoreConfig] =
    decode(ConfigSource.fromFile(path.toString))

  val embeddedConf: String =
    """server { host = "0.0.0.0", port = 8080 }
      |catalog { featuredBook = "Zionomicon", pageSize = 20 }
      |db {
      |  url = "jdbc:postgresql://localhost:5432/bookstore"
      |  user = "bookstore"
      |  password = "dev-password"
      |  poolSize = 10
      |}
      |""".stripMargin

  val bundledFiles: Map[String, String] = Map(
    "application.conf" ->
      """include "server.conf"
        |include "catalog.conf"
        |db {
        |  include "db.conf"
        |}
        |""".stripMargin,
    "server.conf"  -> "server { host = \"0.0.0.0\", port = 8080 }\n",
    "catalog.conf" -> "catalog { featuredBook = \"Zionomicon\", pageSize = 20 }\n",
    "db.conf"      ->
      """url = "jdbc:postgresql://localhost:5432/bookstore"
        |user = "bookstore"
        |password = "dev-password"
        |poolSize = 10
        |""".stripMargin
  )

  def parseBundled(files: Map[String, String], root: String): Either[ConfigError, ConfigSource] =
    HoconParser.parse(files(root), files.get) match {
      case Right(tree) => Right(ConfigSource.fromMap(HoconValue.flatten(tree), s"hocon:$root"))
      case Left(error) => Left(ConfigError.ParseError("", s"hocon:$root", "valid HOCON", Some(error)))
    }

  def main(args: Array[String]): Unit = {
    println(decode(ConfigSource.fromHocon(embeddedConf, "embedded.conf")))
    println(decode(parseBundled(bundledFiles, "application.conf")))
  }
}
```

</details>

`config-examples` runs on the JVM, so this example runs the platform-independent part, the embedded files and `HoconParser.parse`, on the JVM. Run it from the `config-examples` directory:

```bash
sbt "runMain loadconfigurationfiles.Step8EmbeddedConfig"
```

### The Complete Example

<details>
  <summary>config-examples/src/main/scala/loadconfigurationfiles/CompleteExample.scala</summary>

```scala title="config-examples/src/main/scala/loadconfigurationfiles/CompleteExample.scala" showLineNumbers
package loadconfigurationfiles

/**
 * How to Load Configuration Files on the JVM and in the Browser — Putting It
 * Together
 *
 * Writes a split application.conf and a defaults.conf into a temporary folder,
 * loads them with fromFile locked to that folder, layers system properties and
 * BOOKSTORE_* environment variables over them, prints the settings and where
 * three of them came from, and shows the errors for an include outside the
 * folder and for a broken YAML file.
 *
 * Run with: cd config-examples && BOOKSTORE_DB_PASSWORD=production-password sbt
 * "runMain loadconfigurationfiles.CompleteExample"
 */
object CompleteExample {

  final case class ServerConfig(host: String, port: Int)
  object ServerConfig {
    implicit val schema: Schema[ServerConfig] = Schema.derived
  }

  final case class CatalogConfig(featuredBook: String, pageSize: Int)
  object CatalogConfig {
    implicit val schema: Schema[CatalogConfig] = Schema.derived
  }

  final case class DbConfig(url: String, user: String, password: Secret, poolSize: Int)
  object DbConfig {
    implicit val schema: Schema[DbConfig] = Schema.derived
  }

  final case class BookstoreConfig(server: ServerConfig, catalog: CatalogConfig, db: DbConfig)
  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  val workDir: Path = Files.createTempDirectory("bookstore").toRealPath()
  val confDir: Path = Files.createDirectories(workDir.resolve("conf"))

  def write(name: String, text: String): Path = {
    val file = confDir.resolve(name)
    Files.createDirectories(file.getParent)
    Files.write(file, text.getBytes("UTF-8"))
  }

  def report(error: ConfigError): Unit =
    println(error.getMessage.replace(workDir.toString, "<work>"))

  def decode(source: Either[ConfigError, ConfigSource]): Either[List[ConfigError], BookstoreConfig] =
    source.left.map(List(_)).flatMap(Config.load[BookstoreConfig](_))

  def loadBookstore(path: Path): Either[List[ConfigError], BookstoreConfig] =
    decode(ConfigSource.fromFile(path.toString))

  val productionEnv: ConfigSource = EnvSource.keyFormat(KeyFormat.UpperSnakeCase).prefix("bookstore")

  val systemProperties: ConfigSource = SysPropSource.prefix("bookstore")

  def main(args: Array[String]): Unit = {
    // 1. Write a split application.conf and a shared defaults.conf.
    write(
      "defaults.conf",
      """server {
        |  host = "0.0.0.0"
        |  port = 8080
        |}
        |catalog {
        |  featuredBook = "Programming in Scala"
        |  pageSize = 20
        |}
        |db {
        |  host = "localhost"
        |  port = 5432
        |  name = "bookstore"
        |  url = "jdbc:postgresql://"${db.host}":"${db.port}"/"${db.name}
        |  user = "bookstore"
        |  password = "dev-password"
        |  poolSize = 10
        |}
        |""".stripMargin
    )
    write("db/pool.conf", "poolSize = 10\n")
    write(
      "db/db.conf",
      """user = "bookstore"
        |password = "dev-password"
        |include "pool.conf"
        |""".stripMargin
    )
    val applicationConf = write(
      "application.conf",
      """include "defaults.conf"
        |catalog.featuredBook = "Zionomicon"
        |db {
        |  host = "db.internal"
        |  include "db/db.conf"
        |}
        |""".stripMargin
    )

    // 2. Load the file locked to the configuration folder, then layer the process settings over it.
    val fileSource = ConfigSource.fromFile(applicationConf.toString, allowedBase = Some(confDir.toFile)) match {
      case Right(source) => source
      case Left(error)   => throw error
    }
    val settings = systemProperties.orElse(productionEnv).orElse(fileSource)

    val loaded = Config.loadWithProvenance[BookstoreConfig](settings) match {
      case Right(provenance) => provenance
      case Left(errors)      => throw errors.head
    }

    // 3. The settings and where three non-secret values came from.
    println(loaded.value)
    println(loaded.provenanceOf("catalog.pageSize"))
    println(loaded.provenanceOf("catalog.featuredBook"))
    println(loaded.provenanceOf("server.port"))

    // 4. An include that leaves the folder fails the whole load.
    Files.write(workDir.resolve("secrets.conf"), "db.password = \"production-password\"\n".getBytes("UTF-8"))
    val escapeConf = write("escape.conf", "include \"../secrets.conf\"\n")
    ConfigSource.fromFile(escapeConf.toString, allowedBase = Some(confDir.toFile)) match {
      case Left(ConfigError.ParseError(_, _, _, Some(cause))) =>
        println(cause.getMessage.replace(workDir.toString, "<work>"))
      case other => println(other.map(_.sourceId))
    }

    // 5. A broken YAML file parses, but the typed load reports every missing key.
    val brokenYaml = write(
      "broken.yaml",
      """server:
        |  host: 0.0.0.0
        |  port: 8080
        |catalog:
        |  featuredBook: Zionomicon
        |  pageSize:20
        |db:
        |  url: jdbc:postgresql://localhost:5432/bookstore
        |  user: bookstore
        |  password: dev-password
        |  poolSize: 10
        |""".stripMargin
    )
    val yamlText = new String(Files.readAllBytes(brokenYaml), "UTF-8")
    decode(ConfigSource.fromYaml(yamlText, brokenYaml.getFileName.toString)).left.foreach(_.foreach(report))
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
BOOKSTORE_DB_PASSWORD=production-password sbt "runMain loadconfigurationfiles.CompleteExample"
```

## Going Further

- [Formats](../reference/config/formats.md) lists every HOCON, JSON and YAML feature the format modules support.
- [ConfigSource](../reference/config/config-source.md) covers the other ways to build and combine sources, key formats and provenance.
- [Errors](../reference/config/errors.md) describes every `ConfigError` case and how a failed load is reported.
- [Coming from PureConfig, Typesafe Config or zio-config](./config-library-migration.md) moves an existing Typesafe Config or PureConfig application over, including its kebab-case keys.
- [Build a Small JSON Backend from Blocks](./small-json-backend-from-blocks.md) loads its settings as part of a complete service.

---

## How to Map Custom Column Types

In this guide, we store a bookstore's `Book` in one SQL table, field by field, without flattening the model or writing conversions by hand. By the end, you will have a repository that writes and reads a validated ISBN newtype, an opaque ID, an enum, a nested case class stored as JSON, a list of tags stored as a PostgreSQL array, and a publication timestamp that comes back exactly as it went in, on SQLite and on PostgreSQL.

We take one column type at a time:

- A newtype and an opaque type, each in one plain column
- An enum, stored as the name of its case
- A nested case class, stored as JSON (`TEXT` on SQLite, `jsonb` on PostgreSQL)
- A `List[String]`, stored as JSON on SQLite and as a `text[]` array on PostgreSQL
- An `Instant`, stored without time-zone shifts
- Nullable versions of these columns
- One repository with every custom column

This guide builds on [Build Your First Database Repository on SQLite](./your-first-sqlite-repository.md). We use `JdbcTransactor`, `Repo`, `Table` and `sql"..."` the way that tutorial introduces them and don't explain them again.

## The Problem

The `sql` module maps simple fields such as `String`, `Int` and `Long` to columns without any work. A real model has more than that: an `Isbn` that must be valid, an enum for the format, a nested record of publisher details, a list of tags and a timestamp. Without a mapping for those types, we either flatten the model into strings and numbers, or we read and write every column by hand.

Written with plain JDBC, reading one book looks like this:

```scala

enum Format { case Hardcover, Paperback, Ebook }
case class Metadata(publisher: String, edition: Int, pages: Int)
case class Book(isbn: String, title: String, format: Format, metadata: Metadata, tags: List[String], publishedAt: Instant)

def findBook(conn: Connection, isbn: String): Option[Book] = {
  val query = conn.prepareStatement("SELECT isbn, title, format, metadata, tags, published_at FROM books WHERE isbn = ?")
  query.setString(1, isbn)
  val rs = query.executeQuery()
  try {
    if (!rs.next()) None
    else {
      // The JSON document: parse it by hand, or pull in a JSON library
      val json      = rs.getString("metadata")
      val fields    = json.stripPrefix("{").stripSuffix("}").split(",").map(_.split(":", 2)).map(kv => kv(0).trim.stripPrefix("\"").stripSuffix("\"") -> kv(1).trim).toMap
      val metadata  = Metadata(fields("publisher").stripPrefix("\"").stripSuffix("\""), fields("edition").toInt, fields("pages").toInt)
      // The array: one more JDBC call, and a cast
      val tags      = rs.getArray("tags").getArray.asInstanceOf[Array[String]].toList
      // The timestamp: forget the calendar and the JVM's time zone shifts it
      val utc       = Calendar.getInstance(TimeZone.getTimeZone("UTC"))
      val published = rs.getTimestamp("published_at", utc).toInstant
      Some(Book(rs.getString("isbn"), rs.getString("title"), Format.valueOf(rs.getString("format")), metadata, tags, published))
    }
  } finally query.close()
}
```

Every custom type adds its own conversion, and each one is a place to get it wrong. The ISBN is a plain `String`, so nothing checks it. `Format.valueOf` throws on a value written by another program. The hand-written JSON parser breaks on the first comma inside a publisher's name. Writing the row back needs the same conversions in reverse, and they have to stay in sync with this function.

In this guide, we give each type a single description, mostly through its `Schema`, and let the `sql` module derive the column mapping from it, in both directions.

## Prerequisites

Add the `sql` module and the JDBC drivers to your `build.sbt`. [How to Add ZIO Blocks to Your Build](../getting-started/installation.md) explains the two artifact names:

```scala
// Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-sql" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-sql" % "0.0.56"
scalacOptions += "-experimental"

// The JDBC drivers, for every Scala version
libraryDependencies += "org.xerial"     % "sqlite-jdbc" % "3.53.4.0"
libraryDependencies += "org.postgresql" % "postgresql"  % "42.7.14"
```

The `sql` module is published for Scala 3 only, and `JdbcTransactor` runs on the JVM and needs JDK 21 or newer. Opaque types and `enum` are Scala 3 features, so the code in this guide uses Scala 3 syntax throughout.

The SQLite parts run on a temporary database file. For the PostgreSQL parts, you need a PostgreSQL server; [How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md) shows a `docker run` command that starts one with a `bookstore` database, user and password. The PostgreSQL code reads the password from the `BOOKSTORE_DB_PASSWORD` environment variable.

## The Bookstore Table

Our `books` table holds one row per book, and every field except the title needs a decision about its column. This table is the plan for the rest of the guide:

| Field         | Scala type                  | SQLite column            | PostgreSQL column | What we write                                     |
| ------------- | --------------------------- | ------------------------ | ----------------- | ------------------------------------------------- |
| `id`          | `opaque type BookId = Long` | `INTEGER`                | `BIGINT`          | Nothing                                           |
| `isbn`        | `Isbn` newtype              | `TEXT`                   | `TEXT`            | A `Schema` built with `Schema#transform`          |
| `format`      | `enum Format`               | `TEXT`                   | `TEXT`            | Nothing (the case name is stored)                 |
| `metadata`    | `case class Metadata`       | `TEXT` holding JSON      | `JSONB`           | A `Repo` built from the codec's columns           |
| `tags`        | `List[String]`              | `TEXT` holding JSON      | `TEXT[]`          | `PgCodec.listStringCodec` on PostgreSQL           |
| `publishedAt` | `java.time.Instant`         | `INTEGER` (milliseconds) | `TIMESTAMPTZ`     | Nothing, once the column type is right            |

One thing applies to every row of this table: we write the `CREATE TABLE` statement ourselves. `Table#createTable` declares no primary key in version 0.0.56, and it picks the wrong column type for some of these fields, as the sections below show.

## Store a Newtype or an Opaque Type in One Column

A newtype is a one-field case class that gives a plain value its own type, so an `Isbn` can't be passed where a title is expected. The obvious version, a case class with a derived `Schema`, doesn't map to one column. To the `sql` module it is a record with a field called `value`:

```scala

final case class PlainIsbn(value: String)

object PlainIsbn {
  given Schema[PlainIsbn] = Schema.derived
}

case class PlainBook(id: Long, isbn: PlainIsbn)

object PlainBook {
  given Schema[PlainBook] = Schema.derived
}
```

The table expects a column named after the record's field, and the codec writes the value as a JSON object:

```scala
println(Table.derived[PlainBook].columns)
// Vector(id, isbn_value)
println(DbCodec.derived[PlainBook].toDbValues(PlainBook(2L, PlainIsbn("9780981531601"))))
// Vector(DbLong(2), DbString({"value":"9780981531601"}))
```

What we want is a `TEXT` column that holds `9780981531601`. The fix is to describe `Isbn` as a `String` with a check on the way in. This is the validated newtype from [How to Define Validated Newtypes](./validated-newtypes.md): a private constructor, a smart constructor `Isbn.parse` that returns `Either[SchemaError, Isbn]`, and a `Schema` built with `Schema#transform`, which takes a function from `String` to `Isbn` and one back:

```scala
final case class Isbn private (value: String)

object Isbn {
  def parse(input: String): Either[SchemaError, Isbn] = {
    val digits = input.filterNot(c => c == '-' || c == ' ')
    if (digits.length != 13 || !digits.forall(_.isDigit))
      Left(SchemaError.validationFailed(s"ISBN must have 13 digits, got '$input'"))
    else {
      val weighted = digits.init.zipWithIndex.map { case (c, i) => c.asDigit * (if (i % 2 == 0) 1 else 3) }
      if ((10 - weighted.sum % 10) % 10 == digits.last.asDigit) Right(new Isbn(digits))
      else Left(SchemaError.validationFailed(s"ISBN check digit is wrong in '$input'"))
    }
  }

  def unsafe(input: String): Isbn = parse(input).fold(e => throw e, identity)

  given Schema[Isbn] = Schema[String].transform(unsafe, _.value)
}
```

A Scala 3 opaque type needs even less. `BookId` is a `Long` at runtime, and the `Schema` and `DbCodec` macros see through it, so it maps to an integer column with no `Schema` of its own. Its companion needs `BookId.apply` and `BookId.unwrap`, which the `DbCodec` for the ID column uses to convert to and from `Long`:

```scala
object ids {
  opaque type BookId = Long

  object BookId {
    def apply(value: Long): BookId = value
    def unwrap(id: BookId): Long   = id
  }
}

case class Book(id: BookId, title: String, isbn: Isbn)

object Book {
  given Schema[Book] = Schema.derived
}

val books = Repo.derived[Book, BookId]("books", "id", _.id)

val dbFile     = java.nio.file.Files.createTempFile("bookstore", ".db")
val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
```

With both fields mapped, the row stores plain values, and `Repo#all` turns them back into `BookId` and `Isbn`:

```scala
transactor.connect {
  sql"""CREATE TABLE books (
    id    INTEGER PRIMARY KEY,
    title TEXT NOT NULL,
    isbn  TEXT NOT NULL
  )""".update
  books.insert(Book(BookId(1L), "Programming in Scala", Isbn.unsafe("978-0-9815316-0-1")))
  books.insert(Book(BookId(2L), "Functional Programming in Scala", Isbn.unsafe("978-1-61729-065-7")))
  println(sql"SELECT id, title, isbn FROM books ORDER BY id".query[(Long, String, String)])
  books.all.foreach(println)
}
// List((1,Programming in Scala,9780981531601), (2,Functional Programming in Scala,9781617290657))
// Book(1,Programming in Scala,Isbn(9780981531601))
// Book(2,Functional Programming in Scala,Isbn(9781617290657))
```

`Isbn.parse` stored the bare digits, so `978-0-9815316-0-1` went in as `9780981531601`. To filter by ISBN in our own SQL, we pass the underlying `String`. The `sql"..."` interpolator has no parameter instance for `Isbn`, because the `DbCodec` it looks for can't be derived for a case class with a private constructor:

```scala
transactor.connect {
  val isbn = Isbn.unsafe("9780981531601")
  println(sql"SELECT title FROM books WHERE isbn = ${isbn.value}".query[String])
}
// List(Programming in Scala)
```

The check also runs when we read. If another program writes an ISBN with a wrong check digit, every query that returns that row fails with the `SchemaError` from `Isbn.parse`, while queries that skip the row still work:

```scala
transactor.connect {
  sql"INSERT INTO books (id, title, isbn) VALUES (3, 'Scala with Cats', '9781617290658')".update
  println(Try(books.all))
  println(books.find(BookId(2L)))
}
// Failure(zio.blocks.schema.SchemaError: ISBN check digit is wrong in '9781617290658')
// Book(2,Functional Programming in Scala,Isbn(9781617290657))
```

:::tip
A newtype doesn't have to validate. For a plain wrapper, `Schema[String].transform(Isbn(_), _.value)` stores it the same way.
:::

## Store an Enum as Its Case Name

An enum whose cases have no fields needs no work at all: the derived codec stores the name of the case in a `TEXT` column and reads it back by name. We start a fresh scope for this section, with its own database file:

```scala

enum Format {
  case Hardcover, Paperback, Ebook
}

object Format {
  given Schema[Format] = Schema.derived
}

case class Book(id: Long, title: String, format: Format)

object Book {
  given Schema[Book] = Schema.derived
}

val books = Repo.derived[Book, Long]("books", "id", _.id)

val dbFile     = java.nio.file.Files.createTempFile("bookstore", ".db")
val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
```

The column holds `Ebook` and `Paperback`, and an enum value works as a parameter in our own SQL:

```scala
transactor.connect {
  sql"CREATE TABLE books (id INTEGER PRIMARY KEY, title TEXT NOT NULL, format TEXT NOT NULL)".update
  books.insert(Book(1L, "Zionomicon", Format.Ebook))
  books.insert(Book(2L, "Programming in Scala", Format.Paperback))
  println(sql"SELECT title, format FROM books ORDER BY id".query[(String, String)])
  println(sql"SELECT title FROM books WHERE format = ${Format.Ebook}".query[String])
}
// List((Zionomicon,Ebook), (Programming in Scala,Paperback))
// List(Zionomicon)
```

The codec never stores the ordinal, so reordering or inserting cases doesn't change what existing rows mean. Renaming a case does. When the stored values don't match the case names, for example in an existing table that uses lower case, `@Modifier.rename` sets the stored name of a case:

```scala
enum LegacyFormat {
  @Modifier.rename("hardcover") case Hardcover
  @Modifier.rename("paperback") case Paperback
  @Modifier.rename("ebook") case Ebook
}

object LegacyFormat {
  given Schema[LegacyFormat] = Schema.derived
}

case class LegacyBook(id: Long, title: String, format: LegacyFormat)

object LegacyBook {
  given Schema[LegacyBook] = Schema.derived
}

val legacyBooks = Repo.derived[LegacyBook, Long]("legacy_books", "id", _.id)
```

The renamed case is written as `ebook` and read back as `LegacyFormat.Ebook`:

```scala
transactor.connect {
  sql"CREATE TABLE legacy_books (id INTEGER PRIMARY KEY, title TEXT NOT NULL, format TEXT NOT NULL)".update
  legacyBooks.insert(LegacyBook(1L, "Zionomicon", LegacyFormat.Ebook))
  println(sql"SELECT title, format FROM legacy_books".query[(String, String)])
  println(legacyBooks.all)
}
// List((Zionomicon,ebook))
// List(LegacyBook(1,Zionomicon,Ebook))
```

A value that matches no case fails the read with an `IllegalArgumentException` that names the value and the cases it expected:

```scala
transactor.connect {
  sql"INSERT INTO books (id, title, format) VALUES (3, 'Functional Programming in Scala', 'Audiobook')".update
  println(Try(books.all))
}
// Failure(java.lang.IllegalArgumentException: Unknown enum variant 'Audiobook'. Expected one of: Hardcover, Paperback, Ebook)
```

:::warning
Add a new case to the enum before any program writes its name to the table. Until then, every query that returns such a row fails.
:::

## Store a Nested Case Class as JSON

The `Metadata` record has three fields, and we want it in one column. The derived codec already does this: a field whose type is a case class is stored as one JSON column. The catch, in version 0.0.56, is that `Repo.derived` and `Table.derived` compute the table's columns separately from the codec, and they flatten the record into one column per field:

```scala

case class Metadata(publisher: String, edition: Int, pages: Int)

object Metadata {
  given Schema[Metadata] = Schema.derived
}

case class Book(id: Long, title: String, metadata: Metadata)

object Book {
  given Schema[Book] = Schema.derived
}

// Page counts are illustrative
val zionomicon         = Book(1L, "Zionomicon", Metadata("Ziverge", 1, 700))
val programmingInScala = Book(2L, "Programming in Scala", Metadata("Artima", 1, 776))

val dbFile     = java.nio.file.Files.createTempFile("bookstore", ".db")
val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
```

The table and the codec disagree about the columns:

```scala
println(Table.derived[Book].columns)
// Vector(id, title, metadata_publisher, metadata_edition, metadata_pages)
println(DbCodec.derived[Book].columns)
// Vector(id, title, metadata)
```

A `Repo` from `Repo.derived` names five columns in its `INSERT` but has only three values to bind, so every insert fails:

```scala
transactor.connect {
  sql"""CREATE TABLE books (
    id       INTEGER PRIMARY KEY,
    title    TEXT NOT NULL,
    metadata TEXT NOT NULL
  )""".update
  println(Try(Repo.derived[Book, Long]("books", "id", _.id).insert(zionomicon)))
}
// Failure(java.lang.IllegalArgumentException: requirement failed: Repo.buildInsertFrag: columns/value count mismatch)
```

Until this is fixed, we build the `Table` from the codec's own columns and pass it to `Repo.apply`. `Table` takes a name, a codec and one `ColumnMeta` per column; the column type in a `ColumnMeta` is read only by `Table#createTable`, which we don't call:

```scala
def tableFor[A](name: String, codec: DbCodec[A]): Table[A] =
  Table(name, codec, codec.columns.map(column => ColumnMeta(column, DbValue.DbString(""), nullable = false)))

val bookCodec: DbCodec[Book] = DbCodec.derived[Book]
val books                    = Repo(tableFor("books", bookCodec), "id", DbCodec[Long], (book: Book) => book.id)
```

Now the column holds a JSON document, and `Repo#all` decodes it back into `Metadata`. SQLite's `json_extract` reads a field inside the document, so we can filter on the publisher in our own SQL:

```scala
transactor.connect {
  books.insert(zionomicon)
  books.insert(programmingInScala)
  println(sql"SELECT title, metadata FROM books ORDER BY id".query[(String, String)])
  books.all.foreach(println)

  val publisher = "Artima"
  println(sql"SELECT id, title, metadata FROM books WHERE json_extract(metadata, '$$.publisher') = $publisher".query[Book])
}
// List((Zionomicon,{"publisher":"Ziverge","edition":1,"pages":700}), (Programming in Scala,{"publisher":"Artima","edition":1,"pages":776}))
// Book(1,Zionomicon,Metadata(Ziverge,1,700))
// Book(2,Programming in Scala,Metadata(Artima,1,776))
// List(Book(2,Programming in Scala,Metadata(Artima,1,776)))
```

The `$$` in `'$$.publisher'` is an escaped `$`, so SQLite receives the JSON path `$.publisher`. A document that doesn't match `Metadata` fails when the row is read, with the JSON error inside the message:

```scala
transactor.connect {
  sql"""INSERT INTO books (id, title, metadata) VALUES (3, 'Functional Programming in Scala', '{"publisher":"Manning"}')""".update
  println(Try(books.find(3L)))
}
// Failure(java.lang.RuntimeException: JSONB decode error: zio.blocks.schema.SchemaError: missing required field "edition" at: .)
```

:::tip
To store the record's fields as separate columns instead, annotate the field with `@Modifier.config("sql.inline", "true")`. The codec then writes `metadata_publisher`, `metadata_edition` and `metadata_pages`, the same columns `Table.derived` lists, and `Repo.derived` works unchanged.
:::

## Use a jsonb Column on PostgreSQL

On PostgreSQL, the natural column type for a JSON document is `jsonb`: it is stored parsed, so the server can index it and query inside it. The codec binds the JSON as a string parameter, and by default the PostgreSQL JDBC driver sends string parameters as `character varying`, which PostgreSQL refuses to put into a `jsonb` column without a cast. The driver's `stringtype=unspecified` URL setting sends them untyped instead, and PostgreSQL then converts the text to `jsonb` itself. The same `books` repository works on PostgreSQL once the URL has that setting:

```scala
val baseUrl  = sys.env.getOrElse("BOOKSTORE_DB_URL", "jdbc:postgresql://localhost:5432/bookstore")
val user     = sys.env.getOrElse("BOOKSTORE_DB_USER", "bookstore")
val password = sys.env("BOOKSTORE_DB_PASSWORD")

// Without the setting: the insert fails
JdbcTransactor.fromUrl(baseUrl, user, password, SqlDialect.PostgreSQL).connect {
  sql"DROP TABLE IF EXISTS books".update
  sql"""CREATE TABLE books (
    id       BIGINT PRIMARY KEY,
    title    TEXT NOT NULL,
    metadata JSONB NOT NULL
  )""".update
  println(Try(books.insert(zionomicon)).fold(_.getMessage.linesIterator.next(), n => s"Inserted $n"))
}

// With the setting: the same insert works, and ->> reads a field inside the document
val jsonbUrl   = baseUrl + (if (baseUrl.contains("?")) "&" else "?") + "stringtype=unspecified"
val postgresTx = JdbcTransactor.fromUrl(jsonbUrl, user, password, SqlDialect.PostgreSQL)
postgresTx.connect {
  books.insert(zionomicon)
  books.insert(programmingInScala)
  println(sql"SELECT title, metadata::text FROM books ORDER BY id".query[(String, String)])
  books.all.foreach(println)

  val publisher = "Artima"
  println(sql"SELECT id, title, metadata FROM books WHERE metadata->>'publisher' = $publisher".query[Book])
}
```

Run against PostgreSQL 16 in Docker, it prints:

```text
ERROR: column "metadata" is of type jsonb but expression is of type character varying
List((Zionomicon,{"pages": 700, "edition": 1, "publisher": "Ziverge"}), (Programming in Scala,{"pages": 776, "edition": 1, "publisher": "Artima"}))
Book(1,Zionomicon,Metadata(Ziverge,1,700))
Book(2,Programming in Scala,Metadata(Artima,1,776))
List(Book(2,Programming in Scala,Metadata(Artima,1,776)))
```

PostgreSQL reorders the keys when it stores a `jsonb` document, which doesn't matter to the decoder. `stringtype=unspecified` applies to every string parameter on the connection, and PostgreSQL then infers each one's type from where it appears. If you'd rather keep the default, write the `INSERT` yourself with a cast, `${json}::jsonb`, instead of using `Repo#insert`.

## Store a List as a PostgreSQL Array

A `List[String]` field goes through the same JSON path as a nested record: the derived codec stores it as a JSON array in one column. `Table.derived` hits the same bug in a different way and throws when it meets the list, so we use `tableFor` again:

```scala

case class Book(id: Long, title: String, tags: List[String])

object Book {
  given Schema[Book] = Schema.derived
}

def tableFor[A](name: String, codec: DbCodec[A]): Table[A] =
  Table(name, codec, codec.columns.map(column => ColumnMeta(column, DbValue.DbString(""), nullable = false)))

val bookCodec: DbCodec[Book] = DbCodec.derived[Book]
val books                    = Repo(tableFor("books", bookCodec), "id", DbCodec[Long], (book: Book) => book.id)

val dbFile     = java.nio.file.Files.createTempFile("bookstore", ".db")
val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
```

On SQLite, the tags land in a `TEXT` column as JSON, and SQLite's `json_each` lets us filter on one tag:

```scala
println(Try(Table.derived[Book]))
// Failure(java.lang.UnsupportedOperationException: Typed DDL does not support reflect node List[String])

transactor.connect {
  sql"CREATE TABLE books (id INTEGER PRIMARY KEY, title TEXT NOT NULL, tags TEXT NOT NULL)".update
  books.insert(Book(1L, "Zionomicon", List("zio", "effects", "concurrency")))
  books.insert(Book(2L, "Programming in Scala", List("scala", "fundamentals")))
  books.insert(Book(3L, "Functional Programming in Scala", List("scala", "functional-programming")))
  println(sql"SELECT title, tags FROM books ORDER BY id".query[(String, String)])

  val tag = "scala"
  println(
    sql"SELECT id, title, tags FROM books WHERE EXISTS (SELECT 1 FROM json_each(books.tags) WHERE value = $tag) ORDER BY id"
      .query[Book]
  )
}
// List((Zionomicon,["zio","effects","concurrency"]), (Programming in Scala,["scala","fundamentals"]), (Functional Programming in Scala,["scala","functional-programming"]))
// List(Book(2,Programming in Scala,List(scala, fundamentals)), Book(3,Functional Programming in Scala,List(scala, functional-programming)))
```

PostgreSQL has real arrays. A `text[]` column holds the tags as SQL values, so `ANY` and the overlap operator `&&` work on it and a GIN index can speed them up. `PgCodec`, which is JVM-only, provides the codec: `PgCodec.listStringCodec` reads and writes a `List[String]` as a SQL array (there is also `PgCodec.iArrayStringCodec` for `IArray[String]`; strings are the only element type it reads).

The row codec gets each field's codec from the field's `Schema`, so a `DbCodec[List[String]]` in scope doesn't change how `tags` is stored. To swap one field's codec, `DbCodec.derivedWith` derives the row codec with overrides: `DerivationBuilder#instance` takes the record's `TypeId`, the field name and the codec to use for it:

```scala
val pgBookCodec: DbCodec[Book] =
  DbCodec.derivedWith[Book](_.instance(TypeId.of[Book], "tags", PgCodec.listStringCodec))

val pgBooks = Repo(tableFor("books", pgBookCodec), "id", DbCodec[Long], (book: Book) => book.id)
```

`pgBooks` writes the tags as an array. In our own queries, `query[Book]` needs the same codec, so we make it the `given DbCodec[Book]` in that scope. To pass a Scala list as an array parameter, `import PgCodec.given` brings parameter instances for lists of `String`, `Int`, `Long`, `Double`, `Boolean` and `UUID`:

```scala
val baseUrl  = sys.env.getOrElse("BOOKSTORE_DB_URL", "jdbc:postgresql://localhost:5432/bookstore")
val user     = sys.env.getOrElse("BOOKSTORE_DB_USER", "bookstore")
val password = sys.env("BOOKSTORE_DB_PASSWORD")

JdbcTransactor.fromUrl(baseUrl, user, password, SqlDialect.PostgreSQL).connect {
  given DbCodec[Book] = pgBookCodec

  sql"DROP TABLE IF EXISTS books".update
  sql"CREATE TABLE books (id BIGINT PRIMARY KEY, title TEXT NOT NULL, tags TEXT[])".update

  // The derived codec writes JSON text, which a text[] column refuses
  println(Try(books.insert(Book(1L, "Zionomicon", List("zio")))).fold(_.getMessage.linesIterator.next(), n => s"Inserted $n"))

  pgBooks.insert(Book(1L, "Zionomicon", List("zio", "effects", "concurrency")))
  pgBooks.insert(Book(2L, "Programming in Scala", List("scala", "fundamentals")))
  pgBooks.insert(Book(3L, "Functional Programming in Scala", List("scala", "functional-programming")))
  println(sql"SELECT title, tags::text FROM books ORDER BY id".query[(String, String)])

  val tag = "scala"
  println(sql"SELECT id, title, tags FROM books WHERE $tag = ANY(tags) ORDER BY id".query[Book])

  val wanted = List("effects", "functional-programming")
  println(
    Try(sql"SELECT title FROM books WHERE tags && $wanted ORDER BY id".query[String])
      .fold(_.getMessage.linesIterator.next(), _.toString)
  )
  println(sql"SELECT title FROM books WHERE tags && $wanted::text[] ORDER BY id".query[String])
}
```

Run against PostgreSQL 16 in Docker, it prints:

```text
ERROR: column "tags" is of type text[] but expression is of type character varying
List((Zionomicon,{zio,effects,concurrency}), (Programming in Scala,{scala,fundamentals}), (Functional Programming in Scala,{scala,functional-programming}))
List(Book(2,Programming in Scala,List(scala, fundamentals)), Book(3,Functional Programming in Scala,List(scala, functional-programming)))
ERROR: operator does not exist: text[] && character varying[]
List(Zionomicon, Functional Programming in Scala)
```

`PgCodec` binds a list of strings as a `varchar[]` array. Inserting it into a `text[]` column works, because PostgreSQL converts it on assignment, but an operator such as `&&` needs both sides to have the same type, so the parameter needs a `::text[]` cast. A list of numbers works with `ANY` as it is: `id = ANY($ids)` with `ids: List[Long]` binds a `bigint[]`.

## Store Timestamps Without Time-Zone Shifts

An `Instant` is a moment on the timeline, with no time zone. The `sql` module writes it with JDBC's `setTimestamp` and reads it with `getTimestamp`, both with a UTC calendar, so the JVM's default time zone never shifts the value. What can still go wrong is the column type, and the right one differs between SQLite and PostgreSQL.

### Use an INTEGER Column on SQLite

SQLite has no timestamp type, and its JDBC driver stores the timestamp as the number of milliseconds since 1970. We start a fresh scope with a `Book` that has a publication date:

```scala

case class Book(id: Long, title: String, publishedAt: Instant)

object Book {
  given Schema[Book] = Schema.derived
}

val books = Repo.derived[Book, Long]("books", "id", _.id)

// Illustrative date, with microseconds to show the precision SQLite keeps
val zionomicon = Book(1L, "Zionomicon", Instant.parse("2021-07-15T09:30:00.123456Z"))

val dbFile     = java.nio.file.Files.createTempFile("bookstore", ".db")
val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
```

`Table#createTable` declares the column as `TEXT`. SQLite then stores the milliseconds as text, and the driver can't parse them back, so the row is written but can't be read:

```scala
println(books.table.createTable(SqlDialect.SQLite).sql(SqlDialect.SQLite))
// CREATE TABLE IF NOT EXISTS books (
//   id INTEGER NOT NULL,
//   title TEXT NOT NULL,
//   published_at TEXT NOT NULL
// )

transactor.connect {
  sql"CREATE TABLE books (id INTEGER PRIMARY KEY, title TEXT NOT NULL, published_at TEXT NOT NULL)".update
  books.insert(zionomicon)
  println(sql"SELECT published_at, typeof(published_at) FROM books".query[(String, String)])
  println(Try(books.all))
}
// List((1626341400123,text))
// Failure(java.sql.SQLException: Error parsing time stamp)
```

We replace the table with one whose column is `INTEGER`. It stores the same number as a number, and it reads back. Comparisons in SQL work too, because an `Instant` parameter is bound the same way:

```scala
transactor.connect {
  sql"DROP TABLE books".update
  sql"CREATE TABLE books (id INTEGER PRIMARY KEY, title TEXT NOT NULL, published_at INTEGER NOT NULL)".update
  books.insert(zionomicon)
  println(sql"SELECT published_at, typeof(published_at) FROM books".query[(Long, String)])
  println(books.all)

  val cutoff = Instant.parse("2020-01-01T00:00:00Z")
  println(sql"SELECT title FROM books WHERE published_at >= $cutoff".query[String])
}
// List((1626341400123,integer))
// List(Book(1,Zionomicon,2021-07-15T09:30:00.123Z))
// List(Zionomicon)
```

The value comes back as `09:30:00.123Z`: SQLite keeps milliseconds, so the microseconds are gone. If you need finer precision on SQLite, store the value yourself, for example as a `Long` of microseconds.

### Use a timestamptz Column on PostgreSQL

PostgreSQL has two timestamp types. `timestamptz` (`timestamp with time zone`) stores a moment, like `Instant`. `timestamp` (`timestamp without time zone`) stores a date and a clock time with no zone. Our own `Repo` round-trips an `Instant` through either one, because it always writes and reads UTC clock times. The trouble starts when anything else writes to a `timestamp` column: PostgreSQL converts a moment to the clock time of that session's time zone, and our UTC reader then gets a different moment.

To see it, we store the same `Instant` in one column of each type, then let plain SQL write the same moment into both from a session that runs in Tehran:

```scala
case class Release(id: Long, withZone: Instant, withoutZone: Instant)

object Release {
  given Schema[Release] = Schema.derived
}

val releases = Repo.derived[Release, Long]("releases", "id", _.id)

val baseUrl  = sys.env.getOrElse("BOOKSTORE_DB_URL", "jdbc:postgresql://localhost:5432/bookstore")
val user     = sys.env.getOrElse("BOOKSTORE_DB_USER", "bookstore")
val password = sys.env("BOOKSTORE_DB_PASSWORD")

// Illustrative date
val publishedAt = Instant.parse("2021-07-15T09:30:00.123456Z")

JdbcTransactor.fromUrl(baseUrl, user, password, SqlDialect.PostgreSQL).connect {
  sql"DROP TABLE IF EXISTS releases".update
  sql"""CREATE TABLE releases (
    id           BIGINT PRIMARY KEY,
    with_zone    TIMESTAMPTZ NOT NULL,
    without_zone TIMESTAMP NOT NULL
  )""".update

  // Written by our Repo
  releases.insert(Release(1L, publishedAt, publishedAt))

  // Written by another client whose session runs in Tehran
  sql"SET TIME ZONE 'Asia/Tehran'".update
  sql"""INSERT INTO releases VALUES (2,
    TIMESTAMPTZ '2021-07-15 09:30:00.123456+00',
    TIMESTAMPTZ '2021-07-15 09:30:00.123456+00')""".update
  sql"SET TIME ZONE 'UTC'".update

  println(sql"SELECT id, with_zone::text, without_zone::text FROM releases ORDER BY id".query[(Long, String, String)])
  releases.all.foreach(println)
}
```

Run against PostgreSQL 16 in Docker, it prints:

```text
List((1,2021-07-15 09:30:00.123456+00,2021-07-15 09:30:00.123456), (2,2021-07-15 09:30:00.123456+00,2021-07-15 14:00:00.123456))
Release(1,2021-07-15T09:30:00.123456Z,2021-07-15T09:30:00.123456Z)
Release(2,2021-07-15T09:30:00.123456Z,2021-07-15T14:00:00.123456Z)
```

Row 2's `timestamp` column holds 14:00, the clock time in Tehran (UTC+04:30 in July 2021), and we read it back as `14:00:00Z`, four and a half hours late. The `timestamptz` column gives the right moment in both rows. The same shift happens with a column default such as `DEFAULT now()`, or with a client whose session time zone isn't UTC. Use `timestamptz` for every `Instant`; it is also the type `Table#createTable` picks for one on PostgreSQL. PostgreSQL keeps microseconds, so the value round-trips exactly.

:::tip
If a field is a date and clock time with no zone, such as a store's opening time, model it as `java.time.LocalDateTime` with a `timestamp` column. The `sql` module writes a `LocalDateTime` without any time-zone conversion.
:::

## Make a Custom Column Nullable

A nullable column needs an `Option` field. A non-optional field doesn't fail on SQL `NULL`: a `String` reads it as `null` and an `Int` as `0`, so a `NULL` turns into a value that looks real:

```scala

val dbFile     = java.nio.file.Files.createTempFile("bookstore", ".db")
val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
```

Reading two `NULL`s into a `String` and an `Int`:

```scala
transactor.connect {
  println(sql"SELECT CAST(NULL AS TEXT), CAST(NULL AS INTEGER)".query[(String, Int)])
}
// List((null,0))
```

`Option` fields of plain types, such as `Option[String]`, read `NULL` as `None`. Two custom types need more care in version 0.0.56. We bring back the `Isbn` newtype and the `Metadata` record. `Metadata` also gets a `JsonCodec`, for the JSON codec we use further down:

```scala
final case class Isbn private (value: String)

object Isbn {
  def parse(input: String): Either[SchemaError, Isbn] = {
    val digits = input.filterNot(c => c == '-' || c == ' ')
    if (digits.length != 13 || !digits.forall(_.isDigit))
      Left(SchemaError.validationFailed(s"ISBN must have 13 digits, got '$input'"))
    else {
      val weighted = digits.init.zipWithIndex.map { case (c, i) => c.asDigit * (if (i % 2 == 0) 1 else 3) }
      if ((10 - weighted.sum % 10) % 10 == digits.last.asDigit) Right(new Isbn(digits))
      else Left(SchemaError.validationFailed(s"ISBN check digit is wrong in '$input'"))
    }
  }

  def unsafe(input: String): Isbn = parse(input).fold(e => throw e, identity)

  given Schema[Isbn] = Schema[String].transform(unsafe, _.value)
}

case class Metadata(publisher: String, edition: Int, pages: Int)

object Metadata {
  given Schema[Metadata]    = Schema.derived
  given JsonCodec[Metadata] = Schema[Metadata].derive(JsonFormat)
}

case class Book(id: Long, title: String, isbn: Option[Isbn], metadata: Option[Metadata])

object Book {
  given Schema[Book] = Schema.derived
}

case class ShelfBook(id: Long, title: String, isbn: Option[Isbn])

object ShelfBook {
  given Schema[ShelfBook] = Schema.derived
}
```

The derived codec rejects `Option[Metadata]` outright, because an `Option` field must map to one column and it treats the optional record as three. For `Option[Isbn]`, a known bug in 0.0.56 makes the codec read the column before it checks for `NULL`, so a `NULL` reaches `Isbn.unsafe` as `null` and the read throws. An `Option` of an enum fails the same way, with `Unknown enum variant 'null'`:

```scala
println(Try(DbCodec.derived[Book]).fold(_.getMessage, _ => "derived"))
// Option[A] where A has 3 columns is not supported. Option is only supported for single-column types.

transactor.connect {
  sql"""CREATE TABLE books (
    id       INTEGER PRIMARY KEY,
    title    TEXT NOT NULL,
    isbn     TEXT,
    metadata TEXT
  )""".update
  sql"INSERT INTO books (id, title, isbn) VALUES (1, 'Zionomicon', NULL)".update
  println(Try(sql"SELECT id, title, isbn FROM books".query[ShelfBook]))
}
// Failure(java.lang.NullPointerException: Cannot invoke "String.length()" because "$this" is null)
```

Both fields get a codec of their own through `DbCodec.derivedWith`. For the ISBN, `DbCodec#transform` builds a codec on top of `DbCodec[Option[String]]`, which handles `NULL` correctly, so `Isbn.unsafe` only ever sees a real string. For the metadata, `DbCodec.jsonbOption` stores `Some` as JSON and `None` as `NULL`, using the `JsonCodec` we gave `Metadata`:

```scala
def tableFor[A](name: String, codec: DbCodec[A]): Table[A] =
  Table(name, codec, codec.columns.map(column => ColumnMeta(column, DbValue.DbString(""), nullable = false)))

val optionalIsbn: DbCodec[Option[Isbn]] =
  DbCodec[Option[String]].transform(_.map(Isbn.unsafe))(_.map(_.value))

val bookCodec: DbCodec[Book] =
  DbCodec.derivedWith[Book](
    _.instance(TypeId.of[Book], "isbn", optionalIsbn)
      .instance(TypeId.of[Book], "metadata", DbCodec.jsonbOption[Metadata])
  )

val books = Repo(tableFor("books", bookCodec), "id", DbCodec[Long], (book: Book) => book.id)
```

Zionomicon has no ISBN in our records and no metadata yet, and both come back as `None`:

```scala
transactor.connect {
  sql"DELETE FROM books".update
  // Page counts are illustrative
  books.insert(Book(1L, "Zionomicon", None, None))
  books.insert(Book(2L, "Programming in Scala", Some(Isbn.unsafe("978-0-9815316-0-1")), Some(Metadata("Artima", 1, 776))))
  println(sql"SELECT title, isbn, metadata FROM books ORDER BY id".query[(String, Option[String], Option[String])])
  books.all.foreach(println)
}
// List((Zionomicon,None,None), (Programming in Scala,Some(9780981531601),Some({"publisher":"Artima","edition":1,"pages":776})))
// Book(1,Zionomicon,None,None)
// Book(2,Programming in Scala,Some(Isbn(9780981531601)),Some(Metadata(Artima,1,776)))
```

:::warning
`PgCodec.listStringCodec` reads a `NULL` array as a `null` list, not as an empty one. Declare array columns `NOT NULL`, or check for `null` after reading.
:::

## Putting It Together

The complete example puts every column type from this guide into one `books` table on SQLite: the opaque `BookId`, the optional `Isbn`, the `Format` enum, `Metadata` as JSON, the tags as a JSON array and the publication date as milliseconds. It inserts three books, checks that they read back equal to what went in, and runs one query per column type:

```scala title="sql-examples/src/main/scala/mapcustomcolumntypes/CompleteExample.scala"
package mapcustomcolumntypes

/**
 * How to Map Custom Column Types — Complete Example (SQLite)
 *
 * One `books` table with every custom column type from the guide: an opaque
 * `BookId`, a validated `Isbn` newtype, a `Format` enum, `Metadata` as JSON, a
 * `List[String]` of tags as a JSON array and an `Instant` stored as epoch
 * milliseconds. Inserts three books, reads them back and runs one query per
 * column type.
 *
 * Run with: cd sql-examples && sbt "runMain
 * mapcustomcolumntypes.CompleteExample"
 */
object CompleteExample {

  object ids {
    opaque type BookId = Long

    object BookId {
      def apply(value: Long): BookId = value
      def unwrap(id: BookId): Long   = id
    }
  }

  final case class Isbn private (value: String)

  object Isbn {
    def parse(input: String): Either[SchemaError, Isbn] = {
      val digits = input.filterNot(c => c == '-' || c == ' ')
      if (digits.length != 13 || !digits.forall(_.isDigit))
        Left(SchemaError.validationFailed(s"ISBN must have 13 digits, got '$input'"))
      else {
        val weighted = digits.init.zipWithIndex.map { case (c, i) => c.asDigit * (if (i % 2 == 0) 1 else 3) }
        if ((10 - weighted.sum % 10) % 10 == digits.last.asDigit) Right(new Isbn(digits))
        else Left(SchemaError.validationFailed(s"ISBN check digit is wrong in '$input'"))
      }
    }

    def unsafe(input: String): Isbn = parse(input).fold(e => throw e, identity)

    given Schema[Isbn] = Schema[String].transform(unsafe, _.value)
  }

  enum Format {
    case Hardcover, Paperback, Ebook
  }

  object Format {
    given Schema[Format] = Schema.derived
  }

  case class Metadata(publisher: String, edition: Int, pages: Int)

  object Metadata {
    given Schema[Metadata] = Schema.derived
  }

  case class Book(
    id: BookId,
    isbn: Option[Isbn],
    title: String,
    format: Format,
    metadata: Metadata,
    tags: List[String],
    publishedAt: Instant
  )

  object Book {
    given Schema[Book] = Schema.derived
  }

  // Columns taken from the codec, so the Repo writes what the codec writes.
  // ColumnMeta's type is only read by createTable, which we don't call.
  def tableFor[A](name: String, codec: DbCodec[A]): Table[A] =
    Table(name, codec, codec.columns.map(column => ColumnMeta(column, DbValue.DbString(""), nullable = false)))

  // Read the column as Option[String] first, so NULL never reaches Isbn.unsafe
  val optionalIsbn: DbCodec[Option[Isbn]] =
    DbCodec[Option[String]].transform(_.map(Isbn.unsafe))(_.map(_.value))

  val bookCodec: DbCodec[Book] = DbCodec.derivedWith[Book](_.instance(TypeId.of[Book], "isbn", optionalIsbn))
  val books                    = Repo(tableFor("books", bookCodec), "id", DbCodec[BookId], (book: Book) => book.id)

  // Page counts and dates are illustrative
  val shelf = List(
    Book(
      BookId(1L),
      None,
      "Zionomicon",
      Format.Ebook,
      Metadata("Ziverge", 1, 700),
      List("zio", "effects", "concurrency"),
      Instant.parse("2021-07-15T09:30:00Z")
    ),
    Book(
      BookId(2L),
      Some(Isbn.unsafe("978-0-9815316-0-1")),
      "Programming in Scala",
      Format.Paperback,
      Metadata("Artima", 1, 776),
      List("scala", "fundamentals"),
      Instant.parse("2008-11-26T00:00:00Z")
    ),
    Book(
      BookId(3L),
      Some(Isbn.unsafe("978-1-61729-065-7")),
      "Functional Programming in Scala",
      Format.Paperback,
      Metadata("Manning", 1, 320),
      List("scala", "functional-programming"),
      Instant.parse("2014-09-14T00:00:00Z")
    )
  )

  def main(args: Array[String]): Unit = {
    val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.transact {
      sql"""CREATE TABLE books (
        id           INTEGER PRIMARY KEY,
        isbn         TEXT,
        title        TEXT NOT NULL,
        format       TEXT NOT NULL,
        metadata     TEXT NOT NULL,
        tags         TEXT NOT NULL,
        published_at INTEGER NOT NULL
      )""".update
      books.insertBatch(shelf)
    }

    transactor.connect {
      println(sql"SELECT * FROM books WHERE id = 2".query[(Long, String, String, String, String, String, Long)])
      println(books.all == shelf)
      println(books.find(BookId(1L)))

      val format = Format.Paperback
      println(sql"SELECT title FROM books WHERE format = $format ORDER BY id".query[String])

      val publisher = "Ziverge"
      println(sql"SELECT title FROM books WHERE json_extract(metadata, '$$.publisher') = $publisher".query[String])

      val tag = "scala"
      println(
        sql"SELECT title FROM books WHERE EXISTS (SELECT 1 FROM json_each(books.tags) WHERE value = $tag) ORDER BY id"
          .query[String]
      )

      val cutoff = Instant.parse("2010-01-01T00:00:00Z")
      println(sql"SELECT title FROM books WHERE published_at >= $cutoff ORDER BY published_at".query[String])
    }
  }
}
```

On PostgreSQL, three things change: the `CREATE TABLE` statement uses `JSONB`, `TEXT[]` and `TIMESTAMPTZ`; the row codec also overrides `tags` with `PgCodec.listStringCodec`; and the JDBC URL has `stringtype=unspecified`. `CompletePostgresExample.scala`, listed in the next section, is the same program with those changes.

## Running the Examples

All examples in this guide have corresponding runnable Scala files in the `sql-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/sql-examples
```

To check that every example compiles without running any of them, use `sbt compile` instead of `sbt "runMain ..."`:

```bash
sbt compile
```

The SQLite examples each create their own temporary database file, so you can run them in any order. The PostgreSQL examples need the server from the [Prerequisites](#prerequisites) and the `BOOKSTORE_DB_PASSWORD` environment variable; set `BOOKSTORE_DB_URL` and `BOOKSTORE_DB_USER` too if they differ from `jdbc:postgresql://localhost:5432/bookstore` and `bookstore`. Each PostgreSQL example drops the tables it creates.

### Store a Newtype or an Opaque Type in One Column

This example shows the `isbn_value` column a derived newtype `Schema` produces, then stores a validated `Isbn` and an opaque `BookId` as plain columns and reads back a row with a bad check digit.

<details>
  <summary>sql-examples/src/main/scala/mapcustomcolumntypes/Step1NewtypeExample.scala</summary>

```scala title="sql-examples/src/main/scala/mapcustomcolumntypes/Step1NewtypeExample.scala" showLineNumbers
package mapcustomcolumntypes

/**
 * How to Map Custom Column Types — Step 1: Store a Newtype or an Opaque Type
 * in One Column
 *
 * Shows that a one-field case class with a derived Schema is a record (its
 * column becomes `isbn_value` and its value JSON), then stores a validated
 * `Isbn` newtype and an opaque `BookId` as plain columns, queries by ISBN and
 * reads back a row whose ISBN fails validation.
 *
 * Run with: cd sql-examples && sbt "runMain
 * mapcustomcolumntypes.Step1NewtypeExample"
 */
object Step1NewtypeExample {

  // A one-field case class with a derived Schema is a record, not a column
  final case class PlainIsbn(value: String)

  object PlainIsbn {
    given Schema[PlainIsbn] = Schema.derived
  }

  case class PlainBook(id: Long, isbn: PlainIsbn)

  object PlainBook {
    given Schema[PlainBook] = Schema.derived
  }

  // A validated newtype: its Schema is a String with a check on the way in
  final case class Isbn private (value: String)

  object Isbn {
    def parse(input: String): Either[SchemaError, Isbn] = {
      val digits = input.filterNot(c => c == '-' || c == ' ')
      if (digits.length != 13 || !digits.forall(_.isDigit))
        Left(SchemaError.validationFailed(s"ISBN must have 13 digits, got '$input'"))
      else {
        val weighted = digits.init.zipWithIndex.map { case (c, i) => c.asDigit * (if (i % 2 == 0) 1 else 3) }
        if ((10 - weighted.sum % 10) % 10 == digits.last.asDigit) Right(new Isbn(digits))
        else Left(SchemaError.validationFailed(s"ISBN check digit is wrong in '$input'"))
      }
    }

    def unsafe(input: String): Isbn = parse(input).fold(e => throw e, identity)

    given Schema[Isbn] = Schema[String].transform(unsafe, _.value)
  }

  // An opaque type: the macros see through it to Long
  object ids {
    opaque type BookId = Long

    object BookId {
      def apply(value: Long): BookId = value
      def unwrap(id: BookId): Long   = id
    }
  }

  case class Book(id: BookId, title: String, isbn: Isbn)

  object Book {
    given Schema[Book] = Schema.derived
  }

  val books = Repo.derived[Book, BookId]("books", "id", _.id)

  def main(args: Array[String]): Unit = {
    println(Table.derived[PlainBook].columns)
    println(DbCodec.derived[PlainBook].toDbValues(PlainBook(2L, PlainIsbn("9780981531601"))))

    val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.connect {
      sql"""CREATE TABLE books (
        id    INTEGER PRIMARY KEY,
        title TEXT NOT NULL,
        isbn  TEXT NOT NULL
      )""".update
      books.insert(Book(BookId(1L), "Programming in Scala", Isbn.unsafe("978-0-9815316-0-1")))
      books.insert(Book(BookId(2L), "Functional Programming in Scala", Isbn.unsafe("978-1-61729-065-7")))
      println(sql"SELECT id, title, isbn FROM books ORDER BY id".query[(Long, String, String)])
      books.all.foreach(println)

      // The sql interpolator has no parameter instance for Isbn: pass the String
      val isbn = Isbn.unsafe("9780981531601")
      println(sql"SELECT title FROM books WHERE isbn = ${isbn.value}".query[String])

      // Another program writes an ISBN with a wrong check digit
      sql"INSERT INTO books (id, title, isbn) VALUES (3, 'Scala with Cats', '9781617290658')".update
      println(Try(books.all))
      println(books.find(BookId(2L)))
    }
  }
}
```

</details>

```bash
sbt "runMain mapcustomcolumntypes.Step1NewtypeExample"
```

### Store an Enum as Its Case Name

This example stores `Format` by case name, renames the stored values of `LegacyFormat` with `@Modifier.rename`, and reads back a value that matches no case.

<details>
  <summary>sql-examples/src/main/scala/mapcustomcolumntypes/Step2EnumExample.scala</summary>

```scala title="sql-examples/src/main/scala/mapcustomcolumntypes/Step2EnumExample.scala" showLineNumbers
package mapcustomcolumntypes

/**
 * How to Map Custom Column Types — Step 2: Store an Enum as Its Case Name
 *
 * Stores a Scala 3 enum as the name of its case in a TEXT column, renames the
 * stored value of one case with `@Modifier.rename`, and reads back a row whose
 * value matches no case.
 *
 * Run with: cd sql-examples && sbt "runMain
 * mapcustomcolumntypes.Step2EnumExample"
 */
object Step2EnumExample {

  enum Format {
    case Hardcover, Paperback, Ebook
  }

  object Format {
    given Schema[Format] = Schema.derived
  }

  // The same cases, stored in lower case to match an existing table
  enum LegacyFormat {
    @Modifier.rename("hardcover") case Hardcover
    @Modifier.rename("paperback") case Paperback
    @Modifier.rename("ebook") case Ebook
  }

  object LegacyFormat {
    given Schema[LegacyFormat] = Schema.derived
  }

  case class Book(id: Long, title: String, format: Format)

  object Book {
    given Schema[Book] = Schema.derived
  }

  case class LegacyBook(id: Long, title: String, format: LegacyFormat)

  object LegacyBook {
    given Schema[LegacyBook] = Schema.derived
  }

  val books       = Repo.derived[Book, Long]("books", "id", _.id)
  val legacyBooks = Repo.derived[LegacyBook, Long]("legacy_books", "id", _.id)

  def main(args: Array[String]): Unit = {
    val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.connect {
      sql"CREATE TABLE books (id INTEGER PRIMARY KEY, title TEXT NOT NULL, format TEXT NOT NULL)".update
      books.insert(Book(1L, "Zionomicon", Format.Ebook))
      books.insert(Book(2L, "Programming in Scala", Format.Paperback))
      println(sql"SELECT title, format FROM books ORDER BY id".query[(String, String)])
      println(sql"SELECT title FROM books WHERE format = ${Format.Ebook}".query[String])

      sql"CREATE TABLE legacy_books (id INTEGER PRIMARY KEY, title TEXT NOT NULL, format TEXT NOT NULL)".update
      legacyBooks.insert(LegacyBook(1L, "Zionomicon", LegacyFormat.Ebook))
      println(sql"SELECT title, format FROM legacy_books".query[(String, String)])
      println(legacyBooks.all)

      // Another program adds a format the enum doesn't know
      sql"INSERT INTO books (id, title, format) VALUES (3, 'Functional Programming in Scala', 'Audiobook')".update
      println(Try(books.all))
    }
  }
}
```

</details>

```bash
sbt "runMain mapcustomcolumntypes.Step2EnumExample"
```

### Store a Nested Case Class as JSON on SQLite

This example shows `Repo.derived` failing on the `Metadata` field, then stores it as JSON through a `Repo` built from the codec's columns and filters with `json_extract`.

<details>
  <summary>sql-examples/src/main/scala/mapcustomcolumntypes/Step3JsonColumnExample.scala</summary>

```scala title="sql-examples/src/main/scala/mapcustomcolumntypes/Step3JsonColumnExample.scala" showLineNumbers
package mapcustomcolumntypes

/**
 * How to Map Custom Column Types — Step 3: Store a Nested Case Class as JSON
 * (SQLite)
 *
 * Shows that `Repo.derived` expects flattened `metadata_*` columns while the
 * row codec writes one JSON `metadata` column, builds the Repo from the codec's
 * own columns instead, round-trips the JSON column and filters on a field
 * inside it with SQLite's `json_extract`.
 *
 * Run with: cd sql-examples && sbt "runMain
 * mapcustomcolumntypes.Step3JsonColumnExample"
 */
object Step3JsonColumnExample {

  case class Metadata(publisher: String, edition: Int, pages: Int)

  object Metadata {
    given Schema[Metadata] = Schema.derived
  }

  case class Book(id: Long, title: String, metadata: Metadata)

  object Book {
    given Schema[Book] = Schema.derived
  }

  // Columns taken from the codec, so the Repo writes what the codec writes.
  // ColumnMeta's type is only read by createTable, which we don't call.
  def tableFor[A](name: String, codec: DbCodec[A]): Table[A] =
    Table(name, codec, codec.columns.map(column => ColumnMeta(column, DbValue.DbString(""), nullable = false)))

  val bookCodec: DbCodec[Book] = DbCodec.derived[Book]
  val books                    = Repo(tableFor("books", bookCodec), "id", DbCodec[Long], (book: Book) => book.id)

  // Page counts are illustrative
  val zionomicon         = Book(1L, "Zionomicon", Metadata("Ziverge", 1, 700))
  val programmingInScala = Book(2L, "Programming in Scala", Metadata("Artima", 1, 776))

  def main(args: Array[String]): Unit = {
    println(Table.derived[Book].columns)
    println(bookCodec.columns)

    val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.connect {
      sql"""CREATE TABLE books (
        id       INTEGER PRIMARY KEY,
        title    TEXT NOT NULL,
        metadata TEXT NOT NULL
      )""".update

      val derivedRepo = Repo.derived[Book, Long]("books", "id", _.id)
      println(Try(derivedRepo.insert(zionomicon)))

      books.insert(zionomicon)
      books.insert(programmingInScala)
      println(sql"SELECT title, metadata FROM books ORDER BY id".query[(String, String)])
      books.all.foreach(println)

      val publisher = "Artima"
      println(sql"SELECT id, title, metadata FROM books WHERE json_extract(metadata, '$$.publisher') = $publisher".query[Book])

      // A row whose JSON is missing a field fails when it is read
      sql"""INSERT INTO books (id, title, metadata) VALUES (3, 'Functional Programming in Scala', '{"publisher":"Manning"}')""".update
      println(Try(books.find(3L)))
    }
  }
}
```

</details>

```bash
sbt "runMain mapcustomcolumntypes.Step3JsonColumnExample"
```

### Use a jsonb Column on PostgreSQL

This example inserts into a `jsonb` column without and with `stringtype=unspecified` and filters with `->>`. It needs PostgreSQL.

<details>
  <summary>sql-examples/src/main/scala/mapcustomcolumntypes/Step4JsonbPostgresExample.scala</summary>

```scala title="sql-examples/src/main/scala/mapcustomcolumntypes/Step4JsonbPostgresExample.scala" showLineNumbers
package mapcustomcolumntypes

/**
 * How to Map Custom Column Types — Step 4: Store a Nested Case Class in a
 * PostgreSQL jsonb Column
 *
 * Shows the insert into a jsonb column failing on a default connection, then
 * succeeding once the JDBC URL has `stringtype=unspecified`, and filters on a
 * field inside the JSON with PostgreSQL's `->>` operator.
 *
 * Needs a running PostgreSQL; see the guide for the docker command. Set
 * BOOKSTORE_DB_PASSWORD (and BOOKSTORE_DB_URL / BOOKSTORE_DB_USER if they
 * differ from the defaults).
 *
 * Run with: cd sql-examples && sbt "runMain
 * mapcustomcolumntypes.Step4JsonbPostgresExample"
 */
object Step4JsonbPostgresExample {

  case class Metadata(publisher: String, edition: Int, pages: Int)

  object Metadata {
    given Schema[Metadata] = Schema.derived
  }

  case class Book(id: Long, title: String, metadata: Metadata)

  object Book {
    given Schema[Book] = Schema.derived
  }

  def tableFor[A](name: String, codec: DbCodec[A]): Table[A] =
    Table(name, codec, codec.columns.map(column => ColumnMeta(column, DbValue.DbString(""), nullable = false)))

  val bookCodec: DbCodec[Book] = DbCodec.derived[Book]
  val books                    = Repo(tableFor("books", bookCodec), "id", DbCodec[Long], (book: Book) => book.id)

  // Page counts are illustrative
  val zionomicon         = Book(1L, "Zionomicon", Metadata("Ziverge", 1, 700))
  val programmingInScala = Book(2L, "Programming in Scala", Metadata("Artima", 1, 776))

  val baseUrl  = sys.env.getOrElse("BOOKSTORE_DB_URL", "jdbc:postgresql://localhost:5432/bookstore")
  val user     = sys.env.getOrElse("BOOKSTORE_DB_USER", "bookstore")
  val password = sys.env("BOOKSTORE_DB_PASSWORD")

  // Let PostgreSQL infer the type of string parameters, so JSON text can go into jsonb
  val jsonbUrl = baseUrl + (if (baseUrl.contains("?")) "&" else "?") + "stringtype=unspecified"

  def main(args: Array[String]): Unit = {
    val plain = JdbcTransactor.fromUrl(baseUrl, user, password, SqlDialect.PostgreSQL)
    plain.connect {
      sql"DROP TABLE IF EXISTS books".update
      sql"""CREATE TABLE books (
        id       BIGINT PRIMARY KEY,
        title    TEXT NOT NULL,
        metadata JSONB NOT NULL
      )""".update
      println(Try(books.insert(zionomicon)).fold(_.getMessage.linesIterator.next(), n => s"Inserted $n"))
    }

    val transactor = JdbcTransactor.fromUrl(jsonbUrl, user, password, SqlDialect.PostgreSQL)
    transactor.connect {
      books.insert(zionomicon)
      books.insert(programmingInScala)
      println(sql"SELECT title, metadata::text FROM books ORDER BY id".query[(String, String)])
      books.all.foreach(println)

      val publisher = "Artima"
      println(sql"SELECT id, title, metadata FROM books WHERE metadata->>'publisher' = $publisher".query[Book])
      sql"DROP TABLE books".update
    }
  }
}
```

</details>

```bash
sbt "runMain mapcustomcolumntypes.Step4JsonbPostgresExample"
```

### Store a List as JSON on SQLite

This example shows `Table.derived` throwing on a `List[String]` field, then stores the tags as a JSON array and filters with `json_each`.

<details>
  <summary>sql-examples/src/main/scala/mapcustomcolumntypes/Step5ListColumnExample.scala</summary>

```scala title="sql-examples/src/main/scala/mapcustomcolumntypes/Step5ListColumnExample.scala" showLineNumbers
package mapcustomcolumntypes

/**
 * How to Map Custom Column Types — Step 5: Store a List as JSON (SQLite)
 *
 * Shows that `Table.derived` throws for a `List[String]` field, then stores
 * the list as a JSON array in a TEXT column through a Repo built from the
 * codec's columns, and filters on one tag with SQLite's `json_each`.
 *
 * Run with: cd sql-examples && sbt "runMain
 * mapcustomcolumntypes.Step5ListColumnExample"
 */
object Step5ListColumnExample {

  case class Book(id: Long, title: String, tags: List[String])

  object Book {
    given Schema[Book] = Schema.derived
  }

  def tableFor[A](name: String, codec: DbCodec[A]): Table[A] =
    Table(name, codec, codec.columns.map(column => ColumnMeta(column, DbValue.DbString(""), nullable = false)))

  val bookCodec: DbCodec[Book] = DbCodec.derived[Book]
  val books                    = Repo(tableFor("books", bookCodec), "id", DbCodec[Long], (book: Book) => book.id)

  def main(args: Array[String]): Unit = {
    println(Try(Table.derived[Book]))

    val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.connect {
      sql"CREATE TABLE books (id INTEGER PRIMARY KEY, title TEXT NOT NULL, tags TEXT NOT NULL)".update
      books.insert(Book(1L, "Zionomicon", List("zio", "effects", "concurrency")))
      books.insert(Book(2L, "Programming in Scala", List("scala", "fundamentals")))
      books.insert(Book(3L, "Functional Programming in Scala", List("scala", "functional-programming")))
      println(sql"SELECT title, tags FROM books ORDER BY id".query[(String, String)])

      val tag = "scala"
      println(
        sql"SELECT id, title, tags FROM books WHERE EXISTS (SELECT 1 FROM json_each(books.tags) WHERE value = $tag) ORDER BY id"
          .query[Book]
      )
    }
  }
}
```

</details>

```bash
sbt "runMain mapcustomcolumntypes.Step5ListColumnExample"
```

### Store a List as a PostgreSQL Array

This example overrides the `tags` codec with `PgCodec.listStringCodec`, filters with `ANY` and `&&`, and reads back an empty array and a `NULL` one. It needs PostgreSQL.

<details>
  <summary>sql-examples/src/main/scala/mapcustomcolumntypes/Step6ArrayPostgresExample.scala</summary>

```scala title="sql-examples/src/main/scala/mapcustomcolumntypes/Step6ArrayPostgresExample.scala" showLineNumbers
package mapcustomcolumntypes

/**
 * How to Map Custom Column Types — Step 6: Store a List as a PostgreSQL Array
 *
 * Shows the derived JSON codec failing against a `text[]` column, swaps in
 * `PgCodec.listStringCodec` for the `tags` field, filters with `ANY` and `&&`,
 * and reads back an empty array and a NULL array.
 *
 * Needs a running PostgreSQL; see the guide for the docker command. Set
 * BOOKSTORE_DB_PASSWORD (and BOOKSTORE_DB_URL / BOOKSTORE_DB_USER if they
 * differ from the defaults).
 *
 * Run with: cd sql-examples && sbt "runMain
 * mapcustomcolumntypes.Step6ArrayPostgresExample"
 */
object Step6ArrayPostgresExample {

  case class Book(id: Long, title: String, tags: List[String])

  object Book {
    given Schema[Book] = Schema.derived
  }

  def tableFor[A](name: String, codec: DbCodec[A]): Table[A] =
    Table(name, codec, codec.columns.map(column => ColumnMeta(column, DbValue.DbString(""), nullable = false)))

  // The derived codec writes tags as JSON text
  val jsonBookCodec: DbCodec[Book] = DbCodec.derived[Book]

  // The same codec with a PostgreSQL array for the tags field
  given bookCodec: DbCodec[Book] =
    DbCodec.derivedWith[Book](_.instance(TypeId.of[Book], "tags", PgCodec.listStringCodec))

  val books = Repo(tableFor("books", bookCodec), "id", DbCodec[Long], (book: Book) => book.id)

  val baseUrl  = sys.env.getOrElse("BOOKSTORE_DB_URL", "jdbc:postgresql://localhost:5432/bookstore")
  val user     = sys.env.getOrElse("BOOKSTORE_DB_USER", "bookstore")
  val password = sys.env("BOOKSTORE_DB_PASSWORD")

  def main(args: Array[String]): Unit = {
    val transactor = JdbcTransactor.fromUrl(baseUrl, user, password, SqlDialect.PostgreSQL)

    transactor.connect {
      sql"DROP TABLE IF EXISTS books".update
      sql"CREATE TABLE books (id BIGINT PRIMARY KEY, title TEXT NOT NULL, tags TEXT[])".update

      val jsonBooks = Repo(tableFor("books", jsonBookCodec), "id", DbCodec[Long], (book: Book) => book.id)
      println(Try(jsonBooks.insert(Book(1L, "Zionomicon", List("zio")))).fold(_.getMessage.linesIterator.next(), n => s"Inserted $n"))

      books.insert(Book(1L, "Zionomicon", List("zio", "effects", "concurrency")))
      books.insert(Book(2L, "Programming in Scala", List("scala", "fundamentals")))
      books.insert(Book(3L, "Functional Programming in Scala", List("scala", "functional-programming")))
      println(sql"SELECT title, tags::text FROM books ORDER BY id".query[(String, String)])

      val tag = "scala"
      println(sql"SELECT id, title, tags FROM books WHERE $tag = ANY(tags) ORDER BY id".query[Book])

      val wanted = List("effects", "functional-programming")
      println(
        Try(sql"SELECT title FROM books WHERE tags && $wanted ORDER BY id".query[String])
          .fold(_.getMessage.linesIterator.next(), _.toString)
      )
      println(sql"SELECT title FROM books WHERE tags && $wanted::text[] ORDER BY id".query[String])

      books.insert(Book(4L, "Scala with Cats", Nil))
      sql"INSERT INTO books (id, title, tags) VALUES (5, 'Hands-on Scala Programming', NULL)".update
      println(books.find(4L))
      println(books.find(5L))
      sql"DROP TABLE books".update
    }
  }
}
```

</details>

```bash
sbt "runMain mapcustomcolumntypes.Step6ArrayPostgresExample"
```

### Use an INTEGER Column for Timestamps on SQLite

This example stores an `Instant` in a `TEXT` column, which can't be read back, and then in an `INTEGER` column, which can.

<details>
  <summary>sql-examples/src/main/scala/mapcustomcolumntypes/Step7InstantSqliteExample.scala</summary>

```scala title="sql-examples/src/main/scala/mapcustomcolumntypes/Step7InstantSqliteExample.scala" showLineNumbers
package mapcustomcolumntypes

/**
 * How to Map Custom Column Types — Step 7: Store an Instant on SQLite
 *
 * Shows what SQLite stores for an `Instant` (epoch milliseconds), that the
 * TEXT column `createTable` declares can't be read back, and that an INTEGER
 * column round-trips the value to the millisecond.
 *
 * Run with: cd sql-examples && sbt "runMain
 * mapcustomcolumntypes.Step7InstantSqliteExample"
 */
object Step7InstantSqliteExample {

  case class Book(id: Long, title: String, publishedAt: Instant)

  object Book {
    given Schema[Book] = Schema.derived
  }

  val books = Repo.derived[Book, Long]("books", "id", _.id)

  // Illustrative date, with microseconds to show the precision SQLite keeps
  val zionomicon = Book(1L, "Zionomicon", Instant.parse("2021-07-15T09:30:00.123456Z"))

  def main(args: Array[String]): Unit = {
    println(books.table.createTable(SqlDialect.SQLite).sql(SqlDialect.SQLite))

    val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.connect {
      sql"CREATE TABLE books (id INTEGER PRIMARY KEY, title TEXT NOT NULL, published_at TEXT NOT NULL)".update
      books.insert(zionomicon)
      println(sql"SELECT published_at, typeof(published_at) FROM books".query[(String, String)])
      println(Try(books.all))
      sql"DROP TABLE books".update

      sql"CREATE TABLE books (id INTEGER PRIMARY KEY, title TEXT NOT NULL, published_at INTEGER NOT NULL)".update
      books.insert(zionomicon)
      println(sql"SELECT published_at, typeof(published_at) FROM books".query[(Long, String)])
      println(books.all)

      val cutoff = Instant.parse("2020-01-01T00:00:00Z")
      println(sql"SELECT title FROM books WHERE published_at >= $cutoff".query[String])
    }
  }
}
```

</details>

```bash
sbt "runMain mapcustomcolumntypes.Step7InstantSqliteExample"
```

### Use a timestamptz Column on PostgreSQL

This example writes the same moment to a `timestamptz` and a `timestamp` column, from our `Repo` and from a session in another time zone. It needs PostgreSQL.

<details>
  <summary>sql-examples/src/main/scala/mapcustomcolumntypes/Step8InstantPostgresExample.scala</summary>

```scala title="sql-examples/src/main/scala/mapcustomcolumntypes/Step8InstantPostgresExample.scala" showLineNumbers
package mapcustomcolumntypes

/**
 * How to Map Custom Column Types — Step 8: Store an Instant on PostgreSQL
 *
 * Stores the same `Instant` in a `timestamptz` and a `timestamp` column, then
 * lets plain SQL write the same moment into both columns from a session in
 * another time zone, and reads all four values back.
 *
 * Needs a running PostgreSQL; see the guide for the docker command. Set
 * BOOKSTORE_DB_PASSWORD (and BOOKSTORE_DB_URL / BOOKSTORE_DB_USER if they
 * differ from the defaults).
 *
 * Run with: cd sql-examples && sbt "runMain
 * mapcustomcolumntypes.Step8InstantPostgresExample"
 */
object Step8InstantPostgresExample {

  case class Release(id: Long, withZone: Instant, withoutZone: Instant)

  object Release {
    given Schema[Release] = Schema.derived
  }

  val releases = Repo.derived[Release, Long]("releases", "id", _.id)

  val baseUrl  = sys.env.getOrElse("BOOKSTORE_DB_URL", "jdbc:postgresql://localhost:5432/bookstore")
  val user     = sys.env.getOrElse("BOOKSTORE_DB_USER", "bookstore")
  val password = sys.env("BOOKSTORE_DB_PASSWORD")

  // Illustrative date
  val publishedAt = Instant.parse("2021-07-15T09:30:00.123456Z")

  def main(args: Array[String]): Unit = {
    val transactor = JdbcTransactor.fromUrl(baseUrl, user, password, SqlDialect.PostgreSQL)

    transactor.connect {
      sql"DROP TABLE IF EXISTS releases".update
      sql"""CREATE TABLE releases (
        id           BIGINT PRIMARY KEY,
        with_zone    TIMESTAMPTZ NOT NULL,
        without_zone TIMESTAMP NOT NULL
      )""".update

      // Written by our Repo
      releases.insert(Release(1L, publishedAt, publishedAt))

      // Written by another client whose session runs in Tehran (UTC+03:30)
      sql"SET TIME ZONE 'Asia/Tehran'".update
      sql"""INSERT INTO releases VALUES (2,
        TIMESTAMPTZ '2021-07-15 09:30:00.123456+00',
        TIMESTAMPTZ '2021-07-15 09:30:00.123456+00')""".update
      sql"SET TIME ZONE 'UTC'".update

      println(sql"SELECT id, with_zone::text, without_zone::text FROM releases ORDER BY id".query[(Long, String, String)])
      releases.all.foreach(println)
      sql"DROP TABLE releases".update
    }
  }
}
```

</details>

```bash
sbt "runMain mapcustomcolumntypes.Step8InstantPostgresExample"
```

### Make a Custom Column Nullable

This example reads `NULL` into non-optional fields, shows the two `Option` fields that fail, and fixes both with `DbCodec.derivedWith`.

<details>
  <summary>sql-examples/src/main/scala/mapcustomcolumntypes/Step9NullableExample.scala</summary>

```scala title="sql-examples/src/main/scala/mapcustomcolumntypes/Step9NullableExample.scala" showLineNumbers
package mapcustomcolumntypes

/**
 * How to Map Custom Column Types — Step 9: Make a Custom Column Nullable
 *
 * Shows what a non-optional field reads from SQL NULL, that the derived codec
 * fails on NULL for an `Option` of a newtype and rejects an `Option` of a
 * nested case class, then fixes both fields with `DbCodec.derivedWith`.
 *
 * Run with: cd sql-examples && sbt "runMain
 * mapcustomcolumntypes.Step9NullableExample"
 */
object Step9NullableExample {

  final case class Isbn private (value: String)

  object Isbn {
    def parse(input: String): Either[SchemaError, Isbn] = {
      val digits = input.filterNot(c => c == '-' || c == ' ')
      if (digits.length != 13 || !digits.forall(_.isDigit))
        Left(SchemaError.validationFailed(s"ISBN must have 13 digits, got '$input'"))
      else {
        val weighted = digits.init.zipWithIndex.map { case (c, i) => c.asDigit * (if (i % 2 == 0) 1 else 3) }
        if ((10 - weighted.sum % 10) % 10 == digits.last.asDigit) Right(new Isbn(digits))
        else Left(SchemaError.validationFailed(s"ISBN check digit is wrong in '$input'"))
      }
    }

    def unsafe(input: String): Isbn = parse(input).fold(e => throw e, identity)

    given Schema[Isbn] = Schema[String].transform(unsafe, _.value)
  }

  case class Metadata(publisher: String, edition: Int, pages: Int)

  object Metadata {
    given Schema[Metadata]    = Schema.derived
    given JsonCodec[Metadata] = Schema[Metadata].derive(JsonFormat)
  }

  case class Book(id: Long, title: String, isbn: Option[Isbn], metadata: Option[Metadata])

  object Book {
    given Schema[Book] = Schema.derived
  }

  case class ShelfBook(id: Long, title: String, isbn: Option[Isbn])

  object ShelfBook {
    given Schema[ShelfBook] = Schema.derived
  }

  def tableFor[A](name: String, codec: DbCodec[A]): Table[A] =
    Table(name, codec, codec.columns.map(column => ColumnMeta(column, DbValue.DbString(""), nullable = false)))

  // Read the column as Option[String] first, so NULL never reaches Isbn.unsafe
  val optionalIsbn: DbCodec[Option[Isbn]] =
    DbCodec[Option[String]].transform(_.map(Isbn.unsafe))(_.map(_.value))

  val bookCodec: DbCodec[Book] =
    DbCodec.derivedWith[Book](
      _.instance(TypeId.of[Book], "isbn", optionalIsbn)
        .instance(TypeId.of[Book], "metadata", DbCodec.jsonbOption[Metadata])
    )

  val books = Repo(tableFor("books", bookCodec), "id", DbCodec[Long], (book: Book) => book.id)

  def main(args: Array[String]): Unit = {
    val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.connect {
      // Non-optional fields don't fail on NULL
      println(sql"SELECT CAST(NULL AS TEXT), CAST(NULL AS INTEGER)".query[(String, Int)])

      println(Try(DbCodec.derived[Book]).fold(_.getMessage, _ => "derived"))

      sql"""CREATE TABLE books (
        id       INTEGER PRIMARY KEY,
        title    TEXT NOT NULL,
        isbn     TEXT,
        metadata TEXT
      )""".update
      sql"INSERT INTO books (id, title, isbn) VALUES (1, 'Zionomicon', NULL)".update
      println(Try(sql"SELECT id, title, isbn FROM books".query[ShelfBook]))

      sql"DELETE FROM books".update
      // Page counts are illustrative
      books.insert(Book(1L, "Zionomicon", None, None))
      books.insert(Book(2L, "Programming in Scala", Some(Isbn.unsafe("978-0-9815316-0-1")), Some(Metadata("Artima", 1, 776))))
      println(sql"SELECT title, isbn, metadata FROM books ORDER BY id".query[(String, Option[String], Option[String])])
      books.all.foreach(println)
    }
  }
}
```

</details>

```bash
sbt "runMain mapcustomcolumntypes.Step9NullableExample"
```

### The Complete Example on SQLite

This example puts every column type from this guide into one `books` table on SQLite, inserts three books, checks that they read back equal and runs one query per column type.

<details>
  <summary>sql-examples/src/main/scala/mapcustomcolumntypes/CompleteExample.scala</summary>

```scala title="sql-examples/src/main/scala/mapcustomcolumntypes/CompleteExample.scala" showLineNumbers
package mapcustomcolumntypes

/**
 * How to Map Custom Column Types — Complete Example (SQLite)
 *
 * One `books` table with every custom column type from the guide: an opaque
 * `BookId`, a validated `Isbn` newtype, a `Format` enum, `Metadata` as JSON, a
 * `List[String]` of tags as a JSON array and an `Instant` stored as epoch
 * milliseconds. Inserts three books, reads them back and runs one query per
 * column type.
 *
 * Run with: cd sql-examples && sbt "runMain
 * mapcustomcolumntypes.CompleteExample"
 */
object CompleteExample {

  object ids {
    opaque type BookId = Long

    object BookId {
      def apply(value: Long): BookId = value
      def unwrap(id: BookId): Long   = id
    }
  }

  final case class Isbn private (value: String)

  object Isbn {
    def parse(input: String): Either[SchemaError, Isbn] = {
      val digits = input.filterNot(c => c == '-' || c == ' ')
      if (digits.length != 13 || !digits.forall(_.isDigit))
        Left(SchemaError.validationFailed(s"ISBN must have 13 digits, got '$input'"))
      else {
        val weighted = digits.init.zipWithIndex.map { case (c, i) => c.asDigit * (if (i % 2 == 0) 1 else 3) }
        if ((10 - weighted.sum % 10) % 10 == digits.last.asDigit) Right(new Isbn(digits))
        else Left(SchemaError.validationFailed(s"ISBN check digit is wrong in '$input'"))
      }
    }

    def unsafe(input: String): Isbn = parse(input).fold(e => throw e, identity)

    given Schema[Isbn] = Schema[String].transform(unsafe, _.value)
  }

  enum Format {
    case Hardcover, Paperback, Ebook
  }

  object Format {
    given Schema[Format] = Schema.derived
  }

  case class Metadata(publisher: String, edition: Int, pages: Int)

  object Metadata {
    given Schema[Metadata] = Schema.derived
  }

  case class Book(
    id: BookId,
    isbn: Option[Isbn],
    title: String,
    format: Format,
    metadata: Metadata,
    tags: List[String],
    publishedAt: Instant
  )

  object Book {
    given Schema[Book] = Schema.derived
  }

  // Columns taken from the codec, so the Repo writes what the codec writes.
  // ColumnMeta's type is only read by createTable, which we don't call.
  def tableFor[A](name: String, codec: DbCodec[A]): Table[A] =
    Table(name, codec, codec.columns.map(column => ColumnMeta(column, DbValue.DbString(""), nullable = false)))

  // Read the column as Option[String] first, so NULL never reaches Isbn.unsafe
  val optionalIsbn: DbCodec[Option[Isbn]] =
    DbCodec[Option[String]].transform(_.map(Isbn.unsafe))(_.map(_.value))

  val bookCodec: DbCodec[Book] = DbCodec.derivedWith[Book](_.instance(TypeId.of[Book], "isbn", optionalIsbn))
  val books                    = Repo(tableFor("books", bookCodec), "id", DbCodec[BookId], (book: Book) => book.id)

  // Page counts and dates are illustrative
  val shelf = List(
    Book(
      BookId(1L),
      None,
      "Zionomicon",
      Format.Ebook,
      Metadata("Ziverge", 1, 700),
      List("zio", "effects", "concurrency"),
      Instant.parse("2021-07-15T09:30:00Z")
    ),
    Book(
      BookId(2L),
      Some(Isbn.unsafe("978-0-9815316-0-1")),
      "Programming in Scala",
      Format.Paperback,
      Metadata("Artima", 1, 776),
      List("scala", "fundamentals"),
      Instant.parse("2008-11-26T00:00:00Z")
    ),
    Book(
      BookId(3L),
      Some(Isbn.unsafe("978-1-61729-065-7")),
      "Functional Programming in Scala",
      Format.Paperback,
      Metadata("Manning", 1, 320),
      List("scala", "functional-programming"),
      Instant.parse("2014-09-14T00:00:00Z")
    )
  )

  def main(args: Array[String]): Unit = {
    val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.transact {
      sql"""CREATE TABLE books (
        id           INTEGER PRIMARY KEY,
        isbn         TEXT,
        title        TEXT NOT NULL,
        format       TEXT NOT NULL,
        metadata     TEXT NOT NULL,
        tags         TEXT NOT NULL,
        published_at INTEGER NOT NULL
      )""".update
      books.insertBatch(shelf)
    }

    transactor.connect {
      println(sql"SELECT * FROM books WHERE id = 2".query[(Long, String, String, String, String, String, Long)])
      println(books.all == shelf)
      println(books.find(BookId(1L)))

      val format = Format.Paperback
      println(sql"SELECT title FROM books WHERE format = $format ORDER BY id".query[String])

      val publisher = "Ziverge"
      println(sql"SELECT title FROM books WHERE json_extract(metadata, '$$.publisher') = $publisher".query[String])

      val tag = "scala"
      println(
        sql"SELECT title FROM books WHERE EXISTS (SELECT 1 FROM json_each(books.tags) WHERE value = $tag) ORDER BY id"
          .query[String]
      )

      val cutoff = Instant.parse("2010-01-01T00:00:00Z")
      println(sql"SELECT title FROM books WHERE published_at >= $cutoff ORDER BY published_at".query[String])
    }
  }
}
```

</details>

```bash
sbt "runMain mapcustomcolumntypes.CompleteExample"
```

### The Complete Example on PostgreSQL

This example is the complete example with a `jsonb`, a `text[]` and a `timestamptz` column. It needs PostgreSQL.

<details>
  <summary>sql-examples/src/main/scala/mapcustomcolumntypes/CompletePostgresExample.scala</summary>

```scala title="sql-examples/src/main/scala/mapcustomcolumntypes/CompletePostgresExample.scala" showLineNumbers
package mapcustomcolumntypes

/**
 * How to Map Custom Column Types — Complete Example (PostgreSQL)
 *
 * The same `books` table as CompleteExample on PostgreSQL: `jsonb` for the
 * metadata, `text[]` for the tags and `timestamptz` for the publication date.
 * Inserts three books, reads them back and runs one query per column type.
 *
 * Needs a running PostgreSQL; see the guide for the docker command. Set
 * BOOKSTORE_DB_PASSWORD (and BOOKSTORE_DB_URL / BOOKSTORE_DB_USER if they
 * differ from the defaults).
 *
 * Run with: cd sql-examples && sbt "runMain
 * mapcustomcolumntypes.CompletePostgresExample"
 */
object CompletePostgresExample {

  object ids {
    opaque type BookId = Long

    object BookId {
      def apply(value: Long): BookId = value
      def unwrap(id: BookId): Long   = id
    }
  }

  final case class Isbn private (value: String)

  object Isbn {
    def parse(input: String): Either[SchemaError, Isbn] = {
      val digits = input.filterNot(c => c == '-' || c == ' ')
      if (digits.length != 13 || !digits.forall(_.isDigit))
        Left(SchemaError.validationFailed(s"ISBN must have 13 digits, got '$input'"))
      else {
        val weighted = digits.init.zipWithIndex.map { case (c, i) => c.asDigit * (if (i % 2 == 0) 1 else 3) }
        if ((10 - weighted.sum % 10) % 10 == digits.last.asDigit) Right(new Isbn(digits))
        else Left(SchemaError.validationFailed(s"ISBN check digit is wrong in '$input'"))
      }
    }

    def unsafe(input: String): Isbn = parse(input).fold(e => throw e, identity)

    given Schema[Isbn] = Schema[String].transform(unsafe, _.value)
  }

  enum Format {
    case Hardcover, Paperback, Ebook
  }

  object Format {
    given Schema[Format] = Schema.derived
  }

  case class Metadata(publisher: String, edition: Int, pages: Int)

  object Metadata {
    given Schema[Metadata] = Schema.derived
  }

  case class Book(
    id: BookId,
    isbn: Option[Isbn],
    title: String,
    format: Format,
    metadata: Metadata,
    tags: List[String],
    publishedAt: Instant
  )

  object Book {
    given Schema[Book] = Schema.derived
  }

  def tableFor[A](name: String, codec: DbCodec[A]): Table[A] =
    Table(name, codec, codec.columns.map(column => ColumnMeta(column, DbValue.DbString(""), nullable = false)))

  // Read the column as Option[String] first, so NULL never reaches Isbn.unsafe
  val optionalIsbn: DbCodec[Option[Isbn]] =
    DbCodec[Option[String]].transform(_.map(Isbn.unsafe))(_.map(_.value))

  // tags as a PostgreSQL array, isbn as above; everything else as derived
  given bookCodec: DbCodec[Book] =
    DbCodec.derivedWith[Book](
      _.instance(TypeId.of[Book], "isbn", optionalIsbn)
        .instance(TypeId.of[Book], "tags", PgCodec.listStringCodec)
    )

  val books = Repo(tableFor("books", bookCodec), "id", DbCodec[BookId], (book: Book) => book.id)

  // Page counts and dates are illustrative
  val shelf = List(
    Book(
      BookId(1L),
      None,
      "Zionomicon",
      Format.Ebook,
      Metadata("Ziverge", 1, 700),
      List("zio", "effects", "concurrency"),
      Instant.parse("2021-07-15T09:30:00Z")
    ),
    Book(
      BookId(2L),
      Some(Isbn.unsafe("978-0-9815316-0-1")),
      "Programming in Scala",
      Format.Paperback,
      Metadata("Artima", 1, 776),
      List("scala", "fundamentals"),
      Instant.parse("2008-11-26T00:00:00Z")
    ),
    Book(
      BookId(3L),
      Some(Isbn.unsafe("978-1-61729-065-7")),
      "Functional Programming in Scala",
      Format.Paperback,
      Metadata("Manning", 1, 320),
      List("scala", "functional-programming"),
      Instant.parse("2014-09-14T00:00:00Z")
    )
  )

  val baseUrl  = sys.env.getOrElse("BOOKSTORE_DB_URL", "jdbc:postgresql://localhost:5432/bookstore")
  val user     = sys.env.getOrElse("BOOKSTORE_DB_USER", "bookstore")
  val password = sys.env("BOOKSTORE_DB_PASSWORD")

  // Let PostgreSQL infer the type of string parameters, so JSON text can go into jsonb
  val jsonbUrl = baseUrl + (if (baseUrl.contains("?")) "&" else "?") + "stringtype=unspecified"

  def main(args: Array[String]): Unit = {
    val transactor = JdbcTransactor.fromUrl(jsonbUrl, user, password, SqlDialect.PostgreSQL)

    transactor.transact {
      sql"DROP TABLE IF EXISTS books".update
      sql"""CREATE TABLE books (
        id           BIGINT PRIMARY KEY,
        isbn         TEXT,
        title        TEXT NOT NULL,
        format       TEXT NOT NULL,
        metadata     JSONB NOT NULL,
        tags         TEXT[] NOT NULL,
        published_at TIMESTAMPTZ NOT NULL
      )""".update
      books.insertBatch(shelf)
    }

    transactor.connect {
      println(
        sql"SELECT id, isbn, title, format, metadata::text, tags::text, published_at FROM books WHERE id = 2"
          .query[(Long, String, String, String, String, String, Instant)]
      )
      println(books.all == shelf)
      println(books.find(BookId(1L)))

      val format = Format.Paperback
      println(sql"SELECT title FROM books WHERE format = $format ORDER BY id".query[String])

      val publisher = "Ziverge"
      println(sql"SELECT title FROM books WHERE metadata->>'publisher' = $publisher".query[String])

      val tag = "scala"
      println(sql"SELECT title FROM books WHERE $tag = ANY(tags) ORDER BY id".query[String])

      val cutoff = Instant.parse("2010-01-01T00:00:00Z")
      println(sql"SELECT title FROM books WHERE published_at >= $cutoff ORDER BY published_at".query[String])
      sql"DROP TABLE books".update
    }
  }
}
```

</details>

```bash
sbt "runMain mapcustomcolumntypes.CompletePostgresExample"
```

## Going Further

- [DbCodec](../reference/sql/db-codec.md) lists every built-in codec and the `DbCodec#transform`, `DbCodec.jsonb` and `DbCodec.derivedWith` constructors in full.
- [DbCodecDeriver](../reference/sql/db-codec-deriver.md) describes how the row codec maps each kind of field, including the `sql.inline` and `sql.inline_fields` settings.
- [SqlDialect](../reference/sql/sql-dialect.md) shows the column type `Table#createTable` picks for each value on SQLite and PostgreSQL.
- [How to Test Database Code Against SQLite](./test-database-code-against-sqlite.md) runs code like this in tests, with a fresh database per test.
- [How to Define Validated Newtypes](./validated-newtypes.md) collects every validation error of a record at once, instead of stopping at the first.
- [How to Store and Search Vector Embeddings in PostgreSQL](./store-and-search-vector-embeddings.md) uses the same `DbCodec#transform` pattern for a pgvector `vector` column, and needs a cast for the same reason as the `jsonb` column above.

---

## Coming from Magnum or doobie

This guide moves a data-access layer written with [Magnum](https://github.com/AugustNagro/magnum) or [doobie](https://typelevel.org/doobie/) to the ZIO Blocks `sql` module. By the end, you will have a bookshelf repository whose entities, queries, transactions and streams run on ZIO Blocks, and a checklist for porting the rest of your code.

Magnum users will find most names unchanged: `DbCodec`, `Repo`, `DbCon`, `DbTx`, `Frag` and `SqlLiteral` all exist in ZIO Blocks with the same roles, so the [Magnum section](#coming-from-magnum) concentrates on the few real differences. doobie users change their effect model as well as their names, so the [doobie section](#coming-from-doobie) is longer and starts with that shift.

The guide covers:

- A concept map from Magnum and doobie names to ZIO Blocks names
- Coming from Magnum: entities, repositories, transactors, the `sql"..."` interpolator, `Spec` queries and custom column types
- Coming from doobie: the effect model, transactors, `Read`/`Write`/`Meta`, fragments, query results, streaming and query checking
- Running the same code from ZIO or cats-effect
- What you gain, what's different, and a migration checklist

## The Problem

A data-access layer is tied to its library in many small places: how a case class becomes a row, how a connection is opened, how a query returns zero, one or many rows, and how errors surface. Porting it means finding the new name for every one of those places. When the two libraries differ in their effect model, as doobie and ZIO Blocks do, the shape of every data-access method changes too.

For example, a doobie repository describes every query as a `ConnectionIO` program and only runs it when an `IO` is executed:

```scala
// Before: doobie 1.0.0-RC13 (not compiled here)

case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String])

def booksBefore(year: Int): ConnectionIO[List[Book]] =
  sql"SELECT id, title, author, year, isbn FROM book WHERE year < $year".query[Book].to[List]

val xa = Transactor.fromDriverManager[IO](
  driver = "org.sqlite.JDBC",
  url = "jdbc:sqlite:bookshelf.db",
  logHandler = None
)

val classics: IO[List[Book]] = booksBefore(2010).transact(xa)
```

Every method returns a `ConnectionIO`, every caller composes them in a for-comprehension, and the result only exists inside `IO`. Porting that code is mostly mechanical once you know the target shape, and this guide gives you that shape for each construct.

## Prerequisites

Replace the Magnum or doobie dependencies with the ZIO Blocks `sql` module and your JDBC driver. The `sql` module is published for Scala 3 only:

```scala
// Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-sql" % "0.0.56"

// Scala 3.9 and newer, which also needs -experimental
libraryDependencies += "dev.zio" %% "zio-blocks-next-sql" % "0.0.56"
scalacOptions += "-experimental"

// Your JDBC driver, for example SQLite
libraryDependencies += "org.xerial" % "sqlite-jdbc" % "3.53.4.0"
```

The module is cross-built for the JVM and Scala.js, but `JdbcTransactor`, the part that talks to a database, runs on the JVM only. The built-in dialects are `SqlDialect.PostgreSQL` and `SqlDialect.SQLite`.

This guide assumes you know the basics from [Build Your First Database Repository on SQLite](./your-first-sqlite-repository.md): a `Schema` for each entity, a `JdbcTransactor`, `Repo` and `sql"..."`. This guide does not re-teach them; it maps what you already write onto them. The "before" snippets follow Magnum 2.0.0-M4 and doobie 1.0.0-RC13. They are plain Scala blocks, because neither library is on this site's build. Every ZIO Blocks snippet is compiled, and most of them run against a SQLite file.

## Concept Map

The table below lists the constructs you use every day, with the name each library gives them. The sections that follow show each row in code:

| Concept                 | Magnum                                       | doobie                                             | ZIO Blocks                                                |
| ----------------------- | -------------------------------------------- | -------------------------------------------------- | --------------------------------------------------------- |
| Row mapping             | `derives DbCodec`                            | `Read` / `Write`, derived automatically            | `Schema.derived`; `DbCodec` is derived from it            |
| Custom column type      | `DbCodec[A].biMap`                           | `Meta[A].timap`, `Get` / `Put`                     | `DbCodec[A].transform`                                    |
| Table metadata          | `@Table(dbType, nameMapper)`, `TableInfo`    | none, SQL is written by hand                       | `Table.derived[A]`                                        |
| Column naming           | `SqlNameMapper.CamelToSnakeCase`, `@SqlName` | none, SQL is written by hand                       | snake_case by default, `@Modifier.rename`                 |
| Repository              | `Repo[EC, E, ID]`, `ImmutableRepo[E, ID]`    | none                                               | `Repo.derived[E, ID]`                                     |
| Dynamic criteria        | `Spec[E]`                                    | `Fragments.whereAndOpt`                            | `SqlQuery.from(table)`, or `Frag` composition             |
| SQL fragment            | `Frag`, `sql"..."`                           | `Fragment`, `sql"..."`, `fr"..."`, `fr0"..."`      | `Frag`, `sql"..."`                                        |
| Raw SQL splice          | `SqlLiteral`                                 | `Fragment.const`                                   | `SqlLiteral`                                              |
| Database program        | a block with `using DbCon` / `DbTx`          | `ConnectionIO[A]`                                  | a block with `using DbCon` / `DbTx`                       |
| Transactor              | `Transactor(dataSource)`                     | `Transactor.fromDriverManager`, `HikariTransactor` | `JdbcTransactor.fromUrl`, `JdbcTransactor.fromDataSource` |
| Run on a connection     | `connect(xa) { ... }`                        | none, `transact` always opens a transaction        | `transactor.connect { ... }`                              |
| Run in a transaction    | `transact(xa) { ... }`                       | `program.transact(xa)`                             | `transactor.transact { ... }`                             |
| All rows                | `.query[A].run(): Vector[A]`                 | `.query[A].to[List]`                               | `Frag#query[A]: List[A]`                                  |
| At most one row         | `repo.findById(id): Option[E]`               | `.query[A].option`                                 | `Frag#queryOne[A]: Maybe[A]`, `Repo#find`                 |
| Streaming               | `.query[A].iterator()`                       | `.query[A].stream` (fs2)                           | `Frag#queryStream`, `Frag#queryChunked`                   |
| Insert, update, delete  | `.update.run(): Int`                         | `.update.run`                                      | `Frag#update: Int`                                        |
| Generated keys          | `.returning[K]`, `.returningKeys`            | `.update.withUniqueGeneratedKeys[K]`               | `Frag#updateReturningKeys[K]: List[K]`                    |
| Query checking          | none                                         | `.check` against a live database                   | `StringContext(...).sql(table)(...)` at compile time      |
| ZIO integration         | `magnumzio`: `TransactorZIO`                 | none                                               | `zio-blocks-sql-zio`: `TransactorZIO`                     |
| cats-effect integration | `magnum-cats-effect`: `TransactorCats`       | built in                                           | none; wrap calls in `IO.blocking`                         |

## The Shared Bookshelf

Both migration paths end at the same ZIO Blocks code, so the setup comes first. It reuses the bookshelf from [the SQLite tutorial](./your-first-sqlite-repository.md): a `Book` with a derived `Schema`, a transactor for a temporary SQLite file, the table and the repository:

```scala

case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String])

object Book {
  given Schema[Book] = Schema.derived
}

val dbFile     = java.nio.file.Files.createTempFile("bookshelf", ".db")
val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

val bookTable = Table.derived[Book]
val books     = Repo.derived[Book, Long]
```

The `author` field holds each book's first author. Each run of this page creates a new temporary file, so the database always starts empty.

:::caution
Every `JdbcTransactor#connect` and `JdbcTransactor#transact` call opens a new connection. With `jdbc:sqlite::memory:`, each new connection gets its own empty database, so use a file, or a `DataSource` that hands out the same connection, when you move SQLite tests over. [How to Test Database Code Against SQLite](./test-database-code-against-sqlite.md) shows both.
:::

## Coming from Magnum

Magnum and the ZIO Blocks `sql` module share their core design: entities are plain case classes, database work runs in a context function that receives a `DbCon` or `DbTx`, and queries are written with a `sql"..."` interpolator that returns a `Frag`. Most of a Magnum code base therefore ports by changing imports and a handful of method names. The sections below cover the places where the two libraries really differ.

### Entities and Tables

A Magnum entity carries its database type and naming policy in a `@Table` annotation, marks its ID with `@Id` and derives a `DbCodec`:

```scala
// Before: Magnum 2.0.0-M4 (not compiled here)

@Table(SqliteDbType, SqlNameMapper.CamelToSnakeCase)
case class Book(
  @Id id: Long,
  title: String,
  author: String,
  year: Int,
  isbn: Option[String]
) derives DbCodec
```

In ZIO Blocks, the entity only derives a `Schema`, as in the shared bookshelf above. Three things move elsewhere. The database type becomes a `SqlDialect` on the transactor, so the same entity works with PostgreSQL and SQLite. Column names are snake_case by default, which matches `SqlNameMapper.CamelToSnakeCase`. And the ID needs no annotation when the field is named `id`, is the only field of the ID type, or is named `<entity>Id`.

ZIO Blocks also generates the `CREATE TABLE` statement from the same schema, which Magnum leaves to you. The next block prints the statement and runs it:

```scala
println(bookTable.createTable(SqlDialect.SQLite).sql(SqlDialect.SQLite))
// CREATE TABLE IF NOT EXISTS book (
//   id INTEGER NOT NULL,
//   title TEXT NOT NULL,
//   author TEXT NOT NULL,
//   year INTEGER NOT NULL,
//   isbn TEXT
// )

transactor.connect {
  bookTable.createTable(SqlDialect.SQLite).update
  println(s"Rows in ${bookTable.name}: ${books.count}")
}
// Rows in book: 0
```

The generated statement declares no `PRIMARY KEY`, not even on the ID column. If you rely on a primary key, for example for `Repo#insertOrUpdate`, keep writing that table's DDL yourself.

When the defaults don't fit, annotations from `zio.blocks.schema.Modifier` take over the job of Magnum's `@SqlName`, `@Table` name mapper and `@Id`:

```scala
@Modifier.config("sql.table_name", "library_books")
case class LibraryBook(
  @Modifier.id bookNumber: Long,
  @Modifier.rename("book_title") title: String,
  firstAuthor: String
)

object LibraryBook {
  given Schema[LibraryBook] = Schema.derived
}
```

The code above sets the table name with `@Modifier.config("sql.table_name", ...)`, picks the ID field with `@Modifier.id` and renames one column with `@Modifier.rename`. The other field still follows the snake_case rule:

```scala
println(Table.derived[LibraryBook].createTable(SqlDialect.SQLite).sql(SqlDialect.SQLite))
// CREATE TABLE IF NOT EXISTS library_books (
//   book_number INTEGER NOT NULL,
//   book_title TEXT NOT NULL,
//   first_author TEXT NOT NULL
// )
println(Table.derived[Book](TableNamingPolicy.Plural).name)
// books
```

The second line shows `TableNamingPolicy.Plural`, which derives plural table names such as `books`. `Table.derived` and `Repo.derived` always use snake_case column names; if your columns use another convention, such as Magnum's default `SqlNameMapper.SameCase`, either rename the fields with `@Modifier.rename` or build the codec with a custom mapper, as [SqlNameMapper](../reference/sql/sql-name-mapper.md) shows.

### Repositories

Magnum's `Repo[EC, E, ID]` takes a separate creator type `EC` for inserts, so the database can generate columns such as the ID. ZIO Blocks has no creator type: `Repo.derived[E, ID]` inserts whole entities, so a Magnum `Repo[Book, Book, Long]` maps directly, and a repository with a creator type needs a hand-written insert, shown in [Generated Keys](#generated-keys) below:

```scala
// Before: Magnum 2.0.0-M4 (not compiled here)
val bookRepo = Repo[Book, Book, Long]
```

The ZIO Blocks repository is the `books` value from the shared bookshelf. Most methods keep their meaning but change their name or return type:

| Magnum `Repo` / `ImmutableRepo`          | ZIO Blocks `Repo`                                           |
| ---------------------------------------- | ----------------------------------------------------------- |
| `findById(id): Option[E]`                | `find(id): Maybe[E]`                                        |
| `findAll: Vector[E]`                     | `all: List[E]`                                              |
| `findAll(spec): Vector[E]`               | `SqlQuery` or `sql"..."`, see [Spec Queries](#spec-queries) |
| `findAllById(ids): Vector[E]`            | `findAll(ids): List[E]`                                     |
| `existsById(id): Boolean`                | `exists(id): Boolean`                                       |
| `count: Long`                            | `count: Long`                                               |
| `insert(creator): Unit`                  | `insert(entity): Int`                                       |
| `insertAll(creators): Unit`              | `insertBatch(entities): Int`                                |
| `insertReturning(creator): E`            | `insertReturning(entity): E`                                |
| `update(entity): Unit`                   | `update(entity): Int`                                       |
| `updateAll(entities): BatchUpdateResult` | `Repo#update` in a loop inside `JdbcTransactor#transact`    |
| `deleteById(id): Unit`                   | `delete(id): Int`                                           |
| `delete(entity): Unit`                   | `delete(entity.id): Int`                                    |
| `deleteAllById(ids): BatchUpdateResult`  | `deleteAll(ids): Int`                                       |
| `truncate(): Unit`                       | `clear(): Int`                                              |

:::warning
`findAll` means different things in the two libraries. Magnum's `ImmutableRepo#findAll` returns every row; ZIO Blocks' `Repo#findAll(ids)` returns the rows with the given IDs, and `Repo#all` returns every row.
:::

ZIO Blocks has no `ImmutableRepo`. To hand out a read-only view, expose a trait of your own with only the read methods. In return, `Repo` adds `Repo#insertOrUpdate` and `Repo#insertOrUpdateBatch` for upserts, and `Repo#pageAfter` for keyset pagination, described in [How to Upsert Rows and Paginate with Keyset Cursors](./upsert-and-paginate-with-keyset-cursors.md).

### Connections and Transactions

Magnum builds a `Transactor` from a `DataSource` and runs blocks with `connect` and `transact`:

```scala
// Before: Magnum 2.0.0-M4 (not compiled here)
val xa = Transactor(dataSource)

transact(xa) {
  bookRepo.insert(Book(1L, "Zionomicon", "John A. De Goes", 2024, None))
}

val total: Long = connect(xa) {
  bookRepo.count
}
```

ZIO Blocks has the same two blocks as methods on the transactor. `JdbcTransactor.fromDataSource(dataSource, dialect)` replaces `Transactor(dataSource)`, and `JdbcTransactor.fromUrl` skips the `DataSource` entirely. We put three books on the shelf:

```scala
val zionomicon         = Book(1L, "Zionomicon", "John A. De Goes", 2024, None)
val programmingInScala = Book(2L, "Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"))
val fpInScala          = Book(3L, "Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
```

Then we insert them in one transaction and count the rows on a plain connection:

```scala
transactor.transact {
  val inserted = books.insertBatch(List(zionomicon, programmingInScala, fpInScala))
  println(s"Rows inserted: $inserted")
}
// Rows inserted: 3

transactor.connect {
  println(s"Books on the shelf: ${books.count}")
}
// Books on the shelf: 3
```

Methods that take `(using DbCon)` or `(using DbTx)` port unchanged, because `DbTx` extends `DbCon` in both libraries. Two Magnum settings move. A `SqlLogger` becomes the third constructor argument of `JdbcTransactor`, as [SqlLogger](../reference/sql/sql-logger.md) shows. A `connectionConfig` that sets the isolation level becomes an argument of the transaction itself, `transactor.transact(TransactionIsolation.RepeatableRead, readOnly = false) { ... }`, described in [SQL Transactions](./sql-transactions.md).

### The sql Interpolator

The `sql"..."` interpolator works the same way in both libraries: every `$value` becomes a bound parameter, and a `Frag` interpolated into another `Frag` is spliced in. The difference is in how the result runs. Magnum turns a `Frag` into a `Query` or `Update` and calls `run()`; ZIO Blocks runs the statement as soon as you call `Frag#query` or `Frag#update`:

```scala
// Before: Magnum 2.0.0-M4 (not compiled here)
connect(xa) {
  val classics: Vector[Book] =
    sql"SELECT * FROM book WHERE year < 2010".query[Book].run()
  val changed: Int =
    sql"UPDATE book SET author = 'Martin Odersky et al.' WHERE id = 2".update.run()
}
```

The ZIO Blocks version drops `run()` and returns a `List` instead of a `Vector`:

```scala
val publishedBefore = 2010
// publishedBefore: Int = 2010

transactor.connect {
  val classics: List[Book] =
    sql"SELECT id, title, author, year, isbn FROM book WHERE year < $publishedBefore".query[Book]
  classics.foreach(println)
}
// Book(2,Programming in Scala,Martin Odersky,2008,Some(9780981531601))
```

Magnum's `TableInfo` interpolates column lists and table names, as in `sql"SELECT ${u.all} FROM $u"`. In ZIO Blocks, `Table#columns` and `Table#name` are plain strings, so splice them with `SqlLiteral`; a plain `String` would become a bound parameter:

```scala
val bookColumns = SqlLiteral(bookTable.columns.mkString(", "))
// bookColumns: SqlLiteral = SqlLiteral("id, title, author, year, isbn")
val bookName    = SqlLiteral(bookTable.name)
// bookName: SqlLiteral = SqlLiteral("book")

transactor.connect {
  val newest = sql"SELECT $bookColumns FROM $bookName ORDER BY year DESC".queryOne[Book]
  println(newest.fold("The shelf is empty")(book => s"Newest: ${book.title}"))
}
// Newest: Zionomicon
```

`SqlLiteral` splices raw text, exactly as it does in Magnum, so only use it for SQL your program builds itself, never for user input.

### Spec Queries

Magnum's `Spec[E]` adds `WHERE`, `ORDER BY` and `LIMIT` clauses to a repository's `SELECT`:

```scala
// Before: Magnum 2.0.0-M4 (not compiled here)
val spec = Spec[Book]
  .where(sql"year < 2010")
  .orderBy("year")
  .limit(10)

val classics: Vector[Book] = connect(xa)(bookRepo.findAll(spec))
```

The closest ZIO Blocks construct is `SqlQuery`, a query builder over a `Table`. Like `Spec`, it takes its conditions as `Frag`s and its column names as strings. `SqlQuery#toFrag` renders it for a dialect, and the result runs like any other `Frag`:

```scala
val classicsQuery =
  SqlQuery
    .from(bookTable)
    .where(sql"year < $publishedBefore")
    .orderBy("year")
    .limit(10)
```

The next block prints the SQL it renders for SQLite and runs it:

```scala
println(classicsQuery.sql(SqlDialect.SQLite))
// SELECT t0."id", t0."title", t0."author", t0."year", t0."isbn" FROM "book" AS t0 WHERE year < ? ORDER BY t0."year" ASC LIMIT 10

transactor.connect {
  classicsQuery.toFrag(SqlDialect.SQLite).query[Book].foreach(println)
}
// Book(2,Programming in Scala,Martin Odersky,2008,Some(9780981531601))
```

The [SqlQuery](../reference/sql/sql-query.md) reference lists every builder method. For Magnum's `Spec#seek`, which pages by key, use `Repo#pageAfter(cursorId, limit)`. For joins between tables, see [How to Write Joins and Inspect the Generated SQL](./write-joins-and-inspect-generated-sql.md). To build `WHERE` clauses from typed Scala expressions instead of SQL strings, see [Query DSL with Reified Optics — Part 2: SQL Generation](./query-dsl-sql.md).

### Custom Column Types

Magnum maps a wrapper type onto an existing codec with `DbCodec#biMap`. ZIO Blocks has the same idea under the name `DbCodec#transform`, with the two functions in separate parameter lists:

```scala
// Before: Magnum 2.0.0-M4 (not compiled here)
case class Isbn(value: String)
given DbCodec[Isbn] = DbCodec[String].biMap(Isbn(_), _.value)
```

The ZIO Blocks codec is used to read the column and, through the same `given`, to bind an `Isbn` as a parameter:

```scala
case class Isbn(value: String)

given DbCodec[Isbn] = DbCodec[String].transform(Isbn(_))(_.value)

transactor.connect {
  val isbns = sql"SELECT isbn FROM book WHERE isbn IS NOT NULL ORDER BY id".query[Isbn]
  println(isbns)

  val wanted = Isbn("9781617290657")
  val title  = sql"SELECT title FROM book WHERE isbn = $wanted".queryOne[String]
  println(title.getOrElse("No book with that ISBN"))
}
// List(Isbn(9780981531601), Isbn(9781617290657))
// Functional Programming in Scala
```

Scala 3 opaque types usually need no codec at all, because ZIO Blocks derives one from the underlying type; see [How to Map Custom Column Types](./map-custom-column-types.md) for opaque types, enums and JSON columns, and [DbCodec](../reference/sql/db-codec.md) for the rules.

## Coming from doobie

doobie and ZIO Blocks both let you write plain SQL with a `sql"..."` interpolator and read rows into case classes. The big change is the effect model, so we start there; after that, most of the work is renaming.

### From ConnectionIO to Direct Style

In doobie, a database operation is a value of type `ConnectionIO[A]`: a description of JDBC work that does nothing until you run it. You compose these descriptions with `flatMap` or a for-comprehension, then `.transact(xa)` turns the whole program into an `IO[A]` that opens a connection, runs the program in a transaction and commits or rolls back:

```scala
// Before: doobie 1.0.0-RC13 (not compiled here)
def addBook(book: Book): ConnectionIO[Int] =
  sql"INSERT INTO book (id, title, author, year, isbn) VALUES (${book.id}, ${book.title}, ${book.author}, ${book.year}, ${book.isbn})".update.run

def countBooks: ConnectionIO[Long] =
  sql"SELECT COUNT(*) FROM book".query[Long].unique

val program: ConnectionIO[Long] =
  for {
    _     <- addBook(Book(4L, "Hands-on Scala Programming", "Li Haoyi", 2020, None))
    total <- countBooks
  } yield total

val total: IO[Long] = program.transact(xa)
```

ZIO Blocks is direct style: a database method is an ordinary method that needs a `DbCon` and returns its result. It runs as soon as it is called, on the calling thread, and blocks until the database answers. The `ConnectionIO[A]` return type becomes `A` plus a `(using DbCon)` parameter:

```scala
val handsOnScala = Book(4L, "Hands-on Scala Programming", "Li Haoyi", 2020, None)

def addBook(book: Book)(using DbCon): Int =
  sql"INSERT INTO book (id, title, author, year, isbn) VALUES (${book.id}, ${book.title}, ${book.author}, ${book.year}, ${book.isbn})".update

def countBooks(using DbCon): Long =
  sql"SELECT COUNT(*) FROM book".queryOne[Long].getOrElse(0L)
```

Calling them inside one `JdbcTransactor#transact` block replaces the for-comprehension and `.transact(xa)`:

```scala
val total: Long =
  transactor.transact {
    addBook(handsOnScala)
    countBooks
  }
// total: Long = 4L

println(s"Books on the shelf: $total")
// Books on the shelf: 4
```

The compiler still tracks database access for you: `addBook` can only be called where a `DbCon` is in scope, which means inside `JdbcTransactor#connect` or `JdbcTransactor#transact`. A method that needs a transaction, such as one that uses savepoints, asks for `(using DbTx)` instead, and then only compiles inside `JdbcTransactor#transact`.

Two more consequences follow. Errors are exceptions: a failed statement throws the JDBC driver's exception, usually a `java.sql.SQLException`, instead of failing an `IO`, and `JdbcTransactor#transact` rolls back and rethrows it, so use `try`/`catch`, `scala.util.Try`, or the effect wrappers in [Effect Integration](#effect-integration) to handle it. You also choose between `JdbcTransactor#connect` and `JdbcTransactor#transact`: doobie's `.transact(xa)` always runs in a transaction, so `JdbcTransactor#transact` is its direct equivalent, while `JdbcTransactor#connect` runs in auto-commit mode, where each statement is saved as soon as it runs, which suits single reads.

### Transactors

doobie builds a transactor for an effect type, either from the `DriverManager` or from a Hikari pool in a `Resource`:

```scala
// Before: doobie 1.0.0-RC13 (not compiled here)
val xa = Transactor.fromDriverManager[IO](
  driver = "org.postgresql.Driver",
  url = "jdbc:postgresql:bookshelf",
  user = "postgres",
  password = "secret",
  logHandler = None
)

val pooled: Resource[IO, HikariTransactor[IO]] =
  HikariTransactor.fromHikariConfig[IO](hikariConfig)
```

A `JdbcTransactor` has no effect type and no driver class name, and takes a `SqlDialect` instead. Each doobie constructor maps as follows:

| doobie                                                                      | ZIO Blocks                                                             |
| --------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| `Transactor.fromDriverManager[IO](driver, url, logHandler)`                 | `JdbcTransactor.fromUrl(url, dialect)`                                 |
| `Transactor.fromDriverManager[IO](driver, url, user, password, logHandler)` | `JdbcTransactor.fromUrl(url, user, password, dialect)`                 |
| `Transactor.fromDataSource[IO](dataSource, connectEC)`                      | `JdbcTransactor.fromDataSource(dataSource, dialect)`                   |
| `HikariTransactor.fromHikariConfig[IO](config)`                             | `JdbcTransactor.fromDataSource(new HikariDataSource(config), dialect)` |
| `logHandler = Some(handler)`                                                | `new JdbcTransactor(connectionFactory, dialect, sqlLogger)`            |

With Hikari, you now own the pool: create the `HikariDataSource` at startup and close it at shutdown, which the `Resource` used to do for you. [How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md) shows the whole setup.

### Read, Write and Meta

doobie derives `Read` and `Write` instances for case classes, and uses `Meta`, `Get` and `Put` for single columns. ZIO Blocks replaces all five with `DbCodec`, which both reads and writes, and derives it from the entity's `Schema`. The `given Schema[Book]` in the shared bookshelf is all that `Frag#query[Book]` needs, and the same schema serves the other ZIO Blocks modules, such as the JSON codecs.

A custom single-column mapping moves from `Meta#timap` to `DbCodec#transform`:

```scala
// Before: doobie 1.0.0-RC13 (not compiled here)
case class Isbn(value: String)
given Meta[Isbn] = Meta[String].timap(Isbn(_))(_.value)
```

The ZIO Blocks version is the `given DbCodec[Isbn]` shown in [Custom Column Types](#custom-column-types), and it also lets you bind an `Isbn` as a query parameter. Unlike doobie's `Read`, which reads columns by position, a derived `DbCodec` matches result columns by name, so the order of the columns in a `SELECT` doesn't matter as long as their names match the snake_case field names.

### Fragments

doobie has three interpolators: `fr"..."` adds a trailing space, while `fr0"..."` and `sql"..."` don't. ZIO Blocks has only `sql"..."`, and `Frag#++` joins two fragments exactly as written, without adding a space, so write the spaces yourself. doobie's `Fragments.whereAndOpt` becomes a few lines of ordinary Scala:

```scala
// Before: doobie 1.0.0-RC13 (not compiled here)
def search(author: Option[String], before: Option[Int]): Fragment =
  fr"SELECT id, title, author, year, isbn FROM book" ++
    Fragments.whereAndOpt(author.map(a => fr"author = $a"), before.map(y => fr"year < $y")) ++
    fr"ORDER BY year"
```

The ZIO Blocks version collects the conditions and joins them with `AND`:

```scala
def search(author: Option[String], before: Option[Int]): Frag = {
  val conditions = author.map(a => sql"author = $a").toList ++ before.map(y => sql"year < $y").toList
  val where =
    if (conditions.isEmpty) Frag.empty
    else sql" WHERE " ++ conditions.reduce((left, right) => left ++ sql" AND " ++ right)
  sql"SELECT id, title, author, year, isbn FROM book" ++ where ++ sql" ORDER BY year"
}

val byOdersky = search(Some("Martin Odersky"), None)
// byOdersky: Frag = Frag(
//   parts = Vector(
//     "SELECT id, title, author, year, isbn FROM book WHERE author = ",
//     " ORDER BY year"
//   ),
//   params = Vector(DbString("Martin Odersky"))
// )
println(byOdersky.sql(SqlDialect.SQLite))
// SELECT id, title, author, year, isbn FROM book WHERE author = ? ORDER BY year

transactor.connect {
  byOdersky.query[Book].foreach(println)
}
// Book(2,Programming in Scala,Martin Odersky,2008,Some(9780981531601))
```

`Fragment.const` becomes `SqlLiteral`, shown in [The sql Interpolator](#the-sql-interpolator). There is no counterpart to `Fragments.in`: for ID lists, use `Repo#findAll(ids)` and `Repo#deleteAll(ids)`, which render the `IN (...)` list for you. For a whole search form's `WHERE` clause, with an `IN (...)` list of strings, a `LIKE` pattern and paging, see [How to Build a Query from Optional Search Filters](./build-a-query-from-optional-search-filters.md).

### Query Results

A doobie `Query0` picks its result shape with a method such as `.to[List]` or `.option`. In ZIO Blocks, the method on `Frag` picks the shape and runs the query:

| doobie                                     | ZIO Blocks                                               |
| ------------------------------------------ | -------------------------------------------------------- |
| `.query[A].to[List]`                       | `Frag#query[A]: List[A]`                                 |
| `.query[A].to[Vector]`                     | `Frag#query[A].toVector`                                 |
| `.query[A].option`                         | `Frag#queryOne[A]: Maybe[A]`                             |
| `.query[A].unique`                         | `Frag#queryOne[A]` plus `getOrElse(throw ...)`           |
| `.query[A].stream`                         | `Frag#queryStream[A]`, `Frag#queryChunked[A](chunkSize)` |
| `.update.run`                              | `Frag#update: Int`                                       |
| `.update.withUniqueGeneratedKeys[K]("id")` | `Frag#updateReturningKeys[K]: List[K]`                   |
| `Update[A](sql).updateMany(rows)`          | `Repo#insertBatch(rows)`                                 |

`Frag#queryOne` returns a `Maybe`, the ZIO Blocks counterpart of `Option`. Unlike doobie's `.option`, it doesn't fail when the query returns several rows; it returns the first one:

```scala
transactor.connect {
  val first   = sql"SELECT id, title, author, year, isbn FROM book ORDER BY id".queryOne[Book]
  val missing = sql"SELECT id, title, author, year, isbn FROM book WHERE id = 42".queryOne[Book]
  println(first.fold("No books")(book => s"First: ${book.title}"))
  println(missing.fold("No book with id 42")(book => s"Found: ${book.title}"))
}
// First: Zionomicon
// No book with id 42
```

#### Generated Keys

doobie's `.withUniqueGeneratedKeys` and Magnum's `.returningKeys` read the keys the database generates during an insert. In ZIO Blocks, `Frag#updateReturningKeys[K]` runs the insert and returns every generated key. The bookshelf table has no generated keys, so we create a reading log whose `INTEGER PRIMARY KEY` SQLite fills in:

```scala
transactor.connect {
  sql"CREATE TABLE IF NOT EXISTS reading_log (id INTEGER PRIMARY KEY, book_id INTEGER NOT NULL)".update

  val keys = sql"INSERT INTO reading_log (book_id) VALUES (${zionomicon.id})".updateReturningKeys[Long]
  println(s"Generated keys: $keys")
}
// Generated keys: List(1)
```

This is also the pattern for a Magnum repository with a creator type: write the `INSERT` without the generated columns and read the keys back.

#### Streaming

doobie streams rows as an `fs2.Stream[ConnectionIO, A]` and runs it with `.transact(xa)`. ZIO Blocks returns a ZIO Blocks `Stream[Throwable, Chunk[A]]` from `Frag#queryStream` or `Frag#queryChunked`; each element is a chunk of rows. The statement opens on the first pull, so consume the stream inside the `JdbcTransactor#connect` or `JdbcTransactor#transact` block that created it. Terminal operations such as `runFold` return an `Either` instead of throwing:

```scala
transactor.connect {
  val rowCount =
    sql"SELECT id, title, author, year, isbn FROM book ORDER BY id"
      .queryChunked[Book](2)
      .runFold(0)((count, chunk) => count + chunk.size)
  println(s"Streamed rows: $rowCount")
}
// Streamed rows: Right(4)
```

The code above reads the books in chunks of two and counts them; `Right(4)` means the stream finished without an error and saw four rows. [How to Stream Large Query Results](./stream-large-query-results.md) covers aggregating, exporting and handling errors in such streams.

### Checking Queries

doobie's `.check` and its `IOChecker` test helpers prepare each query against a live database and compare the SQL, the parameter types, the column types and their nullability with your Scala types. That catches type mismatches, but only in a test run that can reach a database.

ZIO Blocks checks at compile time instead, without a database. Pass one or more `Table` values to the interpolator, written as `StringContext(...).sql(table)(args)`, and the compiler checks every table and column name in the SQL against them, with a did-you-mean suggestion for typos. A misspelled column fails the build:

```scala
StringContext("SELECT title, auther FROM book WHERE year < ", "").sql(bookTable)(publishedBefore)
// error:
// Unknown identifier 'auther' at position 14; did you mean 'author'?
// StringContext("SELECT title, auther FROM book WHERE year < ", "").sql(bookTable)(publishedBefore)
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

The check is lint-grade: it catches misspelled tables and columns, but it doesn't check SQL types or nullability, so doobie-style analysis tests still have a place for complex queries. [sql(table) Checked Interpolation](./sql-checked-interpolation.md) lists exactly what is checked and how to opt out for one fragment.

## Effect Integration

ZIO Blocks runs database work as blocking calls, so in an effect system you run them on a thread pool meant for blocking work.

ZIO users add the `zio-blocks-sql-zio` module, which provides `TransactorZIO`. Its `TransactorZIO#connect` and `TransactorZIO#transact` take the same blocks as `JdbcTransactor` and return a `Task`, running the block with `ZIO.attemptBlocking`. `TransactorZIO#connectZIO` and `TransactorZIO#transactZIO` take a block that returns a `ZIO`, and close the connection even when the fiber is interrupted. Magnum's `magnumzio` users will recognize the class name:

```scala

case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String])

object Book {
  given Schema[Book] = Schema.derived
}

val books = Repo.derived[Book, Long]
val xa    = TransactorZIO.fromUrl("jdbc:sqlite:bookshelf.db", SqlDialect.SQLite)

val countBooks: Task[Long] = xa.connect(books.count)
val addBook: Task[Int]     = xa.transact(books.insert(Book(5L, "Zionomicon", "John A. De Goes", 2024, None)))
```

Keep `import zio.blocks.sql.*` after every other `zio` import, as above: once `sql-zio` is on the classpath, it brings the subpackage `zio.blocks.sql.zio` into scope under the name `zio`, and a later `import zio.blocks...` fails to compile. `TransactorZIO.layer(url, dialect)` provides the transactor as a `ZLayer`. [How to Use the sql Module from a ZIO Application](./sql-from-zio.md) covers layers and mixing database writes with other effects, and [TransactorZIO](../reference/sql/transactor-zio.md) lists every method.

cats-effect users don't get an integration module: ZIO Blocks has no counterpart to doobie's `Transactor[IO]` or Magnum's `TransactorCats`. Wrap each `JdbcTransactor#connect` or `JdbcTransactor#transact` call in `IO.blocking` yourself:

```scala
// cats-effect wrapper (not compiled here: cats-effect is not on this site's build)

def countBooks: IO[Long] =
  IO.blocking(transactor.connect(books.count))

def addBook(book: Book): IO[Int] =
  IO.blocking(transactor.transact(books.insert(book)))
```

`IO.blocking` runs the call on cats-effect's blocking pool and turns a thrown exception into a failed `IO`. The JDBC call itself is not interruptible, so cancelling the fiber waits for the statement to finish.

## What You Gain and What's Different

Moving to ZIO Blocks gives you:

- **One schema for everything.** The `Schema` that maps a row also drives JSON and the other ZIO Blocks formats, so an entity is described once.
- **Generated DDL and repositories.** `Table#createTable` writes the `CREATE TABLE` statement, and `Repo.derived` adds upserts and keyset pagination to the usual CRUD operations.
- **Compile-time query checking.** Misspelled tables and columns fail the build, without a database.
- **No effect system required.** The core module depends on no effect library; ZIO users add `zio-blocks-sql-zio`, and everyone else calls it directly.
- **Chunked streaming.** `Frag#queryChunked` streams rows in chunks of a size you choose.

Things that work differently, or don't exist yet:

- **Scala 3 only.** doobie users on Scala 2.13 have to move to Scala 3 first; Magnum is already Scala 3 only.
- **Two dialects.** `SqlDialect` supports PostgreSQL and SQLite. Magnum also supports MySQL, H2, Oracle, ClickHouse and SQL Server, and doobie works with any JDBC driver.
- **Collections and optional results.** Queries return `List` instead of Magnum's `Vector`, and single results are `Maybe` instead of `Option`.
- **No creator entities and no generated primary keys in DDL.** Inserts write every column; use `Frag#updateReturningKeys` and your own DDL for database-generated IDs.
- **No type analysis of SQL.** The compile-time check covers names only, not types or nullability.

## Migration Checklist

Work through these steps in order; each one leaves the project compiling:

1. Upgrade to Scala 3 if you are coming from doobie on Scala 2.
2. Replace the Magnum or doobie dependencies with `zio-blocks-sql` (or `zio-blocks-next-sql` with `-experimental` on Scala 3.9 and newer), and add `zio-blocks-sql-zio` if you use ZIO.
3. Give every entity a `given Schema[E] = Schema.derived`, and remove `@Table`, `@Id`, `derives DbCodec`, `Read` and `Write`.
4. Replace `@SqlName` with `@Modifier.rename`, `@Id` on a field that isn't named `id` with `@Modifier.id`, and custom table names with `@Modifier.config("sql.table_name", ...)`.
5. Port custom column types from `biMap` or `Meta#timap` to `DbCodec#transform`.
6. Build one `JdbcTransactor` with `JdbcTransactor.fromUrl` or `JdbcTransactor.fromDataSource`, passing the right `SqlDialect`.
7. Replace Magnum repositories with `Repo.derived[E, ID]` and rename their methods using the table in [Repositories](#repositories), watching out for `Repo#findAll`.
8. Turn every doobie `ConnectionIO[A]` method into a method returning `A` with `(using DbCon)`, and every `.transact(xa)` into `transactor.transact { ... }`.
9. Drop `run()` after Magnum queries; replace doobie's `.to[List]`, `.option`, `.unique` and `.update.run` with `Frag#query`, `Frag#queryOne` and `Frag#update`.
10. Rewrite `Spec` queries with `SqlQuery`, and `Fragments` helpers with `Frag#++` and explicit spaces.
11. Move fs2 or iterator streaming to `Frag#queryChunked`, consumed inside the `JdbcTransactor#connect` or `JdbcTransactor#transact` block.
12. Wrap transactor calls in `TransactorZIO` for ZIO, or in `IO.blocking` for cats-effect.
13. Pass `Table` values to `sql"..."` for the queries you want checked at compile time.

## Putting It Together

The complete example below is the bookshelf after the migration: entities, a repository, a hand-written query, a dynamic search, a stream and a transaction that rolls back, all on SQLite:

```scala title="sql-examples/src/main/scala/migratefrommagnumordoobie/CompleteExample.scala"
package migratefrommagnumordoobie

/**
 * Coming from Magnum or doobie — Complete Example
 *
 * The bookshelf after a migration from Magnum or doobie: a derived Repo, a
 * custom column type, a SqlQuery in place of a Magnum Spec, a dynamic search in
 * place of doobie's Fragments, a chunked stream and a transaction that rolls
 * back.
 *
 * Run with: cd sql-examples && sbt "runMain
 * migratefrommagnumordoobie.CompleteExample"
 */
object CompleteExample extends App {

  case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String])

  object Book {
    given Schema[Book] = Schema.derived
  }

  // Replaces Magnum's DbCodec#biMap and doobie's Meta#timap
  case class Isbn(value: String)

  given DbCodec[Isbn] = DbCodec[String].transform(Isbn(_))(_.value)

  val dbFile = java.nio.file.Files.createTempFile("bookshelf", ".db")
  dbFile.toFile.deleteOnExit()

  // Replaces Magnum's Transactor(dataSource) and doobie's Transactor.fromDriverManager[IO]
  val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

  val bookTable = Table.derived[Book]

  // Replaces Magnum's Repo[Book, Book, Long]
  val books = Repo.derived[Book, Long]

  // Replaces doobie's ConnectionIO[List[Book]]
  def booksBefore(year: Int)(using DbCon): List[Book] =
    sql"SELECT id, title, author, year, isbn FROM book WHERE year < $year".query[Book]

  // Replaces doobie's Fragments.whereAndOpt
  def search(author: Option[String], before: Option[Int]): Frag = {
    val conditions = author.map(a => sql"author = $a").toList ++ before.map(y => sql"year < $y").toList
    val where      =
      if (conditions.isEmpty) Frag.empty
      else sql" WHERE " ++ conditions.reduce((left, right) => left ++ sql" AND " ++ right)
    sql"SELECT id, title, author, year, isbn FROM book" ++ where ++ sql" ORDER BY year"
  }

  transactor.transact {
    bookTable.createTable(SqlDialect.SQLite).update
    books.insertBatch(
      List(
        Book(1L, "Zionomicon", "John A. De Goes", 2024, None),
        Book(2L, "Programming in Scala", "Martin Odersky", 2008, Some("9780981531601")),
        Book(3L, "Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
      )
    )
  }

  transactor.connect {
    println(s"Books on the shelf: ${books.count}")

    // Magnum's findById returned Option; Repo#find returns Maybe
    val found: Maybe[Book] = books.find(1L)
    println(found.fold("No book with id 1")(book => s"Found: ${book.title}"))

    println("Published before 2010:")
    booksBefore(2010).foreach(book => println(s"  ${book.title}"))

    // Replaces Magnum's Spec[Book].where(...).orderBy(...).limit(...)
    val oldest = SqlQuery.from(bookTable).orderBy("year").limit(2)
    println("Two oldest books:")
    oldest.toFrag(SqlDialect.SQLite).query[Book].foreach(book => println(s"  ${book.title}"))

    println("By Martin Odersky:")
    search(Some("Martin Odersky"), None).query[Book].foreach(book => println(s"  ${book.title}"))

    val isbns = sql"SELECT isbn FROM book WHERE isbn IS NOT NULL ORDER BY id".query[Isbn]
    println(s"Known ISBNs: ${isbns.map(_.value).mkString(", ")}")

    // Replaces doobie's .stream and Magnum's .iterator(); consumed inside connect
    val streamed =
      sql"SELECT id, title, author, year, isbn FROM book ORDER BY id"
        .queryChunked[Book](2)
        .runFold(0)((count, chunk) => count + chunk.size)
    println(s"Streamed rows: $streamed")
  }

  // Replaces doobie's program.transact(xa) and Magnum's transact(xa)
  Try {
    transactor.transact {
      books.insert(Book(4L, "Hands-on Scala Programming", "Li Haoyi", 2020, None))
      if (books.count > 3) throw new IllegalStateException("The shelf holds at most 3 books")
    }
  } match {
    case Failure(error) => println(s"Transaction rolled back: ${error.getMessage}")
    case Success(_)     => println("Transaction committed")
  }

  transactor.connect {
    println(s"Books after the rollback: ${books.count}")
  }
}
```

## Running the Examples

The complete example has a runnable Scala file in the `sql-examples` project. To get it, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/sql-examples
```

The example creates its own temporary database file, so you can run it as often as you like.

<details>
  <summary>sql-examples/src/main/scala/migratefrommagnumordoobie/CompleteExample.scala</summary>

```scala title="sql-examples/src/main/scala/migratefrommagnumordoobie/CompleteExample.scala" showLineNumbers
package migratefrommagnumordoobie

/**
 * Coming from Magnum or doobie — Complete Example
 *
 * The bookshelf after a migration from Magnum or doobie: a derived Repo, a
 * custom column type, a SqlQuery in place of a Magnum Spec, a dynamic search in
 * place of doobie's Fragments, a chunked stream and a transaction that rolls
 * back.
 *
 * Run with: cd sql-examples && sbt "runMain
 * migratefrommagnumordoobie.CompleteExample"
 */
object CompleteExample extends App {

  case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String])

  object Book {
    given Schema[Book] = Schema.derived
  }

  // Replaces Magnum's DbCodec#biMap and doobie's Meta#timap
  case class Isbn(value: String)

  given DbCodec[Isbn] = DbCodec[String].transform(Isbn(_))(_.value)

  val dbFile = java.nio.file.Files.createTempFile("bookshelf", ".db")
  dbFile.toFile.deleteOnExit()

  // Replaces Magnum's Transactor(dataSource) and doobie's Transactor.fromDriverManager[IO]
  val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

  val bookTable = Table.derived[Book]

  // Replaces Magnum's Repo[Book, Book, Long]
  val books = Repo.derived[Book, Long]

  // Replaces doobie's ConnectionIO[List[Book]]
  def booksBefore(year: Int)(using DbCon): List[Book] =
    sql"SELECT id, title, author, year, isbn FROM book WHERE year < $year".query[Book]

  // Replaces doobie's Fragments.whereAndOpt
  def search(author: Option[String], before: Option[Int]): Frag = {
    val conditions = author.map(a => sql"author = $a").toList ++ before.map(y => sql"year < $y").toList
    val where      =
      if (conditions.isEmpty) Frag.empty
      else sql" WHERE " ++ conditions.reduce((left, right) => left ++ sql" AND " ++ right)
    sql"SELECT id, title, author, year, isbn FROM book" ++ where ++ sql" ORDER BY year"
  }

  transactor.transact {
    bookTable.createTable(SqlDialect.SQLite).update
    books.insertBatch(
      List(
        Book(1L, "Zionomicon", "John A. De Goes", 2024, None),
        Book(2L, "Programming in Scala", "Martin Odersky", 2008, Some("9780981531601")),
        Book(3L, "Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
      )
    )
  }

  transactor.connect {
    println(s"Books on the shelf: ${books.count}")

    // Magnum's findById returned Option; Repo#find returns Maybe
    val found: Maybe[Book] = books.find(1L)
    println(found.fold("No book with id 1")(book => s"Found: ${book.title}"))

    println("Published before 2010:")
    booksBefore(2010).foreach(book => println(s"  ${book.title}"))

    // Replaces Magnum's Spec[Book].where(...).orderBy(...).limit(...)
    val oldest = SqlQuery.from(bookTable).orderBy("year").limit(2)
    println("Two oldest books:")
    oldest.toFrag(SqlDialect.SQLite).query[Book].foreach(book => println(s"  ${book.title}"))

    println("By Martin Odersky:")
    search(Some("Martin Odersky"), None).query[Book].foreach(book => println(s"  ${book.title}"))

    val isbns = sql"SELECT isbn FROM book WHERE isbn IS NOT NULL ORDER BY id".query[Isbn]
    println(s"Known ISBNs: ${isbns.map(_.value).mkString(", ")}")

    // Replaces doobie's .stream and Magnum's .iterator(); consumed inside connect
    val streamed =
      sql"SELECT id, title, author, year, isbn FROM book ORDER BY id"
        .queryChunked[Book](2)
        .runFold(0)((count, chunk) => count + chunk.size)
    println(s"Streamed rows: $streamed")
  }

  // Replaces doobie's program.transact(xa) and Magnum's transact(xa)
  Try {
    transactor.transact {
      books.insert(Book(4L, "Hands-on Scala Programming", "Li Haoyi", 2020, None))
      if (books.count > 3) throw new IllegalStateException("The shelf holds at most 3 books")
    }
  } match {
    case Failure(error) => println(s"Transaction rolled back: ${error.getMessage}")
    case Success(_)     => println("Transaction committed")
  }

  transactor.connect {
    println(s"Books after the rollback: ${books.count}")
  }
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "runMain migratefrommagnumordoobie.CompleteExample"
```

To compile the example without running it:

```bash
sbt compile
```

## Going Further

- [Build Your First Database Repository on SQLite](./your-first-sqlite-repository.md) teaches the basics this guide builds on.
- [SQL Transactions](./sql-transactions.md) covers isolation levels, read-only transactions and nested transactions with savepoints.
- [sql(table) Checked Interpolation](./sql-checked-interpolation.md) explains the compile-time query check in detail.
- [Repo](../reference/sql/repo.md) and [Frag](../reference/sql/frag.md) list every repository and fragment operation.
- [SQL Module](../reference/sql/index.md) links to every `sql` reference page.

---

## How to Migrate Table Rows to a New Version Online

In this guide, we change the row type of a live `customers` table from `CustomerV1`, which stores one `full_name` column, to `CustomerV2`, which stores `first_name` and `last_name`, and we convert every existing row while the application keeps reading and writing. By the end, you will have backfilled a small SQLite table in place with `SmallMigrator`, copied a 20,000-row PostgreSQL table into a shadow table with `LargeMigrator` while another thread writes to it, and swapped the new table in without losing a write.

We describe the row change once, as a typed [`Migration`](../reference/schema/migration.md), and let the `dataMigration` module run it in batches from a queue of row keys. Along the way, we:

- Describe the old and new rows and the conversion between them
- Preview the SQL a migration would run, before running anything
- Backfill a small table in place with `SmallMigrator`
- Migrate a large PostgreSQL table through a shadow table with `LargeMigrator`
- Recover from a failed batch and restart a migration
- Choose an execution model for your own table

This guide assumes you know SQL, have built a repository as in [Your First SQLite Repository](./your-first-sqlite-repository.md), and have written a `Migration` as in [Evolve Your Data](./evolve-your-data.md). We don't explain `Repo`, `JdbcTransactor` or `Migration.newBuilder` again here.

## The Problem

Our `customers` table has a `full_name` column, and the next version of the application needs `first_name` and `last_name` instead. The table is live: customers sign up and change their details all day, and we can't take the application down while we rewrite every row.

The usual first attempt is a backfill loop that reads rows in pages and updates them. This is that loop in plain JDBC:

```scala

def backfillNames(connection: Connection): Unit = {
  val select = connection.prepareStatement(
    "SELECT id, full_name FROM customers WHERE first_name IS NULL ORDER BY id LIMIT 500"
  )
  val update = connection.prepareStatement("UPDATE customers SET first_name = ?, last_name = ? WHERE id = ?")
  var done   = false
  while (!done) {
    val rows = select.executeQuery()
    done = true
    while (rows.next()) {
      done = false
      val fullName = rows.getString("full_name")
      val space    = fullName.indexOf(' ')
      update.setString(1, fullName.substring(0, space))
      update.setString(2, fullName.substring(space + 1))
      update.setLong(3, rows.getLong("id"))
      update.executeUpdate()
    }
    rows.close()
  }
}
```

This loop has three problems. A customer who changes their name after the loop has passed their row keeps the old `first_name`, because `first_name IS NULL` never selects that row again. A single-word name such as "Cher" makes `String#substring` throw, and every rerun throws at the same row. And the loop only works while the new columns sit beside the old one: when `full_name` has to go and the new columns must be `NOT NULL`, we need a new table and a moment when the application switches to it, and the loop gives us neither.

In this guide, we let the `dataMigration` module do the bookkeeping. It keeps the keys of the rows still to convert in a queue table, converts them in batches with a typed `Migration`, and, for a new table, records every concurrent write in the same queue with database triggers until we swap the tables.

## Prerequisites

Add the `dataMigration` module, which brings in the `sql` and `schema` modules, and the JDBC drivers of the databases you use to your `build.sbt`. [How to Add ZIO Blocks to Your Build](../getting-started/installation.md) explains the two artifact names:

```scala
// Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-data-migration" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-data-migration" % "0.0.56"
scalacOptions += "-experimental"

// JDBC drivers, for every Scala version
libraryDependencies += "org.xerial"     % "sqlite-jdbc" % "3.53.4.0"
libraryDependencies += "org.postgresql" % "postgresql"  % "42.7.14"
```

The module is published for Scala 3 only; there is no Scala 2.13 artifact. It runs on the JVM, through [`JdbcTransactor`](../reference/sql/transactor.md); the code in this guide runs on JDK 17 and newer.

The large-table section needs PostgreSQL 14 or newer, because the capture triggers use `CREATE OR REPLACE TRIGGER`. If you have Docker, this command starts PostgreSQL 16 with a `shop` database, user and password:

```bash
docker run --rm -d --name shop-db -p 5432:5432 \
  -e POSTGRES_USER=shop -e POSTGRES_PASSWORD=shop -e POSTGRES_DB=shop \
  postgres:16
```

The PostgreSQL code reads the database URL, user and password from `SHOP_DB_URL`, `SHOP_DB_USER` and `SHOP_DB_PASSWORD`, with defaults that match the Docker command, so it runs without any variables set. The SQLite code needs no server.

## Describe the Old and New Rows

A migration needs three things: the row type the table holds today, the row type it should hold tomorrow, and a conversion from one to the other. We describe all three once and use them for every database in this guide:

```scala

final case class CustomerV1(id: Long, fullName: String, email: String)

object CustomerV1 extends CompanionOptics[CustomerV1] {
  implicit val schema: Schema[CustomerV1] = Schema.derived

  val fullName: Lens[CustomerV1, String] = $(_.fullName)
}

final case class CustomerV2(id: Long, firstName: String, lastName: String, email: String)

object CustomerV2 {
  implicit val schema: Schema[CustomerV2] = Schema.derived
}

val customersV1 = Repo.derived[CustomerV1, Long]("customers", "id", _.id)
val customersV2 = Repo.derived[CustomerV2, Long]("customers", "id", _.id)

val fullName   = SchemaExpr.optic[CustomerV1, String](CustomerV1.fullName.toDynamic, CustomerV1.schema)
val firstSpace = SchemaExpr.stringIndexOf(fullName, SchemaExpr.literal(" "))

val splitName: Migration[CustomerV1, CustomerV2] =
  Migration
    .newBuilder[CustomerV1, CustomerV2]
    .addField(_.firstName, SchemaExpr.stringSubstring(fullName, SchemaExpr.literal(0), firstSpace))
    .addField(
      _.lastName,
      SchemaExpr.stringTrim(SchemaExpr.stringSubstring(fullName, firstSpace, SchemaExpr.stringLength(fullName)))
    )
    .dropField(_.fullName, SchemaExpr.literal(""))
    .build
```

We keep `import zio.blocks.sql.*` after the other `zio.blocks` imports: if your project also uses the `sql-zio` module, it brings a package named `zio` into scope, and a `zio.blocks` import after it fails to compile.

Both repositories name the same table, `customers`. `customersV1` reads and writes the old columns (`id`, `full_name`, `email`), and `customersV2` the new ones (`id`, `first_name`, `last_name`, `email`). The migrators read each row through the first and write it through the second, and for a shadow table they derive the shadow table's name from the second.

`splitName` is built from [`SchemaExpr`](../reference/schema/schema-expr.md) expressions instead of a Scala function, and the migration evaluates them on each row. `firstSpace` is the index of the first space in the full name. `MigrationBuilder#addField` computes each new field from the whole `CustomerV1` value: `firstName` is the text before the first space, and `lastName` is the rest, trimmed. `MigrationBuilder#dropField` removes `fullName`, and `MigrationBuilder#build` checks at compile time that every field of `CustomerV2` is accounted for.

Applying the migration to two customers shows the split:

```scala
println(splitName(CustomerV1(1, "Ada Lovelace", "ada@example.com")))
// Right(CustomerV2(1,Ada,Lovelace,ada@example.com))
println(splitName(CustomerV1(2, "Grace Brewster Hopper", "grace@example.com")))
// Right(CustomerV2(2,Grace,Brewster Hopper,grace@example.com))
```

`Migration#apply` returns an `Either`, so a row that the conversion can't handle becomes a `Left`, not an exception. We meet such a row in [Recover from a Failed Batch and Restart](#recover-from-a-failed-batch-and-restart).

All the examples in this guide share these definitions, together with three pieces we show in later sections, in one file:

<details>
  <summary>The shared model: <code>Customers.scala</code></summary>

```scala title="sql-examples/src/main/scala/migratetablerowsonline/Customers.scala" showLineNumbers
package migratetablerowsonline

/**
 * How to Migrate Table Rows to a New Version Online — Shared Model
 *
 * The old and new customer rows, their repositories on the `customers` table,
 * the migrations that split `fullName` into `firstName` and `lastName`, and the
 * DDL of the old table and the startup migration that adds the new columns. It
 * has no main method.
 */
final case class CustomerV1(id: Long, fullName: String, email: String)

object CustomerV1 extends CompanionOptics[CustomerV1] {
  implicit val schema: Schema[CustomerV1] = Schema.derived

  val fullName: Lens[CustomerV1, String] = $(_.fullName)
}

final case class CustomerV2(id: Long, firstName: String, lastName: String, email: String)

object CustomerV2 {
  implicit val schema: Schema[CustomerV2] = Schema.derived
}

val customersV1 = Repo.derived[CustomerV1, Long]("customers", "id", _.id)
val customersV2 = Repo.derived[CustomerV2, Long]("customers", "id", _.id)

val fullName   = SchemaExpr.optic[CustomerV1, String](CustomerV1.fullName.toDynamic, CustomerV1.schema)
val firstSpace = SchemaExpr.stringIndexOf(fullName, SchemaExpr.literal(" "))

val splitName: Migration[CustomerV1, CustomerV2] =
  Migration
    .newBuilder[CustomerV1, CustomerV2]
    .addField(_.firstName, SchemaExpr.stringSubstring(fullName, SchemaExpr.literal(0), firstSpace))
    .addField(
      _.lastName,
      SchemaExpr.stringTrim(SchemaExpr.stringSubstring(fullName, firstSpace, SchemaExpr.stringLength(fullName)))
    )
    .dropField(_.fullName, SchemaExpr.literal(""))
    .build

val paddedName  = SchemaExpr.stringConcat(fullName, SchemaExpr.literal(" "))
val paddedSpace = SchemaExpr.stringIndexOf(paddedName, SchemaExpr.literal(" "))

val splitNameSafely: Migration[CustomerV1, CustomerV2] =
  Migration
    .newBuilder[CustomerV1, CustomerV2]
    .addField(_.firstName, SchemaExpr.stringSubstring(paddedName, SchemaExpr.literal(0), paddedSpace))
    .addField(
      _.lastName,
      SchemaExpr.stringTrim(SchemaExpr.stringSubstring(paddedName, paddedSpace, SchemaExpr.stringLength(paddedName)))
    )
    .dropField(_.fullName, SchemaExpr.literal(""))
    .build

def createCustomersV1()(using DbCon): Unit =
  Frag
    .literal("CREATE TABLE customers (id BIGINT PRIMARY KEY, full_name TEXT NOT NULL, email TEXT NOT NULL)")
    .update

class AddNameColumns(transactor: Transactor) extends TinyMigrator(transactor) {
  def run()(using DbTx): Unit = {
    Frag.literal("ALTER TABLE customers ADD COLUMN first_name TEXT").update
    Frag.literal("ALTER TABLE customers ADD COLUMN last_name TEXT").update
  }
}
```

</details>

## Preview the SQL Before Running Anything

Before a migration touches a production database, we want to read the SQL it will run. `SmallMigrator#previewSql` and `LargeMigrator#previewSql` render that SQL for the dialect in scope without opening a connection, so the transactor they ask for doesn't need a running database.

For PostgreSQL, we write the transactor once, as a helper that the large-table section reuses:

```scala
def shopTransactor(): JdbcTransactor =
  JdbcTransactor.fromUrl(
    sys.env.getOrElse("SHOP_DB_URL", "jdbc:postgresql://localhost:5432/shop"),
    sys.env.getOrElse("SHOP_DB_USER", "shop"),
    sys.env.getOrElse("SHOP_DB_PASSWORD", "shop"),
    SqlDialect.PostgreSQL
  )
```

The first preview is an in-place migration on SQLite, which updates the rows of `customers` where they are:

```scala
val inPlace = {
  given Transactor = JdbcTransactor.fromUrl("jdbc:sqlite:customers.db", SqlDialect.SQLite)
  given Dialect    = Dialect.SQLite
  SmallMigrator[CustomerV1, CustomerV2, Long, Long](
    repoV1 = customersV1,
    repoV2 = customersV2,
    migration = splitName,
    queueTable = "customers_split_q",
    batchSize = 500,
    target = TargetStrategy.InPlace
  )
}
```

`SmallMigrator` takes the two repositories, the migration, the name of the queue table that holds the keys still to convert, the number of keys per batch and a `TargetStrategy`. The `Dialect` given picks the SQL dialect of the generated statements. Printing its preview:

```scala
inPlace.previewSql().foreach(println)
// CREATE TABLE IF NOT EXISTS customers_split_q (
//   id TEXT NOT NULL PRIMARY KEY,
//   op TEXT NOT NULL DEFAULT 'I',
//   payload TEXT
// )
// SELECT id FROM customers_split_q ORDER BY id LIMIT 500
```

The preview shows only the queue table and the query that claims a batch of keys: an in-place migration creates no other table.

The second preview is a shadow-table migration on PostgreSQL, which copies the rows into a new table and swaps it in at the end:

```scala
val shadow = {
  given Transactor = shopTransactor()
  given Dialect    = Dialect.Postgres
  LargeMigrator[CustomerV1, CustomerV2, Long, Long](
    repoV1 = customersV1,
    repoV2 = customersV2,
    migration = splitName,
    queueTable = "customers_split_q",
    batchSize = 500,
    target = TargetStrategy.ShadowTable("v2"),
    captureTriggers = true
  )
}
```

Its preview is longer:

```scala
shadow.previewSql().foreach(println)
// CREATE TABLE IF NOT EXISTS customers_split_q (
//   id TEXT NOT NULL PRIMARY KEY,
//   op TEXT NOT NULL DEFAULT 'I',
//   payload TEXT
// )
// CREATE TABLE IF NOT EXISTS customers_v2 (LIKE customers INCLUDING ALL)
// CREATE OR REPLACE FUNCTION customers_split_q_notify()
// RETURNS TRIGGER AS $$
// BEGIN
//   IF TG_OP = 'DELETE' THEN
//     INSERT INTO customers_split_q (id, op, payload) VALUES (OLD.id, 'D', row_to_json(OLD)::text) ON CONFLICT (id) DO UPDATE SET op = 'D', payload = EXCLUDED.payload;
//   ELSIF TG_OP = 'INSERT' THEN
//     INSERT INTO customers_split_q (id, op) VALUES (NEW.id, 'I') ON CONFLICT (id) DO NOTHING;
//   ELSIF TG_OP = 'UPDATE' THEN
//     INSERT INTO customers_split_q (id, op) VALUES (NEW.id, 'U') ON CONFLICT (id) DO NOTHING;
//   END IF;
//   RETURN NULL;
// END;
// $$ LANGUAGE plpgsql;
// CREATE OR REPLACE TRIGGER trg_customers_split_q_mod AFTER INSERT OR UPDATE OR DELETE ON customers FOR EACH ROW EXECUTE FUNCTION customers_split_q_notify();
// SELECT id FROM customers_split_q ORDER BY id LIMIT 500 FOR UPDATE SKIP LOCKED
// ALTER TABLE customers RENAME TO customers_old_v2
// ALTER TABLE customers_v2 RENAME TO customers
```

This preview tells us three things we have to act on before we run anything:

- **Nobody creates the queue table for us.** The preview lists its `CREATE TABLE`, but `SmallMigrator#init` and `LargeMigrator#init` don't run it. We create it with `QueueTable.create` first.
- **The shadow table copies the old shape.** `CREATE TABLE IF NOT EXISTS customers_v2 (LIKE customers INCLUDING ALL)` makes `customers_v2` a copy of today's `customers`, with `full_name` and without `first_name` or `last_name`, so writing a `CustomerV2` row into it would fail. We create `customers_v2` ourselves with the new columns before `LargeMigrator#init`, and the `IF NOT EXISTS` then leaves our table alone.
- **The triggers record every write.** With `captureTriggers = true`, a trigger on `customers` adds the key of every inserted, updated or deleted row to the queue, inside the writer's own transaction. The swap at the end is the two `ALTER TABLE ... RENAME` statements.

The claim query differs by database: PostgreSQL claims keys with `FOR UPDATE SKIP LOCKED`, so several workers can share a queue, and SQLite claims them with a plain `LIMIT`, so one worker processes a SQLite queue.

<details>
  <summary>Runnable version, with both previews in one program: <code>Step2PreviewSqlExample.scala</code></summary>

```scala title="sql-examples/src/main/scala/migratetablerowsonline/Step2PreviewSqlExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-data-migration:0.0.56
//> using dep org.postgresql:postgresql:42.7.14
//> using file Customers.scala
//> using file ShopDatabase.scala
// Run from this folder:    scala-cli run Step2PreviewSqlExample.scala
// Or from sql-examples:    sbt "runMain migratetablerowsonline.Step2PreviewSqlExample"

package migratetablerowsonline

/**
 * How to Migrate Table Rows to a New Version Online — Preview the SQL Before
 * Running Anything
 *
 * Prints the statements an in-place SQLite migration and a shadow-table
 * PostgreSQL migration would run, without opening a connection.
 */
object Step2PreviewSqlExample {
  def main(args: Array[String]): Unit = {
    val inPlace = {
      given Transactor = JdbcTransactor.fromUrl("jdbc:sqlite:customers.db", SqlDialect.SQLite)
      given Dialect    = Dialect.SQLite
      SmallMigrator[CustomerV1, CustomerV2, Long, Long](
        repoV1 = customersV1,
        repoV2 = customersV2,
        migration = splitName,
        queueTable = "customers_split_q",
        batchSize = 500,
        target = TargetStrategy.InPlace
      )
    }

    inPlace.previewSql().foreach(println)

    val shadow = {
      given Transactor = shopTransactor()
      given Dialect    = Dialect.Postgres
      LargeMigrator[CustomerV1, CustomerV2, Long, Long](
        repoV1 = customersV1,
        repoV2 = customersV2,
        migration = splitName,
        queueTable = "customers_split_q",
        batchSize = 500,
        target = TargetStrategy.ShadowTable("v2"),
        captureTriggers = true
      )
    }

    shadow.previewSql().foreach(println)
  }
}
```

</details>

<details>
  <summary>The shared PostgreSQL setup, including helpers from the large-table section: <code>ShopDatabase.scala</code></summary>

```scala title="sql-examples/src/main/scala/migratetablerowsonline/ShopDatabase.scala" showLineNumbers
package migratetablerowsonline

/**
 * How to Migrate Table Rows to a New Version Online — Shared PostgreSQL Setup
 *
 * The transactor for the `shop` database, the cleanup that lets the examples
 * run again, and the generated customers of the large table. It has no main
 * method.
 */
def shopTransactor(): JdbcTransactor =
  JdbcTransactor.fromUrl(
    sys.env.getOrElse("SHOP_DB_URL", "jdbc:postgresql://localhost:5432/shop"),
    sys.env.getOrElse("SHOP_DB_USER", "shop"),
    sys.env.getOrElse("SHOP_DB_PASSWORD", "shop"),
    SqlDialect.PostgreSQL
  )

def resetShop()(using DbCon): Unit = {
  Frag.literal("DROP TABLE IF EXISTS customers, customers_v2, customers_old_v2, customers_split_q CASCADE").update
  Frag.literal("DROP FUNCTION IF EXISTS customers_split_q_notify() CASCADE").update
}

val firstNames = Vector("Ada", "Alan", "Grace", "Edsger", "Barbara", "Donald", "Margaret", "Ken")
val lastNames  = Vector("Lovelace", "Turing", "Hopper", "Dijkstra", "Liskov", "Knuth", "Hamilton", "Thompson")

def sampleCustomer(id: Long): CustomerV1 =
  CustomerV1(
    id,
    s"${firstNames(((id - 1) % 8).toInt)} ${lastNames(((id - 1) / 8 % 8).toInt)}",
    s"customer$id@example.com"
  )
```

</details>

## Backfill a Small Table in Place

When the new columns can live beside the old one for a while, we convert the rows where they are. This is the classic expand-and-contract change: we add `first_name` and `last_name` next to `full_name`, fill them in while the old code keeps writing `full_name`, and drop `full_name` when the new code starts. `SmallMigrator` does the filling, and we drive it batch by batch. We use SQLite here; the same code runs on PostgreSQL with `Dialect.Postgres`.

The table holds three customers to start with:

```scala

def createCustomersV1()(using DbCon): Unit =
  Frag
    .literal("CREATE TABLE customers (id BIGINT PRIMARY KEY, full_name TEXT NOT NULL, email TEXT NOT NULL)")
    .update

val dbFile     = Files.createTempFile("customers", ".db")
val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
given Transactor = transactor
given Dialect    = Dialect.SQLite

transactor.transact {
  createCustomersV1()
  customersV1.insertBatch(
    List(
      CustomerV1(1, "Ada Lovelace", "ada@example.com"),
      CustomerV1(2, "Alan Turing", "alan@example.com"),
      CustomerV1(3, "Grace Brewster Hopper", "grace@example.com")
    )
  )
}
```

The migrators find the `Transactor` and `Dialect` as givens, so we declare both once.

### Add the New Columns

An in-place migration writes `first_name` and `last_name` into `customers`, so the table needs those columns before the first batch. A `TinyMigrator` runs schema changes like this one in a single transaction, typically at application startup. We extend it and put the statements in `TinyMigrator#run`:

```scala
class AddNameColumns(transactor: Transactor) extends TinyMigrator(transactor) {
  def run()(using DbTx): Unit = {
    Frag.literal("ALTER TABLE customers ADD COLUMN first_name TEXT").update
    Frag.literal("ALTER TABLE customers ADD COLUMN last_name TEXT").update
  }
}

AddNameColumns(transactor).migrate()
```

The new columns are nullable, so adding them rewrites no row, and the old code, which never names them, keeps working.

### Queue Every Row

The migrator converts only the rows whose keys are in the queue table. We create the queue and add the key of every existing customer:

```scala
QueueTable.create[Long]("customers_split_q", transactor)
transactor.transact(QueueTable.enqueue("customers_split_q", customersV1.all.map(_.id)))
```

`QueueTable.create[Long]` takes the key type, and the queue stores each key as text. `QueueTable.enqueue` ignores keys that are already queued, so queuing the same row twice converts it once.

:::warning
Don't set `captureTriggers = true` for an in-place migration. The triggers sit on the table the migrator writes to, so every converted row queues itself again and the queue never empties. In place, the application queues its own writes, as the next part shows.
:::

### Convert the Rows While the Old Code Writes

We create the migrator with a batch size of two, so three rows take two batches:

```scala
val migrator = SmallMigrator[CustomerV1, CustomerV2, Long, Long](
  repoV1 = customersV1,
  repoV2 = customersV2,
  migration = splitName,
  queueTable = "customers_split_q",
  batchSize = 2,
  target = TargetStrategy.InPlace
)
```

We call `SmallMigrator#init` once and convert the first batch:

```scala
migrator.init()
println(s"first batch: ${migrator.processBatch()} rows")
// first batch: 2 rows
```

`SmallMigrator#processBatch` claims up to two keys, reads their rows through `customersV1`, applies `splitName` and writes the results through `customersV2`, all in one transaction.

Meanwhile, the old version of the application is still running. It signs up Edsger and changes Ada's name, and, because no trigger watches the table, it adds both keys to the queue in the same transaction as the writes:

```scala
transactor.transact {
  customersV1.insert(CustomerV1(4, "Edsger Dijkstra", "edsger@example.com"))
  customersV1.update(CustomerV1(1, "Ada King", "ada@example.com"))
  QueueTable.enqueue("customers_split_q", List(4L, 1L))
}
```

Ada's row was converted in the first batch, so without the second key her `first_name` and `last_name` would keep the old name. We convert batches until the queue is empty and then call `SmallMigrator#complete`:

```scala
while (transactor.connect(QueueTable.pending("customers_split_q")) > 0)
  println(s"next batch: ${migrator.processBatch()} rows")
// next batch: 2 rows
// next batch: 1 rows

migrator.complete()
```

`SmallMigrator#complete` refuses to run while keys are still queued. For an in-place migration, it changes no table; for a shadow table, it swaps the tables.

:::note
The loop asks `QueueTable.pending` whether keys are left instead of stopping when `SmallMigrator#processBatch` returns `0`. The count is the number of rows written, which is `0` for a batch whose rows were deleted before it ran, even while more keys wait behind it.
:::

Every row now has its new columns filled in, Ada's from her new name:

```scala
transactor.connect {
  Frag
    .literal("SELECT id, full_name, first_name, last_name FROM customers ORDER BY id")
    .query[(Long, String, String, String)]
    .foreach(println)
}
// (1,Ada King,Ada,King)
// (2,Alan Turing,Alan,Turing)
// (3,Grace Brewster Hopper,Grace,Brewster Hopper)
// (4,Edsger Dijkstra,Edsger,Dijkstra)
```

### Drop the Old Column

The contract step removes `full_name`. Once it is gone, old code that writes `full_name` fails, so this step belongs to the start of the new version, after the last old process has stopped. A second `TinyMigrator` runs it, and the new code then reads `CustomerV2` rows:

```scala
class DropFullName(transactor: Transactor) extends TinyMigrator(transactor) {
  def run()(using DbTx): Unit =
    Frag.literal("ALTER TABLE customers DROP COLUMN full_name").update
}

DropFullName(transactor).migrate()

transactor.connect(customersV2.all.foreach(println))
// CustomerV2(1,Ada,King,ada@example.com)
// CustomerV2(2,Alan,Turing,alan@example.com)
// CustomerV2(3,Grace,Brewster Hopper,grace@example.com)
// CustomerV2(4,Edsger,Dijkstra,edsger@example.com)
```

`ALTER TABLE ... DROP COLUMN` needs SQLite 3.35 or newer; the `sqlite-jdbc` driver above bundles a newer one. The example deletes its database file at the end:

```scala
Files.deleteIfExists(dbFile)
```

<details>
  <summary>Runnable version, with all steps of this section in one program: <code>Step3SmallTableExample.scala</code></summary>

```scala title="sql-examples/src/main/scala/migratetablerowsonline/Step3SmallTableExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-data-migration:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
//> using file Customers.scala
// Run from this folder:    scala-cli run Step3SmallTableExample.scala
// Or from sql-examples:    sbt "runMain migratetablerowsonline.Step3SmallTableExample"

package migratetablerowsonline

/**
 * How to Migrate Table Rows to a New Version Online — Backfill a Small Table in
 * Place
 *
 * Adds the new columns to a SQLite table, backfills them with SmallMigrator
 * while the old code keeps writing, and drops the old column.
 */
object Step3SmallTableExample {
  def main(args: Array[String]): Unit = {
    val dbFile       = Files.createTempFile("customers", ".db")
    val transactor   = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
    given Transactor = transactor
    given Dialect    = Dialect.SQLite

    transactor.transact {
      createCustomersV1()
      customersV1.insertBatch(
        List(
          CustomerV1(1, "Ada Lovelace", "ada@example.com"),
          CustomerV1(2, "Alan Turing", "alan@example.com"),
          CustomerV1(3, "Grace Brewster Hopper", "grace@example.com")
        )
      )
    }

    AddNameColumns(transactor).migrate()

    QueueTable.create[Long]("customers_split_q", transactor)
    transactor.transact(QueueTable.enqueue("customers_split_q", customersV1.all.map(_.id)))

    val migrator = SmallMigrator[CustomerV1, CustomerV2, Long, Long](
      repoV1 = customersV1,
      repoV2 = customersV2,
      migration = splitName,
      queueTable = "customers_split_q",
      batchSize = 2,
      target = TargetStrategy.InPlace
    )

    migrator.init()
    println(s"first batch: ${migrator.processBatch()} rows")

    transactor.transact {
      customersV1.insert(CustomerV1(4, "Edsger Dijkstra", "edsger@example.com"))
      customersV1.update(CustomerV1(1, "Ada King", "ada@example.com"))
      QueueTable.enqueue("customers_split_q", List(4L, 1L))
    }

    while (transactor.connect(QueueTable.pending("customers_split_q")) > 0)
      println(s"next batch: ${migrator.processBatch()} rows")

    migrator.complete()

    transactor.connect {
      Frag
        .literal("SELECT id, full_name, first_name, last_name FROM customers ORDER BY id")
        .query[(Long, String, String, String)]
        .foreach(println)
    }

    class DropFullName(transactor: Transactor) extends TinyMigrator(transactor) {
      def run()(using DbTx): Unit =
        Frag.literal("ALTER TABLE customers DROP COLUMN full_name").update
    }

    DropFullName(transactor).migrate()

    transactor.connect(customersV2.all.foreach(println))

    Files.deleteIfExists(dbFile)
  }
}
```

</details>

## Migrate a Large Table Through a Shadow Table

When the new shape can't live beside the old one, we copy the rows into a new table and swap it in. In `CustomerV2`, `full_name` is gone and both name columns are `NOT NULL`, so the new table is `customers_v2`, built from scratch. `LargeMigrator` copies `customers` into it while the application keeps writing, uses capture triggers to queue every concurrent write, and swaps the tables with a short cutover: `LargeMigrator#fence`, `LargeMigrator#drain` and `LargeMigrator#complete`. Shadow tables need PostgreSQL; SQLite can't create them.

The overall flow looks like this:

```
 old app ──writes──► customers ──trigger──► customers_split_q ◄── INSERT ... SELECT (existing rows)
                         │                         │
                         │ read by key             │ claim a batch (SKIP LOCKED)
                         ▼                         ▼
                    LargeMigrator: splitName ──writes──► customers_v2
                                                           │
 cutover: stop writers → fence() → drain() → complete()    │ rename: customers → customers_old_v2,
                                                           ▼         customers_v2 → customers
 new app ──reads and writes──► customers (CustomerV2 shape)
```

The code in this section needs a running PostgreSQL server, so the page compiles it but doesn't run it. Each step is a method, and the output at the end of the section is from running them against PostgreSQL 16.

### Create a Large Table

The `customers` table gets 20,000 generated customers. `resetShop()` drops everything the migration leaves behind, so the example can run again:

```scala
def resetShop()(using DbCon): Unit = {
  Frag.literal("DROP TABLE IF EXISTS customers, customers_v2, customers_old_v2, customers_split_q CASCADE").update
  Frag.literal("DROP FUNCTION IF EXISTS customers_split_q_notify() CASCADE").update
}

val firstNames = Vector("Ada", "Alan", "Grace", "Edsger", "Barbara", "Donald", "Margaret", "Ken")
val lastNames  = Vector("Lovelace", "Turing", "Hopper", "Dijkstra", "Liskov", "Knuth", "Hamilton", "Thompson")

def sampleCustomer(id: Long): CustomerV1 =
  CustomerV1(id, s"${firstNames(((id - 1) % 8).toInt)} ${lastNames(((id - 1) / 8 % 8).toInt)}", s"customer$id@example.com")

def createLargeTable()(using transactor: Transactor): Unit =
  transactor.transact {
    resetShop()
    createCustomersV1()
    customersV1.insertBatch((1L to 20000L).map(sampleCustomer))
  }
```

### Prepare the Shadow Table and the Queue

We create `customers_v2` with the new columns and the queue table, then create the migrator and call `LargeMigrator#init`, which installs the capture triggers on `customers`. The triggers queue only the writes that happen from now on, so we then queue the key of every existing row:

```scala
def prepareShadowCopy()(using
  transactor: Transactor,
  dialect: Dialect
): LargeMigrator[CustomerV1, CustomerV2, Long, Long] = {
  transactor.transact {
    Frag
      .literal(
        "CREATE TABLE customers_v2 (id BIGINT PRIMARY KEY, first_name TEXT NOT NULL, last_name TEXT NOT NULL, email TEXT NOT NULL)"
      )
      .update
  }
  QueueTable.create[Long]("customers_split_q", transactor)

  val migrator = LargeMigrator[CustomerV1, CustomerV2, Long, Long](
    repoV1 = customersV1,
    repoV2 = customersV2,
    migration = splitName,
    queueTable = "customers_split_q",
    batchSize = 500,
    target = TargetStrategy.ShadowTable("v2"),
    captureTriggers = true
  )

  migrator.init()

  val queued = transactor.transact {
    Frag
      .literal("INSERT INTO customers_split_q (id) SELECT CAST(id AS TEXT) FROM customers ON CONFLICT DO NOTHING")
      .update
  }
  println(s"queued $queued existing customers")

  migrator
}
```

The order matters. Installing the triggers before queuing the existing rows means that no write can fall between the two: a row written in between is queued by its trigger, and `ON CONFLICT DO NOTHING` skips its key when the `INSERT ... SELECT` reaches it. We queue the existing rows with one SQL statement, because `QueueTable.enqueue` takes the keys as a Scala collection, and loading the keys of a large table into memory first is what we want to avoid. The queue stores keys as text, hence the `CAST`.

### Copy the Rows While the Application Writes

`LargeMigrator#run` claims batches and converts them until a batch writes no rows, then returns the number of rows it wrote. To show writes arriving during the copy, a second thread plays the old application: it signs up 100 new customers and changes the email address of 100 existing ones while the copy runs:

```scala
def copyWhileTheAppWrites(migrator: LargeMigrator[CustomerV1, CustomerV2, Long, Long])(using
  transactor: Transactor
): Unit = {
  val writer = new Thread(() =>
    for (i <- 1L to 100L) {
      transactor.transact {
        customersV1.insert(sampleCustomer(20000L + i))
        customersV1.update(sampleCustomer(i).copy(email = s"customer$i@new-domain.example"))
      }
      Thread.sleep(5)
    }
  )
  writer.start()

  val copied = migrator.run()
  println(s"first pass copied $copied rows")

  writer.join()
  println(s"writer stopped with ${migrator.pendingCount} keys still queued")
}
```

Each write fires the trigger, which queues the row's key in the writer's transaction. An update to a row that was already copied queues it again, and the migrator copies it again over the earlier copy, so the first pass writes more rows than the table holds. A write that lands after `LargeMigrator#run` returns waits in the queue for the cutover. `LargeMigrator#pendingCount` counts the keys still queued.

`LargeMigrator#run` returning doesn't prove the queue is empty: in 0.0.56 it also returns when a batch holds only keys of rows that were deleted in the meantime (a known bug). The cutover below doesn't rely on it, because `LargeMigrator#drain` checks the queue with a count.

### Cut Over to the New Table

Cutover is the one moment the application has to cooperate. We stop every writer to `customers` (in a real application: put it into a read-only mode, or stop the old instances), then fence, drain and complete:

```scala
def cutOver(migrator: LargeMigrator[CustomerV1, CustomerV2, Long, Long])(using transactor: Transactor): Unit = {
  migrator.fence()
  transactor.transact(customersV1.insert(CustomerV1(30001L, "Frances Allen", "frances@example.com")))

  val drained = migrator.drain()
  println(s"drain() applied $drained more rows")

  require(migrator.pendingCount == 0, "a write reached the queue after drain(); stop all writers first")
  migrator.complete()
}
```

`LargeMigrator#fence` only moves the migrator into its cutover state; it doesn't block writes. The insert of Frances Allen after it shows that: the trigger still queues her key, and `LargeMigrator#drain`, which converts batches until a count finds the queue empty, copies her row. `LargeMigrator#complete` then renames `customers` to `customers_old_v2` and `customers_v2` to `customers` in one transaction.

:::danger
A write that lands after `LargeMigrator#drain` returns and before `LargeMigrator#complete` runs is lost from the new table. `LargeMigrator#complete` doesn't check the queue in 0.0.56 (a known bug; `SmallMigrator#complete` does), so it swaps the tables anyway, and the row exists only in `customers_old_v2`. Stop every writer before `LargeMigrator#fence`, and check `LargeMigrator#pendingCount` right before `LargeMigrator#complete`, as `cutOver` does.
:::

### Check the Result and Clean Up

After the swap, `customers` has the new shape, and the old code can't write to it anymore. We count the rows in both tables, look up a row from each kind of write, and try an old-style insert:

```scala
def checkTheSwap()(using transactor: Transactor): Unit = {
  transactor.connect {
    val oldCount = Frag.literal("SELECT COUNT(*) FROM customers_old_v2").queryOne[Long].getOrElse(0L)
    println(s"customers: ${customersV2.count} rows, customers_old_v2: $oldCount rows")
    println(s"updated during the copy: ${customersV2.find(1L)}")
    println(s"inserted during the copy: ${customersV2.find(20001L)}")
    println(s"inserted after fence(): ${customersV2.find(30001L)}")
  }

  try transactor.transact(customersV1.insert(CustomerV1(30002L, "John Backus", "john@example.com")))
  catch { case e: java.sql.SQLException => println(s"old code after the swap: ${e.getMessage.linesIterator.next()}") }
}
```

The migrator leaves the old table, the queue table and the trigger function behind, so that we can compare the tables before we drop anything. Renaming `customers` moved its trigger to `customers_old_v2`, so dropping that table drops the trigger too:

```scala
def dropMigrationLeftovers()(using transactor: Transactor): Unit =
  transactor.transact {
    Frag.literal("DROP TABLE customers_old_v2").update
    Frag.literal("DROP TABLE customers_split_q").update
    Frag.literal("DROP FUNCTION customers_split_q_notify()").update
  }
```

One method runs the steps in order against the `shop` database:

```scala
def migrateLargeTable(): Unit = {
  given Transactor = shopTransactor()
  given Dialect    = Dialect.Postgres

  createLargeTable()
  val migrator = prepareShadowCopy()
  copyWhileTheAppWrites(migrator)
  cutOver(migrator)
  checkTheSwap()
  dropMigrationLeftovers()
}
```

Running `migrateLargeTable()` against PostgreSQL 16 prints the following. The second to fourth lines vary from run to run, because they depend on how the writer thread and the copy interleave:

```text
queued 20000 existing customers
first pass copied 20102 rows
writer stopped with 0 keys still queued
drain() applied 1 more rows
customers: 20101 rows, customers_old_v2: 20101 rows
updated during the copy: CustomerV2(1,Ada,Lovelace,customer1@new-domain.example)
inserted during the copy: CustomerV2(20001,Ada,Liskov,customer20001@example.com)
inserted after fence(): CustomerV2(30001,Frances,Allen,frances@example.com)
old code after the swap: ERROR: column "full_name" of relation "customers" does not exist
```

Both tables hold the 20,000 original rows, the 100 sign-ups and Frances Allen. Customer 1 has the email address the writer gave them during the copy, and the old code's insert fails with a clear error instead of writing a row in the old shape. Deploy the code that reads and writes `CustomerV2` as part of the cutover.

<details>
  <summary>Runnable version, with the methods and a <code>main</code> that calls <code>migrateLargeTable()</code>: <code>Step4LargeTableExample.scala</code></summary>

```scala title="sql-examples/src/main/scala/migratetablerowsonline/Step4LargeTableExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-data-migration:0.0.56
//> using dep org.postgresql:postgresql:42.7.14
//> using file Customers.scala
//> using file ShopDatabase.scala
// Needs PostgreSQL 14 or newer, see the guide's Prerequisites for a Docker command.
// Run from this folder:    scala-cli run Step4LargeTableExample.scala
// Or from sql-examples:    sbt "runMain migratetablerowsonline.Step4LargeTableExample"

package migratetablerowsonline

/**
 * How to Migrate Table Rows to a New Version Online — Migrate a Large Table
 * Through a Shadow Table
 *
 * Copies 20,000 PostgreSQL rows into a shadow table with LargeMigrator while
 * another thread keeps writing, then swaps the shadow table in.
 */
object Step4LargeTableExample {

  def createLargeTable()(using transactor: Transactor): Unit =
    transactor.transact {
      resetShop()
      createCustomersV1()
      customersV1.insertBatch((1L to 20000L).map(sampleCustomer))
    }

  def prepareShadowCopy()(using
    transactor: Transactor,
    dialect: Dialect
  ): LargeMigrator[CustomerV1, CustomerV2, Long, Long] = {
    transactor.transact {
      Frag
        .literal(
          "CREATE TABLE customers_v2 (id BIGINT PRIMARY KEY, first_name TEXT NOT NULL, last_name TEXT NOT NULL, email TEXT NOT NULL)"
        )
        .update
    }
    QueueTable.create[Long]("customers_split_q", transactor)

    val migrator = LargeMigrator[CustomerV1, CustomerV2, Long, Long](
      repoV1 = customersV1,
      repoV2 = customersV2,
      migration = splitName,
      queueTable = "customers_split_q",
      batchSize = 500,
      target = TargetStrategy.ShadowTable("v2"),
      captureTriggers = true
    )

    migrator.init()

    val queued = transactor.transact {
      Frag
        .literal("INSERT INTO customers_split_q (id) SELECT CAST(id AS TEXT) FROM customers ON CONFLICT DO NOTHING")
        .update
    }
    println(s"queued $queued existing customers")

    migrator
  }

  def copyWhileTheAppWrites(migrator: LargeMigrator[CustomerV1, CustomerV2, Long, Long])(using
    transactor: Transactor
  ): Unit = {
    val writer = new Thread(() =>
      for (i <- 1L to 100L) {
        transactor.transact {
          customersV1.insert(sampleCustomer(20000L + i))
          customersV1.update(sampleCustomer(i).copy(email = s"customer$i@new-domain.example"))
        }
        Thread.sleep(5)
      }
    )
    writer.start()

    val copied = migrator.run()
    println(s"first pass copied $copied rows")

    writer.join()
    println(s"writer stopped with ${migrator.pendingCount} keys still queued")
  }

  def cutOver(migrator: LargeMigrator[CustomerV1, CustomerV2, Long, Long])(using transactor: Transactor): Unit = {
    migrator.fence()
    transactor.transact(customersV1.insert(CustomerV1(30001L, "Frances Allen", "frances@example.com")))

    val drained = migrator.drain()
    println(s"drain() applied $drained more rows")

    require(migrator.pendingCount == 0, "a write reached the queue after drain(); stop all writers first")
    migrator.complete()
  }

  def checkTheSwap()(using transactor: Transactor): Unit = {
    transactor.connect {
      val oldCount = Frag.literal("SELECT COUNT(*) FROM customers_old_v2").queryOne[Long].getOrElse(0L)
      println(s"customers: ${customersV2.count} rows, customers_old_v2: $oldCount rows")
      println(s"updated during the copy: ${customersV2.find(1L)}")
      println(s"inserted during the copy: ${customersV2.find(20001L)}")
      println(s"inserted after fence(): ${customersV2.find(30001L)}")
    }

    try transactor.transact(customersV1.insert(CustomerV1(30002L, "John Backus", "john@example.com")))
    catch { case e: java.sql.SQLException => println(s"old code after the swap: ${e.getMessage.linesIterator.next()}") }
  }

  def dropMigrationLeftovers()(using transactor: Transactor): Unit =
    transactor.transact {
      Frag.literal("DROP TABLE customers_old_v2").update
      Frag.literal("DROP TABLE customers_split_q").update
      Frag.literal("DROP FUNCTION customers_split_q_notify()").update
    }

  def migrateLargeTable(): Unit = {
    given Transactor = shopTransactor()
    given Dialect    = Dialect.Postgres

    createLargeTable()
    val migrator = prepareShadowCopy()
    copyWhileTheAppWrites(migrator)
    cutOver(migrator)
    checkTheSwap()
    dropMigrationLeftovers()
  }

  def main(args: Array[String]): Unit = migrateLargeTable()
}
```

</details>

## Recover from a Failed Batch and Restart

Production data has rows that the migration didn't expect. When the migration returns a `Left` for a row, the migrator throws, its transaction rolls back, and the keys of the whole batch stay in the queue. There is no automatic retry and no dead-letter table: we fix the cause and run again. A customer named "Cher" shows how that looks. We start a fresh SQLite database with her between two other customers:

```scala
val dbFile     = Files.createTempFile("customers", ".db")
val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
given Transactor = transactor
given Dialect    = Dialect.SQLite

transactor.transact {
  createCustomersV1()
  customersV1.insertBatch(
    List(
      CustomerV1(1, "Ada Lovelace", "ada@example.com"),
      CustomerV1(2, "Cher", "cher@example.com"),
      CustomerV1(3, "Alan Turing", "alan@example.com")
    )
  )
}
AddNameColumns(transactor).migrate()
QueueTable.create[Long]("customers_split_q", transactor)
transactor.transact(QueueTable.enqueue("customers_split_q", customersV1.all.map(_.id)))
```

We migrate with `splitName` again, two keys per batch:

```scala
val firstTry = SmallMigrator[CustomerV1, CustomerV2, Long, Long](
  repoV1 = customersV1,
  repoV2 = customersV2,
  migration = splitName,
  queueTable = "customers_split_q",
  batchSize = 2,
  target = TargetStrategy.InPlace
)
```

The first batch holds Ada and Cher. "Cher" has no space, so `firstSpace` is `-1` and the substring fails:

```scala
firstTry.init()
try println(s"batch: ${firstTry.processBatch()} rows")
catch { case e: RuntimeException => println(s"batch failed: ${e.getMessage}") }
// batch failed: Migration failed: zio.blocks.schema.SchemaError: Substring indices must be non-negative: start=0, end=-1

transactor.connect {
  val migrated =
    Frag.literal("SELECT COUNT(*) FROM customers WHERE first_name IS NOT NULL").queryOne[Long].getOrElse(0L)
  println(s"rows migrated: $migrated, keys still queued: ${QueueTable.pending("customers_split_q")}")
}
// rows migrated: 0, keys still queued: 3
```

Ada's row, which converted fine, rolled back with Cher's, and all three keys are still queued. Calling `SmallMigrator#processBatch` again would claim the same two keys and fail the same way, because the queue hands out keys in key order.

The fix is a migration that handles single-word names. Appending a space to the name before searching for it means there is always a space to find; a single-word name then gets an empty `lastName`:

```scala
val paddedName  = SchemaExpr.stringConcat(fullName, SchemaExpr.literal(" "))
val paddedSpace = SchemaExpr.stringIndexOf(paddedName, SchemaExpr.literal(" "))

val splitNameSafely: Migration[CustomerV1, CustomerV2] =
  Migration
    .newBuilder[CustomerV1, CustomerV2]
    .addField(_.firstName, SchemaExpr.stringSubstring(paddedName, SchemaExpr.literal(0), paddedSpace))
    .addField(
      _.lastName,
      SchemaExpr.stringTrim(SchemaExpr.stringSubstring(paddedName, paddedSpace, SchemaExpr.stringLength(paddedName)))
    )
    .dropField(_.fullName, SchemaExpr.literal(""))
    .build
```

A new migrator with the fixed migration picks up the same queue:

```scala
val secondTry = SmallMigrator[CustomerV1, CustomerV2, Long, Long](
  repoV1 = customersV1,
  repoV2 = customersV2,
  migration = splitNameSafely,
  queueTable = "customers_split_q",
  batchSize = 2,
  target = TargetStrategy.InPlace
)
```

It converts the three queued rows and finishes the job:

```scala
secondTry.init()
while (transactor.connect(QueueTable.pending("customers_split_q")) > 0)
  println(s"batch: ${secondTry.processBatch()} rows")
// batch: 2 rows
// batch: 1 rows
secondTry.complete()

transactor.connect(customersV2.all.foreach(println))
// CustomerV2(1,Ada,Lovelace,ada@example.com)
// CustomerV2(2,Cher,,cher@example.com)
// CustomerV2(3,Alan,Turing,alan@example.com)
```

As before, the example deletes its database file at the end:

```scala
Files.deleteIfExists(dbFile)
```

The same holds when the process itself dies mid-migration. The queue is a table, and a batch claims its keys and writes its rows in one transaction, so a crash leaves the keys of the unfinished batch queued and the finished batches written. After the restart, we create a new migrator with the same arguments and call `LargeMigrator#init` again: it creates nothing twice, because the shadow table's DDL uses `CREATE TABLE IF NOT EXISTS` and the triggers `CREATE OR REPLACE TRIGGER`. A shadow table makes reruns safe, too: a row copied twice is updated, not inserted again. A `LargeMigrator` keeps its cutover state in memory, so a new one starts before `LargeMigrator#fence` and needs its own `LargeMigrator#fence`, `LargeMigrator#drain` and `LargeMigrator#complete`. To stop a `LargeMigrator` on purpose, call `LargeMigrator#pause` from another thread: `LargeMigrator#run` finishes its current batch and returns, and `LargeMigrator#resume` lets the next `LargeMigrator#run` continue.

<details>
  <summary>Runnable version, with the failing run and the fixed run in one program: <code>Step5FailureAndRestartExample.scala</code></summary>

```scala title="sql-examples/src/main/scala/migratetablerowsonline/Step5FailureAndRestartExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-data-migration:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
//> using file Customers.scala
// Run from this folder:    scala-cli run Step5FailureAndRestartExample.scala
// Or from sql-examples:    sbt "runMain migratetablerowsonline.Step5FailureAndRestartExample"

package migratetablerowsonline

/**
 * How to Migrate Table Rows to a New Version Online — Recover from a Failed
 * Batch
 *
 * Lets a single-word name fail a batch, shows that the batch rolled back and
 * its keys stayed queued, and finishes the migration with a fixed migration.
 */
object Step5FailureAndRestartExample {
  def main(args: Array[String]): Unit = {
    val dbFile       = Files.createTempFile("customers", ".db")
    val transactor   = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
    given Transactor = transactor
    given Dialect    = Dialect.SQLite

    transactor.transact {
      createCustomersV1()
      customersV1.insertBatch(
        List(
          CustomerV1(1, "Ada Lovelace", "ada@example.com"),
          CustomerV1(2, "Cher", "cher@example.com"),
          CustomerV1(3, "Alan Turing", "alan@example.com")
        )
      )
    }
    AddNameColumns(transactor).migrate()
    QueueTable.create[Long]("customers_split_q", transactor)
    transactor.transact(QueueTable.enqueue("customers_split_q", customersV1.all.map(_.id)))

    val firstTry = SmallMigrator[CustomerV1, CustomerV2, Long, Long](
      repoV1 = customersV1,
      repoV2 = customersV2,
      migration = splitName,
      queueTable = "customers_split_q",
      batchSize = 2,
      target = TargetStrategy.InPlace
    )

    firstTry.init()
    try println(s"batch: ${firstTry.processBatch()} rows")
    catch { case e: RuntimeException => println(s"batch failed: ${e.getMessage}") }

    transactor.connect {
      val migrated =
        Frag.literal("SELECT COUNT(*) FROM customers WHERE first_name IS NOT NULL").queryOne[Long].getOrElse(0L)
      println(s"rows migrated: $migrated, keys still queued: ${QueueTable.pending("customers_split_q")}")
    }

    val secondTry = SmallMigrator[CustomerV1, CustomerV2, Long, Long](
      repoV1 = customersV1,
      repoV2 = customersV2,
      migration = splitNameSafely,
      queueTable = "customers_split_q",
      batchSize = 2,
      target = TargetStrategy.InPlace
    )

    secondTry.init()
    while (transactor.connect(QueueTable.pending("customers_split_q")) > 0)
      println(s"batch: ${secondTry.processBatch()} rows")
    secondTry.complete()

    transactor.connect(customersV2.all.foreach(println))

    Files.deleteIfExists(dbFile)
  }
}
```

</details>

## Choose an Execution Model

The module has three migrators. The size of the table matters less than two other questions: does the change rewrite rows at all, and can the new columns sit beside the old ones until the old code is gone?

| Your change                                     | Use                                                 | Database              |
| ----------------------------------------------- | --------------------------------------------------- | --------------------- |
| Schema only, such as a nullable column or index | `TinyMigrator`                                      | SQLite and PostgreSQL |
| New columns beside the old ones                 | `SmallMigrator` with `TargetStrategy.InPlace`       | SQLite and PostgreSQL |
| A new shape that replaces the old one           | `LargeMigrator` with `TargetStrategy.ShadowTable`   | PostgreSQL only       |

`TinyMigrator` runs your DDL in one transaction, usually at startup, and converts no rows. `SmallMigrator` converts rows in batches that you drive with `SmallMigrator#processBatch`, and `SmallMigrator#complete` refuses to run while keys are queued; in place, the application queues its own writes. `LargeMigrator` adds trigger-captured writes (`captureTriggers = true`), `LargeMigrator#run` with `LargeMigrator#pause` and `LargeMigrator#resume`, the `LargeMigrator#fence`, `LargeMigrator#drain` and `LargeMigrator#complete` cutover, and several workers on one PostgreSQL queue.

A few combinations fall between the rows. `SmallMigrator` also takes `TargetStrategy.ShadowTable` on PostgreSQL, when you want a shadow table but drive the batches yourself. `LargeMigrator` also takes `TargetStrategy.InPlace`, when a big in-place backfill should run with `LargeMigrator#run`, `LargeMigrator#pause` and `LargeMigrator#resume`; without triggers, the application then queues its own writes, as in the small-table section. On SQLite, only in-place migrations work, with one worker.

## Putting It Together

The complete example is the large-table migration from [Migrate a Large Table Through a Shadow Table](#migrate-a-large-table-through-a-shadow-table) as one program, with one change: it uses `splitNameSafely` from [Recover from a Failed Batch and Restart](#recover-from-a-failed-batch-and-restart), so a single-word name can't stop it:

```scala title="sql-examples/src/main/scala/migratetablerowsonline/CompleteExample.scala"
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-next-data-migration:0.0.56
//> using dep org.postgresql:postgresql:42.7.14
//> using file Customers.scala
//> using file ShopDatabase.scala
// Needs PostgreSQL 14 or newer, see the guide's Prerequisites for a Docker command.
// Run from this folder:    scala-cli run CompleteExample.scala
// Or from sql-examples:    sbt "runMain migratetablerowsonline.CompleteExample"

package migratetablerowsonline

/**
 * How to Migrate Table Rows to a New Version Online — Complete Example
 *
 * Splits the full name of 20,000 PostgreSQL customers into first and last names
 * with LargeMigrator while another thread keeps writing, using the migration
 * that also handles single-word names.
 */
object CompleteExample {
  def main(args: Array[String]): Unit = {
    val transactor   = shopTransactor()
    given Transactor = transactor
    given Dialect    = Dialect.Postgres

    transactor.transact {
      resetShop()
      createCustomersV1()
      customersV1.insertBatch((1L to 20000L).map(sampleCustomer))
    }

    transactor.transact {
      Frag
        .literal(
          "CREATE TABLE customers_v2 (id BIGINT PRIMARY KEY, first_name TEXT NOT NULL, last_name TEXT NOT NULL, email TEXT NOT NULL)"
        )
        .update
    }
    QueueTable.create[Long]("customers_split_q", transactor)

    val migrator = LargeMigrator[CustomerV1, CustomerV2, Long, Long](
      repoV1 = customersV1,
      repoV2 = customersV2,
      migration = splitNameSafely,
      queueTable = "customers_split_q",
      batchSize = 500,
      target = TargetStrategy.ShadowTable("v2"),
      captureTriggers = true
    )

    migrator.init()

    val queued = transactor.transact {
      Frag
        .literal("INSERT INTO customers_split_q (id) SELECT CAST(id AS TEXT) FROM customers ON CONFLICT DO NOTHING")
        .update
    }
    println(s"queued $queued existing customers")

    val writer = new Thread(() =>
      for (i <- 1L to 100L) {
        transactor.transact {
          customersV1.insert(sampleCustomer(20000L + i))
          customersV1.update(sampleCustomer(i).copy(email = s"customer$i@new-domain.example"))
        }
        Thread.sleep(5)
      }
    )
    writer.start()

    val copied = migrator.run()
    println(s"first pass copied $copied rows")

    writer.join()
    println(s"writer stopped with ${migrator.pendingCount} keys still queued")

    migrator.fence()
    transactor.transact(customersV1.insert(CustomerV1(30001L, "Frances Allen", "frances@example.com")))

    val drained = migrator.drain()
    println(s"drain() applied $drained more rows")

    require(migrator.pendingCount == 0, "a write reached the queue after drain(); stop all writers first")
    migrator.complete()

    transactor.connect {
      val oldCount = Frag.literal("SELECT COUNT(*) FROM customers_old_v2").queryOne[Long].getOrElse(0L)
      println(s"customers: ${customersV2.count} rows, customers_old_v2: $oldCount rows")
      println(s"updated during the copy: ${customersV2.find(1L)}")
      println(s"inserted during the copy: ${customersV2.find(20001L)}")
      println(s"inserted after fence(): ${customersV2.find(30001L)}")
    }

    try transactor.transact(customersV1.insert(CustomerV1(30002L, "John Backus", "john@example.com")))
    catch { case e: java.sql.SQLException => println(s"old code after the swap: ${e.getMessage.linesIterator.next()}") }

    transactor.transact {
      Frag.literal("DROP TABLE customers_old_v2").update
      Frag.literal("DROP TABLE customers_split_q").update
      Frag.literal("DROP FUNCTION customers_split_q_notify()").update
    }
  }
}
```

Running it against PostgreSQL 16 prints the same lines as the large-table section. The second to fourth lines vary from run to run:

```text
queued 20000 existing customers
first pass copied 20102 rows
writer stopped with 0 keys still queued
drain() applied 1 more rows
customers: 20101 rows, customers_old_v2: 20101 rows
updated during the copy: CustomerV2(1,Ada,Lovelace,customer1@new-domain.example)
inserted during the copy: CustomerV2(20001,Ada,Liskov,customer20001@example.com)
inserted after fence(): CustomerV2(30001,Frances,Allen,frances@example.com)
old code after the swap: ERROR: column "full_name" of relation "customers" does not exist
```

## Running the Examples

All runnable code from this guide is in [`sql-examples/src/main/scala/migratetablerowsonline/`](https://github.com/zio/zio-blocks/tree/main/sql-examples/src/main/scala/migratetablerowsonline). Run any file from that folder with `scala-cli run <File>.scala`, or from `sql-examples` with `sbt "runMain migratetablerowsonline.<ObjectName>"`. `Step4LargeTableExample` and `CompleteExample` need the PostgreSQL server from [Prerequisites](#prerequisites); stop it afterwards with `docker stop shop-db`.

## Where to Go Next

- [Data Migration](../reference/data-migration.md) is the reference for the three migrators, the target strategies and the queue primitives.
- [Migration](../reference/schema/migration.md) lists every builder operation, and [SchemaExpr](../reference/schema/schema-expr.md) every expression you can compute a new field with.
- [How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md) builds the transactor on a HikariCP pool, which a migration running for hours next to the application should use.
- [How to Test Database Code Against SQLite](./test-database-code-against-sqlite.md) shows how to test code like the in-place backfill in a unit test.

---

## Navigate and Update Nested Data with Optics

Welcome! This tutorial is for Scala developers who have finished [Your First Schema](./your-first-schema.md), so you know how to derive a `Schema` for a case class. You don't need any prior experience with optics, or with optics libraries such as Monocle or Quicklens, to follow along.

## 1. Introduction

By the end of this tutorial, you will understand:

- What an optic is, and how to declare one from a path with the `$(...)` macro
- How a `Lens` reads, replaces and modifies one field, however deeply it is nested
- How `.when[B]`, `.each` and `.atKey(k)` reach one case of a sealed trait, every element of a list and one entry of a map
- How `modifyOrFail` explains why an update found nothing to change
- How the same paths look as strings, with the `p"..."` interpolator

We learn these concepts by managing the stock of a small bookstore, one step at a time:

1. Put a lens on one field
2. Follow a path into nested data
3. Focus on one case with `.when[B]`
4. Update every element with `.each`
5. Reach into a map with `.atKey`
6. See why an update failed with `modifyOrFail`
7. Write the same paths as strings with `p"..."`

We recommend reading from top to bottom — each section builds on the previous one.

To follow along, add the schema module to your `build.sbt`. Optics are part of it, so there is nothing else to install:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56"
```

The code in this tutorial is the same for Scala 2.13 and Scala 3.

## 2. Background: Paths Into Immutable Data

Case classes are immutable, so "changing" a field means building a new value with that field replaced. For a field at the top level, `copy` does the job. For a field two or three levels down, every level needs its own `copy`, and a field inside every element of a list needs a `map` around the `copy`. The path to the field — "the author's name of this book", "the stock of every book on every shelf" — gets lost in the plumbing.

An optic turns that path into a value. You describe the path once, for example "the `name` of the `author` of a `Book`", and the optic gives you operations to read the value at the end of the path and to build an updated copy of the whole structure. ZIO Blocks builds optics from your `Schema`, so each optic also knows the structure it walks through. That is how it can tell you, in words, why an update found nothing to change.

## 3. Put a Lens on One Field

A `Lens[S, A]` focuses on exactly one field of type `A` inside a value of type `S`. Because a case class always has every one of its fields, a lens always finds its target.

We declare optics with the `$(...)` macro, which comes from the `CompanionOptics[S]` trait. We extend it in the companion object, next to the schema, and pass `$` a lambda that selects a path. Here is the model for one book, with two lenses. The page counts, file sizes and stock levels in this tutorial are sample data:

```scala

case class Author(name: String)

sealed trait Format
case class Paperback(pages: Int)  extends Format
case class Ebook(sizeMb: Double) extends Format

case class Book(title: String, author: Author, format: Format, stock: Int)

object Book extends CompanionOptics[Book] {
  implicit val schema: Schema[Book] = Schema.derived

  val title: Lens[Book, String]      = $(_.title)
  val authorName: Lens[Book, String] = $(_.author.name)
}
```

The code above:

- `import zio.blocks.schema._` brings `Schema`, `CompanionOptics` and the optic types into scope.
- `Author`, `Format` and `Book` are ordinary case classes and a sealed trait; `author` holds the book's first author.
- `object Book extends CompanionOptics[Book]` gives the companion object the `$` macro for paths that start at a `Book`.
- `implicit val schema` comes first, because the macro uses the schema to build each optic.
- `$(_.title)` reads as "from a book, go to `title`" and returns a `Lens[Book, String]`.
- `$(_.author.name)` is a longer path; we use it in the next section.

Let's read the title of a book with `Lens#get`:

```scala
val programmingInScala = Book("Programming in Scala", Author("Martin Odersky"), Paperback(860), 7)
// programmingInScala: Book = Book(
//   title = "Programming in Scala",
//   author = Author("Martin Odersky"),
//   format = Paperback(860),
//   stock = 7
// )

println(Book.title.get(programmingInScala))
// Programming in Scala
```

The code above builds a `Book` and passes it to `Book.title.get`, which returns the value at the end of the lens's path.

To update the field, we have two methods. `Lens#replace` puts a new value in place, and `Lens#modify` applies a function to the current value. Both return a new `Book`:

```scala
println(Book.title.replace(programmingInScala, "Programming in Scala, Fifth Edition"))
// Book(Programming in Scala, Fifth Edition,Author(Martin Odersky),Paperback(860),7)

println(Book.title.modify(programmingInScala, _.toUpperCase))
// Book(PROGRAMMING IN SCALA,Author(Martin Odersky),Paperback(860),7)

println(programmingInScala)
// Book(Programming in Scala,Author(Martin Odersky),Paperback(860),7)
```

The code above:

- `Book.title.replace(programmingInScala, "...")` returns a copy of the book with the new title.
- `Book.title.modify(programmingInScala, _.toUpperCase)` returns a copy whose title is the old title in upper case.
- `println(programmingInScala)` shows that the original book did not change: optics never mutate, they build new values.

An optic is an ordinary value, so we can print it too:

```scala
println(Book.title)
// Lens(_.title)
```

Notice that the printed form matches the path we passed to `$`. When a program holds many optics, this makes it easy to see which one is which.

## 4. Follow a Path Into Nested Data

A path can go through as many fields as we like. Each `.field` step goes one level deeper, and the result is still a `Lens`, because every step always finds its field. The `Book.authorName` lens from the previous section goes from a `Book` to its `author`, then to the author's `name`:

```scala
println(Book.authorName.get(programmingInScala))
// Martin Odersky
```

The code above follows both steps of the path and returns the name of the book's first author.

Now let's update the name and compare the result with the same update written by hand:

```scala
val shouting = Book.authorName.modify(programmingInScala, _.toUpperCase)

val byHand = programmingInScala.copy(author = programmingInScala.author.copy(name = programmingInScala.author.name.toUpperCase))
```

```scala
println(shouting)
// Book(Programming in Scala,Author(MARTIN ODERSKY),Paperback(860),7)
println(shouting == byHand)
// true
```

The code above:

- `Book.authorName.modify(...)` builds a new `Author` with the upper-case name, then a new `Book` that holds it.
- `byHand` does the same with two nested `copy` calls; notice how the path `author.name` is spread across the whole expression.
- `shouting == byHand` confirms that both give the same book.

The optic only rebuilds the values on its path. Everything off the path is shared with the original:

```scala
println(shouting.author eq programmingInScala.author)
// false
println(shouting.format eq programmingInScala.format)
// true
```

The first line prints `false`, because the `Author` is on the path and was rebuilt. The second prints `true`, because `format` is not on the path, so the new book points at the very same `Paperback` object.

## 5. Focus on One Case with `.when[B]`

Some books are paperbacks and some are ebooks. A path step `.when[B]` selects one case of a sealed trait. The value might be a different case, so the path no longer always finds its target: `$(_.format.when[Paperback].pages)` gives an `Optional[Book, Int]`, an optic that focuses on zero or one value.

Let's add this optic to the companion object. We start a fresh scope so we can redefine `Book`:

```scala

case class Author(name: String)

sealed trait Format
case class Paperback(pages: Int)  extends Format
case class Ebook(sizeMb: Double) extends Format

case class Book(title: String, author: Author, format: Format, stock: Int)

object Book extends CompanionOptics[Book] {
  implicit val schema: Schema[Book] = Schema.derived

  val title: Lens[Book, String]          = $(_.title)
  val authorName: Lens[Book, String]     = $(_.author.name)
  val paperbackPages: Optional[Book, Int] = $(_.format.when[Paperback].pages)
}

val programmingInScala = Book("Programming in Scala", Author("Martin Odersky"), Paperback(860), 7)

val zionomicon = Book("Zionomicon", Author("John A. De Goes"), Ebook(5.2), 3)
```

The code above reads `$(_.format.when[Paperback].pages)` as "the format, if it is a `Paperback`, then its page count", and defines `programmingInScala` as a paperback and `zionomicon` as an ebook in our sample data.

:::caution
Path steps such as `.when[B]`, `.each` and `.atKey(k)` exist only inside an object that extends `CompanionOptics`. Declare optics that use them in the companion object, as we do here. Outside it, the compiler reports an error such as ``value each is not a member of List[Shelf]``.
:::

An `Optional` reads with `Optional#getOption`, which returns `Some` when the target is there and `None` when it isn't:

```scala
println(Book.paperbackPages.getOption(programmingInScala))
// Some(860)
println(Book.paperbackPages.getOption(zionomicon))
// None
```

The code above returns the page count of the paperback in a `Some`, and `None` for the ebook.

Updates follow the same rule. `modify` changes the value when the target is there, and returns the input unchanged when it isn't:

```scala
println(Book.paperbackPages.modify(programmingInScala, _ + 16))
// Book(Programming in Scala,Author(Martin Odersky),Paperback(876),7)

println(Book.paperbackPages.modify(zionomicon, _ + 16) == zionomicon)
// true
```

The code above adds 16 pages to the paperback, and leaves the ebook exactly as it was: the comparison prints `true`.

## 6. Update Every Element with `.each`

Let's put books on shelves. A path step `.each` goes into every element of a collection. Because a collection can hold any number of elements — including none — a path with `.each` gives a `Traversal[S, A]`, an optic that focuses on zero or more values at once.

We add a shelf, and an inventory that holds the shelves and a price list. The companion object of `Inventory` declares a traversal over the stock of every book on every shelf:

```scala
case class Shelf(name: String, books: List[Book])

case class Inventory(store: String, shelves: List[Shelf], prices: Map[String, BigDecimal])

object Inventory extends CompanionOptics[Inventory] {
  implicit val schema: Schema[Inventory] = Schema.derived

  val allStock: Traversal[Inventory, Int] = $(_.shelves.each.books.each.stock)
}

val fpInScala = Book("Functional Programming in Scala", Author("Paul Chiusano"), Paperback(320), 0)

val inventory = Inventory(
  store = "Lambda Books",
  shelves = List(
    Shelf("Scala", List(programmingInScala, fpInScala)),
    Shelf("ZIO", List(zionomicon))
  ),
  prices = Map("Programming in Scala" -> BigDecimal("59.95"), "Zionomicon" -> BigDecimal("49.99"))
)
```

The code above:

- `Shelf` holds a list of books, and `Inventory` holds a list of shelves and a `Map` from title to price.
- `Inventory.schema` is derived like `Book.schema`, and reuses it for the books on each shelf.
- `$(_.shelves.each.books.each.stock)` reads as "every shelf, then every book on it, then its stock".
- `inventory` has two shelves with three books, and prices for two of them.

A `Traversal` has no `get`, because there can be many values. Instead, `Traversal#fold` combines them, here into a list:

```scala
println(Inventory.allStock.fold(inventory)(List.empty[Int], (stocks, stock) => stocks :+ stock))
// List(7, 0, 3)
```

The code above starts from an empty list and appends each stock value, shelf by shelf and book by book.

A delivery has arrived with one more copy of every book. `Traversal#modify` applies the function to every focused value:

```scala
val restocked = Inventory.allStock.modify(inventory, _ + 1)
```

```scala
println(Inventory.allStock.fold(restocked)(List.empty[Int], (stocks, stock) => stocks :+ stock))
// List(8, 1, 4)
```

The code above restocks all three books in a single call. Written by hand, the same update needs a `map` over the shelves, a `map` over each shelf's books and two `copy` calls.

## 7. Reach Into a Map with `.atKey`

A path step `.atKey(k)` focuses on the value stored under key `k` in a `Map`. The key might be missing, so this gives an `Optional`, like `.when[B]`.

Let's add two price optics to the companion object of `Inventory`: one for Zionomicon, and one for Functional Programming in Scala, which has no entry in our price list. We start a fresh scope again, so this block repeats the model and the data:

```scala

case class Author(name: String)

sealed trait Format
case class Paperback(pages: Int)  extends Format
case class Ebook(sizeMb: Double) extends Format

case class Book(title: String, author: Author, format: Format, stock: Int)

object Book extends CompanionOptics[Book] {
  implicit val schema: Schema[Book] = Schema.derived

  val title: Lens[Book, String]          = $(_.title)
  val authorName: Lens[Book, String]     = $(_.author.name)
  val paperbackPages: Optional[Book, Int] = $(_.format.when[Paperback].pages)
}

case class Shelf(name: String, books: List[Book])

case class Inventory(store: String, shelves: List[Shelf], prices: Map[String, BigDecimal])

object Inventory extends CompanionOptics[Inventory] {
  implicit val schema: Schema[Inventory] = Schema.derived

  val allStock: Traversal[Inventory, Int]              = $(_.shelves.each.books.each.stock)
  val zionomiconPrice: Optional[Inventory, BigDecimal] = $(_.prices.atKey("Zionomicon"))
  val fpInScalaPrice: Optional[Inventory, BigDecimal]  = $(_.prices.atKey("Functional Programming in Scala"))
}

val programmingInScala = Book("Programming in Scala", Author("Martin Odersky"), Paperback(860), 7)

val zionomicon = Book("Zionomicon", Author("John A. De Goes"), Ebook(5.2), 3)

val fpInScala = Book("Functional Programming in Scala", Author("Paul Chiusano"), Paperback(320), 0)

val inventory = Inventory(
  store = "Lambda Books",
  shelves = List(
    Shelf("Scala", List(programmingInScala, fpInScala)),
    Shelf("ZIO", List(zionomicon))
  ),
  prices = Map("Programming in Scala" -> BigDecimal("59.95"), "Zionomicon" -> BigDecimal("49.99"))
)
```

The code above adds `zionomiconPrice` and `fpInScalaPrice`. The path `_.prices.atKey("Zionomicon")` reads as "the price list, then the value under the key `"Zionomicon"`".

Let's look up and change the price of Zionomicon:

```scala
println(Inventory.zionomiconPrice.getOption(inventory))
// Some(49.99)

println(Inventory.zionomiconPrice.replace(inventory, BigDecimal("44.99")).prices)
// Map(Programming in Scala -> 59.95, Zionomicon -> 44.99)
```

The code above looks up the current price with `getOption(inventory)`, which returns it wrapped in `Some`, and builds a new inventory with the new price using `replace(inventory, BigDecimal("44.99"))`; we print only its `prices` to keep the output short.

Now let's see what an `Optional` does with the missing key:

```scala
println(Inventory.fpInScalaPrice.getOption(inventory))
// None

println(Inventory.fpInScalaPrice.modify(inventory, _ + 1) == inventory)
// true
```

The code above returns `None` for the missing key, and `modify` returns the inventory unchanged, just like the ebook in section 5.

## 8. See Why an Update Failed with `modifyOrFail`

Returning the input unchanged is often what we want. Sometimes, though, a missing target is a bug: we expected a paperback, or we expected the price to be listed. For those cases, every optic has `modifyOrFail`, which returns an `Either[OpticCheck, S]`. On success it holds the updated value in a `Right`. On failure it holds an `OpticCheck` in a `Left`, and `OpticCheck#message` says what the optic expected and where it stopped.

Let's try to add pages to the ebook:

```scala
Book.paperbackPages.modifyOrFail(zionomicon, _ + 16) match {
  case Left(check) => println(check.message)
  case Right(book) => println(s"Updated: $book")
}
// During attempted access at .format.when[Paperback].pages, encountered an unexpected case at .format.when[Paperback]: expected Paperback, but got Ebook
```

The code above calls `Book.paperbackPages.modifyOrFail(zionomicon, _ + 16)`, which returns a `Left` because Zionomicon's format is an `Ebook`; `check.message` names the full path of the optic, the step where it stopped, the case it expected and the case it found.

Now let's try to raise the missing price:

```scala
Inventory.fpInScalaPrice.modifyOrFail(inventory, _ + 1) match {
  case Left(check)    => println(check.message)
  case Right(updated) => println(s"Updated: ${updated.prices}")
}
// During attempted access at .prices.atKey("Functional Programming in Scala"), encountered missing key at .prices.atKey("Functional Programming in Scala")
```

This time the message says that the key is missing, and shows the path with the key we asked for. When the target is there, `modifyOrFail` returns the updated value:

```scala
println(Book.paperbackPages.modifyOrFail(programmingInScala, _ + 16).map(_.format))
// Right(Paperback(876))
```

The code above returns a `Right` holding the updated book; we map over it to print only its new format. Every optic has `modifyOrFail` — including lenses, which always return a `Right`, and traversals, which fail when they find no elements.

:::tip
Use `modify` when "nothing to change" is a normal outcome, and `modifyOrFail` when it means the data is not what you expected.
:::

## 9. Write the Same Paths as Strings with `p"..."`

Everything so far was checked by the compiler against our case classes. ZIO Blocks also has a string syntax for paths, the `p"..."` interpolator, for code that works with data generically instead of with one Scala type. The string is checked at compile time too, and the result is a `DynamicOptic`: the same path, without the Scala types at each end.

Let's write the stock path as a string and compare it with the `allStock` traversal from section 6:

```scala
val stockPath: DynamicOptic = p".shelves[*].books[*].stock"
// stockPath: DynamicOptic = DynamicOptic(
//   Vector(Field("shelves"), Elements, Field("books"), Elements, Field("stock"))
// )

println(stockPath)
// .shelves[*].books[*].stock
println(Inventory.allStock.toDynamic == stockPath)
// true
```

The code above writes the path from section 6 as the string `p".shelves[*].books[*].stock"`, where `[*]` means "every element", like `.each`, and turns the typed traversal into its `DynamicOptic` with `Inventory.allStock.toDynamic`; the comparison prints `true`.

A `DynamicOptic` works on `DynamicValue`, the generic data view from [Your First Schema](./your-first-schema.md#8-look-at-the-same-data-two-more-ways). Let's convert the inventory and select the stock values with the string path:

```scala
val dynamicInventory = Inventory.schema.toDynamicValue(inventory)
```

```scala
println(dynamicInventory.get(stockPath).toChunk.mkString(", "))
// 7, 0, 3
```

The code above prints the same three stock values that `allStock` folded in section 6, this time selected as `DynamicValue`s by a path written as a string. The [Path Interpolator](../reference/schema/path-interpolator.md) reference shows the full string syntax, including cases (`<Paperback>`) and map keys.

## 10. Putting It Together

This example combines all the concepts we've learned — lenses in the companion object, a nested path, an optional on one case and on one map key, a traversal over every book, `modifyOrFail` for a missing target and the same path as a `p"..."` string:

```scala title="schema-examples/src/main/scala/nesteddatawithoptics/CompleteExample.scala"
package nesteddatawithoptics

/**
 * Navigate and Update Nested Data with Optics — Complete Example
 *
 * Combines lenses, a nested path, an optional on one case and on one map key, a
 * traversal over every book, modifyOrFail for a missing target and the same
 * path as a p"..." string.
 *
 * Run with: cd schema-examples && sbt "runMain
 * nesteddatawithoptics.CompleteExample"
 */
object CompleteExample extends App {

  case class Author(name: String)

  sealed trait Format
  case class Paperback(pages: Int) extends Format
  case class Ebook(sizeMb: Double) extends Format

  case class Book(title: String, author: Author, format: Format, stock: Int)

  object Book extends CompanionOptics[Book] {
    implicit val schema: Schema[Book] = Schema.derived

    val title: Lens[Book, String]           = $(_.title)
    val authorName: Lens[Book, String]      = $(_.author.name)
    val paperbackPages: Optional[Book, Int] = $(_.format.when[Paperback].pages)
  }

  case class Shelf(name: String, books: List[Book])

  case class Inventory(store: String, shelves: List[Shelf], prices: Map[String, BigDecimal])

  object Inventory extends CompanionOptics[Inventory] {
    implicit val schema: Schema[Inventory] = Schema.derived

    val allStock: Traversal[Inventory, Int]              = $(_.shelves.each.books.each.stock)
    val zionomiconPrice: Optional[Inventory, BigDecimal] = $(_.prices.atKey("Zionomicon"))
    val fpInScalaPrice: Optional[Inventory, BigDecimal]  = $(_.prices.atKey("Functional Programming in Scala"))
  }

  val programmingInScala = Book("Programming in Scala", Author("Martin Odersky"), Paperback(860), 7)

  val zionomicon = Book("Zionomicon", Author("John A. De Goes"), Ebook(5.2), 3)

  val fpInScala = Book("Functional Programming in Scala", Author("Paul Chiusano"), Paperback(320), 0)

  val inventory = Inventory(
    store = "Lambda Books",
    shelves = List(
      Shelf("Scala", List(programmingInScala, fpInScala)),
      Shelf("ZIO", List(zionomicon))
    ),
    prices = Map("Programming in Scala" -> BigDecimal("59.95"), "Zionomicon" -> BigDecimal("49.99"))
  )

  println(Book.title.get(programmingInScala))

  println(Book.title.replace(programmingInScala, "Programming in Scala, Fifth Edition"))

  println(Book.title.modify(programmingInScala, _.toUpperCase))

  println(programmingInScala)

  println(Book.title)

  println(Book.authorName.get(programmingInScala))

  val shouting = Book.authorName.modify(programmingInScala, _.toUpperCase)

  val byHand =
    programmingInScala.copy(author = programmingInScala.author.copy(name = programmingInScala.author.name.toUpperCase))

  println(shouting)
  println(shouting == byHand)

  println(shouting.author eq programmingInScala.author)
  println(shouting.format eq programmingInScala.format)

  println(Book.paperbackPages.getOption(programmingInScala))
  println(Book.paperbackPages.getOption(zionomicon))

  println(Book.paperbackPages.modify(programmingInScala, _ + 16))

  println(Book.paperbackPages.modify(zionomicon, _ + 16) == zionomicon)

  println(Inventory.allStock.fold(inventory)(List.empty[Int], (stocks, stock) => stocks :+ stock))

  val restocked = Inventory.allStock.modify(inventory, _ + 1)

  println(Inventory.allStock.fold(restocked)(List.empty[Int], (stocks, stock) => stocks :+ stock))

  println(Inventory.zionomiconPrice.getOption(inventory))

  println(Inventory.zionomiconPrice.replace(inventory, BigDecimal("44.99")).prices)

  println(Inventory.fpInScalaPrice.getOption(inventory))

  println(Inventory.fpInScalaPrice.modify(inventory, _ + 1) == inventory)

  Book.paperbackPages.modifyOrFail(zionomicon, _ + 16) match {
    case Left(check) => println(check.message)
    case Right(book) => println(s"Updated: $book")
  }

  Inventory.fpInScalaPrice.modifyOrFail(inventory, _ + 1) match {
    case Left(check)    => println(check.message)
    case Right(updated) => println(s"Updated: ${updated.prices}")
  }

  println(Book.paperbackPages.modifyOrFail(programmingInScala, _ + 16).map(_.format))

  val stockPath: DynamicOptic = p".shelves[*].books[*].stock"

  println(stockPath)
  println(Inventory.allStock.toDynamic == stockPath)

  val dynamicInventory = Inventory.schema.toDynamicValue(inventory)

  println(dynamicInventory.get(stockPath).toChunk.mkString(", "))
}
```

## 11. Running the Examples

All examples in this tutorial have corresponding runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

Run them in order to follow the tutorial step by step.

### Put a Lens on One Field

This example declares a lens on a book's title in the companion object, then reads, replaces and modifies the title.

<details>
  <summary>schema-examples/src/main/scala/nesteddatawithoptics/Concept1LensExample.scala</summary>

```scala title="schema-examples/src/main/scala/nesteddatawithoptics/Concept1LensExample.scala" showLineNumbers
package nesteddatawithoptics

/**
 * Navigate and Update Nested Data with Optics — Concept 1: Put a Lens on One
 * Field
 *
 * Declares lenses on a book's title and first author in the companion object,
 * then reads, replaces and modifies the title.
 *
 * Run with: cd schema-examples && sbt "runMain
 * nesteddatawithoptics.Concept1LensExample"
 */
object Concept1LensExample extends App {

  case class Author(name: String)

  sealed trait Format
  case class Paperback(pages: Int) extends Format
  case class Ebook(sizeMb: Double) extends Format

  case class Book(title: String, author: Author, format: Format, stock: Int)

  object Book extends CompanionOptics[Book] {
    implicit val schema: Schema[Book] = Schema.derived

    val title: Lens[Book, String]      = $(_.title)
    val authorName: Lens[Book, String] = $(_.author.name)
  }

  val programmingInScala = Book("Programming in Scala", Author("Martin Odersky"), Paperback(860), 7)

  println(Book.title.get(programmingInScala))

  println(Book.title.replace(programmingInScala, "Programming in Scala, Fifth Edition"))

  println(Book.title.modify(programmingInScala, _.toUpperCase))

  println(programmingInScala)

  println(Book.title)
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain nesteddatawithoptics.Concept1LensExample"
```

### Follow a Path Into Nested Data

This example uses a lens on the first author's name, compares its update with nested `copy` calls, and shows which values are shared with the original.

<details>
  <summary>schema-examples/src/main/scala/nesteddatawithoptics/Concept2NestedPathExample.scala</summary>

```scala title="schema-examples/src/main/scala/nesteddatawithoptics/Concept2NestedPathExample.scala" showLineNumbers
package nesteddatawithoptics

/**
 * Navigate and Update Nested Data with Optics — Concept 2: Follow a Path Into
 * Nested Data
 *
 * Uses a lens on the first author's name, compares its update with nested copy
 * calls and shows which values are shared with the original.
 *
 * Run with: cd schema-examples && sbt "runMain
 * nesteddatawithoptics.Concept2NestedPathExample"
 */
object Concept2NestedPathExample extends App {

  case class Author(name: String)

  sealed trait Format
  case class Paperback(pages: Int) extends Format
  case class Ebook(sizeMb: Double) extends Format

  case class Book(title: String, author: Author, format: Format, stock: Int)

  object Book extends CompanionOptics[Book] {
    implicit val schema: Schema[Book] = Schema.derived

    val title: Lens[Book, String]      = $(_.title)
    val authorName: Lens[Book, String] = $(_.author.name)
  }

  val programmingInScala = Book("Programming in Scala", Author("Martin Odersky"), Paperback(860), 7)

  println(Book.authorName.get(programmingInScala))

  val shouting = Book.authorName.modify(programmingInScala, _.toUpperCase)

  val byHand =
    programmingInScala.copy(author = programmingInScala.author.copy(name = programmingInScala.author.name.toUpperCase))

  println(shouting)
  println(shouting == byHand)

  println(shouting.author eq programmingInScala.author)
  println(shouting.format eq programmingInScala.format)
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain nesteddatawithoptics.Concept2NestedPathExample"
```

### Focus on One Case with `.when[B]`

This example declares an optional on the page count of paperbacks and shows that it finds nothing in an ebook.

<details>
  <summary>schema-examples/src/main/scala/nesteddatawithoptics/Concept3WhenExample.scala</summary>

```scala title="schema-examples/src/main/scala/nesteddatawithoptics/Concept3WhenExample.scala" showLineNumbers
package nesteddatawithoptics

/**
 * Navigate and Update Nested Data with Optics — Concept 3: Focus on One Case
 * with .when[B]
 *
 * Declares an optional on the page count of paperbacks and shows that it finds
 * nothing in an ebook.
 *
 * Run with: cd schema-examples && sbt "runMain
 * nesteddatawithoptics.Concept3WhenExample"
 */
object Concept3WhenExample extends App {

  case class Author(name: String)

  sealed trait Format
  case class Paperback(pages: Int) extends Format
  case class Ebook(sizeMb: Double) extends Format

  case class Book(title: String, author: Author, format: Format, stock: Int)

  object Book extends CompanionOptics[Book] {
    implicit val schema: Schema[Book] = Schema.derived

    val title: Lens[Book, String]           = $(_.title)
    val authorName: Lens[Book, String]      = $(_.author.name)
    val paperbackPages: Optional[Book, Int] = $(_.format.when[Paperback].pages)
  }

  val programmingInScala = Book("Programming in Scala", Author("Martin Odersky"), Paperback(860), 7)

  val zionomicon = Book("Zionomicon", Author("John A. De Goes"), Ebook(5.2), 3)

  println(Book.paperbackPages.getOption(programmingInScala))
  println(Book.paperbackPages.getOption(zionomicon))

  println(Book.paperbackPages.modify(programmingInScala, _ + 16))

  println(Book.paperbackPages.modify(zionomicon, _ + 16) == zionomicon)
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain nesteddatawithoptics.Concept3WhenExample"
```

### Update Every Element with `.each`

This example declares a traversal over the stock of every book on every shelf, folds it into a list and restocks every book in one call.

<details>
  <summary>schema-examples/src/main/scala/nesteddatawithoptics/Concept4EachExample.scala</summary>

```scala title="schema-examples/src/main/scala/nesteddatawithoptics/Concept4EachExample.scala" showLineNumbers
package nesteddatawithoptics

/**
 * Navigate and Update Nested Data with Optics — Concept 4: Update Every Element
 * with .each
 *
 * Declares a traversal over the stock of every book on every shelf, folds it
 * into a list and restocks every book in one call.
 *
 * Run with: cd schema-examples && sbt "runMain
 * nesteddatawithoptics.Concept4EachExample"
 */
object Concept4EachExample extends App {

  case class Author(name: String)

  sealed trait Format
  case class Paperback(pages: Int) extends Format
  case class Ebook(sizeMb: Double) extends Format

  case class Book(title: String, author: Author, format: Format, stock: Int)

  object Book extends CompanionOptics[Book] {
    implicit val schema: Schema[Book] = Schema.derived

    val title: Lens[Book, String]           = $(_.title)
    val authorName: Lens[Book, String]      = $(_.author.name)
    val paperbackPages: Optional[Book, Int] = $(_.format.when[Paperback].pages)
  }

  case class Shelf(name: String, books: List[Book])

  case class Inventory(store: String, shelves: List[Shelf], prices: Map[String, BigDecimal])

  object Inventory extends CompanionOptics[Inventory] {
    implicit val schema: Schema[Inventory] = Schema.derived

    val allStock: Traversal[Inventory, Int] = $(_.shelves.each.books.each.stock)
  }

  val programmingInScala = Book("Programming in Scala", Author("Martin Odersky"), Paperback(860), 7)

  val zionomicon = Book("Zionomicon", Author("John A. De Goes"), Ebook(5.2), 3)

  val fpInScala = Book("Functional Programming in Scala", Author("Paul Chiusano"), Paperback(320), 0)

  val inventory = Inventory(
    store = "Lambda Books",
    shelves = List(
      Shelf("Scala", List(programmingInScala, fpInScala)),
      Shelf("ZIO", List(zionomicon))
    ),
    prices = Map("Programming in Scala" -> BigDecimal("59.95"), "Zionomicon" -> BigDecimal("49.99"))
  )

  println(Inventory.allStock.fold(inventory)(List.empty[Int], (stocks, stock) => stocks :+ stock))

  val restocked = Inventory.allStock.modify(inventory, _ + 1)

  println(Inventory.allStock.fold(restocked)(List.empty[Int], (stocks, stock) => stocks :+ stock))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain nesteddatawithoptics.Concept4EachExample"
```

### Reach Into a Map with `.atKey`

This example reads and replaces one price in the price list, and shows what happens when the key is missing.

<details>
  <summary>schema-examples/src/main/scala/nesteddatawithoptics/Concept5AtKeyExample.scala</summary>

```scala title="schema-examples/src/main/scala/nesteddatawithoptics/Concept5AtKeyExample.scala" showLineNumbers
package nesteddatawithoptics

/**
 * Navigate and Update Nested Data with Optics — Concept 5: Reach Into a Map
 * with .atKey
 *
 * Reads and replaces one price in the price list and shows what happens when
 * the key is missing.
 *
 * Run with: cd schema-examples && sbt "runMain
 * nesteddatawithoptics.Concept5AtKeyExample"
 */
object Concept5AtKeyExample extends App {

  case class Author(name: String)

  sealed trait Format
  case class Paperback(pages: Int) extends Format
  case class Ebook(sizeMb: Double) extends Format

  case class Book(title: String, author: Author, format: Format, stock: Int)

  object Book extends CompanionOptics[Book] {
    implicit val schema: Schema[Book] = Schema.derived

    val title: Lens[Book, String]           = $(_.title)
    val authorName: Lens[Book, String]      = $(_.author.name)
    val paperbackPages: Optional[Book, Int] = $(_.format.when[Paperback].pages)
  }

  case class Shelf(name: String, books: List[Book])

  case class Inventory(store: String, shelves: List[Shelf], prices: Map[String, BigDecimal])

  object Inventory extends CompanionOptics[Inventory] {
    implicit val schema: Schema[Inventory] = Schema.derived

    val allStock: Traversal[Inventory, Int]              = $(_.shelves.each.books.each.stock)
    val zionomiconPrice: Optional[Inventory, BigDecimal] = $(_.prices.atKey("Zionomicon"))
    val fpInScalaPrice: Optional[Inventory, BigDecimal]  = $(_.prices.atKey("Functional Programming in Scala"))
  }

  val programmingInScala = Book("Programming in Scala", Author("Martin Odersky"), Paperback(860), 7)

  val zionomicon = Book("Zionomicon", Author("John A. De Goes"), Ebook(5.2), 3)

  val fpInScala = Book("Functional Programming in Scala", Author("Paul Chiusano"), Paperback(320), 0)

  val inventory = Inventory(
    store = "Lambda Books",
    shelves = List(
      Shelf("Scala", List(programmingInScala, fpInScala)),
      Shelf("ZIO", List(zionomicon))
    ),
    prices = Map("Programming in Scala" -> BigDecimal("59.95"), "Zionomicon" -> BigDecimal("49.99"))
  )

  println(Inventory.zionomiconPrice.getOption(inventory))

  println(Inventory.zionomiconPrice.replace(inventory, BigDecimal("44.99")).prices)

  println(Inventory.fpInScalaPrice.getOption(inventory))

  println(Inventory.fpInScalaPrice.modify(inventory, _ + 1) == inventory)
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain nesteddatawithoptics.Concept5AtKeyExample"
```

### See Why an Update Failed with `modifyOrFail`

This example prints the `OpticCheck` message for a wrong case and for a missing key, and the `Right` result of a successful update.

<details>
  <summary>schema-examples/src/main/scala/nesteddatawithoptics/Concept6ModifyOrFailExample.scala</summary>

```scala title="schema-examples/src/main/scala/nesteddatawithoptics/Concept6ModifyOrFailExample.scala" showLineNumbers
package nesteddatawithoptics

/**
 * Navigate and Update Nested Data with Optics — Concept 6: See Why an Update
 * Failed with modifyOrFail
 *
 * Prints the OpticCheck message for a wrong case and for a missing key, and the
 * Right result of a successful update.
 *
 * Run with: cd schema-examples && sbt "runMain
 * nesteddatawithoptics.Concept6ModifyOrFailExample"
 */
object Concept6ModifyOrFailExample extends App {

  case class Author(name: String)

  sealed trait Format
  case class Paperback(pages: Int) extends Format
  case class Ebook(sizeMb: Double) extends Format

  case class Book(title: String, author: Author, format: Format, stock: Int)

  object Book extends CompanionOptics[Book] {
    implicit val schema: Schema[Book] = Schema.derived

    val title: Lens[Book, String]           = $(_.title)
    val authorName: Lens[Book, String]      = $(_.author.name)
    val paperbackPages: Optional[Book, Int] = $(_.format.when[Paperback].pages)
  }

  case class Shelf(name: String, books: List[Book])

  case class Inventory(store: String, shelves: List[Shelf], prices: Map[String, BigDecimal])

  object Inventory extends CompanionOptics[Inventory] {
    implicit val schema: Schema[Inventory] = Schema.derived

    val allStock: Traversal[Inventory, Int]              = $(_.shelves.each.books.each.stock)
    val zionomiconPrice: Optional[Inventory, BigDecimal] = $(_.prices.atKey("Zionomicon"))
    val fpInScalaPrice: Optional[Inventory, BigDecimal]  = $(_.prices.atKey("Functional Programming in Scala"))
  }

  val programmingInScala = Book("Programming in Scala", Author("Martin Odersky"), Paperback(860), 7)

  val zionomicon = Book("Zionomicon", Author("John A. De Goes"), Ebook(5.2), 3)

  val fpInScala = Book("Functional Programming in Scala", Author("Paul Chiusano"), Paperback(320), 0)

  val inventory = Inventory(
    store = "Lambda Books",
    shelves = List(
      Shelf("Scala", List(programmingInScala, fpInScala)),
      Shelf("ZIO", List(zionomicon))
    ),
    prices = Map("Programming in Scala" -> BigDecimal("59.95"), "Zionomicon" -> BigDecimal("49.99"))
  )

  Book.paperbackPages.modifyOrFail(zionomicon, _ + 16) match {
    case Left(check) => println(check.message)
    case Right(book) => println(s"Updated: $book")
  }

  Inventory.fpInScalaPrice.modifyOrFail(inventory, _ + 1) match {
    case Left(check)    => println(check.message)
    case Right(updated) => println(s"Updated: ${updated.prices}")
  }

  println(Book.paperbackPages.modifyOrFail(programmingInScala, _ + 16).map(_.format))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain nesteddatawithoptics.Concept6ModifyOrFailExample"
```

### Write the Same Paths as Strings with `p"..."`

This example writes the stock path as a `p"..."` string, compares it with the typed traversal and uses it on the generic view of the inventory.

<details>
  <summary>schema-examples/src/main/scala/nesteddatawithoptics/Concept7PathInterpolatorExample.scala</summary>

```scala title="schema-examples/src/main/scala/nesteddatawithoptics/Concept7PathInterpolatorExample.scala" showLineNumbers
package nesteddatawithoptics

/**
 * Navigate and Update Nested Data with Optics — Concept 7: Write the Same Paths
 * as Strings with p"..."
 *
 * Writes the stock path as a p"..." string, compares it with the typed
 * traversal and uses it on the generic view of the inventory.
 *
 * Run with: cd schema-examples && sbt "runMain
 * nesteddatawithoptics.Concept7PathInterpolatorExample"
 */
object Concept7PathInterpolatorExample extends App {

  case class Author(name: String)

  sealed trait Format
  case class Paperback(pages: Int) extends Format
  case class Ebook(sizeMb: Double) extends Format

  case class Book(title: String, author: Author, format: Format, stock: Int)

  object Book extends CompanionOptics[Book] {
    implicit val schema: Schema[Book] = Schema.derived

    val title: Lens[Book, String]           = $(_.title)
    val authorName: Lens[Book, String]      = $(_.author.name)
    val paperbackPages: Optional[Book, Int] = $(_.format.when[Paperback].pages)
  }

  case class Shelf(name: String, books: List[Book])

  case class Inventory(store: String, shelves: List[Shelf], prices: Map[String, BigDecimal])

  object Inventory extends CompanionOptics[Inventory] {
    implicit val schema: Schema[Inventory] = Schema.derived

    val allStock: Traversal[Inventory, Int]              = $(_.shelves.each.books.each.stock)
    val zionomiconPrice: Optional[Inventory, BigDecimal] = $(_.prices.atKey("Zionomicon"))
    val fpInScalaPrice: Optional[Inventory, BigDecimal]  = $(_.prices.atKey("Functional Programming in Scala"))
  }

  val programmingInScala = Book("Programming in Scala", Author("Martin Odersky"), Paperback(860), 7)

  val zionomicon = Book("Zionomicon", Author("John A. De Goes"), Ebook(5.2), 3)

  val fpInScala = Book("Functional Programming in Scala", Author("Paul Chiusano"), Paperback(320), 0)

  val inventory = Inventory(
    store = "Lambda Books",
    shelves = List(
      Shelf("Scala", List(programmingInScala, fpInScala)),
      Shelf("ZIO", List(zionomicon))
    ),
    prices = Map("Programming in Scala" -> BigDecimal("59.95"), "Zionomicon" -> BigDecimal("49.99"))
  )

  val stockPath: DynamicOptic = p".shelves[*].books[*].stock"

  println(stockPath)
  println(Inventory.allStock.toDynamic == stockPath)

  val dynamicInventory = Inventory.schema.toDynamicValue(inventory)

  println(dynamicInventory.get(stockPath).toChunk.mkString(", "))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain nesteddatawithoptics.Concept7PathInterpolatorExample"
```

### The Complete Example

The complete example from "Putting It Together" runs everything from the tutorial in one program:

```bash
sbt "runMain nesteddatawithoptics.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## 12. What You've Learned

In this tutorial, you learned:

- What an optic is — a path into a data structure, stored as a value — and how to declare one with `$(...)` in a companion object that extends `CompanionOptics`
- How a `Lens` reads with `get` and updates with `replace` and `modify`, at any depth, sharing everything off its path
- How `.when[B]` and `.atKey(k)` give an `Optional` that may find nothing, and how `.each` gives a `Traversal` over every element
- How `modifyOrFail` returns an `OpticCheck` that explains why an update found nothing to change
- How `p"..."` writes the same path as a string and gives a `DynamicOptic` for generic data

You now have a solid foundation in ZIO Blocks optics. The next step is to explore the other optic operations, and to see how optics become the building blocks of a query language.

## 13. Where to Go Next

- **Want the full optics API?** Read [Optics](../reference/schema/optics.md), which covers every operation on `Lens`, `Prism`, `Optional` and `Traversal`, building optics by hand, composing optics and their debug-friendly `toString`.
- **Need paths at run time?** Read [DynamicOptic](../reference/schema/dynamic-optic.md) and the [Path Interpolator](../reference/schema/path-interpolator.md) syntax.
- **Want to understand failures in depth?** The [Failure Path in OpticCheck](../reference/schema/dynamic-optic.md#failure-path-in-opticcheck) section of DynamicOptic shows the kinds of failure an optic can report, such as a wrong case, an empty sequence and a missing map key.
- **Ready to build on optics?** The four-part Query DSL series starts with [Writing a Query DSL with Reified Optics](./query-dsl-reified-optics.md), then [SQL](./query-dsl-sql.md), [Extending the DSL](./query-dsl-extending.md) and the [Fluent Builder](./query-dsl-fluent-builder.md).

---

## One Type, Many Formats: Export a Catalog as JSON, YAML, CSV, XML, MessagePack, TOON and Avro

Welcome! This tutorial is for Scala developers who have finished [Your First Schema](./your-first-schema.md) and can derive a `Schema` and round-trip a value through JSON. You don't need any prior experience with YAML, XML, CSV, TOON, MessagePack or Avro to follow along.

## 1. Introduction

By the end of this tutorial, you will understand:

- How one `Schema` value drives a codec for every format ZIO Blocks supports
- How to get a codec with `Schema#derive` and a format object, and how to encode and decode with it
- What the text formats (JSON, YAML, XML, TOON) and the binary formats (MessagePack, Avro) look like for the same data
- Why CSV accepts only flat records, and how a small flat row type solves that

We learn these concepts by exporting a small catalog of Scala books, one format at a time:

1. Model the catalog
2. Encode it to JSON the long way
3. Swap in YAML
4. Swap in XML
5. Swap in TOON
6. Swap in MessagePack
7. Swap in Avro
8. Flatten the catalog for CSV

We recommend reading from top to bottom — each section builds on the previous one.

Each format lives in its own module. To follow along, add the schema module and the six format modules to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-schema"             % "0.0.56",
  "dev.zio" %% "zio-blocks-schema-yaml"        % "0.0.56",
  "dev.zio" %% "zio-blocks-schema-xml"         % "0.0.56",
  "dev.zio" %% "zio-blocks-schema-toon"        % "0.0.56",
  "dev.zio" %% "zio-blocks-schema-messagepack" % "0.0.56",
  "dev.zio" %% "zio-blocks-schema-avro"        % "0.0.56",
  "dev.zio" %% "zio-blocks-schema-csv"         % "0.0.56"
)

// Scala 3.9 and newer (also needs scalacOptions += "-experimental"): the same modules with the zio-blocks-next- prefix
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-schema"             % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema-yaml"        % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema-xml"         % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema-toon"        % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema-messagepack" % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema-avro"        % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema-csv"         % "0.0.56"
)
```

The JSON codec ships inside `zio-blocks-schema` itself, so it needs no extra module. The code in this tutorial is the same for every Scala version.

## 2. Background: One Schema, Many Codecs

In [Your First Schema](./your-first-schema.md), we derived a `Schema[Book]` and got a JSON codec for free. JSON is only one of many formats your data may need: YAML for configuration files, XML for older services, CSV for spreadsheets, MessagePack or Avro for compact messages between services, and TOON for compact prompts to large language models. In most libraries, each of those formats means another set of encoders and decoders to write and keep in sync with the case class.

ZIO Blocks treats a format as a plug-in. Each format module provides a format object — `YamlFormat`, `XmlFormat`, `AvroFormat` and so on — that knows how to turn any `Schema` into a codec for that format. The schema describes the shape of the data once; the format object decides how that shape is written. So adding a format to a program means adding its module, importing its format object, and writing one line that derives the codec. In this tutorial, we watch that one line produce seven very different outputs from the same catalog.

## 3. Model the Catalog

Let's start with the data. A `Book` has a title, an author, a publication year and a list of tags, and a `Catalog` holds a list of books. As in the first tutorial, each type gets a derived schema in its companion object:

```scala

case class Book(title: String, author: String, year: Int, tags: List[String])

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

case class Catalog(books: List[Book])

object Catalog {
  implicit val schema: Schema[Catalog] = Schema.derived
}
```

The code above:

- `Book` stores its authors as one `String` and its tags as a `List[String]`.
- `Catalog` wraps a `List[Book]`, so the whole export is one value with one top-level name.
- `Schema.derived` derives each schema at compile time; `Catalog.schema` finds `Book.schema` for the list's elements.

Next, we fill the catalog with three books about Scala:

```scala
val catalog = Catalog(
  List(
    Book("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", 2024, List("zio", "effects")),
    Book("Programming in Scala", "Martin Odersky, Lex Spoon, Bill Venners", 2008, List("scala", "fundamentals")),
    Book("Functional Programming in Scala", "Paul Chiusano, Rúnar Bjarnason", 2014, List("scala", "fp"))
  )
)
```

The code above builds the one value we export in every section that follows. Notice that every author list contains a comma — that detail matters when we reach CSV.

## 4. Encode to JSON the Long Way

In the first tutorial, `toJsonString` and `fromJson` did all the work. Behind those shortcuts sits a codec — a value that can both encode and decode one type in one format. Let's get the JSON codec explicitly, because the same three steps work for every other format in this tutorial:

```scala

val jsonCodec = Catalog.schema.derive(JsonFormat)

val jsonBytes = jsonCodec.encode(catalog)
```

The code above:

- `import zio.blocks.schema.json.JsonFormat` brings the JSON format object into scope.
- `Catalog.schema.derive(JsonFormat)` walks the catalog's schema and builds a `JsonCodec[Catalog]`.
- `jsonCodec.encode(catalog)` writes the catalog and returns the result as an `Array[Byte]`.

The bytes are UTF-8 text, so we turn them into a `String` to read them, and then decode the same bytes back:

```scala
println(new String(jsonBytes, UTF_8))
// {"books":[{"title":"Zionomicon","author":"John A. De Goes, Adam Fraser and Milad Khajavi","year":2024,"tags":["zio","effects"]},{"title":"Programming in Scala","author":"Martin Odersky, Lex Spoon, Bill Venners","year":2008,"tags":["scala","fundamentals"]},{"title":"Functional Programming in Scala","author":"Paul Chiusano, Rúnar Bjarnason","year":2014,"tags":["scala","fp"]}]}

jsonCodec.decode(jsonBytes) == Right(catalog)
// res1: Boolean = true
```

The code above prints the JSON text and then checks the round trip. `JsonCodec#decode` returns an `Either[SchemaError, Catalog]`, just like `fromJson`, and `true` tells us that decoding gave back exactly the catalog we started with.

These are the three steps we repeat for every format: derive a codec from the schema, encode a value to bytes, decode the bytes back. The `toJsonString` shortcut from the first tutorial asks the schema for this same JSON codec and calls `JsonCodec#encodeToString` on it.

## 5. Swap In YAML

Now for the moment this tutorial is about. To export the catalog as YAML, we change only the format object. This step uses the `zio-blocks-schema-yaml` module:

```scala

val yamlCodec = Catalog.schema.derive(YamlFormat)

val yamlBytes = yamlCodec.encode(catalog)
```

The code above:

- `import zio.blocks.schema.yaml.YamlFormat` brings the YAML format object into scope.
- `Catalog.schema.derive(YamlFormat)` is the JSON line with `JsonFormat` replaced by `YamlFormat`; `Catalog.schema` already says everything the YAML codec needs.
- `yamlCodec.encode(catalog)` writes the catalog as YAML bytes.

Let's print the result and check the round trip:

```scala
println(new String(yamlBytes, UTF_8))
// books:
//   - title: Zionomicon
//     author: John A. De Goes, Adam Fraser and Milad Khajavi
//     year: 2024
//     tags:
//       - zio
//       - effects
//   - title: Programming in Scala
//     author: Martin Odersky, Lex Spoon, Bill Venners
//     year: 2008
//     tags:
//       - scala
//       - fundamentals
//   - title: Functional Programming in Scala
//     author: Paul Chiusano, Rúnar Bjarnason
//     year: 2014
//     tags:
//       - scala
//       - fp

yamlCodec.decode(yamlBytes) == Right(catalog)
// res3: Boolean = true
```

The code above prints the YAML and decodes it back. Notice how the same structure looks in YAML: the `books` field becomes a block sequence with one `-` entry per book, each book's fields become `key: value` lines, and the nested `tags` list becomes a sequence of its own.

## 6. Swap In XML

XML follows the same pattern. This step uses the `zio-blocks-schema-xml` module:

```scala

val xmlCodec = Catalog.schema.derive(XmlFormat)

val xmlBytes = xmlCodec.encode(catalog)
```

The code above:

- `import zio.blocks.schema.xml.XmlFormat` brings the XML format object into scope.
- `Catalog.schema.derive(XmlFormat)` builds the XML codec from the same schema.
- `xmlCodec.encode(catalog)` writes the catalog as XML bytes.

Let's print the XML and decode it back:

```scala
println(new String(xmlBytes, UTF_8))
// <Catalog><books><item><title>Zionomicon</title><author>John A. De Goes, Adam Fraser and Milad Khajavi</author><year>2024</year><tags><item>zio</item><item>effects</item></tags></item><item><title>Programming in Scala</title><author>Martin Odersky, Lex Spoon, Bill Venners</author><year>2008</year><tags><item>scala</item><item>fundamentals</item></tags></item><item><title>Functional Programming in Scala</title><author>Paul Chiusano, Rúnar Bjarnason</author><year>2014</year><tags><item>scala</item><item>fp</item></tags></item></books></Catalog>

xmlCodec.decode(xmlBytes) == Right(catalog)
// res5: Boolean = true
```

The code above prints one compact line of XML. Every XML document needs a single root element, and the codec names it after the type, so the catalog becomes `<Catalog>`. Each field becomes a child element with the field's name, and each list element is wrapped in an `<item>` element — both for the books and for each book's tags.

:::tip
To print indented XML for people to read, pass a writer configuration: `xmlCodec.encodeToString(catalog, WriterConfig.pretty)` with `import zio.blocks.schema.xml.WriterConfig`.
:::

## 7. Swap In TOON

TOON (Token-Oriented Object Notation) is a text format designed to carry structured data to large language models in fewer tokens than JSON. This step uses the `zio-blocks-schema-toon` module:

```scala

val toonCodec = Catalog.schema.derive(ToonFormat)

val toonBytes = toonCodec.encode(catalog)
```

The code above:

- `import zio.blocks.schema.toon.ToonFormat` brings the TOON format object into scope.
- `Catalog.schema.derive(ToonFormat)` builds the TOON codec from the same schema.
- `toonCodec.encode(catalog)` writes the catalog as TOON bytes.

Let's print the TOON text and decode it back:

```scala
println(new String(toonBytes, UTF_8))
// books[3]:
//   - title: Zionomicon
//     author: John A. De Goes, Adam Fraser and Milad Khajavi
//     year: 2024
//     tags[2]: zio,effects
//   - title: Programming in Scala
//     author: Martin Odersky, Lex Spoon, Bill Venners
//     year: 2008
//     tags[2]: scala,fundamentals
//   - title: Functional Programming in Scala
//     author: Paul Chiusano, Rúnar Bjarnason
//     year: 2014
//     tags[2]: scala,fp

toonCodec.decode(toonBytes) == Right(catalog)
// res7: Boolean = true
```

The code above prints the catalog as TOON. Notice what TOON leaves out compared with JSON: there are no braces and no quotes around keys or plain strings. Each list states its length up front — `books[3]` and `tags[2]` — and a list of plain values such as `tags` fits on one line as `zio,effects`.

## 8. Swap In MessagePack

The formats so far produce text. MessagePack is a binary format: it stores the same structure as JSON, but with compact type tags instead of punctuation. This step uses the `zio-blocks-schema-messagepack` module:

```scala

val msgpackCodec = Catalog.schema.derive(MessagePackFormat)

val msgpackBytes = msgpackCodec.encode(catalog)
```

The code above:

- `import zio.blocks.schema.msgpack.MessagePackFormat` brings the MessagePack format object into scope.
- `Catalog.schema.derive(MessagePackFormat)` builds the MessagePack codec from the same schema.
- `msgpackCodec.encode(catalog)` writes the catalog as binary bytes.

Binary bytes aren't readable as text, so instead of printing them as a `String`, we write a small helper that shows the first 16 bytes in hexadecimal:

```scala
def hexPrefix(bytes: Array[Byte]): String =
  bytes.take(16).map(b => f"$b%02x").mkString(" ")
```

The code above:

- `bytes.take(16)` keeps only the first 16 bytes.
- `f"$b%02x"` formats each byte as two hexadecimal digits.
- `mkString(" ")` joins the digits with spaces.

Now let's compare the size with JSON, look at the first bytes and check the round trip:

```scala
println(s"MessagePack: ${msgpackBytes.length} bytes, JSON: ${jsonBytes.length} bytes")
// MessagePack: 317 bytes, JSON: 378 bytes

println(hexPrefix(msgpackBytes))
// 81 a5 62 6f 6f 6b 73 93 84 a5 74 69 74 6c 65 aa

msgpackCodec.decode(msgpackBytes) == Right(catalog)
// res10: Boolean = true
```

The code above shows that the MessagePack export is smaller than the JSON one. The hex prefix still follows the catalog's structure: `81` starts a map with one entry, `a5` starts a 5-byte string, `62 6f 6f 6b 73` spells `books`, `93` starts an array of three books, and `84` starts the first book as a map with four entries, whose first key is `title`. Field names travel inside the bytes, just as they do in JSON.

## 9. Swap In Avro

Avro is a binary format that is popular with Apache Kafka and big-data tools. This step uses the `zio-blocks-schema-avro` module, which builds on the Apache Avro library and runs on the JVM only — it is not available for Scala.js:

```scala

val avroCodec = Catalog.schema.derive(AvroFormat)

val avroBytes = avroCodec.encode(catalog)
```

The code above:

- `import zio.blocks.schema.avro.AvroFormat` brings the Avro format object into scope.
- `Catalog.schema.derive(AvroFormat)` builds the Avro codec from the same schema.
- `avroCodec.encode(catalog)` writes the catalog as binary bytes.

Let's compare the size with the previous two formats, look at the first bytes and check the round trip:

```scala
println(s"Avro: ${avroBytes.length} bytes, MessagePack: ${msgpackBytes.length} bytes, JSON: ${jsonBytes.length} bytes")
// Avro: 237 bytes, MessagePack: 317 bytes, JSON: 378 bytes

println(hexPrefix(avroBytes))
// 06 14 5a 69 6f 6e 6f 6d 69 63 6f 6e 5c 4a 6f 68

avroCodec.decode(avroBytes) == Right(catalog)
// res13: Boolean = true
```

The code above shows that Avro is the smallest export so far. Its bytes contain no field names at all: `06` is the number of books (Avro writes 3 as `06`), `14` is the length of the first title (10), and the next ten bytes spell `Zionomicon`. A reader needs the Avro schema to know which value is which field, and the codec derived that from `Catalog.schema` as well — `AvroCodec#avroSchema` returns it.

ZIO Blocks has two more binary formats, both JVM only: Thrift, derived with `ThriftFormat` from `zio-blocks-schema-thrift`, and BSON, derived with `BsonCodecDeriver` from `zio-blocks-schema-bson`.

## 10. Flatten the Catalog for CSV

CSV is different from every format so far. A CSV file is a table: one row per record, one column per field, and every cell is plain text. There is no place in a cell for a list, so a `Book` with its `tags` doesn't fit. The CSV deriver checks this when we derive the codec and throws an exception. Let's see the message. This step uses the `zio-blocks-schema-csv` module:

```scala

println(scala.util.Try(Book.schema.derive(CsvFormat)))
// Failure(java.lang.UnsupportedOperationException: CSV does not support sequence fields.)
```

The code above imports `CsvFormat` from `zio.blocks.schema.csv` and wraps the derivation in `scala.util.Try(...)`, so we can print the failure instead of stopping the program.

The exception is thrown at run time, when we call `Schema#derive`, not at compile time. The CSV deriver rejects lists, maps, `Option` fields and sealed traits in the same way.

The fix is a flat row type that holds only the columns we want in the table. Let's define `BookRow` and turn each book into a row:

```scala
case class BookRow(title: String, author: String, year: Int)

object BookRow {
  implicit val schema: Schema[BookRow] = Schema.derived
}

val rows = catalog.books.map(book => BookRow(book.title, book.author, book.year))

val csvCodec = BookRow.schema.derive(CsvFormat)
```

The code above derives the CSV codec for `BookRow`, which works because every field is a plain value. Notice that we derive the codec for one row, not for the catalog — a CSV codec encodes and decodes one record at a time.

CSV is also the one text format in this tutorial: `CsvFormat` is a `TextFormat`, so its codec writes characters into a `java.nio.CharBuffer` instead of returning bytes. Let's write a small helper that encodes one row into a buffer and returns the line as a `String`:

```scala

def toCsvLine(row: BookRow): String = {
  val buffer = CharBuffer.allocate(256)
  csvCodec.encode(row, buffer)
  buffer.flip()
  buffer.toString
}

val csvLines = rows.map(toCsvLine)
```

The code above:

- `CharBuffer.allocate(256)` creates a buffer large enough for one line.
- `csvCodec.encode(row, buffer)` writes the row's fields, separated by commas and ending with a line break.
- `buffer.flip()` switches the buffer from writing to reading, so `buffer.toString` returns what was written.

The codec writes data rows only; the header line comes from `CsvCodec#headerNames`, the list of column names. Let's print the header and the three lines:

```scala
println(csvCodec.headerNames.mkString(","))
// title,author,year

csvLines.foreach(line => println(line.stripLineEnd))
// Zionomicon,"John A. De Goes, Adam Fraser and Milad Khajavi",2024
// Programming in Scala,"Martin Odersky, Lex Spoon, Bill Venners",2008
// Functional Programming in Scala,"Paul Chiusano, Rúnar Bjarnason",2014
```

The code above prints a small CSV table. Notice the author column: a value that contains a comma is wrapped in double quotes, so the commas inside `"John A. De Goes, Adam Fraser and Milad Khajavi"` don't split the cell. The derived codec always uses the default CSV settings — a comma between fields, double quotes and a `\r\n` line ending.

Finally, let's decode each line back into a `BookRow` and check the round trip:

```scala
csvLines.map(line => csvCodec.decode(CharBuffer.wrap(line))) == rows.map(Right(_))
// res17: Boolean = true
```

The code above decodes each line with `CsvCodec#decode` and compares the results with the original rows. The tags are not in the CSV, so the round trip holds for `BookRow`, not for `Book` — choosing which fields go into the table is part of designing a CSV export.

## 11. Putting It Together

This example combines all the concepts we've learned — one `Catalog` schema, a codec derived for each of the six structured formats with the same `Schema#derive` call, a round trip through each, and a flat `BookRow` for CSV:

```scala title="schema-examples/src/main/scala/onetypemanyformats/CompleteExample.scala"
package onetypemanyformats

/**
 * One Type, Many Formats — Complete Example
 *
 * Derives JSON, YAML, XML, TOON, MessagePack and Avro codecs from one Catalog
 * schema, round-trips each, then exports a flat BookRow as CSV.
 *
 * Run with: cd schema-examples && sbt "runMain
 * onetypemanyformats.CompleteExample"
 */
object CompleteExample extends App {

  case class Book(title: String, author: String, year: Int, tags: List[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalog = Catalog(
    List(
      Book("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", 2024, List("zio", "effects")),
      Book("Programming in Scala", "Martin Odersky, Lex Spoon, Bill Venners", 2008, List("scala", "fundamentals")),
      Book("Functional Programming in Scala", "Paul Chiusano, Rúnar Bjarnason", 2014, List("scala", "fp"))
    )
  )

  // One schema, six structured formats: only the format object changes
  val jsonCodec = Catalog.schema.derive(JsonFormat)
  val yamlCodec = Catalog.schema.derive(YamlFormat)
  val xmlCodec  = Catalog.schema.derive(XmlFormat)
  val toonCodec = Catalog.schema.derive(ToonFormat)
  val msgCodec  = Catalog.schema.derive(MessagePackFormat)
  val avroCodec = Catalog.schema.derive(AvroFormat)

  val jsonBytes = jsonCodec.encode(catalog)
  println(new String(jsonBytes, UTF_8))
  println(jsonCodec.decode(jsonBytes) == Right(catalog))

  val yamlBytes = yamlCodec.encode(catalog)
  println(new String(yamlBytes, UTF_8))
  println(yamlCodec.decode(yamlBytes) == Right(catalog))

  val xmlBytes = xmlCodec.encode(catalog)
  println(new String(xmlBytes, UTF_8))
  println(xmlCodec.decode(xmlBytes) == Right(catalog))

  val toonBytes = toonCodec.encode(catalog)
  println(new String(toonBytes, UTF_8))
  println(toonCodec.decode(toonBytes) == Right(catalog))

  val msgBytes = msgCodec.encode(catalog)
  println(s"MessagePack: ${msgBytes.length} bytes, round trip: ${msgCodec.decode(msgBytes) == Right(catalog)}")

  val avroBytes = avroCodec.encode(catalog)
  println(s"Avro: ${avroBytes.length} bytes, round trip: ${avroCodec.decode(avroBytes) == Right(catalog)}")

  // CSV accepts only flat records, so export a BookRow per book
  case class BookRow(title: String, author: String, year: Int)

  object BookRow {
    implicit val schema: Schema[BookRow] = Schema.derived
  }

  val rows     = catalog.books.map(book => BookRow(book.title, book.author, book.year))
  val csvCodec = BookRow.schema.derive(CsvFormat)

  def toCsvLine(row: BookRow): String = {
    val buffer = CharBuffer.allocate(256)
    csvCodec.encode(row, buffer)
    buffer.flip()
    buffer.toString
  }

  val csvLines = rows.map(toCsvLine)
  println(csvCodec.headerNames.mkString(","))
  csvLines.foreach(line => println(line.stripLineEnd))
  println(csvLines.map(line => csvCodec.decode(CharBuffer.wrap(line))) == rows.map(Right(_)))
}
```

## 12. Running the Examples

All examples in this tutorial have corresponding runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

Run them in order to follow the tutorial step by step.

### Model the Catalog and Encode to JSON

This example models the catalog, derives its JSON codec with `Schema#derive`, prints the JSON and checks the round trip.

<details>
  <summary>schema-examples/src/main/scala/onetypemanyformats/Concept1JsonExample.scala</summary>

```scala title="schema-examples/src/main/scala/onetypemanyformats/Concept1JsonExample.scala" showLineNumbers
package onetypemanyformats

/**
 * One Type, Many Formats — Concept 1: Model the Catalog and Encode to JSON
 *
 * Models the catalog, derives its JSON codec with Schema#derive, prints the
 * JSON and checks the round trip.
 *
 * Run with: cd schema-examples && sbt "runMain
 * onetypemanyformats.Concept1JsonExample"
 */
object Concept1JsonExample extends App {

  case class Book(title: String, author: String, year: Int, tags: List[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalog = Catalog(
    List(
      Book("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", 2024, List("zio", "effects")),
      Book("Programming in Scala", "Martin Odersky, Lex Spoon, Bill Venners", 2008, List("scala", "fundamentals")),
      Book("Functional Programming in Scala", "Paul Chiusano, Rúnar Bjarnason", 2014, List("scala", "fp"))
    )
  )

  val jsonCodec = Catalog.schema.derive(JsonFormat)

  val jsonBytes = jsonCodec.encode(catalog)

  println(new String(jsonBytes, UTF_8))

  println(jsonCodec.decode(jsonBytes) == Right(catalog))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain onetypemanyformats.Concept1JsonExample"
```

### Swap In YAML

This example derives the YAML codec from the same schema, prints the YAML and checks the round trip.

<details>
  <summary>schema-examples/src/main/scala/onetypemanyformats/Concept2YamlExample.scala</summary>

```scala title="schema-examples/src/main/scala/onetypemanyformats/Concept2YamlExample.scala" showLineNumbers
package onetypemanyformats

/**
 * One Type, Many Formats — Concept 2: Swap In YAML
 *
 * Derives the YAML codec from the same schema, prints the YAML and checks the
 * round trip.
 *
 * Run with: cd schema-examples && sbt "runMain
 * onetypemanyformats.Concept2YamlExample"
 */
object Concept2YamlExample extends App {

  case class Book(title: String, author: String, year: Int, tags: List[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalog = Catalog(
    List(
      Book("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", 2024, List("zio", "effects")),
      Book("Programming in Scala", "Martin Odersky, Lex Spoon, Bill Venners", 2008, List("scala", "fundamentals")),
      Book("Functional Programming in Scala", "Paul Chiusano, Rúnar Bjarnason", 2014, List("scala", "fp"))
    )
  )

  val yamlCodec = Catalog.schema.derive(YamlFormat)

  val yamlBytes = yamlCodec.encode(catalog)

  println(new String(yamlBytes, UTF_8))

  println(yamlCodec.decode(yamlBytes) == Right(catalog))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain onetypemanyformats.Concept2YamlExample"
```

### Swap In XML

This example derives the XML codec, prints the XML with its `<Catalog>` root element and checks the round trip.

<details>
  <summary>schema-examples/src/main/scala/onetypemanyformats/Concept3XmlExample.scala</summary>

```scala title="schema-examples/src/main/scala/onetypemanyformats/Concept3XmlExample.scala" showLineNumbers
package onetypemanyformats

/**
 * One Type, Many Formats — Concept 3: Swap In XML
 *
 * Derives the XML codec, prints the XML with its <Catalog> root element and
 * checks the round trip.
 *
 * Run with: cd schema-examples && sbt "runMain
 * onetypemanyformats.Concept3XmlExample"
 */
object Concept3XmlExample extends App {

  case class Book(title: String, author: String, year: Int, tags: List[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalog = Catalog(
    List(
      Book("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", 2024, List("zio", "effects")),
      Book("Programming in Scala", "Martin Odersky, Lex Spoon, Bill Venners", 2008, List("scala", "fundamentals")),
      Book("Functional Programming in Scala", "Paul Chiusano, Rúnar Bjarnason", 2014, List("scala", "fp"))
    )
  )

  val xmlCodec = Catalog.schema.derive(XmlFormat)

  val xmlBytes = xmlCodec.encode(catalog)

  println(new String(xmlBytes, UTF_8))

  println(xmlCodec.decode(xmlBytes) == Right(catalog))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain onetypemanyformats.Concept3XmlExample"
```

### Swap In TOON

This example derives the TOON codec, prints the TOON text and checks the round trip.

<details>
  <summary>schema-examples/src/main/scala/onetypemanyformats/Concept4ToonExample.scala</summary>

```scala title="schema-examples/src/main/scala/onetypemanyformats/Concept4ToonExample.scala" showLineNumbers
package onetypemanyformats

/**
 * One Type, Many Formats — Concept 4: Swap In TOON
 *
 * Derives the TOON codec, prints the TOON text and checks the round trip.
 *
 * Run with: cd schema-examples && sbt "runMain
 * onetypemanyformats.Concept4ToonExample"
 */
object Concept4ToonExample extends App {

  case class Book(title: String, author: String, year: Int, tags: List[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalog = Catalog(
    List(
      Book("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", 2024, List("zio", "effects")),
      Book("Programming in Scala", "Martin Odersky, Lex Spoon, Bill Venners", 2008, List("scala", "fundamentals")),
      Book("Functional Programming in Scala", "Paul Chiusano, Rúnar Bjarnason", 2014, List("scala", "fp"))
    )
  )

  val toonCodec = Catalog.schema.derive(ToonFormat)

  val toonBytes = toonCodec.encode(catalog)

  println(new String(toonBytes, UTF_8))

  println(toonCodec.decode(toonBytes) == Right(catalog))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain onetypemanyformats.Concept4ToonExample"
```

### Swap In MessagePack

This example derives the MessagePack codec, compares its size with JSON, prints the first bytes in hexadecimal and checks the round trip.

<details>
  <summary>schema-examples/src/main/scala/onetypemanyformats/Concept5MessagePackExample.scala</summary>

```scala title="schema-examples/src/main/scala/onetypemanyformats/Concept5MessagePackExample.scala" showLineNumbers
package onetypemanyformats

/**
 * One Type, Many Formats — Concept 5: Swap In MessagePack
 *
 * Derives the MessagePack codec, compares its size with JSON, prints the first
 * bytes in hexadecimal and checks the round trip.
 *
 * Run with: cd schema-examples && sbt "runMain
 * onetypemanyformats.Concept5MessagePackExample"
 */
object Concept5MessagePackExample extends App {

  case class Book(title: String, author: String, year: Int, tags: List[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalog = Catalog(
    List(
      Book("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", 2024, List("zio", "effects")),
      Book("Programming in Scala", "Martin Odersky, Lex Spoon, Bill Venners", 2008, List("scala", "fundamentals")),
      Book("Functional Programming in Scala", "Paul Chiusano, Rúnar Bjarnason", 2014, List("scala", "fp"))
    )
  )

  val jsonCodec = Catalog.schema.derive(JsonFormat)

  val jsonBytes = jsonCodec.encode(catalog)

  val msgpackCodec = Catalog.schema.derive(MessagePackFormat)

  val msgpackBytes = msgpackCodec.encode(catalog)

  def hexPrefix(bytes: Array[Byte]): String =
    bytes.take(16).map(b => f"$b%02x").mkString(" ")

  println(s"MessagePack: ${msgpackBytes.length} bytes, JSON: ${jsonBytes.length} bytes")

  println(hexPrefix(msgpackBytes))

  println(msgpackCodec.decode(msgpackBytes) == Right(catalog))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain onetypemanyformats.Concept5MessagePackExample"
```

### Swap In Avro

This example derives the Avro codec, compares its size with MessagePack and JSON, prints the first bytes in hexadecimal and checks the round trip.

<details>
  <summary>schema-examples/src/main/scala/onetypemanyformats/Concept6AvroExample.scala</summary>

```scala title="schema-examples/src/main/scala/onetypemanyformats/Concept6AvroExample.scala" showLineNumbers
package onetypemanyformats

/**
 * One Type, Many Formats — Concept 6: Swap In Avro
 *
 * Derives the Avro codec, compares its size with MessagePack and JSON, prints
 * the first bytes in hexadecimal and checks the round trip.
 *
 * Run with: cd schema-examples && sbt "runMain
 * onetypemanyformats.Concept6AvroExample"
 */
object Concept6AvroExample extends App {

  case class Book(title: String, author: String, year: Int, tags: List[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalog = Catalog(
    List(
      Book("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", 2024, List("zio", "effects")),
      Book("Programming in Scala", "Martin Odersky, Lex Spoon, Bill Venners", 2008, List("scala", "fundamentals")),
      Book("Functional Programming in Scala", "Paul Chiusano, Rúnar Bjarnason", 2014, List("scala", "fp"))
    )
  )

  val jsonCodec = Catalog.schema.derive(JsonFormat)

  val jsonBytes = jsonCodec.encode(catalog)

  val msgpackCodec = Catalog.schema.derive(MessagePackFormat)

  val msgpackBytes = msgpackCodec.encode(catalog)

  def hexPrefix(bytes: Array[Byte]): String =
    bytes.take(16).map(b => f"$b%02x").mkString(" ")

  val avroCodec = Catalog.schema.derive(AvroFormat)

  val avroBytes = avroCodec.encode(catalog)

  println(
    s"Avro: ${avroBytes.length} bytes, MessagePack: ${msgpackBytes.length} bytes, JSON: ${jsonBytes.length} bytes"
  )

  println(hexPrefix(avroBytes))

  println(avroCodec.decode(avroBytes) == Right(catalog))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain onetypemanyformats.Concept6AvroExample"
```

### Flatten the Catalog for CSV

This example shows why a `Book` with tags can't become CSV, derives the codec for the flat `BookRow`, prints the CSV table and checks the round trip.

<details>
  <summary>schema-examples/src/main/scala/onetypemanyformats/Concept7CsvExample.scala</summary>

```scala title="schema-examples/src/main/scala/onetypemanyformats/Concept7CsvExample.scala" showLineNumbers
package onetypemanyformats

/**
 * One Type, Many Formats — Concept 7: Flatten the Catalog for CSV
 *
 * Shows why a Book with tags cannot become CSV, derives the codec for the flat
 * BookRow, prints the CSV table and checks the round trip.
 *
 * Run with: cd schema-examples && sbt "runMain
 * onetypemanyformats.Concept7CsvExample"
 */
object Concept7CsvExample extends App {

  case class Book(title: String, author: String, year: Int, tags: List[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalog = Catalog(
    List(
      Book("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", 2024, List("zio", "effects")),
      Book("Programming in Scala", "Martin Odersky, Lex Spoon, Bill Venners", 2008, List("scala", "fundamentals")),
      Book("Functional Programming in Scala", "Paul Chiusano, Rúnar Bjarnason", 2014, List("scala", "fp"))
    )
  )

  println(scala.util.Try(Book.schema.derive(CsvFormat)))

  case class Maybe(title: String, subtitle: Option[String])

  object Maybe {
    implicit val schema: Schema[Maybe] = Schema.derived
  }

  println(scala.util.Try(Maybe.schema.derive(CsvFormat)))

  case class BookRow(title: String, author: String, year: Int)

  object BookRow {
    implicit val schema: Schema[BookRow] = Schema.derived
  }

  val rows = catalog.books.map(book => BookRow(book.title, book.author, book.year))

  val csvCodec = BookRow.schema.derive(CsvFormat)

  def toCsvLine(row: BookRow): String = {
    val buffer = CharBuffer.allocate(256)
    csvCodec.encode(row, buffer)
    buffer.flip()
    buffer.toString
  }

  val csvLines = rows.map(toCsvLine)

  println(csvCodec.headerNames.mkString(","))

  csvLines.foreach(line => println(line.stripLineEnd))

  println(csvLines.map(line => csvCodec.decode(CharBuffer.wrap(line))) == rows.map(Right(_)))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain onetypemanyformats.Concept7CsvExample"
```

### The Complete Example

The complete example from "Putting It Together" runs everything from the tutorial in one program:

```bash
sbt "runMain onetypemanyformats.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## 13. What You've Learned

In this tutorial, you learned:

- How one `Schema` drives a codec for every format: adding a format means adding its module, importing its format object and deriving the codec with one `Schema#derive` call
- How to encode a value with a codec, decode it back, and check that the round trip returns the original value
- What JSON, YAML, XML and TOON look like for the same data, and how the binary formats MessagePack and Avro trade readability for size
- Why CSV accepts only flat records, and how a flat `BookRow` type turns a nested model into a table

The table below sums up the formats in this tutorial:

| Format      | Format object       | Module                          | Output | Notes                                 |
| ----------- | ------------------- | ------------------------------- | ------ | ------------------------------------- |
| JSON        | `JsonFormat`        | `zio-blocks-schema`             | Bytes  | Built into the schema module          |
| YAML        | `YamlFormat`        | `zio-blocks-schema-yaml`        | Bytes  | Readable configuration format         |
| XML         | `XmlFormat`         | `zio-blocks-schema-xml`         | Bytes  | Root element named after the type     |
| TOON        | `ToonFormat`        | `zio-blocks-schema-toon`        | Bytes  | Compact text for language models      |
| MessagePack | `MessagePackFormat` | `zio-blocks-schema-messagepack` | Bytes  | Binary, keeps field names             |
| Avro        | `AvroFormat`        | `zio-blocks-schema-avro`        | Bytes  | Binary, no field names, JVM only      |
| CSV         | `CsvFormat`         | `zio-blocks-schema-csv`         | Chars  | Flat records only, one row per encode |

You now know how to take one Scala type to any format ZIO Blocks supports. The next step is to tune a format's output — its field names, indentation or array layout — with that format's configuration.

## 14. Where to Go Next

- **Want an overview of every format?** The [Built-in Codecs](../reference/schema/built-in-codecs/index.md) page lists each format with its codec type, MIME type and module.
- **Curious how format objects work?** Read [Format](../reference/schema/format.md) for the `Format` abstraction behind `Schema#derive`.
- **Working with YAML in depth?** Read [How to Read and Write YAML Documents](./read-and-write-yaml.md) to read YAML into case classes, write CI-style YAML and guard against YAML the reader silently misreads.
- **Ready to tune one format?** Each format has its own reference page: [JSON](../reference/schema/built-in-codecs/json/index.md), [YAML](../reference/schema/built-in-codecs/yaml.md), [XML](../reference/schema/built-in-codecs/xml.md), [TOON](../reference/schema/built-in-codecs/toon.md), [MessagePack](../reference/schema/built-in-codecs/messagepack.md), [Avro](../reference/schema/built-in-codecs/avro.md) and [CSV](../reference/schema/built-in-codecs/csv.md).
- **Need Thrift or BSON?** Read [Thrift](../reference/schema/built-in-codecs/thrift.md) and [BSON](../reference/schema/built-in-codecs/bson.md).
- **Want to know how derivation works?** Read [Type Class Derivation](../reference/schema/type-class-derivation.md) to see how a `Deriver` turns a schema into a codec.

---

## How to Override the Codec for One Type Everywhere

In this guide, we make every derived codec in a small Scala bookstore write `java.time.LocalDate` as `dd/MM/yyyy` instead of the ISO-8601 default. We write one custom codec for `LocalDate`, register it once on the deriver with `withInstance`, and derive every other codec through that deriver. Nested types pick up the override automatically, and the same recipe works for formats other than JSON.

We cover:

- Writing a custom `JsonCodec[LocalDate]`
- Registering it once on `JsonCodecDeriver`
- Deriving codecs for nested types through that deriver
- Doing the same for TOON, a second format
- Narrower scopes: one field, or one derivation
- Choosing the right tool

## The Problem

The derived JSON codec writes a `LocalDate` as an ISO-8601 string such as `2025-03-14`. Our bookstore's partner system expects `14/03/2025`, and it expects that format in every message we send it, whichever case class the date sits in.

Without a way to change one type's codec globally, every type that contains a date needs its own encoding code, and every one of them repeats the date format. Here is that approach written by hand:

```scala

val partnerDateFormat = DateTimeFormatter.ofPattern("dd/MM/yyyy")

case class Order(title: String, orderedOn: LocalDate, price: BigDecimal)
case class Shelf(owner: String, openedOn: LocalDate, orders: List[Order])

def orderToJson(order: Order): String =
  s"""{"title":"${order.title}","orderedOn":"${partnerDateFormat.format(order.orderedOn)}","price":${order.price}}"""

def shelfToJson(shelf: Shelf): String =
  s"""{"owner":"${shelf.owner}","openedOn":"${partnerDateFormat.format(shelf.openedOn)}","orders":[${shelf.orders
      .map(orderToJson)
      .mkString(",")}]}"""
```

This approach doesn't scale. Each new case class with a date needs another hand-written function, the decoders need the same treatment, and nothing stops one function from forgetting the custom format. It also covers JSON only; a second wire format means writing it all again.

In ZIO Blocks, a deriver builds a codec for every type from its schema. We give the deriver one custom codec for `LocalDate`, and it uses that codec wherever a `LocalDate` appears.

## Prerequisites

This guide builds on [Your First Schema](./your-first-schema.md): you should know how to derive a `Schema` and round-trip a value through JSON.

Add the schema module, and the TOON module for the second-format section, to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-schema"      % "0.0.56",
  "dev.zio" %% "zio-blocks-schema-toon" % "0.0.56"
)

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-schema"      % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema-toon" % "0.0.56"
)
```

We use these imports throughout:

```scala

```

## The Bookstore Model

Our model has two case classes. An `Order` records one book purchase, and a `Shelf` groups a reader's orders. Both contain a `LocalDate`, and `Shelf` nests `Order` inside a list:

```scala
case class Order(title: String, orderedOn: LocalDate, price: BigDecimal)

object Order {
  implicit val schema: Schema[Order] = Schema.derived
}

case class Shelf(owner: String, openedOn: LocalDate, orders: List[Order])

object Shelf {
  implicit val schema: Schema[Shelf] = Schema.derived
}

val shelf = Shelf(
  owner = "Ada",
  openedOn = LocalDate.of(2025, 1, 10),
  orders = List(
    Order("Zionomicon", LocalDate.of(2025, 3, 14), BigDecimal("39.99")),
    Order("Programming in Scala", LocalDate.of(2025, 4, 2), BigDecimal("54.50")),
    Order("Functional Programming in Scala", LocalDate.of(2025, 5, 21), BigDecimal("44.00"))
  )
)
```

The order dates and prices are sample data. With the default codec, both dates come out in ISO-8601 form:

```scala
println(shelf.toJsonString)
// {"owner":"Ada","openedOn":"2025-01-10","orders":[{"title":"Zionomicon","orderedOn":"2025-03-14","price":39.99},{"title":"Programming in Scala","orderedOn":"2025-04-02","price":54.50},{"title":"Functional Programming in Scala","orderedOn":"2025-05-21","price":44.00}]}
```

## Writing the Codec for `LocalDate`

A `JsonCodec[A]` needs two methods: `decodeValue(in: JsonReader)` reads a value from the streaming reader, and `encodeValue(x, out: JsonWriter)` writes it. Two more methods convert to and from the `Json` value type; their defaults throw, so we override them too, which keeps the codec usable with `Json` values:

```scala
val partnerDateFormat = DateTimeFormatter.ofPattern("dd/MM/yyyy")

val partnerDateCodec: JsonCodec[LocalDate] = new JsonCodec[LocalDate] {
  private def parse(s: String): LocalDate =
    try LocalDate.parse(s, partnerDateFormat)
    catch { case _: DateTimeParseException => error(s"expected a dd/MM/yyyy date, got: $s") }

  def decodeValue(in: JsonReader): LocalDate =
    parse(in.readString())

  def encodeValue(x: LocalDate, out: JsonWriter): Unit =
    out.writeVal(partnerDateFormat.format(x))

  override def decodeValue(json: Json): LocalDate = json match {
    case s: Json.String => parse(s.value)
    case _              => error("expected a JSON string")
  }

  override def encodeValue(x: LocalDate): Json =
    Json.String(partnerDateFormat.format(x))
}
```

The codec reads and writes a JSON string, the same JSON type the default `LocalDate` codec uses; only the text inside changes. `error` is a method on `JsonCodec` that reports a decode failure, so a malformed date becomes a `SchemaError` instead of an uncaught exception.

## Registering the Codec on the Deriver

`JsonCodecDeriver` is the deriver behind the default JSON codecs. Its `withInstance` method returns a new deriver that uses our codec for `LocalDate` and derives everything else as before:

```scala
val partnerJson = JsonCodecDeriver.withInstance[LocalDate](partnerDateCodec)
```

`withInstance[A]` finds the type through an implicit `TypeId[A]`, which ZIO Blocks provides for `LocalDate` and the other built-in types. The original `JsonCodecDeriver` is unchanged, so the default codecs keep working.

To use a money type as well, chain another call, for example `.withInstance[BigDecimal](priceAsStringCodec)`. Each call adds one override.

## Deriving Codecs Through the Deriver

`Schema#deriving` starts a derivation with a given deriver, and `.derive` builds the codec. We derive one codec per top-level type we send:

```scala
val orderCodec: JsonCodec[Order] = Schema[Order].deriving(partnerJson).derive
val shelfCodec: JsonCodec[Shelf] = Schema[Shelf].deriving(partnerJson).derive
```

The `Shelf` codec uses the custom date format for its own `openedOn` field and for the `orderedOn` field of every `Order` inside the list:

```scala
println(orderCodec.encodeToString(shelf.orders.head))
// {"title":"Zionomicon","orderedOn":"14/03/2025","price":39.99}
println(shelfCodec.encodeToString(shelf))
// {"owner":"Ada","openedOn":"10/01/2025","orders":[{"title":"Zionomicon","orderedOn":"14/03/2025","price":39.99},{"title":"Programming in Scala","orderedOn":"02/04/2025","price":54.50},{"title":"Functional Programming in Scala","orderedOn":"21/05/2025","price":44.00}]}
```

The deriver matches the override by `TypeId` at every level of the schema, so we never mention `Order` or its fields when we register the override. Decoding uses the same codec, so the round trip returns the original value:

```scala
println(shelfCodec.decode(shelfCodec.encodeToString(shelf)) == Right(shelf))
// true
```

:::warning
`toJsonString` and `fromJson` always use the default codec from `Schema#jsonCodec`, not a deriver you configured. Call `encodeToString` and `decode` on the codec you derived. An `implicit JsonCodec[LocalDate]` in scope doesn't change derivation either: the deriver never searches for codec implicits, so the override must be registered with `withInstance`.
:::

## Doing the Same for TOON

Every format in ZIO Blocks has its own deriver, and `withInstance` is defined on the shared `Deriver` trait, so the recipe is the same for each one. Here it is for [TOON](../reference/schema/built-in-codecs/toon.md), a compact text format. A `ToonCodec` has the same `decodeValue`/`encodeValue` pair, over `ToonReader` and `ToonWriter`:

```scala

val partnerDateToonCodec: ToonCodec[LocalDate] = new ToonCodec[LocalDate] {
  def decodeValue(in: ToonReader): LocalDate = {
    in.skipBlankLines()
    val s = in.readString()
    try LocalDate.parse(s, partnerDateFormat)
    catch { case _: DateTimeParseException => error(s"expected a dd/MM/yyyy date, got: $s") }
  }

  def encodeValue(x: LocalDate, out: ToonWriter): Unit =
    out.writeString(partnerDateFormat.format(x))
}

val partnerToon = ToonCodecDeriver.withInstance[LocalDate](partnerDateToonCodec)

val shelfToonCodec: ToonCodec[Shelf] = Schema[Shelf].deriving(partnerToon).derive
```

`skipBlankLines()` mirrors the built-in TOON `LocalDate` codec, which skips blank lines before reading a value. The TOON output carries the custom date format for both `Shelf` and the nested orders:

```scala
println(shelfToonCodec.encodeToString(shelf))
// owner: Ada
// openedOn: 10/01/2025
// orders[3]:
//   - title: Zionomicon
//     orderedOn: 14/03/2025
//     price: 39.99
//   - title: Programming in Scala
//     orderedOn: 02/04/2025
//     price: 54.5
//   - title: Functional Programming in Scala
//     orderedOn: 21/05/2025
//     price: 44
```

## Narrower Scope: One Field

Sometimes only one field should use the custom format. The second `withInstance` overload takes the parent type's `TypeId`, the field name and the codec. Here only `Order.orderedOn` changes, and `Shelf.openedOn` keeps the ISO-8601 format:

```scala
val orderDateOnly = JsonCodecDeriver.withInstance(TypeId.of[Order], "orderedOn", partnerDateCodec)

val shelfCodecOrderDateOnly: JsonCodec[Shelf] = Schema[Shelf].deriving(orderDateOnly).derive
```

```scala
println(shelfCodecOrderDateOnly.encodeToString(shelf))
// {"owner":"Ada","openedOn":"2025-01-10","orders":[{"title":"Zionomicon","orderedOn":"14/03/2025","price":39.99},{"title":"Programming in Scala","orderedOn":"02/04/2025","price":54.50},{"title":"Functional Programming in Scala","orderedOn":"21/05/2025","price":44.00}]}
```

The field type isn't checked against the codec at compile time, so make sure the codec's type matches the field's type.

## Narrower Scope: One Derivation

To override for a single derivation without building a new deriver, call `instance` on the `DerivationBuilder` that `deriving` returns. It accepts a `TypeId` (with or without a field name) or an optic:

```scala
val oneOffShelfCodec: JsonCodec[Shelf] =
  Schema[Shelf]
    .deriving(JsonCodecDeriver)
    .instance(TypeId.of[LocalDate], partnerDateCodec)
    .derive
```

```scala
println(oneOffShelfCodec.encodeToString(shelf))
// {"owner":"Ada","openedOn":"10/01/2025","orders":[{"title":"Zionomicon","orderedOn":"14/03/2025","price":39.99},{"title":"Programming in Scala","orderedOn":"02/04/2025","price":54.50},{"title":"Functional Programming in Scala","orderedOn":"21/05/2025","price":44.00}]}
```

When the deriver and the builder both override the same type or field, the deriver's override wins. See [Custom Type-class Instances](../reference/schema/type-class-derivation.md#custom-type-class-instances) for the optic overload and the full resolution order.

The deriver also accepts field modifiers through `withModifier`, for example `withModifier(TypeId.of[Order], "title", Modifier.rename("bookTitle"))`. For JSON field names, ADT encoding and optional fields, see [How to Customize JSON Field Names, ADT Encoding and Optional Fields](./customize-json-encoding.md).

## Which Tool When

| Goal                                                      | Tool                                                                             | Scope                                                |
|-----------------------------------------------------------|----------------------------------------------------------------------------------|------------------------------------------------------|
| One type encoded the same way in every derived codec      | `Deriver#withInstance[A](codec)`                                                 | Every codec derived through that deriver             |
| One field of one type encoded differently                 | `Deriver#withInstance(TypeId.of[P], "field", codec)`                             | That field, in every codec from that deriver         |
| A one-off override for a single codec                     | `DerivationBuilder#instance(typeId or optic, codec)`                             | The one `.derive` call                               |
| Full control of a whole type's wire format, used directly | A hand-written `JsonCodec[A]`, called directly or plugged in with `withInstance` | Only where you call it; derivation ignores implicits |

## Putting It Together

This example combines everything from the guide: a custom `LocalDate` codec for JSON and TOON, registered once on each deriver, then used for the nested bookstore model and checked with a round trip:

```scala title="schema-examples/src/main/scala/overridecodecforonetype/CompleteExample.scala"
package overridecodecforonetype

/**
 * How to Override the Codec for One Type Everywhere — Putting It Together
 *
 * Combines custom LocalDate codecs for JSON and TOON, registered once on each
 * deriver, for the nested bookstore model with round-trip checks.
 *
 * Run with: cd schema-examples && sbt "runMain
 * overridecodecforonetype.CompleteExample"
 */
object CompleteExample extends App {

  case class Order(title: String, orderedOn: LocalDate, price: BigDecimal)

  object Order {
    implicit val schema: Schema[Order] = Schema.derived
  }

  case class Shelf(owner: String, openedOn: LocalDate, orders: List[Order])

  object Shelf {
    implicit val schema: Schema[Shelf] = Schema.derived
  }

  val shelf = Shelf(
    owner = "Ada",
    openedOn = LocalDate.of(2025, 1, 10),
    orders = List(
      Order("Zionomicon", LocalDate.of(2025, 3, 14), BigDecimal("39.99")),
      Order("Programming in Scala", LocalDate.of(2025, 4, 2), BigDecimal("54.50")),
      Order("Functional Programming in Scala", LocalDate.of(2025, 5, 21), BigDecimal("44.00"))
    )
  )

  val partnerDateFormat = DateTimeFormatter.ofPattern("dd/MM/yyyy")

  val partnerDateCodec: JsonCodec[LocalDate] = new JsonCodec[LocalDate] {
    private def parse(s: String): LocalDate =
      try LocalDate.parse(s, partnerDateFormat)
      catch { case _: DateTimeParseException => error(s"expected a dd/MM/yyyy date, got: $s") }

    def decodeValue(in: JsonReader): LocalDate =
      parse(in.readString())

    def encodeValue(x: LocalDate, out: JsonWriter): Unit =
      out.writeVal(partnerDateFormat.format(x))

    override def decodeValue(json: Json): LocalDate = json match {
      case s: Json.String => parse(s.value)
      case _              => error("expected a JSON string")
    }

    override def encodeValue(x: LocalDate): Json =
      Json.String(partnerDateFormat.format(x))
  }

  val partnerDateToonCodec: ToonCodec[LocalDate] = new ToonCodec[LocalDate] {
    def decodeValue(in: ToonReader): LocalDate = {
      in.skipBlankLines()
      val s = in.readString()
      try LocalDate.parse(s, partnerDateFormat)
      catch { case _: DateTimeParseException => error(s"expected a dd/MM/yyyy date, got: $s") }
    }

    def encodeValue(x: LocalDate, out: ToonWriter): Unit =
      out.writeString(partnerDateFormat.format(x))
  }

  val partnerJson = JsonCodecDeriver.withInstance[LocalDate](partnerDateCodec)
  val partnerToon = ToonCodecDeriver.withInstance[LocalDate](partnerDateToonCodec)

  val shelfCodec: JsonCodec[Shelf]     = Schema[Shelf].deriving(partnerJson).derive
  val shelfToonCodec: ToonCodec[Shelf] = Schema[Shelf].deriving(partnerToon).derive

  println(s"default JSON: ${shelf.toJsonString}")
  println(s"partner JSON: ${shelfCodec.encodeToString(shelf)}")
  println(s"partner TOON:\n${shelfToonCodec.encodeToString(shelf)}")
  println(s"JSON round trip: ${shelfCodec.decode(shelfCodec.encodeToString(shelf)) == Right(shelf)}")
  println(s"TOON round trip: ${shelfToonCodec.decode(shelfToonCodec.encodeToString(shelf)) == Right(shelf)}")
}
```

## Running the Examples

All examples in this guide have corresponding runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

Run them in order to follow the guide step by step.

### Writing the Codec for `LocalDate`

This example writes the custom `JsonCodec[LocalDate]` and uses it on its own to encode and decode a single date.

<details>
  <summary>schema-examples/src/main/scala/overridecodecforonetype/Step1WriteCodecExample.scala</summary>

```scala title="schema-examples/src/main/scala/overridecodecforonetype/Step1WriteCodecExample.scala" showLineNumbers
package overridecodecforonetype

/**
 * How to Override the Codec for One Type Everywhere — Writing the Codec for LocalDate
 *
 * Writes a custom JsonCodec[LocalDate] for dd/MM/yyyy and uses it on its own.
 *
 * Run with: cd schema-examples && sbt "runMain
 * overridecodecforonetype.Step1WriteCodecExample"
 */
object Step1WriteCodecExample extends App {

  val partnerDateFormat = DateTimeFormatter.ofPattern("dd/MM/yyyy")

  val partnerDateCodec: JsonCodec[LocalDate] = new JsonCodec[LocalDate] {
    private def parse(s: String): LocalDate =
      try LocalDate.parse(s, partnerDateFormat)
      catch { case _: DateTimeParseException => error(s"expected a dd/MM/yyyy date, got: $s") }

    def decodeValue(in: JsonReader): LocalDate =
      parse(in.readString())

    def encodeValue(x: LocalDate, out: JsonWriter): Unit =
      out.writeVal(partnerDateFormat.format(x))

    override def decodeValue(json: Json): LocalDate = json match {
      case s: Json.String => parse(s.value)
      case _              => error("expected a JSON string")
    }

    override def encodeValue(x: LocalDate): Json =
      Json.String(partnerDateFormat.format(x))
  }

  println(partnerDateCodec.encodeToString(LocalDate.of(2025, 3, 14)))
  println(partnerDateCodec.decode("\"14/03/2025\""))
  println(partnerDateCodec.decode("\"2025-03-14\""))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain overridecodecforonetype.Step1WriteCodecExample"
```

### Registering the Codec and Deriving Nested Types

This example registers the codec on `JsonCodecDeriver`, derives codecs for `Order` and `Shelf`, and prints both next to the default output.

<details>
  <summary>schema-examples/src/main/scala/overridecodecforonetype/Step2DeriveEverywhereExample.scala</summary>

```scala title="schema-examples/src/main/scala/overridecodecforonetype/Step2DeriveEverywhereExample.scala" showLineNumbers
package overridecodecforonetype

/**
 * How to Override the Codec for One Type Everywhere — Registering the Codec and Deriving Nested Types
 *
 * Registers the custom date codec on JsonCodecDeriver and derives Order and Shelf codecs through it.
 *
 * Run with: cd schema-examples && sbt "runMain
 * overridecodecforonetype.Step2DeriveEverywhereExample"
 */
object Step2DeriveEverywhereExample extends App {

  case class Order(title: String, orderedOn: LocalDate, price: BigDecimal)

  object Order {
    implicit val schema: Schema[Order] = Schema.derived
  }

  case class Shelf(owner: String, openedOn: LocalDate, orders: List[Order])

  object Shelf {
    implicit val schema: Schema[Shelf] = Schema.derived
  }

  val shelf = Shelf(
    owner = "Ada",
    openedOn = LocalDate.of(2025, 1, 10),
    orders = List(
      Order("Zionomicon", LocalDate.of(2025, 3, 14), BigDecimal("39.99")),
      Order("Programming in Scala", LocalDate.of(2025, 4, 2), BigDecimal("54.50")),
      Order("Functional Programming in Scala", LocalDate.of(2025, 5, 21), BigDecimal("44.00"))
    )
  )

  val partnerDateFormat = DateTimeFormatter.ofPattern("dd/MM/yyyy")

  val partnerDateCodec: JsonCodec[LocalDate] = new JsonCodec[LocalDate] {
    private def parse(s: String): LocalDate =
      try LocalDate.parse(s, partnerDateFormat)
      catch { case _: DateTimeParseException => error(s"expected a dd/MM/yyyy date, got: $s") }

    def decodeValue(in: JsonReader): LocalDate =
      parse(in.readString())

    def encodeValue(x: LocalDate, out: JsonWriter): Unit =
      out.writeVal(partnerDateFormat.format(x))

    override def decodeValue(json: Json): LocalDate = json match {
      case s: Json.String => parse(s.value)
      case _              => error("expected a JSON string")
    }

    override def encodeValue(x: LocalDate): Json =
      Json.String(partnerDateFormat.format(x))
  }

  val partnerJson = JsonCodecDeriver.withInstance[LocalDate](partnerDateCodec)

  val orderCodec: JsonCodec[Order] = Schema[Order].deriving(partnerJson).derive
  val shelfCodec: JsonCodec[Shelf] = Schema[Shelf].deriving(partnerJson).derive

  println(s"default:  ${shelf.toJsonString}")
  println(s"order:    ${orderCodec.encodeToString(shelf.orders.head)}")
  println(s"shelf:    ${shelfCodec.encodeToString(shelf)}")
  println(s"round trip: ${shelfCodec.decode(shelfCodec.encodeToString(shelf)) == Right(shelf)}")
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain overridecodecforonetype.Step2DeriveEverywhereExample"
```

### Doing the Same for TOON

This example applies the same recipe to `ToonCodecDeriver`.

<details>
  <summary>schema-examples/src/main/scala/overridecodecforonetype/Step3ToonExample.scala</summary>

```scala title="schema-examples/src/main/scala/overridecodecforonetype/Step3ToonExample.scala" showLineNumbers
package overridecodecforonetype

/**
 * How to Override the Codec for One Type Everywhere — Doing the Same for TOON
 *
 * Registers a custom TOON date codec on ToonCodecDeriver and derives a Shelf codec through it.
 *
 * Run with: cd schema-examples && sbt "runMain
 * overridecodecforonetype.Step3ToonExample"
 */
object Step3ToonExample extends App {

  case class Order(title: String, orderedOn: LocalDate, price: BigDecimal)

  object Order {
    implicit val schema: Schema[Order] = Schema.derived
  }

  case class Shelf(owner: String, openedOn: LocalDate, orders: List[Order])

  object Shelf {
    implicit val schema: Schema[Shelf] = Schema.derived
  }

  val shelf = Shelf(
    owner = "Ada",
    openedOn = LocalDate.of(2025, 1, 10),
    orders = List(
      Order("Zionomicon", LocalDate.of(2025, 3, 14), BigDecimal("39.99")),
      Order("Programming in Scala", LocalDate.of(2025, 4, 2), BigDecimal("54.50")),
      Order("Functional Programming in Scala", LocalDate.of(2025, 5, 21), BigDecimal("44.00"))
    )
  )

  val partnerDateFormat = DateTimeFormatter.ofPattern("dd/MM/yyyy")

  val partnerDateToonCodec: ToonCodec[LocalDate] = new ToonCodec[LocalDate] {
    def decodeValue(in: ToonReader): LocalDate = {
      in.skipBlankLines()
      val s = in.readString()
      try LocalDate.parse(s, partnerDateFormat)
      catch { case _: DateTimeParseException => error(s"expected a dd/MM/yyyy date, got: $s") }
    }

    def encodeValue(x: LocalDate, out: ToonWriter): Unit =
      out.writeString(partnerDateFormat.format(x))
  }

  val partnerToon = ToonCodecDeriver.withInstance[LocalDate](partnerDateToonCodec)

  val shelfToonCodec: ToonCodec[Shelf] = Schema[Shelf].deriving(partnerToon).derive

  println(shelfToonCodec.encodeToString(shelf))
  println(s"round trip: ${shelfToonCodec.decode(shelfToonCodec.encodeToString(shelf)) == Right(shelf)}")
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain overridecodecforonetype.Step3ToonExample"
```

### Narrower Scopes

This example overrides only `Order.orderedOn`, then overrides `LocalDate` for a single derivation with `DerivationBuilder#instance`.

<details>
  <summary>schema-examples/src/main/scala/overridecodecforonetype/Step4NarrowerScopesExample.scala</summary>

```scala title="schema-examples/src/main/scala/overridecodecforonetype/Step4NarrowerScopesExample.scala" showLineNumbers
package overridecodecforonetype

/**
 * How to Override the Codec for One Type Everywhere — Narrower Scopes: One Field or One Derivation
 *
 * Overrides only Order.orderedOn, then overrides LocalDate for a single derivation with DerivationBuilder#instance.
 *
 * Run with: cd schema-examples && sbt "runMain
 * overridecodecforonetype.Step4NarrowerScopesExample"
 */
object Step4NarrowerScopesExample extends App {

  case class Order(title: String, orderedOn: LocalDate, price: BigDecimal)

  object Order {
    implicit val schema: Schema[Order] = Schema.derived
  }

  case class Shelf(owner: String, openedOn: LocalDate, orders: List[Order])

  object Shelf {
    implicit val schema: Schema[Shelf] = Schema.derived
  }

  val shelf = Shelf(
    owner = "Ada",
    openedOn = LocalDate.of(2025, 1, 10),
    orders = List(
      Order("Zionomicon", LocalDate.of(2025, 3, 14), BigDecimal("39.99")),
      Order("Programming in Scala", LocalDate.of(2025, 4, 2), BigDecimal("54.50")),
      Order("Functional Programming in Scala", LocalDate.of(2025, 5, 21), BigDecimal("44.00"))
    )
  )

  val partnerDateFormat = DateTimeFormatter.ofPattern("dd/MM/yyyy")

  val partnerDateCodec: JsonCodec[LocalDate] = new JsonCodec[LocalDate] {
    private def parse(s: String): LocalDate =
      try LocalDate.parse(s, partnerDateFormat)
      catch { case _: DateTimeParseException => error(s"expected a dd/MM/yyyy date, got: $s") }

    def decodeValue(in: JsonReader): LocalDate =
      parse(in.readString())

    def encodeValue(x: LocalDate, out: JsonWriter): Unit =
      out.writeVal(partnerDateFormat.format(x))

    override def decodeValue(json: Json): LocalDate = json match {
      case s: Json.String => parse(s.value)
      case _              => error("expected a JSON string")
    }

    override def encodeValue(x: LocalDate): Json =
      Json.String(partnerDateFormat.format(x))
  }

  val orderDateOnly = JsonCodecDeriver.withInstance(TypeId.of[Order], "orderedOn", partnerDateCodec)

  val shelfCodecOrderDateOnly: JsonCodec[Shelf] = Schema[Shelf].deriving(orderDateOnly).derive

  println(s"field only: ${shelfCodecOrderDateOnly.encodeToString(shelf)}")

  val oneOffShelfCodec: JsonCodec[Shelf] =
    Schema[Shelf]
      .deriving(JsonCodecDeriver)
      .instance(TypeId.of[LocalDate], partnerDateCodec)
      .derive

  println(s"one-off:    ${oneOffShelfCodec.encodeToString(shelf)}")
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain overridecodecforonetype.Step4NarrowerScopesExample"
```

### The Complete Example

The complete example from "Putting It Together" runs the JSON and TOON overrides in one program:

```bash
sbt "runMain overridecodecforonetype.CompleteExample"
```

To check that every example compiles without running it, use:

```bash
sbt compile
```

## Going Further

- [Customizing Derivation with Instance and Modifier Overrides](../reference/schema/type-class-derivation.md#customizing-derivation-with-instance-and-modifier-overrides) covers every `withInstance` and `withModifier` overload and how overrides chain.
- [How to Derive Your Own Type Class from a Schema](./derive-your-own-type-class.md) writes a `Deriver` for a type class of your own, beyond codecs.
- [Codec](../reference/schema/codec.md) explains how formats, derivers and codecs fit together.
- [TOON](../reference/schema/built-in-codecs/toon.md) documents the TOON format and its deriver settings.

---

## How to Process Binary Data Without Boxing

In this guide, we process the binary files a vibration sensor on a pump writes: one big-endian `Long` timestamp and one little-endian `Double` acceleration per sample. We read them with ZIO Blocks Streams, so each value travels as a primitive, never as a boxed `java.lang.Long` or `java.lang.Double`. By the end, you will have a pipeline that reads a memory-mapped file or a gzip `InputStream`, transforms and aggregates 100,000 samples without allocating per sample, writes the calibrated samples to a new file, and measures what each step allocates.

We start with a `ByteBuffer` and build up to files, custom sinks and writers:

- [Reading Values From a ByteBuffer](#reading-values-from-a-bytebuffer)
- [Matching the File's Byte Order](#matching-the-files-byte-order)
- [Mapping a File Instead of Copying It](#mapping-a-file-instead-of-copying-it)
- [Transforming Values on the Primitive Lane](#transforming-values-on-the-primitive-lane)
- [Aggregating With the Built-in Folds](#aggregating-with-the-built-in-folds)
- [Writing a Custom Primitive Sink](#writing-a-custom-primitive-sink)
- [Ending the Loop Without Losing Values](#ending-the-loop-without-losing-values)
- [Decoding Values From an InputStream](#decoding-values-from-an-inputstream)
- [Writing Values Back Out](#writing-values-back-out)
- [Measuring Bytes per Element](#measuring-bytes-per-element)

## The Problem

Binary data arrives as bytes: a file of sensor samples, a market-data capture, a network packet. To compute anything we decode the bytes into numbers, and the usual Scala tools store each number as an object. A `Vector[Double]`, an `Iterator[Long]` or an `Iterator#foldLeft` over them boxes every element, so a file of 100,000 samples becomes 100,000 small objects for the garbage collector. The JDK's `DataInputStream` also reads only big-endian values, so a little-endian file needs a manual byte swap.

For example, reading the sensor's samples and computing their root mean square with the standard library looks like this:

```scala

def readSamples(file: Path): Vector[Double] = {
  val in      = new DataInputStream(new BufferedInputStream(Files.newInputStream(file)))
  val samples = new VectorBuilder[Double]
  try {
    while (true) {
      // DataInputStream is big-endian only, so swap the bytes of each little-endian value
      samples += java.lang.Double.longBitsToDouble(java.lang.Long.reverseBytes(in.readLong()))
    }
  } catch { case _: EOFException => () }
  finally in.close()
  samples.result()
}

def rms(samples: Vector[Double]): Double =
  math.sqrt(samples.map(g => g * g).sum / samples.size)
```

This works, but it holds the whole file in memory as boxed `Double`s, `Vector#map` boxes a second copy, and the end of the file is an exception. On a busy service that decodes files or packets all day, those allocations turn into garbage collection pauses.

In this guide, we read the same bytes as a stream on a primitive *lane*: ZIO Blocks Streams tags a stream of `Long` or `Double` values with its element type and pulls each value as a JVM primitive, from the source through `Stream#map` and `Stream#filter` to the sink.

## Prerequisites

Add the streams module to your `build.sbt`. With ZIO Blocks 0.0.56, the artifact name depends on the Scala version:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-streams" % "0.0.56"

// Scala 3.9 and newer, which also needs the -experimental compiler flag
libraryDependencies += "dev.zio" %% "zio-blocks-next-streams" % "0.0.56"
scalacOptions += "-experimental"
```

The streams module needs JDK 21 or newer at runtime. `NioStreams`, `NioReaders`, `NioSinks`, `NioWriters`, `Sink.create` and the blocking terminals such as `Stream#runFold` exist only on the JVM.

This guide assumes you have built and run a [`Stream`](../reference/streams/core/stream.md) before, as in the tutorial [Your First Streaming Pipeline](./your-first-streaming-pipeline.md), and that you know `java.nio.ByteBuffer`. Every terminal returns an `Either`, with typed failures on the `Left`.

The snippets on this page share these imports:

```scala

```

## The Sensor Files

The sensor samples at 1 kHz. For every sample it appends the time in milliseconds since the epoch to `timestamps.bin`, as a big-endian `Long`, and the acceleration in g to `samples.bin`, as a little-endian `Double`. When the acceleration is out of range, it writes `Double.MaxValue` instead of a reading. We write 100 seconds of such data to a temporary directory, with three overrange readings, so every snippet runs on any machine:

```scala
val SampleCount = 100000
val StartMillis = 1767225600000L // 2026-01-01T00:00:00Z
val Overrange   = Double.MaxValue

def sample(i: Int): Double =
  if (i > 0 && i % 25000 == 0) Overrange
  else 0.2 * math.sin(2 * math.Pi * 50 * i / 1000.0) + 0.05 * math.sin(2 * math.Pi * 120 * i / 1000.0)

val sensorDir: Path      = Files.createTempDirectory("pump-sensor")
val timestampsFile: Path = sensorDir.resolve("timestamps.bin")
val samplesFile: Path    = sensorDir.resolve("samples.bin")

locally {
  val times = ByteBuffer.allocate(SampleCount * 8) // big-endian, the ByteBuffer default
  val accel = ByteBuffer.allocate(SampleCount * 8).order(ByteOrder.LITTLE_ENDIAN)
  (0 until SampleCount).foreach { i => times.putLong(StartMillis + i); accel.putDouble(sample(i)) }
  Files.write(timestampsFile, times.array())
  Files.write(samplesFile, accel.array())
}
```

## Reading Values From a ByteBuffer

`NioStreams` turns a `ByteBuffer` into a stream of primitives: `NioStreams.fromByteBufferLong` reads 8 bytes per `Long`, and `NioStreams.fromByteBufferInt`, `NioStreams.fromByteBufferFloat` and `NioStreams.fromByteBufferDouble` do the same for the other numeric types. `NioStreams.fromByteBuffer` reads plain bytes. When the file fits in memory, we read it into a buffer and stream the timestamps:

```scala
val timestampBytes: ByteBuffer = ByteBuffer.wrap(Files.readAllBytes(timestampsFile))
```

`Stream#runFold` with a `Long` start value counts the timestamps without boxing, and `Stream#head` and `Stream#last` return the first and last one:

```scala
NioStreams.fromByteBufferLong(timestampBytes.duplicate()).runFold(0L)((n, _) => n + 1L)
// res2: Either[Nothing, Long] = Right(100000L)
NioStreams.fromByteBufferLong(timestampBytes.duplicate()).head
// res3: Either[Nothing, Option[Long]] = Right(Some(1767225600000L))
NioStreams.fromByteBufferLong(timestampBytes.duplicate()).last
// res4: Either[Nothing, Option[Long]] = Right(Some(1767225699999L))
```

The stream reads from the buffer's position to its limit and moves the position as it goes, so a second stream over the same buffer finds nothing left to read. Each stream above therefore reads its own `ByteBuffer#duplicate`, which shares the bytes but has its own position. A trailing group of fewer than 8 bytes is not a whole `Long` and is not emitted.

`NioReaders` has the same five `ByteBuffer` factories, for code that pulls values by hand from a [`Reader`](../reference/streams/primitives/reader.md) instead of running a stream.

## Matching the File's Byte Order

A `ByteBuffer` reads multi-byte values in its own byte order, and every new buffer starts big-endian. The timestamps are big-endian, so they decode correctly. The samples are little-endian:

```scala
val sampleBytes: Array[Byte] = Files.readAllBytes(samplesFile)
```

Read with the default order, they decode to nonsense:

```scala
println(NioStreams.fromByteBufferDouble(ByteBuffer.wrap(sampleBytes)).take(3).runCollect)
// Right(Chunk(0.0,6.837871985336776E36,1.791739153893799E-190))
```

Setting the buffer's order with `ByteBuffer#order` before the stream reads it fixes the values:

```scala
println(NioStreams.fromByteBufferDouble(ByteBuffer.wrap(sampleBytes).order(ByteOrder.LITTLE_ENDIAN)).take(3).runCollect)
// Right(Chunk(0.0,0.09603075417142391,0.16745838687990822))
```

:::warning[`ByteBuffer#duplicate` and `ByteBuffer#slice` reset the byte order]
`ByteBuffer#duplicate` and `ByteBuffer#slice` return a big-endian buffer, whatever the original's order. Call `ByteBuffer#order` on the copy every time: `buf.duplicate().order(ByteOrder.LITTLE_ENDIAN)`. A `MappedByteBuffer` from `FileChannel#map` also starts big-endian.
:::

The same applies to writing: the typed sinks and writers later in this guide write in the target buffer's order.

## Mapping a File Instead of Copying It

`Files.readAllBytes` copies the whole file onto the heap. `FileChannel#map` maps the file into memory instead, so the operating system pages it in as the stream reads it, and nothing is copied onto the heap. We open the channel with `Stream.fromAcquireRelease`, which closes it when the run ends, whether the run finishes, fails or stops early:

```scala
def samples(file: Path): Stream[Nothing, Double] =
  Stream.fromAcquireRelease(FileChannel.open(file)) { ch =>
    val mapped = ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size()).order(ByteOrder.LITTLE_ENDIAN)
    NioStreams.fromByteBufferDouble(mapped)
  }

def timestamps(file: Path): Stream[Nothing, Long] =
  Stream.fromAcquireRelease(FileChannel.open(file)) { ch =>
    NioStreams.fromByteBufferLong(ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size()))
  }
```

Each run opens the file again and maps it afresh, so the stream can run any number of times:

```scala
samples(samplesFile).runFold(0L)((n, _) => n + 1L)
// res7: Either[Nothing, Long] = Right(100000L)
println(samples(samplesFile).take(3).runCollect)
// Right(Chunk(0.0,0.09603075417142391,0.16745838687990822))
```

The mapping stays valid after the channel closes, until the garbage collector frees the buffer. One mapping covers at most `Int.MaxValue` bytes (2 GiB), so a larger file needs several mappings, which `Stream#flatMap` can join into one stream; it keeps the inner streams' `Double` lane.

The guide [How to Stream Files and Sockets Without Leaking Resources](./stream-files-and-sockets.md) covers `Stream.fromAcquireRelease` and the other ways to close a resource.

## Transforming Values on the Primitive Lane

`Stream#map`, `Stream#filter`, `Stream#scan`, `Stream#take`, `Stream#drop`, `Stream#takeWhile` and `Stream#tapEach` keep a primitive stream on its lane, as long as the function returns a primitive. To convert the first second of samples from g to m/s², drop the overrange readings and track the running peak:

```scala
val StandardGravity = 9.80665 // m/s² per g

val firstSecond: Stream[Nothing, Double] =
  samples(samplesFile)
    .take(1000)
    .filter(_ != Overrange)
    .map(_ * StandardGravity)

val peak: Stream[Nothing, Double] =
  firstSecond.scan(0.0)((peak, a) => math.max(peak, math.abs(a)))
```

The stream carries each sample as a `Double` from the mapped file to the terminal:

```scala
println(firstSecond.take(3).runCollect)
// Right(Chunk(0.0,0.9417399953951943,1.6422057896958517))
peak.last
// res10: Either[Nothing, Option[Double]] = Right(Some(2.427663919276313))
```

A function that returns an object moves the stream to the reference lane, and from then on every element is an object. A `map(g => (g, g))` builds a tuple per sample, which measured 64 bytes per sample on our machine. Keep values primitive for as long as the pipeline runs, and build objects only for the results.

## Aggregating With the Built-in Folds

A *sink* consumes a stream and produces one result. `Stream#runFold` and `Sink.foldLeft` with a `Long` or `Int` start value fold a primitive stream without boxing either the element or the accumulator. To count the overrange readings and find the first and last timestamp:

```scala
samples(samplesFile).filter(_ == Overrange).runFold(0L)((n, _) => n + 1L)
// res11: Either[Nothing, Long] = Right(3L)

val first = timestamps(timestampsFile).run(Sink.foldLeft(Long.MaxValue)((lo: Long, t: Long) => math.min(lo, t)))
// first: Either[Nothing, Long] = Right(1767225600000L)
val last  = timestamps(timestampsFile).run(Sink.foldLeft(Long.MinValue)((hi: Long, t: Long) => math.max(hi, t)))
// last: Either[Nothing, Long] = Right(1767225699999L)
```

:::warning[Some built-in sinks box the accumulator]
A known bug in 0.0.56: `Sink.sumLong`, `Sink.sumInt`, `Sink.sumFloat`, `Sink.sumDouble`, `Sink.count` and `Stream#count` read the elements unboxed, but box their accumulator for every element, which measured 24 to 48 bytes per element on our machine. `Stream#runFold` and `Sink.foldLeft` with a `Double` start value box the same way when the stream is on the `Long` or `Double` lane. Until the fix ships, count and sum with a `Long` start value, as above, and fold `Double` values with a custom sink, as in the next section.
:::

## Writing a Custom Primitive Sink

When no built-in sink computes what we need, `Sink.create` builds one from a function that pulls from the stream's `Reader.SyncReader`. A custom sink also helps when we want several results in one pass: a fold would need a tuple accumulator, which allocates per element, while a sink keeps each result in its own primitive `var`. This sink computes the vibration statistics of the whole recording in one pass:

```scala
final case class VibrationStats(samples: Long, overrange: Long, min: Double, max: Double, rms: Double)

val vibrationStats: Sink[Nothing, Double, VibrationStats] =
  Sink.create[Nothing, Double, VibrationStats] { reader =>
    val one       = new Array[Double](1)
    var samples   = 0L
    var overrange = 0L
    var min       = Double.PositiveInfinity
    var max       = Double.NegativeInfinity
    var sumSq     = 0.0
    while (reader.readDoubles(one, 0, 1) > 0) {
      val g = one(0)
      if (g == Overrange) overrange += 1
      else {
        samples += 1
        if (g < min) min = g
        if (g > max) max = g
        sumSq += g * g
      }
    }
    VibrationStats(samples, overrange, min, max, math.sqrt(sumSq / samples))
  }
```

`Reader#readDoubles` copies the next sample into `one` and returns `1`, or returns `-1` once the stream has ended. The one-element array is allocated once per run, outside the loop. Running the sink over the mapped file allocates nothing per sample, only the result:

```scala
samples(samplesFile).run(vibrationStats)
// res12: Either[Nothing, VibrationStats] = Right(
//   VibrationStats(
//     samples = 99997L,
//     overrange = 3L,
//     min = -0.24755282581508503,
//     max = 0.24755282581481453,
//     rms = 0.14577598402729725
//   )
// )
```

The `var`s are local to one run, so the same `vibrationStats` value can run any number of times. A sink built with `Sink.create` exists only on the JVM, and at an asynchronous terminal its callback blocks the thread that drives the run; `Sink.createBoth` takes a synchronous and an asynchronous callback for code that also runs on Scala.js or must not block at asynchronous terminals. The [`Sink`](../reference/streams/core/sink.md) reference covers both.

## Ending the Loop Without Losing Values

A `Reader` also has a scalar read for each type, such as `Reader#readDouble`, which returns the next value or a *sentinel*, a value we pick to mean "the stream has ended". On the `Long` and `Double` lanes every value can be real data, so any sentinel can collide with an element. The sensor writes `Double.MaxValue` for an overrange reading, so a sink that uses it as the sentinel stops at the first overrange sample, without an error. To compare the two loops, we count the samples both ways:

```scala
val countWithSentinel: Sink[Nothing, Double, Long] =
  Sink.create[Nothing, Double, Long] { reader =>
    var n = 0L
    var g = reader.readDouble(Double.MaxValue)
    while (g != Double.MaxValue) { n += 1; g = reader.readDouble(Double.MaxValue) }
    n
  }

val countWithCount: Sink[Nothing, Double, Long] =
  Sink.create[Nothing, Double, Long] { reader =>
    val one = new Array[Double](1)
    var n   = 0L
    while (reader.readDoubles(one, 0, 1) > 0) n += 1
    n
  }
```

The sentinel loop silently drops three quarters of the recording:

```scala
samples(samplesFile).run(countWithSentinel)
// res13: Either[Nothing, Long] = Right(25000L)
samples(samplesFile).run(countWithCount)
// res14: Either[Nothing, Long] = Right(100000L)
```

On the `Long` and `Double` lanes, end the loop on the count that `Reader#readLongs` or `Reader#readDoubles` returns, as `vibrationStats` does. The count is reported apart from the data, so no value is lost, and the one-element array, allocated once per run, keeps the loop free of per-element allocation. Do not add a check against the sentinel inside the loop to work around a collision. The other lanes have no such risk: `Reader#readInt` returns a `Long`, `Reader#readFloat` a `Double`, `Reader#readShort` an `Int` and `Reader#readByte` an `Int` from `0` to `255`, so their usual sentinels (`Long.MinValue`, `Double.MaxValue`, `Int.MinValue` and `-1`) lie outside every element's range. The library's own sinks, `NioSinks` included, use the count on the `Long` and `Double` lanes and these widened sentinels on the others, so they keep every value. The [zero-boxing reference](../reference/streams/execution-and-compatibility/zero-boxing.md#eof-signalling-per-lane) lists the end-of-stream signal of every lane.

## Decoding Values From an InputStream

Not every source is a file we can map. A gzip file, an HTTP response body or a socket arrives as a `java.io.InputStream`, and `Stream.fromInputStream` turns it into a stream of bytes, not of `Double`s. To decode it, we read the input in blocks into one reusable `ByteBuffer` with `Stream.unfold`, and turn each block into a `Double` stream with `Stream#flatMap`:

```scala

def samplesFrom(in: InputStream, order: ByteOrder): Stream[Nothing, Double] = {
  val block = ByteBuffer.allocate(64 * 1024).order(order)
  Stream
    .unfold(in) { in =>
      val n = in.readNBytes(block.array(), 0, block.capacity())
      if (n <= 0) None
      else {
        block.clear()
        block.limit(n - n % 8)
        Some((block, in))
      }
    }
    .flatMap(b => NioStreams.fromByteBufferDouble(b))
}
```

`InputStream#readNBytes` fills the whole block unless the input ends, so only the last block can be shorter, and `block.limit(n - n % 8)` drops an incomplete value at the very end. `Stream#flatMap` reads each block to its end before `Stream.unfold` refills it, which is what makes one buffer per run safe. The decoded stream stays on the `Double` lane, so the same `vibrationStats` sink reads it unboxed. We write a gzip copy of the samples next to the original:

```scala

val samplesGz: Path = sensorDir.resolve("samples.bin.gz")
locally {
  val out = new GZIPOutputStream(Files.newOutputStream(samplesGz))
  try out.write(sampleBytes)
  finally out.close()
}
```

Decoded through `GZIPInputStream`, the copy gives the same statistics as the mapped file:

```scala
Stream
  .fromAcquireRelease(new GZIPInputStream(Files.newInputStream(samplesGz))) { in =>
    samplesFrom(in, ByteOrder.LITTLE_ENDIAN)
  }
  .run(vibrationStats)
// res16: Either[Nothing, VibrationStats] = Right(
//   VibrationStats(
//     samples = 99997L,
//     overrange = 3L,
//     min = -0.24755282581508503,
//     max = 0.24755282581481453,
//     rms = 0.14577598402729725
//   )
// )
```

`Stream.fromAcquireRelease` closes the `GZIPInputStream`, and with it the file, when the run ends. An `IOException` from `InputStream#readNBytes` is not a typed error here: it ends the run as a thrown exception. A socket works the same way through its `InputStream`, or through `java.nio.channels.Channels.newInputStream` for a channel.

## Writing Values Back Out

The write side mirrors the read side. This section writes the calibrated samples with a typed sink, records values that a driver pushes one at a time with typed writers, and lists who closes which channel.

### Into a Mapped File

`NioSinks.fromByteBufferDouble` writes each `Double` of a stream into a `ByteBuffer`, 8 bytes per element in the buffer's byte order; `NioSinks.fromByteBufferLong`, `NioSinks.fromByteBufferInt`, `NioSinks.fromByteBufferFloat` and `NioSinks.fromByteBuffer` cover the other types. The buffer must have room for every element, or the sink throws `BufferOverflowException`. We know the output size, so we map an output file of that size with `FileChannel.MapMode.READ_WRITE` and let the sink fill it. The calibrated file is big-endian, the default order of the mapping:

```scala

val calibratedFile: Path = sensorDir.resolve("calibrated.bin")
```

The sink returns `Right(())` once every sample is in the mapping:

```scala
Using.resource(FileChannel.open(calibratedFile, CREATE, TRUNCATE_EXISTING, READ, WRITE)) { ch =>
  val target = ch.map(FileChannel.MapMode.READ_WRITE, 0, SampleCount * 8L)
  val result = samples(samplesFile)
    .map(g => if (g == Overrange) g else g * StandardGravity)
    .run(NioSinks.fromByteBufferDouble(target))
  target.force()
  result
}
// res17: Either[Nothing, Unit] = Right(())
```

`MappedByteBuffer#force` writes the changes to the storage device. The sink ends its loop on the count from `Reader#readDoubles`, so the `Double.MaxValue` markers are written like any other value. Reading the output back finds all three:

```scala
Stream
  .fromAcquireRelease(FileChannel.open(calibratedFile)) { ch =>
    NioStreams.fromByteBufferDouble(ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size()))
  }
  .filter(_ == Double.MaxValue)
  .runFold(0L)((n, _) => n + 1L)
// res18: Either[Nothing, Long] = Right(3L)
```

### From Pushed Values With Writers

A sink pulls from a stream. When values are pushed to us instead, one call per sample from a device driver, a [`Writer`](../reference/streams/primitives/writer.md) accepts them: `NioWriters.fromByteBufferLong` and `NioWriters.fromByteBufferDouble` return a `Writer[Long]` and a `Writer[Double]` whose `Writer#writeLong` and `Writer#writeDouble` take the primitive unboxed. A `ByteBuffer` writer returns `false` and closes itself once its buffer is full, so this recorder writes each full block to its channel and starts a new writer on the emptied buffer:

```scala

final class Recorder(times: WritableByteChannel, samples: WritableByteChannel, blockSize: Int) {
  private val timeBlock   = ByteBuffer.allocate(blockSize * 8)
  private val sampleBlock = ByteBuffer.allocate(blockSize * 8).order(ByteOrder.LITTLE_ENDIAN)
  private var timeOut: Writer[Long]     = NioWriters.fromByteBufferLong(timeBlock)
  private var sampleOut: Writer[Double] = NioWriters.fromByteBufferDouble(sampleBlock)

  def onSample(millis: Long, g: Double): Unit = {
    if (!timeOut.writeLong(millis)) { flush(); timeOut.writeLong(millis) }
    if (!sampleOut.writeDouble(g)) { flush(); sampleOut.writeDouble(g) }
  }

  def flush(): Unit = {
    drain(timeBlock, times)
    drain(sampleBlock, samples)
    timeOut = NioWriters.fromByteBufferLong(timeBlock)
    sampleOut = NioWriters.fromByteBufferDouble(sampleBlock)
  }

  private def drain(block: ByteBuffer, ch: WritableByteChannel): Unit = {
    block.flip()
    while (block.hasRemaining) ch.write(block)
    block.clear()
  }
}
```

The recorder does not own the channels, so the caller opens and closes them. We record into two new files:

```scala
val recordedTimes: Path   = sensorDir.resolve("recorded-times.bin")
val recordedSamples: Path = sensorDir.resolve("recorded-samples.bin")
```

Recording 10 seconds of samples and reading them back:

```scala
Using.resources(
  FileChannel.open(recordedTimes, CREATE, TRUNCATE_EXISTING, WRITE),
  FileChannel.open(recordedSamples, CREATE, TRUNCATE_EXISTING, WRITE)
) { (timesCh, samplesCh) =>
  val recorder = new Recorder(timesCh, samplesCh, blockSize = 4096)
  (0 until 10000).foreach(i => recorder.onSample(StartMillis + i, sample(i)))
  recorder.flush()
}

samples(recordedSamples).runFold(0L)((n, _) => n + 1L)
// res20: Either[Nothing, Long] = Right(10000L)
```

### Who Closes the Channel

The NIO constructors differ in who closes the channel they wrap, and a channel nobody closes leaks a file handle:

| Constructor                        | Closes the channel                               |
|------------------------------------|--------------------------------------------------|
| `NioStreams.fromChannel`           | Yes, when the run ends                           |
| `NioStreams.fromChannelUnmanaged`  | No, the caller closes it                         |
| `NioReaders.fromChannel`           | Yes, when the reader closes                      |
| `NioSinks.fromChannel`             | No, the caller closes it                         |
| `NioWriters.fromChannel`           | Yes, when the writer closes                      |
| `Stream.fromAcquireRelease`        | Yes, the release function runs when the run ends |

All five NIO channel constructors work with bytes, not with `Long` or `Double` values. A known bug in 0.0.56: after a failed write, closing a `NioWriters.fromChannel` writer, or one from `Writer.fromOutputStream` or `Writer.fromWriter`, does not close the wrapped channel or stream, so close it yourself in that case.

## Measuring Bytes per Element

The only reliable way to know that a pipeline stays on its lane is to measure what it allocates. The JVM counts the bytes each thread allocates, and `com.sun.management.ThreadMXBean` exposes that counter. The block must run a few thousand times first, so that the JIT compiler has compiled it and removed what it can:

```scala

object Allocation {
  private val threads =
    ManagementFactory.getThreadMXBean.asInstanceOf[com.sun.management.ThreadMXBean]

  @volatile private var blackhole: Int = 0

  def bytesPerElement(elements: Int)(body: => Any): Double = {
    var i = 0
    while (i < 1000) { blackhole ^= body.hashCode; i += 1 } // warm up the JIT
    val runs   = 100
    val before = threads.getCurrentThreadAllocatedBytes
    i = 0
    while (i < runs) { blackhole ^= body.hashCode; i += 1 }
    (threads.getCurrentThreadAllocatedBytes - before).toDouble / runs / elements
  }
}
```

We measure one step at a time, such as the statistics over the mapped file:

```scala
println(Allocation.bytesPerElement(SampleCount)(samples(samplesFile).run(vibrationStats)))
```

The companion example measures every step of this guide over the 100,000 samples. On our machine, with JDK 25, the results were:

| Step                                                                        | Bytes per sample |
|-----------------------------------------------------------------------------|------------------|
| `ByteBuffer`, counted with `runFold(0L)`                                    | 0.00             |
| `Stream#filter`, `Stream#map` and `Stream#scan`, counted with `runFold(0L)` | 0.02             |
| Mapped file into `vibrationStats`                                           | 0.02             |
| Gzip `InputStream` into `vibrationStats`                                    | 0.72             |
| `Stream#map` into `NioSinks.fromByteBufferDouble`                           | 0.00             |
| Mapped file into a mapped file                                              | 0.02             |
| `Recorder`, blocks of 4,096 samples                                         | 0.67             |
| `Stream#map` to a tuple, leaving the lane                                   | 63.97            |
| `Sink.sumDouble` (a known bug in 0.0.56)                                    | 24.00            |
| `runFold(0.0)` (a known bug in 0.0.56)                                      | 48.00            |

The small non-zero figures are per-run costs spread over 100,000 samples: the stream's own objects, the result, and in the gzip and recorder rows the 64 KB of buffers each run allocates. None of them grows with the number of samples. Treat the figures as rough sizes, not guarantees: they depend on the JDK, the JIT and the pipeline. For a JMH benchmark with `-prof gc`, which reports the same figure as `gc.alloc.rate.norm`, see [How to Avoid Allocation on Hot Paths](./avoid-allocation-on-hot-paths.md).

## Putting It Together

The complete example writes the sensor files, reports the recording's time span from the timestamps, computes the vibration statistics from the mapped samples and checks that the gzip copy gives the same ones, writes the calibrated samples to a mapped output file, and prints what the statistics and the calibration allocate per sample. It takes the sources, the decoder and the statistics sink from `Pump.scala`, the sensor files from `SensorFiles.scala` and the measuring helper from `Allocation.scala`, which hold the same code as the sections above:

```scala title="streams-examples/src/main/scala/processbinarydata/CompleteExample.scala"
package processbinarydata

/**
 * How to Process Binary Data Without Boxing
 *
 * Pump vibration report: maps a sensor's timestamp and sample files into
 * memory, computes vibration statistics with a custom primitive sink, checks
 * the gzip copy decodes to the same statistics, writes the calibrated samples
 * to a memory-mapped output file, and prints the bytes each run allocates per
 * sample.
 *
 * Run with: sbt "runMain processbinarydata.CompleteExample"
 */
object CompleteExample {

  def main(args: Array[String]): Unit = {
    val dump       = SensorFiles.create()
    val calibrated = dump.dir.resolve("calibrated.bin")
    val n          = SensorFiles.SampleCount
    try {
      // 1. Time span from the big-endian timestamps.
      val first =
        Pump.timestamps(dump.timestamps).run(Sink.foldLeft(Long.MaxValue)((lo: Long, t: Long) => math.min(lo, t)))
      val last =
        Pump.timestamps(dump.timestamps).run(Sink.foldLeft(Long.MinValue)((hi: Long, t: Long) => math.max(hi, t)))
      for { a <- first; b <- last } println(s"recording: ${n} samples over ${(b - a + 1) / 1000.0} s")

      // 2. Vibration statistics from the little-endian samples, mapped and gzipped.
      val fromFile = Pump.samples(dump.samples).run(Pump.vibrationStats)
      val fromGzip = Stream
        .fromAcquireRelease(new GZIPInputStream(Files.newInputStream(dump.samplesGz))) { in =>
          Pump.samplesFrom(in, ByteOrder.LITTLE_ENDIAN)
        }
        .run(Pump.vibrationStats)
      fromFile.foreach { s =>
        println(f"samples:   ${s.samples} (${s.overrange} overrange)")
        println(f"range (g): ${s.min}%.4f to ${s.max}%.4f, rms ${s.rms}%.4f")
      }
      println(s"gzip copy matches: ${fromFile == fromGzip}")

      // 3. Calibrated samples (m/s², big-endian) into a memory-mapped output file.
      def calibrate(target: java.nio.ByteBuffer) =
        Pump
          .samples(dump.samples)
          .map(g => if (g == SensorFiles.Overrange) g else g * Pump.StandardGravity)
          .run(NioSinks.fromByteBufferDouble(target))

      Using.resource(FileChannel.open(calibrated, CREATE, TRUNCATE_EXISTING, READ, WRITE)) { ch =>
        val target = ch.map(FileChannel.MapMode.READ_WRITE, 0, n * 8L)
        calibrate(target)
        target.force()

        // 4. What each run allocates per sample, once the JIT has compiled it.
        val statsBytes     = Allocation.bytesPerElement(n)(Pump.samples(dump.samples).run(Pump.vibrationStats))
        val calibrateBytes = Allocation.bytesPerElement(n) { target.clear(); calibrate(target) }
        println(f"stats:     $statsBytes%.2f bytes per sample")
        println(f"calibrate: $calibrateBytes%.2f bytes per sample")
      }
      println(s"calibrated file: ${Files.size(calibrated)} bytes")
    } finally SensorFiles.delete(dump)
  }
}
```

## Running the Examples

All examples in this guide have runnable Scala files in the `streams-examples` project. Clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/streams-examples
```

Each example writes its own copy of the sensor files to a temporary directory and deletes it at the end. The code the examples share lives in `SensorFiles.scala` (the sensor files), `Allocation.scala` (the measuring helper), `Pump.scala` (the mapped-file sources, the `InputStream` decoder and the statistics sink) and `Recorder.scala`, all in `src/main/scala/processbinarydata/`.

Each section has its own example, run from the `streams-examples` directory:

| Section                                                                                 | Run                                                      |
|-----------------------------------------------------------------------------------------|----------------------------------------------------------|
| [Reading Values From a ByteBuffer](#reading-values-from-a-bytebuffer)                   | `sbt "runMain processbinarydata.Step1ReadBuffer"`        |
| [Matching the File's Byte Order](#matching-the-files-byte-order)                        | `sbt "runMain processbinarydata.Step2ByteOrder"`         |
| [Mapping a File Instead of Copying It](#mapping-a-file-instead-of-copying-it)           | `sbt "runMain processbinarydata.Step3MapFile"`           |
| [Transforming Values on the Primitive Lane](#transforming-values-on-the-primitive-lane) | `sbt "runMain processbinarydata.Step4Transform"`         |
| [Aggregating With the Built-in Folds](#aggregating-with-the-built-in-folds)             | `sbt "runMain processbinarydata.Step5Aggregate"`         |
| [Writing a Custom Primitive Sink](#writing-a-custom-primitive-sink)                     | `sbt "runMain processbinarydata.Step6CustomSink"`        |
| [Ending the Loop Without Losing Values](#ending-the-loop-without-losing-values)         | `sbt "runMain processbinarydata.Step7EndOfStream"`       |
| [Decoding Values From an InputStream](#decoding-values-from-an-inputstream)             | `sbt "runMain processbinarydata.Step8DecodeInputStream"` |
| [Writing Values Back Out](#writing-values-back-out)                                     | `sbt "runMain processbinarydata.Step9WriteBack"`         |
| [Measuring Bytes per Element](#measuring-bytes-per-element)                             | `sbt "runMain processbinarydata.Step10Measure"`          |
| [Putting It Together](#putting-it-together)                                             | `sbt "runMain processbinarydata.CompleteExample"`        |

`Step5Aggregate` also measures `Stream#count` and `Sink.sumLong` against `runFold(0L)`, and `Step10Measure` prints the table in [Measuring Bytes per Element](#measuring-bytes-per-element).

## Going Further

- The [zero-boxing reference](../reference/streams/execution-and-compatibility/zero-boxing.md) explains how a stream picks its lane and where boxing can still happen, including the asynchronous terminals.
- The [`Reader`](../reference/streams/primitives/reader.md) and [`Writer`](../reference/streams/primitives/writer.md) references list every primitive read and write, including the bulk `Reader#readLongs` with a larger array.
- [How to Stream Files and Sockets Without Leaking Resources](./stream-files-and-sockets.md) covers managed and unmanaged handles, sockets and `Scope` resources.
- [How to Avoid Allocation on Hot Paths](./avoid-allocation-on-hot-paths.md) applies the same measuring to `Chunk`, `Option` and telemetry code, and verifies the figures with JMH.
- [Coming from fs2, ZIO Streams or Ox](./streaming-library-migration.md) maps the stream operators of those libraries onto ZIO Blocks Streams.

---

## How to Protect an Endpoint with JWT Bearer Tokens

In this guide, we protect the endpoints of a small Scala bookstore API so that only signed-in users can call them. By the end, you will have a wrapper of about twenty lines that reads the `Authorization: Bearer <token>` header, verifies the JSON Web Token (JWT) with `Jwt.decode`, answers `401 Unauthorized` or `403 Forbidden` the way HTTP clients expect, and hands the verified caller to your handler.

We start from an API with no protection at all and add one piece at a time:

- Read the token from the request with the typed header API.
- Verify it, including issuer, audience and expiry.
- Choose between 401 and 403, with the right `WWW-Authenticate` challenge.
- Wrap handlers so they only ever run for a verified caller.
- Load the signing key from configuration as a `Secret`.

Every section that serves requests starts a real server on a free loopback port, calls it, and prints what came back.

## The Problem

Our bookstore has a public book list, a page with the caller's own orders, and an admin-only route that changes a book's price. Without authentication, the server can only trust what the request says about the caller. Anyone who can reach the server can read anyone's orders and change any price.

A first attempt often checks that an `Authorization` header is present and stops there. Here is that handler, written against the JDK HTTP server API:

```scala

val ordersHandler: HttpHandler = (exchange: HttpExchange) => {
  val authorization = Option(exchange.getRequestHeaders.getFirst("Authorization"))
  val (status, body) =
    if (authorization.exists(_.startsWith("Bearer "))) // Any string after "Bearer " is accepted
      (200, "Orders of reader-1024: Zionomicon, Functional Programming in Scala")
    else
      (403, "Forbidden") // Wrong status, and no WWW-Authenticate challenge for the client
  val bytes = body.getBytes(UTF_8)
  exchange.sendResponseHeaders(status, bytes.length.toLong)
  exchange.getResponseBody.write(bytes)
  exchange.close()
}
```

This handler lets in `Bearer hello`, never checks a signature or an expiry, has no idea *who* the caller is, and answers a missing token with 403, which tells a well-behaved client that signing in again will not help.

Declaring the requirement does not fix it either. ZIO Blocks' endpoint module has `AuthType.Bearer`, and it is easy to read `Endpoint#auth(AuthType.Bearer)` as "this endpoint is protected". It is not: `AuthType` only *describes* the requirement and says how to *extract* the token. Nothing in ZIO Blocks verifies a JWT on your behalf. Verification is your code, and this guide shows that code.

## Prerequisites

Add the HTTP model, the JWT module and the config module to your `build.sbt`. The endpoint module is only needed for the `AuthType` section:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-http-model" % "0.0.56",
  "dev.zio" %% "zio-blocks-jwt"        % "0.0.56",
  "dev.zio" %% "zio-blocks-config"     % "0.0.56",
  "dev.zio" %% "zio-blocks-endpoint"   % "0.0.56" // optional
)

// Scala 3.9 and newer
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-http-model" % "0.0.56",
  "dev.zio" %% "zio-blocks-next-jwt"        % "0.0.56",
  "dev.zio" %% "zio-blocks-next-config"     % "0.0.56",
  "dev.zio" %% "zio-blocks-next-endpoint"   % "0.0.56" // optional
)
scalacOptions += "-experimental"
```

The code is the same for every Scala version. The HTTP model depends on the ZIO Blocks streams module, which is compiled for Java 21, so run the program on **JDK 21 or newer**.

You should know how to sign and verify a token with `Jwt.sign` and `Jwt.decode`, from [How to Issue and Verify JWTs](./issue-and-verify-jwts.md). The configuration section uses `Config.load`, introduced in [Build a Typed Configuration Loader](./typed-config-loader.md).

ZIO Blocks has no HTTP server. We serve requests with the small adapter from the guide [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md): `JdkHttp.withServer(app)(use)` starts the JDK's built-in server on a free loopback port, runs `use` with the port, and stops the server. `Client.call(port, method, target, body, authorization)` sends one request and prints the status, the `WWW-Authenticate` header when there is one, and the body. Both files are shown in full under [Running the Examples](#running-the-examples); copy them into your project.

We use these imports throughout the guide:

```scala

```

## The Bookstore API

The bookstore sells three Scala books and keeps the orders of two fictional users. The responses are plain text so that the output stays short; nothing in this model knows about authentication:

```scala
final case class Book(id: Int, title: String, priceCents: Long)

object Bookstore {
  val books: List[Book] = List(
    Book(1, "Zionomicon", 4500),
    Book(2, "Programming in Scala", 5999),
    Book(3, "Functional Programming in Scala", 4999)
  )

  // Orders per user, keyed by the `sub` claim of their token.
  val orders: Map[String, List[String]] = Map(
    "reader-1024" -> List("Zionomicon", "Functional Programming in Scala"),
    "staff-0007"  -> List("Programming in Scala")
  )

  def price(cents: Long): String = f"$$${cents / 100}%d.${cents % 100}%02d"

  def listBooks: Response =
    Response.text(books.map(book => s"${book.id}. ${book.title} ${price(book.priceCents)}").mkString("\n"))

  def ordersOf(user: String): Response =
    Response.text(s"Orders of $user: ${orders.getOrElse(user, Nil).mkString(", ")}")

  // Answers with the new price; a real store would also save it.
  def setPrice(id: String, body: String): Response =
    (books.find(_.id.toString == id), body.trim.toLongOption.filter(_ > 0)) match {
      case (None, _)                 => Response.text(Status.NotFound, s"No book with id $id")
      case (_, None)                 => Response.text(Status.BadRequest, "The body must be a price in cents")
      case (Some(book), Some(cents)) => Response.text(s"${book.title} now costs ${price(cents)}")
    }

  def notFound(request: Request): Response =
    Response.text(Status.NotFound, s"No route for ${request.method.name} ${request.path.encode}")
}

```

Without authentication, the `GET /orders` route has to take the user's name from the request itself, here from a `?user=` query parameter. The routing matches on the method and the path segments:

```scala
def unprotectedApp(request: Request): Response =
  (request.method, request.path.segments.toList) match {
    case (Method.GET, List("books"))              => listBooks
    case (Method.GET, List("orders"))             => ordersOf(request.queryParams.getFirst("user").getOrElse(""))
    case (Method.PUT, List("books", id, "price")) => setPrice(id, request.body.text)
    case _                                        => notFound(request)
  }
```

When we serve it and call every route without any credentials, every call succeeds, including reading another user's orders and dropping the price of Zionomicon to one cent:

```scala
JdkHttp.withServer(unprotectedApp) { port =>
  Client.call(port, "GET", "/books")
  Client.call(port, "GET", "/orders?user=reader-1024")
  Client.call(port, "PUT", "/books/1/price", "1")
}
// GET /books -> 200 [text/plain; charset=UTF-8, 93 bytes]
// 1. Zionomicon $45.00
// 2. Programming in Scala $59.99
// 3. Functional Programming in Scala $49.99
// GET /orders?user=reader-1024 -> 200 [text/plain; charset=UTF-8, 66 bytes]
// Orders of reader-1024: Zionomicon, Functional Programming in Scala
// PUT /books/1/price -> 200 [text/plain; charset=UTF-8, 26 bytes]
// Zionomicon now costs $0.01
```

The rest of the guide keeps `GET /books` public, makes `GET /orders` require a signed-in user, and makes `PUT /books/{id}/price` require a signed-in admin.

## Declaring the Requirement With AuthType.Bearer

If you describe your API with the [endpoint module](../reference/endpoint/index.md), you declare the requirement with `Endpoint#auth`. [`AuthType.Bearer`](../reference/endpoint/auth-type.md) says that the client must send `Authorization: Bearer <token>`, and `AuthType.Scoped` adds the names of the scopes a caller needs:

```scala

val ordersEndpoint = Endpoint(Method.GET / "orders").auth(AuthType.Bearer)

val priceEndpoint =
  Endpoint(Method.PUT / "books" / PathCodec.int("id") / "price")
    .auth(AuthType.Scoped(AuthType.Bearer, List("admin")))
```

These are descriptions for documentation and for any interpreter that reads them. `AuthType.Bearer` carries a codec that parses the header into a `Header.Authorization.Bearer`, and a status for unauthorized calls:

```scala
println(ordersEndpoint.auth.unauthorizedStatus)
// 404
println(priceEndpoint.auth.unauthorizedStatus)
// 404
```

The default is `404 Not Found`, chosen to hide that a protected endpoint exists (see [Answering 401, 403 or 404](#answering-401-403-or-404)). Neither value checks a signature, an expiry or a scope: ZIO Blocks ships no server interpreter that enforces them. The rest of the guide is the code that does.

:::warning
`Endpoint#auth(AuthType.Bearer)` on its own does not protect anything. Treat it as documentation of what your handler enforces, and keep the two in sync.
:::

## Reading the Bearer Token

The HTTP model parses the `Authorization` header into one of four cases: `Header.Authorization.Basic`, `Bearer`, `Digest` or `Unparsed`. `Request#header` takes the header's companion object and returns an `Option`, so reading a bearer token is a `collect` on that `Option`:

```scala
def bearerToken(request: Request): Option[String] =
  request.header(Header.Authorization).collect { case Header.Authorization.Bearer(token) => token }
```

To see how different headers come out, we build a few requests by hand, with no server involved:

```scala
def ordersRequest(authorization: Option[String]): Request = {
  val request = Request.get(URL.parse("/orders").fold(error => sys.error(error), url => url))
  authorization.fold(request)(value => request.addHeader("authorization", value))
}
```

A bearer header gives the token whatever the case of the scheme name; a Basic header, a header with no token and a missing header all give `None`:

```scala
println(bearerToken(ordersRequest(Some("Bearer eyJhbGciOiJIUzI1NiJ9.e30.c2ln"))))
// Some(eyJhbGciOiJIUzI1NiJ9.e30.c2ln)
println(bearerToken(ordersRequest(Some("bearer eyJhbGciOiJIUzI1NiJ9.e30.c2ln"))))
// Some(eyJhbGciOiJIUzI1NiJ9.e30.c2ln)
println(bearerToken(ordersRequest(Some("Basic cmVhZGVyOnNlY3JldA=="))))
// None
println(bearerToken(ordersRequest(Some("Bearer"))))
// None
println(bearerToken(ordersRequest(None)))
// None
```

`Bearer` with no token is not a valid header, and `Headers#get` skips values it cannot parse (see the [headers reference](../reference/http-model/headers.md)), so it reads as a missing header. All three `None` cases get the same answer: a client that sent no usable token is asked to sign in.

:::warning
Read the token only from the `Authorization` header. Tokens in query strings end up in access logs, browser history and `Referer` headers.
:::

## Verifying the Token With Jwt.decode

Before we can verify tokens, someone has to issue them. In production, an auth service does that. In this guide, a small helper plays the auth service and signs tokens with a shared HMAC secret, at a fixed time so that every result is the same on every run. Each token carries the user's roles in a custom `roles` claim:

```scala
object TestTokens {
  // 2026-01-01T00:00:00Z. The API verifies tokens one minute later.
  val now: Long = 1767225600L

  // Sample data only. A real HS256 secret is at least 32 random bytes, loaded from a secret store.
  val secret: String = "sample-bookstore-secret-32-bytes"

  // Registered claims (iss, sub, aud, iat, exp) go in their own fields, never in `extra`.
  def issue(
    subject: String,
    roles: List[String],
    audience: String = "bookstore-api",
    expiresAt: Option[Long] = Some(now + 3600),
    key: String = secret
  ): String = {
    val claims = JwtClaims(
      iss = Some("bookstore-auth"),
      sub = Some(subject),
      aud = Some(JwtAudience.single(audience)),
      iat = Some(now),
      exp = expiresAt,
      extra = Map("roles" -> JwtValue.Arr(Chunk.fromIterable(roles.map(role => JwtValue.Str(role)))))
    )
    Jwt.sign(claims, key.getBytes(UTF_8), Algorithm.HS256).fold(error => throw error, token => token)
  }

  val reader: String        = issue("reader-1024", List("reader"))
  val admin: String         = issue("staff-0007", List("reader", "admin"))
  val expired: String       = issue("reader-1024", List("reader"), expiresAt = Some(now - 600))
  val otherAudience: String = issue("reader-1024", List("reader"), audience = "partner-api")
  val forged: String        = issue("staff-0007", List("admin"), key = "attacker-guessed-secret-32-bytes")
  val neverExpires: String  = issue("reader-1024", List("reader"), expiresAt = None)
}
```

:::caution
Keep registered claims such as `iss` and `exp` out of `JwtClaims#extra`. If `JwtClaims#extra` also holds one, `Jwt.sign` writes the claim twice and verifiers read the `JwtClaims#extra` value.
:::

On the API side, a handler should not work with raw claims. We turn a verified token into a `Caller`: who the user is and what roles they have. A `Verifier` is any function from a token to a `Caller` or a `JwtError`:

```scala
final case class Caller(subject: String, roles: Set[String])

type Verifier = String => Either[JwtError, Caller]

def rolesOf(claims: JwtClaims): Set[String] =
  claims.extra.get("roles") match {
    case Some(JwtValue.Arr(items)) => items.collect { case JwtValue.Str(role) => role }.toSet
    case _                         => Set.empty
  }

def toCaller(claims: JwtClaims): Either[JwtError, Caller] =
  (claims.sub, claims.exp) match {
    case (_, None)          => Left(JwtError.MissingClaim("exp"))
    case (None, _)          => Left(JwtError.MissingClaim("sub"))
    case (Some(subject), _) => Right(Caller(subject, rolesOf(claims)))
  }

def jwtVerifier(key: Array[Byte], options: JwtDecodeOptions): Verifier =
  token => Jwt.decode(token, key, Algorithm.HS256, options).flatMap(toCaller)
```

`Jwt.decode` checks the algorithm, the signature (in constant time), and then the claims you configure. It checks `exp` only when the token has one, so a token without `exp` never expires. `toCaller` closes that gap by requiring both `sub` and `exp`.

We build the verifier with the options our API needs: tokens from `bookstore-auth`, meant for `bookstore-api`, with 30 seconds of tolerance for clock differences between servers. `nowSeconds` fixes the clock one minute after the tokens were issued:

```scala
val verify: Verifier = jwtVerifier(
  TestTokens.secret.getBytes(UTF_8),
  JwtDecodeOptions(
    issuer = Some("bookstore-auth"),
    expectedAudience = Some("bookstore-api"),
    clockSkewSeconds = 30,
    nowSeconds = Some(TestTokens.now + 60)
  )
)

def report(name: String, token: String): Unit =
  verify(token) match {
    case Right(caller) => println(s"$name: accepted $caller")
    case Left(error)   => println(s"$name: rejected ${error.getClass.getSimpleName}: ${error.message}")
  }
```

Running every test token through it accepts the two good ones and rejects each bad one with its own reason:

```scala
report("reader", TestTokens.reader)
// reader: accepted Caller(reader-1024,Set(reader))
report("admin", TestTokens.admin)
// admin: accepted Caller(staff-0007,Set(reader, admin))
report("expired", TestTokens.expired)
// expired: rejected ExpiredToken: JWT token expired at 1767225000 (now: 1767225660)
report("other audience", TestTokens.otherAudience)
// other audience: rejected InvalidClaim: Invalid JWT claim 'aud': expected bookstore-api but found partner-api
report("forged", TestTokens.forged)
// forged: rejected InvalidSignature: JWT signature verification failed for algorithm HS256
report("never expires", TestTokens.neverExpires)
// never expires: rejected MissingClaim: Missing JWT claim: exp
```

:::note
In production, leave `nowSeconds` unset. `Jwt.decode` then uses the system clock.
:::

## Answering 401, 403 or 404

A protected endpoint has three ways to say no, and HTTP clients react differently to each. RFC 6750, which defines bearer tokens over HTTP, sets the rules:

| Situation                                             | Status             | `WWW-Authenticate`                                          |
| ----------------------------------------------------- | ------------------ | ----------------------------------------------------------- |
| No token, or another scheme such as Basic             | `401 Unauthorized` | `Bearer realm="bookstore"`                                  |
| Token is forged, expired, for another audience, ...   | `401 Unauthorized` | `Bearer realm="bookstore", error="invalid_token"`           |
| Token is valid, but the caller lacks the needed role  | `403 Forbidden`    | `Bearer realm="bookstore", error="insufficient_scope"`      |

A 401 means "who are you?": the client should sign in, or refresh its token, and try again. Every 401 must carry a `WWW-Authenticate` challenge that names the scheme. A 403 means "I know who you are, and the answer is no": a new token for the same user does not help.

`Response.unauthorized` and `Response.forbidden` carry no headers, so we build the three responses ourselves and add the typed `Header.WWWAuthenticate` with `Response#addHeader`:

```scala
val realm = "bookstore"

val missingToken: Response =
  Response
    .text(Status.Unauthorized, "Sign in to continue")
    .addHeader(Header.WWWAuthenticate("Bearer", Map("realm" -> realm)))

val invalidToken: Response =
  Response
    .text(Status.Unauthorized, "The access token is invalid or expired")
    .addHeader(Header.WWWAuthenticate("Bearer", Map("realm" -> realm, "error" -> "invalid_token")))

def missingRole(role: String): Response =
  Response
    .text(Status.Forbidden, s"This action needs the $role role")
    .addHeader(Header.WWWAuthenticate("Bearer", Map("realm" -> realm, "error" -> "insufficient_scope")))
```

The challenge for an invalid token renders like this:

```scala
println(invalidToken.header(Header.WWWAuthenticate).map(_.renderedValue))
// Some(Bearer realm="bookstore", error="invalid_token")
```

The bodies are deliberately vague. Every bad token gets the same answer, so a caller cannot use the API to learn which check failed. The precise reason goes to the server log instead.

:::caution
`Header.WWWAuthenticate` writes each parameter value between double quotes as it is, without escaping, so a value that contains `"` produces an invalid header. Use fixed strings such as the ones above, and never copy a `JwtError` message or any request data into the challenge.
:::

Some APIs answer 404 instead, as if the endpoint did not exist, which is why `AuthType` defaults to `Status.NotFound`. That hides admin routes from someone probing the API, but it costs a lot: clients can no longer tell an expired token from a wrong URL, so they cannot refresh and retry, and debugging gets harder. If you hide endpoints, answer 404 for *every* rejection of that endpoint, including the 403 case, or the difference between the answers gives the secret away. For an API whose routes are documented anyway, prefer 401 and 403.

## Handing the Caller to the Handler

Now we combine the pieces into one wrapper. `authenticated` takes a verifier, an optional role, and a handler that receives the verified `Caller` along with the request. It returns a plain `Request => Response`, so it fits any router, and the handler cannot run without a verified caller:

```scala
def authenticated(verify: Verifier, requiredRole: Option[String] = None)(
  handler: (Caller, Request) => Response
): Request => Response =
  request =>
    bearerToken(request) match {
      case None        => missingToken
      case Some(token) =>
        verify(token) match {
          case Left(error) =>
            // Log why, never the token itself.
            println(s"auth: rejected token: ${error.getClass.getSimpleName}")
            invalidToken
          case Right(caller) if requiredRole.forall(caller.roles.contains) => handler(caller, request)
          case Right(caller)                                               =>
            println(s"auth: ${caller.subject} lacks role ${requiredRole.getOrElse("")}")
            missingRole(requiredRole.getOrElse(""))
        }
    }
```

The rejection log names the kind of failure and, once the token is verified, the user. It never prints the token, which would let anyone who reads the log replay it. In a real service, send these lines to your logger instead of `println`.

The protected bookstore uses the caller's `subject` to pick the orders, so the `?user=` parameter is gone, and it asks for the `admin` role before changing a price:

```scala
def bookstoreApp(verify: Verifier): Request => Response = {
  val showOrders = authenticated(verify)((caller, _) => ordersOf(caller.subject))

  def changePrice(id: String) =
    authenticated(verify, requiredRole = Some("admin"))((_, request) => setPrice(id, request.body.text))

  request =>
    (request.method, request.path.segments.toList) match {
      case (Method.GET, List("books"))              => listBooks
      case (Method.GET, List("orders"))             => showOrders(request)
      case (Method.PUT, List("books", id, "price")) => changePrice(id)(request)
      case _                                        => notFound(request)
    }
}
```

When we call it with no token, as a reader and as an admin, the server's log lines appear just before the response they explain:

```scala
JdkHttp.withServer(bookstoreApp(verify)) { port =>
  Client.call(port, "GET", "/books")
  Client.call(port, "GET", "/orders")
  Client.call(port, "GET", "/orders", authorization = s"Bearer ${TestTokens.reader}")
  Client.call(port, "PUT", "/books/1/price", "3999", authorization = s"Bearer ${TestTokens.reader}")
  Client.call(port, "PUT", "/books/1/price", "3999", authorization = s"Bearer ${TestTokens.admin}")
}
// GET /books -> 200 [text/plain; charset=UTF-8, 93 bytes]
// 1. Zionomicon $45.00
// 2. Programming in Scala $59.99
// 3. Functional Programming in Scala $49.99
// GET /orders -> 401 [text/plain; charset=UTF-8, 19 bytes]
// www-authenticate: Bearer realm="bookstore"
// Sign in to continue
// GET /orders -> 200 [text/plain; charset=UTF-8, 66 bytes]
// Orders of reader-1024: Zionomicon, Functional Programming in Scala
// auth: reader-1024 lacks role admin
// PUT /books/1/price -> 403 [text/plain; charset=UTF-8, 32 bytes]
// www-authenticate: Bearer realm="bookstore", error="insufficient_scope"
// This action needs the admin role
// PUT /books/1/price -> 200 [text/plain; charset=UTF-8, 27 bytes]
// Zionomicon now costs $39.99
```

The public route still answers everyone, a missing token gets a challenge, the reader sees only their own orders and gets 403 on the admin route, and the admin can change the price.

## Loading the Key From Configuration

The signing key, the issuer and the audience belong in configuration, not in code. We describe them as a case class and keep the HMAC secret in a `Secret`, whose `toString` is always `<secret>`:

```scala
final case class AuthConfig(issuer: String, audience: String, hmacSecret: Secret)

object AuthConfig {
  implicit val schema: Schema[AuthConfig] = Schema.derived
}
```

`Secret.unwrap` is the only way to read the value, and we call it exactly once, to turn the secret into key bytes. HS256 needs a key of at least 32 bytes, and `Jwt.decode` rejects a shorter one on every request, so we check the length once at startup and refuse to start instead:

```scala
def verifierFrom(config: AuthConfig, nowSeconds: Option[Long] = None): Either[String, Verifier] = {
  val key = Secret.unwrap(config.hmacSecret).getBytes(UTF_8)
  if (key.length < 32) Left(s"hmacSecret has ${key.length} bytes; HS256 needs at least 32")
  else
    Right(
      jwtVerifier(
        key,
        JwtDecodeOptions(
          issuer = Some(config.issuer),
          expectedAudience = Some(config.audience),
          clockSkewSeconds = 30,
          nowSeconds = nowSeconds
        )
      )
    )
}
```

To keep the output stable, we load the configuration from a map with `ConfigSource.fromMap`. In production the same keys come from environment variables or a YAML or HOCON file, as [Build a Typed Configuration Loader](./typed-config-loader.md) shows. The `startWith` helper plays the startup of our service: it loads the configuration, builds the verifier and checks the reader's token:

```scala
val settings: Map[String, String] = Map(
  "issuer"     -> "bookstore-auth",
  "audience"   -> "bookstore-api",
  "hmacSecret" -> TestTokens.secret
)

def startWith(settings: Map[String, String]): Unit =
  Config.load[AuthConfig](ConfigSource.fromMap(settings, "env")) match {
    case Left(errors)  => errors.foreach(error => println(s"cannot start: ${error.message}"))
    case Right(config) =>
      println(s"loaded $config")
      verifierFrom(config, nowSeconds = Some(TestTokens.now + 60)) match {
        case Left(problem) => println(s"cannot start: $problem")
        case Right(verify) => println(s"reader token: ${verify(TestTokens.reader)}")
      }
  }
```

Starting with the full settings, with no secret, and with a secret that is too short:

```scala
startWith(settings)
// loaded AuthConfig(bookstore-auth,bookstore-api,<secret>)
// reader token: Right(Caller(reader-1024,Set(reader)))
startWith(settings - "hmacSecret")
// cannot start: Missing required key 'hmacSecret' in source 'env'
startWith(settings.updated("hmacSecret", "too-short"))
// loaded AuthConfig(bookstore-auth,bookstore-api,<secret>)
// cannot start: hmacSecret has 9 bytes; HS256 needs at least 32
```

The loaded configuration prints with `<secret>` in place of the key, so it is safe to log at startup.

:::tip
For an RSA, EC or EdDSA key pair, the API needs only the public key, which is not secret: load it as a Base64-encoded X.509 DER `String`, decode it with `java.util.Base64`, and pass the matching `Algorithm` to `Jwt.decode`. Call `JvmJwtCryptoBackend.install()` once at startup to enable these algorithms. [How to Issue and Verify JWTs](./issue-and-verify-jwts.md) shows the key formats. The module has no JWK or JWKS support, so it cannot fetch keys from an identity provider's key-set URL: fetch and convert them yourself.
:::

## Calling the Protected API

With the verifier loaded from configuration, we can send the API every kind of request it will meet in practice: no token, a Basic header, each bad token from the test set, a reader's token and an admin's token:

```scala
val configured: Verifier = Config
  .load[AuthConfig](ConfigSource.fromMap(settings, "env"))
  .left
  .map(_.map(_.message).mkString("; "))
  .flatMap(verifierFrom(_, nowSeconds = Some(TestTokens.now + 60)))
  .fold(sys.error, identity)

def bearer(token: String): String = s"Bearer $token"
```

Serving the bookstore with this verifier and calling `GET /orders` once per case:

```scala
JdkHttp.withServer(bookstoreApp(configured)) { port =>
  Client.call(port, "GET", "/orders")
  Client.call(port, "GET", "/orders", authorization = "Basic cmVhZGVyOnNlY3JldA==")
  Client.call(port, "GET", "/orders", authorization = bearer(TestTokens.forged))
  Client.call(port, "GET", "/orders", authorization = bearer(TestTokens.expired))
  Client.call(port, "GET", "/orders", authorization = bearer(TestTokens.otherAudience))
  Client.call(port, "GET", "/orders", authorization = bearer(TestTokens.neverExpires))
  Client.call(port, "GET", "/orders", authorization = bearer(TestTokens.reader))
}
// GET /orders -> 401 [text/plain; charset=UTF-8, 19 bytes]
// www-authenticate: Bearer realm="bookstore"
// Sign in to continue
// GET /orders -> 401 [text/plain; charset=UTF-8, 19 bytes]
// www-authenticate: Bearer realm="bookstore"
// Sign in to continue
// auth: rejected token: InvalidSignature
// GET /orders -> 401 [text/plain; charset=UTF-8, 38 bytes]
// www-authenticate: Bearer realm="bookstore", error="invalid_token"
// The access token is invalid or expired
// auth: rejected token: ExpiredToken
// GET /orders -> 401 [text/plain; charset=UTF-8, 38 bytes]
// www-authenticate: Bearer realm="bookstore", error="invalid_token"
// The access token is invalid or expired
// auth: rejected token: InvalidClaim
// GET /orders -> 401 [text/plain; charset=UTF-8, 38 bytes]
// www-authenticate: Bearer realm="bookstore", error="invalid_token"
// The access token is invalid or expired
// auth: rejected token: MissingClaim
// GET /orders -> 401 [text/plain; charset=UTF-8, 38 bytes]
// www-authenticate: Bearer realm="bookstore", error="invalid_token"
// The access token is invalid or expired
// GET /orders -> 200 [text/plain; charset=UTF-8, 66 bytes]
// Orders of reader-1024: Zionomicon, Functional Programming in Scala
```

A request with no usable token gets a plain challenge. Each bad token gets the same `invalid_token` answer, while the log records the precise reason: a forged signature, an expired token, the wrong audience, or a token with no expiry.

## What This Code Does Not Do

The wrapper verifies tokens; it is not a complete security setup. Before you rely on it, check these points:

- **Use HTTPS.** A bearer token works for whoever holds it. Over plain HTTP, anyone on the network path can copy it. The loopback server in this guide is for examples only.
- **No revocation.** A signed token stays valid until it expires. Keep `exp` short, minutes rather than days, and refresh tokens on the client.
- **Roles are only as good as the issuer.** The API trusts the `roles` claim because the token is signed. Anyone who holds the signing key can mint an admin token, so prefer an asymmetric key pair when another service issues tokens: the API then holds only the public key.

The verifier also knows only one key, so rotating the signing key needs a little more code. The module does not pick a key by the header's `kid`, so we choose the key ourselves: during a rotation, a verifier tries the new key and then the old one until every token signed with the old key has expired:

```scala
def firstAccepting(verifiers: List[Verifier]): Verifier =
  token => {
    val results = verifiers.map(verify => verify(token))
    results.collectFirst { case accepted @ Right(_) => accepted }.getOrElse(results.head)
  }
```

## Putting It Together

The complete example loads the configuration, builds the verifier, serves the protected bookstore, and calls it with every token from this guide. It uses four shared files from [Running the Examples](#running-the-examples): the adapter `JdkHttp.scala`, the test client `Client.scala`, the model `Bookstore.scala` and the token issuer `TestTokens.scala`:

```scala title="jwt-examples/src/main/scala/protectendpointswithjwt/CompleteExample.scala"
package protectendpointswithjwt

/**
 * How to Protect an Endpoint with JWT Bearer Tokens — Complete Example
 *
 * A bookstore API with a public `GET /books`, a `GET /orders` for signed-in
 * users and a `PUT /books/{id}/price` for admins. The HMAC secret is loaded
 * from configuration as a `Secret`, every bearer token is verified with
 * `Jwt.decode`, and failures are answered with 401 or 403 and a
 * `WWW-Authenticate: Bearer` challenge. The program calls the API with no
 * token, bad tokens and good tokens, then stops the server.
 *
 * Run with: cd jwt-examples && sbt "runMain
 * protectendpointswithjwt.CompleteExample"
 */
object CompleteExample {

  final case class AuthConfig(issuer: String, audience: String, hmacSecret: Secret)

  object AuthConfig {
    implicit val schema: Schema[AuthConfig] = Schema.derived
  }

  final case class Caller(subject: String, roles: Set[String])

  type Verifier = String => Either[JwtError, Caller]

  def rolesOf(claims: JwtClaims): Set[String] =
    claims.extra.get("roles") match {
      case Some(JwtValue.Arr(items)) => items.collect { case JwtValue.Str(role) => role }.toSet
      case _                         => Set.empty
    }

  def toCaller(claims: JwtClaims): Either[JwtError, Caller] =
    (claims.sub, claims.exp) match {
      case (_, None)          => Left(JwtError.MissingClaim("exp"))
      case (None, _)          => Left(JwtError.MissingClaim("sub"))
      case (Some(subject), _) => Right(Caller(subject, rolesOf(claims)))
    }

  def verifierFrom(config: AuthConfig, nowSeconds: Option[Long] = None): Either[String, Verifier] = {
    val key     = Secret.unwrap(config.hmacSecret).getBytes(UTF_8)
    val options = JwtDecodeOptions(
      issuer = Some(config.issuer),
      expectedAudience = Some(config.audience),
      clockSkewSeconds = 30,
      nowSeconds = nowSeconds
    )
    if (key.length < 32) Left(s"hmacSecret has ${key.length} bytes; HS256 needs at least 32")
    else Right(token => Jwt.decode(token, key, Algorithm.HS256, options).flatMap(toCaller))
  }

  val realm = "bookstore"

  val missingToken: Response =
    Response
      .text(Status.Unauthorized, "Sign in to continue")
      .addHeader(Header.WWWAuthenticate("Bearer", Map("realm" -> realm)))

  val invalidToken: Response =
    Response
      .text(Status.Unauthorized, "The access token is invalid or expired")
      .addHeader(Header.WWWAuthenticate("Bearer", Map("realm" -> realm, "error" -> "invalid_token")))

  def missingRole(role: String): Response =
    Response
      .text(Status.Forbidden, s"This action needs the $role role")
      .addHeader(Header.WWWAuthenticate("Bearer", Map("realm" -> realm, "error" -> "insufficient_scope")))

  def bearerToken(request: Request): Option[String] =
    request.header(Header.Authorization).collect { case Header.Authorization.Bearer(token) => token }

  def authenticated(verify: Verifier, requiredRole: Option[String] = None)(
    handler: (Caller, Request) => Response
  ): Request => Response =
    request =>
      bearerToken(request) match {
        case None        => missingToken
        case Some(token) =>
          verify(token) match {
            case Left(error) =>
              // Log why, never the token itself.
              println(s"auth: rejected token: ${error.getClass.getSimpleName}")
              invalidToken
            case Right(caller) if requiredRole.forall(caller.roles.contains) => handler(caller, request)
            case Right(caller)                                               =>
              println(s"auth: ${caller.subject} lacks role ${requiredRole.getOrElse("")}")
              missingRole(requiredRole.getOrElse(""))
          }
      }

  def bookstoreApp(verify: Verifier): Request => Response = {
    val showOrders = authenticated(verify)((caller, _) => ordersOf(caller.subject))

    def changePrice(id: String) =
      authenticated(verify, requiredRole = Some("admin"))((_, request) => setPrice(id, request.body.text))

    request =>
      (request.method, request.path.segments.toList) match {
        case (Method.GET, List("books"))              => listBooks
        case (Method.GET, List("orders"))             => showOrders(request)
        case (Method.PUT, List("books", id, "price")) => changePrice(id)(request)
        case _                                        => notFound(request)
      }
  }

  // In production these values come from the environment or a config file.
  val settings: Map[String, String] = Map(
    "issuer"     -> "bookstore-auth",
    "audience"   -> "bookstore-api",
    "hmacSecret" -> TestTokens.secret
  )

  def main(args: Array[String]): Unit = {
    val config = Config.load[AuthConfig](ConfigSource.fromMap(settings, "env")) match {
      case Right(loaded) => loaded
      case Left(errors)  => sys.error(errors.map(_.message).mkString("; "))
    }
    println(s"loaded $config")

    // A fixed clock keeps the output stable; leave `nowSeconds` unset in production.
    val verify = verifierFrom(config, nowSeconds = Some(TestTokens.now + 60)).fold(sys.error, identity)

    JdkHttp.withServer(bookstoreApp(verify)) { port =>
      def bearer(token: String) = s"Bearer $token"

      Client.call(port, "GET", "/books")
      Client.call(port, "GET", "/orders")
      Client.call(port, "GET", "/orders", authorization = "Basic cmVhZGVyOnNlY3JldA==")
      Client.call(port, "GET", "/orders", authorization = bearer(TestTokens.forged))
      Client.call(port, "GET", "/orders", authorization = bearer(TestTokens.expired))
      Client.call(port, "GET", "/orders", authorization = bearer(TestTokens.otherAudience))
      Client.call(port, "GET", "/orders", authorization = bearer(TestTokens.neverExpires))
      Client.call(port, "GET", "/orders", authorization = bearer(TestTokens.reader))
      Client.call(port, "PUT", "/books/1/price", "3999", authorization = bearer(TestTokens.reader))
      Client.call(port, "PUT", "/books/1/price", "3999", authorization = bearer(TestTokens.admin))
    }
  }
}
```

## Running the Examples

All code from this guide is available as runnable examples in the `jwt-examples` directory. It is a standalone sbt build that uses the published ZIO Blocks artifacts. To run them, clone the repository and move into that directory:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/jwt-examples
```

The examples share four files. `JdkHttp.scala` is the adapter from the [JDK HTTP server guide](./serve-http-model-requests.md), unchanged except for its package. `Client.scala` is that guide's test client with two additions: an `authorization` parameter, and printing the `WWW-Authenticate` header:

<details>
  <summary>jwt-examples/src/main/scala/protectendpointswithjwt/JdkHttp.scala</summary>

```scala title="jwt-examples/src/main/scala/protectendpointswithjwt/JdkHttp.scala" showLineNumbers
package protectendpointswithjwt

/**
 * How to Serve http-model Requests with the JDK HTTP Server — the adapter
 *
 * Glue between the JDK's built-in `com.sun.net.httpserver.HttpServer` and a
 * plain `Request => Response` function. It is meant for examples, tools and
 * small services, not for production traffic.
 */
object JdkHttp {

  /**
   * Reads the method, URL, headers and body of an exchange into a `Request`.
   * Returns `Left` with a ready-made error response when the method is unknown
   * (501) or the request target is not a valid URL (400).
   */
  def toRequest(exchange: HttpExchange): Either[Response, Request] = {
    val uri    = exchange.getRequestURI
    val target = uri.getRawPath + Option(uri.getRawQuery).fold("")("?" + _)
    for {
      method <- Method.fromString(exchange.getRequestMethod).toRight(Response(Status.NotImplemented))
      url    <- URL.parse(target).left.map(error => Response.text(Status.BadRequest, error))
    } yield {
      val headers = exchange.getRequestHeaders.asScala.foldLeft(Headers.empty) { case (acc, (name, values)) =>
        values.asScala.foldLeft(acc)((withValue, value) => withValue.add(name, value))
      }
      val contentType =
        headers.get(Header.ContentType).map(_.value).getOrElse(ContentType.`application/octet-stream`)
      val body    = Body.fromArray(exchange.getRequestBody.readAllBytes(), contentType)
      val version = Version.fromString(exchange.getProtocol).getOrElse(Version.`HTTP/1.1`)
      Request(method, url, headers, body, version)
    }
  }

  /**
   * Writes the status, headers and body of a `Response` to an exchange. The JDK
   * server computes `Content-Length` itself, so the response's own
   * `content-length` and `transfer-encoding` headers are skipped.
   */
  def writeResponse(exchange: HttpExchange, response: Response): Unit = {
    val out = exchange.getResponseHeaders
    response.headers.toList.foreach { case (name, value) =>
      if (name != "content-length" && name != "transfer-encoding") out.add(name, value)
    }
    val bytes = response.body.toArray
    if (bytes.nonEmpty && !response.headers.has("content-type"))
      out.set("content-type", response.body.contentType.render)
    val code   = response.status.code
    val isHead = exchange.getRequestMethod == "HEAD"
    if (isHead && bytes.nonEmpty) out.set("content-length", bytes.length.toString)
    val noBody = isHead || code == 204 || code == 304 || bytes.isEmpty
    exchange.sendResponseHeaders(code, if (noBody) -1L else bytes.length.toLong)
    if (!noBody) exchange.getResponseBody.write(bytes)
  }

  /**
   * Turns an application into an `HttpHandler`. A `HEAD` request runs the `GET`
   * route and drops the body; a non-fatal exception or `LinkageError` in the
   * application becomes a 500 response; a failure while writing the response is
   * logged and, if no headers have gone out yet, also becomes a 500; the
   * exchange is closed on every path.
   */
  def handler(app: Request => Response): HttpHandler = exchange =>
    try {
      val response =
        try
          toRequest(exchange) match {
            case Left(error)                                     => error
            case Right(request) if request.method == Method.HEAD => app(request.copy(method = Method.GET))
            case Right(request)                                  => app(request)
          }
        catch {
          case error @ (NonFatal(_) | _: LinkageError) =>
            System.err.println(s"Handler failed: $error")
            Response.text(Status.InternalServerError, "Internal Server Error")
        }
      try writeResponse(exchange, response)
      catch {
        case error @ (NonFatal(_) | _: LinkageError) =>
          System.err.println(s"Writing the response failed: $error")
          if (exchange.getResponseCode == -1) {
            exchange.getResponseHeaders.clear()
            exchange.sendResponseHeaders(Status.InternalServerError.code, -1L)
          }
      }
    } finally exchange.close()

  /**
   * Starts a server on the loopback interface. Port 0 asks the operating system
   * for a free port; read it back with `server.getAddress.getPort`.
   */
  def start(app: Request => Response, port: Int = 0): HttpServer = {
    val server = HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress, port), 0)
    server.createContext("/", handler(app))
    server.start()
    server
  }

  /**
   * Starts a server on a free loopback port, passes the port to `use`, and
   * stops the server when `use` returns or throws.
   */
  def withServer[A](app: Request => Response)(use: Int => A): A = {
    val server = start(app)
    try use(server.getAddress.getPort)
    finally server.stop(0)
  }
}
```

</details>

<details>
  <summary>jwt-examples/src/main/scala/protectendpointswithjwt/Client.scala</summary>

```scala title="jwt-examples/src/main/scala/protectendpointswithjwt/Client.scala" showLineNumbers
package protectendpointswithjwt

/**
 * How to Serve http-model Requests with the JDK HTTP Server — a test client
 *
 * Sends one request with the JDK's `java.net.http.HttpClient` and prints the
 * status code, the `Content-Type` and `Content-Length` headers, and the body.
 *
 * Copied from the JDK HTTP server guide with two additions for this guide: an
 * optional `Authorization` header value to send, and the `WWW-Authenticate`
 * header printed when the response has one.
 */
object Client {
  private val client = HttpClient.newBuilder().version(HttpClient.Version.HTTP_1_1).build()

  def call(port: Int, method: String, target: String, body: String = "", authorization: String = ""): Unit = {
    val plain   = HttpRequest.newBuilder(URI.create(s"http://127.0.0.1:$port$target"))
    val builder = if (authorization.isEmpty) plain else plain.header("authorization", authorization)
    val request =
      if (body.isEmpty) builder.method(method, HttpRequest.BodyPublishers.noBody()).build()
      else
        builder
          .method(method, HttpRequest.BodyPublishers.ofString(body))
          .header("content-type", "application/json")
          .build()
    val response                     = client.send(request, HttpResponse.BodyHandlers.ofString())
    def header(name: String): String = response.headers.firstValue(name).orElse("none")
    println(s"$method $target -> ${response.statusCode} [${header("content-type")}, ${header("content-length")} bytes]")
    response.headers.firstValue("www-authenticate").ifPresent(value => println(s"www-authenticate: $value"))
    if (response.body.nonEmpty) println(response.body)
  }
}
```

</details>

<details>
  <summary>jwt-examples/src/main/scala/protectendpointswithjwt/Bookstore.scala</summary>

```scala title="jwt-examples/src/main/scala/protectendpointswithjwt/Bookstore.scala" showLineNumbers
package protectendpointswithjwt

/**
 * How to Protect an Endpoint with JWT Bearer Tokens — the bookstore model
 *
 * Three Scala books, the orders of two fictional users, and the plain responses
 * that the bookstore API sends. Nothing here knows about authentication.
 */
final case class Book(id: Int, title: String, priceCents: Long)

object Bookstore {
  val books: List[Book] = List(
    Book(1, "Zionomicon", 4500),
    Book(2, "Programming in Scala", 5999),
    Book(3, "Functional Programming in Scala", 4999)
  )

  // Orders per user, keyed by the `sub` claim of their token.
  val orders: Map[String, List[String]] = Map(
    "reader-1024" -> List("Zionomicon", "Functional Programming in Scala"),
    "staff-0007"  -> List("Programming in Scala")
  )

  def price(cents: Long): String = f"$$${cents / 100}%d.${cents % 100}%02d"

  def listBooks: Response =
    Response.text(books.map(book => s"${book.id}. ${book.title} ${price(book.priceCents)}").mkString("\n"))

  def ordersOf(user: String): Response =
    Response.text(s"Orders of $user: ${orders.getOrElse(user, Nil).mkString(", ")}")

  // Answers with the new price; a real store would also save it.
  def setPrice(id: String, body: String): Response =
    (books.find(_.id.toString == id), body.trim.toLongOption.filter(_ > 0)) match {
      case (None, _)                 => Response.text(Status.NotFound, s"No book with id $id")
      case (_, None)                 => Response.text(Status.BadRequest, "The body must be a price in cents")
      case (Some(book), Some(cents)) => Response.text(s"${book.title} now costs ${price(cents)}")
    }

  def notFound(request: Request): Response =
    Response.text(Status.NotFound, s"No route for ${request.method.name} ${request.path.encode}")
}
```

</details>

<details>
  <summary>jwt-examples/src/main/scala/protectendpointswithjwt/TestTokens.scala</summary>

```scala title="jwt-examples/src/main/scala/protectendpointswithjwt/TestTokens.scala" showLineNumbers
package protectendpointswithjwt

/**
 * How to Protect an Endpoint with JWT Bearer Tokens — test tokens
 *
 * Plays the auth service: signs the tokens that the examples send to the
 * bookstore API. Every token is issued at a fixed time, so the results are the
 * same on every run.
 */
object TestTokens {

  // 2026-01-01T00:00:00Z. The API verifies tokens one minute later.
  val now: Long = 1767225600L

  // Sample data only. A real HS256 secret is at least 32 random bytes, loaded from a secret store.
  val secret: String = "sample-bookstore-secret-32-bytes"

  // Registered claims (iss, sub, aud, iat, exp) go in their own fields, never in `extra`.
  def issue(
    subject: String,
    roles: List[String],
    audience: String = "bookstore-api",
    expiresAt: Option[Long] = Some(now + 3600),
    key: String = secret
  ): String = {
    val claims = JwtClaims(
      iss = Some("bookstore-auth"),
      sub = Some(subject),
      aud = Some(JwtAudience.single(audience)),
      iat = Some(now),
      exp = expiresAt,
      extra = Map("roles" -> JwtValue.Arr(Chunk.fromIterable(roles.map(role => JwtValue.Str(role)))))
    )
    Jwt.sign(claims, key.getBytes(UTF_8), Algorithm.HS256).fold(error => throw error, token => token)
  }

  val reader: String        = issue("reader-1024", List("reader"))
  val admin: String         = issue("staff-0007", List("reader", "admin"))
  val expired: String       = issue("reader-1024", List("reader"), expiresAt = Some(now - 600))
  val otherAudience: String = issue("reader-1024", List("reader"), audience = "partner-api")
  val forged: String        = issue("staff-0007", List("admin"), key = "attacker-guessed-secret-32-bytes")
  val neverExpires: String  = issue("reader-1024", List("reader"), expiresAt = None)
}
```

</details>

### Serve the Unprotected API

The first example serves the bookstore with no authentication and shows that anyone can read any user's orders and change a price:

<details>
  <summary>jwt-examples/src/main/scala/protectendpointswithjwt/Step1UnprotectedApiExample.scala</summary>

```scala title="jwt-examples/src/main/scala/protectendpointswithjwt/Step1UnprotectedApiExample.scala" showLineNumbers
package protectendpointswithjwt

/**
 * How to Protect an Endpoint with JWT Bearer Tokens: Step 1 — The Unprotected
 * API
 *
 * Serves the bookstore with no authentication: the caller names whose orders to
 * show, and anyone can change a price.
 *
 * Run with: cd jwt-examples && sbt "runMain
 * protectendpointswithjwt.Step1UnprotectedApiExample"
 */
object Step1UnprotectedApiExample {

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, List("books"))              => listBooks
      case (Method.GET, List("orders"))             => ordersOf(request.queryParams.getFirst("user").getOrElse(""))
      case (Method.PUT, List("books", id, "price")) => setPrice(id, request.body.text)
      case _                                        => notFound(request)
    }

  def main(args: Array[String]): Unit =
    JdkHttp.withServer(app) { port =>
      Client.call(port, "GET", "/books")
      Client.call(port, "GET", "/orders?user=reader-1024")
      Client.call(port, "PUT", "/books/1/price", "1")
    }
}
```

</details>

To run it:

```bash
sbt "runMain protectendpointswithjwt.Step1UnprotectedApiExample"
```

### Read the Bearer Token

The second example reads the token from five different `Authorization` headers:

<details>
  <summary>jwt-examples/src/main/scala/protectendpointswithjwt/Step2ReadBearerTokenExample.scala</summary>

```scala title="jwt-examples/src/main/scala/protectendpointswithjwt/Step2ReadBearerTokenExample.scala" showLineNumbers
package protectendpointswithjwt

/**
 * How to Protect an Endpoint with JWT Bearer Tokens: Step 2 — Reading the
 * Bearer Token
 *
 * Reads the token from the `Authorization` header with the typed header API,
 * for a Bearer header, a Basic header, a header with no token and no header.
 *
 * Run with: cd jwt-examples && sbt "runMain
 * protectendpointswithjwt.Step2ReadBearerTokenExample"
 */
object Step2ReadBearerTokenExample {

  def bearerToken(request: Request): Option[String] =
    request.header(Header.Authorization).collect { case Header.Authorization.Bearer(token) => token }

  def ordersRequest(authorization: Option[String]): Request = {
    val request = Request.get(URL.parse("/orders").fold(error => sys.error(error), url => url))
    authorization.fold(request)(value => request.addHeader("authorization", value))
  }

  def main(args: Array[String]): Unit = {
    println(bearerToken(ordersRequest(Some("Bearer eyJhbGciOiJIUzI1NiJ9.e30.c2ln"))))
    println(bearerToken(ordersRequest(Some("bearer eyJhbGciOiJIUzI1NiJ9.e30.c2ln"))))
    println(bearerToken(ordersRequest(Some("Basic cmVhZGVyOnNlY3JldA=="))))
    println(bearerToken(ordersRequest(Some("Bearer"))))
    println(bearerToken(ordersRequest(None)))
  }
}
```

</details>

To run it:

```bash
sbt "runMain protectendpointswithjwt.Step2ReadBearerTokenExample"
```

### Verify the Token

The third example verifies every test token and prints the caller or the reason for the rejection:

<details>
  <summary>jwt-examples/src/main/scala/protectendpointswithjwt/Step3VerifyTokenExample.scala</summary>

```scala title="jwt-examples/src/main/scala/protectendpointswithjwt/Step3VerifyTokenExample.scala" showLineNumbers
package protectendpointswithjwt

/**
 * How to Protect an Endpoint with JWT Bearer Tokens: Step 3 — Verifying the
 * Token
 *
 * Verifies each test token with `Jwt.decode`, checks issuer, audience and
 * expiry against a fixed clock, requires `sub` and `exp`, and turns the claims
 * into a `Caller`.
 *
 * Run with: cd jwt-examples && sbt "runMain
 * protectendpointswithjwt.Step3VerifyTokenExample"
 */
object Step3VerifyTokenExample {

  final case class Caller(subject: String, roles: Set[String])

  type Verifier = String => Either[JwtError, Caller]

  def rolesOf(claims: JwtClaims): Set[String] =
    claims.extra.get("roles") match {
      case Some(JwtValue.Arr(items)) => items.collect { case JwtValue.Str(role) => role }.toSet
      case _                         => Set.empty
    }

  def toCaller(claims: JwtClaims): Either[JwtError, Caller] =
    (claims.sub, claims.exp) match {
      case (_, None)          => Left(JwtError.MissingClaim("exp"))
      case (None, _)          => Left(JwtError.MissingClaim("sub"))
      case (Some(subject), _) => Right(Caller(subject, rolesOf(claims)))
    }

  def jwtVerifier(key: Array[Byte], options: JwtDecodeOptions): Verifier =
    token => Jwt.decode(token, key, Algorithm.HS256, options).flatMap(toCaller)

  val verify: Verifier = jwtVerifier(
    TestTokens.secret.getBytes(UTF_8),
    JwtDecodeOptions(
      issuer = Some("bookstore-auth"),
      expectedAudience = Some("bookstore-api"),
      clockSkewSeconds = 30,
      nowSeconds = Some(TestTokens.now + 60)
    )
  )

  def report(name: String, token: String): Unit =
    verify(token) match {
      case Right(caller) => println(s"$name: accepted $caller")
      case Left(error)   => println(s"$name: rejected ${error.getClass.getSimpleName}: ${error.message}")
    }

  def main(args: Array[String]): Unit = {
    report("reader", TestTokens.reader)
    report("admin", TestTokens.admin)
    report("expired", TestTokens.expired)
    report("other audience", TestTokens.otherAudience)
    report("forged", TestTokens.forged)
    report("never expires", TestTokens.neverExpires)
  }
}
```

</details>

To run it:

```bash
sbt "runMain protectendpointswithjwt.Step3VerifyTokenExample"
```

### Protect the Routes

The fourth example adds the 401 and 403 responses and the `authenticated` wrapper, and calls the protected API:

<details>
  <summary>jwt-examples/src/main/scala/protectendpointswithjwt/Step4ProtectedRoutesExample.scala</summary>

```scala title="jwt-examples/src/main/scala/protectendpointswithjwt/Step4ProtectedRoutesExample.scala" showLineNumbers
package protectendpointswithjwt

/**
 * How to Protect an Endpoint with JWT Bearer Tokens: Step 4 — Protecting the
 * Routes
 *
 * Wraps the handlers of `GET /orders` and `PUT /books/{id}/price` in
 * `authenticated`, which answers 401 with a `WWW-Authenticate: Bearer`
 * challenge for a missing or bad token, 403 when the caller lacks the role, and
 * otherwise passes the verified caller to the handler.
 *
 * Run with: cd jwt-examples && sbt "runMain
 * protectendpointswithjwt.Step4ProtectedRoutesExample"
 */
object Step4ProtectedRoutesExample {

  val realm = "bookstore"

  val missingToken: Response =
    Response
      .text(Status.Unauthorized, "Sign in to continue")
      .addHeader(Header.WWWAuthenticate("Bearer", Map("realm" -> realm)))

  val invalidToken: Response =
    Response
      .text(Status.Unauthorized, "The access token is invalid or expired")
      .addHeader(Header.WWWAuthenticate("Bearer", Map("realm" -> realm, "error" -> "invalid_token")))

  def missingRole(role: String): Response =
    Response
      .text(Status.Forbidden, s"This action needs the $role role")
      .addHeader(Header.WWWAuthenticate("Bearer", Map("realm" -> realm, "error" -> "insufficient_scope")))

  def authenticated(verify: Verifier, requiredRole: Option[String] = None)(
    handler: (Caller, Request) => Response
  ): Request => Response =
    request =>
      bearerToken(request) match {
        case None        => missingToken
        case Some(token) =>
          verify(token) match {
            case Left(error) =>
              // Log why, never the token itself.
              println(s"auth: rejected token: ${error.getClass.getSimpleName}")
              invalidToken
            case Right(caller) if requiredRole.forall(caller.roles.contains) => handler(caller, request)
            case Right(caller)                                               =>
              println(s"auth: ${caller.subject} lacks role ${requiredRole.getOrElse("")}")
              missingRole(requiredRole.getOrElse(""))
          }
      }

  def bookstoreApp(verify: Verifier): Request => Response = {
    val showOrders = authenticated(verify)((caller, _) => ordersOf(caller.subject))

    def changePrice(id: String) =
      authenticated(verify, requiredRole = Some("admin"))((_, request) => setPrice(id, request.body.text))

    request =>
      (request.method, request.path.segments.toList) match {
        case (Method.GET, List("books"))              => listBooks
        case (Method.GET, List("orders"))             => showOrders(request)
        case (Method.PUT, List("books", id, "price")) => changePrice(id)(request)
        case _                                        => notFound(request)
      }
  }

  def main(args: Array[String]): Unit =
    JdkHttp.withServer(bookstoreApp(verify)) { port =>
      Client.call(port, "GET", "/books")
      Client.call(port, "GET", "/orders")
      Client.call(port, "GET", "/orders", authorization = s"Bearer ${TestTokens.reader}")
      Client.call(port, "PUT", "/books/1/price", "3999", authorization = s"Bearer ${TestTokens.reader}")
      Client.call(port, "PUT", "/books/1/price", "3999", authorization = s"Bearer ${TestTokens.admin}")
    }
}
```

</details>

To run it:

```bash
sbt "runMain protectendpointswithjwt.Step4ProtectedRoutesExample"
```

### Load the Key From Configuration

The fifth example loads the issuer, the audience and the secret with `Config.load`, and refuses to start without a usable secret:

<details>
  <summary>jwt-examples/src/main/scala/protectendpointswithjwt/Step5KeyFromConfigExample.scala</summary>

```scala title="jwt-examples/src/main/scala/protectendpointswithjwt/Step5KeyFromConfigExample.scala" showLineNumbers
package protectendpointswithjwt

/**
 * How to Protect an Endpoint with JWT Bearer Tokens: Step 5 — Loading the Key
 * From Configuration
 *
 * Loads the issuer, the audience and the HMAC secret with `Config.load`, keeps
 * the secret in a `Secret` so it never shows up in logs, and refuses to start
 * with a secret shorter than HS256 needs.
 *
 * Run with: cd jwt-examples && sbt "runMain
 * protectendpointswithjwt.Step5KeyFromConfigExample"
 */
object Step5KeyFromConfigExample {

  final case class AuthConfig(issuer: String, audience: String, hmacSecret: Secret)

  object AuthConfig {
    implicit val schema: Schema[AuthConfig] = Schema.derived
  }

  def verifierFrom(config: AuthConfig, nowSeconds: Option[Long] = None): Either[String, Verifier] = {
    val key = Secret.unwrap(config.hmacSecret).getBytes(UTF_8)
    if (key.length < 32) Left(s"hmacSecret has ${key.length} bytes; HS256 needs at least 32")
    else
      Right(
        jwtVerifier(
          key,
          JwtDecodeOptions(
            issuer = Some(config.issuer),
            expectedAudience = Some(config.audience),
            clockSkewSeconds = 30,
            nowSeconds = nowSeconds
          )
        )
      )
  }

  // In production these values come from the environment or a config file; see the typed config guide.
  val settings: Map[String, String] = Map(
    "issuer"     -> "bookstore-auth",
    "audience"   -> "bookstore-api",
    "hmacSecret" -> TestTokens.secret
  )

  def startWith(settings: Map[String, String]): Unit =
    Config.load[AuthConfig](ConfigSource.fromMap(settings, "env")) match {
      case Left(errors)  => errors.foreach(error => println(s"cannot start: ${error.message}"))
      case Right(config) =>
        println(s"loaded $config")
        verifierFrom(config, nowSeconds = Some(TestTokens.now + 60)) match {
          case Left(problem) => println(s"cannot start: $problem")
          case Right(verify) => println(s"reader token: ${verify(TestTokens.reader)}")
        }
    }

  def main(args: Array[String]): Unit = {
    startWith(settings)
    startWith(settings - "hmacSecret")
    startWith(settings.updated("hmacSecret", "too-short"))
  }
}
```

</details>

To run it:

```bash
sbt "runMain protectendpointswithjwt.Step5KeyFromConfigExample"
```

### The Complete Example

To run the complete example shown under [Putting It Together](#putting-it-together):

```bash
sbt "runMain protectendpointswithjwt.CompleteExample"
```

### Compile All Examples

To compile all the examples without running them:

```bash
sbt compile
```

## Going Further

- [How to Protect Pages with Basic Auth or a Session Cookie](./protect-pages-with-basic-auth-or-a-session-cookie.md) protects browser pages with Basic credentials or a login form and a session cookie.
- [How to Issue and Verify JWTs](./issue-and-verify-jwts.md) covers asymmetric keys, PEM files, hostile-token limits and every `JwtError` case.
- The [JWT reference](../reference/jwt.md) lists every algorithm, key size rule and decode option.
- The [`AuthType` reference](../reference/endpoint/auth-type.md) shows the other schemes (Basic, Digest, custom headers) and how to combine them.
- The [headers reference](../reference/http-model/headers.md) lists the typed headers, including `Header.Authorization` and `Header.WWWAuthenticate`.
- To protect many routes, give `authenticated` to your router once per route group, or route with [`RoutePattern`](../reference/endpoint/route-pattern.md) and wrap the matched handler.

---

## How to Protect Pages with Basic Auth or a Session Cookie

In this guide, we protect the staff pages of a small coffee roastery's web shop so that only staff can read the open orders. We do it in the two ways most web applications use: HTTP Basic authentication, where the browser asks for a username and password and sends them with its later requests to the protected area, and a login form that sets a session cookie. By the end, you will have a server that sends people to a sign-in page, keeps them signed in with an `HttpOnly` cookie, signs them out on request, and still lets scripts download the orders with Basic credentials.

We start from pages with no protection at all and add one piece at a time:

- Check a password against a stored hash.
- Read Basic credentials and answer `401 Unauthorized` with a `WWW-Authenticate` challenge.
- Issue a random session id in a `ResponseCookie` with the right security attributes.
- Log in with an HTML form, read the cookie back on every request, and log out.
- Choose between the two approaches.

Every section that serves requests starts a real server on a free loopback port, calls it, and prints what came back.

## The Problem

The roastery's staff page lists the open orders, with the customers' names. It must not be public, but the server has no idea who is asking: an HTTP request carries only what the client chooses to send.

A first attempt often lets the login form set a cookie such as `user=ada`, and lets the staff page trust it. Here is that page handler, written against the JDK HTTP server API:

```scala

val adminHandler: HttpHandler = (exchange: HttpExchange) => {
  val cookies  = Option(exchange.getRequestHeaders.getFirst("Cookie")).getOrElse("")
  val signedIn = cookies.split(";").map(_.trim).contains("user=ada") // Set by the login form after a correct password
  val (status, body) =
    if (signedIn) (200, "Open orders: #101 Lena, #102 Marco, #103 Priya")
    else (403, "Forbidden")
  val bytes = body.getBytes(UTF_8)
  exchange.sendResponseHeaders(status, bytes.length.toLong)
  exchange.getResponseBody.write(bytes)
  exchange.close()
}
```

Anyone can send `Cookie: user=ada` without ever seeing the login form, so this page is not protected at all. The cookie also never expires on the server, signing out cannot revoke it, and a browser that is not signed in gets a bare 403 instead of the sign-in page.

ZIO Blocks gives us the typed pieces: `Header.Authorization.Basic` decodes Basic credentials, `Header.WWWAuthenticate` writes the challenge, `ResponseCookie` renders a cookie with its attributes, `Request#cookies` reads cookies back, and `Form` decodes a submitted form. It does not store users, check passwords or keep sessions: that is application code, and this guide shows it.

## Prerequisites

Add the HTTP model and the HTML module to your `build.sbt`. The HTML module builds the pages and escapes the text we put in them:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-http-model" % "0.0.56",
  "dev.zio" %% "zio-blocks-html"       % "0.0.56"
)

// Scala 3.9 and newer
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-http-model" % "0.0.56",
  "dev.zio" %% "zio-blocks-next-html"       % "0.0.56"
)
scalacOptions += "-experimental"
```

The code is the same for every Scala version. Password hashing, random session ids and constant-time comparison come from the JDK (`javax.crypto`, `java.security`), because ZIO Blocks has no module for them. The HTTP model depends on the ZIO Blocks streams module, which is compiled for Java 21, so run the program on **JDK 21 or newer**.

You should know the request and response types from the [HTTP model reference](../reference/http-model/model.md), and the HTML DSL from [Render Your First Server-Side HTML Page](./your-first-html-page.md). The 401 answer follows the pattern of [How to Protect an Endpoint with JWT Bearer Tokens](./protect-endpoints-with-jwt.md).

ZIO Blocks has no HTTP server. We serve requests with the small adapter from the guide [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md): `JdkHttp.withServer(app)(use)` starts the JDK's built-in server on a free loopback port, runs `use` with the port, and stops the server. `Client.call(port, method, target, form, authorization, cookie)` sends one request and prints the status, the `WWW-Authenticate`, `Location` and `Set-Cookie` headers when there are any, and the body. `Client.logIn(port, username, password)` posts the login form and returns the session cookie, ready to send back. Both files are embedded at the end of this section; copy them into your project.

We use these imports throughout the guide:

```scala

```

<details>
  <summary>The shared adapter, unchanged from the serving guide except for its package: <code>JdkHttp.scala</code></summary>

```scala title="http-model-examples/src/main/scala/protectpageswithbasicauthorasessioncookie/JdkHttp.scala" showLineNumbers
package protectpageswithbasicauthorasessioncookie

/**
 * How to Serve http-model Requests with the JDK HTTP Server — the adapter
 *
 * Glue between the JDK's built-in `com.sun.net.httpserver.HttpServer` and a
 * plain `Request => Response` function. It is meant for examples, tools and
 * small services, not for production traffic.
 */
object JdkHttp {

  /**
   * Reads the method, URL, headers and body of an exchange into a `Request`.
   * Returns `Left` with a ready-made error response when the method is unknown
   * (501) or the request target is not a valid URL (400).
   */
  def toRequest(exchange: HttpExchange): Either[Response, Request] = {
    val uri    = exchange.getRequestURI
    val target = uri.getRawPath + Option(uri.getRawQuery).fold("")("?" + _)
    for {
      method <- Method.fromString(exchange.getRequestMethod).toRight(Response(Status.NotImplemented))
      url    <- URL.parse(target).left.map(error => Response.text(Status.BadRequest, error))
    } yield {
      val headers = exchange.getRequestHeaders.asScala.foldLeft(Headers.empty) { case (acc, (name, values)) =>
        values.asScala.foldLeft(acc)((withValue, value) => withValue.add(name, value))
      }
      val contentType =
        headers.get(Header.ContentType).map(_.value).getOrElse(ContentType.`application/octet-stream`)
      val body    = Body.fromArray(exchange.getRequestBody.readAllBytes(), contentType)
      val version = Version.fromString(exchange.getProtocol).getOrElse(Version.`HTTP/1.1`)
      Request(method, url, headers, body, version)
    }
  }

  /**
   * Writes the status, headers and body of a `Response` to an exchange. The JDK
   * server computes `Content-Length` itself, so the response's own
   * `content-length` and `transfer-encoding` headers are skipped.
   */
  def writeResponse(exchange: HttpExchange, response: Response): Unit = {
    val out = exchange.getResponseHeaders
    response.headers.toList.foreach { case (name, value) =>
      if (name != "content-length" && name != "transfer-encoding") out.add(name, value)
    }
    val bytes = response.body.toArray
    if (bytes.nonEmpty && !response.headers.has("content-type"))
      out.set("content-type", response.body.contentType.render)
    val code   = response.status.code
    val isHead = exchange.getRequestMethod == "HEAD"
    if (isHead && bytes.nonEmpty) out.set("content-length", bytes.length.toString)
    val noBody = isHead || code == 204 || code == 304 || bytes.isEmpty
    exchange.sendResponseHeaders(code, if (noBody) -1L else bytes.length.toLong)
    if (!noBody) exchange.getResponseBody.write(bytes)
  }

  /**
   * Turns an application into an `HttpHandler`. A `HEAD` request runs the `GET`
   * route and drops the body; a non-fatal exception or `LinkageError` in the
   * application becomes a 500 response; a failure while writing the response is
   * logged and, if no headers have gone out yet, also becomes a 500; the
   * exchange is closed on every path.
   */
  def handler(app: Request => Response): HttpHandler = exchange =>
    try {
      val response =
        try
          toRequest(exchange) match {
            case Left(error)                                     => error
            case Right(request) if request.method == Method.HEAD => app(request.copy(method = Method.GET))
            case Right(request)                                  => app(request)
          }
        catch {
          case error @ (NonFatal(_) | _: LinkageError) =>
            System.err.println(s"Handler failed: $error")
            Response.text(Status.InternalServerError, "Internal Server Error")
        }
      try writeResponse(exchange, response)
      catch {
        case error @ (NonFatal(_) | _: LinkageError) =>
          System.err.println(s"Writing the response failed: $error")
          if (exchange.getResponseCode == -1) {
            exchange.getResponseHeaders.clear()
            exchange.sendResponseHeaders(Status.InternalServerError.code, -1L)
          }
      }
    } finally exchange.close()

  /**
   * Starts a server on the loopback interface. Port 0 asks the operating system
   * for a free port; read it back with `server.getAddress.getPort`.
   */
  def start(app: Request => Response, port: Int = 0): HttpServer = {
    val server = HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress, port), 0)
    server.createContext("/", handler(app))
    server.start()
    server
  }

  /**
   * Starts a server on a free loopback port, passes the port to `use`, and
   * stops the server when `use` returns or throws.
   */
  def withServer[A](app: Request => Response)(use: Int => A): A = {
    val server = start(app)
    try use(server.getAddress.getPort)
    finally server.stop(0)
  }
}
```

</details>

<details>
  <summary>The shared test client, which sends a form, an <code>Authorization</code> header or a cookie: <code>Client.scala</code></summary>

```scala title="http-model-examples/src/main/scala/protectpageswithbasicauthorasessioncookie/Client.scala" showLineNumbers
package protectpageswithbasicauthorasessioncookie

/**
 * How to Protect Pages with Basic Auth or a Session Cookie — a test client
 *
 * Sends one request with the JDK's `java.net.http.HttpClient`, optionally with
 * a form body, an `Authorization` header or a `Cookie` header, and prints the
 * status code, the `WWW-Authenticate`, `Location` and `Set-Cookie` headers, and
 * the body.
 */
object Client {
  private val client = HttpClient.newBuilder().version(HttpClient.Version.HTTP_1_1).build()

  private def send(
    port: Int,
    method: String,
    target: String,
    form: Form,
    authorization: String,
    cookie: String
  ): HttpResponse[String] = {
    val headers = List("authorization" -> authorization, "cookie" -> cookie).filter(_._2.nonEmpty)
    val builder = headers.foldLeft(HttpRequest.newBuilder(URI.create(s"http://127.0.0.1:$port$target"))) {
      case (withHeaders, (name, value)) => withHeaders.header(name, value)
    }
    val request =
      if (form.isEmpty) builder.method(method, HttpRequest.BodyPublishers.noBody()).build()
      else
        builder
          .method(method, HttpRequest.BodyPublishers.ofString(form.encode))
          .header("content-type", "application/x-www-form-urlencoded")
          .build()
    val response = client.send(request, HttpResponse.BodyHandlers.ofString())
    println(s"$method $target -> ${response.statusCode}")
    List("www-authenticate", "location", "set-cookie").foreach { name =>
      response.headers.allValues(name).forEach(value => println(s"$name: $value"))
    }
    if (response.body.nonEmpty) println(response.body)
    response
  }

  def call(
    port: Int,
    method: String,
    target: String,
    form: Form = Form.empty,
    authorization: String = "",
    cookie: String = ""
  ): Unit = {
    send(port, method, target, form, authorization, cookie)
    ()
  }

  // Posts the login form and returns the session cookie as `name=value`, ready for a `Cookie` header.
  def logIn(port: Int, username: String, password: String): String = {
    val form     = Form("username" -> username, "password" -> password)
    val response = send(port, "POST", "/login", form, "", "")
    response.headers.firstValue("set-cookie").orElse("").takeWhile(_ != ';')
  }
}
```

</details>

## The Roastery Admin Area

The roastery has a public home page, a staff page at `/admin` that lists the open orders, and a CSV export of the same orders at `/orders.csv` for spreadsheets and scripts. The pages are built with the HTML DSL, which escapes every string we pass as text, so a customer's name cannot inject markup. Nothing in this model knows about authentication:

```scala
final case class Order(id: Int, customer: String, coffee: String, grams: Int)

object Roastery {
  val openOrders: List[Order] = List(
    Order(101, "Lena", "Ethiopia Guji", 250),
    Order(102, "Marco", "Colombia Huila", 1000),
    Order(103, "Priya", "Kenya Nyeri", 500)
  )

  def htmlResponse(page: Dom.Element): Response =
    Response.ok(Body.fromArray((doctype.render + page.render(2)).getBytes(UTF_8), ContentType.`text/html`))

  def page(heading: String, content: Dom.Element*): Response =
    htmlResponse(
      html(lang := "en")(
        head(meta(charset := "utf-8"), title(heading)),
        body(h1(heading), content)
      )
    )

  val home: Response = page("Fresh Roasts Every Tuesday", a(href := "/admin")("Staff area"))

  // `note` says who is signed in; the session sections put a sign-out button in it.
  def ordersPage(note: Dom.Element): Response =
    page("Open Orders", ul(openOrders.map(o => li(s"#${o.id} ${o.customer}: ${o.coffee}, ${o.grams} g"))), note)

  val ordersCsv: Response =
    Response
      .text(("id,customer,coffee,grams" :: openOrders.map(o => s"${o.id},${o.customer},${o.coffee},${o.grams}")).mkString("\n"))
      .setHeader("content-type", "text/csv; charset=utf-8")

  def notFound(request: Request): Response =
    Response.text(Status.NotFound, s"No route for ${request.method.name} ${request.path.encode}")
}

```

The routing matches on the method and the path segments:

```scala
def unprotectedApp(request: Request): Response =
  (request.method, request.path.segments.toList) match {
    case (Method.GET, Nil)                => home
    case (Method.GET, List("admin"))      => ordersPage(p("Anyone can read this page."))
    case (Method.GET, List("orders.csv")) => ordersCsv
    case _                                => notFound(request)
  }
```

When we serve it and ask for the staff page and the export without any credentials, both answer with every customer's order:

```scala
JdkHttp.withServer(unprotectedApp) { port =>
  Client.call(port, "GET", "/admin")
  Client.call(port, "GET", "/orders.csv")
}
// GET /admin -> 200
// <!DOCTYPE html><html lang="en">
//   <head>
//     <meta charset="utf-8"/>
//     <title>Open Orders</title>
//   </head>
//   <body>
//     Open Orders
//     
//       #101 Lena: Ethiopia Guji, 250 g
//       #102 Marco: Colombia Huila, 1000 g
//       #103 Priya: Kenya Nyeri, 500 g
//     
//     Anyone can read this page.
//   </body>
// </html>
// GET /orders.csv -> 200
// id,customer,coffee,grams
// 101,Lena,Ethiopia Guji,250
// 102,Marco,Colombia Huila,1000
// 103,Priya,Kenya Nyeri,500
```

The rest of the guide keeps `/` public and lets only signed-in staff reach `/admin` and `/orders.csv`.

## Checking a Password

Both approaches end in the same question: is this the right password for this user? The server should never store the passwords themselves, only a slow hash of each one, so that a leaked user table does not leak the passwords. A salt, a random value stored next to each hash and mixed into it, makes two users with the same password get different hashes. The JDK ships PBKDF2, a standard password-hashing function: it repeats a keyed hash (HMAC-SHA256) hundreds of thousands of times over the password and the salt. We use 600,000 iterations, the OWASP recommendation for PBKDF2-HMAC-SHA256:

```scala
final case class PasswordHash(salt: Array[Byte], hash: Array[Byte])

object PasswordHash {
  val iterations: Int = 600000

  private val random = new SecureRandom()

  def derive(password: String, salt: Array[Byte]): Array[Byte] = {
    val spec = new PBEKeySpec(password.toCharArray, salt, iterations, 256)
    try SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256").generateSecret(spec).getEncoded
    finally spec.clearPassword()
  }

  def create(password: String): PasswordHash = {
    val salt = new Array[Byte](16)
    random.nextBytes(salt)
    PasswordHash(salt, derive(password, salt))
  }

  // MessageDigest.isEqual compares every byte, however early the first difference is.
  def matches(password: String, stored: PasswordHash): Boolean =
    MessageDigest.isEqual(derive(password, stored.salt), stored.hash)
}
```

`MessageDigest.isEqual` takes the same time wherever the first differing byte is, so the response time does not tell an attacker how close a guess came. `==` on strings stops at the first difference.

The staff accounts map each username to its hash. When the username is unknown, `Staff.authenticate` still checks the password against a dummy hash, so that a wrong username takes as long as a wrong password and the response time does not reveal which usernames exist:

```scala
object Staff {
  // Sample data only: a real service loads the salts and hashes from its user store and never sees the passwords.
  private val accounts: Map[String, PasswordHash] = Map(
    "ada"   -> PasswordHash.create("correct horse battery staple"),
    "grace" -> PasswordHash.create("tea and biscuits at four")
  )

  private val unknownUser: PasswordHash = PasswordHash.create("no such user")

  def authenticate(username: String, password: String): Option[String] = {
    val passwordMatches = PasswordHash.matches(password, accounts.getOrElse(username, unknownUser))
    if (passwordMatches && accounts.contains(username)) Some(username) else None
  }
}
```

The right password gives the username; a wrong password and an unknown user both give `None`:

```scala
println(Staff.authenticate("ada", "correct horse battery staple"))
// Some(ada)
println(Staff.authenticate("ada", "Correct horse battery staple"))
// None
println(Staff.authenticate("mallory", "correct horse battery staple"))
// None
```

:::note
Each check takes a noticeable fraction of a second on purpose: that cost is what slows down an attacker who has stolen the hashes. Keep it in mind for Basic authentication, which checks the password on every request.
:::

## Reading Basic Credentials

With HTTP Basic authentication, the client sends `Authorization: Basic <credentials>` with its requests to the protected area, where `<credentials>` is `username:password` in Base64. The HTTP model already decodes it: `Request#header(Header.Authorization)` returns a `Header.Authorization.Basic` with the username and the password, split at the first colon, so a password may contain colons. We only need to pick that case:

```scala
def basicCredentials(request: Request): Option[Header.Authorization.Basic] =
  request.header(Header.Authorization).collect { case basic: Header.Authorization.Basic => basic }
```

Base64 is an encoding, not encryption. `Header.Authorization.Basic` also renders the header for a client:

```scala
val adaHeader = Header.Authorization.Basic("ada", "correct horse battery staple").renderedValue
```

Printing the header and decoding it again shows the password in plain text:

```scala
println(adaHeader)
// Basic YWRhOmNvcnJlY3QgaG9yc2UgYmF0dGVyeSBzdGFwbGU=
println(new String(Base64.getDecoder.decode(adaHeader.stripPrefix("Basic ")), UTF_8))
// ada:correct horse battery staple
```

Anyone who can read the request can read the password, so serve Basic authentication only over HTTPS.

To see how different headers come out, we build a few requests by hand, with no server involved. The `describe` helper prints the username and only the length of the password:

```scala
def adminRequest(authorization: Option[String]): Request = {
  val request = Request.get(URL.parse("/admin").fold(error => sys.error(error), url => url))
  authorization.fold(request)(value => request.addHeader("authorization", value))
}

def describe(request: Request): String =
  basicCredentials(request).fold("no Basic credentials") { basic =>
    s"user ${basic.username}, password of ${basic.password.length} characters"
  }

def base64(text: String): String = Base64.getEncoder.encodeToString(text.getBytes(UTF_8))
```

The scheme name is case-insensitive and a colon in the password survives. Invalid Base64, another scheme and a missing header all give `None`:

```scala
println(describe(adminRequest(Some(adaHeader))))
// user ada, password of 28 characters
println(describe(adminRequest(Some("basic " + base64("ada:correct horse battery staple")))))
// user ada, password of 28 characters
println(describe(adminRequest(Some("Basic " + base64("grace:tea:and:biscuits")))))
// user grace, password of 16 characters
println(describe(adminRequest(Some("Basic not-base64!"))))
// no Basic credentials
println(describe(adminRequest(Some("Bearer eyJhbGciOiJIUzI1NiJ9.e30.c2ln"))))
// no Basic credentials
println(describe(adminRequest(None)))
// no Basic credentials
```

:::warning
`Header.Authorization.Basic` is a case class, so its `toString` prints the password. Never log the header, the request's headers, or a `Header.Authorization.Basic` value.
:::

If you describe your API with the [endpoint module](../reference/endpoint/index.md), [`AuthType.Basic`](../reference/endpoint/auth-type.md) declares that an endpoint needs Basic credentials and extracts them with the same `Header.Authorization.Basic`. Like `AuthType.Bearer` in the JWT guide, it checks nothing, and its default failure status is `404 Not Found`. Checking the credentials is the code in this guide.

## Asking for Credentials With a 401 Challenge

When a request has no credentials or wrong ones, the server answers `401 Unauthorized` with a `WWW-Authenticate` header that names the scheme. RFC 7617, which defines Basic authentication, adds a `realm`, the name of the protected area that a browser shows in its sign-in dialog, and an optional `charset="UTF-8"` that asks the client to encode non-ASCII usernames and passwords as UTF-8, which is what the HTTP model decodes. `Response.unauthorized` carries no headers, so we build the answer ourselves, as the JWT guide does:

```scala
val realm = "Roastery staff"

val basicChallenge: Response =
  Response
    .text(Status.Unauthorized, "Sign in to see the roastery's orders")
    .addHeader(Header.WWWAuthenticate("Basic", Map("realm" -> realm, "charset" -> "UTF-8")))
```

The challenge renders like this:

```scala
println(basicChallenge.header(Header.WWWAuthenticate).map(_.renderedValue))
// Some(Basic realm="Roastery staff", charset="UTF-8")
```

:::caution
`Header.WWWAuthenticate` writes each parameter value between double quotes as it is, without escaping, a known bug in 0.0.56. A realm that contains `"` or `\` produces an invalid header, and one that contains a comma does not parse back. Use a fixed realm such as the one above, and never copy request data into it.
:::

The `basicAuth` wrapper answers the challenge when there are no Basic credentials, checks them otherwise, and hands the authenticated username to the handler. Wrong credentials get the same challenge, so the browser asks again:

```scala
def basicAuth(handler: (String, Request) => Response): Request => Response =
  request =>
    basicCredentials(request) match {
      case None        => basicChallenge
      case Some(basic) =>
        Staff.authenticate(basic.username, basic.password) match {
          case Some(user) => handler(user, request)
          case None       =>
            println("auth: wrong Basic credentials") // Never log the password or the header.
            basicChallenge
        }
    }
```

We protect both the staff page and the export with it:

```scala
def basicApp(request: Request): Response =
  (request.method, request.path.segments.toList) match {
    case (Method.GET, Nil)                => home
    case (Method.GET, List("admin"))      => basicAuth((user, _) => ordersPage(p(s"Signed in as $user")))(request)
    case (Method.GET, List("orders.csv")) => basicAuth((_, _) => ordersCsv)(request)
    case _                                => notFound(request)
  }

def basic(username: String, password: String): String =
  Header.Authorization.Basic(username, password).renderedValue
```

When we call it with no credentials, a wrong password and Grace's account, the server's log line appears just before the response it explains:

```scala
JdkHttp.withServer(basicApp) { port =>
  Client.call(port, "GET", "/admin")
  Client.call(port, "GET", "/admin", authorization = basic("grace", "coffee and cake"))
  Client.call(port, "GET", "/admin", authorization = basic("grace", "tea and biscuits at four"))
  Client.call(port, "GET", "/orders.csv", authorization = basic("ada", "correct horse battery staple"))
}
// GET /admin -> 401
// www-authenticate: Basic realm="Roastery staff", charset="UTF-8"
// Sign in to see the roastery's orders
// auth: wrong Basic credentials
// GET /admin -> 401
// www-authenticate: Basic realm="Roastery staff", charset="UTF-8"
// Sign in to see the roastery's orders
// GET /admin -> 200
// <!DOCTYPE html><html lang="en">
//   <head>
//     <meta charset="utf-8"/>
//     <title>Open Orders</title>
//   </head>
//   <body>
//     Open Orders
//     
//       #101 Lena: Ethiopia Guji, 250 g
//       #102 Marco: Colombia Huila, 1000 g
//       #103 Priya: Kenya Nyeri, 500 g
//     
//     Signed in as grace
//   </body>
// </html>
// GET /orders.csv -> 200
// id,customer,coffee,grams
// 101,Lena,Ethiopia Guji,250
// 102,Marco,Colombia Huila,1000
// 103,Priya,Kenya Nyeri,500
```

A browser that gets the 401 shows its own sign-in dialog, remembers what the user typed, and sends it with later requests to the same protected area. `curl -u ada:'correct horse battery staple' http://localhost:8080/orders.csv` does the same for a script.

## Issuing a Session Cookie

Basic authentication has no sign-in page of our own, and no way to sign out: the browser keeps sending the password until it is closed. A login form with a session cookie fixes both. After a correct password, the server creates a session, sends its id to the browser in a cookie, and the browser sends the cookie back with every request. The cookie holds only a random id, never the username or the password, so a client cannot forge or edit it, although anyone who steals a valid id can use it until the session ends.

The session store keeps each session's user and expiry time, keyed by its id. The id is 32 bytes from `SecureRandom`, encoded as Base64url, which uses only characters that are valid in a cookie value. `TrieMap` is a thread-safe map, which matters once requests run in parallel:

```scala
final case class Session(user: String, expiresAtSeconds: Long)

object Sessions {
  val lifetimeSeconds: Long = 30 * 60

  private val random = new SecureRandom()
  private val active = TrieMap.empty[String, Session]

  private def nowSeconds: Long = System.currentTimeMillis() / 1000

  def start(user: String): String = {
    val bytes = new Array[Byte](32)
    random.nextBytes(bytes)
    val id = Base64.getUrlEncoder.withoutPadding.encodeToString(bytes)
    active.put(id, Session(user, nowSeconds + lifetimeSeconds))
    id
  }

  def find(id: String): Option[Session] =
    active.get(id) match {
      case Some(session) if session.expiresAtSeconds > nowSeconds => Some(session)
      case Some(_)                                                =>
        active.remove(id)
        None
      case None => None
    }

  def end(id: String): Unit = {
    active.remove(id)
    ()
  }
}
```

The session expires on the server after 30 minutes, whatever the browser does with the cookie.

`ResponseCookie` describes the cookie and its attributes; `Response#addCookie` renders it into a `Set-Cookie` header. Each attribute closes a door:

| Attribute      | `ResponseCookie` field           | Effect                                                                                                                                |
| -------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `HttpOnly`     | `isHttpOnly = true`              | JavaScript cannot read the cookie, so an injected script cannot steal the session.                                                    |
| `Secure`       | `isSecure = true`                | The browser sends the cookie only over HTTPS. Chrome and Firefox also treat `http://localhost` as secure, so the example works there. |
| `SameSite=Lax` | `sameSite = Some(SameSite.Lax)`  | The browser leaves the cookie off cross-site requests such as a form posted from another site.                                        |
| `Path=/`       | `path = Some(Path.root)`         | The cookie is sent to every path of the site, not only to the path that set it.                                                       |
| `Max-Age=1800` | `maxAge = Some(lifetimeSeconds)` | The browser drops the cookie after 30 minutes, when the server also expires the session.                                              |

The cookie for a session id:

```scala
val sessionCookieName = "roastery_session"

def sessionCookie(id: String): ResponseCookie =
  ResponseCookie(
    sessionCookieName,
    id,
    path = Some(Path.root),
    maxAge = Some(Sessions.lifetimeSeconds),
    isSecure = true,
    isHttpOnly = true,
    sameSite = Some(SameSite.Lax)
  )
```

Starting a session for Ada and rendering its cookie shows the random id and every attribute:

```scala
println(Cookie.renderResponse(sessionCookie(Sessions.start("ada"))))
// roastery_session=DEAFWvMhKPo3T3a1joMao6xGRcDNtFoMmsuDAR4S9IM; Path=/; Max-Age=1800; Secure; HttpOnly; SameSite=Lax
```

:::warning
Never put a password, or anything you would not show the user, in a cookie. The browser stores cookies in plain text, and every request carries them.
:::

## Logging In With a Form

The sign-in page is an HTML form that posts the username and the password to `/login`. The `password` input type keeps the password hidden while it is typed:

```scala
def loginPage(message: String): Response =
  page(
    "Staff Sign-In",
    p(message),
    form(method := "post", action := "/login")(
      label("Username ", input(name := "username")),
      label("Password ", input(name := "password", typeAttr := "password")),
      button("Sign in")
    )
  )
```

The browser sends the form as an `application/x-www-form-urlencoded` body, which `Form.fromString` decodes. After a correct password, `logIn` starts a session and answers `303 See Other` with the session cookie, which sends the browser on to `/admin` with a `GET`:

```scala
def logIn(request: Request): Response = {
  val submitted = Form.fromString(request.body.text)
  val user      = for {
    username <- submitted.get("username")
    password <- submitted.get("password")
    user     <- Staff.authenticate(username, password)
  } yield user
  user match {
    case Some(name) => Response.seeOther("/admin").addCookie(sessionCookie(Sessions.start(name)))
    case None       => loginPage("Wrong username or password.").copy(status = Status.Forbidden)
  }
}
```

Every successful login starts a new session with a new id, even when the browser already sent one. A server that kept an id it got before login would let an attacker who planted that id in a victim's browser share the victim's session. A failed login gets the form back with `403 Forbidden`, not `401 Unauthorized`: a 401 must carry a `WWW-Authenticate` challenge, and a login form is not an HTTP authentication scheme. The message does not say whether the username or the password was wrong.

To try the handler on its own, we serve `logIn` for every path and post a wrong password, then the right one:

```scala
JdkHttp.withServer(logIn) { port =>
  Client.call(port, "POST", "/login", Form("username" -> "ada", "password" -> "espresso"))
  Client.call(port, "POST", "/login", Form("username" -> "ada", "password" -> "correct horse battery staple"))
}
// POST /login -> 403
// <!DOCTYPE html><html lang="en">
//   <head>
//     <meta charset="utf-8"/>
//     <title>Staff Sign-In</title>
//   </head>
//   <body>
//     Staff Sign-In
//     Wrong username or password.
//     <form method="post" action="/login">
//       <label>
//         Username 
//         <input name="username"/>
//       </label>
//       <label>
//         Password 
//         <input name="password" type="password"/>
//       </label>
//       <button>Sign in</button>
//     </form>
//   </body>
// </html>
// POST /login -> 303
// location: /admin
// set-cookie: roastery_session=oIhzLZdKpGRsq7Z31ohyKM8AuR4RPQcu1uiPST0pNIw; Path=/; Max-Age=1800; Secure; HttpOnly; SameSite=Lax
```

:::tip
Use `Response.seeOther` after a form post. `Response.redirect` answers `307 Temporary Redirect`, which makes the browser post the form, password included, to the new location again.
:::

## Reading the Session Cookie

On every later request, `Request#cookies` gives the cookies the browser sent. We look for ours and find its session:

```scala
def sessionId(request: Request): Option[String] =
  request.cookies.find(_.name == sessionCookieName).map(_.value)
```

The `signedIn` wrapper hands the session to the handler. A request with no cookie, an unknown id or an expired session is sent to the sign-in page with `303 See Other`, which is what a person in a browser needs; a 401 would only show them an error:

```scala
def signedIn(handler: (Session, Request) => Response): Request => Response =
  request =>
    sessionId(request).flatMap(Sessions.find) match {
      case Some(session) => handler(session, request)
      case None          => Response.seeOther("/login")
    }
```

The staff page now shows who is signed in, next to a sign-out button that posts to `/logout`, which the next section implements:

```scala
def signOutForm(user: String): Dom.Element =
  form(method := "post", action := "/logout")(s"Signed in as $user ", button("Sign out"))

def pagesApp(request: Request): Response =
  (request.method, request.path.segments.toList) match {
    case (Method.GET, Nil)            => home
    case (Method.GET, List("login"))  => loginPage("Sign in with your staff account.")
    case (Method.POST, List("login")) => logIn(request)
    case (Method.GET, List("admin"))  => signedIn((session, _) => ordersPage(signOutForm(session.user)))(request)
    case _                            => notFound(request)
  }
```

We ask for the staff page without a cookie, sign in, ask again with the cookie from the login, and then with an id that the server never issues:

```scala
JdkHttp.withServer(pagesApp) { port =>
  Client.call(port, "GET", "/admin")
  val cookie = Client.logIn(port, "ada", "correct horse battery staple")
  Client.call(port, "GET", "/admin", cookie = cookie)
  Client.call(port, "GET", "/admin", cookie = s"$sessionCookieName=made-up-session-id")
}
// GET /admin -> 303
// location: /login
// POST /login -> 303
// location: /admin
// set-cookie: roastery_session=NylEZEBW7EGbEGSrjRc06O32bsEWRv-Fp4zSvUYZW5c; Path=/; Max-Age=1800; Secure; HttpOnly; SameSite=Lax
// GET /admin -> 200
// <!DOCTYPE html><html lang="en">
//   <head>
//     <meta charset="utf-8"/>
//     <title>Open Orders</title>
//   </head>
//   <body>
//     Open Orders
//     
//       #101 Lena: Ethiopia Guji, 250 g
//       #102 Marco: Colombia Huila, 1000 g
//       #103 Priya: Kenya Nyeri, 500 g
//     
//     <form method="post" action="/logout">
//       Signed in as ada 
//       <button>Sign out</button>
//     </form>
//   </body>
// </html>
// GET /admin -> 303
// location: /login
```

Only the id from the login opens the page. A made-up id finds no session, so a client cannot invent a working cookie.

## Logging Out

Signing out has two parts: forget the session on the server, and tell the browser to drop the cookie. A `Set-Cookie` with the same name, domain and path, an empty value and `Max-Age=0` replaces the cookie and expires it at once:

```scala
val expiredSessionCookie: ResponseCookie = sessionCookie("").copy(maxAge = Some(0L))

def logOut(request: Request): Response = {
  sessionId(request).foreach(Sessions.end)
  Response.seeOther("/").addCookie(expiredSessionCookie)
}

def staffApp(request: Request): Response =
  (request.method, request.path.segments.toList) match {
    case (Method.POST, List("logout")) => logOut(request)
    case _                             => pagesApp(request)
  }
```

Sign-out is a `POST`, not a link. With `SameSite=Lax`, the browser leaves the cookie off a `POST` from another site, so a malicious page cannot sign the user out; a `GET` route could be triggered by any `` tag. Ending the session on the server matters most: the cookie the browser had stops working even if a copy of it survives somewhere, as the last request shows:

```scala
JdkHttp.withServer(staffApp) { port =>
  val cookie = Client.logIn(port, "grace", "tea and biscuits at four")
  Client.call(port, "POST", "/logout", cookie = cookie)
  Client.call(port, "GET", "/admin", cookie = cookie)
}
// POST /login -> 303
// location: /admin
// set-cookie: roastery_session=CsqCYGlwdN4IuknxtYvpc8hqHUQp3nJrUN4rndnzT2M; Path=/; Max-Age=1800; Secure; HttpOnly; SameSite=Lax
// POST /logout -> 303
// location: /
// set-cookie: roastery_session=; Path=/; Max-Age=0; Secure; HttpOnly; SameSite=Lax
// GET /admin -> 303
// location: /login
```

## Choosing Between Basic Auth and a Session Cookie

Both approaches check the same password; they differ in how the client proves who it is on the next request:

| Question                        | Basic authentication                                   | Session cookie                                            |
| ------------------------------- | ------------------------------------------------------ | --------------------------------------------------------- |
| Who is the client?              | Scripts, `curl`, internal tools                        | People in a browser                                       |
| Sign-in page                    | The browser's own dialog                               | Your own HTML form                                        |
| What travels on every request   | The password                                           | A random session id                                       |
| Password check                  | On every request                                       | Once, at login                                            |
| Sign out                        | Not possible; the browser forgets when it is closed    | Yes, and the server revokes the session                   |
| State on the server             | None                                                   | A session store                                           |
| Unauthenticated answer          | `401` with `WWW-Authenticate: Basic`                   | `303 See Other` to the sign-in page                       |

Many applications use both, as the complete example does: a login form for people, and Basic credentials for the script that downloads the export.

:::tip
A signed JWT can stand in for the random id: the cookie then carries the user and the expiry itself, and the server needs no session store. The price is that the server cannot revoke one token before it expires, so signing out only deletes the browser's copy. If you go that way, sign and verify the token as [How to Issue and Verify JWTs](./issue-and-verify-jwts.md) and [How to Protect an Endpoint with JWT Bearer Tokens](./protect-endpoints-with-jwt.md) show, and keep the cookie attributes from this guide.
:::

## What This Code Does Not Do

The code in this guide checks credentials; it is not a complete security setup. Before you rely on it, check these points:

- **Use HTTPS.** Basic credentials and session ids work for whoever holds them, and over plain HTTP anyone on the network path can copy them. The loopback server in this guide is for examples only.
- **No rate limiting.** Nothing stops a client from trying thousands of passwords. Limit failed logins per account and per address.
- **One server, in memory.** The session store lives in one JVM: a restart signs everyone out, and two instances behind a load balancer do not share sessions. Use a shared store in that case. `Sessions.find` removes an expired session only when its id comes back, so also purge expired sessions on a timer.
- **Cross-site requests.** A malicious site can make the user's browser send a request to this site, and the browser attaches the cookie: cross-site request forgery (CSRF). `SameSite=Lax` keeps the cookie off cross-site `POST` requests in current browsers, which protects the form routes. It still sends the cookie on cross-site links, so never change data in a `GET` route. For older browsers, add a CSRF token to every form: a random value the server puts in the form and checks on submit, which another site cannot read.
- **Caching.** A shared cache could store a private page. Add `Cache-Control: no-store` to responses that show private data.

## Putting It Together

The complete example serves the whole roastery on port 8080 until you stop it with Ctrl+C: `staffApp` from the last section answers `/`, `/login`, `/admin` and `/logout`, and `/orders.csv` goes through `basicAuth` instead. Its `main` starts the server with `JdkHttp.start(app, 8080)` and blocks the main thread with `Thread.currentThread.join()`. It builds on the adapter and the client embedded under Prerequisites, and on the code of the earlier sections, which the examples keep in one file per section: `Roastery.scala`, `Staff.scala`, `BasicAuth.scala`, `Sessions.scala`, `Login.scala`, `SessionPages.scala` and `Logout.scala`. The complete example:

```scala title="http-model-examples/src/main/scala/protectpageswithbasicauthorasessioncookie/CompleteExample.scala"
package protectpageswithbasicauthorasessioncookie

/**
 * How to Protect Pages with Basic Auth or a Session Cookie — Putting It
 * Together
 *
 * Serves the whole roastery on port 8080 until Ctrl+C: the session-cookie pages
 * at `/login`, `/admin` and `/logout`, and `/orders.csv` behind Basic
 * authentication.
 */
// Run with: cd http-model-examples && sbt "runMain protectpageswithbasicauthorasessioncookie.CompleteExample"
object CompleteExample {
  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, List("orders.csv")) => basicAuth((_, _) => ordersCsv)(request)
      case _                                => staffApp(request)
    }

  def main(args: Array[String]): Unit = {
    JdkHttp.start(app, 8080)
    println("Serving http://localhost:8080/ (Ctrl+C to stop)")
    Thread.currentThread.join() // Block the main thread; the server runs on its own threads.
  }
}
```

While it runs, open `http://localhost:8080/admin` in a browser to be sent to the sign-in page, or call it with `curl`. `curl` does not store cookies unless you give it a cookie jar, so the session cookie is passed by hand with `-b`. One run, with the `Date` headers removed, is in the block below:

<details>
  <summary>A <code>curl</code> session against the complete example</summary>

```text
$ curl -s -i http://localhost:8080/admin
HTTP/1.1 303 See Other
Content-length: 0
Location: /login

$ curl -s -i --data-urlencode username=ada --data-urlencode "password=correct horse battery staple" http://localhost:8080/login
HTTP/1.1 303 See Other
Content-length: 0
Location: /admin
Set-cookie: roastery_session=IHUNG5YYpbQrXnh9LySKLtUKrME9QUl2H-QNCbDkt3c; Path=/; Max-Age=1800; Secure; HttpOnly; SameSite=Lax

$ curl -s -i -b roastery_session=IHUNG5YYpbQrXnh9LySKLtUKrME9QUl2H-QNCbDkt3c http://localhost:8080/admin
HTTP/1.1 200 OK
Content-type: text/html
Content-length: 433

<!DOCTYPE html><html lang="en">
  <head>
    <meta charset="utf-8"/>
    <title>Open Orders</title>
  </head>
  <body>
    Open Orders
    
      #101 Lena: Ethiopia Guji, 250 g
      #102 Marco: Colombia Huila, 1000 g
      #103 Priya: Kenya Nyeri, 500 g
    
    <form method="post" action="/logout">
      Signed in as ada 
      <button>Sign out</button>
    </form>
  </body>
</html>
$ curl -s -i http://localhost:8080/orders.csv
HTTP/1.1 401 Unauthorized
Www-authenticate: Basic realm="Roastery staff", charset="UTF-8"
Content-type: text/plain; charset=UTF-8
Content-length: 36

Sign in to see the roastery's orders
$ curl -s -i -u ada:wrong http://localhost:8080/orders.csv
HTTP/1.1 401 Unauthorized
Www-authenticate: Basic realm="Roastery staff", charset="UTF-8"
Content-type: text/plain; charset=UTF-8
Content-length: 36

Sign in to see the roastery's orders
$ curl -s -i -u "ada:correct horse battery staple" http://localhost:8080/orders.csv
HTTP/1.1 200 OK
Content-type: text/csv; charset=utf-8
Content-length: 107

id,customer,coffee,grams
101,Lena,Ethiopia Guji,250
102,Marco,Colombia Huila,1000
103,Priya,Kenya Nyeri,500
$ curl -s -i -X POST -b roastery_session=IHUNG5YYpbQrXnh9LySKLtUKrME9QUl2H-QNCbDkt3c http://localhost:8080/logout
HTTP/1.1 303 See Other
Content-length: 0
Location: /
Set-cookie: roastery_session=; Path=/; Max-Age=0; Secure; HttpOnly; SameSite=Lax

$ curl -s -i -b roastery_session=IHUNG5YYpbQrXnh9LySKLtUKrME9QUl2H-QNCbDkt3c http://localhost:8080/admin
HTTP/1.1 303 See Other
Content-length: 0
Location: /login
```

</details>

The server prints `Serving http://localhost:8080/ (Ctrl+C to stop)` at startup and `auth: wrong Basic credentials` for the `ada:wrong` request. The JDK server capitalizes header names, so `www-authenticate` arrives as `Www-authenticate`; header names are case-insensitive. The last request shows the server-side logout: the old cookie no longer opens the staff page.

## Running the Examples

All runnable code from this guide is in [`http-model-examples/src/main/scala/protectpageswithbasicauthorasessioncookie/`](https://github.com/zio/zio-blocks/tree/main/http-model-examples/src/main/scala/protectpageswithbasicauthorasessioncookie). Run any file from `http-model-examples` with `sbt "runMain protectpageswithbasicauthorasessioncookie.<ObjectName>"`.

## Going Further

- [How to Protect an Endpoint with JWT Bearer Tokens](./protect-endpoints-with-jwt.md) protects a JSON API with bearer tokens and explains when to answer 401, 403 or 404.
- [How to Issue and Verify JWTs](./issue-and-verify-jwts.md) covers signing keys and token checks, if you carry a JWT in the session cookie.
- The [HTTP model reference](../reference/http-model/model.md) documents `RequestCookie`, `ResponseCookie`, `SameSite` and `Form`.
- The [headers reference](../reference/http-model/headers.md) lists the typed headers, including `Header.Authorization` and `Header.WWWAuthenticate`.
- The [`AuthType` reference](../reference/endpoint/auth-type.md) shows how to declare Basic authentication on an endpoint description.

---

## Query DSL

The Query DSL guides build one type-safe query language in four parts. You write a filter once as ordinary Scala, and the same expression can be evaluated in memory, turned into SQL, extended with new SQL features, and finally assembled into full statements.

Read the parts in order. Each one builds on the expressions and code from the one before.

1. [Expressions](../query-dsl-reified-optics.md) builds query expressions over your domain types with reified optics: comparisons, boolean logic, arithmetic and string operations. You evaluate them against in-memory data.
2. [SQL Generation](../query-dsl-sql.md) translates those expressions into `WHERE` clauses, `SELECT` statements and parameterized queries.
3. [Extending the Expression Language](../query-dsl-extending.md) adds SQL predicates the built-in language lacks, such as `IN`, `BETWEEN`, `IS NULL` and `LIKE`, plus aggregate functions and `CASE WHEN`.
4. [A Fluent SQL Builder](../query-dsl-fluent-builder.md) wraps everything in a builder for `SELECT`, `UPDATE`, `INSERT` and `DELETE` statements.

The reference pages for the optics and schema types the DSL uses are in the [reference](../../reference/index.md).

---

## Query DSL with Reified Optics — Part 3: Extending the Expression Language

In this guide, we will extend the ZIO Blocks query DSL with an expression language that goes beyond what `SchemaExpr` provides out of the box. By the end, you will have an `Expr` ADT that adds SQL-specific predicates (`IN`, `BETWEEN`, `IS NULL`, `LIKE`), type-safe aggregate functions (`COUNT`, `SUM`, `AVG`), and conditional expressions (`CASE WHEN`) — all composable with the built-in `SchemaExpr` operators from Parts 1 and 2.

This is Part 3 of the Query DSL series. [Part 1](./query-dsl-reified-optics.md) covered building query expressions, and [Part 2](./query-dsl-sql.md) covered translating them to SQL. Here, we handle the cases where the built-in expression language is not enough.

**What we'll cover:**

- Why `SchemaExpr` is deliberately closed and what that means for extension
- Designing an `Expr` ADT that is a superset of `SchemaExpr`
- Translating `SchemaExpr` into `Expr` via `Expr.fromSchemaExpr`
- Adding SQL-specific predicates: `IN`, `BETWEEN`, `IS NULL`, `LIKE`
- Writing bridge extension methods for seamless `SchemaExpr` + `Expr` composition
- Building a single unified SQL interpreter
- Adding type-safe aggregate functions and `CASE WHEN` for advanced SQL generation

## The Problem

The built-in `SchemaExpr` operators cover the fundamentals: equality, comparisons, boolean logic, arithmetic, and basic string operations. But real-world SQL requires more. Consider these common queries:

```sql
-- Membership test
SELECT * FROM products WHERE category IN ('Electronics', 'Books', 'Toys')

-- Range check
SELECT * FROM products WHERE price BETWEEN 10.0 AND 100.0

-- Null handling
SELECT * FROM products WHERE description IS NULL

-- Pattern matching with SQL wildcards
SELECT * FROM products WHERE name LIKE 'Lap%'

-- Aggregation
SELECT category, COUNT(*), AVG(price)
FROM products GROUP BY category HAVING COUNT(*) > 2

-- Conditional logic
SELECT name, CASE WHEN price > 100 THEN 'expensive' ELSE 'cheap' END AS tier
FROM products
```

None of these can be expressed with `SchemaExpr` alone. You could generate the SQL strings manually, but then you lose composability — you can no longer mix these operations with the type-safe `SchemaExpr` predicates from Parts 1 and 2.

`SchemaExpr` wraps a `DynamicSchemaExpr` sealed trait that you cannot add new cases to. The public API is `SchemaExpr`; `DynamicSchemaExpr` is the raw serializable AST exposed through `SchemaExpr#dynamic`. Instead of extending `DynamicSchemaExpr` directly, we define an `Expr` ADT that is a superset — it includes equivalent nodes for everything `SchemaExpr` can express, plus our custom SQL-specific operations. A `Expr.fromSchemaExpr` function translates `SchemaExpr` values into `Expr` by reading the underlying `DynamicSchemaExpr`, enabling seamless interoperability with a single unified interpreter.

## Prerequisites

This guide builds on [Part 1: Expressions](./query-dsl-reified-optics.md) and [Part 2: SQL Generation](./query-dsl-sql.md). You should be comfortable building `SchemaExpr` values and translating them to SQL. To add the dependency:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"
```

Then import the schema package:

```scala

```

## Domain Setup

We reuse the product catalog domain from the earlier guides:

```scala
case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]     = optic(_.name)
  val price: Lens[Product, Double]    = optic(_.price)
  val category: Lens[Product, String] = optic(_.category)
  val inStock: Lens[Product, Boolean] = optic(_.inStock)
  val rating: Lens[Product, Int]      = optic(_.rating)
}
```

## Designing the Expr ADT

The key insight is the **translation pattern**: keep your public extension API typed, then provide an `Expr.fromSchemaExpr` function that lifts any built-in `SchemaExpr` into your extended ADT. Ordinary application code stays on `SchemaExpr`, `Optic`, and typed constructors. Only the interpreter internals need to inspect `SchemaExpr#dynamic`. The following diagram shows how the two layers relate:

```
  Built-in API                               Your extension layer
┌───────────────────────────────┐       ┌───────────────────────────────────────┐
│  SchemaExpr[S, A]             │       │  Expr[S, A]                           │
│  built with optics/operators  │──────▶│  ├── Builtin(schemaExpr)              │
└───────────────────────────────┘       │  ├── Column(optic)                    │
         fromSchemaExpr ────────────────│  ├── Lit(value, schema)               │
                                        │  ├── In(expr, values, schema)         │
                                        │  ├── Between(expr, low, high, schema) │
                                        │  ├── IsNull(expr)                     │
                                        │  ├── Like(expr, pattern)              │
                                        │  ├── Agg(function, expr)              │
                                        │  └── CaseWhen(branches, else)         │
                                        └───────────────────────────────────────┘
```

The `Expr` ADT keeps the public surface typed. It can wrap any built-in `SchemaExpr` through `Builtin`, and it adds SQL-specific nodes on top. The interpreter is still free to inspect `schemaExpr.dynamic` internally when it needs to translate the built-in pieces.

Here is the full `Expr` ADT with its supporting types:

```scala
sealed trait Expr[S, A]

object Expr {

  // --- Typed public nodes ---
  final case class Builtin[S, A](schemaExpr: SchemaExpr[S, A]) extends Expr[S, A]
  final case class Column[S, A](optic: Optic[S, A]) extends Expr[S, A]
  final case class Lit[S, A](value: A, schema: Schema[A]) extends Expr[S, A]

  // Relational
  final case class Relational[S, A](left: Expr[S, A], right: Expr[S, A], op: RelOp) extends Expr[S, Boolean]

  // Logical
  final case class And[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Or[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Not[S](expr: Expr[S, Boolean]) extends Expr[S, Boolean]

  // Arithmetic
  final case class Arithmetic[S, A](left: Expr[S, A], right: Expr[S, A], op: ArithOp) extends Expr[S, A]

  // String
  final case class StringConcat[S](left: Expr[S, String], right: Expr[S, String]) extends Expr[S, String]
  final case class StringRegexMatch[S](regex: Expr[S, String], string: Expr[S, String]) extends Expr[S, Boolean]
  final case class StringLength[S](string: Expr[S, String]) extends Expr[S, Int]

  // --- SQL-specific extensions (no SchemaExpr equivalents) ---
  final case class In[S, A](expr: Expr[S, A], values: List[A], schema: Schema[A]) extends Expr[S, Boolean]
  final case class Between[S, A](expr: Expr[S, A], low: A, high: A, schema: Schema[A]) extends Expr[S, Boolean]
  final case class IsNull[S, A](expr: Expr[S, A]) extends Expr[S, Boolean]
  final case class Like[S](expr: Expr[S, String], pattern: String) extends Expr[S, Boolean]

  // --- Aggregates ---
  final case class Agg[S, A, B](function: AggFunction[A, B], expr: Expr[S, A]) extends Expr[S, B]

  // --- Conditional ---
  final case class CaseWhen[S, A](
    branches: List[(Expr[S, Boolean], Expr[S, A])],
    otherwise: Option[Expr[S, A]]
  ) extends Expr[S, A]

  // --- Factory methods ---
  def col[S, A](optic: Optic[S, A]): Expr[S, A] = Column(optic)
  def lit[S, A](value: A)(implicit schema: Schema[A]): Expr[S, A] = Lit(value, schema)

  def count[S, A](expr: Expr[S, A]): Expr[S, Long]   = Agg(AggFunction.Count(), expr)
  def sum[S](expr: Expr[S, Double]): Expr[S, Double]  = Agg(AggFunction.Sum, expr)
  def avg[S](expr: Expr[S, Double]): Expr[S, Double]  = Agg(AggFunction.Avg, expr)
  def min[S, A](expr: Expr[S, A]): Expr[S, A]         = Agg(AggFunction.Min(), expr)
  def max[S, A](expr: Expr[S, A]): Expr[S, A]         = Agg(AggFunction.Max(), expr)

  def caseWhen[S, A](branches: (Expr[S, Boolean], Expr[S, A])*): CaseWhenBuilder[S, A] =
    CaseWhenBuilder(branches.toList)

  case class CaseWhenBuilder[S, A](branches: List[(Expr[S, Boolean], Expr[S, A])]) {
    def otherwise(value: Expr[S, A]): Expr[S, A] = CaseWhen(branches, Some(value))
    def end: Expr[S, A] = CaseWhen(branches, None)
  }

  // --- Translation from SchemaExpr ---
  def fromSchemaExpr[S, A](se: SchemaExpr[S, A]): Expr[S, A] = Builtin(se)
}

// --- Operators ---

sealed trait RelOp
object RelOp {
  case object Equal              extends RelOp
  case object NotEqual           extends RelOp
  case object LessThan           extends RelOp
  case object LessThanOrEqual    extends RelOp
  case object GreaterThan        extends RelOp
  case object GreaterThanOrEqual extends RelOp
}

sealed trait ArithOp
object ArithOp {
  case object Add      extends ArithOp
  case object Subtract extends ArithOp
  case object Multiply extends ArithOp
}

// Typed aggregate functions
sealed trait AggFunction[A, B] {
  def name: String
}
object AggFunction {
  case class Count[A]() extends AggFunction[A, Long]         { val name = "COUNT" }
  case object Sum       extends AggFunction[Double, Double]  { val name = "SUM" }
  case object Avg       extends AggFunction[Double, Double]  { val name = "AVG" }
  case class Min[A]()   extends AggFunction[A, A]            { val name = "MIN" }
  case class Max[A]()   extends AggFunction[A, A]            { val name = "MAX" }
}
```

Here are some keynotes on the design:

- **Type-safe aggregates** — `AggFunction[A, B]` encodes the return type: `COUNT` returns `Long`, `SUM`/`AVG` return `Double`, `MIN`/`MAX` preserve the input type.
- **Typed public constructors** — `Column` stores an `Optic` and `Lit` stores a typed value plus its `Schema`, so extension code stays on the same public abstractions as ordinary `SchemaExpr` code.
- **`Expr.fromSchemaExpr`** — lifts a built-in `SchemaExpr` into the extended ADT. The dynamic representation is only consulted later by the interpreter.

## Extension Methods

To make the new operations feel natural, we define implicit classes on `Optic`, `Expr`, and `SchemaExpr`. The bridge implicit class on `SchemaExpr` auto-translates at the boundary via `Expr.fromSchemaExpr`, so `SchemaExpr` and `Expr` values compose seamlessly with `&&` and `||`:

```scala
implicit final class OpticExprOps[S, A](private val optic: Optic[S, A]) {
  def in(values: A*)(implicit schema: Schema[A]): Expr[S, Boolean]           = Expr.In(Expr.col(optic), values.toList, schema)
  def between(low: A, high: A)(implicit schema: Schema[A]): Expr[S, Boolean] = Expr.Between(Expr.col(optic), low, high, schema)
  def isNull: Expr[S, Boolean]                   = Expr.IsNull(Expr.col(optic))
  def isNotNull: Expr[S, Boolean]                = Expr.Not(Expr.IsNull(Expr.col(optic)))
}

implicit final class StringOpticExprOps[S](private val optic: Optic[S, String]) {
  def like(pattern: String): Expr[S, Boolean] = Expr.Like(Expr.col(optic), pattern)
}

// Boolean combinators — accept both Expr and SchemaExpr on the right
implicit final class ExprBooleanOps[S](private val self: Expr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.And(self, other)
  def &&(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.And(self, Expr.fromSchemaExpr(other))
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.Or(self, other)
  def ||(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.Or(self, Expr.fromSchemaExpr(other))
  def unary_! : Expr[S, Boolean]                          = Expr.Not(self)
}

// Bridge: SchemaExpr on the left, Expr on the right
implicit final class SchemaExprBooleanBridge[S](private val self: SchemaExpr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.And(Expr.fromSchemaExpr(self), other)
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.Or(Expr.fromSchemaExpr(self), other)
  def toExpr: Expr[S, Boolean] = Expr.fromSchemaExpr(self)
}
```

The bridge implicit classes are the key to ergonomic composition. When you write `Product.category.in("Electronics") && (Product.rating >= 4)`, the `&&` on `Expr[S, Boolean]` sees a `SchemaExpr[S, Boolean]` on the right and auto-translates it. Similarly, `(Product.rating >= 4) && Product.category.in("Electronics")` uses the `SchemaExpr` bridge to translate the left side. No explicit `SchemaExprBooleanBridge#toExpr` is needed in most cases.

:::tip
The `SchemaExprBooleanBridge#toExpr` method is still available for cases where you need to explicitly lift a `SchemaExpr[S, Boolean]` — for example, when building `CASE WHEN` branch conditions.
:::

## The Unified SQL Interpreter

With the `Expr` ADT, we write a single interpreter that handles all cases directly. Typed extension nodes stay typed; built-in `SchemaExpr` fragments delegate to a helper that reads `SchemaExpr#dynamic` internally:

```scala
def columnName(optic: zio.blocks.schema.Optic[_, _]): String =
  optic.toDynamic.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")

def columnName(path: DynamicOptic): String =
  path.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")

def sqlLiteral[A](value: A, schema: Schema[A]): String = {
  val dv = schema.toDynamicValue(value)
  dv match {
    case p: DynamicValue.Primitive => p.value match {
      case _: PrimitiveValue.String  => s"'${value.toString.replace("'", "''")}'"
      case b: PrimitiveValue.Boolean => if (b.value) "TRUE" else "FALSE"
      case _                         => value.toString
    }
    case _ => value.toString
  }
}

def sqlLiteralDV(dv: DynamicValue): String = dv match {
  case DynamicValue.Primitive(pv) =>
    pv match {
      case PrimitiveValue.String(s)  => s"'${s.replace("'", "''")}'"
      case PrimitiveValue.Boolean(b) => if (b) "TRUE" else "FALSE"
      case PrimitiveValue.Int(n)     => n.toString
      case PrimitiveValue.Long(n)    => n.toString
      case PrimitiveValue.Double(n)  => n.toString
      case PrimitiveValue.Float(n)   => n.toString
      case PrimitiveValue.Short(n)   => n.toString
      case PrimitiveValue.Byte(n)    => n.toString
      case other                     => other.toString
    }
  case other => other.toString
}

def exprToSql[S, A](expr: Expr[S, A]): String = expr match {
  case Expr.Builtin(schemaExpr) => schemaExprToSql(schemaExpr)
  case Expr.Column(optic)       => columnName(optic)
  case Expr.Lit(value, schema)  => sqlLiteral(value, schema)

  case Expr.Relational(left, right, op) =>
    val sqlOp = op match {
      case RelOp.Equal              => "="
      case RelOp.NotEqual           => "<>"
      case RelOp.LessThan           => "<"
      case RelOp.LessThanOrEqual    => "<="
      case RelOp.GreaterThan        => ">"
      case RelOp.GreaterThanOrEqual => ">="
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"

  case Expr.And(l, r) => s"(${exprToSql(l)} AND ${exprToSql(r)})"
  case Expr.Or(l, r)  => s"(${exprToSql(l)} OR ${exprToSql(r)})"
  case Expr.Not(e)    => s"NOT (${exprToSql(e)})"

  case Expr.Arithmetic(left, right, op) =>
    val sqlOp = op match {
      case ArithOp.Add      => "+"
      case ArithOp.Subtract => "-"
      case ArithOp.Multiply => "*"
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"

  case Expr.StringConcat(l, r)          => s"CONCAT(${exprToSql(l)}, ${exprToSql(r)})"
  case Expr.StringRegexMatch(regex, s)  => s"(${exprToSql(s)} LIKE ${exprToSql(regex)})"
  case Expr.StringLength(s)             => s"LENGTH(${exprToSql(s)})"

  // SQL-specific
  case Expr.In(e, values, schema) =>
    s"${exprToSql(e)} IN (${values.map(v => sqlLiteral(v, schema)).mkString(", ")})"
  case Expr.Between(e, low, high, schema) =>
    s"(${exprToSql(e)} BETWEEN ${sqlLiteral(low, schema)} AND ${sqlLiteral(high, schema)})"
  case Expr.IsNull(e)          => s"${exprToSql(e)} IS NULL"
  case Expr.Like(e, pattern)   => s"${exprToSql(e)} LIKE '${pattern.replace("'", "''")}'"

  // Aggregates
  case Expr.Agg(func, e) => s"${func.name}(${exprToSql(e)})"

  // CASE WHEN
  case Expr.CaseWhen(branches, otherwise) =>
    val cases = branches.map { case (cond, value) =>
      s"WHEN ${exprToSql(cond)} THEN ${exprToSql(value)}"
    }.mkString(" ")
    val elseClause = otherwise.map(e => s" ELSE ${exprToSql(e)}").getOrElse("")
    s"CASE $cases$elseClause END"
}

def schemaExprToSql[S, A](expr: SchemaExpr[S, A]): String =
  toSqlDynamic(expr.dynamic)

def toSqlDynamic(expr: DynamicSchemaExpr): String = expr match {
  case DynamicSchemaExpr.Select(path)   => columnName(path)
  case DynamicSchemaExpr.Literal(value, _) => sqlLiteralDV(value)

  case DynamicSchemaExpr.Relational(left, right, op) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.RelationalOperator.Equal              => "="
      case DynamicSchemaExpr.RelationalOperator.NotEqual           => "<>"
      case DynamicSchemaExpr.RelationalOperator.LessThan           => "<"
      case DynamicSchemaExpr.RelationalOperator.LessThanOrEqual    => "<="
      case DynamicSchemaExpr.RelationalOperator.GreaterThan        => ">"
      case DynamicSchemaExpr.RelationalOperator.GreaterThanOrEqual => ">="
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"

  case DynamicSchemaExpr.Logical(left, right, op) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.LogicalOperator.And => "AND"
      case DynamicSchemaExpr.LogicalOperator.Or  => "OR"
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"

  case DynamicSchemaExpr.Not(inner) =>
    s"NOT (${toSqlDynamic(inner)})"

  case DynamicSchemaExpr.Arithmetic(left, right, op, _) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.ArithmeticOperator.Add      => "+"
      case DynamicSchemaExpr.ArithmeticOperator.Subtract => "-"
      case DynamicSchemaExpr.ArithmeticOperator.Multiply => "*"
      case _                                             => "?"
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"

  case DynamicSchemaExpr.StringConcat(left, right) =>
    s"CONCAT(${toSqlDynamic(left)}, ${toSqlDynamic(right)})"

  case DynamicSchemaExpr.StringRegexMatch(regex, string) =>
    s"(${toSqlDynamic(string)} LIKE ${toSqlDynamic(regex)})"

  case DynamicSchemaExpr.StringLength(string) =>
    s"LENGTH(${toSqlDynamic(string)})"

  case _ => "?"
}
```

The typed `sqlLiteral[A](value, schema)` uses the `Schema` carried by `Lit`, `In`, and `Between` to format values correctly — strings get quoted, booleans become `TRUE`/`FALSE`, numbers stay as-is. `sqlLiteralDV` is only needed inside `toSqlDynamic`, where built-in `SchemaExpr` nodes have already crossed into the dynamic representation.

## SQL-Specific Predicates

With the ADT, extensions, and interpreter in place, the new operators work directly on optics:

```scala
exprToSql(Product.category.in("Electronics", "Books", "Toys"))
// res0: String = "category IN ('Electronics', 'Books', 'Toys')"

exprToSql(Product.price.between(10.0, 100.0))
// res1: String = "(price BETWEEN 10.0 AND 100.0)"

exprToSql(Product.name.isNull)
// res2: String = "name IS NULL"

exprToSql(Product.name.isNotNull)
// res3: String = "NOT (name IS NULL)"

exprToSql(Product.name.like("Lap%"))
// res4: String = "name LIKE 'Lap%'"
```

Each extension method on `Optic` returns an `Expr` node. The interpreter handles it and produces the corresponding SQL fragment.

## Composing with SchemaExpr

The bridge implicit classes handle the translation automatically. You can freely mix `SchemaExpr` predicates (from `===`, `>`, etc.) with `Expr` predicates (from `OpticExprOps#in`, `OpticExprOps#between`, etc.) using `&&` and `||`:

```scala
// SchemaExpr values from built-in operators
val highRated: SchemaExpr[Product, Boolean] = Product.rating >= 4

// Expr values from extension methods
val inCategory: Expr[Product, Boolean] = Product.category.in("Electronics", "Books")
val priceRange: Expr[Product, Boolean] = Product.price.between(10.0, 500.0)

// Seamless composition — bridge auto-translates at the boundary
val combined: Expr[Product, Boolean] =
  inCategory && priceRange && highRated
```

To render the combined predicate:

```scala
exprToSql(combined)
// res5: String = "((category IN ('Electronics', 'Books') AND (price BETWEEN 10.0 AND 500.0)) AND (rating >= 4))"
```

The `&&` between `priceRange` (an `Expr`) and `highRated` (a `SchemaExpr`) triggers the overloaded `&&` that accepts `SchemaExpr` on the right. It calls `Expr.fromSchemaExpr` internally, so no explicit `SchemaExprBooleanBridge#toExpr` call is needed.

You can also start from a `SchemaExpr` on the left — the bridge implicit class handles it:

```scala
val query: Expr[Product, Boolean] =
  Product.category.in("Electronics", "Books") &&
  Product.price.between(10.0, 500.0) &&
  (Product.rating >= 4) &&
  Product.name.like("M%")
```

To render the query:

```scala
exprToSql(query)
// res6: String = "(((category IN ('Electronics', 'Books') AND (price BETWEEN 10.0 AND 500.0)) AND (rating >= 4)) AND name LIKE 'M%')"
```

A helper function to generate full SELECT statements from `Expr` predicates:

```scala
def selectWhere(table: String, predicate: Expr[_, Boolean]): String =
  s"SELECT * FROM $table WHERE ${exprToSql(predicate)}"
```

To produce a full statement:

```scala
selectWhere("products", query)
// res7: String = "SELECT * FROM products WHERE (((category IN ('Electronics', 'Books') AND (price BETWEEN 10.0 AND 500.0)) AND (rating >= 4)) AND name LIKE 'M%')"
```

## Aggregate Expressions

The `Agg` node wraps any column expression with a type-safe aggregate function. The return type reflects SQL semantics: `COUNT` returns `Expr[S, Long]`, `SUM`/`AVG` return `Expr[S, Double]`, and `MIN`/`MAX` preserve the input type:

```scala
exprToSql(Expr.count(Expr.col(Product.name)))
// res8: String = "COUNT(name)"

exprToSql(Expr.avg(Expr.col(Product.price)))
// res9: String = "AVG(price)"

exprToSql(Expr.max(Expr.col(Product.rating)))
// res10: String = "MAX(rating)"
```

Aggregates compose with the rest of the ADT. Build a `GROUP BY` query by combining aggregate SQL fragments with a select builder:

```scala
def selectGroupBy(
  table: String,
  columns: List[String],
  groupBy: List[String],
  having: Option[String] = None
): String = {
  val base = s"SELECT ${columns.mkString(", ")} FROM $table GROUP BY ${groupBy.mkString(", ")}"
  having.fold(base)(h => s"$base HAVING $h")
}
```

To render the grouped query:

```scala
selectGroupBy(
  "products",
  columns = List(
    "category",
    s"${exprToSql(Expr.count(Expr.col(Product.name)))} AS product_count",
    s"${exprToSql(Expr.avg(Expr.col(Product.price)))} AS avg_price"
  ),
  groupBy = List("category"),
  having = Some(s"${exprToSql(Expr.count(Expr.col(Product.name)))} > 2")
)
// res11: String = "SELECT category, COUNT(name) AS product_count, AVG(price) AS avg_price FROM products GROUP BY category HAVING COUNT(name) > 2"
```

## CASE WHEN Expressions

The `CaseWhen` node represents SQL's conditional expression. Use the `Expr.caseWhen` builder with `(condition -> result)` pairs and an optional `CaseWhenBuilder#otherwise` clause:

```scala
val priceLabel: Expr[Product, String] = Expr.caseWhen[Product, String](
  (Product.price > 100.0).toExpr  -> Expr.lit[Product, String]("expensive"),
  (Product.price > 10.0).toExpr   -> Expr.lit[Product, String]("moderate")
).otherwise(Expr.lit[Product, String]("cheap"))
```

To render the CASE expression:

```scala
exprToSql(priceLabel)
// res12: String = "CASE WHEN (price > 100.0) THEN 'expensive' WHEN (price > 10.0) THEN 'moderate' ELSE 'cheap' END"
```

`CASE WHEN` is useful for computed columns in SELECT lists:

```scala
val stockStatus: Expr[Product, String] = Expr.caseWhen[Product, String](
  (Product.inStock === true).toExpr -> Expr.lit[Product, String]("available")
).otherwise(Expr.lit[Product, String]("out of stock"))
```

To print the complete SELECT statement:

```scala
val selectSql = s"SELECT name, price, ${exprToSql(priceLabel)} AS tier, ${exprToSql(stockStatus)} AS status FROM products"
// selectSql: String = "SELECT name, price, CASE WHEN (price > 100.0) THEN 'expensive' WHEN (price > 10.0) THEN 'moderate' ELSE 'cheap' END AS tier, CASE WHEN (inStock = TRUE) THEN 'available' ELSE 'out of stock' END AS status FROM products"
println(selectSql)
// SELECT name, price, CASE WHEN (price > 100.0) THEN 'expensive' WHEN (price > 10.0) THEN 'moderate' ELSE 'cheap' END AS tier, CASE WHEN (inStock = TRUE) THEN 'available' ELSE 'out of stock' END AS status FROM products
```

## Putting It Together

Here is a complete, self-contained example that defines the independent expression ADT, lifts built-in `SchemaExpr` values into it, and generates advanced SQL:

```scala

// --- Domain ---

case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]     = optic(_.name)
  val price: Lens[Product, Double]    = optic(_.price)
  val category: Lens[Product, String] = optic(_.category)
  val inStock: Lens[Product, Boolean] = optic(_.inStock)
  val rating: Lens[Product, Int]      = optic(_.rating)
}

// --- Independent Expr ADT ---

sealed trait Expr[S, A]

object Expr {
  final case class Builtin[S, A](schemaExpr: SchemaExpr[S, A]) extends Expr[S, A]
  final case class Column[S, A](optic: Optic[S, A]) extends Expr[S, A]
  final case class Lit[S, A](value: A, schema: Schema[A]) extends Expr[S, A]

  final case class Relational[S, A](left: Expr[S, A], right: Expr[S, A], op: RelOp) extends Expr[S, Boolean]
  final case class And[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Or[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Not[S](expr: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Arithmetic[S, A](left: Expr[S, A], right: Expr[S, A], op: ArithOp) extends Expr[S, A]
  final case class StringConcat[S](left: Expr[S, String], right: Expr[S, String]) extends Expr[S, String]
  final case class StringRegexMatch[S](regex: Expr[S, String], string: Expr[S, String]) extends Expr[S, Boolean]
  final case class StringLength[S](string: Expr[S, String]) extends Expr[S, Int]

  final case class In[S, A](expr: Expr[S, A], values: List[A], schema: Schema[A]) extends Expr[S, Boolean]
  final case class Between[S, A](expr: Expr[S, A], low: A, high: A, schema: Schema[A]) extends Expr[S, Boolean]
  final case class IsNull[S, A](expr: Expr[S, A]) extends Expr[S, Boolean]
  final case class Like[S](expr: Expr[S, String], pattern: String) extends Expr[S, Boolean]

  final case class Agg[S, A, B](function: AggFunction[A, B], expr: Expr[S, A]) extends Expr[S, B]
  final case class CaseWhen[S, A](
    branches: List[(Expr[S, Boolean], Expr[S, A])],
    otherwise: Option[Expr[S, A]]
  ) extends Expr[S, A]

  def col[S, A](optic: Optic[S, A]): Expr[S, A] = Column(optic)
  def lit[S, A](value: A)(implicit schema: Schema[A]): Expr[S, A] = Lit(value, schema)
  def count[S, A](expr: Expr[S, A]): Expr[S, Long]   = Agg(AggFunction.Count(), expr)
  def sum[S](expr: Expr[S, Double]): Expr[S, Double]  = Agg(AggFunction.Sum, expr)
  def avg[S](expr: Expr[S, Double]): Expr[S, Double]  = Agg(AggFunction.Avg, expr)
  def min[S, A](expr: Expr[S, A]): Expr[S, A]         = Agg(AggFunction.Min(), expr)
  def max[S, A](expr: Expr[S, A]): Expr[S, A]         = Agg(AggFunction.Max(), expr)

  def caseWhen[S, A](branches: (Expr[S, Boolean], Expr[S, A])*): CaseWhenBuilder[S, A] =
    CaseWhenBuilder(branches.toList)

  case class CaseWhenBuilder[S, A](branches: List[(Expr[S, Boolean], Expr[S, A])]) {
    def otherwise(value: Expr[S, A]): Expr[S, A] = CaseWhen(branches, Some(value))
    def end: Expr[S, A] = CaseWhen(branches, None)
  }

  def fromSchemaExpr[S, A](se: SchemaExpr[S, A]): Expr[S, A] = Builtin(se)
}

sealed trait RelOp
object RelOp {
  case object Equal              extends RelOp
  case object NotEqual           extends RelOp
  case object LessThan           extends RelOp
  case object LessThanOrEqual    extends RelOp
  case object GreaterThan        extends RelOp
  case object GreaterThanOrEqual extends RelOp
}

sealed trait ArithOp
object ArithOp {
  case object Add      extends ArithOp
  case object Subtract extends ArithOp
  case object Multiply extends ArithOp
}

sealed trait AggFunction[A, B] { def name: String }
object AggFunction {
  case class Count[A]() extends AggFunction[A, Long]        { val name = "COUNT" }
  case object Sum       extends AggFunction[Double, Double]  { val name = "SUM" }
  case object Avg       extends AggFunction[Double, Double]  { val name = "AVG" }
  case class Min[A]()   extends AggFunction[A, A]            { val name = "MIN" }
  case class Max[A]()   extends AggFunction[A, A]            { val name = "MAX" }
}

// --- Extension methods with bridge ---

implicit final class OpticExprOps[S, A](private val optic: Optic[S, A]) {
  def in(values: A*)(implicit schema: Schema[A]): Expr[S, Boolean]           = Expr.In(Expr.col(optic), values.toList, schema)
  def between(low: A, high: A)(implicit schema: Schema[A]): Expr[S, Boolean] = Expr.Between(Expr.col(optic), low, high, schema)
  def isNull: Expr[S, Boolean]                   = Expr.IsNull(Expr.col(optic))
  def isNotNull: Expr[S, Boolean]                = Expr.Not(Expr.IsNull(Expr.col(optic)))
}

implicit final class StringOpticExprOps[S](private val optic: Optic[S, String]) {
  def like(pattern: String): Expr[S, Boolean] = Expr.Like(Expr.col(optic), pattern)
}

implicit final class ExprBooleanOps[S](private val self: Expr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.And(self, other)
  def &&(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.And(self, Expr.fromSchemaExpr(other))
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.Or(self, other)
  def ||(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.Or(self, Expr.fromSchemaExpr(other))
  def unary_! : Expr[S, Boolean]                          = Expr.Not(self)
}

implicit final class SchemaExprBooleanBridge[S](private val self: SchemaExpr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.And(Expr.fromSchemaExpr(self), other)
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.Or(Expr.fromSchemaExpr(self), other)
  def toExpr: Expr[S, Boolean] = Expr.fromSchemaExpr(self)
}

// --- SQL rendering ---

def columnName(optic: zio.blocks.schema.Optic[_, _]): String =
  optic.toDynamic.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")

def columnName(path: DynamicOptic): String =
  path.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")

def sqlLiteral[A](value: A, schema: Schema[A]): String = {
  val dv = schema.toDynamicValue(value)
  dv match {
    case p: DynamicValue.Primitive => p.value match {
      case _: PrimitiveValue.String  => s"'${value.toString.replace("'", "''")}'"
      case b: PrimitiveValue.Boolean => if (b.value) "TRUE" else "FALSE"
      case _                         => value.toString
    }
    case _ => value.toString
  }
}

def sqlLiteralDV(dv: DynamicValue): String = dv match {
  case DynamicValue.Primitive(pv) =>
    pv match {
      case PrimitiveValue.String(s)  => s"'${s.replace("'", "''")}'"
      case PrimitiveValue.Boolean(b) => if (b) "TRUE" else "FALSE"
      case PrimitiveValue.Int(n)     => n.toString
      case PrimitiveValue.Long(n)    => n.toString
      case PrimitiveValue.Double(n)  => n.toString
      case PrimitiveValue.Float(n)   => n.toString
      case PrimitiveValue.Short(n)   => n.toString
      case PrimitiveValue.Byte(n)    => n.toString
      case other                     => other.toString
    }
  case other => other.toString
}

def exprToSql[S, A](expr: Expr[S, A]): String = expr match {
  case Expr.Builtin(schemaExpr) => schemaExprToSql(schemaExpr)
  case Expr.Column(optic)       => columnName(optic)
  case Expr.Lit(value, schema)  => sqlLiteral(value, schema)
  case Expr.Relational(left, right, op) =>
    val sqlOp = op match {
      case RelOp.Equal => "="; case RelOp.NotEqual => "<>"
      case RelOp.LessThan => "<"; case RelOp.LessThanOrEqual => "<="
      case RelOp.GreaterThan => ">"; case RelOp.GreaterThanOrEqual => ">="
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"
  case Expr.And(l, r)                   => s"(${exprToSql(l)} AND ${exprToSql(r)})"
  case Expr.Or(l, r)                    => s"(${exprToSql(l)} OR ${exprToSql(r)})"
  case Expr.Not(e)                      => s"NOT (${exprToSql(e)})"
  case Expr.Arithmetic(left, right, op) =>
    val sqlOp = op match {
      case ArithOp.Add => "+"; case ArithOp.Subtract => "-"; case ArithOp.Multiply => "*"
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"
  case Expr.StringConcat(l, r)         => s"CONCAT(${exprToSql(l)}, ${exprToSql(r)})"
  case Expr.StringRegexMatch(regex, s) => s"(${exprToSql(s)} LIKE ${exprToSql(regex)})"
  case Expr.StringLength(s)            => s"LENGTH(${exprToSql(s)})"
  case Expr.In(e, values, schema) =>
    s"${exprToSql(e)} IN (${values.map(v => sqlLiteral(v, schema)).mkString(", ")})"
  case Expr.Between(e, low, high, schema) =>
    s"(${exprToSql(e)} BETWEEN ${sqlLiteral(low, schema)} AND ${sqlLiteral(high, schema)})"
  case Expr.IsNull(e)        => s"${exprToSql(e)} IS NULL"
  case Expr.Like(e, pattern) => s"${exprToSql(e)} LIKE '${pattern.replace("'", "''")}'"
  case Expr.Agg(func, e)     => s"${func.name}(${exprToSql(e)})"
  case Expr.CaseWhen(branches, otherwise) =>
    val cases = branches.map { case (cond, value) =>
      s"WHEN ${exprToSql(cond)} THEN ${exprToSql(value)}"
    }.mkString(" ")
    val elseClause = otherwise.map(e => s" ELSE ${exprToSql(e)}").getOrElse("")
    s"CASE $cases$elseClause END"
}

def schemaExprToSql[S, A](expr: SchemaExpr[S, A]): String =
  toSqlDynamic(expr.dynamic)

def toSqlDynamic(expr: DynamicSchemaExpr): String = expr match {
  case DynamicSchemaExpr.Select(path)   => columnName(path)
  case DynamicSchemaExpr.Literal(value, _) => sqlLiteralDV(value)
  case DynamicSchemaExpr.Relational(left, right, op) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.RelationalOperator.Equal              => "="
      case DynamicSchemaExpr.RelationalOperator.NotEqual           => "<>"
      case DynamicSchemaExpr.RelationalOperator.LessThan           => "<"
      case DynamicSchemaExpr.RelationalOperator.LessThanOrEqual    => "<="
      case DynamicSchemaExpr.RelationalOperator.GreaterThan        => ">"
      case DynamicSchemaExpr.RelationalOperator.GreaterThanOrEqual => ">="
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.Logical(left, right, op) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.LogicalOperator.And => "AND"
      case DynamicSchemaExpr.LogicalOperator.Or  => "OR"
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.Not(inner) =>
    s"NOT (${toSqlDynamic(inner)})"
  case DynamicSchemaExpr.Arithmetic(left, right, op, _) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.ArithmeticOperator.Add      => "+"
      case DynamicSchemaExpr.ArithmeticOperator.Subtract => "-"
      case DynamicSchemaExpr.ArithmeticOperator.Multiply => "*"
      case _                                             => "?"
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.StringConcat(left, right) =>
    s"CONCAT(${toSqlDynamic(left)}, ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.StringRegexMatch(regex, string) =>
    s"(${toSqlDynamic(string)} LIKE ${toSqlDynamic(regex)})"
  case DynamicSchemaExpr.StringLength(string) =>
    s"LENGTH(${toSqlDynamic(string)})"
  case _ => "?"
}

// --- Usage ---

// 1. SQL-specific predicates — seamless composition
val q1 = Product.category.in("Electronics", "Books") &&
  Product.price.between(10.0, 500.0) &&
  (Product.rating >= 4) &&
  Product.name.like("M%")

println(s"SELECT * FROM products WHERE ${exprToSql(q1)}")

// 2. Type-safe aggregation
val countExpr: Expr[Product, Long]   = Expr.count(Expr.col(Product.name))
val avgExpr: Expr[Product, Double]   = Expr.avg(Expr.col(Product.price))
val countSql = exprToSql(countExpr)
val avgSql   = exprToSql(avgExpr)
println(s"SELECT category, $countSql AS cnt, $avgSql AS avg_price FROM products GROUP BY category HAVING $countSql > 2")

// 3. CASE WHEN
val tier = Expr.caseWhen[Product, String](
  (Product.price > 100.0).toExpr -> Expr.lit[Product, String]("expensive"),
  (Product.price > 10.0).toExpr  -> Expr.lit[Product, String]("moderate")
).otherwise(Expr.lit[Product, String]("cheap"))

println(s"SELECT name, price, ${exprToSql(tier)} AS tier FROM products")
```

## Going Further

- **[Part 1: Expressions](./query-dsl-reified-optics.md)** -- Building query expressions with reified optics
- **[Part 2: SQL Generation](./query-dsl-sql.md)** -- Translating built-in expressions to SQL
- **[Part 4: A Fluent SQL Builder](./query-dsl-fluent-builder.md)** -- Type-safe SELECT, UPDATE, INSERT, DELETE with seamless condition mixing
- **[SchemaExpr Reference](../reference/schema/schema-expr.md)** -- Full API coverage of expression types
- **[Optics Reference](../reference/schema/optics.md)** -- Lens, Prism, Optional, and Traversal

The translation pattern shown here extends to any domain where `SchemaExpr` falls short. The same approach works for MongoDB operators (`$in`, `$exists`, `$elemMatch`), Elasticsearch queries (`terms`, `range`, `exists`), or GraphQL filters. Define an independent ADT, provide a `Expr.fromSchemaExpr` lift for built-in expressions, add your domain-specific nodes, and let your interpreter inspect `SchemaExpr#dynamic` only inside its internal translation helpers.

---

## Query DSL with Reified Optics — Part 4: A Fluent SQL Builder

In this guide, we will build a fluent, type-safe SQL statement builder on top of the query expression language from Parts 1–3. By the end, you will have builder functions for SELECT, UPDATE, INSERT, and DELETE that compose naturally with both built-in [`SchemaExpr`](../reference/schema/schema-expr.md) operators and extended `Expr` operators — no manual `SchemaExprBooleanBridge#toExpr` lifting required.

We'll take an incremental approach: starting with seamless condition composition, then adding table references and fluent statement builders for each SQL operation until we have a complete, native-feeling SQL DSL.

This is Part 4 of the Query DSL series. [Part 1](./query-dsl-reified-optics.md) covered expressions, [Part 2](./query-dsl-sql.md) covered SQL generation, and [Part 3](./query-dsl-extending.md) covered extending the expression language.

**What we'll cover:**

- Bridging `SchemaExpr` and `Expr` for seamless `&&` / `||` composition
- Defining table references with `Table[S]`
- Deriving table names from `Schema` metadata via `Modifier.config` and auto-pluralization
- Building fluent SELECT queries with `SelectStmt#columns`, `SelectStmt#where`, `SelectStmt#orderBy`, `SelectStmt#limit`
- Building UPDATE statements with `UpdateStmt#set` and `UpdateStmt#where`
- Building INSERT and DELETE statements
- Rendering all statement types to SQL strings

## The Problem

Parts 1–3 gave us a powerful expression language for building WHERE clauses. But constructing full SQL statements still requires string concatenation:

```scala
val whereSql = exprToSql(myCondition)
val sql = s"UPDATE products SET price = 9.99 WHERE $whereSql"
```

Column names in SET clauses are hand-written strings that can drift from your schema, table names are repeated magic strings, and there is no compile-time connection between the statement structure and your domain types.

Additionally, mixing built-in `SchemaExpr` operators (`===`, `>=`) with extended `Expr` operators (`between`, `like`) required explicit `SchemaExprBooleanBridge#toExpr` calls in Part 3:

```scala
// Part 3: explicit .toExpr required
val q = (Product.rating >= 4).toExpr && Product.price.between(10.0, 500.0)
```

In a fluent builder, this friction breaks the flow. We want to write:

```scala
update(Product.table)
  .set(Product.price, 9.99)
  .where(
    Product.category === "Books" &&
      Product.rating >= 4 &&
      Product.price.between(10.0, 30.0) &&
      Product.name.like("M%")
  )
```

In this guide, we solve both problems: bridge extensions eliminate `SchemaExprBooleanBridge#toExpr`, and type-safe statement builders eliminate string manipulation for SQL construction.

## Prerequisites

This guide builds on [Part 1: Expressions](./query-dsl-reified-optics.md), [Part 2: SQL Generation](./query-dsl-sql.md), and [Part 3: Extending the Expression Language](./query-dsl-extending.md). To add the schema module to an sbt build:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"
```

## Domain Setup

We carry forward the product catalog domain and the Part 3 independent `Expr` ADT. The key additions in Part 4 are bridge extension methods, schema-driven table names, and statement builder types. As in Part 3, the public expression layer stays typed: built-in logic remains `SchemaExpr`, and the SQL builder only drops to the dynamic AST inside interpreter helpers. The following block defines the table type, the domain, the `Expr` ADT, the bridge classes, and the SQL interpreter:

```scala

// --- Table reference ---

case class Table[S](name: String)

object Table {
  def derived[S](implicit schema: Schema[S]): Table[S] = Table(tableName(schema))
}

// --- Table name derivation ---

def tableName[S](schema: Schema[S]): String =
  schema.reflect.modifiers.collectFirst {
    case Modifier.config(key, value) if key == "sql.table_name" => value
  }.getOrElse(pluralize(schema.reflect.typeId.name.toLowerCase))

def pluralize(word: String): String =
  if (word.endsWith("s") || word.endsWith("x") || word.endsWith("z") ||
      word.endsWith("ch") || word.endsWith("sh")) word + "es"
  else if (word.endsWith("y") && !word.endsWith("ay") && !word.endsWith("ey") &&
           !word.endsWith("oy") && !word.endsWith("uy")) word.dropRight(1) + "ies"
  else word + "s"

// --- Domain ---

case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val table: Table[Product] = Table.derived  // "Product" → "products"

  val name: Lens[Product, String]     = optic(_.name)
  val price: Lens[Product, Double]    = optic(_.price)
  val category: Lens[Product, String] = optic(_.category)
  val inStock: Lens[Product, Boolean] = optic(_.inStock)
  val rating: Lens[Product, Int]      = optic(_.rating)
}

case class OrderItem(
  orderId: Int,
  productId: Int,
  quantity: Int,
  unitPrice: Double
)

object OrderItem extends CompanionOptics[OrderItem] {
  implicit val schema: Schema[OrderItem] = Schema.derived
    .modifier(Modifier.config("sql.table_name", "order_items"))

  val table: Table[OrderItem] = Table.derived  // annotation → "order_items"

  val orderId: Lens[OrderItem, Int]      = optic(_.orderId)
  val productId: Lens[OrderItem, Int]    = optic(_.productId)
  val quantity: Lens[OrderItem, Int]     = optic(_.quantity)
  val unitPrice: Lens[OrderItem, Double] = optic(_.unitPrice)
}

// --- Expr ADT (from Part 3) ---

sealed trait Expr[S, A]

object Expr {
  final case class Builtin[S, A](schemaExpr: SchemaExpr[S, A]) extends Expr[S, A]
  final case class Column[S, A](optic: Optic[S, A]) extends Expr[S, A]
  final case class Lit[S, A](value: A, schema: Schema[A]) extends Expr[S, A]

  final case class Relational[S, A](left: Expr[S, A], right: Expr[S, A], op: RelOp) extends Expr[S, Boolean]
  final case class And[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Or[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Not[S](expr: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Arithmetic[S, A](left: Expr[S, A], right: Expr[S, A], op: ArithOp) extends Expr[S, A]
  final case class StringConcat[S](left: Expr[S, String], right: Expr[S, String]) extends Expr[S, String]
  final case class StringRegexMatch[S](regex: Expr[S, String], string: Expr[S, String]) extends Expr[S, Boolean]
  final case class StringLength[S](string: Expr[S, String]) extends Expr[S, Int]

  final case class In[S, A](expr: Expr[S, A], values: List[A], schema: Schema[A]) extends Expr[S, Boolean]
  final case class Between[S, A](expr: Expr[S, A], low: A, high: A, schema: Schema[A]) extends Expr[S, Boolean]
  final case class IsNull[S, A](expr: Expr[S, A]) extends Expr[S, Boolean]
  final case class Like[S](expr: Expr[S, String], pattern: String) extends Expr[S, Boolean]

  def col[S, A](optic: Optic[S, A]): Expr[S, A] = Column(optic)
  def lit[S, A](value: A)(implicit schema: Schema[A]): Expr[S, A] = Lit(value, schema)

  def fromSchemaExpr[S, A](se: SchemaExpr[S, A]): Expr[S, A] = Builtin(se)
}

sealed trait RelOp
object RelOp {
  case object Equal              extends RelOp
  case object NotEqual           extends RelOp
  case object LessThan           extends RelOp
  case object LessThanOrEqual    extends RelOp
  case object GreaterThan        extends RelOp
  case object GreaterThanOrEqual extends RelOp
}

sealed trait ArithOp
object ArithOp {
  case object Add      extends ArithOp
  case object Subtract extends ArithOp
  case object Multiply extends ArithOp
}

// --- SQL helpers ---

def columnName(optic: zio.blocks.schema.Optic[_, _]): String =
  optic.toDynamic.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")

def columnName(path: DynamicOptic): String =
  path.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")

def sqlLiteral[A](value: A, schema: Schema[A]): String = {
  val dv = schema.toDynamicValue(value)
  dv match {
    case p: DynamicValue.Primitive => p.value match {
      case _: PrimitiveValue.String  => s"'${value.toString.replace("'", "''")}'"
      case b: PrimitiveValue.Boolean => if (b.value) "TRUE" else "FALSE"
      case _                         => value.toString
    }
    case _ => value.toString
  }
}

def sqlLiteralDV(dv: DynamicValue): String = dv match {
  case DynamicValue.Primitive(pv) =>
    pv match {
      case PrimitiveValue.String(s)  => s"'${s.replace("'", "''")}'"
      case PrimitiveValue.Boolean(b) => if (b) "TRUE" else "FALSE"
      case PrimitiveValue.Int(n)     => n.toString
      case PrimitiveValue.Long(n)    => n.toString
      case PrimitiveValue.Double(n)  => n.toString
      case PrimitiveValue.Float(n)   => n.toString
      case PrimitiveValue.Short(n)   => n.toString
      case PrimitiveValue.Byte(n)    => n.toString
      case other                     => other.toString
    }
  case other => other.toString
}

// --- Optic extension methods ---

implicit final class OpticExprOps[S, A](private val optic: Optic[S, A]) {
  def in(values: A*)(implicit schema: Schema[A]): Expr[S, Boolean]           = Expr.In(Expr.col(optic), values.toList, schema)
  def between(low: A, high: A)(implicit schema: Schema[A]): Expr[S, Boolean] = Expr.Between(Expr.col(optic), low, high, schema)
  def isNull: Expr[S, Boolean]                   = Expr.IsNull(Expr.col(optic))
  def isNotNull: Expr[S, Boolean]                = Expr.Not(Expr.IsNull(Expr.col(optic)))
}

implicit final class StringOpticExprOps[S](private val optic: Optic[S, String]) {
  def like(pattern: String): Expr[S, Boolean] = Expr.Like(Expr.col(optic), pattern)
}

// --- Boolean combinators with bridge extensions ---

implicit final class ExprBooleanOps[S](private val self: Expr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.And(self, other)
  def &&(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.And(self, Expr.fromSchemaExpr(other))
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.Or(self, other)
  def ||(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.Or(self, Expr.fromSchemaExpr(other))
  def unary_! : Expr[S, Boolean]                          = Expr.Not(self)
}

implicit final class SchemaExprBooleanBridge[S](private val self: SchemaExpr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.And(Expr.fromSchemaExpr(self), other)
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.Or(Expr.fromSchemaExpr(self), other)
  def toExpr: Expr[S, Boolean] = Expr.fromSchemaExpr(self)
}

// --- Single unified SQL interpreter ---

def exprToSql[S, A](expr: Expr[S, A]): String = expr match {
  case Expr.Builtin(schemaExpr) => schemaExprToSql(schemaExpr)
  case Expr.Column(optic)       => columnName(optic)
  case Expr.Lit(value, schema)  => sqlLiteral(value, schema)
  case Expr.Relational(left, right, op) =>
    val sqlOp = op match {
      case RelOp.Equal              => "="
      case RelOp.NotEqual           => "<>"
      case RelOp.LessThan           => "<"
      case RelOp.LessThanOrEqual    => "<="
      case RelOp.GreaterThan        => ">"
      case RelOp.GreaterThanOrEqual => ">="
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"
  case Expr.And(l, r) => s"(${exprToSql(l)} AND ${exprToSql(r)})"
  case Expr.Or(l, r)  => s"(${exprToSql(l)} OR ${exprToSql(r)})"
  case Expr.Not(e)    => s"NOT (${exprToSql(e)})"
  case Expr.Arithmetic(left, right, op) =>
    val sqlOp = op match {
      case ArithOp.Add      => "+"
      case ArithOp.Subtract => "-"
      case ArithOp.Multiply => "*"
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"
  case Expr.StringConcat(l, r)         => s"CONCAT(${exprToSql(l)}, ${exprToSql(r)})"
  case Expr.StringRegexMatch(regex, s) => s"(${exprToSql(s)} LIKE ${exprToSql(regex)})"
  case Expr.StringLength(s)            => s"LENGTH(${exprToSql(s)})"
  case Expr.In(e, values, schema) =>
    s"${exprToSql(e)} IN (${values.map(v => sqlLiteral(v, schema)).mkString(", ")})"
  case Expr.Between(e, low, high, schema) =>
    s"(${exprToSql(e)} BETWEEN ${sqlLiteral(low, schema)} AND ${sqlLiteral(high, schema)})"
  case Expr.IsNull(e)        => s"${exprToSql(e)} IS NULL"
  case Expr.Like(e, pattern) => s"${exprToSql(e)} LIKE '${pattern.replace("'", "''")}'"
}

def schemaExprToSql[S, A](expr: SchemaExpr[S, A]): String =
  toSqlDynamic(expr.dynamic)

def toSqlDynamic(expr: DynamicSchemaExpr): String = expr match {
  case DynamicSchemaExpr.Select(path)   => columnName(path)
  case DynamicSchemaExpr.Literal(value, _) => sqlLiteralDV(value)
  case DynamicSchemaExpr.Relational(left, right, op) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.RelationalOperator.Equal              => "="
      case DynamicSchemaExpr.RelationalOperator.NotEqual           => "<>"
      case DynamicSchemaExpr.RelationalOperator.LessThan           => "<"
      case DynamicSchemaExpr.RelationalOperator.LessThanOrEqual    => "<="
      case DynamicSchemaExpr.RelationalOperator.GreaterThan        => ">"
      case DynamicSchemaExpr.RelationalOperator.GreaterThanOrEqual => ">="
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.Logical(left, right, op) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.LogicalOperator.And => "AND"
      case DynamicSchemaExpr.LogicalOperator.Or  => "OR"
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.Not(inner) =>
    s"NOT (${toSqlDynamic(inner)})"
  case DynamicSchemaExpr.Arithmetic(left, right, op, _) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.ArithmeticOperator.Add      => "+"
      case DynamicSchemaExpr.ArithmeticOperator.Subtract => "-"
      case DynamicSchemaExpr.ArithmeticOperator.Multiply => "*"
      case _                                             => "?"
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.StringConcat(left, right) =>
    s"CONCAT(${toSqlDynamic(left)}, ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.StringRegexMatch(regex, string) =>
    s"(${toSqlDynamic(string)} LIKE ${toSqlDynamic(regex)})"
  case DynamicSchemaExpr.StringLength(string) =>
    s"LENGTH(${toSqlDynamic(string)})"
}
```

## Seamless Condition Composition

In Part 3, composing `SchemaExpr` and `Expr` values required explicit `SchemaExprBooleanBridge#toExpr` calls because `SchemaExpr.&&` is a direct method that only accepts other `SchemaExpr` values. Part 4 adds **bridge implicit classes** that handle the cross-type case automatically:

```scala
// When SchemaExpr.&& receives an Expr argument, the direct method
// doesn't match (Expr is not SchemaExpr), so this implicit class kicks in:
implicit final class SchemaExprBooleanBridge[S](
  private val self: SchemaExpr[S, Boolean]
) extends AnyVal {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean] = ...
}

// And vice versa — Expr.&& can accept SchemaExpr directly:
implicit final class ExprBooleanOps[S](
  private val self: Expr[S, Boolean]
) extends AnyVal {
  def &&(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = ...
}
```

Scala's implicit conversion resolution makes this work:

1. **`SchemaExpr && SchemaExpr`** — the built-in direct method matches, returns `SchemaExpr`
2. **`SchemaExpr && Expr`** — the direct method doesn't match, the bridge implicit class kicks in, returns `Expr`
3. **`Expr && SchemaExpr`** — the bridge overload on `ExprBooleanOps` matches, returns `Expr`
4. **`Expr && Expr`** — the standard `ExprBooleanOps.&&` overload matches, returns `Expr`

The result type automatically widens to `Expr` whenever an `Expr` value enters the chain. No `SchemaExprBooleanBridge#toExpr` needed:

```scala
val condition =
  Product.price.between(10.0, 500.0) &&
  (Product.category === "Electronics") &&
  (Product.rating >= 4) &&
  Product.name.like("L%")
// condition: Expr[Product, Boolean] = And(
//   left = And(
//     left = And(
//       left = Between(
//         expr = Column(
//           LensImpl(
//             sources = Array(
//               Record(
//                 fields = Vector(
//                   Term(
//                     name = "name",
//                     value = Primitive(
//                       primitiveType = String(None),
//                       typeId = String,
//                       primitiveBinding = Primitive(),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List(),
//                       storedDefaultValue = None,
//                       storedExamples = List()
//                     ),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                     modifiers = List()
//                   ),
//                   Term(
//                     name = "price",
//                     value = Primitive(
//                       primitiveType = Double(None),
//                       typeId = Double,
//                       primitiveBinding = Primitive(),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List(),
//                       storedDefaultValue = None,
//                       storedExamples = List()
//                     ),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                     modifiers = List()
//                   ),
//                   Term(
//                     name = "category",
//                     value = Primitive(
//                       primitiveType = String(None),
//                       typeId = String,
//                       primitiveBinding = Primitive(),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List(),
//                       storedDefaultValue = None,
//                       storedExamples = List()
//                     ),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
// ...

exprToSql(condition)
// res0: String = "((((price BETWEEN 10.0 AND 500.0) AND (category = 'Electronics')) AND (rating >= 4)) AND name LIKE 'L%')"
```

The first `&&` already involves `between`, so the bridge activates as soon as an `Expr` enters the chain. From that point on, the whole condition is represented as `Expr`.

## Schema-Driven Table Names

The `Table[S]` type associates a SQL table name with a schema type. Rather than hand-writing each table name, `Table.derived` extracts it from `Schema` metadata:

```scala
case class Table[S](name: String)

object Table {
  def derived[S](implicit schema: Schema[S]): Table[S] = Table(tableName(schema))
}
```

The `tableName` function checks the schema's modifiers for an explicit `sql.table_name` annotation. If none is found, it lowercases the case class name and applies English pluralization:

```scala
def tableName[S](schema: Schema[S]): String =
  schema.reflect.modifiers.collectFirst {
    case Modifier.config(key, value) if key == "sql.table_name" => value
  }.getOrElse(pluralize(schema.reflect.typeId.name.toLowerCase))
```

This gives you two ways to control the table name:

**Auto-pluralization (zero configuration):**

```scala
object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived
  val table: Table[Product] = Table.derived  // "Product" → "products"
}
```

**Explicit annotation (for compound names or irregular plurals):**

```scala
object OrderItem extends CompanionOptics[OrderItem] {
  implicit val schema: Schema[OrderItem] = Schema.derived
    .modifier(Modifier.config("sql.table_name", "order_items"))
  val table: Table[OrderItem] = Table.derived  // annotation → "order_items"
}
```

Without the annotation, `OrderItem` would become `orderitems`. The `Modifier.config("sql.table_name", "order_items")` annotation gives you precise control when the default doesn't match your database convention. To print both derived table names:

```scala
Product.table.name
// res1: String = "products"

OrderItem.table.name
// res2: String = "order_items"
```

You can still use the explicit constructor `Table[S]("my_table")` when you need a name that doesn't come from a schema at all.

`Table[S]` is the entry point for every statement builder. The type parameter `S` connects the table to its domain type, ensuring you can only use `Product` lenses in queries against `Product.table`.

## The SELECT Builder

The SELECT builder uses immutable case classes with `copy` for a fluent API:

```scala
sealed trait SortOrder
object SortOrder {
  case object Asc  extends SortOrder
  case object Desc extends SortOrder
}

case class SelectStmt[S](
  table: Table[S],
  columnList: List[String] = List("*"),
  whereExpr: Option[Expr[S, Boolean]] = None,
  orderByList: List[(String, SortOrder)] = Nil,
  limitCount: Option[Int] = None
) {
  def columns(optics: Optic[S, _]*): SelectStmt[S] =
    copy(columnList = optics.map(columnName).toList)
  def where(cond: Expr[S, Boolean]): SelectStmt[S] =
    copy(whereExpr = Some(cond))
  def where(cond: SchemaExpr[S, Boolean]): SelectStmt[S] =
    copy(whereExpr = Some(Expr.fromSchemaExpr(cond)))
  def orderBy(optic: Optic[S, _], order: SortOrder = SortOrder.Asc): SelectStmt[S] =
    copy(orderByList = orderByList :+ (columnName(optic), order))
  def limit(n: Int): SelectStmt[S] =
    copy(limitCount = Some(n))
}

def select[S](table: Table[S]): SelectStmt[S] = SelectStmt(table)

def renderSelect[S](stmt: SelectStmt[S]): String = {
  val cols = stmt.columnList.mkString(", ")
  val where = stmt.whereExpr.map(c => s" WHERE ${exprToSql(c)}").getOrElse("")
  val orderBy = if (stmt.orderByList.isEmpty) "" else {
    val orders = stmt.orderByList.map { case (col, order) =>
      val dir = order match { case SortOrder.Asc => "ASC"; case SortOrder.Desc => "DESC" }
      s"$col $dir"
    }.mkString(", ")
    s" ORDER BY $orders"
  }
  val limit = stmt.limitCount.map(n => s" LIMIT $n").getOrElse("")
  s"SELECT $cols FROM ${stmt.table.name}$where$orderBy$limit"
}
```

Each builder method returns a new `SelectStmt` with the updated field. The `SelectStmt#where` method is overloaded to accept both `Expr` and `SchemaExpr` — `SchemaExpr` values are translated via `Expr.fromSchemaExpr`, so pure `SchemaExpr` chains and mixed chains both work:

```scala
// Pure SchemaExpr conditions
val basicSelect = select(Product.table)
  .columns(Product.name, Product.price)
  .where(Product.inStock === true)
// basicSelect: SelectStmt[Product] = SelectStmt(
//   table = Table("products"),
//   columnList = List("name", "price"),
//   whereExpr = Some(
//     Builtin(
//       SchemaExpr(
//         dynamic = Relational(
//           left = Select(DynamicOptic(ArraySeq(Field("inStock")))),
//           right = Literal(
//             value = Primitive(Boolean(true)),
//             schema = Schema(
//               Primitive(
//                 primitiveType = Boolean(None),
//                 typeId = Boolean,
//                 primitiveBinding = Primitive(),
//                 doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                 modifiers = List(),
//                 storedDefaultValue = None,
//                 storedExamples = List()
//               )
//             )
//           ),
//           operator = Equal
//         ),
//         inputSchema = Schema(
//           Record(
//             fields = Vector(
//               Term(
//                 name = "name",
//                 value = Primitive(
//                   primitiveType = String(None),
//                   typeId = String,
//                   primitiveBinding = Primitive(),
//                   doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                   modifiers = List(),
//                   storedDefaultValue = None,
//                   storedExamples = List()
//                 ),
//                 doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                 modifiers = List()
//               ),
//               Term(
//                 name = "price",
//                 value = Primitive(
//                   primitiveType = Double(None),
//                   typeId = Double,
//                   primitiveBinding = Primitive(),
//                   doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                   modifiers = List(),
// ...

renderSelect(basicSelect)
// res3: String = "SELECT name, price FROM products WHERE (inStock = TRUE)"

// Mixed conditions with ordering and limit
val advancedSelect = select(Product.table)
  .columns(Product.name, Product.price, Product.category)
  .where(
    Product.price.between(10.0, 500.0) &&
    (Product.category === "Electronics") &&
    (Product.rating >= 4)
  )
  .orderBy(Product.price, SortOrder.Desc)
  .limit(10)
// advancedSelect: SelectStmt[Product] = SelectStmt(
//   table = Table("products"),
//   columnList = List("name", "price", "category"),
//   whereExpr = Some(
//     And(
//       left = And(
//         left = Between(
//           expr = Column(
//             LensImpl(
//               sources = Array(
//                 Record(
//                   fields = Vector(
//                     Term(
//                       name = "name",
//                       value = Primitive(
//                         primitiveType = String(None),
//                         typeId = String,
//                         primitiveBinding = Primitive(),
//                         doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                         modifiers = List(),
//                         storedDefaultValue = None,
//                         storedExamples = List()
//                       ),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List()
//                     ),
//                     Term(
//                       name = "price",
//                       value = Primitive(
//                         primitiveType = Double(None),
//                         typeId = Double,
//                         primitiveBinding = Primitive(),
//                         doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                         modifiers = List(),
//                         storedDefaultValue = None,
//                         storedExamples = List()
//                       ),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List()
//                     ),
//                     Term(
//                       name = "category",
//                       value = Primitive(
//                         primitiveType = String(None),
//                         typeId = String,
//                         primitiveBinding = Primitive(),
//                         doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                         modifiers = List(),
//                         storedDefaultValue = None,
// ...

renderSelect(advancedSelect)
// res4: String = "SELECT name, price, category FROM products WHERE (((price BETWEEN 10.0 AND 500.0) AND (category = 'Electronics')) AND (rating >= 4)) ORDER BY price DESC LIMIT 10"
```

## The UPDATE Builder

The UPDATE builder accumulates `UpdateStmt#set` calls, each pairing an optic with a value:

```scala
case class Assignment(column: String, value: String)

case class UpdateStmt[S](
  table: Table[S],
  assignments: List[Assignment] = Nil,
  whereExpr: Option[Expr[S, Boolean]] = None
) {
  def set[A](optic: Optic[S, A], value: A)(implicit schema: Schema[A]): UpdateStmt[S] =
    copy(assignments = assignments :+ Assignment(columnName(optic), sqlLiteral(value, schema)))
  def where(cond: Expr[S, Boolean]): UpdateStmt[S] =
    copy(whereExpr = Some(cond))
  def where(cond: SchemaExpr[S, Boolean]): UpdateStmt[S] =
    copy(whereExpr = Some(Expr.fromSchemaExpr(cond)))
}

def update[S](table: Table[S]): UpdateStmt[S] = UpdateStmt(table)

def renderUpdate[S](stmt: UpdateStmt[S]): String = {
  val sets = stmt.assignments.map(a => s"${a.column} = ${a.value}").mkString(", ")
  val where = stmt.whereExpr.map(c => s" WHERE ${exprToSql(c)}").getOrElse("")
  s"UPDATE ${stmt.table.name} SET $sets$where"
}
```

The `UpdateStmt#set` method uses the optic to extract the column name and `sqlLiteral` (via the implicit `Schema[A]`) to render the value. The type parameter `A` on `set[A](optic: Optic[S, A], value: A)` ensures you cannot assign a `String` to a `Double` field. To build an UPDATE with a condition that mixes `SchemaExpr` and `Expr` operators:

```scala
val basicUpdate =
  update(Product.table)
    .set(Product.price, 9.99)
    .where(
      Product.price.between(10.0, 30.0) &&
        Product.name.like("M%") &&
        (Product.category === "Books") &&
        (Product.rating >= 4) &&
        (Product.inStock === true)
    )
// basicUpdate: UpdateStmt[Product] = UpdateStmt(
//   table = Table("products"),
//   assignments = List(Assignment(column = "price", value = "9.99")),
//   whereExpr = Some(
//     And(
//       left = And(
//         left = And(
//           left = And(
//             left = Between(
//               expr = Column(
//                 LensImpl(
//                   sources = Array(
//                     Record(
//                       fields = Vector(
//                         Term(
//                           name = "name",
//                           value = Primitive(
//                             primitiveType = String(None),
//                             typeId = String,
//                             primitiveBinding = Primitive(),
//                             doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                             modifiers = List(),
//                             storedDefaultValue = None,
//                             storedExamples = List()
//                           ),
//                           doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                           modifiers = List()
//                         ),
//                         Term(
//                           name = "price",
//                           value = Primitive(
//                             primitiveType = Double(None),
//                             typeId = Double,
//                             primitiveBinding = Primitive(),
//                             doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                             modifiers = List(),
//                             storedDefaultValue = None,
//                             storedExamples = List()
//                           ),
//                           doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                           modifiers = List()
//                         ),
//                         Term(
//                           name = "category",
//                           value = Primitive(
//                             primitiveType = String(None),
//                             typeId = String,
//                             primitiveBinding = Primitive(),
//                             doc = Doc(blocks = IndexedSeq(), metadata = Map()),
// ...

renderUpdate(basicUpdate)
// res5: String = "UPDATE products SET price = 9.99 WHERE (((((price BETWEEN 10.0 AND 30.0) AND name LIKE 'M%') AND (category = 'Books')) AND (rating >= 4)) AND (inStock = TRUE))"
```

Multiple `UpdateStmt#set` calls accumulate:

```scala
val multiUpdate =
  update(Product.table)
    .set(Product.price, 19.99)
    .set(Product.inStock, false)
    .where(Product.category === "Clearance")
// multiUpdate: UpdateStmt[Product] = UpdateStmt(
//   table = Table("products"),
//   assignments = List(
//     Assignment(column = "price", value = "19.99"),
//     Assignment(column = "inStock", value = "FALSE")
//   ),
//   whereExpr = Some(
//     Builtin(
//       SchemaExpr(
//         dynamic = Relational(
//           left = Select(DynamicOptic(ArraySeq(Field("category")))),
//           right = Literal(
//             value = Primitive(String("Clearance")),
//             schema = Schema(
//               Primitive(
//                 primitiveType = String(None),
//                 typeId = String,
//                 primitiveBinding = Primitive(),
//                 doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                 modifiers = List(),
//                 storedDefaultValue = None,
//                 storedExamples = List()
//               )
//             )
//           ),
//           operator = Equal
//         ),
//         inputSchema = Schema(
//           Record(
//             fields = Vector(
//               Term(
//                 name = "name",
//                 value = Primitive(
//                   primitiveType = String(None),
//                   typeId = String,
//                   primitiveBinding = Primitive(),
//                   doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                   modifiers = List(),
//                   storedDefaultValue = None,
//                   storedExamples = List()
//                 ),
//                 doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                 modifiers = List()
//               ),
//               Term(
//                 name = "price",
//                 value = Primitive(
//                   primitiveType = Double(None),
//                   typeId = Double,
// ...

renderUpdate(multiUpdate)
// res6: String = "UPDATE products SET price = 19.99, inStock = FALSE WHERE (category = 'Clearance')"
```

## INSERT and DELETE

The INSERT builder collects column-value pairs via `InsertStmt#set`:

```scala
case class InsertStmt[S](
  table: Table[S],
  assignments: List[Assignment] = Nil
) {
  def set[A](optic: Optic[S, A], value: A)(implicit schema: Schema[A]): InsertStmt[S] =
    copy(assignments = assignments :+ Assignment(columnName(optic), sqlLiteral(value, schema)))
}

def insertInto[S](table: Table[S]): InsertStmt[S] = InsertStmt(table)

def renderInsert[S](stmt: InsertStmt[S]): String = {
  val cols = stmt.assignments.map(_.column).mkString(", ")
  val vals = stmt.assignments.map(_.value).mkString(", ")
  s"INSERT INTO ${stmt.table.name} ($cols) VALUES ($vals)"
}
```

The DELETE builder takes an optional WHERE clause:

```scala
case class DeleteStmt[S](
  table: Table[S],
  whereExpr: Option[Expr[S, Boolean]] = None
) {
  def where(cond: Expr[S, Boolean]): DeleteStmt[S] =
    copy(whereExpr = Some(cond))
  def where(cond: SchemaExpr[S, Boolean]): DeleteStmt[S] =
    copy(whereExpr = Some(Expr.fromSchemaExpr(cond)))
}

def deleteFrom[S](table: Table[S]): DeleteStmt[S] = DeleteStmt(table)

def renderDelete[S](stmt: DeleteStmt[S]): String = {
  val where = stmt.whereExpr.map(c => s" WHERE ${exprToSql(c)}").getOrElse("")
  s"DELETE FROM ${stmt.table.name}$where"
}
```

To build and render an INSERT and a DELETE statement:

```scala
val ins = insertInto(Product.table)
  .set(Product.name, "Wireless Mouse")
  .set(Product.price, 29.99)
  .set(Product.category, "Electronics")
  .set(Product.inStock, true)
  .set(Product.rating, 4)
// ins: InsertStmt[Product] = InsertStmt(
//   table = Table("products"),
//   assignments = List(
//     Assignment(column = "name", value = "'Wireless Mouse'"),
//     Assignment(column = "price", value = "29.99"),
//     Assignment(column = "category", value = "'Electronics'"),
//     Assignment(column = "inStock", value = "TRUE"),
//     Assignment(column = "rating", value = "4")
//   )
// )

renderInsert(ins)
// res7: String = "INSERT INTO products (name, price, category, inStock, rating) VALUES ('Wireless Mouse', 29.99, 'Electronics', TRUE, 4)"

val del = deleteFrom(Product.table)
  .where(
    Product.price.between(0.0, 1.0) &&
    (Product.inStock === false)
  )
// del: DeleteStmt[Product] = DeleteStmt(
//   table = Table("products"),
//   whereExpr = Some(
//     And(
//       left = Between(
//         expr = Column(
//           LensImpl(
//             sources = Array(
//               Record(
//                 fields = Vector(
//                   Term(
//                     name = "name",
//                     value = Primitive(
//                       primitiveType = String(None),
//                       typeId = String,
//                       primitiveBinding = Primitive(),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List(),
//                       storedDefaultValue = None,
//                       storedExamples = List()
//                     ),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                     modifiers = List()
//                   ),
//                   Term(
//                     name = "price",
//                     value = Primitive(
//                       primitiveType = Double(None),
//                       typeId = Double,
//                       primitiveBinding = Primitive(),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List(),
//                       storedDefaultValue = None,
//                       storedExamples = List()
//                     ),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                     modifiers = List()
//                   ),
//                   Term(
//                     name = "category",
//                     value = Primitive(
//                       primitiveType = String(None),
//                       typeId = String,
//                       primitiveBinding = Primitive(),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List(),
//                       storedDefaultValue = None,
//                       storedExamples = List()
//                     ),
// ...

renderDelete(del)
// res8: String = "DELETE FROM products WHERE ((price BETWEEN 0.0 AND 1.0) AND (inStock = FALSE))"
```

:::tip
For batch inserts, create one `InsertStmt` per row and render each separately. The builder pattern keeps each row type-safe.
:::

## Putting It Together

Here is a complete example combining schema-driven table names, bridge extensions, all four statement builders, and the renderers. The block is self-contained: it repeats the `Expr` ADT, extension methods, SQL rendering, and builder types so the usage code at the end stays focused on building queries with typed optics and `SchemaExpr`, not on manipulating `Dynamic*` nodes directly:

```scala

// --- Table reference with schema-driven names ---

case class Table[S](name: String)

object Table {
  def derived[S](implicit schema: Schema[S]): Table[S] = Table(tableName(schema))
}

def tableName[S](schema: Schema[S]): String =
  schema.reflect.modifiers.collectFirst {
    case Modifier.config(key, value) if key == "sql.table_name" => value
  }.getOrElse(pluralize(schema.reflect.typeId.name.toLowerCase))

def pluralize(word: String): String =
  if (word.endsWith("s") || word.endsWith("x") || word.endsWith("z") ||
      word.endsWith("ch") || word.endsWith("sh")) word + "es"
  else if (word.endsWith("y") && !word.endsWith("ay") && !word.endsWith("ey") &&
           !word.endsWith("oy") && !word.endsWith("uy")) word.dropRight(1) + "ies"
  else word + "s"

// --- Domain ---

case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val table: Table[Product] = Table.derived  // "Product" → "products"

  val name: Lens[Product, String]     = optic(_.name)
  val price: Lens[Product, Double]    = optic(_.price)
  val category: Lens[Product, String] = optic(_.category)
  val inStock: Lens[Product, Boolean] = optic(_.inStock)
  val rating: Lens[Product, Int]      = optic(_.rating)
}

case class OrderItem(
  orderId: Int,
  productId: Int,
  quantity: Int,
  unitPrice: Double
)

object OrderItem extends CompanionOptics[OrderItem] {
  implicit val schema: Schema[OrderItem] = Schema.derived
    .modifier(Modifier.config("sql.table_name", "order_items"))

  val table: Table[OrderItem] = Table.derived  // annotation → "order_items"

  val orderId: Lens[OrderItem, Int]      = optic(_.orderId)
  val productId: Lens[OrderItem, Int]    = optic(_.productId)
  val quantity: Lens[OrderItem, Int]     = optic(_.quantity)
  val unitPrice: Lens[OrderItem, Double] = optic(_.unitPrice)
}

// --- Expr ADT ---

sealed trait Expr[S, A]

object Expr {
  final case class Builtin[S, A](schemaExpr: SchemaExpr[S, A]) extends Expr[S, A]
  final case class Column[S, A](optic: Optic[S, A]) extends Expr[S, A]
  final case class Lit[S, A](value: A, schema: Schema[A]) extends Expr[S, A]

  final case class Relational[S, A](left: Expr[S, A], right: Expr[S, A], op: RelOp) extends Expr[S, Boolean]
  final case class And[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Or[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Not[S](expr: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Arithmetic[S, A](left: Expr[S, A], right: Expr[S, A], op: ArithOp) extends Expr[S, A]
  final case class StringConcat[S](left: Expr[S, String], right: Expr[S, String]) extends Expr[S, String]
  final case class StringRegexMatch[S](regex: Expr[S, String], string: Expr[S, String]) extends Expr[S, Boolean]
  final case class StringLength[S](string: Expr[S, String]) extends Expr[S, Int]

  final case class In[S, A](expr: Expr[S, A], values: List[A], schema: Schema[A]) extends Expr[S, Boolean]
  final case class Between[S, A](expr: Expr[S, A], low: A, high: A, schema: Schema[A]) extends Expr[S, Boolean]
  final case class IsNull[S, A](expr: Expr[S, A]) extends Expr[S, Boolean]
  final case class Like[S](expr: Expr[S, String], pattern: String) extends Expr[S, Boolean]

  def col[S, A](optic: Optic[S, A]): Expr[S, A] = Column(optic)
  def lit[S, A](value: A)(implicit schema: Schema[A]): Expr[S, A] = Lit(value, schema)

  def fromSchemaExpr[S, A](se: SchemaExpr[S, A]): Expr[S, A] = Builtin(se)
}

sealed trait RelOp
object RelOp {
  case object Equal              extends RelOp
  case object NotEqual           extends RelOp
  case object LessThan           extends RelOp
  case object LessThanOrEqual    extends RelOp
  case object GreaterThan        extends RelOp
  case object GreaterThanOrEqual extends RelOp
}

sealed trait ArithOp
object ArithOp {
  case object Add      extends ArithOp
  case object Subtract extends ArithOp
  case object Multiply extends ArithOp
}

// --- Extension methods with bridge ---

implicit final class OpticExprOps[S, A](private val optic: Optic[S, A]) {
  def in(values: A*)(implicit schema: Schema[A]): Expr[S, Boolean]           = Expr.In(Expr.col(optic), values.toList, schema)
  def between(low: A, high: A)(implicit schema: Schema[A]): Expr[S, Boolean] = Expr.Between(Expr.col(optic), low, high, schema)
  def isNull: Expr[S, Boolean]                   = Expr.IsNull(Expr.col(optic))
  def isNotNull: Expr[S, Boolean]                = Expr.Not(Expr.IsNull(Expr.col(optic)))
}

implicit final class StringOpticExprOps[S](private val optic: Optic[S, String]) {
  def like(pattern: String): Expr[S, Boolean] = Expr.Like(Expr.col(optic), pattern)
}

implicit final class ExprBooleanOps[S](private val self: Expr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.And(self, other)
  def &&(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.And(self, Expr.fromSchemaExpr(other))
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.Or(self, other)
  def ||(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.Or(self, Expr.fromSchemaExpr(other))
  def unary_! : Expr[S, Boolean]                          = Expr.Not(self)
}

implicit final class SchemaExprBooleanBridge[S](private val self: SchemaExpr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.And(Expr.fromSchemaExpr(self), other)
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.Or(Expr.fromSchemaExpr(self), other)
  def toExpr: Expr[S, Boolean] = Expr.fromSchemaExpr(self)
}

// --- SQL rendering ---

def columnName(optic: zio.blocks.schema.Optic[_, _]): String =
  optic.toDynamic.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")

def columnName(path: DynamicOptic): String =
  path.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")

def sqlLiteral[A](value: A, schema: Schema[A]): String = {
  val dv = schema.toDynamicValue(value)
  dv match {
    case p: DynamicValue.Primitive => p.value match {
      case _: PrimitiveValue.String  => s"'${value.toString.replace("'", "''")}'"
      case b: PrimitiveValue.Boolean => if (b.value) "TRUE" else "FALSE"
      case _                         => value.toString
    }
    case _ => value.toString
  }
}

def sqlLiteralDV(dv: DynamicValue): String = dv match {
  case DynamicValue.Primitive(pv) =>
    pv match {
      case PrimitiveValue.String(s)  => s"'${s.replace("'", "''")}'"
      case PrimitiveValue.Boolean(b) => if (b) "TRUE" else "FALSE"
      case PrimitiveValue.Int(n)     => n.toString
      case PrimitiveValue.Long(n)    => n.toString
      case PrimitiveValue.Double(n)  => n.toString
      case PrimitiveValue.Float(n)   => n.toString
      case PrimitiveValue.Short(n)   => n.toString
      case PrimitiveValue.Byte(n)    => n.toString
      case other                     => other.toString
    }
  case other => other.toString
}

def exprToSql[S, A](expr: Expr[S, A]): String = expr match {
  case Expr.Builtin(schemaExpr) => schemaExprToSql(schemaExpr)
  case Expr.Column(optic)       => columnName(optic)
  case Expr.Lit(value, schema)  => sqlLiteral(value, schema)
  case Expr.Relational(left, right, op) =>
    val sqlOp = op match {
      case RelOp.Equal              => "="
      case RelOp.NotEqual           => "<>"
      case RelOp.LessThan           => "<"
      case RelOp.LessThanOrEqual    => "<="
      case RelOp.GreaterThan        => ">"
      case RelOp.GreaterThanOrEqual => ">="
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"
  case Expr.And(l, r) => s"(${exprToSql(l)} AND ${exprToSql(r)})"
  case Expr.Or(l, r)  => s"(${exprToSql(l)} OR ${exprToSql(r)})"
  case Expr.Not(e)    => s"NOT (${exprToSql(e)})"
  case Expr.Arithmetic(left, right, op) =>
    val sqlOp = op match {
      case ArithOp.Add      => "+"
      case ArithOp.Subtract => "-"
      case ArithOp.Multiply => "*"
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"
  case Expr.StringConcat(l, r)         => s"CONCAT(${exprToSql(l)}, ${exprToSql(r)})"
  case Expr.StringRegexMatch(regex, s) => s"(${exprToSql(s)} LIKE ${exprToSql(regex)})"
  case Expr.StringLength(s)            => s"LENGTH(${exprToSql(s)})"
  case Expr.In(e, values, schema) =>
    s"${exprToSql(e)} IN (${values.map(v => sqlLiteral(v, schema)).mkString(", ")})"
  case Expr.Between(e, low, high, schema) =>
    s"(${exprToSql(e)} BETWEEN ${sqlLiteral(low, schema)} AND ${sqlLiteral(high, schema)})"
  case Expr.IsNull(e)        => s"${exprToSql(e)} IS NULL"
  case Expr.Like(e, pattern) => s"${exprToSql(e)} LIKE '${pattern.replace("'", "''")}'"
}

def schemaExprToSql[S, A](expr: SchemaExpr[S, A]): String =
  toSqlDynamic(expr.dynamic)

def toSqlDynamic(expr: DynamicSchemaExpr): String = expr match {
  case DynamicSchemaExpr.Select(path)   => columnName(path)
  case DynamicSchemaExpr.Literal(value, _) => sqlLiteralDV(value)
  case DynamicSchemaExpr.Relational(left, right, op) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.RelationalOperator.Equal              => "="
      case DynamicSchemaExpr.RelationalOperator.NotEqual           => "<>"
      case DynamicSchemaExpr.RelationalOperator.LessThan           => "<"
      case DynamicSchemaExpr.RelationalOperator.LessThanOrEqual    => "<="
      case DynamicSchemaExpr.RelationalOperator.GreaterThan        => ">"
      case DynamicSchemaExpr.RelationalOperator.GreaterThanOrEqual => ">="
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.Logical(left, right, op) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.LogicalOperator.And => "AND"
      case DynamicSchemaExpr.LogicalOperator.Or  => "OR"
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.Not(inner) =>
    s"NOT (${toSqlDynamic(inner)})"
  case DynamicSchemaExpr.Arithmetic(left, right, op, _) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.ArithmeticOperator.Add      => "+"
      case DynamicSchemaExpr.ArithmeticOperator.Subtract => "-"
      case DynamicSchemaExpr.ArithmeticOperator.Multiply => "*"
      case _                                             => "?"
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.StringConcat(left, right) =>
    s"CONCAT(${toSqlDynamic(left)}, ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.StringRegexMatch(regex, string) =>
    s"(${toSqlDynamic(string)} LIKE ${toSqlDynamic(regex)})"
  case DynamicSchemaExpr.StringLength(string) =>
    s"LENGTH(${toSqlDynamic(string)})"
}

// --- Statement builders ---

sealed trait SortOrder
object SortOrder {
  case object Asc  extends SortOrder
  case object Desc extends SortOrder
}

case class Assignment(column: String, value: String)

case class SelectStmt[S](
  table: Table[S],
  columnList: List[String] = List("*"),
  whereExpr: Option[Expr[S, Boolean]] = None,
  orderByList: List[(String, SortOrder)] = Nil,
  limitCount: Option[Int] = None
) {
  def columns(optics: Optic[S, _]*): SelectStmt[S] =
    copy(columnList = optics.map(columnName).toList)
  def where(cond: Expr[S, Boolean]): SelectStmt[S] =
    copy(whereExpr = Some(cond))
  def where(cond: SchemaExpr[S, Boolean]): SelectStmt[S] =
    copy(whereExpr = Some(Expr.fromSchemaExpr(cond)))
  def orderBy(optic: Optic[S, _], order: SortOrder = SortOrder.Asc): SelectStmt[S] =
    copy(orderByList = orderByList :+ (columnName(optic), order))
  def limit(n: Int): SelectStmt[S] =
    copy(limitCount = Some(n))
}

case class UpdateStmt[S](
  table: Table[S],
  assignments: List[Assignment] = Nil,
  whereExpr: Option[Expr[S, Boolean]] = None
) {
  def set[A](optic: Optic[S, A], value: A)(implicit schema: Schema[A]): UpdateStmt[S] =
    copy(assignments = assignments :+ Assignment(columnName(optic), sqlLiteral(value, schema)))
  def where(cond: Expr[S, Boolean]): UpdateStmt[S] =
    copy(whereExpr = Some(cond))
  def where(cond: SchemaExpr[S, Boolean]): UpdateStmt[S] =
    copy(whereExpr = Some(Expr.fromSchemaExpr(cond)))
}

case class InsertStmt[S](
  table: Table[S],
  assignments: List[Assignment] = Nil
) {
  def set[A](optic: Optic[S, A], value: A)(implicit schema: Schema[A]): InsertStmt[S] =
    copy(assignments = assignments :+ Assignment(columnName(optic), sqlLiteral(value, schema)))
}

case class DeleteStmt[S](
  table: Table[S],
  whereExpr: Option[Expr[S, Boolean]] = None
) {
  def where(cond: Expr[S, Boolean]): DeleteStmt[S] =
    copy(whereExpr = Some(cond))
  def where(cond: SchemaExpr[S, Boolean]): DeleteStmt[S] =
    copy(whereExpr = Some(Expr.fromSchemaExpr(cond)))
}

def select[S](table: Table[S]): SelectStmt[S]       = SelectStmt(table)
def update[S](table: Table[S]): UpdateStmt[S]       = UpdateStmt(table)
def insertInto[S](table: Table[S]): InsertStmt[S]   = InsertStmt(table)
def deleteFrom[S](table: Table[S]): DeleteStmt[S]   = DeleteStmt(table)

// --- Renderers ---

def renderSelect[S](stmt: SelectStmt[S]): String = {
  val cols = stmt.columnList.mkString(", ")
  val where = stmt.whereExpr.map(c => s" WHERE ${exprToSql(c)}").getOrElse("")
  val orderBy = if (stmt.orderByList.isEmpty) "" else {
    val orders = stmt.orderByList.map { case (col, order) =>
      val dir = order match { case SortOrder.Asc => "ASC"; case SortOrder.Desc => "DESC" }
      s"$col $dir"
    }.mkString(", ")
    s" ORDER BY $orders"
  }
  val limit = stmt.limitCount.map(n => s" LIMIT $n").getOrElse("")
  s"SELECT $cols FROM ${stmt.table.name}$where$orderBy$limit"
}

def renderUpdate[S](stmt: UpdateStmt[S]): String = {
  val sets = stmt.assignments.map(a => s"${a.column} = ${a.value}").mkString(", ")
  val where = stmt.whereExpr.map(c => s" WHERE ${exprToSql(c)}").getOrElse("")
  s"UPDATE ${stmt.table.name} SET $sets$where"
}

def renderInsert[S](stmt: InsertStmt[S]): String = {
  val cols = stmt.assignments.map(_.column).mkString(", ")
  val vals = stmt.assignments.map(_.value).mkString(", ")
  s"INSERT INTO ${stmt.table.name} ($cols) VALUES ($vals)"
}

def renderDelete[S](stmt: DeleteStmt[S]): String = {
  val where = stmt.whereExpr.map(c => s" WHERE ${exprToSql(c)}").getOrElse("")
  s"DELETE FROM ${stmt.table.name}$where"
}

// --- Usage ---

// Schema-driven table names
println(Product.table.name)    // "products" (auto-pluralized)
println(OrderItem.table.name)  // "order_items" (from annotation)

// SELECT with mixed conditions
val q = select(Product.table)
  .columns(Product.name, Product.price, Product.category)
  .where(
    Product.category.in("Electronics", "Books") &&
    Product.price.between(10.0, 500.0) &&
    (Product.rating >= 4)
  )
  .orderBy(Product.price, SortOrder.Desc)
  .limit(20)

println(renderSelect(q))

// UPDATE with seamless condition mixing
val u = update(Product.table)
  .set(Product.price, 9.99)
  .where(
    Product.price.between(10.0, 30.0) &&
      Product.name.like("M%") &&
      (Product.category === "Books") &&
      (Product.rating >= 4) &&
      (Product.inStock === true)
  )

println(renderUpdate(u))

// INSERT
val i = insertInto(Product.table)
  .set(Product.name, "Wireless Mouse")
  .set(Product.price, 29.99)
  .set(Product.category, "Electronics")
  .set(Product.inStock, true)
  .set(Product.rating, 4)

println(renderInsert(i))

// DELETE
val d = deleteFrom(Product.table)
  .where(Product.price.between(0.0, 1.0) && (Product.inStock === false))

println(renderDelete(d))

// Cross-table query with annotated table name
val orderQuery = select(OrderItem.table)
  .columns(OrderItem.orderId, OrderItem.productId, OrderItem.quantity)
  .where(OrderItem.quantity >= 3)
  .orderBy(OrderItem.unitPrice, SortOrder.Desc)

println(renderSelect(orderQuery))
```

## Going Further

- **[Part 1: Expressions](./query-dsl-reified-optics.md)** — Building query expressions with reified optics
- **[Part 2: SQL Generation](./query-dsl-sql.md)** — Translating built-in expressions to SQL
- **[Part 3: Extending the Expression Language](./query-dsl-extending.md)** — Adding custom operators beyond SchemaExpr
- **[SchemaExpr Reference](../reference/schema/schema-expr.md)** — Full API coverage of expression types
- **[Optics Reference](../reference/schema/optics.md)** — Lens, Prism, Optional, and Traversal

The builder pattern shown here extends naturally to JOIN clauses (using lenses from multiple table types), subqueries (nesting `SelectStmt` in WHERE conditions), and parameterized queries (collecting `?` placeholders and parameter values during rendering). Each of these builds on the same foundation: optics for column names, `Expr` for conditions, and immutable builders for statement structure.

---

## Query DSL with Reified Optics — Part 1: Expressions

In this guide, we will build a type-safe query DSL for filtering, comparing, and computing over domain data using ZIO Blocks' reified optics and schema expressions. By the end, you will have a composable query language that works on any schema-equipped data type, supporting equality checks, comparisons, boolean logic, arithmetic, and string operations.

We'll take an incremental approach: starting with simple field-level equality checks, then adding comparison operators, boolean combinators, arithmetic expressions, and string operations until we have a complete, expressive query DSL.

**What we'll cover:**

- Defining domain types with schemas and optics
- Building equality and comparison queries using `===`, `>`, `<`, `>=`, `<=`
- Combining queries with `&&` and `||`
- Using arithmetic operators (`+`, `-`, `*`) in expressions
- Working with string operations (`matches`, `concat`, `length`)
- Querying through nested structures and collections
- Evaluating queries against data

## The Problem

When you need to query or filter collections of structured data in Scala, you typically write ad-hoc predicate functions:

```scala
case class Product(name: String, price: Double, category: String, inStock: Boolean)

val products: List[Product] = loadProducts()

// Filtering with ad-hoc predicates
val results = products.filter(p =>
  p.category == "Electronics" && p.price < 500.0 && p.inStock
)
```

This works, but the predicate `p => p.category == "Electronics" && p.price < 500.0 && p.inStock` is an opaque function. You cannot inspect it, serialize it, translate it to SQL, send it to a remote service, or optimize it. It is a black box.

If you need to build a query builder for a database, a filter language for an API, or a rule engine, you need queries as **data** -- inspectable, composable, serializable expression trees. Building these by hand means defining an AST, writing an evaluator, and maintaining type safety across all operators -- a significant amount of boilerplate for every domain type.

In this guide, we'll solve this by using ZIO Blocks' `SchemaExpr` and reified optics, which give us composable, type-safe query expressions for free, derived directly from your data model's schema.

## Prerequisites

Add the ZIO Blocks Schema dependency:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"
```

```scala

```

This guide assumes familiarity with ZIO Blocks schemas and basic optics. See the [Schema](../reference/schema/schema.md) and [Optics](../reference/schema/optics.md) reference pages for background.

## Defining Your Domain

We'll build a product catalog query DSL. First, define the domain types with schemas and optics:

```scala
case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]      = optic(_.name)
  val price: Lens[Product, Double]     = optic(_.price)
  val category: Lens[Product, String]  = optic(_.category)
  val inStock: Lens[Product, Boolean]  = optic(_.inStock)
  val rating: Lens[Product, Int]       = optic(_.rating)
}
```

Three things are required for each queryable type:

1. **`Schema.derived`** -- captures the type's structure at runtime
2. **`CompanionOptics[T]`** -- provides the `optic` macro for deriving lenses
3. **Named lenses** for each field you want to query on

Each lens is a *reified* field accessor: unlike `_.price`, the lens `Product.price` is a first-class value that carries the field name, the source schema, and the focus schema. This metadata is what makes the query DSL possible.

## Equality and Comparison Queries

Every optic in ZIO Blocks has built-in operators that create `SchemaExpr` values -- expression trees representing queries:

```scala
// Equality check
val isElectronics: SchemaExpr[Product, Boolean] =
  Product.category === "Electronics"

// Greater-than comparison
val expensiveItems: SchemaExpr[Product, Boolean] =
  Product.price > 100.0

// Less-than-or-equal
val budgetFriendly: SchemaExpr[Product, Boolean] =
  Product.price <= 50.0

// Comparison against a literal
val highRated: SchemaExpr[Product, Boolean] =
  Product.rating >= 4
```

The full set of comparison operators:

| Operator | Meaning                |
|----------|------------------------|
| `===`    | Equal to               |
| `!=`     | Not equal to           |
| `>`      | Greater than           |
| `>=`     | Greater than or equal  |
| `<`      | Less than              |
| `<=`     | Less than or equal     |

Each operator works in two forms: an optic against a literal, as in `Product.price > 100.0`, which compares a field to a value, and an optic against another optic, as in `Product.rating === Product.rating`, which compares two fields.

## Evaluating Queries

A `SchemaExpr[A, Boolean]` is a predicate over `A`. Evaluate it with `SchemaExpr#eval`:

```scala
val laptop = Product("Laptop", 999.99, "Electronics", true, 5)
val pen    = Product("Pen", 2.50, "Office", true, 3)
```

```scala
isElectronics.eval(laptop)
// res0: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(true))
isElectronics.eval(pen)
// res1: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(false))

expensiveItems.eval(laptop)
// res2: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(true))
expensiveItems.eval(pen)
// res3: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(false))
```

The result type is `Either[OpticCheck, Seq[Boolean]]`. A successful evaluation gives `Right(Seq(true))` or `Right(Seq(false))`, and `Left(opticCheck)` means the optic path failed (for example, a prism encountered the wrong variant case).

For lenses, evaluation always succeeds because lenses always resolve.

## Combining Queries with Boolean Logic

Combine queries with `&&` (and), `||` (or), and `!` (not):

```scala
// AND: electronics under $500
val affordableElectronics: SchemaExpr[Product, Boolean] =
  (Product.category === "Electronics") && (Product.price < 500.0)

// OR: either cheap or highly rated
val goodDeal: SchemaExpr[Product, Boolean] =
  (Product.price < 10.0) || (Product.rating >= 5)

// NOT: items that are out of stock
val outOfStock: SchemaExpr[Product, Boolean] =
  !Product.inStock
```

```scala
affordableElectronics.eval(laptop)
// res4: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(false))
goodDeal.eval(laptop)
// res5: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(true))
goodDeal.eval(pen)
// res6: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(true))
outOfStock.eval(laptop)
// res7: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(false))
```

Boolean combinators also work on `SchemaExpr` values, not just optics. This means you can build complex compound queries:

```scala
val complexQuery: SchemaExpr[Product, Boolean] =
  ((Product.category === "Electronics") && (Product.price < 500.0)) ||
  ((Product.category === "Office") && (Product.rating >= 4))
```

```scala
complexQuery.eval(laptop)
// res8: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(false))
complexQuery.eval(pen)
// res9: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(false))
```

## Arithmetic Expressions

Optics on numeric fields support `+`, `-`, and `*`:

```scala
val discountedPrice: SchemaExpr[Product, Double] =
  Product.price * 0.9

val priceWithTax: SchemaExpr[Product, Double] =
  Product.price * 1.08
```

```scala
discountedPrice.eval(laptop)
// res10: Either[OpticCheck, Seq[Double]] = Right(IndexedSeq(899.991))
priceWithTax.eval(pen)
// res11: Either[OpticCheck, Seq[Double]] = Right(IndexedSeq(2.7))
```

Arithmetic operators are available for all numeric types: `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, and `BigDecimal`.

## String Operations

Optics on `String` fields provide string-specific operations:

```scala
// Regex matching
val startsWithL: SchemaExpr[Product, Boolean] =
  Product.name.matches("L.*")

// String concatenation
val labeledName: SchemaExpr[Product, String] =
  Product.name.concat(" [SALE]")

// String length
val nameLength: SchemaExpr[Product, Int] =
  Product.name.length
```

```scala
startsWithL.eval(laptop)
// res12: Either[OpticCheck, Seq[Boolean]] = Right(List(true))
startsWithL.eval(pen)
// res13: Either[OpticCheck, Seq[Boolean]] = Right(List(false))

labeledName.eval(laptop)
// res14: Either[OpticCheck, Seq[String]] = Right(IndexedSeq("Laptop [SALE]"))

nameLength.eval(laptop)
// res15: Either[OpticCheck, Seq[Int]] = Right(IndexedSeq(6))
nameLength.eval(pen)
// res16: Either[OpticCheck, Seq[Int]] = Right(IndexedSeq(3))
```

| String Operation | Signature                                   | Description     |
| ---------------- | ------------------------------------------- | --------------- |
| `matches`        | `(regex: String) => SchemaExpr[S, Boolean]` | Regex match     |
| `concat`         | `(suffix: String) => SchemaExpr[S, String]` | Append a string |
| `length`         | `SchemaExpr[S, Int]`                        | String length   |

## Dynamic Evaluation

Every `SchemaExpr` can also evaluate to `DynamicValue`, which is useful when you need format-agnostic results (e.g., serializing query results to JSON):

```scala
val priceExpr: SchemaExpr[Product, Double] = Product.price * 0.9
```

```scala
priceExpr.evalDynamic(laptop)
// res17: Either[OpticCheck, Seq[DynamicValue]] = Right(
//   IndexedSeq(Primitive(Double(899.991)))
// )
```

The `SchemaExpr#evalDynamic` method converts results to `DynamicValue` representations, enabling integration with serialization formats without knowing the concrete type.

## Querying Nested Structures

The real power of reified optics emerges with nested data. Define a richer domain:

```scala

case class Address(city: String, country: String)
object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

case class Seller(name: String, address: Address, rating: Double)
object Seller extends CompanionOptics[Seller] {
  implicit val schema: Schema[Seller] = Schema.derived

  val name: Lens[Seller, String]       = optic(_.name)
  val rating: Lens[Seller, Double]     = optic(_.rating)
  // Compose through nested structure directly
  val city: Lens[Seller, String]       = optic(_.address.city)
  val country: Lens[Seller, String]    = optic(_.address.country)
}
```

The `optic(_.address.city)` macro composes a lens from `Seller` to `Address` with a lens from `Address` to `String`, producing a single `Lens[Seller, String]`. Now we can query nested fields as if they were top-level:

```scala
val localSeller: SchemaExpr[Seller, Boolean] =
  (Seller.city === "Berlin") && (Seller.rating >= 4.0)

val seller = Seller("TechShop", Address("Berlin", "Germany"), 4.5)
```

```scala
localSeller.eval(seller)
// res19: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(true))
```

## Querying Through Collections

For collection fields, use traversals to query across all elements:

```scala

case class LineItem(sku: String, price: Double, quantity: Int)
object LineItem {
  implicit val schema: Schema[LineItem] = Schema.derived
}

case class Order(id: String, items: List[LineItem])
object Order extends CompanionOptics[Order] {
  implicit val schema: Schema[Order] = Schema.derived

  val id: Lens[Order, String]                  = optic(_.id)
  val allPrices: Traversal[Order, Double]      = optic(_.items.each.price)
  val allSkus: Traversal[Order, String]        = optic(_.items.each.sku)
  val allQuantities: Traversal[Order, Int]     = optic(_.items.each.quantity)
}
```

Traversals produce `SchemaExpr` values that evaluate to **multiple results**:

```scala
val order = Order("ORD-1", List(
  LineItem("SKU-A", 29.99, 2),
  LineItem("SKU-B", 149.99, 1),
  LineItem("SKU-C", 9.99, 5)
))

// Each element is evaluated independently
val hasExpensiveItem: SchemaExpr[Order, Boolean] =
  Order.allPrices > 100.0
```

```scala
hasExpensiveItem.eval(order)
// res21: Either[OpticCheck, Seq[Boolean]] = Right(
//   IndexedSeq(false, true, false)
// )
```

:::tip
When a traversal-based expression produces multiple results, each element in the sequence corresponds to one focused value from the traversal. This makes it straightforward to check whether *any* or *all* elements satisfy a condition by examining the result sequence.
:::

## Filtering a Collection

With `SchemaExpr` as a reified predicate, you can build a generic filter function that works with any schema-equipped type:

```scala

case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]      = optic(_.name)
  val price: Lens[Product, Double]     = optic(_.price)
  val category: Lens[Product, String]  = optic(_.category)
  val inStock: Lens[Product, Boolean]  = optic(_.inStock)
  val rating: Lens[Product, Int]       = optic(_.rating)
}

def filter[A](items: List[A], predicate: SchemaExpr[A, Boolean]): List[A] =
  items.filter(item =>
    predicate.eval(item) match {
      case Right(results) => results.forall(_ == true)
      case Left(_)        => false
    }
  )

val catalog = List(
  Product("Laptop", 999.99, "Electronics", true, 5),
  Product("Mouse", 29.99, "Electronics", true, 4),
  Product("Pen", 2.50, "Office", true, 3),
  Product("Monitor", 349.99, "Electronics", false, 5),
  Product("Notebook", 5.99, "Office", true, 4)
)

val query = (Product.category === "Electronics") && (Product.inStock === true) && (Product.price < 500.0)
```

```scala
filter(catalog, query).map(_.name)
// res23: List[String] = List("Mouse")
```

The `filter` function knows nothing about `Product` -- it works with any `SchemaExpr[A, Boolean]`. The query is data, not a lambda, so it could be serialized, logged, or translated to a database query.

## Putting It Together

Here is a complete, self-contained example combining all the techniques from this guide:

```scala

// --- Domain ---

case class Address(city: String, country: String)
object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int,
  warehouse: Address
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]      = optic(_.name)
  val price: Lens[Product, Double]     = optic(_.price)
  val category: Lens[Product, String]  = optic(_.category)
  val inStock: Lens[Product, Boolean]  = optic(_.inStock)
  val rating: Lens[Product, Int]       = optic(_.rating)
  val city: Lens[Product, String]      = optic(_.warehouse.city)
}

// --- Generic query filter ---

def filter[A](items: List[A], predicate: SchemaExpr[A, Boolean]): List[A] =
  items.filter(item =>
    predicate.eval(item) match {
      case Right(results) => results.forall(_ == true)
      case Left(_)        => false
    }
  )

// --- Usage ---

val catalog = List(
  Product("Laptop", 999.99, "Electronics", true, 5, Address("Berlin", "Germany")),
  Product("Mouse", 29.99, "Electronics", true, 4, Address("Berlin", "Germany")),
  Product("Pen", 2.50, "Office", true, 3, Address("London", "UK")),
  Product("Monitor", 349.99, "Electronics", false, 5, Address("Berlin", "Germany")),
  Product("Notebook", 5.99, "Office", true, 4, Address("London", "UK"))
)

// Compose a query: in-stock electronics under $500, from Berlin, highly rated
val query =
  (Product.category === "Electronics") &&
  (Product.inStock === true) &&
  (Product.price < 500.0) &&
  (Product.city === "Berlin") &&
  (Product.rating >= 4)

val results = filter(catalog, query)
// results: List(Product("Mouse", 29.99, "Electronics", true, 4, Address("Berlin", "Germany")))

// String operations
val searchQuery = Product.name.matches(".*top$")
val matches = filter(catalog, searchQuery)
// matches: List(Product("Laptop", ...))

// Arithmetic: compute discounted prices
val discounted = Product.price * 0.8
catalog.foreach { p =>
  println(s"${Product.name.get(p)}: ${discounted.eval(p)}")
}
```

## Going Further

- **[Part 2: SQL Generation](./query-dsl-sql.md)** -- Translating query expressions to SQL
- **[Part 3: Extending the Expression Language](./query-dsl-extending.md)** -- Adding custom operators (IN, BETWEEN, aggregates) beyond SchemaExpr
- **[Part 4: A Fluent SQL Builder](./query-dsl-fluent-builder.md)** -- Type-safe SELECT, UPDATE, INSERT, DELETE with seamless condition mixing
- **[Optics Reference](../reference/schema/optics.md)** -- Full API coverage of Lens, Prism, Optional, and Traversal
- **[DynamicOptic Reference](../reference/schema/dynamic-optic.md)** -- Runtime optic paths for programmatic query construction
- **[Schema Reference](../reference/schema/schema.md)** -- Schema derivation and type-level metadata
- **[Path Interpolator](../reference/schema/path-interpolator.md)** -- String-based path construction with `p"..."` syntax

`SchemaExpr` is a typed wrapper around a serializable `DynamicSchemaExpr` tree, making it straightforward to write interpreters that translate queries to SQL, MongoDB filters, Elasticsearch queries, or any other target language. Because each optic carries its `DynamicOptic` path (via `toDynamic`), you can extract field names and paths programmatically for these translations.

---

## Query DSL with Reified Optics — Part 2: SQL Generation

In this guide, we will build a SQL query generator that translates ZIO Blocks' `SchemaExpr` expression trees into SQL WHERE clauses, SELECT statements, and parameterized queries. By the end, you will have an interpreter that takes any `SchemaExpr`-based query and produces executable SQL, covering comparisons, boolean logic, arithmetic, string operations, nested structures, and safe parameterization.

This is Part 2 of the Query DSL series. [Part 1](./query-dsl-reified-optics.md) covered building query expressions with reified optics. Here, we interpret those expressions as SQL.

**What we'll cover:**

- Interpreting `SchemaExpr` while keeping the typed API at the boundary
- Extracting column names from optic paths using `DynamicOptic`
- Translating relational, logical, arithmetic, and string operations to SQL
- Building complete `SELECT ... FROM ... WHERE ...` statements
- Generating parameterized queries for SQL injection safety
- Handling nested structures with table-qualified column names
- Building upserts, inspecting generated SQL, and dumping SQL at compile time

## The Problem

In Part 1, we built composable query expressions as data -- `SchemaExpr` values that can be inspected, combined, and evaluated in-memory. But in real applications, data lives in databases. You need to translate those same queries into SQL.

The naive approach is to write SQL strings by hand for every query:

```scala
// Manual SQL for each query variant
def findProducts(category: Option[String], maxPrice: Option[Double], inStock: Option[Boolean]): String = {
  val conditions = List.newBuilder[String]
  category.foreach(c => conditions += s"category = '$c'")      // SQL injection!
  maxPrice.foreach(p => conditions += s"price < $p")
  inStock.foreach(s => conditions += s"in_stock = $s")
  val where = conditions.result().mkString(" AND ")
  s"SELECT * FROM products" + (if (where.nonEmpty) s" WHERE $where" else "")
}
```

This is fragile, repetitive, and vulnerable to SQL injection. Every new query shape requires new string-building code. The query logic is duplicated -- once as a `SchemaExpr` for in-memory filtering, and again as hand-written SQL for the database.

`SchemaExpr` is the user-facing query API. Internally it wraps a `DynamicSchemaExpr` — a sealed trait whose cases represent the expression AST. That means we can write a single interpreter that accepts `SchemaExpr`, then crosses into the dynamic AST internally to translate *any* query expression into SQL. Write the interpreter once, and every query you build with the Part 1 DSL automatically gets a SQL translation.

## Prerequisites

This guide builds on [Part 1: Expressions](./query-dsl-reified-optics.md). You should be comfortable building `SchemaExpr` values with optic operators (`===`, `>`, `&&`, etc.).

Add the dependency:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"
```

Then import the schema package:

```scala

```

## Domain Setup

We reuse the product catalog domain from Part 1:

```scala
case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]      = optic(_.name)
  val price: Lens[Product, Double]     = optic(_.price)
  val category: Lens[Product, String]  = optic(_.category)
  val inStock: Lens[Product, Boolean]  = optic(_.inStock)
  val rating: Lens[Product, Int]       = optic(_.rating)
}
```

## The SchemaExpr API

Before we build the interpreter, keep the API boundary in mind: application code builds `SchemaExpr[A, B]` values, while interpreter code may inspect the underlying `DynamicSchemaExpr` through `SchemaExpr#dynamic`, as the following outline shows:

```
SchemaExpr[A, B]                        -- user-facing, typed API
└── .dynamic: DynamicSchemaExpr         -- interpreter/runtime boundary (cases used in this guide)
    ├── Select(path: DynamicOptic)      -- field reference
    ├── Literal(value: DynamicValue, schema: Schema[_])  -- constant value
    ├── Relational(left, right, op)     -- comparisons
    ├── Logical(left, right, op)        -- boolean operators
    ├── Not(expr)                       -- negation
    ├── Arithmetic(left, right, op, _)  -- numeric operators
    ├── StringConcat(left, right)       -- string concatenation
    ├── StringRegexMatch(regex, string) -- pattern matching
    └── StringLength(string)            -- string length
```

Most users never need to construct `DynamicSchemaExpr` directly. The normal workflow is:

1. Build a typed `SchemaExpr` with optics and operators.
2. Pass that `SchemaExpr` to your interpreter.
3. Let the interpreter read `SchemaExpr#dynamic` internally.

The dynamic cases are still worth understanding because they are what your interpreter will pattern-match on:

```
RelationalOperator
├── LessThan
├── GreaterThan
├── LessThanOrEqual
├── GreaterThanOrEqual
├── Equal
└── NotEqual

LogicalOperator
├── And
└── Or

ArithmeticOperator
├── Add
├── Subtract
├── Multiply
├── Divide
├── Pow
└── Modulo
```

The interpreter below handles `Add`, `Subtract` and `Multiply` and emits `?` for every other case.

Each dynamic case carries enough information to produce SQL: `Select` nodes carry field paths, `Literal` nodes carry values, and operator nodes carry the operation type. Our interpreter walks this tree and emits SQL fragments.

## Extracting Column Names from Optics

The first challenge is turning a reified optic into a SQL column name. Every `Optic[S, A]` has a `toDynamic` method that returns a `DynamicOptic` -- a sequence of path nodes. For a lens like `Product.price`, the path is `[Field("price")]`. We extract the field name from the last `Field` node:

```scala
def columnName(optic: zio.blocks.schema.Optic[?, ?]): String = {
  val nodes = optic.toDynamic.nodes
  nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")
}

def columnName(path: DynamicOptic): String =
  path.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")
```

This converts the optic path to a column name. For a simple field like `Product.price`, it produces `"price"`. For a nested path, it joins field names with underscores (we will refine this for table-qualified names later).

Evaluating it on the Part 1 lenses gives:

```scala
columnName(Product.price)
// res0: String = "price"
columnName(Product.name)
// res1: String = "name"
columnName(Product.category)
// res2: String = "category"
```

## Translating Literals to SQL

Once we cross the interpreter boundary, literal values appear as `DynamicValue`. We need a function to format them as SQL:

```scala
def sqlLiteralDV(dv: DynamicValue): String = dv match {
  case DynamicValue.Primitive(pv) =>
    pv match {
      case PrimitiveValue.String(s)  => s"'${s.replace("'", "''")}'"
      case PrimitiveValue.Boolean(b) => if (b) "TRUE" else "FALSE"
      case PrimitiveValue.Int(n)     => n.toString
      case PrimitiveValue.Long(n)    => n.toString
      case PrimitiveValue.Double(n)  => n.toString
      case PrimitiveValue.Float(n)   => n.toString
      case PrimitiveValue.Short(n)   => n.toString
      case PrimitiveValue.Byte(n)    => n.toString
      case other                     => other.toString
    }
  case other => other.toString
}
```

## Building the SQL Interpreter

Now we build the core interpreter. The public entry point accepts `SchemaExpr`; the internal helper does the `DynamicSchemaExpr` pattern matching:

```scala
def toSql[A, B](expr: SchemaExpr[A, B]): String = toSqlDynamic(expr.dynamic)

private def toSqlDynamic(expr: DynamicSchemaExpr): String = expr match {

  // Field reference → column name
  case DynamicSchemaExpr.Select(path) =>
    columnName(path)

  // Constant value → SQL literal
  case DynamicSchemaExpr.Literal(value, _) =>
    sqlLiteralDV(value)

  // Comparison operators → SQL relational operators
  case DynamicSchemaExpr.Relational(left, right, op) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.RelationalOperator.Equal              => "="
      case DynamicSchemaExpr.RelationalOperator.NotEqual           => "<>"
      case DynamicSchemaExpr.RelationalOperator.LessThan           => "<"
      case DynamicSchemaExpr.RelationalOperator.LessThanOrEqual    => "<="
      case DynamicSchemaExpr.RelationalOperator.GreaterThan        => ">"
      case DynamicSchemaExpr.RelationalOperator.GreaterThanOrEqual => ">="
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"

  // Boolean operators → AND / OR
  case DynamicSchemaExpr.Logical(left, right, op) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.LogicalOperator.And => "AND"
      case DynamicSchemaExpr.LogicalOperator.Or  => "OR"
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"

  // Negation → NOT
  case DynamicSchemaExpr.Not(inner) =>
    s"NOT (${toSqlDynamic(inner)})"

  // Arithmetic → SQL math operators
  case DynamicSchemaExpr.Arithmetic(left, right, op, _) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.ArithmeticOperator.Add      => "+"
      case DynamicSchemaExpr.ArithmeticOperator.Subtract => "-"
      case DynamicSchemaExpr.ArithmeticOperator.Multiply => "*"
      case _                                             => "?"
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"

  // String concatenation → CONCAT()
  case DynamicSchemaExpr.StringConcat(left, right) =>
    s"CONCAT(${toSqlDynamic(left)}, ${toSqlDynamic(right)})"

  // Regex match → LIKE (simplified: regex semantics are not preserved)
  case DynamicSchemaExpr.StringRegexMatch(regex, string) =>
    s"(${toSqlDynamic(string)} LIKE ${toSqlDynamic(regex)})"

  // String length → LENGTH()
  case DynamicSchemaExpr.StringLength(string) =>
    s"LENGTH(${toSqlDynamic(string)})"

  case _ => "?"
}
```

The mapping from `DynamicSchemaExpr` to SQL is direct, but that dynamic matching stays inside the interpreter implementation:

| DynamicSchemaExpr Case    | SQL Output                                             |
| ------------------------- | ------------------------------------------------------ |
| `Select(path)`            | Column name from `DynamicOptic`                        |
| `Literal(value, schema)`  | SQL literal (`'text'`, `42`, `TRUE`)                   |
| `Relational(_, _, op)`    | `=`, `<>`, `<`, `>`, `<=`, `>=`                        |
| `Logical(_, _, op)`       | `AND`, `OR`                                            |
| `Not(expr)`               | `NOT (...)`                                            |
| `Arithmetic(_, _, op, _)` | `+`, `-`, `*`                                          |
| `StringConcat`            | `CONCAT(a, b)`                                         |
| `StringRegexMatch`        | `LIKE` (simplified: regex semantics are not preserved) |
| `StringLength`            | `LENGTH(col)`                                          |

## Generating SQL from Queries

Now we can translate any query expression into a SQL WHERE clause. Let's try it with the queries from Part 1:

```scala
val isElectronics = Product.category === "Electronics"
val expensiveItems = Product.price > 100.0
val highRated = Product.rating >= 4
```

Each query translates to a WHERE-clause fragment:

```scala
toSql(isElectronics)
// res3: String = "(category = 'Electronics')"
toSql(expensiveItems)
// res4: String = "(price > 100.0)"
toSql(highRated)
// res5: String = "(rating >= 4)"
```

## Compound Queries

Boolean combinators translate to `AND`, `OR`, and `NOT`:

```scala
val affordableElectronics =
  (Product.category === "Electronics") && (Product.price < 500.0)

val goodDeal =
  (Product.price < 10.0) || (Product.rating >= 5)

val outOfStock = !Product.inStock
```

Each compound query renders with nested parentheses:

```scala
toSql(affordableElectronics)
// res6: String = "((category = 'Electronics') AND (price < 500.0))"
toSql(goodDeal)
// res7: String = "((price < 10.0) OR (rating >= 5))"
toSql(outOfStock)
// res8: String = "NOT (inStock)"
```

Complex nested queries compose naturally:

```scala
val complexQuery =
  ((Product.category === "Electronics") && (Product.price < 500.0)) ||
  ((Product.category === "Office") && (Product.rating >= 4))
```

The nested query renders as one expression:

```scala
toSql(complexQuery)
// res9: String = "(((category = 'Electronics') AND (price < 500.0)) OR ((category = 'Office') AND (rating >= 4)))"
```

## Arithmetic in SQL

Arithmetic expressions translate directly to SQL math:

```scala
val discountedPrice = Product.price * 0.9
val priceWithTax = Product.price * 1.08
```

Arithmetic expressions render as SQL math:

```scala
toSql(discountedPrice)
// res10: String = "(price * 0.9)"
toSql(priceWithTax)
// res11: String = "(price * 1.08)"
```

## String Operations in SQL

String operations map to SQL string functions:

```scala
// Regex match → LIKE
val startsWithL = Product.name.matches("L%")

// Concatenation → CONCAT()
val labeledName = Product.name.concat(" [SALE]")

// String length → LENGTH()
val nameLength = Product.name.length
```

String operations render as SQL functions:

```scala
toSql(startsWithL)
// res12: String = "(name LIKE 'L%')"
toSql(labeledName)
// res13: String = "CONCAT(name, ' [SALE]')"
toSql(nameLength)
// res14: String = "LENGTH(name)"
```

:::tip
The `matches` operator uses regex syntax in the `SchemaExpr` evaluator, but SQL's `LIKE` uses `%` and `_` wildcards. When building queries intended for SQL, use SQL-style patterns (`L%` instead of `L.*`). If you need full regex support, replace the `LIKE` translation with your database's regex function (e.g., `REGEXP` in MySQL, `~` in PostgreSQL).
:::

## Building Complete SELECT Statements

With the `toSql` interpreter, building complete SQL statements is straightforward:

```scala
def select(table: String, predicate: SchemaExpr[?, Boolean]): String =
  s"SELECT * FROM $table WHERE ${toSql(predicate)}"

def selectColumns(table: String, columns: List[String], predicate: SchemaExpr[?, Boolean]): String =
  s"SELECT ${columns.mkString(", ")} FROM $table WHERE ${toSql(predicate)}"

def selectWithLimit(
  table: String,
  predicate: SchemaExpr[?, Boolean],
  orderBy: Option[String] = None,
  limit: Option[Int] = None
): String = {
  val base = s"SELECT * FROM $table WHERE ${toSql(predicate)}"
  val ordered = orderBy.fold(base)(col => s"$base ORDER BY $col")
  limit.fold(ordered)(n => s"$ordered LIMIT $n")
}
```

```scala
val query = (Product.category === "Electronics") && (Product.inStock === true) && (Product.price < 500.0)
```

The statement builders render complete queries:

```scala
select("products", query)
// res15: String = "SELECT * FROM products WHERE (((category = 'Electronics') AND (inStock = TRUE)) AND (price < 500.0))"

selectColumns("products", List("name", "price"), query)
// res16: String = "SELECT name, price FROM products WHERE (((category = 'Electronics') AND (inStock = TRUE)) AND (price < 500.0))"

selectWithLimit("products", query, orderBy = Some("price ASC"), limit = Some(10))
// res17: String = "SELECT * FROM products WHERE (((category = 'Electronics') AND (inStock = TRUE)) AND (price < 500.0)) ORDER BY price ASC LIMIT 10"
```

## Parameterized Queries

The `toSql` function above inlines literal values directly into the SQL string. For production use, you need parameterized queries to prevent SQL injection. We modify the interpreter to collect parameters separately:

```scala
case class SqlQuery(sql: String, params: List[Any])

def toParameterized[A, B](expr: SchemaExpr[A, B]): SqlQuery = toParameterizedDynamic(expr.dynamic)

private def toParameterizedDynamic(expr: DynamicSchemaExpr): SqlQuery = expr match {

  case DynamicSchemaExpr.Select(path) =>
    SqlQuery(columnName(path), Nil)

  case DynamicSchemaExpr.Literal(value, _) =>
    val param = value match {
      case DynamicValue.Primitive(pv) => pv match {
        case PrimitiveValue.String(s)     => s
        case PrimitiveValue.Boolean(b)    => b
        case PrimitiveValue.Int(n)        => n
        case PrimitiveValue.Long(n)       => n
        case PrimitiveValue.Double(n)     => n
        case PrimitiveValue.Float(n)      => n
        case PrimitiveValue.Short(n)      => n
        case PrimitiveValue.Byte(n)       => n
        case PrimitiveValue.BigInt(n)     => n
        case PrimitiveValue.BigDecimal(n) => n
        case PrimitiveValue.Char(c)       => c
        case other                        => other.toString
      }
      case other => other.toString
    }
    SqlQuery("?", List(param))

  case DynamicSchemaExpr.Relational(left, right, op) =>
    val l = toParameterizedDynamic(left); val r = toParameterizedDynamic(right)
    val sqlOp = op match {
      case DynamicSchemaExpr.RelationalOperator.Equal              => "="
      case DynamicSchemaExpr.RelationalOperator.NotEqual           => "<>"
      case DynamicSchemaExpr.RelationalOperator.LessThan           => "<"
      case DynamicSchemaExpr.RelationalOperator.LessThanOrEqual    => "<="
      case DynamicSchemaExpr.RelationalOperator.GreaterThan        => ">"
      case DynamicSchemaExpr.RelationalOperator.GreaterThanOrEqual => ">="
    }
    SqlQuery(s"(${l.sql} $sqlOp ${r.sql})", l.params ++ r.params)

  case DynamicSchemaExpr.Logical(left, right, op) =>
    val l = toParameterizedDynamic(left); val r = toParameterizedDynamic(right)
    val sqlOp = op match {
      case DynamicSchemaExpr.LogicalOperator.And => "AND"
      case DynamicSchemaExpr.LogicalOperator.Or  => "OR"
    }
    SqlQuery(s"(${l.sql} $sqlOp ${r.sql})", l.params ++ r.params)

  case DynamicSchemaExpr.Not(inner) =>
    val i = toParameterizedDynamic(inner)
    SqlQuery(s"NOT (${i.sql})", i.params)

  case DynamicSchemaExpr.Arithmetic(left, right, op, _) =>
    val l = toParameterizedDynamic(left); val r = toParameterizedDynamic(right)
    val sqlOp = op match {
      case DynamicSchemaExpr.ArithmeticOperator.Add      => "+"
      case DynamicSchemaExpr.ArithmeticOperator.Subtract => "-"
      case DynamicSchemaExpr.ArithmeticOperator.Multiply => "*"
      case _                                             => "?"
    }
    SqlQuery(s"(${l.sql} $sqlOp ${r.sql})", l.params ++ r.params)

  case DynamicSchemaExpr.StringConcat(left, right) =>
    val l = toParameterizedDynamic(left); val r = toParameterizedDynamic(right)
    SqlQuery(s"CONCAT(${l.sql}, ${r.sql})", l.params ++ r.params)

  case DynamicSchemaExpr.StringRegexMatch(regex, string) =>
    val s = toParameterizedDynamic(string); val r = toParameterizedDynamic(regex)
    SqlQuery(s"(${s.sql} LIKE ${r.sql})", s.params ++ r.params)

  case DynamicSchemaExpr.StringLength(string) =>
    val s = toParameterizedDynamic(string)
    SqlQuery(s"LENGTH(${s.sql})", s.params)

  case _ => SqlQuery("?", Nil)
}
```

Now literals become `?` placeholders, with the actual values collected in a parameter list:

```scala
val q = (Product.category === "Electronics") && (Product.price < 500.0) && (Product.rating >= 4)
val paramQuery = toParameterized(q)
```

The SQL holds placeholders and the parameter list holds the values:

```scala
paramQuery.sql
// res18: String = "(((category = ?) AND (price < ?)) AND (rating >= ?))"
paramQuery.params
// res19: List[Any] = List("Electronics", 500.0, 4)
```

You can use this with JDBC's `PreparedStatement`:

```scala
val ps = connection.prepareStatement(s"SELECT * FROM products WHERE ${paramQuery.sql}")
paramQuery.params.zipWithIndex.foreach { case (value, idx) =>
  value match {
    case s: String  => ps.setString(idx + 1, s)
    case d: Double  => ps.setDouble(idx + 1, d)
    case i: Int     => ps.setInt(idx + 1, i)
    case b: Boolean => ps.setBoolean(idx + 1, b)
    case l: Long    => ps.setLong(idx + 1, l)
  }
}
val rs = ps.executeQuery()
```

:::warning
Always use parameterized queries for user-supplied values. The inline `toSql` function is suitable for logging and debugging, but use `toParameterized` for actual database execution.
:::

## Nested Structures and Table-Qualified Columns

When domain types have nested structures, optic paths contain multiple `Field` nodes. For SQL, these often map to JOIN-based queries with table-qualified column names.

```scala

case class Address(city: String, country: String)
object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

case class Seller(name: String, address: Address, rating: Double)
object Seller extends CompanionOptics[Seller] {
  implicit val schema: Schema[Seller] = Schema.derived

  val name: Lens[Seller, String]       = optic(_.name)
  val rating: Lens[Seller, Double]     = optic(_.rating)
  val city: Lens[Seller, String]       = optic(_.address.city)
  val country: Lens[Seller, String]    = optic(_.address.country)
}
```

The lens `Seller.city` has the path `[Field("address"), Field("city")]`. We can translate multi-segment paths into table-qualified column names:

```scala
def qualifiedColumnName(optic: zio.blocks.schema.Optic[?, ?]): String = {
  val fields = optic.toDynamic.nodes.collect {
    case f: DynamicOptic.Node.Field => f.name
  }
  // Single field: use as-is. Multiple fields: table.column convention
  if (fields.length <= 1) fields.mkString
  else s"${fields.init.mkString("_")}.${fields.last}"
}
```

The qualified names join nested fields with a dot:

```scala
qualifiedColumnName(Seller.name)
// res21: String = "name"
qualifiedColumnName(Seller.city)
// res22: String = "address.city"
qualifiedColumnName(Seller.country)
// res23: String = "address.country"
```

This produces `address.city` for nested fields, which maps naturally to a SQL JOIN:

```sql
SELECT sellers.*, address.city, address.country
FROM sellers
JOIN addresses AS address ON sellers.id = address.seller_id
WHERE address.city = 'Berlin' AND sellers.rating >= 4.0
```

To generate full JOIN queries, you would extend the interpreter to inspect the optic paths, detect multi-segment paths, and emit appropriate JOIN clauses. The path structure from `DynamicOptic` gives you all the information needed.

## Putting It Together

Here is a complete, self-contained example that defines a domain, builds queries, and generates both inline SQL and parameterized queries:

```scala

// --- Domain ---

case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]      = optic(_.name)
  val price: Lens[Product, Double]     = optic(_.price)
  val category: Lens[Product, String]  = optic(_.category)
  val inStock: Lens[Product, Boolean]  = optic(_.inStock)
  val rating: Lens[Product, Int]       = optic(_.rating)
}

// --- SQL Interpreter ---

def columnName(optic: zio.blocks.schema.Optic[?, ?]): String =
  optic.toDynamic.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")

def columnName(path: DynamicOptic): String =
  path.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")

def sqlLiteralDV(dv: DynamicValue): String = dv match {
  case DynamicValue.Primitive(pv) =>
    pv match {
      case PrimitiveValue.String(s)  => s"'${s.replace("'", "''")}'"
      case PrimitiveValue.Boolean(b) => if (b) "TRUE" else "FALSE"
      case PrimitiveValue.Int(n)     => n.toString
      case PrimitiveValue.Long(n)    => n.toString
      case PrimitiveValue.Double(n)  => n.toString
      case PrimitiveValue.Float(n)   => n.toString
      case PrimitiveValue.Short(n)   => n.toString
      case PrimitiveValue.Byte(n)    => n.toString
      case other                     => other.toString
    }
  case other => other.toString
}

def toSql[A, B](expr: SchemaExpr[A, B]): String = toSqlDynamic(expr.dynamic)

private def toSqlDynamic(expr: DynamicSchemaExpr): String = expr match {
  case DynamicSchemaExpr.Select(path)              => columnName(path)
  case DynamicSchemaExpr.Literal(value, _)         => sqlLiteralDV(value)
  case DynamicSchemaExpr.Relational(left, right, op) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.RelationalOperator.Equal              => "="
      case DynamicSchemaExpr.RelationalOperator.NotEqual           => "<>"
      case DynamicSchemaExpr.RelationalOperator.LessThan           => "<"
      case DynamicSchemaExpr.RelationalOperator.LessThanOrEqual    => "<="
      case DynamicSchemaExpr.RelationalOperator.GreaterThan        => ">"
      case DynamicSchemaExpr.RelationalOperator.GreaterThanOrEqual => ">="
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.Logical(left, right, op) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.LogicalOperator.And => "AND"
      case DynamicSchemaExpr.LogicalOperator.Or  => "OR"
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.Not(inner)                      => s"NOT (${toSqlDynamic(inner)})"
  case DynamicSchemaExpr.Arithmetic(left, right, op, _) =>
    val sqlOp = op match {
      case DynamicSchemaExpr.ArithmeticOperator.Add      => "+"
      case DynamicSchemaExpr.ArithmeticOperator.Subtract => "-"
      case DynamicSchemaExpr.ArithmeticOperator.Multiply => "*"
      case _                                             => "?"
    }
    s"(${toSqlDynamic(left)} $sqlOp ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.StringConcat(left, right)       => s"CONCAT(${toSqlDynamic(left)}, ${toSqlDynamic(right)})"
  case DynamicSchemaExpr.StringRegexMatch(regex, string) => s"(${toSqlDynamic(string)} LIKE ${toSqlDynamic(regex)})"
  case DynamicSchemaExpr.StringLength(string)            => s"LENGTH(${toSqlDynamic(string)})"
  case _                                                 => "?"
}

// --- Parameterized queries ---

case class SqlQuery(sql: String, params: List[Any])

def toParameterized[A, B](expr: SchemaExpr[A, B]): SqlQuery = toParameterizedDynamic(expr.dynamic)

private def toParameterizedDynamic(expr: DynamicSchemaExpr): SqlQuery = expr match {
  case DynamicSchemaExpr.Select(path)   => SqlQuery(columnName(path), Nil)
  case DynamicSchemaExpr.Literal(value, _) =>
    val param = value match {
      case DynamicValue.Primitive(pv) => pv match {
        case PrimitiveValue.String(s)     => s
        case PrimitiveValue.Boolean(b)    => b
        case PrimitiveValue.Int(n)        => n
        case PrimitiveValue.Long(n)       => n
        case PrimitiveValue.Double(n)     => n
        case PrimitiveValue.Float(n)      => n
        case PrimitiveValue.Short(n)      => n
        case PrimitiveValue.Byte(n)       => n
        case PrimitiveValue.BigInt(n)     => n
        case PrimitiveValue.BigDecimal(n) => n
        case PrimitiveValue.Char(c)       => c
        case other                        => other.toString
      }
      case other => other.toString
    }
    SqlQuery("?", List(param))
  case DynamicSchemaExpr.Relational(left, right, op) =>
    val l = toParameterizedDynamic(left); val r = toParameterizedDynamic(right)
    val sqlOp = op match {
      case DynamicSchemaExpr.RelationalOperator.Equal              => "="
      case DynamicSchemaExpr.RelationalOperator.NotEqual           => "<>"
      case DynamicSchemaExpr.RelationalOperator.LessThan           => "<"
      case DynamicSchemaExpr.RelationalOperator.LessThanOrEqual    => "<="
      case DynamicSchemaExpr.RelationalOperator.GreaterThan        => ">"
      case DynamicSchemaExpr.RelationalOperator.GreaterThanOrEqual => ">="
    }
    SqlQuery(s"(${l.sql} $sqlOp ${r.sql})", l.params ++ r.params)
  case DynamicSchemaExpr.Logical(left, right, op) =>
    val l = toParameterizedDynamic(left); val r = toParameterizedDynamic(right)
    val sqlOp = op match {
      case DynamicSchemaExpr.LogicalOperator.And => "AND"
      case DynamicSchemaExpr.LogicalOperator.Or  => "OR"
    }
    SqlQuery(s"(${l.sql} $sqlOp ${r.sql})", l.params ++ r.params)
  case DynamicSchemaExpr.Not(inner) =>
    val i = toParameterizedDynamic(inner)
    SqlQuery(s"NOT (${i.sql})", i.params)
  case DynamicSchemaExpr.Arithmetic(left, right, op, _) =>
    val l = toParameterizedDynamic(left); val r = toParameterizedDynamic(right)
    val sqlOp = op match {
      case DynamicSchemaExpr.ArithmeticOperator.Add      => "+"
      case DynamicSchemaExpr.ArithmeticOperator.Subtract => "-"
      case DynamicSchemaExpr.ArithmeticOperator.Multiply => "*"
      case _                                             => "?"
    }
    SqlQuery(s"(${l.sql} $sqlOp ${r.sql})", l.params ++ r.params)
  case DynamicSchemaExpr.StringConcat(left, right) =>
    val l = toParameterizedDynamic(left); val r = toParameterizedDynamic(right)
    SqlQuery(s"CONCAT(${l.sql}, ${r.sql})", l.params ++ r.params)
  case DynamicSchemaExpr.StringRegexMatch(regex, string) =>
    val s = toParameterizedDynamic(string); val r = toParameterizedDynamic(regex)
    SqlQuery(s"(${s.sql} LIKE ${r.sql})", s.params ++ r.params)
  case DynamicSchemaExpr.StringLength(string) =>
    val s = toParameterizedDynamic(string)
    SqlQuery(s"LENGTH(${s.sql})", s.params)
  case _ => SqlQuery("?", Nil)
}

// --- Complete SELECT builder ---

def select(table: String, predicate: SchemaExpr[?, Boolean]): String =
  s"SELECT * FROM $table WHERE ${toSql(predicate)}"

// --- Usage ---

val query =
  (Product.category === "Electronics") &&
  (Product.inStock === true) &&
  (Product.price < 500.0) &&
  (Product.rating >= 4)

// Inline SQL for debugging
println(select("products", query))
// SELECT * FROM products WHERE ((((category = 'Electronics') AND (inStock = TRUE)) AND (price < 500.0)) AND (rating >= 4))

// Parameterized SQL for execution
val pq = toParameterized(query)
println(s"SQL:    ${pq.sql}")
println(s"Params: ${pq.params}")
// SQL:    ((((category = ?) AND (inStock = ?)) AND (price < ?)) AND (rating >= ?))
// Params: List(Electronics, true, 500.0, 4)

// String operations in SQL
println(toSql(Product.name.matches("L%")))
// (name LIKE 'L%')

// Arithmetic in SQL
println(toSql(Product.price * 0.9))
// (price * 0.9)
```

## Upsert (ON CONFLICT)

Upsert (insert-or-update) combines an `INSERT` with a conflict handler so the
statement is idempotent. When a row with the conflicting key already exists, the
database either skips the insert or updates specified columns.

All identifiers (table name, column names, conflict column) are validated through
`SqlIdentifier.validate` and assignment columns are additionally checked against
`Table.columns`. Invalid or unknown names throw `IllegalArgumentException` at
build time, not at execution time.

For a task-oriented walkthrough of the table-aware builders (`Upsert.insertDoNothing`,
`Upsert.insertDoUpdate` with and without `updateColumns`) and the `Repo#insertOrUpdate`
and `Repo#insertOrUpdateBatch` wrappers, run against a real SQLite database, see
[How to Upsert Rows and Paginate with Keyset Cursors](./upsert-and-paginate-with-keyset-cursors.md).
The builders below take column names and `DbValue`s directly, without a `Table`.

### Low-Level Builders

When you need full control over column names and values (e.g. computed or
transformed data), use the low-level `Upsert.doNothing`, `Upsert.doNothingRaw`,
and `Upsert.doUpdate` builders:

```scala

// Low-level DO NOTHING with explicit columns and values
val frag1: Frag = Upsert.doNothing(
  tableName   = "users",
  columns     = IndexedSeq("id", "name", "email"),
  values      = IndexedSeq(DbValue.DbInt(1), DbValue.DbString("Bob"), DbValue.DbString("bob@example.com")),
  conflictColumn = "id"
)

// Comma-joined column string variant
val frag2: Frag = Upsert.doNothingRaw(
  tableName   = "users",
  allColumns  = "id, name, email",
  values      = IndexedSeq(DbValue.DbInt(1), DbValue.DbString("Bob"), DbValue.DbString("bob@example.com")),
  conflictColumn = "id"
)

// Low-level DO UPDATE with explicit assignments
val frag3: Frag = Upsert.doUpdate(
  tableName      = "users",
  columns        = IndexedSeq("id", "name", "email"),
  values         = IndexedSeq(DbValue.DbInt(1), DbValue.DbString("Bob"), DbValue.DbString("bob@example.com")),
  conflictColumn = "id",
  assignments    = IndexedSeq("name" -> DbValue.DbString("Bob"), "email" -> DbValue.DbString("bob@example.com"))
)
```

### Suffix Builders

To append an `ON CONFLICT` clause to an existing `INSERT` `Frag`, use the suffix
builders:

```scala

val base: Frag = Frag.literal("INSERT INTO users (id, name) VALUES (?, ?)")

// Append DO NOTHING suffix
val withNothing: Frag = base ++ Upsert.doNothingSuffix(conflictColumn = "id")

// Append DO UPDATE suffix with explicit assignments
val withUpdate: Frag = base ++ Upsert.doUpdateSuffix(
  conflictColumn = "id",
  assignments    = IndexedSeq("name" -> DbValue.DbString("updated"))
)
```

## Keyset Pagination

Keyset (cursor) pagination with `Repo#pageAfter` and `Frag.keysetAfter`, including how
to page through a filtered query and why it beats `OFFSET`, is covered in
[How to Upsert Rows and Paginate with Keyset Cursors](./upsert-and-paginate-with-keyset-cursors.md).

## Inspecting SQL

The custom interpreter above produces raw strings useful for debugging. The `sql` module's query IR (`zio.blocks.sql.query.SqlQuery`) provides richer inspection APIs.

### explain(dialect): String

The `explain` method renders the full SQL text with numbered parameter placeholders (`?1`, `?2`, ...) and a trailing comment listing each parameter's position and type:

```scala

val userTable = Table.derived[User]
val repoTable = Table.derived[Repo]

val q = Select
  .from(userTable)
  .innerJoin(Rel(repoTable, "owner_id", userTable, "id"))
  .filter(Frag(IndexedSeq("""t0."name" = """, ""), IndexedSeq(DbValue.DbString("alice"))))

println(q.explain(SqlDialect.PostgreSQL))
// SELECT t0."id", t0."name", t1."id", t1."owner_id", t1."name" FROM "user" AS t0 INNER JOIN "repo" AS t1 ON t1."owner_id" = t0."id" WHERE t0."name" = ?1
// -- params: 1:String
```

`explain` renders a single-line SQL string with numbered `?N` placeholders. The `?N`
placeholders correspond one-to-one with the parameter list you can obtain separately via
`toFrag(dialect).params`. This makes `explain` useful for logging and visual debugging without touching a
database.

### Inspecting the Query IR

The query itself is a case class, so every part is available programmatically:

```scala
q.source       // Table[User] for "user"
q.joins        // Vector(JoinNode(repoTable, "t1", Inner, <on-frag>))
q.filters      // Vector(Frag(t0."name" = ?, DbString(alice)))
q.groupBy      // Vector.empty
q.orderBy      // Vector.empty
q.limit        // None
q.offset       // None
q.toFrag       // Frag (re-renderable to SQL via frag.sql(dialect))
```

Inspecting the IR directly lets you examine joins, filters, ordering, and limits programmatically. This is useful for building monitoring dashboards, query analyzers, or dynamic query modification layers.

### sql(dialect): String

The query IR also provides a simpler `SqlQuery#sql` method that renders `?` placeholders for execution:

```scala
println(q.sql(SqlDialect.PostgreSQL))
// SELECT t0."id", t0."name", t1."id", t1."owner_id", t1."name" FROM "user" AS t0 INNER JOIN "repo" AS t1 ON t1."owner_id" = t0."id" WHERE t0."name" = ?
```

### previewSql() for Migrations

When using `SmallMigrator` or `LargeMigrator` from the `data-migration` module, `previewSql()` returns the full sequence of SQL statements the migrator would execute, without opening any database connection:

```scala

given transactor: Transactor = tx

val migrator = SmallMigrator(
  repoV1 = userRepo,
  repoV2 = userRepoV2,
  migration = userMigration,
  queueTable = "migration_queue",
  batchSize = 100,
  target = TargetStrategy.InPlace
)

val statements: Vector[String] = migrator.previewSql()
// Vector(
//   "CREATE TABLE ...",     -- queue DDL
//   "CREATE TABLE ...",     -- shadow table
//   "CREATE TRIGGER ...",   -- capture triggers
//   "SELECT ...",           -- dequeue template
//   "ALTER TABLE ..."       -- finalize (rename)
// )
```

This gives you a dry run of the migration SQL before any schema changes are applied.

## Compile-Time SQL Dumps

To run joins against a real database and dump their SQL, see [How to Write Joins and Inspect the Generated SQL](./write-joins-and-inspect-generated-sql.md).

The `Dump` object emits SQL files at compile time. When the JVM property `zib.sql.dumpDir` is set, inline macro calls to `Dump.dumpTable` or `Dump.dumpQuery` write `.sql` files to that directory. When the property is absent, the calls become no-ops with zero runtime cost.

### Enabling Dumps

The `Dump` macros read `System.getProperty("zib.sql.dumpDir")` at compile time from the JVM running sbt's compiler. This is a JVM system property, not a scalac flag. Passing it via `scalacOptions` does nothing.

The simplest way to set it is as a JVM flag on the sbt command line:

```bash
sbt -Dzib.sql.dumpDir=target/sql-dumps "++3.8.3; sqlJVM/compile"
```

If you prefer not to type `-D` every time, export it through `SBT_OPTS`:

```bash
SBT_OPTS="-Dzib.sql.dumpDir=target/sql-dumps" sbt compile
```

Both approaches set the property on the sbt process, which is the same JVM that runs the Scala compiler and the macros within it.

### Entry Points

There are two inline macro entry points, all in `zio.blocks.sql.Dump`:

```scala

// Dump a Table's CREATE TABLE DDL (both PostgreSQL and SQLite)
Dump.dumpTable(userTable)

// Dump a query IR's SELECT
Dump.dumpQuery(queryIr)
```

Each call emits one file per dialect (PostgreSQL and SQLite by default).

### Naming Scheme

Files are named `<owner>-<dialect>.sql` where `<owner>` is derived from the enclosing symbol and `<dialect>` is the lowercased dialect name:

```
target/sql-dumps/
  user-postgresql.sql
  user-sqlite.sql
  repo-postgresql.sql
  repo-sqlite.sql
```

For `Dump.dumpTable`, the owner comes from the `Table`'s type name. For `Dump.dumpQuery`, the macro walks the call site to extract the enclosing method, val name, or argument name. If the macro cannot determine a meaningful name, it falls back to `query`.

### Content-Hash Skip

Each dump file is written only when its content differs from the existing file. The macro compares the new bytes against any existing file at the target path. If they match, the write is skipped. This means incremental compilations do not produce noisy diffs or unnecessary filesystem writes.

All files use UTF-8 encoding with a trailing newline.

### Limitations

Compile-time dumps work well for statically constructed queries, but some SQL patterns cannot be dumped:

- **Dynamic `Frag` chains.** Fragments built at runtime from user input, database lookups, or conditional branching are invisible to the macro. Only the static structure known at compile time appears in the dump.
- **Repo internals.** The `Repo` abstraction's generated queries (insert, update, delete, select-by-id) are assembled at runtime from the `DbCodec` and `Table` metadata. `Dump.dumpTable` captures the DDL, but the CRUD queries themselves are not emitted.
- **Phase 2 note.** A future phase may extend `Dump` to cover `Repo`-level CRUD operations and dynamic fragment composition. For now, treat the dump as a DDL and static-query snapshot, not a complete representation of every SQL statement your application will execute.

:::tip
Pair `Dump.dumpTable` with `SmallMigrator#previewSql` for a fuller picture: `Dump.dumpTable` captures the schema DDL at compile time, while `SmallMigrator#previewSql` captures the migration SQL at runtime before execution.
:::

## Going Further

- **[Part 1: Expressions](./query-dsl-reified-optics.md)** -- Building query expressions with reified optics
- **[Part 3: Extending the Expression Language](./query-dsl-extending.md)** -- Adding custom operators (IN, BETWEEN, aggregates) beyond SchemaExpr
- **[Part 4: A Fluent SQL Builder](./query-dsl-fluent-builder.md)** -- Type-safe SELECT, UPDATE, INSERT, DELETE with seamless condition mixing
- **[SchemaExpr Reference](../reference/schema/schema-expr.md)** -- Full API coverage of expression types
- **[Optics Reference](../reference/schema/optics.md)** -- Lens, Prism, Optional, and Traversal
- **[DynamicOptic Reference](../reference/schema/dynamic-optic.md)** -- Runtime optic paths for programmatic field extraction
- **[How to Upsert Rows and Paginate with Keyset Cursors](./upsert-and-paginate-with-keyset-cursors.md)** -- Insert-or-update rows and page through large tables with keyset cursors

The interpreter pattern shown here extends naturally to other query targets. Because `SchemaExpr` wraps a `DynamicSchemaExpr` sealed trait and `DynamicOptic` carries full path metadata, you can write interpreters for MongoDB filters, Elasticsearch queries, GraphQL filters, or any other query language using the same approach: access `SchemaExpr#dynamic`, pattern match on the AST, map operators, and extract field names from optic paths.

---

## How to Read and Write CSV and TSV Files

In this guide, we will read a bookshop's inventory file into Scala records and write it back out, using the CSV module of ZIO Blocks. By the end, you will have a handful of small functions that read a CSV or TSV file by column name, list every bad row with its row number, write records with correct quoting, and process a file of any size one row at a time.

We start with a three-line CSV file and grow the code step by step:

- Read a CSV file with a header into records.
- Report every bad row instead of stopping at the first one.
- Write records back out as CSV.
- Read and write TSV, which needs a different delimiter.
- Process a large file without loading it into memory.
- Turn records with lists and optional fields into flat rows.

## The Problem

A CSV file looks easy to read: split each line at the commas, then convert each cell. That works until a cell contains a comma. The authors of a book are a good example, because a CSV writer wraps such a cell in double quotes:

```scala
val inventoryLine = "Programming in Scala,2008,\"Martin Odersky, Lex Spoon, Bill Venners\",7"
// inventoryLine: String = "Programming in Scala,2008,\"Martin Odersky, Lex Spoon, Bill Venners\",7"

println(inventoryLine.split(",").toList)
// List(Programming in Scala, 2008, "Martin Odersky,  Lex Spoon,  Bill Venners", 7)
```

The code above splits a line with four columns into six cells, and the quote characters end up inside the cells. Converting the fourth cell to the number of copies now throws a `NumberFormatException`. A correct reader also has to handle doubled quotes (`""`) inside a quoted cell, line breaks inside a quoted cell, files whose columns come in a different order, and files that use tabs instead of commas. Hand-written parsers rarely handle all of these, and they fail on real exports from spreadsheets and databases.

In this guide, we solve this with `zio-blocks-schema-csv`: `CsvReader` and `CsvWriter` split and join rows by the rules of RFC 4180, and a codec derived from a `Schema` converts each row to and from a case class.

## Prerequisites

To follow along, add the schema and CSV modules to your build:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-schema"     % "0.0.56",
  "dev.zio" %% "zio-blocks-schema-csv" % "0.0.56"
)

// Scala 3.9 and newer (also needs scalacOptions += "-experimental"): the same modules with the zio-blocks-next- prefix
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-schema"     % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema-csv" % "0.0.56"
)
```

This guide assumes you can derive a `Schema` for a case class, as shown in [Your First Schema](./your-first-schema.md). The CSV section of [One Type, Many Formats](./one-type-many-formats.md#10-flatten-the-catalog-for-csv) shows how a CSV codec encodes and decodes a single row; here we build whole files on top of it.

## The Inventory Row

Each row of the inventory file describes one book: its title, its authors, the year it was published and the number of copies in stock. We model one row as a flat case class and derive its CSV codec with `Schema#derive(CsvFormat)`:

```scala

case class BookRow(title: String, authors: String, year: Int, copies: Int)

object BookRow {
  implicit val schema: Schema[BookRow] = Schema.derived
}

val bookCodec: CsvCodec[BookRow] = BookRow.schema.derive(CsvFormat)
```

The codec names one column for each field, in declaration order. `CsvCodec#headerNames` returns these names:

```scala
println(bookCodec.headerNames)
// Vector(title, authors, year, copies)
```

The `authors` field holds all the authors of a book in one string, such as `"Martin Odersky, Lex Spoon, Bill Venners"`. A CSV cell holds one value, so a list of authors has to become a single string; we come back to this in [Flatten Lists and Optional Fields](#flatten-lists-and-optional-fields).

## Read a CSV File With a Header

Our sample file lists its columns in a different order from `BookRow`: the year comes before the authors. Files exported by other tools often do this, so we match columns by their header name, not by position. First, we write the file to a temporary path:

```scala
val inventoryCsv: String =
  """title,year,authors,copies
    |Zionomicon,2024,"John A. De Goes, Adam Fraser and Milad Khajavi",12
    |Programming in Scala,2008,"Martin Odersky, Lex Spoon, Bill Venners",7
    |Functional Programming in Scala,2014,"Paul Chiusano, Rúnar Bjarnason",5
    |""".stripMargin

val inventoryPath: Path = Files.createTempFile("inventory", ".csv")
Files.writeString(inventoryPath, inventoryCsv, UTF_8)
```

`CsvReader.readAll` splits a whole document into a header and a list of rows, where each row is a list of cell strings. It takes the text and a `CsvConfig`; `CsvConfig.default` means commas between cells and double quotes around cells that need them:

```scala
val inventory = CsvReader.readAll(Files.readString(inventoryPath, UTF_8), CsvConfig.default)
// inventory: Either[CsvError, Tuple2[IndexedSeq[String], IndexedSeq[IndexedSeq[String]]]] = Right(
//   (
//     Vector("title", "year", "authors", "copies"),
//     Vector(
//       Vector(
//         "Zionomicon",
//         "2024",
//         "John A. De Goes, Adam Fraser and Milad Khajavi",
//         "12"
//       ),
//       Vector(
//         "Programming in Scala",
//         "2008",
//         "Martin Odersky, Lex Spoon, Bill Venners",
//         "7"
//       ),
//       Vector(
//         "Functional Programming in Scala",
//         "2014",
//         "Paul Chiusano, Rúnar Bjarnason",
//         "5"
//       )
//     )
//   )
// )
```

The quoted authors are now single cells, without their quotes. Next, we turn each row of cells into a `BookRow`. The derived codec decodes one CSV line whose cells are in `CsvCodec#headerNames` order, so we pick the cells in that order, join them back into a line with `CsvWriter.writeRow`, and decode the line. A small `RowDecoder` does this, and its `RowDecoder.fromHeader` constructor finds the position of each column in the file's header:

```scala
final class RowDecoder[A](codec: CsvCodec[A], width: Int, positions: IndexedSeq[Int]) {
  def decode(cells: IndexedSeq[String]): Either[String, A] =
    if (cells.length != width) Left(s"expected $width cells, got ${cells.length}")
    else {
      val line = CsvWriter.writeRow(positions.map(cells), CsvConfig.default)
      codec.decode(CharBuffer.wrap(line)).left.map(_.getMessage)
    }
}

object RowDecoder {
  def fromHeader[A](codec: CsvCodec[A], header: IndexedSeq[String]): Either[String, RowDecoder[A]] = {
    val missing = codec.headerNames.filterNot(header.contains)
    if (missing.nonEmpty) Left(s"missing column(s): ${missing.mkString(", ")}")
    else Right(new RowDecoder(codec, header.length, codec.headerNames.map(name => header.indexOf(name))))
  }
}
```

In the code above, `positions` holds, for each column of `BookRow`, its index in the file. `RowDecoder#decode` first checks that the row has as many cells as the header, because a short row would otherwise put cells in the wrong columns. `CsvWriter.writeRow` quotes any cell that needs it, so a cell such as `John A. De Goes, Adam Fraser and Milad Khajavi` survives the trip back into a line. With the decoder in place, we decode every row of the inventory:

```scala
inventory.foreach { case (header, rows) =>
  RowDecoder.fromHeader(bookCodec, header) match {
    case Left(error)    => println(error)
    case Right(decoder) => rows.map(decoder.decode).foreach(println)
  }
}
// Right(BookRow(Zionomicon,John A. De Goes, Adam Fraser and Milad Khajavi,2024,12))
// Right(BookRow(Programming in Scala,Martin Odersky, Lex Spoon, Bill Venners,2008,7))
// Right(BookRow(Functional Programming in Scala,Paul Chiusano, Rúnar Bjarnason,2014,5))
```

Each row decodes to a `BookRow`, with the year and the number of copies parsed as `Int` values.

## Report Every Bad Row

Real files contain mistakes, and a reader that stops at the first one makes the user fix a file one error at a time. We decode every row, keep the good records, and collect a message with the row number for each bad one. The header is row 1, so the first data row is row 2:

```scala
final case class Report[A](records: IndexedSeq[A], errors: IndexedSeq[String])

def readCsv[A](codec: CsvCodec[A], text: String, config: CsvConfig): Either[String, Report[A]] =
  CsvReader.readAll(text, config) match {
    case Left(error)           => Left(error.getMessage)
    case Right((header, rows)) =>
      RowDecoder.fromHeader(codec, header).map { decoder =>
        val results = rows.zipWithIndex.map { case (cells, index) =>
          decoder.decode(cells).left.map(error => s"row ${index + 2}: $error")
        }
        Report(results.collect { case Right(a) => a }, results.collect { case Left(e) => e })
      }
  }
```

The following file has two mistakes: a year written as a Roman numeral, and a row that lost its last cell:

```scala
val damagedCsv: String =
  """title,year,authors,copies
    |Zionomicon,2024,"John A. De Goes, Adam Fraser and Milad Khajavi",12
    |Programming in Scala,MMVIII,"Martin Odersky, Lex Spoon, Bill Venners",7
    |Functional Programming in Scala,2014,"Paul Chiusano, Rúnar Bjarnason"
    |""".stripMargin
```

Reading it returns the one good record and both errors:

```scala
readCsv(bookCodec, damagedCsv, CsvConfig.default).foreach { report =>
  report.records.foreach(println)
  report.errors.foreach(println)
}
// BookRow(Zionomicon,John A. De Goes, Adam Fraser and Milad Khajavi,2024,12)
// row 3: Cannot parse 'MMVIII' as Int: For input string: "MMVIII" at: .
// row 4: expected 4 cells, got 3
```

The message for row 3 comes from the derived codec. It quotes the bad value, but it doesn't name the column: the trailing `at: .` is the position inside the record, which the CSV codec leaves empty. The cell count message for row 4 comes from our `RowDecoder`. A file that lacks a whole column fails before any row is decoded:

```scala
println(readCsv(bookCodec, "title,year,authors\nZionomicon,2024,John A. De Goes\n", CsvConfig.default))
// Left(missing column(s): copies)
```

A quote that is opened but never closed is a different kind of error. Everything after the opening quote, up to the end of the file, belongs to one cell, so no later row can be trusted. `CsvReader.readAll` returns a `CsvError.ParseError` for the whole document. In the following file, the authors cell on the last line has no closing quote:

```scala
val unclosedCsv: String =
  """title,year,authors,copies
    |Zionomicon,2024,"John A. De Goes, Adam Fraser and Milad Khajavi",12
    |Programming in Scala,2008,"Martin Odersky, Lex Spoon, Bill Venners,7
    |""".stripMargin
```

Reading it returns the parse error and no records:

```scala
println(readCsv(bookCodec, unclosedCsv, CsvConfig.default))
// Left(CSV error at row 2, column 1: Unclosed quoted field)
```

:::warning
The row and column in a `CsvError.ParseError` count from the start of the row that failed, not from the start of the file. Above, the unclosed quote is on line 3 of the file, but the error says `row 2, column 1`: the count starts again at 1 for the row being parsed, and the open cell runs past that row's line break to the end of the file. To report file line numbers, count the lines yourself, as the reader in [Process a Large File Row by Row](#process-a-large-file-row-by-row) does.
:::

## Write Records as CSV

Writing is the reverse: the derived codec encodes one record as one CSV line, ending with `\r\n`. `CsvCodec#encode` writes into a `java.nio.CharBuffer`, so we allocate a buffer, encode, and read the line back out:

```scala
def encodeLine[A](codec: CsvCodec[A], value: A): String = {
  val buffer = CharBuffer.allocate(4096)
  codec.encode(value, buffer)
  buffer.flip()
  buffer.toString
}

def writeCsv[A](codec: CsvCodec[A], values: Iterable[A]): String =
  CsvWriter.writeHeader(codec.headerNames, CsvConfig.default) + values.map(value => encodeLine(codec, value)).mkString
```

The buffer must be large enough for one line; `CharBuffer#put` throws a `java.nio.BufferOverflowException` when a line doesn't fit. `CsvWriter.writeHeader` writes the column names as the first line, quoted by the same rules as data. Here is the inventory after a delivery of new copies, written as CSV text:

```scala
val restocked: List[BookRow] = List(
  BookRow("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", 2024, 20),
  BookRow("Programming in Scala", "Martin Odersky, Lex Spoon, Bill Venners", 2008, 7),
  BookRow("Functional Programming in Scala", "Paul Chiusano, Rúnar Bjarnason", 2014, 9)
)

val restockedCsv: String = writeCsv(bookCodec, restocked)
```

Printing the text line by line shows the CSV file:

```scala
restockedCsv.linesIterator.foreach(println)
// title,authors,year,copies
// Zionomicon,"John A. De Goes, Adam Fraser and Milad Khajavi",2024,20
// Programming in Scala,"Martin Odersky, Lex Spoon, Bill Venners",2008,7
// Functional Programming in Scala,"Paul Chiusano, Rúnar Bjarnason",2014,9
```

The writer quotes only the cells that need it: the authors contain commas, so they are wrapped in double quotes, and the titles are not. A cell that contains a double quote is quoted too, and each quote inside it is doubled:

```scala
println(CsvWriter.writeRow(IndexedSeq("Zionomicon", "the \"ZIO\" book"), CsvConfig.default).stripLineEnd)
// Zionomicon,"the ""ZIO"" book"
```

Finally, we save the file:

```scala
val restockedPath: Path = Files.createTempFile("restocked", ".csv")
Files.writeString(restockedPath, restockedCsv, UTF_8)
```

Then we read it back with `readCsv` from the previous section, to check that every record survives the round trip:

```scala
readCsv(bookCodec, Files.readString(restockedPath, UTF_8), CsvConfig.default).map(_.records == restocked)
// res10: Either[String, Boolean] = Right(true)
```

## Read and Write TSV

A tab-separated file (TSV) is a CSV file with a tab between the cells. `CsvConfig.tsv` is the preset for it, and `CsvReader` and `CsvWriter` accept it in place of `CsvConfig.default`. The derived codec does not: it always reads and writes commas, whatever `CsvConfig` the rest of your code uses. A TSV line given straight to the codec is split at its commas, not at its tabs:

```scala
println(bookCodec.decode(CharBuffer.wrap("Zionomicon\tJohn A. De Goes, Adam Fraser and Milad Khajavi\t2024\t12")))
// Left(zio.blocks.schema.SchemaError: Expected 4 fields, got 2 at: .)
```

Our `readCsv` already works for TSV, because it splits rows with `CsvReader` and the configuration we pass, and only hands the codec lines that `CsvWriter.writeRow` joined with commas. For writing, we turn each record into a list of cells: we encode it with the codec and split the CSV line again with `CsvReader.readRow`. `CsvWriter.writeAll` then joins the header and the rows with any delimiter:

```scala
def encodeCells[A](codec: CsvCodec[A], value: A): IndexedSeq[String] =
  CsvReader.readRow(encodeLine(codec, value), 0, CsvConfig.default) match {
    case Right((cells, _)) => cells
    case Left(error)       => throw error
  }

def writeDelimited[A](codec: CsvCodec[A], values: Iterable[A], config: CsvConfig): String =
  CsvWriter.writeAll(codec.headerNames, values.map(value => encodeCells(codec, value)), config)

val restockedTsv: String = writeDelimited(bookCodec, restocked, CsvConfig.tsv)
```

`CsvReader.readRow` parses one row starting at a character offset (here 0) and returns the cells and the offset after the row. Since the codec wrote the line, it always parses, and the `throw` never runs. Printing the TSV text shows tabs between the cells:

```scala
restockedTsv.linesIterator.foreach(println)
// title	authors	year	copies
// Zionomicon	John A. De Goes, Adam Fraser and Milad Khajavi	2024	20
// Programming in Scala	Martin Odersky, Lex Spoon, Bill Venners	2008	7
// Functional Programming in Scala	Paul Chiusano, Rúnar Bjarnason	2014	9
```

The authors are no longer quoted, because a comma is an ordinary character in a TSV file; only a cell that contains a tab, a quote or a line break is quoted. Reading the TSV text back with `CsvConfig.tsv` returns the same records:

```scala
readCsv(bookCodec, restockedTsv, CsvConfig.tsv).map(_.records == restocked)
// res13: Either[String, Boolean] = Right(true)
```

The same two functions work for any delimiter, such as `CsvConfig(delimiter = ';')` for the semicolon-separated files that spreadsheets write in many European locales.

## Process a Large File Row by Row

`CsvReader.readAll` needs the whole file in one `String`, and returns all rows at once. For a file of several gigabytes, we read one record at a time instead and decode each one as it arrives. The module has no file or stream support of its own, so we read lines with a `java.io.BufferedReader` and give each record to `CsvReader.readRow`.

There is one catch: a quoted cell may contain a line break, so one record can span several lines. RFC 4180 doubles every quote inside a quoted cell, so a record is complete when it contains an even number of quote characters. The following iterator joins lines until that is true, and remembers the line number where each record starts:

```scala

def csvRecords(reader: BufferedReader, quoteChar: Char): Iterator[(Int, String)] = {
  val lines = Iterator.continually(reader.readLine()).takeWhile(_ != null).zip(Iterator.from(1))

  @tailrec
  def complete(record: String): String =
    if (record.count(_ == quoteChar) % 2 == 0 || !lines.hasNext) record
    else complete(record + "\n" + lines.next()._1)

  lines.map { case (firstLine, lineNumber) => (lineNumber, complete(firstLine)) }
}
```

`csvRecords` pairs each line with its number, then maps every line that starts a record to the complete record. `complete` takes the following lines from the same `lines` iterator, so a line that continues a record is never seen as the start of a new one.

The parity rule relies on the file following RFC 4180. A file with a stray quote in the middle of an unquoted cell, such as `5" floppy`, breaks it; `CsvReader` accepts such a cell as text, but the iterator then joins lines until it finds another quote. `BufferedReader#readLine` also drops the line ending, so a line break inside a cell comes back as `\n` even if the file has `\r\n`.

Next, `withCsvFile` opens a file, reads its header, and hands the caller an iterator of decoded rows, each with its line number. The file stays open only while the caller's function runs, so the caller must consume the iterator inside it:

```scala
def withCsvFile[A, B](path: Path, codec: CsvCodec[A], config: CsvConfig)(
  use: Iterator[(Int, Either[String, A])] => B
): Either[String, B] =
  Using.resource(Files.newBufferedReader(path, UTF_8)) { reader =>
    val records = csvRecords(reader, config.quoteChar)
    if (!records.hasNext) Left("the file is empty")
    else {
      val (_, headerText) = records.next()
      CsvReader.readRow(headerText, 0, config).left.map(_.getMessage).flatMap { case (header, _) =>
        RowDecoder.fromHeader(codec, header).map { decoder =>
          use(records.filter(_._2.nonEmpty).map { case (line, text) =>
            val decoded = CsvReader.readRow(text, 0, config) match {
              case Left(error)       => Left(s"line ${line + error.row - 1}, column ${error.column}: ${error.message}")
              case Right((cells, _)) => decoder.decode(cells).left.map(error => s"line $line: $error")
            }
            (line, decoded)
          })
        }
      }
    }
  }
```

Because each record is parsed on its own, a parse error such as a stray character after a closing quote affects only that record. `CsvError#row` counts lines from the start of the record, so `line + error.row - 1` is the line in the file, and `CsvError#column` is the column within that line. Blank lines are skipped. To try it, we write a small file with a cell that spans two lines, a bad year and a stray character after a closing quote:

```scala
val awkwardPath: Path = Files.createTempFile("awkward", ".csv")
Files.writeString(
  awkwardPath,
  """title,year,authors,copies
    |Zionomicon,2024,"John A. De Goes,
    |Adam Fraser and Milad Khajavi",12
    |Programming in Scala,MMVIII,"Martin Odersky, Lex Spoon, Bill Venners",7
    |"Functional Programming in Scala"!,2014,"Paul Chiusano, Rúnar Bjarnason",5
    |""".stripMargin,
  UTF_8
)
```

Reading it prints each result as it is decoded:

```scala
withCsvFile(awkwardPath, bookCodec, CsvConfig.default)(rows => rows.foreach(println))
// (2,Right(BookRow(Zionomicon,John A. De Goes,
// Adam Fraser and Milad Khajavi,2024,12)))
// (4,Left(line 4: Cannot parse 'MMVIII' as Int: For input string: "MMVIII" at: .))
// (5,Left(line 5, column 34: Unexpected character '!' after closing quote))
// res15: Either[String, Unit] = Right(())
```

The first record spans lines 2 and 3, and its authors cell keeps the line break. The errors point to lines 4 and 5 of the file.

Writing a large file needs no new code: open a `java.io.BufferedWriter`, write the header, then write one encoded line per record. The following code writes 100,000 rows by cycling through the restocked inventory:

```scala
def writeCsvFile[A](path: Path, codec: CsvCodec[A], values: Iterator[A]): Unit =
  Using.resource(Files.newBufferedWriter(path, UTF_8)) { writer =>
    writer.write(CsvWriter.writeHeader(codec.headerNames, CsvConfig.default))
    values.foreach(value => writer.write(encodeLine(codec, value)))
  }

val largePath: Path = Files.createTempFile("large-inventory", ".csv")
writeCsvFile(largePath, bookCodec, Iterator.continually(restocked).flatten.take(100000))
```

Then we read it back and add up the copies in stock, holding one record in memory at a time:

```scala
withCsvFile(largePath, bookCodec, CsvConfig.default) { rows =>
  rows.foldLeft((0, 0L)) {
    case ((count, copies), (_, Right(book))) => (count + 1, copies + book.copies)
    case (totals, (_, Left(error)))          => println(error); totals
  }
}
// res17: Either[String, Tuple2[Int, Long]] = Right((100000, 1200008L))
```

The result is the number of rows and the total number of copies. Both functions stream: `Iterator#foldLeft` pulls one record at a time from the reader, and `writeCsvFile` writes each line as soon as it is encoded.

## Flatten Lists and Optional Fields

A CSV codec can only be derived for records whose fields are plain values: strings, numbers, dates and the other primitive types, or wrappers around them. A field of type `List`, `Map` or `Option`, or a sealed trait, has no single cell to go into. The deriver rejects such a record when `Schema#derive` runs, with an `UnsupportedOperationException`; the code compiles, and the error comes at run time:

```scala
case class Book(title: String, authors: List[String], year: Int, isbn: Option[String])

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

println(scala.util.Try(Book.schema.derive(CsvFormat)))
// Failure(java.lang.UnsupportedOperationException: CSV does not support variant/sum types. Use a flat case class instead.)
```

The message is about the `Option` field: the schema models `Option` as a sum type of `Some` and `None`, and CSV has no way to say which one a cell holds. A record whose only problem is a `List` field fails with `CSV does not support sequence fields.` instead. Neither message names the field, so check every field type when a derivation fails.

The workaround is a flat row type for the file, and a pair of functions that convert between it and the domain type. Each conversion is a choice about the file format: here the authors are joined with `"; "`, and a missing ISBN becomes an empty cell:

```scala
case class BookExportRow(title: String, authors: String, year: Int, isbn: String)

object BookExportRow {
  implicit val schema: Schema[BookExportRow] = Schema.derived

  def fromBook(book: Book): BookExportRow =
    BookExportRow(book.title, book.authors.mkString("; "), book.year, book.isbn.getOrElse(""))

  def toBook(row: BookExportRow): Book =
    Book(row.title, row.authors.split("; ").toList, row.year, Some(row.isbn).filter(_.nonEmpty))
}

val exportCodec: CsvCodec[BookExportRow] = BookExportRow.schema.derive(CsvFormat)

val catalog: List[Book] = List(
  Book("Zionomicon", List("John A. De Goes", "Adam Fraser", "Milad Khajavi"), 2024, None),
  Book("Programming in Scala", List("Martin Odersky", "Lex Spoon", "Bill Venners"), 2008, Some("978-0-9815316-0-1"))
)

val catalogCsv: String = writeCsv(exportCodec, catalog.map(BookExportRow.fromBook))
```

The CSV text has one cell per field, and the empty ISBN is an empty cell:

```scala
catalogCsv.linesIterator.foreach(println)
// title,authors,year,isbn
// Zionomicon,John A. De Goes; Adam Fraser; Milad Khajavi,2024,
// Programming in Scala,Martin Odersky; Lex Spoon; Bill Venners,2008,978-0-9815316-0-1
```

Reading the text back and converting each row restores the original books:

```scala
readCsv(exportCodec, catalogCsv, CsvConfig.default).map(_.records.map(BookExportRow.toBook) == catalog)
// res20: Either[String, Boolean] = Right(true)
```

:::warning
A field whose type is another case class is not rejected, but it doesn't become separate columns either. Take a row with a nested `Edition` record:

```scala
case class Edition(number: Int, year: Int)

object Edition {
  implicit val schema: Schema[Edition] = Schema.derived
}

case class EditionRow(title: String, edition: Edition)

object EditionRow {
  implicit val schema: Schema[EditionRow] = Schema.derived
}

val editionCodec: CsvCodec[EditionRow] = EditionRow.schema.derive(CsvFormat)
```

The codec writes the nested record as one quoted cell that ends with a line break (shown here as `\r\n`), and `CsvCodec#headerNames` has a single column for it:

```scala
println(editionCodec.headerNames)
// Vector(title, edition)
println(encodeLine(editionCodec, EditionRow("Programming in Scala", Edition(5, 2021))).replace("\r\n", "\\r\\n"))
// Programming in Scala,"5,2021\r\n"\r\n
```

Spreadsheets show that cell as a two-line value. Flatten nested records into separate fields, such as `editionNumber` and `editionYear`, in the same way as lists and options.
:::

## Putting It Together

The complete example reads an inventory file row by row, reports its bad rows with their line numbers, and writes the good records to a TSV file:

```scala title="schema-examples/src/main/scala/readandwritecsvandtsvfiles/CompleteExample.scala"
package readandwritecsvandtsvfiles

/**
 * How to Read and Write CSV and TSV Files — Complete Example
 *
 * Reads an inventory CSV file one record at a time, matching columns by header
 * name, reports every bad row with its line in the file, and writes each good
 * record to a TSV file as soon as it is decoded.
 *
 * Run with: cd schema-examples && sbt "runMain
 * readandwritecsvandtsvfiles.CompleteExample"
 */
object CompleteExample {

  case class BookRow(title: String, authors: String, year: Int, copies: Int)

  object BookRow {
    implicit val schema: Schema[BookRow] = Schema.derived
  }

  /** Decodes rows whose columns may come in any order, by header name. */
  final class RowDecoder[A](codec: CsvCodec[A], width: Int, positions: IndexedSeq[Int]) {
    def decode(cells: IndexedSeq[String]): Either[String, A] =
      if (cells.length != width) Left(s"expected $width cells, got ${cells.length}")
      else {
        val line = CsvWriter.writeRow(positions.map(cells), CsvConfig.default)
        codec.decode(CharBuffer.wrap(line)).left.map(_.getMessage)
      }
  }

  object RowDecoder {
    def fromHeader[A](codec: CsvCodec[A], header: IndexedSeq[String]): Either[String, RowDecoder[A]] = {
      val missing = codec.headerNames.filterNot(header.contains)
      if (missing.nonEmpty) Left(s"missing column(s): ${missing.mkString(", ")}")
      else Right(new RowDecoder(codec, header.length, codec.headerNames.map(name => header.indexOf(name))))
    }
  }

  /** Joins physical lines into CSV records, each with its first line number. */
  def csvRecords(reader: BufferedReader, quoteChar: Char): Iterator[(Int, String)] = {
    val lines = Iterator.continually(reader.readLine()).takeWhile(_ != null).zip(Iterator.from(1))

    @tailrec
    def complete(record: String): String =
      if (record.count(_ == quoteChar) % 2 == 0 || !lines.hasNext) record
      else complete(record + "\n" + lines.next()._1)

    lines.map { case (firstLine, lineNumber) => (lineNumber, complete(firstLine)) }
  }

  /** Opens a CSV or TSV file and passes its decoded rows to `use`. */
  def withCsvFile[A, B](path: Path, codec: CsvCodec[A], config: CsvConfig)(
    use: Iterator[(Int, Either[String, A])] => B
  ): Either[String, B] =
    Using.resource(Files.newBufferedReader(path, UTF_8)) { reader =>
      val records = csvRecords(reader, config.quoteChar)
      if (!records.hasNext) Left("the file is empty")
      else {
        val (_, headerText) = records.next()
        CsvReader.readRow(headerText, 0, config).left.map(_.getMessage).flatMap { case (header, _) =>
          RowDecoder.fromHeader(codec, header).map { decoder =>
            use(records.filter(_._2.nonEmpty).map { case (line, text) =>
              val decoded = CsvReader.readRow(text, 0, config) match {
                case Left(error)       => Left(s"line ${line + error.row - 1}, column ${error.column}: ${error.message}")
                case Right((cells, _)) => decoder.decode(cells).left.map(error => s"line $line: $error")
              }
              (line, decoded)
            })
          }
        }
      }
    }

  def encodeLine[A](codec: CsvCodec[A], value: A): String = {
    val buffer = CharBuffer.allocate(4096)
    codec.encode(value, buffer)
    buffer.flip()
    buffer.toString
  }

  def encodeCells[A](codec: CsvCodec[A], value: A): IndexedSeq[String] =
    CsvReader.readRow(encodeLine(codec, value), 0, CsvConfig.default) match {
      case Right((cells, _)) => cells
      case Left(error)       => throw error
    }

  def main(args: Array[String]): Unit = {
    val bookCodec: CsvCodec[BookRow] = BookRow.schema.derive(CsvFormat)

    val inventoryPath: Path = Files.createTempFile("inventory", ".csv")
    Files.writeString(
      inventoryPath,
      """title,year,authors,copies
        |Zionomicon,2024,"John A. De Goes, Adam Fraser and Milad Khajavi",12
        |Programming in Scala,MMVIII,"Martin Odersky, Lex Spoon, Bill Venners",7
        |Functional Programming in Scala,2014,"Paul Chiusano, Rúnar Bjarnason",5
        |Scala with Cats,2017,"Noel Welsh, Dave Gurnell"
        |""".stripMargin,
      UTF_8
    )

    val tsvPath: Path = Files.createTempFile("inventory", ".tsv")

    val result = Using.resource(Files.newBufferedWriter(tsvPath, UTF_8)) { writer =>
      writer.write(CsvWriter.writeHeader(bookCodec.headerNames, CsvConfig.tsv))
      withCsvFile(inventoryPath, bookCodec, CsvConfig.default) { rows =>
        rows.foldLeft(0) {
          case (count, (_, Right(book))) =>
            writer.write(CsvWriter.writeRow(encodeCells(bookCodec, book), CsvConfig.tsv))
            count + 1
          case (count, (_, Left(error))) =>
            println(s"skipped $error")
            count
        }
      }
    }

    result match {
      case Left(error)  => println(s"cannot read ${inventoryPath.getFileName}: $error")
      case Right(count) =>
        println(s"wrote $count records to the TSV file:")
        Files.readString(tsvPath, UTF_8).linesIterator.foreach(println)
    }
  }
}
```

## Running the Examples

All examples in this guide have runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

Each example below writes its sample files to temporary paths, so it runs anywhere without setup.

### Read a CSV File With a Header

This example writes the inventory to a temporary file, reads it with `CsvReader.readAll`, and decodes each row by column name.

<details>
  <summary>schema-examples/src/main/scala/readandwritecsvandtsvfiles/Step1ReadCsvExample.scala</summary>

```scala title="schema-examples/src/main/scala/readandwritecsvandtsvfiles/Step1ReadCsvExample.scala" showLineNumbers
package readandwritecsvandtsvfiles

/**
 * How to Read and Write CSV and TSV Files — Read a CSV File With a Header
 *
 * Writes a small inventory file whose columns are in a different order from
 * BookRow, reads it with CsvReader.readAll, and decodes each row by column name
 * with the derived CSV codec.
 *
 * Run with: cd schema-examples && sbt "runMain
 * readandwritecsvandtsvfiles.Step1ReadCsvExample"
 */
object Step1ReadCsvExample {

  case class BookRow(title: String, authors: String, year: Int, copies: Int)

  object BookRow {
    implicit val schema: Schema[BookRow] = Schema.derived
  }

  final class RowDecoder[A](codec: CsvCodec[A], width: Int, positions: IndexedSeq[Int]) {
    def decode(cells: IndexedSeq[String]): Either[String, A] =
      if (cells.length != width) Left(s"expected $width cells, got ${cells.length}")
      else {
        val line = CsvWriter.writeRow(positions.map(cells), CsvConfig.default)
        codec.decode(CharBuffer.wrap(line)).left.map(_.getMessage)
      }
  }

  object RowDecoder {
    def fromHeader[A](codec: CsvCodec[A], header: IndexedSeq[String]): Either[String, RowDecoder[A]] = {
      val missing = codec.headerNames.filterNot(header.contains)
      if (missing.nonEmpty) Left(s"missing column(s): ${missing.mkString(", ")}")
      else Right(new RowDecoder(codec, header.length, codec.headerNames.map(name => header.indexOf(name))))
    }
  }

  def main(args: Array[String]): Unit = {
    val bookCodec: CsvCodec[BookRow] = BookRow.schema.derive(CsvFormat)
    println(bookCodec.headerNames)

    val inventoryCsv: String =
      """title,year,authors,copies
        |Zionomicon,2024,"John A. De Goes, Adam Fraser and Milad Khajavi",12
        |Programming in Scala,2008,"Martin Odersky, Lex Spoon, Bill Venners",7
        |Functional Programming in Scala,2014,"Paul Chiusano, Rúnar Bjarnason",5
        |""".stripMargin

    val inventoryPath: Path = Files.createTempFile("inventory", ".csv")
    Files.writeString(inventoryPath, inventoryCsv, UTF_8)

    val inventory = CsvReader.readAll(Files.readString(inventoryPath, UTF_8), CsvConfig.default)
    println(inventory)

    inventory.foreach { case (header, rows) =>
      RowDecoder.fromHeader(bookCodec, header) match {
        case Left(error)    => println(error)
        case Right(decoder) => rows.map(decoder.decode).foreach(println)
      }
    }
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain readandwritecsvandtsvfiles.Step1ReadCsvExample"
```

### Report Every Bad Row

This example reads a damaged file, keeps the good records, and prints every bad row with its row number, a missing column and an unclosed quote.

<details>
  <summary>schema-examples/src/main/scala/readandwritecsvandtsvfiles/Step2ReportBadRowsExample.scala</summary>

```scala title="schema-examples/src/main/scala/readandwritecsvandtsvfiles/Step2ReportBadRowsExample.scala" showLineNumbers
package readandwritecsvandtsvfiles

/**
 * How to Read and Write CSV and TSV Files — Report Every Bad Row
 *
 * Reads a damaged inventory, keeps the good records and prints every bad row
 * with its row number. Also shows a file with a missing column and a file with
 * an unclosed quote.
 *
 * Run with: cd schema-examples && sbt "runMain
 * readandwritecsvandtsvfiles.Step2ReportBadRowsExample"
 */
object Step2ReportBadRowsExample {

  case class BookRow(title: String, authors: String, year: Int, copies: Int)

  object BookRow {
    implicit val schema: Schema[BookRow] = Schema.derived
  }

  final class RowDecoder[A](codec: CsvCodec[A], width: Int, positions: IndexedSeq[Int]) {
    def decode(cells: IndexedSeq[String]): Either[String, A] =
      if (cells.length != width) Left(s"expected $width cells, got ${cells.length}")
      else {
        val line = CsvWriter.writeRow(positions.map(cells), CsvConfig.default)
        codec.decode(CharBuffer.wrap(line)).left.map(_.getMessage)
      }
  }

  object RowDecoder {
    def fromHeader[A](codec: CsvCodec[A], header: IndexedSeq[String]): Either[String, RowDecoder[A]] = {
      val missing = codec.headerNames.filterNot(header.contains)
      if (missing.nonEmpty) Left(s"missing column(s): ${missing.mkString(", ")}")
      else Right(new RowDecoder(codec, header.length, codec.headerNames.map(name => header.indexOf(name))))
    }
  }

  final case class Report[A](records: IndexedSeq[A], errors: IndexedSeq[String])

  def readCsv[A](codec: CsvCodec[A], text: String, config: CsvConfig): Either[String, Report[A]] =
    CsvReader.readAll(text, config) match {
      case Left(error)           => Left(error.getMessage)
      case Right((header, rows)) =>
        RowDecoder.fromHeader(codec, header).map { decoder =>
          val results = rows.zipWithIndex.map { case (cells, index) =>
            decoder.decode(cells).left.map(error => s"row ${index + 2}: $error")
          }
          Report(results.collect { case Right(a) => a }, results.collect { case Left(e) => e })
        }
    }

  def main(args: Array[String]): Unit = {
    val bookCodec: CsvCodec[BookRow] = BookRow.schema.derive(CsvFormat)

    val damagedCsv: String =
      """title,year,authors,copies
        |Zionomicon,2024,"John A. De Goes, Adam Fraser and Milad Khajavi",12
        |Programming in Scala,MMVIII,"Martin Odersky, Lex Spoon, Bill Venners",7
        |Functional Programming in Scala,2014,"Paul Chiusano, Rúnar Bjarnason"
        |""".stripMargin

    readCsv(bookCodec, damagedCsv, CsvConfig.default).foreach { report =>
      report.records.foreach(println)
      report.errors.foreach(println)
    }

    println(readCsv(bookCodec, "title,year,authors\nZionomicon,2024,John A. De Goes\n", CsvConfig.default))

    val unclosedCsv: String =
      """title,year,authors,copies
        |Zionomicon,2024,"John A. De Goes, Adam Fraser and Milad Khajavi",12
        |Programming in Scala,2008,"Martin Odersky, Lex Spoon, Bill Venners,7
        |""".stripMargin

    println(readCsv(bookCodec, unclosedCsv, CsvConfig.default))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain readandwritecsvandtsvfiles.Step2ReportBadRowsExample"
```

### Write Records as CSV

This example encodes records with the derived codec, writes the CSV text to a file, and reads it back.

<details>
  <summary>schema-examples/src/main/scala/readandwritecsvandtsvfiles/Step3WriteCsvExample.scala</summary>

```scala title="schema-examples/src/main/scala/readandwritecsvandtsvfiles/Step3WriteCsvExample.scala" showLineNumbers
package readandwritecsvandtsvfiles

/**
 * How to Read and Write CSV and TSV Files — Write Records as CSV
 *
 * Encodes records with the derived CSV codec, adds a header with
 * CsvWriter.writeHeader, writes the text to a file and reads it back to check
 * the round trip.
 *
 * Run with: cd schema-examples && sbt "runMain
 * readandwritecsvandtsvfiles.Step3WriteCsvExample"
 */
object Step3WriteCsvExample {

  case class BookRow(title: String, authors: String, year: Int, copies: Int)

  object BookRow {
    implicit val schema: Schema[BookRow] = Schema.derived
  }

  def encodeLine[A](codec: CsvCodec[A], value: A): String = {
    val buffer = CharBuffer.allocate(4096)
    codec.encode(value, buffer)
    buffer.flip()
    buffer.toString
  }

  def writeCsv[A](codec: CsvCodec[A], values: Iterable[A]): String =
    CsvWriter.writeHeader(codec.headerNames, CsvConfig.default) + values.map(value => encodeLine(codec, value)).mkString

  def readRecords[A](codec: CsvCodec[A], text: String): Either[String, IndexedSeq[A]] =
    CsvReader.readAll(text, CsvConfig.default) match {
      case Left(error)      => Left(error.getMessage)
      case Right((_, rows)) =>
        val decoded = rows.map { cells =>
          codec.decode(CharBuffer.wrap(CsvWriter.writeRow(cells, CsvConfig.default))).left.map(_.getMessage)
        }
        decoded.collectFirst { case Left(error) => error }.toLeft(decoded.collect { case Right(a) => a })
    }

  def main(args: Array[String]): Unit = {
    val bookCodec: CsvCodec[BookRow] = BookRow.schema.derive(CsvFormat)

    val restocked: List[BookRow] = List(
      BookRow("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", 2024, 20),
      BookRow("Programming in Scala", "Martin Odersky, Lex Spoon, Bill Venners", 2008, 7),
      BookRow("Functional Programming in Scala", "Paul Chiusano, Rúnar Bjarnason", 2014, 9)
    )

    val restockedCsv: String = writeCsv(bookCodec, restocked)
    restockedCsv.linesIterator.foreach(println)

    println(CsvWriter.writeRow(IndexedSeq("Zionomicon", "the \"ZIO\" book"), CsvConfig.default).stripLineEnd)

    val restockedPath: Path = Files.createTempFile("restocked", ".csv")
    Files.writeString(restockedPath, restockedCsv, UTF_8)

    println(readRecords(bookCodec, Files.readString(restockedPath, UTF_8)).map(_ == restocked))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain readandwritecsvandtsvfiles.Step3WriteCsvExample"
```

### Read and Write TSV

This example writes the inventory with `CsvConfig.tsv` and reads it back, and shows what the derived codec does with a TSV line.

<details>
  <summary>schema-examples/src/main/scala/readandwritecsvandtsvfiles/Step4TsvExample.scala</summary>

```scala title="schema-examples/src/main/scala/readandwritecsvandtsvfiles/Step4TsvExample.scala" showLineNumbers
package readandwritecsvandtsvfiles

/**
 * How to Read and Write CSV and TSV Files — Read and Write TSV
 *
 * Shows that the derived codec only reads commas, then writes the inventory
 * with CsvConfig.tsv through CsvWriter.writeAll and reads it back with
 * CsvReader.readAll.
 *
 * Run with: cd schema-examples && sbt "runMain
 * readandwritecsvandtsvfiles.Step4TsvExample"
 */
object Step4TsvExample {

  case class BookRow(title: String, authors: String, year: Int, copies: Int)

  object BookRow {
    implicit val schema: Schema[BookRow] = Schema.derived
  }

  final class RowDecoder[A](codec: CsvCodec[A], width: Int, positions: IndexedSeq[Int]) {
    def decode(cells: IndexedSeq[String]): Either[String, A] =
      if (cells.length != width) Left(s"expected $width cells, got ${cells.length}")
      else {
        val line = CsvWriter.writeRow(positions.map(cells), CsvConfig.default)
        codec.decode(CharBuffer.wrap(line)).left.map(_.getMessage)
      }
  }

  object RowDecoder {
    def fromHeader[A](codec: CsvCodec[A], header: IndexedSeq[String]): Either[String, RowDecoder[A]] = {
      val missing = codec.headerNames.filterNot(header.contains)
      if (missing.nonEmpty) Left(s"missing column(s): ${missing.mkString(", ")}")
      else Right(new RowDecoder(codec, header.length, codec.headerNames.map(name => header.indexOf(name))))
    }
  }

  final case class Report[A](records: IndexedSeq[A], errors: IndexedSeq[String])

  def readCsv[A](codec: CsvCodec[A], text: String, config: CsvConfig): Either[String, Report[A]] =
    CsvReader.readAll(text, config) match {
      case Left(error)           => Left(error.getMessage)
      case Right((header, rows)) =>
        RowDecoder.fromHeader(codec, header).map { decoder =>
          val results = rows.zipWithIndex.map { case (cells, index) =>
            decoder.decode(cells).left.map(error => s"row ${index + 2}: $error")
          }
          Report(results.collect { case Right(a) => a }, results.collect { case Left(e) => e })
        }
    }

  def encodeLine[A](codec: CsvCodec[A], value: A): String = {
    val buffer = CharBuffer.allocate(4096)
    codec.encode(value, buffer)
    buffer.flip()
    buffer.toString
  }

  def encodeCells[A](codec: CsvCodec[A], value: A): IndexedSeq[String] =
    CsvReader.readRow(encodeLine(codec, value), 0, CsvConfig.default) match {
      case Right((cells, _)) => cells
      case Left(error)       => throw error
    }

  def writeDelimited[A](codec: CsvCodec[A], values: Iterable[A], config: CsvConfig): String =
    CsvWriter.writeAll(codec.headerNames, values.map(value => encodeCells(codec, value)), config)

  def main(args: Array[String]): Unit = {
    val bookCodec: CsvCodec[BookRow] = BookRow.schema.derive(CsvFormat)

    println(bookCodec.decode(CharBuffer.wrap("Zionomicon\tJohn A. De Goes, Adam Fraser and Milad Khajavi\t2024\t12")))

    val restocked: List[BookRow] = List(
      BookRow("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", 2024, 20),
      BookRow("Programming in Scala", "Martin Odersky, Lex Spoon, Bill Venners", 2008, 7),
      BookRow("Functional Programming in Scala", "Paul Chiusano, Rúnar Bjarnason", 2014, 9)
    )

    val restockedTsv: String = writeDelimited(bookCodec, restocked, CsvConfig.tsv)
    restockedTsv.linesIterator.foreach(println)

    println(readCsv(bookCodec, restockedTsv, CsvConfig.tsv).map(_.records == restocked))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain readandwritecsvandtsvfiles.Step4TsvExample"
```

### Process a Large File Row by Row

This example reads a file with a multi-line cell and bad rows one record at a time, then writes and sums a file of 100,000 rows.

<details>
  <summary>schema-examples/src/main/scala/readandwritecsvandtsvfiles/Step5LargeFileExample.scala</summary>

```scala title="schema-examples/src/main/scala/readandwritecsvandtsvfiles/Step5LargeFileExample.scala" showLineNumbers
package readandwritecsvandtsvfiles

/**
 * How to Read and Write CSV and TSV Files — Process a Large File Row by Row
 *
 * Reads a file one record at a time, joining the lines of a quoted cell that
 * spans several lines, and reports bad rows with their line in the file. Then
 * writes 100,000 rows and adds up the copies without loading the file into
 * memory.
 *
 * Run with: cd schema-examples && sbt "runMain
 * readandwritecsvandtsvfiles.Step5LargeFileExample"
 */
object Step5LargeFileExample {

  case class BookRow(title: String, authors: String, year: Int, copies: Int)

  object BookRow {
    implicit val schema: Schema[BookRow] = Schema.derived
  }

  final class RowDecoder[A](codec: CsvCodec[A], width: Int, positions: IndexedSeq[Int]) {
    def decode(cells: IndexedSeq[String]): Either[String, A] =
      if (cells.length != width) Left(s"expected $width cells, got ${cells.length}")
      else {
        val line = CsvWriter.writeRow(positions.map(cells), CsvConfig.default)
        codec.decode(CharBuffer.wrap(line)).left.map(_.getMessage)
      }
  }

  object RowDecoder {
    def fromHeader[A](codec: CsvCodec[A], header: IndexedSeq[String]): Either[String, RowDecoder[A]] = {
      val missing = codec.headerNames.filterNot(header.contains)
      if (missing.nonEmpty) Left(s"missing column(s): ${missing.mkString(", ")}")
      else Right(new RowDecoder(codec, header.length, codec.headerNames.map(name => header.indexOf(name))))
    }
  }

  def encodeLine[A](codec: CsvCodec[A], value: A): String = {
    val buffer = CharBuffer.allocate(4096)
    codec.encode(value, buffer)
    buffer.flip()
    buffer.toString
  }

  def csvRecords(reader: BufferedReader, quoteChar: Char): Iterator[(Int, String)] = {
    val lines = Iterator.continually(reader.readLine()).takeWhile(_ != null).zip(Iterator.from(1))

    @tailrec
    def complete(record: String): String =
      if (record.count(_ == quoteChar) % 2 == 0 || !lines.hasNext) record
      else complete(record + "\n" + lines.next()._1)

    lines.map { case (firstLine, lineNumber) => (lineNumber, complete(firstLine)) }
  }

  def withCsvFile[A, B](path: Path, codec: CsvCodec[A], config: CsvConfig)(
    use: Iterator[(Int, Either[String, A])] => B
  ): Either[String, B] =
    Using.resource(Files.newBufferedReader(path, UTF_8)) { reader =>
      val records = csvRecords(reader, config.quoteChar)
      if (!records.hasNext) Left("the file is empty")
      else {
        val (_, headerText) = records.next()
        CsvReader.readRow(headerText, 0, config).left.map(_.getMessage).flatMap { case (header, _) =>
          RowDecoder.fromHeader(codec, header).map { decoder =>
            use(records.filter(_._2.nonEmpty).map { case (line, text) =>
              val decoded = CsvReader.readRow(text, 0, config) match {
                case Left(error)       => Left(s"line ${line + error.row - 1}, column ${error.column}: ${error.message}")
                case Right((cells, _)) => decoder.decode(cells).left.map(error => s"line $line: $error")
              }
              (line, decoded)
            })
          }
        }
      }
    }

  def writeCsvFile[A](path: Path, codec: CsvCodec[A], values: Iterator[A]): Unit =
    Using.resource(Files.newBufferedWriter(path, UTF_8)) { writer =>
      writer.write(CsvWriter.writeHeader(codec.headerNames, CsvConfig.default))
      values.foreach(value => writer.write(encodeLine(codec, value)))
    }

  def main(args: Array[String]): Unit = {
    val bookCodec: CsvCodec[BookRow] = BookRow.schema.derive(CsvFormat)

    val awkwardPath: Path = Files.createTempFile("awkward", ".csv")
    Files.writeString(
      awkwardPath,
      """title,year,authors,copies
        |Zionomicon,2024,"John A. De Goes,
        |Adam Fraser and Milad Khajavi",12
        |Programming in Scala,MMVIII,"Martin Odersky, Lex Spoon, Bill Venners",7
        |"Functional Programming in Scala"!,2014,"Paul Chiusano, Rúnar Bjarnason",5
        |""".stripMargin,
      UTF_8
    )

    println(withCsvFile(awkwardPath, bookCodec, CsvConfig.default)(rows => rows.foreach(println)))

    val restocked: List[BookRow] = List(
      BookRow("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi", 2024, 20),
      BookRow("Programming in Scala", "Martin Odersky, Lex Spoon, Bill Venners", 2008, 7),
      BookRow("Functional Programming in Scala", "Paul Chiusano, Rúnar Bjarnason", 2014, 9)
    )

    val largePath: Path = Files.createTempFile("large-inventory", ".csv")
    writeCsvFile(largePath, bookCodec, Iterator.continually(restocked).flatten.take(100000))

    val totals = withCsvFile(largePath, bookCodec, CsvConfig.default) { rows =>
      rows.foldLeft((0, 0L)) {
        case ((count, copies), (_, Right(book))) => (count + 1, copies + book.copies)
        case (totals, (_, Left(error)))          => println(error); totals
      }
    }
    println(totals)
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain readandwritecsvandtsvfiles.Step5LargeFileExample"
```

### Flatten Lists and Optional Fields

This example shows the deriver rejecting a `List` field, then exports and imports books through a flat row type.

<details>
  <summary>schema-examples/src/main/scala/readandwritecsvandtsvfiles/Step6FlatRowsExample.scala</summary>

```scala title="schema-examples/src/main/scala/readandwritecsvandtsvfiles/Step6FlatRowsExample.scala" showLineNumbers
package readandwritecsvandtsvfiles

/**
 * How to Read and Write CSV and TSV Files — Flatten Lists and Optional Fields
 *
 * Shows the CSV deriver rejecting a record with a List field at run time, then
 * exports books through a flat row type that joins the authors and turns a
 * missing ISBN into an empty cell, and imports them back.
 *
 * Run with: cd schema-examples && sbt "runMain
 * readandwritecsvandtsvfiles.Step6FlatRowsExample"
 */
object Step6FlatRowsExample {

  case class Book(title: String, authors: List[String], year: Int, isbn: Option[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class BookExportRow(title: String, authors: String, year: Int, isbn: String)

  object BookExportRow {
    implicit val schema: Schema[BookExportRow] = Schema.derived

    def fromBook(book: Book): BookExportRow =
      BookExportRow(book.title, book.authors.mkString("; "), book.year, book.isbn.getOrElse(""))

    def toBook(row: BookExportRow): Book =
      Book(row.title, row.authors.split("; ").toList, row.year, Some(row.isbn).filter(_.nonEmpty))
  }

  def encodeLine[A](codec: CsvCodec[A], value: A): String = {
    val buffer = CharBuffer.allocate(4096)
    codec.encode(value, buffer)
    buffer.flip()
    buffer.toString
  }

  def writeCsv[A](codec: CsvCodec[A], values: Iterable[A]): String =
    CsvWriter.writeHeader(codec.headerNames, CsvConfig.default) + values.map(value => encodeLine(codec, value)).mkString

  def readRecords[A](codec: CsvCodec[A], text: String): Either[String, IndexedSeq[A]] =
    CsvReader.readAll(text, CsvConfig.default) match {
      case Left(error)      => Left(error.getMessage)
      case Right((_, rows)) =>
        val decoded = rows.map { cells =>
          codec.decode(CharBuffer.wrap(CsvWriter.writeRow(cells, CsvConfig.default))).left.map(_.getMessage)
        }
        decoded.collectFirst { case Left(error) => error }.toLeft(decoded.collect { case Right(a) => a })
    }

  def main(args: Array[String]): Unit = {
    println(scala.util.Try(Book.schema.derive(CsvFormat)))

    val exportCodec: CsvCodec[BookExportRow] = BookExportRow.schema.derive(CsvFormat)

    val catalog: List[Book] = List(
      Book("Zionomicon", List("John A. De Goes", "Adam Fraser", "Milad Khajavi"), 2024, None),
      Book("Programming in Scala", List("Martin Odersky", "Lex Spoon", "Bill Venners"), 2008, Some("978-0-9815316-0-1"))
    )

    val catalogCsv: String = writeCsv(exportCodec, catalog.map(BookExportRow.fromBook))
    catalogCsv.linesIterator.foreach(println)

    println(readRecords(exportCodec, catalogCsv).map(_.map(BookExportRow.toBook) == catalog))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain readandwritecsvandtsvfiles.Step6FlatRowsExample"
```

### The Complete Example

Run the complete example from the `schema-examples` directory:

```bash
sbt "runMain readandwritecsvandtsvfiles.CompleteExample"
```

### Compile All Examples

To compile every example in the project without running any of them, run this from the `schema-examples` directory:

```bash
sbt compile
```

## Going Further

- The [CSV reference page](../reference/schema/built-in-codecs/csv.md) lists every `CsvConfig` field and every method of `CsvReader`, `CsvWriter` and `CsvCodec`.
- [One Type, Many Formats](./one-type-many-formats.md) derives JSON, YAML, XML, TOON, MessagePack and Avro codecs from the same kind of schema, for data that doesn't fit in a table.
- [Schema](../reference/schema/schema.md) explains `Schema#derive` and the formats it accepts.

---

## How to Read and Write JSON Lines with Streams

In this guide, we read a sensor log in JSON Lines format, one JSON value per line, with ZIO Blocks Streams. JSON Lines is the usual format for logs, exports and LLM training data, because a writer can append a record without rewriting the file. By the end, you will have a stream that decodes each line into a typed `SensorReading`, two ways to handle a bad line (skip and count it, or fail with its line number), and a writer that puts one record on each line of a new file.

We start with the code a reader writes without a library, then build the stream one section at a time:

- Reading the lines of a file as a stream.
- Decoding each line into a record.
- Skipping bad lines and counting them.
- Failing at the first bad line, with its line number.
- Writing records back, one per line.

## The Problem

A JSON Lines file is easy to produce and easy to get wrong. One truncated line, or one string where a number belongs, can stop an import that otherwise worked. Without a library, a reader usually matches each line against a pattern and turns the captures into values:

```scala

def readSensorLog(path: Path): List[(String, Double, Long)] = {
  val pattern = """\{"sensor":"([^"]*)","celsius":([-0-9.]+),"sample":(\d+)\}""".r
  Files.readAllLines(path).asScala.toList.map { case pattern(sensor, celsius, sample) =>
    (sensor, celsius.toDouble, sample.toLong)
  }
}
```

This version has two problems. A line that does not match the pattern throws a `MatchError`, which ends the whole import, and the pattern breaks as soon as a field is added or reordered. Writing has the same weakness: each record is formatted by string interpolation, so a sensor name that contains a quote produces invalid JSON.

In this guide, we let a `Schema` derive the JSON codec, and let the stream decide what happens to each bad line.

## Prerequisites

Add the streams and schema modules to your `build.sbt`. The streams module does not depend on the schema module, so we add both. The schema module provides `Schema` and the JSON codec:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-schema"  % "0.0.56",
  "dev.zio" %% "zio-blocks-streams" % "0.0.56"
)

// Scala 3.9 and newer, which also needs -experimental
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-schema"  % "0.0.56",
  "dev.zio" %% "zio-blocks-next-streams" % "0.0.56"
)
scalacOptions += "-experimental"
```

[How to Add ZIO Blocks to Your Build](../getting-started/installation.md) explains which artifact name each Scala version uses. On the JVM, the streams module needs JDK 21 or newer. The terminals in this guide, such as `Stream#run`, `Stream#runCollect` and `Stream#runForeach`, run on the JVM. On Scala.js, use the `*Async` terminals instead, such as `Stream#runCollectAsync`.

This guide assumes you have run a stream before, as in [Your First Streaming Pipeline](./your-first-streaming-pipeline.md), and that you can derive a schema with `Schema.derived`, as in [Your First Schema](./your-first-schema.md).

The snippets below share these imports:

```scala

```

## The Core Model

The sensor log holds one reading per line. We model a reading as a case class, and a line that cannot be decoded as a `BadLine` that records where it is and why it failed:

```scala
final case class SensorReading(sensor: String, celsius: Double, sample: Long)

object SensorReading {
  implicit val schema: Schema[SensorReading] = Schema.derived
}

final case class BadLine(line: Long, text: String, reason: String)
```

`Schema.derived` gives `SensorReading` a JSON codec. The methods `SensorReading#toJsonString` and `String#fromJson` use that default codec, so one line of the file maps to one `SensorReading`.

We write a sample log to a folder under the system's temporary directory, so that every snippet runs on any machine. Lines 3 and 5 are bad: line 3 has a string where `celsius` should be a number, and line 5 is cut off before the value of `sample`:

```scala
val shop: Path = Files.createDirectories(Paths.get(System.getProperty("java.io.tmpdir"), "sensor-log"))

val sensorLog: Path = shop.resolve("sensors.jsonl")

Files.writeString(
  sensorLog,
  """{"sensor":"boiler-7","celsius":21.5,"sample":1}
    |{"sensor":"boiler-7","celsius":22.0,"sample":2}
    |{"sensor":"boiler-7","celsius":"hot","sample":3}
    |{"sensor":"attic-2","celsius":14.25,"sample":4}
    |{"sensor":"attic-2","celsius":14.5,"sample":
    |{"sensor":"cellar-1","celsius":9.0,"sample":6}
    |""".stripMargin
)
```

Printing the file shows what the stream will read:

```scala
print(Files.readString(sensorLog))
// {"sensor":"boiler-7","celsius":21.5,"sample":1}
// {"sensor":"boiler-7","celsius":22.0,"sample":2}
// {"sensor":"boiler-7","celsius":"hot","sample":3}
// {"sensor":"attic-2","celsius":14.25,"sample":4}
// {"sensor":"attic-2","celsius":14.5,"sample":
// {"sensor":"cellar-1","celsius":9.0,"sample":6}
```

## Reading the Lines of a File

Streams has no helper that splits a file into lines, so we use the JDK's `BufferedReader#lines` for the splitting, inside `Stream.fromAcquireRelease`, as the streaming guides do. The stream owns the reader: it opens the file when the stream runs, and closes it when the stream ends, fails or stops early. The default `release` closes any `AutoCloseable`, so we do not pass one. The reader becomes a stream like this:

```scala
def readLines(path: Path): Stream[Nothing, String] =
  Stream.fromAcquireRelease(Files.newBufferedReader(path))(reader =>
    Stream.fromIterator(reader.lines().iterator().asScala)
  )
```

Running the stream with `Stream#runForeach` prints each line as it arrives:

```scala
readLines(sensorLog).runForeach(line => println(line))
// {"sensor":"boiler-7","celsius":21.5,"sample":1}
// {"sensor":"boiler-7","celsius":22.0,"sample":2}
// {"sensor":"boiler-7","celsius":"hot","sample":3}
// {"sensor":"attic-2","celsius":14.25,"sample":4}
// {"sensor":"attic-2","celsius":14.5,"sample":
// {"sensor":"cellar-1","celsius":9.0,"sample":6}
// res3: Either[Nothing, Unit] = Right(())
```

Each line prints on its own line, and the terminal returns `Right(())`, which means the run succeeded and returns no value.

## Decoding Each Line into a Record

Each line is a JSON object, so we decode it with `String#fromJson`, which returns `Either[SchemaError, SensorReading]`. We want to know which line failed, so we number the lines first. `Stream#mapAccum` carries a counter from one element to the next, and the decoded error becomes a `BadLine`:

```scala
val numbered: Stream[Nothing, (Long, String)] =
  readLines(sensorLog).mapAccum(1L)((n, text) => (n + 1, (n, text)))

val decoded: Stream[Nothing, Either[BadLine, SensorReading]] =
  numbered.map { case (n, text) =>
    text.fromJson[SensorReading].left.map(error => BadLine(n, text, error.getMessage))
  }
```

Running `decoded` shows one result per line:

```scala
decoded.runForeach(result => println(result))
// Right(SensorReading(boiler-7,21.5,1))
// Right(SensorReading(boiler-7,22.0,2))
// Left(BadLine(3,{"sensor":"boiler-7","celsius":"hot","sample":3},illegal number at: .celsius))
// Right(SensorReading(attic-2,14.25,4))
// Left(BadLine(5,{"sensor":"attic-2","celsius":14.5,"sample":,unexpected end of input at: .sample))
// Right(SensorReading(cellar-1,9.0,6))
// res4: Either[Nothing, Unit] = Right(())
```

Lines 3 and 5 come out as `Left` values. Each message names the field where decoding stopped, the `celsius` and `sample` fields, and the stream keeps going.

## Skipping Bad Lines and Counting Them

Some imports should keep every good line and report the rest. `Sink.foldLeft` reads the stream once, and its function decides what each result adds to the accumulator. A bad line is logged and counted, and a good reading is kept:

```scala
final case class Import(readings: Vector[SensorReading], skipped: Int)

decoded.run(
  Sink.foldLeft(Import(Vector.empty, 0)) { (acc, next) =>
    next match {
      case Right(reading) => acc.copy(readings = acc.readings :+ reading)
      case Left(bad)      =>
        println(s"skipped line ${bad.line}: ${bad.reason}")
        acc.copy(skipped = acc.skipped + 1)
    }
  }
).foreach { imported =>
  println(s"kept ${imported.readings.size} readings, skipped ${imported.skipped} lines")
  imported.readings.foreach(println)
}
// skipped line 3: illegal number at: .celsius
// skipped line 5: unexpected end of input at: .sample
// kept 4 readings, skipped 2 lines
// SensorReading(boiler-7,21.5,1)
// SensorReading(boiler-7,22.0,2)
// SensorReading(attic-2,14.25,4)
// SensorReading(cellar-1,9.0,6)
```

The skipped lines are logged while the stream runs, and the summary comes after. The `Right` from `Stream#run` holds the accumulator, so the code after `Either#foreach` sees the kept readings and the count together.

## Failing at the First Bad Line

Some imports must not continue past a bad line, for example when a later step depends on every record. We turn the decode error into a typed failure with `Stream.fail`, and `Stream#flatMap` replaces each line with either one reading or that failure. The stream's error type is now `BadLine`:

```scala
val strict: Stream[BadLine, SensorReading] =
  numbered.flatMap { case (n, text) =>
    text.fromJson[SensorReading] match {
      case Right(reading) => Stream.succeed(reading)
      case Left(error)    => Stream.fail(BadLine(n, text, error.getMessage))
    }
  }
```

Running `strict` with `Stream#runCollect` returns the first failure, with its line number:

```scala
strict.runCollect
// res6: Either[BadLine, Chunk[SensorReading]] = Left(
//   BadLine(
//     line = 3L,
//     text = "{\"sensor\":\"boiler-7\",\"celsius\":\"hot\",\"sample\":3}",
//     reason = "illegal number at: .celsius"
//   )
// )
```

The readings before line 3 are emitted before the failure, but `Stream#runCollect` returns only the `Left`. A terminal that folds the elements as they arrive, such as the one in the previous section, would see them before the failure.

## Writing One Record Per Line

To write a record, we turn it into its JSON text, add a newline, and send the bytes to the output. `Stream.fromIterable` makes a stream of records, and `Stream#flatMap` turns each record into the bytes of one line. `Sink.fromOutputStream` writes the bytes in order. The sink does not close the output stream, so the code closes it in `finally`:

```scala
val readings: Seq[SensorReading] = Seq(
  SensorReading("boiler-7", 21.5, 1L),
  SensorReading("attic-2", 14.25, 2L),
  SensorReading("cellar-1", 9.0, 3L)
)

def jsonLine(reading: SensorReading): Array[Byte] =
  (reading.toJsonString + "\n").getBytes(java.nio.charset.StandardCharsets.UTF_8)

def writeJsonLines(path: Path, records: Seq[SensorReading]): Either[Nothing, Unit] = {
  val out = Files.newOutputStream(path)
  try
    Stream
      .fromIterable(records)
      .flatMap(reading => Stream.fromIterable(jsonLine(reading).toIndexedSeq))
      .run(Sink.fromOutputStream(out))
  finally out.close()
}
```

We write three readings to a new file and print its content:

```scala
writeJsonLines(shop.resolve("readings.jsonl"), readings)
// res7: Either[Nothing, Unit] = Right(())
print(Files.readString(shop.resolve("readings.jsonl")))
// {"sensor":"boiler-7","celsius":21.5,"sample":1}
// {"sensor":"attic-2","celsius":14.25,"sample":2}
// {"sensor":"cellar-1","celsius":9.0,"sample":3}
```

Each record is one line, ending with a newline, so the file can be appended to and read again with the code above.

:::tip
`Schema` derives the codec that `SensorReading#toJsonString` uses, so the field names in the file are the case class field names. To change them, see [Customize JSON Encoding](./customize-json-encoding.md).
:::

## Putting It Together

The complete example reads the sample log, skips and counts the bad lines, writes the good readings to a new file, and prints that file. It differs from the sections in three ways: it writes to a fresh temporary directory instead of the `sensor-log` folder, it writes `clean.jsonl` instead of `readings.jsonl`, and it deletes its files when it finishes. The complete example is:

```scala title="zio-blocks-examples/json-lines/src/main/scala/readandwritejsonlines/CompleteExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using dep dev.zio::zio-blocks-streams:0.0.56
//> using file Model.scala

package readandwritejsonlines

/**
 * How to Read and Write JSON Lines (NDJSON) with Streams — Complete Example
 *
 * Cleans a sensor log in one pass: reads the lines, decodes each one, logs and
 * counts the bad lines, writes the good readings to a new JSON Lines file, and
 * reads that file back to check it.
 *
 * Run with: cd zio-blocks-examples && sbt "jsonLines/runMain
 * readandwritejsonlines.CompleteExample"
 */
object CompleteExample {

  final case class Import(readings: Vector[SensorReading], skipped: Int)

  def readLines(path: Path): Stream[Nothing, String] =
    Stream.fromAcquireRelease(Files.newBufferedReader(path))(reader =>
      Stream.fromIterator(reader.lines().iterator().asScala)
    )

  def jsonLine(reading: SensorReading): Array[Byte] =
    (reading.toJsonString + "\n").getBytes(UTF_8)

  def writeJsonLines(path: Path, records: Seq[SensorReading]): Either[Nothing, Unit] = {
    val out = Files.newOutputStream(path)
    try
      Stream
        .fromIterable(records)
        .flatMap(reading => Stream.fromIterable(jsonLine(reading).toIndexedSeq))
        .run(Sink.fromOutputStream(out))
    finally out.close()
  }

  def main(args: Array[String]): Unit = {
    val dir    = Files.createTempDirectory("json-lines")
    val input  = Sample.writeTo(dir)
    val output = dir.resolve("clean.jsonl")

    val numbered: Stream[Nothing, (Long, String)] =
      readLines(input).mapAccum(1L)((n, text) => (n + 1, (n, text)))

    val decoded: Stream[Nothing, Either[BadLine, SensorReading]] =
      numbered.map { case (n, text) =>
        text.fromJson[SensorReading].left.map(error => BadLine(n, text, error.getMessage))
      }

    val imported: Either[Nothing, Import] = decoded.run(
      Sink.foldLeft(Import(Vector.empty, 0)) { (acc, next) =>
        next match {
          case Right(reading) => acc.copy(readings = acc.readings :+ reading)
          case Left(bad)      =>
            println(s"skipped line ${bad.line}: ${bad.reason}")
            acc.copy(skipped = acc.skipped + 1)
        }
      }
    )

    imported.foreach { result =>
      writeJsonLines(output, result.readings)
      println(s"wrote ${result.readings.size} readings, skipped ${result.skipped} lines")
    }

    println("clean.jsonl:")
    print(Files.readString(output))

    Files.delete(input)
    Files.delete(output)
    Files.delete(dir)
  }
}
```

Running it prints the skipped lines, then the summary, then the new file:

```text
skipped line 3: illegal number at: .celsius
skipped line 5: unexpected end of input at: .sample
wrote 4 readings, skipped 2 lines
clean.jsonl:
{"sensor":"boiler-7","celsius":21.5,"sample":1}
{"sensor":"boiler-7","celsius":22.0,"sample":2}
{"sensor":"attic-2","celsius":14.25,"sample":4}
{"sensor":"cellar-1","celsius":9.0,"sample":6}
```

## Running the Examples

All runnable code from this guide is in [`zio-blocks-examples/json-lines/src/main/scala/readandwritejsonlines/`](https://github.com/zio/zio-blocks/tree/main/zio-blocks-examples/json-lines/src/main/scala/readandwritejsonlines). Every file uses the shared `Model.scala` in the same folder. Run a file from that folder with `scala-cli run <File>.scala Model.scala`, or from `zio-blocks-examples` with `sbt "jsonLines/runMain readandwritejsonlines.<ObjectName>"`.

## Going Further

- [Stream Files and Sockets Without Leaking Resources](./stream-files-and-sockets.md) covers the resource handling behind `Stream.fromAcquireRelease`, and reading from a socket.
- [Read and Write CSV and TSV Files](./read-and-write-csv-and-tsv-files.md) applies the same bad-row policies to tabular files.
- [Stream reference](../reference/streams/core/stream.md) lists every operator and terminal this guide uses, and [Sink reference](../reference/streams/core/sink.md) covers `Sink.foldLeft` and `Sink.fromOutputStream`.

---

## How to Read and Write YAML Documents

In this guide, we will read a bookstore's catalog from a YAML file into Scala case classes and write a CI workflow file from Scala values, using the same `Schema` that already gives you JSON. By the end, you will have a small set of functions that decode YAML with readable errors, write YAML in the style you choose, convert YAML to and from JSON, and refuse a document that the reader would otherwise load with parts missing.

We start with a short catalog and add one capability at a time:

- Read a YAML catalog into case classes.
- Report decode errors with the path of the bad value.
- Write a CI workflow as YAML.
- Control indentation, flow style and document markers.
- Build YAML values in code.
- Convert between YAML and JSON.
- Learn what the YAML module does not support, and guard against silent data loss.

## The Problem

YAML is the format of Kubernetes manifests, GitHub Actions workflows, docker-compose files, OpenAPI specs and many application configuration files. A first attempt at reading one by hand splits each line at the colon:

```scala
def readFlat(text: String): Map[String, String] =
  text.linesIterator
    .map(_.trim)
    .filter(line => line.nonEmpty && !line.startsWith("#"))
    .flatMap { line =>
      line.split(":", 2) match {
        case Array(key, value) => Some(key.trim -> value.trim.stripPrefix("\"").stripSuffix("\""))
        case _                 => None
      }
    }
    .toMap

val book  = readFlat("title: Zionomicon\npages: 600\n")
val pages = book("pages").toInt
```

This works for one flat record and nothing else. A list of authors, a nested record, a value that contains a colon, or two books in one file all break it, and every field still has to be converted and checked by hand. A YAML library solves the parsing, but most Scala YAML libraries come with their own codec type class, so you end up maintaining JSON codecs and YAML codecs side by side.

In this guide, we solve this with `zio-blocks-schema-yaml`: a codec derived from the `Schema` you already have converts YAML to and from your case classes, and the `Yaml` value type lets you build, inspect and convert documents in code. The module is small and lenient, and some YAML features are not supported; the last two sections show exactly which ones, and how to make sure a document was read in full.

## Prerequisites

Add the schema and YAML modules to your build:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-schema"      % "0.0.56",
  "dev.zio" %% "zio-blocks-schema-yaml" % "0.0.56"
)

// Scala 3.9 and newer (also needs scalacOptions += "-experimental"): the same modules with the zio-blocks-next- prefix
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-schema"      % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema-yaml" % "0.0.56"
)
```

[How to Add ZIO Blocks to Your Build](../getting-started/installation.md) explains the two artifact names and how to choose one per Scala version. Everything in this guide works on Scala 2.13 and Scala 3, except the `yaml"..."` interpolator, which is Scala 3 only in 0.0.56 (see [Build YAML Values in Code](#build-yaml-values-in-code)).

This guide assumes you can derive a `Schema` for a case class, as shown in [Your First Schema](./your-first-schema.md). [One Type, Many Formats](./one-type-many-formats.md) shows a YAML codec next to the other formats; here we go into the details of YAML.

## The Catalog Model

Our catalog file lists books, and each book has a title, its authors, an ISBN, a price, a page count, some tags and the formats it is sold in:

```scala

sealed trait Format

object Format {
  case object Paperback extends Format
  case object Hardcover extends Format
  case object Ebook     extends Format

  val all: List[Format] = List(Paperback, Hardcover, Ebook)

  implicit val schema: Schema[Format] = Schema[String].transform[Format](
    name => all.find(_.toString == name).getOrElse(throw new IllegalArgumentException(s"Unknown format: $name")),
    _.toString
  )
}

case class Book(
  title: String,
  authors: List[String],
  isbn: String,
  price: Double,
  pageCount: Int,
  tags: List[String],
  formats: List[Format]
)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

case class Catalog(store: String, books: List[Book])

object Catalog {
  implicit val schema: Schema[Catalog] = Schema.derived
}
```

`Format` gets its schema from `Schema[String].transform` instead of `Schema.derived`, so each format is written as its name (`Ebook`). A derived schema for a sealed trait writes each case as a mapping with one key, the case name, and the case's fields under it, so a case object becomes `Ebook: {}` and a plain `Ebook` in the file fails to decode with `Expected mapping with single key for variant`. That is the right shape for cases with fields, but YAML files list enumeration values as plain words.

## Read a YAML Catalog Into Case Classes

`Schema#derive(YamlFormat)` derives a `YamlCodec` from the schema. Its `YamlCodec#decode` method takes the YAML text and returns `Either[SchemaError, A]`:

```scala
val catalogCodec: YamlCodec[Catalog] = Catalog.schema.derive(YamlFormat)

val catalogYaml: String =
  """# Books on the front table this month
    |store: Scala Books Berlin
    |books:
    |  - title: Zionomicon
    |    authors: [John A. De Goes, Adam Fraser, Milad Khajavi]
    |    isbn: 978-0-9862-0040-6
    |    price: 49.99
    |    page-count: 744
    |    tags: [zio, effects, concurrency]
    |    formats: [Ebook]
    |  - title: Programming in Scala
    |    authors:
    |      - Martin Odersky
    |      - Lex Spoon
    |      - Bill Venners
    |      - Frank Sommers
    |    isbn: "9780997148008"
    |    price: 59.95
    |    page-count: 852
    |    tags: [scala, language]
    |    formats:
    |      - Paperback
    |      - Ebook
    |""".stripMargin
```

To decode the catalog and print each book with its authors and price:

```scala
catalogCodec.decode(catalogYaml) match {
  case Right(catalog) => catalog.books.foreach(book => println(s"${book.title}: ${book.authors.mkString(", ")}, ${book.price}"))
  case Left(error)    => println(error.getMessage)
}
// Zionomicon: John A. De Goes, Adam Fraser, Milad Khajavi, 49.99
// Programming in Scala: Martin Odersky, Lex Spoon, Bill Venners, Frank Sommers, 59.95
```

The file mixes the two ways YAML writes a list: the authors of Zionomicon are a flow sequence in square brackets, and the authors of Programming in Scala are a block sequence with one `- ` item per line. The codec reads both. It also skips the comment, and ignores keys that the case class doesn't have.

Two rules decide which keys the codec looks for. Keys are the field names in kebab-case: the `pageCount` field is read from the key `page-count`. The YAML codec always converts field names this way, and in 0.0.56 it ignores `@Modifier.rename`, `@Modifier.transient` and the other field annotations that the JSON codec honours, so the [JSON customizations](./customize-json-encoding.md) don't carry over to YAML. A field named `imagePullPolicy` can't be read from a Kubernetes file that spells the key `imagePullPolicy`; [Convert Between YAML and JSON](#convert-between-yaml-and-json) shows a way around this. GitHub issue [#1327](https://github.com/zio/zio-blocks/issues/1327) tracks configurable field names. Scalars are read as text and converted by the field's type: the ISBN of Programming in Scala is quoted because it looks like a number, but an unquoted ISBN is fine too, because a `String` field keeps the text either way and a `Double` field parses it.

If you prefer extension methods, `YamlSyntax` adds `String#fromYaml` to `String`, and `toYaml`, `toYamlString` and `toYamlBytes` to any value with a `Schema`:

```scala

val zionomicon = Book(
  title = "Zionomicon",
  authors = List("John A. De Goes", "Adam Fraser", "Milad Khajavi"),
  isbn = "978-0-9862-0040-6",
  price = 49.99,
  pageCount = 744,
  tags = List("zio", "effects"),
  formats = List(Format.Ebook, Format.Paperback)
)
```

To write the book as YAML and check that it reads back as the same value:

```scala
println(zionomicon.toYamlString)
// title: Zionomicon
// authors:
//   - John A. De Goes
//   - Adam Fraser
//   - Milad Khajavi
// isbn: "978-0-9862-0040-6"
// price: 49.99
// page-count: 744
// tags:
//   - zio
//   - effects
// formats:
//   - Ebook
//   - Paperback
println(zionomicon.toYamlString.fromYaml[Book] == Right(zionomicon))
// true
```

The codec also decodes from `Array[Byte]` and `java.nio.ByteBuffer`, so you can pass the bytes of a file straight in: `catalogCodec.decode(Files.readAllBytes(path))`.

## Report Decode Errors

When a value is missing or has the wrong type, `YamlCodec#decode` returns a `Left` with a `SchemaError`. Its message names what went wrong and the path to the value, written as Scala navigation from the root (`.books.at(1).price` is the `price` field of the second book):

```scala
def report(yaml: String): Unit =
  catalogCodec.decode(yaml) match {
    case Right(catalog) => println(s"OK: ${catalog.books.size} books")
    case Left(error)    => println(error.getMessage)
  }

report(catalogYaml.replace("price: 59.95", "price: cheap"))
// Invalid double value: cheap at: .books.at(1).price
report(catalogYaml.replace("page-count: 744", "page-count: many"))
// Invalid int value: many at: .books.at(0).pageCount
report(catalogYaml.replace("formats: [Ebook]", "formats: [Kindle]"))
// Unknown format: Kindle at: .books.at(0).formats.at(0).wrapped
report(catalogYaml.replace("    isbn: \"9780997148008\"\n", ""))
// Missing required field: isbn at: .books.at(1)
```

The messages read as follows:

- A wrong scalar names the field that holds it. The path uses the Scala field name (`.books.at(0).pageCount`), not the YAML key `page-count`.
- The `Format` error ends in a `wrapped` path step, the step that ZIO Blocks adds for a type built with `Schema#transform`. It is an internal detail that shows up in the path.
- A missing field is named by its YAML key, and the path points at the record that should contain it, not at the field.

None of the messages has a line or column number: the codec decodes a tree of YAML values, and the tree doesn't remember where each value came from. A path is usually enough to find the value in a file of this size.

Malformed YAML is a different story. `YamlReader`, which turns the text into that tree, never reports a syntax error. It is lenient: it reads what it can, and in some cases it drops the rest of the document without a word. [Guard Against Silent Data Loss](#guard-against-silent-data-loss) shows how to detect that.

## Write a CI Workflow as YAML

Writing YAML is the other half. Our bookstore builds its Scala code on GitHub Actions, and we want to generate the workflow file from Scala values, for example to keep the Scala and JDK versions in one place. Kebab-case keys are what GitHub Actions expects (`runs-on`, `timeout-minutes`), so the default field naming works in our favour here:

```scala
case class Branches(branches: List[String])

object Branches {
  implicit val schema: Schema[Branches] = Schema.derived
}

case class Triggers(push: Branches, pullRequest: Branches)

object Triggers {
  implicit val schema: Schema[Triggers] = Schema.derived
}

case class Step(name: String, uses: Option[String] = None, run: Option[String] = None)

object Step {
  implicit val schema: Schema[Step] = Schema.derived
}

case class Job(runsOn: String, timeoutMinutes: Int, steps: List[Step])

object Job {
  implicit val schema: Schema[Job] = Schema.derived
}

case class Workflow(name: String, on: Triggers, jobs: Map[String, Job])

object Workflow {
  implicit val schema: Schema[Workflow] = Schema.derived
}

val workflowCodec: YamlCodec[Workflow] = Workflow.schema.derive(YamlFormat)

val ciWorkflow = Workflow(
  name = "CI",
  on = Triggers(push = Branches(List("main")), pullRequest = Branches(List("main"))),
  jobs = Map(
    "test" -> Job(
      runsOn = "ubuntu-latest",
      timeoutMinutes = 30,
      steps = List(
        Step("Check out", uses = Some("actions/checkout@v4")),
        Step("Set up sbt", uses = Some("sbt/setup-sbt@v1")),
        Step("Test", run = Some("sbt +test"))
      )
    )
  )
)
```

`YamlCodec#encodeToString` writes the value as a YAML string, and `YamlCodec#encode` writes the same text as UTF-8 bytes, ready for `Files.write`:

```scala
println(workflowCodec.encodeToString(ciWorkflow))
// name: CI
// "on":
//   push:
//     branches:
//       - main
//   pull-request:
//     branches:
//       - main
// jobs:
//   test:
//     runs-on: ubuntu-latest
//     timeout-minutes: 30
//     steps:
//       - name: Check out
//         uses: actions/checkout@v4
//       - name: Set up sbt
//         uses: sbt/setup-sbt@v1
//       - name: Test
//         run: sbt +test
```

The output follows a few fixed rules:

- Keys come in field order, and `Map` entries in the map's order. Field names are in kebab-case: `pullRequest` became `pull-request`.
- A `None` field is left out, so each step has only the keys it uses. A field with a default value is always written.
- The key `on` is quoted. YAML 1.1 readers read a bare `on`, `off`, `yes` or `no` as a boolean, so the writer quotes these words wherever they appear. GitHub reads `"on"` as the key `on`.
- Strings that would read back as something else are double-quoted: text that looks like a number, a boolean or `null`, text that starts with a YAML indicator such as `[`, `&` or `!`, and text that contains `: `, ` #` or a line break. `Int`, `Double` and `Boolean` fields are written as plain numbers and booleans.

Two value types are written differently from what you might expect. A string with line breaks is written as a double-quoted string with `\n` escapes, never as a `|` block, so a multi-line `run:` script is valid YAML in this form, just harder to read. `BigDecimal` and `BigInt` fields are written as quoted strings (`price: "49.99"`), so other YAML readers see a string, not a number; use `Double`, `Int` or `Long` for numbers that other tools must read as numbers.

## Control the Output Style

`YamlCodec#encodeToString` always uses the default style. To change it, encode the value to the `Yaml` tree with `YamlCodec#encodeValue`, then write the tree with `YamlWriter.write` and a `YamlOptions`:

```scala
val workflowTree: Yaml = workflowCodec.encodeValue(ciWorkflow)
```

To write the tree with four-space indentation and a leading `---` line:

```scala
println(YamlWriter.write(workflowTree, YamlOptions(indentStep = 4, documentMarkers = true)))
// ---
// name: CI
// "on":
//     push:
//         branches:
//             - main
//     pull-request:
//         branches:
//             - main
// jobs:
//     test:
//         runs-on: ubuntu-latest
//         timeout-minutes: 30
//         steps:
//             - name: Check out
//               uses: actions/checkout@v4
//             - name: Set up sbt
//               uses: sbt/setup-sbt@v1
//             - name: Test
//               run: sbt +test
```

`YamlOptions` has three settings:

| Setting           | Default | Effect                                                                     |
|-------------------|---------|----------------------------------------------------------------------------|
| `indentStep`      | `2`     | Spaces per nesting level                                                   |
| `documentMarkers` | `false` | Start the output with a `---` line                                         |
| `flowStyle`       | `false` | Write every mapping as `{k: v}` and every sequence as `[a, b]` on one line |

`YamlOptions.default`, `YamlOptions.pretty` (document markers on) and `YamlOptions.flow` (flow style on) are ready-made values, and `Yaml#print` and `Yaml#printPretty` are shortcuts for the first two. Flow style suits short values such as one step:

```scala
val testStep = Step("Test", run = Some("sbt +test"))
```

To write the step in flow style:

```scala
println(YamlWriter.write(Step.schema.derive(YamlFormat).encodeValue(testStep), YamlOptions.flow))
// {name: Test, run: sbt +test}
```

:::warning
In 0.0.56, flow style doesn't quote commas, `]` or `}` inside strings. A title such as `Scala, the Language` is written unquoted and reads back as `Scala`, with the rest lost. Use flow style only for values you know contain none of these characters, or check the output as shown in [Guard Against Silent Data Loss](#guard-against-silent-data-loss).
:::

## Build YAML Values in Code

The codec turns case classes into `Yaml` trees, but you can also build a tree directly. `Yaml` has four cases: `Yaml.Mapping`, `Yaml.Sequence`, `Yaml.Scalar` and `Yaml.NullValue`. A scalar holds its text and an optional `YamlTag`; the tag tells the writer that the text is a number or a boolean, so it isn't quoted:

```scala
val deployment: Yaml = Yaml.Mapping.fromStringKeys(
  "service"  -> Yaml.Scalar("bookstore-api"),
  "image"    -> Yaml.Scalar("ghcr.io/scala-books/bookstore-api:1.4.2"),
  "replicas" -> Yaml.Scalar("3", tag = Some(YamlTag.Int)),
  "public"   -> Yaml.Scalar("true", tag = Some(YamlTag.Bool)),
  "regions"  -> Yaml.Sequence(Yaml.Scalar("eu-central"), Yaml.Scalar("us-east"))
)
```

To print the tree in the default block style:

```scala
println(deployment.print)
// service: bookstore-api
// image: ghcr.io/scala-books/bookstore-api:1.4.2
// replicas: 3
// public: true
// regions:
//   - eu-central
//   - us-east
```

To read a tree, match on the four cases. `YamlReader.read` parses text into a tree, which is useful when the shape of a document isn't known in advance:

```scala
YamlReader.read(catalogYaml) match {
  case Yaml.Mapping(entries) => println(entries.map(_._1.print).mkString(", "))
  case other                 => println(s"Not a mapping: ${other.print}")
}
// store, books
```

On Scala 3, the `yaml"..."` interpolator builds a tree from YAML text with Scala values spliced in. Each spliced value needs a `Schema`, which the compiler checks:

```scala
val featuredTitle = "Zionomicon"
val featuredPrice = 49.99

val featured: Yaml =
  yaml"""title: $featuredTitle
price: $featuredPrice
note: "Our book of the month is $featuredTitle"
"""

case class Featured(title: String, price: Double, note: String)

object Featured {
  implicit val schema: Schema[Featured] = Schema.derived
}
```

To print the tree and decode it into `Featured`:

```scala
println(featured.print)
// title: Zionomicon
// price: "49.99"
// note: Our book of the month is Zionomicon
println(Featured.schema.derive(YamlFormat).decodeValue(featured))
// Featured(Zionomicon,49.99,Our book of the month is Zionomicon)
```

The interpolator builds the text and parses it again with `YamlReader`, so its result carries no tags: `price` is printed quoted, like any untyped scalar from the reader. The codec doesn't mind, because it reads scalars as text. This makes the interpolator handy for test inputs, and less so for output.

:::warning
The interpolator has known bugs in 0.0.56:

- On Scala 2.13 it doesn't compile in user code; the error says that `YamlInterpolatorRuntime` cannot be accessed. Build trees with the `Yaml` constructors there.
- Splice only scalar values (strings, numbers, booleans, dates). A spliced `List`, case class or `Yaml` tree is written in block style on the line of its key, so it is misread without an error: a list loses all but its first item and every line after it, and a record's fields move up to the parent mapping.
- Escape sequences in the literal are not processed: `yaml"title: Zionomicon\nprice: 49.99"` is one key whose value contains a backslash and an `n`. Write multi-line YAML in a triple-quoted literal, as above.
- The compile-time check covers the spliced types, not the YAML syntax: a literal with an unclosed `[` still compiles.
:::

## Convert Between YAML and JSON

`Yaml#toJson` converts a tree to a ZIO Blocks `Json` value, and `Yaml.fromJson` converts back. JSON to YAML keeps every type: JSON numbers and booleans become tagged scalars, so they are written unquoted, and JSON strings that look like numbers are quoted:

```scala

val bookJson: Json = Json.from(zionomicon)
```

To print the YAML that the JSON value produces:

```scala
println(Yaml.fromJson(bookJson).print)
// title: Zionomicon
// authors:
//   - John A. De Goes
//   - Adam Fraser
//   - Milad Khajavi
// isbn: "978-0-9862-0040-6"
// price: 49.990
// pageCount: 744
// tags:
//   - zio
//   - effects
// formats:
//   - Ebook
//   - Paperback
```

This tree came from the JSON codec, so the keys are the JSON field names (`pageCount`), not kebab-case. The price prints as `49.990` because of a known bug in 0.0.56: the JSON codec turns a `Double` into a `Json.Number` with an extra trailing zero. The value is the same.

YAML to JSON has to guess, because `YamlReader` keeps the text of each scalar but not whether it was quoted. `Yaml#toJson` makes a scalar a JSON number when its text starts with a digit or `-` and parses as a number, and a JSON boolean or `null` when its text is `true`, `false`, `null` or `~` (in lower, title or upper case). Everything else becomes a JSON string:

```scala
println(YamlReader.read("isbn: \"9780997148008\"\nversion: \"1.10\"\nimage-tag: 1.4.2\nin-stock: \"true\"\n").toJson.print)
// {"isbn":9780997148008,"version":1.10,"image-tag":"1.4.2","in-stock":true}
```

The quotes were lost: the quoted ISBN and version became numbers (`1.10` keeps its trailing zero only because the number is a `BigDecimal`), and the quoted `"true"` became a boolean. `1.4.2` isn't a number, so it stayed a string.

The conversion still has a use when the YAML keys are in camelCase, as in Kubernetes manifests: decoding the JSON with the JSON codec uses the field names as written, and honours `@Modifier.rename`. `Json#as[A]` decodes a `Json` value with the type's schema:

```scala
case class Container(name: String, image: String, imagePullPolicy: String, containerPort: Int)

object Container {
  implicit val schema: Schema[Container] = Schema.derived
}

val containerYaml =
  """name: bookstore-api
    |image: ghcr.io/scala-books/bookstore-api:1.4.2
    |imagePullPolicy: IfNotPresent
    |containerPort: 8080
    |""".stripMargin
```

To decode the YAML through the JSON codec, first with a valid container and then with a name that looks like a number:

```scala
println(YamlReader.read(containerYaml).toJson.as[Container])
// Right(Container(bookstore-api,ghcr.io/scala-books/bookstore-api:1.4.2,IfNotPresent,8080))
println(YamlReader.read(containerYaml.replace("name: bookstore-api", "name: \"2024\"")).toJson.as[Container].left.map(_.getMessage))
// Left(expected Json.String at: .name)
```

The second call, on a container whose name is `"2024"`, shows the limit of this route: a quoted value that looks like a number reaches the JSON codec as a number, and a `String` field rejects it. The error is loud, so no data is lost, but a file that quotes numeric strings can't be read this way.

[How to Write an OpenAPI Document and Export It as JSON or YAML](./write-and-export-openapi.md) uses `Yaml.fromJson` to export an OpenAPI document, and shows a workaround for the quoting bug described in the next section.

## What schema-yaml Does Not Support

`YamlReader` implements the parts of YAML that hand-written files use most: block and flow mappings and sequences, plain, single-quoted and double-quoted scalars, `|` and `>` blocks, comments, and one leading `---`. What it doesn't implement, it doesn't reject either. A sequence at the same indentation as its key is the form hand-written Kubernetes and docker-compose files use most. To see how `YamlReader` reads one:

```scala
println(YamlReader.read("steps:\n- name: Check out\n- name: Test\n"))
// Mapping(Chunk((Scalar(steps,None),NullValue),(Scalar(- name,None),Scalar(Check out,None)),(Scalar(- name,None),Scalar(Test,None))))
```

The table lists what 0.0.56 does with each unsupported form, as run:

| YAML form                                                                                      | What `YamlReader` does in 0.0.56                                                                              |
|------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------|
| A sequence at the same indentation as its key (`steps:` then `- name: a` on the next line)     | Reads `steps` as `null` and the items as odd keys named `- name`. Common in Kubernetes and docker-compose files. |
| A sequence nested directly in a sequence (`- - a`)                                             | Keeps the first item as the text `- a` and drops the rest of the document                                        |
| An anchor on a block value, and `<<` merge keys (`defaults: &defaults` plus an indented block) | Reads the anchor as text and drops the rest of the document                                                      |
| Anchors and aliases on scalars (`&a text`, `*a`)                                               | Keeps them as part of the text                                                                                   |
| Tags (`!!str 123`, `!Ref name`)                                                                | Keeps them as part of the text                                                                                   |
| A second document after `---` or `...`                                                         | Drops it                                                                                                         |
| A tab as indentation                                                                           | Moves the indented keys up to the parent level                                                                   |
| An unterminated `"`, `'`, `[` or `{`                                                           | Accepts it and drops the last character of the value                                                             |
| A line it can't read, such as `pages:744` with no space after the colon                        | Drops that line and every line after it                                                                          |
| The final line break of a literal or folded block                                              | Drops it, and ignores the chomping indicators `-` and `+` that ask to strip or keep line breaks                  |
| `yes`, `no`, `on`, `off`, `True`                                                               | Reads them as text: a `String` field keeps the word, a `Boolean` field accepts only `true` and `false`           |
| An empty value, `null` or `~` in a `String` field                                              | Decodes it as the empty string                                                                                   |

The writer has gaps of its own. It quotes most strings that need quoting, but in 0.0.56 it leaves these unquoted:

- A string that starts with `#`, such as an OpenAPI `$ref` (`#/components/schemas/Book`).
- A string that starts with `- ` or ends with `:`.
- A string with leading or trailing spaces.
- Commas and closing brackets in flow style (see [Control the Output Style](#control-the-output-style)).

The first two are read back correctly by `YamlReader`, which is lenient, but YAML readers that follow the spec misread them or reject the file: the `#` starts a comment, so the value is lost. Leading and trailing spaces are lost by every reader, including `YamlReader`.

## Guard Against Silent Data Loss

The codec can't tell you about data the reader never hands to it, so the defence starts before decoding. Three habits cover the cases above.

**Prefer required fields for data you can't lose.** A field without a default value and without `Option` fails the decode when its key is missing, so a dropped section becomes an error instead of a smaller value. `Catalog` and `Book` follow this rule, so the stray tab below at least causes an error:

```scala
report(catalogYaml.replace("    isbn: 978-0-9862-0040-6", "\tisbn: 978-0-9862-0040-6"))
// Missing required field: isbn at: .books.at(0)
```

A field with a default or an `Option` type would have quietly taken its default.

**Check that every key in the text was read.** A key that the reader drops is in the text but not in the tree. Comparing the keys of both catches every case in the table above that loses keys, and it costs a few lines:

```scala
val keyLine = """^\s*(?:-\s+)?([A-Za-z0-9_.-]+):(?:\s|$)""".r

def keysInText(text: String): List[String] =
  text.linesIterator.flatMap(line => keyLine.findPrefixMatchOf(line).map(_.group(1))).toList

def keysInTree(yaml: Yaml): List[String] = yaml match {
  case Yaml.Mapping(entries) =>
    entries.toList.flatMap {
      case (Yaml.Scalar(key, _), value) => key :: keysInTree(value)
      case (_, value)                   => keysInTree(value)
    }
  case Yaml.Sequence(elements) => elements.toList.flatMap(keysInTree)
  case _                       => Nil
}

def readChecked(text: String): Either[String, Yaml] = {
  val tree   = YamlReader.read(text)
  val unread = keysInText(text).diff(keysInTree(tree))
  if (unread.isEmpty) Right(tree)
  else Left(s"YAML not fully read, missing keys: ${unread.distinct.mkString(", ")}")
}

def decodeChecked[A](codec: YamlCodec[A], text: String): Either[String, A] =
  readChecked(text).flatMap { tree =>
    try Right(codec.decodeValue(tree))
    catch { case _: Exception => codec.decode(text).left.map(_.getMessage) }
  }
```

`List#diff` removes one occurrence per match, so a key that appears in several books is counted once per book. `decodeChecked` decodes the checked tree with `YamlCodec#decodeValue`, which throws on a decode error; in that case it decodes the text again with `YamlCodec#decode` to get the `SchemaError` message. Here it is on three documents that `YamlReader` reads without an error:

```scala
println(decodeChecked(catalogCodec, catalogYaml).map(_.books.map(_.title)))
// Right(List(Zionomicon, Programming in Scala))
println(decodeChecked(catalogCodec, catalogYaml.replace("    page-count: 744", "    page-count:744")))
// Left(YAML not fully read, missing keys: tags, formats, title, authors, isbn, price, page-count)
println(readChecked("steps:\n- name: Check out\n- name: Test\ntimeout-minutes: 30\n"))
// Left(YAML not fully read, missing keys: name)
```

The check matches keys at the start of a line only. It doesn't see keys inside flow mappings or quoted keys, and a `|` block whose lines look like `key: value` makes it report keys that were never meant to be keys. It catches dropped lines, not wrong values, so keep the codec's own checks too.

**Check what you write before you publish it.** Reading the output back with the codec catches the losses that `YamlReader` itself makes, such as leading spaces and commas in flow style. The strings that `YamlReader` reads correctly but other readers misread need a separate look:

```scala
def misreadByOtherReaders(yaml: Yaml): List[String] = yaml match {
  case Yaml.Mapping(entries)   => entries.toList.flatMap { case (key, value) => misreadByOtherReaders(key) ++ misreadByOtherReaders(value) }
  case Yaml.Sequence(elements) => elements.toList.flatMap(misreadByOtherReaders)
  case Yaml.Scalar(text, _)    =>
    if (text.startsWith("#") || text.startsWith("- ") || text == "-" || text.endsWith(":")) List(text) else Nil
  case Yaml.NullValue => Nil
}

def writeChecked[A](codec: YamlCodec[A], value: A, options: YamlOptions = YamlOptions.default): Either[String, String] = {
  val tree  = codec.encodeValue(value)
  val text  = YamlWriter.write(tree, options)
  val risky = misreadByOtherReaders(tree)
  if (risky.nonEmpty) Left(s"Other YAML readers would misread: ${risky.mkString(", ")}")
  else if (codec.decode(text) != Right(value)) Left("The YAML doesn't read back as the same value")
  else Right(text)
}
```

To run both checks, first on a plain book and then on three values that fail:

```scala
println(writeChecked(Book.schema.derive(YamlFormat), zionomicon).map(_ => "OK"))
// Right(OK)
println(writeChecked(Book.schema.derive(YamlFormat), zionomicon.copy(tags = List("#scala", "zio"))))
// Left(Other YAML readers would misread: #scala)
println(writeChecked(Book.schema.derive(YamlFormat), zionomicon.copy(title = "Zionomicon, 2nd edition"), YamlOptions.flow))
// Left(The YAML doesn't read back as the same value)
println(writeChecked(Book.schema.derive(YamlFormat), zionomicon.copy(title = "Zionomicon ")))
// Left(The YAML doesn't read back as the same value)
```

When the check fails, change the value (drop the trailing space), or write in block style instead of flow style. For `#` values in a document that must keep them, quote them after printing, as the OpenAPI guide does for `$ref`.

## Putting It Together

The complete example reads a catalog with the key check, reports its decode errors, and writes the CI workflow after checking the output:

```scala title="schema-examples/src/main/scala/readandwriteyaml/CompleteExample.scala"
package readandwriteyaml

/**
 * How to Read and Write YAML Documents — Complete Example
 *
 * Reads the catalog with the key check, reports decode errors for a damaged
 * copy, and writes the CI workflow through writeChecked and prints it.
 *
 * Run with: cd schema-examples && sbt "runMain
 * readandwriteyaml.CompleteExample"
 */
object CompleteExample {

  sealed trait Format

  object Format {
    case object Paperback extends Format
    case object Hardcover extends Format
    case object Ebook     extends Format

    val all: List[Format] = List(Paperback, Hardcover, Ebook)

    implicit val schema: Schema[Format] = Schema[String].transform[Format](
      name => all.find(_.toString == name).getOrElse(throw new IllegalArgumentException(s"Unknown format: $name")),
      _.toString
    )
  }

  case class Book(
    title: String,
    authors: List[String],
    isbn: String,
    price: Double,
    pageCount: Int,
    tags: List[String],
    formats: List[Format]
  )

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(store: String, books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalogCodec: YamlCodec[Catalog] = Catalog.schema.derive(YamlFormat)

  val catalogYaml: String =
    """# Books on the front table this month
      |store: Scala Books Berlin
      |books:
      |  - title: Zionomicon
      |    authors: [John A. De Goes, Adam Fraser, Milad Khajavi]
      |    isbn: 978-0-9862-0040-6
      |    price: 49.99
      |    page-count: 744
      |    tags: [zio, effects, concurrency]
      |    formats: [Ebook]
      |  - title: Programming in Scala
      |    authors:
      |      - Martin Odersky
      |      - Lex Spoon
      |      - Bill Venners
      |      - Frank Sommers
      |    isbn: "9780997148008"
      |    price: 59.95
      |    page-count: 852
      |    tags: [scala, language]
      |    formats:
      |      - Paperback
      |      - Ebook
      |""".stripMargin

  val zionomicon: Book = Book(
    title = "Zionomicon",
    authors = List("John A. De Goes", "Adam Fraser", "Milad Khajavi"),
    isbn = "978-0-9862-0040-6",
    price = 49.99,
    pageCount = 744,
    tags = List("zio", "effects"),
    formats = List(Format.Ebook, Format.Paperback)
  )

  case class Branches(branches: List[String])

  object Branches {
    implicit val schema: Schema[Branches] = Schema.derived
  }

  case class Triggers(push: Branches, pullRequest: Branches)

  object Triggers {
    implicit val schema: Schema[Triggers] = Schema.derived
  }

  case class Step(name: String, uses: Option[String] = None, run: Option[String] = None)

  object Step {
    implicit val schema: Schema[Step] = Schema.derived
  }

  case class Job(runsOn: String, timeoutMinutes: Int, steps: List[Step])

  object Job {
    implicit val schema: Schema[Job] = Schema.derived
  }

  case class Workflow(name: String, on: Triggers, jobs: Map[String, Job])

  object Workflow {
    implicit val schema: Schema[Workflow] = Schema.derived
  }

  val workflowCodec: YamlCodec[Workflow] = Workflow.schema.derive(YamlFormat)

  val ciWorkflow: Workflow = Workflow(
    name = "CI",
    on = Triggers(push = Branches(List("main")), pullRequest = Branches(List("main"))),
    jobs = Map(
      "test" -> Job(
        runsOn = "ubuntu-latest",
        timeoutMinutes = 30,
        steps = List(
          Step("Check out", uses = Some("actions/checkout@v4")),
          Step("Set up sbt", uses = Some("sbt/setup-sbt@v1")),
          Step("Test", run = Some("sbt +test"))
        )
      )
    )
  )

  val keyLine = """^\s*(?:-\s+)?([A-Za-z0-9_.-]+):(?:\s|$)""".r

  def keysInText(text: String): List[String] =
    text.linesIterator.flatMap(line => keyLine.findPrefixMatchOf(line).map(_.group(1))).toList

  def keysInTree(yaml: Yaml): List[String] = yaml match {
    case Yaml.Mapping(entries) =>
      entries.toList.flatMap {
        case (Yaml.Scalar(key, _), value) => key :: keysInTree(value)
        case (_, value)                   => keysInTree(value)
      }
    case Yaml.Sequence(elements) => elements.toList.flatMap(keysInTree)
    case _                       => Nil
  }

  def readChecked(text: String): Either[String, Yaml] = {
    val tree   = YamlReader.read(text)
    val unread = keysInText(text).diff(keysInTree(tree))
    if (unread.isEmpty) Right(tree)
    else Left(s"YAML not fully read, missing keys: ${unread.distinct.mkString(", ")}")
  }

  def decodeChecked[A](codec: YamlCodec[A], text: String): Either[String, A] =
    readChecked(text).flatMap { tree =>
      try Right(codec.decodeValue(tree))
      catch { case _: Exception => codec.decode(text).left.map(_.getMessage) }
    }

  def misreadByOtherReaders(yaml: Yaml): List[String] = yaml match {
    case Yaml.Mapping(entries) =>
      entries.toList.flatMap { case (key, value) => misreadByOtherReaders(key) ++ misreadByOtherReaders(value) }
    case Yaml.Sequence(elements) => elements.toList.flatMap(misreadByOtherReaders)
    case Yaml.Scalar(text, _)    =>
      if (text.startsWith("#") || text.startsWith("- ") || text == "-" || text.endsWith(":")) List(text) else Nil
    case Yaml.NullValue => Nil
  }

  def writeChecked[A](
    codec: YamlCodec[A],
    value: A,
    options: YamlOptions = YamlOptions.default
  ): Either[String, String] = {
    val tree  = codec.encodeValue(value)
    val text  = YamlWriter.write(tree, options)
    val risky = misreadByOtherReaders(tree)
    if (risky.nonEmpty) Left(s"Other YAML readers would misread: ${risky.mkString(", ")}")
    else if (codec.decode(text) != Right(value)) Left("The YAML doesn't read back as the same value")
    else Right(text)
  }

  def report(yaml: String): Unit =
    catalogCodec.decode(yaml) match {
      case Right(catalog) => println(s"OK: ${catalog.books.size} books")
      case Left(error)    => println(error.getMessage)
    }

  def main(args: Array[String]): Unit = {
    decodeChecked(catalogCodec, catalogYaml) match {
      case Right(catalog) =>
        catalog.books.foreach(book => println(s"${book.title}: ${book.authors.mkString(", ")}, ${book.price}"))
      case Left(message) => println(message)
    }

    report(catalogYaml.replace("price: 59.95", "price: cheap"))
    report(catalogYaml.replace("formats: [Ebook]", "formats: [Kindle]"))
    println(decodeChecked(catalogCodec, catalogYaml.replace("    page-count: 744", "    page-count:744")))

    writeChecked(workflowCodec, ciWorkflow) match {
      case Right(text)   => println(text)
      case Left(message) => println(message)
    }
  }
}
```

## Running the Examples

All examples in this guide have runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

Each example builds its YAML in memory, so it runs anywhere without setup.

### Read a YAML Catalog Into Case Classes

This example decodes the catalog with the derived codec and with the `String#fromYaml` extension method, and writes a book back out.

<details>
  <summary>schema-examples/src/main/scala/readandwriteyaml/Step1ReadCatalogExample.scala</summary>

```scala title="schema-examples/src/main/scala/readandwriteyaml/Step1ReadCatalogExample.scala" showLineNumbers
package readandwriteyaml

/**
 * How to Read and Write YAML Documents — Read a YAML Catalog Into Case Classes
 *
 * Decodes a bookstore catalog with a codec derived from the Schema, then uses
 * the YamlSyntax extension methods to write a book and read it back.
 *
 * Run with: cd schema-examples && sbt "runMain
 * readandwriteyaml.Step1ReadCatalogExample"
 */
object Step1ReadCatalogExample {

  sealed trait Format

  object Format {
    case object Paperback extends Format
    case object Hardcover extends Format
    case object Ebook     extends Format

    val all: List[Format] = List(Paperback, Hardcover, Ebook)

    implicit val schema: Schema[Format] = Schema[String].transform[Format](
      name => all.find(_.toString == name).getOrElse(throw new IllegalArgumentException(s"Unknown format: $name")),
      _.toString
    )
  }

  case class Book(
    title: String,
    authors: List[String],
    isbn: String,
    price: Double,
    pageCount: Int,
    tags: List[String],
    formats: List[Format]
  )

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(store: String, books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalogCodec: YamlCodec[Catalog] = Catalog.schema.derive(YamlFormat)

  val catalogYaml: String =
    """# Books on the front table this month
      |store: Scala Books Berlin
      |books:
      |  - title: Zionomicon
      |    authors: [John A. De Goes, Adam Fraser, Milad Khajavi]
      |    isbn: 978-0-9862-0040-6
      |    price: 49.99
      |    page-count: 744
      |    tags: [zio, effects, concurrency]
      |    formats: [Ebook]
      |  - title: Programming in Scala
      |    authors:
      |      - Martin Odersky
      |      - Lex Spoon
      |      - Bill Venners
      |      - Frank Sommers
      |    isbn: "9780997148008"
      |    price: 59.95
      |    page-count: 852
      |    tags: [scala, language]
      |    formats:
      |      - Paperback
      |      - Ebook
      |""".stripMargin

  val zionomicon: Book = Book(
    title = "Zionomicon",
    authors = List("John A. De Goes", "Adam Fraser", "Milad Khajavi"),
    isbn = "978-0-9862-0040-6",
    price = 49.99,
    pageCount = 744,
    tags = List("zio", "effects"),
    formats = List(Format.Ebook, Format.Paperback)
  )

  def main(args: Array[String]): Unit = {
    catalogCodec.decode(catalogYaml) match {
      case Right(catalog) =>
        catalog.books.foreach(book => println(s"${book.title}: ${book.authors.mkString(", ")}, ${book.price}"))
      case Left(error) => println(error.getMessage)
    }

    println(zionomicon.toYamlString)
    println(zionomicon.toYamlString.fromYaml[Book] == Right(zionomicon))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain readandwriteyaml.Step1ReadCatalogExample"
```

### Report Decode Errors

This example decodes four damaged catalogs and prints each error message with its path.

<details>
  <summary>schema-examples/src/main/scala/readandwriteyaml/Step2DecodeErrorsExample.scala</summary>

```scala title="schema-examples/src/main/scala/readandwriteyaml/Step2DecodeErrorsExample.scala" showLineNumbers
package readandwriteyaml

/**
 * How to Read and Write YAML Documents — Report Decode Errors
 *
 * Decodes four damaged copies of the catalog and prints each error message with
 * the path of the bad value.
 *
 * Run with: cd schema-examples && sbt "runMain
 * readandwriteyaml.Step2DecodeErrorsExample"
 */
object Step2DecodeErrorsExample {

  sealed trait Format

  object Format {
    case object Paperback extends Format
    case object Hardcover extends Format
    case object Ebook     extends Format

    val all: List[Format] = List(Paperback, Hardcover, Ebook)

    implicit val schema: Schema[Format] = Schema[String].transform[Format](
      name => all.find(_.toString == name).getOrElse(throw new IllegalArgumentException(s"Unknown format: $name")),
      _.toString
    )
  }

  case class Book(
    title: String,
    authors: List[String],
    isbn: String,
    price: Double,
    pageCount: Int,
    tags: List[String],
    formats: List[Format]
  )

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(store: String, books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalogCodec: YamlCodec[Catalog] = Catalog.schema.derive(YamlFormat)

  val catalogYaml: String =
    """# Books on the front table this month
      |store: Scala Books Berlin
      |books:
      |  - title: Zionomicon
      |    authors: [John A. De Goes, Adam Fraser, Milad Khajavi]
      |    isbn: 978-0-9862-0040-6
      |    price: 49.99
      |    page-count: 744
      |    tags: [zio, effects, concurrency]
      |    formats: [Ebook]
      |  - title: Programming in Scala
      |    authors:
      |      - Martin Odersky
      |      - Lex Spoon
      |      - Bill Venners
      |      - Frank Sommers
      |    isbn: "9780997148008"
      |    price: 59.95
      |    page-count: 852
      |    tags: [scala, language]
      |    formats:
      |      - Paperback
      |      - Ebook
      |""".stripMargin

  val zionomicon: Book = Book(
    title = "Zionomicon",
    authors = List("John A. De Goes", "Adam Fraser", "Milad Khajavi"),
    isbn = "978-0-9862-0040-6",
    price = 49.99,
    pageCount = 744,
    tags = List("zio", "effects"),
    formats = List(Format.Ebook, Format.Paperback)
  )

  def report(yaml: String): Unit =
    catalogCodec.decode(yaml) match {
      case Right(catalog) => println(s"OK: ${catalog.books.size} books")
      case Left(error)    => println(error.getMessage)
    }

  def main(args: Array[String]): Unit = {
    report(catalogYaml.replace("price: 59.95", "price: cheap"))
    report(catalogYaml.replace("page-count: 744", "page-count: many"))
    report(catalogYaml.replace("formats: [Ebook]", "formats: [Kindle]"))
    report(catalogYaml.replace("    isbn: \"9780997148008\"\n", ""))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain readandwriteyaml.Step2DecodeErrorsExample"
```

### Write a CI Workflow as YAML

This example writes a GitHub Actions workflow from case classes and reads it back.

<details>
  <summary>schema-examples/src/main/scala/readandwriteyaml/Step3WriteWorkflowExample.scala</summary>

```scala title="schema-examples/src/main/scala/readandwriteyaml/Step3WriteWorkflowExample.scala" showLineNumbers
package readandwriteyaml

/**
 * How to Read and Write YAML Documents — Write a CI Workflow as YAML
 *
 * Writes a GitHub Actions workflow from case classes with the derived codec,
 * then reads the text back and compares it with the original value.
 *
 * Run with: cd schema-examples && sbt "runMain
 * readandwriteyaml.Step3WriteWorkflowExample"
 */
object Step3WriteWorkflowExample {

  case class Branches(branches: List[String])

  object Branches {
    implicit val schema: Schema[Branches] = Schema.derived
  }

  case class Triggers(push: Branches, pullRequest: Branches)

  object Triggers {
    implicit val schema: Schema[Triggers] = Schema.derived
  }

  case class Step(name: String, uses: Option[String] = None, run: Option[String] = None)

  object Step {
    implicit val schema: Schema[Step] = Schema.derived
  }

  case class Job(runsOn: String, timeoutMinutes: Int, steps: List[Step])

  object Job {
    implicit val schema: Schema[Job] = Schema.derived
  }

  case class Workflow(name: String, on: Triggers, jobs: Map[String, Job])

  object Workflow {
    implicit val schema: Schema[Workflow] = Schema.derived
  }

  val workflowCodec: YamlCodec[Workflow] = Workflow.schema.derive(YamlFormat)

  val ciWorkflow: Workflow = Workflow(
    name = "CI",
    on = Triggers(push = Branches(List("main")), pullRequest = Branches(List("main"))),
    jobs = Map(
      "test" -> Job(
        runsOn = "ubuntu-latest",
        timeoutMinutes = 30,
        steps = List(
          Step("Check out", uses = Some("actions/checkout@v4")),
          Step("Set up sbt", uses = Some("sbt/setup-sbt@v1")),
          Step("Test", run = Some("sbt +test"))
        )
      )
    )
  )

  def main(args: Array[String]): Unit = {
    val text = workflowCodec.encodeToString(ciWorkflow)
    println(text)
    println(workflowCodec.decode(text) == Right(ciWorkflow))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain readandwriteyaml.Step3WriteWorkflowExample"
```

### Control the Output Style

This example writes the same workflow with a wider indentation and document markers, and one step in flow style.

<details>
  <summary>schema-examples/src/main/scala/readandwriteyaml/Step4OutputStyleExample.scala</summary>

```scala title="schema-examples/src/main/scala/readandwriteyaml/Step4OutputStyleExample.scala" showLineNumbers
package readandwriteyaml

/**
 * How to Read and Write YAML Documents — Control the Output Style
 *
 * Writes the CI workflow with a wider indentation and document markers, and one
 * step in flow style.
 *
 * Run with: cd schema-examples && sbt "runMain
 * readandwriteyaml.Step4OutputStyleExample"
 */
object Step4OutputStyleExample {

  case class Branches(branches: List[String])

  object Branches {
    implicit val schema: Schema[Branches] = Schema.derived
  }

  case class Triggers(push: Branches, pullRequest: Branches)

  object Triggers {
    implicit val schema: Schema[Triggers] = Schema.derived
  }

  case class Step(name: String, uses: Option[String] = None, run: Option[String] = None)

  object Step {
    implicit val schema: Schema[Step] = Schema.derived
  }

  case class Job(runsOn: String, timeoutMinutes: Int, steps: List[Step])

  object Job {
    implicit val schema: Schema[Job] = Schema.derived
  }

  case class Workflow(name: String, on: Triggers, jobs: Map[String, Job])

  object Workflow {
    implicit val schema: Schema[Workflow] = Schema.derived
  }

  val workflowCodec: YamlCodec[Workflow] = Workflow.schema.derive(YamlFormat)

  val ciWorkflow: Workflow = Workflow(
    name = "CI",
    on = Triggers(push = Branches(List("main")), pullRequest = Branches(List("main"))),
    jobs = Map(
      "test" -> Job(
        runsOn = "ubuntu-latest",
        timeoutMinutes = 30,
        steps = List(
          Step("Check out", uses = Some("actions/checkout@v4")),
          Step("Set up sbt", uses = Some("sbt/setup-sbt@v1")),
          Step("Test", run = Some("sbt +test"))
        )
      )
    )
  )

  def main(args: Array[String]): Unit = {
    val workflowTree: Yaml = workflowCodec.encodeValue(ciWorkflow)

    println(YamlWriter.write(workflowTree, YamlOptions(indentStep = 4, documentMarkers = true)))

    val testStep = Step("Test", run = Some("sbt +test"))

    println(YamlWriter.write(Step.schema.derive(YamlFormat).encodeValue(testStep), YamlOptions.flow))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain readandwriteyaml.Step4OutputStyleExample"
```

### Build YAML Values in Code

This example builds a tree with the `Yaml` constructors, walks a parsed tree, and builds a test input with the `yaml"..."` interpolator. Because of the interpolator, this file compiles on Scala 3 only.

<details>
  <summary>schema-examples/src/main/scala/readandwriteyaml/Step5BuildYamlInCodeExample.scala</summary>

```scala title="schema-examples/src/main/scala/readandwriteyaml/Step5BuildYamlInCodeExample.scala" showLineNumbers
package readandwriteyaml

/**
 * How to Read and Write YAML Documents — Build YAML Values in Code
 *
 * Builds a tree with the Yaml constructors, walks a parsed tree, and builds a
 * test input with the yaml interpolator. Scala 3 only: the yaml"..."
 * interpolator does not compile in user code on Scala 2.13.
 *
 * Run with: cd schema-examples && sbt "runMain
 * readandwriteyaml.Step5BuildYamlInCodeExample"
 */
object Step5BuildYamlInCodeExample {

  sealed trait Format

  object Format {
    case object Paperback extends Format
    case object Hardcover extends Format
    case object Ebook     extends Format

    val all: List[Format] = List(Paperback, Hardcover, Ebook)

    implicit val schema: Schema[Format] = Schema[String].transform[Format](
      name => all.find(_.toString == name).getOrElse(throw new IllegalArgumentException(s"Unknown format: $name")),
      _.toString
    )
  }

  case class Book(
    title: String,
    authors: List[String],
    isbn: String,
    price: Double,
    pageCount: Int,
    tags: List[String],
    formats: List[Format]
  )

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(store: String, books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalogCodec: YamlCodec[Catalog] = Catalog.schema.derive(YamlFormat)

  val catalogYaml: String =
    """# Books on the front table this month
      |store: Scala Books Berlin
      |books:
      |  - title: Zionomicon
      |    authors: [John A. De Goes, Adam Fraser, Milad Khajavi]
      |    isbn: 978-0-9862-0040-6
      |    price: 49.99
      |    page-count: 744
      |    tags: [zio, effects, concurrency]
      |    formats: [Ebook]
      |  - title: Programming in Scala
      |    authors:
      |      - Martin Odersky
      |      - Lex Spoon
      |      - Bill Venners
      |      - Frank Sommers
      |    isbn: "9780997148008"
      |    price: 59.95
      |    page-count: 852
      |    tags: [scala, language]
      |    formats:
      |      - Paperback
      |      - Ebook
      |""".stripMargin

  case class Featured(title: String, price: Double, note: String)

  object Featured {
    implicit val schema: Schema[Featured] = Schema.derived
  }

  def main(args: Array[String]): Unit = {
    val deployment: Yaml = Yaml.Mapping.fromStringKeys(
      "service"  -> Yaml.Scalar("bookstore-api"),
      "image"    -> Yaml.Scalar("ghcr.io/scala-books/bookstore-api:1.4.2"),
      "replicas" -> Yaml.Scalar("3", tag = Some(YamlTag.Int)),
      "public"   -> Yaml.Scalar("true", tag = Some(YamlTag.Bool)),
      "regions"  -> Yaml.Sequence(Yaml.Scalar("eu-central"), Yaml.Scalar("us-east"))
    )

    println(deployment.print)

    YamlReader.read(catalogYaml) match {
      case Yaml.Mapping(entries) => println(entries.map(_._1.print).mkString(", "))
      case other                 => println(s"Not a mapping: ${other.print}")
    }

    val featuredTitle = "Zionomicon"
    val featuredPrice = 49.99

    val featured: Yaml =
      yaml"""title: $featuredTitle
price: $featuredPrice
note: "Our book of the month is $featuredTitle"
"""

    println(featured.print)

    println(Featured.schema.derive(YamlFormat).decodeValue(featured))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain readandwriteyaml.Step5BuildYamlInCodeExample"
```

### Convert Between YAML and JSON

This example converts a book from JSON to YAML, shows how quoted scalars change type on the way to JSON, and reads a camelCase container spec through JSON.

<details>
  <summary>schema-examples/src/main/scala/readandwriteyaml/Step6YamlAndJsonExample.scala</summary>

```scala title="schema-examples/src/main/scala/readandwriteyaml/Step6YamlAndJsonExample.scala" showLineNumbers
package readandwriteyaml

/**
 * How to Read and Write YAML Documents — Convert Between YAML and JSON
 *
 * Converts a book from JSON to YAML, shows how quoted scalars change type on
 * the way to JSON, and reads a camelCase container spec through JSON.
 *
 * Run with: cd schema-examples && sbt "runMain
 * readandwriteyaml.Step6YamlAndJsonExample"
 */
object Step6YamlAndJsonExample {

  sealed trait Format

  object Format {
    case object Paperback extends Format
    case object Hardcover extends Format
    case object Ebook     extends Format

    val all: List[Format] = List(Paperback, Hardcover, Ebook)

    implicit val schema: Schema[Format] = Schema[String].transform[Format](
      name => all.find(_.toString == name).getOrElse(throw new IllegalArgumentException(s"Unknown format: $name")),
      _.toString
    )
  }

  case class Book(
    title: String,
    authors: List[String],
    isbn: String,
    price: Double,
    pageCount: Int,
    tags: List[String],
    formats: List[Format]
  )

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(store: String, books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val zionomicon: Book = Book(
    title = "Zionomicon",
    authors = List("John A. De Goes", "Adam Fraser", "Milad Khajavi"),
    isbn = "978-0-9862-0040-6",
    price = 49.99,
    pageCount = 744,
    tags = List("zio", "effects"),
    formats = List(Format.Ebook, Format.Paperback)
  )

  case class Container(name: String, image: String, imagePullPolicy: String, containerPort: Int)

  object Container {
    implicit val schema: Schema[Container] = Schema.derived
  }

  val containerYaml: String =
    """name: bookstore-api
      |image: ghcr.io/scala-books/bookstore-api:1.4.2
      |imagePullPolicy: IfNotPresent
      |containerPort: 8080
      |""".stripMargin

  def main(args: Array[String]): Unit = {
    val bookJson: Json = Json.from(zionomicon)

    println(Yaml.fromJson(bookJson).print)

    println(
      YamlReader.read("isbn: \"9780997148008\"\nversion: \"1.10\"\nimage-tag: 1.4.2\nin-stock: \"true\"\n").toJson.print
    )

    println(YamlReader.read(containerYaml).toJson.as[Container])
    println(
      YamlReader
        .read(containerYaml.replace("name: bookstore-api", "name: \"2024\""))
        .toJson
        .as[Container]
        .left
        .map(_.getMessage)
    )
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain readandwriteyaml.Step6YamlAndJsonExample"
```

### Guard Against Silent Data Loss

This example runs the key check on documents that `YamlReader` reads only in part, and the output check on values the writer doesn't quote.

<details>
  <summary>schema-examples/src/main/scala/readandwriteyaml/Step7GuardAgainstDataLossExample.scala</summary>

```scala title="schema-examples/src/main/scala/readandwriteyaml/Step7GuardAgainstDataLossExample.scala" showLineNumbers
package readandwriteyaml

/**
 * How to Read and Write YAML Documents — Guard Against Silent Data Loss
 *
 * Runs the key check on documents that YamlReader reads only in part, and the
 * output check on values the writer does not quote.
 *
 * Run with: cd schema-examples && sbt "runMain
 * readandwriteyaml.Step7GuardAgainstDataLossExample"
 */
object Step7GuardAgainstDataLossExample {

  sealed trait Format

  object Format {
    case object Paperback extends Format
    case object Hardcover extends Format
    case object Ebook     extends Format

    val all: List[Format] = List(Paperback, Hardcover, Ebook)

    implicit val schema: Schema[Format] = Schema[String].transform[Format](
      name => all.find(_.toString == name).getOrElse(throw new IllegalArgumentException(s"Unknown format: $name")),
      _.toString
    )
  }

  case class Book(
    title: String,
    authors: List[String],
    isbn: String,
    price: Double,
    pageCount: Int,
    tags: List[String],
    formats: List[Format]
  )

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class Catalog(store: String, books: List[Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  val catalogCodec: YamlCodec[Catalog] = Catalog.schema.derive(YamlFormat)

  val catalogYaml: String =
    """# Books on the front table this month
      |store: Scala Books Berlin
      |books:
      |  - title: Zionomicon
      |    authors: [John A. De Goes, Adam Fraser, Milad Khajavi]
      |    isbn: 978-0-9862-0040-6
      |    price: 49.99
      |    page-count: 744
      |    tags: [zio, effects, concurrency]
      |    formats: [Ebook]
      |  - title: Programming in Scala
      |    authors:
      |      - Martin Odersky
      |      - Lex Spoon
      |      - Bill Venners
      |      - Frank Sommers
      |    isbn: "9780997148008"
      |    price: 59.95
      |    page-count: 852
      |    tags: [scala, language]
      |    formats:
      |      - Paperback
      |      - Ebook
      |""".stripMargin

  val zionomicon: Book = Book(
    title = "Zionomicon",
    authors = List("John A. De Goes", "Adam Fraser", "Milad Khajavi"),
    isbn = "978-0-9862-0040-6",
    price = 49.99,
    pageCount = 744,
    tags = List("zio", "effects"),
    formats = List(Format.Ebook, Format.Paperback)
  )

  val keyLine = """^\s*(?:-\s+)?([A-Za-z0-9_.-]+):(?:\s|$)""".r

  def keysInText(text: String): List[String] =
    text.linesIterator.flatMap(line => keyLine.findPrefixMatchOf(line).map(_.group(1))).toList

  def keysInTree(yaml: Yaml): List[String] = yaml match {
    case Yaml.Mapping(entries) =>
      entries.toList.flatMap {
        case (Yaml.Scalar(key, _), value) => key :: keysInTree(value)
        case (_, value)                   => keysInTree(value)
      }
    case Yaml.Sequence(elements) => elements.toList.flatMap(keysInTree)
    case _                       => Nil
  }

  def readChecked(text: String): Either[String, Yaml] = {
    val tree   = YamlReader.read(text)
    val unread = keysInText(text).diff(keysInTree(tree))
    if (unread.isEmpty) Right(tree)
    else Left(s"YAML not fully read, missing keys: ${unread.distinct.mkString(", ")}")
  }

  def decodeChecked[A](codec: YamlCodec[A], text: String): Either[String, A] =
    readChecked(text).flatMap { tree =>
      try Right(codec.decodeValue(tree))
      catch { case _: Exception => codec.decode(text).left.map(_.getMessage) }
    }

  def misreadByOtherReaders(yaml: Yaml): List[String] = yaml match {
    case Yaml.Mapping(entries) =>
      entries.toList.flatMap { case (key, value) => misreadByOtherReaders(key) ++ misreadByOtherReaders(value) }
    case Yaml.Sequence(elements) => elements.toList.flatMap(misreadByOtherReaders)
    case Yaml.Scalar(text, _)    =>
      if (text.startsWith("#") || text.startsWith("- ") || text == "-" || text.endsWith(":")) List(text) else Nil
    case Yaml.NullValue => Nil
  }

  def writeChecked[A](
    codec: YamlCodec[A],
    value: A,
    options: YamlOptions = YamlOptions.default
  ): Either[String, String] = {
    val tree  = codec.encodeValue(value)
    val text  = YamlWriter.write(tree, options)
    val risky = misreadByOtherReaders(tree)
    if (risky.nonEmpty) Left(s"Other YAML readers would misread: ${risky.mkString(", ")}")
    else if (codec.decode(text) != Right(value)) Left("The YAML doesn't read back as the same value")
    else Right(text)
  }

  def report(yaml: String): Unit =
    catalogCodec.decode(yaml) match {
      case Right(catalog) => println(s"OK: ${catalog.books.size} books")
      case Left(error)    => println(error.getMessage)
    }

  def main(args: Array[String]): Unit = {
    report(catalogYaml.replace("    isbn: 978-0-9862-0040-6", "\tisbn: 978-0-9862-0040-6"))

    println(decodeChecked(catalogCodec, catalogYaml).map(_.books.map(_.title)))
    println(decodeChecked(catalogCodec, catalogYaml.replace("    page-count: 744", "    page-count:744")))
    println(readChecked("steps:\n- name: Check out\n- name: Test\ntimeout-minutes: 30\n"))

    println(writeChecked(Book.schema.derive(YamlFormat), zionomicon).map(_ => "OK"))
    println(writeChecked(Book.schema.derive(YamlFormat), zionomicon.copy(tags = List("#scala", "zio"))))
    println(
      writeChecked(Book.schema.derive(YamlFormat), zionomicon.copy(title = "Zionomicon, 2nd edition"), YamlOptions.flow)
    )
    println(writeChecked(Book.schema.derive(YamlFormat), zionomicon.copy(title = "Zionomicon ")))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain readandwriteyaml.Step7GuardAgainstDataLossExample"
```

### The Complete Example

Run the complete example from the `schema-examples` directory:

```bash
sbt "runMain readandwriteyaml.CompleteExample"
```

### Compile All Examples

To compile every example in the project without running any of them, run this from the `schema-examples` directory:

```bash
sbt compile
```

## Going Further

- The [YAML reference page](../reference/schema/built-in-codecs/yaml.md) lists the primitive types the codec supports and the methods of `YamlCodec`, `Yaml` and `YamlOptions`.
- [One Type, Many Formats](./one-type-many-formats.md) derives JSON, YAML, XML, TOON, MessagePack, Avro and CSV codecs from one schema.
- [Build a Typed Configuration Loader](./typed-config-loader.md) loads application settings from YAML with environment overrides and defaults, through the `config-yaml` module, which uses the same `YamlReader`.
- [How to Read and Write CSV and TSV Files](./read-and-write-csv-and-tsv-files.md) does the same job for tabular data.

---

## Coming from cats-effect Resource, ZIO Scope/ZLayer, macwire or Using

This guide moves a small bookstore service from cats-effect `Resource`, ZIO `Scope` and `ZLayer`, macwire, or `scala.util.Using` to the ZIO Blocks [`scope` module](../reference/resource-management/index.md). By the end, the service's database connection and export file are opened, shared and closed by `Scope`, `Resource` and `Wire`, and the compiler rejects any attempt to use them after they close. Each source library gets its own section with the same three parts: the code you wrote, the code you write now, and what changes. Read the concept map, then jump to the section for the library you are leaving.

- [Concept Map](#concept-map)
- [Coming from cats-effect `Resource`](#coming-from-cats-effect-resource)
- [Coming from ZIO `Scope` and `ZLayer`](#coming-from-zio-scope-and-zlayer)
- [Coming from macwire](#coming-from-macwire)
- [Coming from `scala.util.Using`](#coming-from-scalautilusing)
- [What You Gain and What Is Different](#what-you-gain-and-what-is-different)
- [Migration Checklist](#migration-checklist)

## The Problem

All four libraries close resources for you, but none of them stops you from keeping a resource after it closes. A value returned from `use`, `ZIO.scoped` or `Using.resource` compiles fine and fails only when someone calls it. On top of that, lifecycle and wiring live in different places: macwire builds the object graph but closes nothing, `Using` closes things but builds no graph, and the effect systems need an effect type around every constructor.

Here is the leak with `scala.util.Using`, which needs nothing outside the standard library:

```scala

final class Connection extends AutoCloseable {
  private val open = new AtomicBoolean(true)
  def query(sql: String): String = {
    require(open.get, "connection is closed")
    s"rows for $sql"
  }
  def close(): Unit = open.set(false)
}

// Compiles: hands back a connection that is already closed
val leaked: Connection = Using.resource(new Connection)(identity)
```

The same shape compiles with `db.use(IO.pure)` in cats-effect and with `ZIO.scoped(ZIO.fromAutoCloseable(...))` returning the resource in ZIO. Every one of them turns into a runtime failure later.

ZIO Blocks Scope ties each resource to a scope with a type, `$[A]`, and lets a value leave the scope only when it has an `Unscoped` instance, which resources do not. `Resource.from` wires a graph from constructors and closes every `AutoCloseable` in it, in reverse order, when the scope closes.

## Prerequisites

Add the `scope` module:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-scope" % "0.0.56"
```

For Scala.js, use `%%%`. The module supports Scala 2.13 and 3.x. It has no third-party dependencies (it pulls in only the ZIO Blocks `context` and `chunk` modules) and does not depend on cats-effect, ZIO or any other effect library.

Every ZIO Blocks snippet below starts from this import:

```scala

```

This guide assumes you already know one of the four source libraries. It maps their concepts onto ZIO Blocks rather than teaching `Scope` from scratch, so if `Scope#scoped`, `Scope#allocate` or `Scope#$` are new to you, work through [Compile-Time Resource Safety with Scope](./compile-time-resource-safety-with-scope.md) first.

## The Bookstore Model

The running example is a bookstore service with this graph:

```text
DbConfig ──► Db ──┬──► InventoryRepo ──┐
                  └──► PriceService ───┴──► Catalog

ExportFile (a second, independent resource)
```

`Db` is an `AutoCloseable` connection that both `InventoryRepo` and `PriceService` share. `Catalog` sits on top. `ExportFile` is a temporary CSV file that the service writes to and deletes when it is done. Each resource prints a line when it opens and closes, so the order is visible:

```scala

final case class DbConfig(url: String)

final class Db(config: DbConfig) extends AutoCloseable {
  println(s"Db: connected to ${config.url}")

  private val stock: Map[String, Int] = Map(
    "Zionomicon"                      -> 3,
    "Programming in Scala"            -> 7,
    "Functional Programming in Scala" -> 0
  )

  private val prices: Map[String, BigDecimal] = Map(
    "Zionomicon"                      -> BigDecimal("49.99"),
    "Programming in Scala"            -> BigDecimal("59.95"),
    "Functional Programming in Scala" -> BigDecimal("44.99")
  )

  def stockOf(title: String): Int        = stock.getOrElse(title, 0)
  def priceOf(title: String): BigDecimal = prices.getOrElse(title, BigDecimal(0))
  def close(): Unit                      = println("Db: connection closed")
}

final class InventoryRepo(db: Db) {
  def inStock(title: String): Int = db.stockOf(title)
}

final class PriceService(db: Db) {
  def price(title: String): BigDecimal = db.priceOf(title)
}

final class Catalog(inventory: InventoryRepo, prices: PriceService) {
  def describe(title: String): String =
    s"$title: ${inventory.inStock(title)} in stock at $$${prices.price(title)}"
}

final class ExportFile(prefix: String) extends AutoCloseable {
  private val path: Path = Files.createTempFile(prefix, ".csv")
  println("ExportFile: created")

  def write(line: String): Unit = {
    Files.write(path, (line + "\n").getBytes(StandardCharsets.UTF_8), StandardOpenOption.APPEND)
    ()
  }

  def lineCount: Int = Files.readAllLines(path).size

  def close(): Unit = {
    Files.deleteIfExists(path)
    println("ExportFile: deleted")
  }
}
```

`Db` and `ExportFile` are plain classes, not case classes. That matters on Scala 3, where every case class whose fields have `Unscoped` instances gets one too, so a case class resource could leave its scope. Keep resources as plain classes.

## Concept Map

| Task                          | cats-effect 3                          | ZIO 2                                    | macwire 2                     | `scala.util.Using`                  | ZIO Blocks                                         |
|-------------------------------|----------------------------------------|------------------------------------------|-------------------------------|-------------------------------------|----------------------------------------------------|
| Describe acquire and release  | `Resource.make(acq)(rel)`              | `ZIO.acquireRelease(acq)(rel)`           | none                          | a `Releasable[R]` instance          | `Resource.acquireRelease(acq)(rel)`                |
| Wrap an `AutoCloseable`       | `Resource.fromAutoCloseable(IO(r))`    | `ZIO.fromAutoCloseable(ZIO.attempt(r))`  | none                          | built in                            | `Resource.fromAutoCloseable(r)` or `allocate(r)`   |
| Use one resource              | `r.use(f)`                             | `ZIO.scoped(effect)`                     | none                          | `Using.resource(r)(f)`              | `r.use(f)`                                         |
| Several resources, one block  | `for` over resources, then `use`       | `ZIO.scoped` around a `for`              | none                          | `Using.Manager { use => ... }`      | `Scope.global.scoped { scope => ... }`             |
| Compose descriptions          | `map`, `flatMap`                       | `map`, `flatMap`                         | none                          | none                                | `map`, `flatMap`, `zip`                            |
| Acquire in parallel           | `both`, `parTupled`                    | `zipPar`                                 | none                          | none                                | not supported; `zip` is sequential                 |
| Add a finalizer               | `Resource.onFinalize(fin)`             | `ZIO.addFinalizer(fin)`                  | none                          | none                                | `defer(fin)`, cancel with `DeferHandle.cancel()`   |
| Manage the lifetime by hand   | `allocated`                            | `Scope.make`, then `close(exit)`         | none                          | none                                | `open()`, then `OpenScope.close()`                 |
| Several finalizers fail       | first error wins, rest reported        | all kept in the `Cause`                  | none                          | one thrown, rest suppressed         | first failure thrown, rest suppressed              |
| Stop a resource escaping      | no                                     | no                                       | no                            | no                                  | compile error unless the type is `Unscoped`        |
| Build from a constructor      | none                                   | `ZLayer.fromFunction`, `ZLayer.derive`   | `wire[A]`                     | none                                | `Wire.shared[A]`, `Wire.unique[A]`                 |
| Wire a whole graph            | none                                   | `ZLayer.make[A](layers*)`                | `autowire[A](deps*)` (Scala 3) | none                                | `Resource.from[A](wires*)`                         |
| Shared or fresh instances     | none                                   | shared by default, `ZLayer#fresh`        | one `lazy val` per instance   | none                                | `Wire.shared` (default), `Wire.unique`             |
| Close services in the graph   | through `Resource`                     | through `ZLayer.scoped`                  | by hand                       | by hand                             | automatic for `AutoCloseable`                      |

## Coming from cats-effect `Resource`

cats-effect `Resource[F, A]` and ZIO Blocks `Resource[A]` are both lazy descriptions of acquire and release that compose with `flatMap` and run with `use`. The ZIO Blocks one has no `F[_]`: acquisition and release are plain, synchronous Scala.

### You Wrote This

This is the old code, cats-effect 3 (not compiled here):

```scala

object Bookstore extends IOApp.Simple {
  val db: Resource[IO, Db] =
    Resource.fromAutoCloseable(IO(new Db(DbConfig("jdbc:h2:mem:bookstore"))))

  val exportFile: Resource[IO, Path] =
    Resource.make(IO.blocking(Files.createTempFile("catalog", ".csv")))(p =>
      IO.blocking(Files.deleteIfExists(p)).void
    )

  // released in reverse order: exportFile first, then db
  val resources: Resource[IO, (Db, Path)] =
    for {
      d <- db
      f <- exportFile
    } yield (d, f)

  val run: IO[Unit] =
    resources.use { case (d, file) =>
      IO.blocking(Files.writeString(file, s"Zionomicon,${d.stockOf("Zionomicon")}")) *>
        IO.println(s"exported to $file")
    }
}
```

### You Write This

Drop the `IO` wrappers. `Resource.make` becomes `Resource.acquireRelease`, `Resource.fromAutoCloseable` keeps its name, and the `for` comprehension and `use` work the same way:

```scala
val db: Resource[Db] =
  Resource.fromAutoCloseable(new Db(DbConfig("jdbc:h2:mem:bookstore")))

val exportPath: Resource[Path] =
  Resource.acquireRelease(Files.createTempFile("catalog", ".csv")) { p =>
    Files.deleteIfExists(p)
    println("export path: deleted")
  }

val resources: Resource[(Db, Path)] =
  for {
    d <- db
    f <- exportPath
  } yield (d, f)
```

Nothing has run yet. `use` acquires both, runs the function, and releases them in reverse order:

```scala
val linesWritten: Int = resources.use { case (d, file) =>
  Files.write(file, s"Zionomicon,${d.stockOf("Zionomicon")}".getBytes(StandardCharsets.UTF_8))
  Files.readAllLines(file).size
}
// Db: connected to jdbc:h2:mem:bookstore
// export path: deleted
// Db: connection closed
// linesWritten: Int = 1
```

`linesWritten` is an `Int`, so it may leave `use`. The resource itself may not. This is the cats-effect `db.use(IO.pure)` leak, and it does not compile:

```scala
val escaped: Db = db.use(d => d)
// error:
// No given instance of type zio.blocks.scope.Unscoped[repl.MdocSession.MdocApp.Db] was found for parameter ev of method use in trait Resource.
// I found:
// 
//     zio.blocks.scope.Unscoped.derived[repl.MdocSession.MdocApp.Db](
//       /* missing */summon[scala.deriving.Mirror.Of[repl.MdocSession.MdocApp.Db]])
// 
// But Failed to synthesize an instance of type scala.deriving.Mirror.Of[repl.MdocSession.MdocApp.Db]:
// 	* class Db is not a generic product because it is not a case class
// 	* class Db is not a generic sum because it is not a sealed class.
// val escaped: Db = db.use(d => d)
//                                ^
```

### What Changes

- **No effect type.** `Resource[A]` has one type parameter. Acquire and release are by-name Scala expressions, so `IO(...)`, `IO.blocking(...)` and `Resource.eval` go away: call the code directly.
- **The result of `use` must be `Unscoped`.** `use[B](f: A => B)` requires an `Unscoped[B]` instance. `Int`, `String`, collections of them and Scala 3 case classes built from them have one. A plain class such as `Db` does not, so it cannot escape.
- **No parallel acquisition.** `both` and `parTupled` have no equivalent. `zip` acquires left then right and releases right then left.
- **No cancellation or async.** A ZIO Blocks scope is synchronous and runs on the thread that opened it. If you rely on fibers, interruption or `Resource[IO, A]` inside a concurrent program, keep cats-effect for that part.
- **Finalizer errors are kept.** In cats-effect, when several releases fail, the first error wins and the rest go to the runtime's `reportFailure`. A ZIO Blocks scope throws the first failure and attaches the others as suppressed exceptions. The [`scala.util.Using` section](#finalizer-failures) shows this.
- **`allocated` becomes `open()`.** `Scope.global.open()` returns an `OpenScope` whose `close()` runs its finalizers and returns a `Finalization`, with no `F[Unit]` to forget.

## Coming from ZIO `Scope` and `ZLayer`

ZIO splits resource handling into two pieces: `ZIO.acquireRelease` plus `ZIO.scoped` for local resources, and `ZLayer` for the service graph. ZIO Blocks has the same split: `Scope#scoped` plus `Scope#allocate` for local resources, and `Wire` plus `Resource.from` for the graph.

### Scoped Resources

A local resource in ZIO lives inside `ZIO.scoped`; in ZIO Blocks it lives inside `Scope.global.scoped`.

#### You Wrote This

This is the old code, ZIO 2 (not compiled here):

```scala

val program: Task[Int] =
  ZIO.scoped {
    for {
      db   <- ZIO.fromAutoCloseable(ZIO.attempt(new Db(DbConfig("jdbc:h2:mem:bookstore"))))
      file <- ZIO.fromAutoCloseable(ZIO.attempt(new ExportFile("catalog")))
      _    <- ZIO.addFinalizer(ZIO.succeed(println("export finished")))
      _    <- ZIO.attempt(file.write(s"Zionomicon,${db.stockOf("Zionomicon")}"))
    } yield file.lineCount
  }
```

#### You Write This

`ZIO.scoped` becomes `Scope.global.scoped`, `ZIO.fromAutoCloseable` becomes `Scope#allocate`, and `ZIO.addFinalizer` becomes `defer`. Values come back as `$[A]`, and `$` opens them for one call:

```scala
val exportedLines: Int = Scope.global.scoped { scope =>

  val db   = allocate(new Db(DbConfig("jdbc:h2:mem:bookstore")))
  val file = allocate(new ExportFile("catalog"))
  defer(println("export finished"))

  $(db, file)((d, f) => f.write(s"Zionomicon,${d.stockOf("Zionomicon")}"))
  $(file)(_.lineCount)
}
// Db: connected to jdbc:h2:mem:bookstore
// ExportFile: created
// export finished
// ExportFile: deleted
// Db: connection closed
// exportedLines: Int = 1
```

`defer` registered last, so it ran first. Then the export file closed, then the database, in reverse order of acquisition, as in ZIO.

### The Service Graph

A ZIO service graph is a set of `ZLayer`s assembled by `ZLayer.make`; in ZIO Blocks it is one `Resource.from` call.

#### You Wrote This

This is the old code, ZIO 2 (not compiled here):

```scala

val dbLayer: ZLayer[DbConfig, Throwable, Db] =
  ZLayer.scoped[DbConfig](ZIO.serviceWithZIO[DbConfig](c => ZIO.fromAutoCloseable(ZIO.attempt(new Db(c)))))

val catalogLayer: ZLayer[Any, Throwable, Catalog] =
  ZLayer.make[Catalog](
    ZLayer.succeed(DbConfig("jdbc:h2:mem:bookstore")),
    dbLayer,
    ZLayer.fromFunction(new InventoryRepo(_)),
    ZLayer.fromFunction(new PriceService(_)),
    ZLayer.fromFunction(new Catalog(_, _))
  )

val run =
  ZIO.serviceWith[Catalog](_.describe("Zionomicon")).debug.provide(catalogLayer)
```

#### You Write This

`Resource.from[Catalog]` reads the constructors itself, so you list only what it cannot build: values such as `DbConfig` (it never builds a `String`), and services you want to construct differently. `Wire(value)` is the `ZLayer.succeed` counterpart:

```scala
def catalogResource: Resource[Catalog] =
  Resource.from[Catalog](Wire(DbConfig("jdbc:h2:mem:bookstore")))
```

Allocate it in a scope like any other resource. One `Db` serves both repositories, and it closes when the scope does:

```scala
val description: String = Scope.global.scoped { scope =>

  val catalog = allocate(catalogResource)
  $(catalog)(_.describe("Zionomicon"))
}
// Db: connected to jdbc:h2:mem:bookstore
// Db: connection closed
// description: String = "Zionomicon: 3 in stock at $49.99"
```

`Db` extends `AutoCloseable`, so the macro registered its `close()`. There was no `ZLayer.scoped` step to remember.

When `Resource.from` cannot build the graph, the error names the type and the path that needed it. Leaving out the `DbConfig` wire fails because nothing can build its `String` field:

```scala
Resource.from[Catalog]()
// error: 
// ── Scope Error ─────────────────────────────────────────────────────────────────
// 
//   Cannot auto-create java.lang.String
// 
//   This type (primitive, collection, or function) cannot be auto-created.
// 
//   Required by:
//   ├── repl.MdocSession.MdocApp.DbConfig
//     ├── repl.MdocSession.MdocApp.Db
//     ├── repl.MdocSession.MdocApp.InventoryRepo
//     └── repl.MdocSession.MdocApp.Catalog
// 
//   Fix: Provide Wire(value) with the desired value:
// 
//     Resource.from[...](
//       Wire(...),  // provide a value for java.lang.String
//       ...
//     )
// 
// ────────────────────────────────────────────────────────────────────────────────
```

Both libraries check the graph at compile time: ZIO prints a `ZLAYER ERROR` box that lists the missing or ambiguous layers by number, while ZIO Blocks prints a `Scope Error` box with the tree of types that needed the missing one and a suggested `Wire(...)` fix.

### Shared and Fresh Services

ZIO shares a layer across the graph by default and gives you `ZLayer#fresh` for a separate copy. ZIO Blocks works the same way: auto-built services and `Wire.shared[A]` are shared, and `Wire.unique[A]` builds a new instance for each dependent:

```scala
def catalogWithFreshPrices: Resource[Catalog] =
  Resource.from[Catalog](
    Wire(DbConfig("jdbc:h2:mem:bookstore")),
    Wire.unique[PriceService]
  )
```

`ZLayer.derive[A]` maps to `Wire.shared[A]`. One difference: `ZLayer.derive` does not close an `AutoCloseable` unless the class extends `ZLayer.Derive.Scoped`, while `Wire.shared` always registers `close()`.

### Manual Wires

`ZLayer.fromFunction` maps to `Wire.Shared.fromFunction`, for construction the macro cannot express. The function receives the `Scope` and a `Context` holding its inputs. A hand-written wire does not register `close()` for you, so defer it yourself:

```scala
val dbWire: Wire.Shared[DbConfig, Db] =
  Wire.Shared.fromFunction[DbConfig, Db] { (scope, ctx) =>
    val db = new Db(ctx.get[DbConfig])
    scope.defer(db.close())
    db
  }

def catalogFromManualWire: Resource[Catalog] =
  Resource.from[Catalog](Wire(DbConfig("jdbc:h2:mem:bookstore")), dbWire)
```

Allocating the resource builds the graph with the manual wire and prints the description:

```scala
catalogFromManualWire.use(_.describe("Programming in Scala"))
// Db: connected to jdbc:h2:mem:bookstore
// Db: connection closed
// res3: String = "Programming in Scala: 7 in stock at $59.95"
```

:::warning
`Wire(value)` takes its argument by value. `Wire(new Db(...))` opens the connection when the wire is created, not when a scope allocates it. Use `Wire(...)` for configuration values and let `Resource.from`, `Wire.shared` or `Wire.Shared.fromFunction` construct resources.
:::

### What Changes

- **No environment type.** There is no `R` in `Resource[A]`. The graph is resolved once, inside `Resource.from`, and the result is an ordinary `Resource[Catalog]`.
- **Fewer layers to write.** `Resource.from` builds every concrete class it can reach from constructors. You list only leaf values and overrides, much like `ZLayer.derive` for the whole graph.
- **Errors are exceptions.** `Resource[A]` has no error channel. A constructor that throws fails `allocate` with that exception, and the scope still closes everything acquired so far.
- **Build a fresh graph for each run.** Write `def catalogResource`, not `val`. A `Resource` built from shared wires counts its users and is used up once the last one closes; allocating the same value again throws `IllegalStateException: Cannot allocate from a destroyed shared resource`. A `ZLayer` value can be provided again; a `Resource.from` value cannot.
- **Concurrency stays in ZIO.** If services in the graph run fibers, use `ZLayer` and ZIO's `Scope` for them.

## Coming from macwire

macwire is the closest match: `wire[A]` and `Wire.shared[A]` both read a constructor at compile time, with no runtime reflection. What macwire leaves out is lifecycle, and that is what ZIO Blocks adds.

### You Wrote This

This is the old code, macwire 2 (not compiled here):

```scala

class BookstoreModule {
  lazy val config: DbConfig         = DbConfig("jdbc:h2:mem:bookstore")
  lazy val db: Db                   = wire[Db]
  lazy val inventory: InventoryRepo = wire[InventoryRepo]
  lazy val prices: PriceService     = wire[PriceService]
  lazy val catalog: Catalog         = wire[Catalog]
}

val module = new BookstoreModule
try println(module.catalog.describe("Zionomicon"))
finally module.db.close() // macwire has no lifecycle: closing is on you
```

On Scala 3, macwire 2 also has `autowire`, which builds the whole graph from the dependencies you pass it:

```scala

val catalog = autowire[Catalog](DbConfig("jdbc:h2:mem:bookstore"))
```

### You Write This

`autowire[Catalog](deps*)` maps directly to `Resource.from[Catalog](wires*)`, and it works on Scala 2.13 as well as Scala 3. The module of `lazy val`s is no longer needed, and neither is the `try`/`finally`:

```scala
Resource
  .from[Catalog](Wire(DbConfig("jdbc:h2:mem:bookstore")))
  .use(_.describe("Functional Programming in Scala"))
// Db: connected to jdbc:h2:mem:bookstore
// Db: connection closed
// res4: String = "Functional Programming in Scala: 0 in stock at $44.99"
```

To keep one named wire per service, as with `wire[A]`, write `Wire.shared[A]` (or `Wire.unique[A]`) and pass the wires to `Resource.from`. Both libraries resolve dependencies by type, and `Resource.from`, like `autowire`, refuses to build a `String` or a primitive on its own.

### What Lifecycle Adds

A macwire module closes nothing, so every `AutoCloseable` needs its own `try`/`finally`, written in reverse order by hand. With ZIO Blocks:

- **`AutoCloseable` services close automatically**, in reverse order of construction, when the scope closes, including when the body throws. The exception is `Resource.from[T]` without parentheses: when `T`'s constructor takes a `Finalizer` or `Scope`, it does not register `close()` (a known bug in 0.0.56), so write `Resource.from[T]()` instead.
- **Services can register their own cleanup.** A constructor that takes an implicit `Finalizer` (or `Scope`) gets one injected, and can call `defer`:

```scala
final class CachedPriceService(db: Db)(implicit finalizer: Finalizer) {
  defer(println("CachedPriceService: price cache flushed"))

  def price(title: String): BigDecimal = db.priceOf(title)
}

final class CachedCatalog(inventory: InventoryRepo, prices: CachedPriceService) {
  def describe(title: String): String =
    s"$title: ${inventory.inStock(title)} in stock at $$${prices.price(title)}"
}
```

Wiring `CachedCatalog` flushes the price cache before the connection closes:

```scala
Resource
  .from[CachedCatalog](Wire(DbConfig("jdbc:h2:mem:bookstore")))
  .use(_.describe("Zionomicon"))
// Db: connected to jdbc:h2:mem:bookstore
// CachedPriceService: price cache flushed
// Db: connection closed
// res5: String = "Zionomicon: 3 in stock at $49.99"
```

The price cache was flushed before the connection closed, because `CachedPriceService` was built after `Db`.

- **Services cannot outlive the graph.** A macwire module hands out plain references that stay usable after you close `db`. A ZIO Blocks scope hands out `$[Catalog]`, which cannot leave the scope.
- **Sharing is explicit.** In macwire, sharing follows from how many `lazy val`s you write. In ZIO Blocks, shared is the default and `Wire.unique[A]` asks for a new instance per dependent.

## Coming from `scala.util.Using`

`Using` is in the standard library, so the old code on this page compiles too. `Using.resource` maps to `Resource#use`, and `Using.Manager` maps to `Scope#scoped` plus `Scope#allocate`.

### You Wrote This

`Using.Manager` closes every resource passed to `use` when the block ends:

```scala

val managed: Try[Int] = Using.Manager { use =>
  val db   = use(new Db(DbConfig("jdbc:h2:mem:bookstore")))
  val file = use(new ExportFile("catalog"))
  file.write(s"Zionomicon,${db.stockOf("Zionomicon")}")
  file.lineCount
}
// Db: connected to jdbc:h2:mem:bookstore
// ExportFile: created
// ExportFile: deleted
// Db: connection closed
// managed: Try[Int] = Success(1)
```

### You Write This

The same export written with a scope:

```scala
val scopedLines: Int = Scope.global.scoped { scope =>

  val db   = allocate(new Db(DbConfig("jdbc:h2:mem:bookstore")))
  val file = allocate(new ExportFile("catalog"))
  $(db, file)((d, f) => f.write(s"Zionomicon,${d.stockOf("Zionomicon")}"))
  $(file)(_.lineCount)
}
// Db: connected to jdbc:h2:mem:bookstore
// ExportFile: created
// ExportFile: deleted
// Db: connection closed
// scopedLines: Int = 1
```

Both close in reverse order. `Using.Manager` returns a `Try`; `Scope#scoped` returns the value and throws if the body or a finalizer fails. Wrap the call in `Try` if you want the old shape back.

### Finalizer Failures

`Using` and ZIO Blocks handle failing finalizers in nearly the same way. `Using` rethrows one exception and adds the others with `addSuppressed`, ranking fatal errors above ordinary ones. A ZIO Blocks scope throws the first failure in run order and adds the rest as suppressed; when the body itself throws, the body's exception wins and every finalizer failure is suppressed on it. Here are two resources whose `close()` fails:

```scala
final class FlakyResource(name: String) extends AutoCloseable {
  def close(): Unit = throw new IllegalStateException(s"$name failed to close")
}

def describeFailure(t: Throwable): String =
  s"${t.getMessage}; suppressed: ${t.getSuppressed.map(_.getMessage).mkString(", ")}"
```

With `Using.Manager`, the file closes first, so its failure is the one you get:

```scala
Using.Manager { use =>
  use(new FlakyResource("db"))
  use(new FlakyResource("export file"))
  ()
}.failed.foreach(t => println(describeFailure(t)))
// export file failed to close; suppressed: db failed to close
```

A scope gives the same answer:

```scala
Try {
  Scope.global.scoped { scope =>

    allocate(new FlakyResource("db"))
    allocate(new FlakyResource("export file"))
    ()
  }
}.failed.foreach(t => println(describeFailure(t)))
// export file failed to close; suppressed: db failed to close
```

What `Using` has no counterpart for is collecting the failures without throwing. A scope from `open()` returns them as a `Finalization` when you close it:

```scala
val openScope = Scope.global.open()
openScope.scope.allocate(new FlakyResource("db"))
openScope.scope.allocate(new FlakyResource("export file"))

val finalization: Finalization = openScope.close()
```

`close()` threw nothing. Both failures are in `finalization.errors`, in the order the finalizers ran:

```scala
println(finalization.errors.map(_.getMessage).mkString(" | "))
// export file failed to close | db failed to close
```

From there, `finalization.orThrow()` throws the first error with the rest suppressed, and `finalization.suppress(primary)` attaches them all to an exception of your own.

### What Changes

- **Resources cannot escape.** `Using.resource(r)(identity)` compiles; `Scope#allocate` returns a `$[A]`, and `Scope#scoped` rejects a result without an `Unscoped` instance.
- **Use goes through `$`.** Inside the scope, call methods on a resource with `$(db)(_.stockOf(title))`. The lambda's parameter may appear only as the receiver of a method call, so you cannot pass the resource itself to other code. See [`$` and the scoped value type](../reference/resource-management/scope.md) for the rules.
- **More than resources.** `defer` adds any cleanup action, `DeferHandle.cancel()` removes it again, and nested `Scope#scoped` blocks plus `Scope#lower` let a child scope use its parent's resources.

## What You Gain and What Is Different

You gain:

- **A compile-time check on escapes.** `$[A]` and `Unscoped` turn use-after-close into a compile error. None of the four source libraries offers this.
- **Lifecycle and wiring in one model.** The constructor graph from macwire and the reverse-order cleanup from `Using`, cats-effect and ZIO come from the same `Resource`.
- **A compile-time checked graph.** `Resource.from` reports missing dependencies and ambiguous providers at compile time, like `ZLayer.make`. In 0.0.56 a dependency cycle crashes the macro instead of producing an error (a known bug, #1767).
- **No third-party dependencies and no effect type.** The module runs on Scala 2.13 and 3, on the JVM and Scala.js, in direct-style code.

What is different:

- **Synchronous only.** No fibers, async acquisition, interruption or parallel `both`. For those, stay on cats-effect or ZIO.
- **Child scopes belong to a thread.** Only the thread that entered a `Scope#scoped` block can open child scopes on it; a scope from `open()` has no owner.
- **Results need `Unscoped`.** To return your own type from `Scope#scoped` or `Resource#use`, it needs an `Unscoped` instance (see below).
- **`Resource.from` values are single-use.** Build them with `def`.
- **Case classes are `Unscoped` on Scala 3.** Model resources as plain classes so the escape check covers them.

To return your own type from a scope, give it an `Unscoped` instance. On Scala 3, case classes whose fields are `Unscoped` get one automatically, and `derives Unscoped` makes it explicit; on Scala 2, derive it in the companion:

<Tabs groupId="scala-version" defaultValue="scala2">
<TabItem value="scala2" label="Scala 2">

```scala
final case class StockReport(title: String, inStock: Int)

object StockReport {
  implicit val unscoped: Unscoped[StockReport] = Unscoped.derived[StockReport]
}
```

</TabItem>
<TabItem value="scala3" label="Scala 3">

```scala
final case class StockReport(title: String, inStock: Int) derives Unscoped
```

</TabItem>
</Tabs>

Returning `StockReport` from `Scope#scoped` now compiles because it has an `Unscoped` instance:

```scala
val report: StockReport = Scope.global.scoped { scope =>

  val catalog = allocate(Resource.from[InventoryRepo](Wire(DbConfig("jdbc:h2:mem:bookstore"))))
  StockReport("Zionomicon", $(catalog)(_.inStock("Zionomicon")))
}
// Db: connected to jdbc:h2:mem:bookstore
// Db: connection closed
// report: StockReport = StockReport(title = "Zionomicon", inStock = 3)
println(report)
// StockReport(Zionomicon,3)
```

## Migration Checklist

1. Add `zio-blocks-scope` and `import zio.blocks.scope._`.
2. Replace `Resource.make` / `ZIO.acquireRelease` with `Resource.acquireRelease`, and cats-effect or ZIO `fromAutoCloseable` with ZIO Blocks `Resource.fromAutoCloseable` or `allocate(new ...)` (`Scope#allocate`). Remove the effect wrappers around acquire and release.
3. Replace `Resource#use` / `ZIO.scoped` / `Using.Manager` blocks with `Scope.global.scoped { scope => import scope._; ... }` and call resources through `$`.
4. Replace `Resource.onFinalize` / `ZIO.addFinalizer` with `defer`.
5. Replace `allocated` / `Scope.make` with `open()` and `OpenScope.close()`, and handle the returned `Finalization`.
6. Replace `ZLayer.make` or a macwire module with one `Resource.from[Root](wires*)`. Pass `Wire(value)` for configuration values; let the macro build the rest.
7. Replace `ZLayer#fresh` layers with `Wire.unique[A]`, and `ZLayer.fromFunction` with `Wire.Shared.fromFunction`, deferring `close()` yourself.
8. Turn `val` resources built by `Resource.from` into `def`s.
9. Make resource types plain classes, and give returned result types an `Unscoped` instance.
10. Delete the manual `try`/`finally` blocks that closed macwire-built services.
11. Keep cats-effect or ZIO for parts that need fibers, async resources or parallel acquisition.

## Putting It Together

This is the bookstore fully migrated: `Resource.from` wires the graph with one shared `Db`, a service registers its own cleanup through `Finalizer`, the scope opens the export file as a second resource, and everything closes in reverse order:

```scala title="scope-examples/src/main/scala/resourcemigration/BookstoreMigratedExample.scala"
package resourcemigration

/**
 * The bookstore service graph from the "Coming from cats-effect Resource, ZIO
 * Scope/ZLayer, macwire or Using" guide, fully migrated to ZIO Blocks Scope.
 *
 * `Resource.from[Catalog]` wires the graph from constructors, sharing one `Db`
 * between `InventoryRepo` and `PriceService`. The scope opens a temporary
 * export file as a second resource. When the scope closes, finalizers run in
 * LIFO order: the export file is deleted first, then the price cache is
 * flushed, then the database connection is closed.
 *
 * Run with: sbt "runMain resourcemigration.BookstoreMigratedExample"
 */

final case class DbConfig(url: String)

final class Db(config: DbConfig) extends AutoCloseable {
  println(s"Db: connected to ${config.url}")

  private val stock: Map[String, Int] = Map(
    "Zionomicon"                      -> 3,
    "Programming in Scala"            -> 7,
    "Functional Programming in Scala" -> 0
  )

  private val prices: Map[String, BigDecimal] = Map(
    "Zionomicon"                      -> BigDecimal("49.99"),
    "Programming in Scala"            -> BigDecimal("59.95"),
    "Functional Programming in Scala" -> BigDecimal("44.99")
  )

  def stockOf(title: String): Int        = stock.getOrElse(title, 0)
  def priceOf(title: String): BigDecimal = prices.getOrElse(title, BigDecimal(0))
  def close(): Unit                      = println("Db: connection closed")
}

final class InventoryRepo(db: Db) {
  def inStock(title: String): Int = db.stockOf(title)
}

// The implicit Finalizer lets a service register its own cleanup; the wiring
// macro injects it automatically.
final class PriceService(db: Db)(implicit finalizer: Finalizer) {
  defer(println("PriceService: price cache flushed"))

  def price(title: String): BigDecimal = db.priceOf(title)
}

final class Catalog(inventory: InventoryRepo, prices: PriceService) {
  def describe(title: String): String =
    s"$title: ${inventory.inStock(title)} in stock at $$${prices.price(title)}"
}

final class ExportFile(prefix: String) extends AutoCloseable {
  private val path: Path = Files.createTempFile(prefix, ".csv")
  println("ExportFile: created")

  def write(line: String): Unit = {
    Files.write(path, (line + "\n").getBytes(StandardCharsets.UTF_8), StandardOpenOption.APPEND)
    ()
  }

  def lineCount: Int = Files.readAllLines(path).size

  def close(): Unit = {
    Files.deleteIfExists(path)
    println("ExportFile: deleted")
  }
}

object BookstoreMigratedExample {

  val titles: List[String] =
    List("Zionomicon", "Programming in Scala", "Functional Programming in Scala")

  // A `def`, not a `val`: a Resource built from shared wires is single-use once
  // its last scope has closed, so each run builds a fresh graph.
  def catalogResource: Resource[Catalog] =
    Resource.from[Catalog](Wire(DbConfig("jdbc:h2:mem:bookstore")))

  def main(args: Array[String]): Unit = {
    val exported: Int = Scope.global.scoped { scope =>

      val catalog = allocate(catalogResource)
      val file    = allocate(new ExportFile("catalog"))

      titles.foreach { title =>
        $(catalog, file)((c, f) => f.write(c.describe(title)))
      }
      $(file)(_.lineCount)
    }

    println(s"Exported $exported lines")
  }
}
```

## Running the Examples

The complete example is a runnable file in the `scope-examples` project. Clone the repository and change into the project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/scope-examples
```

<details>
  <summary>scope-examples/src/main/scala/resourcemigration/BookstoreMigratedExample.scala</summary>

```scala title="scope-examples/src/main/scala/resourcemigration/BookstoreMigratedExample.scala" showLineNumbers
package resourcemigration

/**
 * The bookstore service graph from the "Coming from cats-effect Resource, ZIO
 * Scope/ZLayer, macwire or Using" guide, fully migrated to ZIO Blocks Scope.
 *
 * `Resource.from[Catalog]` wires the graph from constructors, sharing one `Db`
 * between `InventoryRepo` and `PriceService`. The scope opens a temporary
 * export file as a second resource. When the scope closes, finalizers run in
 * LIFO order: the export file is deleted first, then the price cache is
 * flushed, then the database connection is closed.
 *
 * Run with: sbt "runMain resourcemigration.BookstoreMigratedExample"
 */

final case class DbConfig(url: String)

final class Db(config: DbConfig) extends AutoCloseable {
  println(s"Db: connected to ${config.url}")

  private val stock: Map[String, Int] = Map(
    "Zionomicon"                      -> 3,
    "Programming in Scala"            -> 7,
    "Functional Programming in Scala" -> 0
  )

  private val prices: Map[String, BigDecimal] = Map(
    "Zionomicon"                      -> BigDecimal("49.99"),
    "Programming in Scala"            -> BigDecimal("59.95"),
    "Functional Programming in Scala" -> BigDecimal("44.99")
  )

  def stockOf(title: String): Int        = stock.getOrElse(title, 0)
  def priceOf(title: String): BigDecimal = prices.getOrElse(title, BigDecimal(0))
  def close(): Unit                      = println("Db: connection closed")
}

final class InventoryRepo(db: Db) {
  def inStock(title: String): Int = db.stockOf(title)
}

// The implicit Finalizer lets a service register its own cleanup; the wiring
// macro injects it automatically.
final class PriceService(db: Db)(implicit finalizer: Finalizer) {
  defer(println("PriceService: price cache flushed"))

  def price(title: String): BigDecimal = db.priceOf(title)
}

final class Catalog(inventory: InventoryRepo, prices: PriceService) {
  def describe(title: String): String =
    s"$title: ${inventory.inStock(title)} in stock at $$${prices.price(title)}"
}

final class ExportFile(prefix: String) extends AutoCloseable {
  private val path: Path = Files.createTempFile(prefix, ".csv")
  println("ExportFile: created")

  def write(line: String): Unit = {
    Files.write(path, (line + "\n").getBytes(StandardCharsets.UTF_8), StandardOpenOption.APPEND)
    ()
  }

  def lineCount: Int = Files.readAllLines(path).size

  def close(): Unit = {
    Files.deleteIfExists(path)
    println("ExportFile: deleted")
  }
}

object BookstoreMigratedExample {

  val titles: List[String] =
    List("Zionomicon", "Programming in Scala", "Functional Programming in Scala")

  // A `def`, not a `val`: a Resource built from shared wires is single-use once
  // its last scope has closed, so each run builds a fresh graph.
  def catalogResource: Resource[Catalog] =
    Resource.from[Catalog](Wire(DbConfig("jdbc:h2:mem:bookstore")))

  def main(args: Array[String]): Unit = {
    val exported: Int = Scope.global.scoped { scope =>

      val catalog = allocate(catalogResource)
      val file    = allocate(new ExportFile("catalog"))

      titles.foreach { title =>
        $(catalog, file)((c, f) => f.write(c.describe(title)))
      }
      $(file)(_.lineCount)
    }

    println(s"Exported $exported lines")
  }
}
```

</details>

Run it from the `scope-examples` directory:

```bash
sbt "runMain resourcemigration.BookstoreMigratedExample"
```

To check that the example compiles without running it:

```bash
sbt compile
```

## See Also

- [Compile-Time Resource Safety with Scope](./compile-time-resource-safety-with-scope.md): the tutorial behind every concept on this page.
- [Resource Management](../reference/resource-management/index.md): how `Scope`, `Resource` and `Wire` fit together.
- [Scope](../reference/resource-management/scope.md), [Resource](../reference/resource-management/resource.md) and [Wire](../reference/resource-management/wire.md): the full API.
- [Unscoped](../reference/resource-management/unscoped.md) and [Finalization](../reference/resource-management/finalization.md): escape rules and finalizer errors.
- The how-to guide "How to fix common Scope and Wire compile errors", in progress, will explain each `Resource.from` and `$` compile error.
- [How to Use ZIO Blocks in a ZIO or cats-effect Application](./use-zio-blocks-in-zio-or-cats-effect.md): using ZIO Blocks resources from a ZIO or cats-effect application, with cleanup carried across.

---

## How to Return HTMX Fragments, Full Pages and Response Headers

In this guide, we write the server side of a small tea shop built with htmx: a catalog, an "Add to cart" button, a cart badge, a cart page with a promo code and a checkout. Every handler answers with HTML built by the ZIO Blocks [HTML module](../reference/html.md) and with typed `HX-*` headers from the [HTMX module](../reference/htmx/index.md). By the end, you will have one handler per URL that sends a full page or a fragment depending on who asks, updates the cart badge in the same response, tells other parts of the page that the cart changed, redirects after checkout, puts validation errors next to the form and keeps the address bar in sync.

We assume you have finished the tutorial [Add Interactivity to a Page with HTMX](./add-interactivity-with-htmx.md), so `hxGet`, `hxTarget` and a swap are familiar. This guide covers:

- Answering one URL with a full page or a fragment, including boosted links and the Back button
- Updating a second spot with an out-of-band swap
- Announcing an event to the page with `HX-Trigger`
- Choosing between `HX-Redirect`, `HX-Location` and `HX-Refresh` after a form post
- Sending validation errors to another element with `HX-Retarget` and `HX-Reswap`
- Updating the address bar from the server with `HX-Push-Url` and `HX-Replace-Url`

## The Problem

An htmx page asks the server for many different things, and they all arrive as ordinary HTTP requests. A click on a category link wants the product list only, a reload of the same URL wants the whole page, and the Back button may want either. After a form post, the server often has to do more than return HTML: update the cart badge in the header, refresh a summary panel, move the browser to a receipt or show an error next to the form.

htmx reads all of this from request and response headers whose names and values are plain strings. Without typed headers, a handler for the "Add to cart" button looks like this:

```scala

def addToCart(request: Request, name: String, count: Int): Response = {
  val fromHtmx = request.headers.rawGet("hx-request").contains("true")
  val button   = s"""<button hx-post="/cart/items/sencha" hx-target="this" hx-swap="outerHTML">In cart</button>"""
  val badge    = s"""$count"""
  val body     = if (fromHtmx) button + badge else s"<!DOCTYPE html><html><body>$button</body></html>"
  Response
    .ok(Body.fromArray(body.getBytes("UTF-8"), ContentType.`text/html`))
    .addHeader("HX-Trigger", s"""{"show-message":{"text":"Added $name"}}""")
    .addHeader("HX-Retarget", "#promo-eror")
}
```

Every string in it can be wrong without any error. A misspelled header name or selector, such as `#promo-eror`, is silently ignored by htmx. The JSON in `HX-Trigger` breaks as soon as `name` contains a quote. And the `fromHtmx` check is wrong in two cases we look at below: a boosted link and a Back button whose page htmx no longer has in its cache both send `HX-Request: true` and still need the full page.

In this guide, we replace each string with a typed value: the request headers `HxRequest`, `HxBoosted` and `HxHistoryRestoreRequest`, the `hxSwapOob` attribute, and the response headers `HxTriggerHeader`, `HxLocation`, `HxRedirect`, `HxRefresh`, `HxRetarget`, `HxReswap` and `HxReplaceUrl`.

## Prerequisites

The HTMX module is for Scala 3 only. Add it to your `build.sbt`:

```scala
// Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-http-htmx" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-http-htmx" % "0.0.56"
```

The HTMX module brings the HTML module and the [http-model](../reference/http-model/index.md) module with it, so `Dom`, `Request` and `Response` are already on the classpath. The http-model module is compiled for Java 21, so the programs need JDK 21 or newer. The handlers are plain `Request => Response` functions; the companion examples serve them with the small JDK adapter from [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md). The guide loads htmx 2.0.11 in the browser, the version the HTMX module's attributes follow, and every browser behavior described below was observed with it.

We use these imports throughout the guide:

```scala

```

`zio.http.htmx._` brings the `hx*` attribute keys and their value types, such as `HxTarget` and `HxSwap`, and `zio.http.htmx.headers._` brings the typed headers.

## The Tea Shop

The data is five teas in three categories, prices in cents, and one in-memory cart that every handler shares:

```scala
final case class Product(id: String, name: String, category: String, priceCents: Int)

val products: List[Product] = List(
  Product("sencha", "Sencha", "Green", 1200),
  Product("matcha", "Matcha", "Green", 2400),
  Product("assam", "Assam", "Black", 900),
  Product("earl-grey", "Earl Grey", "Black", 1000),
  Product("rooibos", "Rooibos", "Herbal", 800)
)

val categories: List[String] = List("Green", "Black", "Herbal")

def price(cents: Int): String = "$" + (cents / 100) + "." + f"${cents % 100}%02d"

final class Cart {
  private val lines = new AtomicReference(Map.empty[String, Int])
  private val promo = new AtomicReference(Option.empty[String])

  def add(productId: String): Int =
    lines.updateAndGet(current => current.updated(productId, current.getOrElse(productId, 0) + 1))(productId)
  def quantity(productId: String): Int = lines.get.getOrElse(productId, 0)
  def count: Int                       = lines.get.values.sum
  def applyPromo(code: String): Unit   = promo.set(Some(code))
  def totalCents: Int = {
    val gross = lines.get.map { case (id, n) => products.find(_.id == id).fold(0)(_.priceCents) * n }.sum
    if (promo.get.contains("TEA10")) gross * 9 / 10 else gross
  }
  def clear(): Unit = { lines.set(Map.empty); promo.set(None) }
}

val cart = new Cart
```

The handlers run on the server's threads, so the cart keeps its state in `AtomicReference`s and changes it atomically. A real shop keeps one cart per session; one shared cart keeps the guide short.

Now the page itself. The catalog has category links and an `#catalog` element with sort links and the product list. The layout around it has the cart badge in the header, a `#message` line, the main area and a cart summary on the side:

```scala
val htmxScript: Dom.Element = script(src := "https://cdn.jsdelivr.net/npm/htmx.org@2.0.11/dist/htmx.min.js")

def catalogUrl(category: String, sort: String): String = {
  val params = List("category" -> category, "sort" -> sort).filter { case (k, v) => v.nonEmpty && !(k == "sort" && v == "name") }
  if (params.isEmpty) "/" else params.map { case (k, v) => s"$k=$v" }.mkString("/?", "&", "")
}

def addButton(product: Product): Dom.Element = {
  val quantity = cart.quantity(product.id)
  button(
    id       := s"add-${product.id}",
    hxPost   := s"/cart/items/${product.id}",
    hxTarget := HxTarget.this_,
    hxSwap   := HxSwap.OuterHTML
  )(if (quantity == 0) "Add to cart" else s"In cart ($quantity)")
}

def catalog(category: String, sort: String): Dom.Element = {
  val shown  = products.filter(product => category.isEmpty || product.category == category)
  val sorted = if (sort == "price") shown.sortBy(_.priceCents) else shown.sortBy(_.name)
  div(id := "catalog")(
    p(
      "Sort by ",
      a(hxGet := catalogUrl(category, "name"), hxTarget := HxTarget.css("#catalog"), hxSwap := HxSwap.OuterHTML)("name"),
      " | ",
      a(hxGet := catalogUrl(category, "price"), hxTarget := HxTarget.css("#catalog"), hxSwap := HxSwap.OuterHTML)("price")
    ),
    ul(sorted.map(product => li(strong(product.name), s" ${price(product.priceCents)} ", addButton(product))))
  )
}

def categoryLinks: Dom.Element =
  p(
    ("" :: categories).map { category =>
      a(
        href      := catalogUrl(category, ""),
        hxGet     := catalogUrl(category, ""),
        hxTarget  := HxTarget.css("#catalog"),
        hxSwap    := HxSwap.OuterHTML,
        hxPushUrl := HxUrlUpdate(true)
      )(if (category.isEmpty) "All teas " else s"$category ")
    }
  )

def catalogContent(category: String, sort: String): Dom.Element =
  div(h1("Teas"), categoryLinks, catalog(category, sort))

def cartBadge: Dom.Element = span(id := "cart-count")(cart.count.toString)

def cartSummary: Dom.Element =
  aside(
    id        := "cart-summary",
    hxGet     := "/cart/summary",
    hxTrigger := HxTrigger("cart-updated").from("body"),
    hxSwap    := HxSwap.OuterHTML
  )(s"${cart.count} items, ${price(cart.totalCents)}")

def layout(heading: String, content: Dom.Element): Dom.Element =
  html(lang := "en")(
    head(meta(charset := "utf-8"), title(s"$heading - Tea Shop"), htmxScript),
    body(hxOn("show-message") := Js("document.getElementById('message').textContent = event.detail.text"))(
      header(nav(hxBoost := true)(a(href := "/")("Teas"), " ", a(href := "/cart")("Cart (", cartBadge, ")"))),
      p(id := "message")(""),
      main(id := "main")(content),
      cartSummary
    )
  )

def htmlResponse(html: String): Response =
  Response.ok(Body.fromArray(html.getBytes("UTF-8"), ContentType.`text/html`))
```

The code above builds every element the later sections talk to:

- `catalogUrl` builds the address of a catalog view and leaves out empty and default values, so "all teas sorted by name" is just `/`.
- `addButton` posts to `/cart/items/<id>` and replaces itself (`hxTarget := HxTarget.this_` with `HxSwap.OuterHTML`) with whatever the server sends back.
- `catalog` is the `#catalog` element. The category links and the sort links ask for a new `#catalog` and swap it in with `outerHTML`, so the fragment the server sends is exactly the element the page already contains.
- `categoryLinks` carries both `href` and `hxGet`: htmx takes over the click, and the `href` still works without JavaScript. `hxPushUrl` puts each category into the address bar.
- `cartBadge` is the `#cart-count` element in the header, `cartSummary` refreshes itself when a `cart-updated` event reaches the page, and the `hxOn("show-message")` handler on `<body>` writes a message into `#message`. Sections below explain each one when the server starts using it.
- `nav(hxBoost := true)` turns the two header links into boosted links: htmx fetches the next page with an AJAX request and swaps its `<body>` in, instead of a full browser navigation.

Render the layout around an empty main area to check the hooks:

```scala
println(layout("Teas", div).render(indent = 2))
// <html lang="en">
//   <head>
//     <meta charset="utf-8"/>
//     <title>Teas - Tea Shop</title>
//     <script src="https://cdn.jsdelivr.net/npm/htmx.org@2.0.11/dist/htmx.min.js"></script>
//   </head>
//   <body hx-on:show-message="document.getElementById(&#x27;message&#x27;).textContent = event.detail.text">
//     <header>
//       <nav hx-boost="true">
//         Teas
//          
//         
//           Cart (
//           0
//           )
//         
//       </nav>
//     </header>
//     
//     <main id="main">
//       
//     </main>
//     <aside id="cart-summary" hx-get="/cart/summary" hx-trigger="cart-updated from:body" hx-swap="outerHTML">0 items, $0.00</aside>
//   </body>
// </html>
```

Finally, two helpers let us send requests to a handler without a server and print what htmx would receive: the status, every `hx-*` and `vary` header, and the body:

```scala
def get(target: String, headers: Header*): Request =
  headers.foldLeft(Request.get(URL.parse(target).toOption.get))((request, header) => request.addHeader(header))

def post(target: String, form: Form, headers: Header*): Request =
  headers.foldLeft(Request.post(URL.parse(target).toOption.get, Body.fromString(form.encode)))((request, header) =>
    request.addHeader(header)
  )

def show(response: Response): Unit = {
  println(s"status: ${response.status.code}")
  response.headers.toList.foreach { case (name, value) =>
    if (name.startsWith("hx-") || name == "vary") println(s"$name: $value")
  }
  println(response.body.asString())
}
```

## Answer One URL with a Full Page or a Fragment

The category links push URLs such as `/?category=Green` into the address bar, so `/` must answer three kinds of visitors. A browser that loads, reloads or bookmarks the URL needs the full page. A category link needs only the new `#catalog`. And htmx itself sometimes asks for the URL on the reader's behalf and needs the whole page back:

| Request                                         | `HX-Request` | `HX-Boosted` | `HX-History-Restore-Request` | Needs     |
| ----------------------------------------------- | ------------ | ------------ | ---------------------------- | --------- |
| Address bar, reload, bookmark                   | absent       | absent       | absent                       | full page |
| A click on an `hxGet` link                      | `true`       | absent       | absent                       | fragment  |
| A click on a boosted link (`hxBoost`)           | `true`       | `true`       | absent                       | full page |
| Back button, page not in htmx's history cache   | `true`       | absent       | `true`                       | full page |

A boosted link swaps the response into `<body>` and takes the page title from it, and a history restore puts the response in place of the whole `<body>` as well. Only the second row wants a fragment, so the check reads all three headers:

```scala
def wantsFragment(request: Request): Boolean =
  request.header(HxRequest).exists(_.enabled) &&
    !request.header(HxBoosted).exists(_.enabled) &&
    !request.header(HxHistoryRestoreRequest).exists(_.enabled)

def pageOrFragment(request: Request, heading: String)(fragment: => Dom.Element, content: => Dom.Element): Response = {
  val html =
    if (wantsFragment(request)) fragment.render
    else doctype.render + layout(heading, content).render
  htmlResponse(html).addHeader("vary", "hx-request, hx-boosted, hx-history-restore-request")
}

def catalogPage(request: Request): Response = {
  val category = request.queryParams.getFirst("category").getOrElse("")
  val sort     = request.queryParams.getFirst("sort").getOrElse("name")
  pageOrFragment(request, "Teas")(catalog(category, sort), catalogContent(category, sort))
}
```

The code above:

- `request.header(HxRequest)` parses the `HX-Request` header into an `HxRequest`, and `exists(_.enabled)` is `true` only for the value `true`. A missing header and a malformed one both give `None`, so both count as "not htmx".
- `pageOrFragment` takes the fragment and the main content as by-name parameters, so only the one it needs is built.
- `addHeader("vary", ...)` tells browsers and caches that the answer depends on these three headers, so a cached fragment is never shown as a page.

Ask `wantsFragment` about each row of the table, plus a request whose `HX-Request` value is not a boolean:

```scala
println(wantsFragment(get("/?category=Green")))
// false
println(wantsFragment(get("/?category=Green", HxRequest(true))))
// true
println(wantsFragment(get("/cart", HxRequest(true), HxBoosted(true))))
// false
println(wantsFragment(get("/?category=Green", HxRequest(true), HxHistoryRestoreRequest(true))))
// false
println(get("/", Header.Custom("hx-request", "yes")).header(HxRequest))
// None
```

Only the category link gets a fragment. The last line shows that `HX-Request: yes` does not parse, so `Request#header` returns `None` and that request gets the full page. Send the category link's request to the handler:

```scala
show(catalogPage(get("/?category=Green", HxRequest(true))))
// status: 200
// vary: hx-request, hx-boosted, hx-history-restore-request
// Sort by name | priceMatcha $24.00 <button id="add-matcha" hx-post="/cart/items/matcha" hx-target="this" hx-swap="outerHTML">Add to cart</button>Sencha $12.00 <button id="add-sencha" hx-post="/cart/items/sencha" hx-target="this" hx-swap="outerHTML">Add to cart</button>
```

The response is the `#catalog` element alone, with the two green teas. The same URL without `HX-Request` returns `<!DOCTYPE html>` and the whole layout around the same element.

In a browser, clicking "Green" sends `HX-Request: true` and swaps the new `#catalog` in, and the address bar shows `/?category=Green`. A reload of that address gets the full page. Clicking the boosted "Cart" link sends `HX-Request: true` and `HX-Boosted: true`, gets the full cart page and keeps the header, and the tab title changes to "Cart - Tea Shop". With htmx's history cache turned off (`htmx.config.historyCacheSize = 0`), pressing Back sends `HX-History-Restore-Request: true` and gets the full page back.

:::warning
A check that reads only `HX-Request` answers the last two rows with a fragment. We tried that version in the browser: after a click on the boosted "Cart" link, the header and its navigation were gone and the tab title still said "Teas", and after Back with a history cache miss, the `<body>` held only the product list. The tutorial's check, which reads `HX-Request` and `HX-History-Restore-Request`, is enough for a page without boosted links; once the page uses `hxBoost`, read `HX-Boosted` too.
:::

## Update a Second Spot with an Out-of-Band Swap

When the reader clicks "Add to cart", the button replaces itself with "In cart (1)". The cart badge in the header must change too, but it is outside the button's target. An out-of-band swap updates it in the same response: htmx looks for elements with an `hx-swap-oob` attribute in the response, takes them out, and swaps each one into the page element with the same `id`. The HTMX module types this attribute as `hxSwapOob`:

```scala
def addToCart(productId: String): Response =
  products.find(_.id == productId) match {
    case None => Response.text(Status.NotFound, s"No product $productId")
    case Some(product) =>
      cart.add(product.id)
      val badge = cartBadge(hxSwapOob := HxSwapOob.True)
      htmlResponse(addButton(product).render + badge.render)
  }
```

The response is the new button, which htmx swaps into the button's own place, followed by the badge. `cartBadge(hxSwapOob := HxSwapOob.True)` adds the attribute to the same element the layout renders, so the badge's markup lives in one place. The two elements are rendered separately and concatenated: a fragment can contain any number of top-level elements.

Add a Sencha to the cart:

```scala
show(addToCart("sencha"))
// status: 200
// <button id="add-sencha" hx-post="/cart/items/sencha" hx-target="this" hx-swap="outerHTML">In cart (1)</button>1
```

In a browser, the button changes to "In cart (1)" and the header's badge changes from 0 to 1 after one request. htmx removes `hx-swap-oob` from the element before it inserts it, so the page keeps a plain ``.

`HxSwapOob.True` replaces the whole element with the same `id`. `HxSwapOob.using(HxSwap.InnerHTML)` swaps only its content, and `HxSwapOob.using(HxSwap.InnerHTML, HxTarget.css("#cart-count"))` names the target explicitly, so the element in the response can have another `id` or none.

:::warning
Keep the `HxSwap` you pass to `HxSwapOob.using` bare, without modifiers such as `swap(...)` or `settle(...)`. In 0.0.56, `HxSwapOob.using(HxSwap.InnerHTML.swap(10.millis))` renders `hx-swap-oob="innerHTML swap:10ms"`, which htmx 2 reads as the strategy `innerHTML swap` and the selector `10ms`. In the browser this raised `htmx:swapError`, and htmx swapped nothing from that response, not even the main fragment.
:::

## Tell the Page Something Happened

An out-of-band swap works when the server knows each spot to update. When several parts of the page care about the cart, it is simpler to announce what happened and let each part decide. The `HX-Trigger` response header names client-side events that htmx fires as soon as the response arrives. Its typed form is `HxTriggerHeader`, and its value is an `HxEventPayload`:

```scala
def addToCartAndAnnounce(productId: String): Response = {
  val response = addToCart(productId)
  products.find(_.id == productId).fold(response) { product =>
    val message = Json.Object("show-message" -> Json.Object("text" -> Json.String(s"Added ${product.name}")))
    response
      .addHeader(HxTriggerHeader(HxEventPayload.Event("cart-updated")))
      .addHeader(HxTriggerAfterSettle(HxEventPayload.JsonValue(message)))
  }
}

def cartSummaryFragment: Response = htmlResponse(cartSummary.render)
```

The code above sends two events with two kinds of payload:

- `HxEventPayload.Event("cart-updated")` is a plain event name. The `cartSummary` element from the layout listens for it with `hxTrigger := HxTrigger("cart-updated").from("body")`, which renders as `hx-trigger="cart-updated from:body"`: htmx fires the event on the button that sent the request, the event bubbles up to `<body>`, and the summary sends `GET /cart/summary`. `cartSummaryFragment` answers with a new `<aside>`, and the summary's `hxSwap := HxSwap.OuterHTML` replaces the old one with it. With the default `innerHTML` swap, the new `<aside>` would land inside the old one, and in the browser each nested copy then sent its own request on the next `cart-updated`.
- `HxEventPayload.JsonValue(message)` sends a JSON object whose keys are event names and whose values become the event's `detail`. The `hxOn("show-message")` handler on `<body>` reads `event.detail.text` and writes it into `#message`.
- `HxTriggerAfterSettle` fires its events after htmx has swapped and settled the new content, later than `HxTriggerHeader`. `HxTriggerAfterSwap` sits between the two. Use a later header when the listener needs the new HTML to be in the page already.

Both event names are lowercase with dashes. HTML lowercases attribute names, so an `hx-on:` attribute can't name an event with capital letters; `hx-trigger` values keep their case, but one style for every event is easier to follow.

Add an Assam and read the headers:

```scala
show(addToCartAndAnnounce("assam"))
// status: 200
// hx-trigger: cart-updated
// hx-trigger-after-settle: {"show-message":{"text":"Added Assam"}}
// <button id="add-assam" hx-post="/cart/items/assam" hx-target="this" hx-swap="outerHTML">In cart (1)</button>2
show(cartSummaryFragment)
// status: 200
// <aside id="cart-summary" hx-get="/cart/summary" hx-trigger="cart-updated from:body" hx-swap="outerHTML">2 items, $21.00</aside>
```

In a browser, one click on "Add to cart" sends the `POST`, the button and the badge update, the summary sends `GET /cart/summary` with `HX-Trigger: cart-summary` (the requesting element's id) and swaps in the new totals, and `#message` shows "Added Assam".

The JSON payload must be an object. `HxEventPayload.JsonValue` also accepts other JSON values, such as an array, but htmx then treats the whole header as a comma-separated list of event names: in the browser, `HX-Trigger: ["cart-updated"]` fired no `cart-updated` event. To send several plain events, write them as one name list, `HxEventPayload.Event("cart-updated, show-banner")`, or use several keys in one JSON object.

:::warning
`HxEventPayload.JsonValue` and `HxLocation` print their JSON with `Json#print`, which writes characters outside ASCII as they are. HTTP header values are meant to be ASCII. With the JDK HTTP server from the companion examples, a value with a character above `U+00FF`, such as `✓`, makes the server drop the connection without a response, and characters such as `é` are sent as single ISO-8859-1 bytes. This is a known bug in 0.0.56.
:::

Until the module escapes them, write such a payload with the raw header name and escaped JSON. `Json#print` escapes non-ASCII characters when the `WriterConfig` has `withEscapeUnicode(true)`, and `Response#addHeader` accepts the header name and the escaped text:

```scala
val greeting = Json.Object("show-message" -> Json.Object("text" -> Json.String("Added Sencha ✓")))

val escaped = htmlResponse("").addHeader(HxTriggerHeader.name, greeting.print(WriterConfig.withEscapeUnicode(true)))
```

Print the header value on the wire, then read it back as a typed `HxTriggerHeader`:

```scala
println(escaped.headers.rawGet(HxTriggerHeader.name))
// Some({"show-message":{"text":"Added Sencha \u2713"}})
println(escaped.header(HxTriggerHeader).map(_.renderedValue))
// Some({"show-message":{"text":"Added Sencha ✓"}})
```

The value on the wire is pure ASCII. htmx parses the escaped JSON back into the same text, and the typed `HxTriggerHeader` read gives back the original characters.

## Redirect, Navigate or Refresh After a Form Post

After "Place order" the reader should see a receipt, and after "Empty cart" the cart page should start over. The response to an htmx request can't use an HTTP redirect for this: the browser's `XMLHttpRequest` follows a `303` by itself, and htmx only sees the final response. In the browser, a `303 See Other` to `/orders/1001` was followed with every `HX-*` request header sent again, so our handler answered with a fragment, htmx swapped the receipt into the button's target inside the cart page, and the address bar still said `/cart`.

htmx has three response headers for this instead. Each one makes htmx skip the swap of the response body:

| Header        | Typed value              | What the browser does                                                       |
| ------------- | ------------------------ | --------------------------------------------------------------------------- |
| `HX-Redirect` | `HxRedirect("/")`        | A full page load of the new URL, without `HX-*` request headers             |
| `HX-Location` | `HxLocation("/orders/1001", Some(...))` | An htmx `GET` of the new URL into a target, then pushes the URL |
| `HX-Refresh`  | `HxRefresh(true)`        | A full reload of the current page                                           |

`HX-Location` keeps the page and its scripts and is the smooth choice when the next view fits into the current layout. `HX-Redirect` is the right choice when the next page has another layout, or when the reader leaves the htmx part of the site. The cart page and the three handlers look like this:

```scala
val orderNumbers = new AtomicInteger(1000)

def cartContent: Dom.Element =
  div(
    h1("Your Cart"),
    ul(products.filter(product => cart.quantity(product.id) > 0).map { product =>
      li(s"${cart.quantity(product.id)} x ${product.name}")
    }),
    p(id := "cart-total")(s"Total: ${price(cart.totalCents)}"),
    form(hxPost := "/cart/promo", hxTarget := HxTarget.css("#cart-total"), hxSwap := HxSwap.OuterHTML)(
      input(name := "code", placeholder := "Promo code"),
      button(`type` := "submit")("Apply"),
      span(id := "promo-error")("")
    ),
    button(hxPost := "/checkout")("Place order"),
    button(hxPost := "/cart/clear")("Empty cart")
  )

def orderContent(number: Int): Dom.Element =
  div(h1(s"Order $number Placed"), p("Thank you! Your tea is on its way."), a(href := "/")("Keep shopping"))

def checkout(): Response =
  if (cart.count == 0) htmlResponse("").addHeader(HxRedirect("/"))
  else {
    val number = orderNumbers.incrementAndGet()
    cart.clear()
    htmlResponse("")
      .addHeader(HxLocation(s"/orders/$number", target = Some(HxTarget.css("#main"))))
      .addHeader(HxTriggerHeader(HxEventPayload.Event("cart-updated")))
  }

def clearCart(): Response = {
  cart.clear()
  htmlResponse("").addHeader(HxRefresh(true))
}

def orderPage(request: Request, number: Int): Response =
  pageOrFragment(request, s"Order $number")(
    div(orderContent(number), cartBadge(hxSwapOob := HxSwapOob.True)),
    orderContent(number)
  )
```

The code above:

- `checkout` with an empty cart sends `HxRedirect("/")`, so the browser loads the catalog as a fresh page.
- `checkout` with items places the order and sends `HxLocation` with the receipt's path and `#main` as the target. htmx then requests `/orders/1001` with `HX-Request: true` and `HX-Target: main`, and `orderPage` answers with the fragment for `#main`. A reload of `/orders/1001` gets the full page from the same handler.
- The checkout empties the cart, but htmx never swaps the body of a response with `HX-Location`, so an out-of-band badge there would be ignored. `checkout` announces `cart-updated` instead, which refreshes the summary, and the receipt fragment carries the out-of-band badge. htmx also finds `hx-swap-oob` elements nested inside the main fragment.
- `clearCart` sends `HxRefresh(true)`, and the browser reloads `/cart` with the badge back at 0.

Place an order with the two teas in the cart, and check the header text:

```scala
show(checkout())
// status: 200
// hx-location: {"path":"/orders/1001","target":"#main"}
// hx-trigger: cart-updated
// 
show(orderPage(get("/orders/1001", HxRequest(true), HxTargetId("main")), 1001))
// status: 200
// vary: hx-request, hx-boosted, hx-history-restore-request
// Order 1001 PlacedThank you! Your tea is on its way.Keep shopping0
show(checkout())
// status: 200
// hx-redirect: /
//
```

The first checkout sends `hx-location` with the JSON object htmx expects, and the second one, with the cart now empty, sends `hx-redirect`. In a browser, "Place order" swaps the receipt into `#main`, the header and the summary stay and both drop to 0 items, the address bar shows `/orders/1001`, and Back returns to the cart. The tab title still says "Cart - Tea Shop", because the fragment has no `<title>`. Back shows the cart as htmx saved it in its history cache, with both teas and a badge of 2: a history cache hit restores the saved snapshot without asking the server.

:::warning
Always give `HxLocation` a target that is a plain CSS selector, such as `HxTarget.css("#main")`. Without a target, htmx swaps the answer into `<body>`; our handler sees `HX-Request: true` without `HX-Boosted` and sends the `#main` fragment, so the header and the summary disappeared in the browser. `HxLocation` also accepts relative targets, such as `HxTarget.closest("main")`, but htmx 2 resolves the `HX-Location` target with `document.querySelector`, so in the browser it raised `htmx:targetError` and nothing happened. This is a known bug in 0.0.56.
:::

`HX-Redirect`, `HX-Location` and `HX-Refresh` end htmx's handling of the response before the swap. An `HxTriggerHeader` sent with them still fires, but `HxTriggerAfterSwap` and `HxTriggerAfterSettle` don't: in the browser, a response with `HX-Redirect` and `HX-Trigger-After-Settle` fired only the `HX-Trigger` event before the new page loaded.

## Send Validation Errors Somewhere Else

The promo-code form targets `#cart-total`: a valid code sends the new total, and the form's `hxSwap := HxSwap.OuterHTML` puts it in place. A wrong code should not replace the total. It should put an error message next to the form. `HxRetarget` changes the target for this one response, and `HxReswap` changes how the content is swapped:

```scala
def applyPromo(request: Request): Response = {
  val code = Form.fromString(request.body.asString()).get("code").getOrElse("").trim.toUpperCase
  if (code == "TEA10") {
    cart.applyPromo(code)
    val total      = p(id := "cart-total")(s"Total: ${price(cart.totalCents)} (TEA10 applied)")
    val clearError = span(id := "promo-error", hxSwapOob := HxSwapOob.True)("")
    htmlResponse(total.render + clearError.render).addHeader(HxTriggerHeader(HxEventPayload.Event("cart-updated")))
  } else
    htmlResponse(span(s"$code is not a valid promo code").render)
      .addHeader(HxRetarget(CssSelector.id("promo-error")))
      .addHeader(HxReswap(HxSwap.InnerHTML))
}
```

The code above handles the two outcomes:

- A valid code applies the discount and sends the new `#cart-total`. An out-of-band `#promo-error` clears a message left by an earlier attempt, and `cart-updated` refreshes the summary on the side.
- An invalid code sends the message only. `HxRetarget(CssSelector.id("promo-error"))` sends it to `#promo-error` instead of `#cart-total`, and `HxReswap(HxSwap.InnerHTML)` puts it inside that element instead of replacing the element. The code the reader typed is rendered as text by the HTML module, so a code such as `` can't add markup to the page.
- `Form.fromString` decodes the `application/x-www-form-urlencoded` body that htmx sends for a form.

The checkout above emptied the cart, so put a Matcha in it first:

```scala
cart.add("matcha")
```

Try a wrong code, then the right one:

```scala
show(applyPromo(post("/cart/promo", Form("code" -> "TEA5"), HxRequest(true))))
// status: 200
// hx-retarget: #promo-error
// hx-reswap: innerHTML
// TEA5 is not a valid promo code
show(applyPromo(post("/cart/promo", Form("code" -> "tea10"), HxRequest(true))))
// status: 200
// hx-trigger: cart-updated
// Total: $21.60 (TEA10 applied)
```

In a browser, the wrong code leaves the total alone and shows "TEA5 is not a valid promo code" under the form, and the right code updates the total, clears the message and refreshes the summary.

:::warning
Send the error with status `200`. htmx 2 swaps `2xx` and `3xx` responses by default, except `204`: in the browser, the same response with status `422 Unprocessable Entity` swapped nothing, ignored `HX-Retarget` and `HX-Reswap`, and fired `htmx:responseError`. To keep a `4xx` status, change `htmx.config.responseHandling` on the page so that htmx swaps that status.
:::

## Update the Address Bar from the Server

The category links push their URL with the `hxPushUrl` attribute, so each category is a new entry in the browser history. A sort change is different: the reader is looking at the same list in another order, and Back should not step through every sort. The sort links carry no `hx-push-url`, and the server decides instead: `HxReplaceUrl` replaces the current address without a new history entry, and `HxPushUrl` adds one:

```scala
def catalogRoute(request: Request): Response = {
  val response = catalogPage(request)
  request.queryParams.getFirst("sort") match {
    case Some(sort) if wantsFragment(request) =>
      val category = request.queryParams.getFirst("category").getOrElse("")
      response.addHeader(HxReplaceUrl(HxUrlUpdate(catalogUrl(category, sort))))
    case _ => response
  }
}
```

`HxUrlUpdate` is the same type the `hxPushUrl` attribute takes, so `HxUrlUpdate(true)`, `HxUrlUpdate(false)` and a URL work in both places. The handler adds the header only to an htmx fragment, because a full page load already shows its own URL. `catalogUrl` leaves out the default sort, so a sort by name replaces the address with `/?category=Green` and a sort by price with `/?category=Green&sort=price`:

```scala
show(catalogRoute(get("/?category=Green&sort=price", HxRequest(true))))
// status: 200
// vary: hx-request, hx-boosted, hx-history-restore-request
// hx-replace-url: /?category=Green&sort=price
// Sort by name | priceSencha $12.00 <button id="add-sencha" hx-post="/cart/items/sencha" hx-target="this" hx-swap="outerHTML">Add to cart</button>Matcha $24.00 <button id="add-matcha" hx-post="/cart/items/matcha" hx-target="this" hx-swap="outerHTML">In cart (1)</button>
```

In a browser, a click on "Green" adds a history entry, and a click on "price" changes the address to `/?category=Green&sort=price` without one, so a single Back goes from the sorted green teas straight to the page before "Green". A `HX-Push-Url` header in the same place adds an entry instead, and its URL wins over the request's own URL.

Every URL the server pushes or replaces must work as a full page, because htmx asks for it again on a history cache miss and the reader can reload or bookmark it. Here, `catalogRoute` answers `/?category=Green&sort=price` from the same handler and the same `wantsFragment` check as before, so a reload, a bookmark and a Back with a history restore all get the full page.

## Putting It Together

The complete example serves the whole shop with the JDK HTTP server from [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md). It routes each URL to the handlers from the sections above. It differs from them in four ways: the shop lives in its own `CompleteShop` object, because an object that defines `main` would make the `main(...)` element in the layout call that method; it declares the model and helpers itself instead of importing them from `TeaShop.scala`, so the file stands alone; it adds a `notFound` handler; and a single `app` function routes every URL. `main` starts the server on port 8080 or on the port given as the first argument:

```scala title="zio-blocks-htmx-examples/src/main/scala/returnhtmxfragments/CompleteExample.scala"
package returnhtmxfragments

/**
 * How to Return HTMX Fragments, Full Pages and Response Headers — Complete
 * Example
 *
 * The whole tea shop in one file: the catalog with category and sort links,
 * "Add to cart" with an out-of-band badge and trigger events, the cart page
 * with a promo code, checkout, and the address-bar updates. The shop lives in
 * its own object because an object that defines `main` would make the
 * `main(...)` element in the layout call that method.
 *
 * Run with: cd zio-blocks-htmx-examples && sbt "runMain
 * returnhtmxfragments.CompleteExample", then open http://localhost:8080/
 */
object CompleteShop {
  final case class Product(id: String, name: String, category: String, priceCents: Int)

  val products: List[Product] = List(
    Product("sencha", "Sencha", "Green", 1200),
    Product("matcha", "Matcha", "Green", 2400),
    Product("assam", "Assam", "Black", 900),
    Product("earl-grey", "Earl Grey", "Black", 1000),
    Product("rooibos", "Rooibos", "Herbal", 800)
  )

  val categories: List[String] = List("Green", "Black", "Herbal")

  def price(cents: Int): String = "$" + (cents / 100) + "." + f"${cents % 100}%02d"

  final class Cart {
    private val lines = new AtomicReference(Map.empty[String, Int])
    private val promo = new AtomicReference(Option.empty[String])

    def add(productId: String): Int =
      lines.updateAndGet(current => current.updated(productId, current.getOrElse(productId, 0) + 1))(productId)
    def quantity(productId: String): Int = lines.get.getOrElse(productId, 0)
    def count: Int                       = lines.get.values.sum
    def applyPromo(code: String): Unit   = promo.set(Some(code))
    def totalCents: Int                  = {
      val gross = lines.get.map { case (id, n) => products.find(_.id == id).fold(0)(_.priceCents) * n }.sum
      if (promo.get.contains("TEA10")) gross * 9 / 10 else gross
    }
    def clear(): Unit = { lines.set(Map.empty); promo.set(None) }
  }

  val cart = new Cart

  val htmxScript: Dom.Element = script(src := "https://cdn.jsdelivr.net/npm/htmx.org@2.0.11/dist/htmx.min.js")

  def catalogUrl(category: String, sort: String): String = {
    val params = List("category" -> category, "sort" -> sort).filter { case (k, v) =>
      v.nonEmpty && !(k == "sort" && v == "name")
    }
    if (params.isEmpty) "/" else params.map { case (k, v) => s"$k=$v" }.mkString("/?", "&", "")
  }

  def addButton(product: Product): Dom.Element = {
    val quantity = cart.quantity(product.id)
    button(
      id       := s"add-${product.id}",
      hxPost   := s"/cart/items/${product.id}",
      hxTarget := HxTarget.this_,
      hxSwap   := HxSwap.OuterHTML
    )(if (quantity == 0) "Add to cart" else s"In cart ($quantity)")
  }

  def catalog(category: String, sort: String): Dom.Element = {
    val shown  = products.filter(product => category.isEmpty || product.category == category)
    val sorted = if (sort == "price") shown.sortBy(_.priceCents) else shown.sortBy(_.name)
    div(id := "catalog")(
      p(
        "Sort by ",
        a(hxGet := catalogUrl(category, "name"), hxTarget := HxTarget.css("#catalog"), hxSwap := HxSwap.OuterHTML)(
          "name"
        ),
        " | ",
        a(hxGet := catalogUrl(category, "price"), hxTarget := HxTarget.css("#catalog"), hxSwap := HxSwap.OuterHTML)(
          "price"
        )
      ),
      ul(sorted.map(product => li(strong(product.name), s" ${price(product.priceCents)} ", addButton(product))))
    )
  }

  def categoryLinks: Dom.Element =
    p(
      ("" :: categories).map { category =>
        a(
          href      := catalogUrl(category, ""),
          hxGet     := catalogUrl(category, ""),
          hxTarget  := HxTarget.css("#catalog"),
          hxSwap    := HxSwap.OuterHTML,
          hxPushUrl := HxUrlUpdate(true)
        )(if (category.isEmpty) "All teas " else s"$category ")
      }
    )

  def catalogContent(category: String, sort: String): Dom.Element =
    div(h1("Teas"), categoryLinks, catalog(category, sort))

  def cartBadge: Dom.Element = span(id := "cart-count")(cart.count.toString)

  def cartSummary: Dom.Element =
    aside(
      id        := "cart-summary",
      hxGet     := "/cart/summary",
      hxTrigger := HxTrigger("cart-updated").from("body"),
      hxSwap    := HxSwap.OuterHTML
    )(s"${cart.count} items, ${price(cart.totalCents)}")

  def layout(heading: String, content: Dom.Element): Dom.Element =
    html(lang := "en")(
      head(meta(charset := "utf-8"), title(s"$heading - Tea Shop"), htmxScript),
      body(hxOn("show-message") := Js("document.getElementById('message').textContent = event.detail.text"))(
        header(nav(hxBoost := true)(a(href := "/")("Teas"), " ", a(href := "/cart")("Cart (", cartBadge, ")"))),
        p(id := "message")(""),
        main(id := "main")(content),
        cartSummary
      )
    )

  def htmlResponse(html: String): Response =
    Response.ok(Body.fromArray(html.getBytes("UTF-8"), ContentType.`text/html`))

  def wantsFragment(request: Request): Boolean =
    request.header(HxRequest).exists(_.enabled) &&
      !request.header(HxBoosted).exists(_.enabled) &&
      !request.header(HxHistoryRestoreRequest).exists(_.enabled)

  def pageOrFragment(request: Request, heading: String)(fragment: => Dom.Element, content: => Dom.Element): Response = {
    val html =
      if (wantsFragment(request)) fragment.render
      else doctype.render + layout(heading, content).render
    htmlResponse(html).addHeader("vary", "hx-request, hx-boosted, hx-history-restore-request")
  }

  def catalogPage(request: Request): Response = {
    val category = request.queryParams.getFirst("category").getOrElse("")
    val sort     = request.queryParams.getFirst("sort").getOrElse("name")
    pageOrFragment(request, "Teas")(catalog(category, sort), catalogContent(category, sort))
  }

  def addToCart(productId: String): Response =
    products.find(_.id == productId) match {
      case None          => Response.text(Status.NotFound, s"No product $productId")
      case Some(product) =>
        cart.add(product.id)
        val badge = cartBadge(hxSwapOob := HxSwapOob.True)
        htmlResponse(addButton(product).render + badge.render)
    }

  def addToCartAndAnnounce(productId: String): Response = {
    val response = addToCart(productId)
    products.find(_.id == productId).fold(response) { product =>
      val message = Json.Object("show-message" -> Json.Object("text" -> Json.String(s"Added ${product.name}")))
      response
        .addHeader(HxTriggerHeader(HxEventPayload.Event("cart-updated")))
        .addHeader(HxTriggerAfterSettle(HxEventPayload.JsonValue(message)))
    }
  }

  def cartSummaryFragment: Response = htmlResponse(cartSummary.render)

  val orderNumbers = new AtomicInteger(1000)

  def cartContent: Dom.Element =
    div(
      h1("Your Cart"),
      ul(products.filter(product => cart.quantity(product.id) > 0).map { product =>
        li(s"${cart.quantity(product.id)} x ${product.name}")
      }),
      p(id := "cart-total")(s"Total: ${price(cart.totalCents)}"),
      form(hxPost := "/cart/promo", hxTarget := HxTarget.css("#cart-total"), hxSwap := HxSwap.OuterHTML)(
        input(name := "code", placeholder := "Promo code"),
        button(`type` := "submit")("Apply"),
        span(id := "promo-error")("")
      ),
      button(hxPost := "/checkout")("Place order"),
      button(hxPost := "/cart/clear")("Empty cart")
    )

  def orderContent(number: Int): Dom.Element =
    div(h1(s"Order $number Placed"), p("Thank you! Your tea is on its way."), a(href := "/")("Keep shopping"))

  def checkout(): Response =
    if (cart.count == 0) htmlResponse("").addHeader(HxRedirect("/"))
    else {
      val number = orderNumbers.incrementAndGet()
      cart.clear()
      htmlResponse("")
        .addHeader(HxLocation(s"/orders/$number", target = Some(HxTarget.css("#main"))))
        .addHeader(HxTriggerHeader(HxEventPayload.Event("cart-updated")))
    }

  def clearCart(): Response = {
    cart.clear()
    htmlResponse("").addHeader(HxRefresh(true))
  }

  def orderPage(request: Request, number: Int): Response =
    pageOrFragment(request, s"Order $number")(
      div(orderContent(number), cartBadge(hxSwapOob := HxSwapOob.True)),
      orderContent(number)
    )

  def applyPromo(request: Request): Response = {
    val code = Form.fromString(request.body.asString()).get("code").getOrElse("").trim.toUpperCase
    if (code == "TEA10") {
      cart.applyPromo(code)
      val total      = p(id := "cart-total")(s"Total: ${price(cart.totalCents)} (TEA10 applied)")
      val clearError = span(id := "promo-error", hxSwapOob := HxSwapOob.True)("")
      htmlResponse(total.render + clearError.render).addHeader(HxTriggerHeader(HxEventPayload.Event("cart-updated")))
    } else
      htmlResponse(span(s"$code is not a valid promo code").render)
        .addHeader(HxRetarget(CssSelector.id("promo-error")))
        .addHeader(HxReswap(HxSwap.InnerHTML))
  }

  def catalogRoute(request: Request): Response = {
    val response = catalogPage(request)
    request.queryParams.getFirst("sort") match {
      case Some(sort) if wantsFragment(request) =>
        val category = request.queryParams.getFirst("category").getOrElse("")
        response.addHeader(HxReplaceUrl(HxUrlUpdate(catalogUrl(category, sort))))
      case _ => response
    }
  }

  def notFound(request: Request): Response =
    Response.text(Status.NotFound, s"No route for ${request.method.name} ${request.path.encode}")

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil)                                                    => catalogRoute(request)
      case (Method.POST, List("cart", "items", id))                             => addToCartAndAnnounce(id)
      case (Method.GET, List("cart", "summary"))                                => cartSummaryFragment
      case (Method.GET, List("cart"))                                           => pageOrFragment(request, "Cart")(cartContent, cartContent)
      case (Method.POST, List("cart", "promo"))                                 => applyPromo(request)
      case (Method.POST, List("checkout"))                                      => checkout()
      case (Method.POST, List("cart", "clear"))                                 => clearCart()
      case (Method.GET, List("orders", number)) if number.toIntOption.isDefined =>
        orderPage(request, number.toInt)
      case _ => notFound(request)
    }
}

object CompleteExample {
  def main(args: Array[String]): Unit = {
    val port   = args.headOption.flatMap(_.toIntOption).getOrElse(8080)
    val server = JdkHttp.start(CompleteShop.app, port)
    sys.addShutdownHook(server.stop(0))
    println(s"Open http://localhost:$port/ in a browser. Press Ctrl+C to stop.")
  }
}
```

Run it and open `http://localhost:8080/`. Pick a category, sort by price, add two teas and watch the badge, the summary and the message line change, then open the cart through the header, try the promo codes "TEA5" and "TEA10", and place the order. Open the browser's developer tools on the Network tab to see the `HX-*` headers of each request and response.

## Running the Examples

All examples in this guide have corresponding runnable Scala files in the `zio-blocks-htmx-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/zio-blocks-htmx-examples
```

Each example prints what its handlers answer to the requests from its section, then starts the server on port 8080 and keeps running so you can open `http://localhost:8080/` in a browser. Press Ctrl+C to stop it before running the next one. To use another port, pass it as an argument, for example `sbt "runMain returnhtmxfragments.Step1FullPageOrFragmentExample 9090"`. The examples share the tea shop in `TeaShop.scala`, and `JdkHttp.scala` is the server adapter from [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md), unchanged.

<details>
  <summary>zio-blocks-htmx-examples/src/main/scala/returnhtmxfragments/TeaShop.scala</summary>

```scala title="zio-blocks-htmx-examples/src/main/scala/returnhtmxfragments/TeaShop.scala" showLineNumbers
package returnhtmxfragments

/**
 * How to Return HTMX Fragments, Full Pages and Response Headers — the tea shop
 *
 * Five teas, one in-memory cart, the page layout with its cart badge and
 * summary, the full-page-or-fragment check, and the helpers that print a
 * response and start the server. Every step of the guide reuses these.
 */
object TeaShop {
  final case class Product(id: String, name: String, category: String, priceCents: Int)

  val products: List[Product] = List(
    Product("sencha", "Sencha", "Green", 1200),
    Product("matcha", "Matcha", "Green", 2400),
    Product("assam", "Assam", "Black", 900),
    Product("earl-grey", "Earl Grey", "Black", 1000),
    Product("rooibos", "Rooibos", "Herbal", 800)
  )

  val categories: List[String] = List("Green", "Black", "Herbal")

  def price(cents: Int): String = "$" + (cents / 100) + "." + f"${cents % 100}%02d"

  final class Cart {
    private val lines = new AtomicReference(Map.empty[String, Int])
    private val promo = new AtomicReference(Option.empty[String])

    def add(productId: String): Int =
      lines.updateAndGet(current => current.updated(productId, current.getOrElse(productId, 0) + 1))(productId)
    def quantity(productId: String): Int = lines.get.getOrElse(productId, 0)
    def count: Int                       = lines.get.values.sum
    def applyPromo(code: String): Unit   = promo.set(Some(code))
    def totalCents: Int                  = {
      val gross = lines.get.map { case (id, n) => products.find(_.id == id).fold(0)(_.priceCents) * n }.sum
      if (promo.get.contains("TEA10")) gross * 9 / 10 else gross
    }
    def clear(): Unit = { lines.set(Map.empty); promo.set(None) }
  }

  val cart = new Cart

  val htmxScript: Dom.Element = script(src := "https://cdn.jsdelivr.net/npm/htmx.org@2.0.11/dist/htmx.min.js")

  def catalogUrl(category: String, sort: String): String = {
    val params = List("category" -> category, "sort" -> sort).filter { case (k, v) =>
      v.nonEmpty && !(k == "sort" && v == "name")
    }
    if (params.isEmpty) "/" else params.map { case (k, v) => s"$k=$v" }.mkString("/?", "&", "")
  }

  def addButton(product: Product): Dom.Element = {
    val quantity = cart.quantity(product.id)
    button(
      id       := s"add-${product.id}",
      hxPost   := s"/cart/items/${product.id}",
      hxTarget := HxTarget.this_,
      hxSwap   := HxSwap.OuterHTML
    )(if (quantity == 0) "Add to cart" else s"In cart ($quantity)")
  }

  def catalog(category: String, sort: String): Dom.Element = {
    val shown  = products.filter(product => category.isEmpty || product.category == category)
    val sorted = if (sort == "price") shown.sortBy(_.priceCents) else shown.sortBy(_.name)
    div(id := "catalog")(
      p(
        "Sort by ",
        a(hxGet := catalogUrl(category, "name"), hxTarget := HxTarget.css("#catalog"), hxSwap := HxSwap.OuterHTML)(
          "name"
        ),
        " | ",
        a(hxGet := catalogUrl(category, "price"), hxTarget := HxTarget.css("#catalog"), hxSwap := HxSwap.OuterHTML)(
          "price"
        )
      ),
      ul(sorted.map(product => li(strong(product.name), s" ${price(product.priceCents)} ", addButton(product))))
    )
  }

  def categoryLinks: Dom.Element =
    p(
      ("" :: categories).map { category =>
        a(
          href      := catalogUrl(category, ""),
          hxGet     := catalogUrl(category, ""),
          hxTarget  := HxTarget.css("#catalog"),
          hxSwap    := HxSwap.OuterHTML,
          hxPushUrl := HxUrlUpdate(true)
        )(if (category.isEmpty) "All teas " else s"$category ")
      }
    )

  def catalogContent(category: String, sort: String): Dom.Element =
    div(h1("Teas"), categoryLinks, catalog(category, sort))

  def cartBadge: Dom.Element = span(id := "cart-count")(cart.count.toString)

  def cartSummary: Dom.Element =
    aside(
      id        := "cart-summary",
      hxGet     := "/cart/summary",
      hxTrigger := HxTrigger("cart-updated").from("body"),
      hxSwap    := HxSwap.OuterHTML
    )(s"${cart.count} items, ${price(cart.totalCents)}")

  def layout(heading: String, content: Dom.Element): Dom.Element =
    html(lang := "en")(
      head(meta(charset := "utf-8"), title(s"$heading - Tea Shop"), htmxScript),
      body(hxOn("show-message") := Js("document.getElementById('message').textContent = event.detail.text"))(
        header(nav(hxBoost := true)(a(href := "/")("Teas"), " ", a(href := "/cart")("Cart (", cartBadge, ")"))),
        p(id := "message")(""),
        main(id := "main")(content),
        cartSummary
      )
    )

  def htmlResponse(html: String): Response =
    Response.ok(Body.fromArray(html.getBytes("UTF-8"), ContentType.`text/html`))

  def get(target: String, headers: Header*): Request =
    headers.foldLeft(Request.get(URL.parse(target).toOption.get))((request, header) => request.addHeader(header))

  def post(target: String, form: Form, headers: Header*): Request =
    headers.foldLeft(Request.post(URL.parse(target).toOption.get, Body.fromString(form.encode)))((request, header) =>
      request.addHeader(header)
    )

  def show(response: Response): Unit = {
    println(s"status: ${response.status.code}")
    response.headers.toList.foreach { case (name, value) =>
      if (name.startsWith("hx-") || name == "vary") println(s"$name: $value")
    }
    println(response.body.asString())
  }

  def wantsFragment(request: Request): Boolean =
    request.header(HxRequest).exists(_.enabled) &&
      !request.header(HxBoosted).exists(_.enabled) &&
      !request.header(HxHistoryRestoreRequest).exists(_.enabled)

  def pageOrFragment(request: Request, heading: String)(fragment: => Dom.Element, content: => Dom.Element): Response = {
    val html =
      if (wantsFragment(request)) fragment.render
      else doctype.render + layout(heading, content).render
    htmlResponse(html).addHeader("vary", "hx-request, hx-boosted, hx-history-restore-request")
  }

  def catalogPage(request: Request): Response = {
    val category = request.queryParams.getFirst("category").getOrElse("")
    val sort     = request.queryParams.getFirst("sort").getOrElse("name")
    pageOrFragment(request, "Teas")(catalog(category, sort), catalogContent(category, sort))
  }

  /** Answers 404 for any route the step does not handle. */
  def notFound(request: Request): Response =
    Response.text(Status.NotFound, s"No route for ${request.method.name} ${request.path.encode}")

  /**
   * Starts the server on the port given as the first argument (8080 by default)
   * and returns the port. The server keeps running until the program is stopped
   * with Ctrl+C.
   */
  def serve(app: Request => Response, args: Array[String]): Int = {
    val port   = args.headOption.flatMap(_.toIntOption).getOrElse(8080)
    val server = JdkHttp.start(app, port)
    sys.addShutdownHook(server.stop(0))
    port
  }

  def openInBrowser(port: Int): Unit =
    println(s"Open http://localhost:$port/ in a browser. Press Ctrl+C to stop.")
}
```

</details>

<details>
  <summary>zio-blocks-htmx-examples/src/main/scala/returnhtmxfragments/Step1FullPageOrFragmentExample.scala</summary>

```scala title="zio-blocks-htmx-examples/src/main/scala/returnhtmxfragments/Step1FullPageOrFragmentExample.scala" showLineNumbers
package returnhtmxfragments

/**
 * How to Return HTMX Fragments, Full Pages and Response Headers — Answer One
 * URL with a Full Page or a Fragment
 *
 * Answers `/` with the full page, or with the `#catalog` fragment when an htmx
 * link asks for it. The `wantsFragment` check reads `HX-Request`, `HX-Boosted`
 * and `HX-History-Restore-Request`.
 *
 * Run with: cd zio-blocks-htmx-examples && sbt "runMain
 * returnhtmxfragments.Step1FullPageOrFragmentExample"
 */
object Step1FullPageOrFragmentExample {
  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil) => catalogPage(request)
      case _                 => notFound(request)
    }

  def main(args: Array[String]): Unit = {
    println(wantsFragment(get("/?category=Green")))
    println(wantsFragment(get("/?category=Green", HxRequest(true))))
    println(wantsFragment(get("/cart", HxRequest(true), HxBoosted(true))))
    println(wantsFragment(get("/?category=Green", HxRequest(true), HxHistoryRestoreRequest(true))))
    println(get("/", Header.Custom("hx-request", "yes")).header(HxRequest))
    show(catalogPage(get("/?category=Green", HxRequest(true))))
    val port = serve(app, args)
    openInBrowser(port)
  }
}
```

</details>

```bash
sbt "runMain returnhtmxfragments.Step1FullPageOrFragmentExample"
```

<details>
  <summary>zio-blocks-htmx-examples/src/main/scala/returnhtmxfragments/Step2OutOfBandSwapExample.scala</summary>

```scala title="zio-blocks-htmx-examples/src/main/scala/returnhtmxfragments/Step2OutOfBandSwapExample.scala" showLineNumbers
package returnhtmxfragments

/**
 * How to Return HTMX Fragments, Full Pages and Response Headers — Update a
 * Second Spot with an Out-of-Band Swap
 *
 * "Add to cart" answers with the new button and, out of band, the new cart
 * badge.
 *
 * Run with: cd zio-blocks-htmx-examples && sbt "runMain
 * returnhtmxfragments.Step2OutOfBandSwapExample"
 */
object Step2OutOfBandSwapExample {
  def addToCart(productId: String): Response =
    products.find(_.id == productId) match {
      case None          => Response.text(Status.NotFound, s"No product $productId")
      case Some(product) =>
        cart.add(product.id)
        val badge = cartBadge(hxSwapOob := HxSwapOob.True)
        htmlResponse(addButton(product).render + badge.render)
    }

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil)                        => catalogPage(request)
      case (Method.POST, List("cart", "items", id)) => addToCart(id)
      case _                                        => notFound(request)
    }

  def main(args: Array[String]): Unit = {
    show(addToCart("sencha"))
    val port = serve(app, args)
    openInBrowser(port)
  }
}
```

</details>

```bash
sbt "runMain returnhtmxfragments.Step2OutOfBandSwapExample"
```

<details>
  <summary>zio-blocks-htmx-examples/src/main/scala/returnhtmxfragments/Step3TriggerEventsExample.scala</summary>

```scala title="zio-blocks-htmx-examples/src/main/scala/returnhtmxfragments/Step3TriggerEventsExample.scala" showLineNumbers
package returnhtmxfragments

/**
 * How to Return HTMX Fragments, Full Pages and Response Headers — Tell the Page
 * Something Happened
 *
 * "Add to cart" also announces `cart-updated` and `show-message` with
 * `HX-Trigger` headers, and `/cart/summary` answers the summary's refresh
 * request.
 *
 * Run with: cd zio-blocks-htmx-examples && sbt "runMain
 * returnhtmxfragments.Step3TriggerEventsExample"
 */
object Step3TriggerEventsExample {
  def addToCartAndAnnounce(productId: String): Response = {
    val response = Step2OutOfBandSwapExample.addToCart(productId)
    products.find(_.id == productId).fold(response) { product =>
      val message = Json.Object("show-message" -> Json.Object("text" -> Json.String(s"Added ${product.name}")))
      response
        .addHeader(HxTriggerHeader(HxEventPayload.Event("cart-updated")))
        .addHeader(HxTriggerAfterSettle(HxEventPayload.JsonValue(message)))
    }
  }

  def cartSummaryFragment: Response = htmlResponse(cartSummary.render)

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil)                        => catalogPage(request)
      case (Method.POST, List("cart", "items", id)) => addToCartAndAnnounce(id)
      case (Method.GET, List("cart", "summary"))    => cartSummaryFragment
      case _                                        => notFound(request)
    }

  def main(args: Array[String]): Unit = {
    // Setup from section 2: a Sencha is already in the cart
    Step2OutOfBandSwapExample.addToCart("sencha")
    show(addToCartAndAnnounce("assam"))
    show(cartSummaryFragment)
    val greeting = Json.Object("show-message" -> Json.Object("text" -> Json.String("Added Sencha ✓")))
    val escaped  = htmlResponse("").addHeader(HxTriggerHeader.name, greeting.print(WriterConfig.withEscapeUnicode(true)))
    println(escaped.headers.rawGet(HxTriggerHeader.name))
    println(escaped.header(HxTriggerHeader).map(_.renderedValue))
    val port = serve(app, args)
    openInBrowser(port)
  }
}
```

</details>

```bash
sbt "runMain returnhtmxfragments.Step3TriggerEventsExample"
```

<details>
  <summary>zio-blocks-htmx-examples/src/main/scala/returnhtmxfragments/Step4RedirectAfterPostExample.scala</summary>

```scala title="zio-blocks-htmx-examples/src/main/scala/returnhtmxfragments/Step4RedirectAfterPostExample.scala" showLineNumbers
package returnhtmxfragments

/**
 * How to Return HTMX Fragments, Full Pages and Response Headers — Redirect,
 * Navigate or Refresh After a Form Post
 *
 * Checkout sends `HX-Location` (or `HX-Redirect` for an empty cart), and "Empty
 * cart" sends `HX-Refresh`.
 *
 * Run with: cd zio-blocks-htmx-examples && sbt "runMain
 * returnhtmxfragments.Step4RedirectAfterPostExample"
 */
object Step4RedirectAfterPostExample {
  val orderNumbers = new AtomicInteger(1000)

  def cartContent: Dom.Element =
    div(
      h1("Your Cart"),
      ul(products.filter(product => cart.quantity(product.id) > 0).map { product =>
        li(s"${cart.quantity(product.id)} x ${product.name}")
      }),
      p(id := "cart-total")(s"Total: ${price(cart.totalCents)}"),
      form(hxPost := "/cart/promo", hxTarget := HxTarget.css("#cart-total"), hxSwap := HxSwap.OuterHTML)(
        input(name := "code", placeholder := "Promo code"),
        button(`type` := "submit")("Apply"),
        span(id := "promo-error")("")
      ),
      button(hxPost := "/checkout")("Place order"),
      button(hxPost := "/cart/clear")("Empty cart")
    )

  def orderContent(number: Int): Dom.Element =
    div(h1(s"Order $number Placed"), p("Thank you! Your tea is on its way."), a(href := "/")("Keep shopping"))

  def checkout(): Response =
    if (cart.count == 0) htmlResponse("").addHeader(HxRedirect("/"))
    else {
      val number = orderNumbers.incrementAndGet()
      cart.clear()
      htmlResponse("")
        .addHeader(HxLocation(s"/orders/$number", target = Some(HxTarget.css("#main"))))
        .addHeader(HxTriggerHeader(HxEventPayload.Event("cart-updated")))
    }

  def clearCart(): Response = {
    cart.clear()
    htmlResponse("").addHeader(HxRefresh(true))
  }

  def orderPage(request: Request, number: Int): Response =
    pageOrFragment(request, s"Order $number")(
      div(orderContent(number), cartBadge(hxSwapOob := HxSwapOob.True)),
      orderContent(number)
    )

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil)                                                    => catalogPage(request)
      case (Method.POST, List("cart", "items", id))                             => Step3TriggerEventsExample.addToCartAndAnnounce(id)
      case (Method.GET, List("cart", "summary"))                                => Step3TriggerEventsExample.cartSummaryFragment
      case (Method.GET, List("cart"))                                           => pageOrFragment(request, "Cart")(cartContent, cartContent)
      case (Method.POST, List("checkout"))                                      => checkout()
      case (Method.POST, List("cart", "clear"))                                 => clearCart()
      case (Method.GET, List("orders", number)) if number.toIntOption.isDefined =>
        orderPage(request, number.toInt)
      case _ => notFound(request)
    }

  def main(args: Array[String]): Unit = {
    // Setup from sections 2 and 3: a Sencha and an Assam are in the cart
    Step2OutOfBandSwapExample.addToCart("sencha")
    Step2OutOfBandSwapExample.addToCart("assam")
    show(checkout())
    show(orderPage(get("/orders/1001", HxRequest(true), HxTargetId("main")), 1001))
    show(checkout())
    val port = serve(app, args)
    openInBrowser(port)
  }
}
```

</details>

```bash
sbt "runMain returnhtmxfragments.Step4RedirectAfterPostExample"
```

<details>
  <summary>zio-blocks-htmx-examples/src/main/scala/returnhtmxfragments/Step5ValidationErrorsExample.scala</summary>

```scala title="zio-blocks-htmx-examples/src/main/scala/returnhtmxfragments/Step5ValidationErrorsExample.scala" showLineNumbers
package returnhtmxfragments

/**
 * How to Return HTMX Fragments, Full Pages and Response Headers — Send
 * Validation Errors Somewhere Else
 *
 * A wrong promo code is sent to `#promo-error` with `HX-Retarget` and
 * `HX-Reswap`; a right one updates the total.
 *
 * Run with: cd zio-blocks-htmx-examples && sbt "runMain
 * returnhtmxfragments.Step5ValidationErrorsExample"
 */
object Step5ValidationErrorsExample {
  def applyPromo(request: Request): Response = {
    val code = Form.fromString(request.body.asString()).get("code").getOrElse("").trim.toUpperCase
    if (code == "TEA10") {
      cart.applyPromo(code)
      val total      = p(id := "cart-total")(s"Total: ${price(cart.totalCents)} (TEA10 applied)")
      val clearError = span(id := "promo-error", hxSwapOob := HxSwapOob.True)("")
      htmlResponse(total.render + clearError.render).addHeader(HxTriggerHeader(HxEventPayload.Event("cart-updated")))
    } else
      htmlResponse(span(s"$code is not a valid promo code").render)
        .addHeader(HxRetarget(CssSelector.id("promo-error")))
        .addHeader(HxReswap(HxSwap.InnerHTML))
  }

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil)                        => catalogPage(request)
      case (Method.POST, List("cart", "items", id)) => Step3TriggerEventsExample.addToCartAndAnnounce(id)
      case (Method.GET, List("cart", "summary"))    => Step3TriggerEventsExample.cartSummaryFragment
      case (Method.GET, List("cart"))               =>
        pageOrFragment(request, "Cart")(
          Step4RedirectAfterPostExample.cartContent,
          Step4RedirectAfterPostExample.cartContent
        )
      case (Method.POST, List("cart", "promo"))                                 => applyPromo(request)
      case (Method.POST, List("checkout"))                                      => Step4RedirectAfterPostExample.checkout()
      case (Method.POST, List("cart", "clear"))                                 => Step4RedirectAfterPostExample.clearCart()
      case (Method.GET, List("orders", number)) if number.toIntOption.isDefined =>
        Step4RedirectAfterPostExample.orderPage(request, number.toInt)
      case _ => notFound(request)
    }

  def main(args: Array[String]): Unit = {
    // Setup from sections 2 to 4: the order is placed and the cart is empty again
    cart.add("matcha")
    show(applyPromo(post("/cart/promo", Form("code" -> "TEA5"), HxRequest(true))))
    show(applyPromo(post("/cart/promo", Form("code" -> "tea10"), HxRequest(true))))
    val port = serve(app, args)
    openInBrowser(port)
  }
}
```

</details>

```bash
sbt "runMain returnhtmxfragments.Step5ValidationErrorsExample"
```

<details>
  <summary>zio-blocks-htmx-examples/src/main/scala/returnhtmxfragments/Step6UpdateUrlExample.scala</summary>

```scala title="zio-blocks-htmx-examples/src/main/scala/returnhtmxfragments/Step6UpdateUrlExample.scala" showLineNumbers
package returnhtmxfragments

/**
 * How to Return HTMX Fragments, Full Pages and Response Headers — Update the
 * Address Bar from the Server
 *
 * A sort change on an htmx request replaces the address with `HX-Replace-Url`
 * instead of adding a history entry.
 *
 * Run with: cd zio-blocks-htmx-examples && sbt "runMain
 * returnhtmxfragments.Step6UpdateUrlExample"
 */
object Step6UpdateUrlExample {
  def catalogRoute(request: Request): Response = {
    val response = catalogPage(request)
    request.queryParams.getFirst("sort") match {
      case Some(sort) if wantsFragment(request) =>
        val category = request.queryParams.getFirst("category").getOrElse("")
        response.addHeader(HxReplaceUrl(HxUrlUpdate(catalogUrl(category, sort))))
      case _ => response
    }
  }

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil)                        => catalogRoute(request)
      case (Method.POST, List("cart", "items", id)) => Step3TriggerEventsExample.addToCartAndAnnounce(id)
      case (Method.GET, List("cart", "summary"))    => Step3TriggerEventsExample.cartSummaryFragment
      case _                                        => notFound(request)
    }

  def main(args: Array[String]): Unit = {
    show(catalogRoute(get("/?category=Green&sort=price", HxRequest(true))))
    val port = serve(app, args)
    openInBrowser(port)
  }
}
```

</details>

```bash
sbt "runMain returnhtmxfragments.Step6UpdateUrlExample"
```

The complete example from "Putting It Together" runs the same way:

```bash
sbt "runMain returnhtmxfragments.CompleteExample"
```

## Going Further

These pages cover the pieces this guide uses in more depth:

- [Request and Response Headers](../reference/htmx/response-headers.md) lists every typed `HX-*` header, including `HxReselect`, `HxTriggerAfterSwap` and the request headers `HxCurrentUrl`, `HxTargetId` and `HxTriggerName`.
- [`HxSwap`](../reference/htmx/hx-swap.md), [`HxTarget`](../reference/htmx/hx-target.md) and [`HxUrlUpdate`](../reference/htmx/hx-url-update.md) describe the values `HxReswap`, `HxLocation`, `HxPushUrl` and `HxReplaceUrl` reuse from the attribute DSL.
- [How to Validate Query Parameters and Headers and Answer 400](./validate-query-parameters-and-headers.md) decodes typed query parameters, such as the category and the sort, and reports every mistake at once.
- [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md) explains the adapter the examples use and its limits.

---

## How to Roll Out a Feature Gradually with Flags

In this guide, we roll out two changes to a Scala bookstore with ZIO Blocks flags: a lower free-shipping threshold that only changes on restart, and a new recommendations panel that reaches 10% of readers first, then half of the readers in the `eu` region, then more, without a redeploy. By the end, you will have flags that give each reader the same answer on every request, an expression you can widen while the service runs, a startup report that catches misspelled overrides, and tests that keep the global flag registry clean.

We start with a restart-only knob, turn the panel on for a percentage of readers, narrow it to one region, change it at runtime, check expressions before they ship, print a startup report, and finish with tests:

- [A Restart-Only Knob: The Free-Shipping Threshold](#a-restart-only-knob-the-free-shipping-threshold)
- [Turning the Panel On for 10% of Readers](#turning-the-panel-on-for-10-of-readers)
- [Targeting One Region](#targeting-one-region)
- [Changing the Rollout While the Service Runs](#changing-the-rollout-while-the-service-runs)
- [Checking an Expression Before You Ship It](#checking-an-expression-before-you-ship-it)
- [Catching Typos at Startup](#catching-typos-at-startup)
- [Testing Code That Uses Flags](#testing-code-that-uses-flags)

## The Problem

A gradual rollout needs three things a hand-rolled boolean does not give you: the same reader must get the same answer on every request, the share must be adjustable without a deploy, and a mistyped override must be noticed instead of silently ignored.

Without a flag library, the recommendations panel usually starts like this:

```scala

object Recommendations {
  // Read once at startup: changing the share needs a restart.
  private val share: Double =
    sys.env.get("RECOMMENDATIONS_SHARE").flatMap(_.toDoubleOption).getOrElse(0.0)

  // Re-rolled on every call: a reader sees the panel, reloads the page, and it is gone.
  def showPanel(readerId: String, region: String): Boolean =
    region == "eu" && Random.nextDouble() < share
}
```

The coin flip makes the panel flicker for every reader, a typo such as `RECOMENDATIONS_SHARE` falls back to `0.0` without a word, and the region rule is code, so every change to it is a deploy. This guide replaces the coin flip with sticky percentage bucketing and the hard-coded rule with a rollout expression that can be replaced at runtime.

## Prerequisites

Add the config module to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-config" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-config" % "0.0.56"
```

This guide assumes you have loaded configuration with ZIO Blocks before; [Build a Typed Configuration Loader](./typed-config-loader.md) covers sources and provenance. Most snippets below use one import:

```scala

```

Flags are Scala `object`s, and each flag's name is the object's fully-qualified name. On this page, mdoc nests every object inside `repl.MdocSession.MdocApp`, so that prefix appears in the names below; in your application, the prefix is your package and enclosing objects, as the [companion examples](#running-the-examples) show.

## A Restart-Only Knob: The Free-Shipping Threshold

The free-shipping threshold is a setting the bookstore changes rarely and only on restart, so it is a [`StaticFlag`](../reference/config/flags.md). A `StaticFlag[A]` takes a default and resolves its value once, the first time anything touches the object.

Its value comes from the first place that has one: a source registered in `FlagSource.Registry`, then a system property with the flag's name, then an environment variable with the dots replaced by underscores and uppercased, then the default. Here, an operator lowers the threshold from $35 to $25 through a map source:

```scala
FlagSource.Registry.register(
  FlagSource.fromMap(
    Map("repl.MdocSession.MdocApp.FreeShippingThreshold" -> "25"),
    "ops-overrides"
  )
)

object FreeShippingThreshold extends StaticFlag[Int](35)
```

Touching the flag resolves it. `StaticFlag#source` tells you which step answered:

```scala
println(FreeShippingThreshold.name)
// repl.MdocSession.MdocApp.FreeShippingThreshold
println(FreeShippingThreshold())
// 25
println(FreeShippingThreshold.source)
// FlagSourceValue(ops-overrides)
```

:::warning[Register sources before the first touch]
A `StaticFlag` reads the registry once, when its object initializes. A source registered after that is never consulted for that flag, so register every source at the top of `main`, before any code reads a flag.
:::

A value that does not parse stops the flag at first touch instead of handing out a plausible wrong value. The object initializer throws `ExceptionInInitializerError` whose cause is a `FlagException.FlagValueParseException`:

```scala
FlagSource.Registry.register(
  FlagSource.fromMap(
    Map("repl.MdocSession.MdocApp.MaxBooksPerOrder" -> "ten"),
    "bad-overrides"
  )
)

object MaxBooksPerOrder extends StaticFlag[Int](20)
```

Touching the flag now fails at initialization:

```scala
try println(MaxBooksPerOrder())
catch {
  case e: ExceptionInInitializerError => println(e.getCause.getMessage)
}
// Failed to parse value 'ten' for flag 'repl.MdocSession.MdocApp.MaxBooksPerOrder' (expected Int): Invalid value 'ten' for key 'repl.MdocSession.MdocApp.MaxBooksPerOrder' (expected Int) in source 'flag': For input string: "ten"
```

For a top-level object, the JVM does not retry a failed initializer: every later touch throws `NoClassDefFoundError: Could not initialize class`. Let the error stop the service at startup rather than catching it and carrying on.

## Turning the Panel On for 10% of Readers

The recommendations panel needs a decision per reader that can change while the service runs, so it is a `DynamicFlag`. A `DynamicFlag[A]` takes a default value and a default rollout expression. The expression `true@*/10%; false` reads as two choices, tried left to right: `true` for 10% of any key, otherwise `false`:

```scala
object RecommendationsPanel extends DynamicFlag[Boolean](false, "true@*/10%; false")
```

Call the flag with the reader's id. The id is the bucketing key: [`Rollout.bucketFor`](../reference/config/rollout.md) hashes it with MurmurHash3 into a bucket from 0 to 99, and `10%` matches buckets 0 to 9. The hash depends only on the id, so the answer for a reader never changes between calls, requests or restarts:

```scala
println(Rollout.bucketFor("reader-42"))
// 79
println((1 to 3).map(_ => RecommendationsPanel("reader-42")))
// Vector(false, false, false)
```

Across 1000 synthetic reader ids, the share that sees the panel is close to the 10% the expression names:

```scala
val readers = (1 to 1000).map(i => s"reader-$i")
val panelCohort = readers.filter(RecommendationsPanel.evaluate(_))
```

Printing the cohort size and its first five readers shows the share:

```scala
println(s"${panelCohort.size} of ${readers.size} readers see the panel")
// 97 of 1000 readers see the panel
println(panelCohort.take(5))
// Vector(reader-7, reader-12, reader-24, reader-36, reader-127)
```

`DynamicFlag#apply` also counts each call per key for the [evaluation counters](#changing-the-rollout-while-the-service-runs); `DynamicFlag#evaluate` returns the same answer without counting, which is what a batch like this one should use.

:::warning[Every rollout buckets the same readers]
The bucket depends on the key alone, not on the flag. Two features both at `10%` and keyed by the plain reader id reach exactly the same readers, so a reader in bucket 3 gets every early rollout at once. Prefix the key with the feature's name when rollouts should be independent:

```scala
object WishlistSharing extends DynamicFlag[Boolean](false, "true@*/10%; false")

val wishlistPlain = readers.filter(WishlistSharing.evaluate(_))
val panelSalted = readers.filter(id => RecommendationsPanel.evaluate(s"recommendations:$id"))
val wishlistSalted = readers.filter(id => WishlistSharing.evaluate(s"wishlist:$id"))
```

Comparing the cohorts shows how many readers appear in both rollouts:

```scala
println(s"in both rollouts, plain key:  ${panelCohort.intersect(wishlistPlain).size}")
// in both rollouts, plain key:  97
println(s"in both rollouts, salted key: ${panelSalted.intersect(wishlistSalted).size}")
// in both rollouts, salted key: 17
```

Use the same prefix at every call site of a flag, or the reader moves to another bucket.
:::

## Targeting One Region

To offer the panel to half of the readers in the `eu` region and to nobody else, pass the region as an attribute after the reader id. The flag joins the key and the attributes with slashes into a path such as `reader-42/eu`, and a selector must match that path segment by segment. `*` matches the reader id, `eu` must match the region literally, and `50%` applies to whatever matched:

```scala
object RegionalPanel extends DynamicFlag[Boolean](false, "true@*/eu/50%; false")
```

Counting the readers each region sees shows the targeting:

```scala
println(readers.count(id => RegionalPanel.evaluate(id, "eu")))
// 509
println(readers.count(id => RegionalPanel.evaluate(id, "us")))
// 0
```

Readers outside `eu` fall through to the catch-all `false`. Rules for more regions are more choices, tried in order: `true@*/eu/100%; true@*/us/25%; false` gives the panel to every `eu` reader and a quarter of the `us` readers.

**The segment count must match exactly.** A selector with two segments never matches a path with one or three. A call site that forgets the region builds the one-segment path `reader-42`, which `*/eu/50%` never matches, so the flag silently answers with the catch-all:

```scala
println(readers.count(id => RegionalPanel.evaluate(id)))
// 0
```

The full matching rules are in [Rollout](../reference/config/rollout.md).

## Changing the Rollout While the Service Runs

A rollout widens in steps: 10%, then 25%, then 50%. `DynamicFlag#update` replaces the expression from code and returns `Right(())` when the new expression parses:

```scala
println(RecommendationsPanel.update("true@*/25%; false"))
// Right(())
```

Widening keeps everyone who already had the panel. A reader in bucket 7 matches `bucket < 10` and still matches `bucket < 25`, so nobody loses the panel when the share grows:

```scala
val widened = readers.filter(RecommendationsPanel.evaluate(_))
```

Comparing the two cohorts shows that nobody lost the panel:

```scala
println(s"${widened.size} readers, original cohort kept: ${panelCohort.forall(widened.contains)}")
// 232 readers, original cohort kept: true
```

An expression that does not parse comes back as `Left(ConfigError)`, and the current expression stays in place:

```scala
println(RecommendationsPanel.update("true@*/150%; false"))
// Left(zio.blocks.config.ConfigError$InvalidValue: Invalid value '*/150%' for key 'rollout' (expected percentage <= 100) in source 'rollout')
println(RecommendationsPanel.expression)
// true@*/25%; false
```

**Parsing does not check the values.** `DynamicFlag#update` checks the grammar, not whether each value parses as the flag's type. An expression whose values the flag cannot read is accepted, every matching call answers with the default, and `DynamicFlag#parseErrorCount` counts the fallbacks; see [Checking an Expression Before You Ship It](#checking-an-expression-before-you-ship-it). `DynamicFlag#update` also accepts an empty or blank expression as a no-op and returns `Right(())` without changing anything.

### Reloading From a Source

In production the expression lives in a flag service or a config file, not in code. `DynamicFlag#reload` reads the flag's name from the registered sources and reports the outcome as a `Flag.ReloadResult`. With no source providing the flag, nothing changes:

```scala
println(RecommendationsPanel.reload())
// NoSource
```

Once a source provides an expression, `DynamicFlag#reload` applies it, and reloading the same expression again is a no-op. Because the flag is already initialized, `RecommendationsPanel.name` is safe to use as the key:

```scala
FlagSource.Registry.register(
  FlagSource.fromMap(Map(RecommendationsPanel.name -> "true@*/50%; false"), "flag-service")
)
```

The first reload applies the source's expression, and the second finds nothing to change:

```scala
println(RecommendationsPanel.reload())
// Updated(true@*/25%; false,true@*/50%; false)
println(RecommendationsPanel.reload())
// Unchanged
```

A bad push to the flag service comes back as `Failed`, and the flag keeps its current expression instead of dropping to the default. Registering a source under an existing `FlagSource#sourceId` replaces it in place, which is how this snippet simulates a new push:

```scala
FlagSource.Registry.register(
  FlagSource.fromMap(Map(RecommendationsPanel.name -> "true@*/fifty%"), "flag-service")
)
```

Reloading again rejects the bad push and keeps the current expression:

```scala
println(RecommendationsPanel.reload())
// Failed(zio.blocks.config.ConfigError$InvalidValue: Invalid value '*/fifty%' for key 'rollout' (expected selector path using /NN% percentage syntax) in source 'rollout')
println(RecommendationsPanel.expression)
// true@*/50%; false
```

### Auditing Changes

Every successful `DynamicFlag#update` or `DynamicFlag#reload` adds a `DynamicFlag.UpdateRecord` to `DynamicFlag#updateHistory`, most recent first and capped at the last ten. Each record also carries a `DynamicFlag.UpdateRecord#timestampMillis`:

```scala
RecommendationsPanel.updateHistory.foreach(r => println(s"${r.oldExpression}  ->  ${r.newExpression}"))
// true@*/25%; false  ->  true@*/50%; false
// true@*/10%; false  ->  true@*/25%; false
```

`DynamicFlag#counters` shows how often `DynamicFlag#apply` saw each key; `DynamicFlag#evaluate` calls are not counted. After three more calls, the snapshot holds them next to the three calls for `reader-42` from earlier. The counters stop adding keys at 100 distinct ones and count the rest under `"other"`:

```scala
RecommendationsPanel("reader-1")
RecommendationsPanel("reader-1")
RecommendationsPanel("reader-2")
```

Printing the snapshot shows each key's count:

```scala
println(RecommendationsPanel.counters.toList.sorted)
// List((reader-1,2), (reader-2,1), (reader-42,3))
```

`DynamicFlag#parseErrorCount` counts calls whose matched value could not be parsed as the flag's type. The expression `maybe` parses as a catch-all, but `maybe` is not a `Boolean`, so each call answers with the default `false` and the count grows:

```scala
println(RecommendationsPanel.update("maybe"))
// Right(())
println(RecommendationsPanel("reader-1"))
// false
println(RecommendationsPanel.parseErrorCount)
// 1
```

A non-zero `DynamicFlag#parseErrorCount` always means a misconfigured expression, so it is worth exporting as a metric.

### Reloading on a Schedule

Reload is manual. The library does not poll a source, watch a file or start a thread, so call `DynamicFlag#reload` from a scheduler your service already has. A plain `ScheduledExecutorService` is enough:

```scala

val scheduler = Executors.newSingleThreadScheduledExecutor()

scheduler.scheduleAtFixedRate(
  () =>
    RecommendationsPanel.reload() match {
      case Flag.ReloadResult.Updated(from, to) => println(s"rollout changed: $from -> $to")
      case Flag.ReloadResult.Failed(error)     => println(s"rollout rejected: ${error.message}")
      case Flag.ReloadResult.Unchanged         => ()
      case Flag.ReloadResult.NoSource          => println("no source provides the rollout")
    },
  0,
  30,
  TimeUnit.SECONDS
)
```

Shut the scheduler down when the service stops. For a source that fetches from a remote service, refresh the source's contents on the same schedule before calling `DynamicFlag#reload`.

## Checking an Expression Before You Ship It

`Rollout.validate` parses an expression and returns warnings for rules that parse but are almost certainly wrong. Run it in the admin endpoint or deploy check that accepts expressions from operators; flags do not run it themselves. A catch-all before targeted choices makes them unreachable:

```scala
println(Rollout.validate("false; true@*/eu/50%"))
// Right(List(Choices after index 0 are unreachable (catch-all found)))
```

Percentages for one pattern that add up to more than 100% are flagged too:

```scala
println(Rollout.validate("true@*/eu/60%; true@*/eu/50%; false"))
// Right(List(Cumulative percentage for pattern '*/eu' is 110% (exceeds 100%)))
```

An expression that does not parse comes back as `Left`, and a clean one as an empty list:

```scala
println(Rollout.validate("true@*/eu/150%; false"))
// Left(zio.blocks.config.ConfigError$InvalidValue: Invalid value '*/eu/150%' for key 'rollout' (expected percentage <= 100) in source 'rollout')
println(Rollout.validate("true@*/eu/50%; false"))
// Right(List())
```

:::warning[Percentages are thresholds, not slices]
Each choice matches `bucket < percentage` on its own, and the first match wins. `a@*/10%; b@*/20%; c` gives `a` to buckets 0–9 and `b` only to buckets 10–19, so `b` reaches 10% of keys, not 20%. A later choice with a lower percentage on the same pattern is never reached, and `Rollout.validate` does not warn about it:

```scala
println(Rollout.validate("true@*/eu/30%; false@*/eu/20%; false"))
// Right(List())
```

To split readers between variants, give each choice the cumulative upper bound of its slice: `a@*/10%; b@*/30%; c` is 10% `a`, 20% `b` and 70% `c`.
:::

Validation checks the grammar, so a value of the wrong type still slips through, as the `maybe` example above showed. To catch that too, parse each value with the flag's reader before shipping:

```scala
def valuesParseAsBoolean(expression: String): Boolean =
  Rollout.parseChoices(expression).exists { choices =>
    choices.entries.forall {
      case Rollout.Choice.Targeted(value, _) => Flag.Reader.booleanReader.parse("check", value).isRight
      case Rollout.Choice.CatchAll(value)    => Flag.Reader.booleanReader.parse("check", value).isRight
    }
  }
```

Checking one valid and one invalid expression shows the difference:

```scala
println(valuesParseAsBoolean("true@*/eu/50%; false"))
// true
println(valuesParseAsBoolean("maybe"))
// false
```

## Catching Typos at Startup

An override whose key is spelled differently from the flag's name is ignored, and the flag quietly uses its default. `Flag.nearMissWarnings` looks in three places. In the registered sources it looks for keys that differ from the flag's name in case or in dots versus underscores. In system properties and environment variables it looks only for case differences (for an environment variable, against the flag's name with dots replaced by underscores and uppercased):

```scala
FlagSource.Registry.register(
  FlagSource.fromMap(
    Map("repl.mdocsession.mdocapp.newcheckout" -> "true@*/5%; false"),
    "typo-overrides"
  )
)

object NewCheckout extends DynamicFlag[Boolean](false, "false")
```

Printing the expression shows the default, and the warning names the misspelled key:

```scala
println(NewCheckout.expression)
// false
Flag.nearMissWarnings(NewCheckout.name).foreach(println)
// Near-miss: FlagSource 'typo-overrides' contains key 'repl.mdocsession.mdocapp.newcheckout' similar to flag 'repl.MdocSession.MdocApp.NewCheckout'
```

`Flag.dump()` prints every flag that has been touched, with its value or expression and where the value came from. A flag whose object has not been touched yet is missing from the table, so print the report after startup has read every flag:

```scala
println(Flag.dump())
// ┌────────────────────────────────────────────────┬────────────────────────────────┬──────────────────────┬───────────────┐
// │ Name                                           │ Type                           │ Value                │ Source        │
// ├────────────────────────────────────────────────┼────────────────────────────────┼──────────────────────┼───────────────┤
// │ repl.MdocSession.MdocApp.FreeShippingThreshold │ FlagSourceValue(ops-overrides) │ 25                   │ ops-overrides │
// │ repl.MdocSession.MdocApp.NewCheckout           │ DynamicFlag                    │ false                │ dynamic       │
// │ repl.MdocSession.MdocApp.RecommendationsPanel  │ DynamicFlag                    │ maybe                │ dynamic       │
// │ repl.MdocSession.MdocApp.RegionalPanel         │ DynamicFlag                    │ true@*/eu/50%; false │ dynamic       │
// │ repl.MdocSession.MdocApp.WishlistSharing       │ DynamicFlag                    │ true@*/10%; false    │ dynamic       │
// └────────────────────────────────────────────────┴────────────────────────────────┴──────────────────────┴───────────────┘
```

:::warning[Use `Secret` for credentials in flags]
`Flag.dump()` masks values by type, not by name. A `StaticFlag[Secret]` prints as `<secret>`, but a `StaticFlag[String]` prints its value even when the flag is called `PaymentToken`. Give credentials the type `Secret`. The dump reads only the flags themselves; it does not print the environment or the contents of sources.
:::

## Testing Code That Uses Flags

Flags read two global singletons: `FlagSource.Registry`, which holds the sources, and the flag objects themselves. Tests that register sources therefore share state, so follow three rules:

1. **Clear the registry in setup and in teardown.** A source left behind by one test changes what the next test's flags see.
2. **Run flag tests sequentially.** Two tests registering sources at the same time race on the registry. The config module's own tests turn off parallel execution for this reason.
3. **Register before first touch, then reload.** A `StaticFlag` resolves once per JVM, so it cannot take a different value per test; keep restart-only values out of logic you want to test with several values. A `DynamicFlag` can be reset in each test with `DynamicFlag#reload` or `DynamicFlag#update`.

A small helper keeps the first rule in one place:

```scala
def withFlagSources[A](sources: FlagSource*)(body: => A): A = {
  FlagSource.Registry.clear()
  sources.foreach(FlagSource.Registry.register)
  try body
  finally FlagSource.Registry.clear()
}

def showPanel(readerId: String, region: String): Boolean =
  RegionalPanel(readerId, region)
```

Each test registers the expression it needs and reloads the flag before calling the code under test:

```scala
println(withFlagSources(FlagSource.fromMap(Map(RegionalPanel.name -> "true@*/eu; false"))) {
  RegionalPanel.reload()
  (showPanel("reader-42", "eu"), showPanel("reader-42", "us"))
})
// (true,false)

println(FlagSource.Registry.all.size)
// 0
```

With ZIO Test, wrap each test body in the helper and mark the suite `TestAspect.sequential`. In sbt, also set `Test / parallelExecution := false` when several specs touch flags:

```scala

object RecommendationsSpec extends ZIOSpecDefault {
  def spec = suite("recommendations panel")(
    test("eu readers get the panel") {
      withFlagSources(FlagSource.fromMap(Map(RegionalPanel.name -> "true@*/eu; false"))) {
        RegionalPanel.reload()
        assertTrue(showPanel("reader-42", "eu"), !showPanel("reader-42", "us"))
      }
    },
    test("nobody gets the panel when it is off") {
      withFlagSources(FlagSource.fromMap(Map(RegionalPanel.name -> "false"))) {
        RegionalPanel.reload()
        assertTrue(!showPanel("reader-42", "eu"))
      }
    }
  ) @@ TestAspect.sequential
}
```

## Putting It Together

The complete example declares the bookstore's flags, registers a stand-in for a flag service before touching them, prints the startup report, serves three readers, validates a wider rollout, and picks it up with a scheduled reload:

```scala title="config-examples/src/main/scala/rolloutfeaturegradually/CompleteExample.scala"
package rolloutfeaturegradually

/**
 * How to Roll Out a Feature Gradually with Flags — Putting It Together
 *
 * The Scala bookstore's flags: a restart-only free-shipping threshold, a
 * recommendations panel rolled out to half of the eu readers, a startup report
 * with near-miss warnings, a validated runtime change, and a scheduled reload.
 *
 * Run with: cd config-examples && sbt "runMain
 * rolloutfeaturegradually.CompleteExample"
 */
object CompleteExample {

  object FreeShippingThreshold extends StaticFlag[Int](35)

  object RecommendationsPanel extends DynamicFlag[Boolean](false, "false")

  final case class Book(title: String, price: Int)

  final case class Reader(id: String, region: String)

  val catalog = List(
    Book("Zionomicon", 45),
    Book("Programming in Scala", 40),
    Book("Functional Programming in Scala", 30)
  )

  // Stands in for a remote flag service: reload re-reads whatever it holds now.
  @volatile var flagService: Map[String, String] = Map(
    "rolloutfeaturegradually.CompleteExample.FreeShippingThreshold" -> "40",
    "rolloutfeaturegradually.CompleteExample.RecommendationsPanel"  -> "true@*/eu/50%; false"
  )

  val flagServiceSource: FlagSource = new FlagSource {
    val sourceId: String  = "flag-service"
    def get(name: String) = FlagSource.fromMap(flagService, sourceId).get(name)
  }

  def checkout(reader: Reader, book: Book): String = {
    val shipping = if (book.price >= FreeShippingThreshold()) "free shipping" else "standard shipping"
    val panel    = if (RecommendationsPanel(s"recommendations:${reader.id}", reader.region)) "panel" else "no panel"
    s"${reader.id} (${reader.region}) buys ${book.title}: $shipping, $panel"
  }

  def main(args: Array[String]): Unit = {
    // 1. Register sources before any flag is touched.
    FlagSource.Registry.register(flagServiceSource)

    // 2. Startup report: typos and the resolved values.
    val flags = List(FreeShippingThreshold.name, RecommendationsPanel.name)
    flags.flatMap(Flag.nearMissWarnings).foreach(println)
    println(Flag.dump())

    // 3. Serve some readers.
    val readers = List(Reader("reader-7", "eu"), Reader("reader-8", "eu"), Reader("reader-9", "us"))
    readers.zip(catalog).foreach { case (reader, book) => println(checkout(reader, book)) }

    // 4. Operators push a wider rollout; validate it before the service picks it up.
    val proposed = "true@*/eu/100%; true@*/us/25%; false"
    Rollout.validate(proposed) match {
      case Right(Nil)  => flagService = flagService.updated(RecommendationsPanel.name, proposed)
      case Right(warn) => println(s"held back: ${warn.mkString("; ")}")
      case Left(error) => println(s"rejected: ${error.message}")
    }

    // 5. Reload on a schedule: the library never polls on its own.
    val scheduler = Executors.newSingleThreadScheduledExecutor()
    scheduler.scheduleAtFixedRate(
      () =>
        RecommendationsPanel.reload() match {
          case Flag.ReloadResult.Updated(from, to) => println(s"rollout changed: $from -> $to")
          case Flag.ReloadResult.Failed(error)     => println(s"rollout rejected: ${error.message}")
          case _                                   => ()
        },
      0,
      30,
      TimeUnit.SECONDS
    )
    Thread.sleep(500)
    scheduler.shutdown()

    readers.zip(catalog).foreach { case (reader, book) => println(checkout(reader, book)) }
  }
}
```

## Running the Examples

All examples in this guide have corresponding runnable Scala files in the `config-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/config-examples
```

Each file below matches one section of this guide.

### A Restart-Only Knob

<details>
  <summary>config-examples/src/main/scala/rolloutfeaturegradually/Step1StaticFlag.scala</summary>

```scala title="config-examples/src/main/scala/rolloutfeaturegradually/Step1StaticFlag.scala" showLineNumbers
package rolloutfeaturegradually

/**
 * How to Roll Out a Feature Gradually with Flags — Step 1: A Restart-Only Knob
 *
 * Declares the bookstore's free-shipping threshold as a StaticFlag, overrides
 * it from a registered FlagSource, and shows that a value which does not parse
 * stops the flag at first touch.
 *
 * Run with: cd config-examples && sbt "runMain
 * rolloutfeaturegradually.Step1StaticFlag"
 */
object Step1StaticFlag {

  // Orders at or above this many dollars ship for free.
  object FreeShippingThreshold extends StaticFlag[Int](35)

  // The most books a reader can put in one order.
  object MaxBooksPerOrder extends StaticFlag[Int](20)

  def main(args: Array[String]): Unit = {
    // Register every source before touching any flag.
    FlagSource.Registry.register(
      FlagSource.fromMap(
        Map(
          "rolloutfeaturegradually.Step1StaticFlag.FreeShippingThreshold" -> "25",
          "rolloutfeaturegradually.Step1StaticFlag.MaxBooksPerOrder"      -> "ten"
        ),
        "ops-overrides"
      )
    )

    println(s"name:      ${FreeShippingThreshold.name}")
    println(s"value:     ${FreeShippingThreshold()}")
    println(s"source:    ${FreeShippingThreshold.source}")

    try MaxBooksPerOrder()
    catch {
      case e: ExceptionInInitializerError => println(s"rejected:  ${e.getCause.getMessage}")
    }
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain rolloutfeaturegradually.Step1StaticFlag"
```

### Turning the Panel On for 10% of Readers

<details>
  <summary>config-examples/src/main/scala/rolloutfeaturegradually/Step2PercentageRollout.scala</summary>

```scala title="config-examples/src/main/scala/rolloutfeaturegradually/Step2PercentageRollout.scala" showLineNumbers
package rolloutfeaturegradually

/**
 * How to Roll Out a Feature Gradually with Flags — Step 2: A Percentage Rollout
 *
 * Turns the new recommendations panel on for 10% of readers, shows that each
 * reader always gets the same answer, measures the share across 1000 reader
 * ids, and salts the key so two rollouts pick independent readers.
 *
 * Run with: cd config-examples && sbt "runMain
 * rolloutfeaturegradually.Step2PercentageRollout"
 */
object Step2PercentageRollout {

  object RecommendationsPanel extends DynamicFlag[Boolean](false, "true@*/10%; false")

  object WishlistSharing extends DynamicFlag[Boolean](false, "true@*/10%; false")

  def main(args: Array[String]): Unit = {
    // The same reader id always lands in the same bucket, so the answer never flips.
    println(s"reader-42 bucket: ${Rollout.bucketFor("reader-42")}")
    println(s"reader-42 sees panel: ${(1 to 3).map(_ => RecommendationsPanel("reader-42"))}")

    val readers = (1 to 1000).map(i => s"reader-$i")
    val inPanel = readers.filter(RecommendationsPanel.evaluate(_))
    println(s"readers with the panel: ${inPanel.size} of ${readers.size}")

    // Bucketing hashes only the key, so every 10% rollout picks the same readers.
    val sameKey = readers.filter(WishlistSharing.evaluate(_))
    println(s"in both rollouts (plain key):  ${inPanel.intersect(sameKey).size}")

    // Prefix the key with the feature name to give each rollout its own readers.
    val panelSalted    = readers.filter(id => RecommendationsPanel.evaluate(s"recommendations:$id"))
    val wishlistSalted = readers.filter(id => WishlistSharing.evaluate(s"wishlist:$id"))
    println(s"in both rollouts (salted key): ${panelSalted.intersect(wishlistSalted).size}")
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain rolloutfeaturegradually.Step2PercentageRollout"
```

### Targeting One Region

<details>
  <summary>config-examples/src/main/scala/rolloutfeaturegradually/Step3RegionTargeting.scala</summary>

```scala title="config-examples/src/main/scala/rolloutfeaturegradually/Step3RegionTargeting.scala" showLineNumbers
package rolloutfeaturegradually

/**
 * How to Roll Out a Feature Gradually with Flags — Step 3: Targeting a Region
 *
 * Offers the recommendations panel to half of the readers in the eu region and
 * to nobody else, passing the region as a path attribute after the reader id.
 *
 * Run with: cd config-examples && sbt "runMain
 * rolloutfeaturegradually.Step3RegionTargeting"
 */
object Step3RegionTargeting {

  object RecommendationsPanel extends DynamicFlag[Boolean](false, "true@*/eu/50%; false")

  def main(args: Array[String]): Unit = {
    val readers = (1 to 1000).map(i => s"reader-$i")

    // The path is "<reader id>/<region>", so the selector has two segments.
    val eu = readers.count(id => RecommendationsPanel.evaluate(id, "eu"))
    val us = readers.count(id => RecommendationsPanel.evaluate(id, "us"))
    println(s"eu readers with the panel: $eu of ${readers.size}")
    println(s"us readers with the panel: $us of ${readers.size}")

    // Forgetting the region gives a one-segment path, which never matches */eu.
    val noRegion = readers.count(id => RecommendationsPanel.evaluate(id))
    println(s"readers with no region:    $noRegion of ${readers.size}")
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain rolloutfeaturegradually.Step3RegionTargeting"
```

### Changing the Rollout While the Service Runs

<details>
  <summary>config-examples/src/main/scala/rolloutfeaturegradually/Step4RuntimeChanges.scala</summary>

```scala title="config-examples/src/main/scala/rolloutfeaturegradually/Step4RuntimeChanges.scala" showLineNumbers
package rolloutfeaturegradually

/**
 * How to Roll Out a Feature Gradually with Flags — Step 4: Changing the Rollout
 * While the Service Runs
 *
 * Widens the rollout with update, rejects a malformed expression, reloads the
 * expression from a registered source, reads the update history, counters and
 * parse-error count, and drives reload from a ScheduledExecutorService.
 *
 * Run with: cd config-examples && sbt "runMain
 * rolloutfeaturegradually.Step4RuntimeChanges"
 */
object Step4RuntimeChanges {

  object RecommendationsPanel extends DynamicFlag[Boolean](false, "true@*/10%; false")

  def main(args: Array[String]): Unit = {
    val readers = (1 to 1000).map(i => s"reader-$i")
    val cohort  = readers.filter(RecommendationsPanel.evaluate(_))

    // Widen from 10% to 25%: everyone who had the panel keeps it.
    println(s"update to 25%:  ${RecommendationsPanel.update("true@*/25%; false")}")
    val widened = readers.filter(RecommendationsPanel.evaluate(_))
    println(s"readers with the panel: ${widened.size}, original cohort kept: ${cohort.forall(widened.contains)}")

    // A malformed expression is rejected and the old one stays in place.
    println(s"update to 150%: ${RecommendationsPanel.update("true@*/150%; false")}")
    println(s"expression:     ${RecommendationsPanel.expression}")

    // Reload reads the expression from the registered sources.
    println(s"reload, no source:  ${RecommendationsPanel.reload()}")
    FlagSource.Registry.register(
      FlagSource.fromMap(Map(RecommendationsPanel.name -> "true@*/50%; false"), "flag-service")
    )
    println(s"reload:             ${RecommendationsPanel.reload()}")
    println(s"reload again:       ${RecommendationsPanel.reload()}")
    FlagSource.Registry.register(FlagSource.fromMap(Map(RecommendationsPanel.name -> "true@*/fifty%"), "flag-service"))
    println(s"reload, bad value:  ${RecommendationsPanel.reload()}")

    RecommendationsPanel.updateHistory.foreach(r => println(s"history: ${r.oldExpression} -> ${r.newExpression}"))

    // apply counts each key; evaluate does not.
    RecommendationsPanel("reader-1")
    RecommendationsPanel("reader-1")
    RecommendationsPanel("reader-2")
    println(s"counters: ${RecommendationsPanel.counters.toList.sorted}")

    // A value the reader cannot parse falls back to the default and is counted.
    RecommendationsPanel.update("maybe")
    println(
      s"reader-1 sees panel: ${RecommendationsPanel("reader-1")}, parse errors: ${RecommendationsPanel.parseErrorCount}"
    )

    // Nothing reloads on its own: schedule it yourself.
    FlagSource.Registry.register(
      FlagSource.fromMap(Map(RecommendationsPanel.name -> "true@*/75%; false"), "flag-service")
    )
    val scheduler = Executors.newSingleThreadScheduledExecutor()
    scheduler.scheduleAtFixedRate(
      () =>
        RecommendationsPanel.reload() match {
          case Flag.ReloadResult.Updated(from, to) => println(s"rollout changed: $from -> $to")
          case Flag.ReloadResult.Failed(error)     => println(s"rollout rejected: ${error.message}")
          case Flag.ReloadResult.Unchanged         => ()
          case Flag.ReloadResult.NoSource          => println("no source provides the rollout")
        },
      0,
      30,
      TimeUnit.SECONDS
    )
    Thread.sleep(500)
    scheduler.shutdown()
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain rolloutfeaturegradually.Step4RuntimeChanges"
```

### Checking an Expression Before You Ship It

<details>
  <summary>config-examples/src/main/scala/rolloutfeaturegradually/Step5Validation.scala</summary>

```scala title="config-examples/src/main/scala/rolloutfeaturegradually/Step5Validation.scala" showLineNumbers
package rolloutfeaturegradually

/**
 * How to Roll Out a Feature Gradually with Flags — Step 5: Checking an
 * Expression Before You Ship It
 *
 * Runs Rollout.validate on candidate expressions and refuses to apply one that
 * does not parse or comes back with warnings.
 *
 * Run with: cd config-examples && sbt "runMain
 * rolloutfeaturegradually.Step5Validation"
 */
object Step5Validation {

  object RecommendationsPanel extends DynamicFlag[Boolean](false, "true@*/10%; false")

  def ship(expression: String): Unit =
    Rollout.validate(expression) match {
      case Left(error)                          => println(s"rejected '$expression': ${error.message}")
      case Right(warnings) if warnings.nonEmpty => println(s"held back '$expression': ${warnings.mkString("; ")}")
      case Right(_)                             =>
        RecommendationsPanel.update(expression)
        println(s"shipped '$expression'")
    }

  def main(args: Array[String]): Unit = {
    ship("false; true@*/eu/50%")
    ship("true@*/eu/60%; true@*/eu/50%; false")
    ship("true@*/eu/150%; false")
    ship("true@*/eu/50%; false")
    println(s"expression: ${RecommendationsPanel.expression}")
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain rolloutfeaturegradually.Step5Validation"
```

### Catching Typos at Startup

<details>
  <summary>config-examples/src/main/scala/rolloutfeaturegradually/Step6StartupReport.scala</summary>

```scala title="config-examples/src/main/scala/rolloutfeaturegradually/Step6StartupReport.scala" showLineNumbers
package rolloutfeaturegradually

/**
 * How to Roll Out a Feature Gradually with Flags — Step 6: Catching Typos at
 * Startup
 *
 * Prints near-miss warnings for a flag whose override key has the wrong case,
 * and a table of every flag the service touched, with the payment token kept
 * out of it.
 *
 * Run with: cd config-examples && sbt "runMain
 * rolloutfeaturegradually.Step6StartupReport"
 */
object Step6StartupReport {

  object FreeShippingThreshold extends StaticFlag[Int](35)

  object RecommendationsPanel extends DynamicFlag[Boolean](false, "true@*/10%; false")

  object PaymentToken extends StaticFlag[Secret](Secret("dev-token"))

  def main(args: Array[String]): Unit = {
    FlagSource.Registry.register(
      FlagSource.fromMap(
        Map(
          // Typo: the key is lower-case, so the flag never sees it.
          "rolloutfeaturegradually.step6startupreport.freeshippingthreshold" -> "25",
          "rolloutfeaturegradually.Step6StartupReport.PaymentToken"          -> "tok-live-123"
        ),
        "ops-overrides"
      )
    )

    val flags = List(FreeShippingThreshold.name, RecommendationsPanel.name, PaymentToken.name)
    flags.flatMap(Flag.nearMissWarnings).foreach(println)
    println(Flag.dump())
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain rolloutfeaturegradually.Step6StartupReport"
```

### Testing Code That Uses Flags

<details>
  <summary>config-examples/src/main/scala/rolloutfeaturegradually/Step7Testing.scala</summary>

```scala title="config-examples/src/main/scala/rolloutfeaturegradually/Step7Testing.scala" showLineNumbers
package rolloutfeaturegradually

/**
 * How to Roll Out a Feature Gradually with Flags — Step 7: Testing Code That
 * Uses Flags
 *
 * A helper that registers sources for one block of code and clears the global
 * registry before and after it, used to check the panel decision for two
 * different expressions.
 *
 * Run with: cd config-examples && sbt "runMain
 * rolloutfeaturegradually.Step7Testing"
 */
object Step7Testing {

  object RecommendationsPanel extends DynamicFlag[Boolean](false, "false")

  def withFlagSources[A](sources: FlagSource*)(body: => A): A = {
    FlagSource.Registry.clear()
    sources.foreach(FlagSource.Registry.register)
    try body
    finally FlagSource.Registry.clear()
  }

  def showPanel(readerId: String, region: String): Boolean = RecommendationsPanel(readerId, region)

  def main(args: Array[String]): Unit = {
    val everyoneInEu = withFlagSources(FlagSource.fromMap(Map(RecommendationsPanel.name -> "true@*/eu; false"))) {
      RecommendationsPanel.reload()
      (showPanel("reader-42", "eu"), showPanel("reader-42", "us"))
    }
    println(s"eu, us with 'true@*/eu; false': $everyoneInEu")

    val nobody = withFlagSources(FlagSource.fromMap(Map(RecommendationsPanel.name -> "false"))) {
      RecommendationsPanel.reload()
      (showPanel("reader-42", "eu"), showPanel("reader-42", "us"))
    }
    println(s"eu, us with 'false':            $nobody")
    println(s"sources left registered: ${FlagSource.Registry.all.size}")
  }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain rolloutfeaturegradually.Step7Testing"
```

### The Complete Example

Run the complete example from the `config-examples` directory:

```bash
sbt "runMain rolloutfeaturegradually.CompleteExample"
```

To check that every example compiles without running it, use:

```bash
sbt compile
```

## Going Further

- [Flags](../reference/config/flags.md) covers every member of `StaticFlag`, `DynamicFlag`, `Flag.Reader` and `FlagException`, including custom readers for your own types.
- [Rollout](../reference/config/rollout.md) covers the expression grammar, path matching and the parsed form an admin tool can render.
- [ConfigSource](../reference/config/config-source.md) explains `Secret`; any `ConfigSource` is also a `FlagSource`, so a YAML file or the environment can feed flags directly.
- [Build a Typed Configuration Loader](./typed-config-loader.md) loads the rest of the bookstore's settings.

---

## How to Run a Mux over a Real Socket

In this guide, we run a ZIO Blocks [`Mux`](../reference/mux.mdx) over a real TCP connection. By the end, you will have a small chat server and client that carry three independent streams over one socket on `localhost`: a chat room, a file upload and a heartbeat. Their messages interleave on the wire and each one arrives at the right stream.

`Mux` keeps the streams apart, but it ships no transport. The protocol side pushes each received message into its stream with `MuxStream#offerInbound` and pulls each message to send with `MuxStream#takeOutbound`. That protocol side is what we write here, one piece at a time:

- Define a frame format that tags every message with its stream id
- Keep the bookkeeping `Mux` leaves to the protocol
- Drain every stream's outbound queue onto the socket from one writer loop
- Route incoming frames to their streams from one reader loop, opening a stream the first time its id appears
- Run several streams over the one connection at the same time
- Half-close a stream while the other direction keeps working
- Stop reading when a slow stream's queue is full, instead of dropping data or growing memory
- Refuse streams over the limit, cancel a stream with a reset, and shut the connection down cleanly

This guide assumes you have worked through [Getting Started with Mux](./getting-started-with-mux.md) and know what a TCP socket is.

## The Problem

One TCP connection is one ordered stream of bytes. A program that wants a chat, a file transfer and a heartbeat to share that connection has to cut the bytes into frames, tag each frame with the logical stream it belongs to, and route frames back to the right place on the other side, as HTTP/2 streams and SSH channels do. Without a library for the routing, every program writes its own stream table, and the parts that are easy to leave out are the ones that hurt in production: limits, flow control and stream state.

For example, this is a hand-written reader loop that routes frames to one queue per stream id:

```scala

// One queue per stream id, created the first time the id shows up.
val inboxes  = new ConcurrentHashMap[Int, LinkedBlockingQueue[String]]()
val finished = ConcurrentHashMap.newKeySet[Int]()

def readLoop(in: DataInputStream): Unit =
  while (true) {
    val id     = in.readInt()
    val kind   = in.readByte()
    val length = in.readInt()
    val bytes  = new Array[Byte](length)
    in.readFully(bytes)
    kind match {
      case 0 => inboxes.computeIfAbsent(id, _ => new LinkedBlockingQueue[String]()).put(new String(bytes, UTF_8))
      case 1 => finished.add(id) // the peer is done, but nothing stops our side from reading forever
      case _ => inboxes.remove(id) // reset: the thread reading that queue is never told
    }
  }
```

This loop works on a good day. A peer that opens a million stream ids gets a million queues, because nothing limits how many streams are open. A slow consumer makes its queue grow until the JVM runs out of memory, because the queues are unbounded. A frame that arrives after a reset quietly creates a new queue, because no stream remembers that it is closed. And the sending side, which is not even written yet, needs the same table again.

In this guide, we let `Mux` own the stream table: it limits how many streams are open, bounds every stream's queues, and tracks each stream's half-close state. We write only the glue between it and the socket.

## Prerequisites

Add the mux module to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-mux" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-mux" % "0.0.56"
```

The code in this guide uses only the JDK besides the mux module. It runs on the JVM: `Mux` is also published for Scala.js, but the examples use `java.net.Socket`, threads and blocking reads, which Scala.js does not have.

The code is written for Scala 3, because it uses the union types that `Mux#open`, `MuxStream#send`, `MuxStream#receive`, `MuxStream#offerInbound` and `MuxStream#takeOutbound` return on Scala 3, and the companion examples use top-level definitions. On Scala 2.13 the same five methods return `Either[MuxError, A]`, so each match changes shape. For example, delivering one message, as the reader loop below does:

<Tabs groupId="scala-version" defaultValue="scala2">
<TabItem value="scala2" label="Scala 2">

```scala

@tailrec def deliver(stream: MuxStream[Int, String, String], message: String): Unit =
  stream.offerInbound(message) match {
    case Right(())                   => ()
    case Left(MuxError.QueueFull(_)) =>
      LockSupport.parkNanos(1000000L)
      deliver(stream, message)
    case Left(_) => () // closed or cancelled on our side: drop the frame
  }
```

</TabItem>
<TabItem value="scala3" label="Scala 3">

```scala

@tailrec def deliver(stream: MuxStream[Int, String, String], message: String): Unit =
  stream.offerInbound(message) match {
    case ()                    => ()
    case MuxError.QueueFull(_) =>
      LockSupport.parkNanos(1000000L)
      deliver(stream, message)
    case _: MuxError => () // closed or cancelled on our side: drop the frame
  }
```

</TabItem>
</Tabs>

## How the Pieces Fit

Each side of the connection runs the same machinery. Application threads talk to their own `MuxStream`; two protocol threads move messages between the streams and the socket:

```text
 application threads            Mux (one per connection)              protocol threads          network
 -------------------            ------------------------              ----------------          -------
 chat thread    --send-------->  stream 1 outbound queue  --+
 upload thread  --send-------->  stream 3 outbound queue  --+--takeOutbound--> writer loop --frames--> socket
 heartbeat      --send-------->  stream 5 outbound queue  --+

 chat thread    <--receive----  stream 1 inbound queue   <--+
 upload thread  <--receive----  stream 3 inbound queue   <--+--offerInbound--- reader loop <--frames-- socket
 heartbeat      <--receive----  stream 5 inbound queue   <--+
```

The arrows also show the threading rule this design follows. `MuxStream#send` and `MuxStream#offerInbound` are safe to call from many threads at once. `MuxStream#receive` and `MuxStream#takeOutbound` are single-consumer: on each stream, only one thread at a time may call each of them. In this guide, one application thread per stream calls `MuxStream#receive`, and the writer loop is the only code that calls `MuxStream#takeOutbound`.

## Define a Frame Format

The socket carries bytes, so every message needs a header that says which stream it belongs to and how long it is. `Mux` doesn't care what the bytes look like; the frame format below is glue, and any format with a stream id and a length works.

Each frame is a 9-byte header followed by the payload: the stream id (4 bytes), the kind (1 byte: data, end or reset), the payload length (4 bytes), and then the payload as UTF-8 text. `Message` is what the application sends and receives on a stream; its `End` case is the in-band "I am done sending" marker that [Half-Close a Stream](#half-close-a-stream) explains:

```scala

// What the application sends and receives on one stream. `End` is the
// in-band "I am done sending" marker; it travels the same FIFO as the data.
sealed trait Message
object Message {
  final case class Text(text: String) extends Message
  case object End                     extends Message
}

// Every stream in this guide carries `Message` in both directions.
type MessageStream = MuxStream[Int, Message, Message]

// One frame on the wire: stream id (4 bytes), kind (1 byte),
// payload length (4 bytes), payload (UTF-8 text).
sealed trait Frame { def streamId: Int }

object Frame {
  final case class Data(streamId: Int, text: String)    extends Frame
  final case class End(streamId: Int)                   extends Frame
  final case class Reset(streamId: Int, reason: String) extends Frame

  val MaxPayload: Int = 64 * 1024

  def write(out: DataOutputStream, frame: Frame): Unit = {
    val (kind, payload) = frame match {
      case Data(_, text)    => (0, text.getBytes(UTF_8))
      case End(_)           => (1, Array.emptyByteArray)
      case Reset(_, reason) => (2, reason.getBytes(UTF_8))
    }
    if (payload.length > MaxPayload) throw new IOException(s"payload of ${payload.length} bytes is too large")
    out.writeInt(frame.streamId)
    out.writeByte(kind)
    out.writeInt(payload.length)
    out.write(payload)
  }

  // Returns None when the peer closed the connection between two frames.
  def read(in: DataInputStream): Option[Frame] = {
    val streamId =
      try in.readInt()
      catch { case _: EOFException => return None }
    val kind   = in.readByte()
    val length = in.readInt()
    if (length < 0 || length > MaxPayload) throw new IOException(s"frame length $length is out of range")
    val payload = new Array[Byte](length)
    in.readFully(payload)
    kind match {
      case 0     => Some(Data(streamId, new String(payload, UTF_8)))
      case 1     => Some(End(streamId))
      case 2     => Some(Reset(streamId, new String(payload, UTF_8)))
      case other => throw new IOException(s"unknown frame kind $other")
    }
  }
}
```

`Frame.read` returns `None` when the connection ends cleanly between two frames, and throws when it ends in the middle of one. It also rejects a length over `Frame.MaxPayload` before allocating, so a corrupt header can't make the reader allocate gigabytes.

To check the format, we encode three frames into a byte array and open two readers on it, one of them on a copy cut off 7 bytes in, in the middle of the first frame:

```scala

val bytes = new ByteArrayOutputStream()
val out   = new DataOutputStream(bytes)
Frame.write(out, Frame.Data(1, "hi"))
Frame.write(out, Frame.End(1))
Frame.write(out, Frame.Reset(3, "cancelled"))
out.flush()
val wire = bytes.toByteArray
val in   = new DataInputStream(new ByteArrayInputStream(wire))
val cut  = new DataInputStream(new ByteArrayInputStream(wire.take(7))) // a frame cut off after 7 bytes
```

Decoding the bytes gives the three frames back, then `None` at the clean end, and an `EOFException` for the cut frame:

```scala
println(s"first frame on the wire: ${wire.take(11).map(b => f"$b%02x").mkString(" ")}")
// first frame on the wire: 00 00 00 01 00 00 00 00 02 68 69
println(Frame.read(in))
// Some(Data(1,hi))
println(Frame.read(in))
// Some(End(1))
println(Frame.read(in))
// Some(Reset(3,cancelled))
println(Frame.read(in))
// None
try println(Frame.read(cut))
catch { case e: EOFException => println(s"cut mid-frame: $e") }
// cut mid-frame: java.io.EOFException
```

The first frame is stream id `1`, kind `0` (data), length `2`, and the two bytes of `"hi"`.

## Keep the Bookkeeping the Mux Leaves to You

`Mux` tracks open streams, but a protocol needs three more things it doesn't keep. First, the writer loop has to visit every stream that may have messages to send, and `Mux` has no method that lists its streams: it offers only `Mux#get` by id and `Mux#activeCount`. Second, a stream leaves the `Mux` as soon as it is fully closed, while its outbound queue may still hold the last messages the application sent, so the writer has to keep its own reference until that queue is empty. Third, `Mux#open` accepts an id again once its stream has been closed or cancelled, so a frame that arrives late for a finished stream would open a new one unless the protocol remembers the ids it has used.

`StreamTable` holds those three things, plus a queue of control frames (resets) that belong to no stream's queue:

```scala

// The bookkeeping a Mux leaves to the protocol:
//   draining - the streams the writer loop drains. A Mux can't list its
//              streams, and a closed stream leaves the Mux before its
//              outbound queue is empty.
//   usedIds  - every id this connection has used, so a late frame for a
//              finished stream can't reopen it.
//   control  - frames that belong to no stream's queue (RESET), waiting
//              for the writer loop.
final class StreamTable {
  val draining = new ConcurrentHashMap[Int, MessageStream]()
  val usedIds  = ConcurrentHashMap.newKeySet[Int]()
  val control  = new ConcurrentLinkedQueue[Frame]()
  val writer   = new AtomicReference[Thread]()

  def track(stream: MessageStream): Unit = {
    usedIds.add(stream.id)
    draining.put(stream.id, stream)
    wakeWriter()
  }

  def sendControl(frame: Frame): Unit = {
    control.add(frame)
    wakeWriter()
  }

  // The writer parks while it has nothing to write; wake it when there is.
  def wakeWriter(): Unit = {
    val thread = writer.get
    if (thread != null) LockSupport.unpark(thread)
  }
}
```

The writer loop parks while there is nothing to write. `StreamTable#wakeWriter` unparks it, so a new stream, a reset or a sent message reaches the socket without waiting for the next poll.

## Drain Outbound Queues onto the Socket

The writer loop is the only code that calls `MuxStream#takeOutbound`, and it runs on one thread, so every stream's outbound queue has exactly one consumer. Each pass sends pending control frames first, then takes up to 16 messages from each stream, encodes them and flushes the socket once:

```scala

// The only code that calls `takeOutbound`, on one thread, so every stream's
// outbound queue has exactly one consumer.
final class WriterLoop(out: DataOutputStream, table: StreamTable, running: AtomicBoolean) extends Runnable {
  // At most this many messages per stream per pass, so one busy stream
  // can't starve the others.
  private val Batch = 16

  def run(): Unit = {
    table.writer.set(Thread.currentThread())
    try loop()
    catch {
      // The connection is gone. Closing the stream closes the socket, which
      // ends the reader loop, which closes every stream in the Mux.
      case _: IOException =>
        running.set(false)
        try out.close()
        catch { case _: IOException => () }
    }
  }

  // After `running` turns false, keep going until a pass writes nothing, so
  // messages queued before the shutdown still reach the peer.
  @tailrec private def loop(): Unit = {
    val controlFrames = writeControlFrames(0)
    val messages      = table.draining.values.asScala.foldLeft(0)((sum, stream) => sum + drain(stream, 0))
    val wrote         = controlFrames + messages > 0
    if (wrote) out.flush()
    else if (running.get) LockSupport.parkNanos(10_000_000L) // idle: wait up to 10 ms or until woken
    if (running.get || wrote) loop()
  }

  @tailrec private def writeControlFrames(written: Int): Int =
    table.control.poll() match {
      case null  => written
      case frame =>
        Frame.write(out, frame)
        frame match {
          case Frame.Reset(id, _) => table.draining.remove(id) // drop whatever the reset stream still had queued
          case _                  => ()
        }
        writeControlFrames(written + 1)
    }

  @tailrec private def drain(stream: MessageStream, written: Int): Int =
    if (written == Batch) written
    else
      stream.takeOutbound() match {
        case Some(Message.Text(text)) =>
          Frame.write(out, Frame.Data(stream.id, text))
          drain(stream, written + 1)
        case Some(Message.End) =>
          Frame.write(out, Frame.End(stream.id))
          drain(stream, written + 1)
        case None =>
          written
        case _: MuxError => // closed, and its queue is empty: nothing more will come
          table.draining.remove(stream.id)
          written
      }
}
```

Three details matter here. The batch limit keeps one busy stream from starving the others: the large file upload later in this guide gets 16 frames per pass, then the chat and heartbeat streams get their turn. A stream whose `MuxStream#takeOutbound` returns an error is closed and has an empty queue, so the loop forgets it. And after shutdown begins, the loop keeps going until a pass writes nothing, so the last messages the application sent still reach the peer.

:::warning
Never call `MuxStream#takeOutbound` (or `MuxStream#receive`) on one stream from two threads at the same time. The queues are single-consumer ring buffers, and nothing detects the mistake. In a test with two threads draining one stream while a third sent 200,000 numbered messages, the drainers took some messages twice and lost others, and then the queue stopped accepting messages for good, so the sender stalled long before it had sent them all. One writer loop per connection avoids the problem by construction.
:::

## Route Incoming Frames to Their Streams

The reader loop reads one frame at a time and hands it to its stream. It is the only code that calls `MuxStream#offerInbound` and `MuxStream#signalRemoteClose`:

```scala

// Reads frames off the socket and hands each one to its stream. It is the
// only code that calls `offerInbound` and `signalRemoteClose`.
final class ReaderLoop(
  in: DataInputStream,
  mux: Mux[Int, Message, Message],
  table: StreamTable,
  incoming: BlockingQueue[MessageStream],
  acceptStreams: Boolean,
  running: AtomicBoolean
) extends Runnable {
  // How long this loop stopped reading because a stream's inbound queue was full.
  val pausedMillis = new AtomicLong(0L)

  def run(): Unit =
    try loop()
    catch {
      case _: IOException => () // the connection broke, or we closed the socket ourselves
    } finally {
      running.set(false)
      mux.closeAll(MuxError.Cancelled("connection", "connection closed"))
      table.wakeWriter()
    }

  @tailrec private def loop(): Unit =
    Frame.read(in) match {
      case Some(frame) =>
        handle(frame)
        loop()
      case None => () // the peer closed the connection
    }

  private def handle(frame: Frame): Unit = frame match {
    case Frame.Data(id, text) =>
      streamFor(id).foreach(deliver(_, Message.Text(text)))
    case Frame.End(id) =>
      streamFor(id).foreach { stream =>
        deliver(stream, Message.End) // tell the application, in order, after the data
        stream.signalRemoteClose()   // and tell the Mux: the peer sends nothing more
      }
    case Frame.Reset(id, reason) =>
      mux.cancel(id, MuxError.Cancelled(id, reason))
      table.draining.remove(id)
  }

  // The open stream for `id`, or a new one the first time the peer uses `id`.
  private def streamFor(id: Int): Option[MessageStream] =
    mux.get(id) match {
      case found @ Some(_)                                      => found
      case None if !acceptStreams || table.usedIds.contains(id) => None // a late frame for a finished stream: drop it
      case None                                                 =>
        mux.open(id) match {
          case stream: MessageStream =>
            table.track(stream)
            incoming.put(stream)
            Some(stream)
          case error: MuxError => // CapacityExceeded: refuse the stream
            table.usedIds.add(id)
            table.sendControl(Frame.Reset(id, s"refused: $error"))
            None
        }
    }

  // Back-pressure: while the application is behind, stop reading the socket.
  // Nothing is dropped and memory stays bounded, but every stream on this
  // connection waits for the slow one.
  @tailrec private def deliver(stream: MessageStream, message: Message): Unit =
    stream.offerInbound(message) match {
      case ()                    => ()
      case MuxError.QueueFull(_) =>
        pausedMillis.incrementAndGet()
        LockSupport.parkNanos(1_000_000L)
        deliver(stream, message)
      case _: MuxError => () // closed or cancelled on our side: drop the frame
    }
}
```

`ReaderLoop#streamFor` opens a stream the first time the peer uses an id, but only on the side that accepts streams (the server here) and only for an id this connection has never used. When `Mux#open` returns `CapacityExceeded`, which happens when `Mux#activeCount` has reached the capacity given to `Mux.apply`, the loop refuses the stream with a `Frame.Reset`, as HTTP/2 refuses streams over its limit.

An end frame becomes two calls, in this order: `ReaderLoop#deliver` puts `Message.End` into the inbound queue behind the data, and then `MuxStream#signalRemoteClose` tells the `Mux` that the peer sends nothing more. The order matters, because after `MuxStream#signalRemoteClose` the stream's `MuxStream#offerInbound` returns `StreamClosed`. A reset frame becomes `Mux#cancel` with a `MuxError.Cancelled` reason, which the application sees from `MuxStream#receive` once it has read the messages already queued.

When the connection ends, for any reason, the `finally` block calls `Mux#closeAll`, so every application thread waiting on a stream sees an error instead of waiting forever.

:::note
Only the client opens streams in this guide. If both sides of your protocol open streams, split the ids between them so that the two can never pick the same one: HTTP/2 gives clients the odd ids and servers the even ids.
:::

## Run Both Loops on One Connection

`MuxConnection` puts the pieces together for one socket: it creates the `Mux`, starts the reader and writer loops on their own threads, and gives the application a small API over them:

```scala

// One TCP connection carrying many streams: a Mux, a reader thread and a
// writer thread. `acceptStreams` is true on the side that lets the peer
// open streams (the server).
final class MuxConnection(socket: Socket, name: String, maxStreams: Int, acceptStreams: Boolean) {
  val mux: Mux[Int, Message, Message] = Mux[Int, Message, Message](maxStreams)

  private val table    = new StreamTable
  private val running  = new AtomicBoolean(true)
  private val incoming = new LinkedBlockingQueue[MessageStream]()
  private val in       = new DataInputStream(new BufferedInputStream(socket.getInputStream))
  private val out      = new DataOutputStream(new BufferedOutputStream(socket.getOutputStream))

  val reader: ReaderLoop = new ReaderLoop(in, mux, table, incoming, acceptStreams, running)
  // How many times `send` found the outbound queue full and waited.
  val sendWaits = new AtomicLong(0L)

  private val readerThread = new Thread(reader, s"$name-reader")
  private val writerThread = new Thread(new WriterLoop(out, table, running), s"$name-writer")
  readerThread.start()
  writerThread.start()

  // Opens a stream from this side.
  def open(id: Int): MessageStream | MuxError =
    mux.open(id) match {
      case stream: MessageStream =>
        table.track(stream)
        stream
      case error: MuxError => error
    }

  // The next stream the peer opened, or None once the connection is closed.
  @tailrec def accept(): Option[MessageStream] = {
    val stream = incoming.poll(10, TimeUnit.MILLISECONDS)
    if (stream != null) Some(stream)
    else if (!running.get && incoming.isEmpty) None
    else accept()
  }

  // Waits while the outbound queue is full instead of dropping the message.
  @tailrec def send(stream: MessageStream, message: Message): Unit | MuxError =
    stream.send(message) match {
      case MuxError.QueueFull(_) =>
        sendWaits.incrementAndGet()
        LockSupport.parkNanos(1_000_000L)
        send(stream, message)
      case result =>
        table.wakeWriter()
        result
    }

  // Half-close: queue End behind the data already sent, then stop sending.
  // Call it from the thread that sends on this stream.
  def finish(stream: MessageStream): Unit | MuxError =
    send(stream, Message.End) match {
      case ()              => stream.halfClose()
      case error: MuxError => error
    }

  // Cancels a stream on both ends: drop it here, then send RESET to the peer.
  def reset(id: Int, reason: String): Unit = {
    mux.cancel(id, MuxError.Cancelled(id, reason))
    table.sendControl(Frame.Reset(id, reason))
  }

  // Clean shutdown: fail every stream, let the writer flush what was already
  // queued, then close the socket, which ends the reader loop.
  def close(): Unit = {
    mux.closeAll(MuxError.Cancelled("connection", s"$name closed the connection"))
    running.set(false)
    table.wakeWriter()
    writerThread.join(5000)
    socket.close()
    readerThread.join(5000)
  }

  // Blocks until the peer closes the connection.
  def awaitPeerClose(): Unit = readerThread.join()
}

object MuxConnection {
  // `receive` never blocks: poll it, parking for 1 ms while the queue is empty.
  @tailrec def next(stream: MessageStream): Message | MuxError =
    stream.receive() match {
      case Some(message) => message
      case None          =>
        LockSupport.parkNanos(1_000_000L)
        next(stream)
      case error: MuxError => error
    }
}
```

`MuxConnection#send` retries while the stream's outbound queue is full, and `MuxConnection#finish` and `MuxConnection#reset` are covered in their own sections below. `MuxConnection.next` waits for the next inbound message: `MuxStream#receive` never blocks, it returns `None` when the queue is empty, so the loop polls and parks for a millisecond between tries.

The loops run on plain JDK threads, not on `Async.start` from the async module. Both loops block for long stretches (the reader in a socket read, the writer in a socket write when the peer is slow), and `Async.start` runs its body on the JVM's shared `ForkJoinPool.commonPool()`, where a handful of blocked loops can use up every worker on a machine with few cores. A thread per loop is also what a socket's blocking API expects. For the same reason, the loops read and write through `DataInputStream` and `DataOutputStream` rather than a zio-blocks `Stream`: a frame decoder needs to read exactly 4, 1, 4 and `length` bytes at a time, which `DataInputStream#readFully` does directly.

## Serve Several Streams over One Connection

With `MuxConnection` in place, the application code only sends and receives. On the server, each stream gets a handler that runs on its own thread. The chat room echoes every line back, and the heartbeat answers every ping:

```scala

// Server side, stream 1: the chat room. Echoes every line back to the room.
def chatRoom(conn: MuxConnection, stream: MessageStream): Unit = {
  @tailrec def loop(): Unit =
    MuxConnection.next(stream) match {
      case Message.Text(line) =>
        conn.send(stream, Message.Text(s"[room] $line"))
        loop()
      case Message.End     => conn.finish(stream) // the client left: end our side too
      case error: MuxError => println(s"server | chat: $error")
    }
  loop()
}

// Server side, stream 5: the heartbeat. Answers every "ping n" with "pong n".
def heartbeat(conn: MuxConnection, stream: MessageStream): Unit = {
  @tailrec def loop(): Unit =
    MuxConnection.next(stream) match {
      case Message.Text(ping) =>
        conn.send(stream, Message.Text(ping.replace("ping", "pong")))
        loop()
      case Message.End     => conn.finish(stream)
      case error: MuxError => println(s"server | heartbeat: $error")
    }
  loop()
}

// Client side: open a stream from our end, or stop with the reason.
def openStream(conn: MuxConnection, id: Int): MessageStream =
  conn.open(id) match {
    case stream: MessageStream => stream
    case error: MuxError       => throw new IllegalStateException(s"cannot open stream $id: $error")
  }

// Client side: send one line and wait for the reply on the same stream.
def ask(conn: MuxConnection, stream: MessageStream, text: String): Message | MuxError =
  conn.send(stream, Message.Text(text)) match {
    case ()              => MuxConnection.next(stream)
    case error: MuxError => error
  }

def inThread(name: String)(body: => Unit): Thread = {
  val thread = new Thread(() => body, name)
  thread.start()
  thread
}

def chatSession(conn: MuxConnection, chat: MessageStream): Unit = {
  for (line <- Seq("alice: the build is green", "alice: shipping it")) {
    println(s"client | stream 1: ${ask(conn, chat, line)}")
    Thread.sleep(300)
  }
  conn.finish(chat)
  println(s"client | stream 1: ${MuxConnection.next(chat)}") // End: the server finished its side too
}

def heartbeatSession(conn: MuxConnection, heartbeat: MessageStream): Unit = {
  val roundTrips = (1 to 10).map { n =>
    val start  = System.nanoTime()
    val reply  = ask(conn, heartbeat, s"ping $n")
    val millis = (System.nanoTime() - start) / 1000000
    println(s"client | stream 5: $reply after $millis ms")
    Thread.sleep(200)
    millis
  }
  conn.finish(heartbeat)
  println(s"client | stream 5: ${MuxConnection.next(heartbeat)}; slowest round trip ${roundTrips.max} ms")
}
```

The client side of the same file opens streams with `MuxConnection#open`, sends a line and waits for its reply with `ask`, and runs one session per stream on its own thread. Each session thread is the only thread that calls `MuxStream#receive` on its stream, which keeps the single-consumer rule.

The client opens streams 1, 3 and 5, asks one question on each, and then runs the chat, upload and heartbeat sessions at once (the complete client is in [Shut Down Cleanly](#shut-down-cleanly)). The output of one run shows messages from the three streams interleaved on the one connection, each reply arriving on the stream that asked for it:

```text
server | listening on port 38335
server | client opened stream 1
client | stream 1: Text([room] alice: hi all)
server | client opened stream 3
client | stream 3: Text(ready for release-notes.txt)
server | client opened stream 5
client | stream 5: Text(pong 0)
client | stream 7: Cancelled(7,refused: CapacityExceeded(3))
client | stream 1: Text([room] alice: the build is green)
client | stream 5: Text(pong 1) after 2 ms
client | stream 5: Text(pong 2) after 302 ms
server | upload: stored 500 lines
client | stream 1: Text([room] alice: shipping it)
client | stream 3: sent 1500 lines, then End
client | stream 5: Text(pong 3) after 311 ms
server | upload: stored 1000 lines
client | stream 1: End
client | stream 5: Text(pong 4) after 108 ms
client | stream 5: Text(pong 5) after 2 ms
server | upload: stored 1500 lines
server | upload: the client finished sending; replying on the same stream
client | stream 3: Text(stored release-notes.txt: 1500 lines, 63000 bytes)
client | stream 3: End
client | stream 3: send after both sides finished: StreamClosed(3)
client | stream 5: Text(pong 6) after 1 ms
client | stream 5: Text(pong 7) after 1 ms
client | stream 5: Text(pong 8) after 1 ms
client | stream 5: Text(pong 9) after 1 ms
client | stream 5: Text(pong 10) after 2 ms
client | stream 5: End; slowest round trip 311 ms
```

The thread scheduling decides the exact order, so each run interleaves the lines differently.

## Half-Close a Stream

Half-closing a stream says "I am done sending" while the other direction stays open, so the peer can still answer. The upload uses it: the client sends the file, half-closes, and then reads the server's summary on the same stream.

`MuxStream#halfClose` changes what the stream allows, and so does `MuxStream#signalRemoteClose`. To see what each operation returns in each state, we open two streams on a `Mux` of their own, with no socket involved:

```scala

val mux = Mux[Int, Message, Message](4)

def open(id: Int): MessageStream = mux.open(id) match {
  case stream: MessageStream => stream
  case error: MuxError       => throw new IllegalStateException(error.toString)
}

val transfer = open(3)
val room     = open(1)
```

The first stream half-closes its own side and then hears that the peer has finished too; the second hears from the peer first:

```scala
transfer.halfClose() // we finished sending
println(s"after halfClose: send = ${transfer.send(Message.Text("more"))}")
// after halfClose: send = StreamClosed(3)
println(s"after halfClose: offerInbound = ${transfer.offerInbound(Message.Text("summary"))}")
// after halfClose: offerInbound = ()
println(s"after halfClose: receive = ${transfer.receive()}")
// after halfClose: receive = Some(Text(summary))
transfer.signalRemoteClose() // the peer finished too
println(s"after both: isClosed = ${transfer.isClosed}, mux.get(3) = ${mux.get(3)}")
// after both: isClosed = true, mux.get(3) = None
println(s"after both: receive = ${transfer.receive()}")
// after both: receive = StreamClosed(3)

room.signalRemoteClose() // the peer finished first
println(s"peer finished first: receive = ${room.receive()}, isHalfClosed = ${room.isHalfClosed}")
// peer finished first: receive = None, isHalfClosed = true
println(s"peer finished first: offerInbound = ${room.offerInbound(Message.Text("late"))}")
// peer finished first: offerInbound = StreamClosed(1)
println(s"peer finished first: send = ${room.send(Message.Text("still talking"))}")
// peer finished first: send = ()
```

After `MuxStream#halfClose`, `MuxStream#send` fails with `StreamClosed`, while inbound messages are still accepted and received. Once both sides are done, the stream is closed and gone from the `Mux`, which frees its slot for a new stream, and `MuxStream#receive` returns the terminal error after the queue drains.

The second stream shows why this guide sends `Message.End` in-band. When the peer finishes first, `MuxStream#receive` returns `None` once the queue is empty, exactly as it does when the next message has not arrived yet, and `MuxStream#isHalfClosed` is `true` whichever side finished. Nothing on the stream tells the application that the peer is done, so the reader loop delivers `Message.End` as the last inbound message before it calls `MuxStream#signalRemoteClose`. On the sending side, `MuxConnection#finish` sends `Message.End` through the same outbound queue as the data, so it can't overtake the data, and only then calls `MuxStream#halfClose`.

Over the socket, the server's upload handler stores lines until it receives `Message.End`, then replies with a summary and finishes its own side. The client sends the file, finishes, and keeps receiving on the same stream:

```scala

// Server side, stream 3: a file upload. The first line names the file; every
// later line is file content. Storing a line takes 1 ms, so this handler is
// slower than the client. When the client half-closes, reply with a summary
// on the same stream, then half-close our side.
def upload(conn: MuxConnection, stream: MessageStream): Unit = {
  @tailrec def store(fileName: String, lines: Int, bytes: Long): Unit =
    MuxConnection.next(stream) match {
      case Message.Text(header) if fileName.isEmpty =>
        val name = header.stripPrefix("upload ")
        conn.send(stream, Message.Text(s"ready for $name"))
        store(name, lines, bytes)
      case Message.Text(line) =>
        Thread.sleep(1) // a slow disk
        if ((lines + 1) % 500 == 0) println(s"server | upload: stored ${lines + 1} lines")
        store(fileName, lines + 1, bytes + line.length)
      case Message.End =>
        println("server | upload: the client finished sending; replying on the same stream")
        conn.send(stream, Message.Text(s"stored $fileName: $lines lines, $bytes bytes"))
        conn.finish(stream)
      case error: MuxError =>
        println(s"server | upload: $error")
    }
  store("", 0, 0L)
}

// Client side: send the file, half-close, and keep reading the reply.
def uploadSession(conn: MuxConnection, upload: MessageStream): Unit = {
  val lines = 1500
  val sent  = (1 to lines).takeWhile { n =>
    conn.send(upload, Message.Text(f"$n%04d: fixed a race in the release pipeline")) match {
      case ()              => true
      case error: MuxError =>
        println(s"client | stream 3: $error")
        false
    }
  }.size
  conn.finish(upload)
  println(s"client | stream 3: sent $sent lines, then End")
  println(s"client | stream 3: ${MuxConnection.next(upload)}") // the server's summary
  println(s"client | stream 3: ${MuxConnection.next(upload)}") // End
  println(s"client | stream 3: send after both sides finished: ${conn.send(upload, Message.Text("one more line"))}")
}
```

The upload's lines from the run above show the reply arriving after the client's half-close, and `StreamClosed` once both sides are done:

```text
server | upload: stored 500 lines
client | stream 3: sent 1500 lines, then End
server | upload: stored 1000 lines
server | upload: stored 1500 lines
server | upload: the client finished sending; replying on the same stream
client | stream 3: Text(stored release-notes.txt: 1500 lines, 63000 bytes)
client | stream 3: End
client | stream 3: send after both sides finished: StreamClosed(3)
```

## Handle Back-Pressure Without Dropping Data

Every stream's queues are bounded: each direction holds 256 messages, a limit fixed by the library. When a queue is full, `MuxStream#send` and `MuxStream#offerInbound` return `MuxError.QueueFull` instead of adding the message. To see where that starts, we open one stream with no socket involved:

```scala

val queueMux   = Mux[Int, Message, Message](1)
val slowStream = queueMux.open(3) match {
  case stream: MessageStream => stream
  case error: MuxError       => throw new IllegalStateException(error.toString)
}
```

Then we fill its inbound queue, try one message more, and receive one to make room:

```scala
println(s"accepted: ${(1 to 256).count(n => slowStream.offerInbound(Message.Text(s"line $n")) == ())} of 256")
// accepted: 256 of 256
println(s"offerInbound #257: ${slowStream.offerInbound(Message.Text("line 257"))}")
// offerInbound #257: QueueFull(256)
println(s"receive: ${slowStream.receive()}")
// receive: Some(Text(line 1))
println(s"offerInbound #257 again: ${slowStream.offerInbound(Message.Text("line 257"))}")
// offerInbound #257 again: ()
```

Over a socket, the inbound queue fills when the application reads a stream more slowly than the peer sends. The upload handler above stores one line per millisecond, while the client sends 1,500 lines as fast as it can. The reader loop then gets `QueueFull` from `MuxStream#offerInbound`, and it has to do something with the frame it is holding. These are the options:

| Strategy                        | Data | Memory    | Other streams on the connection          |
| ------------------------------- | ---- | --------- | ---------------------------------------- |
| Drop the frame                  | Lost | Bounded   | Unaffected                               |
| Buffer it outside the `Mux`     | Kept | Unbounded | Unaffected                               |
| Stop reading until there's room | Kept | Bounded   | Wait behind the slow stream              |
| Per-stream flow control         | Kept | Bounded   | Unaffected, at the cost of more protocol |

`ReaderLoop#deliver` stops reading: it parks for a millisecond and retries the same frame until the queue has room. While it waits, it reads nothing from the socket, so the kernel's receive buffer fills, TCP tells the client to stop sending, the client's writer loop blocks in its socket write, the client's outbound queues fill, and `MuxConnection#send` on the client starts to wait on `QueueFull`. The pressure travels all the way back to the code that produces the data, and no queue grows past its bound.

The price is head-of-line blocking: while the reader waits for the upload, it can't deliver the heartbeat's pings either. The run above shows both effects. The heartbeat's round trips grow from a few milliseconds to a few hundred while the upload is in flight, and drop back once it ends; the closing lines count how long each side waited:

```text
client | stream 5: Text(pong 1) after 2 ms
client | stream 5: Text(pong 2) after 302 ms
client | stream 5: Text(pong 3) after 311 ms
client | stream 5: Text(pong 4) after 108 ms
client | stream 5: Text(pong 5) after 2 ms
client | stream 5: Text(pong 6) after 1 ms
client | stream 5: Text(pong 7) after 1 ms
client | stream 5: Text(pong 8) after 1 ms
client | stream 5: Text(pong 9) after 1 ms
client | stream 5: Text(pong 10) after 2 ms
client | stream 5: End; slowest round trip 311 ms
client | connection closed; send waited 645 times for a full outbound queue
server | connection closed; the reader paused 1226 ms for full inbound queues
```

The examples shrink the socket buffers to 4 KiB (`ServerSocket#setReceiveBufferSize` and `Socket#setSendBufferSize`) so that the effect shows up within seconds; with the default buffers the same thing happens, only later.

If one slow stream must not hold up the others, the fix is per-stream flow control, as HTTP/2 does with its flow-control windows: the receiver grants each stream a credit of messages, the sender's writer loop skips a stream that has used its credit, and the receiver grants more credit as the application calls `MuxStream#receive`. It is more protocol than this guide needs, but it plugs into the same two places: the writer loop decides which streams it may drain, and the reader loop never meets a full queue.

## Refuse and Cancel Streams

Two things end a stream abruptly: the server refusing it, and either side cancelling it. Both travel as a reset frame and arrive as `MuxError.Cancelled`. The server's job handler runs any stream that isn't the chat, upload or heartbeat, and the client has one function for each case:

```scala

// Server side, any other stream: a job. Accepts each command, until the client
// ends the stream or cancels it.
def job(conn: MuxConnection, stream: MessageStream): Unit = {
  @tailrec def loop(): Unit =
    MuxConnection.next(stream) match {
      case Message.Text(command) =>
        println(s"server | job ${stream.id}: started '$command'")
        conn.send(stream, Message.Text(s"accepted $command"))
        loop()
      case Message.End     => conn.finish(stream)
      case error: MuxError => println(s"server | job ${stream.id}: $error")
    }
  loop()
}

// Client side: a fourth stream is over the server's limit of three, so the
// server refuses it with a RESET.
def refuseOverLimit(conn: MuxConnection): Unit = {
  val extra = openStream(conn, 7)
  println(s"client | stream 7: ${ask(conn, extra, "hello?")}")
}

// Client side: start a job, then cancel it on both ends.
def cancelJob(conn: MuxConnection): Unit = {
  val job = openStream(conn, 9)
  println(s"client | stream 9: ${ask(conn, job, "rebuild search index")}")
  conn.reset(9, "client cancelled the job")
  println(s"client | stream 9 after reset: ${job.receive()}")
}
```

`refuseOverLimit` opens a fourth stream while the first three are still open. The client's own `Mux` accepts it, because the client allows four streams, but the server's allows three, so the server's `Mux#open` returns `CapacityExceeded(3)` and the reader loop answers with a reset. `cancelJob` starts a job and calls `MuxConnection#reset`, which cancels the stream locally with `Mux#cancel` and sends a reset frame, so the server's handler sees the same reason:

```text
client | stream 7: Cancelled(7,refused: CapacityExceeded(3))
server | client opened stream 9
server | job 9: started 'rebuild search index'
client | stream 9: Text(accepted rebuild search index)
client | stream 9 after reset: Cancelled(9,client cancelled the job)
server | job 9: Cancelled(9,client cancelled the job)
```

The cancelled side's `MuxStream#receive` returns the reason passed to `Mux#cancel`, which is why both sides print the same `Cancelled`. `MuxStream#takeOutbound` on a cancelled stream returns `StreamClosed` instead, once its queue is empty; `MuxConnection#reset` doesn't wait for that, because the writer drops the stream as soon as it sends the reset frame.

## Shut Down Cleanly

The last piece is the program around the streams. The server listens, serves one client connection, routes each new stream to its handler, and returns when the client closes the connection. The client connects, runs every session from the previous sections, and closes the connection:

```scala

// Port 0 picks any free port.
def listen(port: Int): ServerSocket = {
  val server = new ServerSocket()
  server.setReceiveBufferSize(4096) // small buffers make back-pressure show up within seconds
  server.setSoTimeout(60000)        // stop waiting for a client after a minute
  server.bind(new InetSocketAddress("localhost", port))
  server
}

// Serves one client connection: one handler thread per stream the client opens,
// at most three streams at a time. Returns when the client closes the connection.
def serve(server: ServerSocket): Unit = {
  println(s"server | listening on port ${server.getLocalPort}")
  val conn = new MuxConnection(server.accept(), "server", maxStreams = 3, acceptStreams = true)

  @tailrec def startHandlers(started: List[Thread]): List[Thread] =
    conn.accept() match {
      case Some(stream) =>
        println(s"server | client opened stream ${stream.id}")
        val handler = inThread(s"server-stream-${stream.id}") {
          stream.id match {
            case 1 => chatRoom(conn, stream)
            case 3 => upload(conn, stream)
            case 5 => heartbeat(conn, stream)
            case _ => job(conn, stream)
          }
        }
        startHandlers(handler :: started)
      case None => started // the client closed the connection
    }

  startHandlers(Nil).foreach(_.join())
  conn.close()
  println(s"server | connection closed; the reader paused ${conn.reader.pausedMillis.get} ms for full inbound queues")
}

def runClient(port: Int): Unit = {
  val socket = new Socket()
  socket.setSendBufferSize(4096) // small buffers make back-pressure show up within seconds
  socket.connect(new InetSocketAddress("localhost", port), 5000)
  val conn = new MuxConnection(socket, "client", maxStreams = 4, acceptStreams = false)

  // Three streams over the one connection, one request and reply on each.
  val chat      = openStream(conn, 1)
  val upload    = openStream(conn, 3)
  val heartbeat = openStream(conn, 5)
  println(s"client | stream 1: ${ask(conn, chat, "alice: hi all")}")
  println(s"client | stream 3: ${ask(conn, upload, "upload release-notes.txt")}")
  println(s"client | stream 5: ${ask(conn, heartbeat, "ping 0")}")

  refuseOverLimit(conn)

  // All three streams at once, one thread per stream.
  val sessions = Seq(
    inThread("client-chat")(chatSession(conn, chat)),
    inThread("client-upload")(uploadSession(conn, upload)),
    inThread("client-heartbeat")(heartbeatSession(conn, heartbeat))
  )
  sessions.foreach(_.join())
  println(s"client | active streams: ${conn.mux.activeCount}")

  cancelJob(conn)

  conn.close()
  println(s"client | connection closed; send waited ${conn.sendWaits.get} times for a full outbound queue")
}
```

`MuxConnection#close` shuts down in a fixed order. `Mux#closeAll` first fails every stream, so no new message can be queued and every waiting thread wakes up with an error. Then the writer loop flushes what was already queued and exits, and closing the socket ends the reader loop. On the server, the reader loop sees the client's close as the end of the stream, calls `Mux#closeAll` itself, and `MuxConnection#accept` returns `None`, so `serve` joins the handler threads and closes its side. Nothing is left running: both programs exit on their own, and the port is free again.

Each side prints a last line on the way out:

```text
client | connection closed; send waited 645 times for a full outbound queue
server | connection closed; the reader paused 1226 ms for full inbound queues
```

## Putting It Together

The complete example runs the server and the client from the previous sections in one JVM: the server on a background thread and the client on the main thread, over a localhost connection. It differs from the separate `ChatServer` and `ChatClient` programs in one way: the server listens on port 0, so the operating system picks a free port, and the client connects to that port:

```scala title="mux-examples/src/main/scala/muxoverasocket/CompleteExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-mux:0.0.56
//> using file Frame.scala
//> using file StreamTable.scala
//> using file WriterLoop.scala
//> using file ReaderLoop.scala
//> using file MuxConnection.scala
//> using file ChatStreams.scala
//> using file Upload.scala
//> using file Jobs.scala
//> using file ServerAndClient.scala
// Run from this folder:  scala-cli run CompleteExample.scala
// Or from mux-examples:  sbt "runMain muxoverasocket.CompleteExample"

package muxoverasocket

/**
 * How to Run a Mux over a Real Socket — Complete Example
 *
 * Runs the server and the client in one JVM over a localhost TCP connection on
 * a free port, and exits when both are done.
 */
object CompleteExample {

  def main(args: Array[String]): Unit = {
    val server       = listen(0)
    val serverThread = inThread("server") {
      try serve(server)
      finally server.close()
    }
    runClient(server.getLocalPort)
    serverThread.join()
  }
}
```

Running it prints the whole session. The numbers and the interleaving of the middle lines change from run to run:

```text
server | listening on port 38335
server | client opened stream 1
client | stream 1: Text([room] alice: hi all)
server | client opened stream 3
client | stream 3: Text(ready for release-notes.txt)
server | client opened stream 5
client | stream 5: Text(pong 0)
client | stream 7: Cancelled(7,refused: CapacityExceeded(3))
client | stream 1: Text([room] alice: the build is green)
client | stream 5: Text(pong 1) after 2 ms
client | stream 5: Text(pong 2) after 302 ms
server | upload: stored 500 lines
client | stream 1: Text([room] alice: shipping it)
client | stream 3: sent 1500 lines, then End
client | stream 5: Text(pong 3) after 311 ms
server | upload: stored 1000 lines
client | stream 1: End
client | stream 5: Text(pong 4) after 108 ms
client | stream 5: Text(pong 5) after 2 ms
server | upload: stored 1500 lines
server | upload: the client finished sending; replying on the same stream
client | stream 3: Text(stored release-notes.txt: 1500 lines, 63000 bytes)
client | stream 3: End
client | stream 3: send after both sides finished: StreamClosed(3)
client | stream 5: Text(pong 6) after 1 ms
client | stream 5: Text(pong 7) after 1 ms
client | stream 5: Text(pong 8) after 1 ms
client | stream 5: Text(pong 9) after 1 ms
client | stream 5: Text(pong 10) after 2 ms
client | stream 5: End; slowest round trip 311 ms
client | active streams: 0
server | client opened stream 9
server | job 9: started 'rebuild search index'
client | stream 9: Text(accepted rebuild search index)
client | stream 9 after reset: Cancelled(9,client cancelled the job)
server | job 9: Cancelled(9,client cancelled the job)
client | connection closed; send waited 645 times for a full outbound queue
server | connection closed; the reader paused 1226 ms for full inbound queues
```

## Running the Examples

All runnable code from this guide is in [`mux-examples/src/main/scala/muxoverasocket/`](https://github.com/zio/zio-blocks/tree/main/mux-examples/src/main/scala/muxoverasocket). Run any file from that folder with `scala-cli run <File>.scala`, or from `mux-examples` with `sbt "runMain muxoverasocket.<ObjectName>"`. To run the server and the client as two processes, start `ChatServer` with a port (`sbt "runMain muxoverasocket.ChatServer 7070"`), then `ChatClient` with the same port in a second terminal; the server serves one client and exits.

## Going Further

- The [`Mux` reference](../reference/mux.mdx) lists every operation, error and state transition.
- [How to Stream Files and Sockets Without Leaking Resources](./stream-files-and-sockets.md) moves bytes over a socket with zio-blocks streams, for a single stream of data per connection.
- To serve many clients, accept connections in a loop and give each one its own `MuxConnection`; each connection has its own `Mux`, so stream ids and limits are per connection.
- To serve many streams per connection without a thread per stream, replace the per-stream handler threads with one application thread that polls `MuxStream#receive` on every stream it owns, keeping each stream's `MuxStream#receive` on that one thread.

---

## How to Run Streams Without Blocking

In this guide, we total the price of a coffee order by asking a price service for each item, without parking a thread while we wait. By the end, you will have a pipeline written once against `Stream` and `Async` that runs on the JVM and on Scala.js, and you will know where it is safe to block, how its errors and cancellation behave, and how to read a browser or Node `ReadableStream` such as a `fetch` response body as a stream.

We start from the blocking terminal that causes the trouble, swap it for its `Async` twin, add async work per element with `runForeachAsync` and `runFoldAsync`, and compose the result. Then we look at errors and cancellation, run the same pipeline on Scala.js, and read a `ReadableStream`.

## The Problem

A blocking terminal such as `Stream#runCollect` or `Stream#runFold` makes one promise: when it returns, the stream is finished and the answer is in your hand. On the JVM, the promise is kept by parking the calling thread until the stream completes. If the stream does a remote call per element, the thread sits idle for every call. On Scala.js the promise cannot be kept at all, because the single thread that would wait is the one that has to run the callbacks that deliver the answer.

For example, here is a price total over a list of items, where each price comes from a call that returns a `Future`. A caller that wants one number back has nothing to do but wait for every price in turn:

```scala

def totalBlocking(items: List[String], price: String => Future[Long]): Long =
  items.foldLeft(0L)((sum, item) => sum + Await.result(price(item), 5.seconds))
```

This parks the calling thread once per item, one item at a time, so a hundred 100 ms lookups hold a thread for ten seconds doing nothing. A web server that does this per request runs out of threads long before it runs out of CPU. A blocking stream terminal such as `Stream#runCollect` or `Stream#runFold` has the same cost, and on Scala.js it does not even compile, because those terminals do not exist there.

In this guide, we replace each blocking call with an asynchronous one that returns immediately with an `Async` and let the caller decide when, and whether, to wait.

## Prerequisites

To add the streams module to your project, include one of these dependencies in your `build.sbt`. The async module comes with it:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-streams" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental"): the same module with the zio-blocks-next- prefix
libraryDependencies += "dev.zio" %% "zio-blocks-next-streams" % "0.0.56"
```

For a Scala.js or cross-built project, use `%%%` instead of `%%`. [How to Add ZIO Blocks to Your Build](../getting-started/installation.md) explains the two artifact names. Streams need JDK 21 at runtime on the JVM.

This guide assumes you have worked through [Your First Streaming Pipeline](./your-first-streaming-pipeline.md) and [Getting Started with Async](./async-getting-started.md). To follow along, we need a stream of order lines and a price service. `MenuPrices.priceInCents` answers after a 100 ms delay on another thread, the way a remote call would, and fails for an item the menu does not list. The prices are sample data:

```scala

final case class OrderLine(item: String, table: Int)

object MenuPrices {
  val order: List[OrderLine] = List(
    OrderLine("Espresso", 4),
    OrderLine("Flat White", 7),
    OrderLine("Croissant", 4)
  )

  private val cents: Map[String, Long] = Map(
    "Espresso"   -> 350L,
    "Flat White" -> 480L,
    "Croissant"  -> 420L
  )

  def priceInCents(item: String): Async[Long] =
    Async.fromCompletionStage(
      CompletableFuture.supplyAsync(
        () => cents.getOrElse(item, throw new NoSuchElementException(s"No price for $item")),
        CompletableFuture.delayedExecutor(100, MILLISECONDS)
      )
    )
}
```

`priceInCents` wraps a `CompletableFuture` with `Async.fromCompletionStage`. [How to Wrap Callback and Future-Based APIs in Async](./wrap-callback-and-future-apis.md) covers that bridge and its siblings for `Future`, JavaScript promises and callbacks.

## Swapping a Blocking Terminal for Its Async Twin

Every blocking terminal has a twin whose name ends in `Async`: `Stream#runCollect` becomes `Stream#runCollectAsync`, `Stream#runDrain` becomes `Stream#runDrainAsync`, and `Stream#runFold` becomes `Stream#runFoldAsync`. The twin returns `Async[Either[E, Z]]`: the same `Either` the blocking terminal returns, wrapped in an `Async` that completes later.

To collect the price of every line in the order:

```scala
val prices: Stream[Nothing, Long] =
  Stream.fromIterator(MenuPrices.order.iterator).mapAsync(line => MenuPrices.priceInCents(line.item))

val pending: Async[Either[Nothing, Chunk[Long]]] = prices.runCollectAsync
```

Creating `pending` takes no time, because the terminal returns at once and the work happens when something drives the `Async`. To get the answer, we drive it with `Async#block`:

```scala
println(pending.block)
// Right(Chunk(350,480,420))
```

Running the same stream with the blocking `runCollect` and with `runCollectAsync` and timing the two calls shows the difference:

```text
runCollect returned after 434 ms with Right(Chunk(350,480,420))
runCollectAsync returned after 1 ms, nothing has run yet
the result is Right(Chunk(350,480,420))
```

The blocking call holds its thread for the three lookups; the async call hands back an `Async` immediately. The availability table in [Platform Differences](../reference/streams/execution-and-compatibility/platform-differences.md) lists the replacement for every blocking member.

:::warning
An async terminal does not make a blocking source non-blocking. The `Async` runs on whatever thread drives it, so a source whose pull blocks, such as an iterator that sleeps, a file read or `Stream.fromInputStream`, still blocks that thread. In a probe with an iterator that sleeps 200 ms per element, `runCollectAsync` returns after about 21 ms, and `Async#block` then takes about 654 ms because the sleeping runs inside it. Only sources and callbacks that report completion through an `Async`, like `mapAsync` over `MenuPrices.priceInCents`, wait without holding a thread.
:::

## Doing Async Work for Each Element

`mapAsync` lets a stage return an `Async`, but often we only want the final answer. `runForeachAsync` and `runFoldAsync` take a step function that returns an `Async` and run it one element at a time, so no thread waits for the price of an item.

`runForeachAsync` runs an effect for each element and returns `Async[Either[E, Unit]]`. To print each order line with its price:

```scala
val order: Stream[Nothing, OrderLine] = Stream.fromIterator(MenuPrices.order.iterator)

val printed: Async[Either[Nothing, Unit]] =
  order.runForeachAsync { line =>
    MenuPrices.priceInCents(line.item).map(cents => println(s"${line.item} for table ${line.table}: $cents cents"))
  }
```

To run `printed` and print one line per order line:

```scala
printed.block
// Espresso for table 4: 350 cents
// Flat White for table 7: 480 cents
// Croissant for table 4: 420 cents
// res2: Either[Nothing, Unit] = Right(())
```

`runFoldAsync` threads an accumulator through the stream. To total the order:

```scala
val total: Async[Either[Nothing, Long]] =
  order.runFoldAsync(0L)((sum, line) => MenuPrices.priceInCents(line.item).map(sum + _))
```

To print the total:

```scala
println(total.block)
// Right(1250)
```

`runFoldAsync` has an overload for each primitive accumulator, `Double`, `Float`, `Int` and `Long`, and a generic one. The compiler picks the overload from the static type of the zero, which is why we wrote `0L` and not `0`. The primitive overloads keep the accumulator unboxed, so a fold over a primitive total does not allocate a box per step. The `Float` overload has no blocking twin.

The step function runs for one element at a time. Each lookup finishes before the next one starts, so this total takes about 300 ms for three order lines. To overlap the lookups, use `mapParAsync` on the stream; [Asynchronous Stream Execution](../reference/streams/execution-and-compatibility/async-execution.md) covers the bounded-concurrency operator.

## Using the Result Without Waiting

An `Async` is a value you pass around, so most of the time we do not block at all. We write the pipeline as a function that returns an `Async`, and let the caller compose it:

```scala
def totalInCents(items: Stream[Nothing, String]): Async[Either[Nothing, Long]] =
  items.runFoldAsync(0L)((sum, item) => MenuPrices.priceInCents(item).map(sum + _))
```

`Async#map` transforms the result when it arrives, with no thread waiting in between:

```scala
val items: Stream[Nothing, String] = Stream.fromIterator(MenuPrices.order.iterator.map(_.item))

val label: Async[Either[Nothing, String]] =
  totalInCents(items).map(_.map(cents => f"The order costs ${cents / 100.0}%.2f"))
```

To print the label:

```scala
println(label.block)
// Right(The order costs 12.50)
```

Code that has not moved to `Async` usually wants a `Future`. On the JVM, `Async#toFuture` takes an implicit `ExecutionContext` and returns one. On Scala.js, `Async#toJsPromise` returns a `js.Promise`. To hand the label to a `Future`-based caller:

```scala

val future: Future[Either[Nothing, String]] = label.toFuture
println(Await.result(future, 5.seconds))
```

To run the `Async` in the background and keep a handle that you can join or cancel, use `Async#start`, which returns an `Async.Running`. [Getting Started with Async](./async-getting-started.md#8-forking-and-cancellation-start-and-asyncrunning) covers it.

:::tip
Block only at the outermost edge of a JVM program, such as the end of `main`, where there is nothing left to do but wait. A call to `Async#block` anywhere else holds a thread, which is what we set out to avoid. On Scala.js, `Async#block` returns a value that is already complete and throws `IllegalStateException` when it would have to wait, so there is no edge to block at. The host drives the `Async`, as the next sections show.
:::

## Errors and Cancellation

A stream has two kinds of failure, and the `Async` keeps them apart. A typed error, the `E` of `Stream[E, A]`, arrives as `Left` inside the result. A defect, an exception thrown by user code or a failed `Async` inside a step, fails the `Async` itself.

A typed error lands in the `Either`:

```scala
val typed: Stream[String, Long] = Stream.fail("the price service is down")
// typed: Stream[String, Long] = Stream.fail(...)
println(typed.runFoldAsync(0L)((sum, cents) => Async.succeed(sum + cents)).block)
// Left(the price service is down)
```

A defect fails the `Async`. We total the order with an item the menu does not list, and look at it through `Async#either`, then through `Async#block`:

```scala
val unlisted: Async[Either[Nothing, Long]] =
  order.runFoldAsync(0L) { (sum, line) =>
    val item = if (line.item == "Flat White") "Flat White XL" else line.item
    MenuPrices.priceInCents(item).map(sum + _)
  }
```

To see the defect as an `Either` and as a thrown exception:

```scala
println(unlisted.either.block)
// Left(java.util.NoSuchElementException: No price for Flat White XL)
println(scala.util.Try(unlisted.block))
// Failure(java.util.NoSuchElementException: No price for Flat White XL)
```

`Async#either` moves the failure into an `Either[Throwable, ...]` value, and `Async#block` rethrows it, so one `try` or `Try` at the edge covers every defect.

Cancelling a running terminal stops the pulls and runs the stream's finalizers. To see it, we drain an endless stream of lookups with a finalizer, start it, and cancel it after 350 ms:

```scala

val pulls     = new AtomicInteger
val finalized = new AtomicInteger

val endless: Stream[Nothing, Long] =
  Stream
    .fromIterator(Iterator.continually(MenuPrices.order.head))
    .mapAsync { line =>
      pulls.incrementAndGet()
      MenuPrices.priceInCents(line.item)
    }
    .ensuring { finalized.incrementAndGet(); () }

val running = endless.runDrainAsync.start
Thread.sleep(350)
running.cancel()
val pullsAtCancel = pulls.get()
Thread.sleep(300)
println(s"pulls at cancel: $pullsAtCancel, pulls 300 ms later: ${pulls.get()}, finalizer ran: ${finalized.get()}")
```

Running it prints:

```text
pulls at cancel: 4, pulls 300 ms later: 4, finalizer ran: 1
```

No pull starts after the cancel, and the finalizer runs exactly once. One thing the cancel does not do is reach inside the lookup that was in flight: a lookup built from a `Future`, `CompletionStage` or JavaScript promise keeps running after you stop waiting for it. [How to Wrap Callback and Future-Based APIs in Async](./wrap-callback-and-future-apis.md#cancel-the-underlying-call-when-the-async-is-cancelled) explains why and how to carry a cancel handle through.

## Running the Same Pipeline on Scala.js

Everything above used only the `*Async` terminals, `Async#map` and `Async.fromCompletionStage`, so only one piece is JVM-specific: the price lookup. The pipeline itself, `OrderTotalJs.totalInCents`, does not change if we pass the lookup in as a parameter. The Scala.js code in the rest of this guide is not compiled by the docs build; each snippet runs on Node 24 with Scala.js 1.22 on Scala 2.13.18 and 3.9.0.

On Scala.js, the lookup wraps a `js.Promise` with `Async.fromJsPromise`:

```scala
// Scala.js code

object OrderTotalJs {
  private val cents: Map[String, Long] = Map(
    "Espresso"   -> 350L,
    "Flat White" -> 480L,
    "Croissant"  -> 420L
  )

  def priceInCents(item: String): Async[Long] =
    Async.fromJsPromise(new js.Promise[Long]((resolve, reject) => {
      js.timers.setTimeout(100) {
        cents.get(item) match {
          case Some(price) => resolve(price)
          case None        => reject(new NoSuchElementException(s"No price for $item"))
        }
      }
      ()
    }))

  def totalInCents(items: Stream[Nothing, String], price: String => Async[Long]): Async[Either[Nothing, Long]] =
    items.runFoldAsync(0L)((sum, item) => price(item).map(sum + _))

  def main(args: Array[String]): Unit = {
    val items = Stream("Espresso", "Flat White", "Croissant")

    totalInCents(items, priceInCents).map(total => println(s"The total is $total")).toJsPromise

    println("main has returned; the total prints when the timers fire")
  }
}
```

Running it under Node prints:

```text
main has returned; the total prints when the timers fire
The total is Right(1250)
```

`main` returns before the first timer fires, which is the point: nothing waits. `Async#toJsPromise` hands the `Async` to the JavaScript event loop, so Node stays alive until the promise settles. In a browser, you would keep the `Async` or the `js.Promise` and update the page in the callback.

Two things differ from the JVM, and the compiler or the runtime tells you at once. A blocking terminal does not exist on Scala.js, so the call fails to compile:

```text
value runCollect is not a member of zio.blocks.streams.Stream[Nothing, Int]
```

`Async#block` does exist in shared source, so it cannot be removed. It returns a value that is already complete and throws when it would have to wait:

```text
A ready value: 1
A pending value: Async.block: suspension did not complete synchronously and JavaScript cannot block. Drive the Pollable from a non-blocking entry point instead.
```

## Reading a ReadableStream as a Source

In a browser or Node 18 and later, a `fetch` response body is a `ReadableStream` of bytes. `ReadableStreamReaders.fromReadableStream` wraps one as a `Reader.AsyncReader[Byte]`, and `Stream.fromReader` turns the reader into a `Stream[E, Byte]`. The adapter is Scala.js-only and byte-only: every chunk the JavaScript stream delivers is a `Uint8Array`, and the stream you get has element type `Byte`. It calls `read()` on the JavaScript stream only when the zio-blocks stream pulls for more bytes, and has at most one read in flight, so a slow consumer slows the reads rather than piling up chunks.

To fetch an order list, one item per line, and total its prices with the same `OrderTotalJs.totalInCents`, we first read the bytes and then decode them. The URL is a `data:` URL, so the example needs no network. Node and the browsers fetch it like any other:

```scala
// Scala.js code

object OrderListJs {
  private val orderUrl =
    "data:text/plain,Espresso%0AFlat%20White%0ACroissant%0A"

  def itemsIn(bytes: Chunk[Byte]): Stream[Nothing, String] =
    Stream.fromIterator(new String(bytes.toArray, "UTF-8").split("\n").iterator.filter(_.nonEmpty))

  def main(args: Array[String]): Unit = {
    val response = js.Dynamic.global.fetch(orderUrl).asInstanceOf[js.Promise[js.Dynamic]]

    val total: Async[Either[Nothing, Long]] =
      Async.fromJsPromise(response).flatMap { resp =>
        val body = resp.body.asInstanceOf[ReadableStreamReaders.ReadableStream]
        val bytes: Stream[Nothing, Byte] =
          Stream.fromReader[Nothing, Byte](ReadableStreamReaders.fromReadableStream(body))

        bytes.runCollectAsync.flatMap {
          case Right(all) => OrderTotalJs.totalInCents(itemsIn(all), OrderTotalJs.priceInCents)
          case Left(e)    => Async.succeed(Left(e))
        }
      }

    total.map(result => println(s"The order list costs $result")).toJsPromise
    ()
  }
}
```

```text
The order list costs Right(1250)
```

`ReadableStreamReaders` declares its own small `@js.native` facades, `ReadableStreamReaders.ReadableStream` among them, so we cast the `body` of the untyped `fetch` result to that facade. The factory takes the stream and calls `getReader()` itself.

`ReadableStreamReaders.fromReadableStream` owns the stream. When the zio-blocks stream closes early, for example under `take(3)`, the adapter cancels the JavaScript stream and releases its lock. To see it, we give the stream a `cancel` hook and take three of its five bytes:

```scala
// Scala.js code

object EarlyCloseJs {
  def main(args: Array[String]): Unit = {
    val cancelled = new AtomicBoolean(false)

    val source = js.Dynamic.literal(
      start = { (controller: js.Dynamic) =>
        controller.enqueue(new Uint8Array(js.Array[Short](1, 2, 3, 4, 5)))
      }: js.Function1[js.Dynamic, Unit],
      cancel = { (reason: js.Any) =>
        cancelled.set(true)
      }: js.Function1[js.Any, Unit]
    )
    val stream = js.Dynamic
      .newInstance(js.Dynamic.global.ReadableStream)(source)
      .asInstanceOf[ReadableStreamReaders.ReadableStream]

    Stream
      .fromReader[Nothing, Byte](ReadableStreamReaders.fromReadableStream(stream))
      .take(3)
      .runCollectAsync
      .map(result => println(s"take(3) gave $result, stream cancelled: ${cancelled.get()}"))
      .toJsPromise
    ()
  }
}
```

Running it prints:

```text
take(3) gave Right(Chunk(1,2,3)), stream cancelled: true
```

If the stream is not yours to cancel, use `ReadableStreamReaders.fromReadableStreamUnmanaged`, which releases the lock and never cancels, so the code that creates the stream can keep using it. [Reader](../reference/streams/primitives/reader.md#from-native-asynchronous-sources) documents both factories.

:::warning
In 0.0.56, a read that the JavaScript stream rejects, for example a dropped connection, does not arrive as `Left` in the `Either` the way the JVM channel adapter's failures do. It fails the `Async` itself with an internal `StreamError`, whose message is empty, so handle it with `Async#either` or a `catch` at the edge. This is a known bug in 0.0.56:

```text
The read failed: Left(zio.blocks.streams.internal.StreamError)
```

That line is the result of `runCollectAsync.either` on a stream whose second read is rejected with `new js.Error("network reset")`.
:::

## Putting It Together

The complete example puts the pipeline behind a `PriceLookup` function, so the same `receipt` could take the Scala.js lookup. It uses the `OrderLine` case class and the `MenuPrices` price service from the support file `MenuPrices.scala`, which Running the Examples embeds first. It prints a line per order line with `runForeachAsync`, totals the order with `runFoldAsync`, joins the two with `Async#flatMap`, and blocks once, at the end of `main`. A second run adds an item the menu does not list and looks at the failure through `Async#either`. The complete example, embedded from its companion file, follows:

```scala title="streams-examples/src/main/scala/runstreamswithoutblocking/CompleteExample.scala"
package runstreamswithoutblocking

/**
 * How to Run Streams Without Blocking — Putting It Together
 *
 * One pipeline written against a price lookup that returns an `Async`: it
 * prints a line per order line and totals the order without ever parking a
 * thread. `main` blocks once, at the very end.
 *
 * Run with: cd streams-examples && sbt "runMain
 * runstreamswithoutblocking.CompleteExample"
 */
object CompleteExample {
  type PriceLookup = String => Async[Long]

  def receipt(order: List[OrderLine], price: PriceLookup): Async[Either[Nothing, Long]] = {
    val lines: Stream[Nothing, OrderLine] = Stream.fromIterator(order.iterator)

    val printed: Async[Either[Nothing, Unit]] =
      lines.runForeachAsync { line =>
        price(line.item).map(cents => println(s"${line.item} for table ${line.table}: $cents cents"))
      }

    val total: Async[Either[Nothing, Long]] =
      lines.runFoldAsync(0L)((sum, line) => price(line.item).map(sum + _))

    printed.flatMap(_ => total)
  }

  def main(args: Array[String]): Unit = {
    val ok: Async[Either[Nothing, Long]] = receipt(MenuPrices.order, MenuPrices.priceInCents)
    println(s"Total for the order: ${ok.block} cents")

    // An item the menu does not list fails the Async itself, so `either` is the way to look at it.
    val unlisted = OrderLine("Cold Brew", 9)
    val failing  = receipt(MenuPrices.order :+ unlisted, MenuPrices.priceInCents)
    println(s"With an unlisted item: ${failing.either.block}")
  }
}
```

## Running the Examples

Every JVM snippet in this guide has a standalone companion file in the `streams-examples` build. Clone the repository and move into that build:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/streams-examples
```

Every example uses the `OrderLine` case class and the `MenuPrices` price service:

<details>
<summary>The Price Service</summary>

The price service and the `OrderLine` case class:

```scala title="streams-examples/src/main/scala/runstreamswithoutblocking/MenuPrices.scala" showLineNumbers
package runstreamswithoutblocking

final case class OrderLine(item: String, table: Int)

/**
 * A fictional price service for a coffee shop menu. `priceInCents` answers
 * after a 100 ms delay on another thread, the way a remote call would, and
 * fails for an item the menu does not list. Prices are sample data.
 */
object MenuPrices {
  val order: List[OrderLine] = List(
    OrderLine("Espresso", 4),
    OrderLine("Flat White", 7),
    OrderLine("Croissant", 4)
  )

  private val cents: Map[String, Long] = Map(
    "Espresso"   -> 350L,
    "Flat White" -> 480L,
    "Croissant"  -> 420L
  )

  def priceInCents(item: String): Async[Long] =
    Async.fromCompletionStage(
      CompletableFuture.supplyAsync(
        () => cents.getOrElse(item, throw new NoSuchElementException(s"No price for $item")),
        CompletableFuture.delayedExecutor(100, MILLISECONDS)
      )
    )
}
```

</details>

<details>
<summary>Swap a Blocking Terminal for Its Async Twin</summary>

The terminal swap, timed against the blocking call:

```scala title="streams-examples/src/main/scala/runstreamswithoutblocking/Concept1SwapTerminalExample.scala" showLineNumbers
package runstreamswithoutblocking

/**
 * How to Run Streams Without Blocking — Swap a Blocking Terminal for Its Async
 * Twin
 *
 * Runs one stream of price lookups with `runCollect` and with
 * `runCollectAsync`, and shows when each call returns.
 *
 * Run with: cd streams-examples && sbt "runMain
 * runstreamswithoutblocking.Concept1SwapTerminalExample"
 */
object Concept1SwapTerminalExample {
  def main(args: Array[String]): Unit = {
    val prices: Stream[Nothing, Long] =
      Stream.fromIterator(MenuPrices.order.iterator).mapAsync(line => MenuPrices.priceInCents(line.item))

    val blockedAt = System.nanoTime()
    val blocking  = prices.runCollect
    println(s"runCollect returned after ${elapsedMillis(blockedAt)} ms with $blocking")

    val startedAt                                    = System.nanoTime()
    val pending: Async[Either[Nothing, Chunk[Long]]] = prices.runCollectAsync
    println(s"runCollectAsync returned after ${elapsedMillis(startedAt)} ms, nothing has run yet")
    println(s"the result is ${pending.block}")
  }

  private def elapsedMillis(since: Long): Long = (System.nanoTime() - since) / 1000000
}
```

Run it with:

```bash
sbt "runMain runstreamswithoutblocking.Concept1SwapTerminalExample"
```

</details>

<details>
<summary>Do Async Work for Each Element</summary>

Per-element async work with `runForeachAsync` and `runFoldAsync`:

```scala title="streams-examples/src/main/scala/runstreamswithoutblocking/Concept2PerElementExample.scala" showLineNumbers
package runstreamswithoutblocking

/**
 * How to Run Streams Without Blocking — Do Async Work for Each Element
 *
 * Totals the price of every line of an order with `runFoldAsync`, and prints
 * each price as it arrives with `runForeachAsync`. The step function returns an
 * `Async`, so no thread waits for a price.
 *
 * Run with: cd streams-examples && sbt "runMain
 * runstreamswithoutblocking.Concept2PerElementExample"
 */
object Concept2PerElementExample {
  def main(args: Array[String]): Unit = {
    val order: Stream[Nothing, OrderLine] = Stream.fromIterator(MenuPrices.order.iterator)

    val printed: Async[Either[Nothing, Unit]] =
      order.runForeachAsync { line =>
        MenuPrices.priceInCents(line.item).map(cents => println(s"${line.item} for table ${line.table}: $cents cents"))
      }

    val total: Async[Either[Nothing, Long]] =
      order.runFoldAsync(0L)((sum, line) => MenuPrices.priceInCents(line.item).map(sum + _))

    println(s"printed: ${printed.block}")
    println(s"total: ${total.block} cents")
  }
}
```

Run it with:

```bash
sbt "runMain runstreamswithoutblocking.Concept2PerElementExample"
```

</details>

<details>
<summary>Use the Result Without Waiting</summary>

Composing the result and handing it to a `Future`:

```scala title="streams-examples/src/main/scala/runstreamswithoutblocking/Concept3UsingResultExample.scala" showLineNumbers
package runstreamswithoutblocking

/**
 * How to Run Streams Without Blocking — Use the Result Without Waiting
 *
 * Composes the result of a terminal with `map`, hands it to code that expects a
 * `Future`, and blocks only at the end of `main`.
 *
 * Run with: cd streams-examples && sbt "runMain
 * runstreamswithoutblocking.Concept3UsingResultExample"
 */
object Concept3UsingResultExample {
  def totalInCents(items: Stream[Nothing, String]): Async[Either[Nothing, Long]] =
    items.runFoldAsync(0L)((sum, item) => MenuPrices.priceInCents(item).map(sum + _))

  def main(args: Array[String]): Unit = {
    val items: Stream[Nothing, String] = Stream.fromIterator(MenuPrices.order.iterator.map(_.item))

    // `map` on the Async: no thread waits, the function runs when the total is ready.
    val label: Async[Either[Nothing, String]] =
      totalInCents(items).map(_.map(cents => f"The order costs ${cents / 100.0}%.2f"))

    // Hand the same Async to code that wants a Future.
    val future = label.toFuture
    println(s"Future completed yet: ${future.isCompleted}")

    // The one blocking call: the end of main, which has nothing left to do.
    println(Await.result(future, 5.seconds))
    println(label.block)
  }
}
```

Run it with:

```bash
sbt "runMain runstreamswithoutblocking.Concept3UsingResultExample"
```

</details>

<details>
<summary>Errors and Cancellation</summary>

Typed errors, defects and cancellation:

```scala title="streams-examples/src/main/scala/runstreamswithoutblocking/Concept4ErrorsAndCancellationExample.scala" showLineNumbers
package runstreamswithoutblocking

/**
 * How to Run Streams Without Blocking — Errors and Cancellation
 *
 * Shows where a typed error, a defect and a cancel end up, and that cancelling
 * a running terminal stops the pulls and runs the stream's finalizers.
 *
 * Run with: cd streams-examples && sbt "runMain
 * runstreamswithoutblocking.Concept4ErrorsAndCancellationExample"
 */
object Concept4ErrorsAndCancellationExample {
  def main(args: Array[String]): Unit = {
    // A typed error lands in the Either.
    val typed: Stream[String, Long] = Stream.fail("the price service is down")
    println(s"typed error: ${typed.runFoldAsync(0L)((sum, cents) => Async.succeed(sum + cents)).block}")

    // A defect lands in the Async's own failure channel.
    val lines: Stream[Nothing, OrderLine] = Stream.fromIterator(MenuPrices.order.iterator)
    val unlisted                          = lines.runFoldAsync(0L) { (sum, line) =>
      MenuPrices.priceInCents(if (line.item == "Flat White") "Flat White XL" else line.item).map(sum + _)
    }
    println(s"defect, as an Either: ${unlisted.either.block}")
    println(s"defect, through block: ${scala.util.Try(unlisted.block)}")

    // Cancel a long run: the pulls stop and the finalizer runs.
    val pulls                          = new AtomicInteger
    val finalized                      = new AtomicInteger
    val endless: Stream[Nothing, Long] =
      Stream
        .fromIterator(Iterator.continually(MenuPrices.order.head))
        .mapAsync { line =>
          pulls.incrementAndGet()
          MenuPrices.priceInCents(line.item)
        }
        .ensuring { finalized.incrementAndGet(); () }

    val running = endless.runDrainAsync.start
    Thread.sleep(350)
    running.cancel()
    val pullsAtCancel = pulls.get()
    Thread.sleep(300)
    println(s"pulls at cancel: $pullsAtCancel, pulls 300 ms later: ${pulls.get()}, finalizer ran: ${finalized.get()}")
  }
}
```

Run it with:

```bash
sbt "runMain runstreamswithoutblocking.Concept4ErrorsAndCancellationExample"
```

</details>

<details>
<summary>Complete Example: An Order Receipt</summary>

The complete order receipt:

```scala title="streams-examples/src/main/scala/runstreamswithoutblocking/CompleteExample.scala" showLineNumbers
package runstreamswithoutblocking

/**
 * How to Run Streams Without Blocking — Putting It Together
 *
 * One pipeline written against a price lookup that returns an `Async`: it
 * prints a line per order line and totals the order without ever parking a
 * thread. `main` blocks once, at the very end.
 *
 * Run with: cd streams-examples && sbt "runMain
 * runstreamswithoutblocking.CompleteExample"
 */
object CompleteExample {
  type PriceLookup = String => Async[Long]

  def receipt(order: List[OrderLine], price: PriceLookup): Async[Either[Nothing, Long]] = {
    val lines: Stream[Nothing, OrderLine] = Stream.fromIterator(order.iterator)

    val printed: Async[Either[Nothing, Unit]] =
      lines.runForeachAsync { line =>
        price(line.item).map(cents => println(s"${line.item} for table ${line.table}: $cents cents"))
      }

    val total: Async[Either[Nothing, Long]] =
      lines.runFoldAsync(0L)((sum, line) => price(line.item).map(sum + _))

    printed.flatMap(_ => total)
  }

  def main(args: Array[String]): Unit = {
    val ok: Async[Either[Nothing, Long]] = receipt(MenuPrices.order, MenuPrices.priceInCents)
    println(s"Total for the order: ${ok.block} cents")

    // An item the menu does not list fails the Async itself, so `either` is the way to look at it.
    val unlisted = OrderLine("Cold Brew", 9)
    val failing  = receipt(MenuPrices.order :+ unlisted, MenuPrices.priceInCents)
    println(s"With an unlisted item: ${failing.either.block}")
  }
}
```

Run it with:

```bash
sbt "runMain runstreamswithoutblocking.CompleteExample"
```

</details>

To check that every example compiles without running any of them, run:

```bash
sbt compile
```

The `streams-examples` build is JVM-only, so the Scala.js snippets have no companion files. To run one, make a scratch sbt project with `addSbtPlugin("org.scala-js" % "sbt-scalajs" % "1.22.0")` in `project/plugins.sbt` and this `build.sbt`, copy a snippet into `src/main/scala`, and run it with `sbt run`, which needs Node on your path:

```scala
scalaVersion := "3.9.0"
enablePlugins(ScalaJSPlugin)
scalaJSUseMainModuleInitializer := true
scalacOptions += "-experimental"
libraryDependencies += "dev.zio" %%% "zio-blocks-streams" % "0.0.56"
```

## Going Further

- [Asynchronous Stream Execution](../reference/streams/execution-and-compatibility/async-execution.md) lists every async constructor, operator and terminal, and [Platform Differences](../reference/streams/execution-and-compatibility/platform-differences.md) lists what exists on each platform.
- [How to Wrap Callback and Future-Based APIs in Async](./wrap-callback-and-future-apis.md) brings `Future`, `CompletionStage`, promise and callback clients into `Async`, with cancellation.
- [How to Stream Files and Sockets Without Leaking Resources](./stream-files-and-sockets.md) covers the JVM sources that do block, and how they close their handles.
- [Reader](../reference/streams/primitives/reader.md#from-native-asynchronous-sources) documents the JVM NIO and Scala.js `ReadableStream` adapters.

---

## How to Send Server-Sent Events with http-model

In this guide, we build a "new arrivals" feed for a Scala bookshop that pushes one book at a time to every open browser tab, using `ServerSentEvent` and `Body.fromStream` from the ZIO Blocks [HTTP model](../reference/http-model/index.md). By the end, you will have a `text/event-stream` endpoint that streams each event the moment it is produced, resumes where a reconnecting client left off, and notices when a client goes away, plus a small HTML page that shows the books as they arrive.

We start from one event on the wire, turn a stream of events into a `Response`, serve it through a streaming version of the adapter from [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md), and then add event ids, resuming, a browser client, and disconnect handling.

## The Problem

With Server-Sent Events (SSE) the server keeps one connection open and writes small text messages made of `event:`, `id:`, `retry:` and `data:` lines, which the browser's built-in `EventSource` reads and reconnects on by itself. The wire format looks simple enough to write by hand, but it is line-based, and every line break in a value changes its meaning. A server also has to send each event as soon as it exists; a body that is collected in full before it is sent never reaches the client while the feed is still running.

For example, a first attempt at formatting an arrival might look like this:

```scala
def formatArrival(id: String, title: String, authors: String): String =
  s"event: arrival\nid: $id\ndata: $title\n$authors\n\n"
```

For the book `Zionomicon`, the authors line has no `data:` prefix, so `EventSource` reads `John A. De Goes, Adam Fraser and Milad Khajavi` as the name of an unknown field and drops it: the browser receives only the title. An id with a line break in it would inject a field of its own. Even with the text right, the [JDK server adapter](./serve-http-model-requests.md) collects the whole body with `Body#toArray` before it sends anything, so a client of a feed that never ends receives nothing at all, not even the status line.

In this guide, we let `ServerSentEvent` produce the wire text, stream it with `Body.fromStream`, and change the adapter so it writes each event as soon as it is rendered.

## Prerequisites

Add the HTTP model to your build. It brings the streams module with it:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-http-model" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-http-model" % "0.0.56"
scalacOptions += "-experimental"
```

The code in this guide is the same on Scala 2.13 and Scala 3. The HTTP model depends on the ZIO Blocks streams module, which is compiled for Java 21, so run the program on **JDK 21 or newer**. The server parts use `com.sun.net.httpserver`, so they are JVM only.

You should have read [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md), because we reuse its adapter. For streams, [Your First Streaming Pipeline](./your-first-streaming-pipeline.md) covers creating and running a `Stream`.

We use these imports throughout the guide:

```scala

```

## The New-Arrivals Feed

The bookshop announces six Scala books. Each book becomes one event named `arrival`, whose id is the book's position in the list and whose data has two lines, the title and the authors:

```scala
final case class Book(title: String, authors: String)

val books: Vector[Book] = Vector(
  Book("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi"),
  Book("Programming in Scala", "Martin Odersky, Lex Spoon, Bill Venners and Frank Sommers"),
  Book("Functional Programming in Scala", "Michael Pilquist, Rúnar Bjarnason and Paul Chiusano"),
  Book("Scala with Cats", "Noel Welsh and Dave Gurnell"),
  Book("Hands-on Scala Programming", "Li Haoyi"),
  Book("Essential Effects", "Adam Rosien")
)

def arrival(position: Int, book: Book): ServerSentEvent[String] =
  ServerSentEvent(s"${book.title}\n${book.authors}", "arrival").id(position.toString)

def arrivalsAfter(lastSeen: Int): Vector[ServerSentEvent[String]] =
  books.zipWithIndex.drop(lastSeen).map { case (book, index) => arrival(index + 1, book) }
```

`arrivalsAfter(n)` returns the events for every book after the first `n`. We use it later to resume a feed.

## Rendering One Event

`ServerSentEvent[A]` is an immutable value: the data plus the optional `event`, `id` and `retry` fields. `ServerSentEvent#render` turns it into the exact text that goes on the wire, ending with the blank line that marks the end of an event. The simplest event has only data:

```scala
ServerSentEvent("Zionomicon").render == "data: Zionomicon\n\n"
// res1: Boolean = true
```

`ServerSentEvent.apply` takes an optional event name, and `ServerSentEvent#id` and `ServerSentEvent#retry` add the other fields. They are always rendered in the order `event`, `id`, `retry`, `data`:

```scala
ServerSentEvent("Zionomicon", "arrival").id("1").retry(2000).render ==
  "event: arrival\nid: 1\nretry: 2000\ndata: Zionomicon\n\n"
// res2: Boolean = true
```

Data is rendered through an implicit `SseDataEncoder[A]`. The encoder for `String` splits the text at every `\n`, `\r` or `\r\n` and writes each piece on its own `data:` line, which is exactly what our two-line arrival needs:

```scala
println(arrival(1, books.head).render)
// event: arrival
// id: 1
// data: Zionomicon
// data: John A. De Goes, Adam Fraser and Milad Khajavi
// 
//
```

`EventSource` joins the `data:` lines back together with `\n`, so the browser sees the title and the authors separated by a line break. The other fields must fit on one line, and `ServerSentEvent#event` and `ServerSentEvent#id` throw an `IllegalArgumentException` for a value that contains a line break:

```scala
val rejected = scala.util.Try(ServerSentEvent("Zionomicon").id("1\n2"))
// rejected: Try[ServerSentEvent[String]] = Failure(
//   java.lang.IllegalArgumentException: SSE id must not contain CR or LF characters
// )
```

`ServerSentEvent#retry` likewise rejects a negative number. To send a payload of your own type, such as JSON, write an `SseDataEncoder` instance that returns its lines; the [Server-Sent Events reference](../reference/http-model/server-sent-event.md) shows how.

## Turning a Stream of Events into a Response

An SSE response is an ordinary `Response`: status `200 OK`, the content type `text/event-stream`, and a body that is a stream of rendered events. The media type registry has no constant for `text/event-stream`, so we build the `ContentType` from a `MediaType`:

```scala
val eventStream: ContentType = ContentType(MediaType("text", "event-stream"))
```

`Body.fromStream` takes a `Stream[Nothing, Byte]`, a byte stream that cannot fail, and a content type. We render each event and encode its text as UTF-8, which is the only encoding SSE allows. `Cache-Control: no-cache` keeps proxies and the browser from caching the feed:

```scala
def encode(events: Stream[Nothing, ServerSentEvent[String]]): Stream[Nothing, Byte] =
  events.flatMap(event => Stream.fromChunk(Chunk.fromArray(event.render.getBytes(StandardCharsets.UTF_8))))

def eventResponse(events: Stream[Nothing, ServerSentEvent[String]]): Response =
  Response.ok(Body.fromStream(encode(events), eventStream)).addHeader(Header.CacheControl.NoCache)
```

`Response.ok` sets the `content-type` header from the body. Nothing runs yet: a `Stream` is a description, and the events are produced only when the server pulls the body. To see the result without a server, we build a response for the first two books and read its body back in the same program:

```scala
val firstTwo = eventResponse(Stream.fromIterable(arrivalsAfter(0).take(2)))
```

Reading the body runs the stream to the end, which is fine for a finite stream:

```scala
println(s"status: ${firstTwo.status.code}")
// status: 200
firstTwo.headers.toList.foreach { case (name, value) => println(s"$name: $value") }
// content-type: text/event-stream
// cache-control: no-cache
println(s"known body length: ${firstTwo.body.length}")
// known body length: None
print(firstTwo.body.asString())
// event: arrival
// id: 1
// data: Zionomicon
// data: John A. De Goes, Adam Fraser and Milad Khajavi
// 
// event: arrival
// id: 2
// data: Programming in Scala
// data: Martin Odersky, Lex Spoon, Bill Venners and Frank Sommers
//
```

The body length is `None` because the body is built with `Stream#flatMap`, so its length is not known until the stream has been read. That is why an SSE response is sent with chunked transfer encoding instead of a `Content-Length` header.

## Sending One Book at a Time

A real feed produces events over time. The streams module has no built-in timer, so the simplest honest way to pace a feed is to block before each element with `Stream#tapEach`:

```scala
def paced(events: Vector[ServerSentEvent[String]], intervalMillis: Long): Stream[Nothing, ServerSentEvent[String]] =
  Stream.fromIterable(events).tapEach(_ => Thread.sleep(intervalMillis))
```

`Thread.sleep` runs on the thread that pulls the stream, which is the server thread writing the response. Every open feed therefore holds one thread for as long as it stays open. A feed that never ends is the same shape with an endless source, such as `Stream.fromIterator(Iterator.continually(...))`. In a real application, the source would be a queue that new arrivals are put into, and pulling the next element would block on that queue instead of sleeping.

## Streaming the Response Through the JDK Server

The `JdkHttp` adapter cannot serve this response. `JdkHttp.writeResponse` calls `Body#toArray`, which pulls the whole stream before it writes a single byte, and the JDK server without an executor runs every exchange on one dispatcher thread, so an open feed blocks every other request. Served through that adapter, an endless feed sends its client no bytes at all, not even the status line; a request to another path waits until the feed ends; and the server keeps producing events after the client has left, because it never writes and so never learns that the connection is gone.

The streaming adapter, `StreamingJdkHttp`, reuses `JdkHttp.toRequest` and `JdkHttp.writeResponse` from the original adapter and changes three things for `text/event-stream` responses. The first change writes the body as the stream produces it. A length of `0` passed to `HttpExchange#sendResponseHeaders` tells the JDK server that the length is unknown, so it sends the headers at once and uses chunked transfer encoding. `Stream#run` with `Sink.fromOutputStream` then pulls the stream and writes each byte as it arrives:

```scala

final class FlushPerEvent(out: OutputStream) extends FilterOutputStream(out) {
  private var previous = -1

  override def write(b: Int): Unit = {
    out.write(b)
    if (b == '\n' && previous == '\n') out.flush()
    previous = b
  }
}

def writeEventStream(exchange: HttpExchange, response: Response): Unit = {
  val out = exchange.getResponseHeaders
  response.headers.toList.foreach { case (name, value) =>
    if (name != "content-length" && name != "transfer-encoding") out.add(name, value)
  }
  if (!response.headers.has("content-type")) out.set("content-type", response.body.contentType.render)
  exchange.sendResponseHeaders(response.status.code, 0L)
  response.body.stream.run(Sink.fromOutputStream(new FlushPerEvent(exchange.getResponseBody)))
  ()
}
```

The second change is `FlushPerEvent`. The JDK's response stream buffers what we write, so without a flush the first events would sit in the buffer. `FlushPerEvent` flushes at every blank line, the end of an event as `ServerSentEvent#render` writes it, so each event leaves in its own chunk. The third change gives the server an executor, so each open feed runs on its own thread and other requests are not blocked. The threads are daemon threads, so an open feed never keeps the JVM alive after `main` returns:

```scala

val daemonThreads: ThreadFactory = runnable => {
  val thread = new Thread(runnable, "sse-exchange")
  thread.setDaemon(true)
  thread
}

def useThreadPerExchange(server: HttpServer): Unit =
  server.setExecutor(Executors.newCachedThreadPool(daemonThreads))
```

The handler sends `text/event-stream` responses through `writeEventStream` and every other response through `JdkHttp.writeResponse`, and treats an `IOException` while writing as the client having gone away. Any other failure while writing is logged, and becomes a 500 when no headers have gone out yet, the same as in the serving guide's adapter. The whole adapter is about a hundred lines; it is embedded under [Running the Examples](#running-the-examples).

The live-feed example serves three books, one per second, first through `JdkHttp` and then through the streaming adapter, and its client prints when each event arrives. Its output, from one run:

```text
== JdkHttp (collects the body first)
GET /arrivals -> 200 [content-type: text/event-stream, cache-control: no-cache, transfer-encoding: none]
event: arrival
id: 1
data: Zionomicon
data: John A. De Goes, Adam Fraser and Milad Khajavi
-- event 1 received after 4.0 s
event: arrival
id: 2
data: Programming in Scala
data: Martin Odersky, Lex Spoon, Bill Venners and Frank Sommers
-- event 2 received after 4.0 s
event: arrival
id: 3
data: Functional Programming in Scala
data: Michael Pilquist, Rúnar Bjarnason and Paul Chiusano
-- event 3 received after 4.0 s
== Streaming adapter (StreamingJdkHttp)
GET /arrivals -> 200 [content-type: text/event-stream, cache-control: no-cache, transfer-encoding: chunked]
event: arrival
id: 1
data: Zionomicon
data: John A. De Goes, Adam Fraser and Milad Khajavi
-- event 1 received after 1.0 s
event: arrival
id: 2
data: Programming in Scala
data: Martin Odersky, Lex Spoon, Bill Venners and Frank Sommers
-- event 2 received after 2.0 s
event: arrival
id: 3
data: Functional Programming in Scala
data: Michael Pilquist, Rúnar Bjarnason and Paul Chiusano
-- event 3 received after 3.0 s
```

Through `JdkHttp` all three events arrive together, after the stream has ended. Through the streaming adapter they arrive one second apart.

To watch the feed from a terminal, start the complete example (see [Putting It Together](#putting-it-together)) and open the feed with `curl -N`, which turns off curl's output buffering. The events appear one every two seconds:

```bash
curl -N -i http://localhost:8080/arrivals
```

The first lines it prints:

```text
HTTP/1.1 200 OK
Date: Fri, 23 Oct 2026 19:28:05 GMT
Transfer-encoding: chunked
Content-type: text/event-stream
Cache-control: no-cache

event: arrival
id: 1
retry: 2000
data: Zionomicon
data: John A. De Goes, Adam Fraser and Milad Khajavi

event: arrival
id: 2
data: Programming in Scala
data: Martin Odersky, Lex Spoon, Bill Venners and Frank Sommers
```

The JDK server capitalizes only the first letter of each header name; HTTP header names are case-insensitive. The `retry:` line comes from the next section.

## Naming, Numbering and Resuming the Feed

The three metadata fields are what make a feed survive a dropped connection. The event name selects the listener on the client: an event with an `event:` field is delivered only to listeners for that name, not to `EventSource#onmessage`. The id is remembered by the client, and when `EventSource` reconnects it sends the last id it saw in a `Last-Event-ID` request header. The `retry:` field tells the client how many milliseconds to wait before it reconnects.

http-model has no typed codec for `Last-Event-ID`, so we read it with `Headers#rawGet`, which matches header names case-insensitively. The feed then streams only the books after that id, asks the client to wait two seconds before reconnecting, and answers `204 No Content` when the client already has every book, which tells `EventSource` to stop reconnecting:

```scala
def lastSeen(request: Request): Int =
  request.headers.rawGet("last-event-id").flatMap(_.toIntOption).getOrElse(0)

def arrivals(request: Request, intervalMillis: Long): Response = {
  val remaining = arrivalsAfter(lastSeen(request))
  if (remaining.isEmpty) Response(Status.NoContent)
  else eventResponse(paced(remaining.updated(0, remaining.head.retry(2000)), intervalMillis))
}
```

The `204` matters for a finite feed. When the stream ends, the server closes the connection, and `EventSource` treats that like any other dropped connection: it reconnects. Without the `204`, a feed that replays from the start would send the same books again on every reconnect.

Because `arrivals` is a plain function, we can call it with a request that carries the header, without a server:

```scala
def call(lastEventId: String): Unit = {
  val request  = Request.get(URL.fromPath(Path("/arrivals"))).addHeader("last-event-id", lastEventId)
  val response = arrivals(request, intervalMillis = 0)
  println(s"status: ${response.status.code}")
  print(response.body.asString())
}

call("4")
// status: 200
// event: arrival
// id: 5
// retry: 2000
// data: Hands-on Scala Programming
// data: Li Haoyi
// 
// event: arrival
// id: 6
// data: Essential Effects
// data: Adam Rosien
// 
call("6")
// status: 204
```

With `curl`, pass the header yourself to see the same resume against the running server:

```bash
curl -N -H 'Last-Event-ID: 4' http://localhost:8080/arrivals
```

It prints the last two books and exits when the server closes the stream:

```text
event: arrival
id: 5
retry: 2000
data: Hands-on Scala Programming
data: Li Haoyi

event: arrival
id: 6
data: Essential Effects
data: Adam Rosien
```

## Watching the Feed in a Browser

`EventSource` is built into every current browser, so the client needs no JavaScript library. The page opens the feed, adds a list item for each `arrival` event, and shows the connection state. We keep it as a Scala string so the server can send it:

```scala
val page: String =
  """<!DOCTYPE html>
    |<html>
    |<head><meta charset="utf-8"><title>New arrivals</title></head>
    |<body>
    |New arrivals
    |Connecting...
    |
    |<script>
    |  const status = document.getElementById("status");
    |  const source = new EventSource("/arrivals");
    |  source.addEventListener("arrival", (event) => {
    |    const [title, authors] = event.data.split("\n");
    |    const item = document.createElement("li");
    |    item.textContent = title + " by " + authors;
    |    document.getElementById("books").append(item);
    |    status.textContent = "Last event id: " + event.lastEventId;
    |  });
    |  source.onerror = () => {
    |    status.textContent = source.readyState === EventSource.CLOSED ? "All books have arrived." : "Reconnecting...";
    |  };
    |</script>
    |</body>
    |</html>
    |""".stripMargin
```

`event.data` is the two `data:` lines joined with `\n`, and `event.lastEventId` is the `id:` field. `EventSource#onerror` fires both when the connection drops and when the browser gives up; `EventSource#readyState` tells the two apart. The application serves the page at `/` and the feed at `/arrivals`. `Body.fromString` would label the page `text/plain`, so we build the body from bytes with ``ContentType.`text/html` ``:

```scala
def app(intervalMillis: Long): Request => Response = request =>
  (request.method, request.path.encode) match {
    case (Method.GET, "/") =>
      Response.ok(Body.fromArray(page.getBytes(StandardCharsets.UTF_8), ContentType.`text/html`))
    case (Method.GET, "/arrivals") => arrivals(request, intervalMillis)
    case _                         => Response.notFound
  }
```

Serving the page from the same server as the feed keeps the request same-origin; a feed on another origin needs CORS headers. Opened in Chromium against the complete example, the page lists the six books two seconds apart. When the stream ends, the browser reconnects once with `Last-Event-ID: 6`, receives the `204`, and the status line changes to "All books have arrived.". The server log of that visit:

```text
stream 5: opened after event 0
stream 5: closed
stream 6: the client has every book, 204
```

## Noticing When a Client Leaves

A client closes an event stream by closing the connection: the user closes the tab, or the page calls `EventSource#close`. The server finds out only when it next writes, and the write fails with an `IOException`. `Stream#run` then stops pulling the stream and closes it, so a finalizer registered with `Stream#ensuring` runs, and the streaming adapter's handler catches the `IOException` and closes the exchange. To see when that happens, the disconnect example counts the events it produces and logs when the stream closes:

```scala

def countedFeed(request: Request): Response = {
  val produced = new AtomicInteger(0)
  val events   = paced(arrivalsAfter(0), intervalMillis = 1000)
    .tapEach(_ => produced.incrementAndGet())
    .ensuring(println(s"server: stream closed after producing ${produced.get} events"))
  eventResponse(events)
}
```

Its client reads two events and disconnects. The output:

```text
GET /arrivals -> 200 [content-type: text/event-stream, cache-control: no-cache, transfer-encoding: chunked]
event: arrival
id: 1
data: Zionomicon
data: John A. De Goes, Adam Fraser and Milad Khajavi
-- event 1 received after 2.2 s
event: arrival
id: 2
data: Programming in Scala
data: Martin Odersky, Lex Spoon, Bill Venners and Frank Sommers
-- event 2 received after 3.2 s
client: disconnected
server: stream closed after producing 4 events
```

The server produces four events, not three. The first write after the client leaves still succeeds, because the operating system accepts the bytes before the client's reset arrives; the second write fails. So the server notices a departed client within about two intervals, and until then the feed keeps its thread. A feed that can stay quiet for minutes should send a small event, such as one named `ping`, at a fixed interval, so a departed client is noticed in bounded time.

:::tip
Each open feed holds one thread. The cached thread pool in the streaming adapter creates a platform thread per open feed; on JDK 21 or newer, a virtual thread per exchange makes a blocked feed cheap:

```scala
server.setExecutor(Executors.newVirtualThreadPerTaskExecutor())
```
:::

## Putting It Together

The complete example serves the page at `/` and the resumable feed at `/arrivals` on port 8080, sends one book every two seconds, and logs when each stream opens and closes. It builds on the shared files embedded under Running the Examples below: the feed model `Bookshop.scala`, the streaming adapter `StreamingJdkHttp.scala`, and the page from `Step5BrowserPageExample.scala`. The complete example:

```scala title="http-model-examples/src/main/scala/sendserversentevents/CompleteExample.scala"
package sendserversentevents

/**
 * How to Send Server-Sent Events with http-model — Complete Example
 *
 * A new-arrivals feed for a Scala bookshop: an HTML page at / that listens with
 * `EventSource`, and an event stream at /arrivals that sends one book every two
 * seconds, resumes from `Last-Event-ID`, answers 204 once the client has every
 * book, and logs when each stream opens and closes. Watch it with
 * `curl -N http://localhost:8080/arrivals` or open http://localhost:8080/ in a
 * browser.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * sendserversentevents.CompleteExample"
 */
object CompleteExample {
  private val opened = new AtomicInteger(0)

  def arrivals(request: Request): Response = {
    val number    = opened.incrementAndGet()
    val lastSeen  = request.headers.rawGet("last-event-id").flatMap(_.toIntOption).getOrElse(0)
    val remaining = arrivalsAfter(lastSeen)
    if (remaining.isEmpty) {
      println(s"stream $number: the client has every book, 204")
      Response(Status.NoContent)
    } else {
      println(s"stream $number: opened after event $lastSeen")
      val events = paced(remaining.updated(0, remaining.head.retry(2000)), intervalMillis = 2000)
        .ensuring(println(s"stream $number: closed"))
      eventResponse(events)
    }
  }

  val app: Request => Response = request =>
    (request.method, request.path.encode) match {
      case (Method.GET, "/") =>
        Response.ok(
          Body.fromArray(Step5BrowserPageExample.page.getBytes(StandardCharsets.UTF_8), ContentType.`text/html`)
        )
      case (Method.GET, "/arrivals") => arrivals(request)
      case _                         => Response.notFound
    }

  def main(args: Array[String]): Unit = {
    val port = args.headOption.flatMap(_.toIntOption).getOrElse(8080)
    StreamingJdkHttp.start(app, port)
    println(s"Serving http://localhost:$port/ and http://localhost:$port/arrivals (Ctrl+C to stop)")
    Thread.currentThread().join()
  }
}
```

While it runs, open `http://localhost:8080/` in a browser or watch the feed with `curl -N http://localhost:8080/arrivals`. A full `curl` run, a resumed one with `Last-Event-ID: 4`, a finished one with `Last-Event-ID: 6`, and a `curl` stopped after three seconds produce this log:

```text
Serving http://localhost:8080/ and http://localhost:8080/arrivals (Ctrl+C to stop)
stream 1: opened after event 0
stream 1: closed
stream 2: opened after event 4
stream 2: closed
stream 3: the client has every book, 204
stream 4: opened after event 0
stream 4: closed
```

## Running the Examples

All code from this guide is available as runnable examples in the `http-model-examples` directory. It is a standalone sbt build that uses the published ZIO Blocks artifacts. To run them, clone the repository and move into that directory:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/http-model-examples
```

The examples share three files of their own, the feed model, the streaming adapter and a test client, and use the adapter `JdkHttp.scala` from the serving guide, which lives in the same build:

<details>
  <summary>http-model-examples/src/main/scala/sendserversentevents/Bookshop.scala</summary>

```scala title="http-model-examples/src/main/scala/sendserversentevents/Bookshop.scala" showLineNumbers
package sendserversentevents

/**
 * How to Send Server-Sent Events with http-model — the new-arrivals feed
 *
 * Six Scala books that a bookshop announces one at a time, the event for each
 * arrival, the `text/event-stream` content type, and the helpers that turn a
 * stream of events into a streaming `Response`.
 */
final case class Book(title: String, authors: String)

object Bookshop {
  val books: Vector[Book] = Vector(
    Book("Zionomicon", "John A. De Goes, Adam Fraser and Milad Khajavi"),
    Book("Programming in Scala", "Martin Odersky, Lex Spoon, Bill Venners and Frank Sommers"),
    Book("Functional Programming in Scala", "Michael Pilquist, Rúnar Bjarnason and Paul Chiusano"),
    Book("Scala with Cats", "Noel Welsh and Dave Gurnell"),
    Book("Hands-on Scala Programming", "Li Haoyi"),
    Book("Essential Effects", "Adam Rosien")
  )

  /** The event for the book at `position` (1-based) in the arrivals list. */
  def arrival(position: Int, book: Book): ServerSentEvent[String] =
    ServerSentEvent(s"${book.title}\n${book.authors}", "arrival").id(position.toString)

  /** Every arrival after the first `lastSeen` books. */
  def arrivalsAfter(lastSeen: Int): Vector[ServerSentEvent[String]] =
    books.zipWithIndex.drop(lastSeen).map { case (book, index) => arrival(index + 1, book) }

  /** Emits the events one at a time, waiting `intervalMillis` before each. */
  def paced(events: Vector[ServerSentEvent[String]], intervalMillis: Long): Stream[Nothing, ServerSentEvent[String]] =
    Stream.fromIterable(events).tapEach(_ => Thread.sleep(intervalMillis))

  /** `text/event-stream`; the media type registry has no constant for it. */
  val eventStream: ContentType = ContentType(MediaType("text", "event-stream"))

  /** Renders each event to its wire text and encodes it as UTF-8 bytes. */
  def encode(events: Stream[Nothing, ServerSentEvent[String]]): Stream[Nothing, Byte] =
    events.flatMap(event => Stream.fromChunk(Chunk.fromArray(event.render.getBytes(StandardCharsets.UTF_8))))

  /** A `200 OK` whose body streams the events as they are produced. */
  def eventResponse(events: Stream[Nothing, ServerSentEvent[String]]): Response =
    Response.ok(Body.fromStream(encode(events), eventStream)).addHeader(Header.CacheControl.NoCache)
}
```

</details>

<details>
  <summary>http-model-examples/src/main/scala/sendserversentevents/StreamingJdkHttp.scala</summary>

```scala title="http-model-examples/src/main/scala/sendserversentevents/StreamingJdkHttp.scala" showLineNumbers
package sendserversentevents

/**
 * How to Send Server-Sent Events with http-model — the streaming adapter
 *
 * The adapter from How to Serve http-model Requests with the JDK HTTP Server
 * (`servehttpmodelrequests.JdkHttp`) with two changes for event streams: a
 * `text/event-stream` body is written as it is produced, with chunked transfer
 * encoding and a flush after every event, instead of being collected with
 * `Body#toArray`; and each exchange runs on its own thread, so one open stream
 * does not block every other request. Other responses still go through
 * `JdkHttp.writeResponse`.
 */
object StreamingJdkHttp {

  /** Flushes the underlying stream at every blank line, the end of an event. */
  final class FlushPerEvent(out: OutputStream) extends FilterOutputStream(out) {
    private var previous = -1

    override def write(b: Int): Unit = {
      out.write(b)
      if (b == '\n' && previous == '\n') out.flush()
      previous = b
    }
  }

  def isEventStream(response: Response): Boolean =
    response.body.contentType.mediaType.fullType == "text/event-stream"

  /**
   * Sends the status and headers at once, then pulls the body stream and writes
   * each event as soon as it is rendered. A response length of 0 tells the JDK
   * server that the length is unknown, so it uses chunked transfer encoding.
   * Throws `IOException` when the client has gone away.
   */
  def writeEventStream(exchange: HttpExchange, response: Response): Unit = {
    val out = exchange.getResponseHeaders
    response.headers.toList.foreach { case (name, value) =>
      if (name != "content-length" && name != "transfer-encoding") out.add(name, value)
    }
    if (!response.headers.has("content-type")) out.set("content-type", response.body.contentType.render)
    exchange.sendResponseHeaders(response.status.code, 0L)
    response.body.stream.run(Sink.fromOutputStream(new FlushPerEvent(exchange.getResponseBody)))
    ()
  }

  /**
   * Like `JdkHttp.handler`, but sends event streams with `writeEventStream`. An
   * `IOException` while writing means the client closed the connection, which
   * is how every event stream ends from the client's side.
   */
  def handler(app: Request => Response): HttpHandler = exchange =>
    try {
      val response =
        try JdkHttp.toRequest(exchange).fold(identity, app)
        catch {
          case error @ (NonFatal(_) | _: LinkageError) =>
            System.err.println(s"Handler failed: $error")
            Response.text(Status.InternalServerError, "Internal Server Error")
        }
      try
        if (isEventStream(response)) writeEventStream(exchange, response)
        else JdkHttp.writeResponse(exchange, response)
      catch {
        case _: IOException                          => ()
        case error @ (NonFatal(_) | _: LinkageError) =>
          System.err.println(s"Writing the response failed: $error")
          if (exchange.getResponseCode == -1) {
            exchange.getResponseHeaders.clear()
            exchange.sendResponseHeaders(Status.InternalServerError.code, -1L)
          }
      }
    } finally exchange.close()

  private val daemonThreads: ThreadFactory = runnable => {
    val thread = new Thread(runnable, "sse-exchange")
    thread.setDaemon(true)
    thread
  }

  /**
   * Starts a server on the loopback interface with one thread per open
   * exchange. Port 0 asks the operating system for a free port.
   */
  def start(app: Request => Response, port: Int = 0): HttpServer = {
    val server = HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress, port), 0)
    server.createContext("/", handler(app))
    server.setExecutor(Executors.newCachedThreadPool(daemonThreads))
    server.start()
    server
  }

  /**
   * Starts a server on a free port, passes the port to `use`, then stops it.
   */
  def withServer[A](app: Request => Response)(use: Int => A): A = {
    val server = start(app)
    try use(server.getAddress.getPort)
    finally server.stop(0)
  }
}
```

</details>

<details>
  <summary>http-model-examples/src/main/scala/sendserversentevents/FeedClient.scala</summary>

```scala title="http-model-examples/src/main/scala/sendserversentevents/FeedClient.scala" showLineNumbers
package sendserversentevents

/**
 * How to Send Server-Sent Events with http-model — a test client
 *
 * Opens an event stream with `HttpURLConnection`, prints the status and the
 * headers that matter for SSE, then prints each line as it arrives. After each
 * event it prints how long after the request the event arrived.
 */
object FeedClient {
  def watch(port: Int, target: String, lastEventId: Option[String] = None, maxEvents: Int = Int.MaxValue): Unit = {
    val connection = URI.create(s"http://127.0.0.1:$port$target").toURL.openConnection().asInstanceOf[HttpURLConnection]
    connection.setRequestProperty("accept", "text/event-stream")
    lastEventId.foreach(id => connection.setRequestProperty("last-event-id", id))
    val started = System.nanoTime()
    try {
      val status                       = connection.getResponseCode
      def header(name: String): String = Option(connection.getHeaderField(name)).getOrElse("none")
      println(
        s"GET $target -> $status [content-type: ${header("content-type")}, cache-control: ${header("cache-control")}, " +
          s"transfer-encoding: ${header("transfer-encoding")}]"
      )
      if (status == 200) {
        val reader = new BufferedReader(new InputStreamReader(connection.getInputStream, StandardCharsets.UTF_8))
        var events = 0
        var line   = reader.readLine()
        while (line != null && events < maxEvents) {
          if (line.isEmpty) {
            events += 1
            println(f"-- event $events received after ${(System.nanoTime() - started) / 1e9}%.1f s")
          } else println(line)
          if (events < maxEvents) line = reader.readLine()
        }
      }
    } finally connection.disconnect()
  }
}
```

</details>

The first example prints the wire text of a few events, with every line break shown as `\n`, and the message for a rejected id:

<details>
  <summary>http-model-examples/src/main/scala/sendserversentevents/Step1RenderEventExample.scala</summary>

```scala title="http-model-examples/src/main/scala/sendserversentevents/Step1RenderEventExample.scala" showLineNumbers
package sendserversentevents

/**
 * How to Send Server-Sent Events with http-model — Rendering One Event
 *
 * Builds `ServerSentEvent` values and prints the exact text each one puts on
 * the wire, with every line break shown as `\n`, including a two-line payload,
 * then shows that an `id` containing a line break is rejected.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * sendserversentevents.Step1RenderEventExample"
 */
object Step1RenderEventExample {
  def show(event: ServerSentEvent[String]): Unit =
    println(event.render.replace("\n", "\\n"))

  def main(args: Array[String]): Unit = {
    show(ServerSentEvent("Zionomicon"))
    show(ServerSentEvent("Zionomicon", "arrival").id("1").retry(2000))
    show(arrival(1, books.head))
    try ServerSentEvent("Zionomicon").id("1\n2")
    catch { case error: IllegalArgumentException => println(s"rejected: ${error.getMessage}") }
  }
}
```

</details>

To run it:

```bash
sbt "runMain sendserversentevents.Step1RenderEventExample"
```

The second example builds the response for two books and reads it back without a server:

<details>
  <summary>http-model-examples/src/main/scala/sendserversentevents/Step2EventResponseExample.scala</summary>

```scala title="http-model-examples/src/main/scala/sendserversentevents/Step2EventResponseExample.scala" showLineNumbers
package sendserversentevents

/**
 * How to Send Server-Sent Events with http-model — Returning a Stream of Events
 *
 * Turns the six arrivals into a `text/event-stream` `Response` and reads it
 * back in the same program, without a server: the status, the headers, and the
 * body text.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * sendserversentevents.Step2EventResponseExample"
 */
object Step2EventResponseExample {
  def main(args: Array[String]): Unit = {
    val response = eventResponse(paced(arrivalsAfter(0).take(2), intervalMillis = 0))
    println(s"status: ${response.status.code}")
    response.headers.toList.foreach { case (name, value) => println(s"$name: $value") }
    println(s"known body length: ${response.body.length}")
    print(response.body.asString())
  }
}
```

</details>

To run it:

```bash
sbt "runMain sendserversentevents.Step2EventResponseExample"
```

The third example serves the paced feed through both adapters and prints when each event arrives:

<details>
  <summary>http-model-examples/src/main/scala/sendserversentevents/Step3LiveFeedExample.scala</summary>

```scala title="http-model-examples/src/main/scala/sendserversentevents/Step3LiveFeedExample.scala" showLineNumbers
package sendserversentevents

/**
 * How to Send Server-Sent Events with http-model — Streaming the Feed as It
 * Happens
 *
 * Serves one book per second, first through `JdkHttp`, the adapter from How to
 * Serve http-model Requests with the JDK HTTP Server, which collects the whole
 * body before sending it, then through the streaming adapter, which sends each
 * event as soon as it is produced. The client prints when each event arrived.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * sendserversentevents.Step3LiveFeedExample"
 */
object Step3LiveFeedExample {
  val app: Request => Response =
    _ => eventResponse(paced(arrivalsAfter(0).take(3), intervalMillis = 1000))

  def main(args: Array[String]): Unit = {
    println("== JdkHttp (collects the body first)")
    JdkHttp.withServer(app)(port => FeedClient.watch(port, "/arrivals"))
    println("== Streaming adapter (StreamingJdkHttp)")
    StreamingJdkHttp.withServer(app)(port => FeedClient.watch(port, "/arrivals"))
  }
}
```

</details>

To run it:

```bash
sbt "runMain sendserversentevents.Step3LiveFeedExample"
```

The fourth example resumes the feed from `Last-Event-ID: 4`, then asks again with `Last-Event-ID: 6` and gets `204`:

<details>
  <summary>http-model-examples/src/main/scala/sendserversentevents/Step4ResumeExample.scala</summary>

```scala title="http-model-examples/src/main/scala/sendserversentevents/Step4ResumeExample.scala" showLineNumbers
package sendserversentevents

/**
 * How to Send Server-Sent Events with http-model — Resuming from Last-Event-ID
 *
 * Reads the `Last-Event-ID` header that a reconnecting client sends, streams
 * only the books after that id, asks the client to wait two seconds before it
 * reconnects, and answers `204 No Content` once the client has every book.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * sendserversentevents.Step4ResumeExample"
 */
object Step4ResumeExample {
  def lastSeen(request: Request): Int =
    request.headers.rawGet("last-event-id").flatMap(_.toIntOption).getOrElse(0)

  def arrivals(request: Request, intervalMillis: Long): Response = {
    val remaining = arrivalsAfter(lastSeen(request))
    if (remaining.isEmpty) Response(Status.NoContent)
    else eventResponse(paced(remaining.updated(0, remaining.head.retry(2000)), intervalMillis))
  }

  def main(args: Array[String]): Unit =
    StreamingJdkHttp.withServer(arrivals(_, intervalMillis = 200)) { port =>
      FeedClient.watch(port, "/arrivals", lastEventId = Some("4"))
      FeedClient.watch(port, "/arrivals", lastEventId = Some("6"))
    }
}
```

</details>

To run it:

```bash
sbt "runMain sendserversentevents.Step4ResumeExample"
```

The fifth example serves the browser page and the feed on port 8080 until you stop it with Ctrl+C; pass another port as the first argument if 8080 is taken:

<details>
  <summary>http-model-examples/src/main/scala/sendserversentevents/Step5BrowserPageExample.scala</summary>

```scala title="http-model-examples/src/main/scala/sendserversentevents/Step5BrowserPageExample.scala" showLineNumbers
package sendserversentevents

/**
 * How to Send Server-Sent Events with http-model — Watching the Feed in a
 * Browser
 *
 * Serves a small HTML page at / whose script opens the feed at /arrivals with
 * the browser's built-in `EventSource` and lists each book as it arrives. The
 * server keeps running until you stop it with Ctrl+C.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * sendserversentevents.Step5BrowserPageExample" and open http://localhost:8080/
 */
object Step5BrowserPageExample {
  val page: String =
    """<!DOCTYPE html>
      |<html>
      |<head><meta charset="utf-8"><title>New arrivals</title></head>
      |<body>
      |New arrivals
      |Connecting...
      |
      |<script>
      |  const status = document.getElementById("status");
      |  const source = new EventSource("/arrivals");
      |  source.addEventListener("arrival", (event) => {
      |    const [title, authors] = event.data.split("\n");
      |    const item = document.createElement("li");
      |    item.textContent = title + " by " + authors;
      |    document.getElementById("books").append(item);
      |    status.textContent = "Last event id: " + event.lastEventId;
      |  });
      |  source.onerror = () => {
      |    status.textContent = source.readyState === EventSource.CLOSED ? "All books have arrived." : "Reconnecting...";
      |  };
      |</script>
      |</body>
      |</html>
      |""".stripMargin

  def app(intervalMillis: Long): Request => Response = request =>
    (request.method, request.path.encode) match {
      case (Method.GET, "/") =>
        Response.ok(Body.fromArray(page.getBytes(StandardCharsets.UTF_8), ContentType.`text/html`))
      case (Method.GET, "/arrivals") => Step4ResumeExample.arrivals(request, intervalMillis)
      case _                         => Response.notFound
    }

  def main(args: Array[String]): Unit = {
    val port = args.headOption.flatMap(_.toIntOption).getOrElse(8080)
    StreamingJdkHttp.start(app(intervalMillis = 1000), port)
    println(s"Open http://localhost:$port/ (Ctrl+C to stop)")
    Thread.currentThread().join()
  }
}
```

</details>

To run it, then open `http://localhost:8080/`:

```bash
sbt "runMain sendserversentevents.Step5BrowserPageExample"
```

The sixth example disconnects after two events and shows when the server notices:

<details>
  <summary>http-model-examples/src/main/scala/sendserversentevents/Step6DisconnectExample.scala</summary>

```scala title="http-model-examples/src/main/scala/sendserversentevents/Step6DisconnectExample.scala" showLineNumbers
package sendserversentevents

/**
 * How to Send Server-Sent Events with http-model — Noticing When a Client
 * Leaves
 *
 * Counts the events the server produces and logs, with `Stream#ensuring`, when
 * the stream closes. The client reads two events and disconnects. The first
 * write after that still succeeds; the server notices on the second one, about
 * two seconds later.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * sendserversentevents.Step6DisconnectExample"
 */
object Step6DisconnectExample {
  def app(request: Request): Response = {
    val produced = new AtomicInteger(0)
    val events   = paced(arrivalsAfter(0), intervalMillis = 1000)
      .tapEach(_ => produced.incrementAndGet())
      .ensuring(println(s"server: stream closed after producing ${produced.get} events"))
    eventResponse(events)
  }

  def main(args: Array[String]): Unit =
    StreamingJdkHttp.withServer(app) { port =>
      FeedClient.watch(port, "/arrivals", maxEvents = 2)
      println("client: disconnected")
      Thread.sleep(3500)
    }
}
```

</details>

To run it:

```bash
sbt "runMain sendserversentevents.Step6DisconnectExample"
```

To run the complete example, which also serves on port 8080 until you stop it:

```bash
sbt "runMain sendserversentevents.CompleteExample"
```

To check that all of the examples compile without running any of them:

```bash
sbt compile
```

## Going Further

The feed in this guide is deliberately small. These are the next things you are likely to need:

- **Typed payloads**: An `SseDataEncoder` for your own type can render JSON into the `data:` lines; see the [Server-Sent Events reference](../reference/http-model/server-sent-event.md).
- **A real event source**: Replace the paced `Vector` with a stream that blocks on a queue, so events are sent when they happen rather than on a timer.
- **Reading streams**: [How to Stream Files and Sockets Without Leaking Resources](./stream-files-and-sockets.md) covers `Stream#ensuring` and resource safety in more depth.
- **Production traffic**: The JDK server is fine for tools and examples. For production, hand the same `Response` values to a full HTTP server library that streams bodies natively.

Datastar, the hypermedia framework, also uses `text/event-stream` responses on the server side, so the techniques in this guide carry over to it. For the types used here, see the [Server-Sent Events reference](../reference/http-model/server-sent-event.md), the [HTTP model reference](../reference/http-model/model.md) and the [headers reference](../reference/http-model/headers.md).

---

## How to Serve http-model Requests with the JDK HTTP Server

In this guide, we serve a small bookstore over HTTP using the ZIO Blocks [HTTP model](../reference/http-model/index.md) and the HTTP server that ships with the JDK (`com.sun.net.httpserver.HttpServer`). By the end, you will have an adapter of about sixty lines that turns any `Request => Response` function into a running server, and a bookstore that answers with an HTML catalog, JSON lists, single books, validated JSON input and clean 404s.

We take an incremental approach: first we write the adapter and serve a one-line handler, then we add routing, query parameters, a JSON request body and an HTML page. Every section starts a real server on a free port, calls it with the JDK's `java.net.http.HttpClient` and prints what came back.

:::warning
This adapter is glue for examples, tools, tests and small internal services. It is not a production server: ZIO Blocks ships no HTTP server, and the JDK server offers no HTTP/2, no request-body size limit and little tuning. For production traffic, hand your `Request` and `Response` values to a full HTTP server library.
:::

## The Problem

ZIO Blocks has an HTTP *data model* (`Request`, `Response`, `Body`, `Headers`, `URL`, `Method`, `Status`, `ContentType`), but no server that reads requests from a socket. If you build pages or APIs on that model, nothing serves them yet.

The JDK has had a small HTTP server since Java 6, but its `HttpExchange` is a low-level API: the raw request URI, a mutable header map, an `InputStream` and an `OutputStream`. Writing handlers directly against it means every handler repeats the same fragile plumbing.

For example, consider a handler for `GET /books?limit=2` written directly against the JDK API:

```scala

val booksHandler: HttpHandler = (exchange: HttpExchange) => {
  val query = Option(exchange.getRequestURI.getRawQuery).getOrElse("")
  // Hand-rolled query parsing: no percent-decoding, no repeated keys
  val limit = query.split('&').collectFirst { case s"limit=$value" => value.toInt }.getOrElse(3)
  val json  = List("Zionomicon", "Programming in Scala", "Functional Programming in Scala")
    .take(limit)
    .map(title => "\"" + title + "\"") // Hand-written JSON: breaks on a quote in a title
    .mkString("[", ",", "]")
  val bytes = json.getBytes(StandardCharsets.UTF_8)
  exchange.getResponseHeaders.set("Content-Type", "application/json")
  exchange.sendResponseHeaders(200, bytes.length.toLong) // Wrong for HEAD and for empty bodies
  exchange.getResponseBody.write(bytes)
  exchange.close() // Never reached if anything above throws, e.g. `?limit=abc`
}
```

This approach mixes protocol details with application logic. A bad `limit` throws before `HttpExchange#close`, so the client hangs instead of getting a 400. The handler can't be tested without a socket, and the next handler copies the same lines with new bugs.

In this guide, we solve this by writing the plumbing once: one adapter converts an `HttpExchange` to a `Request` and a `Response` back to the exchange. Everything else is an ordinary, socket-free `Request => Response` function.

## Prerequisites

Add the HTTP model to your `build.sbt`, plus the HTML and schema modules that the later sections use for the catalog page and the JSON codecs:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-http-model" % "0.0.56",
  "dev.zio" %% "zio-blocks-html"       % "0.0.56",
  "dev.zio" %% "zio-blocks-schema"     % "0.0.56"
)

// Scala 3.9 and newer
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-http-model" % "0.0.56",
  "dev.zio" %% "zio-blocks-next-html"       % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema"     % "0.0.56"
)
scalacOptions += "-experimental"
```

The code in this guide is the same for every Scala version. It needs only the JDK for the server and the client, with no extra dependency. The HTTP model depends on the ZIO Blocks streams module, which is compiled for Java 21, so run the program on **JDK 21 or newer**. This guide is JVM only: the HTTP model also builds for Scala.js, but `com.sun.net.httpserver` does not exist there.

You should know the HTTP model types from the [HTTP model reference](../reference/http-model/model.md). The JSON section uses a derived `Schema`, introduced in [Your First Schema](./your-first-schema.md), and the HTML section uses the DSL from [Render Your First Server-Side HTML Page](./your-first-html-page.md).

We use these imports throughout the guide:

```scala

```

## Turning an HttpExchange into a Request

The JDK hands each incoming request to an `HttpHandler` as an `HttpExchange`. Our first job is to read the four parts of a `Request` out of it: the method, the URL, the headers and the body. Two of them can be invalid, so the function returns `Either[Response, Request]`: a `Left` holds the error response to send instead of calling the application:

```scala
def toRequest(exchange: HttpExchange): Either[Response, Request] = {
  val uri    = exchange.getRequestURI
  val target = uri.getRawPath + Option(uri.getRawQuery).fold("")("?" + _)
  for {
    method <- Method.fromString(exchange.getRequestMethod).toRight(Response(Status.NotImplemented))
    url    <- URL.parse(target).left.map(error => Response.text(Status.BadRequest, error))
  } yield {
    val headers = exchange.getRequestHeaders.asScala.foldLeft(Headers.empty) { case (acc, (name, values)) =>
      values.asScala.foldLeft(acc)((withValue, value) => withValue.add(name, value))
    }
    val contentType =
      headers.get(Header.ContentType).map(_.value).getOrElse(ContentType.`application/octet-stream`)
    val body    = Body.fromArray(exchange.getRequestBody.readAllBytes(), contentType)
    val version = Version.fromString(exchange.getProtocol).getOrElse(Version.`HTTP/1.1`)
    Request(method, url, headers, body, version)
  }
}
```

Each part maps onto an http-model constructor:

- **Method**: `Method.fromString` returns `None` for a method the model doesn't know, such as WebDAV's `PROPFIND`. `Method` is a closed set, so we answer 501 Not Implemented.
- **URL**: We pass the *raw* path and query (`URI#getRawPath`, `URI#getRawQuery`) to `URL.parse`, which percent-decodes every path segment and query value itself. The decoded `URI#getPath` would decode `%2F` into a `/` before the path is split into segments. `URL.parse` returns `Left` with a message instead of throwing, which becomes a 400.
- **Headers**: `Headers#add` takes each name and value. Header names in `Headers` are case-insensitive and stored in lower case.
- **Body**: `InputStream#readAllBytes` reads the whole body into memory, and `Body.fromArray` wraps it with the request's `Content-Type`. A body built from an array has a known length, so reading it later with `Body#toArray` or `Body#asString` never blocks.
- **Version**: `Version.fromString` parses `HTTP/1.1` and similar protocol strings.

:::note
`InputStream#readAllBytes` puts no limit on the size of the body. That is fine for a local tool, but it is one of the reasons this adapter is not meant for untrusted traffic.
:::

## Writing a Response to the Exchange

Going the other way, we copy the status, the headers and the body of a `Response` onto the exchange. The JDK server writes `Content-Length` itself from the length we pass to `HttpExchange#sendResponseHeaders`, so we skip any `content-length` and `transfer-encoding` header the response carries:

```scala
def writeResponse(exchange: HttpExchange, response: Response): Unit = {
  val out = exchange.getResponseHeaders
  response.headers.toList.foreach { case (name, value) =>
    if (name != "content-length" && name != "transfer-encoding") out.add(name, value)
  }
  val bytes = response.body.toArray
  if (bytes.nonEmpty && !response.headers.has("content-type"))
    out.set("content-type", response.body.contentType.render)
  val code   = response.status.code
  val isHead = exchange.getRequestMethod == "HEAD"
  if (isHead && bytes.nonEmpty) out.set("content-length", bytes.length.toString)
  val noBody = isHead || code == 204 || code == 304 || bytes.isEmpty
  exchange.sendResponseHeaders(code, if (noBody) -1L else bytes.length.toLong)
  if (!noBody) exchange.getResponseBody.write(bytes)
}
```

The length argument of `HttpExchange#sendResponseHeaders` has three meanings, and getting it wrong is the most common bug with this server:

| Length given  | What the JDK does                                    | When we use it                                  |
| ------------- | ---------------------------------------------------- | ----------------------------------------------- |
| `n > 0`       | Sends `Content-Length: n`; we must write `n` bytes   | A response with a body                          |
| `0`           | Uses chunked transfer encoding of unknown length     | Never: we always know the length                |
| `-1`          | Sends no body                                        | Empty bodies, `HEAD`, 204 No Content, 304       |

A few more details matter here:

- **Content type**: `Response.text`, `Response.json` and `Response.ok(body)` set the `content-type` header for us. A `Response` built with its case-class constructor has none, so we fall back to `Body#contentType`.
- **HEAD**: A `HEAD` response has no body but should report the length the `GET` body would have, so we set `content-length` ourselves and pass `-1`.
- **Streaming bodies**: `Body#toArray` collects the whole body before the status line is sent. For a body built with `Body.fromStream` from a stream of unknown length, this call blocks the server thread until the stream ends. A body that never ends never reaches the client, not even its headers, and its producer never stops: the adapter writes nothing, so it never learns that the client has left. [How to Send Server-Sent Events with http-model](./send-server-sent-events.md) shows a write path that streams.
- **Server-Sent Events**: A response whose body never ends needs a different write path. [How to Send Server-Sent Events with http-model](./send-server-sent-events.md) streams `ServerSentEvent` values through a copy of this adapter that writes each event as it is produced and runs each exchange on its own thread.

## Wrapping a Handler Function

Now we can turn any `Request => Response` function into a JDK `HttpHandler`. The handler must close the exchange on every path: if it doesn't, the client waits forever. It also turns a non-fatal exception or a `LinkageError` thrown by the application into a 500 response, so one bad request can't leave a connection hanging. Writing the response can fail too, and that failure gets the same treatment, as the paragraph after the code explains:

```scala
def handler(app: Request => Response): HttpHandler = exchange =>
  try {
    val response =
      try
        toRequest(exchange) match {
          case Left(error)                                     => error
          case Right(request) if request.method == Method.HEAD => app(request.copy(method = Method.GET))
          case Right(request)                                  => app(request)
        }
      catch {
        case error @ (NonFatal(_) | _: LinkageError) =>
          System.err.println(s"Handler failed: $error")
          Response.text(Status.InternalServerError, "Internal Server Error")
      }
    try writeResponse(exchange, response)
    catch {
      case error @ (NonFatal(_) | _: LinkageError) =>
        System.err.println(s"Writing the response failed: $error")
        if (exchange.getResponseCode == -1) {
          exchange.getResponseHeaders.clear()
          exchange.sendResponseHeaders(Status.InternalServerError.code, -1L)
        }
    }
  } finally exchange.close()
```

The handler runs a `HEAD` request as a `GET`, and `writeResponse` drops the body afterwards. That way every `GET` route also answers `HEAD` without writing it twice. The endpoint module's [`RouteTree`](../reference/endpoint/route-tree.md) uses the same fallback. The error message goes to standard error and the client only sees `Internal Server Error`, so exception details never leak to callers.

The pattern `NonFatal(_) | _: LinkageError` is deliberate. `NonFatal` leaves out `LinkageError`, the family of errors that includes `ExceptionInInitializerError` (an `object` whose initializer throws), `NoClassDefFoundError` and `UnsupportedClassVersionError` (a jar built for a newer JDK than the one running). A handler can hit these through no fault of the request, and if the adapter let them escape, the `finally` would close the connection with no response and nothing in the log: the client would only see an empty reply. Now the error is logged and the client gets a 500. Everything else that `NonFatal` leaves out still goes through: `VirtualMachineError` (such as `OutOfMemoryError` and `StackOverflowError`), `InterruptedException`, `ThreadDeath` and Scala's `ControlThrowable`. They mean the JVM is failing or someone asked the thread to stop, and turning them into a 500 would hide that.

One case stays out of reach: the 500 response is built with http-model classes, so if the broken class is one the adapter itself needs (the http-model or streams jar on a JDK that is too old), building the 500 fails with the same error. The log line still names the cause, but the client sees an empty reply.

The second `try` covers `writeResponse`. Writing can fail after the handler has returned, for example when a header value contains a character the JDK refuses, such as a `✓` in `HX-Trigger`. Without that `try`, the exception escaped to the JDK's dispatcher: the client got an empty reply, and nothing was logged. Now the failure is logged, and when no header has gone out yet the client gets a 500. If the response is already partly written, there is no status to change, so the adapter only logs it and the `finally` still closes the exchange.

## Starting and Stopping the Server

`HttpServer.create` binds a socket address, `HttpServer#createContext` attaches our handler to every path under `/`, and `HttpServer#start` starts accepting connections on a background thread. We bind to the loopback interface so that the server is reachable only from this machine. Port 0 asks the operating system for a free port, which we read back from `HttpServer#getAddress`:

```scala
def start(app: Request => Response, port: Int = 0): HttpServer = {
  val server = HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress, port), 0)
  server.createContext("/", handler(app))
  server.start()
  server
}
```

A server holds a socket and a thread, so it must be stopped when we are done with it. `HttpServer#stop` takes the number of seconds to wait for exchanges still in progress; 0 stops at once. The `withServer` helper starts a server, hands its port to a block of code, and stops it when the block returns or throws:

```scala
def withServer[A](app: Request => Response)(use: Int => A): A = {
  val server = start(app)
  try use(server.getAddress.getPort)
  finally server.stop(0)
}
```

`withServer` uses `try`/`finally` to stay free of dependencies. If your program already manages resources with [Scope](./compile-time-resource-safety-with-scope.md), acquire the server as a resource whose release calls `server.stop(0)` instead.

:::tip
Use `start(app, 8080)` to run on a fixed port and open `http://localhost:8080/` in a browser. Call `server.stop(0)` from a JVM shutdown hook to stop it on Ctrl+C.
:::

## Calling the Server from the Same Program

To see real responses, we call the server with `java.net.http.HttpClient`, which is also part of the JDK. The `call` helper sends one request, then prints the status code, the `Content-Type` and `Content-Length` headers, and the body. We pin the client to HTTP/1.1 because the JDK server does not speak HTTP/2:

```scala
val client = HttpClient.newBuilder().version(HttpClient.Version.HTTP_1_1).build()

def call(port: Int, method: String, target: String, body: String = ""): Unit = {
  val builder = HttpRequest.newBuilder(URI.create(s"http://127.0.0.1:$port$target"))
  val request =
    if (body.isEmpty) builder.method(method, HttpRequest.BodyPublishers.noBody()).build()
    else
      builder
        .method(method, HttpRequest.BodyPublishers.ofString(body))
        .header("content-type", "application/json")
        .build()
  val response = client.send(request, HttpResponse.BodyHandlers.ofString())
  def header(name: String): String = response.headers.firstValue(name).orElse("none")
  println(s"$method $target -> ${response.statusCode} [${header("content-type")}, ${header("content-length")} bytes]")
  if (response.body.nonEmpty) println(response.body)
}
```

## Serving a Hello World Handler

With the adapter in place, an application is a plain function. Our first one greets the caller with the method and path it received:

```scala
val hello: Request => Response =
  request => Response.text(s"Hello from ${request.method.name} ${request.path.encode}")
```

To call the handler once with `GET` and once with `HEAD`:

```scala
withServer(hello) { port =>
  call(port, "GET", "/books")
  call(port, "HEAD", "/books")
}
// GET /books -> 200 [text/plain; charset=UTF-8, 21 bytes]
// Hello from GET /books
// HEAD /books -> 200 [text/plain; charset=UTF-8, 21 bytes]
```

The `HEAD` request reports the same type and length as the `GET`, without a body, and the server is stopped by the time `withServer` returns.

To check the failure path, define an application that always throws:

```scala
val broken: Request => Response =
  _ => throw new IllegalStateException("the catalog is not loaded")
```

Serving it shows what the client receives when the application fails:

```scala
withServer(broken) { port =>
  call(port, "GET", "/books")
}
// GET /books -> 500 [text/plain; charset=UTF-8, 21 bytes]
// Internal Server Error
```

The client gets a complete 500 response instead of a hanging connection.

## The Bookstore Model

The rest of the guide serves a tiny bookstore. A `Book` has an id, a title, its authors and the year of publication, and its derived `Schema` gives us JSON codecs for one book and for a list of books:

```scala

final case class Book(id: Int, title: String, authors: List[String], year: Int)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

val books: List[Book] = List(
  Book(1, "Zionomicon", List("John A. De Goes", "Adam Fraser", "Milad Khajavi"), 2024),
  Book(2, "Programming in Scala", List("Martin Odersky", "Lex Spoon", "Bill Venners"), 2008),
  Book(3, "Functional Programming in Scala", List("Paul Chiusano", "Rúnar Bjarnason"), 2014)
)

val bookJson: JsonCodec[Book]           = Schema[Book].jsonCodec
val bookListJson: JsonCodec[List[Book]] = Schema[List[Book]].jsonCodec
```

We import `Schema` by name and call `JsonCodec#encodeToString` and `JsonCodec#decode` on the codecs directly, rather than importing `zio.blocks.schema._` for the `toJsonString` and `fromJson` extension methods. On Scala 3, the wildcard import together with `zio.blocks.html._` makes `p(...)` ambiguous, because the schema package also defines a `p"..."` interpolator.

## Routing a Few Paths

A router is a function from a request to a response, so the simplest one is a pattern match. `Request#path` returns a `Path` whose `segments` are already percent-decoded, so `/books/2` becomes `List("books", "2")`. We match on the method and the segments together, and a final case answers 404 for everything else:

```scala
def notFound(request: Request): Response =
  Response.text(Status.NotFound, s"No route for ${request.method.name} ${request.path.encode}")

def getBook(id: String): Response =
  books.find(_.id.toString == id) match {
    case Some(book) => Response.json(bookJson.encodeToString(book))
    case None       => Response.text(Status.NotFound, s"No book with id $id")
  }

def routes(request: Request): Response =
  (request.method, request.path.segments.toList) match {
    case (Method.GET, List("books"))     => Response.json(bookListJson.encodeToString(books))
    case (Method.GET, List("books", id)) => getBook(id)
    case _                               => notFound(request)
  }
```

`Response.json` sets the status to 200 and the content type to `application/json`. To call a known book, an unknown book and a method the router doesn't handle:

```scala
withServer(routes) { port =>
  call(port, "GET", "/books/1")
  call(port, "GET", "/books/42")
  call(port, "DELETE", "/books/1")
}
// GET /books/1 -> 200 [application/json, 101 bytes]
// {"id":1,"title":"Zionomicon","authors":["John A. De Goes","Adam Fraser","Milad Khajavi"],"year":2024}
// GET /books/42 -> 404 [text/plain; charset=UTF-8, 18 bytes]
// No book with id 42
// DELETE /books/1 -> 404 [text/plain; charset=UTF-8, 28 bytes]
// No route for DELETE /books/1
```

Pattern matching keeps every route visible in one place and needs no extra module. When the number of routes grows, the endpoint module's [`RouteTree`](../reference/endpoint/route-tree.md) looks up a handler by method and path in a prefix tree and also falls back from `HEAD` to `GET`.

## Reading Query Parameters

`Request#queryParams` returns the decoded query string. `QueryParams#getFirst` returns the first value of a key as an `Option[String]`, so a missing `limit` and a bad `limit` are two different cases. A bad value gets a 400 with a message that says what was wrong:

```scala
def listBooks(request: Request): Response =
  request.queryParams.getFirst("limit") match {
    case None      => Response.json(bookListJson.encodeToString(books))
    case Some(raw) =>
      raw.toIntOption.filter(_ >= 0) match {
        case Some(limit) => Response.json(bookListJson.encodeToString(books.take(limit)))
        case None        => Response.text(Status.BadRequest, s"limit must be a non-negative integer, got '$raw'")
      }
  }
```

To serve `listBooks` on its own and call it with a valid and an invalid limit:

```scala
withServer(listBooks) { port =>
  call(port, "GET", "/books?limit=1")
  call(port, "GET", "/books?limit=all")
}
// GET /books?limit=1 -> 200 [application/json, 103 bytes]
// [{"id":1,"title":"Zionomicon","authors":["John A. De Goes","Adam Fraser","Milad Khajavi"],"year":2024}]
// GET /books?limit=all -> 400 [text/plain; charset=UTF-8, 47 bytes]
// limit must be a non-negative integer, got 'all'
```

:::note
`QueryParams` decodes `%20` to a space but keeps `+` as a plus sign. Browsers encode a space as `+` when they submit an HTML form with `method="get"`, so a search for `zio blocks` arrives as `zio+blocks`. If your page has such a form, decode the raw query string (`URI#getRawQuery` in `toRequest`) with `Form.fromString`, which treats `+` as a space.
:::

## Decoding a JSON Body

`Request#body` holds the bytes that `toRequest` read. `JsonCodec#decode` turns them into a `Book` and returns `Either[SchemaError, Book]`, so malformed or incomplete JSON never throws. `SchemaError#message` describes what was wrong and where, which makes a clear 400 response:

```scala
def addBook(request: Request): Response =
  bookJson.decode(request.body.toArray) match {
    case Right(book) => Response.json(Status.Created, bookJson.encodeToString(book))
    case Left(error) => Response.text(Status.BadRequest, s"Invalid book: ${error.message}")
  }
```

This handler only echoes the book back with 201 Created; a real service would store it first. To post a complete book, one with a missing field and one with a field of the wrong type:

```scala
withServer(addBook) { port =>
  call(port, "POST", "/books", """{"id":4,"title":"Scala with Cats","authors":["Noel Welsh","Dave Gurnell"],"year":2017}""")
  call(port, "POST", "/books", """{"id":4,"title":"Scala with Cats"}""")
  call(port, "POST", "/books", """{"id":"four"}""")
}
// POST /books -> 201 [application/json, 86 bytes]
// {"id":4,"title":"Scala with Cats","authors":["Noel Welsh","Dave Gurnell"],"year":2017}
// POST /books -> 400 [text/plain; charset=UTF-8, 49 bytes]
// Invalid book: missing required field "year" at: .
// POST /books -> 400 [text/plain; charset=UTF-8, 36 bytes]
// Invalid book: illegal number at: .id
```

The error message names the missing field and the path of the bad value (`.id`).

## Serving an HTML Page

The last route serves an HTML catalog at `/`. We build the page with the [HTML DSL](./your-first-html-page.md), render it to a string, and wrap the UTF-8 bytes in a `Body` with the `text/html` content type. We need `Body.fromArray` here because `Body.fromString` always uses `text/plain`:

```scala

def catalogPage(books: List[Book]): Dom.Element =
  html(lang := "en")(
    head(
      meta(charset := "utf-8"),
      title("Scala Bookshelf")
    ),
    body(
      h1("Scala Bookshelf"),
      ul(books.map(book => li(a(href := s"/books/${book.id}")(s"${book.title} (${book.year})"))))
    )
  )

def htmlResponse(page: String): Response =
  Response.ok(Body.fromArray(page.getBytes("UTF-8"), ContentType.`text/html`))
```

`Response.ok(body)` copies the body's content type into the `content-type` header. The `text/html` constant on `ContentType` carries no charset, so the page declares UTF-8 itself with `<meta charset="utf-8">`. To serve the page:

```scala
withServer(_ => htmlResponse(catalogPage(books).render)) { port =>
  call(port, "GET", "/")
}
// GET / -> 200 [text/html, 313 bytes]
// <html lang="en"><head><meta charset="utf-8"/><title>Scala Bookshelf</title></head><body>Scala BookshelfZionomicon (2024)Programming in Scala (2008)Functional Programming in Scala (2014)</body></html>
```

Each link is built with `href := ...` and points at the JSON route for that book.

## Putting It Together

The complete example combines every route into one bookstore application and calls each route once. It builds on three shared files embedded under Running the Examples below: the adapter `JdkHttp.scala`, which you can copy into your own project unchanged, the test client `Client.scala` and the model `Bookstore.scala`. The complete example:

```scala title="http-model-examples/src/main/scala/servehttpmodelrequests/CompleteExample.scala"
package servehttpmodelrequests

/**
 * How to Serve http-model Requests with the JDK HTTP Server — Complete Example
 *
 * A tiny bookstore served by the JDK HTTP server through the http-model
 * adapter: an HTML catalog at /, a JSON list at /books with an optional
 * `?limit=`, one book at /books/{id}, JSON input at POST /books, and a 404
 * fallback. The program calls every route with `java.net.http.HttpClient` and
 * stops the server when it is done.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * servehttpmodelrequests.CompleteExample"
 */
object CompleteExample {
  def notFound(request: Request): Response =
    Response.text(Status.NotFound, s"No route for ${request.method.name} ${request.path.encode}")

  def listBooks(request: Request): Response =
    request.queryParams.getFirst("limit") match {
      case None      => Response.json(bookListJson.encodeToString(books))
      case Some(raw) =>
        raw.toIntOption.filter(_ >= 0) match {
          case Some(limit) => Response.json(bookListJson.encodeToString(books.take(limit)))
          case None        => Response.text(Status.BadRequest, s"limit must be a non-negative integer, got '$raw'")
        }
    }

  def getBook(id: String): Response =
    books.find(_.id.toString == id) match {
      case Some(book) => Response.json(bookJson.encodeToString(book))
      case None       => Response.text(Status.NotFound, s"No book with id $id")
    }

  def addBook(request: Request): Response =
    bookJson.decode(request.body.toArray) match {
      case Right(book) => Response.json(Status.Created, bookJson.encodeToString(book))
      case Left(error) => Response.text(Status.BadRequest, s"Invalid book: ${error.message}")
    }

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil) =>
        Response.ok(Body.fromArray(catalogPage(books).render.getBytes("UTF-8"), ContentType.`text/html`))
      case (Method.GET, List("books"))     => listBooks(request)
      case (Method.GET, List("books", id)) => getBook(id)
      case (Method.POST, List("books"))    => addBook(request)
      case _                               => notFound(request)
    }

  def main(args: Array[String]): Unit =
    JdkHttp.withServer(app) { port =>
      Client.call(port, "GET", "/")
      Client.call(port, "GET", "/books?limit=2")
      Client.call(port, "GET", "/books/1")
      Client.call(port, "HEAD", "/books/1")
      Client.call(port, "GET", "/books/42")
      Client.call(
        port,
        "POST",
        "/books",
        """{"id":4,"title":"Scala with Cats","authors":["Noel Welsh","Dave Gurnell"],"year":2017}"""
      )
      Client.call(port, "POST", "/books", """{"id":4,"title":"Scala with Cats"}""")
      Client.call(port, "GET", "/authors")
    }
}
```

## Running the Examples

All code from this guide is available as runnable examples in the `http-model-examples` directory. It is a standalone sbt build that uses the published ZIO Blocks artifacts. To run them, clone the repository and move into that directory:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/http-model-examples
```

The examples share four files: the adapter, the test client, and the bookstore model with its catalog page.

<details>
  <summary>http-model-examples/src/main/scala/servehttpmodelrequests/JdkHttp.scala</summary>

```scala title="http-model-examples/src/main/scala/servehttpmodelrequests/JdkHttp.scala" showLineNumbers
package servehttpmodelrequests

/**
 * How to Serve http-model Requests with the JDK HTTP Server — the adapter
 *
 * Glue between the JDK's built-in `com.sun.net.httpserver.HttpServer` and a
 * plain `Request => Response` function. It is meant for examples, tools and
 * small services, not for production traffic.
 */
object JdkHttp {

  /**
   * Reads the method, URL, headers and body of an exchange into a `Request`.
   * Returns `Left` with a ready-made error response when the method is unknown
   * (501) or the request target is not a valid URL (400).
   */
  def toRequest(exchange: HttpExchange): Either[Response, Request] = {
    val uri    = exchange.getRequestURI
    val target = uri.getRawPath + Option(uri.getRawQuery).fold("")("?" + _)
    for {
      method <- Method.fromString(exchange.getRequestMethod).toRight(Response(Status.NotImplemented))
      url    <- URL.parse(target).left.map(error => Response.text(Status.BadRequest, error))
    } yield {
      val headers = exchange.getRequestHeaders.asScala.foldLeft(Headers.empty) { case (acc, (name, values)) =>
        values.asScala.foldLeft(acc)((withValue, value) => withValue.add(name, value))
      }
      val contentType =
        headers.get(Header.ContentType).map(_.value).getOrElse(ContentType.`application/octet-stream`)
      val body    = Body.fromArray(exchange.getRequestBody.readAllBytes(), contentType)
      val version = Version.fromString(exchange.getProtocol).getOrElse(Version.`HTTP/1.1`)
      Request(method, url, headers, body, version)
    }
  }

  /**
   * Writes the status, headers and body of a `Response` to an exchange. The JDK
   * server computes `Content-Length` itself, so the response's own
   * `content-length` and `transfer-encoding` headers are skipped.
   */
  def writeResponse(exchange: HttpExchange, response: Response): Unit = {
    val out = exchange.getResponseHeaders
    response.headers.toList.foreach { case (name, value) =>
      if (name != "content-length" && name != "transfer-encoding") out.add(name, value)
    }
    val bytes = response.body.toArray
    if (bytes.nonEmpty && !response.headers.has("content-type"))
      out.set("content-type", response.body.contentType.render)
    val code   = response.status.code
    val isHead = exchange.getRequestMethod == "HEAD"
    if (isHead && bytes.nonEmpty) out.set("content-length", bytes.length.toString)
    val noBody = isHead || code == 204 || code == 304 || bytes.isEmpty
    exchange.sendResponseHeaders(code, if (noBody) -1L else bytes.length.toLong)
    if (!noBody) exchange.getResponseBody.write(bytes)
  }

  /**
   * Turns an application into an `HttpHandler`. A `HEAD` request runs the `GET`
   * route and drops the body; a non-fatal exception or `LinkageError` in the
   * application becomes a 500 response; a failure while writing the response is
   * logged and, if no headers have gone out yet, also becomes a 500; the
   * exchange is closed on every path.
   */
  def handler(app: Request => Response): HttpHandler = exchange =>
    try {
      val response =
        try
          toRequest(exchange) match {
            case Left(error)                                     => error
            case Right(request) if request.method == Method.HEAD => app(request.copy(method = Method.GET))
            case Right(request)                                  => app(request)
          }
        catch {
          case error @ (NonFatal(_) | _: LinkageError) =>
            System.err.println(s"Handler failed: $error")
            Response.text(Status.InternalServerError, "Internal Server Error")
        }
      try writeResponse(exchange, response)
      catch {
        case error @ (NonFatal(_) | _: LinkageError) =>
          System.err.println(s"Writing the response failed: $error")
          if (exchange.getResponseCode == -1) {
            exchange.getResponseHeaders.clear()
            exchange.sendResponseHeaders(Status.InternalServerError.code, -1L)
          }
      }
    } finally exchange.close()

  /**
   * Starts a server on the loopback interface. Port 0 asks the operating system
   * for a free port; read it back with `server.getAddress.getPort`.
   */
  def start(app: Request => Response, port: Int = 0): HttpServer = {
    val server = HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress, port), 0)
    server.createContext("/", handler(app))
    server.start()
    server
  }

  /**
   * Starts a server on a free loopback port, passes the port to `use`, and
   * stops the server when `use` returns or throws.
   */
  def withServer[A](app: Request => Response)(use: Int => A): A = {
    val server = start(app)
    try use(server.getAddress.getPort)
    finally server.stop(0)
  }
}
```

</details>

<details>
  <summary>http-model-examples/src/main/scala/servehttpmodelrequests/Client.scala</summary>

```scala title="http-model-examples/src/main/scala/servehttpmodelrequests/Client.scala" showLineNumbers
package servehttpmodelrequests

/**
 * How to Serve http-model Requests with the JDK HTTP Server — a test client
 *
 * Sends one request with the JDK's `java.net.http.HttpClient` and prints the
 * status code, the `Content-Type` and `Content-Length` headers, and the body.
 */
object Client {
  private val client = HttpClient.newBuilder().version(HttpClient.Version.HTTP_1_1).build()

  def call(port: Int, method: String, target: String, body: String = ""): Unit = {
    val builder = HttpRequest.newBuilder(URI.create(s"http://127.0.0.1:$port$target"))
    val request =
      if (body.isEmpty) builder.method(method, HttpRequest.BodyPublishers.noBody()).build()
      else
        builder
          .method(method, HttpRequest.BodyPublishers.ofString(body))
          .header("content-type", "application/json")
          .build()
    val response                     = client.send(request, HttpResponse.BodyHandlers.ofString())
    def header(name: String): String = response.headers.firstValue(name).orElse("none")
    println(s"$method $target -> ${response.statusCode} [${header("content-type")}, ${header("content-length")} bytes]")
    if (response.body.nonEmpty) println(response.body)
  }
}
```

</details>

<details>
  <summary>http-model-examples/src/main/scala/servehttpmodelrequests/Bookstore.scala</summary>

```scala title="http-model-examples/src/main/scala/servehttpmodelrequests/Bookstore.scala" showLineNumbers
package servehttpmodelrequests

/**
 * How to Serve http-model Requests with the JDK HTTP Server — the bookstore
 * model
 *
 * Three Scala books, the JSON codecs derived from their schema, and an HTML
 * catalog page built with the html module.
 */
final case class Book(id: Int, title: String, authors: List[String], year: Int)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

object Bookstore {
  val books: List[Book] = List(
    Book(1, "Zionomicon", List("John A. De Goes", "Adam Fraser", "Milad Khajavi"), 2024),
    Book(2, "Programming in Scala", List("Martin Odersky", "Lex Spoon", "Bill Venners"), 2008),
    Book(3, "Functional Programming in Scala", List("Paul Chiusano", "Rúnar Bjarnason"), 2014)
  )

  val bookJson: JsonCodec[Book]           = Schema[Book].jsonCodec
  val bookListJson: JsonCodec[List[Book]] = Schema[List[Book]].jsonCodec

  def catalogPage(books: List[Book]): Dom.Element =
    html(lang := "en")(
      head(
        meta(charset := "utf-8"),
        title("Scala Bookshelf")
      ),
      body(
        h1("Scala Bookshelf"),
        ul(books.map(book => li(a(href := s"/books/${book.id}")(s"${book.title} (${book.year})"))))
      )
    )
}
```

</details>

The first example serves the hello world handler and the failing handler:

<details>
  <summary>http-model-examples/src/main/scala/servehttpmodelrequests/Step1HelloWorldExample.scala</summary>

```scala title="http-model-examples/src/main/scala/servehttpmodelrequests/Step1HelloWorldExample.scala" showLineNumbers
package servehttpmodelrequests

/**
 * How to Serve http-model Requests with the JDK HTTP Server — Serving a Hello
 * World Handler
 *
 * Serves one `Request => Response` function on a free loopback port, calls it
 * with GET and HEAD, then shows that a failing handler becomes a 500 response.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * servehttpmodelrequests.Step1HelloWorldExample"
 */
object Step1HelloWorldExample {
  val hello: Request => Response =
    request => Response.text(s"Hello from ${request.method.name} ${request.path.encode}")

  val broken: Request => Response =
    _ => throw new IllegalStateException("the catalog is not loaded")

  def main(args: Array[String]): Unit = {
    JdkHttp.withServer(hello) { port =>
      Client.call(port, "GET", "/books")
      Client.call(port, "HEAD", "/books")
    }
    JdkHttp.withServer(broken) { port =>
      Client.call(port, "GET", "/books")
    }
  }
}
```

</details>

To run it:

```bash
sbt "runMain servehttpmodelrequests.Step1HelloWorldExample"
```

The second example routes `GET /books` and `GET /books/{id}` with a 404 fallback:

<details>
  <summary>http-model-examples/src/main/scala/servehttpmodelrequests/Step2RoutingExample.scala</summary>

```scala title="http-model-examples/src/main/scala/servehttpmodelrequests/Step2RoutingExample.scala" showLineNumbers
package servehttpmodelrequests

/**
 * How to Serve http-model Requests with the JDK HTTP Server — Routing a Few
 * Paths
 *
 * Routes GET /books and GET /books/{id} by pattern matching on the method and
 * the path segments, with a 404 fallback for everything else.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * servehttpmodelrequests.Step2RoutingExample"
 */
object Step2RoutingExample {
  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, List("books")) =>
        Response.json(bookListJson.encodeToString(books))
      case (Method.GET, List("books", id)) =>
        books.find(_.id.toString == id) match {
          case Some(book) => Response.json(bookJson.encodeToString(book))
          case None       => Response.text(Status.NotFound, s"No book with id $id")
        }
      case _ =>
        Response.text(Status.NotFound, s"No route for ${request.method.name} ${request.path.encode}")
    }

  def main(args: Array[String]): Unit =
    JdkHttp.withServer(app) { port =>
      Client.call(port, "GET", "/books")
      Client.call(port, "GET", "/books/1")
      Client.call(port, "GET", "/books/42")
      Client.call(port, "DELETE", "/books/1")
    }
}
```

</details>

To run it:

```bash
sbt "runMain servehttpmodelrequests.Step2RoutingExample"
```

The third example reads `?limit=` and decodes a JSON request body:

<details>
  <summary>http-model-examples/src/main/scala/servehttpmodelrequests/Step3QueryAndJsonExample.scala</summary>

```scala title="http-model-examples/src/main/scala/servehttpmodelrequests/Step3QueryAndJsonExample.scala" showLineNumbers
package servehttpmodelrequests

/**
 * How to Serve http-model Requests with the JDK HTTP Server — Reading Query
 * Parameters and a JSON Body
 *
 * Reads `?limit=` from the query string and decodes a JSON request body with
 * the codec derived from `Book`'s schema, answering 400 on bad input.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * servehttpmodelrequests.Step3QueryAndJsonExample"
 */
object Step3QueryAndJsonExample {
  def listBooks(request: Request): Response =
    request.queryParams.getFirst("limit") match {
      case None      => Response.json(bookListJson.encodeToString(books))
      case Some(raw) =>
        raw.toIntOption.filter(_ >= 0) match {
          case Some(limit) => Response.json(bookListJson.encodeToString(books.take(limit)))
          case None        => Response.text(Status.BadRequest, s"limit must be a non-negative integer, got '$raw'")
        }
    }

  def addBook(request: Request): Response =
    bookJson.decode(request.body.toArray) match {
      case Right(book) => Response.json(Status.Created, bookJson.encodeToString(book))
      case Left(error) => Response.text(Status.BadRequest, s"Invalid book: ${error.message}")
    }

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, List("books"))  => listBooks(request)
      case (Method.POST, List("books")) => addBook(request)
      case _                            =>
        Response.text(Status.NotFound, s"No route for ${request.method.name} ${request.path.encode}")
    }

  def main(args: Array[String]): Unit =
    JdkHttp.withServer(app) { port =>
      Client.call(port, "GET", "/books?limit=1")
      Client.call(port, "GET", "/books?limit=all")
      Client.call(
        port,
        "POST",
        "/books",
        """{"id":4,"title":"Scala with Cats","authors":["Noel Welsh","Dave Gurnell"],"year":2017}"""
      )
      Client.call(port, "POST", "/books", """{"id":4,"title":"Scala with Cats"}""")
      Client.call(port, "POST", "/books", """{"id":"four"}""")
    }
}
```

</details>

To run it:

```bash
sbt "runMain servehttpmodelrequests.Step3QueryAndJsonExample"
```

The fourth example serves the HTML catalog page:

<details>
  <summary>http-model-examples/src/main/scala/servehttpmodelrequests/Step4HtmlPageExample.scala</summary>

```scala title="http-model-examples/src/main/scala/servehttpmodelrequests/Step4HtmlPageExample.scala" showLineNumbers
package servehttpmodelrequests

/**
 * How to Serve http-model Requests with the JDK HTTP Server — Serving an HTML
 * Page
 *
 * Renders the catalog page with the html module and serves it as text/html.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * servehttpmodelrequests.Step4HtmlPageExample"
 */
object Step4HtmlPageExample {
  def htmlResponse(page: String): Response =
    Response.ok(Body.fromArray(page.getBytes("UTF-8"), ContentType.`text/html`))

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, Nil) => htmlResponse(catalogPage(books).render)
      case _                 =>
        Response.text(Status.NotFound, s"No route for ${request.method.name} ${request.path.encode}")
    }

  def main(args: Array[String]): Unit =
    JdkHttp.withServer(app) { port =>
      Client.call(port, "GET", "/")
    }
}
```

</details>

To run it:

```bash
sbt "runMain servehttpmodelrequests.Step4HtmlPageExample"
```

To run the complete example:

```bash
sbt "runMain servehttpmodelrequests.CompleteExample"
```

To compile all the examples without running them:

```bash
sbt compile
```

## Going Further

The adapter is deliberately small. These are the knobs you are most likely to reach for next:

- **Concurrency**: With no executor set, the JDK server runs every handler on its single dispatcher thread, one request at a time. Call `server.setExecutor(...)` before `server.start()` to handle requests in parallel. On JDK 21 or newer, a virtual thread per request is a good fit for blocking handlers: `server.setExecutor(Executors.newVirtualThreadPerTaskExecutor())`.
- **TLS**: `com.sun.net.httpserver.HttpsServer` accepts the same handler; configure it with an `HttpsConfigurator` that wraps your `SSLContext`.
- **Shutdown**: Pass a positive delay to `HttpServer#stop` to let in-flight requests finish.
- **Routing**: The [`RoutePattern`](../reference/endpoint/route-pattern.md) and [`RouteTree`](../reference/endpoint/route-tree.md) pages show typed path parameters and prefix-tree lookup.
- **Typed headers and query values**: The [http-model schema integration](../reference/http-model/schema.md) decodes query parameters and headers into typed values with a `Schema`.

For the types this guide relies on, see the [HTTP model overview](../reference/http-model/index.md), the [HTTP model reference](../reference/http-model/model.md) and the [headers reference](../reference/http-model/headers.md).

---

## Build a Small JSON Backend from Blocks

Welcome! This tutorial is for Scala 3 developers who want to see the ZIO Blocks modules work together in one small, real program. Every other guide teaches one block; here we build a bookstore backend that answers JSON over HTTP, and each section adds one block to it. You don't need to have used any of the blocks before: each section explains just enough to move on, and links to the guide that teaches that block in depth.

## 1. Introduction

By the end of this tutorial, you will understand:

- How one derived `Schema` gives a case class its JSON codec, its SQL table mapping and its typed settings
- How layered settings (environment over a YAML file over defaults) reach each service as its own typed value
- How `Resource.from` builds a graph of services from their constructors and closes them in reverse order
- How a plain `Request => Response` function becomes an HTTP backend with checked input, logs, spans and a request counter
- How to test that backend without a server, and once through the real one

We build the backend in nine steps:

1. Model the books
2. Load the settings
3. Store books in SQLite
4. Check the query parameters
5. Route requests
6. Log, trace and count requests
7. Serve the routes over HTTP
8. Wire everything together and shut down
9. Test the backend

We recommend reading from top to bottom — each section builds on the previous one.

To follow along, add the modules the backend uses, plus the SQLite JDBC driver, to your `build.sbt`. Scala 3.3 to 3.8 use the `zio-blocks-*` artifacts:

```scala
libraryDependencies ++= Seq(
  "dev.zio"   %% "zio-blocks-schema"            % "0.0.56",
  "dev.zio"   %% "zio-blocks-config"            % "0.0.56",
  "dev.zio"   %% "zio-blocks-config-yaml"       % "0.0.56",
  "dev.zio"   %% "zio-blocks-scope"             % "0.0.56",
  "dev.zio"   %% "zio-blocks-sql"               % "0.0.56",
  "dev.zio"   %% "zio-blocks-http-model"        % "0.0.56",
  "dev.zio"   %% "zio-blocks-http-model-schema" % "0.0.56",
  "dev.zio"   %% "zio-blocks-telemetry"         % "0.0.56",
  "org.xerial" % "sqlite-jdbc"                  % "3.53.4.0"
)
```

Scala 3.9 and newer use the `zio-blocks-next-*` artifacts instead, and need the `-experimental` compiler flag:

```scala
libraryDependencies ++= Seq(
  "dev.zio"   %% "zio-blocks-next-schema"            % "0.0.56",
  "dev.zio"   %% "zio-blocks-next-config"            % "0.0.56",
  "dev.zio"   %% "zio-blocks-next-config-yaml"       % "0.0.56",
  "dev.zio"   %% "zio-blocks-next-scope"             % "0.0.56",
  "dev.zio"   %% "zio-blocks-next-sql"               % "0.0.56",
  "dev.zio"   %% "zio-blocks-next-http-model"        % "0.0.56",
  "dev.zio"   %% "zio-blocks-next-http-model-schema" % "0.0.56",
  "dev.zio"   %% "zio-blocks-next-telemetry"         % "0.0.56",
  "org.xerial" % "sqlite-jdbc"                       % "3.53.4.0"
)
scalacOptions += "-experimental"
```

The tutorial is Scala 3 only, because the `sql` module is published for Scala 3 only. Use **JDK 21 or newer**, to compile as well as to run. The `telemetry` module is compiled for Java 21, and its `log.*` macros run inside the compiler, so on JDK 17 code that calls `log.info` doesn't even compile. The `http-model` and `sql` modules depend on the `streams` module, which is compiled for Java 21 too. [How to Add ZIO Blocks to Your Build](../getting-started/installation.md) explains the two artifact names, cross-building and the JDK requirements in full.

## 2. Background: One Backend, Seven Blocks

A backend is mostly glue: read settings, open a database, turn HTTP requests into database calls and database rows into JSON, and write down what happened. ZIO Blocks gives you one small library for each of those jobs, with no dependencies between them that you didn't ask for. This tutorial is about the glue: how the blocks meet.

Here is the bookstore we build. Settings flow down from one layered source; each service receives only its own section; requests flow in from the JDK HTTP server through the telemetry wrapper to the routes and the repository:

```
       BOOKSTORE_* env  >  bookstore.yaml  >  defaults      (config)
                          │ Config.wire
        ┌─────────────────┼──────────────────┐
  ServerConfig      CatalogConfig         DbConfig
        │                 │                  │
   HttpService ──▶ Routes ──────▶ BookRepo ──▶ Db ──▶ bookstore.db
   (JdkHttp +   (http-model,     (sql: Repo,      (SQLite)
    telemetry)   JSON from        Table)
        ▲        schema)
        │
   HTTP client
```

| Block               | Its job in the bookstore                          | Learn it in depth                                                                       |
| ------------------- | ------------------------------------------------- | --------------------------------------------------------------------------------------- |
| `schema`            | JSON codecs for books and errors                  | [Your First Schema](./your-first-schema.md)                                             |
| `config`            | Typed, layered settings with a clear error report | [Build a Typed Configuration Loader](./typed-config-loader.md)                          |
| `scope`             | Builds the services and closes them in order      | [Compile-Time Resource Safety with Scope](./compile-time-resource-safety-with-scope.md) |
| `sql`               | The SQLite repository                             | [Build Your First Database Repository on SQLite](./your-first-sqlite-repository.md)     |
| `http-model`        | `Request`, `Response` and routing                 | [HTTP Model reference](../reference/http-model/index.md)                                |
| `http-model-schema` | Typed query parameters                            | [Schema-Based Typed Access](../reference/http-model/schema.md)                          |
| `telemetry`         | Request logs, spans and a counter                 | [Telemetry Guide](./telemetry-guide.md)                                                 |

:::note
ZIO Blocks ships no HTTP server. The `http-model` module describes requests and responses; it doesn't open sockets. In section 9 we connect it to the HTTP server built into the JDK through a small adapter. That adapter is glue for examples, tools and small internal services, not a production server: it has no TLS, no HTTP/2 and no limit on request body size.
:::

## 3. Model the Books

Everything starts with the data. A `Book` is what the catalog stores and what `GET` returns; a `NewBook` is what a client sends to add one, without an id, because the server picks the id; an `ApiError` is the body of every error response. Each one derives a `Schema`, and the JSON codecs come from those schemas:

```scala

final case class Book(
  id: Long,
  title: String,
  firstAuthor: String,
  year: Int,
  isbn: Option[String],
  stock: Int,
  priceCents: Int
)
object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

final case class NewBook(
  title: String,
  firstAuthor: String,
  year: Int,
  isbn: Option[String],
  stock: Int,
  priceCents: Int
)
object NewBook {
  implicit val schema: Schema[NewBook] = Schema.derived
}

final case class ApiError(error: String)
object ApiError {
  implicit val schema: Schema[ApiError] = Schema.derived
}

object Json {
  val book: JsonCodec[Book]        = Schema[Book].jsonCodec
  val books: JsonCodec[List[Book]] = Schema[List[Book]].jsonCodec
  val newBook: JsonCodec[NewBook]  = Schema[NewBook].jsonCodec
  val error: JsonCodec[ApiError]   = Schema[ApiError].jsonCodec
}
```

The code above:

- `Schema.derived` describes each case class: its fields, their names and their types. We write no JSON code at all.
- `firstAuthor` holds the first author only, and `isbn` is an `Option`, because not every book in our catalog has an ISBN on record. Stock and prices are made up; prices are whole cents, so no rounding sneaks in.
- `Schema[Book].jsonCodec` asks the schema for its JSON codec. The `Json` object keeps one codec per type, so the routes don't look them up on every request.

Let's encode a book, decode a request body, and decode a body that is missing fields:

```scala
val zionomicon = Book(1, "Zionomicon", "John A. De Goes", 2024, None, 3, 4500)
// zionomicon: Book = Book(
//   id = 1L,
//   title = "Zionomicon",
//   firstAuthor = "John A. De Goes",
//   year = 2024,
//   isbn = None,
//   stock = 3,
//   priceCents = 4500
// )
println(Json.book.encodeToString(zionomicon))
// {"id":1,"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024,"stock":3,"priceCents":4500}

Json.newBook.decode("""{"title":"Scala with Cats","firstAuthor":"Noel Welsh","year":2017,"stock":2,"priceCents":3500}""")
// res5: Either[SchemaError, NewBook] = Right(
//   NewBook(
//     title = "Scala with Cats",
//     firstAuthor = "Noel Welsh",
//     year = 2017,
//     isbn = None,
//     stock = 2,
//     priceCents = 3500
//   )
// )

Json.newBook.decode("""{"title":"Scala with Cats"}""") match {
  case Right(book) => println(book)
  case Left(error) => println(error.message)
}
// missing required field "firstAuthor" at: .
```

Notice that the encoded `Zionomicon` has no `isbn` field: a `None` is left out of the JSON, and a missing optional field decodes back to `None`. A missing required field is an error with a readable message, which is exactly what we send back to a client as a 400 in section 7. The [Your First Schema](./your-first-schema.md) tutorial explains what else a schema gives you.

## 4. Load the Settings

The backend needs three settings: the HTTP port, the database file and the default page size of the catalog. Each service gets its own settings type. That keeps every service's constructor honest about what it reads, and it matters for the wiring in section 10: the dependency graph finds values by type, so two services that both took a plain `Int` would silently share one value.

We read the settings from three layers. Environment variables win over a YAML file, which wins over built-in defaults:

```scala

final case class ServerConfig(port: Int)
object ServerConfig {
  implicit val schema: Schema[ServerConfig] = Schema.derived
}

final case class DbConfig(file: String)
object DbConfig {
  implicit val schema: Schema[DbConfig] = Schema.derived
}

final case class CatalogConfig(pageSize: Int)
object CatalogConfig {
  implicit val schema: Schema[CatalogConfig] = Schema.derived
}

final case class BookstoreConfig(server: ServerConfig, db: DbConfig, catalog: CatalogConfig)
object BookstoreConfig {
  implicit val schema: Schema[BookstoreConfig] = Schema.derived
}

object Settings {
  val bookstoreYaml: String =
    """server:
      |  port: 0
      |db:
      |  file: bookstore.db
      |""".stripMargin

  val yaml: ConfigSource = ConfigSource.fromYaml(bookstoreYaml, "bookstore.yaml") match {
    case Right(source) => source
    case Left(error)   => throw error
  }

  val defaults: ConfigSource =
    ConfigSource.fromMap(Map("server.port" -> "8080", "catalog.pageSize" -> "2"), "defaults")

  val environment: ConfigSource = EnvSource.keyFormat(KeyFormat.UpperSnakeCase).prefix("bookstore")

  def layered(env: ConfigSource): ConfigSource = env.orElse(yaml).orElse(defaults)
}
```

The code above:

- `ServerConfig`, `DbConfig` and `CatalogConfig` are one settings section each. `BookstoreConfig` groups them so we can check all settings at once at startup.
- The YAML file sets `server.port` to `0`, which asks the operating system for any free port. That keeps the examples from clashing with whatever already runs on your machine; production sets the port through the environment.
- `ConfigSource.fromMap` holds the defaults. The default page size is `2` only so the output in this tutorial stays short.
- `EnvSource.keyFormat(KeyFormat.UpperSnakeCase).prefix("bookstore")` reads `catalog.pageSize` from `BOOKSTORE_CATALOG_PAGE_SIZE`, `server.port` from `BOOKSTORE_SERVER_PORT` and `db.file` from `BOOKSTORE_DB_FILE`. The prefix matters: it keeps the bookstore from reading, or printing, any other variable of the process.
- `ConfigSource#orElse` layers two sources: a key comes from the first source that has it. `Settings.layered` takes the environment as a parameter, so the rest of this page and the tests can pass a stand-in.

To see the layers at work, let's stand in for the environment with a map that sets `db.file`, load the whole `BookstoreConfig`, and print where each setting came from:

```scala
val demoEnv = ConfigSource.fromMap(Map("db.file" -> "/tmp/bookstore-tutorial.db"), "env")
// demoEnv: ConfigSource = MapSource(
//   map = Map("db.file" -> "/tmp/bookstore-tutorial.db"),
//   sourceId = "env"
// )

Config.loadWithProvenance[BookstoreConfig](Settings.layered(demoEnv)) match {
  case Right(loaded) =>
    println(loaded.value)
    println(loaded.dump())
  case Left(errors) => errors.foreach(error => println(error.message))
}
// BookstoreConfig(ServerConfig(0),DbConfig(/tmp/bookstore-tutorial.db),CatalogConfig(2))
// ┌──────────────────┬────────────────────────────┬────────────────┐
// │ Key              │ Value                      │ Source         │
// ├──────────────────┼────────────────────────────┼────────────────┤
// │ catalog.pageSize │ 2                          │ defaults       │
// │ db.file          │ /tmp/bookstore-tutorial.db │ env            │
// │ server.port      │ 0                          │ bookstore.yaml │
// └──────────────────┴────────────────────────────┴────────────────┘
```

Each layer wins once: the page size comes from the defaults, the port from the YAML file, and the database file from the environment, which overrides the YAML file's `bookstore.db`. `Config.loadWithProvenance` returns the decoded value together with the sources it came from, and `ProvenanceMap#dump` prints them as a table. A program can print this table at startup, so whoever runs it sees which settings are in effect. The [Build a Typed Configuration Loader](./typed-config-loader.md) tutorial covers sources, formats and secrets in depth.

## 5. Store Books in SQLite

Now the books need somewhere to live. The `sql` module turns the same `Book` schema into a table mapping and a repository with ready-made reads and writes. We split storage into two services: `Db` owns the database file, and `BookRepo` owns the queries.

`Db` is the first service with something to close, so it extends `AutoCloseable`:

```scala

class Db(config: DbConfig) extends AutoCloseable {
  val transactor: JdbcTransactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:${config.file}", SqlDialect.SQLite)

  transactor.connect {
    sql"""CREATE TABLE IF NOT EXISTS book (
      id INTEGER PRIMARY KEY,
      title TEXT NOT NULL,
      first_author TEXT NOT NULL,
      year INTEGER NOT NULL,
      isbn TEXT,
      stock INTEGER NOT NULL,
      price_cents INTEGER NOT NULL
    )""".update
  }
  println("Db: opened")

  def close(): Unit = println("Db: closed")
}

class BookRepo(db: Db) {
  private val books = Repo.derived[Book, Long]

  db.transactor.transact {
    if (books.count == 0) books.insertBatch(BookRepo.starterCatalog)
  }

  def page(limit: Int, offset: Int): List[Book] =
    db.transactor.connect {
      sql"""SELECT id, title, first_author, year, isbn, stock, price_cents FROM book
            ORDER BY id LIMIT $limit OFFSET $offset""".query[Book]
    }

  def find(id: Long): Option[Book] =
    db.transactor.connect(books.find(id).toOption)

  def add(book: NewBook): Book =
    db.transactor.transact {
      val id = sql"SELECT COALESCE(MAX(id), 0) + 1 FROM book".queryOne[Long].getOrElse(1L)
      books.insertReturning(
        Book(id, book.title, book.firstAuthor, book.year, book.isbn, book.stock, book.priceCents)
      )
    }
}

object BookRepo {
  // Stock and prices are made up.
  val starterCatalog: List[Book] = List(
    Book(1, "Zionomicon", "John A. De Goes", 2024, None, 3, 4500),
    Book(2, "Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"), 5, 3999),
    Book(3, "Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"), 4, 4499)
  )
}

object TempDb {
  def file(): String = {
    val path = Files.createTempFile("bookstore", ".db")
    path.toFile.deleteOnExit()
    path.toString
  }
}
```

The code above:

- `JdbcTransactor.fromUrl` points at the SQLite file named in `DbConfig`. It opens a connection for each `JdbcTransactor#connect` or `JdbcTransactor#transact` call and closes it afterwards, so `Db#close` has nothing to release here and only reports that it ran. A connection pool, such as the one in [How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md), would be closed in `Db#close`.
- We write the `CREATE TABLE` statement by hand, because it declares `id` as the `PRIMARY KEY`; `Table#createTable` doesn't, a known bug in version 0.0.56. The column names are the field names in snake case, which is the naming `Repo.derived` expects.
- `Repo.derived[Book, Long]` derives a repository for `Book`, keyed by its `id` field. `BookRepo` fills an empty catalog with three starter books in one transaction.
- `BookRepo#page` writes its own query, because `LIMIT` and `OFFSET` paging isn't one of the ready-made operations. The `$limit` and `$offset` values become query parameters, not text spliced into the SQL.
- `BookRepo#add` picks the next free id and inserts the book in one transaction, so two clients adding books at once can't get the same id.
- `TempDb.file()` creates an empty temporary file. SQLite's `jdbc:sqlite::memory:` would give each connection its own empty database, so a temporary file is the simplest way to start every run from the starter catalog.

Let's open a fresh database:

```scala
val storageDb = new Db(DbConfig(TempDb.file()))
val repo      = new BookRepo(storageDb)
```

Now we can try the repository:

```scala
repo.page(limit = 2, offset = 0).foreach(println)
// Book(1,Zionomicon,John A. De Goes,2024,None,3,4500)
// Book(2,Programming in Scala,Martin Odersky,2008,Some(9780981531601),5,3999)
repo.find(3)
// res9: Option[Book] = Some(
//   Book(
//     id = 3L,
//     title = "Functional Programming in Scala",
//     firstAuthor = "Paul Chiusano",
//     year = 2014,
//     isbn = Some("9781617290657"),
//     stock = 4,
//     priceCents = 4499
//   )
// )
repo.find(42)
// res10: Option[Book] = None
repo.add(NewBook("Scala with Cats", "Noel Welsh", 2017, None, 2, 3500))
// res11: Book = Book(
//   id = 4L,
//   title = "Scala with Cats",
//   firstAuthor = "Noel Welsh",
//   year = 2017,
//   isbn = None,
//   stock = 2,
//   priceCents = 3500
// )

storageDb.close()
// Db: closed
```

The starter catalog is there, a missing id gives `None`, and the new book gets id `4`. The [Build Your First Database Repository on SQLite](./your-first-sqlite-repository.md) tutorial covers tables, repositories, queries and transactions step by step.

## 6. Check the Query Parameters

Before we route any request, let's decide how to read its input. `GET /books` takes two optional query parameters, `limit` and `offset`. A missing parameter falls back to a default; a malformed or out-of-range one must be rejected with a clear message, never silently replaced.

The `http-model-schema` module reads a query parameter as a typed value with `request.query[Int](name)`. It returns `Right` with the value, or `Left` with a `QueryParamError` that says whether the parameter was missing or malformed:

```scala

final case class Paging(limit: Int, offset: Int)

object Paging {
  val maxLimit = 50

  def from(request: Request, pageSize: Int): Either[ApiError, Paging] =
    for {
      limit  <- intParam(request, "limit", pageSize)
      offset <- intParam(request, "offset", 0)
      _      <- check(limit >= 1 && limit <= maxLimit, s"limit must be between 1 and $maxLimit, got $limit")
      _      <- check(offset >= 0, s"offset must not be negative, got $offset")
    } yield Paging(limit, offset)

  private def intParam(request: Request, name: String, default: Int): Either[ApiError, Int] =
    request.query[Int](name) match {
      case Right(value)                     => Right(value)
      case Left(QueryParamError.Missing(_)) => Right(default)
      case Left(malformed)                  => Left(ApiError(malformed.message))
    }

  private def check(ok: Boolean, message: => String): Either[ApiError, Unit] =
    if (ok) Right(()) else Left(ApiError(message))
}

object Demo {
  def get(target: String): Request = Request.get(url(target))

  def post(target: String, json: String): Request =
    Request.post(url(target), Body.fromArray(json.getBytes("UTF-8"), ContentType.`application/json`))

  def show(response: Response): Unit =
    println(s"${response.status.code} ${response.body.text}")

  private def url(target: String): URL =
    URL.parse(target).fold(error => throw new IllegalArgumentException(error), identity)
}
```

The code above:

- `Paging.intParam` turns `QueryParamError.Missing` into the default and every other error into an `ApiError` with the error's own message.
- `Paging.check` adds the range checks. No decoder checks ranges for us, so `Paging.from` does it after decoding.
- `Paging.from` chains the steps in a `for` comprehension: the first `Left` stops it, and that error is what the client sees.
- `Demo` builds requests from strings and prints responses, so we can try code that takes a `Request` without starting a server. A JSON body needs `Body.fromArray` with the `application/json` content type; `Body.fromString` is always `text/plain`.

Let's read the paging of a few request targets, good and bad:

```scala
List("/books", "/books?limit=1&offset=2", "/books?limit=abc", "/books?limit=500", "/books?offset=-1")
  .foreach(target => println(s"$target -> ${Paging.from(Demo.get(target), pageSize = 2)}"))
// /books -> Right(Paging(2,0))
// /books?limit=1&offset=2 -> Right(Paging(1,2))
// /books?limit=abc -> Left(ApiError(Malformed query parameter 'limit' value 'abc': Cannot parse 'abc' as Int))
// /books?limit=500 -> Left(ApiError(limit must be between 1 and 50, got 500))
// /books?offset=-1 -> Left(ApiError(offset must not be negative, got -1))
```

A plain `/books` gets the default page, and every bad value gets its own message.

:::caution
`request.queryOrElse("limit", 20)` looks like a shorter way to write `Paging.intParam`, but it also falls back to the default when the value is malformed, so `?limit=abc` would quietly return the first page. Match on `request.query` when bad input must be rejected.
:::

[How to Validate Query Parameters and Headers and Answer 400](./validate-query-parameters-and-headers.md) goes further: optional parameters that still reject malformed values, range checks, typed headers, and one JSON 400 that names every bad parameter. The [Schema-Based Typed Access](../reference/http-model/schema.md) reference covers `RequestSchemaOps#query`, `RequestSchemaOps#queryAll` and `RequestSchemaOps#queryOrElse`.

## 7. Route Requests

Now we connect requests to the repository. The routes are one function from `Request` to `Response`. It matches the method and the path segments, calls `BookRepo`, and writes JSON with the codecs from section 3. Bad input gets a 400, and a missing book gets a 404, each with an `ApiError` body:

```scala
class Routes(repo: BookRepo, catalog: CatalogConfig) {

  def handle(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, List("books")) =>
        Paging.from(request, catalog.pageSize) match {
          case Right(paging) => Response.json(Json.books.encodeToString(repo.page(paging.limit, paging.offset)))
          case Left(error)   => Routes.error(Status.BadRequest, error)
        }

      case (Method.GET, List("books", id)) =>
        id.toLongOption.flatMap(repo.find) match {
          case Some(book) => Response.json(Json.book.encodeToString(book))
          case None       => Routes.error(Status.NotFound, ApiError(s"no book with id $id"))
        }

      case (Method.POST, List("books")) =>
        Json.newBook.decode(request.body.toArray) match {
          case Right(newBook) => Response.json(Status.Created, Json.book.encodeToString(repo.add(newBook)))
          case Left(error)    => Routes.error(Status.BadRequest, ApiError(error.message))
        }

      case _ =>
        Routes.error(Status.NotFound, ApiError(s"no route for ${request.method} ${request.path}"))
    }
}

object Routes {
  def error(status: Status, error: ApiError): Response =
    Response.json(status, Json.error.encodeToString(error))
}
```

The code above:

- `request.path.segments` is the path split at `/`, so `/books/2` matches `List("books", "2")` and binds `id` to `"2"`.
- `id.toLongOption.flatMap(repo.find)` turns a non-numeric id into the same 404 as an unknown one.
- `Response.json` sets the status, the `application/json` content type and the body in one call. It takes the JSON as a `String`, so we encode with the derived codec first.
- `Json.newBook.decode(request.body.toArray)` decodes the request body. A body that isn't a valid `NewBook` becomes a 400 with the decoder's message.

The routes are an ordinary function, so we can call them directly, with no server and no sockets. First, let's build them on a fresh database:

```scala
val routesDb = new Db(DbConfig(TempDb.file()))
val routes   = new Routes(new BookRepo(routesDb), CatalogConfig(pageSize = 2))
```

Then we send them a few requests, good and bad:

```scala
Demo.show(routes.handle(Demo.get("/books")))
// 200 [{"id":1,"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024,"stock":3,"priceCents":4500},{"id":2,"title":"Programming in Scala","firstAuthor":"Martin Odersky","year":2008,"isbn":"9780981531601","stock":5,"priceCents":3999}]
Demo.show(routes.handle(Demo.get("/books/2")))
// 200 {"id":2,"title":"Programming in Scala","firstAuthor":"Martin Odersky","year":2008,"isbn":"9780981531601","stock":5,"priceCents":3999}
Demo.show(routes.handle(Demo.get("/books/42")))
// 404 {"error":"no book with id 42"}
Demo.show(
  routes.handle(
    Demo.post(
      "/books",
      """{"title":"Scala with Cats","firstAuthor":"Noel Welsh","year":2017,"stock":2,"priceCents":3500}"""
    )
  )
)
// 201 {"id":4,"title":"Scala with Cats","firstAuthor":"Noel Welsh","year":2017,"stock":2,"priceCents":3500}
Demo.show(routes.handle(Demo.get("/books?limit=abc")))
// 400 {"error":"Malformed query parameter 'limit' value 'abc': Cannot parse 'abc' as Int"}
Demo.show(routes.handle(Demo.post("/books", """{"title":"Scala with Cats"}""")))
// 400 {"error":"missing required field \"firstAuthor\" at: ."}

routesDb.close()
// Db: closed
```

Each line shows the status code and the body. The first page holds two books, because the catalog's page size is `2`; the new book comes back with status `201` and its id; and both kinds of bad input get a `400` with a JSON message. This is the whole backend's behavior. Everything that follows adds operations around it: telemetry, a server, wiring and shutdown.

## 8. Log, Trace and Count Requests

A backend that runs unattended needs to leave a record of what it did. The `telemetry` module gives us three global entry points: `log` for log lines, `trace` for spans and `metric` for counters. Instead of adding telemetry calls to every route, we wrap the routes once. `RequestTelemetry#wrap` takes any `Request => Response` function and returns one that runs each request in its own span, writes one structured log line, and adds one to a counter:

```scala

class RequestTelemetry {
  val requests: Counter = metric.counter("bookstore.http.requests")

  def wrap(app: Request => Response): Request => Response = request =>
    trace.span(s"${request.method} ${request.path}") { _ =>
      val response = app(request)
      requests.add(1L, "status" -> response.status.code)
      log.info(
        "Handled request",
        "method" -> request.method.toString,
        "path"   -> request.path.toString,
        "status" -> response.status.code
      )
      response
    }

  def countsByStatus(): List[(Long, Long)] =
    requests.collect() match {
      case MetricData.SumData(points) =>
        points.map(p => p.attributes.get(RequestTelemetry.status).getOrElse(0L) -> p.value).sorted
      case _ => Nil
    }
}

object RequestTelemetry {
  val status: AttributeKey[Long] = AttributeKey.long("status")
}
```

The code above:

- We import `log`, `metric`, `trace` and the few types we need by name. A wildcard `import zio.blocks.telemetry._` would also bring in telemetry's own `Resource` type, which clashes with the `Resource` from the `scope` module that we use in section 10, a known bug in version 0.0.56.
- `metric.counter("bookstore.http.requests")` creates the counter once, when `RequestTelemetry` is built. Each call to `metric.counter` registers a new instrument, so calling it per request would scatter the counts over many counters.
- `trace.span(...)` runs the request inside a span named after the method and the path, and ends the span when the response is ready.
- `requests.add(1L, "status" -> ...)` adds one to the series for the response's status code.
- `log.info(message, key -> value, ...)` writes one log line with structured fields that a log tool can search without parsing the text.
- `RequestTelemetry#countsByStatus` reads the counter back as `(status, count)` pairs, sorted by status.

Let's wrap a fresh set of routes:

```scala
val telemetryDb = new Db(DbConfig(TempDb.file()))
val telemetry   = new RequestTelemetry
val app         = telemetry.wrap(new Routes(new BookRepo(telemetryDb), CatalogConfig(pageSize = 2)).handle)
```

Now we send three requests through the wrapper, and then read the spans and the counter back:

```scala
Demo.show(app(Demo.get("/books/1")))
// {"timeUnixNano":"1794076391996866648","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Handled request"},"attributes":[{"key":"code.filepath","value":{"stringValue":"small-json-backend-from-blocks.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp.RequestTelemetry"}},{"key":"code.function","value":{"stringValue":"wrap"}},{"key":"code.lineno","value":{"intValue":"483"}},{"key":"method","value":{"stringValue":"GET"}},{"key":"path","value":{"stringValue":"/books/1"}},{"key":"status","value":{"intValue":"200"}}],"traceId":"47c4ba0ea44e4fcda5cbc113c95e5876","spanId":"952f256863631ae7"}
// 200 {"id":1,"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024,"stock":3,"priceCents":4500}
Demo.show(app(Demo.get("/books/42")))
// {"timeUnixNano":"1794076391998626954","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Handled request"},"attributes":[{"key":"code.filepath","value":{"stringValue":"small-json-backend-from-blocks.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp.RequestTelemetry"}},{"key":"code.function","value":{"stringValue":"wrap"}},{"key":"code.lineno","value":{"intValue":"483"}},{"key":"method","value":{"stringValue":"GET"}},{"key":"path","value":{"stringValue":"/books/42"}},{"key":"status","value":{"intValue":"404"}}],"traceId":"103af5ea11e43b223902f6b8904e49ce","spanId":"703a3e28f9f4c688"}
// 404 {"error":"no book with id 42"}
Demo.show(app(Demo.get("/books?limit=abc")))
// {"timeUnixNano":"1794076391999281849","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Handled request"},"attributes":[{"key":"code.filepath","value":{"stringValue":"small-json-backend-from-blocks.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp.RequestTelemetry"}},{"key":"code.function","value":{"stringValue":"wrap"}},{"key":"code.lineno","value":{"intValue":"483"}},{"key":"method","value":{"stringValue":"GET"}},{"key":"path","value":{"stringValue":"/books"}},{"key":"status","value":{"intValue":"400"}}],"traceId":"b0602dc17d6868121ce884edf7f85ad5","spanId":"c8c1e7463f522d92"}
// 400 {"error":"Malformed query parameter 'limit' value 'abc': Cannot parse 'abc' as Int"}

println(trace.collectedSpans.map(_.name))
// List(load-inventory, restock-titles, write-report, restock, GET /books/1, GET /books/42, GET /books)
println(telemetry.countsByStatus())
// List((200,1), (400,1), (404,1))

telemetryDb.close()
// Db: closed
```

Each request writes one log line with its fields, just above the response it describes. By default the lines go to standard output; this page shows the time as `<timestamp>`, because it changes on every run. The spans are kept in memory, and `trace.collectedSpans` returns the finished ones, one per request. The counter has one series per status code.

:::caution
The brackets of a log line should hold the source location: the short class name, the method and the line, such as `[RequestTelemetry.wrap:20]`. In version 0.0.56, a class inside a package prints its full name followed by two numbers instead, as in `[(smalljsonbackendfromblocks.RequestTelemetry,27,43).wrap:20]` from the companion example. On this page, the class also sits inside the `repl.MdocSession` wrapper that renders the examples. This is a known bug; the message and the fields are not affected.
:::

The [Telemetry Guide](./telemetry-guide.md) explains how to send logs as JSON, how records flow through processors, and how to set log levels. The tutorial [Instrument a Program with Logs, Traces and Metrics](./instrument-a-program-with-logs-traces-and-metrics.md) teaches the three signals step by step.

## 9. Serve the Routes over HTTP

So far we've called the routes ourselves. To serve real clients, something has to accept TCP connections, parse HTTP, and write responses back. ZIO Blocks has no server, but the JDK has one built in: `com.sun.net.httpserver.HttpServer`. A small adapter, `JdkHttp`, converts each exchange of that server into a `Request`, calls our function, and writes the `Response` back. We copy it from the guide [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md), which explains every line; it is about sixty lines, so it is folded here:

<details>
  <summary>The JdkHttp adapter</summary>

The adapter, as the guide ships it, without its comments:

```scala

object JdkHttp {

  def toRequest(exchange: HttpExchange): Either[Response, Request] = {
    val uri    = exchange.getRequestURI
    val target = uri.getRawPath + Option(uri.getRawQuery).fold("")("?" + _)
    for {
      method <- Method.fromString(exchange.getRequestMethod).toRight(Response(Status.NotImplemented))
      url    <- URL.parse(target).left.map(error => Response.text(Status.BadRequest, error))
    } yield {
      val headers = exchange.getRequestHeaders.asScala.foldLeft(Headers.empty) { case (acc, (name, values)) =>
        values.asScala.foldLeft(acc)((withValue, value) => withValue.add(name, value))
      }
      val contentType =
        headers.get(Header.ContentType).map(_.value).getOrElse(ContentType.`application/octet-stream`)
      val body    = Body.fromArray(exchange.getRequestBody.readAllBytes(), contentType)
      val version = Version.fromString(exchange.getProtocol).getOrElse(Version.`HTTP/1.1`)
      Request(method, url, headers, body, version)
    }
  }

  def writeResponse(exchange: HttpExchange, response: Response): Unit = {
    val out = exchange.getResponseHeaders
    response.headers.toList.foreach { case (name, value) =>
      if (name != "content-length" && name != "transfer-encoding") out.add(name, value)
    }
    val bytes = response.body.toArray
    if (bytes.nonEmpty && !response.headers.has("content-type"))
      out.set("content-type", response.body.contentType.render)
    val code   = response.status.code
    val isHead = exchange.getRequestMethod == "HEAD"
    if (isHead && bytes.nonEmpty) out.set("content-length", bytes.length.toString)
    val noBody = isHead || code == 204 || code == 304 || bytes.isEmpty
    exchange.sendResponseHeaders(code, if (noBody) -1L else bytes.length.toLong)
    if (!noBody) exchange.getResponseBody.write(bytes)
  }

  def handler(app: Request => Response): HttpHandler = exchange =>
    try {
      val response =
        try
          toRequest(exchange) match {
            case Left(error)                                     => error
            case Right(request) if request.method == Method.HEAD => app(request.copy(method = Method.GET))
            case Right(request)                                  => app(request)
          }
        catch {
          case error @ (NonFatal(_) | _: LinkageError) =>
            System.err.println(s"Handler failed: $error")
            Response.text(Status.InternalServerError, "Internal Server Error")
        }
      try writeResponse(exchange, response)
      catch {
        case error @ (NonFatal(_) | _: LinkageError) =>
          System.err.println(s"Writing the response failed: $error")
          if (exchange.getResponseCode == -1) {
            exchange.getResponseHeaders.clear()
            exchange.sendResponseHeaders(Status.InternalServerError.code, -1L)
          }
      }
    } finally exchange.close()

  def start(app: Request => Response, port: Int = 0): HttpServer = {
    val server = HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress, port), 0)
    server.createContext("/", handler(app))
    server.start()
    server
  }

  def withServer[A](app: Request => Response)(use: Int => A): A = {
    val server = start(app)
    try use(server.getAddress.getPort)
    finally server.stop(0)
  }
}
```

</details>

What matters for the bookstore is `JdkHttp.start(app, port)`: it starts a server on the loopback interface that sends every request to our function, turns a non-fatal exception or a `LinkageError` thrown by the function into a 500 response, logs a failure to write the response (and answers 500 when no headers have gone out yet), and returns the JDK's `HttpServer`. To call the server, we also copy the [guide's](./serve-http-model-requests.md) tiny test client. `Client.call(port, method, target, body)` sends one request with the JDK's `HttpClient` and prints the status code, the content type and length, and the body:

<details>
  <summary>The test client</summary>

The client, as the guide ships it, without its comments:

```scala

object Client {
  private val client = HttpClient.newBuilder().version(HttpClient.Version.HTTP_1_1).build()

  def call(port: Int, method: String, target: String, body: String = ""): Unit = {
    val builder = HttpRequest.newBuilder(URI.create(s"http://127.0.0.1:$port$target"))
    val request =
      if (body.isEmpty) builder.method(method, HttpRequest.BodyPublishers.noBody()).build()
      else
        builder
          .method(method, HttpRequest.BodyPublishers.ofString(body))
          .header("content-type", "application/json")
          .build()
    val response                     = client.send(request, HttpResponse.BodyHandlers.ofString())
    def header(name: String): String = response.headers.firstValue(name).orElse("none")
    println(s"$method $target -> ${response.statusCode} [${header("content-type")}, ${header("content-length")} bytes]")
    if (response.body.nonEmpty) println(response.body)
  }
}
```

</details>

The server is our second service with something to close. `HttpService` starts the server with the instrumented routes on the configured port, and stops it in `HttpService#close`:

```scala
class HttpService(routes: Routes, val telemetry: RequestTelemetry, config: ServerConfig) extends AutoCloseable {
  private val server = JdkHttp.start(telemetry.wrap(routes.handle), config.port)
  println("HttpService: started")

  def port: Int = server.getAddress.getPort

  def close(): Unit = {
    server.stop(0)
    println("HttpService: stopped")
  }
}
```

The code above:

- `telemetry.wrap(routes.handle)` is the application the server runs: the routes from section 7 inside the telemetry from section 8.
- `HttpService#port` reads the real port back from the server, because `ServerConfig(port = 0)` lets the operating system pick one.
- `server.stop(0)` stops accepting connections and returns without waiting for requests in flight. A production service would pass a few seconds, so that requests in progress can finish.

Let's build the services by hand and start the server:

```scala
val serverDb = new Db(DbConfig(TempDb.file()))
val service  = new HttpService(new Routes(new BookRepo(serverDb), CatalogConfig(pageSize = 2)), new RequestTelemetry, ServerConfig(port = 0))
```

Now we send two real HTTP requests, and then close everything:

```scala
Client.call(service.port, "GET", "/books/1")
// {"timeUnixNano":"1794076392014304699","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Handled request"},"attributes":[{"key":"code.filepath","value":{"stringValue":"small-json-backend-from-blocks.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp.RequestTelemetry"}},{"key":"code.function","value":{"stringValue":"wrap"}},{"key":"code.lineno","value":{"intValue":"483"}},{"key":"method","value":{"stringValue":"GET"}},{"key":"path","value":{"stringValue":"/books/1"}},{"key":"status","value":{"intValue":"200"}}],"traceId":"d07dc6965a649d382befc113ce47ac32","spanId":"cdb53e60e6233eb4"}
// GET /books/1 -> 200 [application/json, 101 bytes]
// {"id":1,"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024,"stock":3,"priceCents":4500}
Client.call(service.port, "GET", "/books?limit=abc")
// {"timeUnixNano":"1794076392017335509","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Handled request"},"attributes":[{"key":"code.filepath","value":{"stringValue":"small-json-backend-from-blocks.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp.RequestTelemetry"}},{"key":"code.function","value":{"stringValue":"wrap"}},{"key":"code.lineno","value":{"intValue":"483"}},{"key":"method","value":{"stringValue":"GET"}},{"key":"path","value":{"stringValue":"/books"}},{"key":"status","value":{"intValue":"400"}}],"traceId":"4bd2fa45a18c24cbd6fa4a52379ddc38","spanId":"4a46d3c30b89b763"}
// GET /books?limit=abc -> 400 [application/json, 84 bytes]
// {"error":"Malformed query parameter 'limit' value 'abc': Cannot parse 'abc' as Int"}

service.close()
// HttpService: stopped
serverDb.close()
// Db: closed
```

The server writes its log line before the response goes out, so each log line appears just above the client's view of the same request. Notice the closing at the end: we must stop the server before closing the database, or a request could still arrive while the database is closing. Getting that order right by hand is exactly what the next section automates.

## 10. Wire Everything Together and Shut Down

Building the services by hand works, but `main` must know every constructor, call them in the right order, pass each one its settings, and close them in the reverse order. The `scope` module does that job. `Resource.from[HttpService]` reads the constructors of `HttpService` and everything it needs, builds each service once, and registers `AutoCloseable#close` for every service that is `AutoCloseable`. The `config` module's `Config.wire` turns one section of the settings into a value the graph can use:

```scala

object Bookstore {
  def app(settings: ConfigSource): Resource[HttpService] =
    Resource.from[HttpService](
      Wire(settings),
      Config.wire[ServerConfig]("server"),
      Config.wire[DbConfig]("db"),
      Config.wire[CatalogConfig]("catalog")
    )
}
```

The code above:

- `Wire(settings)` puts the layered `ConfigSource` into the graph, so `Config.wire` can read from it.
- `Config.wire[ServerConfig]("server")` reads the `server` section as a `ServerConfig` when a service asks for one; the other two lines do the same for the `db` and `catalog` sections.
- Everything else, `Db`, `BookRepo`, `Routes`, `RequestTelemetry` and `HttpService`, is built from its constructor. We never call `new` on a service.
- `Bookstore.app` is a `def`, not a `val`. A `Resource` built by `Resource.from` and stored in a `val` can be used only once, a known bug in version 0.0.56; a `def` gives each caller a fresh one.

Now let's run the whole backend. The stand-in environment points `db.file` at a fresh temporary file:

```scala
val wiredEnv = ConfigSource.fromMap(Map("db.file" -> TempDb.file()), "env")
```

`Scope.global.scoped` opens a scope, `Scope#allocate` builds the graph inside it, and when the block ends, the scope closes everything it built, last built first:

```scala
val counts = Scope.global.scoped { scope =>

  val backend = allocate(Bookstore.app(Settings.layered(wiredEnv)))
  val port    = $(backend)(_.port)

  Client.call(port, "GET", "/books")
  Client.call(port, "POST", "/books", """{"title":"Scala with Cats","firstAuthor":"Noel Welsh","year":2017,"stock":2,"priceCents":3500}""")
  Client.call(port, "GET", "/books?offset=2")

  $(backend)(_.telemetry.countsByStatus())
}
// Db: opened
// HttpService: started
// {"timeUnixNano":"1794076392031102682","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Handled request"},"attributes":[{"key":"code.filepath","value":{"stringValue":"small-json-backend-from-blocks.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp.RequestTelemetry"}},{"key":"code.function","value":{"stringValue":"wrap"}},{"key":"code.lineno","value":{"intValue":"483"}},{"key":"method","value":{"stringValue":"GET"}},{"key":"path","value":{"stringValue":"/books"}},{"key":"status","value":{"intValue":"200"}}],"traceId":"6825295a6c1f4beca46ff50a8f3f629d","spanId":"00c2f7b4d36bebcd"}
// GET /books -> 200 [application/json, 237 bytes]
// [{"id":1,"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024,"stock":3,"priceCents":4500},{"id":2,"title":"Programming in Scala","firstAuthor":"Martin Odersky","year":2008,"isbn":"9780981531601","stock":5,"priceCents":3999}]
// {"timeUnixNano":"1794076392035132585","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Handled request"},"attributes":[{"key":"code.filepath","value":{"stringValue":"small-json-backend-from-blocks.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp.RequestTelemetry"}},{"key":"code.function","value":{"stringValue":"wrap"}},{"key":"code.lineno","value":{"intValue":"483"}},{"key":"method","value":{"stringValue":"POST"}},{"key":"path","value":{"stringValue":"/books"}},{"key":"status","value":{"intValue":"201"}}],"traceId":"6f9ed933324b0672a6b68d918d77c57b","spanId":"037d882dc9845e21"}
// POST /books -> 201 [application/json, 101 bytes]
// {"id":4,"title":"Scala with Cats","firstAuthor":"Noel Welsh","year":2017,"stock":2,"priceCents":3500}
// {"timeUnixNano":"1794076392037827964","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Handled request"},"attributes":[{"key":"code.filepath","value":{"stringValue":"small-json-backend-from-blocks.md"}},{"key":"code.namespace","value":{"stringValue":"repl.MdocSession$.MdocApp.RequestTelemetry"}},{"key":"code.function","value":{"stringValue":"wrap"}},{"key":"code.lineno","value":{"intValue":"483"}},{"key":"method","value":{"stringValue":"GET"}},{"key":"path","value":{"stringValue":"/books"}},{"key":"status","value":{"intValue":"200"}}],"traceId":"c5b623f94f3a9b51b289204cf3f92564","spanId":"fa03e3e8b10f32b6"}
// GET /books?offset=2 -> 200 [application/json, 247 bytes]
// [{"id":3,"title":"Functional Programming in Scala","firstAuthor":"Paul Chiusano","year":2014,"isbn":"9781617290657","stock":4,"priceCents":4499},{"id":4,"title":"Scala with Cats","firstAuthor":"Noel Welsh","year":2017,"stock":2,"priceCents":3500}]
// HttpService: stopped
// Db: closed
// counts: List[Tuple2[Long, Long]] = List((200L, 2L), (201L, 1L))
println(s"Requests by status: $counts")
// Requests by status: List((200,2), (201,1))
```

Read the output from the top. `Db: opened` comes first, because everything else depends on the database; `HttpService: started` comes last. After the three requests, the scope closes the services in reverse: the server stops before the database closes, so no request can reach a closed database. That's the order we write by hand in section 9, now guaranteed by the graph.

A few details are worth a second look:

- `Scope#allocate` returns the service wrapped as `$[HttpService]`, a value that can't leave the scope. `$(backend)(_.port)` reads a plain value out of it.
- The block returns the request counts, a `List` of numbers. A scope can return plain data, but not a service: the compiler rejects a block that tries to return `backend`, because the service is closed once the block ends.
- The `POST` adds a fourth book, and `?offset=2` then shows the third and the fourth: the page size from the defaults still applies.

Finally, let's see what happens when a setting is wrong. We set the page size to `ten`:

```scala
val badEnv = ConfigSource.fromMap(Map("db.file" -> TempDb.file(), "catalog.pageSize" -> "ten"), "env")
```

Then we try to start the backend with it:

```scala
try {
  Scope.global.scoped { scope =>

    allocate(Bookstore.app(Settings.layered(badEnv)))
    ()
  }
} catch {
  case error: ConfigLoadException => println(error.getMessage)
}
// Db: opened
// Db: closed
// Configuration loading failed with 1 error(s):
//   - Invalid value 'ten' for key 'pageSize' (expected Int) in source 'env|bookstore.yaml|defaults': For input string: "ten"
//
```

The server never starts. The graph builds `Db` before it reaches the catalog settings, and when `Config.wire` fails, the scope closes the database it has already opened. Then the error report names the bad value, the expected type and the sources it searched. Notice that the key in the report is `pageSize`, not `catalog.pageSize`: `Config.wire` reports keys without their section prefix, a known bug in version 0.0.56. Checking every setting at once with `Config.loadWithProvenance[BookstoreConfig]` before building the graph, as the complete example below does, reports full keys and every bad setting at the same time.

The [How to Feed Configuration into a Wire Dependency Graph](./feed-config-into-a-wire-graph.md) guide shows more of this pattern, and [Compile-Time Resource Safety with Scope](./compile-time-resource-safety-with-scope.md) explains scopes, `$` and `Unscoped` in depth.

## 11. Test the Backend

The routes are a plain function, so most tests need no server at all: build the routes on a fresh database file, call `Routes#handle`, and compare the whole status and body. Let's write the helpers the tests use and run one check here on the page:

```scala
def withRoutes[A](use: Routes => A): A = {
  val db = new Db(DbConfig(TempDb.file()))
  try use(new Routes(new BookRepo(db), CatalogConfig(pageSize = 2)))
  finally db.close()
}

def call(routes: Routes, request: Request): (Int, String) = {
  val response = routes.handle(request)
  (response.status.code, response.body.text)
}

withRoutes(call(_, Demo.get("/books/42"))) == (404, """{"error":"no book with id 42"}""")
// Db: opened
// Db: closed
// res33: Boolean = true
```

The code above:

- `withRoutes` gives each test its own database file with the starter catalog, and closes the database even when the test throws, so tests can't see each other's books.
- `call(routes, request)` reduces a response to its status code and its body as text.
- The check compares the full body with `==`. Checking that the body merely contains `42` would still pass if the message, the field name or the JSON shape changed.

The examples project turns these checks into a ZIO Test spec. Four tests call the routes directly; the last one builds the real graph from the settings and sends one request through the server, to prove the pieces are connected:

```scala title="capstone-examples/src/test/scala/smalljsonbackendfromblocks/BookstoreSpec.scala"
package smalljsonbackendfromblocks

/**
 * Build a Small JSON Backend from Blocks — Test the Backend
 *
 * Calls the routes directly against a fresh database file for each test, and
 * sends one request through the real server built from the settings.
 *
 * Run with: cd capstone-examples && sbt "testOnly
 * smalljsonbackendfromblocks.BookstoreSpec"
 */
object BookstoreSpec extends ZIOSpecDefault {

  def withRoutes[A](use: Routes => A): A = {
    val db = new Db(DbConfig(TempDb.file()))
    try use(new Routes(new BookRepo(db), CatalogConfig(pageSize = 2)))
    finally db.close()
  }

  def call(routes: Routes, request: zio.http.Request): (Int, String) = {
    val response = routes.handle(request)
    (response.status.code, response.body.text)
  }

  def spec = suite("Bookstore")(
    test("GET /books returns the first page") {
      val result = withRoutes(call(_, Demo.get("/books?limit=1")))
      assertTrue(
        result == (
          200,
          """[{"id":1,"title":"Zionomicon","firstAuthor":"John A. De Goes","year":2024,"stock":3,"priceCents":4500}]"""
        )
      )
    },
    test("GET /books/{id} returns 404 for a missing book") {
      val result = withRoutes(call(_, Demo.get("/books/42")))
      assertTrue(result == (404, """{"error":"no book with id 42"}"""))
    },
    test("GET /books returns 400 for a malformed limit") {
      val result = withRoutes(call(_, Demo.get("/books?limit=abc")))
      assertTrue(
        result == (400, """{"error":"Malformed query parameter 'limit' value 'abc': Cannot parse 'abc' as Int"}""")
      )
    },
    test("POST /books stores the book and returns it with its id") {
      val body   = """{"title":"Scala with Cats","firstAuthor":"Noel Welsh","year":2017,"stock":2,"priceCents":3500}"""
      val result = withRoutes { routes =>
        call(routes, Demo.post("/books", body))
        call(routes, Demo.get("/books/4"))
      }
      assertTrue(
        result == (
          200,
          """{"id":4,"title":"Scala with Cats","firstAuthor":"Noel Welsh","year":2017,"stock":2,"priceCents":3500}"""
        )
      )
    },
    test("the server built from the settings answers over HTTP") {
      val env    = ConfigSource.fromMap(Map("db.file" -> TempDb.file()), "env")
      val result = Scope.global.scoped { scope =>

        val service = allocate(Bookstore.app(Settings.layered(env)))
        val port    = $(service)(_.port)
        val client  = HttpClient.newHttpClient()
        val request = HttpRequest.newBuilder(URI.create(s"http://127.0.0.1:$port/books/2")).build()
        val reply   = client.send(request, HttpResponse.BodyHandlers.ofString())
        (reply.statusCode, reply.body)
      }
      assertTrue(
        result == (
          200,
          """{"id":2,"title":"Programming in Scala","firstAuthor":"Martin Odersky","year":2008,"isbn":"9780981531601","stock":5,"priceCents":3999}"""
        )
      )
    }
  ) @@ TestAspect.sequential
}
```

Run it from the `capstone-examples` directory with `sbt test`. The [How to Test Database Code Against SQLite](./test-database-code-against-sqlite.md) guide shows more ways to set up and seed test databases.

## 12. Putting It Together

This example combines all the concepts we've learned into a program: it layers the real `BOOKSTORE_*` environment variables over the YAML file over the defaults, prints the startup report (or every bad setting, and stops), builds the whole service graph in one scope, sends a few requests through the real server, and shuts down in reverse order:

```scala title="capstone-examples/src/main/scala/smalljsonbackendfromblocks/CompleteExample.scala" showLineNumbers
package smalljsonbackendfromblocks

/**
 * Build a Small JSON Backend from Blocks — Putting It Together
 *
 * The bookstore backend as a program: settings from `BOOKSTORE_*` environment
 * variables over the YAML file over the defaults, a startup report, the whole
 * service graph in one scope, a few requests through the real server, and a
 * shutdown in reverse order.
 *
 * Run with: cd capstone-examples && sbt "runMain
 * smalljsonbackendfromblocks.CompleteExample"
 */
object CompleteExample {
  def main(args: Array[String]): Unit = {
    // 1. Layer the real environment over the YAML file over the defaults.
    val settings = Settings.layered(Settings.environment)

    // 2. Report the settings in effect, or every bad one, before starting anything.
    Config.loadWithProvenance[BookstoreConfig](settings) match {
      case Right(loaded) => println(loaded.dump())
      case Left(errors)  =>
        errors.foreach(error => println(error.message))
        sys.exit(1)
    }

    // 3. Build the service graph, serve requests, and close it in reverse order.
    Scope.global.scoped { scope =>

      val service = allocate(Bookstore.app(settings))
      val port    = $(service)(_.port)

      Client.call(port, "GET", "/books")
      Client.call(port, "GET", "/books/1")
      Client.call(
        port,
        "POST",
        "/books",
        """{"title":"Scala with Cats","firstAuthor":"Noel Welsh","year":2017,"stock":2,"priceCents":3500}"""
      )
      Client.call(port, "GET", "/books?limit=abc")
    }
  }
}
```

The classes it wires live in the files next to it, one file per section of this tutorial: `Model.scala`, `Settings.scala`, `Storage.scala`, `Paging.scala`, `Routes.scala`, `RequestTelemetry.scala` and `HttpService.scala`, plus the adapter `JdkHttp.scala` and the client `Client.scala`. Its database file is `bookstore.db` in the working directory, so books added in one run are still there in the next; set `BOOKSTORE_DB_FILE` to use another file, or `BOOKSTORE_SERVER_PORT` to pick the port.

## 13. Running the Examples

All examples in this tutorial have corresponding runnable Scala files in the `capstone-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/capstone-examples
```

Each example except the complete one works on a fresh temporary database file, so you can run them in any order and as often as you like. We recommend running them in order to follow the tutorial step by step.

### Model the Books

This example encodes a book to JSON, decodes a request body, and shows the error for a body with a missing field.

<details>
  <summary>capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept1ModelExample.scala</summary>

```scala title="capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept1ModelExample.scala" showLineNumbers
package smalljsonbackendfromblocks

/**
 * Build a Small JSON Backend from Blocks — Model the Books
 *
 * Encodes a book to JSON with the codec derived from its schema, decodes a
 * request body, and shows the error for a body with a missing field.
 *
 * Run with: cd capstone-examples && sbt "runMain
 * smalljsonbackendfromblocks.Concept1ModelExample"
 */
object Concept1ModelExample {
  def main(args: Array[String]): Unit = {
    val zionomicon = Book(1, "Zionomicon", "John A. De Goes", 2024, None, 3, 4500)
    println(Json.book.encodeToString(zionomicon))

    val body = """{"title":"Scala with Cats","firstAuthor":"Noel Welsh","year":2017,"stock":2,"priceCents":3500}"""
    println(Json.newBook.decode(body))

    Json.newBook.decode("""{"title":"Scala with Cats"}""") match {
      case Right(book) => println(book)
      case Left(error) => println(error.message)
    }
  }
}
```

</details>

Run it from the `capstone-examples` directory:

```bash
sbt "runMain smalljsonbackendfromblocks.Concept1ModelExample"
```

### Load the Settings

This example layers a stand-in environment over the YAML file over the defaults, and prints where each setting came from.

<details>
  <summary>capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept2SettingsExample.scala</summary>

```scala title="capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept2SettingsExample.scala" showLineNumbers
package smalljsonbackendfromblocks

/**
 * Build a Small JSON Backend from Blocks — Load the Settings
 *
 * Layers a stand-in environment over the YAML file over the defaults, loads the
 * whole `BookstoreConfig`, and prints where each setting came from.
 *
 * Run with: cd capstone-examples && sbt "runMain
 * smalljsonbackendfromblocks.Concept2SettingsExample"
 */
object Concept2SettingsExample {
  def main(args: Array[String]): Unit = {
    val env      = ConfigSource.fromMap(Map("db.file" -> "/tmp/bookstore-tutorial.db"), "env")
    val settings = Settings.layered(env)

    Config.loadWithProvenance[BookstoreConfig](settings) match {
      case Right(loaded) =>
        println(loaded.value)
        println(loaded.dump())
      case Left(errors) => errors.foreach(error => println(error.message))
    }
  }
}
```

</details>

Run it from the `capstone-examples` directory:

```bash
sbt "runMain smalljsonbackendfromblocks.Concept2SettingsExample"
```

### Store Books in SQLite

This example opens a fresh database with the starter catalog, reads a page and single books, and adds one.

<details>
  <summary>capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept3StorageExample.scala</summary>

```scala title="capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept3StorageExample.scala" showLineNumbers
package smalljsonbackendfromblocks

/**
 * Build a Small JSON Backend from Blocks — Store Books in SQLite
 *
 * Opens a fresh database file, lets `BookRepo` fill the starter catalog, and
 * reads a page, one book, and a book that was just added.
 *
 * Run with: cd capstone-examples && sbt "runMain
 * smalljsonbackendfromblocks.Concept3StorageExample"
 */
object Concept3StorageExample {
  def main(args: Array[String]): Unit = {
    val db   = new Db(DbConfig(TempDb.file()))
    val repo = new BookRepo(db)

    repo.page(limit = 2, offset = 0).foreach(println)
    println(repo.find(3))
    println(repo.find(42))
    println(repo.add(NewBook("Scala with Cats", "Noel Welsh", 2017, None, 2, 3500)))

    db.close()
  }
}
```

</details>

Run it from the `capstone-examples` directory:

```bash
sbt "runMain smalljsonbackendfromblocks.Concept3StorageExample"
```

### Check the Query Parameters

This example reads `limit` and `offset` from good and bad request targets.

<details>
  <summary>capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept4PagingExample.scala</summary>

```scala title="capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept4PagingExample.scala" showLineNumbers
package smalljsonbackendfromblocks

/**
 * Build a Small JSON Backend from Blocks — Check the Query Parameters
 *
 * Reads `limit` and `offset` from a few request targets, valid and invalid.
 *
 * Run with: cd capstone-examples && sbt "runMain
 * smalljsonbackendfromblocks.Concept4PagingExample"
 */
object Concept4PagingExample {
  def main(args: Array[String]): Unit =
    List("/books", "/books?limit=1&offset=2", "/books?limit=abc", "/books?limit=500", "/books?offset=-1")
      .foreach(target => println(s"$target -> ${Paging.from(Demo.get(target), pageSize = 2)}"))
}
```

</details>

Run it from the `capstone-examples` directory:

```bash
sbt "runMain smalljsonbackendfromblocks.Concept4PagingExample"
```

### Route Requests

This example calls the routes directly, with no server, for good and bad requests.

<details>
  <summary>capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept5RoutesExample.scala</summary>

```scala title="capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept5RoutesExample.scala" showLineNumbers
package smalljsonbackendfromblocks

/**
 * Build a Small JSON Backend from Blocks — Route Requests
 *
 * Calls the routes directly, with no server: a page of books, one book, a
 * missing book, a new book, and two bad requests.
 *
 * Run with: cd capstone-examples && sbt "runMain
 * smalljsonbackendfromblocks.Concept5RoutesExample"
 */
object Concept5RoutesExample {
  def main(args: Array[String]): Unit = {
    val db     = new Db(DbConfig(TempDb.file()))
    val routes = new Routes(new BookRepo(db), CatalogConfig(pageSize = 2))

    Demo.show(routes.handle(Demo.get("/books")))
    Demo.show(routes.handle(Demo.get("/books/2")))
    Demo.show(routes.handle(Demo.get("/books/42")))
    Demo.show(
      routes.handle(
        Demo.post(
          "/books",
          """{"title":"Scala with Cats","firstAuthor":"Noel Welsh","year":2017,"stock":2,"priceCents":3500}"""
        )
      )
    )
    Demo.show(routes.handle(Demo.get("/books?limit=abc")))
    Demo.show(routes.handle(Demo.post("/books", """{"title":"Scala with Cats"}""")))

    db.close()
  }
}
```

</details>

Run it from the `capstone-examples` directory:

```bash
sbt "runMain smalljsonbackendfromblocks.Concept5RoutesExample"
```

### Log, Trace and Count Requests

This example wraps the routes with `RequestTelemetry` and prints the log lines, the spans and the request counts.

<details>
  <summary>capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept6TelemetryExample.scala</summary>

```scala title="capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept6TelemetryExample.scala" showLineNumbers
package smalljsonbackendfromblocks

/**
 * Build a Small JSON Backend from Blocks — Log, Trace and Count Requests
 *
 * Wraps the routes with `RequestTelemetry`, sends three requests, and prints
 * the spans and the request counts by status.
 *
 * Run with: cd capstone-examples && sbt "runMain
 * smalljsonbackendfromblocks.Concept6TelemetryExample"
 */
object Concept6TelemetryExample {
  def main(args: Array[String]): Unit = {
    val db        = new Db(DbConfig(TempDb.file()))
    val telemetry = new RequestTelemetry
    val app       = telemetry.wrap(new Routes(new BookRepo(db), CatalogConfig(pageSize = 2)).handle)

    app(Demo.get("/books/1"))
    app(Demo.get("/books/42"))
    app(Demo.get("/books?limit=abc"))

    println(trace.collectedSpans.map(_.name))
    println(telemetry.countsByStatus())

    db.close()
  }
}
```

</details>

Run it from the `capstone-examples` directory:

```bash
sbt "runMain smalljsonbackendfromblocks.Concept6TelemetryExample"
```

### Serve the Routes over HTTP

This example starts `HttpService` by hand, sends two real HTTP requests, and stops it.

<details>
  <summary>capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept7ServerExample.scala</summary>

```scala title="capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept7ServerExample.scala" showLineNumbers
package smalljsonbackendfromblocks

/**
 * Build a Small JSON Backend from Blocks — Serve the Routes over HTTP
 *
 * Starts `HttpService` on a free port, sends two real HTTP requests, and stops
 * the server.
 *
 * Run with: cd capstone-examples && sbt "runMain
 * smalljsonbackendfromblocks.Concept7ServerExample"
 */
object Concept7ServerExample {
  def main(args: Array[String]): Unit = {
    val db      = new Db(DbConfig(TempDb.file()))
    val routes  = new Routes(new BookRepo(db), CatalogConfig(pageSize = 2))
    val service = new HttpService(routes, new RequestTelemetry, ServerConfig(port = 0))

    Client.call(service.port, "GET", "/books/1")
    Client.call(service.port, "GET", "/books?limit=abc")

    service.close()
    db.close()
  }
}
```

</details>

Run it from the `capstone-examples` directory:

```bash
sbt "runMain smalljsonbackendfromblocks.Concept7ServerExample"
```

### Wire Everything Together and Shut Down

This example builds the whole graph from layered settings in a scope, serves three requests, closes the services in reverse order, and then shows the report for a bad setting.

<details>
  <summary>capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept8WiringExample.scala</summary>

```scala title="capstone-examples/src/main/scala/smalljsonbackendfromblocks/Concept8WiringExample.scala" showLineNumbers
package smalljsonbackendfromblocks

/**
 * Build a Small JSON Backend from Blocks — Wire, Serve and Shut Down
 *
 * Builds the whole service graph from layered settings inside a scope, serves a
 * few requests, and closes the services in reverse order. Then shows the report
 * for a bad setting.
 *
 * Run with: cd capstone-examples && sbt "runMain
 * smalljsonbackendfromblocks.Concept8WiringExample"
 */
object Concept8WiringExample {
  def main(args: Array[String]): Unit = {
    val env = ConfigSource.fromMap(Map("db.file" -> TempDb.file()), "env")

    val counts = Scope.global.scoped { scope =>

      val service = allocate(Bookstore.app(Settings.layered(env)))
      val port    = $(service)(_.port)

      Client.call(port, "GET", "/books")
      Client.call(
        port,
        "POST",
        "/books",
        """{"title":"Scala with Cats","firstAuthor":"Noel Welsh","year":2017,"stock":2,"priceCents":3500}"""
      )
      Client.call(port, "GET", "/books?offset=2")

      $(service)(_.telemetry.countsByStatus())
    }
    println(s"Requests by status: $counts")

    val badEnv = ConfigSource.fromMap(Map("db.file" -> TempDb.file(), "catalog.pageSize" -> "ten"), "env")
    try {
      Scope.global.scoped { scope =>

        allocate(Bookstore.app(Settings.layered(badEnv)))
        ()
      }
    } catch {
      case error: ConfigLoadException => println(error.getMessage)
    }
  }
}
```

</details>

Run it from the `capstone-examples` directory:

```bash
sbt "runMain smalljsonbackendfromblocks.Concept8WiringExample"
```

### The Complete Example

This example runs the backend with the real environment, as shown in [Putting It Together](#12-putting-it-together).

Run it from the `capstone-examples` directory:

```bash
sbt "runMain smalljsonbackendfromblocks.CompleteExample"
```

To check that every example compiles without running any of them, use this command from the `capstone-examples` directory:

```bash
sbt compile
```

The tests from section 11 run with:

```bash
sbt test
```

## 14. What You've Learned

In this tutorial, you learned:

- How one derived `Schema` per case class gives the backend its JSON codecs, and how the same case classes map to SQLite rows and typed settings
- How to layer settings from the environment, a YAML file and defaults, give each service its own settings type, and see where every value came from
- How `Resource.from` builds a service graph from constructors and settings sections, and how a scope closes it in reverse order, even when startup fails halfway
- How a plain `Request => Response` function handles routing, JSON bodies and checked query parameters, and how a wrapper adds logs, spans and a request counter around it
- How to serve that function with the JDK's built-in HTTP server through a small adapter, and test it both directly and through the server

You now have a small backend built entirely from ZIO Blocks, and you know where each block's own guide goes deeper.

## 15. Where to Go Next

- **Go to production storage.** [How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md) replaces SQLite with PostgreSQL and a pool that `Db#close` would shut down.
- **Serve a bigger catalog.** [How to Stream Large Query Results](./stream-large-query-results.md) reads a large table as a stream of small row chunks, instead of loading every row into a `List` at once.
- **Test more of the database code.** [How to Test Database Code Against SQLite](./test-database-code-against-sqlite.md) covers fixtures, schema setup and rolling back instead of cleaning up.
- **Go deeper on settings.** [How to Feed Configuration into a Wire Dependency Graph](./feed-config-into-a-wire-graph.md) and [Build a Typed Configuration Loader](./typed-config-loader.md) cover secrets, formats and testing settings.
- **Go deeper on resources.** [Compile-Time Resource Safety with Scope](./compile-time-resource-safety-with-scope.md) explains how scopes keep services from escaping.
- **Protect the endpoints.** [How to Issue and Verify JWTs](./issue-and-verify-jwts.md) shows how to sign and check tokens; [How to Protect an Endpoint with JWT Bearer Tokens](./protect-endpoints-with-jwt.md) shows how to check them on each request.
- **Look up the APIs.** The reference pages for [the HTTP model](../reference/http-model/index.md), [config](../reference/config/index.md), [sql](../reference/sql/index.md), [Resource](../reference/resource-management/resource.md) and [telemetry](../reference/telemetry/index.md) list every operation.

---

## sql(table) Checked Interpolation

> **Note**: Checked form is `StringContext(...).sql(table, ...)(...)`; no `sqlChecked` alias (removed pre-1.0).

`sql` with tables gives you compile-time typo protection for hand-written SQL: it validates table and column identifiers against the `Table[?]` values you hand it, with did-you-mean suggestions. Without tables it still validates quotes/parentheses. The escape hatch `SqlLiteral` lets you splice unchecked raw SQL when needed. It is lint-grade, not a full SQL parser — zero runtime overhead when unused.

## What it does

The `sql` interpolator composes two compile-time checks:

1. Quote/paren validation from `SqlValidator` (unclosed quotes, unbalanced parentheses) — always.
2. Identifier checking via `SqlIdentifierChecker` when tables are provided: tokenizes literal parts respecting `'...'` / `"..."` and `__HOLE__` placeholders, subtracts the SQL keyword/function allowlist (~160 entries, case-insensitive), tracks aliases introduced via `AS alias`, and requires every remaining identifier to be a known table name or column.

Unknown identifiers emit compile-time errors via `report.error` (up to 5 diagnostics), each message containing the identifier and a Levenshtein `<=2` suggestion when close.

## Usage without and with Table

Tables are derived from `Schema` as elsewhere:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

case class Order(id: Int, userId: Int, amount: Double)
object Order { implicit val schema: Schema[Order] = Schema.derived }

val usersTable  = Table.derived[User]   // name "user"
val ordersTable = Table.derived[Order]  // name "order"
```

Without tables, only quote/paren checks run (identifier checking skipped):

```scala
val plain: Frag = sql"SELECT * FROM users WHERE email = ${"a@b.com"}"
```

With tables, identifiers are checked. Pass the relevant tables before the interpolated args — the macro extracts table/column names from their case class fields at compile time:

```scala
val q1: Frag = StringContext("SELECT * FROM user WHERE email = ", "").sql(usersTable)("a@b.com")
val q2: Frag = StringContext("SELECT user.id, order.amount FROM user JOIN order ON user.id = order.user_id").sql(usersTable, ordersTable)()
val q3: Frag = StringContext("SELECT u.id FROM user AS u WHERE u.email = ", "").sql(usersTable)("x")
```

Positive examples compile and render identically to the plain `sql` form:

```scala
q1.sql(SqlDialect.PostgreSQL)
// res0: String = "SELECT * FROM user WHERE email = ?"
q2.sql(SqlDialect.PostgreSQL)
// res1: String = "SELECT user.id, order.amount FROM user JOIN order ON user.id = order.user_id"
q3.sql(SqlDialect.PostgreSQL)
// res2: String = "SELECT u.id FROM user AS u WHERE u.email = ?"
```

Parameters are handled identically regardless of checking — any `DbParam[T]` (or `DbValue`) can be interpolated and becomes a `?` placeholder:

```scala
val email: String = "alice@example.com"
val frag: Frag = StringContext("SELECT * FROM user WHERE email = ", " AND id = ", "").sql(usersTable)(email, 42)
```

## Did-you-mean example

Typos fail at compile time with a suggestion (only when tables are provided):

```scala
StringContext("SELECT * FROM usre").sql(usersTable)()
// error:
// Unknown identifier 'usre' at position 14; did you mean 'user'?
// StringContext("SELECT * FROM usre").sql(usersTable)()
//                                                   ^
```

```
// error: Unknown identifier 'usre' at position 14; did you mean 'user'?
```

```scala
StringContext("SELECT emial FROM user").sql(usersTable)()
// error:
// Unknown identifier 'emial' at position 7; did you mean 'email'?
// StringContext("SELECT emial FROM user").sql(usersTable)()
//                                                       ^
```

```
// error: Unknown identifier 'emial' at position 7; did you mean 'email'?
```

```scala
StringContext("SELECT * FROM user WHERE badcol = 1").sql(usersTable)()
// error:
// Unknown identifier 'badcol' at position 25
// StringContext("SELECT * FROM user WHERE badcol = 1").sql(usersTable)()
//                                                                    ^
```

```
// error: Unknown identifier 'badcol' at position 28
```

Up to 5 diagnostics are reported per interpolator invocation; fix them iteratively.

## Limits — lint-grade, not a parser

The checked `sql(tables)` is intentionally lightweight:

- **No expression typing.** It does not type-check `WHERE` expressions or enforce that `amount > "foo"` is ill-typed.
- **Subquery internals out of scope.** Identifiers inside nested `SELECT` subqueries are checked as flat tokens; column scoping across subqueries is not modeled.
- **Aliases trusted.** Once an alias is introduced via `AS alias` (case-insensitive, quoted aliases supported), further uses of that alias and `alias.column` are trusted without verifying the column belongs to the aliased table.
- **Allowlist coverage.** ~160 SQL keywords, types, and common functions (SELECT/FROM/WHERE/JOIN/COUNT/SUM/AVG/COALESCE/CASE/etc.) are allowlisted case-insensitively. Uncommon dialect-specific functions may need the escape hatch.
- **Keyword-named tables.** Tables named like `order`/`group` overlap the keyword allowlist; they are trusted as keywords unless passed as a `Table`, so ensure such tables are included in the `Table[?]` arguments — known tables/columns are checked before the allowlist, so `order` is recognized when it is in `knownTables`.
- **Table extraction.** Column names are derived from case class fields via `SqlNameMapper.SnakeCase`. Custom renames (`@Modifier.rename`) or complex `Table("name", codec, cols)` constructions are not fully reflected — prefer `Table.derived` for checked queries. In particular, `Table.name` from `@Modifier.config("sql.table_name")` is only reflected in the checker when the `Table` is constructed with an explicit name literal (e.g. `Table.derived[User]("my_table")` or `Table("my_table", ...)`); otherwise the macro falls back to `SnakeCase(typeName)` which may mismatch the runtime name — see `SqlMacros.addTableMeta` limitation.
- **String literals respected.** Identifiers inside `'...'` are never flagged; `"quoted identifiers"` are validated as identifiers.

If a valid query is flagged, use the escape hatch below.

## Escape hatch: `SqlLiteral`

> **Warning**: `SqlLiteral` is spliced verbatim without escaping — never construct it from untrusted input (user data, request params). Use `DbParam`/`?` placeholders for values; reserve `SqlLiteral` for trusted, dialect-specific SQL.

When you need dynamic SQL or a dialect-specific construct that the lint cannot model, use `SqlLiteral` — unchecked raw SQL.

Standalone unchecked fragment (no validation, no `Table` needed):

```scala
val raw: Frag = SqlLiteral("SELECT MY_CUSTOM_FUNC(id) FROM user").toFrag
// also: SqlLiteral.frag("SELECT ...")
```

Splicing raw SQL inside the `sql` interpolator as a verbatim fragment (not a `?` parameter):

```scala
val qRaw: Frag = sql"SELECT ${SqlLiteral("MY_CUSTOM_FUNC(id)")} FROM user"
val qMixed: Frag = sql"SELECT ${SqlLiteral("MY_FUNC()")}, email FROM user WHERE id = ${42}"
```

`Frag` values are also spliced verbatim: `sql"SELECT * FROM (${myFrag}) WHERE id = ${id}"`.

Spliced `SqlLiteral`/`Frag` content is not identifier-checked; the surrounding literal parts still are (when tables are provided). Interpolated values that are not `SqlLiteral`/`Frag` always become `?` placeholders, so dialect-specific SQL must be spliced as `SqlLiteral`/`Frag`, not as a bound parameter.

```scala
qRaw.sql(SqlDialect.PostgreSQL)
// res6: String = "SELECT MY_CUSTOM_FUNC(id) FROM user"
qMixed.sql(SqlDialect.PostgreSQL)
// res7: String = "SELECT MY_FUNC(), email FROM user WHERE id = ?"
```

Prefer checked `sql(tables)` for all hand-written queries; reserve `SqlLiteral` for genuinely dynamic or dialect-specific cases.

A query whose `WHERE` clause depends on which search fields a user filled in does not need `SqlLiteral`: build one `Frag` per condition and join them, as [How to Build a Query from Optional Search Filters](./build-a-query-from-optional-search-filters.md) shows.

## Comparison

| Form | Quote/paren check | Identifier check | Needs `Table` | Use when |
|------|-------------------|------------------|---------------|----------|
| `sql"..."` (no tables) | yes | no | no | Default, no schema coupling |
| `StringContext(...).sql(table)(...)` | yes | yes (first 5) | yes | Hand-written SQL you want typo-protected |
| `SqlLiteral("...")` / spliced `SqlLiteral` / `Frag` | no | no | no | Escape hatch for dynamic/dialect SQL |

Neither form changes `Frag` rendering; all produce the same `Frag(parts, params)` structure and `sql(dialect)` output.

## See also

- `SqlIdentifierChecker` — pure checker core and `DefaultAllowlist`
- `SqlValidator` — quote/paren validation
- `SqlLiteral` — unchecked raw SQL holder
- `Table.derived` — table derivation and `TableNamingPolicy`

---

## How to Use the sql Module from a ZIO Application

In this guide, we connect the ZIO Blocks [sql module](../reference/sql/index.md) to a ZIO application. By the end, you will have a bookstore `BookRepo` service that ZIO wires with `ZLayer`, a sale that writes to the database and calls a payment service inside one transaction, error handling with retries and timeouts, and a ZIO Test spec that gives every test its own database.

We start from a database provided as a layer, then add one capability at a time:

- A `TransactorZIO` provided as a `ZLayer`, for SQLite and for a pooled PostgreSQL database
- A ZIO service that runs its queries on ZIO's blocking thread pool
- A transaction whose body is a ZIO effect, and the rollback when that effect fails
- Typed errors, retries and timeouts with ZIO's own operators
- A ZIO Test layer that gives each test a fresh database
- Reading a large result set without a `ZStream`

This guide builds on [Build Your First Database Repository on SQLite](./your-first-sqlite-repository.md). We use `Table`, `Repo`, `DbCon` and `sql"..."` the way that tutorial introduces them, and assume you know `ZIO`, `ZLayer` and `ZIO.serviceWithZIO`.

## The Problem

The `sql` module is plain blocking Scala: `JdbcTransactor#connect` and `JdbcTransactor#transact` hold a thread until the database answers. A ZIO application runs its fibers on a small pool of threads, one per CPU core, and expects blocking work to move to a separate blocking pool. A JDBC call made on the main pool stalls every fiber scheduled behind it.

A first attempt usually wraps the transactor by hand:

```scala

case class Book(id: Long, title: String, year: Int, stock: Int)
object Book { given Schema[Book] = Schema.derived }

val books      = Repo.derived[Book, Long]
val transactor = JdbcTransactor.fromUrl("jdbc:sqlite:bookstore.db", SqlDialect.SQLite)

// Runs JDBC on ZIO's main thread pool
val allBooks: Task[List[Book]] =
  ZIO.attempt(transactor.connect(books.all))

// Correct pool, but every call site has to remember it
val count: Task[Long] =
  ZIO.attemptBlocking(transactor.connect(books.count))

// No way to call another ZIO effect between two statements of one transaction
def sell(id: Long): Task[Int] =
  ZIO.attemptBlocking(transactor.transact(sql"UPDATE book SET stock = stock - 1 WHERE id = $id".update))
```

Every query needs the right wrapper, and one forgotten `ZIO.attemptBlocking` slows down the whole application without failing a test. The transaction body is synchronous, so a sale that must also charge a customer through a ZIO service can't do both in one transaction.

In this guide, we solve this with the `sql-zio` module. Its `TransactorZIO` returns ZIO effects that already run on the blocking pool, comes as a `ZLayer`, and has `TransactorZIO#transactZIO` for transactions whose body is a ZIO effect.

## Prerequisites

Add `sql-zio` and the SQLite JDBC driver to your `build.sbt`. `sql-zio` brings in the `sql` module and ZIO 2.1.27:

```scala
// Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-sql-zio" % "0.0.56"

// Scala 3.9 and newer, which also needs -experimental
libraryDependencies += "dev.zio" %% "zio-blocks-next-sql-zio" % "0.0.56"
scalacOptions += "-experimental"

// The SQLite JDBC driver, for every Scala version
libraryDependencies += "org.xerial" % "sqlite-jdbc" % "3.53.4.0"
```

`sql` and `sql-zio` are published for Scala 3 only and run on the JVM. Run your application on JDK 21 or newer.

We import ZIO, the ZIO integration and the `sql` module:

```scala

```

:::warning[Put `import zio.blocks.sql.*` last]
Once `sql-zio` is on the classpath, `import zio.blocks.sql.*` brings the subpackage `zio.blocks.sql.zio` into scope under the name `zio`. After it, `zio` no longer means the root `zio` package, so every later import that starts with `zio`, such as `import zio.blocks.sql.zio.*` or `import zio.blocks.schema.Schema`, fails with `value blocks is not a member of zio.blocks.sql.zio`, and so does a fully qualified name such as `zio.Duration`. This also breaks existing files that compiled before you added `sql-zio`. Put `import zio.blocks.sql.*` after every other `zio` import, as above, or write `_root_.zio` where you need the root package after it.
:::

This page runs its effects inside mdoc, so we use a small helper that runs an effect and returns its result. In an application, `ZIOAppDefault#run` does this for you:

```scala
def runDemo[A](effect: ZIO[Any, Any, A]): A =
  Unsafe.unsafe { implicit unsafe =>
    Runtime.default.unsafe.run(effect).getOrThrowFiberFailure()
  }
```

## The Bookstore Model

We keep a bookstore's stock in a `book` table. Each `Book` has an ID, a title, the year it was published and the copies on the shelf. The stock numbers are made up:

```scala
case class Book(id: Long, title: String, year: Int, stock: Int)

object Book {
  given Schema[Book] = Schema.derived
}

val books: Repo[Book, Long] = Repo.derived[Book, Long]

val shelf = List(
  Book(1L, "Zionomicon", 2024, 5),
  Book(2L, "Programming in Scala", 2008, 3),
  Book(3L, "Functional Programming in Scala", 2014, 4)
)

def createBookTable()(using DbCon): Unit =
  sql"""CREATE TABLE book (
    id INTEGER PRIMARY KEY,
    title TEXT NOT NULL,
    year INTEGER NOT NULL,
    stock INTEGER NOT NULL
  )""".update
```

We write the `CREATE TABLE` statement by hand instead of using `Table#createTable`, because the derived statement has no `PRIMARY KEY`, as [How to Test Database Code Against SQLite](./test-database-code-against-sqlite.md#create-the-schema-from-code) explains. `createBookTable` takes a `(using DbCon)`, so it runs inside any `TransactorZIO#connect` or `TransactorZIO#transact` block.

## Provide the Database as a ZLayer

ZIO applications get their dependencies from layers, so the database is a layer too. `TransactorZIO.layer` takes a JDBC URL and a dialect and returns a `ZLayer[Any, Nothing, TransactorZIO]`. We point it at a temporary SQLite file:

```scala
val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
dbFile.toFile.deleteOnExit()

val transactorLayer: ZLayer[Any, Nothing, TransactorZIO] =
  TransactorZIO.layer(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
```

The layer can't fail because it opens nothing: `TransactorZIO` opens a new JDBC connection for each `TransactorZIO#connect` or `TransactorZIO#transact` call and closes it afterwards. A wrong URL shows up at the first query, not at startup.

:::warning
Don't use `jdbc:sqlite::memory:` here. Each new connection to `:memory:` gets its own empty database, so the table we create in one call is gone in the next. Use a file, as above.
:::

We create the table and put the books on the shelf. `TransactorZIO#connect` runs its synchronous body on ZIO's blocking thread pool and returns a `Task`:

```scala
runDemo {
  ZIO.serviceWithZIO[TransactorZIO] { transactor =>
    transactor.connect {
      createBookTable()
      shelf.foreach(books.insert)
      books.count
    }
  }.provide(transactorLayer)
}
// res2: Long = 3L
```

In production, a PostgreSQL database is usually reached through a connection pool. A pool is a `javax.sql.DataSource` that ZIO opens at startup and closes at shutdown, which is what `ZLayer.scoped` with `ZIO.acquireRelease` expresses. `TransactorZIO.fromDataSource` then builds the transactor on the pool. The pool's settings are the ones from [How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md):

```scala

def poolConfig(): HikariConfig = {
  val config = new HikariConfig()
  config.setJdbcUrl(sys.env.getOrElse("BOOKSTORE_DB_URL", "jdbc:postgresql://localhost:5432/bookstore"))
  config.setUsername(sys.env.getOrElse("BOOKSTORE_DB_USER", "bookstore"))
  config.setPassword(sys.env("BOOKSTORE_DB_PASSWORD"))
  config.setMaximumPoolSize(10)
  config
}

val poolLayer: ZLayer[Any, Throwable, DataSource] =
  ZLayer.scoped {
    ZIO.acquireRelease(ZIO.attemptBlocking(new HikariDataSource(poolConfig())))(pool =>
      ZIO.attemptBlocking(pool.close()).orDie
    )
  }

val postgresTransactorLayer: ZLayer[DataSource, Nothing, TransactorZIO] =
  ZLayer.fromFunction((pool: DataSource) => TransactorZIO.fromDataSource(pool, SqlDialect.PostgreSQL))

val databaseLayer: ZLayer[Any, Throwable, TransactorZIO] =
  poolLayer >>> postgresTransactorLayer
```

`HikariDataSource` connects in its constructor, so `databaseLayer` fails at startup when the database is unreachable, and the pool closes when the application stops. The rest of this guide only depends on `TransactorZIO`, so swapping `transactorLayer` for `databaseLayer` is the only change it needs to run on PostgreSQL.

`sql-zio` also adds `JdbcTransactor.postgresLayer` and `JdbcTransactor.sqliteLayer`, which turn a `DataSource` layer into a layer of the synchronous `Transactor`. They are extension methods defined in `zio.blocks.sql.zio`, so they need `import zio.blocks.sql.zio.*`. A `Transactor` runs its body on the calling thread, so you would have to wrap each call in `ZIO.attemptBlocking` yourself; prefer `TransactorZIO` in ZIO code.

## Write a ZIO Service on the Transactor

The rest of the application shouldn't see SQL. We hide it behind a `BookRepo` service whose methods return ZIO effects:

```scala
trait BookRepo {
  def all: Task[List[Book]]
  def find(id: Long): Task[Option[Book]]
  def add(book: Book): Task[Unit]
}

final case class BookRepoLive(transactor: TransactorZIO) extends BookRepo {
  def all: Task[List[Book]] =
    transactor.connect(books.all)

  def find(id: Long): Task[Option[Book]] =
    transactor.connect(books.find(id).toOption)

  def add(book: Book): Task[Unit] =
    transactor.transact(books.insert(book)).unit
}

object BookRepoLive {
  val layer: ZLayer[TransactorZIO, Nothing, BookRepo] =
    ZLayer.fromFunction(BookRepoLive(_))
}
```

`BookRepoLive` takes the transactor as a constructor parameter, and `ZLayer.fromFunction` turns that constructor into a layer that needs a `TransactorZIO`. Each method is one `TransactorZIO#connect` or `TransactorZIO#transact` call, so each one runs on the blocking pool and fails with the JDBC exception as a `Task` error. `TransactorZIO#transact` turns off auto-commit, commits when its body returns and rolls back when it throws.

Callers ask ZIO for the service and provide both layers:

```scala
runDemo {
  ZIO.serviceWithZIO[BookRepo] { repo =>
    repo.add(Book(4L, "Scala with Cats", 2020, 2)) *> repo.all
  }.provide(BookRepoLive.layer, transactorLayer)
}.foreach(println)
// Book(1,Zionomicon,2024,5)
// Book(2,Programming in Scala,2008,3)
// Book(3,Functional Programming in Scala,2014,4)
// Book(4,Scala with Cats,2020,2)
```

## Mix Database Writes with Other Effects in One Transaction

`TransactorZIO#transact` is enough while the whole transaction is SQL. A sale is not: it lowers the stock and charges the customer, and the charge is a ZIO service. If the charge fails, the stock change must be undone. That calls for `TransactorZIO#transactZIO`, whose body is a ZIO effect: it opens a connection, turns off auto-commit, runs the effect, then commits if the effect succeeds and rolls back if it fails.

We model the payment provider as a ZIO service that declines one customer:

```scala
final case class PaymentDeclined(customer: String) extends Exception(s"Payment declined for $customer")
final case class OutOfStock(bookId: Long)          extends Exception(s"Book $bookId is out of stock")

trait Payments {
  def charge(customer: String, copies: Int): IO[PaymentDeclined, Unit]
}

val paymentsLayer: ZLayer[Any, Nothing, Payments] =
  ZLayer.succeed(new Payments {
    def charge(customer: String, copies: Int): IO[PaymentDeclined, Unit] =
      if (customer == "mallory") ZIO.fail(PaymentDeclined(customer)) else ZIO.unit
  })
```

The sale lowers the stock, fails with `OutOfStock` if there weren't enough copies, then charges the customer, all inside one `TransactorZIO#transactZIO`:

```scala
def sell(bookId: Long, copies: Int, customer: String): ZIO[TransactorZIO & Payments, Throwable, Unit] =
  ZIO.serviceWithZIO[TransactorZIO] { transactor =>
    transactor.transactZIO {
      for {
        updated <- ZIO.attemptBlocking(
                     sql"UPDATE book SET stock = stock - $copies WHERE id = $bookId AND stock >= $copies".update
                   )
        _       <- ZIO.fail(OutOfStock(bookId)).when(updated == 0)
        _       <- ZIO.serviceWithZIO[Payments](_.charge(customer, copies))
      } yield ()
    }
  }

def stockOf(bookId: Long): ZIO[TransactorZIO, Throwable, Int] =
  ZIO.serviceWithZIO[TransactorZIO](
    _.connect(sql"SELECT stock FROM book WHERE id = $bookId".queryOne[Int].getOrElse(0))
  )
```

Alice buys two copies of Zionomicon and Mallory tries to buy one:

```scala
runDemo {
  (for {
    before  <- stockOf(1L)
    alice   <- sell(1L, 2, "alice").either
    mallory <- sell(1L, 1, "mallory").either
    after   <- stockOf(1L)
  } yield {
    println(s"Stock before: $before")
    println(s"Alice: $alice")
    println(s"Mallory: ${mallory.left.map(_.getMessage)}")
    println(s"Stock after: $after")
  }).provide(transactorLayer, paymentsLayer)
}
// Stock before: 5
// Alice: Right(())
// Mallory: Left(Payment declined for mallory)
// Stock after: 3
```

Alice's sale committed and took two copies. Mallory's `UPDATE` ran, but the declined charge failed the body, so `TransactorZIO#transactZIO` rolled the `UPDATE` back and the stock only dropped by Alice's two copies. The error type is `Throwable` here because both of our errors extend `Exception`; in general `TransactorZIO#transactZIO` returns `ZIO[R, E | Throwable, A]`, adding `Throwable` for the JDBC errors to whatever error type the body has.

Inside `TransactorZIO#transactZIO`, wrap every statement in `ZIO.attemptBlocking`. `TransactorZIO#transactZIO` runs only the connection setup, the commit and the rollback on the blocking pool, and the body's effect runs on whatever thread the fiber is on. `ZIO.attemptBlocking` moves the statement to the blocking pool and turns its exception into a typed failure that triggers the rollback.

:::danger[Two ways `TransactorZIO#transactZIO` 0.0.56 commits instead of rolling back]
A statement written directly in the body, such as `transactZIO { sql"...".update; ZIO.unit }`, runs while the body's effect is being built, before `TransactorZIO#transactZIO` watches it for failures. If such a statement throws, `TransactorZIO#transactZIO` does not roll back, and the statements before it in the same body are committed. Wrapping each statement in `ZIO.attemptBlocking`, as above, avoids this.

Interrupting a running `TransactorZIO#transactZIO` from outside, for example with `ZIO#timeout`, `ZIO#race` or by interrupting its fiber, also commits the statements that already ran. Don't put `ZIO#timeout` around a `TransactorZIO#transactZIO`; put a time limit on the slow effect inside the body with `ZIO#timeoutFail`, which fails the body and so rolls the transaction back, as shown in [Limit How Long an Effect Waits](#limit-how-long-an-effect-waits).
:::

`TransactorZIO#transactZIO` keeps the driver's default isolation level. When a transaction needs a specific one, pass it with the two-argument overload, `transactor.transactZIO(TransactionIsolation.Serializable, readOnly = false) { ... }`. [SQL Transactions: Isolation, Read-Only and Savepoints](./sql-transactions.md) explains the levels.

Operations that belong to one transaction must also share its connection: every `TransactorZIO#connect`, `TransactorZIO#transact`, `TransactorZIO#connectZIO` and `TransactorZIO#transactZIO` call takes its own connection. Calling one from inside another, for example a service method that calls `BookRepo#add` while it is inside its own `TransactorZIO#transactZIO`, opens a second connection while the first one still holds its locks. On SQLite, the inner write waits for the outer transaction and fails with `SQLITE_BUSY` after the busy timeout; on PostgreSQL, enough concurrent callers doing this use up the pool and wait for each other.

Instead, write the operations that should share a transaction as functions with a `(using DbCon)` parameter, like `createBookTable` above, and call them inside one `TransactorZIO#transactZIO`:

```scala
def restock(bookId: Long, copies: Int)(using DbCon): Int =
  sql"UPDATE book SET stock = stock + $copies WHERE id = $bookId".update

def restockAll(copies: Int): ZIO[TransactorZIO, Throwable, Int] =
  ZIO.serviceWithZIO[TransactorZIO](_.transactZIO {
    ZIO.foreach(List(1L, 2L, 3L))(bookId => ZIO.attemptBlocking(restock(bookId, copies))).map(_.sum)
  })
```

Both `restock` calls use the `DbTx` that `TransactorZIO#transactZIO` passes to its body, so they run on one connection and commit together.

## Handle Errors, Retries and Timeouts the ZIO Way

Every `TransactorZIO` method returns an ordinary ZIO effect, so ZIO's error operators work on database calls like on any other effect. This section shows the three you need most.

### Turn Database Errors into Domain Errors

A JDBC failure arrives as a `java.sql.SQLException` in the error channel. Callers usually care about a few specific cases, such as a duplicate key, and should see them as their own error type. With SQLite, `SQLiteException#getResultCode` tells the cases apart; on PostgreSQL, check `SQLException#getSQLState`, which is `"23505"` for a unique violation:

```scala

final case class DuplicateBook(id: Long) extends Exception(s"Book $id already exists")

def addNew(book: Book): ZIO[BookRepo, Throwable, Unit] =
  ZIO.serviceWithZIO[BookRepo](_.add(book)).catchSome {
    case e: SQLiteException if e.getResultCode == SQLiteErrorCode.SQLITE_CONSTRAINT_PRIMARYKEY =>
      ZIO.fail(DuplicateBook(book.id))
  }
```

Adding Zionomicon a second time fails with the domain error, not the driver's exception:

```scala
runDemo {
  addNew(Book(1L, "Zionomicon", 2024, 1)).either.provide(BookRepoLive.layer, transactorLayer)
}
// res6: Either[Throwable, Unit] = Left(DuplicateBook(1L))
```

### Retry Transient Failures

Some failures go away on their own: the database is restarting, the network dropped a connection, or another writer held a lock. `ZIO#retry` reruns the effect with a `Schedule`. Retrying a whole `TransactorZIO#connect` or `TransactorZIO#transact` is safe, because each attempt takes a new connection and a failed `TransactorZIO#transact` has already rolled back.

To see a retry without a real outage, we build a `TransactorZIO` whose connection factory fails twice before it connects, the way a database that is still starting up would:

```scala
val attempts = new java.util.concurrent.atomic.AtomicInteger(0)

val startingUp = new TransactorZIO(
  () =>
    if (attempts.incrementAndGet() <= 2) throw new java.sql.SQLException("Connection refused")
    else java.sql.DriverManager.getConnection(s"jdbc:sqlite:$dbFile"),
  SqlDialect.SQLite
)
```

We retry up to five times, waiting 10 milliseconds before the first retry and twice as long before each next one:

```scala
runDemo {
  startingUp
    .connect(books.count)
    .retry(Schedule.exponential(10.millis) && Schedule.recurs(5))
    .map(count => println(s"Counted $count books after ${attempts.get} attempts"))
}
// Counted 4 books after 3 attempts
```

Retry only what is safe to repeat. A `TransactorZIO#transactZIO` body that calls a payment service would charge the customer again on every retry; retry the database part inside the body instead, or make the charge idempotent.

### Limit How Long an Effect Waits

`ZIO#timeout` interrupts an effect that takes too long, but interruption can't stop a JDBC call that is already running. `TransactorZIO` runs statements with `ZIO.attemptBlocking`, which doesn't interrupt its thread, so the fiber waits until the statement finishes and only then reports the timeout. We time a query that counts to three million inside SQLite, with a timeout of 10 milliseconds:

```scala
val slowCount =
  sql"WITH RECURSIVE n(x) AS (SELECT 1 UNION ALL SELECT x + 1 FROM n WHERE x < 3000000) SELECT count(*) FROM n"

def timed[A](effect: Task[A]): Task[(A, Duration)] =
  effect.timed.map(_.swap)
```

The timeout reports `None`, but only after the whole query ran:

```scala
runDemo {
  ZIO.serviceWithZIO[TransactorZIO] { transactor =>
    timed(transactor.connect(slowCount.queryOne[Long]).timeout(10.millis)).map { case (result, elapsed) =>
      println(s"Result: $result")
      println(s"Waited longer than the timeout: ${elapsed.toMillis > 100}")
    }
  }.provide(transactorLayer)
}
// Result: None
// Waited longer than the timeout: true
```

When the caller must get its answer on time, add `ZIO#disconnect` before the timeout. The fiber then stops waiting at the deadline, while the statement finishes in the background and still holds its connection until it does:

```scala
runDemo {
  ZIO.serviceWithZIO[TransactorZIO] { transactor =>
    timed(transactor.connect(slowCount.queryOne[Long]).disconnect.timeout(10.millis)).map { case (result, elapsed) =>
      println(s"Result: $result")
      println(s"Returned before the query finished: ${elapsed.toMillis < 100}")
    }
  }.provide(transactorLayer)
}
// Result: None
// Returned before the query finished: true
```

To stop the statement itself, set a limit in the database, such as PostgreSQL's `statement_timeout` on the pool's connections.

Inside a `TransactorZIO#transactZIO`, put the time limit on the non-database effect with `ZIO#timeoutFail`. The timeout becomes a typed failure of the body, and the transaction rolls back. Here the payment provider hangs, so the sale fails after 50 milliseconds and the stock stays as it was:

```scala
val hangingPayments: ZLayer[Any, Nothing, Payments] =
  ZLayer.succeed(new Payments {
    def charge(customer: String, copies: Int): IO[PaymentDeclined, Unit] = ZIO.never
  })

def sellWithin(bookId: Long, copies: Int, customer: String): ZIO[TransactorZIO & Payments, Throwable, Unit] =
  ZIO.serviceWithZIO[TransactorZIO] { transactor =>
    transactor.transactZIO {
      for {
        _ <- ZIO.attemptBlocking(sql"UPDATE book SET stock = stock - $copies WHERE id = $bookId".update)
        _ <- ZIO
               .serviceWithZIO[Payments](_.charge(customer, copies))
               .timeoutFail(new java.util.concurrent.TimeoutException("Payment took too long"))(50.millis)
      } yield ()
    }
  }
```

Bob's payment never answers:

```scala
runDemo {
  (for {
    before <- stockOf(2L)
    result <- sellWithin(2L, 1, "bob").either
    after  <- stockOf(2L)
  } yield {
    println(s"Sale: $result")
    println(s"Stock before: $before, after: $after")
  }).provide(transactorLayer, hangingPayments)
}
// Sale: Left(java.util.concurrent.TimeoutException: Payment took too long)
// Stock before: 3, after: 3
```

## Test Services Against a Fresh Database

A ZIO Test spec gets its services from layers, so a test database is a layer that creates a temporary SQLite file, builds the schema and seeds the books, and deletes the file when the test ends. `ZLayer.scoped` with `ZIO.acquireRelease` ties the file's lifetime to the layer:

```scala

val testTransactorLayer: ZLayer[Any, Throwable, TransactorZIO] =
  ZLayer.scoped {
    for {
      file      <- ZIO.acquireRelease(ZIO.attemptBlocking(Files.createTempFile("bookstore-test", ".db")))(file =>
                     ZIO.attemptBlocking(Files.deleteIfExists(file)).orDie
                   )
      transactor = TransactorZIO.fromUrl(s"jdbc:sqlite:$file", SqlDialect.SQLite)
      _         <- transactor.connect {
                     createBookTable()
                     shelf.foreach(books.insert)
                   }
    } yield transactor
  }
```

Each time ZIO builds the layer, it gets a new file. Two "tests" that each add a book both start from the same three books:

```scala
List("first test", "second test").foreach { name =>
  val size = runDemo {
    ZIO.serviceWithZIO[BookRepo] { repo =>
      repo.add(Book(10L, "Scala with Cats", 2020, 2)) *> repo.all.map(_.size)
    }.provide(BookRepoLive.layer, testTransactorLayer)
  }
  println(s"$name sees $size books")
}
// first test sees 4 books
// second test sees 4 books
```

In a spec, `Spec#provide` builds the layer once for each test, so every test gets its own database, and `Spec#provideShared` builds it once for the whole suite. The companion spec below uses `Spec#provide`, so its tests can run in any order and in parallel:

<details>
<summary><code>BookRepoSpec.scala</code></summary>

```scala title="sql-examples/src/test/scala/sqlfromzio/BookRepoSpec.scala" showLineNumbers
package sqlfromzio

object BookRepoSpec extends ZIOSpecDefault {

  val testTransactorLayer: ZLayer[Any, Throwable, TransactorZIO] =
    ZLayer.scoped {
      for {
        file      <- ZIO.acquireRelease(ZIO.attemptBlocking(Files.createTempFile("bookstore-test", ".db")))(file =>
                       ZIO.attemptBlocking(Files.deleteIfExists(file)).orDie
                     )
        transactor = TransactorZIO.fromUrl(s"jdbc:sqlite:$file", SqlDialect.SQLite)
        _         <- transactor.connect {
                       createBookTable()
                       shelf.foreach(books.insert)
                     }
      } yield transactor
    }

  def spec: Spec[Any, Throwable] =
    suite("BookRepoSpec")(
      test("starts with the three seeded books") {
        for {
          repo <- ZIO.service[BookRepo]
          all  <- repo.all
        } yield assertTrue(all == shelf)
      },
      test("first test adds a book and sees four books") {
        for {
          repo <- ZIO.service[BookRepo]
          _    <- repo.add(Book(10L, "Scala with Cats", 2020, 2))
          all  <- repo.all
        } yield assertTrue(all.size == 4)
      },
      test("second test adds the same ID and still sees four books") {
        for {
          repo <- ZIO.service[BookRepo]
          _    <- repo.add(Book(10L, "Scala with Cats", 2020, 2))
          all  <- repo.all
        } yield assertTrue(all.size == 4)
      },
      test("find returns a seeded book") {
        for {
          repo  <- ZIO.service[BookRepo]
          found <- repo.find(1L)
        } yield assertTrue(found.map(_.title) == Some("Zionomicon"))
      }
    ).provide(BookRepoLive.layer, testTransactorLayer)
}
```

</details>

[How to Test Database Code Against SQLite](./test-database-code-against-sqlite.md) covers the same ideas for the synchronous API, including fixtures and rollback-only tests.

## Read a Large Result Set

`Frag#queryStream` and `Frag#queryChunked` read a result set in chunks as a ZIO Blocks `Stream`, without loading every row into memory. `sql-zio` has no bridge to `ZStream`, and the stream must be consumed while its connection is open. So we run it inside `TransactorZIO#connect` and reduce it there. A terminal operation such as `Stream#runFold` returns an `Either`, and `ZIO#absolve` turns its `Left` into a failure:

```scala
runDemo {
  ZIO.serviceWithZIO[TransactorZIO] { transactor =>
    transactor.connect {
      sql"SELECT id, title, year, stock FROM book"
        .queryChunked[Book](2)
        .runFold(0)((total, chunk) => total + chunk.foldLeft(0)(_ + _.stock))
    }.absolve
  }.provide(transactorLayer)
}
// res12: Int = 12
```

The fold reads two rows at a time and adds up the stock across the shelf. The whole fold runs inside `TransactorZIO#connect`, so it runs on the blocking pool. When each row needs a ZIO effect, such as publishing it to a queue, collect the rows you need inside `TransactorZIO#connect` and run the effects after it returns, or read the table in pages with `Repo#pageAfter`.

## Putting It Together

The complete example below builds the bookstore application with `ZIOAppDefault`. It provides the transactor, the `BookRepo` service and the payment service as layers, sells books in `TransactorZIO#transactZIO` transactions, and prints the stock after a declined payment:

<details>
<summary><code>CompleteExample.scala</code></summary>

```scala title="sql-examples/src/main/scala/sqlfromzio/CompleteExample.scala" showLineNumbers
package sqlfromzio

/**
 * How to Use the sql Module from a ZIO Application — Complete Example
 *
 * A bookstore application: layers for the transactor, the `BookRepo` service
 * and the payment service, sales in `transactZIO` transactions, and the stock
 * after a declined payment.
 *
 * Run with: cd sql-examples && sbt "runMain sqlfromzio.CompleteExample"
 */
object CompleteExample extends ZIOAppDefault {

  val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
  dbFile.toFile.deleteOnExit()

  val transactorLayer: ZLayer[Any, Nothing, TransactorZIO] =
    TransactorZIO.layer(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

  def sell(bookId: Long, copies: Int, customer: String): ZIO[TransactorZIO & Payments, Throwable, Unit] =
    ZIO.serviceWithZIO[TransactorZIO] { transactor =>
      transactor.transactZIO {
        for {
          updated <- ZIO.attemptBlocking(
                       sql"UPDATE book SET stock = stock - $copies WHERE id = $bookId AND stock >= $copies".update
                     )
          _ <- ZIO.fail(OutOfStock(bookId)).when(updated == 0)
          _ <- ZIO.serviceWithZIO[Payments](_.charge(customer, copies))
        } yield ()
      }
    }

  val program: ZIO[TransactorZIO & BookRepo & Payments, Throwable, Unit] =
    for {
      _ <- ZIO.serviceWithZIO[TransactorZIO](_.connect {
             createBookTable()
             shelf.foreach(books.insert)
           })
      _       <- ZIO.serviceWithZIO[BookRepo](_.add(Book(4L, "Scala with Cats", 2020, 2)))
      _       <- sell(1L, 2, "alice")
      mallory <- sell(1L, 1, "mallory").either
      _       <- Console.printLine(s"Mallory: $mallory")
      all     <- ZIO.serviceWithZIO[BookRepo](_.all)
      _       <- ZIO.foreachDiscard(all)(book => Console.printLine(s"${book.title}: ${book.stock} in stock"))
    } yield ()

  val run =
    program.provide(transactorLayer, BookRepoLive.layer, paymentsLayer)
}
```

</details>

## Running the Examples

The companion examples live in the `sql-examples` build, which uses the published ZIO Blocks artifacts. Clone the repository and change into it:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/sql-examples
```

Every example uses a temporary SQLite file and deletes it when it exits, so you can run them in any order.

The examples share the bookstore model from [The Bookstore Model](#the-bookstore-model), defined once in `Bookstore.scala`:

<details>
<summary><code>Bookstore.scala</code>: the shared <code>Book</code> model, <code>books</code>, <code>shelf</code>, <code>createBookTable</code> and <code>stockOf</code></summary>

```scala title="sql-examples/src/main/scala/sqlfromzio/Bookstore.scala" showLineNumbers
package sqlfromzio

/**
 * How to Use the sql Module from a ZIO Application — the shared bookstore model
 *
 * The `Book` model, its `Repo`, the books on the shelf, the hand-written
 * `CREATE TABLE` statement and a stock lookup. The examples in this package
 * import them.
 */
case class Book(id: Long, title: String, year: Int, stock: Int)

object Book {
  given Schema[Book] = Schema.derived
}

val books: Repo[Book, Long] = Repo.derived[Book, Long]

val shelf = List(
  Book(1L, "Zionomicon", 2024, 5),
  Book(2L, "Programming in Scala", 2008, 3),
  Book(3L, "Functional Programming in Scala", 2014, 4)
)

def createBookTable()(using DbCon): Unit =
  sql"""CREATE TABLE book (
    id INTEGER PRIMARY KEY,
    title TEXT NOT NULL,
    year INTEGER NOT NULL,
    stock INTEGER NOT NULL
  )""".update

def stockOf(bookId: Long): ZIO[TransactorZIO, Throwable, Int] =
  ZIO.serviceWithZIO[TransactorZIO](
    _.connect(sql"SELECT stock FROM book WHERE id = $bookId".queryOne[Int].getOrElse(0))
  )
```

</details>

To check that every example compiles without running any of them:

```bash
sbt compile
```

<details>
<summary><code>BookRepoLayerExample.scala</code>: provide the transactor and the <code>BookRepo</code> service as layers</summary>

```scala title="sql-examples/src/main/scala/sqlfromzio/BookRepoLayerExample.scala" showLineNumbers
package sqlfromzio

trait BookRepo {
  def all: Task[List[Book]]
  def find(id: Long): Task[Option[Book]]
  def add(book: Book): Task[Unit]
}

final case class BookRepoLive(transactor: TransactorZIO) extends BookRepo {
  def all: Task[List[Book]] =
    transactor.connect(books.all)

  def find(id: Long): Task[Option[Book]] =
    transactor.connect(books.find(id).toOption)

  def add(book: Book): Task[Unit] =
    transactor.transact(books.insert(book)).unit
}

object BookRepoLive {
  val layer: ZLayer[TransactorZIO, Nothing, BookRepo] =
    ZLayer.fromFunction(BookRepoLive(_))
}

/**
 * How to Use the sql Module from a ZIO Application — Provide the Database as a
 * ZLayer
 *
 * Provides a `TransactorZIO` over a temporary SQLite file as a layer, creates
 * the table, and reads the books through a `BookRepo` service.
 *
 * Run with: cd sql-examples && sbt "runMain sqlfromzio.BookRepoLayerExample"
 */
object BookRepoLayerExample extends ZIOAppDefault {

  val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
  dbFile.toFile.deleteOnExit()

  val transactorLayer: ZLayer[Any, Nothing, TransactorZIO] =
    TransactorZIO.layer(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

  val run =
    for {
      count <- ZIO
                 .serviceWithZIO[TransactorZIO] { transactor =>
                   transactor.connect {
                     createBookTable()
                     shelf.foreach(books.insert)
                     books.count
                   }
                 }
                 .provide(transactorLayer)
      _   <- ZIO.succeed(println(s"Seeded $count books"))
      all <- ZIO
               .serviceWithZIO[BookRepo] { repo =>
                 repo.add(Book(4L, "Scala with Cats", 2020, 2)) *> repo.all
               }
               .provide(BookRepoLive.layer, transactorLayer)
      _ <- ZIO.succeed(all.foreach(println))
    } yield ()
}
```

</details>

Run it with:

```bash
sbt "runMain sqlfromzio.BookRepoLayerExample"
```

<details>
<summary><code>TransactZIOExample.scala</code>: a sale that charges the customer inside the transaction, and the rollback on a declined payment</summary>

```scala title="sql-examples/src/main/scala/sqlfromzio/TransactZIOExample.scala" showLineNumbers
package sqlfromzio

final case class PaymentDeclined(customer: String) extends Exception(s"Payment declined for $customer")
final case class OutOfStock(bookId: Long)          extends Exception(s"Book $bookId is out of stock")

trait Payments {
  def charge(customer: String, copies: Int): IO[PaymentDeclined, Unit]
}

val paymentsLayer: ZLayer[Any, Nothing, Payments] =
  ZLayer.succeed(new Payments {
    def charge(customer: String, copies: Int): IO[PaymentDeclined, Unit] =
      if (customer == "mallory") ZIO.fail(PaymentDeclined(customer)) else ZIO.unit
  })

/**
 * How to Use the sql Module from a ZIO Application — Mix Database Writes with
 * Other Effects in One Transaction
 *
 * A sale lowers the stock and charges the customer inside one `transactZIO`. A
 * declined payment rolls the stock change back.
 *
 * Run with: cd sql-examples && sbt "runMain sqlfromzio.TransactZIOExample"
 */
object TransactZIOExample extends ZIOAppDefault {

  val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
  dbFile.toFile.deleteOnExit()

  val transactorLayer: ZLayer[Any, Nothing, TransactorZIO] =
    TransactorZIO.layer(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

  def sell(bookId: Long, copies: Int, customer: String): ZIO[TransactorZIO & Payments, Throwable, Unit] =
    ZIO.serviceWithZIO[TransactorZIO] { transactor =>
      transactor.transactZIO {
        for {
          updated <- ZIO.attemptBlocking(
                       sql"UPDATE book SET stock = stock - $copies WHERE id = $bookId AND stock >= $copies".update
                     )
          _ <- ZIO.fail(OutOfStock(bookId)).when(updated == 0)
          _ <- ZIO.serviceWithZIO[Payments](_.charge(customer, copies))
        } yield ()
      }
    }

  val run =
    (for {
      _ <- ZIO.serviceWithZIO[TransactorZIO](_.connect {
             createBookTable()
             shelf.foreach(books.insert)
           })
      before  <- stockOf(1L)
      alice   <- sell(1L, 2, "alice").either
      mallory <- sell(1L, 1, "mallory").either
      after   <- stockOf(1L)
    } yield {
      println(s"Stock before: $before")
      println(s"Alice: $alice")
      println(s"Mallory: $mallory")
      println(s"Stock after: $after")
    }).provide(transactorLayer, paymentsLayer)
}
```

</details>

Run it with:

```bash
sbt "runMain sqlfromzio.TransactZIOExample"
```

<details>
<summary><code>ErrorsRetriesTimeoutsExample.scala</code>: domain errors, retries and timeouts</summary>

```scala title="sql-examples/src/main/scala/sqlfromzio/ErrorsRetriesTimeoutsExample.scala" showLineNumbers
package sqlfromzio

final case class DuplicateBook(id: Long) extends Exception(s"Book $id already exists")

/**
 * How to Use the sql Module from a ZIO Application — Handle Errors, Retries and
 * Timeouts the ZIO Way
 *
 * Maps a primary key violation to a domain error, retries a connection that
 * fails twice, compares `timeout` with `disconnect.timeout` on a slow query,
 * and uses `timeoutFail` inside `transactZIO` so a hanging payment rolls back.
 *
 * Run with: cd sql-examples && sbt "runMain
 * sqlfromzio.ErrorsRetriesTimeoutsExample"
 */
object ErrorsRetriesTimeoutsExample extends ZIOAppDefault {

  val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
  dbFile.toFile.deleteOnExit()

  val transactorLayer: ZLayer[Any, Nothing, TransactorZIO] =
    TransactorZIO.layer(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

  def addNew(book: Book): ZIO[BookRepo, Throwable, Unit] =
    ZIO.serviceWithZIO[BookRepo](_.add(book)).catchSome {
      case e: SQLiteException if e.getResultCode == SQLiteErrorCode.SQLITE_CONSTRAINT_PRIMARYKEY =>
        ZIO.fail(DuplicateBook(book.id))
    }

  val attempts = new java.util.concurrent.atomic.AtomicInteger(0)

  val startingUp = new TransactorZIO(
    () =>
      if (attempts.incrementAndGet() <= 2) throw new java.sql.SQLException("Connection refused")
      else java.sql.DriverManager.getConnection(s"jdbc:sqlite:$dbFile"),
    SqlDialect.SQLite
  )

  val slowCount =
    sql"WITH RECURSIVE n(x) AS (SELECT 1 UNION ALL SELECT x + 1 FROM n WHERE x < 3000000) SELECT count(*) FROM n"

  def timed[A](effect: Task[A]): Task[(A, Duration)] =
    effect.timed.map(_.swap)

  val hangingPayments: ZLayer[Any, Nothing, Payments] =
    ZLayer.succeed(new Payments {
      def charge(customer: String, copies: Int): IO[PaymentDeclined, Unit] = ZIO.never
    })

  def sellWithin(bookId: Long, copies: Int, customer: String): ZIO[TransactorZIO & Payments, Throwable, Unit] =
    ZIO.serviceWithZIO[TransactorZIO] { transactor =>
      transactor.transactZIO {
        for {
          _ <- ZIO.attemptBlocking(sql"UPDATE book SET stock = stock - $copies WHERE id = $bookId".update)
          _ <- ZIO
                 .serviceWithZIO[Payments](_.charge(customer, copies))
                 .timeoutFail(new java.util.concurrent.TimeoutException("Payment took too long"))(50.millis)
        } yield ()
      }
    }

  val run =
    for {
      _ <- ZIO
             .serviceWithZIO[TransactorZIO](_.connect {
               createBookTable()
               shelf.foreach(books.insert)
             })
             .provide(transactorLayer)
      duplicate <- addNew(Book(1L, "Zionomicon", 2024, 1)).either.provide(BookRepoLive.layer, transactorLayer)
      _         <- ZIO.succeed(println(s"Adding Zionomicon again: $duplicate"))
      _         <- startingUp
             .connect(books.count)
             .retry(Schedule.exponential(10.millis) && Schedule.recurs(5))
             .map(count => println(s"Counted $count books after ${attempts.get} attempts"))
      _ <- ZIO
             .serviceWithZIO[TransactorZIO] { transactor =>
               timed(transactor.connect(slowCount.queryOne[Long]).timeout(10.millis)).map { case (result, elapsed) =>
                 println(s"timeout: result $result")
                 println(s"Waited longer than the timeout: ${elapsed.toMillis > 100}")
               }
             }
             .provide(transactorLayer)
      _ <- ZIO
             .serviceWithZIO[TransactorZIO] { transactor =>
               timed(transactor.connect(slowCount.queryOne[Long]).disconnect.timeout(10.millis)).map {
                 case (result, elapsed) =>
                   println(s"disconnect.timeout: result $result")
                   println(s"Returned before the query finished: ${elapsed.toMillis < 100}")
               }
             }
             .provide(transactorLayer)
      before <- stockOf(2L).provide(transactorLayer)
      result <- sellWithin(2L, 1, "bob").either.provide(transactorLayer, hangingPayments)
      after  <- stockOf(2L).provide(transactorLayer)
      _      <- ZIO.succeed {
             println(s"Sale: $result")
             println(s"Stock before: $before, after: $after")
           }
    } yield ()
}
```

</details>

Run it with:

```bash
sbt "runMain sqlfromzio.ErrorsRetriesTimeoutsExample"
```

<details>
<summary><code>LargeResultExample.scala</code>: fold a chunked query inside <code>connect</code></summary>

```scala title="sql-examples/src/main/scala/sqlfromzio/LargeResultExample.scala" showLineNumbers
package sqlfromzio

/**
 * How to Use the sql Module from a ZIO Application — Read a Large Result Set
 *
 * Folds a chunked query inside `TransactorZIO#connect` and turns the `Either`
 * of the fold into a ZIO failure with `absolve`.
 *
 * Run with: cd sql-examples && sbt "runMain sqlfromzio.LargeResultExample"
 */
object LargeResultExample extends ZIOAppDefault {

  val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
  dbFile.toFile.deleteOnExit()

  val transactorLayer: ZLayer[Any, Nothing, TransactorZIO] =
    TransactorZIO.layer(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

  val run =
    (for {
      _ <- ZIO.serviceWithZIO[TransactorZIO](_.connect {
             createBookTable()
             shelf.foreach(books.insert)
           })
      total <- ZIO.serviceWithZIO[TransactorZIO] { transactor =>
                 transactor.connect {
                   sql"SELECT id, title, year, stock FROM book"
                     .queryChunked[Book](2)
                     .runFold(0)((total, chunk) => total + chunk.foldLeft(0)(_ + _.stock))
                 }.absolve
               }
      _ <- ZIO.succeed(println(s"Total stock on the shelf: $total"))
    } yield ()).provide(transactorLayer)
}
```

</details>

Run it with:

```bash
sbt "runMain sqlfromzio.LargeResultExample"
```

The complete example runs with:

```bash
sbt "runMain sqlfromzio.CompleteExample"
```

The ZIO Test spec runs with:

```bash
sbt "testOnly sqlfromzio.BookRepoSpec"
```

## Going Further

- [TransactorZIO](../reference/sql/transactor-zio.md) lists every method of `TransactorZIO` and its companion.
- [ZIO Integration](../reference/sql-zio.md) is the reference page for the `sql-zio` module.
- [How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md) sizes the pool that `databaseLayer` opens.
- [SQL Transactions: Isolation, Read-Only and Savepoints](./sql-transactions.md) covers isolation levels and nested transactions with savepoints.
- [How to Test Database Code Against SQLite](./test-database-code-against-sqlite.md) shows fixtures and rollback-only tests.
- [How to Use ZIO Blocks in a ZIO or cats-effect Application](./use-zio-blocks-in-zio-or-cats-effect.md) covers what `sql-zio` doesn't: cats-effect, and lifting resources, `Async` and streams into ZIO with cancellation carried across.

---

## SQL Transactions: Isolation, Read-Only and Savepoints

This guide covers ZIO Blocks' SQL transaction system: how `transact` works under the hood, which isolation levels are available on each database, and how nested transactions use savepoints. To set up a connection pool, see [How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md).

**What we'll cover:**

- Opening transactions with `transactor.transact { ... }` and tuning isolation + readOnly
- The `TransactionIsolation` enum and how SQLite vs PostgreSQL handle each level
- Nested transactions via SQL savepoints (`zib_tx_1 .. zib_tx_N`)
- Error semantics: inner rollback vs outer commit

## Basic Usage

Every database interaction goes through a `Transactor`. The simplest form opens a connection, disables auto-commit, runs your code, and commits on success (or rolls back on failure):

```scala
transactor.transact {
  sql"INSERT INTO users (name) VALUES ('Alice')".update
  sql"INSERT INTO orders (user_id, total) VALUES (1, 99.50)".update
}
```

Both inserts run inside a single transaction. If either fails, both are rolled back.

To control isolation level and read-only mode, pass them explicitly:

```scala
transactor.transact(TransactionIsolation.RepeatableRead, readOnly = false) {
  val users = sql"SELECT * FROM users WHERE active = true".query[User].toList
  // ... process users ...
}
```

The one-argument `Transactor#transact` sets neither the isolation level nor the read-only flag, so it never forces `Serializable`. It keeps the connection's current settings, which are the JDBC driver's defaults unless your connection pool configures them: on PostgreSQL that is `ReadCommitted`, and on SQLite every transaction behaves as `Serializable`, the only level SQLite supports. To get a specific level, use the two-argument overload shown above.

## TransactionIsolation

The `TransactionIsolation` enum has four values, each mapping to the corresponding `java.sql.Connection.TRANSACTION_*` constant:

| Enum value           | JDBC constant                        |
|----------------------|--------------------------------------|
| `ReadUncommitted`    | `TRANSACTION_READ_UNCOMMITTED`       |
| `ReadCommitted`      | `TRANSACTION_READ_COMMITTED`         |
| `RepeatableRead`     | `TRANSACTION_REPEATABLE_READ`        |
| `Serializable`       | `TRANSACTION_SERIALIZABLE`           |

When you pass a level to `transactor.transact`, it calls `Connection.setTransactionIsolation` before starting the transaction. The previous level is restored in a `finally` block after the transaction completes.

### SQLite vs PostgreSQL

The two supported dialects differ significantly in how they honor isolation levels:

| Level             | PostgreSQL                           | SQLite                               |
|-------------------|--------------------------------------|--------------------------------------|
| `ReadUncommitted` | Accepted, behaves as `ReadCommitted` | Accepted, treated as `SERIALIZABLE`  |
| `ReadCommitted`   | Full support (default PG level)      | Accepted, treated as `SERIALIZABLE`  |
| `RepeatableRead`  | Full support (MVCC snapshot)         | Accepted, treated as `SERIALIZABLE`  |
| `Serializable`    | Full support (SSI)                   | Native (default and only true level) |

SQLite natively supports only `SERIALIZABLE`. The driver accepts the `setTransactionIsolation` call without error, but the engine ignores the requested level and behaves as serializable. This is a SQLite limitation, not a ZIO Blocks one. If your application relies on weaker isolation guarantees (e.g., `ReadCommitted` for higher concurrency), test on PostgreSQL where those semantics are real.

PostgreSQL accepts all four levels but implements three of them: `ReadUncommitted` behaves exactly like `ReadCommitted`, so PostgreSQL never allows dirty reads. `ReadCommitted` is PostgreSQL's default level for every new transaction; `RepeatableRead` gives you a consistent snapshot for the duration of the transaction; `Serializable` also detects serialization anomalies with Serializable Snapshot Isolation (SSI) and fails one of the conflicting transactions, which your application must retry.

## Read-Only Transactions

Passing `readOnly = true` marks the connection as read-only at the JDBC level:

```scala
transactor.transact(TransactionIsolation.ReadCommitted, readOnly = true) {
  sql"SELECT COUNT(*) FROM users".queryOne[Long]
}
```

On PostgreSQL, the JDBC driver starts the transaction with `BEGIN READ ONLY`, so a write inside the block fails with an error such as `ERROR: cannot execute INSERT in a read-only transaction`. SQLite may ignore the flag depending on the driver version, but it's still set on the connection for consistency. The previous `readOnly` value is always restored after the transaction, regardless of dialect.

## Nested Transactions via Savepoints

Nested transactions are emulated via SQL savepoints on the same JDBC connection. When you're already inside `transactor.transact { ... }`, you can nest further work using the ambient `DbTx`:

```scala
transactor.transact {
  sql"INSERT INTO orders (total) VALUES (100)".update

  summon[DbTx].transact {
    sql"INSERT INTO order_items (order_id, product_id) VALUES (1, 42)".update
  }

  sql"UPDATE inventory SET stock = stock - 1 WHERE product_id = 42".update
}
```

The nested block runs inside a savepoint named `zib_tx_1`. If it fails, only the inner work is rolled back (via `ROLLBACK TO SAVEPOINT`). The outer transaction continues and can still commit.

### How Savepoints Work

Each nesting level gets a savepoint name derived from the current depth: `zib_tx_1`, `zib_tx_2`, and so on. The mechanism:

1. **Depth increments** before creating the savepoint
2. **`SAVEPOINT zib_tx_<depth>`** is issued on the connection
3. The nested body executes with the ambient `DbTx` in scope
4. On **success**: `RELEASE SAVEPOINT zib_tx_<depth>` is called
5. On **failure**: `ROLLBACK TO SAVEPOINT zib_tx_<depth>` is called, then the exception is rethrown
6. **Depth decrements** in a `finally` block, guaranteeing the counter resets even after exceptions

The depth counter resets after each inner block finishes, so sibling nested transactions reuse the same name sequence without leaking savepoints.

### Three-Level Example

```scala
transactor.transact {
  // Depth 0: outer transaction (real COMMIT/ROLLBACK)
  sql"INSERT INTO t VALUES (1)".update

  summon[DbTx].transact {
    // Depth 1: savepoint zib_tx_1
    sql"INSERT INTO t VALUES (2)".update

    summon[DbTx].transact {
      // Depth 2: savepoint zib_tx_2
      sql"INSERT INTO t VALUES (3)".update
    }
    // zib_tx_2 released here

    sql"INSERT INTO t VALUES (4)".update
  }
  // zib_tx_1 released here

  sql"INSERT INTO t VALUES (5)".update
}
// COMMIT (all 5 inserts)
```

If the depth-2 block fails, only `INSERT INTO t VALUES (3)` is rolled back. Inserts 1, 2, 4, and 5 remain and will be committed.

### Error Semantics

When an inner block throws:

- The inner savepoint is rolled back via `ROLLBACK TO SAVEPOINT`
- The exception is rethrown to the caller
- The outer transaction is **not** rolled back automatically
- The outer block can catch the exception and continue, or let it propagate (triggering a full rollback at the top level)

```scala
transactor.transact {
  sql"INSERT INTO t VALUES (1)".update

  try {
    summon[DbTx].transact {
      sql"INSERT INTO t VALUES (2)".update
      throw new RuntimeException("inner failure")
      sql"INSERT INTO t VALUES (3)".update  // never reached
    }
  } catch {
    case _: RuntimeException => ()  // swallow inner failure
  }

  sql"INSERT INTO t VALUES (4)".update
}
// COMMIT: rows 1 and 4 only (row 2 rolled back by savepoint)
```

### The `transactNested` Helper

For call sites that prefer `using` parameters over extension receivers, `transactNested` is available as an alias:

```scala

transactor.transact {
  DbTx.transactNested {
    // same as summon[DbTx].transact { ... }
    sql"INSERT INTO t VALUES (1)".update
  }
}
```

Both forms are equivalent. The extension method (`summon[DbTx].transact { ... }`) is more common in practice because it reads naturally inside a `transact` block.

### Given-Priority Trick

When a `DbTx` is already in scope (inside `transactor.transact`), the extension `summon[DbTx].transact { ... }` resolves via the more specific `DbTx` receiver and reuses the same connection with a savepoint. Using `transactor.transact { ... }` instead would open a **new** connection, which is almost never what you want inside an existing transaction. The explicit `summon[DbTx]` form keeps semantics clear without hidden implicit resolution.

## Connection Pooling

To run these transactions on a HikariCP pool, build the transactor with `JdbcTransactor.postgres(dataSource)`; [How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md) covers the pool setup, sizing it for virtual threads, and closing it.

## Summary

| Concept                    | Key Point                                               |
|----------------------------|---------------------------------------------------------|
| `transactor.transact`      | Opens connection, disables auto-commit, commits/rollbacks |
| `TransactionIsolation`     | Four levels; SQLite only true `SERIALIZABLE`            |
| `readOnly` flag            | Sets JDBC read-only; PG rejects writes, SQLite may ignore |
| Nested via `summon[DbTx]`  | Savepoints `zib_tx_1..N`, inner rollback isolated       |
| `transactNested`           | Alias using `using` parameter instead of extension      |

## Going Further

- **[Query DSL with SQL](./query-dsl-sql.md)** -- Building parameterized SQL from `SchemaExpr` queries
- **[Transactor Reference](../reference/sql/transactor)** -- `JdbcTransactor.fromDataSource` / `fromUrl` and `transact(isolation, readOnly)` overloads
- **[DbTx Reference](../reference/sql/db-tx)** -- Savepoint API (`savepoint` / `release` / `rollbackTo`) and `summon[DbTx].transact`
- **[How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md)** -- HikariCP setup, pool sizing for virtual threads, and shutdown

---

## How to Store and Search Vector Embeddings in PostgreSQL

In this guide, we store vector embeddings in a PostgreSQL [pgvector](https://github.com/pgvector/pgvector) column and search them by similarity with the ZIO Blocks `sql` module. By the end, you will have an `Embedding` type that you pass to `sql"..."` like any other parameter, a table of book embeddings, a nearest-neighbour query, and an HNSW index that speeds the query up.

We start from a working PostgreSQL setup and add one piece at a time:

- Describe an `Embedding` as text with a [`DbCodec`](../reference/sql/db-codec.md)
- Create a table with a `vector` column
- Insert embeddings with a cast
- Find the nearest books to a query embedding
- Pick a distance operator
- Read embeddings back
- Reject a vector with the wrong number of dimensions
- Speed up the search with an HNSW index
- Skip the casts with a driver setting

This guide assumes you already have a PostgreSQL repository running, as in [How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md), and you know how to give a type its own column mapping with `DbCodec#transform`, as in [How to Map Custom Column Types](./map-custom-column-types.md). We don't explain either again here.

## The Problem

An embedding is a list of numbers, usually the output of a machine-learning model, that stands for the meaning of a piece of text. Two texts with similar meanings get embeddings that are close to each other, so a search for similar texts becomes a search for the nearest vectors. The pgvector extension adds a `vector(n)` column type and distance operators to PostgreSQL for exactly this.

The JDBC driver has no standard type for `vector`, so plain JDBC code wraps the value in a driver-specific `PGobject` every time it writes one. This is the plain JDBC code, PostgreSQL driver 42.7.14 (not compiled here, because the docs build has no PostgreSQL driver; we compiled and ran it against PostgreSQL 16 with pgvector):

```scala

def insertEmbedding(connection: Connection, title: String, values: Array[Float]): Unit = {
  val vector = new PGobject()
  vector.setType("vector")
  vector.setValue(values.mkString("[", ",", "]"))

  val statement = connection.prepareStatement("INSERT INTO book_embeddings (title, embedding) VALUES (?, ?)")
  try {
    statement.setString(1, title)
    statement.setObject(2, vector)
    statement.executeUpdate()
  } finally statement.close()
}
```

This works, but every call site repeats the wrapping, the text format lives in the caller, and nothing ties the Scala type to the column. The obvious next step, passing the `PGobject` to a `sql"..."` interpolator, doesn't compile, as the next section shows.

In this guide, we describe the embedding once, as a type with a `DbCodec`, and let the `sql` module bind it wherever it appears in a query.

## Prerequisites

Add the `sql` module and the PostgreSQL JDBC driver to your `build.sbt`. [How to Add ZIO Blocks to Your Build](../getting-started/installation.md) explains the two artifact names:

```scala
// Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-sql" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-sql" % "0.0.56"
scalacOptions += "-experimental"

// The PostgreSQL JDBC driver, for every Scala version
libraryDependencies += "org.postgresql" % "postgresql" % "42.7.14"
```

The `sql` module is published for Scala 3 only, and [`JdbcTransactor`](../reference/sql/transactor.md) runs on the JVM and needs JDK 21 or newer, so the code in this guide uses Scala 3 syntax. We don't need a pgvector client library: the codec in this guide is a few lines on top of `String`, and adds no dependency.

You need a PostgreSQL server with the pgvector extension installed. If you have Docker, the `pgvector/pgvector` image is PostgreSQL 16 with the extension built in. This command starts it with the same `bookstore` database, user and password as the pool guide:

```bash
docker run --rm -d --name bookstore-db -p 5432:5432 \
  -e POSTGRES_USER=bookstore -e POSTGRES_PASSWORD=bookstore -e POSTGRES_DB=bookstore \
  pgvector/pgvector:pg16
```

The code in this guide reads the database URL, user and password from `BOOKSTORE_DB_URL`, `BOOKSTORE_DB_USER` and `BOOKSTORE_DB_PASSWORD`, with defaults that match the Docker command, so it runs without any variables set. It builds a `JdbcTransactor` straight from the URL to stay short. To run the same code on a connection pool, build the transactor from the pool as the pool guide shows.

## Describe an Embedding as Text

The interpolator accepts a parameter only if it has a [`DbParam`](../reference/sql/db-param.md), and the driver's `PGobject` has none. Passing one, as in `sql"INSERT INTO book_embeddings (title, embedding) VALUES (${"Zionomicon"}, $vector)"` with `vector` a `PGobject`, stops the compiler with this error:

```text
No DbParam/Frag/SqlLiteral/DbValue instance found for type org.postgresql.util.PGobject. Supported: DbParam[T], Frag (verbatim), SqlLiteral (verbatim), DbValue. Use sql"..." or StringContext(...).sql(table)(...) accordingly.
```

The fix is to give our own type a `DbCodec`, which gives it a `DbParam` for free, instead of reaching for a driver type.

pgvector writes a vector as text, as `[0.8,1.0,0.9]`, and PostgreSQL accepts the same text as input. That makes a `String` the simplest bridge: we convert an `Embedding` to that text on the way in, parse it on the way out, and `DbCodec#transform` builds the codec from the two functions:

```scala

final case class Embedding(values: IndexedSeq[Float])

object Embedding {

  def render(embedding: Embedding): String = embedding.values.mkString("[", ",", "]")

  def parse(text: String): Embedding = {
    val inner = text.substring(1, text.length - 1)
    if (inner.isEmpty) Embedding(IndexedSeq.empty)
    else Embedding(inner.split(',').toIndexedSeq.map(_.toFloat))
  }

  given DbCodec[Embedding] = DbCodec[String].transform(parse)(render)
}
```

The sections below read the database settings and create the transactor the same way each time, so we define that once:

```scala
def shelfTransactor(): JdbcTransactor =
  JdbcTransactor.fromUrl(
    sys.env.getOrElse("BOOKSTORE_DB_URL", "jdbc:postgresql://localhost:5432/bookstore"),
    sys.env.getOrElse("BOOKSTORE_DB_USER", "bookstore"),
    sys.env.getOrElse("BOOKSTORE_DB_PASSWORD", "bookstore"),
    SqlDialect.PostgreSQL
  )
```

The sample data for the rest of the guide is five real Scala books. Real embedding models produce hundreds or thousands of numbers per text; we use three, scored by hand for functional programming, effects and concurrency, so that every output stays readable:

```scala
val shelfBooks: List[(String, Embedding)] = List(
  "Zionomicon"                      -> Embedding(IndexedSeq(0.8f, 1.0f, 0.9f)),
  "Programming in Scala"            -> Embedding(IndexedSeq(0.5f, 0.1f, 0.1f)),
  "Functional Programming in Scala" -> Embedding(IndexedSeq(1.0f, 0.5f, 0.2f)),
  "Scala Cookbook"                  -> Embedding(IndexedSeq(0.3f, 0.1f, 0.2f)),
  "Scala with Cats"                 -> Embedding(IndexedSeq(0.9f, 0.7f, 0.1f))
)
```

Rendering and parsing are inverses, so an embedding that goes through the text form comes back unchanged:

```scala
val zionomicon = shelfBooks.head._2
// zionomicon: Embedding = Embedding(Vector(0.8F, 1.0F, 0.9F))

Embedding.render(zionomicon)
// res0: String = "[0.8,1.0,0.9]"

Embedding.parse(Embedding.render(zionomicon)) == zionomicon
// res1: Boolean = true
```

Passing the embedding to the `sql` interpolator now compiles, because the `DbCodec` gives it a `DbParam`. To see what the interpolator hands to the driver, we render the statement for PostgreSQL and list its parameters:

```scala
val insert = sql"INSERT INTO book_embeddings (title, embedding) VALUES (${"Zionomicon"}, $zionomicon)"
// insert: Frag = Frag(
//   parts = Vector(
//     "INSERT INTO book_embeddings (title, embedding) VALUES (",
//     ", ",
//     ")"
//   ),
//   params = Vector(DbString("Zionomicon"), DbString("[0.8,1.0,0.9]"))
// )

insert.sql(SqlDialect.PostgreSQL)
// res2: String = "INSERT INTO book_embeddings (title, embedding) VALUES (?, ?)"

insert.queryParams
// res3: IndexedSeq[DbValue] = Vector(
//   DbString("Zionomicon"),
//   DbString("[0.8,1.0,0.9]")
// )
```

The embedding travels as one `DbString` holding the text form, which the driver binds as a `character varying` parameter. That detail matters in two sections, because PostgreSQL treats a `varchar` and a `vector` as different types.

## Create the Table

The table keeps one row per book, with a `vector(3)` column whose size must match the embeddings we store. We write the `CREATE TABLE` statement ourselves, because `Table#createTable` has no way to declare a `vector(n)` column, and the `vector` extension has to exist before the table does:

```scala
def createShelf()(using DbCon): Unit = {
  sql"CREATE EXTENSION IF NOT EXISTS vector".update
  sql"DROP TABLE IF EXISTS book_embeddings".update
  sql"""CREATE TABLE book_embeddings (
          id        BIGSERIAL PRIMARY KEY,
          title     TEXT        NOT NULL,
          embedding VECTOR(3)   NOT NULL
        )""".update
}
```

`CREATE EXTENSION` needs a role that is allowed to create extensions; the superuser that the Docker image creates is one. The `DROP TABLE` line makes the examples in this guide repeatable.

:::warning
In version 0.0.56, `Repo.derived` and `Table.derived` throw `UnsupportedOperationException: Typed DDL does not support reflect node IndexedSeq[Float]` for a case class with an `Embedding` field, because they compute the table's columns from the `Schema` and ignore the `DbCodec`. [How to Map Custom Column Types](./map-custom-column-types.md) describes the same limitation for a `List[String]` field. This guide writes its statements with `sql"..."` instead.
:::

## Insert Embeddings with a Cast

An insert passes each embedding as a parameter and casts it to `vector` in the SQL. The cast is what tells PostgreSQL to convert the text into a vector, so we write it right after the parameter:

```scala
def stockShelf()(using DbCon): Unit =
  shelfBooks.foreach { (title, embedding) =>
    sql"INSERT INTO book_embeddings (title, embedding) VALUES ($title, $embedding::vector)".update
  }
```

To run both statements in one transaction and count the rows:

```scala
val transactor = shelfTransactor()

val count = transactor.transact {
  createShelf()
  stockShelf()
  sql"SELECT COUNT(*) FROM book_embeddings".queryOne[Long].getOrElse(0L)
}

println(s"Stored $count embeddings")
```

Run against PostgreSQL 16 with pgvector, it prints:

```text
Stored 5 embeddings
```

Without the cast, the same insert fails. The driver sends the embedding as `character varying`, and PostgreSQL doesn't convert that to `vector` on its own:

```text
ERROR: column "embedding" is of type vector but expression is of type character varying
```

The same cast is needed wherever a query parameter meets a `vector`, not only in an `INSERT`. The next section shows it in a search.

## Find the Nearest Books

A nearest-neighbour search orders the rows by their distance to a query embedding and keeps the first few. pgvector's `<->` operator is the Euclidean (L2) distance between two vectors, so we order by it and take three rows. The query is an embedding for "a book about effects and concurrency", scored on the same three topics:

```scala
val transactor = shelfTransactor()

val query = Embedding(IndexedSeq(0.6f, 0.9f, 1.0f))

val nearest = transactor.transact {
  createShelf()
  stockShelf()
  sql"SELECT title FROM book_embeddings ORDER BY embedding <-> $query::vector LIMIT 3".query[String]
}

nearest.foreach(println)
```

The three closest books, nearest first, are:

```text
Zionomicon
Scala with Cats
Functional Programming in Scala
```

Because `Embedding` has a `DbCodec`, `$query::vector` is an ordinary parameter, and the [`Frag#query`](../reference/sql/frag.md) result type comes from the `[String]` type argument. The rows come back in distance order, which is the order the `ORDER BY` gives them.

## Pick a Distance Operator

pgvector offers three distance operators, and which one ranks the books best depends on what your embedding model means by "close":

| Operator | Distance                | Smaller means         | Pair it with an index using |
| -------- | ----------------------- | --------------------- | --------------------------- |
| `<->`    | Euclidean (L2)          | Closer in space       | `vector_l2_ops`             |
| `<=>`    | Cosine                  | Closer in direction   | `vector_cosine_ops`         |
| `<#>`    | Negative inner product  | Larger inner product  | `vector_ip_ops`             |

Cosine distance compares direction only and ignores length, which is why many embedding models are trained for it; check the documentation of yours. The inner product grows with length as well as with direction. pgvector returns the *negative* inner product, because an index can only order rows from smallest to largest, so the best match has the smallest, most negative value and `ORDER BY embedding <#> ...` stays ascending. For embeddings normalised to length 1, the inner product and cosine distance rank the rows the same way.

To see how the choice changes the ranking, we run the same query with each operator and print the distance next to every title:

```scala
val transactor = shelfTransactor()

val query = Embedding(IndexedSeq(0.6f, 0.9f, 1.0f))

transactor.transact {
  createShelf()
  stockShelf()

  println("Euclidean distance (<->):")
  sql"SELECT title, embedding <-> $query::vector FROM book_embeddings ORDER BY embedding <-> $query::vector LIMIT 3"
    .query[(String, Double)]
    .foreach((title, distance) => println(f"  $title%-32s $distance%.4f"))

  println("Cosine distance (<=>):")
  sql"SELECT title, embedding <=> $query::vector FROM book_embeddings ORDER BY embedding <=> $query::vector LIMIT 3"
    .query[(String, Double)]
    .foreach((title, distance) => println(f"  $title%-32s $distance%.4f"))

  println("Negative inner product (<#>):")
  sql"SELECT title, embedding <#> $query::vector FROM book_embeddings ORDER BY embedding <#> $query::vector LIMIT 3"
    .query[(String, Double)]
    .foreach((title, distance) => println(f"  $title%-32s $distance%.4f"))
}
```

Against the sample data, it prints:

```text
Euclidean distance (<->):
  Zionomicon                       0.2449
  Scala with Cats                  0.9695
  Functional Programming in Scala  0.9798
Cosine distance (<=>):
  Zionomicon                       0.0112
  Scala Cookbook                   0.1473
  Scala with Cats                  0.2468
Negative inner product (<#>):
  Zionomicon                       -2.2800
  Scala with Cats                  -1.2700
  Functional Programming in Scala  -1.2500
```

Zionomicon is first in every list. Scala Cookbook appears only in the cosine list: its scores are small, so it is far from the query in space, but its direction is closer to the query's than Functional Programming in Scala's is.

## Read Embeddings Back

A `vector` column reads back as text, and the same `DbCodec` that wrote the embedding parses it. To read a single embedding, ask for the column and give `Frag#queryOne` the `Embedding` type:

```scala
val transactor = shelfTransactor()

transactor.transact {
  createShelf()
  stockShelf()

  val stored = sql"SELECT embedding FROM book_embeddings WHERE title = ${"Zionomicon"}".queryOne[Embedding]
  println(s"Stored:   ${stored.map(_.values.mkString("[", ", ", "]"))}")
  println(s"Expected: ${Some(zionomicon.values.mkString("[", ", ", "]"))}")
  println(s"Same:     ${stored.contains(zionomicon)}")

  val rows = sql"SELECT title, embedding FROM book_embeddings ORDER BY id LIMIT 2".query[(String, Embedding)]
  rows.foreach((title, embedding) => println(s"$title -> ${embedding.values.size} dimensions"))
}
```

The values come back exactly as stored, and an `Embedding` works as one element of a row tuple:

```text
Stored:   Some([0.8, 1.0, 0.9])
Expected: Some([0.8, 1.0, 0.9])
Same:     true
Zionomicon -> 3 dimensions
Programming in Scala -> 3 dimensions
```

The parser handles the text PostgreSQL produces: a vector that is stored from `0.1` comes back as `0.1`, and spaces that were typed in the input, as in `[1, 2.5 ,3]`, are normalised to `[1,2.5,3]` by the server before we read them. A vector must have at least one dimension, so `Embedding.parse` never sees `[]` from a real column; the `isEmpty` branch only keeps the function total.

## Reject the Wrong Number of Dimensions

The `3` in `vector(3)` is enforced by the server. An embedding with four numbers is rejected, the driver throws an `SQLException`, and the surrounding `JdbcTransactor#transact` rolls the transaction back. To see the error and check that nothing was stored:

```scala

val transactor = shelfTransactor()

transactor.transact {
  createShelf()
  stockShelf()
}

val tooLong = Embedding(IndexedSeq(0.1f, 0.2f, 0.3f, 0.4f))

try
  transactor.transact {
    sql"INSERT INTO book_embeddings (title, embedding) VALUES (${"Too Long"}, $tooLong::vector)".update
  }
catch {
  case e: SQLException => println(s"Rejected: ${e.getMessage}")
}

val count = transactor.connect(sql"SELECT COUNT(*) FROM book_embeddings".queryOne[Long].getOrElse(0L))
println(s"Rows stored: $count")
```

The message names both sizes, and the table still holds the five books:

```text
Rejected: ERROR: expected 3 dimensions, not 4
Rows stored: 5
```

A search with the wrong number of dimensions fails too, with `different vector dimensions 3 and 4` when PostgreSQL compares the two vectors directly. So a model that is replaced by one with a different output size shows up at the first insert or search instead of silently mixing vectors of different sizes in one column.

## Speed Up Search with an HNSW Index

Without an index, PostgreSQL measures the distance to every row for every search. An HNSW index, a graph that pgvector builds over the vectors, finds near neighbours without visiting every row, at the price of returning approximate results. We create the index with the operator class that matches the distance operator the queries use:

```scala
val transactor = shelfTransactor()

transactor.transact {
  createShelf()
  stockShelf()

  sql"CREATE INDEX book_embeddings_hnsw ON book_embeddings USING hnsw (embedding vector_cosine_ops)".update
}
```

Only a query whose `ORDER BY` operator matches the operator class can use the index: `vector_cosine_ops` serves `<=>`, and a search with `<->` ignores it. To check this, we ask PostgreSQL for the query plan of each search. With five rows PostgreSQL would choose a sequential scan whatever the index offers, so the example turns sequential scans off inside the transaction with `SET LOCAL`:

```scala
val transactor = shelfTransactor()

val query = Embedding(IndexedSeq(0.6f, 0.9f, 1.0f))

transactor.transact {
  createShelf()
  stockShelf()

  sql"CREATE INDEX book_embeddings_hnsw ON book_embeddings USING hnsw (embedding vector_cosine_ops)".update
  sql"SET LOCAL enable_seqscan = off".update

  val cosinePlan = sql"EXPLAIN SELECT title FROM book_embeddings ORDER BY embedding <=> $query::vector LIMIT 3".query[String]
  val l2Plan     = sql"EXPLAIN SELECT title FROM book_embeddings ORDER BY embedding <-> $query::vector LIMIT 3".query[String]

  println(s"Cosine search uses the index:    ${cosinePlan.exists(_.contains("book_embeddings_hnsw"))}")
  println(s"Euclidean search uses the index: ${l2Plan.exists(_.contains("book_embeddings_hnsw"))}")
}
```

Only the cosine search uses the cosine index:

```text
Cosine search uses the index:    true
Euclidean search uses the index: false
```

The query also has to order by the distance operator itself, ascending, and end with a `LIMIT`. A query that orders by an expression of the distance, such as `ORDER BY 1 - (embedding <=> ...) DESC` to sort by similarity, can't use the index.

If your queries use two operators, create one index for each operator class. An index is worth its build time and memory only once the table is large; the plan check above forces it onto a five-row table only to show the matching rule.

## Skip the Casts with a Driver Setting

The casts exist because the driver sends every `String` parameter as `character varying`. The driver's `stringtype=unspecified` URL setting sends them without a type instead, and PostgreSQL then infers the type from the place where the parameter appears, which turns the text into a `vector` for both the insert and the search. This is the same setting that the [jsonb section](./map-custom-column-types.md#use-a-jsonb-column-on-postgresql) of the custom-types guide uses. To run the same cast-free statements on both kinds of connection:

```scala

val baseUrl = sys.env.getOrElse("BOOKSTORE_DB_URL", "jdbc:postgresql://localhost:5432/bookstore")
val user    = sys.env.getOrElse("BOOKSTORE_DB_USER", "bookstore")
val secret  = sys.env.getOrElse("BOOKSTORE_DB_PASSWORD", "bookstore")

val query = Embedding(IndexedSeq(0.6f, 0.9f, 1.0f))

def withoutCasts(label: String, url: String): Unit = {
  val transactor = JdbcTransactor.fromUrl(url, user, secret, SqlDialect.PostgreSQL)
  try {
    val nearest = transactor.transact {
      createShelf()
      sql"INSERT INTO book_embeddings (title, embedding) VALUES (${"Zionomicon"}, $query)".update
      sql"SELECT title FROM book_embeddings ORDER BY embedding <-> $query LIMIT 1".query[String]
    }
    println(s"$label: stored and found ${nearest.mkString(", ")}")
  } catch {
    case e: SQLException => println(s"$label: ${e.getMessage.linesIterator.next()}")
  }
}

withoutCasts("default settings          ", baseUrl)
withoutCasts("stringtype=unspecified    ", baseUrl + (if (baseUrl.contains("?")) "&" else "?") + "stringtype=unspecified")
```

The default connection rejects the statements, and the connection with the setting accepts them:

```text
default settings          : ERROR: column "embedding" is of type vector but expression is of type character varying
stringtype=unspecified    : stored and found Zionomicon
```

The setting applies to every `String` parameter on the connection, and PostgreSQL then infers each one's type from where it appears, so a query that depends on a `varchar` parameter, such as a call to a function with several overloads, can become ambiguous. The cast in the SQL changes one statement and nothing else, so it is the safer default; the setting is worth it when many statements bind embeddings and you would rather not repeat the cast.

## Putting It Together

The complete example creates the table, stores five books, searches by cosine distance, reads one embedding back, shows the error for a vector with the wrong number of dimensions and adds an HNSW index:

```scala title="sql-examples/src/main/scala/storeandsearchvectorembeddings/CompleteExample.scala"
package storeandsearchvectorembeddings

/**
 * How to Store and Search Vector Embeddings in PostgreSQL: Complete Example
 *
 * Creates the table, stores five books, searches by cosine distance, reads one
 * embedding back, shows the error for a vector with the wrong number of
 * dimensions and adds an HNSW index.
 *
 * Run with: cd sql-examples && sbt "runMain
 * storeandsearchvectorembeddings.CompleteExample"
 */
object CompleteExample extends App {

  val transactor = shelfTransactor()

  val query = Embedding(IndexedSeq(0.6f, 0.9f, 1.0f))

  transactor.transact {
    createShelf()
    stockShelf()
  }

  val nearest = transactor.connect {
    sql"SELECT title, embedding <=> $query::vector FROM book_embeddings ORDER BY embedding <=> $query::vector LIMIT 3"
      .query[(String, Double)]
  }
  nearest.foreach((title, distance) => println(f"$title%-32s $distance%.4f"))

  val stored = transactor.connect {
    sql"SELECT embedding FROM book_embeddings WHERE title = ${"Zionomicon"}".queryOne[Embedding]
  }
  println(s"Zionomicon: ${stored.map(_.values.mkString("[", ", ", "]"))}")

  try
    transactor.transact {
      val tooLong = Embedding(IndexedSeq(0.1f, 0.2f, 0.3f, 0.4f))
      sql"INSERT INTO book_embeddings (title, embedding) VALUES (${"Too Long"}, $tooLong::vector)".update
    }
  catch {
    case e: SQLException => println(s"Rejected: ${e.getMessage}")
  }

  transactor.transact {
    sql"CREATE INDEX book_embeddings_hnsw ON book_embeddings USING hnsw (embedding vector_cosine_ops)".update
  }
  println("Index book_embeddings_hnsw created")
}
```

## Running the Examples

All examples in this guide have runnable Scala files in the `sql-examples` project. They need a running PostgreSQL server with pgvector, so start one with the Docker command from [Prerequisites](#prerequisites) first. Then clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/sql-examples
```

The examples share the `Embedding` type, the transactor, the table and the five books, which live in one file:

<details>
  <summary>sql-examples/src/main/scala/storeandsearchvectorembeddings/Shelf.scala</summary>

```scala title="sql-examples/src/main/scala/storeandsearchvectorembeddings/Shelf.scala" showLineNumbers
package storeandsearchvectorembeddings

/**
 * How to Store and Search Vector Embeddings in PostgreSQL: shared setup
 *
 * The `Embedding` type with its `DbCodec`, the transactor that reads the
 * database settings from the environment, the `book_embeddings` table and the
 * five books that the examples in this package share.
 */
final case class Embedding(values: IndexedSeq[Float])

object Embedding {

  def render(embedding: Embedding): String = embedding.values.mkString("[", ",", "]")

  def parse(text: String): Embedding = {
    val inner = text.substring(1, text.length - 1)
    if (inner.isEmpty) Embedding(IndexedSeq.empty)
    else Embedding(inner.split(',').toIndexedSeq.map(_.toFloat))
  }

  given DbCodec[Embedding] = DbCodec[String].transform(parse)(render)
}

def shelfTransactor(): JdbcTransactor =
  JdbcTransactor.fromUrl(
    sys.env.getOrElse("BOOKSTORE_DB_URL", "jdbc:postgresql://localhost:5432/bookstore"),
    sys.env.getOrElse("BOOKSTORE_DB_USER", "bookstore"),
    sys.env.getOrElse("BOOKSTORE_DB_PASSWORD", "bookstore"),
    SqlDialect.PostgreSQL
  )

// Each number scores a topic: functional programming, effects, concurrency.
val shelfBooks: List[(String, Embedding)] = List(
  "Zionomicon"                      -> Embedding(IndexedSeq(0.8f, 1.0f, 0.9f)),
  "Programming in Scala"            -> Embedding(IndexedSeq(0.5f, 0.1f, 0.1f)),
  "Functional Programming in Scala" -> Embedding(IndexedSeq(1.0f, 0.5f, 0.2f)),
  "Scala Cookbook"                  -> Embedding(IndexedSeq(0.3f, 0.1f, 0.2f)),
  "Scala with Cats"                 -> Embedding(IndexedSeq(0.9f, 0.7f, 0.1f))
)

def createShelf()(using DbCon): Unit = {
  sql"CREATE EXTENSION IF NOT EXISTS vector".update
  sql"DROP TABLE IF EXISTS book_embeddings".update
  sql"""CREATE TABLE book_embeddings (
          id        BIGSERIAL PRIMARY KEY,
          title     TEXT        NOT NULL,
          embedding VECTOR(3)   NOT NULL
        )""".update
}

def stockShelf()(using DbCon): Unit =
  shelfBooks.foreach { (title, embedding) =>
    sql"INSERT INTO book_embeddings (title, embedding) VALUES ($title, $embedding::vector)".update
  }
```

</details>

### Insert Embeddings

This example creates the table, inserts the five books with a cast and counts the rows.

<details>
  <summary>sql-examples/src/main/scala/storeandsearchvectorembeddings/InsertExample.scala</summary>

```scala title="sql-examples/src/main/scala/storeandsearchvectorembeddings/InsertExample.scala" showLineNumbers
package storeandsearchvectorembeddings

/**
 * How to Store and Search Vector Embeddings in PostgreSQL: Insert Embeddings
 *
 * Creates the table, inserts five books with their embeddings, and counts the
 * rows.
 *
 * Run with: cd sql-examples && sbt "runMain
 * storeandsearchvectorembeddings.InsertExample"
 */
object InsertExample extends App {

  val transactor = shelfTransactor()

  val count = transactor.transact {
    createShelf()
    stockShelf()
    sql"SELECT COUNT(*) FROM book_embeddings".queryOne[Long].getOrElse(0L)
  }

  println(s"Stored $count embeddings")
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "runMain storeandsearchvectorembeddings.InsertExample"
```

### Find the Nearest Books

This example asks for the three books closest to a query embedding by Euclidean distance.

<details>
  <summary>sql-examples/src/main/scala/storeandsearchvectorembeddings/NearestExample.scala</summary>

```scala title="sql-examples/src/main/scala/storeandsearchvectorembeddings/NearestExample.scala" showLineNumbers
package storeandsearchvectorembeddings

/**
 * How to Store and Search Vector Embeddings in PostgreSQL: Find the Nearest
 * Books
 *
 * Asks for the three books closest to a query embedding, by Euclidean (L2)
 * distance.
 *
 * Run with: cd sql-examples && sbt "runMain
 * storeandsearchvectorembeddings.NearestExample"
 */
object NearestExample extends App {

  val transactor = shelfTransactor()

  val query = Embedding(IndexedSeq(0.6f, 0.9f, 1.0f))

  val nearest = transactor.transact {
    createShelf()
    stockShelf()
    sql"SELECT title FROM book_embeddings ORDER BY embedding <-> $query::vector LIMIT 3".query[String]
  }

  nearest.foreach(println)
}
```

</details>

Run it:

```bash
sbt "runMain storeandsearchvectorembeddings.NearestExample"
```

### Pick a Distance Operator

This example ranks the same books with each of the three distance operators and prints the distances.

<details>
  <summary>sql-examples/src/main/scala/storeandsearchvectorembeddings/OperatorsExample.scala</summary>

```scala title="sql-examples/src/main/scala/storeandsearchvectorembeddings/OperatorsExample.scala" showLineNumbers
package storeandsearchvectorembeddings

/**
 * How to Store and Search Vector Embeddings in PostgreSQL: Pick a Distance
 * Operator
 *
 * Ranks the same books with each of the three pgvector distance operators and
 * prints the distance next to every title.
 *
 * Run with: cd sql-examples && sbt "runMain
 * storeandsearchvectorembeddings.OperatorsExample"
 */
object OperatorsExample extends App {

  val transactor = shelfTransactor()

  val query = Embedding(IndexedSeq(0.6f, 0.9f, 1.0f))

  transactor.transact {
    createShelf()
    stockShelf()

    println("Euclidean distance (<->):")
    sql"SELECT title, embedding <-> $query::vector FROM book_embeddings ORDER BY embedding <-> $query::vector LIMIT 3"
      .query[(String, Double)]
      .foreach((title, distance) => println(f"  $title%-32s $distance%.4f"))

    println("Cosine distance (<=>):")
    sql"SELECT title, embedding <=> $query::vector FROM book_embeddings ORDER BY embedding <=> $query::vector LIMIT 3"
      .query[(String, Double)]
      .foreach((title, distance) => println(f"  $title%-32s $distance%.4f"))

    println("Negative inner product (<#>):")
    sql"SELECT title, embedding <#> $query::vector FROM book_embeddings ORDER BY embedding <#> $query::vector LIMIT 3"
      .query[(String, Double)]
      .foreach((title, distance) => println(f"  $title%-32s $distance%.4f"))
  }
}
```

</details>

Run it:

```bash
sbt "runMain storeandsearchvectorembeddings.OperatorsExample"
```

### Read Embeddings Back

This example reads one embedding and two rows, and checks that the values survive the round trip.

<details>
  <summary>sql-examples/src/main/scala/storeandsearchvectorembeddings/ReadBackExample.scala</summary>

```scala title="sql-examples/src/main/scala/storeandsearchvectorembeddings/ReadBackExample.scala" showLineNumbers
package storeandsearchvectorembeddings

/**
 * How to Store and Search Vector Embeddings in PostgreSQL: Read Embeddings Back
 *
 * Reads one stored embedding and a whole row as an `Embedding`, and shows that
 * the values survive the round trip through the `vector` column.
 *
 * Run with: cd sql-examples && sbt "runMain
 * storeandsearchvectorembeddings.ReadBackExample"
 */
object ReadBackExample extends App {

  val transactor = shelfTransactor()

  transactor.transact {
    createShelf()
    stockShelf()

    val stored = sql"SELECT embedding FROM book_embeddings WHERE title = ${"Zionomicon"}".queryOne[Embedding]
    println(s"Stored:   ${stored.map(_.values.mkString("[", ", ", "]"))}")
    println(s"Expected: ${Some(shelfBooks.head._2.values.mkString("[", ", ", "]"))}")
    println(s"Same:     ${stored.contains(shelfBooks.head._2)}")

    val rows = sql"SELECT title, embedding FROM book_embeddings ORDER BY id LIMIT 2".query[(String, Embedding)]
    rows.foreach((title, embedding) => println(s"$title -> ${embedding.values.size} dimensions"))
  }
}
```

</details>

Run it:

```bash
sbt "runMain storeandsearchvectorembeddings.ReadBackExample"
```

### Reject the Wrong Number of Dimensions

This example inserts an embedding with four numbers into a `vector(3)` column and prints the error.

<details>
  <summary>sql-examples/src/main/scala/storeandsearchvectorembeddings/DimensionMismatchExample.scala</summary>

```scala title="sql-examples/src/main/scala/storeandsearchvectorembeddings/DimensionMismatchExample.scala" showLineNumbers
package storeandsearchvectorembeddings

/**
 * How to Store and Search Vector Embeddings in PostgreSQL: Reject the Wrong
 * Number of Dimensions
 *
 * Inserts an embedding with four numbers into a `vector(3)` column, prints the
 * error PostgreSQL raises, and shows that the table still holds only the
 * stocked books.
 *
 * Run with: cd sql-examples && sbt "runMain
 * storeandsearchvectorembeddings.DimensionMismatchExample"
 */
object DimensionMismatchExample extends App {

  val transactor = shelfTransactor()

  transactor.transact {
    createShelf()
    stockShelf()
  }

  val tooLong = Embedding(IndexedSeq(0.1f, 0.2f, 0.3f, 0.4f))

  try
    transactor.transact {
      sql"INSERT INTO book_embeddings (title, embedding) VALUES (${"Too Long"}, $tooLong::vector)".update
    }
  catch {
    case e: SQLException => println(s"Rejected: ${e.getMessage}")
  }

  val count = transactor.connect(sql"SELECT COUNT(*) FROM book_embeddings".queryOne[Long].getOrElse(0L))
  println(s"Rows stored: $count")
}
```

</details>

Run it:

```bash
sbt "runMain storeandsearchvectorembeddings.DimensionMismatchExample"
```

### Speed Up Search with an HNSW Index

This example creates an HNSW index and checks which of two searches can use it.

<details>
  <summary>sql-examples/src/main/scala/storeandsearchvectorembeddings/HnswIndexExample.scala</summary>

```scala title="sql-examples/src/main/scala/storeandsearchvectorembeddings/HnswIndexExample.scala" showLineNumbers
package storeandsearchvectorembeddings

/**
 * How to Store and Search Vector Embeddings in PostgreSQL: Speed Up Search with
 * an HNSW Index
 *
 * Creates an HNSW index with the cosine operator class, then asks PostgreSQL
 * for the query plan of a cosine search and of a Euclidean search. Only the
 * search whose operator matches the operator class can use the index.
 *
 * Run with: cd sql-examples && sbt "runMain
 * storeandsearchvectorembeddings.HnswIndexExample"
 */
object HnswIndexExample extends App {

  val transactor = shelfTransactor()

  val query = Embedding(IndexedSeq(0.6f, 0.9f, 1.0f))

  transactor.transact {
    createShelf()
    stockShelf()

    sql"CREATE INDEX book_embeddings_hnsw ON book_embeddings USING hnsw (embedding vector_cosine_ops)".update

    // With five rows PostgreSQL prefers a sequential scan, so turn it off for this transaction to see what the index can do.
    sql"SET LOCAL enable_seqscan = off".update

    val cosinePlan =
      sql"EXPLAIN SELECT title FROM book_embeddings ORDER BY embedding <=> $query::vector LIMIT 3".query[String]
    val l2Plan =
      sql"EXPLAIN SELECT title FROM book_embeddings ORDER BY embedding <-> $query::vector LIMIT 3".query[String]

    println(s"Cosine search uses the index:    ${cosinePlan.exists(_.contains("book_embeddings_hnsw"))}")
    println(s"Euclidean search uses the index: ${l2Plan.exists(_.contains("book_embeddings_hnsw"))}")
  }
}
```

</details>

Run it:

```bash
sbt "runMain storeandsearchvectorembeddings.HnswIndexExample"
```

### Skip the Casts with a Driver Setting

This example runs the same cast-free statements on a default connection and on one with `stringtype=unspecified`.

<details>
  <summary>sql-examples/src/main/scala/storeandsearchvectorembeddings/StringTypeExample.scala</summary>

```scala title="sql-examples/src/main/scala/storeandsearchvectorembeddings/StringTypeExample.scala" showLineNumbers
package storeandsearchvectorembeddings

/**
 * How to Store and Search Vector Embeddings in PostgreSQL: Skip the Cast with
 * stringtype=unspecified
 *
 * Runs the same cast-free insert and search twice: on a connection with the
 * default settings, where PostgreSQL rejects the `varchar` parameter, and on a
 * connection whose URL sets `stringtype=unspecified`, where it accepts it.
 *
 * Run with: cd sql-examples && sbt "runMain
 * storeandsearchvectorembeddings.StringTypeExample"
 */
object StringTypeExample extends App {

  val baseUrl = sys.env.getOrElse("BOOKSTORE_DB_URL", "jdbc:postgresql://localhost:5432/bookstore")
  val user    = sys.env.getOrElse("BOOKSTORE_DB_USER", "bookstore")
  val secret  = sys.env.getOrElse("BOOKSTORE_DB_PASSWORD", "bookstore")

  val query = Embedding(IndexedSeq(0.6f, 0.9f, 1.0f))

  def withoutCasts(label: String, url: String): Unit = {
    val transactor = JdbcTransactor.fromUrl(url, user, secret, SqlDialect.PostgreSQL)
    try {
      val nearest = transactor.transact {
        createShelf()
        sql"INSERT INTO book_embeddings (title, embedding) VALUES (${"Zionomicon"}, $query)".update
        sql"SELECT title FROM book_embeddings ORDER BY embedding <-> $query LIMIT 1".query[String]
      }
      println(s"$label: stored and found ${nearest.mkString(", ")}")
    } catch {
      case e: SQLException => println(s"$label: ${e.getMessage.linesIterator.next()}")
    }
  }

  withoutCasts("default settings          ", baseUrl)
  withoutCasts(
    "stringtype=unspecified    ",
    baseUrl + (if (baseUrl.contains("?")) "&" else "?") + "stringtype=unspecified"
  )
}
```

</details>

Run it:

```bash
sbt "runMain storeandsearchvectorembeddings.StringTypeExample"
```

### The Complete Example

The complete example from "Putting It Together" runs the whole flow in one program:

```bash
sbt "runMain storeandsearchvectorembeddings.CompleteExample"
```

To compile every example without running them, which needs no database:

```bash
sbt compile
```

When you are done, stop the database container:

```bash
docker stop bookstore-db
```

## Going Further

- [DbCodec](../reference/sql/db-codec.md) describes `DbCodec#transform` and the other ways to build a codec.
- [How to Map Custom Column Types](./map-custom-column-types.md) covers the same pattern for newtypes, enums, JSON documents and arrays.
- [How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md) builds the transactor on a HikariCP pool for production use.
- [The SQL Module](../reference/sql/index.md) overview links to every `sql` reference page.
- The `pgvector` client library for Java, in the `com.pgvector` group, offers a `PGvector` type for the driver. It is an alternative to the `String` codec in this guide; this guide adds no dependency for it.

---

## How to Store Case Classes in MongoDB

In this guide, we store the tickets of a small help desk in MongoDB with the `schema-bson` module of ZIO Blocks Schema. By the end, you will have one codec, derived from a `Schema`, that turns a `Ticket` case class into a `BsonDocument` and back, uses MongoDB's own `ObjectId` as the `_id`, stores a sealed trait in the shape your queries need, and reads tickets that the official MongoDB Java driver finds.

We start from the ticket types and build up in this order:

- Derive the codec, turn a ticket into a BSON document and read it back.
- Use a MongoDB `ObjectId` as the document id.
- Choose how the `Status` sealed trait is stored.
- Store optional fields in a shape other MongoDB programs expect.
- Insert and query tickets with the MongoDB Java driver.
- Read the errors the codec reports for documents it can't decode, and decide what to do with extra fields.

## The Problem

MongoDB stores BSON documents, and the Java driver reads and writes `BsonDocument` or its own `Document` type, not your case classes. Without a codec, every type needs two hand-written functions, one that builds the document and one that reads it back, and a sealed trait needs a tag of your own design.

For example, consider a support ticket with a status and a list of comments. With the `org.bson` classes alone, the mapping looks like this:

```scala

sealed trait Status
case object Open                              extends Status
final case class InProgress(assignee: String) extends Status
final case class Closed(resolution: String)   extends Status

final case class Comment(authorName: String, body: String)
final case class Ticket(id: ObjectId, title: String, status: Status, comments: List[Comment])

def toDocument(ticket: Ticket): BsonDocument = {
  val status = ticket.status match {
    case Open                 => new BsonDocument("type", new BsonString("Open"))
    case InProgress(assignee) =>
      new BsonDocument("type", new BsonString("InProgress")).append("assignee", new BsonString(assignee))
    case Closed(resolution) =>
      new BsonDocument("type", new BsonString("Closed")).append("resolution", new BsonString(resolution))
  }
  val comments = new BsonArray()
  ticket.comments.foreach { comment =>
    comments.add(
      new BsonDocument("authorName", new BsonString(comment.authorName))
        .append("body", new BsonString(comment.body))
    )
  }
  new BsonDocument("_id", new BsonObjectId(ticket.id))
    .append("title", new BsonString(ticket.title))
    .append("status", status)
    .append("comments", comments)
}

// Every field name is repeated here: a typo compiles, and fails only when a document is read
def fromDocument(document: BsonDocument): Ticket = {
  val status = document.getDocument("status")
  Ticket(
    id = document.getObjectId("_id").getValue,
    title = document.getString("title").getValue,
    status = status.getString("type").getValue match {
      case "Open"       => Open
      case "InProgress" => InProgress(status.getString("assignee").getValue)
      case "Closed"     => Closed(status.getString("resolution").getValue)
    },
    comments = document.getArray("comments").getValues.asScala.toList.map { value =>
      val comment = value.asDocument()
      Comment(comment.getString("authorName").getValue, comment.getString("body").getValue)
    }
  )
}
```

Add a field or a new status, and both functions must change in step. A missing field throws `BsonInvalidOperationException` (`Document does not contain key authorName`) from deep inside `fromDocument`, without saying which comment of which ticket, and an unknown status throws a `MatchError`. In this guide, the `Schema` of each type derives the codec instead, so the document shape follows the case class.

## Prerequisites

Add the BSON module and the MongoDB Java driver to your build. The module runs on the JVM only, and the code in this guide compiles on Scala 2.13 and Scala 3:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema-bson" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema-bson" % "0.0.56"

// The synchronous MongoDB Java driver
libraryDependencies += "org.mongodb" % "mongodb-driver-sync" % "5.13.0"
```

The BSON module depends on `org.mongodb:bson` 5.13.0, the BSON library that the driver itself uses, so `BsonDocument` and `ObjectId` are on the classpath without the driver. Driver 5.13.0 depends on the same `bson` version. We assume you can derive a `Schema` for a case class, as shown in [Your First Schema](./your-first-schema.md).

The sections that talk to MongoDB need a running server. If you have Docker, this command starts MongoDB 7 on the default port:

```bash
docker run --rm -d -p 27017:27017 --name mongo mongo:7
```

Every code block in this guide uses these imports:

```scala

```

## The Ticket Model

A help-desk ticket has a MongoDB id, a title, a status, an optional customer email and a list of comments. `Status` is a sealed trait whose cases carry different data:

```scala
sealed trait Status
case object Open                              extends Status
final case class InProgress(assignee: String) extends Status
final case class Closed(resolution: String)   extends Status

object Status {
  implicit val schema: Schema[Status] = Schema.derived
}

final case class Comment(authorName: String, body: String)

object Comment {
  implicit val schema: Schema[Comment] = Schema.derived
}

final case class Ticket(
  @Modifier.rename("_id") id: ObjectId,
  title: String,
  status: Status,
  customerEmail: Option[String] = None,
  comments: List[Comment] = Nil
)

object Ticket {

  implicit val schema: Schema[Ticket] = Schema.derived
}
```

Two lines in `Ticket` are there for MongoDB. `Modifier.rename("_id")` stores the `id` field under `_id`, the field MongoDB uses as the primary key, while Scala code keeps the name `id`. The `ObjectIdSupport.objectIdSchema` import gives `Schema.derived` a schema for `org.bson.types.ObjectId`, which has none of its own; without it, derivation fails with `Cannot derive schema for 'org.bson.types.ObjectId'`. The defaults on `customerEmail` and `comments` matter when a stored document lacks those fields, as [Store Optional Fields](#store-optional-fields) shows.

The cases of `Status` are declared next to the trait, not inside its companion object. Both layouts work in a normal source file, but `Schema.derived` crashes the compiler when the trait is declared inside a class and its cases sit in its companion, a known bug in 0.0.56, and this page's code is compiled that way.

## Turn a Ticket into a BsonDocument and Back

`Schema#derive` with `BsonCodecDeriver` derives a `BsonCodec`, which holds a `BsonEncoder` and a `BsonDecoder`. `BsonEncoder#toBsonValue` turns a value into a `BsonValue`, and for a case class that value is a document:

```scala
val ticketCodec: BsonCodec[Ticket] = Schema[Ticket].derive(BsonCodecDeriver)

val loginBug = Ticket(
  id = new ObjectId("6720f1a2b3c4d5e6f7a8b9c0"),
  title = "Login fails on Safari",
  status = InProgress("ana"),
  customerEmail = Some("kim@example.com"),
  comments = List(Comment("kim", "Only on Safari 17"))
)

val document: BsonDocument = ticketCodec.encoder.toBsonValue(loginBug).asDocument()
```

`BsonDocument#toJson` prints the document in MongoDB's Extended JSON, the form `mongosh` shows:

```scala
println(document.toJson)
// {"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Login fails on Safari", "status": {"InProgress": {"assignee": "ana"}}, "customerEmail": {"Some": {"value": "kim@example.com"}}, "comments": [{"authorName": "kim", "body": "Only on Safari 17"}]}
```

Each case class field becomes a document field with the same name, `List` becomes an array, and nested case classes become nested documents. The `status` and `customerEmail` shapes are the module's defaults for sealed traits and for `Option`; the next sections change both.

To read the document back, `BsonDecoder#fromBsonValue` returns `Either[BsonDecoder.Error, Ticket]`:

```scala
println(ticketCodec.decoder.fromBsonValue(document))
// Right(Ticket(6720f1a2b3c4d5e6f7a8b9c0,Login fails on Safari,InProgress(ana),Some(kim@example.com),List(Comment(kim,Only on Safari 17))))
println(ticketCodec.decoder.fromBsonValue(document) == Right(loginBug))
// true
```

<details>
  <summary>Runnable version, with the shared model: <code>Step1EncodeDecodeExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/storecaseclassesinmongodb/Model.scala" showLineNumbers
package storecaseclassesinmongodb

/**
 * How to Store Case Classes in MongoDB — the ticket model shared by every
 * example in this folder.
 */
sealed trait Status
case object Open                              extends Status
final case class InProgress(assignee: String) extends Status
final case class Closed(resolution: String)   extends Status

object Status {
  implicit val schema: Schema[Status] = Schema.derived
}

final case class Comment(authorName: String, body: String)

object Comment {
  implicit val schema: Schema[Comment] = Schema.derived
}

final case class Ticket(
  @Modifier.rename("_id") id: ObjectId,
  title: String,
  status: Status,
  customerEmail: Option[String] = None,
  comments: List[Comment] = Nil
)

object Ticket {

  implicit val schema: Schema[Ticket] = Schema.derived
}
```

```scala title="schema-examples/src/main/scala/storecaseclassesinmongodb/Step1EncodeDecodeExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-bson:0.0.56
//> using options -experimental
//> using file Model.scala
// Run from this folder:  scala-cli run Step1EncodeDecodeExample.scala
// Or from schema-examples:  sbt "runMain storecaseclassesinmongodb.Step1EncodeDecodeExample"

package storecaseclassesinmongodb

/**
 * How to Store Case Classes in MongoDB — Turn a Ticket into a BsonDocument and
 * Back
 *
 * Derives the ticket codec, prints one ticket as a BSON document and decodes it
 * back.
 */
object Step1EncodeDecodeExample {
  def main(args: Array[String]): Unit = {
    val ticketCodec: BsonCodec[Ticket] = Schema[Ticket].derive(BsonCodecDeriver)

    val loginBug = Ticket(
      id = new ObjectId("6720f1a2b3c4d5e6f7a8b9c0"),
      title = "Login fails on Safari",
      status = InProgress("ana"),
      customerEmail = Some("kim@example.com"),
      comments = List(Comment("kim", "Only on Safari 17"))
    )

    val document: BsonDocument = ticketCodec.encoder.toBsonValue(loginBug).asDocument()

    println(document.toJson)

    println(ticketCodec.decoder.fromBsonValue(document))
    println(ticketCodec.decoder.fromBsonValue(document) == Right(loginBug))
  }
}
```

</details>

## Use MongoDB's ObjectId as the Document Id

Every MongoDB document has an `_id`, and most collections use an `ObjectId`: 12 bytes that start with a timestamp, generated without asking the server. With the `ObjectIdSupport` import, the codec stores an `ObjectId` field as BSON's own `ObjectId` type, not as its hex string:

```scala
println(document.get("_id").getBsonType)
// OBJECT_ID
```

That is the type the `_id` index and other MongoDB tools expect, and a document whose `_id` holds the hex string as a plain string does not decode:

```scala
println(ticketCodec.decoder.fromBsonValue(BsonDocument.parse(
  """{"_id": "6720f1a2b3c4d5e6f7a8b9c0", "title": "Login fails on Safari", "status": {"Open": {}}}"""
)))
// Left(zio.blocks.schema.bson.BsonDecoder$Error: Path: ._id, error: Expected OBJECT_ID but got STRING)
```

We create the id in the application with `new ObjectId()` before inserting the ticket, so the `Ticket` value and the stored document carry the same id from the start. The Java driver adds a generated `_id` to a document that has none, but a `Ticket` always has an id, so the codec always writes one.

:::warning
Don't make the id an `Option[ObjectId]` so the driver can fill it in. In 0.0.56 an `Option` is stored as a sub-document (see [Store Optional Fields](#store-optional-fields)), so `None` becomes the id `{"None": {}}`, and the second such insert fails with `E11000 duplicate key error ... dup key: { _id: { None: {} } }`.
:::

You don't need `BsonSchemaCodec.Config#withNativeObjectId` for this: the `ObjectIdSupport` import already selects the native type, and in 0.0.56 the setting changes nothing.

<details>
  <summary>Runnable version, with setup from the previous section: <code>Step2ObjectIdExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/storecaseclassesinmongodb/Step2ObjectIdExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-bson:0.0.56
//> using options -experimental
//> using file Model.scala
// Run from this folder:  scala-cli run Step2ObjectIdExample.scala
// Or from schema-examples:  sbt "runMain storecaseclassesinmongodb.Step2ObjectIdExample"

package storecaseclassesinmongodb

/**
 * How to Store Case Classes in MongoDB — Use MongoDB's ObjectId as the Document
 * Id
 *
 * Shows that the id is stored as BSON's own ObjectId type and that a string id
 * does not decode.
 */
object Step2ObjectIdExample {
  def main(args: Array[String]): Unit = {
    // Setup from "Turn a Ticket into a BsonDocument and Back": the codec and one ticket
    val ticketCodec: BsonCodec[Ticket] = Schema[Ticket].derive(BsonCodecDeriver)

    val loginBug = Ticket(
      id = new ObjectId("6720f1a2b3c4d5e6f7a8b9c0"),
      title = "Login fails on Safari",
      status = InProgress("ana"),
      customerEmail = Some("kim@example.com"),
      comments = List(Comment("kim", "Only on Safari 17"))
    )

    val document: BsonDocument = ticketCodec.encoder.toBsonValue(loginBug).asDocument()

    println(document.get("_id").getBsonType)

    println(
      ticketCodec.decoder.fromBsonValue(
        BsonDocument.parse(
          """{"_id": "6720f1a2b3c4d5e6f7a8b9c0", "title": "Login fails on Safari", "status": {"Open": {}}}"""
        )
      )
    )
  }
}
```

</details>

## Choose How Sealed Traits Are Stored

A sealed trait needs a marker in the document that says which case it holds, and the shape of that marker decides how you query by case. `BsonSchemaCodec.bsonCodec` takes a `BsonSchemaCodec.Config`, and `BsonSchemaCodec.Config#withSumTypeHandling` picks one of three shapes. To compare them, we encode each `Status` case under each setting, inside a `status` field as it appears in a ticket:

```scala
def storedAs(config: Config, status: Status): String =
  new BsonDocument("status", BsonSchemaCodec.bsonCodec(Schema[Status], config).encoder.toBsonValue(status)).toJson

val statuses = List[Status](Open, InProgress("ana"), Closed("refunded"))
```

Printing every case under each setting, plus one with lower-case case names, shows the shapes side by side:

```scala
println("WrapperWithClassNameField (the default):")
// WrapperWithClassNameField (the default):
statuses.foreach(status => println("  " + storedAs(Config, status)))
//   {"status": {"Open": {}}}
//   {"status": {"InProgress": {"assignee": "ana"}}}
//   {"status": {"Closed": {"resolution": "refunded"}}}

println("DiscriminatorField(\"type\"):")
// DiscriminatorField("type"):
statuses.foreach(status => println("  " + storedAs(Config.withSumTypeHandling(SumTypeHandling.DiscriminatorField("type")), status)))
//   {"status": {"type": "Open"}}
//   {"status": {"assignee": "ana", "type": "InProgress"}}
//   {"status": {"resolution": "refunded", "type": "Closed"}}

println("NoDiscriminator:")
// NoDiscriminator:
statuses.foreach(status => println("  " + storedAs(Config.withSumTypeHandling(SumTypeHandling.NoDiscriminator), status)))
//   {"status": "Open"}
//   {"status": {"assignee": "ana"}}
//   {"status": {"resolution": "refunded"}}

println("WrapperWithClassNameField with withClassNameMapping(_.toLowerCase):")
// WrapperWithClassNameField with withClassNameMapping(_.toLowerCase):
statuses.foreach(status => println("  " + storedAs(Config.withClassNameMapping(_.toLowerCase), status)))
//   {"status": {"open": {}}}
//   {"status": {"inprogress": {"assignee": "ana"}}}
//   {"status": {"closed": {"resolution": "refunded"}}}
```

`BsonSchemaCodec.Config#withClassNameMapping` changes the stored case names, here to lower case. The three shapes lead to different queries:

| Setting                      | Stored shape                             | Query for closed tickets                           |
| ---------------------------- | ---------------------------------------- | -------------------------------------------------- |
| `WrapperWithClassNameField`  | the case name wraps the case's fields    | `Filters.exists("status.Closed")`                  |
| `DiscriminatorField("type")` | a `type` field next to the case's fields | `Filters.eq("status.type", "Closed")`              |
| `NoDiscriminator`            | the case's fields only                   | none by case name; a case object is a plain string |

`NoDiscriminator` stores no case name, so the decoder tries the cases in order and keeps the first one that decodes; two cases with the same fields can't be told apart.

A `Config` applies to every sealed trait in the codec, and in 0.0.56 that includes `Option`, so a whole-codec `DiscriminatorField` setting also changes how `customerEmail` is stored:

```scala
println(
  BsonSchemaCodec
    .bsonCodec(Schema[Ticket], Config.withSumTypeHandling(SumTypeHandling.DiscriminatorField("type")))
    .encoder
    .toBsonValue(loginBug.copy(customerEmail = None))
    .asDocument()
    .toJson
)
// {"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Login fails on Safari", "status": {"assignee": "ana", "type": "InProgress"}, "customerEmail": {"type": "None"}, "comments": [{"authorName": "kim", "body": "Only on Safari 17"}]}
```

To change one sealed trait only, put modifiers on the type itself instead. `Modifier.discriminator` stores a discriminator field for this trait only, and `Modifier.caseNaming` renames its cases; on a case class, `Modifier.fieldNaming` renames every field. Here they are on separate copies of `Status` and `Comment`, declared inside an object so the names don't clash with the model:

```scala
object annotated {
  @Modifier.discriminator("type")
  @Modifier.caseNaming("snake_case")
  sealed trait Status
  case object Open                              extends Status
  final case class InProgress(assignee: String) extends Status

  object Status {
    implicit val schema: Schema[Status] = Schema.derived
  }

  @Modifier.fieldNaming("snake_case")
  final case class Comment(authorName: String, body: String)

  object Comment {
    implicit val schema: Schema[Comment] = Schema.derived
  }
}

val annotatedStatusCodec = Schema[annotated.Status].derive(BsonCodecDeriver)
```

The annotated `Status` stores a `type` field with snake_case case names, and the annotated `Comment` stores snake_case field names:

```scala
println(annotatedStatusCodec.encoder.toBsonValue(annotated.InProgress("ana")).asDocument().toJson)
// {"assignee": "ana", "type": "in_progress"}
println(annotatedStatusCodec.encoder.toBsonValue(annotated.Open).asDocument().toJson)
// {"type": "open"}

println(Schema[annotated.Comment].derive(BsonCodecDeriver).encoder.toBsonValue(annotated.Comment("kim", "Thanks")).asDocument().toJson)
// {"author_name": "kim", "body": "Thanks"}
```

A modifier on a type wins over the `Config` for that type. The same modifiers also change JSON and the other formats derived from the schema; [How to Customize JSON Encoding](./customize-json-encoding.md) covers them in detail. The rest of this guide keeps the default wrapper shape for `Status`.

<details>
  <summary>Runnable version: <code>Step3SumTypesExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/storecaseclassesinmongodb/Step3SumTypesExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-bson:0.0.56
//> using options -experimental
//> using file Model.scala
// Run from this folder:  scala-cli run Step3SumTypesExample.scala
// Or from schema-examples:  sbt "runMain storecaseclassesinmongodb.Step3SumTypesExample"

package storecaseclassesinmongodb

object annotated {
  @Modifier.discriminator("type")
  @Modifier.caseNaming("snake_case")
  sealed trait Status
  case object Open                              extends Status
  final case class InProgress(assignee: String) extends Status

  object Status {
    implicit val schema: Schema[Status] = Schema.derived
  }

  @Modifier.fieldNaming("snake_case")
  final case class Comment(authorName: String, body: String)

  object Comment {
    implicit val schema: Schema[Comment] = Schema.derived
  }
}

/**
 * How to Store Case Classes in MongoDB — Choose How Sealed Traits Are Stored
 *
 * Encodes each Status case under each sum-type setting, shows that a codec-wide
 * setting also changes Option, and applies modifiers to single types.
 */
object Step3SumTypesExample {
  def storedAs(config: Config, status: Status): String =
    new BsonDocument("status", BsonSchemaCodec.bsonCodec(Schema[Status], config).encoder.toBsonValue(status)).toJson

  def main(args: Array[String]): Unit = {
    // Setup from "Turn a Ticket into a BsonDocument and Back": one ticket
    val loginBug = Ticket(
      id = new ObjectId("6720f1a2b3c4d5e6f7a8b9c0"),
      title = "Login fails on Safari",
      status = InProgress("ana"),
      customerEmail = Some("kim@example.com"),
      comments = List(Comment("kim", "Only on Safari 17"))
    )

    val statuses = List[Status](Open, InProgress("ana"), Closed("refunded"))

    println("WrapperWithClassNameField (the default):")
    statuses.foreach(status => println("  " + storedAs(Config, status)))

    println("DiscriminatorField(\"type\"):")
    statuses.foreach(status =>
      println("  " + storedAs(Config.withSumTypeHandling(SumTypeHandling.DiscriminatorField("type")), status))
    )

    println("NoDiscriminator:")
    statuses.foreach(status =>
      println("  " + storedAs(Config.withSumTypeHandling(SumTypeHandling.NoDiscriminator), status))
    )

    println("WrapperWithClassNameField with withClassNameMapping(_.toLowerCase):")
    statuses.foreach(status => println("  " + storedAs(Config.withClassNameMapping(_.toLowerCase), status)))

    println(
      BsonSchemaCodec
        .bsonCodec(Schema[Ticket], Config.withSumTypeHandling(SumTypeHandling.DiscriminatorField("type")))
        .encoder
        .toBsonValue(loginBug.copy(customerEmail = None))
        .asDocument()
        .toJson
    )

    val annotatedStatusCodec = Schema[annotated.Status].derive(BsonCodecDeriver)
    println(annotatedStatusCodec.encoder.toBsonValue(annotated.InProgress("ana")).asDocument().toJson)
    println(annotatedStatusCodec.encoder.toBsonValue(annotated.Open).asDocument().toJson)

    println(
      Schema[annotated.Comment]
        .derive(BsonCodecDeriver)
        .encoder
        .toBsonValue(annotated.Comment("kim", "Thanks"))
        .asDocument()
        .toJson
    )
  }
}
```

</details>

## Store Optional Fields

In 0.0.56 the codec stores an `Option` like any other sealed trait: `Some` and `None` are its cases, so `customerEmail` becomes a small document:

```scala
def storedEmail(codec: BsonCodec[Ticket], ticket: Ticket): String =
  new BsonDocument("customerEmail", codec.encoder.toBsonValue(ticket).asDocument().get("customerEmail")).toJson

println(storedEmail(ticketCodec, loginBug))
// {"customerEmail": {"Some": {"value": "kim@example.com"}}}
println(storedEmail(ticketCodec, loginBug.copy(customerEmail = None)))
// {"customerEmail": {"None": {}}}
```

The codec reads its own documents back, and the `= None` default on `customerEmail` lets it read a document that has no `customerEmail` field. A document written by another program, which stores a missing email as `null` and a present one as a plain string, fails to decode:

```scala
def decodeEmail(json: String) = ticketCodec.decoder.fromBsonValue(BsonDocument.parse(json)).map(_.customerEmail)

println(decodeEmail("""{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Login fails on Safari", "status": {"Open": {}}}"""))
// Right(None)
println(decodeEmail("""{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Login fails on Safari", "status": {"Open": {}}, "customerEmail": null}"""))
// Left(zio.blocks.schema.bson.BsonDecoder$Error: Path: .customerEmail, error: Expected DOCUMENT but got NULL)
println(decodeEmail("""{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Login fails on Safari", "status": {"Open": {}}, "customerEmail": "kim@example.com"}"""))
// Left(zio.blocks.schema.bson.BsonDecoder$Error: Path: .customerEmail, error: Expected DOCUMENT but got STRING)
```

:::warning
Storing `Option` as `{"Some": {"value": ...}}` and `{"None": {}}` is a known bug in 0.0.56. It also means a query for tickets without an email must match `{"None": {}}` instead of `null`. Without the `= None` default, a document that lacks the field fails with `Missing required field: customerEmail`.
:::

Until that is fixed, we give `Option[String]` a codec of our own that stores `None` as BSON `null` and `Some` as the plain string. A `BsonCodec` is a pair of an encoder and a decoder; each has a method for the streaming `BsonWriter`/`BsonReader` path and one for whole `BsonValue`s:

```scala

val nullableString: BsonCodec[Option[String]] = BsonCodec(
  new BsonEncoder[Option[String]] {
    def encode(writer: BsonWriter, value: Option[String], ctx: BsonEncoder.EncoderContext): Unit =
      value match {
        case Some(text) => writer.writeString(text)
        case None       => writer.writeNull()
      }

    def toBsonValue(value: Option[String]): BsonValue =
      value.fold[BsonValue](BsonNull.VALUE)(text => new BsonString(text))
  },
  new BsonDecoder[Option[String]] {
    def decodeUnsafe(
      reader: BsonReader,
      trace: List[BsonTrace],
      ctx: BsonDecoder.BsonDecoderContext
    ): Option[String] =
      reader.getCurrentBsonType match {
        case BsonType.NULL   => reader.readNull(); None
        case BsonType.STRING => Some(reader.readString())
        case other           => throw BsonDecoder.Error(trace, s"Expected STRING or NULL but got $other")
      }

    def fromBsonValueUnsafe(
      value: BsonValue,
      trace: List[BsonTrace],
      ctx: BsonDecoder.BsonDecoderContext
    ): Option[String] =
      if (value.isNull) None
      else if (value.isString) Some(value.asString().getValue)
      else throw BsonDecoder.Error(trace, s"Expected STRING or NULL but got ${value.getBsonType}")
  }
)
```

`Schema#deriving` starts a derivation we can adjust, and `DerivationBuilder#instance` with a `TypeId` uses our codec for every `Option[String]` in the ticket:

```scala
val nullableTicketCodec: BsonCodec[Ticket] =
  Schema[Ticket]
    .deriving(BsonCodecDeriver)
    .instance(TypeId.of[Option[String]], nullableString)
    .derive
```

The ticket now stores the email the way other MongoDB programs do, and reads their documents:

```scala
println(storedEmail(nullableTicketCodec, loginBug))
// {"customerEmail": "kim@example.com"}
println(storedEmail(nullableTicketCodec, loginBug.copy(customerEmail = None)))
// {"customerEmail": null}

def decodeEmailNullable(json: String) = nullableTicketCodec.decoder.fromBsonValue(BsonDocument.parse(json)).map(_.customerEmail)

println(decodeEmailNullable("""{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Login fails on Safari", "status": {"Open": {}}, "customerEmail": null}"""))
// Right(None)
println(decodeEmailNullable("""{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Login fails on Safari", "status": {"Open": {}}, "customerEmail": "kim@example.com"}"""))
// Right(Some(kim@example.com))
```

The override covers `Option[String]` only; another optional type, such as `Option[Int]`, needs a codec of its own. A missing field still decodes through the `= None` default, because the codec reads a missing field from the field's default value. The rest of this guide stores tickets with `nullableTicketCodec`. [How to Override the Codec for One Type](./override-codec-for-one-type.md) shows more ways to replace part of a derived codec.

<details>
  <summary>Runnable version, with the codec in a shared file: <code>Step4OptionalFieldsExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/storecaseclassesinmongodb/TicketCodec.scala" showLineNumbers
package storecaseclassesinmongodb

/**
 * How to Store Case Classes in MongoDB — Store Optional Fields
 *
 * A codec for `Option[String]` that stores `None` as BSON null and `Some` as a
 * plain string, and the ticket codec that uses it.
 */
object TicketCodec {
  val nullableString: BsonCodec[Option[String]] = BsonCodec(
    new BsonEncoder[Option[String]] {
      def encode(writer: BsonWriter, value: Option[String], ctx: BsonEncoder.EncoderContext): Unit =
        value match {
          case Some(text) => writer.writeString(text)
          case None       => writer.writeNull()
        }

      def toBsonValue(value: Option[String]): BsonValue =
        value.fold[BsonValue](BsonNull.VALUE)(text => new BsonString(text))
    },
    new BsonDecoder[Option[String]] {
      def decodeUnsafe(
        reader: BsonReader,
        trace: List[BsonTrace],
        ctx: BsonDecoder.BsonDecoderContext
      ): Option[String] =
        reader.getCurrentBsonType match {
          case BsonType.NULL   => reader.readNull(); None
          case BsonType.STRING => Some(reader.readString())
          case other           => throw BsonDecoder.Error(trace, s"Expected STRING or NULL but got $other")
        }

      def fromBsonValueUnsafe(
        value: BsonValue,
        trace: List[BsonTrace],
        ctx: BsonDecoder.BsonDecoderContext
      ): Option[String] =
        if (value.isNull) None
        else if (value.isString) Some(value.asString().getValue)
        else throw BsonDecoder.Error(trace, s"Expected STRING or NULL but got ${value.getBsonType}")
    }
  )

  val nullableTicketCodec: BsonCodec[Ticket] =
    Schema[Ticket]
      .deriving(BsonCodecDeriver)
      .instance(TypeId.of[Option[String]], nullableString)
      .derive
}
```

```scala title="schema-examples/src/main/scala/storecaseclassesinmongodb/Step4OptionalFieldsExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-bson:0.0.56
//> using options -experimental
//> using file Model.scala
//> using file TicketCodec.scala
// Run from this folder:  scala-cli run Step4OptionalFieldsExample.scala
// Or from schema-examples:  sbt "runMain storecaseclassesinmongodb.Step4OptionalFieldsExample"

package storecaseclassesinmongodb

/**
 * How to Store Case Classes in MongoDB — Store Optional Fields
 *
 * Shows how the derived codec stores an Option, which documents it cannot read,
 * and how the codec from TicketCodec.scala stores null and plain strings.
 */
object Step4OptionalFieldsExample {
  def main(args: Array[String]): Unit = {
    // Setup from "Turn a Ticket into a BsonDocument and Back": the codec and one ticket
    val ticketCodec: BsonCodec[Ticket] = Schema[Ticket].derive(BsonCodecDeriver)

    val loginBug = Ticket(
      id = new ObjectId("6720f1a2b3c4d5e6f7a8b9c0"),
      title = "Login fails on Safari",
      status = InProgress("ana"),
      customerEmail = Some("kim@example.com"),
      comments = List(Comment("kim", "Only on Safari 17"))
    )

    def storedEmail(codec: BsonCodec[Ticket], ticket: Ticket): String =
      new BsonDocument("customerEmail", codec.encoder.toBsonValue(ticket).asDocument().get("customerEmail")).toJson

    println(storedEmail(ticketCodec, loginBug))
    println(storedEmail(ticketCodec, loginBug.copy(customerEmail = None)))

    def decodeEmail(json: String) = ticketCodec.decoder.fromBsonValue(BsonDocument.parse(json)).map(_.customerEmail)

    println(
      decodeEmail(
        """{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Login fails on Safari", "status": {"Open": {}}}"""
      )
    )
    println(
      decodeEmail(
        """{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Login fails on Safari", "status": {"Open": {}}, "customerEmail": null}"""
      )
    )
    println(
      decodeEmail(
        """{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Login fails on Safari", "status": {"Open": {}}, "customerEmail": "kim@example.com"}"""
      )
    )

    val nullableTicketCodec = TicketCodec.nullableTicketCodec

    println(storedEmail(nullableTicketCodec, loginBug))
    println(storedEmail(nullableTicketCodec, loginBug.copy(customerEmail = None)))

    def decodeEmailNullable(json: String) =
      nullableTicketCodec.decoder.fromBsonValue(BsonDocument.parse(json)).map(_.customerEmail)

    println(
      decodeEmailNullable(
        """{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Login fails on Safari", "status": {"Open": {}}, "customerEmail": null}"""
      )
    )
    println(
      decodeEmailNullable(
        """{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Login fails on Safari", "status": {"Open": {}}, "customerEmail": "kim@example.com"}"""
      )
    )
  }
}
```

</details>

## Insert and Query with the MongoDB Java Driver

ZIO Blocks has no MongoDB `CodecProvider` or `CodecRegistry`, the driver's own way to plug in codecs. We don't need one: the driver reads and writes `BsonDocument` natively, so we open the collection as a `MongoCollection[BsonDocument]` and convert at the boundary. This glue is our own code, not a ZIO Blocks API, and it is short enough to copy:

```scala title="schema-examples/src/main/scala/storecaseclassesinmongodb/MongoGlue.scala" showLineNumbers
package storecaseclassesinmongodb

/**
 * How to Store Case Classes in MongoDB — Insert and Query with the MongoDB Java
 * Driver
 *
 * Our own glue between a schema-derived `BsonCodec` and the driver: ZIO Blocks
 * has no MongoDB `CodecProvider`, so we convert through `BsonDocument`.
 */
final class TypedCollection[A](val raw: MongoCollection[BsonDocument], codec: BsonCodec[A]) {
  def insert(value: A): Unit =
    raw.insertOne(codec.encoder.toBsonValue(value).asDocument())

  def find(filter: Bson): Either[BsonDecoder.Error, List[A]] = {
    val decoded = raw.find(filter).asScala.toList.map(document => codec.decoder.fromBsonValue(document))
    decoded.collectFirst { case Left(error) => Left(error) }
      .getOrElse(Right(decoded.collect { case Right(value) => value }))
  }
}

object Mongo {
  val uri: String =
    sys.env.getOrElse("MONGODB_URI", "mongodb://localhost:27017/?serverSelectionTimeoutMS=2000")

  def withDatabase[A](name: String)(use: MongoDatabase => A): A = {
    val client = MongoClients.create(uri)
    try use(client.getDatabase(name))
    catch {
      case _: MongoTimeoutException =>
        System.err.println(
          s"No MongoDB server answered at $uri. Start one with: docker run --rm -d -p 27017:27017 --name mongo mongo:7"
        )
        sys.exit(1)
    } finally client.close()
  }
}
```

`TypedCollection#insert` encodes a value and calls `MongoCollection#insertOne`. `TypedCollection#find` runs a driver filter and decodes every document it returns and, if any fail to decode, returns the error of the first one. `Mongo.withDatabase` opens a client for one block of work and closes it afterwards. Its default connection string sets `serverSelectionTimeoutMS=2000`, so with no server running the program stops after two seconds with a one-line message instead of waiting 30 seconds, the driver's default. Set `MONGODB_URI` to use another server.

The example below inserts three tickets and queries them with the driver's `Filters`. Each filter names fields as the codec stores them: `status.Closed` comes from the wrapper shape of `Status`, `customerEmail` is a plain string thanks to `nullableTicketCodec`, and `comments.authorName` matches any element of the `comments` array:

```scala title="schema-examples/src/main/scala/storecaseclassesinmongodb/Step5InsertAndQueryExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-bson:0.0.56
//> using dep org.mongodb:mongodb-driver-sync:5.13.0
//> using options -experimental
//> using file Model.scala
//> using file TicketCodec.scala
//> using file MongoGlue.scala
// Needs MongoDB:  docker run --rm -d -p 27017:27017 --name mongo mongo:7
// Run from this folder:  scala-cli run Step5InsertAndQueryExample.scala
// Or from schema-examples:  sbt "runMain storecaseclassesinmongodb.Step5InsertAndQueryExample"

package storecaseclassesinmongodb

/**
 * How to Store Case Classes in MongoDB — Insert and Query with the MongoDB Java
 * Driver
 *
 * Inserts three tickets into a real MongoDB collection and finds them by id, by
 * status, by customer email and by comment author.
 */
object Step5InsertAndQueryExample {
  def main(args: Array[String]): Unit =
    Mongo.withDatabase("helpdesk") { database =>
      val tickets = new TypedCollection(
        database.getCollection("tickets", classOf[BsonDocument]),
        TicketCodec.nullableTicketCodec
      )
      tickets.raw.drop() // start every run from an empty collection

      val loginBug = Ticket(
        id = new ObjectId(),
        title = "Login fails on Safari",
        status = InProgress("ana"),
        customerEmail = Some("kim@example.com"),
        comments = List(Comment("kim", "Only on Safari 17"))
      )
      val wrongInvoice = Ticket(new ObjectId(), "Invoice total is wrong", Closed("refunded"))
      val darkMode     = Ticket(new ObjectId(), "Add a dark mode", Open, Some("lee@example.com"))
      List(loginBug, wrongInvoice, darkMode).foreach(tickets.insert)

      println(s"stored: ${tickets.raw.find(Filters.eq("_id", loginBug.id)).first().toJson}")
      println(s"by id: ${tickets.find(Filters.eq("_id", loginBug.id))}")
      println(s"closed: ${tickets.find(Filters.exists("status.Closed")).map(_.map(_.title))}")
      println(s"from kim: ${tickets.find(Filters.eq("customerEmail", "kim@example.com")).map(_.map(_.title))}")
      println(s"no email: ${tickets.find(Filters.eq("customerEmail", null)).map(_.map(_.title))}")
      println(s"kim commented: ${tickets.find(Filters.eq("comments.authorName", "kim")).map(_.map(_.title))}")
    }
}
```

Against `mongo:7`, it prints the following (the ids differ on each run):

```text
stored: {"_id": {"$oid": "6ae0dfc103971ce61e94881e"}, "title": "Login fails on Safari", "status": {"InProgress": {"assignee": "ana"}}, "customerEmail": "kim@example.com", "comments": [{"authorName": "kim", "body": "Only on Safari 17"}]}
by id: Right(List(Ticket(6ae0dfc103971ce61e94881e,Login fails on Safari,InProgress(ana),Some(kim@example.com),List(Comment(kim,Only on Safari 17)))))
closed: Right(List(Invoice total is wrong))
from kim: Right(List(Login fails on Safari))
no email: Right(List(Invoice total is wrong))
kim commented: Right(List(Login fails on Safari))
```

`Filters.eq("customerEmail", null)` matches documents whose field is `null` or missing, which is how MongoDB itself treats `null` in a query. The driver also prints an SLF4J warning to standard error when no SLF4J logging backend is on the classpath; add one to your build to route the driver's logs.

## Handle Decode Errors and Extra Fields

A collection is shared over time: older versions of your program, scripts and other services write to it, so a stored document can differ from the case class. `BsonDecoder#fromBsonValue` returns a `BsonDecoder.Error` for such a document, and `BsonDecoder.Error#render` gives the path to the bad value followed by the message. The path is empty when the problem is in the top-level document:

```scala
def readTicket(json: String): Unit =
  ticketCodec.decoder.fromBsonValue(BsonDocument.parse(json)) match {
    case Right(ticket) => println(s"decoded: ${ticket.title}")
    case Left(error)   => println(s"failed: ${error.render}")
  }

readTicket("""{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "status": {"Open": {}}}""")
// failed: : Missing required field: title
readTicket("""{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": 42, "status": {"Open": {}}}""")
// failed: .title: Expected STRING but got INT32
readTicket("""{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Export to CSV", "status": {"Open": {}}, "comments": [{"author": "sam", "body": "+1"}]}""")
// failed: .comments[0]: Missing required field: authorName
readTicket("""{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Export to CSV", "status": {"Pending": {}}}""")
// failed: .status.Pending: Unknown case name: Pending
```

`BsonDecoder.Error#message` holds the message without the path, and `BsonDecoder.Error#toString` combines both as `Path: <path>, error: <message>`.

A field the case class doesn't have is skipped by default, so a document that another program extended still decodes. Here is such a document, with a `priority` field that `Ticket` lacks:

```scala
val withPriority =
  """{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Export to CSV", "status": {"Open": {}}, "priority": 2}"""
```

The default codec decodes it and ignores `priority`:

```scala
readTicket(withPriority)
// decoded: Export to CSV
```

To reject such documents instead, `BsonSchemaCodec.Config#withIgnoreExtraFields(false)` makes every record in the codec strict:

```scala
println(BsonSchemaCodec.bsonCodec(Schema[Ticket], Config.withIgnoreExtraFields(false)).decoder.fromBsonValue(BsonDocument.parse(withPriority)).left.map(_.render))
// Left(.priority: Invalid extra field.)
```

`Modifier.noExtraFields` does the same for one case class, whatever the `Config` says:

```scala
object strict {
  @Modifier.noExtraFields()
  final case class Comment(authorName: String, body: String)

  object Comment {
    implicit val schema: Schema[Comment] = Schema.derived
  }
}

println(
  Schema[strict.Comment]
    .derive(BsonCodecDeriver)
    .decoder
    .fromBsonValue(BsonDocument.parse("""{"authorName": "sam", "body": "+1", "likes": 3}"""))
    .left
    .map(_.render)
)
// Left(.likes: Invalid extra field.)
```

Ignoring extra fields is the safer default for a shared collection: a newer program can add a field without breaking older readers. Strict decoding suits collections that only your program writes, where an unknown field points to a bug.

<details>
  <summary>Runnable version: <code>Step6DecodeErrorsExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/storecaseclassesinmongodb/Step6DecodeErrorsExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-bson:0.0.56
//> using options -experimental
//> using file Model.scala
// Run from this folder:  scala-cli run Step6DecodeErrorsExample.scala
// Or from schema-examples:  sbt "runMain storecaseclassesinmongodb.Step6DecodeErrorsExample"

package storecaseclassesinmongodb

object strict {
  @Modifier.noExtraFields()
  final case class Comment(authorName: String, body: String)

  object Comment {
    implicit val schema: Schema[Comment] = Schema.derived
  }
}

/**
 * How to Store Case Classes in MongoDB — Handle Decode Errors and Extra Fields
 *
 * Decodes documents that don't match the ticket model and shows how extra
 * fields are skipped or rejected.
 */
object Step6DecodeErrorsExample {
  // Setup from "Turn a Ticket into a BsonDocument and Back": the codec
  val ticketCodec: BsonCodec[Ticket] = Schema[Ticket].derive(BsonCodecDeriver)

  def readTicket(json: String): Unit =
    ticketCodec.decoder.fromBsonValue(BsonDocument.parse(json)) match {
      case Right(ticket) => println(s"decoded: ${ticket.title}")
      case Left(error)   => println(s"failed: ${error.render}")
    }

  def main(args: Array[String]): Unit = {
    readTicket("""{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "status": {"Open": {}}}""")
    readTicket("""{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": 42, "status": {"Open": {}}}""")
    readTicket(
      """{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Export to CSV", "status": {"Open": {}}, "comments": [{"author": "sam", "body": "+1"}]}"""
    )
    readTicket("""{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Export to CSV", "status": {"Pending": {}}}""")

    val withPriority =
      """{"_id": {"$oid": "6720f1a2b3c4d5e6f7a8b9c0"}, "title": "Export to CSV", "status": {"Open": {}}, "priority": 2}"""

    readTicket(withPriority)

    println(
      BsonSchemaCodec
        .bsonCodec(Schema[Ticket], Config.withIgnoreExtraFields(false))
        .decoder
        .fromBsonValue(BsonDocument.parse(withPriority))
        .left
        .map(_.render)
    )

    println(
      Schema[strict.Comment]
        .derive(BsonCodecDeriver)
        .decoder
        .fromBsonValue(BsonDocument.parse("""{"authorName": "sam", "body": "+1", "likes": 3}"""))
        .left
        .map(_.render)
    )
  }
}
```

</details>

## Putting It Together

The complete example stores the three tickets from the insert section with `nullableTicketCodec` and queries them. It differs from the sections in one way: it also inserts, through the raw collection, a ticket that another program wrote with an extra `priority` field and a comment whose author field is named `author`. The extra field is skipped, but the misnamed comment field makes the query for open tickets fail with the path to the bad value:

```scala title="schema-examples/src/main/scala/storecaseclassesinmongodb/CompleteExample.scala" showLineNumbers
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema-bson:0.0.56
//> using dep org.mongodb:mongodb-driver-sync:5.13.0
//> using options -experimental
//> using file Model.scala
//> using file TicketCodec.scala
//> using file MongoGlue.scala
// Needs MongoDB:  docker run --rm -d -p 27017:27017 --name mongo mongo:7
// Run from this folder:  scala-cli run CompleteExample.scala
// Or from schema-examples:  sbt "runMain storecaseclassesinmongodb.CompleteExample"

package storecaseclassesinmongodb

/**
 * How to Store Case Classes in MongoDB — Putting It Together
 *
 * Stores tickets in MongoDB, queries them, and reports a document that another
 * program wrote in a shape the codec cannot read.
 */
object CompleteExample {
  def main(args: Array[String]): Unit =
    Mongo.withDatabase("helpdesk") { database =>
      val tickets = new TypedCollection(
        database.getCollection("tickets", classOf[BsonDocument]),
        TicketCodec.nullableTicketCodec
      )
      tickets.raw.drop() // start every run from an empty collection

      List(
        Ticket(
          id = new ObjectId(),
          title = "Login fails on Safari",
          status = InProgress("ana"),
          customerEmail = Some("kim@example.com"),
          comments = List(Comment("kim", "Only on Safari 17"))
        ),
        Ticket(new ObjectId(), "Invoice total is wrong", Closed("refunded")),
        Ticket(new ObjectId(), "Add a dark mode", Open, Some("lee@example.com"))
      ).foreach(tickets.insert)

      // Another program wrote this ticket: it adds "priority" and names the comment author "author"
      tickets.raw.insertOne(
        BsonDocument.parse(
          """{"title": "Export to CSV", "status": {"Open": {}}, "priority": 2, "comments": [{"author": "sam", "body": "+1"}]}"""
        )
      )

      println(s"closed: ${tickets.find(Filters.exists("status.Closed")).map(_.map(_.title))}")
      println(s"from kim: ${tickets.find(Filters.eq("customerEmail", "kim@example.com")).map(_.map(_.title))}")
      tickets.find(Filters.exists("status.Open")) match {
        case Right(open) => println(s"open: ${open.map(_.title)}")
        case Left(error) => println(s"open: cannot read a stored ticket, ${error.render}")
      }
    }
}
```

Against `mongo:7`, it prints:

```text
closed: Right(List(Invoice total is wrong))
from kim: Right(List(Login fails on Safari))
open: cannot read a stored ticket, .comments[0]: Missing required field: authorName
```

## Running the Examples

All runnable code from this guide is in [`schema-examples/src/main/scala/storecaseclassesinmongodb/`](https://github.com/zio/zio-blocks/tree/main/schema-examples/src/main/scala/storecaseclassesinmongodb). Run any file from that folder with `scala-cli run <File>.scala`, or from `schema-examples` with `sbt "runMain storecaseclassesinmongodb.<ObjectName>"`. `Step5InsertAndQueryExample` and `CompleteExample` need the MongoDB server from [Prerequisites](#prerequisites).

## Going Further

- The [BSON codec reference](../reference/schema/built-in-codecs/bson.md) covers every `BsonSchemaCodec.Config` setting, the type mappings and the `BsonEncoder` and `BsonDecoder` traits.
- [One Type, Many Formats](./one-type-many-formats.md) derives JSON, YAML, Avro and other codecs from the same kind of `Schema`, so the ticket model can also be an HTTP payload.
- [How to Encode and Decode Records with Avro](./encode-and-decode-records-with-avro.md) applies the same approach to Avro records for Kafka and files.
- [How to Customize JSON Encoding](./customize-json-encoding.md) explains the `Modifier` annotations this guide uses on `Status` and `Comment`.

---

## How to Stream Files and Sockets Without Leaking Resources

In this guide, we read a Scala bookshop's inventory export with ZIO Blocks Streams so that every file, channel and socket the stream opens is closed again, whether the stream finishes, fails, stops early or never finds its input. By the end, you will have a stream that opens a file only when it runs, a typed error for a missing file, two resources that close in reverse order, a file copy that closes both ends, and a socket read that disconnects when it is done.

We start with one file and one acquire/release pair, then break the stream on purpose in three ways, move to scope `Resource` values, compare the managed and unmanaged I/O constructors, and finish with sinks, sockets and finalizers:

- [Reading a File That Always Closes](#reading-a-file-that-always-closes)
- [Stopping Early Still Closes the File](#stopping-early-still-closes-the-file)
- [Failing Mid-Stream Still Closes the File](#failing-mid-stream-still-closes-the-file)
- [Reporting a Missing File as a Typed Error](#reporting-a-missing-file-as-a-typed-error)
- [Reusing and Composing Scope Resources](#reusing-and-composing-scope-resources)
- [Managed and Unmanaged Handles](#managed-and-unmanaged-handles)
- [Copying a File Through a Sink](#copying-a-file-through-a-sink)
- [Reading from a Socket](#reading-from-a-socket)
- [Cleanup That Is Not a Resource](#cleanup-that-is-not-a-resource)
- [Pulling Elements by Hand](#pulling-elements-by-hand)

## The Problem

A file handle must be closed exactly once, after the last row is read, and also when reading stops early or fails halfway. A lazy sequence makes that hard: the code that opens the file returns before the code that reads it runs, so there is no single place where `try`/`finally` covers both.

Without a streaming library, reading the inventory usually goes one of three ways:

```scala

// Lazy, but nobody ever closes the reader.
def inventoryRowsLeaky(path: Path): Iterator[String] = {
  val reader: BufferedReader = Files.newBufferedReader(path)
  reader.lines().iterator().asScala
}

// Closes the reader, but before the caller has read a single row.
def inventoryRowsClosedTooSoon(path: Path): Iterator[String] = {
  val reader: BufferedReader = Files.newBufferedReader(path)
  try reader.lines().iterator().asScala
  finally reader.close()
}

// Correct, but every caller repeats it, and the rows must not escape the block.
def lowStockTitles(path: Path): List[String] = {
  val reader: BufferedReader = Files.newBufferedReader(path)
  try {
    reader
      .lines()
      .iterator()
      .asScala
      .filter(row => row.split(",")(1).toInt < 5)
      .map(row => row.split(",")(0))
      .toList
  } finally reader.close()
}
```

The first version leaks one file handle per call until the process runs out of them. The second fails on the first read with `IOException: Stream closed`. The third works, but it reads everything into a `List`, and each new question about the inventory needs another copy of the `try`/`finally`. Sockets have the same problem with a connection instead of a file.

In this guide, we let the stream own the handle: it opens the handle when it starts, and closes it when it stops, for whatever reason it stops.

## Prerequisites

Add the streams module to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-streams" % "0.0.56"

// Scala 3.9 and newer, which also needs -experimental
libraryDependencies += "dev.zio" %% "zio-blocks-next-streams" % "0.0.56"
scalacOptions += "-experimental"
```

[How to Add ZIO Blocks to Your Build](../getting-started/installation.md) explains which artifact name each Scala version uses.

The streams module depends on the scope module, so `zio.blocks.scope.Resource` and `Scope` are available without another dependency. On the JVM, the streams module needs JDK 21 or newer. `NioStreams`, `NioSinks`, `Stream#start` and the blocking terminals such as `Stream#runCollect` exist only on the JVM; the other constructors on this page also work on Scala.js, where you run the stream with an asynchronous terminal such as `Stream#runCollectAsync`.

This guide assumes you have run a [`Stream`](../reference/streams/core/stream.md) before and know that every terminal returns an `Either` with typed failures on the `Left`. The beginner tutorial "Your First Streaming Pipeline" covers that ground. For `Resource` and `Scope`, see [Compile-Time Resource Safety with Scope](./compile-time-resource-safety-with-scope.md).

Most snippets below share these imports:

```scala

```

The bookshop exports its inventory as one `title,copies` row per book. We write a copy to a `bookshop` directory under the system's temporary directory so that every snippet runs on any machine:

```scala

val shop: Path = Files.createDirectories(Paths.get(System.getProperty("java.io.tmpdir"), "bookshop"))

val inventory: Path = shop.resolve("inventory.csv")

Files.writeString(
  inventory,
  """Zionomicon,12
    |Programming in Scala,7
    |Functional Programming in Scala,3
    |Scala for the Impatient,0
    |Hands-on Scala Programming,4
    |""".stripMargin
)
```

## Reading a File That Always Closes

`Stream.fromAcquireRelease` takes three parts: `acquire` opens the resource, `release` closes it, and a function receives the open resource and returns the stream that reads it. We print a line in `acquire` and in `release` so that the output shows when each one runs:

```scala
def inventoryRows(path: Path): Stream[Nothing, String] =
  Stream.fromAcquireRelease(
    acquire = {
      println(s"opened ${path.getFileName}")
      Files.newBufferedReader(path)
    },
    release = (reader: BufferedReader) => {
      reader.close()
      println(s"closed ${path.getFileName}")
    }
  )(reader => Stream.fromIterator(reader.lines().iterator().asScala))
```

`acquire` is a by-name parameter, so calling `inventoryRows` opens nothing. Each run of the stream evaluates `acquire` once, builds the inner stream from `BufferedReader#lines`, and calls `release` exactly once when that run ends.

Running it with `Stream#runForeach` prints each row as it arrives:

```scala
inventoryRows(inventory).runForeach(row => println(row))
// opened inventory.csv
// Zionomicon,12
// Programming in Scala,7
// Functional Programming in Scala,3
// Scala for the Impatient,0
// Hands-on Scala Programming,4
// closed inventory.csv
// res2: Either[Nothing, Unit] = Right(())
```

The file opens before the first row and closes after the last one. When the resource is an `AutoCloseable`, as a `BufferedReader` is, you can leave `release` out: its default calls `AutoCloseable#close`. We keep the explicit `release` on this page only for the printed line.

## Stopping Early Still Closes the File

`Stream#take` stops pulling after a given number of elements. The rest of the file is never read, and the stream still closes it:

```scala
inventoryRows(inventory).take(2).runForeach(row => println(row))
// opened inventory.csv
// Zionomicon,12
// Programming in Scala,7
// closed inventory.csv
// res3: Either[Nothing, Unit] = Right(())
```

`closed inventory.csv` appears after two rows, while three rows are still unread. The same holds for `Stream#takeWhile`, for `Stream#head`, and for any sink that finishes before the input ends.

## Failing Mid-Stream Still Closes the File

A second export has a damaged row, where the copy count is a word instead of a number:

```scala
val damaged: Path = shop.resolve("inventory-damaged.csv")

Files.writeString(
  damaged,
  """Zionomicon,12
    |Programming in Scala,seven
    |Functional Programming in Scala,3
    |""".stripMargin
)

final case class Stock(title: String, copies: Int)
```

Parsing with `String#toInt` throws on the damaged row. A thrown exception is a defect: the terminal rethrows it instead of returning a `Left`, and the stream closes the file on the way out:

```scala
def parseOrThrow(row: String): Stock = {
  val fields = row.split(",")
  Stock(fields(0), fields(1).toInt)
}

try println(inventoryRows(damaged).map(parseOrThrow).runCollect)
catch { case e: NumberFormatException => println(s"thrown: $e") }
// opened inventory-damaged.csv
// closed inventory-damaged.csv
// thrown: java.lang.NumberFormatException: For input string: "seven"
```

To make a bad row an expected outcome instead of a crash, we fail the stream with a typed error. `Stream.fail` emits no elements and fails with its argument, and `Stream#flatMap` lets each row choose between a one-element stream and a failed one:

```scala
final case class BadRow(row: String)

def parse(row: String): Stream[BadRow, Stock] = {
  val fields = row.split(",")
  fields(1).toIntOption match {
    case Some(copies) => Stream.succeed(Stock(fields(0), copies))
    case None         => Stream.fail(BadRow(row))
  }
}

println(inventoryRows(damaged).flatMap(parse).runCollect)
// opened inventory-damaged.csv
// closed inventory-damaged.csv
// Left(BadRow(Programming in Scala,seven))
```

The run ends with `Left(BadRow(...))`, and the file is closed before the terminal returns, exactly as on success.

:::warning[Recover on the outer stream]
If you add recovery with `Stream#catchAll`, attach it to the outer stream, as in `inventoryRows(damaged).flatMap(parse).catchAll(...)`, not to the stream that `parse` returns. In 0.0.56, a blocking terminal such as `Stream#runCollect` throws the internal `Stream$AsyncBoundaryRequired$` when `Stream#catchAll` sits inside `Stream#flatMap` and a stateful operator such as `Stream#grouped` follows it.
:::

## Reporting a Missing File as a Typed Error

`acquire` can fail too. When the file does not exist, `Files.newBufferedReader` throws `NoSuchFileException`, and like any exception thrown by `acquire`, it is a defect that the terminal rethrows:

```scala
val missing: Path = shop.resolve("inventory-2025.csv")
// missing: Path = /tmp/bookshop/inventory-2025.csv

try println(inventoryRows(missing).runCollect)
catch { case e: java.nio.file.NoSuchFileException => println(s"thrown: $e") }
// opened inventory-2025.csv
// thrown: java.nio.file.NoSuchFileException: /tmp/bookshop/inventory-2025.csv
```

`opened inventory-2025.csv` is printed because our `println` runs before `Files.newBufferedReader` throws. No `closed` line follows: nothing was acquired, so there is nothing to release.

A missing input file is often a normal condition, such as an export that has not arrived yet. `Stream.attempt` runs a block and turns a non-fatal exception into a typed `Throwable` error. Opening the file with `Stream.attempt` and handing the open reader to `Stream.fromAcquireRelease` makes the failure a `Left` and keeps the close guarantee once the file is open:

```scala
def inventoryRowsOrError(path: Path): Stream[Throwable, String] =
  Stream.attempt(Files.newBufferedReader(path)).flatMap { reader =>
    Stream.fromAcquireRelease(reader)(r => Stream.fromIterator(r.lines().iterator().asScala))
  }

println(inventoryRowsOrError(missing).runCollect)
// Left(java.nio.file.NoSuchFileException: /tmp/bookshop/inventory-2025.csv)
println(inventoryRowsOrError(inventory).take(1).runCollect)
// Right(Chunk(Zionomicon,12))
```

The missing file comes back as `Left(java.nio.file.NoSuchFileException: ...)`, and the existing file reads as before. Here `release` is left out, so the default closes the reader.

## Reusing and Composing Scope Resources

A [`Resource`](../reference/resource-management/resource.md) from the scope module describes how to acquire and release a value, independently of any stream. `Stream.fromResource` runs one in a fresh `Scope` for each run of the stream, passes the acquired value to the function, and closes the scope when the run ends. `Resource.acquireRelease` builds a `Resource` from an acquire block and a release function:

```scala

def openReader(path: Path): Resource[BufferedReader] =
  Resource.acquireRelease {
    println(s"opened ${path.getFileName}")
    Files.newBufferedReader(path)
  } { reader =>
    reader.close()
    println(s"closed ${path.getFileName}")
  }

def openWriter(path: Path): Resource[BufferedWriter] =
  Resource.acquireRelease {
    println(s"opened ${path.getFileName}")
    Files.newBufferedWriter(path)
  } { writer =>
    writer.close()
    println(s"closed ${path.getFileName}")
  }
```

Reading the inventory through `openReader` behaves like `inventoryRows`:

```scala
println(
  Stream
    .fromResource(openReader(inventory))(reader => Stream.fromIterator(reader.lines().iterator().asScala))
    .take(1)
    .runCollect
)
// opened inventory.csv
// closed inventory.csv
// Right(Chunk(Zionomicon,12))
```

The difference shows when one stream needs several resources. `Resource#flatMap` acquires the second resource after the first, so a `for` comprehension describes both. Here the stream reads the inventory and writes every title with fewer than five copies to a restock list:

```scala
val restock: Path = shop.resolve("restock.csv")

val inventoryAndRestock: Resource[(BufferedReader, BufferedWriter)] =
  for {
    reader <- openReader(inventory)
    writer <- openWriter(restock)
  } yield (reader, writer)
```

Running the stream with `Stream#runDrain` writes the list, and the output shows the order in which the two files open and close:

```scala
println(
  Stream
    .fromResource(inventoryAndRestock) { case (reader, writer) =>
      Stream
        .fromIterator(reader.lines().iterator().asScala)
        .filter(row => row.split(",")(1).toInt < 5)
        .tapEach { row =>
          writer.write(row)
          writer.newLine()
        }
    }
    .runDrain
)
// opened inventory.csv
// opened restock.csv
// closed restock.csv
// closed inventory.csv
// Right(())

print(Files.readString(restock))
// Functional Programming in Scala,3
// Scala for the Impatient,0
// Hands-on Scala Programming,4
```

The writer opens last and closes first: the scope releases resources in the reverse order of acquisition, so the restock file is flushed and closed before the inventory it was read from. `Resource#zip` combines two resources with the same order.

The two constructors differ in who describes the resource:

| Situation                                                                                  | Use                         |
| ------------------------------------------------------------------------------------------ | --------------------------- |
| One resource, opened and closed right where the stream is built                            | `Stream.fromAcquireRelease` |
| A `Resource` value that already exists, for example one shared with code that uses `Scope` | `Stream.fromResource`       |
| Several resources that must open in order and close in reverse order                       | `Stream.fromResource`       |

## Managed and Unmanaged Handles

The streams module also builds streams from handles you have already opened: `Stream.fromInputStream` and `NioStreams.fromChannel` are **managed**, so the stream closes the handle when the run ends. `Stream.fromInputStreamUnmanaged` and `NioStreams.fromChannelUnmanaged` read the same way but never close it; the caller owns the handle. All four emit `Byte` elements and report read failures as a typed `IOException`.

Unlike `acquire` in `Stream.fromAcquireRelease`, the handle is an ordinary argument, so it is open from the moment the stream is built. A managed stream closes it after the first run, and a second run reads a closed handle. The helper `text` turns the collected bytes back into a `String`:

```scala
def text(bytes: Chunk[Byte]): String = new String(bytes.toArray, UTF_8)

val managed: Stream[java.io.IOException, Byte] =
  Stream.fromInputStream(Files.newInputStream(inventory))
// managed: Stream[IOException, Byte] = Stream.fromAcquireRelease(...)

println(managed.take(10).runCollect.map(text))
// Right(Zionomicon)
println(managed.take(10).runCollect.map(text))
// Left(java.nio.channels.ClosedChannelException)
```

The second run fails with `ClosedChannelException`, because the first run closes the handle from `Files.newInputStream`. A managed stream built from a handle is good for one run, and if it never runs, the handle is never closed. To get a stream that opens a fresh handle on every run, open the handle in `acquire` and read it with the unmanaged constructor, so that only `Stream.fromAcquireRelease` closes it:

```scala
val rerunnable: Stream[java.io.IOException, Byte] =
  Stream.fromAcquireRelease(Files.newInputStream(inventory))(in => Stream.fromInputStreamUnmanaged(in))
// rerunnable: Stream[IOException, Byte] = Stream.fromAcquireRelease(...)

println(rerunnable.take(10).runCollect.map(text))
// Right(Zionomicon)
println(rerunnable.take(10).runCollect.map(text))
// Right(Zionomicon)
```

The unmanaged constructors are right when the stream borrows a handle that something else owns: `System.in`, a socket that the program keeps using after the stream ends, or a channel managed by a surrounding `Using` or `Scope`. Here the caller opens a `FileChannel` and keeps it:

```scala

val channel: FileChannel = FileChannel.open(inventory)
```

Reading ten bytes through the unmanaged constructor leaves the channel open:

```scala
println(NioStreams.fromChannelUnmanaged(channel).take(10).runCollect.map(text))
// Right(Zionomicon)
println(channel.isOpen)
// true

channel.close()
```

`NioStreams.fromChannel` is the managed counterpart: given the same channel, it closes the channel at the end of the run.

| Constructor                                  | Who closes the handle      | Running the stream twice             |
| -------------------------------------------- | -------------------------- | ------------------------------------ |
| `Stream.fromInputStream(is)`                 | The stream, after each run | The second run reads a closed handle |
| `Stream.fromInputStreamUnmanaged(is)`        | The caller                 | Both runs read the same open handle  |
| `NioStreams.fromChannel(ch)`                 | The stream, after each run | The second run reads a closed handle |
| `NioStreams.fromChannelUnmanaged(ch)`        | The caller                 | Both runs read the same open handle  |
| `Stream.fromAcquireRelease(open)(unmanaged)` | The stream, after each run | Each run opens a new handle          |

`Stream.fromJavaReader` and `Stream.fromJavaReaderUnmanaged` follow the same rule for a `java.io.Reader` of characters.

## Copying a File Through a Sink

On the writing side, `NioSinks.fromChannel` writes every byte of a stream to a `WritableByteChannel` and does not close it; `Sink.fromOutputStream` does the same for an `OutputStream`. The stream owns the input, so the code that runs the stream owns the output. Here `scala.util.Using.resource` closes the backup file even if the copy throws:

```scala

val backup: Path = shop.resolve("inventory-backup.csv")
// backup: Path = /tmp/bookshop/inventory-backup.csv

val copied: Either[java.io.IOException, Unit] =
  Using.resource(FileChannel.open(backup, CREATE, TRUNCATE_EXISTING, WRITE)) { out =>
    NioStreams.fromChannel(FileChannel.open(inventory)).run(NioSinks.fromChannel(out))
  }
// copied: Either[IOException, Unit] = Right(())

println(Files.readString(backup) == Files.readString(inventory))
// true
```

`Stream#run` returns the sink's result in a `Right`, here `Right(())`, or the first `IOException` in a `Left`. The input channel is opened inside the `Using` block and run at once, so the managed `NioStreams.fromChannel` closes it after this single run.

## Reading from a Socket

A `SocketChannel` is a `ReadableByteChannel`, so the NIO constructors read sockets the same way they read files. To keep the example self-contained, a small server on the loopback address sends the inventory to every client and hangs up:

```scala

val server = new ServerSocket(0, 50, InetAddress.getLoopbackAddress)
val serverThread = new Thread(() =>
  for (_ <- 1 to 2) {
    val client = server.accept()
    try client.getOutputStream.write(Files.readAllBytes(inventory))
    catch { case _: java.io.IOException => () }
    finally client.close()
  }
)
serverThread.start()

val address = new InetSocketAddress(InetAddress.getLoopbackAddress, server.getLocalPort)
```

`NioStreams.fromChannel(SocketChannel.open(address))` would connect as soon as the stream is built. Connecting in `acquire` instead connects on each run and disconnects when the run ends:

```scala
val inventoryFeed: Stream[java.io.IOException, Byte] =
  Stream.fromAcquireRelease(
    acquire = {
      val socket = SocketChannel.open(address)
      println("connected")
      socket
    },
    release = (socket: SocketChannel) => {
      socket.close()
      println("disconnected")
    }
  )(socket => NioStreams.fromChannelUnmanaged(socket))
```

The first run reads until the server hangs up, and the second stops after ten bytes:

```scala
print(inventoryFeed.runCollect.map(text).getOrElse(""))
// connected
// disconnected
// Zionomicon,12
// Programming in Scala,7
// Functional Programming in Scala,3
// Scala for the Impatient,0
// Hands-on Scala Programming,4
println(inventoryFeed.take(10).runCollect.map(text))
// connected
// disconnected
// Right(Zionomicon)
```

Both runs disconnect, including the one that stops long before the server is done. We stop the server once both clients are served:

```scala
serverThread.join()
server.close()
```

## Cleanup That Is Not a Resource

Some cleanup has no handle to close: deleting a temporary file, releasing a lock, or logging that a job is done. `Stream#ensuring` runs a block exactly once when a run of the stream ends, after completion, failure or early termination. Here the stream reads the backup made earlier and deletes it when it is done:

```scala
val fromBackup: Stream[Nothing, String] =
  inventoryRows(backup).ensuring {
    Files.deleteIfExists(backup)
    println("deleted inventory-backup.csv")
  }
// fromBackup: Stream[Nothing, String] = Stream.fromAcquireRelease(...).ensuring(...)

println(fromBackup.take(1).runCollect)
// opened inventory-backup.csv
// closed inventory-backup.csv
// deleted inventory-backup.csv
// Right(Chunk(Zionomicon,12))
println(Files.exists(backup))
// false
```

The reader closes first, because it belongs to the inner stream, and the finalizer runs after it. An exception thrown by the finalizer is a defect. `Stream#ensuringAsync` takes an `Async[Unit]` finalizer for asynchronous runs.

## Pulling Elements by Hand

Sometimes code needs to read a stream piece by piece instead of handing it to a terminal, for example to read a header first and decide what to do with the rest. On the JVM, `Stream#start` opens the stream as a `Reader` that belongs to a `Scope`, and closing the scope closes the reader and every resource behind it:

```scala

println(
  Scope.global.scoped { implicit scope =>

    val reader = inventoryRows(inventory).start
    $(reader)(r => r.readN[String](2).mkString(" | "))
  }
)
// opened inventory.csv
// closed inventory.csv
// Zionomicon,12 | Programming in Scala,7
```

`closed inventory.csv` is printed when the `scoped` block ends, not when `Reader#readN` returns. `Scope#$` is how a scope lets you use a value it owns; the [Scope tutorial](./compile-time-resource-safety-with-scope.md) explains why the reader cannot leave the block.

## Putting It Together

The complete example is a nightly restock job. It downloads the inventory export from a loopback server into a staging file, reads the staging file and writes a restock list through two composed `Resource` values, fails with a typed error on a damaged row, and deletes the staging file with `Stream#ensuring`:

```scala title="streams-examples/src/main/scala/streamfilesandsockets/CompleteExample.scala"
package streamfilesandsockets

/**
 * How to Stream Files and Sockets Without Leaking Resources
 *
 * Nightly restock job: downloads the inventory export from a loopback server
 * into a staging file, reads it and writes a restock list through two composed
 * `Resource` values, fails with a typed error on a damaged row, and deletes the
 * staging file with `Stream#ensuring`.
 *
 * Run with: sbt "runMain streamfilesandsockets.CompleteExample"
 */
object CompleteExample {

  final case class Stock(title: String, copies: Int)

  final case class BadRow(row: String)

  sealed trait RestockError
  final case class DownloadFailed(cause: IOException) extends RestockError
  final case class DamagedExport(row: BadRow)         extends RestockError

  val cleanExport: String =
    """Zionomicon,12
      |Programming in Scala,7
      |Functional Programming in Scala,3
      |Scala for the Impatient,0
      |Hands-on Scala Programming,4
      |""".stripMargin

  val damagedExport: String =
    """Zionomicon,12
      |Programming in Scala,seven
      |Functional Programming in Scala,3
      |""".stripMargin

  def parse(row: String): Stream[BadRow, Stock] = {
    val fields = row.split(",")
    fields(1).toIntOption match {
      case Some(copies) => Stream.succeed(Stock(fields(0), copies))
      case None         => Stream.fail(BadRow(row))
    }
  }

  def openReader(path: Path): Resource[BufferedReader] =
    Resource.acquireRelease {
      println(s"opened ${path.getFileName}")
      Files.newBufferedReader(path)
    } { reader =>
      reader.close()
      println(s"closed ${path.getFileName}")
    }

  def openWriter(path: Path): Resource[BufferedWriter] =
    Resource.acquireRelease {
      println(s"opened ${path.getFileName}")
      Files.newBufferedWriter(path)
    } { writer =>
      writer.close()
      println(s"closed ${path.getFileName}")
    }

  def connect(address: InetSocketAddress): Stream[IOException, Byte] =
    Stream.fromAcquireRelease(
      acquire = {
        val socket = SocketChannel.open(address)
        println("connected")
        socket
      },
      release = (socket: SocketChannel) => {
        socket.close()
        println("disconnected")
      }
    )(socket => NioStreams.fromChannelUnmanaged(socket))

  def download(address: InetSocketAddress, staging: Path): Either[RestockError, Unit] =
    Using
      .resource(FileChannel.open(staging, CREATE, TRUNCATE_EXISTING, WRITE)) { out =>
        connect(address).run(NioSinks.fromChannel(out))
      }
      .left
      .map(DownloadFailed(_))

  def restock(staging: Path, restockList: Path): Either[RestockError, Unit] = {
    val staged: Resource[(BufferedReader, BufferedWriter)] =
      for {
        reader <- openReader(staging)
        writer <- openWriter(restockList)
      } yield (reader, writer)

    Stream
      .fromResource(staged) { case (reader, writer) =>
        Stream
          .fromIterator(reader.lines().iterator().asScala)
          .flatMap(parse)
          .filter(_.copies < 5)
          .tapEach { stock =>
            writer.write(stock.title)
            writer.newLine()
          }
      }
      .ensuring {
        Files.deleteIfExists(staging)
        println(s"deleted ${staging.getFileName}")
      }
      .mapError(DamagedExport(_))
      .runDrain
  }

  def restockJob(address: InetSocketAddress, dir: Path): Either[RestockError, Unit] = {
    val staging     = dir.resolve("inventory-staging.csv")
    val restockList = dir.resolve("restock.csv")

    val result = download(address, staging).flatMap(_ => restock(staging, restockList))

    println(result)
    print(Files.readString(restockList))
    println(s"staging exists: ${Files.exists(staging)}")
    result
  }

  def main(args: Array[String]): Unit = {
    val dir = Files.createTempDirectory("restock")

    val server       = new ServerSocket(0, 50, InetAddress.getLoopbackAddress)
    val serverThread = new Thread(() =>
      for (payload <- List(cleanExport, damagedExport)) {
        val client = server.accept()
        try client.getOutputStream.write(payload.getBytes(UTF_8))
        catch { case _: IOException => () }
        finally client.close()
      }
    )
    serverThread.start()

    val address = new InetSocketAddress(InetAddress.getLoopbackAddress, server.getLocalPort)

    println("--- clean export ---")
    restockJob(address, dir)

    println("--- damaged export ---")
    restockJob(address, dir)

    serverThread.join()
    server.close()
  }
}
```

## Running the Examples

All examples in this guide have runnable Scala files in the `streams-examples` project. Clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/streams-examples
```

The examples share the sample inventory files, which live in one file:

<details>
  <summary>streams-examples/src/main/scala/streamfilesandsockets/SampleFiles.scala</summary>

```scala title="streams-examples/src/main/scala/streamfilesandsockets/SampleFiles.scala" showLineNumbers
package streamfilesandsockets

final case class Stock(title: String, copies: Int)

final case class BadRow(row: String)

/**
 * Shared sample data and helpers for the "How to Stream Files and Sockets
 * Without Leaking Resources" guide.
 */
object SampleFiles {

  /** Creates a temporary directory for one example run. */
  def newShop(): Path = Files.createTempDirectory("bookshop")

  /** Writes the clean inventory export into `shop` and returns its path. */
  def writeInventory(shop: Path): Path = {
    val inventory: Path = shop.resolve("inventory.csv")
    Files.writeString(
      inventory,
      """Zionomicon,12
        |Programming in Scala,7
        |Functional Programming in Scala,3
        |Scala for the Impatient,0
        |Hands-on Scala Programming,4
        |""".stripMargin
    )
    inventory
  }

  /** Writes the damaged inventory export into `shop` and returns its path. */
  def writeDamaged(shop: Path): Path = {
    val damaged: Path = shop.resolve("inventory-damaged.csv")
    Files.writeString(
      damaged,
      """Zionomicon,12
        |Programming in Scala,seven
        |Functional Programming in Scala,3
        |""".stripMargin
    )
    damaged
  }

  def inventoryRows(path: Path): Stream[Nothing, String] =
    Stream.fromAcquireRelease(
      acquire = {
        println(s"opened ${path.getFileName}")
        Files.newBufferedReader(path)
      },
      release = (reader: BufferedReader) => {
        reader.close()
        println(s"closed ${path.getFileName}")
      }
    )(reader => Stream.fromIterator(reader.lines().iterator().asScala))

  def parse(row: String): Stream[BadRow, Stock] = {
    val fields = row.split(",")
    fields(1).toIntOption match {
      case Some(copies) => Stream.succeed(Stock(fields(0), copies))
      case None         => Stream.fail(BadRow(row))
    }
  }

  def text(bytes: Chunk[Byte]): String = new String(bytes.toArray, UTF_8)
}
```

</details>

Each example is an object with a `def main` method rather than `extends App`: on the JVM, a blocking terminal that runs inside an object initializer can deadlock once `Stream#catchAll` is in the pipeline.

### Read a File and Stop Early

This example reads the inventory with `Stream.fromAcquireRelease`, then reads only the first two rows, and prints when the file opens and closes.

<details>
  <summary>streams-examples/src/main/scala/streamfilesandsockets/Step1ReadInventory.scala</summary>

```scala title="streams-examples/src/main/scala/streamfilesandsockets/Step1ReadInventory.scala" showLineNumbers
package streamfilesandsockets

/**
 * How to Stream Files and Sockets Without Leaking Resources
 *
 * Read a File and Stop Early: `Stream.fromAcquireRelease` closes the file after
 * the last row and after an early `take`.
 *
 * Run with: sbt "runMain streamfilesandsockets.Step1ReadInventory"
 */
object Step1ReadInventory {
  def main(args: Array[String]): Unit = {
    val inventory: Path = writeInventory(newShop())

    println(inventoryRows(inventory).runForeach(row => println(row)))

    println(inventoryRows(inventory).take(2).runForeach(row => println(row)))
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain streamfilesandsockets.Step1ReadInventory"
```

### Fail Mid-Stream

This example parses the damaged export twice, once with a thrown exception and once with a typed `BadRow` error, and shows the file closed both times.

<details>
  <summary>streams-examples/src/main/scala/streamfilesandsockets/Step2FailMidStream.scala</summary>

```scala title="streams-examples/src/main/scala/streamfilesandsockets/Step2FailMidStream.scala" showLineNumbers
package streamfilesandsockets

/**
 * How to Stream Files and Sockets Without Leaking Resources
 *
 * Fail Mid-Stream: a thrown exception is a defect, a typed `BadRow` is a
 * `Left`; the file is closed both times.
 *
 * Run with: sbt "runMain streamfilesandsockets.Step2FailMidStream"
 */
object Step2FailMidStream {
  def parseOrThrow(row: String): Stock = {
    val fields = row.split(",")
    Stock(fields(0), fields(1).toInt)
  }

  def main(args: Array[String]): Unit = {
    val damaged: Path = writeDamaged(newShop())

    try println(inventoryRows(damaged).map(parseOrThrow).runCollect)
    catch { case e: NumberFormatException => println(s"thrown: $e") }

    println(inventoryRows(damaged).flatMap(parse).runCollect)
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain streamfilesandsockets.Step2FailMidStream"
```

### Report a Missing File

This example reads a file that does not exist, first as a thrown defect and then as a typed error with `Stream.attempt`.

<details>
  <summary>streams-examples/src/main/scala/streamfilesandsockets/Step3MissingFile.scala</summary>

```scala title="streams-examples/src/main/scala/streamfilesandsockets/Step3MissingFile.scala" showLineNumbers
package streamfilesandsockets

/**
 * How to Stream Files and Sockets Without Leaking Resources
 *
 * Report a Missing File: a defect thrown by `acquire`, then a typed error with
 * `Stream.attempt`.
 *
 * Run with: sbt "runMain streamfilesandsockets.Step3MissingFile"
 */
object Step3MissingFile {
  def inventoryRowsOrError(path: Path): Stream[Throwable, String] =
    Stream.attempt(Files.newBufferedReader(path)).flatMap { reader =>
      Stream.fromAcquireRelease(reader)(r => Stream.fromIterator(r.lines().iterator().asScala))
    }

  def main(args: Array[String]): Unit = {
    val shop: Path      = newShop()
    val inventory: Path = writeInventory(shop)

    val missing: Path = shop.resolve("inventory-2025.csv")

    try println(inventoryRows(missing).runCollect)
    catch { case e: java.nio.file.NoSuchFileException => println(s"thrown: $e") }

    println(inventoryRowsOrError(missing).runCollect)
    println(inventoryRowsOrError(inventory).take(1).runCollect)
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain streamfilesandsockets.Step3MissingFile"
```

### Compose Scope Resources

This example reads the inventory through a `Resource`, then writes the restock list with a reader and a writer composed in a `for` comprehension, and prints the reverse closing order.

<details>
  <summary>streams-examples/src/main/scala/streamfilesandsockets/Step4ComposeResources.scala</summary>

```scala title="streams-examples/src/main/scala/streamfilesandsockets/Step4ComposeResources.scala" showLineNumbers
package streamfilesandsockets

/**
 * How to Stream Files and Sockets Without Leaking Resources
 *
 * Compose Scope Resources: `Stream.fromResource` with one resource, then with a
 * reader and a writer that close in reverse order.
 *
 * Run with: sbt "runMain streamfilesandsockets.Step4ComposeResources"
 */
object Step4ComposeResources {
  def openReader(path: Path): Resource[BufferedReader] =
    Resource.acquireRelease {
      println(s"opened ${path.getFileName}")
      Files.newBufferedReader(path)
    } { reader =>
      reader.close()
      println(s"closed ${path.getFileName}")
    }

  def openWriter(path: Path): Resource[BufferedWriter] =
    Resource.acquireRelease {
      println(s"opened ${path.getFileName}")
      Files.newBufferedWriter(path)
    } { writer =>
      writer.close()
      println(s"closed ${path.getFileName}")
    }

  def main(args: Array[String]): Unit = {
    val shop: Path      = newShop()
    val inventory: Path = writeInventory(shop)

    println(
      Stream
        .fromResource(openReader(inventory))(reader => Stream.fromIterator(reader.lines().iterator().asScala))
        .take(1)
        .runCollect
    )

    val restock: Path = shop.resolve("restock.csv")

    val inventoryAndRestock: Resource[(BufferedReader, BufferedWriter)] =
      for {
        reader <- openReader(inventory)
        writer <- openWriter(restock)
      } yield (reader, writer)

    println(
      Stream
        .fromResource(inventoryAndRestock) { case (reader, writer) =>
          Stream
            .fromIterator(reader.lines().iterator().asScala)
            .filter(row => row.split(",")(1).toInt < 5)
            .tapEach { row =>
              writer.write(row)
              writer.newLine()
            }
        }
        .runDrain
    )

    print(Files.readString(restock))
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain streamfilesandsockets.Step4ComposeResources"
```

### Managed and Unmanaged Handles

This example runs a managed `Stream.fromInputStream` twice, a stream that opens a new handle on each run, and an unmanaged `NioStreams.fromChannelUnmanaged` that leaves its channel open.

<details>
  <summary>streams-examples/src/main/scala/streamfilesandsockets/Step5ManagedAndUnmanaged.scala</summary>

```scala title="streams-examples/src/main/scala/streamfilesandsockets/Step5ManagedAndUnmanaged.scala" showLineNumbers
package streamfilesandsockets

/**
 * How to Stream Files and Sockets Without Leaking Resources
 *
 * Managed and Unmanaged Handles: which constructor closes the handle, and what
 * that means for running a stream twice.
 *
 * Run with: sbt "runMain streamfilesandsockets.Step5ManagedAndUnmanaged"
 */
object Step5ManagedAndUnmanaged {
  def main(args: Array[String]): Unit = {
    val inventory: Path = writeInventory(newShop())

    val managed: Stream[java.io.IOException, Byte] =
      Stream.fromInputStream(Files.newInputStream(inventory))

    println(managed.take(10).runCollect.map(text))
    println(managed.take(10).runCollect.map(text))

    val rerunnable: Stream[java.io.IOException, Byte] =
      Stream.fromAcquireRelease(Files.newInputStream(inventory))(in => Stream.fromInputStreamUnmanaged(in))

    println(rerunnable.take(10).runCollect.map(text))
    println(rerunnable.take(10).runCollect.map(text))

    val channel: FileChannel = FileChannel.open(inventory)

    println(NioStreams.fromChannelUnmanaged(channel).take(10).runCollect.map(text))
    println(channel.isOpen)

    channel.close()

    val managedChannel: FileChannel = FileChannel.open(inventory)

    println(NioStreams.fromChannel(managedChannel).take(10).runCollect.map(text))
    println(managedChannel.isOpen)
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain streamfilesandsockets.Step5ManagedAndUnmanaged"
```

### Copy a File

This example copies the inventory to a backup file with `NioSinks.fromChannel` and closes the backup with `Using.resource`.

<details>
  <summary>streams-examples/src/main/scala/streamfilesandsockets/Step6CopyFile.scala</summary>

```scala title="streams-examples/src/main/scala/streamfilesandsockets/Step6CopyFile.scala" showLineNumbers
package streamfilesandsockets

/**
 * How to Stream Files and Sockets Without Leaking Resources
 *
 * Copy a File Through a Sink: `NioSinks.fromChannel` writes the bytes and
 * `Using.resource` closes the backup file.
 *
 * Run with: sbt "runMain streamfilesandsockets.Step6CopyFile"
 */
object Step6CopyFile {
  def main(args: Array[String]): Unit = {
    val shop: Path      = newShop()
    val inventory: Path = writeInventory(shop)

    val backup: Path = shop.resolve("inventory-backup.csv")

    val copied: Either[java.io.IOException, Unit] =
      Using.resource(FileChannel.open(backup, CREATE, TRUNCATE_EXISTING, WRITE)) { out =>
        NioStreams.fromChannel(FileChannel.open(inventory)).run(NioSinks.fromChannel(out))
      }

    println(copied)
    println(Files.readString(backup) == Files.readString(inventory))
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain streamfilesandsockets.Step6CopyFile"
```

### Read from a Socket

This example starts a server on the loopback address, reads the inventory from it twice over a `SocketChannel`, and prints when each connection opens and closes.

<details>
  <summary>streams-examples/src/main/scala/streamfilesandsockets/Step7ReadSocket.scala</summary>

```scala title="streams-examples/src/main/scala/streamfilesandsockets/Step7ReadSocket.scala" showLineNumbers
package streamfilesandsockets

/**
 * How to Stream Files and Sockets Without Leaking Resources
 *
 * Read from a Socket: connect in `acquire`, disconnect in `release`, on every
 * run.
 *
 * Run with: sbt "runMain streamfilesandsockets.Step7ReadSocket"
 */
object Step7ReadSocket {
  def main(args: Array[String]): Unit = {
    val inventory: Path = writeInventory(newShop())

    val server       = new ServerSocket(0, 50, InetAddress.getLoopbackAddress)
    val serverThread = new Thread(() =>
      for (_ <- 1 to 2) {
        val client = server.accept()
        try client.getOutputStream.write(Files.readAllBytes(inventory))
        catch { case _: java.io.IOException => () }
        finally client.close()
      }
    )
    serverThread.start()

    val address = new InetSocketAddress(InetAddress.getLoopbackAddress, server.getLocalPort)

    val inventoryFeed: Stream[java.io.IOException, Byte] =
      Stream.fromAcquireRelease(
        acquire = {
          val socket = SocketChannel.open(address)
          println("connected")
          socket
        },
        release = (socket: SocketChannel) => {
          socket.close()
          println("disconnected")
        }
      )(socket => NioStreams.fromChannelUnmanaged(socket))

    print(inventoryFeed.runCollect.map(text).getOrElse(""))
    println(inventoryFeed.take(10).runCollect.map(text))

    serverThread.join()
    server.close()
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain streamfilesandsockets.Step7ReadSocket"
```

### Finalize and Pull by Hand

This example deletes a staging file with `Stream#ensuring`, then reads two rows by hand with `Stream#start` inside a `Scope`.

<details>
  <summary>streams-examples/src/main/scala/streamfilesandsockets/Step8EnsuringAndStart.scala</summary>

```scala title="streams-examples/src/main/scala/streamfilesandsockets/Step8EnsuringAndStart.scala" showLineNumbers
package streamfilesandsockets

/**
 * How to Stream Files and Sockets Without Leaking Resources
 *
 * Cleanup That Is Not a Resource, and Pulling Elements by Hand:
 * `Stream#ensuring` deletes a staging file, `Stream#start` reads two rows
 * inside a `Scope`.
 *
 * Run with: sbt "runMain streamfilesandsockets.Step8EnsuringAndStart"
 */
object Step8EnsuringAndStart {
  def main(args: Array[String]): Unit = {
    val shop: Path      = newShop()
    val inventory: Path = writeInventory(shop)

    val staging: Path = shop.resolve("inventory-staging.csv")
    Files.copy(inventory, staging)

    val fromStaging: Stream[Nothing, String] =
      inventoryRows(staging).ensuring {
        Files.deleteIfExists(staging)
        println("deleted inventory-staging.csv")
      }

    println(fromStaging.take(1).runCollect)
    println(Files.exists(staging))

    val firstTwo: String =
      Scope.global.scoped { implicit scope =>

        val reader = inventoryRows(inventory).start
        $(reader)(r => r.readN[String](2).mkString(" | "))
      }

    println(firstTwo)
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain streamfilesandsockets.Step8EnsuringAndStart"
```

### The Complete Restock Job

This example runs the nightly restock job from [Putting It Together](#putting-it-together).

```bash
sbt "runMain streamfilesandsockets.CompleteExample"
```

## Going Further

The reference pages cover each constructor in full:

- [`Stream` resource management](../reference/streams/core/stream.md#resource-management) lists `Stream.fromAcquireRelease`, `Stream.fromResource`, `Stream#ensuring` and `Stream#ensuringAsync` with their signatures.
- [`Sink`](../reference/streams/core/sink.md) covers `NioSinks.fromChannel`, `NioSinks.fromByteBuffer` and `Sink.fromOutputStream`.
- [Async Execution](../reference/streams/execution-and-compatibility/async-execution.md) covers `Stream.fromAcquireReleaseAsync`, whose `acquire` and `release` return `Async` values. The asynchronous NIO readers in `AsyncNioReaders` are covered in [Reader](../reference/streams/primitives/reader.md).
- [`Resource`](../reference/resource-management/resource.md) covers `Resource.fromAutoCloseable`, `Resource#zip` and shared resources.
- The streams [overview](../reference/streams/index.md#resource-management) explains how streams integrate with `Scope`.

Streaming rows from a SQL query uses the same guarantee for a database cursor; [How to Stream Large Query Results](./stream-large-query-results.md) covers it.

For a file of one JSON value per line, see [How to Read and Write JSON Lines with Streams](./read-and-write-json-lines.md).

---

## How to Stream Large Query Results

In this guide, we read a large `sales` table as a stream of row chunks with `Frag#queryStream` and `Frag#queryChunked`. By the end, you will have a report of copies sold per title and a CSV export of every sale, both computed while only a small batch of rows is held in memory at a time.

We start from a table of 12,000 sales and add one capability at a time:

- Aggregating the whole table with `runFold`, one chunk at a time
- Choosing the chunk size
- Exporting every row to a CSV file through a `Sink`
- Filtering and reshaping rows on their way to the file
- Keeping the stream inside the block that owns the connection
- Handling a bad query or a row that can't be decoded

This guide builds on [Build Your First Database Repository on SQLite](./your-first-sqlite-repository.md). We use `JdbcTransactor`, `Table`, `Repo` and `sql"..."` the way that tutorial introduces them and don't explain them again. The streaming operators we use (`map`, `filter`, `flatMap`, `grouped`, `runFold`) are the ones the tutorial "Build Your First Streaming Pipeline" teaches.

## The Problem

`Frag#query` returns a `List`: it reads every row of the result into memory before it gives you the first one. That is fine for a page of results, but a report over a whole table holds the entire table on the heap, and an export of a table that doesn't fit in memory fails with an `OutOfMemoryError` halfway through.

Without a streaming API, the usual way out is a hand-written JDBC cursor loop:

```scala

val connection = DriverManager.getConnection("jdbc:sqlite:bookstore.db")
try {
  val statement = connection.prepareStatement("SELECT title, quantity FROM sale")
  try {
    statement.setFetchSize(500)
    val rows = statement.executeQuery()
    try {
      val copiesPerTitle = mutable.Map.empty[String, Int]
      while (rows.next()) {
        val title    = rows.getString("title")
        val quantity = rows.getInt("quantity")
        copiesPerTitle(title) = copiesPerTitle.getOrElse(title, 0) + quantity
      }
      println(copiesPerTitle)
    } finally rows.close()
  } finally statement.close()
} finally connection.close()
```

This keeps memory flat, but every report repeats the same three nested `try`/`finally` blocks, decodes columns by hand, and mixes the reading, the decoding and the aggregation in one loop. Writing a second consumer, such as a file export, means copying the whole loop again.

In this guide, we solve this by asking `Frag` for a `Stream` of decoded rows and building each consumer from stream operators.

## Prerequisites

Add the `sql` module, the `streams` module and the SQLite JDBC driver to your `build.sbt`:

```scala
// Scala 3.9 and newer, with scalacOptions += "-experimental"; on Scala 3.3 to 3.8 the artifacts are "zio-blocks-sql" and "zio-blocks-streams"
libraryDependencies += "dev.zio"   %% "zio-blocks-next-sql"     % "0.0.56"
libraryDependencies += "dev.zio"   %% "zio-blocks-next-streams" % "0.0.56"
libraryDependencies += "org.xerial" % "sqlite-jdbc"             % "3.53.4.0"
```

The `sql` module already depends on `streams`; listing it makes the dependency explicit, since the code imports `zio.blocks.streams` directly. The `sql` module is published for Scala 3 only, and `JdbcTransactor` runs on the JVM.

We use these imports throughout the guide:

```scala

```

## The Sales Table

Our bookstore records one row per sale: which book, how many copies, and on which day. The sales data is fictional; the titles are real Scala books:

```scala
case class Sale(id: Long, title: String, quantity: Int, soldOn: LocalDate)

object Sale {
  given Schema[Sale] = Schema.derived
}

val saleTable = Table.derived[Sale]
val sales     = Repo.derived[Sale, Long]

val titles = Vector("Zionomicon", "Programming in Scala", "Functional Programming in Scala")
```

We store the table in a temporary SQLite file. `jdbc:sqlite::memory:` would not work here: SQLite gives every new connection its own empty in-memory database, and each `connect` block opens a new connection. Then we create the table and insert 12,000 sales in one transaction:

```scala
val dbFile     = Files.createTempFile("bookstore-sales", ".db")
val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

val saleCount = 12000

transactor.transact {
  saleTable.createTable(SqlDialect.SQLite).update
  // In every 7 sales, 3 are of Zionomicon and 2 of each other title
  sales.insertBatch((1 to saleCount).map { i =>
    Sale(i.toLong, titles(i % 7 % 3), 1 + i % 4, LocalDate.of(2026, 1, 1).plusDays((i % 365).toLong))
  })
}
```

We also name the query we stream from in most sections. A `Frag` is only SQL text and parameters, so defining it touches no connection:

```scala
val allSales = sql"SELECT id, title, quantity, sold_on FROM sale ORDER BY id"
```

## Aggregate a Large Table One Chunk at a Time

`Frag#queryStream[A]` returns a `Stream[Throwable, Chunk[A]]`. Each element of the stream is a `Chunk` of up to 64 decoded rows, and a terminal operation such as `runFold` pulls the chunks one after another and returns an `Either`. We add up the copies sold per title, and also record how many chunks arrived and how large the biggest one was:

```scala
case class Report(copiesPerTitle: Map[String, Int], rows: Int, chunks: Int, largestChunk: Int)

val emptyReport = Report(Map.empty, 0, 0, 0)

def addChunk(report: Report, chunk: Chunk[Sale]): Report = {
  val copies = chunk.foldLeft(report.copiesPerTitle) { (totals, sale) =>
    totals.updated(sale.title, totals.getOrElse(sale.title, 0) + sale.quantity)
  }
  Report(copies, report.rows + chunk.size, report.chunks + 1, report.largestChunk.max(chunk.size))
}

val report: Either[Throwable, Report] =
  transactor.connect {
    allSales.queryStream[Sale].runFold(emptyReport)(addChunk)
  }
```

Then we print it:

```scala
report.foreach { r =>
  r.copiesPerTitle.toList.sorted.foreach((title, copies) => println(s"$title: $copies copies"))
  println(s"${r.rows} rows in ${r.chunks} chunks, at most ${r.largestChunk} rows per chunk")
}
// Functional Programming in Scala: 8569 copies
// Programming in Scala: 8572 copies
// Zionomicon: 12859 copies
// 12000 rows in 188 chunks, at most 64 rows per chunk
```

The fold saw all 12,000 rows, but each step only received one chunk of at most 64 of them. The fold keeps the running totals and drops each chunk once it has added it in, so the rows the stream holds at any moment are bounded by the chunk size, not by the size of the table.

## Choose a Chunk Size

`queryStream` is `queryChunked` with the default chunk size of 64. When you want larger or smaller batches, call `Frag#queryChunked[A](chunkSize)` instead. We run the same fold with chunks of 1,000 rows:

```scala
val bigChunks: Either[Throwable, Report] =
  transactor.connect {
    allSales.queryChunked[Sale](1000).runFold(emptyReport)(addChunk)
  }
```

The totals don't change, but the batches do:

```scala
bigChunks.foreach(r => println(s"${r.rows} rows in ${r.chunks} chunks, at most ${r.largestChunk} rows per chunk"))
// 12000 rows in 12 chunks, at most 1000 rows per chunk
```

The chunk size trades memory for per-chunk overhead. Larger chunks mean fewer steps through the stream and suit consumers that work best in batches, such as a batch insert into another table or a buffered write. Smaller chunks keep fewer rows in memory and deliver the first rows sooner, which matters when each row is large or when the consumer is slow. The chunk size bounds only the rows the stream holds; how many rows the JDBC driver fetches from the server per round trip is the driver's own setting.

The size must be at least 1. `queryChunked` checks it as soon as it is called, before any stream exists, so a bad size throws instead of returning a `Left`:

```scala
println(Try(transactor.connect(allSales.queryChunked[Sale](0))))
// Failure(java.lang.IllegalArgumentException: requirement failed: queryChunked: chunkSize must be >= 1, got 0)
```

## Export Every Row to a CSV File

To export the table, we write each chunk to a file as it arrives. `Sink.foreach` builds a `Sink` that runs a function for every element, and `Stream#run(sink)` drains the stream into it. We open a buffered writer, write a header line, and turn every sale into one CSV line:

```scala
def csvLine(sale: Sale): String =
  s"${sale.id},${sale.title},${sale.quantity},${sale.soldOn}"

val exportFile = Files.createTempFile("sales-export", ".csv")

val exported: Either[Throwable, Unit] = {
  val writer = Files.newBufferedWriter(exportFile)
  try {
    writer.write("id,title,quantity,sold_on")
    writer.newLine()
    transactor.connect {
      allSales.queryStream[Sale].run(Sink.foreach[Chunk[Sale]] { chunk =>
        chunk.foreach { sale =>
          writer.write(csvLine(sale))
          writer.newLine()
        }
      })
    }
  } finally writer.close()
}
```

`Sink.foreach` doesn't close the writer, so we close it in `finally`, whether the export succeeds or fails. Then we check the file:

```scala
println(exported)
// Right(())
println(s"Lines in the file: ${Files.readAllLines(exportFile).size}")
// Lines in the file: 12001
Files.readAllLines(exportFile).subList(0, 4).forEach(line => println(line))
// id,title,quantity,sold_on
// 1,Programming in Scala,2,2026-01-02
// 2,Functional Programming in Scala,3,2026-01-03
// 3,Zionomicon,4,2026-01-04
```

The file has the header and one line per sale. None of our titles contains a comma or a quote, so `csvLine` doesn't quote its fields; for real data, quote any field that may contain them.

## Filter and Reshape Rows on the Way Out

Stream operators work between the query and the sink, so the file receives only the rows and the columns we want. To export only Zionomicon sales as `date,copies` lines, we first flatten the chunks into single rows with `flatMap(Stream.fromChunk(_))`, then filter, map each sale to its line, and batch the lines again with `grouped` so the writer gets 500 lines per call:

```scala
val zionomiconFile = Files.createTempFile("zionomicon-sales", ".csv")

val zionomiconBatches: Either[Throwable, Long] = {
  val writer = Files.newBufferedWriter(zionomiconFile)
  try
    transactor.connect {
      allSales
        .queryChunked[Sale](1000)
        .flatMap(chunk => Stream.fromChunk(chunk))
        .filter(_.title == "Zionomicon")
        .map(sale => s"${sale.soldOn},${sale.quantity}")
        .grouped(500)
        .tapEach(lines => lines.foreach { line => writer.write(line); writer.newLine() })
        .count
    }
  finally writer.close()
}
```

`tapEach` writes each batch and passes it on, and `count` returns how many batches went through:

```scala
println(s"Batches written: $zionomiconBatches")
// Batches written: Right(11)
println(s"Lines in the file: ${Files.readAllLines(zionomiconFile).size}")
// Lines in the file: 5142
Files.readAllLines(zionomiconFile).subList(0, 3).forEach(line => println(line))
// 2026-01-04,4
// 2026-01-07,3
// 2026-01-08,4
```

The file has one line per Zionomicon sale. `grouped(500)` emits full batches of 500 lines and a smaller last batch for the rest.

:::warning
A known bug: after a `flatMap` over objects such as rows, a `map` to a primitive (`Int`, `Long`, `Double`, `Boolean`, `Char`) followed by `grouped`, `sliding`, `take`, `takeWhile` or `scan` throws a `ClassCastException` from blocking terminals such as `runCollect` or `count`. For example, `.flatMap(chunk => Stream.fromChunk(chunk)).map(_.quantity).grouped(500)` fails. Keep the elements as objects or strings until after the stateful operator, as above, or map to the primitive inside the batches (`.grouped(500).map(_.map(_.quantity))`).
:::

## Keep the Stream Inside the Connection Block

A query stream is lazy. `queryStream` only prepares a description of the work; the statement runs and the result set opens on the first pull, and both close when the stream is fully drained or closed early. The connection itself belongs to the `connect` (or `transact`) block that provided the `DbCon`, and that block closes it when it returns.

So a stream that leaves its block can't be read any more. Here we return the stream itself from `connect` and run it afterwards:

```scala
val escaped: Stream[Throwable, Chunk[Sale]] =
  transactor.connect(allSales.queryStream[Sale])
```

The first pull happens after the connection is closed:

```scala
println(escaped.runFold(0)((rows, chunk) => rows + chunk.size))
// Left(java.sql.SQLException: database connection closed)
```

The failure is a `Left` from the terminal operation, not an exception. The fix is to run the terminal operation inside the block and return its result instead of the stream:

```scala
println(transactor.connect(allSales.queryStream[Sale].runFold(0)((rows, chunk) => rows + chunk.size)))
// Right(12000)
```

Because the stream is a description, every terminal operation runs the query again. If you need two results from the same rows, compute both in one fold, as the report above does, rather than running the stream twice.

:::caution
Consume the stream on the connection that opened it. Calling `connect` or `transact` again inside an open block, for example to write each chunk to another table, opens a second connection while the first one still holds the result set open. On SQLite the write then waits for the read's lock and fails with `SQLITE_BUSY`; with a connection pool on PostgreSQL, enough nested calls exhaust the pool and time out. Do the writes through the same `DbCon`, inside the same block.
:::

## Handle Query and Decoding Errors

A query that fails surfaces as a `Left` from the terminal operation, carrying the driver's exception. A misspelled column fails when the statement runs on the first pull:

```scala
val badColumn = transactor.connect {
  sql"SELECT id, titel, quantity, sold_on FROM sale".queryStream[Sale].runFold(0)((rows, chunk) => rows + chunk.size)
}
```

We match on the result:

```scala
badColumn match {
  case Left(error)  => println(s"Query failed: ${error.getMessage}")
  case Right(rows)  => println(s"Read $rows rows")
}
// Query failed: [SQLITE_ERROR] SQL error or missing database (no such column: titel)
```

A row that can't be decoded fails the same way, at the row where decoding fails. Here the query hands the `title` column to the `soldOn` field, and a book title isn't a date:

```scala
val badRow = transactor.connect {
  sql"SELECT id, title, quantity, title AS sold_on FROM sale".queryStream[Sale].runFold(0)((rows, chunk) => rows + chunk.size)
}
```

The first row already fails:

```scala
badRow.left.foreach(error => println(s"${error.getClass.getSimpleName}: ${error.getMessage}"))
// DateTimeParseException: Text 'Programming in Scala' could not be parsed at index 0
```

Both errors arrive as `Throwable` values, so you decide in one place whether to log them, retry, or turn them into a domain error with `Either#left.map`.

:::warning
A known bug: a query that returns fewer columns than the row type has (for example `SELECT id, title FROM sale` decoded as `Sale`) throws a `java.sql.SQLException` out of the terminal operation instead of returning a `Left`. Wrap such calls in `Try` if the column list comes from outside your code.
:::

## Putting It Together

The complete example creates the table, prints the per-title report, writes the CSV export, and deletes its temporary files:

```scala title="sql-examples/src/main/scala/streamlargequeryresults/CompleteExample.scala"
package streamlargequeryresults

/**
 * How to Stream Large Query Results — Complete Example
 *
 * Creates the table, prints the per-title report, writes the CSV export, and
 * deletes its temporary files.
 *
 * Run with: cd sql-examples && sbt "runMain
 * streamlargequeryresults.CompleteExample"
 */
object CompleteExample {
  def main(args: Array[String]): Unit =
    withSalesDb { transactor =>
      val report: Either[Throwable, Report] =
        transactor.connect {
          allSales.queryStream[Sale].runFold(emptyReport)(addChunk)
        }

      report.foreach { r =>
        r.copiesPerTitle.toList.sorted.foreach((title, copies) => println(s"$title: $copies copies"))
        println(s"${r.rows} rows in ${r.chunks} chunks, at most ${r.largestChunk} rows per chunk")
      }

      val exportFile = Files.createTempFile("sales-export", ".csv")
      try {
        val exported: Either[Throwable, Unit] = {
          val writer = Files.newBufferedWriter(exportFile)
          try {
            writer.write("id,title,quantity,sold_on")
            writer.newLine()
            transactor.connect {
              allSales
                .queryStream[Sale]
                .run(Sink.foreach[Chunk[Sale]] { chunk =>
                  chunk.foreach { sale =>
                    writer.write(csvLine(sale))
                    writer.newLine()
                  }
                })
            }
          } finally writer.close()
        }

        println(exported)
        println(s"Lines in the file: ${Files.readAllLines(exportFile).size}")
        Files.readAllLines(exportFile).subList(0, 4).forEach(line => println(line))
      } finally Files.deleteIfExists(exportFile)
    }
}
```

## Running the Examples

All the code in this guide has runnable companions in the `sql-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/sql-examples
```

Each example creates its own temporary database and files and deletes them when it finishes, so you can run them in any order. To compile all the examples without running them:

```bash
sbt compile
```

### The Shared Setup

The `Sale` row type, the 12,000 seeded sales and the temporary database every example uses are in this file:

<details>
  <summary>sql-examples/src/main/scala/streamlargequeryresults/Sales.scala</summary>

```scala title="sql-examples/src/main/scala/streamlargequeryresults/Sales.scala" showLineNumbers
package streamlargequeryresults

/**
 * How to Stream Large Query Results: shared setup
 *
 * The Sale row type with its Schema, the Table and Repo for it, the query the
 * examples stream from, the report helpers, and withSalesDb, which seeds 12,000
 * sales into a temporary SQLite file and deletes the file when the block ends.
 */
case class Sale(id: Long, title: String, quantity: Int, soldOn: LocalDate)

object Sale {
  given Schema[Sale] = Schema.derived
}

val saleTable = Table.derived[Sale]
val sales     = Repo.derived[Sale, Long]

val titles = Vector("Zionomicon", "Programming in Scala", "Functional Programming in Scala")

val saleCount = 12000

val allSales = sql"SELECT id, title, quantity, sold_on FROM sale ORDER BY id"

case class Report(copiesPerTitle: Map[String, Int], rows: Int, chunks: Int, largestChunk: Int)

val emptyReport = Report(Map.empty, 0, 0, 0)

def addChunk(report: Report, chunk: Chunk[Sale]): Report = {
  val copies = chunk.foldLeft(report.copiesPerTitle) { (totals, sale) =>
    totals.updated(sale.title, totals.getOrElse(sale.title, 0) + sale.quantity)
  }
  Report(copies, report.rows + chunk.size, report.chunks + 1, report.largestChunk.max(chunk.size))
}

def csvLine(sale: Sale): String =
  s"${sale.id},${sale.title},${sale.quantity},${sale.soldOn}"

def withSalesDb[A](f: JdbcTransactor => A): A = {
  val dbFile     = Files.createTempFile("bookstore-sales", ".db")
  val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
  try {
    transactor.transact {
      saleTable.createTable(SqlDialect.SQLite).update
      // In every 7 sales, 3 are of Zionomicon and 2 of each other title
      sales.insertBatch((1 to saleCount).map { i =>
        Sale(i.toLong, titles(i % 7 % 3), 1 + i % 4, LocalDate.of(2026, 1, 1).plusDays((i % 365).toLong))
      })
    }
    f(transactor)
  } finally Files.deleteIfExists(dbFile)
}
```

</details>

### Aggregate a Large Table One Chunk at a Time

<details>
  <summary>sql-examples/src/main/scala/streamlargequeryresults/Step1AggregateExample.scala</summary>

```scala title="sql-examples/src/main/scala/streamlargequeryresults/Step1AggregateExample.scala" showLineNumbers
package streamlargequeryresults

/**
 * How to Stream Large Query Results — Step 1: Aggregate a Large Table One Chunk
 * at a Time
 *
 * Folds the 12,000 sales into copies sold per title with runFold, one chunk at
 * a time.
 *
 * Run with: cd sql-examples && sbt "runMain
 * streamlargequeryresults.Step1AggregateExample"
 */
object Step1AggregateExample {
  def main(args: Array[String]): Unit =
    withSalesDb { transactor =>
      val report: Either[Throwable, Report] =
        transactor.connect {
          allSales.queryStream[Sale].runFold(emptyReport)(addChunk)
        }

      report.foreach { r =>
        r.copiesPerTitle.toList.sorted.foreach((title, copies) => println(s"$title: $copies copies"))
        println(s"${r.rows} rows in ${r.chunks} chunks, at most ${r.largestChunk} rows per chunk")
      }
    }
}
```

</details>

It prints the copies sold per title and the chunk statistics:

```bash
sbt "runMain streamlargequeryresults.Step1AggregateExample"
```

### Choose a Chunk Size

<details>
  <summary>sql-examples/src/main/scala/streamlargequeryresults/Step2ChunkSizeExample.scala</summary>

```scala title="sql-examples/src/main/scala/streamlargequeryresults/Step2ChunkSizeExample.scala" showLineNumbers
package streamlargequeryresults

/**
 * How to Stream Large Query Results — Step 2: Choose a Chunk Size
 *
 * Runs the same fold with the default chunk size of 64, with 1,000 and with 10,
 * and shows that a chunk size of 0 is rejected up front.
 *
 * Run with: cd sql-examples && sbt "runMain
 * streamlargequeryresults.Step2ChunkSizeExample"
 */
object Step2ChunkSizeExample {
  def main(args: Array[String]): Unit =
    withSalesDb { transactor =>
      def show(label: String, report: Either[Throwable, Report]): Unit =
        report.foreach(r =>
          println(s"$label: ${r.rows} rows in ${r.chunks} chunks, at most ${r.largestChunk} rows per chunk")
        )

      val defaultChunks: Either[Throwable, Report] =
        transactor.connect {
          allSales.queryStream[Sale].runFold(emptyReport)(addChunk)
        }

      val bigChunks: Either[Throwable, Report] =
        transactor.connect {
          allSales.queryChunked[Sale](1000).runFold(emptyReport)(addChunk)
        }

      val smallChunks: Either[Throwable, Report] =
        transactor.connect {
          allSales.queryChunked[Sale](10).runFold(emptyReport)(addChunk)
        }

      show("Default (64)", defaultChunks)
      show("1000", bigChunks)
      show("10", smallChunks)

      println(Try(transactor.connect(allSales.queryChunked[Sale](0))))
    }
}
```

</details>

It reads the table with three chunk sizes and shows the rejected size 0:

```bash
sbt "runMain streamlargequeryresults.Step2ChunkSizeExample"
```

### Export Every Row to a CSV File

<details>
  <summary>sql-examples/src/main/scala/streamlargequeryresults/Step3ExportExample.scala</summary>

```scala title="sql-examples/src/main/scala/streamlargequeryresults/Step3ExportExample.scala" showLineNumbers
package streamlargequeryresults

/**
 * How to Stream Large Query Results — Step 3: Export Every Row to a CSV File
 *
 * Writes every sale to a CSV file through Sink.foreach, then prints the line
 * count and the first lines of the file.
 *
 * Run with: cd sql-examples && sbt "runMain
 * streamlargequeryresults.Step3ExportExample"
 */
object Step3ExportExample {
  def main(args: Array[String]): Unit =
    withSalesDb { transactor =>
      val exportFile = Files.createTempFile("sales-export", ".csv")
      try {
        val exported: Either[Throwable, Unit] = {
          val writer = Files.newBufferedWriter(exportFile)
          try {
            writer.write("id,title,quantity,sold_on")
            writer.newLine()
            transactor.connect {
              allSales
                .queryStream[Sale]
                .run(Sink.foreach[Chunk[Sale]] { chunk =>
                  chunk.foreach { sale =>
                    writer.write(csvLine(sale))
                    writer.newLine()
                  }
                })
            }
          } finally writer.close()
        }

        println(exported)
        println(s"Lines in the file: ${Files.readAllLines(exportFile).size}")
        Files.readAllLines(exportFile).subList(0, 4).forEach(line => println(line))
      } finally Files.deleteIfExists(exportFile)
    }
}
```

</details>

It writes `sales.csv` and prints its first lines:

```bash
sbt "runMain streamlargequeryresults.Step3ExportExample"
```

### Filter and Reshape Rows on the Way Out

<details>
  <summary>sql-examples/src/main/scala/streamlargequeryresults/Step4TransformExample.scala</summary>

```scala title="sql-examples/src/main/scala/streamlargequeryresults/Step4TransformExample.scala" showLineNumbers
package streamlargequeryresults

/**
 * How to Stream Large Query Results — Step 4: Filter and Reshape Rows on the
 * Way Out
 *
 * Exports only the Zionomicon sales as date,copies lines, written in batches of
 * 500 lines.
 *
 * Run with: cd sql-examples && sbt "runMain
 * streamlargequeryresults.Step4TransformExample"
 */
object Step4TransformExample {
  def main(args: Array[String]): Unit =
    withSalesDb { transactor =>
      val zionomiconFile = Files.createTempFile("zionomicon-sales", ".csv")
      try {
        val zionomiconBatches: Either[Throwable, Long] = {
          val writer = Files.newBufferedWriter(zionomiconFile)
          try
            transactor.connect {
              allSales
                .queryChunked[Sale](1000)
                .flatMap(chunk => Stream.fromChunk(chunk))
                .filter(_.title == "Zionomicon")
                .map(sale => s"${sale.soldOn},${sale.quantity}")
                .grouped(500)
                .tapEach(lines => lines.foreach { line => writer.write(line); writer.newLine() })
                .count
            }
          finally writer.close()
        }

        println(s"Batches written: $zionomiconBatches")
        println(s"Lines in the file: ${Files.readAllLines(zionomiconFile).size}")
        Files.readAllLines(zionomiconFile).subList(0, 3).forEach(line => println(line))
      } finally Files.deleteIfExists(zionomiconFile)
    }
}
```

</details>

It exports only the Zionomicon sales, in batches of 500 lines:

```bash
sbt "runMain streamlargequeryresults.Step4TransformExample"
```

### Keep the Stream Inside the Connection Block

<details>
  <summary>sql-examples/src/main/scala/streamlargequeryresults/Step5ScopeExample.scala</summary>

```scala title="sql-examples/src/main/scala/streamlargequeryresults/Step5ScopeExample.scala" showLineNumbers
package streamlargequeryresults

/**
 * How to Stream Large Query Results — Step 5: Keep the Stream Inside the
 * Connection Block
 *
 * Returns a stream out of its connect block and fails on the first pull, then
 * runs the terminal operation inside the block.
 *
 * Run with: cd sql-examples && sbt "runMain
 * streamlargequeryresults.Step5ScopeExample"
 */
object Step5ScopeExample {
  def main(args: Array[String]): Unit =
    withSalesDb { transactor =>
      val escaped: Stream[Throwable, Chunk[Sale]] =
        transactor.connect(allSales.queryStream[Sale])

      println(escaped.runFold(0)((rows, chunk) => rows + chunk.size))

      println(transactor.connect(allSales.queryStream[Sale].runFold(0)((rows, chunk) => rows + chunk.size)))
    }
}
```

</details>

It runs a stream outside its `connect` block, then inside it:

```bash
sbt "runMain streamlargequeryresults.Step5ScopeExample"
```

### Handle Query and Decoding Errors

<details>
  <summary>sql-examples/src/main/scala/streamlargequeryresults/Step6ErrorsExample.scala</summary>

```scala title="sql-examples/src/main/scala/streamlargequeryresults/Step6ErrorsExample.scala" showLineNumbers
package streamlargequeryresults

/**
 * How to Stream Large Query Results — Step 6: Handle Query and Decoding Errors
 *
 * Shows the Left produced by a misspelled column and by a row that can't be
 * decoded.
 *
 * Run with: cd sql-examples && sbt "runMain
 * streamlargequeryresults.Step6ErrorsExample"
 */
object Step6ErrorsExample {
  def main(args: Array[String]): Unit =
    withSalesDb { transactor =>
      val badColumn = transactor.connect {
        sql"SELECT id, titel, quantity, sold_on FROM sale"
          .queryStream[Sale]
          .runFold(0)((rows, chunk) => rows + chunk.size)
      }

      badColumn match {
        case Left(error) => println(s"Query failed: ${error.getMessage}")
        case Right(rows) => println(s"Read $rows rows")
      }

      val badRow = transactor.connect {
        sql"SELECT id, title, quantity, title AS sold_on FROM sale"
          .queryStream[Sale]
          .runFold(0)((rows, chunk) => rows + chunk.size)
      }

      badRow.left.foreach(error => println(s"${error.getClass.getSimpleName}: ${error.getMessage}"))
    }
}
```

</details>

It prints the `Left` for a misspelled column and for a row that can't be decoded:

```bash
sbt "runMain streamlargequeryresults.Step6ErrorsExample"
```

### The Complete Example

```bash
sbt "runMain streamlargequeryresults.CompleteExample"
```

## Going Further

- [`Frag`](../reference/sql/frag.md) documents `queryStream`, `queryChunked` and the other execution methods.
- [`Transactor`](../reference/sql/transactor.md) explains how `connect` and `transact` open and close connections.
- [How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md) moves the same code to PostgreSQL, where pool size limits how many streams can be open at once.
- [How to Test Database Code Against SQLite](./test-database-code-against-sqlite.md) shows how to give each test its own database file.
- The `sql-zio` module has no bridge from a query stream to a ZIO `ZStream` yet. To use a query stream from ZIO, run the whole stream, terminal operation included, inside [`TransactorZIO#connect`](../reference/sql/transactor-zio.md), which runs its block on ZIO's blocking thread pool.

---

## Coming from fs2, ZIO Streams or Ox

This guide moves a bookstore's order pipeline from fs2, ZIO Streams or Ox to the ZIO Blocks [`streams` module](../reference/streams/index.md). By the end, the pipeline reads an order log, parses it with typed errors, prices the orders in parallel and totals the sales per title, using `Stream`, `Sink` and `Pipeline` in plain Scala, with no effect type and no runtime to start. Each source library gets its own section with the same three parts: the code you wrote, the code you write now, and what changes. Read the concept map, then jump to the section for the library you are leaving. Every migration should also read [Traps When Migrating](#traps-when-migrating): those are the places where code that looks like a direct translation behaves differently.

- [Concept Map](#concept-map)
- [Coming from ZIO Streams](#coming-from-zio-streams)
- [Coming from fs2](#coming-from-fs2)
- [Coming from Ox](#coming-from-ox)
- [Traps When Migrating](#traps-when-migrating)
- [What You Gain and What Is Different](#what-you-gain-and-what-is-different)
- [Migration Checklist](#migration-checklist)

## The Problem

fs2 and ZIO Streams tie every stream to an effect system. Each stage that calls ordinary code is wrapped in `IO` or `ZIO`, and nothing runs until an `IOApp`, a `ZIOAppDefault` or an unsafe runtime call interprets it. That is the right trade when the program is already built on cats-effect or ZIO. When the code around the stream is plain, synchronous Scala, the effect type is overhead: one more dependency, one more runtime, and wrappers around functions that never needed them.

Here is a small fs2 pipeline that totals copies sold per title. `parseOrder` and `addOrder` are pure functions from [the bookstore model](#the-bookstore-order-log) below, yet the stream still needs `IO`, `evalMap`, `compile` and `IOApp` to produce a `Map`.

This is the old code, fs2 3.14.0 (not compiled here):

```scala

object CopiesPerTitle extends IOApp.Simple {
  val run: IO[Unit] =
    Stream
      .emits(orderLines)
      .evalMap(line => IO(parseOrder(line)))
      .collect { case Right(order) => order }
      .compile
      .fold(Map.empty[String, Int])(addOrder)
      .flatMap(totals => IO.println(s"copies per title: $totals"))
}
```

The ZIO Streams version has the same shape with `ZStream`, `runFold` and `ZIOAppDefault`. Ox removes the effect type, but it is Scala 3.9+ and JDK 21+ only, and it reports failures as exceptions rather than typed errors.

ZIO Blocks Streams runs the same pipeline as a direct call: `runFold` blocks the calling thread and returns `Either[E, Z]`, with the stream's typed error on the left. The same `Stream` compiles on Scala 2.13 and Scala 3, and on Scala.js it runs through the asynchronous terminals.

## Prerequisites

The module you add depends on your Scala version.

- **Scala 2.13 and Scala 3.3 to 3.8:**

  ```scala
  libraryDependencies += "dev.zio" %% "zio-blocks-streams" % "0.0.56"
  ```

- **Scala 3.9 and later:** use the `next` artifact and turn on experimental features:

  ```scala
  libraryDependencies += "dev.zio" %% "zio-blocks-next-streams" % "0.0.56"
  scalacOptions += "-experimental"
  ```

On the JVM, the module is compiled to Java 21 bytecode, so it needs JDK 21 or later at runtime. For Scala.js, use `%%%` and Scala.js 1.22 or later. The module depends only on other ZIO Blocks modules (`scope`, `chunk`, `combinators`, `ringbuffer` and `async`), not on cats-effect, ZIO or Ox.

Every ZIO Blocks snippet below starts from this import:

```scala

```

This guide assumes you know one of the three source libraries. It maps their concepts onto ZIO Blocks rather than teaching streams from scratch. The [Streams reference](../reference/streams/index.md) covers `Stream`, `Sink` and `Pipeline` in depth. The how-to guides [How to Stream Large Query Results](./stream-large-query-results.md) and [How to Stream Files and Sockets Without Leaking Resources](./stream-files-and-sockets.md) apply streams to databases and I/O. The tutorial "Build your first streaming pipeline" is in progress.

## The Bookstore Order Log

The running example is an order log, one CSV line per order: an order ID, a book title and the number of copies. The sales data is fictional. One line is malformed on purpose, so every library has to decide what to do with it. `priceOf` stands in for a slow call to a pricing service, and `ordersPath` is a temporary file holding the same log:

```scala

final case class OrderEvent(orderId: Long, title: String, copies: Int)

// Fictional order log: one CSV line per order, and one malformed line
val orderLines: List[String] = List(
  "1,Zionomicon,2",
  "2,Programming in Scala,1",
  "3,Functional Programming in Scala,3",
  "not-an-order",
  "4,Zionomicon,1",
  "5,Programming in Scala,4"
)

def parseOrder(line: String): Either[String, OrderEvent] =
  line.split(',') match {
    case Array(id, title, copies) =>
      (id.toLongOption, copies.toIntOption) match {
        case (Some(i), Some(c)) => Right(OrderEvent(i, title, c))
        case _                  => Left(s"malformed order: $line")
      }
    case _ => Left(s"malformed order: $line")
  }

def addOrder(totals: Map[String, Int], order: OrderEvent): Map[String, Int] =
  totals.updated(order.title, totals.getOrElse(order.title, 0) + order.copies)

private val listPrices: Map[String, BigDecimal] = Map(
  "Zionomicon"                      -> BigDecimal("49.99"),
  "Programming in Scala"            -> BigDecimal("59.95"),
  "Functional Programming in Scala" -> BigDecimal("44.99")
)

// A slow lookup, standing in for a call to a pricing service
def priceOf(title: String): BigDecimal = {
  Thread.sleep(20)
  listPrices(title)
}

def addRevenue(totals: Map[String, BigDecimal], sale: (String, BigDecimal)): Map[String, BigDecimal] =
  totals.updated(sale._1, totals.getOrElse(sale._1, BigDecimal(0)) + sale._2)

val ordersPath: Path = Files.write(Files.createTempFile("orders", ".csv"), orderLines.asJava)
```

None of this code uses a streaming library, so the "before" snippets for fs2, ZIO Streams and Ox below use it unchanged.

## Concept Map

| Task                        | fs2 3.14                                | ZIO Streams 2.1                                | Ox 1.1                             | ZIO Blocks                                                                     |
|-----------------------------|-----------------------------------------|------------------------------------------------|------------------------------------|--------------------------------------------------------------------------------|
| Stream type                 | `Stream[F, O]`                          | `ZStream[R, E, A]`                             | `Flow[T]`                          | `Stream[E, A]`                                                                 |
| Reusable transformation     | `Pipe[F, I, O]`, `through`              | `ZPipeline[R, E, In, Out]`, `via`              | a function `Flow[A] => Flow[B]`    | `Pipeline[In, Out]`, `via`                                                     |
| Consumer                    | `compile.*`                             | `ZSink[R, E, In, L, Z]`, `run(sink)`           | `run*` methods                     | `Sink[E, A, Z]`, `run(sink)`                                                   |
| From a collection           | `Stream.emits(xs)`                      | `ZStream.fromIterable(xs)`                     | `Flow.fromIterable(xs)`            | `Stream.fromIterable(xs)`, `Stream(a, b, c)`                                   |
| From an iterator            | `Stream.fromIterator[F](it, chunkSize)` | `ZStream.fromIterator(it)`                     | `Flow.fromIterator(it)`            | `Stream.fromIterator(it)`                                                      |
| Unfold a state              | `Stream.unfold(s)(f)`                   | `ZStream.unfold(s)(f)`                         | `Flow.unfold(s)(f)`                | `Stream.unfold(s)(f)`                                                          |
| Push elements from a loop   | none covered here                       | none covered here                              | `Flow.usingEmit(emit => ...)`      | `Stream.unfold`, `Stream.fromIterator`, or a `Reader` with `Stream.fromReader` |
| Call code for each element  | `evalMap(a => IO(f(a)))`                | `mapZIO(a => ZIO.attempt(f(a)))`               | `map(f)`                           | `map(f)`; `mapAsync` for code that returns `Async`                             |
| Parallel map                | `parEvalMap(n)` (ordered)               | `mapZIOPar(n)` (ordered)                       | `mapPar(n)` (ordered)              | `mapPar(n)`, `mapParAsync(n)` (unordered only)                                 |
| Parallel map, unordered     | `parEvalMapUnordered(n)`                | `mapZIOParUnordered(n)`                        | `mapParUnordered(n)`               | `mapPar(n)`, `mapParAsync(n)`                                                  |
| Parallel flatMap            | `map(f).parJoin(n)`                     | `flatMapPar(n)(f)`                             | `map(f).flattenPar(n)`             | `flatMapPar(n)(f)`                                                             |
| Merge two streams           | `a.merge(b)`                            | `a.merge(b)`                                   | `a.merge(b)`                       | `Stream.mergeAll(2)(Stream(a, b))`                                             |
| Acquire and release         | `Stream.bracket`, `Stream.resource`     | `ZStream.acquireReleaseWith`, `ZStream.scoped` | `usingEmit` with `try`/`finally`   | `Stream.fromAcquireRelease`, `Stream.fromResource`                             |
| Fixed-size batches          | `chunkN(n)`                             | `grouped(n)`                                   | `grouped(n)` (a `Seq`)             | `grouped(n)` or `chunked(n)` (a `Chunk`)                                       |
| Running state               | `scan`, `mapAccumulate`                 | `scan`, `mapAccum`                             | `scan`, `mapStateful`              | `scan`, `mapAccum`                                                             |
| Custom pull loop            | `Pull`, `stream.pull.uncons1`           | none covered here                              | none                               | `start` plus `Reader#read`, or `mapAccum`                                      |
| Typed error                 | none (`Throwable` in `F`)               | the `E` channel                                | none (exceptions; `either` blocks) | the `E` channel, `Left` from every terminal                                    |
| Recover from an error       | `handleErrorWith`                       | `catchAll`, `orElse`                           | `recover`, `recoverWith`           | `catchAll`, `orElse` (alias `\|\|`), `catchDefect`                             |
| Turn exceptions into errors | `attempt`                               | `ZStream.fromZIO(ZIO.attempt(...))`            | `either.catching`                  | `Stream.attempt(f)`                                                            |
| Collect everything          | `compile.toList`                        | `runCollect`                                   | `runToList()`                      | `runCollect`                                                                   |
| Fold                        | `compile.fold(z)(f)`                    | `runFold(z)(f)`                                | `runFold(z)(f)`                    | `runFold(z)(f)`                                                                |
| Drain                       | `compile.drain`                         | `runDrain`                                     | `runDrain()`                       | `runDrain`                                                                     |
| Count                       | `compile.count`                         | `runCount`                                     | none                               | `count`                                                                        |
| Run the program             | `IOApp`, `unsafeRunSync()`              | `ZIOAppDefault`, `Unsafe.unsafe`               | a direct call                      | a direct call on the JVM; `*Async` terminals on Scala.js                       |

## Coming from ZIO Streams

`ZStream[R, E, A]`, `ZSink` and `ZPipeline` map almost one to one onto `Stream[E, A]`, `Sink[E, A, Z]` and `Pipeline[In, Out]`. The `E` channel works the same way. What goes away is the environment `R` and the `ZIO` effect around every step.

### A First Pipeline

#### You Wrote This

This is the old code, ZIO Streams 2.1.27 (not compiled here):

```scala

object CopiesPerTitle extends ZIOAppDefault {
  val run =
    ZStream
      .fromIterable(orderLines)
      .map(parseOrder)
      .collect { case Right(order) => order }
      .runFold(Map.empty[String, Int])(addOrder)
      .debug("copies per title")
}
```

#### You Write This

The operators keep their names. `runFold` runs the stream on the spot instead of returning an effect for `ZIOAppDefault` to run:

```scala
val copiesPerTitle: Either[Nothing, Map[String, Int]] =
  Stream
    .fromIterable(orderLines)
    .map(parseOrder)
    .collect { case Right(order) => order }
    .runFold(Map.empty[String, Int])(addOrder)
```

```scala
println(copiesPerTitle)
// Right(Map(Zionomicon -> 3, Programming in Scala -> 5, Functional Programming in Scala -> 3))
```

The result is an `Either` even though this stream cannot fail: every terminal returns `Either[E, Z]`, and `E` is `Nothing` here.

### Typed Errors

#### You Wrote This

ZIO Streams carries a parse failure in the error channel with `ZIO.fromEither`, and recovers with `catchAll` (ZIO Streams 2.1.27, not compiled here):

```scala

val orders: ZStream[Any, String, OrderEvent] =
  ZStream.fromIterable(orderLines).mapZIO(line => ZIO.fromEither(parseOrder(line)))

// fails with "malformed order: not-an-order"
val strict: IO[String, Map[String, Int]] =
  orders.runFold(Map.empty[String, Int])(addOrder)

// stops at the malformed line and keeps what it has
val lenient: UIO[Map[String, Int]] =
  orders.catchAll(_ => ZStream.empty).runFold(Map.empty[String, Int])(addOrder)
```

#### You Write This

There is no `mapZIO`, so turn each `Either` into a one-element stream or a failed stream with `flatMap`. The error type is inferred from `Stream.fail`:

```scala
def orderOrFailure(line: String): Stream[String, OrderEvent] =
  parseOrder(line).fold(error => Stream.fail(error), order => Stream.succeed(order))

val orders: Stream[String, OrderEvent] =
  Stream.fromIterable(orderLines).flatMap(orderOrFailure)
```

The failure comes back as a `Left` from the terminal, and `catchAll` recovers from it on the outer stream, as in ZIO:

```scala
println(orders.runFold(Map.empty[String, Int])(addOrder))
// Left(malformed order: not-an-order)

println(orders.catchAll(_ => Stream.empty).runFold(Map.empty[String, Int])(addOrder))
// Right(Map(Zionomicon -> 2, Programming in Scala -> 1, Functional Programming in Scala -> 3))
```

:::warning
Put `catchAll` on the outer stream, as above, not on the inner stream inside `flatMap`. Recovery inside `flatMap` followed by `grouped`, `sliding`, `scan` or `mapAccum` throws an internal error from blocking terminals. See [Recovery inside `flatMap`](#recovery-inside-flatmap).
:::

### Resources and Parallel Work

#### You Wrote This

The order log comes from a file opened with `ZStream.acquireReleaseWith`, and each order is priced in parallel with `mapZIOPar` (ZIO Streams 2.1.27, not compiled here):

```scala

def orderFile(path: Path): ZStream[Any, Throwable, String] =
  ZStream
    .acquireReleaseWith(ZIO.attempt(Files.newBufferedReader(path)))(reader =>
      ZIO.succeed { reader.close(); println("order file closed") }
    )
    .flatMap(reader => ZStream.fromIterator(reader.lines().iterator().asScala))

val revenue: Task[Map[String, BigDecimal]] =
  orderFile(ordersPath)
    .map(parseOrder)
    .collect { case Right(order) => order }
    .mapZIOPar(4)(order => ZIO.attemptBlocking(order.title -> priceOf(order.title) * order.copies))
    .runFold(Map.empty[String, BigDecimal])(addRevenue)
```

#### You Write This

`ZStream.acquireReleaseWith(acquire)(release).flatMap(use)` becomes `Stream.fromAcquireRelease(acquire, release)(use)`. Acquire and release are plain code, and `mapZIOPar` becomes `mapPar`:

```scala

def orderFile(path: Path): Stream[Nothing, String] =
  Stream.fromAcquireRelease(
    Files.newBufferedReader(path),
    (reader: BufferedReader) => { reader.close(); println("order file closed") }
  )(reader => Stream.fromIterator(reader.lines().iterator().asScala))

val revenue: Stream[Nothing, (String, BigDecimal)] =
  orderFile(ordersPath)
    .map(parseOrder)
    .collect { case Right(order) => order }
    .mapPar(4)(order => order.title -> priceOf(order.title) * order.copies)
```

Each run opens the file, and the release runs when the run ends. `mapPar` finishes the orders in any order, so the totals are sorted by title before printing:

```scala
println(revenue.runFold(Map.empty[String, BigDecimal])(addRevenue).map(_.toList.sortBy(_._1)))
// order file closed
// Right(List((Functional Programming in Scala,134.97), (Programming in Scala,299.75), (Zionomicon,149.97)))
```

The release argument is optional. Without it, `fromAcquireRelease` calls `close()` on an `AutoCloseable` resource. A file that cannot be opened throws from the terminal, because exceptions in `acquire` are defects; wrap the open in `Stream.attempt` to get a `Stream[Throwable, A]` instead. For a resource built with the ZIO Blocks [`scope` module](../reference/resource-management/index.md), `Stream.fromResource(resource)(use)` is the `ZStream.scoped` counterpart: it opens a scope for each run and closes it when the run ends.

:::warning
`mapZIOPar` keeps input order; `mapPar` does not. The fold above does not care about order. If yours does, read [`mapPar` emits in completion order](#mappar-emits-in-completion-order).
:::

### ZIO Effects in the Pipeline

A `ZIO` step becomes one of two things:

- **A blocking call.** `ZIO.attempt` and `ZIO.attemptBlocking` around ordinary code disappear: call the code in `map`, or in `mapPar` to run several calls at once. On the JVM, `mapPar` and `flatMapPar` run their workers on virtual threads, so blocking in them is cheap.
- **An `Async` value.** When the code you call is itself asynchronous, for example a client that returns a `Future`, convert it to `zio.blocks.async.Async` and use `mapAsync` or `mapParAsync`. On the JVM, `Async.fromFuture` and `Async.fromCompletionStage` convert from the standard types:

```scala

// A pricing client with a Future-based API
def priceFromService(title: String)(implicit ec: ExecutionContext): Future[BigDecimal] =
  Future(priceOf(title))
```

```scala
implicit val ec: ExecutionContext = ExecutionContext.global

val asyncRevenue: Either[Nothing, Map[String, BigDecimal]] =
  Stream
    .fromIterable(orderLines)
    .map(parseOrder)
    .collect { case Right(order) => order }
    .mapParAsync(4)(order =>
      Async.fromFuture(priceFromService(order.title).map(price => order.title -> price * order.copies))
    )
    .runFold(Map.empty[String, BigDecimal])(addRevenue)
```

```scala
println(asyncRevenue.map(_.toList.sortBy(_._1)))
// Right(List((Functional Programming in Scala,134.97), (Programming in Scala,299.75), (Zionomicon,149.97)))
```

A ZIO program that keeps ZIO elsewhere can still call a ZIO Blocks stream: the blocking terminals are plain calls, so wrap one in `ZIO.attemptBlocking`.

### What Has No Equivalent

These ZIO Streams features have no ZIO Blocks counterpart. Keep ZIO Streams for the parts of a program that need them:

- **The environment.** There is no `R`, so `ZStream.service`, `serviceWith` and `serviceWithZIO` have no counterpart. Pass dependencies as ordinary parameters, or wire them with the `scope` module's `Resource.from`.
- **Schedules and time.** There is no `Schedule`, so `ZStream.repeatWithSchedule`, `repeat(schedule)`, `schedule`, `ZStream.tick`, `throttleShape`, `throttleEnforce` and `groupedWithin` have no counterpart. `Stream#repeated` repeats a whole stream with no delay, and `Stream.repeat(a)` repeats one value forever.
- **Fibers and interruption.** There is no fiber to fork, join or interrupt. Concurrency lives inside `mapPar`, `mapParAsync`, `flatMapPar` and `Stream.mergeAll`.
- **Merge strategies.** `merge` with a `HaltStrategy` has no counterpart. `Stream.mergeAll(n)(streams)` ends when every input ends.
- **Sink leftovers and channels.** A `Sink[E, A, Z]` has no leftover type, and there is no `ZChannel`. For custom stateful operators, use `mapAccum`, `scan` or a manual pull with `start` (see [Custom Operators with `Pull`](#custom-operators-with-pull)).

### What Changes

- **No effect type and no runtime.** `Stream[E, A]` has two type parameters. Stages are plain functions, and a terminal returns its result instead of an effect.
- **Every terminal returns `Either[E, Z]`.** `runCollect`, `runFold`, `runDrain`, `count` and `run(sink)` return `Left` for a typed error. An exception thrown inside the pipeline is a defect: blocking terminals rethrow it, and `catchDefect` recovers from it.
- **`fromInputStream` closes its stream.** ZIO's `ZStream.fromInputStream` never closes the `InputStream`. ZIO Blocks' `Stream.fromInputStream` closes it when the run ends, and `Stream.fromInputStreamUnmanaged` leaves it open. See [Managed handles are opened once](#managed-handles-are-opened-once).
- **`merge` becomes `Stream.mergeAll`.** `a.merge(b)` becomes `Stream.mergeAll(2)(Stream(a, b))`, and `ZStream.mergeAll(n)(streams*)` keeps its name with the streams passed as a stream.

## Coming from fs2

fs2's `Stream[F, O]` and ZIO Blocks' `Stream[E, A]` are both pull-based. The ZIO Blocks one has no `F[_]`: where fs2 puts `IO` in the type and `Throwable` in `F`, ZIO Blocks puts a typed error `E`.

### You Wrote This

The order log is read with `Stream.bracket`, parsed with a `Pipe`, and priced in parallel with `parEvalMap`. This is the old code, fs2 3.14.0 (not compiled here):

```scala

def orderFile(path: Path): Stream[IO, String] =
  Stream
    .bracket(IO.blocking(Files.newBufferedReader(path)))(reader =>
      IO.blocking { reader.close(); println("order file closed") }
    )
    .flatMap(reader => Stream.fromIterator[IO](reader.lines().iterator().asScala, 64))

val parseOrders: Pipe[IO, String, OrderEvent] =
  _.map(parseOrder).collect { case Right(order) => order }

val revenue: IO[Map[String, BigDecimal]] =
  orderFile(ordersPath)
    .through(parseOrders)
    .parEvalMap(4)(order => IO.blocking(order.title -> priceOf(order.title) * order.copies))
    .compile
    .fold(Map.empty[String, BigDecimal])(addRevenue)
```

### You Write This

`Stream.bracket(acquire)(release).flatMap(use)` becomes `Stream.fromAcquireRelease(acquire)(use)`; with no release function, the `BufferedReader` is closed as an `AutoCloseable`. A `Pipe` becomes a `Pipeline` built from `Pipeline.map`, `Pipeline.collect` and `andThen`, and `through` becomes `via`. `parEvalMap` becomes `mapPar`, and `compile.fold` becomes `runFold`:

```scala
def orderLog(path: Path): Stream[Nothing, String] =
  Stream.fromAcquireRelease(Files.newBufferedReader(path))(reader =>
    Stream.fromIterator(reader.lines().iterator().asScala)
  )

val parseOrders: Pipeline[String, OrderEvent] =
  Pipeline
    .map[String, Either[String, OrderEvent]](parseOrder)
    .andThen(Pipeline.collect[Either[String, OrderEvent], OrderEvent] { case Right(order) => order })
```

```scala
val revenueFromLog: Either[Nothing, Map[String, BigDecimal]] =
  orderLog(ordersPath)
    .via(parseOrders)
    .mapPar(4)(order => order.title -> priceOf(order.title) * order.copies)
    .runFold(Map.empty[String, BigDecimal])(addRevenue)
```

```scala
println(revenueFromLog.map(_.toList.sortBy(_._1)))
// Right(List((Functional Programming in Scala,134.97), (Programming in Scala,299.75), (Zionomicon,149.97)))
```

`Stream.fromIterator` needs no chunk size: ZIO Blocks pulls one element at a time and specializes primitive elements instead of batching them into chunks. `Stream.resource(r)` maps to `Stream.fromResource(resource)(use)`, with a ZIO Blocks `Resource` in place of a cats-effect one.

### Custom Operators with `Pull`

#### You Wrote This

A `Pull` writes an operator that fs2 does not have, here "emit orders until `limit` copies have been sold". This is the old code, fs2 3.14.0 (not compiled here):

```scala

// Emit orders until `limit` copies have been sold
def untilCopies(limit: Int): Pipe[IO, OrderEvent, OrderEvent] = {
  def go(orders: Stream[IO, OrderEvent], sold: Int): Pull[IO, OrderEvent, Unit] =
    orders.pull.uncons1.flatMap {
      case Some((order, rest)) if sold < limit => Pull.output1(order) >> go(rest, sold + order.copies)
      case _                                   => Pull.done
    }
  orders => go(orders, 0).stream
}

val firstOrders: IO[List[Long]] =
  Stream
    .emits(orderLines)
    .map(parseOrder)
    .collect { case Right(order) => order }
    .through(untilCopies(5))
    .map(_.orderId)
    .compile
    .toList
```

#### You Write This

Most `Pull` loops carry a piece of state from one element to the next. `mapAccum` does that and stays a stream: it pairs each order with the copies sold before it, and `takeWhile` stops at the limit:

```scala
val parsedOrders: Stream[Nothing, OrderEvent] =
  Stream.fromIterable(orderLines).via(parseOrders)

def untilCopies(orders: Stream[Nothing, OrderEvent], limit: Int): Stream[Nothing, OrderEvent] =
  orders
    .mapAccum(0)((sold, order) => (sold + order.copies, (sold, order)))
    .takeWhile { case (soldBefore, _) => soldBefore < limit }
    .map { case (_, order) => order }
```

```scala
println(untilCopies(parsedOrders, 5).map(_.orderId).runCollect)
// Right(Chunk(1,2,3))
```

When the loop does not fit an operator, pull by hand. On the JVM, `start` opens the stream as a `Reader` inside a [`Scope`](../reference/resource-management/scope.md), and `read(sentinel)` returns the next element, or the sentinel you pass once the stream ends. `OrderEvent` is a reference type, so `null` makes a safe sentinel:

```scala

val firstOrderIds: List[Long] = Scope.global.scoped { scope =>

  val reader = parsedOrders.start(using scope)

  $(reader) { r =>
    var sold  = 0
    var ids   = List.empty[Long]
    var order = r.read(null)
    while (order != null && sold < 5) {
      ids = order.orderId :: ids
      sold += order.copies
      order = r.read(null)
    }
    ids.reverse
  }
}
```

```scala
println(firstOrderIds)
// List(1, 2, 3)
```

On Scala 2.13, pass the scope as `parsedOrders.start(scope)`. The reader closes with the scope. For a primitive stream, read each lane with its own method, as [Primitive Streams Signal the End Without Boxing](#primitive-streams-signal-the-end-without-boxing) describes; the [Reader reference](../reference/streams/primitives/reader.md) covers the primitive read methods.

### Batches

#### You Wrote This

`chunkN` groups elements into batches for a bulk write (fs2 3.14.0, not compiled here):

```scala

// Write the order log two lines at a time
val batches: IO[Unit] =
  Stream
    .emits(orderLines)
    .chunkN(2)
    .evalMap(batch => IO.println(s"writing ${batch.size} lines"))
    .compile
    .drain
```

#### You Write This

`chunkN(n)` becomes `grouped(n)` (or its alias `chunked(n)`), which emits a ZIO Blocks `Chunk`. The last batch may be smaller:

```scala
println(
  Stream
    .fromIterable(orderLines)
    .grouped(2)
    .runForeach(batch => println(s"writing ${batch.size} lines"))
)
// writing 2 lines
// writing 2 lines
// writing 2 lines
// Right(())
```

`chunks` has no counterpart, because a ZIO Blocks stream does not carry chunk boundaries. `groupWithin` has none either, because there are no timers.

### What Changes

- **No `F[_]`.** `Stream[IO, O]` becomes `Stream[E, A]`, and `IO(...)`, `IO.blocking(...)` and `evalMap` turn into plain calls in `map`. `Stream.eval(io)` becomes `Stream.attempt(f)` for code that can throw.
- **A typed error instead of `Throwable`.** fs2 fails the stream with a `Throwable`. ZIO Blocks has a typed `E` returned as `Left`, and treats thrown exceptions as defects. `handleErrorWith` maps to `catchAll` for typed errors and to `catchDefect` for exceptions.
- **`compile.*` becomes a terminal.** `compile.toList` becomes `runCollect` (a `Chunk`, inside `Either`), `compile.fold` becomes `runFold`, `compile.drain` becomes `runDrain`, and `compile.count` becomes `count`.
- **`parJoin` becomes `flatMapPar` or `Stream.mergeAll`.** `stream.map(f).parJoin(n)` becomes `stream.flatMapPar(n)(f)`, and a stream of streams merges with `Stream.mergeAll(n)(streams)`.
- **No `Pull`, `Topic` or `Signal`.** Use `mapAccum`, `scan` or `start` for custom loops, and keep fs2 for pub-sub and signals.

## Coming from Ox

Ox and ZIO Blocks are both direct style: a `Flow` and a `Stream` are lazy descriptions, and a `run*` call runs them on the calling thread. What differs is the error model, the concurrency model and the platforms.

### You Wrote This

The order log arrives from a paged API, pushed into the flow with `Flow.usingEmit`, and priced with `mapPar`. This is the old code, Ox 1.1.0 (not compiled here):

```scala

// A paged API: page 0, 1, 2, ... until a page comes back empty
def fetchPage(page: Int): List[String] =
  orderLines.grouped(2).toList.lift(page).getOrElse(Nil)

val pagedOrders: Flow[String] = Flow.usingEmit { emit =>
  var page  = 0
  var lines = fetchPage(page)
  while (lines.nonEmpty) {
    lines.foreach(emit(_))
    page += 1
    lines = fetchPage(page)
  }
}

// mapPar keeps input order: results come back sorted by orderId
val pricedOrders: List[(Long, BigDecimal)] =
  pagedOrders
    .map(parseOrder)
    .collect { case Right(order) => order }
    .mapPar(4)(order => order.orderId -> priceOf(order.title) * order.copies)
    .runToList()
```

### You Write This

A `usingEmit` loop that walks pages is an unfold: `Stream.unfold` produces one page per step, and `flatMap` flattens each page into lines. `runToList()` becomes `runCollect`:

```scala
def fetchPage(page: Int): List[String] =
  orderLines.grouped(2).toList.lift(page).getOrElse(Nil)

val pagedOrders: Stream[Nothing, String] =
  Stream
    .unfold(0)(page => Option(fetchPage(page)).filter(_.nonEmpty).map(lines => (lines, page + 1)))
    .flatMap(lines => Stream.fromIterable(lines))
```

`mapPar` emits results in completion order, so sort them if order matters:

```scala
val pricedOrders: Either[Nothing, Chunk[(Long, BigDecimal)]] =
  pagedOrders
    .map(parseOrder)
    .collect { case Right(order) => order }
    .mapPar(4)(order => order.orderId -> priceOf(order.title) * order.copies)
    .runCollect
    .map(_.sortBy(_._1))
```

```scala
println(pricedOrders)
// Right(Chunk((1,99.98),(2,59.95),(3,134.97),(4,49.99),(5,239.80)))
```

A `usingEmit` loop over an iterator becomes `Stream.fromIterator`. For cursor-based pages, typed errors and `unfoldAsync`, see [How to Turn a Paginated API into a Stream](./turn-a-paginated-api-into-a-stream.md). A loop that does not fit either shape can implement a `Reader` and pass it to `Stream.fromReader`; see the [Reader reference](../reference/streams/primitives/reader.md). A `usingEmit` with `try`/`finally` around a resource becomes `Stream.fromAcquireRelease`, as in [the ZIO Streams section](#resources-and-parallel-work).

### Errors

#### You Wrote This

A `Flow` has no error type. A failure is an exception, and the `run*` method rethrows it (Ox 1.1.0, not compiled here):

```scala

// An exception thrown inside the flow is rethrown by the run method
def strict: Map[String, Int] =
  Flow
    .fromIterable(orderLines)
    .map(line => parseOrder(line).fold(msg => throw new IllegalArgumentException(msg), identity))
    .runFold(Map.empty[String, Int])(addOrder)
```

#### You Write This

ZIO Blocks keeps the same behavior for exceptions: a throw inside `map` is a defect, and a blocking terminal rethrows it. The better translation is a typed error, so the failure arrives as a value. `orders`, from the [ZIO Streams section](#typed-errors), fails with the parse error:

```scala
println(orders.runFold(Map.empty[String, Int])(addOrder))
// Left(malformed order: not-an-order)
```

`Stream.attempt(f)` turns code that throws into a `Stream[Throwable, A]`, the counterpart of Ox's `either.catching`:

```scala
println(Stream.attempt("not-a-number".toInt).runCollect)
// Left(java.lang.NumberFormatException: For input string: "not-a-number")
```

Ox's `recover` and `recoverWith` map to `catchDefect` for exceptions. Ox's `orElse` is a different operator: it substitutes a flow only when the flow is empty. ZIO Blocks' `orElse` (alias `||`) falls back on the first typed error and does not handle defects.

### Merging and Structured Concurrency

#### You Wrote This

`merge` runs two flows at once and interleaves their elements (Ox 1.1.0, not compiled here):

```scala

val onlineOrders: Flow[String]  = Flow.fromIterable(orderLines.take(3))
val inStoreOrders: Flow[String] = Flow.fromIterable(orderLines.drop(3))

// merge runs both flows concurrently and interleaves their elements
val allCopies: Map[String, Int] =
  onlineOrders
    .merge(inStoreOrders)
    .map(parseOrder)
    .collect { case Right(order) => order }
    .runFold(Map.empty[String, Int])(addOrder)
```

#### You Write This

`a.merge(b)` becomes `Stream.mergeAll(2)(Stream(a, b))`, which pulls from up to two streams at once and emits in arrival order:

```scala
val onlineOrders: Stream[Nothing, String]  = Stream.fromIterable(orderLines.take(3))
val inStoreOrders: Stream[Nothing, String] = Stream.fromIterable(orderLines.drop(3))

val allCopies: Either[Nothing, Map[String, Int]] =
  Stream
    .mergeAll(2)(Stream(onlineOrders, inStoreOrders))
    .map(parseOrder)
    .collect { case Right(order) => order }
    .runFold(Map.empty[String, Int])(addOrder)
```

```scala
println(allCopies.map(_.toList.sortBy(_._1)))
// Right(List((Functional Programming in Scala,3), (Programming in Scala,5), (Zionomicon,3)))
```

Ox builds its operators on structured concurrency: `mapPar`, `merge` and `flattenPar` open a scope when the flow runs, and a failing branch cancels the others before the exception reaches the caller. Ox also gives you `supervised` and `fork` to run your own concurrent code next to a flow. ZIO Blocks has no user-facing concurrency scope and no `fork`. Its concurrency lives inside `mapPar`, `mapParAsync`, `flatMapPar` and `Stream.mergeAll`, which run workers on virtual threads on the JVM. When one worker fails, the operator stops the others and closes upstream before the terminal returns. A ZIO Blocks `Scope` manages resources, not threads. Keep Ox's `supervised` for code that forks work outside a stream.

### What Changes

- **Typed errors.** An Ox failure is an exception. A ZIO Blocks failure can be a typed `E` returned as `Left`, or a defect that the terminal rethrows.
- **`mapPar` is unordered.** Ox's `mapPar` keeps input order, and its `mapParUnordered` does not. ZIO Blocks' `mapPar` behaves like `mapParUnordered`. See [`mapPar` emits in completion order](#mappar-emits-in-completion-order).
- **`Chunk`, not `Seq`.** `grouped(n)` emits a `Chunk`, and `runCollect` returns one.
- **`flattenPar` becomes `flatMapPar` or `Stream.mergeAll`.**
- **More platforms.** Ox 1.1.0 needs Scala 3.9 and JDK 21. ZIO Blocks Streams also runs on Scala 2.13, Scala 3.3 and Scala.js; only the JVM needs JDK 21.
- **No time-based operators.** Ox's `throttle`, `debounce`, `groupedWithin`, `Flow.tick` and `retry` have no counterpart.

## Traps When Migrating

Each trap below is code that looks like a direct translation and behaves differently. The first two are about ordering and resources. The next three are known library bugs, listed in the project's issue log, that affect the blocking terminals; the asynchronous terminals give the right answer in each case.

### `mapPar` Emits in Completion Order

fs2's `parEvalMap`, ZIO's `mapZIOPar` and Ox's `mapPar` all keep input order. ZIO Blocks' `mapPar`, `mapParAsync`, `flatMapPar` and `Stream.mergeAll` emit each result as soon as it is ready. A fold that adds numbers, as in this guide, does not care. Code that writes results in order, or that zips them with the input, does. Carry a key through the parallel step and sort afterwards, as in [the Ox section](#coming-from-ox), or keep the order-sensitive part sequential with `map`.

### Recovery Inside `flatMap`

A blocking terminal throws the internal `Stream$AsyncBoundaryRequired$` when `catchAll` or `||` sits on the inner stream of a `flatMap` and a stateful operator (`grouped`, `chunked`, `sliding`, `scan` or `mapAccum`) follows. This is the shape a ZIO `mapZIO(...).catchAll` often turns into:

```scala
// Throws Stream$AsyncBoundaryRequired$ from runCollect
Stream
  .fromIterable(orderLines)
  .flatMap(line => orderOrFailure(line).catchAll(_ => Stream.empty))
  .grouped(2)
  .runCollect
```

Recover on the outer stream instead, as in [Typed Errors](#typed-errors); that stops at the first malformed line. To skip bad lines and keep going, filter the `Either` values before they become stream errors:

```scala
println(
  Stream
    .fromIterable(orderLines)
    .map(parseOrder)
    .collect { case Right(order) => order }
    .grouped(2)
    .map(_.size)
    .runCollect
)
// Right(Chunk(2,2,1))
```

The asynchronous terminals, such as `runCollectAsync`, handle the inner `catchAll` correctly.

### A Primitive `map` after `flatMap`

After a `flatMap` over reference elements, a `map` to a primitive (`Int`, `Long`, `Double`, `Boolean`, `Char`) followed by `grouped`, `chunked`, `sliding`, `take`, `takeWhile`, `scan` or `mapAccum` throws a `ClassCastException` from blocking terminals. `pagedOrders` from the Ox section is built with `flatMap`, so the obvious "copies per batch" code crashes:

```scala
// Throws java.lang.ClassCastException from runCollect
pagedOrders
  .map(parseOrder)
  .collect { case Right(order) => order }
  .map(_.copies)
  .grouped(2)
  .runCollect
```

Keep the elements as objects until after the stateful operator, and map to the primitive inside each batch. [How to Stream Large Query Results](./stream-large-query-results.md) gives the same advice for database rows:

```scala
println(
  pagedOrders
    .map(parseOrder)
    .collect { case Right(order) => order }
    .grouped(2)
    .map(_.map(_.copies))
    .runCollect
)
// Right(Chunk(Chunk(2,1),Chunk(3,1),Chunk(4)))
```

A fold such as `runFold(0)(_ + _)` after the primitive `map` is not affected. The asynchronous terminals give the right answer for the original pipeline.

### Blocking Inside an `App` Object

A blocking terminal can deadlock inside the body of a Scala 3 `object Main extends App` once `catchAll` is in the pipeline. The object's body runs inside its class initializer. A worker thread that touches the object waits for the initializer to finish, while the initializer waits for the worker. Put the program in a `main` method:

```scala
object CopiesReport {
  def main(args: Array[String]): Unit =
    println(orders.catchAll(_ => Stream.empty).runFold(Map.empty[String, Int])(addOrder))
}
```

### Managed Handles Are Opened Once

`Stream.fromInputStream(in)` and `NioStreams.fromChannel(channel)` take a handle that is already open. They close it when the first run ends, so the stream leaks the handle if it never runs, and a second run fails:

```scala
val bytesOnce: Stream[java.io.IOException, Byte] =
  Stream.fromInputStream(Files.newInputStream(ordersPath))
```

The first run reads the file and closes it. The second run reads a closed handle:

```scala
println(bytesOnce.count)
// Right(129)
println(bytesOnce.count)
// Left(java.nio.channels.ClosedChannelException)
```

Open the handle in `Stream.fromAcquireRelease` and read it with the unmanaged constructor, so that each run opens a fresh handle and `fromAcquireRelease` alone closes it:

```scala
val bytesPerRun: Stream[java.io.IOException, Byte] =
  Stream.fromAcquireRelease(Files.newInputStream(ordersPath))(in => Stream.fromInputStreamUnmanaged(in))
```

```scala
println(bytesPerRun.count)
// Right(129)
println(bytesPerRun.count)
// Right(129)
```

The same pattern works for a channel with `NioStreams.fromChannelUnmanaged`; [How to Stream Files and Sockets Without Leaking Resources](./stream-files-and-sockets.md) covers files, channels and sockets in depth. In ZIO Streams, `ZStream.fromInputStream` never closes the stream at all.

### Primitive Streams Signal the End Without Boxing

A stream of `Int`, `Long`, `Double`, `Float`, `Byte`, `Short`, `Char` or `Boolean` runs without boxing, so the end of the stream cannot be an `Option` or a boxed marker. Each lane signals it in a way no real element can match: `Byte` returns `-1` from `Reader#readByte` (bytes come back as `0` to `255`), `Boolean` and `Char` return a negative `Int`, `Short`, `Int` and `Float` return a sentinel in a wider carrier type (`Int.MinValue`, `Long.MinValue`, `Double.MaxValue`), and `Long` and `Double` use no sentinel at all: a one-element `Reader#readLongs` or `Reader#readDoubles` returns a negative count at the end. Every lane is lossless, so `Long.MaxValue` or `Double.MaxValue` in your data is an ordinary element.

None of this shows up in user code unless you pull by hand with `Stream#start`. Then read each lane with its own method, and on a `Long` or `Double` reader loop on `Reader#readLongs(scratch, 0, 1)` or `Reader#readDoubles(scratch, 0, 1)` rather than `Reader#readLong(sentinel)` or `Reader#readDouble(sentinel)`, since any sentinel you pick there is also a value the stream may carry. [EOF Signalling Per Lane](../reference/streams/execution-and-compatibility/zero-boxing.md#eof-signalling-per-lane) lists every lane's signal, and the [Reader reference](../reference/streams/primitives/reader.md) describes the primitive read methods.

## What You Gain and What Is Different

You gain:

- **No effect system.** No `IO`, `ZIO`, `IOApp` or runtime. Stages are plain functions and terminals return values, so a stream fits into synchronous code as easily as a collection does.
- **Typed errors without an effect type.** `Stream[E, A]` carries `E`, and every terminal returns `Either[E, Z]`.
- **Primitive lanes with zero boxing.** A `Stream[Nothing, Int]` keeps its elements as `int`s from source to sink. See [zero-boxing](../reference/streams/execution-and-compatibility/zero-boxing.md).
- **Scope integration.** `Stream.fromResource` runs each pass in its own ZIO Blocks `Scope`, and `start` hands out a reader that the scope closes.
- **Wide platform support.** Scala 2.13, Scala 3 and Scala.js, from the same source.

What is different:

- **Blocking and asynchronous terminals.** On the JVM, `runCollect`, `runFold`, `runDrain`, `count`, `run(sink)` and `start` block the calling thread. Every platform has the asynchronous versions (`runCollectAsync`, `runFoldAsync`, `runAsync(sink)`, `countAsync` and so on), which return `Async[Either[E, Z]]`.
- **Scala.js has only the asynchronous terminals.** JavaScript cannot block, so there is no `Stream#runCollect` or `Stream#start` there, and no work in `Stream#mapPar`, `Stream#flatMapPar` or `Stream.mergeAll` runs in parallel: only asynchronous waits overlap.
- **No time, schedules or environment.** No `Schedule`, timers, `groupedWithin`, throttling or `R` channel.
- **Parallel operators are unordered.** `mapPar` is `mapParUnordered` under another name.
- **No `Pull`, `ZChannel` or fibers.** Custom operators use `mapAccum`, `scan` or `start`, and concurrency lives inside the parallel operators.

On Scala.js, run the stream with an asynchronous terminal and convert the `Async` result to a `Future` with `toFuture` (or to a `js.Promise` with `toJsPromise`). `totals` is a `def`, so each call builds a new `Async`. This code compiles on both platforms:

```scala
def totals: Async[Either[String, Map[String, Int]]] =
  orders.runAsync(Sink.foldLeft[OrderEvent, Map[String, Int]](Map.empty)(addOrder))

val totalsFuture: Future[Either[String, Map[String, Int]]] = totals.toFuture
```

On the JVM, `block` waits for the result instead:

```scala
println(totals.block)
// Left(malformed order: not-an-order)
```

:::warning
Consume each `Async` value once. Calling `toFuture` and `block` on the same value drives it from two places, which is undefined behaviour: one of the two can wait forever. Build a new `Async` for each consumer, as the `def` above does, or call `start` once and share the `Async.Running` it returns. See [Concurrent Fan-Out via Running](../reference/async.md#concurrent-fan-out-via-running).
:::

## Migration Checklist

1. Add `zio-blocks-streams` (Scala 2.13, 3.3 to 3.8) or `zio-blocks-next-streams` plus `-experimental` (Scala 3.9+), and `import zio.blocks.streams._`. Run on JDK 21 or later.
2. Replace `Stream[IO, A]`, `ZStream[R, E, A]` or `Flow[A]` with `Stream[E, A]`. Use `Nothing` for `E` when the stream cannot fail.
3. Replace `evalMap(a => IO(f(a)))` and `mapZIO(a => ZIO.attempt(f(a)))` with `map(f)`. Use `Stream.attempt` where the code can throw and you want a typed error.
4. Replace `parEvalMap`, `mapZIOPar` and Ox `mapPar` with `mapPar` (or `mapParAsync` for `Async` code), and sort the output if order matters.
5. Replace `Stream.bracket`, `ZStream.acquireReleaseWith` and `usingEmit` with `try`/`finally` by `Stream.fromAcquireRelease`; replace `Stream.resource` and `ZStream.scoped` with `Stream.fromResource`.
6. Wrap `Stream.fromInputStream` and `NioStreams.fromChannel` in `Stream.fromAcquireRelease` with the `Unmanaged` constructor if the stream may run more than once.
7. Replace `Pipe` and `ZPipeline` with `Pipeline`, `through` with `via`, and `chunkN` with `grouped`.
8. Replace `compile.toList`, `runToList()` and friends with `runCollect`, `runFold`, `runDrain` or `count`, and handle the `Either` they return.
9. Put `catchAll` on the outer stream, not inside `flatMap`, when a stateful operator follows.
10. After a `flatMap`, keep elements as objects through `grouped`, `take`, `scan` and similar operators; map to primitives inside the batches.
11. Run blocking terminals from a `main` method, not from the body of an `object ... extends App`.
12. On Scala.js, use the `*Async` terminals and `toFuture`.
13. Keep fs2, ZIO Streams or Ox for the parts that need schedules, timers, the ZIO environment, fibers or pub-sub.

## Putting It Together

This is the bookstore pipeline fully migrated. `Stream.fromAcquireRelease` opens the order log once per run, so the same stream runs twice: once to report the malformed lines, and once to price the orders in parallel with `mapPar` and fold the revenue per title with `runFold`. The program runs from a `main` method:

```scala title="streams-examples/src/main/scala/streammigration/BookstoreRevenueMigrated.scala"
package streammigration

/**
 * The bookstore order pipeline from the "Coming from fs2, ZIO Streams or Ox"
 * guide, fully migrated to ZIO Blocks Streams. The sales data is fictional.
 *
 * The program runs from `main`, not from the body of an `object ... extends
 * App`, so blocking terminals never run inside a class initializer.
 *
 * Run it from the `streams-examples` directory:
 * {{{
 * sbt "runMain streammigration.BookstoreRevenueMigrated"
 * }}}
 */
object BookstoreRevenueMigrated {

  final case class OrderEvent(orderId: Long, title: String, copies: Int)

  // Fictional order log: one CSV line per order, and one malformed line
  private val orderLines: List[String] = List(
    "1,Zionomicon,2",
    "2,Programming in Scala,1",
    "3,Functional Programming in Scala,3",
    "not-an-order",
    "4,Zionomicon,1",
    "5,Programming in Scala,4"
  )

  private val listPrices: Map[String, BigDecimal] = Map(
    "Zionomicon"                      -> BigDecimal("49.99"),
    "Programming in Scala"            -> BigDecimal("59.95"),
    "Functional Programming in Scala" -> BigDecimal("44.99")
  )

  def parseOrder(line: String): Either[String, OrderEvent] =
    line.split(',') match {
      case Array(id, title, copies) =>
        (id.toLongOption, copies.toIntOption) match {
          case (Some(i), Some(c)) => Right(OrderEvent(i, title, c))
          case _                  => Left(s"malformed order: $line")
        }
      case _ => Left(s"malformed order: $line")
    }

  // A slow lookup, standing in for a call to a pricing service
  def priceOf(title: String): BigDecimal = {
    Thread.sleep(20)
    listPrices(title)
  }

  def addRevenue(totals: Map[String, BigDecimal], sale: (String, BigDecimal)): Map[String, BigDecimal] =
    totals.updated(sale._1, totals.getOrElse(sale._1, BigDecimal(0)) + sale._2)

  // Opens the log on every run and closes it when the run ends
  def orderLog(path: Path): Stream[Nothing, String] =
    Stream.fromAcquireRelease(
      Files.newBufferedReader(path),
      (reader: BufferedReader) => { reader.close(); println("order log closed") }
    )(reader => Stream.fromIterator(reader.lines().iterator().asScala))

  def main(args: Array[String]): Unit = {
    val path = Files.write(Files.createTempFile("orders", ".csv"), orderLines.asJava)
    try {
      val parsed: Stream[Nothing, Either[String, OrderEvent]] = orderLog(path).map(parseOrder)

      // First run: report the malformed lines
      parsed.collect { case Left(error) => error }
        .runForeach(error => println(s"skipped: $error"))

      // Second run: the log is opened again, and orders are priced in parallel.
      // mapPar emits in completion order; the fold does not depend on it.
      val revenue: Either[Nothing, Map[String, BigDecimal]] =
        parsed.collect { case Right(order) => order }
          .mapPar(4)(order => order.title -> priceOf(order.title) * order.copies)
          .runFold(Map.empty[String, BigDecimal])(addRevenue)

      revenue.foreach { totals =>
        totals.toList.sortBy(_._1).foreach { case (title, total) => println(s"$title: $$$total") }
      }
    } finally {
      Files.deleteIfExists(path)
      ()
    }
  }
}
```

## Running the Examples

The complete example is a runnable file in the `streams-examples` project. Clone the repository and change into the project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/streams-examples
```

<details>
  <summary>streams-examples/src/main/scala/streammigration/BookstoreRevenueMigrated.scala</summary>

```scala title="streams-examples/src/main/scala/streammigration/BookstoreRevenueMigrated.scala" showLineNumbers
package streammigration

/**
 * The bookstore order pipeline from the "Coming from fs2, ZIO Streams or Ox"
 * guide, fully migrated to ZIO Blocks Streams. The sales data is fictional.
 *
 * The program runs from `main`, not from the body of an `object ... extends
 * App`, so blocking terminals never run inside a class initializer.
 *
 * Run it from the `streams-examples` directory:
 * {{{
 * sbt "runMain streammigration.BookstoreRevenueMigrated"
 * }}}
 */
object BookstoreRevenueMigrated {

  final case class OrderEvent(orderId: Long, title: String, copies: Int)

  // Fictional order log: one CSV line per order, and one malformed line
  private val orderLines: List[String] = List(
    "1,Zionomicon,2",
    "2,Programming in Scala,1",
    "3,Functional Programming in Scala,3",
    "not-an-order",
    "4,Zionomicon,1",
    "5,Programming in Scala,4"
  )

  private val listPrices: Map[String, BigDecimal] = Map(
    "Zionomicon"                      -> BigDecimal("49.99"),
    "Programming in Scala"            -> BigDecimal("59.95"),
    "Functional Programming in Scala" -> BigDecimal("44.99")
  )

  def parseOrder(line: String): Either[String, OrderEvent] =
    line.split(',') match {
      case Array(id, title, copies) =>
        (id.toLongOption, copies.toIntOption) match {
          case (Some(i), Some(c)) => Right(OrderEvent(i, title, c))
          case _                  => Left(s"malformed order: $line")
        }
      case _ => Left(s"malformed order: $line")
    }

  // A slow lookup, standing in for a call to a pricing service
  def priceOf(title: String): BigDecimal = {
    Thread.sleep(20)
    listPrices(title)
  }

  def addRevenue(totals: Map[String, BigDecimal], sale: (String, BigDecimal)): Map[String, BigDecimal] =
    totals.updated(sale._1, totals.getOrElse(sale._1, BigDecimal(0)) + sale._2)

  // Opens the log on every run and closes it when the run ends
  def orderLog(path: Path): Stream[Nothing, String] =
    Stream.fromAcquireRelease(
      Files.newBufferedReader(path),
      (reader: BufferedReader) => { reader.close(); println("order log closed") }
    )(reader => Stream.fromIterator(reader.lines().iterator().asScala))

  def main(args: Array[String]): Unit = {
    val path = Files.write(Files.createTempFile("orders", ".csv"), orderLines.asJava)
    try {
      val parsed: Stream[Nothing, Either[String, OrderEvent]] = orderLog(path).map(parseOrder)

      // First run: report the malformed lines
      parsed.collect { case Left(error) => error }
        .runForeach(error => println(s"skipped: $error"))

      // Second run: the log is opened again, and orders are priced in parallel.
      // mapPar emits in completion order; the fold does not depend on it.
      val revenue: Either[Nothing, Map[String, BigDecimal]] =
        parsed.collect { case Right(order) => order }
          .mapPar(4)(order => order.title -> priceOf(order.title) * order.copies)
          .runFold(Map.empty[String, BigDecimal])(addRevenue)

      revenue.foreach { totals =>
        totals.toList.sortBy(_._1).foreach { case (title, total) => println(s"$title: $$$total") }
      }
    } finally {
      Files.deleteIfExists(path)
      ()
    }
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain streammigration.BookstoreRevenueMigrated"
```

To check that the example compiles without running it:

```bash
sbt compile
```

## See Also

- [Streams](../reference/streams/index.md): the module overview, the mental model and the comparison table.
- [Stream](../reference/streams/core/stream.md), [Sink](../reference/streams/core/sink.md) and [Pipeline](../reference/streams/core/pipeline.md): the full API.
- [Reader](../reference/streams/primitives/reader.md): manual pulls with `start` and the sentinel protocol.
- [Async execution](../reference/streams/execution-and-compatibility/async-execution.md) and [platform differences](../reference/streams/execution-and-compatibility/platform-differences.md): the asynchronous terminals and what changes on Scala.js.
- [Coming from cats-effect Resource, ZIO Scope/ZLayer, macwire or Using](./resource-and-di-migration.md): the resource side of the same migration.
- [How to Stream Large Query Results](./stream-large-query-results.md) and [How to Stream Files and Sockets Without Leaking Resources](./stream-files-and-sockets.md): streams over database results, files and sockets.
- The tutorial "Build your first streaming pipeline" is in progress.
- [How to Use ZIO Blocks in a ZIO or cats-effect Application](./use-zio-blocks-in-zio-or-cats-effect.md): calling ZIO Blocks streams from a ZIO or cats-effect application, with cancellation.

---

## Telemetry: Architecture, Patterns, and Real-World Usage

`zio-blocks-telemetry` is an effect-free, zero-allocation observability library that gives you structured logging, distributed tracing, and metrics without pulling in the OpenTelemetry SDK or any effect system. This guide explains why it works the way it does, shows what happens inside each operation, and walks through patterns for production systems.

New to telemetry? Start with the tutorial [Instrument a Program with Logs, Traces and Metrics](./instrument-a-program-with-logs-traces-and-metrics.md), which builds up a small batch job one pillar at a time, then come back here for the architecture and trade-offs.

If you're looking for API signatures and quick copy-paste snippets, the [Telemetry Reference](../reference/telemetry/index.md) has those. This guide focuses on the "how" and "why": architecture, design trade-offs, and the kind of understanding that helps when things go wrong at 3 AM.

## Installation

```scala
// Core: logging, tracing, metrics
libraryDependencies += "dev.zio" %% "zio-blocks-telemetry" % "0.0.56"

// OTLP JSON export over HTTP (JVM only)
libraryDependencies += "dev.zio" %% "zio-blocks-telemetry-otel" % "0.0.56"
```

On the JVM, the module needs JDK 21 or newer. The core module also cross-builds for Scala.js:

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-telemetry" % "0.0.56"
```

---

## Part 1: Architecture and Design Philosophy

This part explains why the library is effect-free and how a signal travels from a call site to an exporter.

### Why effect-free?

Most Scala observability libraries wrap logging or tracing calls in an effect type. ZIO Logging needs `ZIO[R, E, Unit]`. Odin uses `F[Unit]`. Even scribe's direct API leans on Cats effect in production setups.

Effect-wrapping is a real cost. You can't call `log.info(...)` from a callback, a `Future`, a background thread, or a Java library without threading the effect runtime through. You can't log in a constructor, an `AutoCloseable.close()`, or any other place where effects are inconvenient.

`zio-blocks-telemetry` takes a different position: logging and metrics are inherently synchronous, low-level operations. They should work anywhere, with no ceremony. You import the package, call `log.info(...)`, and it works. The library is responsible for making that cheap, not for constraining where you can use it.

This is the same approach `logback` and `java.util.logging` take, but with structured attributes, zero boxing on the fast path, and a modern Scala macro interface.

### The provider-instrument-processor pipeline

Every signal (log, span, metric) flows through the same three-stage pipeline:

```
Call site
    |
    v
[Provider]  -- creates and configures instruments --
    |                                                  LoggerProvider
    |                                                  TracerProvider
    |                                                  MeterProvider
    v
[Instrument]  -- the thing you call at call sites --
    |                                                  Logger  (wraps LogRecord)
    |                                                  Tracer  (wraps Span)
    |                                                  Counter / Histogram / Gauge
    v
[Processor]  -- receives the completed signal --
    |                                                  LogRecordProcessor
    |                                                  SpanProcessor
    |                                                  MetricReader (pull-based)
    v
[Exporter]  -- sends data out of process --
    |                                                  OtlpJsonLogExporter
    |                                                  OtlpJsonTraceExporter
    |                                                  OtlpJsonMetricExporter
    v
OTLP collector / file / console
```

Processors are composable: you can chain several. A `LoggerProvider` built with `.addLogRecordProcessor(console).addLogRecordProcessor(otelExporter)` will call both processors for every log record.

The `log` object and the `Global*` singletons sit in front of this pipeline. They hold a reference to the current provider (an `AtomicReference`) and forward calls through it. You can swap the provider at runtime via `install()` without touching any call sites.

### Zero-allocation design

Two separate things must both be cheap: the fast path when a level is disabled, and the slow path when it's enabled.

**Disabled level fast path.** The first check in every `log.*` call is:

```scala
if (severity.number >= GlobalLogState.globalMinLevel) { ... }
```

`globalMinLevel` is a `@volatile Int` held inside the library-private `GlobalLogState` (you set it through `log.install`, `log.setMinSeverity`, and `log.withMinSeverity` rather than reading it directly). Reading it costs roughly one memory fence. If the level is disabled, the compiler has already eliminated all argument evaluation (the arguments are passed by-name or inlined via macro). No string is built, no object is allocated.

**Enabled level slow path.** When a level is enabled, attribute pairs need to be assembled into a `LogRecord`. The library uses a pooled `AttributeBuilder` (one per thread, reused), backed by parallel primitive arrays. Primitive values (Long, Double, Boolean) are stored as unboxed longs using type discriminator bytes. A `(String, Long)` attribute doesn't box the Long.

For metrics, `Counter` uses a `ConcurrentHashMap[Attributes, LongAdder]`. `LongAdder` is Java's striped counter, designed for high-concurrency increment without contention. `Gauge` uses `AtomicLong` storing the double bits directly. `Histogram` uses `ReentrantLock` per attribute set because it needs to update count, sum, min, max, and bucket counts atomically.

**What actually allocates on the slow path:**
- `Attributes.empty` is a singleton: no allocation
- Attribute builders are pooled: no allocation per record
- The `LogRecord` itself is allocated (unavoidably), but only when the level is enabled
- `SpanContext` stores trace ID as two `Long` fields, not a `UUID` object

### Global singletons vs explicit provider wiring

The three global entry points (`log`, `trace`, `metric`) work without any setup. On first use, each creates a sensible default:

- `log` creates a `ConsoleLogRecordProcessor` that writes to stdout
- `trace` creates an `InMemorySpanProcessor` that stores spans in a ring buffer (useful for testing)
- `metric` creates a basic `MeterProvider` with a `MetricReader`

For tests and exploratory code, the defaults are exactly right. For production, call `install()` once at startup to replace each default with your configured provider.

The global singletons are intentionally not safe to use across threads during `install()`. Call `install()` before starting any request processing. After installation, the `AtomicReference` read on every call is safe and cheap.

---

## Part 2: Logging Deep Dive

This part covers the logging API, from the first call to multiple outputs and testing.

### Getting started

The simplest possible start:

```scala

log.info("server started")
```

No imports beyond the package. No provider setup. Output goes to stdout as human-readable text. This is intentional: the zero-configuration default is production-safe for development.

Moving to a more realistic setup:

```scala

// Structured attributes alongside the message
log.info("user signed in", "user_id" -> "u-7890", "method" -> "oauth2")

// Numbers are unboxed on the fast path
log.info("payment processed", "amount_cents" -> 4999L, "currency" -> "EUR")

// Exceptions get type, message, and stacktrace automatically.
// Bind to a Throwable-typed val; an inline `new SomeException(...)` does not resolve.
val declined: Throwable = new IllegalStateException("card declined")
log.error("payment failed", declined)

// Mix freely
val timeout: Throwable = new RuntimeException("timeout")
log.warn(
  "upstream degraded",
  "service" -> "inventory",
  "latency_ms" -> 850L,
  timeout
)
```

Adding a file output at startup. The module ships stdout, stderr, and no-op writers; a file destination is a few lines of your own:

```scala

final class AppFileWriter(path: String) extends LogWriter {
  private val out = new PrintWriter(path)
  def write(content: CharSequence): Unit = out.println(content)
  override def flush(): Unit             = out.flush()
  override def close(): Unit             = out.close()
}

// Human-readable text to one file, OTLP JSON to another
log.writer(TextLogFormatter, new AppFileWriter("logs/app.log"))
log.writer(JsonLogFormatter, new AppFileWriter("logs/app.jsonl"))
```

Both outputs are active simultaneously, alongside the default console output — `writer()` is additive.

### Macro mechanics explained

Every `log.*` method is a Scala inline macro. The macro generates something like this for `log.info("msg", "key" -> 42L)`:

```
Conceptual expansion of:
  log.info("request received", "user_id" -> userId, "attempt" -> 3)

Expands to (pseudocode):
  if (Severity.Info.number >= GlobalLogState.globalMinLevel) {
    val state = log.getState()
    if (state != null && Severity.Info.number >= state.effectiveLevel(<current class name>)) {
      val now = EpochClock.epochNanos()
      val builder = AttributeBuilderPool.get()
        .put("code.filepath", "<source file>")
        .put("code.namespace", "<current package>")
        .put("code.function", "<current method>")
        .put("code.lineno", <line number>.toLong)
      // for each enrichment argument, the macro calls the right builder method:
      builder.put("user_id", userId)      // resolved as (String, String)
      builder.put("attempt", 3.toLong)    // resolved as (String, Int) -> toLong
      // reads annotations from scoped context
      val annotations = LogAnnotations.get()
      // reads span context from ContextStorage
      val spanCtx = state.logger.currentSpanContext()
      state.logger.emitRaw(now, Severity.Info, "INFO", "request received",
                           builder, traceIdHi, traceIdLo, spanId, traceFlags,
                           resource, scope, None)
    }
  }
```

The important things the macro does:
- Captures `code.filepath`, `code.namespace`, `code.function`, `code.lineno` at compile time
- Resolves each enrichment argument type at compile time and calls the specific builder method (no runtime dispatch on the fast path)
- For rate-limited variants (`infoEvery`, `warnAtMost`), hashes the source position into a `siteId` constant so each call site gets its own rate-limit slot

For `*Every(N, msg)`, the macro emits a guard against a shared global table:

```scala
// siteId is a compile-time constant: (sourcePath + ":" + line).hashCode
if (LogRateLimit.shouldLogEvery(siteId, N)) { log.emit(...) }

// LogRateLimit keeps one AtomicLongArray of 4096 counters:
//   counters.incrementAndGet(siteId & 4095) % N == 0
```

For `*AtMost(windowMs, msg)`, it's the same shape against a parallel timestamp array:

```scala
if (LogRateLimit.shouldLogAtMost(siteId, windowMs)) { log.emit(...) }

// timestamps.get(idx); if (now - last >= windowMs) timestamps.compareAndSet(idx, last, now)
```

Two consequences of the shared table: `*Every(N, …)` emits on the *Nth* call rather than the first (the counter is incremented before the modulo), and two call sites whose `siteId` collides modulo 4096 share one slot.

### Custom enrichment types

The `LogEnrichment[A]` typeclass lets the macro delegate argument resolution to your code.

Say you want to log `UUID` values directly:

```scala

// The implicit must be in scope at log call sites
implicit val uuidEnrichment: LogEnrichment[(String, UUID)] =
  new LogEnrichment[(String, UUID)] {
    def enrich(record: LogRecord, value: (String, UUID)): LogRecord =
      record.copy(
        attributes = record.attributes ++ Attributes.of(
          AttributeKey.string(value._1), value._2.toString
        )
      )
  }

// Now this compiles and works
val requestId = UUID.randomUUID()
log.info("request started", "request_id" -> requestId)
```

For `java.time.Instant`:

```scala

implicit val instantEnrichment: LogEnrichment[(String, Instant)] =
  new LogEnrichment[(String, Instant)] {
    def enrich(record: LogRecord, value: (String, Instant)): LogRecord =
      record.copy(
        attributes = record.attributes ++ Attributes.of(
          AttributeKey.string(value._1), value._2.toString
        )
      )
  }

log.info("event occurred", "timestamp" -> Instant.now())
```

**Performance note.** The built-in enrichments for `(String, String)`, `(String, Long)`, `(String, Int)`, `(String, Double)`, and `(String, Boolean)` go through the attribute builder's primitive path: no `AttributeValue` boxing. Custom enrichments through `LogEnrichment` go through `record.copy(attributes = ...)`, which does allocate an `Attributes` merge. For very hot paths, prefer the built-in types.

### Multiple outputs simultaneously

Every call to `log.writer(formatter, writer)` adds another output. The state is maintained inside the `log` object and composited into the active logger:

```scala

final class AppFileWriter(path: String) extends LogWriter {
  private val out = new PrintWriter(path)
  def write(content: CharSequence): Unit = out.println(content)
  override def flush(): Unit             = out.flush()
  override def close(): Unit             = out.close()
}

// Setup at startup; all three are active simultaneously
log.writer(TextLogFormatter, new AppFileWriter("logs/app.log"))
log.writer(JsonLogFormatter, new AppFileWriter("logs/app.jsonl"))
// The OTEL exporter is a LogRecordProcessor, added via LoggerProvider (see Part 5)
```

Internally, `log.writer(...)` creates a `FormattedLogRecordProcessor` (which pairs a formatter with a writer) and calls `rebuildState()`, which reconstructs the active `Logger` with the accumulated processor list. The original provider-based processors are preserved alongside the writer processors.

To remove all file writers:

```scala

log.clearWriters()
```

This shuts down each `FormattedLogRecordProcessor` cleanly and reverts to processor-only routing.

### Log output format comparison

The same call:

```scala

log.warn("slow query", "table" -> "orders", "duration_ms" -> 450L)
```

The text formatter writes one line per record:
```
2026-05-14T09:30:00.123Z WARN  [PaymentService.handleRequest:87] slow query {table="orders", duration_ms=450}
```

The JSON formatter writes the same record in the OTLP log data model:
```json
{"timeUnixNano":"1747216200123000000","severityNumber":13,"severityText":"WARN","body":{"stringValue":"slow query"},"attributes":[{"key":"code.filepath","value":{"stringValue":"src/main/scala/com/example/PaymentService.scala"}},{"key":"code.namespace","value":{"stringValue":"com.example.PaymentService"}},{"key":"code.function","value":{"stringValue":"handleRequest"}},{"key":"code.lineno","value":{"intValue":"87"}},{"key":"table","value":{"stringValue":"orders"}},{"key":"duration_ms","value":{"intValue":"450"}}]}
```

The JSON format follows the OTLP log data model directly, making it compatible with any OTLP-aware log processor (Grafana Loki, OpenTelemetry Collector, etc.).

The text formatter caches the timestamp prefix per second (the part before milliseconds) using two `@volatile` fields, so it avoids repeated date formatting work on high-throughput paths.

### Testing with telemetry

For unit tests, you usually want to assert that specific log calls happened. The cleanest approach is a custom `LogRecordProcessor`:

```scala

class CapturingLogProcessor extends LogRecordProcessor {
  val records = new CopyOnWriteArrayList[LogRecord]()

  def onEmit(record: LogRecord): Unit = records.add(record)
  def shutdown(): Unit = ()
  def forceFlush(): Unit = ()
}

// In your test setup:
val capturing = new CapturingLogProcessor()

val loggerProvider = LoggerProvider.builder
  .addLogRecordProcessor(capturing)
  .build()

log.install(loggerProvider.get("test"))

// Call the code under test
MyService.processPayment(...)

// Assert on the captured records
val warns = capturing.records.toArray.collect {
  case r: LogRecord if r.severity == Severity.Warn => r
}
assert(warns.exists(_.body.value.contains("payment")))
```

Don't clean up the global logger with `log.removeAll()`. It silences logging, and in version 0.0.56 a later `log.addProcessor` or `log.writer` does not bring it back, so every test after it captures nothing (a known bug in that version). Instead, install a fresh capturing logger with `log.install` at the start of each test, and run tests that share the global logger one at a time. Passing a `Logger` to the code under test avoids the shared state altogether. [How to Test Code That Logs, Traces and Records Metrics](./test-telemetry.md) shows both patterns.

---

## Part 3: Distributed Tracing Deep Dive

This part covers how spans are created, sampled, propagated and tested.

### Trace lifecycle

When you call `tracer.span("name") { span => ... }`, here's the complete sequence:

1. **Read parent context.** `contextStorage.get()` returns `Option[SpanContext]`. If a span is already active on this thread (set by a parent `span` call), it becomes the parent.

2. **Sampler decision.** `sampler.shouldSample(...)` is called with the parent context, new trace ID bits, span name, and kind. The result is one of `Drop`, `RecordOnly`, or `RecordAndSample`.

3. **Drop path.** If `Drop`, your block runs with `Span.NoOp`. All `setAttribute`, `addEvent`, `setStatus` calls are no-ops, and no processor is notified. A dropped span is still cheap but not free: a fresh `SpanId` and `SpanContext` are allocated and scoped so that nested spans and log records stay on the same trace.

4. **Record path.** A `RecordingSpan` is built with a new `SpanContext`. The span gets:
   - The parent's `traceIdHi`/`traceIdLo` if a valid parent exists, or a fresh random pair
   - A fresh random `spanId`
   - `traceFlags` set to `sampled` (for `RecordAndSample`) or `none` (for `RecordOnly`)

5. **Processors on start.** Each `SpanProcessor.onStart(span)` is called.

6. **Context storage scoping.** `contextStorage.scoped(Some(span.spanContext)) { f(span) }` runs your block. While your block is executing, any inner `span` calls will see this span as their parent.

7. **End and export.** When the block exits (normally or via exception), `span.end()` records the end timestamp. Then `SpanProcessor.onEnd(spanData)` is called on each processor. The OTLP exporter's `onEnd` enqueues the `SpanData` in the `BatchProcessor`.

```
tracer.span("checkout") { span =>
    |
    +-- sampler.shouldSample() -> RecordAndSample
    |
    +-- SpanProcessor.onStart(span)  [each processor]
    |
    +-- contextStorage.scoped(Some(span.spanContext)) {
    |       tracer.span("validate-cart") { ... }  // sees checkout as parent
    |       tracer.span("charge-card")  { ... }   // sees checkout as parent
    |   }
    |
    +-- span.end()
    |
    +-- SpanProcessor.onEnd(spanData)  [each processor]
         |
         +-- OtlpJsonTraceExporter.onEnd(spanData)
              |
              +-- BatchProcessor.enqueue(spanData)
                   |
                   (background flush every 5s, draining maxBatchSize per request)
                   |
                   +-- HTTP POST /v1/traces to OTLP collector
```

### Sampling strategies

**`AlwaysOnSampler`** returns `RecordAndSample` for every span. Use this in development or when you have low enough traffic that 100% sampling is affordable. It's the default.

**`AlwaysOffSampler`** returns `Drop` for every span. Useful when you want tracing code paths to exist but not actually produce data, for instance in a test that doesn't care about traces.

**`ParentBasedSampler(root)`** defers to the parent's sampling decision. If there's no parent, it falls back to `root`. If there is a parent:
- Parent sampled (`traceFlags.isSampled == true`) → `RecordAndSample`
- Parent not sampled → `Drop`

This is the right choice for most production services. The gateway or entry service decides sampling, and all downstream services follow automatically. Configure it with `AlwaysOnSampler` as the root if you want the gateway to sample everything, or with a `RateLimitedSampler` (if you implement one) for head-based sampling:

```scala

val tracerProvider = TracerProvider.builder
  .setSampler(ParentBasedSampler(root = AlwaysOnSampler))
  .addSpanProcessor(SpanProcessor.noop)  // your exporter here
  .build()

trace.install(tracerProvider)
```

### SpanContext internals

```scala
final case class SpanContext(
  traceIdHi: Long,    // high 64 bits of the 128-bit trace ID
  traceIdLo: Long,    // low  64 bits of the 128-bit trace ID
  spanId: SpanId,     // AnyVal wrapping a Long
  traceFlags: TraceFlags, // AnyVal wrapping a Byte
  traceState: String,
  isRemote: Boolean
)
```

The 128-bit trace ID is stored as two `Long` fields rather than a `UUID` object. A `UUID` would be 32 bytes of heap allocation per span, plus GC pressure. Two `Long` fields on a case class cost nothing beyond the case class itself, and they're JIT-friendly (no dereference).

A fresh trace ID is generated by `TraceId.random()`, which draws two `Long`s from `scala.util.Random`, retrying in the vanishingly unlikely case that both come out zero (an all-zero trace ID is invalid by the spec).

`SpanId` is an `AnyVal` wrapping a single `Long`, generated the same way and likewise retried if it comes out zero. At runtime on the JVM, `AnyVal`s are unboxed wherever possible, so a `SpanId` in a `SpanContext` field costs nothing extra.

`traceIdHex` is computed on demand by `TraceId.toHex(traceIdHi, traceIdLo)`, which formats the two longs into 32 hex characters. No caching: hex formatting is only needed when building headers or log output.

### Context propagation across process boundaries

Here's a complete round trip. The server receives an HTTP request with W3C `traceparent`, continues the trace, and the client injects context into an outbound call.

**HTTP server (receiving a trace):**

```scala

// Simulate incoming HTTP headers
val incomingHeaders = Map(
  "traceparent" -> "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
)

// Extract SpanContext from the traceparent header
val parentCtx: Option[SpanContext] =
  W3CTraceContextPropagator.extract(incomingHeaders, (m, k) => m.get(k))

// Wire the extracted context as the parent by scoping it in ContextStorage
// A provider's own contextStorage is internal, so create one and pass it in
val contextStorage = ContextStorage.create[Option[SpanContext]](None)

val tracerProvider = TracerProvider.builder
  .addSpanProcessor(SpanProcessor.noop)  // your exporter here
  .setContextStorage(contextStorage)
  .build()

trace.install(tracerProvider)

// Take the tracer after trace.install: it binds to the provider installed when it runs
val tracer = trace.get("api-service")

contextStorage.scoped(parentCtx) {
  tracer.span("handle-checkout", SpanKind.Server) { span =>
    span.setAttribute("http.method", "POST")
    span.setAttribute("http.route", "/checkout")
    // Your handler logic here
    ()  // your handler logic here
  }
}
```

**HTTP client (injecting trace into outbound call):**

```scala

val tracer = trace.get("api-service")

tracer.span("call-inventory", SpanKind.Client) { span =>
  // Inject the current span's context into outbound headers
  val headers = W3CTraceContextPropagator.inject(
    span.spanContext,
    Map.empty[String, String],
    (carrier, k, v) => carrier + (k -> v)
  )

  // headers now contains: "traceparent" -> "00-<traceId>-<spanId>-01"
  headers  // hand these to your HTTP client
}
```

The `traceparent` header encodes: `{version}-{traceId32hex}-{spanId16hex}-{flags2hex}`. The inventory service receives this, calls `extract`, gets back a `SpanContext` with `isRemote = true`, and starts its own child span under the same trace.

For Zipkin/B3 compatibility, swap `W3CTraceContextPropagator` for `B3Propagator.single` or `B3Propagator.multi`.

### Testing traces

The default `trace` provider stores spans in an in-memory processor:

```scala

// Reset before each test
trace.clearSpans()

// Call code under test
val tracer = trace.get("test")
tracer.span("outer") { _ =>
  tracer.span("inner") { span =>
    span.setAttribute("result", "ok")
  }
}

// Inspect what was recorded
val spans = trace.collectedSpans

assert(spans.size == 2)
val inner = spans.find(_.name == "inner").get
assert(inner.attributes.get(AttributeKey.string("result")) == Some("ok"))

// Check parent-child relationship
val outer = spans.find(_.name == "outer").get
assert(inner.parentSpanContext.spanId == outer.spanContext.spanId)
```

`trace.collectedSpans` returns a `List[SpanData]` in the order spans ended. No setup needed; this works out of the box.

### Span-log correlation explained

When you log inside an active span, the logger automatically attaches `traceId` and `spanId` to the record. This works because both the `Logger` and the `Tracer` share the same `ContextStorage[Option[SpanContext]]`.

At provider creation time:

```scala
// TracerProvider.builder.build() falls back to ContextStorage.defaultSpanContextStorage
// unless you pass one; here we create an explicit instance to share
val cs = ContextStorage.create[Option[SpanContext]](None)
val tracerProvider = TracerProvider.builder
  .setContextStorage(cs)
  .build()

// LoggerProvider can share the same ContextStorage
val loggerProvider = LoggerProvider.builder
  .setContextStorage(cs)  // <-- same instance
  .build()
```

When `tracer.span(...)` runs, it calls `contextStorage.scoped(Some(span.spanContext)) { ... }`. This scopes the span context for the duration of the block using `ScopedValue` on the JVM. On Scala.js there are no threads, so the storage is a plain mutable variable saved and restored around the block. Inside that block, `logger.currentSpanContext()` reads from the same storage and sees the active span. No explicit linkage code at the call site.

This works out of the box with the global singletons: a provider built without an explicit `ContextStorage` falls back to the same `ContextStorage.defaultSpanContextStorage` singleton, so the default tracer and the default logger already read and write the same storage. You only need to pass one explicitly when you want the opposite — isolating correlation between test cases, or fitting a custom runtime.

---

## Part 4: Metrics Deep Dive

This part covers choosing, pre-binding and collecting metric instruments.

### Instrument selection guide

| What you're measuring | Instrument | Example |
|---|---|---|
| Monotonically increasing total | `Counter` | Total requests served |
| Value that can go up or down | `UpDownCounter` | Active connections, queue depth |
| Distribution of values | `Histogram` | Request latency, payload size |
| Current point-in-time value | `Gauge` | CPU usage, memory used, pool size |

More specifically:

- Use **`Counter`** when you're counting things that only ever increase: requests processed, errors encountered, bytes sent. Negative deltas are silently ignored.
- Use **`UpDownCounter`** when the value can decrease: pending jobs in a queue (enqueue = +1, dequeue = -1), active WebSocket connections.
- Use **`Histogram`** when you care about the distribution, not just the total: latency percentiles (p50, p95, p99), request sizes. Histograms record min, max, sum, count, and configurable bucket counts.
- Use **`Gauge`** when you're sampling the current value of something external: a JVM heap usage check, a pool size read from a connection pool object. Last-write wins.

```scala

val requests     = metric.counter("http.requests.total")
val activeConns  = metric.upDownCounter("http.connections.active")
val latency      = metric.histogram("http.request.duration_ms")
val heapUsed     = metric.gauge("jvm.heap.used_bytes")

// Counter: only increases
requests.add(1, "method" -> "GET", "status" -> "200")

// UpDownCounter: tracks net change
activeConns.add(1)   // connection opened
activeConns.add(-1)  // connection closed

// Histogram: records the value into the appropriate bucket
latency.record(42.5, "method" -> "GET")

// Gauge: always reflects the current value
heapUsed.record(Runtime.getRuntime.totalMemory() - Runtime.getRuntime.freeMemory())
```

### Labeled instruments with pre-bound attributes

When you record against the same label combination at high frequency, pre-binding avoids the `Attributes` lookup on every call:

```scala

val requests = metric.counter("http.requests.total")

// Bind once, typically at initialization
val getOk   = requests.bind(Attributes.builder.put("method", "GET").put("status", "200").build)
val get404  = requests.bind(Attributes.builder.put("method", "GET").put("status", "404").build)
val postOk  = requests.bind(Attributes.builder.put("method", "POST").put("status", "201").build)

// Hot path: just add to the pre-bound LongAdder
def onGetOk():  Unit = getOk.add(1)
def onGet404(): Unit = get404.add(1)
def onPostOk(): Unit = postOk.add(1)
```

A `BoundCounter` holds a direct reference to the `LongAdder` for that attribute set. The call `add(1)` is just `adder.add(1)`: no map lookup, no attribute construction.

The same pattern works for histograms and gauges:

```scala

val latency    = metric.histogram("http.request.duration_ms")
val getLatency = latency.bind(Attributes.builder.put("method", "GET").build)

val poolSize     = metric.gauge("db.pool.size")
val primaryPool  = poolSize.bind(Attributes.builder.put("pool", "primary").build)

// Hot paths
def recordGetLatency(ms: Double): Unit = getLatency.record(ms)
def updatePrimaryPoolSize(n: Double): Unit = primaryPool.record(n)
```

### MetricData and collection

`Counter.collect()`, `Histogram.collect()`, and `Gauge.collect()` snapshot the current state of the instrument without resetting it:

```scala

val requests = metric.counter("http.requests.total")
requests.add(42, "method" -> "GET", "status" -> "200")
requests.add(7,  "method" -> "POST", "status" -> "201")

val data = requests.collect()
// data: MetricData.SumData(List(
//   SumDataPoint(Attributes{method="GET",status="200"}, 0L, nowNanos, 42),
//   SumDataPoint(Attributes{method="POST",status="201"}, 0L, nowNanos, 7)
// ))
// startTimeNanos is always 0 — the instruments do not track a start time.
```

Each `SumDataPoint` contains the full attribute set plus the accumulated value. `MetricData` doesn't carry the instrument's name, so to export a snapshot, pair each one with its name, description and unit in a `NamedMetric` and encode the batch with `OtlpJsonEncoder.encodeMetrics`:

```scala

val requests = metric.counter("http.requests.total")
val latency  = metric.histogram("http.request.duration_ms")
val poolSize = metric.gauge("db.pool.size")

def snapshot(): Seq[NamedMetric] = Seq(
  NamedMetric("http.requests.total", "Total HTTP requests", "1", requests.collect()),
  NamedMetric("http.request.duration_ms", "Request duration", "ms", latency.collect()),
  NamedMetric("db.pool.size", "DB pool size", "1", poolSize.collect())
)

val payload: Array[Byte] =
  OtlpJsonEncoder.encodeMetrics(snapshot(), Resource.empty, InstrumentationScope("my-service"))
```

Nothing collects metrics in the background: a snapshot happens only when you call `collect()`. Part 5 takes the snapshot on the same `PlatformExecutor` schedule that flushes the other signals.

### Thread safety internals

| Instrument | Internal structure | Thread safety |
|---|---|---|
| `Counter` | `ConcurrentHashMap[Attributes, LongAdder]` | Lock-free reads, CAS on first write per label set |
| `UpDownCounter` | same as Counter, allows negative | same |
| `Gauge` | `ConcurrentHashMap[Attributes, AtomicLong]` | Lock-free: `AtomicLong.set(doubleToLongBits(v))` |
| `Histogram` | `ConcurrentHashMap[Attributes, State]` + `ReentrantLock` per State | `ReentrantLock` per label set during `record` and `collect` |

`LongAdder` is the right choice for `Counter` because it reduces contention under high concurrency: rather than competing for a single `AtomicLong`, threads maintain per-stripe counters and sum them on read. At high add rates, `LongAdder` outperforms `AtomicLong.addAndGet` significantly.

`Histogram` needs a lock because updating count, sum, min, max, and bucket counts is a multi-field operation. No single atomic primitive covers all of them.

---

## Part 5: Production Setup

This part wires an exporter built from the public OTLP pieces into a production service and covers tuning and shutdown.

### Full production wiring

The module's built-in OTLP exporters (`OtlpJsonTraceExporter`, `OtlpJsonLogExporter` and `OtlpJsonMetricExporter`) are `private[otel]`, so user code can't construct them. The pieces they are built from are public: `OtlpJsonEncoder` turns spans, log records and metrics into OTLP JSON, `HttpSender` posts the bytes, and `ExportResult` says whether the collector accepted them. [Building an Exporter](../reference/telemetry/otel/custom-exporter.md) covers each piece; this section wires them into a service.

:::warning[Don't declare the library's package]
Scala lets your own file declare `package zio.blocks.telemetry.otel` and call the `private[otel]` exporters directly. That is an unsupported workaround, not API: their constructors can change or disappear in any release, without notice. Build on the public pieces instead.
:::

`OtlpBuffer` is the exporter. It queues one signal, drops the oldest item once `ExporterConfig#maxQueueSize` is reached, and on each flush sends what it holds in chunks of `ExporterConfig#maxBatchSize`. `Telemetry` creates one buffer per signal, feeds the span and log buffers from a `SpanProcessor` and a `LogRecordProcessor`, installs the providers, and flushes all three buffers every `ExporterConfig#flushIntervalMillis` on a `PlatformExecutor`:

```scala

// Queues one signal and posts it as OTLP JSON to `<endpoint><path>`.
final class OtlpBuffer[A](
  config: ExporterConfig,
  sender: HttpSender,
  path: String,
  encode: Seq[A] => Array[Byte]
) {
  private val url     = config.endpoint + path
  private val headers = config.headers + ("Content-Type" -> "application/json")
  private val queue   = new ArrayBlockingQueue[A](config.maxQueueSize)

  def add(item: A): Unit =
    while (!queue.offer(item)) queue.poll() // full: drop the oldest item

  def flush(): Unit = {
    val drained = new java.util.ArrayList[A]()
    queue.drainTo(drained)
    drained.asScala.toSeq.grouped(config.maxBatchSize).foreach(send)
  }

  private def send(batch: Seq[A]): Unit = {
    val result =
      try ExportResult.fromHttpResponse(sender.send(url, headers, encode(batch)))
      catch {
        case NonFatal(e) => ExportResult.Failure(retryable = true, message = String.valueOf(e))
      }
    result match {
      case ExportResult.Success             => ()
      case ExportResult.Failure(_, message) => System.err.println(s"OTLP export to $url failed: $message")
    }
  }
}

final class Telemetry(config: ExporterConfig, resource: Resource, metrics: () => Seq[NamedMetric]) {
  private val scope    = InstrumentationScope("my-service")
  private val sender   = HttpSender.jdk(config.timeout)
  private val executor = PlatformExecutor.create()

  private val spans =
    new OtlpBuffer[SpanData](config, sender, "/v1/traces", OtlpJsonEncoder.encodeTraces(_, resource, scope))
  private val logs =
    new OtlpBuffer[LogRecord](config, sender, "/v1/logs", OtlpJsonEncoder.encodeLogs(_, resource, scope))
  private val metricBuffer =
    new OtlpBuffer[NamedMetric](config, sender, "/v1/metrics", OtlpJsonEncoder.encodeMetrics(_, resource, scope))

  // Shared ContextStorage so log records inside spans get traceId/spanId
  private val contextStorage = ContextStorage.create[Option[SpanContext]](None)

  // --- Tracing ---
  private val spanProcessor = new SpanProcessor {
    def onStart(span: Span): Unit       = ()
    def onEnd(spanData: SpanData): Unit = spans.add(spanData)
    def forceFlush(): Unit              = spans.flush()
    def shutdown(): Unit                = spans.flush()
  }

  val tracerProvider: TracerProvider = TracerProvider.builder
    .setSampler(ParentBasedSampler(root = AlwaysOnSampler))
    .addSpanProcessor(spanProcessor)
    .setResource(resource)
    .setContextStorage(contextStorage)
    .build()

  trace.install(tracerProvider)

  // --- Logging ---
  private val logProcessor = new LogRecordProcessor {
    def onEmit(logRecord: LogRecord): Unit = logs.add(logRecord)
    def forceFlush(): Unit                 = logs.flush()
    def shutdown(): Unit                   = logs.flush()
  }

  val loggerProvider: LoggerProvider = LoggerProvider.builder
    .addLogRecordProcessor(logProcessor)
    .setContextStorage(contextStorage) // same storage for correlation
    .setResource(resource)
    .build()

  log.install(loggerProvider.get("my-service"), minSeverity = Severity.Info)

  // Optional: also emit JSON on stderr (a file needs your own LogWriter, see Part 2)
  log.writer(JsonLogFormatter, StderrWriter)

  // --- Metrics ---
  val meterProvider: MeterProvider = MeterProvider.builder
    .setResource(resource)
    .build()

  metric.install(meterProvider)

  private def flushMetrics(): Unit = {
    metrics().foreach(metricBuffer.add)
    metricBuffer.flush()
  }

  // --- Periodic flush ---
  private val scheduledFlush = executor.schedule(config.flushIntervalMillis, config.flushIntervalMillis, TimeUnit.MILLISECONDS) { () =>
    try {
      spans.flush()
      logs.flush()
      flushMetrics()
    } catch {
      case NonFatal(e) => System.err.println(s"OTLP flush failed: $e") // an escaping exception stops the schedule
    }
  }

  // Call at JVM shutdown, in the order explained under "Graceful shutdown order"
  def shutdown(): Unit = {
    scheduledFlush.cancel(false) // no new periodic flushes
    executor.shutdown()
    executor.executor.awaitTermination(config.timeout.toMillis, TimeUnit.MILLISECONDS) // let a running flush finish
    loggerProvider.shutdown()    // final log flush
    tracerProvider.shutdown()    // final span flush
    flushMetrics()               // last metric snapshot
    meterProvider.shutdown()
    sender.shutdown()
    log.clearWriters()
  }
}

object Telemetry {
  def fromEnv(metrics: () => Seq[NamedMetric]): Telemetry = {
    val config = ExporterConfig(
      endpoint            = sys.env.getOrElse("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4318"),
      headers             = parseHeaders(sys.env.getOrElse("OTEL_EXPORTER_OTLP_HEADERS", "")),
      timeout             = java.time.Duration.ofSeconds(10),
      maxQueueSize        = 4096,
      maxBatchSize        = 512,
      flushIntervalMillis = 5000L
    )

    val resource = Resource.create(
      Attributes.builder
        .put("service.name", sys.env.getOrElse("OTEL_SERVICE_NAME", "my-service"))
        .put("service.version", sys.env.getOrElse("SERVICE_VERSION", "unknown"))
        .put("deployment.environment", sys.env.getOrElse("ENVIRONMENT", "development"))
        .build
    )

    new Telemetry(config, resource, metrics)
  }

  private def parseHeaders(raw: String): Map[String, String] =
    if (raw.isEmpty) Map.empty
    else
      raw
        .split(',')
        .flatMap { pair =>
          pair.split('=') match {
            case Array(k, v) => Some(k.trim -> v.trim)
            case _           => None
          }
        }
        .toMap
}
```

`OtlpBuffer` reports a failed batch on standard error and drops it. To retry the failures that `ExportResult` marks retryable, and for the other duties of the internal batch processor, see [What You Are Reimplementing](../reference/telemetry/otel/custom-exporter.md#what-you-are-reimplementing). `MetricData` doesn't carry the instrument's name, so `Telemetry` takes a function that pairs each instrument's snapshot with its name in a `NamedMetric`.

The application supplies that function from the instruments it creates, and shuts telemetry down when the JVM stops:

```scala

object AppMetrics {
  val requests = metric.counter("http.requests.total")
  val latency  = metric.histogram("http.request.duration_ms")

  def snapshot(): Seq[NamedMetric] = Seq(
    NamedMetric("http.requests.total", "Total HTTP requests", "1", requests.collect()),
    NamedMetric("http.request.duration_ms", "Request duration", "ms", latency.collect())
  )
}

object Main {
  def main(args: Array[String]): Unit = {
    val telemetry = Telemetry.fromEnv(() => AppMetrics.snapshot())
    sys.addShutdownHook(telemetry.shutdown())

    log.info("application started")
    AppMetrics.requests.add(1, "method" -> "GET", "status" -> "200")
    // ... your app
  }
}
```

### ExporterConfig tuning guide

The default values work for most services. Here's when to change them:

**High-throughput services (>1000 RPS):**
```scala

ExporterConfig(
  maxQueueSize        = 8192,   // larger buffer for spikes
  maxBatchSize        = 1024,   // send bigger payloads less often
  flushIntervalMillis = 5000L   // keep at 5s; collector handles large batches fine
)
```

**Low-latency services (real-time data required):**
```scala

ExporterConfig(
  maxQueueSize        = 1024,
  maxBatchSize        = 128,    // smaller batches = lower end-to-end latency
  flushIntervalMillis = 1000L   // flush more frequently
)
```

**Development / debugging:**
```scala

ExporterConfig(
  maxQueueSize        = 256,
  maxBatchSize        = 32,
  flushIntervalMillis = 500L    // see data almost immediately
)
```

`ExporterConfig#maxQueueSize` and `ExporterConfig#maxBatchSize` interact: if records arrive faster than you export, the queue fills, and once it is full the oldest items are dropped. Set `ExporterConfig#maxQueueSize` to at least four times `ExporterConfig#maxBatchSize` to absorb transient spikes without dropping.

`timeout` is per HTTP request. If your OTLP collector is slow or flaky, a low timeout means you drop data rather than blocking. 10 seconds is usually right for LAN deployments; 30 seconds for cross-region.

### BatchProcessor behavior

The built-in exporters run on an internal `BatchProcessor`. `OtlpBuffer` in the wiring above copies its queue, chunking and interval rules but not its retries. The internal processor behaves like this:

It schedules its flush on the `ScheduledExecutorService` it is handed — `PlatformExecutor.create()` supplies a single-threaded scheduled pool whose threads are virtual (`Executors.newScheduledThreadPool(1, Thread.ofVirtual()…factory())`). This means:

- The scheduled flush (every `ExporterConfig#flushIntervalMillis`) runs on a virtual thread
- Export HTTP calls and their retry sleeps run on that same virtual thread, so backoff doesn't pin a platform thread — but the pool has one thread, so a retrying export delays every other flush sharing the executor. Give a high-volume signal its own `PlatformExecutor` if that matters.
- A full batch does not trigger a send: `ExporterConfig#maxBatchSize` only chunks what a flush drains. Data leaves on the interval, on `forceFlush()`, or on `shutdown()`.
- Queue overflow drops the oldest item, prints to stderr, and continues

The retry schedule uses exponential backoff: attempt 0 waits 1s, attempt 1 waits 2s, attempt 2 waits 4s... up to 30s maximum. `maxRetries` (default 5) counts retries, so a permanently failing batch is attempted 6 times before it is dropped. Non-retryable failures (e.g., 400 Bad Request from the collector) are dropped immediately.

On `shutdown()`, `BatchProcessor` cancels the periodic flush and does a final synchronous flush of all pending items. Retries are skipped once shutdown has started — a failing batch is dropped with a message on stderr rather than delaying exit. The executor is not owned by the processor; shut down the `PlatformExecutor` yourself, after the providers.

### Graceful shutdown order

Shut down in this order; `Telemetry#shutdown` in the wiring above does steps 2 to 6:

1. **Stop accepting new work.** Close your HTTP listener or message consumer.
2. **Stop the periodic flush.** Cancel the scheduled task, shut down the `PlatformExecutor` and wait for a running flush to finish, so it doesn't race the final one.
3. **`loggerProvider.shutdown()`.** Calls the log processor's `shutdown`, which sends the remaining log records.
4. **`tracerProvider.shutdown()`.** Calls the span processor's `shutdown`, which sends the remaining spans.
5. **Take a last metric snapshot, then `meterProvider.shutdown()`.** Sends the final cumulative values.
6. **`HttpSender#shutdown`.** Releases the sender.

The built-in exporters need a different order: their `BatchProcessor` flushes and retries on the executor, so shut the executor down last, after the providers.

### Environment-based configuration

The standard OTLP environment variables map naturally:

```scala

val config = ExporterConfig(
  endpoint = sys.env.getOrElse("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4318"),
  headers  = sys.env.get("OTEL_EXPORTER_OTLP_HEADERS")
    .map(parseOtlpHeaders)
    .getOrElse(Map.empty),
  timeout  = sys.env.get("OTEL_EXPORTER_OTLP_TIMEOUT")
    .map(s => java.time.Duration.ofMillis(s.toLong))
    .getOrElse(java.time.Duration.ofSeconds(30))
)

val serviceName = sys.env.getOrElse("OTEL_SERVICE_NAME", "my-service")

// OTLP headers format: "key1=value1,key2=value2"
def parseOtlpHeaders(raw: String): Map[String, String] =
  raw.split(',').flatMap(_.split('=') match {
    case Array(k, v) => Some(k.trim -> v.trim)
    case _           => None
  }).toMap
```

For minimum log level:

```scala

val minLevel = sys.env.get("LOG_LEVEL").flatMap {
  case "TRACE" => Some(Severity.Trace)
  case "DEBUG" => Some(Severity.Debug)
  case "INFO"  => Some(Severity.Info)
  case "WARN"  => Some(Severity.Warn)
  case "ERROR" => Some(Severity.Error)
  case _       => None
}.getOrElse(Severity.Info)

log.install(LoggerProvider.builder.build().get("my-service"), minSeverity = minLevel)
```

---

## Part 6: Integration Patterns

This part shows how to connect telemetry to HTTP frameworks, Java logging and Scala.js.

### With HTTP frameworks

A typical middleware pattern extracts incoming trace context, starts a server span, and injects context into outbound calls. Here's the conceptual shape (adapting to your specific HTTP library):

```scala

// Incoming request middleware
// `storage` is the ContextStorage passed to the installed provider's setContextStorage
def traceIncoming[Req, Resp](
  request: Req,
  getHeader: (Req, String) => Option[String],
  storage: ContextStorage[Option[SpanContext]],
  handle: Req => Resp
): Resp = {
  val parentCtx = W3CTraceContextPropagator.extract(request, getHeader)
  val tracer    = trace.get("http-server")

  // Scope the remote parent in the tracer's storage,
  // then start a child span under it
  storage.scoped(parentCtx) {
    tracer.span("http.request", SpanKind.Server) { span =>
      // Attach standard HTTP attributes
      // ... then run the handler
      handle(request)
    }
  }
}

// Outbound call wrapper
def traceOutgoing[Resp](
  name: String,
  call: Map[String, String] => Resp
): Resp = {
  val tracer = trace.get("http-client")
  tracer.span(name, SpanKind.Client) { span =>
    val headers = W3CTraceContextPropagator.inject(
      span.spanContext,
      Map.empty[String, String],
      (c, k, v) => c + (k -> v)
    )
    call(headers)
  }
}
```

For the incoming case, the remote parent has to be scoped in the same `ContextStorage` the tracer reads. A provider's own `contextStorage` field is library-private, so build one with `ContextStorage.create` and hand it to the builder via `setContextStorage` — as the server example above does — then wrap the handler in `contextStorage.scoped(parentCtx) { … }`. [How to Correlate Logs and Traces Across Services](./correlate-logs-and-traces.md) builds this pattern end to end with the http-model `Request`, including the log lines of both services.

### With existing Java logging (SLF4J/JUL)

There's no bridge today, but the shape would be a `SLF4JLogRecordProcessor` that consumes `LogRecord` from `zio-blocks-telemetry` and forwards it to the SLF4J backend, or vice versa. If you need Java library logs captured alongside your Scala logs, the simplest current approach is a Logback appender that writes to your file alongside `zio-blocks-telemetry`'s file output.

### Cross-platform considerations

The core telemetry module (`zio-blocks-telemetry`) compiles for JVM and Scala.js. What works on both:

- `log.*`, `trace.*`, `metric.*`
- `TracerProvider`, `LoggerProvider`, `MeterProvider` builders
- `Sampler`, `SpanContext`, `Attributes`, `MetricData`
- `TextLogFormatter`, `JsonLogFormatter`
- All the in-memory processors and default providers

JVM-only:
- File output: a custom `LogWriter` backed by `java.io` or `FileChannel` works on the JVM only
- `ContextStorage` using `ScopedValue`: final in JDK 25 and a preview API in JDK 21 to 24, which the module calls without needing `--enable-preview`; on Scala.js, a simpler mutable-variable implementation is used
- `PlatformExecutor`: uses `ScheduledExecutorService` with virtual threads
- The entire `zio-blocks-telemetry-otel` module (OTLP HTTP export, `BatchProcessor`)

If you're writing cross-platform code that uses telemetry, keep your call sites in the shared module and put all provider wiring in JVM-specific code.

---

## Part 7: FAQ / Troubleshooting

**"Why don't I see any log output?"**

Three common causes:

1. The global minimum level is filtering it out. The default floor is `Severity.Trace` (severity number 1), so everything passes. If you called `log.install(logger, minSeverity = Severity.Info)` or `log.setMinSeverity(Severity.Info)`, then trace and debug calls are suppressed.

2. The logger was installed but the processor chain is empty. Verify your `LoggerProvider` builder has at least one processor:
   ```scala
   LoggerProvider.builder
     .addLogRecordProcessor(new ConsoleLogRecordProcessor)  // don't forget this
     .build()
   ```

3. A namespace override is suppressing your package. Check for `log.setMinSeverity(...)` calls.

**"Why is my log call not compiling?"**

You're probably passing a type that has no `LogEnrichment` instance. Common culprits:
- `Float` (convert to `Double` with `.toDouble`)
- `Short`, `Byte`, `Char` (convert to `Long` or `String`)
- `UUID`, `Instant` (provide a `LogEnrichment[(String, YourType)]` implicit)
- Custom case classes (same: provide an implicit)

The compiler error will say something like `no implicit value for LogEnrichment[(String, UUID)]`.

**"How do I log a UUID, Instant, or custom type?"**

Define an implicit in your package object or companion:

```scala

implicit val uuidEnrichment: LogEnrichment[(String, UUID)] =
  (record, kv) => record.copy(
    attributes = record.attributes ++ Attributes.of(AttributeKey.string(kv._1), kv._2.toString)
  )
```

Then `log.info("action", "id" -> someUuid)` compiles and works.

**"What's the overhead of a disabled log level?"**

Exactly one volatile read (the global minimum level) plus, if the level passes that check, one array scan through the namespace overrides (usually empty, so O(1)). Both happen before any argument is evaluated. No string formatting, no object allocation, no method calls on your arguments.

**"Can I use this with ZIO, Cats Effect, or any other effect system?"**

Yes. The library is effect-free. `log.info(...)` is a plain synchronous call. Call it from anywhere: inside `ZIO.succeed`, inside `IO.apply`, inside a `Future`, inside a background thread. The only threading concern is that `ContextStorage` uses `ScopedValue` for span correlation, which is scoped to the current call stack. If you hop threads between starting a span and logging inside it, the log may not see the active span context. Most effect systems either stay on one thread or have ways to propagate context.

**"Why are the OTEL exporter classes private[otel]?"**

The module doesn't offer them as public API yet, so their constructors can change between releases. Code that declares the `zio.blocks.telemetry.otel` package can still reach them, but that is an unsupported workaround. To export telemetry today, build an exporter from the public `OtlpJsonEncoder`, `HttpSender` and `ExportResult`, as Part 5 does; [Building an Exporter](../reference/telemetry/otel/custom-exporter.md) covers each piece. Everything downstream uses the installed global providers and never needs the exporter types.

**"How do I test my logging and tracing?"**

For logging: use a custom `LogRecordProcessor` that stores records (see "Testing with telemetry" in Part 2), install it via `log.install(...)`, run your code, assert on the captured records.

For tracing: use `trace.collectedSpans` and `trace.clearSpans()`. The default in-memory processor accumulates spans automatically.

**"What happens when the export queue fills up?"**

When `queueSize` exceeds `ExporterConfig#maxQueueSize`, `BatchProcessor#enqueue` polls the head of the queue (the oldest item), decrements the size, and prints a warning to stderr:

```
[zio-blocks-telemetry] BatchProcessor queue full (2048). Dropping oldest item.
```

The new item is still enqueued. This is a best-effort, head-dropping strategy. You won't get backpressure; callers are never blocked. If you see frequent drops, increase `maxQueueSize` or decrease `flushIntervalMillis`.

**"Is this virtual-thread safe?"**

Yes, in the sense that a span's context is bound to the current call stack whether that stack belongs to a platform or a virtual thread: `ContextStorage` on the JVM uses `ScopedValue`, and reads and writes from the running thread are safe. Inheritance is narrower than an `InheritableThreadLocal`, though — a `ScopedValue` binding reaches a child thread only when the child is forked from a `StructuredTaskScope`. A plain `Thread.ofVirtual().start(...)` inside a span's block sees no span context, so records logged there are not correlated; open the span inside the child, or pass the `SpanContext` across explicitly. The `PlatformExecutor` that drives OTLP export runs its tasks on virtual threads, so retry sleeps don't pin platform threads.

---

## Where to Go Next

- **Complete API reference:** [Telemetry Reference](../reference/telemetry/index.md) for all types, methods, and parameter documentation
- **Installation and quick start:** same reference doc, Installation section
- **Context and dependency injection:** [Context Reference](../reference/context.md) for integrating `OtelContext` with the Context module
- **Resource management:** [Compile-Time Resource Safety with Scope](./compile-time-resource-safety-with-scope.md) — the telemetry module has no dependency on `Scope`; the providers are plain `AutoCloseable`s you shut down yourself, as in the wiring recipe above

## See Also

- [TracerProvider](../reference/telemetry/tracing/tracer-provider.md) — configure and install the tracing pipeline
- [LoggerProvider](../reference/telemetry/logging/logger-provider.md) — configure and install the logging pipeline
- [MeterProvider](../reference/telemetry/metrics/meter-provider.md) — configure and install the metrics pipeline
- [Attributes](../reference/telemetry/common/attributes.md) — the unboxed key-value type carried by every signal
- [Common Types](../reference/telemetry/common/index.md) — `Attributes`, `AttributeKey`, `Resource`, and `InstrumentationScope`

---

## How to Test Database Code Against SQLite

In this guide, we test a small bookshop repository against a real SQLite database with ZIO Test. By the end, you will have a reusable `withTestDb` helper that gives every test its own seeded database and deletes it afterwards, a `rollingBack` helper for tests that share one database, and a spec whose tests can't see each other's rows.

We start with the trap that makes the obvious in-memory database fail, then build the helpers one at a time:

- A fresh SQLite file for every test
- A schema created from code, with the primary key the derived table leaves out
- Fixture books that every test starts from
- A transaction that always rolls back, for tests that share a database
- A complete ZIO Test spec that uses all of them

This guide builds on [Build Your First Database Repository on SQLite](./your-first-sqlite-repository.md). We use `JdbcTransactor`, `Table`, `Repo` and `sql"..."` the way that tutorial introduces them and don't explain them again.

## The Problem

Database tests are only trustworthy when each test starts from a known state. A test suite that shares one database lets every test see the rows the previous tests left behind, so a test can pass or fail depending on the order the suite runs in.

A common first attempt keeps one database file for the whole suite and cleans it by hand at the start of each test:

```scala

// One database file shared by every test in the suite
val connection = DriverManager.getConnection("jdbc:sqlite:bookshop-test.db")
val statement  = connection.createStatement()

// Each test first deletes what the previous tests wrote, then inserts its own rows
statement.executeUpdate("DELETE FROM book")
statement.executeUpdate("INSERT INTO book VALUES (1, 'Zionomicon', 'John A. De Goes', 2024, NULL, 5)")
statement.executeUpdate("INSERT INTO book VALUES (2, 'Programming in Scala', 'Martin Odersky', 2008, '9780981531601', 3)")
```

This approach breaks down as the suite grows. Every new table needs another `DELETE`, a test that crashes before its cleanup leaks rows into the next one, and two tests running at the same time overwrite each other's data. The schema itself is created somewhere else, so nothing checks that it still matches the case classes the code reads.

In this guide, we solve this by giving every test a database of its own, built from code, and throwing it away when the test ends.

## Prerequisites

Add the `sql` module, the SQLite JDBC driver and ZIO Test to your `build.sbt`:

```scala
// Scala 3.9 and newer, with scalacOptions += "-experimental"; on Scala 3.3 to 3.8 the artifact is "zio-blocks-sql"
libraryDependencies += "dev.zio"   %% "zio-blocks-next-sql" % "0.0.56"
libraryDependencies += "org.xerial" % "sqlite-jdbc"         % "3.53.4.0"

libraryDependencies += "dev.zio" %% "zio-test"     % "2.1.27" % Test
libraryDependencies += "dev.zio" %% "zio-test-sbt" % "2.1.27" % Test
```

The `sql` module is published for Scala 3 only, and `JdbcTransactor` runs on the JVM. The code under test lives in `src/main`; the helpers and the spec we write in this guide live in `src/test`, so they never ship with the application.

## The Code Under Test

We test a bookshop. Its `Book` is the one from the SQLite tutorial with one more field, `stock`, which counts the copies on the shelf. As in the tutorial, `author` holds each book's first author:

```scala

case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String], stock: Int)

object Book {
  given Schema[Book] = Schema.derived
}

object Bookshop {

  val books: Repo[Book, Long] = Repo.derived[Book, Long]

  def findByTitle(title: String)(using DbCon): Maybe[Book] =
    sql"SELECT id, title, author, year, isbn, stock FROM book WHERE title = $title".queryOne[Book]

  def restock(id: Long, copies: Int)(using DbCon): Int =
    sql"UPDATE book SET stock = stock + $copies WHERE id = $id".update

  def addAll(newBooks: Seq[Book])(using DbCon): Int =
    newBooks.map(books.insert).sum
}
```

Every operation takes a `(using DbCon)` instead of a transactor. The caller decides where the connection comes from, which is what lets a test run the same code against a database it controls.

## Give Each Test Its Own Database

`jdbc:sqlite::memory:` looks like the natural choice for tests: nothing touches the disk and nothing needs deleting. It doesn't work with `JdbcTransactor`, though. Every call to `JdbcTransactor#connect` or `JdbcTransactor#transact` opens a new connection and closes it at the end of the block, and SQLite gives each new connection to `:memory:` its own empty database. We create the `book` table in one block:

```scala

val inMemory = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)

inMemory.connect(Table.derived[Book].createTable(SqlDialect.SQLite).update)
```

Now we count the books in the next block:

```scala
println(Try(inMemory.connect(Bookshop.books.count)))
// Failure(org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (no such table: book))
```

The table is gone by the time the second block runs. Instead, we give each test a temporary SQLite file. `Files.createTempFile` picks a name no other test uses, and a `finally` block deletes the file when the test ends, whether it passes or throws:

```scala

def withTempDb[A](f: JdbcTransactor => A): A = {
  val file = Files.createTempFile("bookshop-test", ".db")
  try f(JdbcTransactor.fromUrl(s"jdbc:sqlite:$file", SqlDialect.SQLite))
  finally Files.deleteIfExists(file)
}
```

A file behaves exactly like the database the application uses: tables outlive a single block, transactions commit and roll back for real, and SQLite's locking is the same. We run two "tests" and ask SQLite which file each one uses:

```scala
val usedFiles = List("first test", "second test").map { _ =>
  withTempDb { transactor =>
    transactor.connect(sql"SELECT file FROM pragma_database_list WHERE name = 'main'".queryOne[String].getOrElse(""))
  }
}
```

Then we compare the two paths and look for the files on disk:

```scala
println(s"Each test used its own file: ${usedFiles.distinct.size == usedFiles.size}")
// Each test used its own file: true
println(s"Files left behind: ${usedFiles.count(file => Files.exists(Paths.get(file)))}")
// Files left behind: 0
```

`pragma_database_list` is SQLite's list of open database files, so each test reports the path of its own file. The two paths differ, and after `withTempDb` returns, neither file exists any more. Because the names are unique, tests can also run concurrently without seeing each other's rows.

## Create the Schema from Code

A test database starts empty, so each test needs the tables before it can run. The SQLite tutorial creates them with `Table#createTable`, which derives a `CREATE TABLE` statement from the schema. We derive the table for `Book`:

```scala
val bookTable = Table.derived[Book]
```

Then we print its `CREATE TABLE` statement for SQLite:

```scala
println(bookTable.createTable(SqlDialect.SQLite).sql(SqlDialect.SQLite))
// CREATE TABLE IF NOT EXISTS book (
//   id INTEGER NOT NULL,
//   title TEXT NOT NULL,
//   author TEXT NOT NULL,
//   year INTEGER NOT NULL,
//   isbn TEXT,
//   stock INTEGER NOT NULL
// )
```

The statement has a column for every field, but no `PRIMARY KEY`. That matters in tests: a database with no primary key accepts two books with the same ID, so a test that checks that duplicates are rejected can never pass. `Repo#insertOrUpdate` doesn't work on such a table either, because its `ON CONFLICT` clause needs a primary key to detect the conflict. We try both on a table created from the derived statement:

```scala
withTempDb { transactor =>
  transactor.connect {
    bookTable.createTable(SqlDialect.SQLite).update
    Bookshop.books.insert(Book(1L, "Zionomicon", "John A. De Goes", 2024, None, 5))
    Bookshop.books.insert(Book(1L, "Zionomicon", "John A. De Goes", 2024, None, 5))
    println(s"Rows with id 1 in the derived table: ${Bookshop.books.count}")
    println(Try(Bookshop.books.insertOrUpdate(Book(1L, "Zionomicon", "John A. De Goes", 2024, None, 7))))
  }
}
// Rows with id 1 in the derived table: 2
// Failure(org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (ON CONFLICT clause does not match any PRIMARY KEY or UNIQUE constraint))
```

Both inserts succeed, and the upsert fails. So we write the `CREATE TABLE` statement by hand with `sql"..."`, keeping the derived column types and adding `PRIMARY KEY` to `id`:

```scala
def createSchema()(using DbCon): Unit =
  sql"""CREATE TABLE book (
    id INTEGER PRIMARY KEY,
    title TEXT NOT NULL,
    author TEXT NOT NULL,
    year INTEGER NOT NULL,
    isbn TEXT,
    stock INTEGER NOT NULL
  )""".update
```

A hand-written schema can drift from the case class, so we check it against the derived table too. `pragma_table_info('book')` lists the columns of the table SQLite holds, and `Table#columns` lists the columns the repository reads and writes. With the hand-written schema, the duplicate insert now fails and the columns still match:

```scala
withTempDb { transactor =>
  transactor.connect {
    createSchema()
    Bookshop.books.insert(Book(1L, "Zionomicon", "John A. De Goes", 2024, None, 5))
    println(Try(Bookshop.books.insert(Book(1L, "Zionomicon", "John A. De Goes", 2024, None, 5))))

    val columns = sql"SELECT name FROM pragma_table_info('book')".query[String]
    println(s"Columns match the derived table: ${columns == bookTable.columns.toList}")
  }
}
// Failure(org.sqlite.SQLiteException: [SQLITE_CONSTRAINT_PRIMARYKEY] A PRIMARY KEY constraint failed (UNIQUE constraint failed: book.id))
// Columns match the derived table: true
```

The `SQLiteException` that SQLite raises is the same one the application would see in production, so a test can rely on it. The spec at the end of this guide keeps the column check as a test of its own, so adding a field to `Book` without updating `createSchema` breaks the build.

## Seed Fixtures for Every Test

Most tests need some data to work on. We define the fixture books once and share them across the whole suite. Since they are immutable values, a test can compare the database against them directly:

```scala
val zionomicon         = Book(1L, "Zionomicon", "John A. De Goes", 2024, None, 5)
val programmingInScala = Book(2L, "Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"), 3)
val fpInScala          = Book(3L, "Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"), 0)

val fixtures: List[Book] = List(zionomicon, programmingInScala, fpInScala)
```

`resetFixtures` empties the table with `Repo#clear` and inserts the fixtures in one JDBC batch with `Repo#insertBatch`. `withTestDb` builds on `withTempDb`: it creates the schema and the fixtures in one transaction, then hands the transactor to the test:

```scala
def resetFixtures()(using DbCon): Unit = {
  Bookshop.books.clear()
  Bookshop.books.insertBatch(fixtures)
}

def withTestDb[A](f: JdbcTransactor => A): A =
  withTempDb { transactor =>
    transactor.transact {
      createSchema()
      resetFixtures()
    }
    f(transactor)
  }
```

Each test that calls `withTestDb` starts from exactly these three books, whatever the tests before it did. `resetFixtures` is useful on its own when several steps share one database and each must start from the fixtures again:

```scala
withTestDb { transactor =>
  transactor.connect {
    Bookshop.books.all.foreach(println)

    Bookshop.books.delete(zionomicon.id)
    println(s"After a test deletes a book: ${Bookshop.books.count} books")

    resetFixtures()
    println(s"After resetFixtures: ${Bookshop.books.count} books")
  }
}
// Book(1,Zionomicon,John A. De Goes,2024,None,5)
// Book(2,Programming in Scala,Martin Odersky,2008,Some(9780981531601),3)
// Book(3,Functional Programming in Scala,Paul Chiusano,2014,Some(9781617290657),0)
// After a test deletes a book: 2 books
// After resetFixtures: 3 books
```

## Roll Back Instead of Cleaning Up

Sometimes a fresh database per test is too slow, for example when the fixtures are large and many tests only read them. Then the tests share one seeded database, and each test must leave it exactly as it found it.

The usual answer is to run each test inside a transaction and roll it back at the end. `JdbcTransactor` has no rollback-only flag: `JdbcTransactor#transact` commits when its block returns and rolls back only when the block throws. So we throw on purpose. `rollingBack` runs the test body inside `JdbcTransactor#transact` and wraps its result in a private exception; `JdbcTransactor#transact` rolls back and rethrows it, and `rollingBack` catches it and returns the result:

```scala
private final class Rollback(val result: Any) extends RuntimeException(null, null, false, false)

def rollingBack[A](transactor: JdbcTransactor)(f: DbTx ?=> A): A =
  try transactor.transact(throw new Rollback(f))
  catch { case rollback: Rollback => rollback.result.asInstanceOf[A] }
```

The `Rollback` class is private, so no code under test can catch it by accident, and its `RuntimeException` constructor skips the stack trace, which a signal like this never needs. When the test body itself throws, `JdbcTransactor#transact` rolls back too and the original exception reaches the test. We add a book and restock another inside `rollingBack`, then look at the database afterwards:

```scala
withTestDb { transactor =>
  val (booksInside, stockInside) = rollingBack(transactor) {
    Bookshop.books.insert(Book(4L, "Hands-on Scala Programming", "Li Haoyi", 2020, None, 2))
    Bookshop.restock(fpInScala.id, 10)
    (Bookshop.books.count, Bookshop.books.find(fpInScala.id).map(_.stock).getOrElse(-1))
  }
  println(s"Inside the test: $booksInside books, $stockInside copies of FP in Scala")

  transactor.connect {
    val stock = Bookshop.books.find(fpInScala.id).map(_.stock).getOrElse(-1)
    println(s"After the test: ${Bookshop.books.count} books, $stock copies of FP in Scala")
  }
}
// Inside the test: 4 books, 10 copies of FP in Scala
// After the test: 3 books, 0 copies of FP in Scala
```

Inside the block, the test sees its own writes. Afterwards, the shelf is back to three books and _Functional Programming in Scala_ has no copies again.

:::warning
`rollingBack` only works when the code under test uses the connection it is given, as every `(using DbCon)` method in `Bookshop` does. Code that calls `JdbcTransactor#transact` itself opens a second connection, which can't see the uncommitted rows and waits for the test's write lock until SQLite gives up with `SQLITE_BUSY` after five seconds. When such code needs its own all-or-nothing block, give it a `(using DbTx)` and use the nested `DbTx#transact`, which runs on the same connection with a savepoint; [SQL Transactions](./sql-transactions.md) explains savepoints.
:::

A fresh database per test is still the simpler default. It needs no discipline from the code under test, and a test that fails halfway can't affect any other.

## Putting It Together

The complete spec runs four tests, each on its own database from `withTestDb`. `dbTest` wraps the helper in `ZIO.attemptBlocking`, because JDBC calls block the thread they run on; an exception from the database becomes a test failure instead of a defect:

```scala title="sql-examples/src/test/scala/testdatabasecode/BookshopSpec.scala"
package testdatabasecode

/**
 * How to Test Database Code Against SQLite — the complete spec
 *
 * Every test runs against its own SQLite file, created by `TestDb.withTestDb`
 * with the schema and the three fixture books, and deleted afterwards.
 *
 * Run with: cd sql-examples && sbt "testOnly testdatabasecode.BookshopSpec"
 */
object BookshopSpec extends ZIOSpecDefault {

  private def dbTest(label: String)(body: JdbcTransactor => TestResult) =
    test(label)(ZIO.attemptBlocking(withTestDb(body)))

  private val handsOnScala = Book(4L, "Hands-on Scala Programming", "Li Haoyi", 2020, None, 2)

  def spec = suite("Bookshop")(
    dbTest("finds a book by its title") { transactor =>
      val found   = transactor.connect(Bookshop.findByTitle("Zionomicon"))
      val missing = transactor.connect(Bookshop.findByTitle("Scala Puzzlers"))
      assertTrue(found == Maybe.present(zionomicon), missing == Maybe.absent[Book])
    },
    dbTest("restock adds copies to one book only") { transactor =>
      val updated = transactor.transact(Bookshop.restock(fpInScala.id, 4))
      val shelf   = transactor.connect(Bookshop.books.all.sortBy(_.id))
      assertTrue(updated == 1, shelf == List(zionomicon, programmingInScala, fpInScala.copy(stock = 4)))
    },
    dbTest("a failing insert rolls back the books added before it") { transactor =>
      val result = Try(transactor.transact(Bookshop.addAll(List(handsOnScala, zionomicon))))
      val shelf  = transactor.connect(Bookshop.books.all.sortBy(_.id))
      assertTrue(result.isFailure, shelf == fixtures)
    },
    dbTest("the hand-written table has the columns Book needs") { transactor =>
      val columns = transactor.connect(sql"SELECT name FROM pragma_table_info('book')".query[String])
      assertTrue(columns == Table.derived[Book].columns.toList)
    }
  )
}
```

Each of the four tests checks one behaviour:

- `finds a book by its title` reads a fixture book and checks that an unknown title finds nothing.
- `restock adds copies to one book only` compares the whole shelf, so it also catches an `UPDATE` without a `WHERE` clause.
- `a failing insert rolls back the books added before it` inserts _Hands-on Scala Programming_ and then a duplicate _Zionomicon_; the primary key rejects the duplicate, `JdbcTransactor#transact` rolls back both inserts, and the shelf still holds exactly the fixtures.
- `the hand-written table has the columns Book needs` guards `createSchema` against drift.

Assertions compare whole values with `==` — the full `Maybe[Book]`, the full shelf as a `List[Book]` — so a wrong field anywhere fails the test.

## What SQLite Can't Test

SQLite is a good stand-in for code that sticks to portable SQL, but not for features only PostgreSQL has. These stay untested on SQLite:

- Array and `jsonb` columns read through `PgCodec`, which exists only for PostgreSQL.
- Shadow tables: `Dialect.Postgres#createShadowTableDDL` copies a table with `CREATE TABLE ... LIKE`, which SQLite doesn't support.
- Queue claims with `FOR UPDATE SKIP LOCKED`, which SQLite doesn't have.
- Isolation levels: SQLite runs every transaction as `SERIALIZABLE`, so code that depends on PostgreSQL's `READ COMMITTED` behaviour can't be checked here.

SQLite's own dialect says so when we ask it for a shadow table:

```scala
println(Try(Dialect.SQLite.createShadowTableDDL("book_shadow", "book")))
// Failure(java.lang.UnsupportedOperationException: SQLite does not support CREATE TABLE ... LIKE. Create shadow table 'book_shadow' manually with matching column definitions.)
```

For these features, test against a real PostgreSQL server. Testcontainers is the usual way to start a throwaway PostgreSQL in Docker from a test suite. A separate guide covers connecting to PostgreSQL through a connection pool.

## Running the Examples

All the code in this guide has runnable companions in the `sql-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/sql-examples
```

The code under test is in `src/main` and everything else is in `src/test`, so the examples run with `Test/runMain`. Each one creates and deletes its own temporary database files, so you can run them in any order. To compile all the examples without running them:

```bash
sbt "Test/compile"
```

### The Code Under Test

`Book` and the `Bookshop` operations the tests exercise are in this file:

<details>
  <summary>sql-examples/src/main/scala/testdatabasecode/Bookshop.scala</summary>

```scala title="sql-examples/src/main/scala/testdatabasecode/Bookshop.scala" showLineNumbers
package testdatabasecode

/**
 * How to Test Database Code Against SQLite — the code under test
 *
 * A bookshop repository: find a book by its title, add copies to its stock and
 * add several books at once. The tests in `src/test/scala/testdatabasecode`
 * run it against a fresh SQLite database.
 */
case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String], stock: Int)

object Book {
  given Schema[Book] = Schema.derived
}

object Bookshop {

  val books: Repo[Book, Long] = Repo.derived[Book, Long]

  def findByTitle(title: String)(using DbCon): Maybe[Book] =
    sql"SELECT id, title, author, year, isbn, stock FROM book WHERE title = $title".queryOne[Book]

  def restock(id: Long, copies: Int)(using DbCon): Int =
    sql"UPDATE book SET stock = stock + $copies WHERE id = $id".update

  def addAll(newBooks: Seq[Book])(using DbCon): Int =
    newBooks.map(books.insert).sum
}
```

</details>

### The Test Helpers

`withTempDb`, `createSchema`, the fixtures, `resetFixtures`, `withTestDb` and `rollingBack` are in one object:

<details>
  <summary>sql-examples/src/test/scala/testdatabasecode/TestDb.scala</summary>

```scala title="sql-examples/src/test/scala/testdatabasecode/TestDb.scala" showLineNumbers
package testdatabasecode

/**
 * How to Test Database Code Against SQLite — the test helpers
 *
 * `withTempDb` gives each caller its own SQLite file and deletes it afterwards.
 * `withTestDb` adds the schema and the fixture books on top. `rollingBack`
 * runs a block in a transaction that is always rolled back.
 */
object TestDb {

  val zionomicon         = Book(1L, "Zionomicon", "John A. De Goes", 2024, None, 5)
  val programmingInScala = Book(2L, "Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"), 3)
  val fpInScala          = Book(3L, "Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"), 0)

  val fixtures: List[Book] = List(zionomicon, programmingInScala, fpInScala)

  def withTempDb[A](f: JdbcTransactor => A): A = {
    val file = Files.createTempFile("bookshop-test", ".db")
    try f(JdbcTransactor.fromUrl(s"jdbc:sqlite:$file", SqlDialect.SQLite))
    finally Files.deleteIfExists(file)
  }

  def createSchema()(using DbCon): Unit =
    sql"""CREATE TABLE book (
      id INTEGER PRIMARY KEY,
      title TEXT NOT NULL,
      author TEXT NOT NULL,
      year INTEGER NOT NULL,
      isbn TEXT,
      stock INTEGER NOT NULL
    )""".update

  def resetFixtures()(using DbCon): Unit = {
    Bookshop.books.clear()
    Bookshop.books.insertBatch(fixtures)
  }

  def withTestDb[A](f: JdbcTransactor => A): A =
    withTempDb { transactor =>
      transactor.transact {
        createSchema()
        resetFixtures()
      }
      f(transactor)
    }

  private final class Rollback(val result: Any) extends RuntimeException(null, null, false, false)

  def rollingBack[A](transactor: JdbcTransactor)(f: DbTx ?=> A): A =
    try transactor.transact(throw new Rollback(f))
    catch { case rollback: Rollback => rollback.result.asInstanceOf[A] }
}
```

</details>

### Give Each Test Its Own Database

This example shows the `:memory:` table disappearing between blocks and two tests using two temporary files that are gone afterwards:

<details>
  <summary>sql-examples/src/test/scala/testdatabasecode/FreshDatabaseExample.scala</summary>

```scala title="sql-examples/src/test/scala/testdatabasecode/FreshDatabaseExample.scala" showLineNumbers
package testdatabasecode

/**
 * How to Test Database Code Against SQLite — Give Each Test Its Own Database
 *
 * Shows that `jdbc:sqlite::memory:` loses its tables between blocks, then runs
 * two "tests" with `TestDb.withTempDb` and checks that each one used its own
 * file and that no file is left behind.
 *
 * Run with: cd sql-examples && sbt "Test/runMain
 * testdatabasecode.FreshDatabaseExample"
 */
object FreshDatabaseExample extends App {

  val inMemory = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)

  inMemory.connect(TestDb.createSchema())
  println(Try(inMemory.connect(Bookshop.books.count)))

  val usedFiles = List("first test", "second test").map { _ =>
    TestDb.withTempDb { transactor =>
      transactor.connect(sql"SELECT file FROM pragma_database_list WHERE name = 'main'".queryOne[String].getOrElse(""))
    }
  }

  println(s"Each test used its own file: ${usedFiles.distinct.size == usedFiles.size}")
  println(s"Files left behind: ${usedFiles.count(file => Files.exists(Paths.get(file)))}")
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "Test/runMain testdatabasecode.FreshDatabaseExample"
```

### Create the Schema from Code

This example prints the derived `CREATE TABLE` statement, shows the derived table accepting a duplicate ID and rejecting an upsert, and shows the hand-written schema rejecting the duplicate:

<details>
  <summary>sql-examples/src/test/scala/testdatabasecode/SchemaExample.scala</summary>

```scala title="sql-examples/src/test/scala/testdatabasecode/SchemaExample.scala" showLineNumbers
package testdatabasecode

/**
 * How to Test Database Code Against SQLite — Create the Schema from Code
 *
 * Prints the `CREATE TABLE` statement derived from `Book`, shows that a table
 * created from it accepts two books with the same ID and can't run
 * `Repo#insertOrUpdate`, and that the
 * hand-written schema in `TestDb.createSchema` rejects the duplicate and still
 * has the columns the derived table expects.
 *
 * Run with: cd sql-examples && sbt "Test/runMain
 * testdatabasecode.SchemaExample"
 */
object SchemaExample extends App {

  val bookTable = Table.derived[Book]

  println(bookTable.createTable(SqlDialect.SQLite).sql(SqlDialect.SQLite))

  TestDb.withTempDb { transactor =>
    transactor.connect {
      bookTable.createTable(SqlDialect.SQLite).update
      Bookshop.books.insert(TestDb.zionomicon)
      Bookshop.books.insert(TestDb.zionomicon)
      println(s"Rows with id 1 in the derived table: ${Bookshop.books.count}")
      println(Try(Bookshop.books.insertOrUpdate(TestDb.zionomicon.copy(stock = 7))))
    }
  }

  TestDb.withTempDb { transactor =>
    transactor.connect {
      TestDb.createSchema()
      Bookshop.books.insert(TestDb.zionomicon)
      println(Try(Bookshop.books.insert(TestDb.zionomicon)))

      val columns = sql"SELECT name FROM pragma_table_info('book')".query[String]
      println(s"Columns match the derived table: ${columns == bookTable.columns.toList}")
    }
  }
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "Test/runMain testdatabasecode.SchemaExample"
```

### Seed Fixtures for Every Test

This example lists the fixture books, deletes one and restores them with `resetFixtures`:

<details>
  <summary>sql-examples/src/test/scala/testdatabasecode/FixturesExample.scala</summary>

```scala title="sql-examples/src/test/scala/testdatabasecode/FixturesExample.scala" showLineNumbers
package testdatabasecode

/**
 * How to Test Database Code Against SQLite — Seed Fixtures for Every Test
 *
 * Opens a database with `TestDb.withTestDb`, changes the fixture data the way a
 * test would, and restores it with `TestDb.resetFixtures`.
 *
 * Run with: cd sql-examples && sbt "Test/runMain
 * testdatabasecode.FixturesExample"
 */
object FixturesExample extends App {

  TestDb.withTestDb { transactor =>
    transactor.connect {
      Bookshop.books.all.foreach(println)

      Bookshop.books.delete(TestDb.zionomicon.id)
      println(s"After a test deletes a book: ${Bookshop.books.count} books")

      TestDb.resetFixtures()
      println(s"After resetFixtures: ${Bookshop.books.count} books")
    }
  }
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "Test/runMain testdatabasecode.FixturesExample"
```

### Roll Back Instead of Cleaning Up

This example writes inside `rollingBack` and shows that the writes are gone afterwards:

<details>
  <summary>sql-examples/src/test/scala/testdatabasecode/RollbackExample.scala</summary>

```scala title="sql-examples/src/test/scala/testdatabasecode/RollbackExample.scala" showLineNumbers
package testdatabasecode

/**
 * How to Test Database Code Against SQLite — Roll Back Instead of Cleaning Up
 *
 * Runs a test body with `TestDb.rollingBack`, so its writes are visible inside
 * the block and gone afterwards, while the block's result still reaches the
 * caller.
 *
 * Run with: cd sql-examples && sbt "Test/runMain
 * testdatabasecode.RollbackExample"
 */
object RollbackExample extends App {

  TestDb.withTestDb { transactor =>
    val (booksInside, stockInside) = TestDb.rollingBack(transactor) {
      Bookshop.books.insert(Book(4L, "Hands-on Scala Programming", "Li Haoyi", 2020, None, 2))
      Bookshop.restock(TestDb.fpInScala.id, 10)
      (Bookshop.books.count, Bookshop.books.find(TestDb.fpInScala.id).map(_.stock).getOrElse(-1))
    }
    println(s"Inside the test: $booksInside books, $stockInside copies of FP in Scala")

    transactor.connect {
      val stock = Bookshop.books.find(TestDb.fpInScala.id).map(_.stock).getOrElse(-1)
      println(s"After the test: ${Bookshop.books.count} books, $stock copies of FP in Scala")
    }
  }
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "Test/runMain testdatabasecode.RollbackExample"
```

### The Complete Spec

The spec from "Putting It Together" runs with sbt's test runner:

```bash
sbt "testOnly testdatabasecode.BookshopSpec"
```

## Going Further

- [SQL Transactions](./sql-transactions.md) covers isolation levels, read-only transactions and nested transactions with savepoints.
- [Transactor](../reference/sql/transactor.md) documents `JdbcTransactor#connect`, `JdbcTransactor#transact` and the SQLite connection settings.
- [Repo](../reference/sql/repo.md) lists every repository operation, including `Repo#insertOrUpdate` and its primary-key requirement.
- [Ddl](../reference/sql/ddl.md) and [Table](../reference/sql/table.md) describe the generated `CREATE TABLE` statements.
- [sql(table) Checked Interpolation](./sql-checked-interpolation.md) lets the compiler check table and column names in queries like `Bookshop.findByTitle`.

---

## How to Test Code That Logs, Traces and Records Metrics

In this guide, we test the telemetry of a bookstore's order service: when a payment fails, the service must log an `ERROR`, mark its span as failed and add one to the `orders.failed` counter. We capture every span, log record and counter value in memory while a test runs, then assert on them like any other value. By the end, you will have a ZIO Test spec whose tests each record their own telemetry, so they can run in parallel without seeing each other's spans, logs or counts.

We take an incremental approach:

- Record the service's spans and assert on their names, attributes, status and parent.
- Capture its log records and assert on their severity, message and fields.
- Read its counters back and assert on their totals.
- Bundle the three into a per-test kit and use it in a ZIO Test spec.
- Test code that uses the global `log`, `trace` and `metric` objects, and reset them safely between tests.

## The Problem

Telemetry is often a real requirement. "When a payment fails, log an `ERROR` and bump `orders.failed`" is a rule that an on-call engineer's alert depends on. Yet it usually goes untested, because the output vanishes into the console or a monitoring system, and the only way to check it is to read text that was meant for people.

For example, a test of an order service that logs with `println` has to capture standard output and search it:

```scala

object OrderLog {
  def placeOrder(orderId: String, declined: Boolean): Unit =
    if (declined) println(s"ERROR Payment failed for $orderId")
    else println(s"INFO Order placed: $orderId")
}

val out = new ByteArrayOutputStream
Console.withOut(out)(OrderLog.placeOrder("order-2", declined = true))
assert(out.toString.contains("ERROR"))
```

This test breaks when someone changes the line format, passes when "ERROR" appears anywhere in the output, and sees the output of every other test that prints at the same time. It also can't check anything that isn't text: how long a step took, which step a line belongs to, or how many orders failed.

In this guide, we solve this by giving the code under test a tracer, a logger and a meter that record into memory, so a test reads back typed values: `SpanData`, `LogRecord` and `MetricData`.

## Prerequisites

Add the `telemetry` module and ZIO Test to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-telemetry" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-telemetry" % "0.0.56"
scalacOptions += "-experimental"

// Either way
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-test"     % "2.1.27" % Test,
  "dev.zio" %% "zio-test-sbt" % "2.1.27" % Test
)
```

The `telemetry` module needs **JDK 21 or newer** on the JVM, to compile as well as to run. [How to Add ZIO Blocks to Your Build](../getting-started/installation.md) explains the two artifact names and the JDK each module needs.

The order service and its tests compile on Scala 2.13 and Scala 3. The section on the global `log` object is Scala 3 only: in version 0.0.56, the `log.*` calls don't compile on Scala 2.13 outside the library's own package, a known bug in that version.

We assume you have seen the `log`, `trace` and `metric` objects before. If not, the tutorial [Instrument a Program with Logs, Traces and Metrics](./instrument-a-program-with-logs-traces-and-metrics.md) introduces them.

## The Order Service Under Test

The service places one order: it charges the customer through a `PaymentGateway`, then records the outcome three ways. It runs inside a `placeOrder` span with a `charge` child span, logs one line, and adds one to `orders.placed` or `orders.failed`.

The key design choice is that the service receives its `Tracer`, `Logger` and `Meter` as constructor parameters instead of reaching for the global objects. The [tracing reference](../reference/telemetry/tracing/index.md) recommends this for library code, and the [`Logger`](../reference/telemetry/logging/logger.md) page says why: then telemetry is an ordinary dependency, and a test can pass in one whose output it inspects.

Here is the model and the service:

```scala

final case class Order(id: String, title: String, copies: Int, priceCents: Long) {
  def totalCents: Long = copies * priceCents
}

trait PaymentGateway {
  def charge(orderId: String, amountCents: Long): Either[String, String]
}

object PaymentGateway {
  val approveAll: PaymentGateway = (orderId, _) => Right(s"receipt-$orderId")

  def declineAll(reason: String): PaymentGateway = (_, _) => Left(reason)
}

final class OrderService(payments: PaymentGateway, tracer: Tracer, logger: Logger, meter: Meter) {
  val ordersPlaced: Counter = meter.counterBuilder("orders.placed").build()
  val ordersFailed: Counter = meter.counterBuilder("orders.failed").build()

  def placeOrder(order: Order): Either[String, String] =
    tracer.span("placeOrder") { span =>
      span.setAttribute("order.id", order.id)
      span.setAttribute("book.title", order.title)

      val charged = tracer.span("charge") { chargeSpan =>
        chargeSpan.setAttribute("amount.cents", order.totalCents)
        payments.charge(order.id, order.totalCents)
      }

      charged match {
        case Right(receipt) =>
          ordersPlaced.add(1L, "title" -> order.title)
          logger.info(
            "Order placed",
            "order.id" -> AttributeValue.StringValue(order.id),
            "receipt"  -> AttributeValue.StringValue(receipt)
          )
          span.setStatus(SpanStatus.Ok)
        case Left(reason) =>
          ordersFailed.add(1L, "reason" -> reason)
          logger.error(
            "Payment failed",
            "order.id" -> AttributeValue.StringValue(order.id),
            "reason"   -> AttributeValue.StringValue(reason)
          )
          span.setStatus(SpanStatus.Error(reason))
      }
      charged
    }
}
```

The code above:

- `PaymentGateway.approveAll` and `PaymentGateway.declineAll` are two fake gateways for the tests; a real one would call a payment provider.
- The counters are public `val`s, created once from the meter. A test reads them back, and keeping one instance per counter matters: every call to a counter builder registers a new instrument.
- `Logger#info` and `Logger#error` take their fields as `(String, AttributeValue)` pairs. Unlike the global `log.*` calls, they are plain methods, so they also compile on Scala 2.13.
- `SpanStatus.Error(reason)` marks the span as failed. Nothing sets it for us: a span whose block throws still ends with the status `Unset`.

In production, the service gets the global tracer and meter, and a logger that prints to standard output:

```scala
val consoleLogger: Logger =
  LoggerProvider.builder
    .addLogRecordProcessor(new ConsoleLogRecordProcessor)
    .build()
    .get("bookstore.orders")

val orders = new OrderService(
  PaymentGateway.approveAll,
  trace.get("bookstore.orders"),
  consoleLogger,
  metric.get("bookstore.orders")
)
```

`trace.get` and `metric.get` return a tracer and a meter from whatever provider is installed when they run, so build the service after any `trace.install` or `metric.install` call at startup.

## Record Spans in Memory

A `TracerProvider` sends every finished span to its `SpanProcessor`s. The default provider behind `trace` keeps them in an in-memory buffer, but that buffer is internal and shared by the whole program. A test needs a buffer of its own, so we write a processor that keeps the spans in a queue:

```scala

final class SpanRecorder extends SpanProcessor {
  private val ended = new ConcurrentLinkedQueue[SpanData]()

  def onStart(span: Span): Unit       = ()
  def onEnd(spanData: SpanData): Unit = { ended.add(spanData); () }
  def shutdown(): Unit                = ()
  def forceFlush(): Unit              = ()

  def spans: List[SpanData] = ended.asScala.toList

  def tree: List[(String, Option[String])] = {
    val nameOf = spans.map(s => s.spanContext.spanId -> s.name).toMap
    spans.map(s => s.name -> nameOf.get(s.parentSpanContext.spanId))
  }
}
```

The code above:

- `onEnd` receives each span as a `SpanData` once it has ended. We ignore `onStart`, because a test only cares about finished spans.
- `ConcurrentLinkedQueue` keeps the spans in the order they ended and is safe to write from any thread.
- `SpanRecorder#tree` pairs each span's name with its parent's name, or `None` for a root span. Ids and timestamps change on every run, so this leaves only what a test should assert on.

Next, we build a tracer that sends its spans to a recorder, give it to the service, and place an order that the gateway declines. The logger and meter here record nothing; we use them in the next sections:

```scala
val spanRecorder = new SpanRecorder

val tracedOrders = new OrderService(
  PaymentGateway.declineAll("card declined"),
  TracerProvider.builder.addSpanProcessor(spanRecorder).build().get("test"),
  LoggerProvider.builder.build().get("test"),
  MeterProvider.builder.build().get("test")
)

val bookTitle = AttributeKey.string("book.title")

def placeOrderSpan: Option[SpanData] = spanRecorder.spans.find(_.name == "placeOrder")
```

`AttributeKey.string("book.title")` is a typed key: `Attributes#get` with it returns an `Option[String]`. Use `AttributeKey.long` for a `Long` attribute such as `amount.cents`.

Place the order and look at the spans it recorded:

```scala
tracedOrders.placeOrder(Order("order-2", "Programming in Scala", 2, 5000L))
// res2: Either[String, String] = Left("card declined")

println(placeOrderSpan.map(_.status))
// Some(Error(card declined))
println(placeOrderSpan.flatMap(_.attributes.get(bookTitle)))
// Some(Programming in Scala)
println(spanRecorder.tree)
// List((charge,Some(placeOrder)), (placeOrder,None))
```

The code above:

- `SpanData#status` is the status the service set: `Error(card declined)`.
- `spanRecorder.tree` lists `charge` first, because a child span ends before its parent, and shows that `charge` is a child of `placeOrder`.

In a ZIO Test spec, each of these becomes one line of an `assertTrue`, such as `placeOrderSpan.map(_.status) == Some(SpanStatus.Error("card declined"))`. The [`SpanData`](../reference/telemetry/tracing/span-data.md) page lists everything else a span records, including its events and start and end times.

:::warning
Only spans started with `Tracer#span` (or `trace.span`) reach the processors. A span started by hand with `Tracer#spanBuilder` and `SpanBuilder#startSpan` is never recorded, so a test can't see it.
:::

## Capture Log Records

A `LoggerProvider` sends every log record to its `LogRecordProcessor`s, so capturing logs works the same way as capturing spans. The processor receives the whole `LogRecord`, not a formatted line, so a test can assert on each part separately:

```scala
final class LogRecorder extends LogRecordProcessor {
  private val emitted = new ConcurrentLinkedQueue[LogRecord]()

  def onEmit(logRecord: LogRecord): Unit = { emitted.add(logRecord); () }
  def shutdown(): Unit                   = ()
  def forceFlush(): Unit                 = ()

  def records: List[LogRecord] = emitted.asScala.toList
}
```

A `LogWriter` would also work as an output, but it receives the formatted text, and asserting on text brings back every problem from the start of this guide.

This time we record both spans and logs, so we can also check that the log line belongs to the `placeOrder` span:

```scala
val logSpans    = new SpanRecorder
val logRecorder = new LogRecorder

val loggedOrders = new OrderService(
  PaymentGateway.declineAll("card declined"),
  TracerProvider.builder.addSpanProcessor(logSpans).build().get("test"),
  LoggerProvider.builder.addLogRecordProcessor(logRecorder).build().get("test"),
  MeterProvider.builder.build().get("test")
)

val reason = AttributeKey.string("reason")

def placeOrderContext: Option[SpanContext] =
  logSpans.spans.find(_.name == "placeOrder").map(_.spanContext)
```

Place the declined order and read the log record back:

```scala
loggedOrders.placeOrder(Order("order-2", "Programming in Scala", 2, 5000L))
// res6: Either[String, String] = Left("card declined")

logRecorder.records.foreach { r =>
  val inSpan = placeOrderContext.exists { s =>
    r.spanId == s.spanId.value && r.traceIdHi == s.traceIdHi && r.traceIdLo == s.traceIdLo
  }
  println(s"${r.severity} ${r.body.value} reason=${r.attributes.get(reason)}")
  println(s"in placeOrder span: $inSpan")
}
// Error Payment failed reason=Some(card declined)
// in placeOrder span: true
```

The code above:

- `LogRecord#severity` is a `Severity` value, here `Severity.Error`, so a test compares it with `==` instead of parsing a level name.
- `LogRecord#body` holds the message; `body.value` is its text.
- `LogRecord#attributes` holds the fields, read with the same typed keys as span attributes.
- `LogRecord#spanId`, `LogRecord#traceIdHi` and `LogRecord#traceIdLo` identify the span that was active when the line was written. The tracer and the logger share the active span without any setup, so the record carries the ids of `placeOrder`.

Assert on the severity, the message and the fields, never on a formatted line. Formatted lines also carry a timestamp, and in version 0.0.56 the text formatter prints a class inside a package with its full name and two numbers in the location brackets, a known bug in that version.

## Read a Counter's Value

A counter keeps a running total per distinct set of attributes. `Counter#collect` returns a snapshot of those totals as `MetricData.SumData`, with one point per series. We add the points that match the attributes we care about:

```scala
def total(counter: Counter, attributes: (String, String)*): Long =
  counter.collect() match {
    case MetricData.SumData(points) =>
      points.filter { p =>
        attributes.forall { case (key, value) => p.attributes.get(AttributeKey.string(key)).contains(value) }
      }
        .map(_.value)
        .sum
    case _ => 0L
  }
```

We read the counter through its own instance because `MetricData` carries no instrument name. `MeterProvider#reader.collectAllMetrics()` returns the data of every instrument on the provider, and nothing in it says which point belongs to `orders.failed`.

Next, we give a service a meter of its own, so its counters start at zero, and place three orders, one of them declined:

```scala
val countedOrders = new OrderService(
  (orderId, _) => if (orderId == "order-2") Left("card declined") else Right(s"receipt-$orderId"),
  TracerProvider.builder.build().get("test"),
  LoggerProvider.builder.build().get("test"),
  MeterProvider.builder.build().get("test")
)
```

Place the orders and read the totals:

```scala
countedOrders.placeOrder(Order("order-1", "Zionomicon", 1, 4500L))
// res8: Either[String, String] = Right("receipt-order-1")
countedOrders.placeOrder(Order("order-2", "Programming in Scala", 2, 5000L))
// res9: Either[String, String] = Left("card declined")
countedOrders.placeOrder(Order("order-3", "Functional Programming in Scala", 1, 4800L))
// res10: Either[String, String] = Right("receipt-order-3")

println(s"placed: ${total(countedOrders.ordersPlaced)}")
// placed: 2
println(s"placed, Zionomicon: ${total(countedOrders.ordersPlaced, "title" -> "Zionomicon")}")
// placed, Zionomicon: 1
println(s"failed, card declined: ${total(countedOrders.ordersFailed, "reason" -> "card declined")}")
// failed, card declined: 1
```

The gateway here is a lambda that declines only `order-2`. Each counter's total is now exact, because nothing else in the program writes to it. A histogram works the same way: `Histogram#collect` returns `MetricData.HistogramData`, whose points carry a `count`, a `sum` and the bucket counts. The [`MetricData`](../reference/telemetry/metrics/metric-data.md) page describes every point type.

## Give Every Test Its Own Telemetry

ZIO Test runs the tests of a suite in parallel by default. Tests that share one span buffer, one logger or one counter see each other's output, and fail at random depending on timing. The fix is to build the tracer, logger and meter inside each test, so each test owns everything it records. We bundle them into one class:

```scala
final class TestTelemetry {
  val spanRecorder = new SpanRecorder
  val logRecorder  = new LogRecorder

  val tracer: Tracer = TracerProvider.builder.addSpanProcessor(spanRecorder).build().get("test")
  val logger: Logger = LoggerProvider.builder.addLogRecordProcessor(logRecorder).build().get("test")
  val meter: Meter   = MeterProvider.builder.build().get("test")

  def spans: List[SpanData]                    = spanRecorder.spans
  def logs: List[LogRecord]                    = logRecorder.records
  def spanTree: List[(String, Option[String])] = spanRecorder.tree
}
```

To show that two kits don't share anything, we create two:

```scala
val first  = new TestTelemetry
val second = new TestTelemetry
```

Then we run one order through each and print what each kit recorded:

```scala
new OrderService(PaymentGateway.approveAll, first.tracer, first.logger, first.meter)
  .placeOrder(Order("order-1", "Zionomicon", 1, 4500L))
// res14: Either[String, String] = Right("receipt-order-1")
new OrderService(PaymentGateway.declineAll("card declined"), second.tracer, second.logger, second.meter)
  .placeOrder(Order("order-2", "Programming in Scala", 2, 5000L))
// res15: Either[String, String] = Left("card declined")

println(s"first:  ${first.logs.map(_.body.value)} ${first.spans.map(_.status)}")
// first:  List(Order placed) List(Unset, Ok)
println(s"second: ${second.logs.map(_.body.value)} ${second.spans.map(_.status)}")
// second: List(Payment failed) List(Unset, Error(card declined))
```

Each kit holds only the telemetry of its own order. In a spec, every test starts with `val telemetry = new TestTelemetry` and builds the service from it; nothing needs to be reset afterwards, because nothing outlives the test. The complete spec is in [Putting It Together](#putting-it-together).

:::warning
The active span is tracked per call stack and is not passed to other threads. If the code under test starts a span on a plain `Thread` or a thread pool, that span still reaches the recorder, but as the root of a new trace instead of a child of the span that started the work. Assert on the tree only for work that runs on the calling thread.
:::

## Test Code That Uses the Global `log`, `trace` and `metric`

Application code often uses the global objects directly instead of taking a tracer, logger and meter. Such code can still be tested, but every test then shares one span buffer, one logger and every counter, so the tests must reset what they read and must not run at the same time. Here is a checkout written that way:

```scala
object Checkout {
  val ordersFailed: Counter = metric.counter("checkout.orders.failed")

  def placeOrder(order: Order, payments: PaymentGateway): Either[String, String] =
    trace.span("checkout") { span =>
      span.setAttribute("order.id", order.id)
      val charged = payments.charge(order.id, order.totalCents)
      charged match {
        case Right(receipt) =>
          log.info("Order placed", "order.id" -> order.id, "receipt" -> receipt)
          span.setStatus(SpanStatus.Ok)
        case Left(reason) =>
          ordersFailed.add(1L, "reason" -> reason)
          span.setStatus(SpanStatus.Error(reason))
          log.error("Payment failed", "order.id" -> order.id, "reason" -> reason)
      }
      charged
    }
}
```

`Checkout.ordersFailed` is created once, in a `val`. Each call to `metric.counter` registers a new instrument, so a counter created inside `placeOrder` would scatter the count over many instruments and leave the test reading the wrong one.

Each test prepares the three globals in its own way. It empties the global span buffer with `trace.clearSpans()`, installs a logger that records into a fresh `LogRecorder` with `log.install`, and reads the counter's total before the call, because a global counter can't be reset:

```scala
trace.clearSpans()
val globalLogs = new LogRecorder
log.install(LoggerProvider.builder.addLogRecordProcessor(globalLogs).build().get("test"))
val failedBefore = total(Checkout.ordersFailed)
```

Run the checkout and read what it recorded:

```scala
Checkout.placeOrder(Order("order-2", "Programming in Scala", 2, 5000L), PaymentGateway.declineAll("card declined"))
// res20: Either[String, String] = Left("card declined")

println(trace.collectedSpans.map(s => s.name -> s.status))
// List((checkout,Error(card declined)))
println(globalLogs.records.map(r => r.severity -> r.body.value))
// List((Error,Payment failed))
println(s"orders.failed went up by ${total(Checkout.ordersFailed) - failedBefore}")
// orders.failed went up by 1
```

`trace.collectedSpans` returns the spans in the global buffer, which holds only this test's span because we cleared it first. `log.install` replaces the global logger, so the `log.error` call reaches the recorder and nothing is printed.

In a ZIO Test spec, these tests share state, so the suite must run them one at a time. Add `@@ TestAspect.sequential` to the suite, as `GlobalTelemetrySpec` in the companion examples does, and keep every test that touches the globals in that one suite.

:::danger
Never "clean up" the globals with `removeAll` or `install` the wrong way. Each of these breaks every test that runs after it in the same JVM:

- `trace.install(...)` and `trace.removeAll()` disconnect `trace.collectedSpans` for good. Spans go to the new provider (or nowhere, after `removeAll`), and no public call reconnects the in-memory buffer, so later tests read an empty list.
- `log.removeAll()` silences logging, and in version 0.0.56 a later `log.addProcessor` or `log.writer` doesn't bring it back, a known bug in that version. Use `log.install` at the start of each test instead; it replaces the previous test's logger.
:::

Two more limits apply to the global buffer. It holds the last 1,024 spans and drops the oldest, so read it right after the code under test, not at the end of a long suite. And `log.install` replaces the default console output for the rest of the run; if later code should print again, install a logger with a `ConsoleLogRecordProcessor`.

## Putting It Together

The complete spec tests the order service's telemetry for an approved order and a declined one. Every test builds its own `TestTelemetry`, so ZIO Test runs them in parallel. The helpers it uses, `SpanRecorder`, `LogRecorder`, `TestTelemetry` and `Metrics.total`, are in `TestTelemetry.scala` (see [Running the Examples](#running-the-examples)):

```scala title="telemetry-examples/src/test/scala/testtelemetry/OrderServiceSpec.scala"
package testtelemetry

/**
 * How to Test Code That Logs, Traces and Records Metrics: the complete spec.
 *
 * Every test creates its own `TestTelemetry`, so the tests share no state and
 * ZIO Test can run them in parallel.
 *
 * Run with: sbt "testOnly testtelemetry.OrderServiceSpec"
 */
object OrderServiceSpec extends ZIOSpecDefault {
  private val reasonKey = AttributeKey.string("reason")

  private def orderService(payments: PaymentGateway, telemetry: TestTelemetry): OrderService =
    new OrderService(payments, telemetry.tracer, telemetry.logger, telemetry.meter)

  def spec = suite("OrderService telemetry")(
    test("an approved order is traced, logged at INFO and counted per title") {
      val telemetry = new TestTelemetry
      val orders    = orderService(PaymentGateway.approveAll, telemetry)

      val result = orders.placeOrder(Order("order-1", "Zionomicon", 1, 4500L))

      assertTrue(
        result == Right("receipt-order-1"),
        telemetry.spanTree == List("charge" -> Some("placeOrder"), "placeOrder" -> None),
        telemetry.spans.find(_.name == "placeOrder").map(_.status) == Some(SpanStatus.Ok),
        telemetry.logs.map(r => r.severity -> r.body.value) == List(Severity.Info -> "Order placed"),
        Metrics.total(orders.ordersPlaced, "title" -> "Zionomicon") == 1L,
        Metrics.total(orders.ordersFailed) == 0L
      )
    },
    test("a declined payment sets an error status, logs an ERROR and bumps orders.failed") {
      val telemetry = new TestTelemetry
      val orders    = orderService(PaymentGateway.declineAll("card declined"), telemetry)

      val result = orders.placeOrder(Order("order-2", "Programming in Scala", 2, 5000L))

      val error = telemetry.logs.find(_.severity == Severity.Error)
      assertTrue(
        result == Left("card declined"),
        telemetry.spans.find(_.name == "placeOrder").map(_.status) == Some(SpanStatus.Error("card declined")),
        error.map(_.body.value) == Some("Payment failed"),
        error.flatMap(_.attributes.get(reasonKey)) == Some("card declined"),
        Metrics.total(orders.ordersFailed, "reason" -> "card declined") == 1L,
        Metrics.total(orders.ordersPlaced) == 0L
      )
    },
    test("each test sees only its own telemetry") {
      val telemetry = new TestTelemetry
      val orders    = orderService(PaymentGateway.approveAll, telemetry)

      (1 to 3).foreach(i => orders.placeOrder(Order(s"order-$i", "Zionomicon", 1, 4500L)))

      assertTrue(
        telemetry.spans.count(_.name == "placeOrder") == 3,
        telemetry.logs.size == 3,
        Metrics.total(orders.ordersPlaced) == 3L
      )
    }
  )
}
```

## Running the Examples

All the code from this guide is available as runnable examples in the `telemetry-examples` module. They need JDK 21 or newer.

To run the examples, clone the ZIO Blocks repository and navigate to the examples module:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/telemetry-examples
```

Each example below is one file. Expand one to see its code, then run it with the command beneath it.

<details>
<summary>1. The order service and its production wiring</summary>

```scala title="telemetry-examples/src/main/scala/testtelemetry/OrderService.scala" showLineNumbers
package testtelemetry

/** One order: `copies` copies of the book `title`, at `priceCents` each. */
final case class Order(id: String, title: String, copies: Int, priceCents: Long) {
  def totalCents: Long = copies * priceCents
}

/**
 * Charges a customer and returns a receipt id, or the reason the payment was
 * declined.
 */
trait PaymentGateway {
  def charge(orderId: String, amountCents: Long): Either[String, String]
}

object PaymentGateway {
  val approveAll: PaymentGateway = (orderId, _) => Right(s"receipt-$orderId")

  def declineAll(reason: String): PaymentGateway = (_, _) => Left(reason)
}

/**
 * The code under test. It receives its tracer, logger and meter as constructor
 * parameters, so a test can pass in ones that record into memory.
 */
final class OrderService(payments: PaymentGateway, tracer: Tracer, logger: Logger, meter: Meter) {
  val ordersPlaced: Counter = meter.counterBuilder("orders.placed").build()
  val ordersFailed: Counter = meter.counterBuilder("orders.failed").build()

  def placeOrder(order: Order): Either[String, String] =
    tracer.span("placeOrder") { span =>
      span.setAttribute("order.id", order.id)
      span.setAttribute("book.title", order.title)

      val charged = tracer.span("charge") { chargeSpan =>
        chargeSpan.setAttribute("amount.cents", order.totalCents)
        payments.charge(order.id, order.totalCents)
      }

      charged match {
        case Right(receipt) =>
          ordersPlaced.add(1L, "title" -> order.title)
          logger.info(
            "Order placed",
            "order.id" -> AttributeValue.StringValue(order.id),
            "receipt"  -> AttributeValue.StringValue(receipt)
          )
          span.setStatus(SpanStatus.Ok)
        case Left(reason) =>
          ordersFailed.add(1L, "reason" -> reason)
          logger.error(
            "Payment failed",
            "order.id" -> AttributeValue.StringValue(order.id),
            "reason"   -> AttributeValue.StringValue(reason)
          )
          span.setStatus(SpanStatus.Error(reason))
      }
      charged
    }
}

/**
 * How to Test Code That Logs, Traces and Records Metrics: the order service
 * wired for production.
 *
 * The tracer and meter come from the global `trace` and `metric` objects, and
 * the logger prints text lines to standard output.
 *
 * Run with: sbt "runMain testtelemetry.OrderServiceApp"
 */
object OrderServiceApp {
  def main(args: Array[String]): Unit = {
    val logger = LoggerProvider.builder
      .addLogRecordProcessor(new ConsoleLogRecordProcessor)
      .build()
      .get("bookstore.orders")

    def service(payments: PaymentGateway) =
      new OrderService(payments, trace.get("bookstore.orders"), logger, metric.get("bookstore.orders"))

    println(service(PaymentGateway.approveAll).placeOrder(Order("order-1", "Zionomicon", 1, 4500L)))
    println(
      service(PaymentGateway.declineAll("card declined"))
        .placeOrder(Order("order-2", "Programming in Scala", 2, 5000L))
    )
  }
}
```

</details>

This places one approved and one declined order and prints their log lines:

```bash
sbt "runMain testtelemetry.OrderServiceApp"
```

<details>
<summary>2. The in-memory recorders and the per-test kit</summary>

```scala title="telemetry-examples/src/test/scala/testtelemetry/TestTelemetry.scala" showLineNumbers
package testtelemetry

/**
 * A span processor that keeps every finished span in memory, in the order the
 * spans ended.
 */
final class SpanRecorder extends SpanProcessor {
  private val ended = new ConcurrentLinkedQueue[SpanData]()

  def onStart(span: Span): Unit       = ()
  def onEnd(spanData: SpanData): Unit = { ended.add(spanData); () }
  def shutdown(): Unit                = ()
  def forceFlush(): Unit              = ()

  def spans: List[SpanData] = ended.asScala.toList

  /**
   * Each span's name paired with its parent's name, or `None` for a root span.
   */
  def tree: List[(String, Option[String])] = {
    val nameOf = spans.map(s => s.spanContext.spanId -> s.name).toMap
    spans.map(s => s.name -> nameOf.get(s.parentSpanContext.spanId))
  }
}

/**
 * A log record processor that keeps every log record in memory, in the order
 * they were emitted.
 */
final class LogRecorder extends LogRecordProcessor {
  private val emitted = new ConcurrentLinkedQueue[LogRecord]()

  def onEmit(logRecord: LogRecord): Unit = { emitted.add(logRecord); () }
  def shutdown(): Unit                   = ()
  def forceFlush(): Unit                 = ()

  def records: List[LogRecord] = emitted.asScala.toList
}

object Metrics {

  /**
   * The counter's total over the series whose string attributes match
   * `attributes`.
   */
  def total(counter: Counter, attributes: (String, String)*): Long =
    counter.collect() match {
      case MetricData.SumData(points) =>
        points.filter { p =>
          attributes.forall { case (key, value) => p.attributes.get(AttributeKey.string(key)).contains(value) }
        }
          .map(_.value)
          .sum
      case _ => 0L
    }
}

/**
 * A tracer, a logger and a meter that record into memory and belong to one test
 * alone. Create a new one in every test.
 */
final class TestTelemetry {
  val spanRecorder = new SpanRecorder
  val logRecorder  = new LogRecorder

  val tracer: Tracer = TracerProvider.builder.addSpanProcessor(spanRecorder).build().get("test")
  val logger: Logger = LoggerProvider.builder.addLogRecordProcessor(logRecorder).build().get("test")
  val meter: Meter   = MeterProvider.builder.build().get("test")

  def spans: List[SpanData]                    = spanRecorder.spans
  def logs: List[LogRecord]                    = logRecorder.records
  def spanTree: List[(String, Option[String])] = spanRecorder.tree
}
```

</details>

The specs below use these helpers; the file has nothing to run on its own.

<details>
<summary>3. Record spans in memory</summary>

```scala title="telemetry-examples/src/test/scala/testtelemetry/RecordSpansSpec.scala" showLineNumbers
package testtelemetry

/**
 * How to Test Code That Logs, Traces and Records Metrics: assert on spans.
 *
 * Builds a tracer that records into a `SpanRecorder`, runs `placeOrder` and
 * asserts on the span's name, attributes, status and parent.
 *
 * Run with: sbt "testOnly testtelemetry.RecordSpansSpec"
 */
object RecordSpansSpec extends ZIOSpecDefault {
  def spec = suite("placeOrder spans")(
    test("a declined payment ends the placeOrder span with an error status") {
      val recorder = new SpanRecorder
      val tracer   = TracerProvider.builder.addSpanProcessor(recorder).build().get("test")
      val service  = new OrderService(
        PaymentGateway.declineAll("card declined"),
        tracer,
        LoggerProvider.builder.build().get("test"),
        MeterProvider.builder.build().get("test")
      )

      service.placeOrder(Order("order-2", "Programming in Scala", 2, 5000L))

      val placeOrder = recorder.spans.find(_.name == "placeOrder")
      assertTrue(
        placeOrder.map(_.status) == Some(SpanStatus.Error("card declined")),
        placeOrder.flatMap(_.attributes.get(AttributeKey.string("order.id"))) == Some("order-2"),
        placeOrder.flatMap(_.attributes.get(AttributeKey.string("book.title"))) == Some("Programming in Scala")
      )
    },
    test("charge is a child of placeOrder") {
      val recorder = new SpanRecorder
      val tracer   = TracerProvider.builder.addSpanProcessor(recorder).build().get("test")
      val service  = new OrderService(
        PaymentGateway.approveAll,
        tracer,
        LoggerProvider.builder.build().get("test"),
        MeterProvider.builder.build().get("test")
      )

      service.placeOrder(Order("order-1", "Zionomicon", 1, 4500L))

      val charge = recorder.spans.find(_.name == "charge")
      assertTrue(
        recorder.tree == List("charge" -> Some("placeOrder"), "placeOrder" -> None),
        charge.flatMap(_.attributes.get(AttributeKey.long("amount.cents"))) == Some(4500L)
      )
    }
  )
}
```

</details>

Run it with:

```bash
sbt "testOnly testtelemetry.RecordSpansSpec"
```

<details>
<summary>4. Capture log records</summary>

```scala title="telemetry-examples/src/test/scala/testtelemetry/CaptureLogsSpec.scala" showLineNumbers
package testtelemetry

/**
 * How to Test Code That Logs, Traces and Records Metrics: assert on log
 * records.
 *
 * Builds a logger that records into a `LogRecorder`, runs `placeOrder` and
 * asserts on the record's severity, message, fields and span.
 *
 * Run with: sbt "testOnly testtelemetry.CaptureLogsSpec"
 */
object CaptureLogsSpec extends ZIOSpecDefault {
  def spec = suite("placeOrder logs")(
    test("a declined payment logs one ERROR with the order id and the reason") {
      val logs    = new LogRecorder
      val logger  = LoggerProvider.builder.addLogRecordProcessor(logs).build().get("test")
      val service = new OrderService(
        PaymentGateway.declineAll("card declined"),
        TracerProvider.builder.build().get("test"),
        logger,
        MeterProvider.builder.build().get("test")
      )

      service.placeOrder(Order("order-2", "Programming in Scala", 2, 5000L))

      val record = logs.records.head
      assertTrue(
        logs.records.map(r => r.severity -> r.body.value) == List(Severity.Error -> "Payment failed"),
        record.attributes.get(AttributeKey.string("order.id")) == Some("order-2"),
        record.attributes.get(AttributeKey.string("reason")) == Some("card declined")
      )
    },
    test("the log record carries the placeOrder span's ids") {
      val spans   = new SpanRecorder
      val logs    = new LogRecorder
      val service = new OrderService(
        PaymentGateway.approveAll,
        TracerProvider.builder.addSpanProcessor(spans).build().get("test"),
        LoggerProvider.builder.addLogRecordProcessor(logs).build().get("test"),
        MeterProvider.builder.build().get("test")
      )

      service.placeOrder(Order("order-1", "Zionomicon", 1, 4500L))

      val span   = spans.spans.find(_.name == "placeOrder").get.spanContext
      val record = logs.records.head
      assertTrue(
        record.severity == Severity.Info,
        record.spanId == span.spanId.value,
        record.traceIdHi == span.traceIdHi,
        record.traceIdLo == span.traceIdLo
      )
    }
  )
}
```

</details>

Run it with:

```bash
sbt "testOnly testtelemetry.CaptureLogsSpec"
```

<details>
<summary>5. Read a counter's value</summary>

```scala title="telemetry-examples/src/test/scala/testtelemetry/ReadCountersSpec.scala" showLineNumbers
package testtelemetry

/**
 * How to Test Code That Logs, Traces and Records Metrics: assert on a counter.
 *
 * Gives the service a meter of its own, runs `placeOrder` and reads the
 * counters back with `Metrics.total`.
 *
 * Run with: sbt "testOnly testtelemetry.ReadCountersSpec"
 */
object ReadCountersSpec extends ZIOSpecDefault {
  private def service(payments: PaymentGateway): OrderService =
    new OrderService(
      payments,
      TracerProvider.builder.build().get("test"),
      LoggerProvider.builder.build().get("test"),
      MeterProvider.builder.build().get("test")
    )

  def spec = suite("placeOrder counters")(
    test("a declined payment adds one to orders.failed, tagged with the reason") {
      val orders = service(PaymentGateway.declineAll("card declined"))

      orders.placeOrder(Order("order-2", "Programming in Scala", 2, 5000L))

      assertTrue(
        Metrics.total(orders.ordersFailed) == 1L,
        Metrics.total(orders.ordersFailed, "reason" -> "card declined") == 1L,
        Metrics.total(orders.ordersPlaced) == 0L
      )
    },
    test("orders.placed keeps one total per title") {
      val orders = service(PaymentGateway.approveAll)

      orders.placeOrder(Order("order-1", "Zionomicon", 1, 4500L))
      orders.placeOrder(Order("order-3", "Zionomicon", 3, 4500L))
      orders.placeOrder(Order("order-4", "Programming in Scala", 1, 5000L))

      assertTrue(
        Metrics.total(orders.ordersPlaced, "title" -> "Zionomicon") == 2L,
        Metrics.total(orders.ordersPlaced, "title" -> "Programming in Scala") == 1L,
        Metrics.total(orders.ordersFailed) == 0L
      )
    }
  )
}
```

</details>

Run it with:

```bash
sbt "testOnly testtelemetry.ReadCountersSpec"
```

<details>
<summary>6. Test code that uses the global objects</summary>

The code under test:

```scala title="telemetry-examples/src/main/scala/testtelemetry/Checkout.scala" showLineNumbers
package testtelemetry

/**
 * Application code that records telemetry through the global `log`, `trace` and
 * `metric` objects instead of taking a tracer, logger and meter as parameters.
 * `GlobalTelemetrySpec` shows how to test it.
 */
object Checkout {
  // Created once: every call to `metric.counter` registers a new instrument.
  val ordersFailed: Counter = metric.counter("checkout.orders.failed")

  def placeOrder(order: Order, payments: PaymentGateway): Either[String, String] =
    trace.span("checkout") { span =>
      span.setAttribute("order.id", order.id)
      val charged = payments.charge(order.id, order.totalCents)
      charged match {
        case Right(receipt) =>
          log.info("Order placed", "order.id" -> order.id, "receipt" -> receipt)
          span.setStatus(SpanStatus.Ok)
        case Left(reason) =>
          ordersFailed.add(1L, "reason" -> reason)
          span.setStatus(SpanStatus.Error(reason))
          log.error("Payment failed", "order.id" -> order.id, "reason" -> reason)
      }
      charged
    }
}
```

The spec that tests it:

```scala title="telemetry-examples/src/test/scala/testtelemetry/GlobalTelemetrySpec.scala" showLineNumbers
package testtelemetry

/**
 * How to Test Code That Logs, Traces and Records Metrics: code that uses the
 * global `log`, `trace` and `metric` objects.
 *
 * Every test resets the global span store, installs a recording logger and
 * reads the counter's change. The suite runs its tests one at a time, because
 * they share the global state.
 *
 * Run with: sbt "testOnly testtelemetry.GlobalTelemetrySpec"
 */
object GlobalTelemetrySpec extends ZIOSpecDefault {
  def spec = suite("Checkout (global telemetry)")(
    test("a declined payment is traced, logged and counted") {
      trace.clearSpans()
      val logs = new LogRecorder
      log.install(LoggerProvider.builder.addLogRecordProcessor(logs).build().get("test"))
      val failedBefore = Metrics.total(Checkout.ordersFailed)

      Checkout.placeOrder(
        Order("order-2", "Programming in Scala", 2, 5000L),
        PaymentGateway.declineAll("card declined")
      )

      assertTrue(
        trace.collectedSpans.map(s => s.name -> s.status) == List("checkout" -> SpanStatus.Error("card declined")),
        logs.records.map(r => r.severity -> r.body.value) == List(Severity.Error -> "Payment failed"),
        Metrics.total(Checkout.ordersFailed) - failedBefore == 1L
      )
    },
    test("an approved payment is traced and logged, and not counted as failed") {
      trace.clearSpans()
      val logs = new LogRecorder
      log.install(LoggerProvider.builder.addLogRecordProcessor(logs).build().get("test"))
      val failedBefore = Metrics.total(Checkout.ordersFailed)

      Checkout.placeOrder(Order("order-1", "Zionomicon", 1, 4500L), PaymentGateway.approveAll)

      assertTrue(
        trace.collectedSpans.map(s => s.name -> s.status) == List("checkout" -> SpanStatus.Ok),
        logs.records.map(r => r.severity -> r.body.value) == List(Severity.Info -> "Order placed"),
        Metrics.total(Checkout.ordersFailed) - failedBefore == 0L
      )
    }
  ) @@ TestAspect.sequential
}
```

</details>

Run it with:

```bash
sbt "testOnly testtelemetry.GlobalTelemetrySpec"
```

To run every spec, including the complete `OrderServiceSpec`:

```bash
sbt test
```

## Going Further

- **Learn the three signals step by step.** The tutorial [Instrument a Program with Logs, Traces and Metrics](./instrument-a-program-with-logs-traces-and-metrics.md) adds logs, spans, a counter and a histogram to a batch job.
- **Look up the types.** The reference pages for [`SpanProcessor`](../reference/telemetry/tracing/span-processor.md), [`LogRecordProcessor`](../reference/telemetry/logging/log-record-processor.md), [`TracerProvider`](../reference/telemetry/tracing/tracer-provider.md), [`LoggerProvider`](../reference/telemetry/logging/logger-provider.md) and [`MeterProvider`](../reference/telemetry/metrics/meter-provider.md) list every method used here.
- **See how the module works inside.** The [Telemetry Guide](./telemetry-guide.md) explains sampling, span context propagation and what is safe across threads.
- **Test the rest of the service.** [How to Test Database Code Against SQLite](./test-database-code-against-sqlite.md) gives every test its own database in the same per-test style.

---

## How to Tune JSON Encoding and Decoding for Throughput

In this guide, we take a JSON request handler that turns every body into a `String` and back, and make it read and write bytes directly. By the end, you will know which `JsonCodec` entry point to call for an `Array[Byte]`, a `ByteBuffer`, an `InputStream` or an `OutputStream`, how to keep derived codecs from being rebuilt on every call, which `ReaderConfig` and `WriterConfig` settings change speed and memory use (and which ones are safety limits you should leave alone), and how to measure the result with JMH.

We use one running example throughout: a bookshop catalog of a few thousand `Book` records that a service receives and sends as JSON. Each section removes one source of copying or garbage, and the last section shows how to check each change with a benchmark instead of guessing.

This guide covers:

- Decoding from bytes, a `ByteBuffer` and an `InputStream`
- Encoding to bytes, a `ByteBuffer` and an `OutputStream`
- Deriving each codec once and reusing it
- Sizing the reader and writer buffers, and keeping the safety limits
- Measuring with the repository's benchmarks and a minimal JMH harness

## The Problem

Services that move a lot of JSON spend real CPU time and memory on it, and much of that work is avoidable copying. Network libraries hand us bytes and want bytes back, but the most familiar way to call a JSON library is through `String`: decode the body as text, parse the text, render the reply as text, and encode the text as bytes. Each step makes a full copy of the payload, and each copy becomes garbage as soon as the request ends.

For example, a handler that receives a list of books and returns the recent ones often looks like this:

```scala

case class Book(id: Long, title: String, authors: List[String], year: Int, tags: List[String])
object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

// Called once per request with the raw request body
def recentBooks(requestBody: Array[Byte]): Array[Byte] = {
  val text  = new String(requestBody, UTF_8)
  val books = text.fromJson[List[Book]]
  books match {
    case Right(all)  => all.filter(_.year >= 2020).toJsonString.getBytes(UTF_8)
    case Left(error) => error.getMessage.getBytes(UTF_8)
  }
}
```

This code is correct, and on a small payload nobody notices its cost. At a few hundred kilobytes per request it copies the body into a `String`, has the decoder copy that `String` back into UTF-8 bytes, writes the reply into a buffer, copies the buffer into a `String`, and copies that `String` into the reply bytes. It also derives a new codec for `List[Book]` on both `fromJson` and `toJsonString`, on every request, for a reason we look at in [Derive Each Codec Once and Reuse It](#derive-each-codec-once-and-reuse-it).

In this guide, we keep the same `Schema` and change only how we call its codec: bytes in, bytes out, one codec per type, and buffer sizes that match our payloads.

## Prerequisites

We use the schema module from the [Your First Schema](./your-first-schema.md) tutorial; [How to Add ZIO Blocks to Your Build](../getting-started/installation.md) explains which artifact each Scala version needs:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56"
```

This guide assumes you can derive a schema with `Schema.derived` and round-trip a value through JSON, as shown in [Your First Schema](./your-first-schema.md). Every example imports `zio.blocks.schema._` for `Schema` and `zio.blocks.schema.json._` for `JsonCodec`, `ReaderConfig` and `WriterConfig`. Everything in this guide works on the JVM and on Scala.js, except JMH, which runs only on the JVM.

## The Book Catalog Model

Our catalog holds `Book` records. A bookshop lists many copies of a few real Scala titles, so we generate 4,000 listings by cycling through eight books and giving each listing its own `id`:

```scala

case class Book(id: Long, title: String, authors: List[String], year: Int, tags: List[String])
object Book {
  implicit val schema: Schema[Book] = Schema.derived

  val codec: JsonCodec[Book]              = schema.jsonCodec
  val catalogCodec: JsonCodec[List[Book]] = Schema[List[Book]].jsonCodec
}

val classics: List[Book] = List(
  Book(1L, "Zionomicon", List("John A. De Goes", "Adam Fraser", "Milad Khajavi"), 2024, List("zio", "effects")),
  Book(2L, "Programming in Scala", List("Martin Odersky", "Lex Spoon", "Bill Venners"), 2008, List("scala")),
  Book(3L, "Functional Programming in Scala", List("Paul Chiusano", "Rúnar Bjarnason"), 2014, List("fp")),
  Book(4L, "Scala for the Impatient", List("Cay S. Horstmann"), 2012, List("scala")),
  Book(5L, "Hands-on Scala Programming", List("Li Haoyi"), 2020, List("scala", "tooling")),
  Book(6L, "Essential Effects", List("Adam Rosien"), 2021, List("cats-effect", "effects")),
  Book(7L, "Practical FP in Scala", List("Gabriel Volpe"), 2020, List("fp", "cats-effect")),
  Book(8L, "Scala with Cats", List("Noel Welsh", "Dave Gurnell"), 2017, List("cats", "fp"))
)

val catalog: List[Book] =
  List.tabulate(4000)(i => classics(i % classics.size).copy(id = i + 1L))

val catalogBytes: Array[Byte] = Book.catalogCodec.encode(catalog)
```

The companion holds two codecs in `val`s, one for a single `Book` and one for the whole catalog. The rest of the guide calls them directly instead of going through `toJsonString` and `fromJson`. The encoded catalog is a few hundred kilobytes, big enough for the costs in this guide to matter:

```scala
println(catalogBytes.length)
// 466894
```

## Decode From Bytes, a ByteBuffer or an InputStream

`JsonCodec#decode` has an overload for each common input type. All of them return `Either[SchemaError, A]`, and none of them needs a `String`. Pick the one that matches what your network or storage layer already gives you.

### Decode From an Array of Bytes

`JsonCodec#decode(input: Array[Byte])` is the cheapest entry point. The reader parses the array in place: it borrows our array for the duration of the call instead of copying it into a buffer of its own. `JsonCodec#decode(input: String)`, by contrast, first calls `String#getBytes` with `UTF_8` on the string and then decodes the new array, so text input always costs one extra full copy.

To decode the whole catalog straight from its bytes:

```scala
println(Book.catalogCodec.decode(catalogBytes).map(_.size))
// Right(4000)
```

When the JSON sits inside a larger array, such as a message with a length prefix, `JsonCodec#decode(input, from, to)` decodes only the bytes from index `from` up to, but not including, index `to`. There is no need to copy the slice out first. For a frame that starts with a four-byte length:

```scala
val payload: Array[Byte] = Book.codec.encode(classics.head)
val frame: Array[Byte]   = ByteBuffer.allocate(4 + payload.length).putInt(payload.length).put(payload).array()
val length: Int          = ByteBuffer.wrap(frame).getInt(0)
```

The codec reads the book from the frame without touching the prefix:

```scala
println(Book.codec.decode(frame, 4, 4 + length))
// Right(Book(1,Zionomicon,List(John A. De Goes, Adam Fraser, Milad Khajavi),2024,List(zio, effects)))
```

### Decode From a ByteBuffer

`JsonCodec#decode(input: ByteBuffer)` reads the bytes between `ByteBuffer#position` and `ByteBuffer#limit`, and moves the position to the end of what it read. A heap buffer (one with a backing array) is parsed in place, like an array. A direct buffer is copied into the reader's internal buffer in chunks, so it works but costs one copy.

Because the position moves only as far as the value it decoded, one buffer can hold several values back to back. By default the reader rejects anything but whitespace after the value; `ReaderConfig#withCheckForEndOfInput(false)` turns that check off so that we can read one value at a time:

```scala
val twoBooks: ByteBuffer = ByteBuffer.wrap(Book.codec.encode(classics(0)) ++ Book.codec.encode(classics(1)))

val oneValueAtATime: ReaderConfig = ReaderConfig.withCheckForEndOfInput(false)
```

Each call decodes the next book and leaves the buffer positioned after it:

```scala
println(Book.codec.decode(twoBooks, oneValueAtATime).map(_.title))
// Right(Zionomicon)
println(Book.codec.decode(twoBooks, oneValueAtATime).map(_.title))
// Right(Programming in Scala)
println(twoBooks.remaining())
// 0
```

A failed decode also moves the position, to just past the byte where the error was found. If you want to retry or skip the value, or call `ByteBuffer#flip` on the buffer afterwards, save `ByteBuffer#position` before the call and restore it afterwards.

### Decode From an InputStream

`JsonCodec#decode(input: InputStream)` reads the stream in chunks into the reader's internal buffer, which stays at its preferred size (32 KiB by default) for ordinary input. That means a large request body never has to be collected into one array first: the stream goes straight into the decoder. The codec does not close the stream; the code that opened it still owns it.

To decode the catalog from a stream:

```scala
println(Book.catalogCodec.decode(new ByteArrayInputStream(catalogBytes)).map(_.size))
// Right(4000)
```

One stream holds one value. The reader fills its buffer with as much input as the stream offers, so after the first value it has already consumed bytes that belong to the next one, and they are gone when the call returns. Even with `ReaderConfig#checkForEndOfInput` turned off, the second read of a stream holding two books fails:

```scala
val booksStream = new ByteArrayInputStream(Book.codec.encode(classics(0)) ++ Book.codec.encode(classics(1)))
```

The first call returns the first book, and the second finds nothing left to read:

```scala
println(Book.codec.decode(booksStream, oneValueAtATime).map(_.title))
// Right(Zionomicon)
println(Book.codec.decode(booksStream, oneValueAtATime).map(_.title))
// Left(zio.blocks.schema.SchemaError: unexpected end of input at: .)
```

To read several values from one source, read it into a `ByteBuffer` and decode from that, as in the previous section, or wrap the stream's values in a JSON array and decode a `List`.

## Encode to Bytes, a ByteBuffer or an OutputStream

Encoding has the same choice of targets. Unlike `JsonCodec#decode`, the `JsonCodec#encode` methods don't return an `Either`: a value the codec can't write, or a target that is too small, throws an exception.

### Encode to an Array of Bytes

`JsonCodec#encode(value)` writes into the writer's internal buffer and returns a copy trimmed to the exact length, so it makes one copy. `JsonCodec#encodeToString` writes the same bytes and then decodes them into a `String`, and calling `String#getBytes` with `UTF_8` on that string copies them once more. When the result goes to the network, encode to bytes:

```scala
println(Book.catalogCodec.encode(catalog).length)
// 466894
```

### Encode Into a ByteBuffer

`JsonCodec#encode(value, output: ByteBuffer)` writes at `ByteBuffer#position` and moves the position past the JSON. A heap buffer receives the bytes straight into its backing array; a direct buffer receives them in chunks through the writer's internal buffer. Either way, call `ByteBuffer#flip` before reading the buffer back, or the reader starts at the end of the JSON:

```scala
val out = ByteBuffer.allocate(1024)
Book.codec.encode(classics.head, out)
```

The position now sits after the JSON, so a reader that starts there finds only the empty rest of the buffer. We decode a `ByteBuffer#duplicate`, which has its own position, to show the mistake without disturbing `out`:

```scala
println(out.position())
// 126
println(Book.codec.decode(out.duplicate()))
// Left(zio.blocks.schema.SchemaError: expected '{' at: .)
```

After `ByteBuffer#flip`, the limit is the end of the JSON and the position is back at the start:

```scala
out.flip()
```

Now the same buffer decodes to the book:

```scala
println(Book.codec.decode(out).map(_.title))
// Right(Zionomicon)
```

When the buffer is too small, `JsonCodec#encode` throws `java.nio.BufferOverflowException`. By then it has already written part of the JSON and moved the position past it, so the buffer holds a truncated value:

```scala
val tooSmall = ByteBuffer.allocate(16)
```

The exception arrives after six bytes are already in the buffer:

```scala
try Book.codec.encode(classics.head, tooSmall)
catch { case _: BufferOverflowException => println(s"overflow at position ${tooSmall.position()}") }
// overflow at position 6
```

To keep the buffer usable, save its position before encoding and restore it when you catch the exception:

```scala
def encodeOrRewind(book: Book, buffer: ByteBuffer): Boolean = {
  val start = buffer.position()
  try {
    Book.codec.encode(book, buffer)
    true
  } catch {
    case _: BufferOverflowException =>
      buffer.position(start)
      false
  }
}

val small = ByteBuffer.allocate(16)
```

The failed write now leaves the buffer exactly as it was:

```scala
println(encodeOrRewind(classics.head, small))
// false
println(small.position())
// 0
```

### Encode to an OutputStream

`JsonCodec#encode(value, output: OutputStream)` writes the JSON in chunks through the writer's internal buffer, so a large reply never exists as one array. The codec neither closes nor flushes the stream. If you wrap the stream in a `BufferedOutputStream`, or your server buffers the response body, call `OutputStream#flush` when you are done, or the bytes may never leave the buffer:

```scala
val sink     = new ByteArrayOutputStream()
val response = new BufferedOutputStream(sink, 1024 * 1024)
```

With a buffer larger than the reply, nothing reaches the underlying stream until we flush:

```scala
Book.catalogCodec.encode(catalog, response)
println(sink.size())
// 0

response.flush()
println(sink.size())
// 466894
```

## Derive Each Codec Once and Reuse It

Deriving a codec walks the whole schema and builds a tree of codec objects, one per type and field. That is far more work than decoding a small record, so a codec should be built once and reused, not rebuilt per request. Codecs are safe to share between threads: each thread gets its own reader and writer from a per-thread pool.

`Schema#jsonCodec` caches the codec it derives, but only on that `Schema` instance. `toJsonString`, `toJsonBytes` and `fromJson` all go through `Schema#jsonCodec` on whatever schema the compiler finds. For `Book`, that is the `implicit val` in its companion, so the cache works. For `List[Book]`, `Option[Book]`, `Vector[Book]` or `Map[String, Book]`, the compiler builds a new schema from the element's schema at each call site, so every call starts with an empty cache. `Schema#derive` never caches. We can check this by comparing instances:

```scala
val derivedOnce: JsonCodec[Book]  = Book.schema.derive(JsonFormat)
val derivedTwice: JsonCodec[Book] = Book.schema.derive(JsonFormat)
```

The first comparison below is the cached `Book` codec, the second the summoned `List[Book]` schema, and the third two calls of `Schema#derive`:

```scala
println(Book.schema.jsonCodec eq Book.schema.jsonCodec)
// true
println(Schema[List[Book]].jsonCodec eq Schema[List[Book]].jsonCodec)
// false
println(derivedOnce eq derivedTwice)
// false
```

The second and third lines print `false`: each of those expressions derives a new codec every time it runs. That is why `fromJson[List[Book]]` and `toJsonString` on a `List[Book]` in [The Problem](#the-problem) pay for two derivations per request.

The fix is to derive each codec you use on a hot path once and keep it in a `val`, as the `Book` companion in [The Book Catalog Model](#the-book-catalog-model) does with `Book.codec` and `Book.catalogCodec`. The same applies to a codec with custom settings from `JsonCodecDeriver`, as in [How to Customize JSON Field Names, ADT Encoding and Optional Fields](./customize-json-encoding.md):

```scala
object CatalogCodecs {
  val snakeCaseCatalog: JsonCodec[List[Book]] =
    Schema[List[Book]].derive(JsonCodecDeriver.withFieldNameMapper(NameMapper.SnakeCase))
}
```

Calling `CatalogCodecs.snakeCaseCatalog` returns the same codec every time; only its first use pays for the derivation.

## Tune ReaderConfig and WriterConfig

Every `JsonCodec#decode` overload has a twin that takes a `ReaderConfig`, and every `JsonCodec#encode` overload has a twin that takes a `WriterConfig`. These two classes control the reader's and writer's buffers and a few output options. They are separate from `JsonCodecDeriver`, which decides the shape of the JSON (field names, discriminators, empty collections) when a codec is derived.

Both classes are immutable. The defaults are the companion objects themselves, so `ReaderConfig` and `WriterConfig` can be passed wherever a config is expected, and each `with...` method returns a new config. Build tuned configs once, in `val`s, next to the codecs that use them. These are all the settings and their defaults:

| Setting                                  | Default             | Applies to                                       | Purpose                                                                    |
| ---------------------------------------- | ------------------- | ------------------------------------------------ | -------------------------------------------------------------------------- |
| `ReaderConfig#preferredBufSize`          | 32768 bytes         | `InputStream` and `ByteBuffer` input             | The size the reader's byte buffer starts at and shrinks back to            |
| `ReaderConfig#maxBufSize`                | 33554432 bytes      | `InputStream` and `ByteBuffer` input             | Safety limit: the largest that buffer may grow                             |
| `ReaderConfig#preferredCharBufSize`      | 4096 chars          | All input                                        | The size the buffer for string values starts at and shrinks back to        |
| `ReaderConfig#maxCharBufSize`            | 4194304 chars       | All input                                        | Safety limit: the longest string value the reader accepts                  |
| `ReaderConfig#checkForEndOfInput`        | `true`              | All input                                        | Rejects anything but whitespace after the value                            |
| `WriterConfig#preferredBufSize`          | 32768 bytes         | All output except a heap `ByteBuffer`            | The size the writer's byte buffer starts at and shrinks back to            |
| `WriterConfig#indentionStep`             | 0                   | All output                                       | Spaces per indentation level; 0 writes compact JSON                        |
| `WriterConfig#escapeUnicode`             | `false`             | All output                                       | Writes every non-ASCII character as an escape sequence                     |

### Size the Writer Buffer for Large Outputs

Each thread keeps one writer with a buffer of `WriterConfig#preferredBufSize` bytes. When a value needs more room, the writer doubles the buffer until the value fits, and when the call ends it replaces the grown buffer with a new one of the preferred size. A service that routinely writes replies larger than 32 KiB therefore allocates and copies several buffers on every call. Raising `WriterConfig#preferredBufSize` to the size of a typical reply lets the buffer stay large:

```scala
val largeReplies: WriterConfig = WriterConfig.withPreferredBufSize(1024 * 1024)
```

The output is the same; only the buffer handling changes:

```scala
println(Book.catalogCodec.encode(catalog, largeReplies).sameElements(catalogBytes))
// true
```

The trade-off is memory: every thread that encodes JSON keeps a buffer of that size for as long as it lives. Encoding into a heap `ByteBuffer` ignores this setting, because the writer uses the buffer's own array.

### Size the Reader Buffers for Streams and Long Strings

The reader has two buffers. Its byte buffer is used only for `InputStream` and `ByteBuffer` input; array and `String` input is parsed in place. It grows past `ReaderConfig#preferredBufSize` only when the reader must hold one part of the input whole: a very long number, or a sealed-trait object written with a discriminator field (`DiscriminatorKind.Field`) whose discriminator isn't the first field, because the reader can't pick the case until it has seen the discriminator. Its char buffer holds the string value being decoded and grows past `ReaderConfig#preferredCharBufSize` for any string longer than 4,096 characters, whatever the input type.

As with the writer, a grown buffer is replaced by a preferred-size one after each call. If your documents regularly contain long strings, such as book descriptions of 10,000 characters, raise the preferred char buffer size:

```scala
val longDescriptions: ReaderConfig = ReaderConfig.withPreferredCharBufSize(16 * 1024)
```

Each setting is validated when you set it. A preferred size can't exceed its maximum, so to raise both, raise the maximum first:

```scala
scala.util.Try(ReaderConfig.withPreferredBufSize(64 * 1024 * 1024)).failed.foreach(e => println(e.getMessage))
// 'preferredBufSize' should be not greater than 'maxBufSize'

println(ReaderConfig.withMaxBufSize(64 * 1024 * 1024).withPreferredBufSize(64 * 1024 * 1024).preferredBufSize)
// 67108864
```

Use one config per kind of call. The pooled reader and writer resize to the preferred sizes of whichever config the current call passes, so alternating two configs with different sizes on one thread reallocates the buffers on every switch.

### Keep the Safety Limits

`ReaderConfig#maxBufSize` and `ReaderConfig#maxCharBufSize` limit how much memory one request can make the reader allocate. Without them, a client could send a 2 GB string value, or an object whose discriminator field comes after gigabytes of other fields, and the reader would try to buffer all of it. When a limit is hit, `JsonCodec#decode` returns a `Left` instead of allocating more. To see one fire, we decode a book with a 5,000-character title under a 2,048-character limit:

```scala
val strict: ReaderConfig = ReaderConfig.withPreferredCharBufSize(1024).withMaxCharBufSize(2048)

val longTitle: Array[Byte] = Book.codec.encode(classics.head.copy(title = "Z" * 5000))
```

The reader stops at the title and names the limit:

```scala
println(Book.codec.decode(longTitle, strict))
// Left(zio.blocks.schema.SchemaError: too long string exceeded 'maxCharBufSize' at: .title)
```

Raise these limits only for input you trust, and only as far as your largest legitimate value needs; for a public endpoint, lowering them is the safer change. Keep `ReaderConfig#maxBufSize` at 32768 bytes or more, though: each thread's reader starts with a 32 KiB byte buffer, and in 0.0.56 a smaller limit is applied only when an earlier call on the same thread has shrunk that buffer (a known bug), so the same request can pass on one thread and fail on another. `ReaderConfig#maxBufSize` doesn't apply to array or `String` input, which is already fully in memory, so limit the size of request bodies in your server before you read them into an array.

Leave `ReaderConfig#checkForEndOfInput` on for request bodies. It costs little, and it turns a body with trailing garbage, such as two JSON documents pasted together, into an error instead of silently decoding the first one. Turn it off only when you deliberately read several values from one `ByteBuffer`, as shown in [Decode From a ByteBuffer](#decode-from-a-bytebuffer).

### Keep Production Output Compact

`WriterConfig#withIndentionStep` pretty-prints JSON with that many spaces per level, and `WriterConfig#withEscapeUnicode` writes every non-ASCII character as an ASCII escape sequence of at least six bytes, instead of its two to four UTF-8 bytes. Both make the output larger, and every extra byte is written, sent and parsed again on the other side. We can compare the sizes for *Functional Programming in Scala*, whose author list contains a non-ASCII `ú`:

```scala
val fpInScala = classics(2)
```

The three sizes are the default compact output, two-space indentation and escaped Unicode:

```scala
println(Book.codec.encode(fpInScala).length)
// 123
println(Book.codec.encode(fpInScala, WriterConfig.withIndentionStep(2)).length)
// 165
println(Book.codec.encode(fpInScala, WriterConfig.withEscapeUnicode(true)).length)
// 127
```

Keep the default compact output for traffic between services. Use indentation for logs, files and debugging, and escaping only when the receiver can't read UTF-8.

## Measure the Difference With JMH

The only reliable way to know whether a change helps is to measure it on your own data and hardware. [JMH](https://github.com/openjdk/jmh), the Java Microbenchmark Harness, runs code long enough for the JIT compiler to settle, repeats it in separate forked JVMs, and reports operations per second with an error margin. A loop timed with `System.nanoTime` does none of that, and its numbers are mostly noise.

### Run the Repository's JSON Benchmarks

The ZIO Blocks repository has a `benchmarks` project with JMH benchmarks for its JSON codec. It runs on the JVM with Scala 3.9.0 only. Four of its benchmarks measure JSON:

| Benchmark                    | What it measures                                                       | Compared with                             |
| ---------------------------- | ---------------------------------------------------------------------- | ----------------------------------------- |
| `JsonListOfRecordsBenchmark` | Reading and writing a list of `size` flat records                      | jsoniter-scala, zio-json, zio-schema-json |
| `JsonNestedRecordsBenchmark` | Reading, writing and failing to read records nested `size` levels deep | Nothing (zio-blocks only)                 |
| `JsonListOfJsonsBenchmark`   | Reading and writing a list of `Json` values                            | zio-json                                  |
| `MaybeBoxingBenchmark`       | Encoding and decoding an `Option[String]` against a `Maybe[String]`    | Nothing (zio-blocks only)                 |

The zio-blocks and jsoniter-scala benchmarks read from and write to `Array[Byte]`; the zio-json and zio-schema-json ones go through a `String`. To run the list-of-records benchmark from a clone of the repository:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
sbt "++3.9.0; benchmarks/Jmh/run -wi 5 -i 5 -f 1 -w 1s -r 1s -p size=1000 zio.blocks.schema.json.JsonListOfRecordsBenchmark"
```

The flags set five warm-up and five measured iterations of one second each, one forked JVM, and a list of 1,000 records (the `size` parameter takes 1, 10, 100, 1000, 10000 or 100000). The forked JVM starts with a 3 GB heap, so close other memory-hungry programs first. Add `-prof gc` to report the bytes allocated per operation next to the throughput; for the changes in this guide, the allocation rate is often the clearer signal.

The repository doesn't publish results, and you shouldn't trust anyone else's: run the benchmarks on the machine and JDK you deploy to.

### Write a Minimal Benchmark for Your Own Types

To measure your own payloads, add the [sbt-jmh](https://github.com/sbt/sbt-jmh) plugin to a separate benchmark project and write a class like the one below. It builds the input once, when JMH creates the state object for each benchmark thread, so each benchmark method measures only the call we care about, and it returns each result so that the JIT compiler can't remove the work. JMH refuses benchmark classes in the default package, so the class lives in a package, next to the `Book` model from [The Book Catalog Model](#the-book-catalog-model). This is a JMH class for an sbt-jmh project, not part of the companion examples:

```scala
package bookshop

@State(Scope.Thread)
@BenchmarkMode(Array(Mode.Throughput))
@OutputTimeUnit(TimeUnit.SECONDS)
@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@Fork(1)
class CatalogBenchmark {
  val catalog: List[Book] =
    List.tabulate(4000)(i => Book(i.toLong, "Zionomicon", List("John A. De Goes"), 2024, List("zio")))
  val bytes: Array[Byte] = Book.catalogCodec.encode(catalog)
  val text: String       = new String(bytes, UTF_8)

  @Benchmark
  def decodeFromBytes(): Either[SchemaError, List[Book]] = Book.catalogCodec.decode(bytes)

  @Benchmark
  def decodeFromString(): Either[SchemaError, List[Book]] = Book.catalogCodec.decode(text)

  @Benchmark
  def encodeToBytes(): Array[Byte] = Book.catalogCodec.encode(catalog)

  @Benchmark
  def encodeThroughString(): Array[Byte] = Book.catalogCodec.encodeToString(catalog).getBytes(UTF_8)
}
```

Run it with `sbt "Jmh/run -prof gc CatalogBenchmark"` from the benchmark project. For a quick experiment without an sbt project, scala-cli runs the same file with `scala-cli --power run --jmh <file> -- -prof gc CatalogBenchmark`. Benchmark one change at a time, against a baseline, and keep the payload close to what your service really sees: the gap between two entry points depends heavily on payload size.

## Putting It Together

The complete example is a catalog endpoint built from every technique in this guide. It keeps its codecs and configs in `val`s, decodes the request body straight from an `InputStream` with tight safety limits, encodes the reply to an `OutputStream` with a writer buffer sized for the reply and flushes it, and reuses one heap `ByteBuffer` for small messages, restoring its position after an overflow:

```scala title="schema-examples/src/main/scala/tunejsonthroughput/CompleteExample.scala"
package tunejsonthroughput

/**
 * Tune JSON Throughput: Putting It Together
 *
 * A catalog endpoint that keeps its codecs and configs in vals, decodes the
 * request straight from an InputStream with tight limits, encodes the reply to
 * an OutputStream with a sized writer buffer, and reuses one heap ByteBuffer
 * for small messages.
 *
 * Run with: cd schema-examples && sbt "runMain
 * tunejsonthroughput.CompleteExample"
 */
object CompleteExample extends App {

  case class Book(id: Long, title: String, authors: List[String], year: Int, tags: List[String])
  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  case class ErrorReply(message: String)
  object ErrorReply {
    implicit val schema: Schema[ErrorReply] = Schema.derived
  }

  object CatalogEndpoint {
    val codec: JsonCodec[Book]              = Book.schema.jsonCodec
    val catalogCodec: JsonCodec[List[Book]] = Schema[List[Book]].jsonCodec
    val errorCodec: JsonCodec[ErrorReply]   = ErrorReply.schema.jsonCodec

    // Tight limits for untrusted request bodies
    val requestConfig: ReaderConfig = ReaderConfig.withPreferredCharBufSize(1024).withMaxCharBufSize(64 * 1024)

    // A writer buffer sized for a typical reply
    val replyConfig: WriterConfig = WriterConfig.withPreferredBufSize(1024 * 1024)

    def handle(request: InputStream, response: OutputStream): Unit = {
      catalogCodec.decode(request, requestConfig) match {
        case Right(books) => catalogCodec.encode(books.filter(_.year >= 2020), response, replyConfig)
        case Left(error)  => errorCodec.encode(ErrorReply(error.getMessage), response, replyConfig)
      }
      response.flush()
    }

    // Reuse one heap buffer for small messages, restoring its position on overflow
    def encodeOrRewind(book: Book, buffer: ByteBuffer): Boolean = {
      val start = buffer.position()
      try {
        codec.encode(book, buffer)
        true
      } catch {
        case _: BufferOverflowException =>
          buffer.position(start)
          false
      }
    }
  }

  val classics: List[Book] = List(
    Book(1L, "Zionomicon", List("John A. De Goes", "Adam Fraser", "Milad Khajavi"), 2024, List("zio", "effects")),
    Book(2L, "Programming in Scala", List("Martin Odersky", "Lex Spoon", "Bill Venners"), 2008, List("scala")),
    Book(3L, "Functional Programming in Scala", List("Paul Chiusano", "Rúnar Bjarnason"), 2014, List("fp")),
    Book(4L, "Scala for the Impatient", List("Cay S. Horstmann"), 2012, List("scala")),
    Book(5L, "Hands-on Scala Programming", List("Li Haoyi"), 2020, List("scala", "tooling")),
    Book(6L, "Essential Effects", List("Adam Rosien"), 2021, List("cats-effect", "effects")),
    Book(7L, "Practical FP in Scala", List("Gabriel Volpe"), 2020, List("fp", "cats-effect")),
    Book(8L, "Scala with Cats", List("Noel Welsh", "Dave Gurnell"), 2017, List("cats", "fp"))
  )

  val catalog: List[Book] = List.tabulate(4000)(i => classics(i % classics.size).copy(id = i + 1L))

  def serve(body: Array[Byte]): Array[Byte] = {
    val sink     = new ByteArrayOutputStream()
    val response = new BufferedOutputStream(sink, 1024 * 1024)
    CatalogEndpoint.handle(new ByteArrayInputStream(body), response)
    sink.toByteArray
  }

  // A valid catalog: keep the books published in 2020 or later
  val reply = serve(CatalogEndpoint.catalogCodec.encode(catalog))
  println(s"Reply: ${reply.length} bytes, ${CatalogEndpoint.catalogCodec.decode(reply).map(_.size)} books")

  // A malformed body gets a small JSON error
  println(new String(serve("""[{"id":1,"title":"Zionomicon"""".getBytes("UTF-8")), "UTF-8"))

  // A 100,000-character title exceeds the string limit
  val hugeTitle = CatalogEndpoint.catalogCodec.encode(List(classics.head.copy(title = "Z" * 100000)))
  println(new String(serve(hugeTitle), "UTF-8"))

  // One heap buffer for small messages
  val buffer = ByteBuffer.allocate(512)
  classics
    .take(3)
    .foreach(book => println(s"${book.title}: ${CatalogEndpoint.encodeOrRewind(book, buffer)} at ${buffer.position()}"))

  val bigBook = classics.head.copy(title = "Z" * 600)
  println(
    s"${bigBook.title.length}-char title: ${CatalogEndpoint.encodeOrRewind(bigBook, buffer)} at ${buffer.position()}"
  )
}
```

## Running the Examples

All examples in this guide have corresponding runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

To compile every example without running one, use `sbt compile`. The section examples share the catalog model in `BookCatalog.scala`; `CompleteExample` defines its own model and runs on its own.

<details>
  <summary>schema-examples/src/main/scala/tunejsonthroughput/BookCatalog.scala</summary>

```scala title="schema-examples/src/main/scala/tunejsonthroughput/BookCatalog.scala" showLineNumbers
package tunejsonthroughput

/**
 * Tune JSON Throughput: The Book Catalog Model
 *
 * The shared Book model, its codecs and a 4,000-book catalog used by the
 * section examples. It has no main method.
 */
case class Book(id: Long, title: String, authors: List[String], year: Int, tags: List[String])
object Book {
  implicit val schema: Schema[Book] = Schema.derived

  val codec: JsonCodec[Book]              = schema.jsonCodec
  val catalogCodec: JsonCodec[List[Book]] = Schema[List[Book]].jsonCodec
}

object BookCatalog {
  val classics: List[Book] = List(
    Book(1L, "Zionomicon", List("John A. De Goes", "Adam Fraser", "Milad Khajavi"), 2024, List("zio", "effects")),
    Book(2L, "Programming in Scala", List("Martin Odersky", "Lex Spoon", "Bill Venners"), 2008, List("scala")),
    Book(3L, "Functional Programming in Scala", List("Paul Chiusano", "Rúnar Bjarnason"), 2014, List("fp")),
    Book(4L, "Scala for the Impatient", List("Cay S. Horstmann"), 2012, List("scala")),
    Book(5L, "Hands-on Scala Programming", List("Li Haoyi"), 2020, List("scala", "tooling")),
    Book(6L, "Essential Effects", List("Adam Rosien"), 2021, List("cats-effect", "effects")),
    Book(7L, "Practical FP in Scala", List("Gabriel Volpe"), 2020, List("fp", "cats-effect")),
    Book(8L, "Scala with Cats", List("Noel Welsh", "Dave Gurnell"), 2017, List("cats", "fp"))
  )

  val catalog: List[Book] =
    List.tabulate(4000)(i => classics(i % classics.size).copy(id = i + 1L))

  val catalogBytes: Array[Byte] = Book.catalogCodec.encode(catalog)
}
```

</details>

### The String Baseline

This example runs the `String`-based handler from [The Problem](#the-problem) and the byte-based handler that replaces it, and checks that both return the same reply:

<details>
  <summary>schema-examples/src/main/scala/tunejsonthroughput/StringBaselineExample.scala</summary>

```scala title="schema-examples/src/main/scala/tunejsonthroughput/StringBaselineExample.scala" showLineNumbers
package tunejsonthroughput

/**
 * Tune JSON Throughput: The String Baseline
 *
 * Runs the String-based handler and the byte-based handler that replaces it,
 * and checks that both return the same reply.
 *
 * Run with: cd schema-examples && sbt "runMain
 * tunejsonthroughput.StringBaselineExample"
 */
object StringBaselineExample extends App {

  // Called once per request with the raw request body
  def recentBooks(requestBody: Array[Byte]): Array[Byte] = {
    val text  = new String(requestBody, UTF_8)
    val books = text.fromJson[List[Book]]
    books match {
      case Right(all)  => all.filter(_.year >= 2020).toJsonString.getBytes(UTF_8)
      case Left(error) => error.getMessage.getBytes(UTF_8)
    }
  }

  // Bytes in, bytes out, one codec for the whole catalog
  def recentBooksFromBytes(requestBody: Array[Byte]): Array[Byte] =
    Book.catalogCodec.decode(requestBody) match {
      case Right(all)  => Book.catalogCodec.encode(all.filter(_.year >= 2020))
      case Left(error) => error.getMessage.getBytes(UTF_8)
    }

  val stringReply = recentBooks(catalogBytes)
  val bytesReply  = recentBooksFromBytes(catalogBytes)

  println(s"Request body: ${catalogBytes.length} bytes")
  println(s"String-based reply: ${stringReply.length} bytes")
  println(s"Byte-based reply: ${bytesReply.length} bytes")
  println(s"Same reply: ${stringReply.sameElements(bytesReply)}")
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain tunejsonthroughput.StringBaselineExample"
```

### Decode From Bytes, a ByteBuffer or an InputStream

This example decodes the catalog from an array, a slice of a framed array, a heap and a direct `ByteBuffer` holding two books, and an `InputStream`:

<details>
  <summary>schema-examples/src/main/scala/tunejsonthroughput/DecodeFromBytesExample.scala</summary>

```scala title="schema-examples/src/main/scala/tunejsonthroughput/DecodeFromBytesExample.scala" showLineNumbers
package tunejsonthroughput

/**
 * Tune JSON Throughput: Decode From Bytes, a ByteBuffer or an InputStream
 *
 * Decodes from an array, a slice of a framed array, heap and direct ByteBuffers
 * holding two books, and an InputStream.
 *
 * Run with: cd schema-examples && sbt "runMain
 * tunejsonthroughput.DecodeFromBytesExample"
 */
object DecodeFromBytesExample extends App {

  // From an array of bytes
  println(Book.catalogCodec.decode(catalogBytes).map(_.size))

  // From a slice of a length-prefixed frame
  val payload: Array[Byte] = Book.codec.encode(classics.head)
  val frame: Array[Byte]   = ByteBuffer.allocate(4 + payload.length).putInt(payload.length).put(payload).array()
  val length: Int          = ByteBuffer.wrap(frame).getInt(0)

  println(Book.codec.decode(frame, 4, 4 + length))

  // From a heap ByteBuffer holding two books back to back
  val oneValueAtATime: ReaderConfig = ReaderConfig.withCheckForEndOfInput(false)

  val twoBytes: Array[Byte] = Book.codec.encode(classics(0)) ++ Book.codec.encode(classics(1))
  val twoBooks: ByteBuffer  = ByteBuffer.wrap(twoBytes)

  println(Book.codec.decode(twoBooks, oneValueAtATime).map(_.title))
  println(Book.codec.decode(twoBooks, oneValueAtATime).map(_.title))
  println(twoBooks.remaining())

  // From a direct ByteBuffer holding the same two books
  val twoDirect: ByteBuffer = ByteBuffer.allocateDirect(twoBytes.length)
  twoDirect.put(twoBytes)
  twoDirect.flip()

  println(Book.codec.decode(twoDirect, oneValueAtATime).map(_.title))
  println(Book.codec.decode(twoDirect, oneValueAtATime).map(_.title))
  println(twoDirect.remaining())

  // From an InputStream
  println(Book.catalogCodec.decode(new ByteArrayInputStream(catalogBytes)).map(_.size))

  // One stream holds one value: the second read finds nothing left
  val booksStream = new ByteArrayInputStream(twoBytes)

  println(Book.codec.decode(booksStream, oneValueAtATime).map(_.title))
  println(Book.codec.decode(booksStream, oneValueAtATime).map(_.title))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain tunejsonthroughput.DecodeFromBytesExample"
```

### Encode to Bytes, a ByteBuffer or an OutputStream

This example encodes to an array, into a `ByteBuffer` (including recovering from an overflow), and to a buffered `OutputStream` that it flushes:

<details>
  <summary>schema-examples/src/main/scala/tunejsonthroughput/EncodeToBytesExample.scala</summary>

```scala title="schema-examples/src/main/scala/tunejsonthroughput/EncodeToBytesExample.scala" showLineNumbers
package tunejsonthroughput

/**
 * Tune JSON Throughput: Encode to Bytes, a ByteBuffer or an OutputStream
 *
 * Encodes to an array, into a ByteBuffer (including recovering from an
 * overflow), and to a buffered OutputStream that it flushes.
 *
 * Run with: cd schema-examples && sbt "runMain
 * tunejsonthroughput.EncodeToBytesExample"
 */
object EncodeToBytesExample extends App {

  // To an array of bytes
  println(Book.catalogCodec.encode(catalog).length)

  // Into a ByteBuffer: reading before flip() starts after the JSON, reading after it returns the book
  val out = ByteBuffer.allocate(1024)
  Book.codec.encode(classics.head, out)

  println(out.position())
  println(Book.codec.decode(out.duplicate()))

  out.flip()
  println(Book.codec.decode(out).map(_.title))

  // A buffer that is too small throws and leaves a truncated value behind
  val tooSmall = ByteBuffer.allocate(16)

  try Book.codec.encode(classics.head, tooSmall)
  catch { case _: BufferOverflowException => println(s"overflow at position ${tooSmall.position()}") }

  // Save the position and restore it on overflow
  def encodeOrRewind(book: Book, buffer: ByteBuffer): Boolean = {
    val start = buffer.position()
    try {
      Book.codec.encode(book, buffer)
      true
    } catch {
      case _: BufferOverflowException =>
        buffer.position(start)
        false
    }
  }

  val small = ByteBuffer.allocate(16)

  println(encodeOrRewind(classics.head, small))
  println(small.position())

  // To an OutputStream: nothing reaches the sink until we flush
  val sink     = new ByteArrayOutputStream()
  val response = new BufferedOutputStream(sink, 1024 * 1024)

  Book.catalogCodec.encode(catalog, response)
  println(sink.size())

  response.flush()
  println(sink.size())
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain tunejsonthroughput.EncodeToBytesExample"
```

### Derive Each Codec Once and Reuse It

This example shows which expressions return a cached codec and which derive a new one on every call:

<details>
  <summary>schema-examples/src/main/scala/tunejsonthroughput/ReuseCodecExample.scala</summary>

```scala title="schema-examples/src/main/scala/tunejsonthroughput/ReuseCodecExample.scala" showLineNumbers
package tunejsonthroughput

/**
 * Tune JSON Throughput: Derive Each Codec Once and Reuse It
 *
 * Shows which expressions return a cached codec and which derive a new one on
 * every call.
 *
 * Run with: cd schema-examples && sbt "runMain
 * tunejsonthroughput.ReuseCodecExample"
 */
object ReuseCodecExample extends App {

  println(Book.schema.jsonCodec eq Book.schema.jsonCodec)
  println(Schema[List[Book]].jsonCodec eq Schema[List[Book]].jsonCodec)

  // Schema#derive never caches: each call builds a new codec
  val derivedOnce: JsonCodec[Book]  = Book.schema.derive(JsonFormat)
  val derivedTwice: JsonCodec[Book] = Book.schema.derive(JsonFormat)
  println(derivedOnce eq derivedTwice)

  object CatalogCodecs {
    val snakeCaseCatalog: JsonCodec[List[Book]] =
      Schema[List[Book]].derive(JsonCodecDeriver.withFieldNameMapper(NameMapper.SnakeCase))
  }

  println(CatalogCodecs.snakeCaseCatalog eq CatalogCodecs.snakeCaseCatalog)
  println(new String(CatalogCodecs.snakeCaseCatalog.encode(catalog.take(1)), "UTF-8").take(80))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain tunejsonthroughput.ReuseCodecExample"
```

### Tune ReaderConfig and WriterConfig

This example builds tuned reader and writer configs, shows the validation order for buffer sizes, triggers both safety limits, and compares the size of compact, indented and escaped output:

<details>
  <summary>schema-examples/src/main/scala/tunejsonthroughput/TuneConfigExample.scala</summary>

```scala title="schema-examples/src/main/scala/tunejsonthroughput/TuneConfigExample.scala" showLineNumbers
package tunejsonthroughput

/**
 * Tune JSON Throughput: Tune ReaderConfig and WriterConfig
 *
 * Builds tuned reader and writer configs, shows the validation order for buffer
 * sizes, triggers both safety limits, and compares compact, indented and
 * escaped output.
 *
 * Run with: cd schema-examples && sbt "runMain
 * tunejsonthroughput.TuneConfigExample"
 */
object TuneConfigExample extends App {

  // A bigger writer buffer: same bytes
  val largeReplies: WriterConfig = WriterConfig.withPreferredBufSize(1024 * 1024)

  println(Book.catalogCodec.encode(catalog, largeReplies).sameElements(catalogBytes))

  // A bigger char buffer for long strings
  val longDescriptions: ReaderConfig = ReaderConfig.withPreferredCharBufSize(16 * 1024)

  println(Book.catalogCodec.decode(catalogBytes, longDescriptions).map(_.size))

  // Validation order: raise the maximum first
  scala.util.Try(ReaderConfig.withPreferredBufSize(64 * 1024 * 1024)).failed.foreach(e => println(e.getMessage))

  println(ReaderConfig.withMaxBufSize(64 * 1024 * 1024).withPreferredBufSize(64 * 1024 * 1024).preferredBufSize)

  // maxCharBufSize: a 5,000-character title under a 2,048-character limit
  val strict: ReaderConfig = ReaderConfig.withPreferredCharBufSize(1024).withMaxCharBufSize(2048)

  val longTitle: Array[Byte] = Book.codec.encode(classics.head.copy(title = "Z" * 5000))

  println(Book.codec.decode(longTitle, strict))

  // maxBufSize: a discriminator that comes last forces the reader to buffer the whole object
  sealed trait Edition
  object Edition {
    case class Print(isbn: String, pages: Int) extends Edition
    case class Ebook(url: String)              extends Edition

    implicit val schema: Schema[Edition] = Schema.derived
    val codec: JsonCodec[Edition]        =
      schema.derive(JsonCodecDeriver.withDiscriminatorKind(DiscriminatorKind.Field("type")))
  }

  // A 100 KB object whose discriminator comes last, under a 64 KiB limit. Keep maxBufSize at or above the
  // reader's initial 32 KiB buffer: in 0.0.56 a smaller limit is not applied reliably.
  val lastDiscriminator: Array[Byte] =
    s"""{"isbn":"${"9" * 100000}","pages":852,"type":"Print"}""".getBytes("UTF-8")
  val strictBuffer: ReaderConfig = ReaderConfig.withMaxBufSize(64 * 1024)

  println(Edition.codec.decode(new ByteArrayInputStream(lastDiscriminator), strictBuffer).left.map(_.getMessage))
  println(Edition.codec.decode(lastDiscriminator).isRight)

  // checkForEndOfInput rejects trailing garbage
  println(Book.codec.decode(Book.codec.encode(classics.head) ++ "garbage".getBytes("UTF-8")).left.map(_.getMessage))

  // Compact, indented and escaped output
  val fpInScala = classics(2)

  println(Book.codec.encode(fpInScala).length)
  println(Book.codec.encode(fpInScala, WriterConfig.withIndentionStep(2)).length)
  println(Book.codec.encode(fpInScala, WriterConfig.withEscapeUnicode(true)).length)
  println(new String(Book.codec.encode(fpInScala, WriterConfig.withEscapeUnicode(true)), "UTF-8"))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain tunejsonthroughput.TuneConfigExample"
```

### Putting It Together

The complete example combines every technique in one catalog endpoint:

```bash
sbt "runMain tunejsonthroughput.CompleteExample"
```

## Going Further

- [JSON Configuration](../reference/schema/built-in-codecs/json/json-config.md) covers `WriterConfig` and `ReaderConfig` together with `Json#print` and `Json.parse`.
- [Codec](../reference/schema/codec.md) lists every encode and decode overload that `JsonCodec` shares with the other binary formats.
- [How to Customize JSON Field Names, ADT Encoding and Optional Fields](./customize-json-encoding.md) changes the shape of the JSON with `JsonCodecDeriver`; derive those codecs once, as shown above.
- [Coming from circe, zio-json or jsoniter-scala](./json-library-migration.md) moves existing codecs to one `Schema`, which you can then tune with this guide.

---

## How to Turn a Paginated API into a Stream

In this guide, we will turn a page-by-page HTTP API into a `Stream` that yields one decoded item at a time. We will walk the pages with `Stream.unfold`, switch to `Stream.unfoldAsync` so that each page request does not block a thread, stop when the API reports no next page, decode every item with a `Schema`, and turn a failed request into a typed error. By the end, you will have a stream of `Issue` values read from a tracker API that serves three pages.

We'll take an incremental approach. We start with a walk that produces pages, then flatten the pages into items, then add error handling, and finally replace the blocking walk with a non-blocking one. The examples use a fake API that serves pages from memory, so they run without a network.

The [streaming migration guide](./streaming-library-migration.md) already shows a synchronous paged walk with `Stream.unfold`, using page numbers and stopping on an empty page. This guide covers the cases that walk leaves out: an opaque next-page cursor, stopping on the last page without an extra request, `Stream.unfoldAsync`, typed errors, and decoding each item.

## The Problem

A paginated API returns part of a collection per request, plus a token that names the next part. A caller who wants the whole collection has to loop: request a page, process its items, keep the token, and stop when the token is missing. Each caller writes that loop again, and each one has to get the cursor handling, the end condition and the error handling right.

For example, consider reading every issue from a tracker that returns a `next` cursor on each page:

```scala
// Without a stream library: a loop with mutable state that you keep correct by hand.
def fetchPage(cursor: Option[String]): Either[String, (List[String], Option[String])] = ???

var cursor: Option[String] = None
var done                   = false
val titles                 = scala.collection.mutable.ListBuffer.empty[String]
while (!done) {
  fetchPage(cursor) match {
    case Left(error) => throw new RuntimeException(error)
    case Right((items, next)) =>
      titles ++= items
      cursor = next
      done = next.isEmpty
  }
}
```

This loop works, but the state lives in two `var`s, the error path throws, and the items are still raw text. Every caller that needs the issues as values repeats the loop, and a bug in the end condition shows up as an extra request or a missed last page. A stream makes the walk a value that the caller composes with other stream operators, and it keeps the cursor inside the walk.

## Prerequisites

The examples need the `streams`, `async` and `schema` modules. Add them to your `build.sbt`:

```scala
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-schema" % "0.0.56",
  "dev.zio" %% "zio-blocks-streams" % "0.0.56",
  "dev.zio" %% "zio-blocks-async" % "0.0.56"
)
```

The code below uses these imports:

```scala

```

This guide assumes you have read [Your First Streaming Pipeline](./your-first-streaming-pipeline.md) and know how to build a `Stream` and run it with `runCollect`. For the schema part, [Your First Schema](./your-first-schema.md) shows how `Schema.derived` and `fromJson` work.

## The Sample API

The tracker's endpoint returns a page of issues and a `next` cursor. Each item is JSON text, as an HTTP body carries it. The last page has no `next`. The fake API below serves three pages from a map and records each cursor it was asked for, so we can count the requests:

```scala
/** An issue as the tracker's JSON API returns it. */
final case class Issue(id: Long, title: String, status: String)

object Issue {
  implicit val schema: Schema[Issue] = Schema.derived
}

/**
 * One response of the paged endpoint. `items` holds each issue as JSON text,
 * and `next` is the cursor of the following page, or `None` on the last page.
 */
final case class RawPage(items: List[String], next: Option[String])

/** Why reading the issues failed: a page request or a single item. */
sealed trait IssueError
final case class PageFailed(cursor: Option[String], reason: String) extends IssueError
final case class ItemFailed(item: String, reason: String)           extends IssueError

/** Where the walk goes next: to the first page, to the page after a cursor, or nowhere. */
sealed trait Position
object Position {
  case object First                      extends Position
  final case class After(cursor: String) extends Position
  case object Done                       extends Position
}

/**
 * A stand-in for an HTTP client that serves `pages` from memory, so the
 * examples run without a network. It records each cursor it is asked for.
 */
final class FakeTrackerApi(pages: Map[Option[String], Either[String, RawPage]]) {
  private val log = scala.collection.mutable.ListBuffer.empty[Option[String]]

  /** The cursors requested so far, in order. */
  def requests: List[Option[String]] = log.synchronized(log.toList)

  def fetchPage(cursor: Option[String]): Either[PageFailed, RawPage] = {
    log.synchronized { log += cursor }
    pages.get(cursor) match {
      case Some(Right(page))  => Right(page)
      case Some(Left(reason)) => Left(PageFailed(cursor, reason))
      case None               => Left(PageFailed(cursor, "unknown cursor"))
    }
  }

  /** The same request, answered on another thread instead of the caller's. */
  def fetchPageAsync(cursor: Option[String]): Async[Either[PageFailed, RawPage]] =
    Async.start(fetchPage(cursor)).map(identity)
}

object FakeTrackerApi {
  def item(id: Long, title: String, status: String): String =
    s"""{"id":$id,"title":"$title","status":"$status"}"""

  /** Three pages of issues; the third page has no `next` cursor. */
  def threePages: FakeTrackerApi =
    new FakeTrackerApi(
      Map(
        None -> Right(
          RawPage(
            List(item(1, "Login fails on Safari", "open"), item(2, "Typo in footer", "closed")),
            Some("c2")
          )
        ),
        Some("c2") -> Right(
          RawPage(
            List(item(3, "Add dark mode", "open"), item(4, "Crash on empty cart", "open")),
            Some("c3")
          )
        ),
        Some("c3") -> Right(RawPage(List(item(5, "Rotate API keys", "closed")), None))
      )
    )
}
```

`FakeTrackerApi#fetchPage` returns an `Either` for a failed request, so the walk can see the failure as a value. `FakeTrackerApi#fetchPageAsync` gives the same answer as an `Async`, which is what the non-blocking walk later in this guide consumes.

## Walking Pages with Stream.unfold

`Stream.unfold` produces elements from a state. We start with the state `Position.First`, and each step returns the element to emit together with the state for the next step. Here, the element is the response of one page request, and the state is the cursor of the next page.

`Stream.unfold` has this signature:

```scala
def unfold[S, A](s: S)(f: S => Option[(A, S)]): Stream[Nothing, A]
```

To start a walk, the state is the first position. To continue, we turn the position into a cursor, request that page, and return the page together with the next position:

```scala
/** Turns one page response into the next walk step: a failure ends the walk, and a page without `next` is the last. */
def advance(result: Either[PageFailed, RawPage]): (Either[PageFailed, RawPage], Position) =
  result match {
    case Left(failure) => (Left(failure), Position.Done)
    case Right(page)   => (Right(page), page.next.fold[Position](Position.Done)(Position.After(_)))
  }

def cursorOf(position: Position): Option[String] =
  position match {
    case Position.After(cursor) => Some(cursor)
    case _                      => None
  }

/** One element per page request, walked with `unfold`. */
def pages(api: FakeTrackerApi): Stream[Nothing, Either[PageFailed, RawPage]] =
  Stream.unfold[Position, Either[PageFailed, RawPage]](Position.First) {
    case Position.Done => None
    case position      => Some(advance(api.fetchPage(cursorOf(position))))
  }
```

The stream's element type is `Either[PageFailed, RawPage]` and not `RawPage`. A failed request must end the walk, and `Stream.unfold` cannot fail: its error type is `Nothing`, and an exception thrown inside `f` is a defect. Carrying the failure as an element lets the next section turn it into a typed error.

Running the stream walks the pages in order, and the fake records each cursor:

```scala
def walkPages(): Unit = {
  val api     = FakeTrackerApi.threePages
  val emitted = pages(api).runCollect
  println(s"pages emitted: ${emitted.map(_.size)}")
  println(s"cursors requested: ${api.requests}")
}

walkPages()
// pages emitted: Right(3)
// cursors requested: List(None, Some(c2), Some(c3))
```

The first request has no cursor, and each later one uses the `next` value of the page before it.

## Stopping on the Last Page

The walk must stop when a page has no `next` cursor. `advance` handles that by moving to `Position.Done`, and the `Stream.unfold` step returns `None` for `Position.Done`, which ends the stream. The last page is emitted with state `Done`, so the walk ends without asking the API for a page that does not exist.

The output above shows three requests for three pages. The last request is for `c3`, the cursor that the second page carries in `next`, and nothing asks for a fourth page. If the state after the last page were `First` or a page number instead, the walk would make one extra request that comes back empty.

:::warning
Do not treat an empty page as the end of the walk unless the API documents that. Many cursor APIs return a last page with items and no `next`, so an empty-page check requests one page too many and can hide a real empty result. Use the `next` value.
:::

## Flattening Pages and Decoding Items

A caller usually wants issues, not pages. We flatten each page into its items with `flatMap`, then decode each item with the `Issue` schema. The `fromJson` extension method on `String` parses the text and returns `Either[SchemaError, Issue]`, so a bad item becomes a `Left` that we can turn into a stream failure:

```scala
/** Decodes each item of a page with the `Issue` schema; a bad item fails the stream. */
def decodeItems(page: RawPage): Stream[IssueError, Issue] =
  Stream.fromIterable(page.items).flatMap { raw =>
    raw.fromJson[Issue] match {
      case Right(issue) => Stream(issue)
      case Left(error)  => Stream.fail(ItemFailed(raw, error.message))
    }
  }

private def flatten(pages: Stream[Nothing, Either[PageFailed, RawPage]]): Stream[IssueError, Issue] =
  pages.flatMap {
    case Left(failure) => Stream.fail(failure)
    case Right(page)   => decodeItems(page)
  }

/** Every issue of every page, or the first typed error. */
def issues(api: FakeTrackerApi): Stream[IssueError, Issue] = flatten(pages(api))
```

The `flatMap` in `flatten` does two things. A page failure becomes `Stream.fail` with the `PageFailed` value, which ends the stream with that error. A successful page goes to `decodeItems`, which emits its issues one at a time. The stream's error type is now `IssueError`, so `runCollect` returns `Either[IssueError, Chunk[Issue]]`:

```scala
def listTitles(): Unit = {
  val api = FakeTrackerApi.threePages
  println(issues(api).runCollect.map(_.map(_.title)))
  println(s"cursors requested: ${api.requests}")
}

listTitles()
// Right(Chunk(Login fails on Safari,Typo in footer,Add dark mode,Crash on empty cart,Rotate API keys))
// cursors requested: List(None, Some(c2), Some(c3))
```

The titles come back in page order, and the decode step gives us values of the `Issue` case class instead of JSON text.

## Handling Errors

Two things can fail. A page request can fail, and an item can fail to decode. Both reach the caller as typed values in the `Left` of `runCollect`, so a caller matches on them and does not catch exceptions.

A failed request ends the stream at that page, so the walk does not request the pages after it. The fake below fails on the second page:

```scala
def failOnSecondPage(): Unit = {
  val flaky = new FakeTrackerApi(
    Map(
      None       -> Right(RawPage(List(FakeTrackerApi.item(1, "Login fails on Safari", "open")), Some("c2"))),
      Some("c2") -> Left("503 Service Unavailable")
    )
  )
  println(issues(flaky).runCollect)
  println(s"cursors requested: ${flaky.requests}")
}

failOnSecondPage()
// Left(PageFailed(Some(c2),503 Service Unavailable))
// cursors requested: List(None, Some(c2))
```

The result is a `PageFailed` for cursor `c2`, and the requests stop there. Nothing asked for a page after `c2`.

A malformed item fails the same way, with an `ItemFailed` that holds the item's text and the decoder's message:

```scala
def decodeBadItem(): Unit = {
  val malformed = new FakeTrackerApi(
    Map(None -> Right(RawPage(List("""{"id":"six","title":"Broken","status":"open"}"""), None)))
  )
  println(issues(malformed).runCollect)
}

decodeBadItem()
// Left(ItemFailed({"id":"six","title":"Broken","status":"open"},illegal number at: .id))
```

The message names the path to the field that failed, here `.id`, so the caller can log the exact item.

A stream does not remember how far it got. Running the same stream value again starts the walk from `Position.First`, so a retry is a second run. Here the same value runs twice and both runs request all three pages:

```scala
def runTwice(): Unit = {
  val api = FakeTrackerApi.threePages
  val s   = issues(api)
  println(s.runCollect.map(_.size))
  println(s.runCollect.map(_.size))
  println(s"cursors requested: ${api.requests}")
}

runTwice()
// Right(5)
// Right(5)
// cursors requested: List(None, Some(c2), Some(c3), None, Some(c2), Some(c3))
```

The count of requests shows six entries: the first run asks for three pages and the second run asks for the same three again. To retry a failed walk, call `runCollect` again, with a limit on the attempts that you choose.

:::tip
To skip bad items instead of failing, replace the `Stream.fail` in `decodeItems` with `Stream.empty` and count the skipped items in your own code. Skipping hides data, so do it only when the caller can live without the missing items.
:::

## Fetching Without Blocking with unfoldAsync

The walk so far calls `FakeTrackerApi#fetchPage` directly, so the thread that runs the stream waits for each page. `Stream.unfoldAsync` takes the same walk with each step returning an `Async`, and the waiting happens on the `Async`, not on the caller's thread. Its signature is:

```scala
def unfoldAsync[S, A](s: S)(f: S => Async[Option[(A, S)]]): Stream[Nothing, A]
```

Each step returns an `Async` that yields `Some((element, nextState))` or `None`. At most one step runs at a time, so the walk requests the next page only after the previous page arrived. A failed `Async` is a defect, which is why the fetch returns an `Either` and never fails the `Async` itself:

```scala
/** The same walk with `unfoldAsync`: each page request returns an `Async`. */
def pagesAsync(api: FakeTrackerApi): Stream[Nothing, Either[PageFailed, RawPage]] =
  Stream.unfoldAsync[Position, Either[PageFailed, RawPage]](Position.First) {
    case Position.Done => Async.succeed(Option.empty[(Either[PageFailed, RawPage], Position)])
    case position      => api.fetchPageAsync(cursorOf(position)).map(result => Some(advance(result)))
  }

def issuesAsync(api: FakeTrackerApi): Stream[IssueError, Issue] = flatten(pagesAsync(api))
```

Only the walk changed. `advance`, `cursorOf`, `decodeItems` and `flatten` are the same, so the typed errors, the last-page rule and the decoding work the same way. The fake's `FakeTrackerApi#fetchPageAsync` starts the request with `Async.start`, which runs it on another thread, so the walk does not hold the caller's thread while the page loads. Running the walk looks the same as before:

```scala
def listTitlesAsync(): Unit = {
  val api = FakeTrackerApi.threePages
  println(issuesAsync(api).runCollect.map(_.map(_.title)))
  println(s"cursors requested: ${api.requests}")
}

listTitlesAsync()
// Right(Chunk(Login fails on Safari,Typo in footer,Add dark mode,Crash on empty cart,Rotate API keys))
// cursors requested: List(None, Some(c2), Some(c3))
```

The output matches the synchronous walk. Use `Stream.unfoldAsync` when each page request is itself asynchronous, for example an HTTP client that returns a future or a callback. Use `Stream.unfold` when the fetch is a plain function that returns its result. For how to wrap a callback-based client in an `Async`, see [Wrap Callback and Future APIs](./wrap-callback-and-future-apis.md).

## Putting It Together

The complete program puts the pieces in one file: the non-blocking walk, the typed errors and the decoding, run once on the three-page API and once on a tracker whose second page fails. It has the same sample API as the sections above, so the only difference from them is that the program runs both cases in order:

```scala title="zio-blocks-examples/paginated-api/src/main/scala/paginatedapi/CompleteExample.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using dep dev.zio::zio-blocks-streams:0.0.56
//> using dep dev.zio::zio-blocks-async:0.0.56
//> using file Model.scala
//> using file Pipeline.scala
// Run from this folder:  scala-cli run CompleteExample.scala
// Or from zio-blocks-examples:  sbt "paginatedApi/runMain paginatedapi.CompleteExample"

package paginatedapi

/**
 * How to Turn a Paginated API into a Stream: the complete program
 *
 * Reads the issues with the non-blocking walk and prints the titles, then reads
 * a tracker whose second page fails.
 */
object CompleteExample {
  def main(args: Array[String]): Unit = {
    val api = FakeTrackerApi.threePages
    println(Pipeline.issuesAsync(api).runCollect.map(_.map(_.title)))
    println(s"cursors requested: ${api.requests}")

    val failing = new FakeTrackerApi(
      Map(
        None       -> Right(RawPage(List(FakeTrackerApi.item(1, "Login fails on Safari", "open")), Some("c2"))),
        Some("c2") -> Left("503 Service Unavailable")
      )
    )
    println(Pipeline.issuesAsync(failing).runCollect)
  }
}
```

Running it prints the titles of all five issues, the three cursors requested, and the `PageFailed` for the failing second page:

```text
Right(Chunk(Login fails on Safari,Typo in footer,Add dark mode,Crash on empty cart,Rotate API keys))
cursors requested: List(None, Some(c2), Some(c3))
Left(PageFailed(Some(c2),503 Service Unavailable))
```

## Running the Examples

All runnable code from this guide is in [`zio-blocks-examples/paginated-api/src/main/scala/paginatedapi/`](https://github.com/zio/zio-blocks/tree/main/zio-blocks-examples/paginated-api/src/main/scala/paginatedapi). Run any file from that folder with `scala-cli run <File>.scala`, or from `zio-blocks-examples` with `sbt "paginatedApi/runMain paginatedapi.<ObjectName>"`.

## Going Further

To read a real API, replace `FakeTrackerApi#fetchPage` with a call to your HTTP client and parse the response body into a `RawPage`. Keep the `Either` result, so a non-2xx status becomes a `PageFailed` and the walk stops with a typed error.

If the API returns the next cursor in a response header instead of the body, read that header in `FakeTrackerApi#fetchPage` and fill `RawPage#next` from it. The walk does not change.

For the full operator list on `Stream`, see the [Stream reference](../reference/streams/core/stream.md). For the rules of `Async`, including cancellation, see the [Async reference](../reference/async.md).

---

## Build a Typed Configuration Loader

Welcome! This tutorial is for Scala developers who want to load their application's settings into a typed value instead of reading strings one key at a time. If you have worked through [Your First Schema](./your-first-schema.md), you have everything you need: we reuse the same idea of deriving a `Schema` for a case class. You don't need any prior experience with configuration libraries to follow along.

## 1. Introduction

By the end of this tutorial, you will understand:

- How a case class with a derived `Schema` becomes a typed configuration value
- How keys, default values and optional fields decide what a load needs
- How to read configuration from YAML and layer it under environment overrides and over defaults
- How a load reports every error at once instead of stopping at the first one
- How to print where each value came from, with secrets kept out of the output

We learn these concepts by building the configuration loader for a small Scala bookstore service, one step at a time:

1. Describe the configuration as a case class
2. Load it from a map, letting defaults fill the gaps
3. Read the same settings from YAML
4. Layer environment overrides, YAML and defaults
5. Collect every error at once
6. See where each value came from

We recommend reading from top to bottom — each section builds on the previous one.

To follow along, add the config module and its YAML adapter to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-config"      % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-config-yaml" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-config"      % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-next-config-yaml" % "0.0.56"
```

Scala 3.9 and newer use the `zio-blocks-next-*` artifact names; the code in this tutorial is the same for every Scala version.

## 2. Background: Configuration Is Just Typed Data

A service reads its settings from several places. A file checked into the repository holds the values every deployment shares, a deployment adds its own file, and the environment supplies passwords and last-minute overrides. Most programs glue these together by hand: read a string, parse it as a number, fall back to another place when it is missing, and throw on the first problem. Each step is one more chance to get a key name or a type wrong.

ZIO Blocks treats configuration as typed data, like any other data with a `Schema`. Every source — a map, a YAML document, the environment — is flattened into the same simple shape: dot-separated keys such as `db.host` pointing to strings. Sources stack on top of each other, so a key missing from one is looked up in the next. One call then walks your case class, reads one key per field, converts each string to the field's type, and either returns the finished value or every problem it found. Along the way it remembers which source answered each key, so "why is the port 9090?" always has an answer.

## 3. Describe the Configuration as a Case Class

Our bookstore service needs a name, a port, the title it features on its home page, a database connection, a list of tags and an optional discount. We describe all of that as ordinary case classes and derive a `Schema` for the top-level one, exactly as we did for `Book` in [Your First Schema](./your-first-schema.md):

```scala

case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

case class BookstoreConfig(
  name: String,
  port: Int = 8080,
  featuredTitle: String,
  db: DbConfig,
  tags: List[String],
  discountPercent: Option[Int]
)

object BookstoreConfig {
  implicit val schema: Schema[BookstoreConfig] = Schema.derived
}
```

The code above:

- `import zio.blocks.config._` brings in `Config`, [`ConfigSource`](../reference/config/config-source.md), `Secret` and the rest of the config module.
- `DbConfig` groups the database settings. `port: Int = 5432` gives the database port a default value.
- `password: Secret` stores the password in a `Secret` — a wrapper around a `String` whose `Secret#toString` never shows the value.
- `BookstoreConfig` nests a `DbConfig`, gives its own `port` a default of `8080`, holds a `List[String]` of tags, and makes `discountPercent` an `Option[Int]` because not every deployment runs a discount.
- `implicit val schema` derives the schema at compile time. `Schema.derived` also derives the schemas for `DbConfig`, `List[String]` and `Option[Int]`, and `Secret` already ships with one.

The configuration is a plain value, so we can build one by hand and print it:

```scala
println(BookstoreConfig("Scala Bookstore", 8080, "Zionomicon", DbConfig("localhost", 5432, "bookstore", Secret("dev-password")), List("scala"), None))
// BookstoreConfig(Scala Bookstore,8080,Zionomicon,DbConfig(localhost,5432,bookstore,<secret>),List(scala),None)
```

The code above prints a hand-built `BookstoreConfig`. Notice that the password appears as `<secret>`: a `Secret` keeps its value out of every log line that prints the configuration. That's all the description the loader needs. Next, we let it build this value for us.

## 4. Load It From a Map, Letting Defaults Fill the Gaps

The loader reads from a `ConfigSource`: a flat set of keys and string values. The simplest one is `ConfigSource.fromMap`, which wraps a `Map[String, String]` and takes a name for the source. The keys follow the shape of the case class: a nested field such as `db.host` is written with a dot, and list elements are numbered from zero, as in `tags.0`.

Let's write the values every deployment of our bookstore shares and call the source `defaults`:

```scala
val defaultSettings = Map(
  "name"            -> "Scala Bookstore",
  "featuredTitle"   -> "Programming in Scala",
  "db.host"         -> "localhost",
  "db.user"         -> "bookstore",
  "db.password"     -> "dev-password",
  "tags.0"          -> "scala",
  "tags.1"          -> "books",
  "discountPercent" -> "10"
)

val defaults = ConfigSource.fromMap(defaultSettings, "defaults")
```

The code above:

- `defaultSettings` is an ordinary `Map[String, String]`. Every value is a string, even the numbers — converting them is the loader's job.
- `"db.host"` and `"db.password"` fill fields of the nested `DbConfig`, and `"tags.0"` and `"tags.1"` are the two elements of `tags`.
- `ConfigSource.fromMap(defaultSettings, "defaults")` turns the map into a source named `defaults`. The name shows up later in error messages and in the provenance table.

`Config.load` derives a decoder from the schema and runs it. A load can fail, so it returns an `Either`: `Right` with the configuration, or `Left` with a non-empty list of errors. Let's write a small helper that prints either side, and load our defaults:

```scala
def report(result: Either[::[ConfigError], BookstoreConfig]): Unit =
  result match {
    case Right(config) => println(config)
    case Left(errors)  => errors.foreach(error => println(error.message))
  }
```

Now let's load our defaults with it:

```scala
report(Config.load[BookstoreConfig](defaults))
// BookstoreConfig(Scala Bookstore,8080,Programming in Scala,DbConfig(localhost,5432,bookstore,<secret>),List(scala, books),Some(10))
```

The code above:

- `Config.load[BookstoreConfig](defaults)` reads one key per field and converts each string to the field's type.
- `report` prints the configuration on `Right`, and the message of each error on `Left`.

Notice the two ports in the output. Our map has no `port` or `db.port` key, so the loader uses the default values from the case classes: `8080` and `5432`. The string `"10"` becomes the number `10`, wrapped in `Some` because `discountPercent` is optional.

What happens when an optional field has no key at all? Let's remove `discountPercent` from the map and load again:

```scala
report(Config.load[BookstoreConfig](ConfigSource.fromMap(defaultSettings - "discountPercent", "defaults")))
// BookstoreConfig(Scala Bookstore,8080,Programming in Scala,DbConfig(localhost,5432,bookstore,<secret>),List(scala, books),None)
```

The code above loads the same settings without the `discountPercent` key. Notice that the load still succeeds and `discountPercent` is `None`. A missing key is an error only for a field that has neither a default value nor an `Option` type.

## 5. Read the Same Settings From YAML

Deployments usually describe their settings in a file. The `config-yaml` module adds `ConfigSource.fromYaml`, which parses YAML text and flattens it into the same dot-separated keys we wrote by hand: a nested mapping becomes `db.host`, and a sequence becomes `tags.0`, `tags.1` and so on. It takes the YAML as a `String`, so our program reads the file and passes its content in. Here we keep the YAML in a string to make the example self-contained:

```scala

val bookstoreYaml =
  """name: Functional Bookshelf
    |featuredTitle: Zionomicon
    |db:
    |  host: db.staging.internal
    |  user: bookshelf_app
    |tags:
    |  - scala
    |  - functional-programming
    |  - zio
    |""".stripMargin

val yaml: ConfigSource = ConfigSource.fromYaml(bookstoreYaml, "bookstore.yaml") match {
  case Right(source) => source
  case Left(error)   => throw error
}
```

The code above:

- `import zio.blocks.config.yaml._` adds the `ConfigSource.fromYaml` method.
- `bookstoreYaml` holds the file content: a store name, a featured title, a nested `db` mapping and a sequence of three tags.
- `ConfigSource.fromYaml(bookstoreYaml, "bookstore.yaml")` parses the text and returns `Either[ConfigError, ConfigSource]`, because the text might not be valid YAML. We name the source after the file it would come from.
- `case Left(error) => throw error` stops the program if the YAML doesn't parse. A `ConfigError` is also a `Throwable`, so it can be thrown as it is.

:::tip
To read a real file, pass its content to `ConfigSource.fromYaml`, for example `java.nio.file.Files.readString(java.nio.file.Path.of("bookstore.yaml"))`.
:::

Now we load the configuration from the YAML source alone:

```scala
report(Config.load[BookstoreConfig](yaml))
// Missing required key 'db.password' in source 'bookstore.yaml'
```

The load fails, and the message says exactly why: the YAML has no `db.password` key. That is on purpose — a password doesn't belong in a file that is checked into a repository. In the next section, we let another source supply it.

## 6. Layer Environment Overrides, YAML and Defaults

`ConfigSource#orElse` stacks two sources. `primary.orElse(fallback)` looks up each key in `primary` first and asks `fallback` only for keys that `primary` doesn't have. The lookup happens key by key, so one configuration value can mix fields from several sources. Chaining `ConfigSource#orElse` builds a whole stack, with the highest priority on the left.

In production, the top layer is the environment. ZIO Blocks reads it with `EnvSource`, which looks up a key such as `db.password` as an environment variable such as `DB_PASSWORD`. A documentation page can't set environment variables, so we stand in for the environment with a map source named `env` that holds two keys:

```scala
val env = ConfigSource.fromMap(Map("db.password" -> "s3cr3t-from-env", "port" -> "9090"), "env")

val layered = env.orElse(yaml).orElse(defaults)
```

The code above:

- `env` holds the password and the port, the two values a deployment would export as environment variables.
- `env.orElse(yaml).orElse(defaults)` builds a three-layer stack: environment first, then the YAML file, then the defaults.

Let's load the layered source:

```scala
report(Config.load[BookstoreConfig](layered))
// BookstoreConfig(Functional Bookshelf,9090,Zionomicon,DbConfig(db.staging.internal,5432,bookshelf_app,<secret>),List(scala, functional-programming, zio),Some(10))
```

Notice where each field came from. The port `9090` and the password come from `env`. The store name, featured title, database host and user and all three tags come from the YAML file. The YAML has no `discountPercent`, so `Some(10)` comes from the defaults. The database port has no key in any layer, so it falls back to the case-class default of `5432`.

:::note
In a real application, replace the stand-in with `EnvSource` and give it a prefix: `EnvSource.prefix("bookstore").orElse(yaml).orElse(defaults)`. `ConfigSource#prefix` puts `bookstore.` in front of every key, and `EnvSource` turns the result into a variable name by replacing dots with underscores and upper-casing it. So `db.password` is read from `BOOKSTORE_DB_PASSWORD` and `port` from `BOOKSTORE_PORT`, and unrelated variables such as `PATH`, or a `PORT` set for another program, never reach the configuration.
:::

## 7. Collect Every Error at Once

Configuration mistakes rarely come alone. A loader that stops at the first one makes us fix, redeploy and fail again for each mistake. `Config.load` keeps going instead: it checks every field and returns every error it found.

Let's break two things at once. We type the port as a word, and we forget to export the password:

```scala
val brokenEnv = ConfigSource.fromMap(Map("port" -> "ninety"), "env")

val defaultsWithoutPassword = ConfigSource.fromMap(defaultSettings - "db.password", "defaults")

val brokenLayers = brokenEnv.orElse(yaml).orElse(defaultsWithoutPassword)
```

Let's load the broken layers:

```scala
report(Config.load[BookstoreConfig](brokenLayers))
// Invalid value 'ninety' for key 'port' (expected Int) in source 'env|bookstore.yaml|defaults': For input string: "ninety"
// Missing required key 'db.password' in source 'env|bookstore.yaml|defaults'
```

The code above:

- `brokenEnv` sets `port` to `"ninety"`, which is not an `Int`.
- `defaultsWithoutPassword` is our defaults map without `db.password`, so no layer has a password.
- `Config.load` reports both problems in one result: the invalid port value and the missing password key.

Each message names the key, the problem and the source that was searched. For a layered source, that name is the whole stack, `env|bookstore.yaml|defaults`, because the loader asked the stack as a whole. The invalid value also ends with the reason the conversion failed, here the JDK's own `For input string: "ninety"`.

Notice that the bad port is reported even though `port` has a default value of `8080`. A default replaces a missing key, never a broken one, so a typo can't silently fall back to a default.

Sometimes a program has nothing better to do than stop when its configuration is broken — for example, at startup. `Config.loadOrThrow` returns the configuration directly, or throws a `ConfigLoadException` whose message is a report of the errors. Let's call it on the YAML source from the previous section, which has no password:

```scala
try Config.loadOrThrow[BookstoreConfig](yaml)
catch { case e: ConfigLoadException => println(e.getMessage) }
// Configuration loading failed with 1 error(s):
//   - Missing required key 'db.password' in source 'bookstore.yaml'
//
```

The code above calls `Config.loadOrThrow` on the YAML source alone and prints the exception's message — the same report a service that fails at startup would log.

## 8. See Where Each Value Came From

When a value surprises us, the first question is "which source set this?". `Config.loadWithProvenance` loads the configuration and also keeps the source stack, so we can ask where each key was found. Its `ProvenanceMap#dump` method prints a table of every key, its value and the source that answered:

```scala
Config.loadWithProvenance[BookstoreConfig](layered) match {
  case Right(provenance) => println(provenance.dump())
  case Left(errors)      => errors.foreach(error => println(error.message))
}
// ┌─────────────────┬────────────────────────┬────────────────┐
// │ Key             │ Value                  │ Source         │
// ├─────────────────┼────────────────────────┼────────────────┤
// │ db.host         │ db.staging.internal    │ bookstore.yaml │
// │ db.password     │ <secret>               │ env            │
// │ db.user         │ bookshelf_app          │ bookstore.yaml │
// │ discountPercent │ 10                     │ defaults       │
// │ featuredTitle   │ Zionomicon             │ bookstore.yaml │
// │ name            │ Functional Bookshelf   │ bookstore.yaml │
// │ port            │ 9090                   │ env            │
// │ tags.0          │ scala                  │ bookstore.yaml │
// │ tags.1          │ functional-programming │ bookstore.yaml │
// │ tags.2          │ zio                    │ bookstore.yaml │
// └─────────────────┴────────────────────────┴────────────────┘
```

The code above:

- `Config.loadWithProvenance[BookstoreConfig](layered)` returns `Either[::[ConfigError], ProvenanceMap[BookstoreConfig]]`. On success, `provenance.value` holds the loaded configuration.
- `provenance.dump()` builds the table as a `String`, sorted by key.

Notice two things in the table. First, each key names the layer that supplied it: `env`, `bookstore.yaml` or `defaults`. Second, the value of `db.password` is shown as `<secret>`. `ProvenanceMap#dump` hides the value of every key whose name looks sensitive — names containing words such as `password`, `secret` or `token`. Redaction goes by the key name, not by the field's type, so give every secret setting a name that says what it is.

The table lists keys that a source supplied. The database port came from the case-class default, not from a source, so it has no row.

Inside our program, we still need the real password to connect to the database. `Secret.unwrap` returns it, and that is the only way to read it:

```scala
val config = Config.loadOrThrow[BookstoreConfig](layered)
```

Next, we print the password field and the length of the real password:

```scala
println(config.db.password)
// <secret>
println(Secret.unwrap(config.db.password).length)
// 15
```

The code above loads the layered configuration, prints the password field, which shows `<secret>`, and then prints the length of the real password that `Secret.unwrap` returns — a value our code can use without ever logging the password itself.

## 9. Putting It Together

This example combines all the concepts we've learned — a typed configuration with defaults, an optional field and a secret, a YAML source layered between the real environment and the defaults, error reporting and the provenance table. It reads the environment with `EnvSource.prefix("bookstore")`, as the note in section 6 recommends, so the provenance table lists only the `BOOKSTORE_*` variables:

```scala title="config-examples/src/main/scala/typedconfigloader/CompleteExample.scala"
package typedconfigloader

/**
 * Build a Typed Configuration Loader — Putting It Together
 *
 * Combines a typed configuration with defaults, an optional field and a secret,
 * a YAML source layered between the real environment (EnvSource) and the
 * defaults, error reporting, and the provenance table.
 *
 * Run with: cd config-examples && sbt "runMain
 * typedconfigloader.CompleteExample"
 *
 * Override settings with: BOOKSTORE_DB_PASSWORD=s3cr3t BOOKSTORE_PORT=9090 sbt
 * "runMain typedconfigloader.CompleteExample"
 */
object CompleteExample extends App {

  case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

  case class BookstoreConfig(
    name: String,
    port: Int = 8080,
    featuredTitle: String,
    db: DbConfig,
    tags: List[String],
    discountPercent: Option[Int]
  )

  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  val defaultSettings = Map(
    "name"            -> "Scala Bookstore",
    "featuredTitle"   -> "Programming in Scala",
    "db.host"         -> "localhost",
    "db.user"         -> "bookstore",
    "db.password"     -> "dev-password",
    "tags.0"          -> "scala",
    "tags.1"          -> "books",
    "discountPercent" -> "10"
  )

  val defaults = ConfigSource.fromMap(defaultSettings, "defaults")

  val bookstoreYaml =
    """name: Functional Bookshelf
      |featuredTitle: Zionomicon
      |db:
      |  host: db.staging.internal
      |  user: bookshelf_app
      |tags:
      |  - scala
      |  - functional-programming
      |  - zio
      |""".stripMargin

  val yaml: ConfigSource = ConfigSource.fromYaml(bookstoreYaml, "bookstore.yaml") match {
    case Right(source) => source
    case Left(error)   => throw error
  }

  // `prefix("bookstore")` scopes EnvSource to BOOKSTORE_* variables: `db.password` is read
  // from BOOKSTORE_DB_PASSWORD and `port` from BOOKSTORE_PORT.
  val layered = EnvSource.prefix("bookstore").orElse(yaml).orElse(defaults)

  Config.loadWithProvenance[BookstoreConfig](layered) match {
    case Right(provenance) =>
      println(provenance.value)
      println(provenance.dump())
    case Left(errors) =>
      errors.foreach(error => println(error.message))
  }

  // A broken load: the port is not a number and no layer has a password.
  val brokenEnv = ConfigSource.fromMap(Map("port" -> "ninety"), "env")

  val defaultsWithoutPassword = ConfigSource.fromMap(defaultSettings - "db.password", "defaults")

  val brokenLayers = brokenEnv.orElse(yaml).orElse(defaultsWithoutPassword)

  Config.load[BookstoreConfig](brokenLayers) match {
    case Right(config) => println(config)
    case Left(errors)  => errors.foreach(error => println(error.message))
  }
}
```

## 10. Running the Examples

All examples in this tutorial have corresponding runnable Scala files in the `config-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/config-examples
```

Run them in order to follow the tutorial step by step.

### Describe the Configuration as a Case Class

This example defines `BookstoreConfig` and `DbConfig`, derives the schema, and prints a hand-built configuration with its password redacted.

<details>
  <summary>config-examples/src/main/scala/typedconfigloader/Concept1DescribeExample.scala</summary>

```scala title="config-examples/src/main/scala/typedconfigloader/Concept1DescribeExample.scala" showLineNumbers
package typedconfigloader

/**
 * Build a Typed Configuration Loader — Concept 1: Describe the Configuration as
 * a Case Class
 *
 * Describes a bookstore service's settings as case classes, derives a Schema
 * for the top-level one, and prints a hand-built configuration whose password
 * is redacted by Secret.
 *
 * Run with: cd config-examples && sbt "runMain
 * typedconfigloader.Concept1DescribeExample"
 */
object Concept1DescribeExample extends App {

  case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

  case class BookstoreConfig(
    name: String,
    port: Int = 8080,
    featuredTitle: String,
    db: DbConfig,
    tags: List[String],
    discountPercent: Option[Int]
  )

  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  println(
    BookstoreConfig(
      "Scala Bookstore",
      8080,
      "Zionomicon",
      DbConfig("localhost", 5432, "bookstore", Secret("dev-password")),
      List("scala"),
      None
    )
  )
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain typedconfigloader.Concept1DescribeExample"
```

### Load It From a Map

This example loads the configuration from a map of defaults and shows how default values and an optional field fill missing keys.

<details>
  <summary>config-examples/src/main/scala/typedconfigloader/Concept2LoadFromMapExample.scala</summary>

```scala title="config-examples/src/main/scala/typedconfigloader/Concept2LoadFromMapExample.scala" showLineNumbers
package typedconfigloader

/**
 * Build a Typed Configuration Loader — Concept 2: Load It From a Map, Letting
 * Defaults Fill the Gaps
 *
 * Loads BookstoreConfig from a map of defaults, showing how case-class default
 * values and an Option field fill keys the map does not have.
 *
 * Run with: cd config-examples && sbt "runMain
 * typedconfigloader.Concept2LoadFromMapExample"
 */
object Concept2LoadFromMapExample extends App {

  case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

  case class BookstoreConfig(
    name: String,
    port: Int = 8080,
    featuredTitle: String,
    db: DbConfig,
    tags: List[String],
    discountPercent: Option[Int]
  )

  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  val defaultSettings = Map(
    "name"            -> "Scala Bookstore",
    "featuredTitle"   -> "Programming in Scala",
    "db.host"         -> "localhost",
    "db.user"         -> "bookstore",
    "db.password"     -> "dev-password",
    "tags.0"          -> "scala",
    "tags.1"          -> "books",
    "discountPercent" -> "10"
  )

  val defaults = ConfigSource.fromMap(defaultSettings, "defaults")

  def report(result: Either[::[ConfigError], BookstoreConfig]): Unit =
    result match {
      case Right(config) => println(config)
      case Left(errors)  => errors.foreach(error => println(error.message))
    }

  report(Config.load[BookstoreConfig](defaults))

  report(Config.load[BookstoreConfig](ConfigSource.fromMap(defaultSettings - "discountPercent", "defaults")))
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain typedconfigloader.Concept2LoadFromMapExample"
```

### Read the Same Settings From YAML

This example parses YAML with `ConfigSource.fromYaml` and loads it alone, which reports the missing password.

<details>
  <summary>config-examples/src/main/scala/typedconfigloader/Concept3YamlExample.scala</summary>

```scala title="config-examples/src/main/scala/typedconfigloader/Concept3YamlExample.scala" showLineNumbers
package typedconfigloader

/**
 * Build a Typed Configuration Loader — Concept 3: Read the Same Settings From
 * YAML
 *
 * Parses YAML text with ConfigSource.fromYaml and loads it alone, which reports
 * the missing db.password key.
 *
 * Run with: cd config-examples && sbt "runMain
 * typedconfigloader.Concept3YamlExample"
 */
object Concept3YamlExample extends App {

  case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

  case class BookstoreConfig(
    name: String,
    port: Int = 8080,
    featuredTitle: String,
    db: DbConfig,
    tags: List[String],
    discountPercent: Option[Int]
  )

  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  def report(result: Either[::[ConfigError], BookstoreConfig]): Unit =
    result match {
      case Right(config) => println(config)
      case Left(errors)  => errors.foreach(error => println(error.message))
    }

  val bookstoreYaml =
    """name: Functional Bookshelf
      |featuredTitle: Zionomicon
      |db:
      |  host: db.staging.internal
      |  user: bookshelf_app
      |tags:
      |  - scala
      |  - functional-programming
      |  - zio
      |""".stripMargin

  val yaml: ConfigSource = ConfigSource.fromYaml(bookstoreYaml, "bookstore.yaml") match {
    case Right(source) => source
    case Left(error)   => throw error
  }

  report(Config.load[BookstoreConfig](yaml))
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain typedconfigloader.Concept3YamlExample"
```

### Layer Environment Overrides, YAML and Defaults

This example stacks an environment stand-in, the YAML source and the defaults with `ConfigSource#orElse` and loads the result.

<details>
  <summary>config-examples/src/main/scala/typedconfigloader/Concept4LayeringExample.scala</summary>

```scala title="config-examples/src/main/scala/typedconfigloader/Concept4LayeringExample.scala" showLineNumbers
package typedconfigloader

/**
 * Build a Typed Configuration Loader — Concept 4: Layer Environment Overrides,
 * YAML and Defaults
 *
 * Stacks an environment stand-in, the YAML source and the defaults with orElse,
 * then loads the layered source key by key.
 *
 * Run with: cd config-examples && sbt "runMain
 * typedconfigloader.Concept4LayeringExample"
 */
object Concept4LayeringExample extends App {

  case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

  case class BookstoreConfig(
    name: String,
    port: Int = 8080,
    featuredTitle: String,
    db: DbConfig,
    tags: List[String],
    discountPercent: Option[Int]
  )

  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  val defaultSettings = Map(
    "name"            -> "Scala Bookstore",
    "featuredTitle"   -> "Programming in Scala",
    "db.host"         -> "localhost",
    "db.user"         -> "bookstore",
    "db.password"     -> "dev-password",
    "tags.0"          -> "scala",
    "tags.1"          -> "books",
    "discountPercent" -> "10"
  )

  val defaults = ConfigSource.fromMap(defaultSettings, "defaults")

  def report(result: Either[::[ConfigError], BookstoreConfig]): Unit =
    result match {
      case Right(config) => println(config)
      case Left(errors)  => errors.foreach(error => println(error.message))
    }

  val bookstoreYaml =
    """name: Functional Bookshelf
      |featuredTitle: Zionomicon
      |db:
      |  host: db.staging.internal
      |  user: bookshelf_app
      |tags:
      |  - scala
      |  - functional-programming
      |  - zio
      |""".stripMargin

  val yaml: ConfigSource = ConfigSource.fromYaml(bookstoreYaml, "bookstore.yaml") match {
    case Right(source) => source
    case Left(error)   => throw error
  }

  val env = ConfigSource.fromMap(Map("db.password" -> "s3cr3t-from-env", "port" -> "9090"), "env")

  val layered = env.orElse(yaml).orElse(defaults)

  report(Config.load[BookstoreConfig](layered))
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain typedconfigloader.Concept4LayeringExample"
```

### Collect Every Error at Once

This example loads layers with a bad port and no password, prints every error, and then shows the report from `Config.loadOrThrow`.

<details>
  <summary>config-examples/src/main/scala/typedconfigloader/Concept5ErrorsExample.scala</summary>

```scala title="config-examples/src/main/scala/typedconfigloader/Concept5ErrorsExample.scala" showLineNumbers
package typedconfigloader

/**
 * Build a Typed Configuration Loader — Concept 5: Collect Every Error at Once
 *
 * Loads layers with a non-numeric port and no password, prints every error from
 * Config.load, then prints the report carried by Config.loadOrThrow on the YAML source alone.
 *
 * Run with: cd config-examples && sbt "runMain
 * typedconfigloader.Concept5ErrorsExample"
 */
object Concept5ErrorsExample extends App {

  case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

  case class BookstoreConfig(
    name: String,
    port: Int = 8080,
    featuredTitle: String,
    db: DbConfig,
    tags: List[String],
    discountPercent: Option[Int]
  )

  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  val defaultSettings = Map(
    "name"            -> "Scala Bookstore",
    "featuredTitle"   -> "Programming in Scala",
    "db.host"         -> "localhost",
    "db.user"         -> "bookstore",
    "db.password"     -> "dev-password",
    "tags.0"          -> "scala",
    "tags.1"          -> "books",
    "discountPercent" -> "10"
  )

  def report(result: Either[::[ConfigError], BookstoreConfig]): Unit =
    result match {
      case Right(config) => println(config)
      case Left(errors)  => errors.foreach(error => println(error.message))
    }

  val bookstoreYaml =
    """name: Functional Bookshelf
      |featuredTitle: Zionomicon
      |db:
      |  host: db.staging.internal
      |  user: bookshelf_app
      |tags:
      |  - scala
      |  - functional-programming
      |  - zio
      |""".stripMargin

  val yaml: ConfigSource = ConfigSource.fromYaml(bookstoreYaml, "bookstore.yaml") match {
    case Right(source) => source
    case Left(error)   => throw error
  }

  val brokenEnv = ConfigSource.fromMap(Map("port" -> "ninety"), "env")

  val defaultsWithoutPassword = ConfigSource.fromMap(defaultSettings - "db.password", "defaults")

  val brokenLayers = brokenEnv.orElse(yaml).orElse(defaultsWithoutPassword)

  report(Config.load[BookstoreConfig](brokenLayers))

  try Config.loadOrThrow[BookstoreConfig](yaml)
  catch { case e: ConfigLoadException => println(e.getMessage) }
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain typedconfigloader.Concept5ErrorsExample"
```

### See Where Each Value Came From

This example prints the provenance table for the layered configuration and reads the password with `Secret.unwrap`.

<details>
  <summary>config-examples/src/main/scala/typedconfigloader/Concept6ProvenanceExample.scala</summary>

```scala title="config-examples/src/main/scala/typedconfigloader/Concept6ProvenanceExample.scala" showLineNumbers
package typedconfigloader

/**
 * Build a Typed Configuration Loader — Concept 6: See Where Each Value Came
 * From
 *
 * Loads the layered configuration with Config.loadWithProvenance, prints the
 * provenance table with secrets redacted, and reads the real password with
 * Secret.unwrap.
 *
 * Run with: cd config-examples && sbt "runMain
 * typedconfigloader.Concept6ProvenanceExample"
 */
object Concept6ProvenanceExample extends App {

  case class DbConfig(host: String, port: Int = 5432, user: String, password: Secret)

  case class BookstoreConfig(
    name: String,
    port: Int = 8080,
    featuredTitle: String,
    db: DbConfig,
    tags: List[String],
    discountPercent: Option[Int]
  )

  object BookstoreConfig {
    implicit val schema: Schema[BookstoreConfig] = Schema.derived
  }

  val defaultSettings = Map(
    "name"            -> "Scala Bookstore",
    "featuredTitle"   -> "Programming in Scala",
    "db.host"         -> "localhost",
    "db.user"         -> "bookstore",
    "db.password"     -> "dev-password",
    "tags.0"          -> "scala",
    "tags.1"          -> "books",
    "discountPercent" -> "10"
  )

  val defaults = ConfigSource.fromMap(defaultSettings, "defaults")

  val bookstoreYaml =
    """name: Functional Bookshelf
      |featuredTitle: Zionomicon
      |db:
      |  host: db.staging.internal
      |  user: bookshelf_app
      |tags:
      |  - scala
      |  - functional-programming
      |  - zio
      |""".stripMargin

  val yaml: ConfigSource = ConfigSource.fromYaml(bookstoreYaml, "bookstore.yaml") match {
    case Right(source) => source
    case Left(error)   => throw error
  }

  val env = ConfigSource.fromMap(Map("db.password" -> "s3cr3t-from-env", "port" -> "9090"), "env")

  val layered = env.orElse(yaml).orElse(defaults)

  Config.loadWithProvenance[BookstoreConfig](layered) match {
    case Right(provenance) => println(provenance.dump())
    case Left(errors)      => errors.foreach(error => println(error.message))
  }

  val config = Config.loadOrThrow[BookstoreConfig](layered)

  println(config.db.password)
  println(Secret.unwrap(config.db.password).length)
}
```

</details>

Run it from the `config-examples` directory:

```bash
sbt "runMain typedconfigloader.Concept6ProvenanceExample"
```

### The Complete Example

The complete example from "Putting It Together" reads the real environment with `EnvSource.prefix("bookstore")`, so you can override settings with `BOOKSTORE_*` variables when you run it:

```bash
sbt "runMain typedconfigloader.CompleteExample"
BOOKSTORE_DB_PASSWORD=s3cr3t BOOKSTORE_PORT=9090 sbt "runMain typedconfigloader.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## 11. What You've Learned

In this tutorial, you learned:

- How a case class with a derived `Schema` describes a typed configuration, and how `Secret` keeps a password out of printed values
- How `Config.load` reads dot-separated keys from a `ConfigSource`, and how default values and `Option` fields decide which keys are required
- How `ConfigSource.fromYaml` flattens YAML into the same keys, and how `ConfigSource#orElse` layers environment overrides, YAML and defaults key by key
- How `Config.load` returns every error at once, and how `Config.loadOrThrow` turns them into one report at startup
- How `Config.loadWithProvenance` and `ProvenanceMap#dump` show which source supplied each value, with sensitive values redacted

You now have a solid foundation in typed configuration with ZIO Blocks. The next step is to explore the other sources and formats, and to wire configuration into the rest of your application.

## 12. Where to Go Next

- **Want every way to build and combine sources?** Read [Config Source](../reference/config/config-source.md), which covers `EnvSource`, `SysPropSource`, `ConfigSource#prefix` and key-format mapping for environment-style names.
- **Curious how fields map to keys?** [Config Decoder](../reference/config/config-decoder.md) documents the rules for nested records, sealed traits, lists and maps.
- **Want to handle errors by kind?** [Errors](../reference/config/errors.md) lists every `ConfigError` case and how errors are combined.
- **Using JSON or HOCON instead of YAML?** [File Formats](../reference/config/formats.md) covers all three adapters.
- **Wiring configuration into a dependency graph?** `Config.wire[A]` turns a `ConfigSource` into a typed value as a node in a `scope` wire graph; the [Configuration & Feature Flags](../reference/config/index.md) overview shows it in action, and [How to Feed Configuration into a Wire Dependency Graph](./feed-config-into-a-wire-graph.md) walks through it end to end.
- **Need runtime switches?** [Flags](../reference/config/flags.md) and [Rollout](../reference/config/rollout.md) cover feature flags with percentage-based rollouts, built on the same sources.
- **Coming from PureConfig, Typesafe Config or zio-config?** [Coming from PureConfig, Typesafe Config or zio-config](./config-library-migration.md) maps the concepts and APIs you already know onto the ones in this tutorial.
- **Ready to put it all together?** [Build a Small JSON Backend from Blocks](./small-json-backend-from-blocks.md) is the capstone tutorial that combines schema, config, scope, SQL and telemetry into one small JSON backend.

---

## How to Upsert Rows and Paginate with Keyset Cursors

In this guide, we keep a small bookstore inventory up to date and read a large sales table back in pages. By the end, you will have a table that accepts upserts (one statement that inserts a row, or updates it when the row is already there), a nightly supplier import that applies a mix of new and changed books in one batch, and a loop that walks 10,000 rows 50 at a time, each page starting right after the last row of the previous one.

We build it one recipe at a time:

- A table with a real primary key, which every upsert needs
- `Repo#insertOrUpdate` and `Repo#insertOrUpdateBatch` for "insert or replace"
- `Upsert.insertDoNothing` for "insert only if it's new"
- `Upsert.insertDoUpdate` for "insert, or update only these columns"
- The upsert SQL on SQLite and on PostgreSQL, side by side
- `Repo#pageAfter` for keyset pagination by ID, and why it beats `OFFSET`
- `Frag.keysetAfter` for keyset pagination over your own query
- What keyset cursors can't do

This guide builds on [Build Your First Database Repository on SQLite](./your-first-sqlite-repository.md). We use `JdbcTransactor`, `Table`, `Repo` and `sql"..."` the way that tutorial introduces them and don't explain them again.

## The Problem

Two jobs come up in almost every database-backed application. The first is "save this row": insert it if it's new, update it if it isn't. The second is "read this big table in pages": show 50 rows, then the next 50, and so on.

Written with plain JDBC, both look harmless:

```scala

// "Save this book": look first, then INSERT or UPDATE
def saveBook(conn: Connection, id: Long, title: String, price: BigDecimal, stock: Int): Unit = {
  val check = conn.prepareStatement("SELECT 1 FROM book WHERE id = ?")
  check.setLong(1, id)
  val exists = check.executeQuery().next()
  check.close()

  val write =
    if (exists) conn.prepareStatement("UPDATE book SET title = ?, price = ?, stock = ? WHERE id = ?")
    else conn.prepareStatement("INSERT INTO book (title, price, stock, id) VALUES (?, ?, ?, ?)")
  write.setString(1, title)
  write.setBigDecimal(2, price.bigDecimal)
  write.setInt(3, stock)
  write.setLong(4, id)
  write.executeUpdate()
  write.close()
}

// "Read page n": skip n * size rows, then take size rows
def salesPage(conn: Connection, pageNumber: Int, size: Int): List[Long] = {
  val query = conn.prepareStatement("SELECT id FROM sale ORDER BY id LIMIT ? OFFSET ?")
  query.setInt(1, size)
  query.setInt(2, pageNumber * size)
  val rs  = query.executeQuery()
  val ids = Iterator.continually(rs).takeWhile(_.next()).map(_.getLong(1)).toList
  query.close()
  ids
}
```

Both break down under real load. `saveBook` makes two round trips, lists every column twice, and has a race: two writers can both see "no such book" and both `INSERT`, and the second one fails. `salesPage` asks the database to read and throw away `pageNumber * size` rows before it returns anything, so every page costs more than the one before it, and a row inserted or deleted while the reader is on page 3 shifts every later page by one: a row is shown twice, or never.

In this guide, we let the database do both jobs in one statement each: `INSERT ... ON CONFLICT` for the upsert, and `WHERE id > ?` with the last ID the reader saw for the page.

## Prerequisites

Add the `sql` module and the SQLite JDBC driver to your `build.sbt`:

```scala
// Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-sql" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-sql" % "0.0.56"
scalacOptions += "-experimental"

// The SQLite JDBC driver, for every Scala version
libraryDependencies += "org.xerial" % "sqlite-jdbc" % "3.53.4.0"
```

The `sql` module is published for Scala 3 only. `JdbcTransactor` runs on the JVM and needs JDK 21 or newer. The code in this guide is the same for both artifact names.

For the database setup itself, see [How to Test Database Code Against SQLite](./test-database-code-against-sqlite.md) (a fresh database per test) and [How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md) (the PostgreSQL side). Everything below runs unchanged on PostgreSQL once the table exists there.

## The Bookstore Inventory

Our inventory is one `Book` per title. `isbn` is optional because Zionomicon has none in our records, so the inventory is keyed by `id` instead. The prices and stock counts are made up:

```scala

case class Book(id: Long, title: String, isbn: Option[String], price: BigDecimal, stock: Int)

object Book {
  given Schema[Book] = Schema.derived
}

val books = Repo.derived[Book, Long]

// Prices and stock are fictional
val zionomicon         = Book(1L, "Zionomicon", None, BigDecimal("39.99"), 5)
val programmingInScala = Book(2L, "Programming in Scala", Some("9780981531601"), BigDecimal("49.95"), 3)
val fpInScala          = Book(3L, "Functional Programming in Scala", Some("9781617290657"), BigDecimal("44.99"), 8)
```

Each recipe prints the inventory, so we add a helper that lists every book in `id` order. The database is a temporary file, because SQLite's `jdbc:sqlite::memory:` URL gives every connection its own empty database:

```scala
val dbFile     = java.nio.file.Files.createTempFile("bookstore", ".db")
val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

def printBooks()(using DbCon): Unit =
  sql"SELECT id, title, isbn, price, stock FROM book ORDER BY id".query[Book].foreach { b =>
    println(f"${b.id}%2d  ${b.title}%-32s ${b.isbn.getOrElse("-")}%-14s ${b.price}%6s  stock ${b.stock}%2d")
  }
```

## Create a Table with a Real Primary Key

An upsert works by letting the `INSERT` collide with an existing row, so the table needs a `PRIMARY KEY` or `UNIQUE` constraint on the column it collides on. `Table#createTable` doesn't declare one. Here is what it generates for `Book`, and what happens when we upsert into that table:

```scala
println(books.table.createTable(SqlDialect.SQLite).sql(SqlDialect.SQLite))
// CREATE TABLE IF NOT EXISTS book (
//   id INTEGER NOT NULL,
//   title TEXT NOT NULL,
//   isbn TEXT,
//   price TEXT NOT NULL,
//   stock INTEGER NOT NULL
// )

transactor.connect {
  books.table.createTable(SqlDialect.SQLite).update
  println(Try(books.insertOrUpdate(zionomicon)))
}
// Failure(org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (ON CONFLICT clause does not match any PRIMARY KEY or UNIQUE constraint))
```

SQLite refuses the statement because nothing on `id` can conflict. The fix is to write the `CREATE TABLE` statement ourselves. The easiest way is to copy what `createTable` printed and add `PRIMARY KEY` to the `id` column. We drop the table `createTable` made and create the keyed one in its place:

```scala
transactor.connect {
  sql"DROP TABLE book".update
  sql"""CREATE TABLE book (
    id    INTEGER PRIMARY KEY,
    title TEXT NOT NULL,
    isbn  TEXT,
    price TEXT NOT NULL,
    stock INTEGER NOT NULL
  )""".update
  println(Try(books.insertOrUpdate(zionomicon)))
  printBooks()
}
// Success(1)
//  1  Zionomicon                       -               39.99  stock  5
```

:::tip
The `price TEXT` column is what `createTable` picks for a `BigDecimal` on SQLite: it keeps the exact decimal digits. On PostgreSQL the same column is `NUMERIC`; see [Compare the Upsert SQL for SQLite and PostgreSQL](#compare-the-upsert-sql-for-sqlite-and-postgresql).
:::

## Insert or Update One Book

`Repo#insertOrUpdate` saves an entity in one statement: it inserts the row, or, when a row with the same ID exists, overwrites every other column with the entity's values. It returns the affected row count:

```scala
transactor.connect {
  println(books.insertOrUpdate(programmingInScala))
  println(books.insertOrUpdate(fpInScala))
  println(books.insertOrUpdate(zionomicon.copy(stock = 4)))
  printBooks()
}
// 1
// 1
// 1
//  1  Zionomicon                       -               39.99  stock  4
//  2  Programming in Scala             9780981531601   49.95  stock  3
//  3  Functional Programming in Scala  9781617290657   44.99  stock  8
```

The first two calls insert new books; the third finds Zionomicon already there and updates its stock from 5 to 4. All three return `1`, so the count can't tell an insert from an update. If your code needs to know which one happened, check with `books.exists(id)` first, inside the same transaction.

## Apply a Nightly Supplier Batch

A supplier sends a nightly feed: some books changed price or stock, and some are new to the store. `Repo#insertOrUpdateBatch` sends the whole feed as one JDBC batch of upserts. We run it inside `transact`, so a failure halfway through leaves the inventory as it was:

```scala
val supplierFeed = List(
  programmingInScala.copy(price = BigDecimal("44.95")),             // price drop
  fpInScala.copy(stock = 12),                                       // restock
  Book(4L, "Hands-on Scala Programming", None, BigDecimal("39.00"), 6) // new title
)
```

```scala
transactor.transact {
  println("Before:")
  printBooks()
  println(s"Rows affected: ${books.insertOrUpdateBatch(supplierFeed)}")
  println("After:")
  printBooks()
}
// Before:
//  1  Zionomicon                       -               39.99  stock  4
//  2  Programming in Scala             9780981531601   49.95  stock  3
//  3  Functional Programming in Scala  9781617290657   44.99  stock  8
// Rows affected: 3
// After:
//  1  Zionomicon                       -               39.99  stock  4
//  2  Programming in Scala             9780981531601   44.95  stock  3
//  3  Functional Programming in Scala  9781617290657   44.99  stock 12
//  4  Hands-on Scala Programming       -               39.00  stock  6
```

The count is the sum over the batch, `1` per row whether it was inserted or updated. An empty feed returns `0` without touching the database.

## Keep Existing Rows Untouched

Sometimes the row already in the database must win. A catalog import that runs every time the application starts should add the books it doesn't know yet, but never reset the price or stock the shop has changed since. `Upsert.insertDoNothing(table, entity, conflictColumn)` builds an `INSERT ... ON CONFLICT (...) DO NOTHING` statement for that. It returns a `Frag`, which we run with `update`:

```scala
val catalogDefaults = List(
  zionomicon,                                                   // stock 5 in the catalog, 4 in the shop
  Book(5L, "Programming Scala", None, BigDecimal("59.99"), 2)   // not in the shop yet
)
```

```scala
transactor.connect {
  catalogDefaults.foreach { book =>
    val inserted = Upsert.insertDoNothing(books.table, book, "id").update
    println(s"${book.title}: $inserted")
  }
  printBooks()
}
// Zionomicon: 0
// Programming Scala: 1
//  1  Zionomicon                       -               39.99  stock  4
//  2  Programming in Scala             9780981531601   44.95  stock  3
//  3  Functional Programming in Scala  9781617290657   44.99  stock 12
//  4  Hands-on Scala Programming       -               39.00  stock  6
//  5  Programming Scala                -               59.99  stock  2
```

Here the count does tell the two cases apart: `0` means the row already existed and nothing changed, `1` means the row was inserted. Zionomicon keeps the stock of 4 that the shop set earlier.

## Update Only the Columns You Choose

A supplier is the source of truth for price and stock, but not for titles: its feed spells Zionomicon in capitals with an edition note. The four-argument `Upsert.insertDoUpdate(table, entity, conflictColumn, updateColumns)` inserts the full row when the book is new, and updates only `updateColumns` when it exists:

```scala
val supplierRow   = zionomicon.copy(title = "ZIONOMICON (PAPERBACK)", price = BigDecimal("34.99"), stock = 20)
val priceAndStock = Upsert.insertDoUpdate(books.table, supplierRow, "id", Seq("price", "stock"))
```

```scala
println(priceAndStock.sql(SqlDialect.SQLite))
// INSERT INTO "book" ("id", "title", "isbn", "price", "stock") VALUES (?, ?, ?, ?, ?) ON CONFLICT ("id") DO UPDATE SET "price" = ?, "stock" = ?

transactor.connect {
  println(priceAndStock.update)
  printBooks()
}
// 1
//  1  Zionomicon                       -               34.99  stock 20
//  2  Programming in Scala             9780981531601   44.95  stock  3
//  3  Functional Programming in Scala  9781617290657   44.99  stock 12
//  4  Hands-on Scala Programming       -               39.00  stock  6
//  5  Programming Scala                -               59.99  stock  2
```

The title stays "Zionomicon"; price and stock come from the feed. The three-argument `Upsert.insertDoUpdate(table, entity, conflictColumn)` updates every column except the conflict column, which is what `Repo#insertOrUpdate` uses.

The column names are checked when the `Frag` is built, not when it runs. A column the table doesn't have, the conflict column itself, a duplicate, or an empty list throws `IllegalArgumentException`:

```scala
println(Try(Upsert.insertDoUpdate(books.table, supplierRow, "id", Seq("cost"))))
// Failure(java.lang.IllegalArgumentException: Assignment column 'cost' (validated as 'cost') not found in table 'book' columns: id, title, isbn, price, stock)
println(Try(Upsert.insertDoUpdate(books.table, supplierRow, "id", Seq("id", "price"))))
// Failure(java.lang.IllegalArgumentException: Assignment columns must not contain conflict column 'id')
```

:::note
The conflict column doesn't have to be the ID. Any column with a `UNIQUE` constraint works, for example `Upsert.insertDoNothing(books.table, book, "isbn")` on a table that declares `isbn TEXT UNIQUE`. Without the constraint, the database rejects the statement just as it did for the table `createTable` made.
:::

## Compare the Upsert SQL for SQLite and PostgreSQL

SQLite (3.24 and newer) and PostgreSQL (9.5 and newer) accept the same `INSERT ... ON CONFLICT (column) DO UPDATE SET ...` syntax, and both take `?` placeholders through JDBC. So the upsert statement is the same text for both dialects:

```scala
val upsert = Upsert.insertDoUpdate(books.table, zionomicon, "id")
```

```scala
println(upsert.sql(SqlDialect.SQLite))
// INSERT INTO "book" ("id", "title", "isbn", "price", "stock") VALUES (?, ?, ?, ?, ?) ON CONFLICT ("id") DO UPDATE SET "title" = ?, "isbn" = ?, "price" = ?, "stock" = ?
println(upsert.sql(SqlDialect.PostgreSQL))
// INSERT INTO "book" ("id", "title", "isbn", "price", "stock") VALUES (?, ?, ?, ?, ?) ON CONFLICT ("id") DO UPDATE SET "title" = ?, "isbn" = ?, "price" = ?, "stock" = ?
println(upsert.sql(SqlDialect.SQLite) == upsert.sql(SqlDialect.PostgreSQL))
// true
println(upsert.queryParams)
// Vector(DbLong(1), DbString(Zionomicon), DbNull, DbBigDecimal(39.99), DbInt(5), DbString(Zionomicon), DbNull, DbBigDecimal(39.99), DbInt(5))
```

The parameters show how the update works: the entity's values are bound once for `VALUES (...)` and once more for the `SET` columns, instead of reading them back through PostgreSQL's `EXCLUDED` row. That keeps one statement valid in both databases.

What does differ is the table you create. The column types come from the dialect:

```scala
println(books.table.createTable(SqlDialect.PostgreSQL).sql(SqlDialect.PostgreSQL))
// CREATE TABLE IF NOT EXISTS book (
//   id BIGINT NOT NULL,
//   title TEXT NOT NULL,
//   isbn TEXT,
//   price NUMERIC NOT NULL,
//   stock INTEGER NOT NULL
// )
```

On PostgreSQL, write the same statement with the primary key added, `id BIGINT PRIMARY KEY`, and every upsert in this guide runs there unchanged.

## Walk a Large Table with pageAfter

Now for the second job. The store logs every sale, and a nightly report reads the whole log. We generate 10,000 fictional sales and insert them in one batch:

```scala
case class Sale(id: Long, bookId: Long, quantity: Int)

object Sale {
  given Schema[Sale] = Schema.derived
}

val sales = Repo.derived[Sale, Long]
```

```scala
transactor.transact {
  sql"CREATE TABLE sale (id INTEGER PRIMARY KEY, book_id INTEGER NOT NULL, quantity INTEGER NOT NULL)".update
  val generated = (1L to 10000L).map(i => Sale(i, i % 3 + 1, (i % 5 + 1).toInt))
  println(sales.insertBatch(generated))
}
// 10000
```

`Repo#pageAfter(cursorId, limit)` returns the next `limit` rows whose ID is strictly greater than `cursorId`, in ascending ID order. The cursor is the ID of the last row on the previous page; for the first page, pass any value smaller than every ID, here `0L`:

```scala
transactor.connect {
  val first  = sales.pageAfter(0L, 50)
  val second = sales.pageAfter(first.last.id, 50)
  println(s"First page:  ${first.size} rows, ids ${first.head.id} to ${first.last.id}")
  println(s"Second page: ${second.size} rows, ids ${second.head.id} to ${second.last.id}")
  println(s"After the last id: ${sales.pageAfter(10000L, 50)}")
}
// First page:  50 rows, ids 1 to 50
// Second page: 50 rows, ids 51 to 100
// After the last id: List()
```

To walk the whole table, keep passing the last ID back in until a page comes back empty:

```scala
def walkSales(cursor: Long, pages: Int, rows: Int, sold: Long)(using DbCon): (Int, Int, Long) = {
  val page = sales.pageAfter(cursor, 50)
  if (page.isEmpty) (pages, rows, sold)
  else walkSales(page.last.id, pages + 1, rows + page.size, sold + page.map(_.quantity).sum)
}
```

```scala
transactor.connect {
  val (pages, rows, sold) = walkSales(0L, 0, 0, 0L)
  println(s"$pages pages, $rows rows, $sold copies sold")
}
// 200 pages, 10000 rows, 30000 copies sold
```

Each call sends `SELECT ... FROM "sale" WHERE "id" > ? ORDER BY "id" ASC LIMIT 50`. Because the row with the cursor's ID is excluded (`>`, not `>=`), consecutive pages never repeat the boundary row. A limit of zero or less throws `IllegalArgumentException`.

## Why Keyset Pagination Beats OFFSET

`LIMIT 50 OFFSET 9950` and `WHERE id > 9950 LIMIT 50` return the same 50 rows, but the database reaches them differently. SQLite's `EXPLAIN QUERY PLAN` shows how:

```scala
transactor.connect {
  val plan = (q: Frag) => q.query[(Int, Int, Int, String)].map(_._4)
  println(plan(sql"EXPLAIN QUERY PLAN SELECT * FROM sale WHERE id > 9950 ORDER BY id ASC LIMIT 50"))
  println(plan(sql"EXPLAIN QUERY PLAN SELECT * FROM sale ORDER BY id ASC LIMIT 50 OFFSET 9950"))
}
// List(SEARCH sale USING INTEGER PRIMARY KEY (rowid>?))
// List(SCAN sale)
```

The keyset query is a `SEARCH`: SQLite jumps straight to the first ID after the cursor through the primary key and reads 50 rows. The `OFFSET` query is a `SCAN`: SQLite walks the table from the start and discards the first 9,950 rows before it returns any. So the cost of an `OFFSET` page grows with its page number, while a keyset page reads only the rows it returns. PostgreSQL behaves the same way: its `OFFSET` also has to produce and skip every earlier row.

Keyset pages are also stable while the table changes. A row inserted before the cursor doesn't push later rows onto the next page, and a deleted row doesn't pull one back, because each page starts from a value, not from a position.

The trade-off: a keyset reader can only go to the next page, not jump to page 137. That fits infinite scrolling, exports and batch jobs; a numbered page picker still needs `OFFSET`.

## Page Through Your Own Query with keysetAfter

`pageAfter` always reads the whole table. To page through a filtered query, such as the sales of one book, build the query yourself and append `Frag.keysetAfter(table, orderCol, lastValue, limit)`. It produces `WHERE <orderCol> > ? ORDER BY <orderCol> ASC LIMIT <limit>` with `lastValue` as the one bound parameter:

```scala
println(Frag.keysetAfter(sales.table, "id", DbValue.DbLong(0L), 5).sql(SqlDialect.SQLite))
//  WHERE id > ? ORDER BY id ASC LIMIT 5
```

Because the fragment starts with its own `WHERE`, our filter goes into a subquery, and `keysetAfter` pages through its result. The alias `AS s` is optional on SQLite but required by PostgreSQL before version 16:

```scala
def salesOfBookAfter(bookId: Long, cursor: Long, limit: Int): Frag =
  sql"SELECT * FROM (SELECT id, book_id, quantity FROM sale WHERE book_id = $bookId) AS s" ++
    Frag.keysetAfter(sales.table, "id", DbValue.DbLong(cursor), limit)
```

```scala
println(salesOfBookAfter(2L, 0L, 5).sql(SqlDialect.SQLite))
// SELECT * FROM (SELECT id, book_id, quantity FROM sale WHERE book_id = ?) AS s WHERE id > ? ORDER BY id ASC LIMIT 5

transactor.connect {
  val first = salesOfBookAfter(2L, 0L, 5).query[Sale]
  val next  = salesOfBookAfter(2L, first.last.id, 5).query[Sale]
  println(first)
  println(next)
}
// List(Sale(1,2,2), Sale(4,2,5), Sale(7,2,3), Sale(10,2,1), Sale(13,2,4))
// List(Sale(16,2,2), Sale(19,2,5), Sale(22,2,3), Sale(25,2,1), Sale(28,2,4))
```

The cursor is a `DbValue`, so it works for any column type: `DbValue.DbLong` here, `DbValue.DbString` for a text key, `DbValue.DbInstant` for a timestamp. `keysetAfter` checks `orderCol` when the fragment is built: it must be a plain identifier (letters, digits and underscores) and, with the table argument, one of the table's columns. The overload without a table, `Frag.keysetAfter(orderCol, lastValue, limit)`, checks only the identifier, for queries that don't map to one `Table`:

```scala
println(Try(Frag.keysetAfter(sales.table, "sold_at", DbValue.DbLong(0L), 5)))
// Failure(java.lang.IllegalArgumentException: Column 'sold_at' not found in table 'sale' columns: id, book_id, quantity)
println(Try(Frag.keysetAfter(sales.table, "id", DbValue.DbLong(0L), 0)))
// Failure(java.lang.IllegalArgumentException: requirement failed: Frag.keysetAfter: limit must be > 0, got 0)
```

## What Keyset Cursors Can't Do

Keep these limits in mind when you choose a cursor column:

- **One column only.** Both `pageAfter` and `keysetAfter` take a single cursor column. Cursors over several columns, such as `(price, id)`, are not supported.
- **The column must be ordered and unique.** Rows that share a cursor value can be split across a page boundary, and the ones after the boundary are then skipped, because the next page starts strictly after that value. Page by the ID, or by another unique column.
- **Ascending, forward only.** Both helpers emit `ORDER BY ... ASC` and `>`. There is no descending order and no "previous page".
- **`keysetAfter` brings its own `WHERE`.** Put your filters in a subquery, as shown above.

The fragment below is the SQL `keysetAfter` produces: always `>`, always `ASC`, with one bound parameter, the cursor value:

```scala
println(Frag.keysetAfter("id", DbValue.DbLong(50L), 5).sql(SqlDialect.SQLite))
//  WHERE id > ? ORDER BY id ASC LIMIT 5
```

:::caution
`Frag.keysetAfter` writes the column name without quotes, unlike `Repo` and `Upsert`, which double-quote every identifier. A column whose name is an SQL keyword, such as `order`, passes validation but produces invalid SQL (`near "order": syntax error` on SQLite). `Repo#pageAfter` quotes its ID column and isn't affected; for `keysetAfter`, use a column name that isn't a keyword.
:::

## Putting It Together

The complete example creates the keyed inventory table, applies a supplier batch, imports catalog defaults with `DO NOTHING`, applies a price-and-stock feed, and then walks the 10,000-row sales log both with `pageAfter` and through one book's sales with `keysetAfter`:

```scala title="sql-examples/src/main/scala/upsertandpaginatewithkeysetcursors/CompleteExample.scala"
package upsertandpaginatewithkeysetcursors

/**
 * How to Upsert Rows and Paginate with Keyset Cursors — Complete Example
 *
 * Creates the keyed inventory table, applies a supplier batch, imports catalog
 * defaults with DO NOTHING, applies a price-and-stock feed, then walks the
 * 10,000-row sales log with pageAfter and through one book's sales with
 * keysetAfter.
 *
 * Run with: cd sql-examples && sbt "runMain
 * upsertandpaginatewithkeysetcursors.CompleteExample"
 */
object CompleteExample {

  case class Book(id: Long, title: String, isbn: Option[String], price: BigDecimal, stock: Int)

  object Book {
    given Schema[Book] = Schema.derived
  }

  val books = Repo.derived[Book, Long]

  // Prices and stock are fictional
  val zionomicon         = Book(1L, "Zionomicon", None, BigDecimal("39.99"), 5)
  val programmingInScala = Book(2L, "Programming in Scala", Some("9780981531601"), BigDecimal("49.95"), 3)
  val fpInScala          = Book(3L, "Functional Programming in Scala", Some("9781617290657"), BigDecimal("44.99"), 8)

  case class Sale(id: Long, bookId: Long, quantity: Int)

  object Sale {
    given Schema[Sale] = Schema.derived
  }

  val sales = Repo.derived[Sale, Long]

  @tailrec
  def walkSales(cursor: Long, pages: Int, rows: Int, sold: Long)(using DbCon): (Int, Int, Long) = {
    val page = sales.pageAfter(cursor, 50)
    if (page.isEmpty) (pages, rows, sold)
    else walkSales(page.last.id, pages + 1, rows + page.size, sold + page.map(_.quantity).sum)
  }

  def salesOfBookAfter(bookId: Long, cursor: Long, limit: Int): Frag =
    sql"SELECT * FROM (SELECT id, book_id, quantity FROM sale WHERE book_id = $bookId) AS s" ++
      Frag.keysetAfter(sales.table, "id", DbValue.DbLong(cursor), limit)

  def printBooks()(using DbCon): Unit =
    sql"SELECT id, title, isbn, price, stock FROM book ORDER BY id".query[Book].foreach { b =>
      println(f"${b.id}%2d  ${b.title}%-32s ${b.isbn.getOrElse("-")}%-14s ${b.price}%6s  stock ${b.stock}%2d")
    }

  def main(args: Array[String]): Unit = {
    val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.connect {
      sql"""CREATE TABLE book (
        id    INTEGER PRIMARY KEY,
        title TEXT NOT NULL,
        isbn  TEXT,
        price TEXT NOT NULL,
        stock INTEGER NOT NULL
      )""".update
      println(books.insertOrUpdate(zionomicon.copy(stock = 4)))
      println(books.insertOrUpdate(programmingInScala))
      println(books.insertOrUpdate(fpInScala))
      printBooks()
    }

    val supplierFeed = List(
      programmingInScala.copy(price = BigDecimal("44.95")),                // price drop
      fpInScala.copy(stock = 12),                                          // restock
      Book(4L, "Hands-on Scala Programming", None, BigDecimal("39.00"), 6) // new title
    )

    transactor.transact {
      println("Before:")
      printBooks()
      println(s"Rows affected: ${books.insertOrUpdateBatch(supplierFeed)}")
      println("After:")
      printBooks()
    }

    val catalogDefaults = List(
      zionomicon,                                                 // stock 5 in the catalog, 4 in the shop
      Book(5L, "Programming Scala", None, BigDecimal("59.99"), 2) // not in the shop yet
    )

    transactor.connect {
      catalogDefaults.foreach { book =>
        val inserted = Upsert.insertDoNothing(books.table, book, "id").update
        println(s"${book.title}: $inserted")
      }
      printBooks()
    }

    val supplierRow   = zionomicon.copy(title = "ZIONOMICON (PAPERBACK)", price = BigDecimal("34.99"), stock = 20)
    val priceAndStock = Upsert.insertDoUpdate(books.table, supplierRow, "id", Seq("price", "stock"))

    println(priceAndStock.sql(SqlDialect.SQLite))

    transactor.connect {
      println(priceAndStock.update)
      printBooks()
    }

    transactor.transact {
      sql"CREATE TABLE sale (id INTEGER PRIMARY KEY, book_id INTEGER NOT NULL, quantity INTEGER NOT NULL)".update
      val generated = (1L to 10000L).map(i => Sale(i, i % 3 + 1, (i % 5 + 1).toInt))
      println(sales.insertBatch(generated))
    }

    transactor.connect {
      val first  = sales.pageAfter(0L, 50)
      val second = sales.pageAfter(first.last.id, 50)
      println(s"First page:  ${first.size} rows, ids ${first.head.id} to ${first.last.id}")
      println(s"Second page: ${second.size} rows, ids ${second.head.id} to ${second.last.id}")
      val (pages, rows, sold) = walkSales(0L, 0, 0, 0L)
      println(s"$pages pages, $rows rows, $sold copies sold")
    }

    transactor.connect {
      val first = salesOfBookAfter(2L, 0L, 5).query[Sale]
      val next  = salesOfBookAfter(2L, first.last.id, 5).query[Sale]
      println(first)
      println(next)
    }
  }
}
```

## Running the Examples

All examples in this guide have corresponding runnable Scala files in the `sql-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/sql-examples
```

Each example creates its own temporary database file, so you can run them in any order and as often as you like.

### Create a Table with a Real Primary Key

This example shows the upsert failing on the table `createTable` makes, then succeeding on a table with a primary key.

<details>
  <summary>sql-examples/src/main/scala/upsertandpaginatewithkeysetcursors/Step1KeyedTableExample.scala</summary>

```scala title="sql-examples/src/main/scala/upsertandpaginatewithkeysetcursors/Step1KeyedTableExample.scala" showLineNumbers
package upsertandpaginatewithkeysetcursors

/**
 * How to Upsert Rows and Paginate with Keyset Cursors — Step 1: Create a Table
 * with a Real Primary Key
 *
 * Shows the upsert failing on the table createTable makes, then succeeding on a
 * table with a primary key.
 *
 * Run with: cd sql-examples && sbt "runMain
 * upsertandpaginatewithkeysetcursors.Step1KeyedTableExample"
 */
object Step1KeyedTableExample {

  case class Book(id: Long, title: String, isbn: Option[String], price: BigDecimal, stock: Int)

  object Book {
    given Schema[Book] = Schema.derived
  }

  val books = Repo.derived[Book, Long]

  // Prices and stock are fictional
  val zionomicon         = Book(1L, "Zionomicon", None, BigDecimal("39.99"), 5)
  val programmingInScala = Book(2L, "Programming in Scala", Some("9780981531601"), BigDecimal("49.95"), 3)
  val fpInScala          = Book(3L, "Functional Programming in Scala", Some("9781617290657"), BigDecimal("44.99"), 8)

  def printBooks()(using DbCon): Unit =
    sql"SELECT id, title, isbn, price, stock FROM book ORDER BY id".query[Book].foreach { b =>
      println(f"${b.id}%2d  ${b.title}%-32s ${b.isbn.getOrElse("-")}%-14s ${b.price}%6s  stock ${b.stock}%2d")
    }

  def main(args: Array[String]): Unit = {
    val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    println(books.table.createTable(SqlDialect.SQLite).sql(SqlDialect.SQLite))

    transactor.connect {
      books.table.createTable(SqlDialect.SQLite).update
      println(Try(books.insertOrUpdate(zionomicon)))
      sql"DROP TABLE book".update
    }

    transactor.connect {
      sql"""CREATE TABLE book (
        id    INTEGER PRIMARY KEY,
        title TEXT NOT NULL,
        isbn  TEXT,
        price TEXT NOT NULL,
        stock INTEGER NOT NULL
      )""".update
      println(Try(books.insertOrUpdate(zionomicon)))
      printBooks()
    }
  }
}
```

</details>

```bash
sbt "runMain upsertandpaginatewithkeysetcursors.Step1KeyedTableExample"
```

### Insert or Update Books

This example saves single books with `insertOrUpdate` and then applies a nightly supplier batch with `insertOrUpdateBatch`.

<details>
  <summary>sql-examples/src/main/scala/upsertandpaginatewithkeysetcursors/Step2InsertOrUpdateExample.scala</summary>

```scala title="sql-examples/src/main/scala/upsertandpaginatewithkeysetcursors/Step2InsertOrUpdateExample.scala" showLineNumbers
package upsertandpaginatewithkeysetcursors

/**
 * How to Upsert Rows and Paginate with Keyset Cursors — Step 2: Insert or
 * Update Books
 *
 * Saves single books with insertOrUpdate, then applies a nightly supplier batch
 * with insertOrUpdateBatch.
 *
 * Run with: cd sql-examples && sbt "runMain
 * upsertandpaginatewithkeysetcursors.Step2InsertOrUpdateExample"
 */
object Step2InsertOrUpdateExample {

  case class Book(id: Long, title: String, isbn: Option[String], price: BigDecimal, stock: Int)

  object Book {
    given Schema[Book] = Schema.derived
  }

  val books = Repo.derived[Book, Long]

  // Prices and stock are fictional
  val zionomicon         = Book(1L, "Zionomicon", None, BigDecimal("39.99"), 5)
  val programmingInScala = Book(2L, "Programming in Scala", Some("9780981531601"), BigDecimal("49.95"), 3)
  val fpInScala          = Book(3L, "Functional Programming in Scala", Some("9781617290657"), BigDecimal("44.99"), 8)

  def printBooks()(using DbCon): Unit =
    sql"SELECT id, title, isbn, price, stock FROM book ORDER BY id".query[Book].foreach { b =>
      println(f"${b.id}%2d  ${b.title}%-32s ${b.isbn.getOrElse("-")}%-14s ${b.price}%6s  stock ${b.stock}%2d")
    }

  def main(args: Array[String]): Unit = {
    val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.connect {
      sql"""CREATE TABLE book (
        id    INTEGER PRIMARY KEY,
        title TEXT NOT NULL,
        isbn  TEXT,
        price TEXT NOT NULL,
        stock INTEGER NOT NULL
      )""".update
      books.insertOrUpdate(zionomicon)
    }

    transactor.connect {
      println(books.insertOrUpdate(programmingInScala))
      println(books.insertOrUpdate(fpInScala))
      println(books.insertOrUpdate(zionomicon.copy(stock = 4)))
      printBooks()
    }

    val supplierFeed = List(
      programmingInScala.copy(price = BigDecimal("44.95")),                // price drop
      fpInScala.copy(stock = 12),                                          // restock
      Book(4L, "Hands-on Scala Programming", None, BigDecimal("39.00"), 6) // new title
    )

    transactor.transact {
      println("Before:")
      printBooks()
      println(s"Rows affected: ${books.insertOrUpdateBatch(supplierFeed)}")
      println("After:")
      printBooks()
    }
  }
}
```

</details>

```bash
sbt "runMain upsertandpaginatewithkeysetcursors.Step2InsertOrUpdateExample"
```

### Keep Existing Rows Untouched

This example imports catalog defaults with `Upsert.insertDoNothing` and prints which rows were inserted.

<details>
  <summary>sql-examples/src/main/scala/upsertandpaginatewithkeysetcursors/Step3DoNothingExample.scala</summary>

```scala title="sql-examples/src/main/scala/upsertandpaginatewithkeysetcursors/Step3DoNothingExample.scala" showLineNumbers
package upsertandpaginatewithkeysetcursors

/**
 * How to Upsert Rows and Paginate with Keyset Cursors — Step 3: Keep Existing
 * Rows Untouched
 *
 * Imports catalog defaults with Upsert.insertDoNothing and prints which rows
 * were inserted.
 *
 * Run with: cd sql-examples && sbt "runMain
 * upsertandpaginatewithkeysetcursors.Step3DoNothingExample"
 */
object Step3DoNothingExample {

  case class Book(id: Long, title: String, isbn: Option[String], price: BigDecimal, stock: Int)

  object Book {
    given Schema[Book] = Schema.derived
  }

  val books = Repo.derived[Book, Long]

  // Prices and stock are fictional
  val zionomicon         = Book(1L, "Zionomicon", None, BigDecimal("39.99"), 5)
  val programmingInScala = Book(2L, "Programming in Scala", Some("9780981531601"), BigDecimal("49.95"), 3)
  val fpInScala          = Book(3L, "Functional Programming in Scala", Some("9781617290657"), BigDecimal("44.99"), 8)

  def printBooks()(using DbCon): Unit =
    sql"SELECT id, title, isbn, price, stock FROM book ORDER BY id".query[Book].foreach { b =>
      println(f"${b.id}%2d  ${b.title}%-32s ${b.isbn.getOrElse("-")}%-14s ${b.price}%6s  stock ${b.stock}%2d")
    }

  def main(args: Array[String]): Unit = {
    val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.connect {
      sql"""CREATE TABLE book (
        id    INTEGER PRIMARY KEY,
        title TEXT NOT NULL,
        isbn  TEXT,
        price TEXT NOT NULL,
        stock INTEGER NOT NULL
      )""".update
      books.insertOrUpdate(programmingInScala)
      books.insertOrUpdate(fpInScala)
      books.insertOrUpdate(zionomicon.copy(stock = 4))
    }

    val catalogDefaults = List(
      zionomicon,                                                 // stock 5 in the catalog, 4 in the shop
      Book(5L, "Programming Scala", None, BigDecimal("59.99"), 2) // not in the shop yet
    )

    transactor.connect {
      catalogDefaults.foreach { book =>
        val inserted = Upsert.insertDoNothing(books.table, book, "id").update
        println(s"${book.title}: $inserted")
      }
      printBooks()
    }
  }
}
```

</details>

```bash
sbt "runMain upsertandpaginatewithkeysetcursors.Step3DoNothingExample"
```

### Update Only the Columns You Choose

This example applies a supplier row that updates price and stock but keeps the title.

<details>
  <summary>sql-examples/src/main/scala/upsertandpaginatewithkeysetcursors/Step4ChosenColumnsExample.scala</summary>

```scala title="sql-examples/src/main/scala/upsertandpaginatewithkeysetcursors/Step4ChosenColumnsExample.scala" showLineNumbers
package upsertandpaginatewithkeysetcursors

/**
 * How to Upsert Rows and Paginate with Keyset Cursors — Step 4: Update Only the
 * Columns You Choose
 *
 * Applies a supplier row that updates price and stock but keeps the title.
 *
 * Run with: cd sql-examples && sbt "runMain
 * upsertandpaginatewithkeysetcursors.Step4ChosenColumnsExample"
 */
object Step4ChosenColumnsExample {

  case class Book(id: Long, title: String, isbn: Option[String], price: BigDecimal, stock: Int)

  object Book {
    given Schema[Book] = Schema.derived
  }

  val books = Repo.derived[Book, Long]

  // Prices and stock are fictional
  val zionomicon         = Book(1L, "Zionomicon", None, BigDecimal("39.99"), 5)
  val programmingInScala = Book(2L, "Programming in Scala", Some("9780981531601"), BigDecimal("49.95"), 3)
  val fpInScala          = Book(3L, "Functional Programming in Scala", Some("9781617290657"), BigDecimal("44.99"), 8)

  def printBooks()(using DbCon): Unit =
    sql"SELECT id, title, isbn, price, stock FROM book ORDER BY id".query[Book].foreach { b =>
      println(f"${b.id}%2d  ${b.title}%-32s ${b.isbn.getOrElse("-")}%-14s ${b.price}%6s  stock ${b.stock}%2d")
    }

  def main(args: Array[String]): Unit = {
    val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.connect {
      sql"""CREATE TABLE book (
        id    INTEGER PRIMARY KEY,
        title TEXT NOT NULL,
        isbn  TEXT,
        price TEXT NOT NULL,
        stock INTEGER NOT NULL
      )""".update
      books.insertOrUpdate(zionomicon)
    }

    val supplierRow   = zionomicon.copy(title = "ZIONOMICON (PAPERBACK)", price = BigDecimal("34.99"), stock = 20)
    val priceAndStock = Upsert.insertDoUpdate(books.table, supplierRow, "id", Seq("price", "stock"))

    println(priceAndStock.sql(SqlDialect.SQLite))

    transactor.connect {
      println(priceAndStock.update)
      printBooks()
    }

    println(Try(Upsert.insertDoUpdate(books.table, supplierRow, "id", Seq("cost"))))
    println(Try(Upsert.insertDoUpdate(books.table, supplierRow, "id", Seq("id", "price"))))
  }
}
```

</details>

```bash
sbt "runMain upsertandpaginatewithkeysetcursors.Step4ChosenColumnsExample"
```

### Compare the Upsert SQL for SQLite and PostgreSQL

This example prints the upsert statement and the `CREATE TABLE` statement for both dialects. It needs no database.

<details>
  <summary>sql-examples/src/main/scala/upsertandpaginatewithkeysetcursors/Step5DialectSqlExample.scala</summary>

```scala title="sql-examples/src/main/scala/upsertandpaginatewithkeysetcursors/Step5DialectSqlExample.scala" showLineNumbers
package upsertandpaginatewithkeysetcursors

/**
 * How to Upsert Rows and Paginate with Keyset Cursors — Step 5: Compare the
 * Upsert SQL for SQLite and PostgreSQL
 *
 * Prints the upsert statement and the CREATE TABLE statement for both dialects.
 * Needs no database.
 *
 * Run with: cd sql-examples && sbt "runMain
 * upsertandpaginatewithkeysetcursors.Step5DialectSqlExample"
 */
object Step5DialectSqlExample {

  case class Book(id: Long, title: String, isbn: Option[String], price: BigDecimal, stock: Int)

  object Book {
    given Schema[Book] = Schema.derived
  }

  val books = Repo.derived[Book, Long]

  // Prices and stock are fictional
  val zionomicon         = Book(1L, "Zionomicon", None, BigDecimal("39.99"), 5)
  val programmingInScala = Book(2L, "Programming in Scala", Some("9780981531601"), BigDecimal("49.95"), 3)
  val fpInScala          = Book(3L, "Functional Programming in Scala", Some("9781617290657"), BigDecimal("44.99"), 8)

  def main(args: Array[String]): Unit = {
    val upsert = Upsert.insertDoUpdate(books.table, zionomicon, "id")

    println(upsert.sql(SqlDialect.SQLite))
    println(upsert.sql(SqlDialect.PostgreSQL))
    println(upsert.sql(SqlDialect.SQLite) == upsert.sql(SqlDialect.PostgreSQL))
    println(upsert.queryParams)

    println(books.table.createTable(SqlDialect.PostgreSQL).sql(SqlDialect.PostgreSQL))
  }
}
```

</details>

```bash
sbt "runMain upsertandpaginatewithkeysetcursors.Step5DialectSqlExample"
```

### Walk a Large Table with pageAfter

This example fills a 10,000-row sales table, walks it in pages of 50, and prints SQLite's query plans for a keyset page and an `OFFSET` page.

<details>
  <summary>sql-examples/src/main/scala/upsertandpaginatewithkeysetcursors/Step6PageAfterExample.scala</summary>

```scala title="sql-examples/src/main/scala/upsertandpaginatewithkeysetcursors/Step6PageAfterExample.scala" showLineNumbers
package upsertandpaginatewithkeysetcursors

/**
 * How to Upsert Rows and Paginate with Keyset Cursors — Step 6: Walk a Large
 * Table with pageAfter
 *
 * Fills a 10,000-row sales table, walks it in pages of 50, and prints SQLite's
 * query plans for a keyset page and an OFFSET page.
 *
 * Run with: cd sql-examples && sbt "runMain
 * upsertandpaginatewithkeysetcursors.Step6PageAfterExample"
 */
object Step6PageAfterExample {

  case class Sale(id: Long, bookId: Long, quantity: Int)

  object Sale {
    given Schema[Sale] = Schema.derived
  }

  val sales = Repo.derived[Sale, Long]

  @tailrec
  def walkSales(cursor: Long, pages: Int, rows: Int, sold: Long)(using DbCon): (Int, Int, Long) = {
    val page = sales.pageAfter(cursor, 50)
    if (page.isEmpty) (pages, rows, sold)
    else walkSales(page.last.id, pages + 1, rows + page.size, sold + page.map(_.quantity).sum)
  }

  def main(args: Array[String]): Unit = {
    val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.transact {
      sql"CREATE TABLE sale (id INTEGER PRIMARY KEY, book_id INTEGER NOT NULL, quantity INTEGER NOT NULL)".update
      val generated = (1L to 10000L).map(i => Sale(i, i % 3 + 1, (i % 5 + 1).toInt))
      println(sales.insertBatch(generated))
    }

    transactor.connect {
      val first  = sales.pageAfter(0L, 50)
      val second = sales.pageAfter(first.last.id, 50)
      println(s"First page:  ${first.size} rows, ids ${first.head.id} to ${first.last.id}")
      println(s"Second page: ${second.size} rows, ids ${second.head.id} to ${second.last.id}")
      println(s"After the last id: ${sales.pageAfter(10000L, 50)}")
    }

    transactor.connect {
      val (pages, rows, sold) = walkSales(0L, 0, 0, 0L)
      println(s"$pages pages, $rows rows, $sold copies sold")
    }

    transactor.connect {
      val plan = (q: Frag) => q.query[(Int, Int, Int, String)].map(_._4)
      println(plan(sql"EXPLAIN QUERY PLAN SELECT * FROM sale WHERE id > 9950 ORDER BY id ASC LIMIT 50"))
      println(plan(sql"EXPLAIN QUERY PLAN SELECT * FROM sale ORDER BY id ASC LIMIT 50 OFFSET 9950"))
    }
  }
}
```

</details>

```bash
sbt "runMain upsertandpaginatewithkeysetcursors.Step6PageAfterExample"
```

### Page Through Your Own Query with keysetAfter

This example pages through the sales of one book with `Frag.keysetAfter`.

<details>
  <summary>sql-examples/src/main/scala/upsertandpaginatewithkeysetcursors/Step7KeysetAfterExample.scala</summary>

```scala title="sql-examples/src/main/scala/upsertandpaginatewithkeysetcursors/Step7KeysetAfterExample.scala" showLineNumbers
package upsertandpaginatewithkeysetcursors

/**
 * How to Upsert Rows and Paginate with Keyset Cursors — Step 7: Page Through
 * Your Own Query with keysetAfter
 *
 * Pages through the sales of one book with Frag.keysetAfter.
 *
 * Run with: cd sql-examples && sbt "runMain
 * upsertandpaginatewithkeysetcursors.Step7KeysetAfterExample"
 */
object Step7KeysetAfterExample {

  case class Sale(id: Long, bookId: Long, quantity: Int)

  object Sale {
    given Schema[Sale] = Schema.derived
  }

  val sales = Repo.derived[Sale, Long]

  def salesOfBookAfter(bookId: Long, cursor: Long, limit: Int): Frag =
    sql"SELECT * FROM (SELECT id, book_id, quantity FROM sale WHERE book_id = $bookId) AS s" ++
      Frag.keysetAfter(sales.table, "id", DbValue.DbLong(cursor), limit)

  def main(args: Array[String]): Unit = {
    val dbFile = java.nio.file.Files.createTempFile("bookstore", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.transact {
      sql"CREATE TABLE sale (id INTEGER PRIMARY KEY, book_id INTEGER NOT NULL, quantity INTEGER NOT NULL)".update
      val generated = (1L to 10000L).map(i => Sale(i, i % 3 + 1, (i % 5 + 1).toInt))
      println(sales.insertBatch(generated))
    }

    println(Frag.keysetAfter(sales.table, "id", DbValue.DbLong(0L), 5).sql(SqlDialect.SQLite))
    println(salesOfBookAfter(2L, 0L, 5).sql(SqlDialect.SQLite))

    transactor.connect {
      val first = salesOfBookAfter(2L, 0L, 5).query[Sale]
      val next  = salesOfBookAfter(2L, first.last.id, 5).query[Sale]
      println(first)
      println(next)
    }

    println(Try(Frag.keysetAfter(sales.table, "sold_at", DbValue.DbLong(0L), 5)))
    println(Try(Frag.keysetAfter(sales.table, "id", DbValue.DbLong(0L), 0)))
  }
}
```

</details>

```bash
sbt "runMain upsertandpaginatewithkeysetcursors.Step7KeysetAfterExample"
```

### The Complete Example

```bash
sbt "runMain upsertandpaginatewithkeysetcursors.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## Going Further

- [Repo](../reference/sql/repo.md) documents `insertOrUpdate`, `insertOrUpdateBatch` and `pageAfter` alongside the rest of the repository API.
- [Frag](../reference/sql/frag.md) covers building, combining and running SQL fragments.
- [Table](../reference/sql/table.md) and [Ddl](../reference/sql/ddl.md) explain what `createTable` generates, and why it has no primary key.
- [SQL Transactions: Isolation, Read-Only and Savepoints](./sql-transactions.md) shows how to group upserts with other writes in one transaction.
- [Query DSL with Reified Optics — Part 2: SQL Generation](./query-dsl-sql.md) builds `SELECT` statements from optics instead of hand-written SQL.

---

## How to Use Iron Refined Types and Typed IDs with Schema

In this guide, we build the input model of a small online shop: customers, products and orders that arrive as JSON. Each field with a rule, such as an email address, a product code or a price, gets an [Iron](https://github.com/Iltotore/iron) refined type like `String :| MinLength[1]`, and Iron's `iron-zio-blocks-schema` module turns each one into a `Schema`. We derive the case classes' schemas as usual, so every JSON codec checks the rules while it decodes. Then we give customer and order IDs their own types with one opaque `Id[+A]`, so the compiler stops us from passing an order ID where a customer ID belongs.

We take an incremental approach:

- Refine the fields of a case class and derive its `Schema` and JSON codec
- Decode JSON and read Iron's constraint failures as `SchemaError`s with field paths
- Name a refined type with Iron's `RefinedType`
- Refine numeric fields with a validated newtype, because of a known bug in ZIO Blocks 0.0.52 to 0.0.56
- See which constraints reach JSON Schema and `DynamicSchema`
- Keep IDs apart with one generic `Id[+A]` and a single `Schema` for every entity
- Choose between Iron and plain validated newtypes

## The Problem

A shop model that stores an email as `String`, a price as `BigDecimal` and every ID as `Long` accepts any string, any number and any ID. A blank customer name, a negative price and a customer ID passed as an order ID all type-check, and nothing stops them until a database constraint or a confused support ticket does.

The usual fix is a validation function and some careful parameter names, and both depend on discipline. Nothing forces a caller to run `validate`, and nothing stops a caller from swapping two `Long` arguments.

For example, a typical model and its checks look like this:

```scala
final case class CustomerRow(id: Long, name: String, email: String)
final case class OrderRow(id: Long, customer: Long, total: BigDecimal)

def validate(customer: CustomerRow): Either[String, CustomerRow] =
  if (customer.name.isEmpty || customer.name.length > 30) Left(s"bad name: ${customer.name}")
  else if (!customer.email.matches("^[^@\\s]+@[^@\\s]+\\.[a-z]+$")) Left(s"bad email: ${customer.email}")
  else Right(customer)

def cancelOrder(orderId: Long, customerId: Long): String = s"order $orderId cancelled for customer $customerId"

// Compiles and runs: the checks are skipped and the IDs are swapped.
val unchecked = CustomerRow(1L, "", "not-an-email")
cancelOrder(unchecked.id, 10L)
```

The rules live apart from the types they protect, the JSON decoder knows nothing about them, and the two IDs are interchangeable.

In this guide, we move each rule into a type. Iron writes the rule in the type itself (`String :| MinLength[1]`), ZIO Blocks enforces it in every codec derived from the schema, and a typed `Id[Order]` can't stand in for an `Id[Customer]`.

## Prerequisites

This guide builds on [Your First Schema](./your-first-schema.md): you should know how to derive a `Schema` for a case class and decode JSON with it. [How to Define Validated Newtypes](./validated-newtypes.md) shows the plain ZIO Blocks way to attach a check to a type, which we compare with Iron at the end.

Iron is a Scala 3 library, and its 3.4 releases are built with Scala 3.9, so this guide needs Scala 3.9 or newer. Add the schema module and Iron's ZIO Blocks module, which brings Iron itself, to your `build.sbt`:

```scala
scalaVersion := "3.9.0"

libraryDependencies ++= Seq(
  "dev.zio"            %% "zio-blocks-next-schema" % "0.0.56",
  ("io.github.iltotore" %% "iron-zio-blocks-schema" % "3.4.1")
    .exclude("dev.zio", "zio-blocks-schema_3")
)
scalacOptions += "-experimental"
```

On Scala 3.9 and newer, ZIO Blocks is published as `zio-blocks-next-*`, and those artifacts need `-experimental`. Iron's module depends on the other artifact line, `zio-blocks-schema` 0.0.51, which holds the same packages. The `exclude` keeps that second copy off the classpath, so Iron's instances run against the `zio-blocks-next-schema` we chose.

We use these imports throughout. The `zioBlocksSchema.given` import brings Iron's `Schema` instances into scope:

```scala

```

## Refine the Fields of a Case Class

An Iron refined type is a base type and a constraint joined by `:|`: `String :| MinLength[1]` is a `String` that Iron has checked to have at least one character. At runtime it is the plain `String`, so it costs nothing to carry around. We name each refined type with a type alias and use the aliases as field types.

For a customer, we refine the display name to 1 to 30 characters and the email to a simple pattern. We keep the pattern in its own alias, `EmailFormat`, so we can reuse it later. The case class derives its schema with `Schema.derived`, exactly as it would with `String` fields:

```scala
type EmailFormat = Match["^[^@\\s]+@[^@\\s]+\\.[a-z]+$"]
type Email       = String :| EmailFormat
type DisplayName = String :| (MinLength[1] & MaxLength[30])

final case class Customer(name: DisplayName, email: Email)

object Customer {
  implicit val schema: Schema[Customer] = Schema.derived
}
```

`Schema.derived` needs a `Schema` for each field type, and Iron's `zioBlocksSchema` provides one for every `A :| C` whose base type `A` has a schema and whose constraint `C` Iron can check at runtime. That instance wraps the base type's schema with a check, so decoding a field tests the constraint. Encoding writes the plain value:

```scala
val ada = Customer("Ada", "ada@example.com")
// ada: Customer = Customer(name = "Ada", email = "ada@example.com")

println(Customer.schema.jsonCodec.encodeToString(ada))
// {"name":"Ada","email":"ada@example.com"}
```

Iron checks a literal at compile time, so `Customer("Ada", "ada@example.com")` needs no explicit refinement. A literal that breaks the constraint doesn't compile:

```scala
val blank: DisplayName = ""
// error:
// -- Constraint Error --------------------------------------------------------
// Could not satisfy a constraint for type java.lang.String.
// 
// Value: 
// Message: Should have a minimum length of 1 & Should have a maximum length of 30
// ----------------------------------------------------------------------------
// val blank: DisplayName = ""
//                          ^^
```

## Read Constraint Failures from the Decoder

When the JSON holds a value that breaks a constraint, the decoder returns a `Left` with a `SchemaError`. Its message is Iron's description of the constraint, followed by the path of the field that failed.

A valid payload decodes to a `Customer`, and a blank name or a malformed email does not:

```scala
println(Customer.schema.jsonCodec.decode("""{"name":"Ada","email":"ada@example.com"}"""))
// Right(Customer(Ada,ada@example.com))
println(Customer.schema.jsonCodec.decode("""{"name":"","email":"ada@example.com"}""").left.map(_.message))
// Left(Should have a minimum length of 1 & Should have a maximum length of 30 at: .name.wrapped)
println(Customer.schema.jsonCodec.decode("""{"name":"Ada","email":"ada-at-example"}""").left.map(_.message))
// Left(Should match ^[^@\s]+@[^@\s]+\.[a-z]+$ at: .email.wrapped)
```

The path ends in `.wrapped` because ZIO Blocks sees a refined type as a wrapper around its base type: `.name.wrapped` is the `String` inside the `DisplayName` field. The message `Should have a minimum length of 1 & Should have a maximum length of 30` comes from Iron and describes both halves of the `&`.

Values that arrive outside JSON, from a form field or a command-line argument, go through Iron's `refineEither` extension method, which runs the same constraint and returns Iron's message on failure:

```scala
val fromForm: String = "grace@example.com"
// fromForm: String = "grace@example.com"

println(fromForm.refineEither[EmailFormat])
// Right(grace@example.com)
println("grace-at-example".refineEither[EmailFormat])
// Left(Should match ^[^@\s]+@[^@\s]+\.[a-z]+$)
```

## Name a Refined Type with RefinedType

A type alias like `DisplayName` is only a name: any `String :| (MinLength[1] & MaxLength[30])` is a `DisplayName`, whatever it was meant for. When a refined value should be its own type, Iron's `RefinedType` creates one, with a companion object that refines values for us. Iron's module gives these new types a `Schema` too.

For products, we make the stock-keeping unit (SKU) a `RefinedType` and refine the price with `Positive`:

```scala
type Sku = Sku.T
object Sku extends RefinedType[String, Match["^[A-Z]{3}-[0-9]{4}$"]]

type Price = BigDecimal :| Positive

final case class Product(sku: Sku, title: DisplayName, price: Price, tags: List[DisplayName])

object Product {
  implicit val schema: Schema[Product] = Schema.derived
}
```

`Sku.apply` refines a literal at compile time and `Sku.either` refines a runtime value, so the companion replaces both the literal check and `refineEither`:

```scala
println(Sku("KEY-0042"))
// KEY-0042
println(Sku.either("key-42"))
// Left(Should match ^[A-Z]{3}-[0-9]{4}$)
```

The decoder checks every refined field, including each element of a collection. A wrong SKU, a negative price and a blank tag each fail with their own path:

```scala
println(Product.schema.jsonCodec.decode("""{"sku":"KEY-0042","title":"Mechanical keyboard","price":89.90,"tags":["usb","wired"]}"""))
// Right(Product(KEY-0042,Mechanical keyboard,89.90,List(usb, wired)))
println(Product.schema.jsonCodec.decode("""{"sku":"key-42","title":"Mechanical keyboard","price":89.90,"tags":[]}""").left.map(_.message))
// Left(Should match ^[A-Z]{3}-[0-9]{4}$ at: .sku.wrapped.wrapped)
println(Product.schema.jsonCodec.decode("""{"sku":"KEY-0042","title":"Mechanical keyboard","price":-1,"tags":[]}""").left.map(_.message))
// Left(Should be strictly positive at: .price.wrapped)
println(Product.schema.jsonCodec.decode("""{"sku":"KEY-0042","title":"Mechanical keyboard","price":89.90,"tags":[""]}""").left.map(_.message))
// Left(Should have a minimum length of 1 & Should have a maximum length of 30 at: .tags.at(0).wrapped)
```

The SKU's path has two `.wrapped` steps: one for the `RefinedType` around the refined `String`, and one for the refinement around the `String` itself. `.tags.at(0)` points at the first element of the list.

## Refine Numeric Fields with a Validated Newtype

An order line needs a quantity, and `Int :| Positive` is the obvious type for it. On ZIO Blocks 0.0.56, a refined field over a JVM primitive doesn't work inside a derived case class.

:::warning[Numeric Iron fields in case classes]
A known bug in ZIO Blocks 0.0.52 to 0.0.56: an Iron field whose base type is `Int`, `Long`, `Double`, `Float`, `Short`, `Byte`, `Char` or `Boolean` decodes as `0` (or `false`) inside a derived case class, even when the input is valid, and the constraint never sees the real value. With fields of several types, decoding can fail with a `ClassCastException` instead. A standalone `Schema[Int :| Positive]` works, and so do refinements of `String`, `BigDecimal` and `BigInt`. Until a fixed release, use a validated newtype for numeric fields, as below.
:::

A validated newtype is a one-field case class with a private constructor and a smart constructor (a function that checks its input before building the value). Its `Schema` comes from `Schema#transform`, which takes a function from the base type and a function back. The `to` function returns a plain `Quantity`, so a failed check rethrows the `SchemaError`, and the decoder turns it into a `Left`:

```scala
final case class Quantity private (value: Int)

object Quantity {
  def parse(value: Int): Either[SchemaError, Quantity] =
    if (value > 0) Right(new Quantity(value))
    else Left(SchemaError.validationFailed(s"quantity must be positive, got $value"))

  implicit val schema: Schema[Quantity] =
    Schema[Int].transform(n => parse(n).fold(e => throw e, identity), _.value)
}

final case class Line(sku: Sku, quantity: Quantity)

object Line {
  implicit val schema: Schema[Line] = Schema.derived
}
```

The line mixes an Iron field and a newtype field, and both rules apply on decode:

```scala
println(Line.schema.jsonCodec.decode("""{"sku":"KEY-0042","quantity":2}"""))
// Right(Line(KEY-0042,Quantity(2)))
println(Line.schema.jsonCodec.decode("""{"sku":"KEY-0042","quantity":0}""").left.map(_.message))
// Left(quantity must be positive, got 0 at: .quantity.wrapped)
```

## See Where the Constraints Go

Iron's constraints protect the decoder, and some of them also travel with the schema. For each constraint that has an exact equivalent among ZIO Blocks' [`Validation`](../reference/schema/validation.md) rules (positive and negative numbers, inclusive bounds, exact values, string patterns and string lengths), Iron's module attaches that `Validation` to the base type's schema. It also stores Iron's message on the schema as a `Modifier.config` entry named `iron.validation.message`.

`Schema#toJsonSchema` doesn't emit `Validation` rules, so the JSON Schema of `Customer` describes both fields as plain strings, without `minLength`, `maxLength` or `pattern`:

```scala
println(Customer.schema.toJsonSchema.toJson)
// {
//   "properties": {
//     "name": {
//       "type": "string"
//     },
//     "email": {
//       "type": "string"
//     }
//   },
//   "type": "object",
//   "required": [
//     "name",
//     "email"
//   ],
//   "title": "Customer"
// }
```

`DynamicSchema` does carry them. `Schema#toDynamicSchema` turns a schema into a value that can be stored or sent to another service, and `DynamicSchema#check` tests a `DynamicValue` against it without a decoder. The blank name fails with the `Validation` rule that Iron attached, not with Iron's own message:

```scala
val blankName = DynamicValue.Record(
  "name"  -> DynamicValue.string(""),
  "email" -> DynamicValue.string("ada@example.com")
)
```

Checking it reports the failed rule and its path:

```scala
println(Customer.schema.toDynamicSchema.check(blankName))
// Some(zio.blocks.schema.SchemaError: Validation failed: string length 0 not in range [1, 30] at: .name.wrapped)
```

When your API contract has to state the limits, write them into a JSON Schema document by hand and load it with `JsonSchema.parse`, as the "Load a Schema from Another Source" section of [How to Generate JSON Schema from a Type and Validate Payloads Against It](./generate-json-schema-and-validate-payloads.md) does.

## Keep IDs Apart with a Typed Id

Refined types check values; IDs need a different kind of safety. A customer ID and an order ID are both `Long`s that pass every check, and the bug is using one where the other belongs. One opaque type with a type parameter fixes that for every entity at once: `Id[Customer]` and `Id[Order]` are different types at compile time and the same `Long` at runtime. This part needs no Iron.

An opaque type is transparent inside the scope that defines it, so we put `Id` and its companion in a small object of their own. The companion has a constructor, an extension method that reads the `Long` back, and one generic `Schema` that serves `Id[A]` for every `A`:

```scala
object ids {
  opaque type Id[+A] = Long

  object Id {
    def apply[A](value: Long): Id[A] = value

    extension [A](id: Id[A]) def value: Long = id

    implicit def schema[A]: Schema[Id[A]] = Schema.long.transform(Id[A](_), _.value)
  }
}

```

:::warning
Write `Schema.long`, not `Schema[Long]`, inside the opaque type's scope. There `Id[A]` is a `Long`, so `Schema[Long]` finds the `implicit def schema` itself: the code compiles with only an "Infinite loop in function body" warning and throws a `StackOverflowError` at runtime.
:::

An order refers to its own ID, its customer's ID and a set of customers who watch its delivery. A field can hold any collection of IDs, such as `Set[Id[Customer]]`, because each `Id[Customer]` has a schema:

```scala
final case class Order(id: Id[Order], customer: Id[Customer], lines: List[Line], watchers: Set[Id[Customer]])

object Order {
  implicit val schema: Schema[Order] = Schema.derived
}

val order = Order(
  id = Id(10L),
  customer = Id(1L),
  lines = List(Line(Sku("KEY-0042"), Quantity.parse(2).toOption.get)),
  watchers = Set(Id(1L), Id(2L))
)

val orderJson = Order.schema.jsonCodec.encodeToString(order)
```

In JSON, each ID is the plain number, and decoding gives the typed IDs back:

```scala
println(orderJson)
// {"id":10,"customer":1,"lines":[{"sku":"KEY-0042","quantity":2}],"watchers":[1,2]}
println(Order.schema.jsonCodec.decode(orderJson))
// Right(Order(10,1,List(Line(KEY-0042,Quantity(2))),Set(1, 2)))
```

The compiler now rejects an ID of the wrong entity. Passing the order's customer ID to a function that cancels an order doesn't compile:

```scala
def cancel(id: Id[Order]): String = s"order ${id.value} cancelled"

cancel(order.customer)
// error:
// Found:    (repl.MdocSession.MdocApp.order.customer :
//   repl.MdocSession.MdocApp.ids.Id[repl.MdocSession.MdocApp.Customer])
// Required: repl.MdocSession.MdocApp.ids.Id[repl.MdocSession.MdocApp.Order]
// cancel(order.customer)
//        ^^^^^^^^^^^^^^
```

The `+` in `Id[+A]` makes `Id` covariant: an `Id[Customer]` is also an `Id[Any]`. That suits code that handles every ID the same way, such as an audit log:

```scala
def audit(id: Id[Any]): String = s"audit entry for id ${id.value}"

println(audit(order.id))
// audit entry for id 10
println(audit(order.customer))
// audit entry for id 1
```

## Choose Between Iron and Validated Newtypes

Iron and a hand-written validated newtype both end with a type that only holds valid values and a `Schema` that checks on decode. They differ in where the rule lives:

| Concern                     | Iron refined type                                         | Validated newtype                                     |
| --------------------------- | --------------------------------------------------------- | ----------------------------------------------------- |
| Scala versions              | Scala 3.9 and newer                                       | Scala 2.13 and Scala 3                                |
| Extra dependency            | `iron-zio-blocks-schema`                                  | None                                                  |
| Writing a rule              | A constraint type, such as `MinLength[1] & MaxLength[30]` | A smart constructor, such as `Quantity.parse`         |
| Literals                    | Checked at compile time                                   | Checked at runtime, by the smart constructor          |
| Normalizing input           | Not supported: the value is kept as written               | Supported, for example trimming or removing hyphens   |
| Error message               | Iron's description of the constraint                      | Your own message                                      |
| `Validation` on the schema  | Added for the constraints that have an exact equivalent   | Not added                                             |
| Numeric fields in 0.0.56    | Broken inside case classes                                | Work                                                  |

Choose Iron when your rules are the common ones (lengths, patterns, ranges, signs) and you want them visible in the type. Choose a validated newtype when a rule needs real code, such as an ISBN check digit, when you normalize the input, when you need Scala 2.13, or for numeric fields on ZIO Blocks 0.0.56. Typed IDs work the same way with either choice.

## Putting It Together

The complete example decodes two incoming orders with the model from this guide: one valid, and one with a lowercase SKU in its first line. The model lives in `Model.scala` and `Ids.scala`, and the program in `CompleteExample.scala`. Unlike the sections, the program wraps decoding in a `receive` function that turns the result into one line of text:

```scala title="zio-blocks-examples/iron-refined-types/src/main/scala/ironrefinedtypes/CompleteExample.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-next-schema:0.0.56
//> using dep io.github.iltotore::iron-zio-blocks-schema:3.4.1,exclude=dev.zio%zio-blocks-schema_3
//> using options -experimental
//> using file Model.scala
//> using file Ids.scala
// Run from this folder:  scala-cli run CompleteExample.scala
// Or from zio-blocks-examples:  sbt "ironRefinedTypes/runMain ironrefinedtypes.CompleteExample"

package ironrefinedtypes

/**
 * How to Use Iron Refined Types and Typed IDs with Schema — Putting It Together
 *
 * Decodes two incoming orders: a valid one, and one whose first line has a
 * lowercase SKU.
 */
object CompleteExample {
  def receive(json: String): String =
    Order.schema.jsonCodec.decode(json) match {
      case Right(order) => s"accepted order ${order.id.value} for customer ${order.customer.value}: ${order.lines}"
      case Left(error)  => s"rejected order: ${error.message}"
    }

  def main(args: Array[String]): Unit = {
    println(receive("""{"id":10,"customer":1,"lines":[{"sku":"KEY-0042","quantity":2}],"watchers":[1,2]}"""))
    println(receive("""{"id":11,"customer":1,"lines":[{"sku":"key-42","quantity":2}],"watchers":[]}"""))
  }
}
```

Running it prints the decoded order and the error of the rejected one:

```text
accepted order 10 for customer 1: List(Line(KEY-0042,Quantity(2)))
rejected order: Should match ^[A-Z]{3}-[0-9]{4}$ at: .lines.at(0).sku.wrapped.wrapped
```

## Running the Examples

All runnable code from this guide is in [`zio-blocks-examples/iron-refined-types/src/main/scala/ironrefinedtypes/`](https://github.com/zio/zio-blocks/tree/main/zio-blocks-examples/iron-refined-types/src/main/scala/ironrefinedtypes). Run any file from that folder with `scala-cli run <File>.scala`, or from `zio-blocks-examples` with `sbt "ironRefinedTypes/runMain ironrefinedtypes.<ObjectName>"`.

## Going Further

- [How to Define Validated Newtypes and Collect Every Validation Error](./validated-newtypes.md) collects every failure of a payload in one pass and turns `SchemaError`s into an error response.
- [Validation](../reference/schema/validation.md) lists the `Validation` rules that Iron's module attaches and where ZIO Blocks enforces them.
- [Allows](../reference/schema/allows.md) restricts which shapes a function accepts at compile time, and treats opaque types as wrappers of their underlying type.
- Iron's [ZIO Blocks module page](https://github.com/Iltotore/iron/blob/main/docs/_docs/modules/zio-blocks-schema.md) shows how to map your own constraints to `Validation` with a `ZioBlocksValidation` given, and how to turn off the attached `Validation` or message with `ZioBlocksSchemaConfig`.

---

## How to Use Scala 3 Named Tuples with Schema

In this guide, we use Scala 3 named tuples as lightweight records for weather readings: a station sends `(city = "Oslo", tempC = 3.8)`, and we encode it to JSON, decode it back, correct one field and turn it into a case class, all without declaring a class for each shape. ZIO Blocks derives a `Schema` for a named tuple the same way it does for a case class, and the schema brings JSON, optics and the other codecs with it.

We take an incremental approach:

- Derive a `Schema` for a named tuple
- Round-trip it through JSON and read decode errors
- Nest named tuples, and add optional and list fields
- Go past 22 fields
- Read and update one field with optics
- Convert between a named tuple and a case class

## The Problem

Named tuples became a stable feature in Scala 3.7. They work like a case class we never declare, which suits query rows, quick JSON payloads and data that changes shape between two steps of a program. But most Scala JSON libraries have no codec for them, so we end up writing one by hand for every shape.

For example, a hand-written codec for one reading often looks like this:

```scala
type Reading = (city: String, tempC: Double)

def toJson(reading: Reading): String =
  s"""{"city":"${reading.city}","tempC":${reading.tempC}}"""

def fromJson(json: String): Option[Reading] =
  for {
    city  <- """"city"\s*:\s*"([^"]*)"""".r.findFirstMatchIn(json).map(_.group(1))
    tempC <- """"tempC"\s*:\s*(-?[0-9.]+)""".r.findFirstMatchIn(json).map(_.group(1).toDouble)
  } yield (city = city, tempC = tempC)
```

The encoder doesn't escape quotes in a city name, the decoder answers `None` without saying which field was wrong, and every new field, nested tuple or new shape needs more of the same code. The usual way out is to declare a case class just to get a codec, which gives up the reason we picked a named tuple in the first place.

In this guide, we derive a `Schema` for the named tuple instead. The field names come from the tuple's labels, and the same schema gives us JSON, decode errors that name the field, optics and conversions.

## Prerequisites

This guide builds on [Your First Schema](./your-first-schema.md) (deriving a `Schema` and its JSON round trip) and [Navigate and Update Nested Data with Optics](./nested-data-with-optics.md) (lenses and `CompanionOptics`). Add the schema module to your `build.sbt` (see [How to Add ZIO Blocks to Your Build](../getting-started/installation.md) for the full setup):

```scala
// Scala 3.7 and 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56"
scalacOptions += "-experimental"
```

Named tuples need Scala 3.7 or newer, the first release where they are a stable language feature. Scala 2.13 has no named tuples, so this guide is for Scala 3 only. We checked every example with the published `zio-blocks-schema` on Scala 3.7.0, 3.7.4 and 3.8.4, and with both artifacts on Scala 3.9.0.

We use this import throughout:

```scala

```

## Derive a Schema for a Named Tuple

A named tuple type is easiest to work with under a type alias. We derive its schema with `Schema.derived`, exactly as for a case class:

```scala
type Reading = (city: String, tempC: Double)

implicit val readingSchema: Schema[Reading] = Schema.derived
```

The schema is a record whose field names are the tuple's labels, in order:

```scala
Schema[Reading].reflect.asRecord.map(_.fields.map(_.name))
// res1: Option[IndexedSeq[String]] = Some(Vector("city", "tempC"))
```

With the schema in scope, every schema-based feature works on a named tuple value. `show` prints it with its labels:

```scala
val oslo: Reading = (city = "Oslo", tempC = 3.8)
// oslo: NamedTuple[Tuple2["city", "tempC"], Tuple2[String, Double]] = (
//   "Oslo",
//   3.8
// )

println(oslo.show)
// {
//   city: "Oslo",
//   tempC: 3.8
// }
```

:::warning[An `object` next to a type alias is not a companion]
For a case class, we usually put the schema in its companion object. A type alias has no companion: in `type Reading = (...)` plus `object Reading { implicit val schema: Schema[Reading] = Schema.derived }`, the schema is not found implicitly, and `oslo.toJsonString` fails with "value toJsonString is not a member of Reading". Define the schema where the code that uses it can see it, as we do here, or write `import Reading._` (with Scala 3 given syntax, write `import Reading.given`).
:::

## Round-Trip a Named Tuple Through JSON

The JSON codec comes from the schema, so a named tuple encodes as a JSON object keyed by its labels:

```scala
val osloJson = oslo.toJsonString
```

The encoded text has one key per label:

```scala
println(osloJson)
// {"city":"Oslo","tempC":3.8}
```

Decoding gives back the same named tuple:

```scala
osloJson.fromJson[Reading]
// res4: Either[SchemaError, Reading] = Right(("Oslo", 3.8))

osloJson.fromJson[Reading] == Right(oslo)
// res5: Boolean = true
```

A named tuple prints like a plain tuple, `(Oslo,3.8)`, because its labels exist only in the type, not in the value. They are still there when we read a field of the decoded value:

```scala
osloJson.fromJson[Reading].map(_.city)
// res6: Either[SchemaError, String] = Right("Oslo")
```

When a payload is wrong, the error names the field. A missing field and a value of the wrong type each give a `SchemaError`:

```scala
println("""{"city":"Oslo"}""".fromJson[Reading].left.map(_.message))
// Left(missing required field "tempC" at: .)

println("""{"city":"Oslo","tempC":"mild"}""".fromJson[Reading].left.map(_.message))
// Left(illegal number at: .tempC)
```

An unknown field is ignored by default, so a station that adds a `wind` field doesn't break our decoder:

```scala
"""{"city":"Oslo","tempC":3.8,"wind":4}""".fromJson[Reading]
// res9: Either[SchemaError, Reading] = Right(("Oslo", 3.8))
```

To change field naming, reject unknown fields or require optional fields, derive a customized codec with `JsonCodecDeriver` settings such as `withFieldNameMapper`, `withRejectExtraFields` and `withRequireOptionFields`, as shown in [How to Customize JSON Encoding](./customize-json-encoding.md). Those global settings apply to named tuples; the per-type `@Modifier` annotations don't, because a named tuple has nothing to annotate.

## Nest Named Tuples and Add Optional Fields

A named tuple can hold another named tuple, an `Option` or a collection. A station's report nests a `Reading`, adds a humidity that some stations don't measure, and lists tags:

```scala
type Station = (id: String, reading: Reading, humidity: Option[Int], tags: List[String])

implicit val stationSchema: Schema[Station] = Schema.derived

val osloStation: Station = (id = "OSL-1", reading = oslo, humidity = Some(81), tags = List("coast"))
val bergenStation: Station =
  (id = "BGO-2", reading = (city = "Bergen", tempC = 7.2), humidity = None, tags = Nil)
```

The nested tuple becomes a nested JSON object. A `None` field is left out, and so is an empty list:

```scala
println(osloStation.toJsonString)
// {"id":"OSL-1","reading":{"city":"Oslo","tempC":3.8},"humidity":81,"tags":["coast"]}

println(bergenStation.toJsonString)
// {"id":"BGO-2","reading":{"city":"Bergen","tempC":7.2}}
```

Decoding fills missing optional fields with `None`, and an empty list for a missing list:

```scala
"""{"id":"BGO-2","reading":{"city":"Bergen","tempC":7.2}}""".fromJson[Station]
// res12: Either[SchemaError, Station] = Right(
//   ("BGO-2", ("Bergen", 7.2), None, List())
// )
```

A named tuple also works as a field of a case class. Deriving the case class's schema derives the schemas of the named tuples inside it, so `DailyReport` needs no extra schema:

```scala
case class DailyReport(day: String, readings: List[(city: String, tempC: Double)])

object DailyReport {
  implicit val schema: Schema[DailyReport] = Schema.derived
}
```

The rows encode as JSON objects inside the case class:

```scala
println(DailyReport("2026-10-27", List(oslo, (city = "Bergen", tempC = 7.2))).toJsonString)
// {"day":"2026-10-27","readings":[{"city":"Oslo","tempC":3.8},{"city":"Bergen","tempC":7.2}]}
```

## Go Past 22 Fields

A plain Scala tuple class stops at `Tuple22`, and longer tuples are stored differently at runtime. Schema derivation handles both, so a reading per hour of the day, 24 fields, works like a short one:

```scala
type HourlyTemps = (
  h00: Double, h01: Double, h02: Double, h03: Double, h04: Double, h05: Double,
  h06: Double, h07: Double, h08: Double, h09: Double, h10: Double, h11: Double,
  h12: Double, h13: Double, h14: Double, h15: Double, h16: Double, h17: Double,
  h18: Double, h19: Double, h20: Double, h21: Double, h22: Double, h23: Double
)

implicit val hourlySchema: Schema[HourlyTemps] = Schema.derived

val osloDay: HourlyTemps = (
  h00 = 1.0, h01 = 0.8, h02 = 0.5, h03 = 0.3, h04 = 0.2, h05 = 0.4,
  h06 = 1.1, h07 = 2.0, h08 = 3.2, h09 = 4.5, h10 = 5.6, h11 = 6.3,
  h12 = 6.9, h13 = 7.1, h14 = 7.0, h15 = 6.4, h16 = 5.5, h17 = 4.6,
  h18 = 3.8, h19 = 3.1, h20 = 2.6, h21 = 2.2, h22 = 1.9, h23 = 1.5
)
```

We encode the day once and keep the text:

```scala
val osloDayJson = osloDay.toJsonString
```

The schema has all 24 fields, and the JSON round trip gives back an equal value:

```scala
Schema[HourlyTemps].reflect.asRecord.map(_.fields.size)
// res14: Option[Int] = Some(24)

osloDayJson.fromJson[HourlyTemps] == Right(osloDay)
// res15: Boolean = true
```

The JSON has one key per hour:

```scala
println(osloDayJson)
// {"h00":1.0,"h01":0.8,"h02":0.5,"h03":0.3,"h04":0.2,"h05":0.4,"h06":1.1,"h07":2.0,"h08":3.2,"h09":4.5,"h10":5.6,"h11":6.3,"h12":6.9,"h13":7.1,"h14":7.0,"h15":6.4,"h16":5.5,"h17":4.6,"h18":3.8,"h19":3.1,"h20":2.6,"h21":2.2,"h22":1.9,"h23":1.5}
```

## Read and Update a Field with Optics

Optics read and change one field without rebuilding the whole tuple by hand. We extend `CompanionOptics` and build lenses with `$`, which finds the schemas we already have in scope. We give the object its own name, `Readings`, because a type alias has no companion:

```scala
object Readings extends CompanionOptics[Reading] {
  val city: Lens[Reading, String]  = $(_.city)
  val tempC: Lens[Reading, Double] = $(_.tempC)
}
```

A lens gets, replaces and modifies its field, and the result is still a `Reading`:

```scala
Readings.tempC.get(oslo)
// res17: Double = 3.8

Readings.tempC.replace(oslo, 4.1)
// res18: NamedTuple[Tuple2["city", "tempC"], Tuple2[String, Double]] = (
//   "Oslo",
//   4.1
// )

Readings.tempC.modify(oslo, _ + 0.5)
// res19: NamedTuple[Tuple2["city", "tempC"], Tuple2[String, Double]] = (
//   "Oslo",
//   4.3
// )
```

A path goes through nested named tuples and into collections, as on case classes:

```scala
object Stations extends CompanionOptics[Station] {
  val tempC: Lens[Station, Double]         = $(_.reading.tempC)
  val humidity: Lens[Station, Option[Int]] = $(_.humidity)
  val tags: Traversal[Station, String]     = $(_.tags.each)
}
```

Each optic updates its target and leaves the rest of the station alone:

```scala
Stations.tempC.modify(osloStation, _ - 0.3)
// res20: NamedTuple[Tuple4["id", "reading", "humidity", "tags"], Tuple4[String, Reading, Option[Int], List[String]]] = (
//   "OSL-1",
//   ("Oslo", 3.5),
//   Some(81),
//   List("coast")
// )

Stations.humidity.modify(osloStation, _.map(_ + 2))
// res21: NamedTuple[Tuple4["id", "reading", "humidity", "tags"], Tuple4[String, Reading, Option[Int], List[String]]] = (
//   "OSL-1",
//   ("Oslo", 3.8),
//   Some(83),
//   List("coast")
// )

Stations.tags.modify(osloStation, _.toUpperCase)
// res22: NamedTuple[Tuple4["id", "reading", "humidity", "tags"], Tuple4[String, Reading, Option[Int], List[String]]] = (
//   "OSL-1",
//   ("Oslo", 3.8),
//   Some(81),
//   List("COAST")
// )
```

:::warning[A `when[...]` step after a named-tuple field fails at runtime]
In a known bug in 0.0.56, a path that selects a case after a named-tuple field, such as `$(_.humidity.when[Some[Int]].value)` or `$(_.sky.when[Cloudy].cover)`, compiles but throws `RuntimeException: Expected a variant` the first time the code reads the optic. The same path on a case class works. Until a fix ships, use a lens to the whole field and change the value inside it, as `Stations.humidity` does with `Option#map` above.
:::

## Convert Between a Named Tuple and a Case Class

Named tuples are handy at the edges of a program, but the rest of the code often has a case class for the same data. `Into` converts between them through the plain tuple: `NamedTuple#toTuple` drops the labels, and `Into` converts a tuple to a case class with matching field types, in order:

```scala
case class WeatherReading(city: String, tempC: Double)

val toWeatherReading: Into[(String, Double), WeatherReading] =
  Into.derived[(String, Double), WeatherReading]

val fromWeatherReading: Into[WeatherReading, (String, Double)] =
  Into.derived[WeatherReading, (String, Double)]
```

The conversion returns an `Either`, as every `Into` does:

```scala
toWeatherReading.into(oslo.toTuple)
// res23: Either[SchemaError, WeatherReading] = Right(
//   WeatherReading(city = "Oslo", tempC = 3.8)
// )
```

The other direction needs no extra step. `fromWeatherReading` produces a plain tuple, and Scala accepts a plain tuple wherever it expects a named tuple with the same element types:

```scala
val bergen: Either[SchemaError, Reading] = fromWeatherReading.into(WeatherReading("Bergen", 7.2))
// bergen: Either[SchemaError, Reading] = Right(("Bergen", 7.2))

bergen.map(_.city)
// res24: Either[SchemaError, String] = Right("Bergen")
```

`As` works the same way, `As.derived[(String, Double), WeatherReading]`, when you need both directions in one value. Both match fields by position, not by label, so keep the tuple's element order the same as the case class's fields. For more about `Into` and `As`, see [Evolve Your Data](./evolve-your-data.md).

:::note
In 0.0.56, `Into.derived` and `As.derived` don't accept a named tuple directly: `Into.derived[Reading, WeatherReading]` fails to compile with "Unsupported type combination", because the macro doesn't recognize a named tuple as a tuple. Converting through the plain tuple, as above, works.
:::

## Putting It Together

The complete example handles a small station feed. It decodes a JSON array of station reports, which uses the schema that ZIO Blocks provides for a `List` of any type with a schema, prints the decode error for a damaged report, corrects each reading's temperature with the `Stations.tempC` lens, converts the readings to `WeatherReading` with `Into`, and encodes a `DailyReport`. Beyond the sections, it decodes the feed with `fromJson[List[Station]]` and stops with `sys.error` if the feed doesn't decode, and it adds a third station, Tromsø. It takes `Reading`, `Station`, `WeatherReading` and `DailyReport` from the shared `Model.scala`:

```scala title="schema-examples/src/main/scala/namedtuples/CompleteExample.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using file Model.scala
// Run from this folder:      scala-cli run CompleteExample.scala
// Or from schema-examples:   sbt "runMain namedtuples.CompleteExample"

package namedtuples

/**
 * How to Use Scala 3 Named Tuples with Schema — Complete Example
 *
 * Decodes a feed of station reports, reports a damaged one, corrects every
 * temperature with a lens, converts the readings to a case class with `Into`
 * and encodes a daily report.
 */
object CompleteExample {

  object Stations extends CompanionOptics[Station] {
    val tempC: Lens[Station, Double] = $(_.reading.tempC)
  }

  val toWeatherReading: Into[(String, Double), WeatherReading] =
    Into.derived[(String, Double), WeatherReading]

  val feed: String =
    """[
      |  {"id":"OSL-1","reading":{"city":"Oslo","tempC":3.8},"humidity":81,"tags":["coast"]},
      |  {"id":"BGO-2","reading":{"city":"Bergen","tempC":7.2}},
      |  {"id":"TOS-1","reading":{"city":"Tromsø","tempC":-1.5},"humidity":74}
      |]""".stripMargin

  def main(args: Array[String]): Unit = {
    val stations: List[Station] = feed.fromJson[List[Station]].fold(e => sys.error(e.message), identity)
    println(s"Decoded ${stations.size} stations")

    val damaged = """{"id":"BGO-2","reading":{"city":"Bergen"}}""".fromJson[Station]
    damaged.left.foreach(error => println(s"Damaged report: ${error.message}"))

    // The stations' sensors read 0.5 degrees low
    val corrected = stations.map(Stations.tempC.modify(_, _ + 0.5))
    println(s"Corrected: ${corrected.map(s => s"${s.id} ${s.reading.tempC}").mkString(", ")}")

    val readings: List[Reading] = corrected.map(_.reading)
    val weather                 = readings.map(r => toWeatherReading.into(r.toTuple))
    println(s"Case classes: ${weather.collect { case Right(w) => w }}")

    println(s"Report: ${DailyReport("2026-10-27", readings).toJsonString}")
  }
}
```

Running it prints:

```text
Decoded 3 stations
Damaged report: missing required field "tempC" at: .reading
Corrected: OSL-1 4.3, BGO-2 7.7, TOS-1 -1.0
Case classes: List(WeatherReading(Oslo,4.3), WeatherReading(Bergen,7.7), WeatherReading(Tromsø,-1.0))
Report: {"day":"2026-10-27","readings":[{"city":"Oslo","tempC":4.3},{"city":"Bergen","tempC":7.7},{"city":"Tromsø","tempC":-1.0}]}
```

## Running the Examples

All runnable code from this guide is in [`schema-examples/src/main/scala/namedtuples/`](https://github.com/zio/zio-blocks/tree/main/schema-examples/src/main/scala/namedtuples). Run any file from that folder with `scala-cli run <File>.scala`, or from `schema-examples` with `sbt "runMain namedtuples.<ObjectName>"`.

## Where to Go Next

- [Your First Schema](./your-first-schema.md) covers deriving a schema and the JSON round trip for case classes.
- [Navigate and Update Nested Data with Optics](./nested-data-with-optics.md) shows every kind of optic path, including `each` and `atKey` steps and the `p"..."` path syntax.
- [How to Customize JSON Encoding](./customize-json-encoding.md) changes field names and optional-field handling.
- [Evolve Your Data](./evolve-your-data.md) explains `Into`, `As` and migrations between versions of a type.

---

## How to Use ZIO Blocks in a ZIO or cats-effect Application

In this guide, we use ZIO Blocks inside an application that already runs on ZIO 2 or cats-effect 3. By the end, you will have a small orders service that parses JSON orders, stores them in SQLite, authorizes payments through an asynchronous gateway, writes an archive file and streams order events, with every ZIO Blocks call running where your effect system expects it, and with interruption and cleanup working across the boundary.

We take five cases one at a time: pure modules, blocking calls, resources, `Async` values and streams.

Every section shows a ZIO tab and a cats-effect tab. Only ZIO has library help, the [`sql-zio` module](./sql-from-zio.md). Every other bridge on this page is a few lines of your own code, which you copy into your project.

## The Problem

ZIO Blocks says it needs no effect system. Then the question is where its calls go in an application that has one. Wrapping each call in the first constructor that compiles works on a laptop, and fails under load or on interruption.

For example, here is a first attempt at three operations of an orders service, written for ZIO 2.1.27 (not compiled here):

```scala
def allOrders(transactor: JdbcTransactor): Task[List[Order]] =
  ZIO.succeed(transactor.connect(OrdersDb.all()))

def authorize(gateway: PaymentGateway, order: Order): Task[String] =
  ZIO.fromFuture(implicit ec => gateway.authorize(order).toFuture)

def events(path: Path): Task[List[OrderEvent]] =
  ZIO.succeed(OrderEvents.fromFile(path).runCollect).absolve.map(_.toList)
```

Each line compiles, and each one is wrong in a way no test notices. `allOrders` runs JDBC on ZIO's compute threads, and an `SQLException` becomes a defect instead of a failure. `authorize` loses cancellation: when the fiber is interrupted, the `Future` from `Async#toFuture` can't cancel the payment authorization, so it runs on. `events` reads the whole file into memory on a compute thread, so it is neither a stream nor interruptible. The cats-effect versions with `IO.delay` and `IO.fromFuture` block a compute thread and lose cancellation in the same way.

In this guide, we put each kind of ZIO Blocks call where it belongs, and write one small bridge per direction for the cases where ZIO Blocks and the effect system both manage something: a resource, a running computation or a stream.

## Prerequisites

Add the ZIO Blocks modules this guide uses, the SQLite JDBC driver, and your effect system. The examples are tested with ZIO 2.1.27, cats-effect 3.7.1 and fs2 3.14.0:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-schema"  % "0.0.56",
  "dev.zio" %% "zio-blocks-sql"     % "0.0.56",
  "dev.zio" %% "zio-blocks-streams" % "0.0.56",
  "dev.zio" %% "zio-blocks-async"   % "0.0.56",
  "dev.zio" %% "zio-blocks-scope"   % "0.0.56"
)

// Scala 3.9 and newer: the same modules as zio-blocks-next-*, plus -experimental
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56" // and so on
scalacOptions += "-experimental"

// The SQLite JDBC driver
libraryDependencies += "org.xerial" % "sqlite-jdbc" % "3.53.4.0"

// For a ZIO application
libraryDependencies ++= Seq(
  "dev.zio" %% "zio"         % "2.1.27",
  "dev.zio" %% "zio-streams" % "2.1.27"
)

// For a cats-effect application
libraryDependencies ++= Seq(
  "org.typelevel" %% "cats-effect" % "3.7.1",
  "co.fs2"        %% "fs2-core"    % "3.14.0"
)
```

The `sql` module is published for Scala 3 only, so the orders model and the programs that use it are Scala 3. The bridges themselves (the resource, `Async` and stream helpers below) compile on Scala 2.13 and Scala 3. The `streams` module needs JDK 21 or newer at runtime.

This guide assumes you know your effect system, and the basics of [`Async`](../reference/async.md), [`Scope` and `Resource`](../reference/resource-management/scope.md) and [`Stream`](../reference/streams/core/stream.md).

## The Orders Model

The service keeps orders in an `orders` table and reads order events from a JSON-lines file. Everything in this file is ZIO Blocks code, with no effect system in sight: `Order` and `OrderEvent` derive a `Schema` for JSON, `OrdersDb` holds the SQL, and `OrderEvents.fromFile` is a ZIO Blocks `Stream` that opens the file when it runs and prints a line when it closes it:

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/Orders.scala"
package effectintegration

final case class Order(id: Long, customer: String, totalCents: Long)

object Order {
  implicit val schema: Schema[Order] = Schema.derived
}

final case class OrderEvent(orderId: Long, status: String)

object OrderEvent {
  implicit val schema: Schema[OrderEvent] = Schema.derived
}

object OrdersDb {
  def createTable()(using DbCon): Unit =
    sql"""CREATE TABLE orders (
      id INTEGER PRIMARY KEY,
      customer TEXT NOT NULL,
      total_cents INTEGER NOT NULL
    )""".update

  def insert(order: Order)(using DbCon): Int =
    sql"INSERT INTO orders (id, customer, total_cents) VALUES (${order.id}, ${order.customer}, ${order.totalCents})".update

  def all()(using DbCon): List[Order] =
    sql"SELECT id, customer, total_cents FROM orders ORDER BY id".query[Order]

  def totals()(using DbCon): Stream[Throwable, Long] =
    sql"SELECT total_cents FROM orders".queryStream[Long].flatMap(chunk => Stream.fromIterable(chunk))

  // A fresh SQLite file with an empty orders table.
  def open(): JdbcTransactor = {
    val file = Files.createTempFile("orders", ".db")
    file.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$file", SqlDialect.SQLite)
    transactor.connect(createTable())
    transactor
  }
}

object OrderEvents {
  // One JSON object per line. The file opens when the stream runs and closes when the run ends.
  def fromFile(path: Path): Stream[SchemaError, OrderEvent] =
    Stream.fromAcquireRelease(
      Files.newBufferedReader(path, UTF_8),
      reader => { reader.close(); println("order-events file closed") }
    ) { reader =>
      Stream
        .fromIterator(reader.lines().iterator().asScala)
        .flatMap(line => line.fromJson[OrderEvent].fold(Stream.fail(_), Stream.succeed(_)))
    }

  def write(lines: String*): Path = {
    val path = Files.createTempFile("order-events", ".jsonl")
    path.toFile.deleteOnExit()
    Files.write(path, lines.asJava, UTF_8)
  }
}
```

`OrdersDb.totals` returns a `Stream` of the order totals from `Frag#queryStream`. Like every `Frag#queryStream` result, it must be consumed inside the `JdbcTransactor#connect` block that opened its connection.

## Call the Pure Modules Directly

`schema`, its codecs, optics, `html` and parsing configuration from a string are pure: they start no threads, do no I/O and hold no resources. Call them from your effect like any other library function. A decode returns `Either[SchemaError, A]`, and `SchemaError` extends `Exception`, so it fits the error channel of `ZIO.fromEither` and of `IO.fromEither` as it is. An encode can't fail, so `ZIO.succeed` and `IO.pure` are enough:

<Tabs groupId="effect-system" defaultValue="zio">
  <TabItem value="zio" label="ZIO">

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/Step1PureZio.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using dep dev.zio::zio-blocks-sql:0.0.56
//> using dep dev.zio::zio-blocks-streams:0.0.56
//> using dep dev.zio::zio:2.1.27
//> using options -experimental
//> using jvm 21
//> using file Orders.scala
// Run from this folder:  scala-cli run Step1PureZio.scala
// Or from zio-blocks-examples:  sbt "effectIntegration/runMain effectintegration.Step1PureZio"

package effectintegration

object Step1PureZio extends ZIOAppDefault {
  def parseOrder(body: String): IO[SchemaError, Order] = ZIO.fromEither(body.fromJson[Order])

  def render(order: Order): UIO[String] = ZIO.succeed(order.toJsonString)

  val run =
    for {
      order  <- parseOrder("""{"id":1,"customer":"Ada","totalCents":4200}""")
      _      <- Console.printLine(s"parsed: $order")
      json   <- render(order.copy(totalCents = 4700))
      _      <- Console.printLine(s"rendered: $json")
      result <- parseOrder("""{"id":2,"customer":"Grace"}""").either
      _      <- Console.printLine(s"missing field: ${result.left.map(_.getMessage)}")
    } yield ()
}
```

  </TabItem>
  <TabItem value="cats" label="cats-effect">

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/Step1PureCats.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using dep dev.zio::zio-blocks-sql:0.0.56
//> using dep dev.zio::zio-blocks-streams:0.0.56
//> using dep org.typelevel::cats-effect:3.7.1
//> using options -experimental
//> using jvm 21
//> using file Orders.scala
// Run from this folder:  scala-cli run Step1PureCats.scala
// Or from zio-blocks-examples:  sbt "effectIntegration/runMain effectintegration.Step1PureCats"

package effectintegration

object Step1PureCats extends IOApp.Simple {
  def parseOrder(body: String): IO[Order] = IO.fromEither(body.fromJson[Order])

  def render(order: Order): IO[String] = IO.pure(order.toJsonString)

  val run =
    for {
      order  <- parseOrder("""{"id":1,"customer":"Ada","totalCents":4200}""")
      _      <- IO.println(s"parsed: $order")
      json   <- render(order.copy(totalCents = 4700))
      _      <- IO.println(s"rendered: $json")
      result <- parseOrder("""{"id":2,"customer":"Grace"}""").attempt
      _      <- IO.println(s"missing field: ${result.left.map(_.getMessage)}")
    } yield ()
}
```

  </TabItem>
</Tabs>

Both programs print the same lines. The missing field arrives in the error channel as a typed `SchemaError`:

```text
parsed: Order(1,Ada,4200)
rendered: {"id":1,"customer":"Ada","totalCents":4700}
missing field: Left(missing required field "totalCents" at: .)
```

## Run Blocking Calls on the Blocking Pool

Some ZIO Blocks calls block the calling thread: every JDBC call, every blocking `Stream` terminal on the JVM, and `Async#block`. Run them with `ZIO.attemptBlocking` or `IO.blocking`, so they use the effect system's pool for blocking work instead of the few compute threads its fibers share. Building a `Stream` or composing a `Resource` blocks nothing, because nothing runs until a terminal or a scope runs it:

| ZIO Blocks call                                                   | Blocks the thread?                     | Where to run it                       |
|-------------------------------------------------------------------|----------------------------------------|---------------------------------------|
| `Schema` codecs, optics, `html` rendering                         | No                                     | Anywhere                              |
| Building a `Stream` (`Stream.fromIterable`, `Stream#map`, ...)    | No, it is a description                | Anywhere                              |
| `Transactor#connect`, `Transactor#transact`, `Frag#query`         | Yes, JDBC                              | `ZIO.attemptBlocking`, `IO.blocking`  |
| `Stream#runCollect`, `Stream#runFold`, `Stream#runForeach`        | Yes, on the JVM                        | `ZIO.attemptBlocking`, `IO.blocking`  |
| `Async#block`                                                     | Yes, until the `Async` completes       | `ZIO.attemptBlocking`, `IO.blocking`  |
| `Scope#scoped`, `Scope#allocate`                                  | Runs acquire and release on the caller | Blocking pool if they do I/O          |
| Building an `Async` (`Async.attempt`, `Async#map`)                | Runs its synchronous work at once      | Like the work it does                 |

Errors map in three shapes. A typed `Either` from a decode or from a stream terminal joins the error channel with `ZIO.fromEither` or `ZIO#absolve`, and with `IO.fromEither` or `IO#rethrow`. An exception that a ZIO Blocks call throws, such as an `SQLException`, becomes a failure inside `ZIO.attemptBlocking` and `IO.blocking`. A defect stays a defect.

The orders service stores two orders, lists them, sums their totals with a stream that it consumes inside `JdbcTransactor#connect`, and then inserts a duplicate id. Each blocking call prints the thread it runs on:

<Tabs groupId="effect-system" defaultValue="zio">
  <TabItem value="zio" label="ZIO">

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/Step2BlockingZio.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using dep dev.zio::zio-blocks-sql:0.0.56
//> using dep dev.zio::zio-blocks-streams:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
//> using dep dev.zio::zio:2.1.27
//> using options -experimental
//> using jvm 21
//> using file Orders.scala
// Run from this folder:  scala-cli run Step2BlockingZio.scala
// Or from zio-blocks-examples:  sbt "effectIntegration/runMain effectintegration.Step2BlockingZio"

package effectintegration

object Step2BlockingZio extends ZIOAppDefault {
  def thread: String = Thread.currentThread.getName

  def saveOrder(transactor: JdbcTransactor, order: Order): Task[Unit] =
    ZIO.attemptBlocking {
      println(s"insert order ${order.id} on $thread")
      transactor.transact(OrdersDb.insert(order))
    }.unit

  // The stream must be consumed inside `connect`; its Either joins the error channel.
  def revenue(transactor: JdbcTransactor): Task[Long] =
    ZIO.attemptBlocking(transactor.connect(OrdersDb.totals().runFold(0L)(_ + _))).absolve

  val run =
    for {
      _          <- ZIO.succeed(println(s"compute work on $thread"))
      transactor <- ZIO.attemptBlocking(OrdersDb.open())
      _          <- saveOrder(transactor, Order(1, "Ada", 4200))
      _          <- saveOrder(transactor, Order(2, "Grace", 1999))
      orders     <- ZIO.attemptBlocking(transactor.connect(OrdersDb.all()))
      _          <- Console.printLine(s"orders: $orders")
      total      <- revenue(transactor)
      _          <- Console.printLine(s"revenue: $total cents")
      duplicate  <- saveOrder(transactor, Order(1, "Ada", 4200)).either
      _          <- Console.printLine(s"duplicate id: ${duplicate.left.map(_.getClass.getName)}")
    } yield ()
}
```

  </TabItem>
  <TabItem value="cats" label="cats-effect">

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/Step2BlockingCats.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using dep dev.zio::zio-blocks-sql:0.0.56
//> using dep dev.zio::zio-blocks-streams:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
//> using dep org.typelevel::cats-effect:3.7.1
//> using options -experimental
//> using jvm 21
//> using file Orders.scala
// Run from this folder:  scala-cli run Step2BlockingCats.scala
// Or from zio-blocks-examples:  sbt "effectIntegration/runMain effectintegration.Step2BlockingCats"

package effectintegration

object Step2BlockingCats extends IOApp.Simple {
  def thread: String = Thread.currentThread.getName

  def saveOrder(transactor: JdbcTransactor, order: Order): IO[Unit] =
    IO.blocking {
      println(s"insert order ${order.id} on $thread")
      transactor.transact(OrdersDb.insert(order))
    }.void

  // The stream must be consumed inside `connect`; its Either joins the error channel.
  def revenue(transactor: JdbcTransactor): IO[Long] =
    IO.blocking(transactor.connect(OrdersDb.totals().runFold(0L)(_ + _))).rethrow

  val run =
    for {
      _          <- IO(println(s"compute work on $thread"))
      transactor <- IO.blocking(OrdersDb.open())
      _          <- saveOrder(transactor, Order(1, "Ada", 4200))
      _          <- saveOrder(transactor, Order(2, "Grace", 1999))
      orders     <- IO.blocking(transactor.connect(OrdersDb.all()))
      _          <- IO.println(s"orders: $orders")
      total      <- revenue(transactor)
      _          <- IO.println(s"revenue: $total cents")
      duplicate  <- saveOrder(transactor, Order(1, "Ada", 4200)).attempt
      _          <- IO.println(s"duplicate id: ${duplicate.left.map(_.getClass.getName)}")
    } yield ()
}
```

  </TabItem>
</Tabs>

The ZIO program prints its compute thread first, then runs every insert on ZIO's blocking pool. The duplicate id fails with the driver's exception in the error channel:

```text
compute work on ZScheduler-Worker-4
insert order 1 on zio-default-blocking-1
insert order 2 on zio-default-blocking-1
orders: List(Order(1,Ada,4200), Order(2,Grace,1999))
revenue: 6199 cents
insert order 1 on zio-default-blocking-1
duplicate id: Left(org.sqlite.SQLiteException)
```

The cats-effect program prints the same results. Its blocking calls run on threads named `io-compute-blocker-<n>`, which is how cats-effect 3 names a compute thread it has handed over to blocking work while a replacement takes its place:

```text
compute work on io-compute-13
insert order 1 on io-compute-blocker-13
insert order 2 on io-compute-blocker-13
orders: List(Order(1,Ada,4200), Order(2,Grace,1999))
revenue: 6199 cents
insert order 1 on io-compute-blocker-13
duplicate id: Left(org.sqlite.SQLiteException)
```

:::tip[On ZIO, `sql-zio` writes the wrappers for you]
The `sql-zio` module's `TransactorZIO` returns ZIO effects that already run on the blocking pool, comes as a `ZLayer`, and runs transactions whose body is a ZIO effect. [How to Use the sql Module from a ZIO Application](./sql-from-zio.md) covers it. There is no cats-effect counterpart, so on cats-effect you write `IO.blocking` as above.
:::

`Frag#queryStream` can't leave its `JdbcTransactor#connect` block: the connection closes when the block returns, and ZIO Blocks has no public way to keep it open for longer. To stream query results through your effect system, consume them inside the block, as `revenue` does, and turn file and socket streams into effect streams with the bridge from [the streams section](#turn-a-zio-blocks-stream-into-a-zstream-or-fs2-stream).

## Lift a ZIO Blocks Resource Into Your Effect System

A ZIO Blocks `Resource` describes how to acquire a value and how to release it, and a `Scope` runs the release when it closes. Your effect system has the same idea, so the bridge hands the lifetime to one of the two. The archive file of placed orders is our resource. It prints a line when it opens and when it closes, so we can count the closes:

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/OrderArchive.scala"
package effectintegration

// An append-only JSON-lines file of placed orders: a resource that must be closed exactly once.
final class OrderArchive(path: Path) extends AutoCloseable {
  private val writer: BufferedWriter = Files.newBufferedWriter(path, UTF_8)
  println("archive opened")

  def append(order: Order): Unit = {
    writer.write(order.toJsonString)
    writer.newLine()
  }

  def close(): Unit = {
    writer.close()
    println("archive closed")
  }
}

object OrderArchive {
  def resource(path: Path): Resource[OrderArchive] = Resource.fromAutoCloseable(new OrderArchive(path))
}
```

ZIO Blocks code normally enters a scope with `Scope#scoped`, whose block keeps every resource inside it. An effect system acquires and releases in two separate steps instead, so the bridge opens a child of `Scope.global` with `Scope#open`, allocates the resource in it, and takes the value out with `Scope#leak`. `Scope#leak` is the escape hatch ZIO Blocks provides for interop; it warns at compile time, and `@nowarn` silences that warning here because the caller closes the scope. Closing the child scope runs the resource's finalizers:

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/Opened.scala"
package effectintegration

// Opens a ZIO Blocks resource outside a `scoped` block, for an effect system that manages the
// lifetime itself. The result pairs the value with a function that releases it.
object Opened {

  @nowarn // `leak` warns on purpose: the value leaves the scope, and the caller closes it
  def resource[A](resource: Resource[A]): (A, () => Unit) = {
    val open = Scope.global.open()
    try (open.scope.leak(open.scope.allocate(resource)), () => open.close().orThrow())
    catch { case t: Throwable => open.close(); throw t }
  }
}
```

The bridges build on `Opened.resource` for one direction. For the other, they run the effect system's own acquire when ZIO Blocks allocates, and its release when the ZIO Blocks scope closes:

<Tabs groupId="effect-system" defaultValue="zio">
  <TabItem value="zio" label="ZIO">

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/ZioResources.scala"
package effectintegration

object ZioResources {

  // ZIO Blocks Resource -> ZIO: acquired on the blocking pool, released when the ZIO Scope closes.
  def fromResource[A](resource: Resource[A]): ZIO[Scope, Throwable, A] =
    ZIO
      .acquireRelease(ZIO.attemptBlocking(Opened.resource(resource))) { case (_, release) =>
        ZIO.attemptBlocking(release()).orDie
      }
      .map(_._1)

  // ZIO -> ZIO Blocks Resource: runs `acquire` in a fresh ZIO Scope, closed with the ZIO Blocks scope.
  def toResource[A](acquire: ZIO[Scope, Throwable, A])(implicit runtime: Runtime[Any]): Resource[A] =
    Resource.acquireRelease {
      Unsafe.unsafe { implicit unsafe =>
        val scope = runtime.unsafe.run(Scope.make).getOrThrow()
        runtime.unsafe.run(scope.extend[Any](acquire)) match {
          case Exit.Success(a)     => (a, scope)
          case Exit.Failure(cause) =>
            runtime.unsafe.run(scope.close(Exit.failCause(cause)))
            throw cause.squash
        }
      }
    } { case (_, scope) =>
      Unsafe.unsafe(implicit unsafe => runtime.unsafe.run(scope.close(Exit.unit)).getOrThrow())
    }
      .map(_._1)
}
```

  </TabItem>
  <TabItem value="cats" label="cats-effect">

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/CatsResources.scala"
package effectintegration

object CatsResources {

  // ZIO Blocks Resource -> cats-effect Resource: acquired and released on the blocking pool.
  def fromResource[A](resource: BlocksResource[A]): Resource[IO, A] =
    Resource
      .make(IO.blocking(Opened.resource(resource))) { case (_, release) => IO.blocking(release()) }
      .map(_._1)

  // cats-effect Resource -> ZIO Blocks Resource: allocated now, released with the ZIO Blocks scope.
  def toResource[A](resource: Resource[IO, A])(implicit dispatcher: Dispatcher[IO]): BlocksResource[A] =
    BlocksResource
      .acquireRelease(dispatcher.unsafeRunSync(resource.allocated)) { case (_, release) =>
        dispatcher.unsafeRunSync(release)
      }
      .map(_._1)
}
```

  </TabItem>
</Tabs>

`ZioResources.fromResource` returns a `ZIO[Scope, Throwable, A]`, so `ZLayer.scoped` turns it into a layer and `ZIO.scoped` runs it. `CatsResources.fromResource` returns a cats-effect `Resource`, which `Resource#use` runs. In the other direction, `ZioResources.toResource` and `CatsResources.toResource` run your effect system from inside ZIO Blocks code with `Runtime#unsafe` or `Dispatcher#unsafeRunSync`, which blocks the thread, so call ZIO Blocks code that allocates it from `ZIO.attemptBlocking` or `IO.blocking`. A `Dispatcher` is cats-effect's tool for running effects from code that is not itself an effect; it comes from `Dispatcher.parallel[IO]`.

The programs below use the archive four times: once successfully, once with a failing body, once interrupted while in use, and once as a resource your effect system manages and ZIO Blocks code uses. Each use opens and closes the archive exactly once:

```text
-- success, through a ZLayer
archive opened
archive closed
-- failure
archive opened
archive closed
result: Left(payment declined)
-- interruption
archive opened
archive closed
-- a ZIO-managed archive used from ZIO Blocks code
archive opened
archive closed
```

The cats-effect program prints the same lines, with `-- success` in place of `-- success, through a ZLayer`, `-- cancellation` in place of `-- interruption`, and "cats-effect-managed" in the last label.

<details>
  <summary>Runnable version, the ZIO tab: <code>Step3ResourcesZio.scala</code></summary>

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/Step3ResourcesZio.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using dep dev.zio::zio-blocks-sql:0.0.56
//> using dep dev.zio::zio-blocks-streams:0.0.56
//> using dep dev.zio::zio-blocks-scope:0.0.56
//> using dep dev.zio::zio:2.1.27
//> using options -experimental
//> using jvm 21
//> using file Orders.scala
//> using file OrderArchive.scala
//> using file Opened.scala
//> using file ZioResources.scala
// Run from this folder:  scala-cli run Step3ResourcesZio.scala
// Or from zio-blocks-examples:  sbt "effectIntegration/runMain effectintegration.Step3ResourcesZio"

package effectintegration

object Step3ResourcesZio extends ZIOAppDefault {
  val order = Order(1, "Ada", 4200)

  val run =
    ZIO.runtime[Any].flatMap { implicit runtime =>
      for {
        path   <- ZIO.attemptBlocking(Files.createTempFile("archive", ".jsonl"))
        archive = ZioResources.fromResource(OrderArchive.resource(path))
        _      <- Console.printLine("-- success, through a ZLayer")
        _      <- ZIO
               .serviceWithZIO[OrderArchive](a => ZIO.attemptBlocking(a.append(order)))
               .provide(ZLayer.scoped(archive))
        _         <- Console.printLine("-- failure")
        failed    <- ZIO.scoped(archive *> ZIO.fail(new Exception("payment declined"))).either
        _         <- Console.printLine(s"result: ${failed.left.map(_.getMessage)}")
        _         <- Console.printLine("-- interruption")
        fiber     <- ZIO.scoped(archive *> ZIO.never).fork
        _         <- ZIO.sleep(100.millis)
        _         <- fiber.interrupt
        _         <- Console.printLine("-- a ZIO-managed archive used from ZIO Blocks code")
        zioManaged = ZioResources.toResource(ZIO.fromAutoCloseable(ZIO.attempt(new OrderArchive(path))))
        _         <- ZIO.attemptBlocking {
               BlocksScope.global.scoped { scope =>

                 val a = allocate(zioManaged)
                 $(a)(_.append(order))
               }
             }
        _ <- ZIO.attemptBlocking(Files.delete(path))
      } yield ()
    }
}
```

</details>

<details>
  <summary>Runnable version, the cats-effect tab: <code>Step3ResourcesCats.scala</code></summary>

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/Step3ResourcesCats.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using dep dev.zio::zio-blocks-sql:0.0.56
//> using dep dev.zio::zio-blocks-streams:0.0.56
//> using dep dev.zio::zio-blocks-scope:0.0.56
//> using dep org.typelevel::cats-effect:3.7.1
//> using options -experimental
//> using jvm 21
//> using file Orders.scala
//> using file OrderArchive.scala
//> using file Opened.scala
//> using file CatsResources.scala
// Run from this folder:  scala-cli run Step3ResourcesCats.scala
// Or from zio-blocks-examples:  sbt "effectIntegration/runMain effectintegration.Step3ResourcesCats"

package effectintegration

object Step3ResourcesCats extends IOApp.Simple {
  val order = Order(1, "Ada", 4200)

  val run =
    Dispatcher.parallel[IO].use { implicit dispatcher =>
      for {
        path       <- IO.blocking(Files.createTempFile("archive", ".jsonl"))
        archive     = CatsResources.fromResource(OrderArchive.resource(path))
        _          <- IO.println("-- success")
        _          <- archive.use(a => IO.blocking(a.append(order)))
        _          <- IO.println("-- failure")
        failed     <- archive.use(_ => IO.raiseError[Unit](new Exception("payment declined"))).attempt
        _          <- IO.println(s"result: ${failed.left.map(_.getMessage)}")
        _          <- IO.println("-- cancellation")
        fiber      <- archive.use(_ => IO.never[Unit]).start
        _          <- IO.sleep(100.millis)
        _          <- fiber.cancel
        _          <- IO.println("-- a cats-effect-managed archive used from ZIO Blocks code")
        catsManaged = CatsResources.toResource(Resource.fromAutoCloseable(IO.blocking(new OrderArchive(path))))
        _          <- IO.blocking {
               Scope.global.scoped { scope =>

                 val a = allocate(catsManaged)
                 $(a)(_.append(order))
               }
             }
        _ <- IO.blocking(Files.delete(path))
      } yield ()
    }
}
```

</details>

## Bridge `Async` and Your Effect, With Cancellation

`Async` is eager: building one runs its synchronous work at once, like a `Future`, while `ZIO` and `IO` values are descriptions that run each time the runtime executes them. So the bridge into your effect takes the `Async` by name and builds it each time the effect runs. Cancellation must also cross in both directions: interrupting the fiber must cancel the `Async`, and cancelling the `Async` must interrupt the effect.

`Async#toFuture` followed by `ZIO.fromFuture` or `IO.fromFuture` loses cancellation in both directions, because a `Future` can't be cancelled. `Async.fromFuture` and `Async.fromCompletionStage` don't forward cancellation to their source either (a known bug in 0.0.56). The bridges therefore use the `cancellable` helper from [How to Wrap Callback and Future-Based APIs](./wrap-callback-and-future-apis.md#cancel-the-underlying-call-when-the-async-is-cancelled), which builds an `Async` whose cancellation calls a function you supply:

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/Cancellable.scala"
package effectintegration

// The `cancellable` helper from "How to Wrap Callback and Future APIs": an `Async` whose
// cancellation calls the function that `register` returns.
object Cancellable {
  def cancellable[A](register: Completer[A] => (() => Unit)): Async[A] =
    new Pollable[A] {
      private val result     = new Completer[A]
      private val cancelCall = register(result)

      def poll(onComplete: Runnable): Async[A] = {
        val polled = result.poll(onComplete)
        // While pending, answer with this Pollable rather than the Completer, so
        // the driver keeps waiting on us and our cancel() is the one it calls.
        if (polled.asInstanceOf[AnyRef] eq result) this else polled
      }

      override def cancel(): Unit = cancelCall()
    }
}
```

The payment gateway is ZIO Blocks code with a callback API. It authorizes a payment after a delay, and cancelling the `Async` cancels the pending authorization and prints a line:

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/PaymentGateway.scala"
package effectintegration

// ZIO Blocks code with an asynchronous API: authorizing a payment takes `latencyMillis`,
// and cancelling the `Async` cancels the pending authorization.
final class PaymentGateway(latencyMillis: Long) {
  private val timer: ScheduledExecutorService =
    Executors.newSingleThreadScheduledExecutor { (r: Runnable) =>
      val thread = new Thread(r, "payment-gateway")
      thread.setDaemon(true)
      thread
    }

  def authorize(order: Order): Async[String] =
    cancellable[String] { completer =>
      val pending = timer.schedule(
        new Runnable { def run(): Unit = completer.succeed(s"auth-${order.id}") },
        latencyMillis,
        TimeUnit.MILLISECONDS
      )
      () => {
        pending.cancel(false)
        println(s"payment gateway: authorization of order ${order.id} cancelled")
      }
    }
}
```

Into your effect, the bridge starts the `Async` with `Async#start`, reports its outcome through `Async#foldCause` to the effect system's callback, and returns `Async.Running#cancel` as the effect's canceller. Out of your effect, it runs the effect with a cancellable future and hands `cancellable` the function that cancels it:

<Tabs groupId="effect-system" defaultValue="zio">
  <TabItem value="zio" label="ZIO">

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/ZioAsync.scala"
package effectintegration

object ZioAsync {

  // ZIO Blocks Async -> ZIO: each run builds the Async; interrupting the fiber cancels it.
  def fromAsync[A](async: => Async[A]): Task[A] =
    ZIO.asyncInterrupt[Any, Throwable, A] { callback =>
      val running = async.foldCause(e => callback(ZIO.fail(e)))(a => callback(ZIO.succeed(a))).start
      Left(ZIO.succeed(running.cancel()))
    }

  // ZIO -> ZIO Blocks Async: starts the effect now; cancelling the Async interrupts its fiber.
  def toAsync[A](task: Task[A])(implicit runtime: Runtime[Any]): Async[A] =
    cancellable[A] { completer =>
      val future                        = Unsafe.unsafe(implicit unsafe => runtime.unsafe.runToFuture(task))
      implicit val ec: ExecutionContext = ExecutionContext.parasitic
      future.onComplete(_.fold(completer.fail, completer.succeed))
      () => { future.cancel(); () }
    }
}
```

  </TabItem>
  <TabItem value="cats" label="cats-effect">

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/CatsAsync.scala"
package effectintegration

object CatsAsync {

  // ZIO Blocks Async -> IO: each run builds the Async; cancelling the fiber cancels it.
  def fromAsync[A](async: => Async[A]): IO[A] =
    IO.async[A] { callback =>
      IO {
        val running = async.foldCause(e => callback(Left(e)))(a => callback(Right(a))).start
        Some(IO(running.cancel()))
      }
    }

  // IO -> ZIO Blocks Async: starts the effect now; cancelling the Async cancels its fiber.
  def toAsync[A](io: IO[A])(implicit dispatcher: Dispatcher[IO]): Async[A] =
    cancellable[A] { completer =>
      val (future, cancel)              = dispatcher.unsafeToFutureCancelable(io)
      implicit val ec: ExecutionContext = ExecutionContext.parasitic
      future.onComplete(_.fold(completer.fail, completer.succeed))
      () => { cancel(); () }
    }
}
```

  </TabItem>
</Tabs>

The programs below authorize one payment, then authorize again under a 100 ms timeout that is shorter than the gateway's 300 ms delay. In the other direction, ZIO Blocks code blocks on a fraud check written as a ZIO or cats-effect effect, then starts a second check and cancels it after 100 ms. The ZIO program prints:

```text
authorized: auth-1
payment gateway: authorization of order 1 cancelled
with a 100 ms timeout: None
fraud check passed: true
fraud check of order 1 interrupted
after cancelling the Async
```

The timeout interrupts the fiber, and the interruption reaches the gateway, which cancels the authorization. Cancelling the `Async` interrupts the fraud check's fiber, whose `ZIO#onInterrupt` prints its line before the marker printed afterwards. The cats-effect program prints `timed out` from `IO#timeoutTo` instead of `None`, and `cancelled` from `IO#onCancel`.

<details>
  <summary>Runnable version, the ZIO tab: <code>Step4AsyncZio.scala</code></summary>

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/Step4AsyncZio.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using dep dev.zio::zio-blocks-sql:0.0.56
//> using dep dev.zio::zio-blocks-streams:0.0.56
//> using dep dev.zio::zio-blocks-async:0.0.56
//> using dep dev.zio::zio:2.1.27
//> using options -experimental
//> using jvm 21
//> using file Orders.scala
//> using file PaymentGateway.scala
//> using file Cancellable.scala
//> using file ZioAsync.scala
// Run from this folder:  scala-cli run Step4AsyncZio.scala
// Or from zio-blocks-examples:  sbt "effectIntegration/runMain effectintegration.Step4AsyncZio"

package effectintegration

object Step4AsyncZio extends ZIOAppDefault {
  val gateway = new PaymentGateway(latencyMillis = 300)
  val order   = Order(1, "Ada", 4200)

  // A ZIO service that ZIO Blocks code wants to call.
  def fraudCheck(order: Order): Task[Boolean] =
    ZIO
      .sleep(1.second)
      .as(order.totalCents < 100000)
      .onInterrupt(Console.printLine(s"fraud check of order ${order.id} interrupted").orDie)

  val run =
    ZIO.runtime[Any].flatMap { implicit runtime =>
      for {
        auth     <- ZioAsync.fromAsync(gateway.authorize(order))
        _        <- Console.printLine(s"authorized: $auth")
        timedOut <- ZioAsync.fromAsync(gateway.authorize(order)).timeout(100.millis)
        _        <- Console.printLine(s"with a 100 ms timeout: $timedOut")
        passed   <- ZIO.attemptBlocking(ZioAsync.toAsync(fraudCheck(order)).block)
        _        <- Console.printLine(s"fraud check passed: $passed")
        running  <- ZIO.succeed(ZioAsync.toAsync(fraudCheck(order)).start)
        _        <- ZIO.sleep(100.millis)
        _        <- ZIO.succeed(running.cancel())
        _        <- ZIO.sleep(100.millis)
        _        <- Console.printLine("after cancelling the Async")
      } yield ()
    }
}
```

</details>

<details>
  <summary>Runnable version, the cats-effect tab: <code>Step4AsyncCats.scala</code></summary>

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/Step4AsyncCats.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using dep dev.zio::zio-blocks-sql:0.0.56
//> using dep dev.zio::zio-blocks-streams:0.0.56
//> using dep dev.zio::zio-blocks-async:0.0.56
//> using dep org.typelevel::cats-effect:3.7.1
//> using options -experimental
//> using jvm 21
//> using file Orders.scala
//> using file PaymentGateway.scala
//> using file Cancellable.scala
//> using file CatsAsync.scala
// Run from this folder:  scala-cli run Step4AsyncCats.scala
// Or from zio-blocks-examples:  sbt "effectIntegration/runMain effectintegration.Step4AsyncCats"

package effectintegration

object Step4AsyncCats extends IOApp.Simple {
  val gateway = new PaymentGateway(latencyMillis = 300)
  val order   = Order(1, "Ada", 4200)

  // A cats-effect service that ZIO Blocks code wants to call.
  def fraudCheck(order: Order): IO[Boolean] =
    IO.sleep(1.second)
      .as(order.totalCents < 100000)
      .onCancel(IO.println(s"fraud check of order ${order.id} cancelled"))

  val run =
    Dispatcher.parallel[IO].use { implicit dispatcher =>
      for {
        auth     <- CatsAsync.fromAsync(gateway.authorize(order))
        _        <- IO.println(s"authorized: $auth")
        timedOut <- CatsAsync.fromAsync(gateway.authorize(order)).timeoutTo(100.millis, IO.pure("timed out"))
        _        <- IO.println(s"with a 100 ms timeout: $timedOut")
        passed   <- IO.blocking(CatsAsync.toAsync(fraudCheck(order)).block)
        _        <- IO.println(s"fraud check passed: $passed")
        running  <- IO(CatsAsync.toAsync(fraudCheck(order)).start)
        _        <- IO.sleep(100.millis)
        _        <- IO(running.cancel())
        _        <- IO.sleep(100.millis)
        _        <- IO.println("after cancelling the Async")
      } yield ()
    }
}
```

</details>

## Turn a ZIO Blocks Stream Into a ZStream or fs2 Stream

A ZIO Blocks `Stream` runs on the thread that pulls it, and a pull can block. To read one from a `ZStream` or an fs2 stream, we open it once with `Stream#start`, pull chunks of elements on the blocking pool, and close it when the effect stream ends, fails or is interrupted. `BlockingPull` wraps that reader:

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/BlockingPull.scala"
package effectintegration

// A ZIO Blocks stream opened for pulling from outside, for an effect system's stream.
// Every call blocks, so call it on a blocking pool.
final class BlockingPull[E, A] private (open: Scope.OpenScope, reader: Reader.SyncReader[A]) {
  @volatile private var failed = false

  // Up to `max` elements, or an empty chunk at the end. A pull throws the stream's typed error
  // wrapped in `StreamError`, an internal class; this unwraps it into a `Left`.
  def next(max: Int): Either[E, Chunk[A]] =
    try Right(reader.readUpToN[A](max))
    catch {
      case error: StreamError => failed = true; Left(error.value.asInstanceOf[E])
      case defect: Throwable  => failed = true; throw defect
    }

  // After a failed pull the stream's finalizers have already run, and closing the reader
  // throws that failure again (ZIO Blocks 0.0.56), so close errors are ignored then.
  def close(): Unit = {
    val finalization = open.close()
    if (!failed) finalization.orThrow()
  }
}

object BlockingPull {
  @nowarn // `leak` warns on purpose: the reader leaves the scope, and `close` closes it
  def apply[E, A](stream: Stream[E, A]): BlockingPull[E, A] = {
    val open = Scope.global.open()
    try new BlockingPull[E, A](open, open.scope.leak(stream.start(open.scope)))
    catch { case t: Throwable => open.close(); throw t }
  }
}
```

`Reader.SyncReader#readUpToN` returns at most `max` elements and an empty chunk at the end, so the effect stream asks for one chunk at a time and the ZIO Blocks stream never runs further ahead than that: this is the back-pressure. Two details are workarounds for 0.0.56. A pull throws the stream's typed error wrapped in `StreamError`, a class in the `internal` package, so `BlockingPull#next` unwraps it into a `Left`. And after a failed pull, closing the reader throws the same failure again, although the stream's finalizers have already run, so `BlockingPull#close` ignores close errors then (a known bug in 0.0.56).

The stream bridges use `BlockingPull` in one direction. In the other, each run of the ZIO Blocks stream starts the effect stream behind a bounded buffer, and closing the run interrupts or cancels it. ZIO has `ZStream#toIterator` for this; fs2 has no equivalent, so `CatsStreams.fromFs2` feeds a bounded `Queue` from a background fiber:

<Tabs groupId="effect-system" defaultValue="zio">
  <TabItem value="zio" label="ZIO">

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/ZioStreams.scala"
package effectintegration

object ZioStreams {

  // ZIO Blocks Stream -> ZStream: pulls up to `chunkSize` elements at a time on the blocking pool,
  // and closes the stream when the ZStream ends, fails or is interrupted.
  def toZStream[E, A](stream: => Stream[E, A], chunkSize: Int = 256): ZStream[Any, E, A] =
    ZStream
      .acquireReleaseWith(ZIO.attemptBlocking(BlockingPull(stream)).orDie)(pull =>
        ZIO.attemptBlocking(pull.close()).orDie
      )
      .flatMap { pull =>
        ZStream.repeatZIOChunkOption {
          ZIO.attemptBlocking(pull.next(chunkSize)).orDie.flatMap {
            case Left(error)                   => ZIO.fail(Some(error))
            case Right(chunk) if chunk.isEmpty => ZIO.fail(None)
            case Right(chunk)                  => ZIO.succeed(Chunk.fromIterable(chunk))
          }
        }
      }

  // ZStream -> ZIO Blocks Stream: each run starts the ZStream; closing the run interrupts it.
  def fromZStream[E, A](zstream: ZStream[Any, E, A])(implicit runtime: Runtime[Any]): Stream[E, A] =
    Stream.fromResource(ZioResources.toResource(zstream.toIterator)) { iterator =>
      Stream.fromIterator(iterator).flatMap(_.fold(Stream.fail(_), Stream.succeed(_)))
    }
}
```

  </TabItem>
  <TabItem value="cats" label="cats-effect">

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/CatsStreams.scala"
package effectintegration

object CatsStreams {

  // ZIO Blocks Stream -> fs2: pulls up to `chunkSize` elements at a time on the blocking pool,
  // and closes the stream when the fs2 stream ends, fails or is cancelled.
  def toFs2[E <: Throwable, A](stream: => Stream[E, A], chunkSize: Int = 256): fs2.Stream[IO, A] =
    fs2.Stream
      .bracket(IO.blocking(BlockingPull(stream)))(pull => IO.blocking(pull.close()))
      .flatMap { pull =>
        fs2.Stream
          .repeatEval(IO.blocking(pull.next(chunkSize)).rethrow)
          .takeWhile(_.nonEmpty)
          .flatMap(chunk => fs2.Stream.iterable(chunk))
      }

  // fs2 -> ZIO Blocks Stream: each run starts the fs2 stream behind a bounded queue, so it
  // runs at most `buffer` elements ahead; closing the run cancels it.
  def fromFs2[A](stream: fs2.Stream[IO, A], buffer: Int = 16)(implicit
    dispatcher: Dispatcher[IO]
  ): Stream[Throwable, A] = {
    // The end of the stream (or its failure) is an element too, so every offer stays cancelable.
    val signals: fs2.Stream[IO, Option[Either[Throwable, A]]] =
      stream.map(a => Some(Right(a))).handleErrorWith(e => fs2.Stream.emit(Some(Left(e)))) ++ fs2.Stream.emit(None)
    val elements = for {
      queue <- cats.effect.Resource.eval(Queue.bounded[IO, Option[Either[Throwable, A]]](buffer))
      _     <- signals.evalMap(queue.offer).compile.drain.background
    } yield Iterator.continually(dispatcher.unsafeRunSync(queue.take)).takeWhile(_.isDefined).flatten
    Stream.fromResource(CatsResources.toResource(elements)) { iterator =>
      Stream.fromIterator(iterator).flatMap(_.fold(Stream.fail(_), Stream.succeed(_)))
    }
  }
}
```

  </TabItem>
</Tabs>

The programs below read the order-events file whole, read only its first event, read a file with a malformed line, and interrupt a stream that never ends. Then they show back-pressure both ways, by printing each element the producer emits while the consumer takes three. The ZIO program prints:

```text
-- whole file
order-events file closed
events: Chunk(placed,paid,shipped)
-- first event only
order-events file closed
events: Chunk(placed)
-- a malformed line
order-events file closed
result: Left(expected '{' at: .)
-- interruption
endless stream closed
-- back-pressure, ZIO Blocks -> ZStream
zio blocks produced 1
zio blocks produced 2
zio blocks produced 3
zio blocks produced 4
zstream took: Chunk(1,2,3)
-- back-pressure, ZStream -> ZIO Blocks
zstream produced 1
zstream produced 2
zstream produced 3
zstream finalized
zio blocks took: Right(Chunk(1,2,3))
```

The file is closed exactly once in each run: after the last event, after `ZStream#take` has what it needs, and after the malformed line, whose `SchemaError` arrives in the `ZStream`'s typed error channel. With a chunk size of 2, the ZIO Blocks stream produces 4 elements for a consumer that takes 3, and no more.

The cats-effect program prints the same lines with `List` in place of `Chunk`, `fs2` in place of `zstream`, and "cancellation" in its label. Its last block shows the bounded queue, of size 2 here, letting the fs2 stream run a few elements ahead before it is cancelled:

```text
-- back-pressure, fs2 -> ZIO Blocks
fs2 produced 1
fs2 produced 2
fs2 produced 3
fs2 produced 4
fs2 produced 5
fs2 produced 6
fs2 finalized
zio blocks took: Right(Chunk(1,2,3))
```

<details>
  <summary>Runnable version, the ZIO tab: <code>Step5StreamsZio.scala</code></summary>

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/Step5StreamsZio.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using dep dev.zio::zio-blocks-sql:0.0.56
//> using dep dev.zio::zio-blocks-streams:0.0.56
//> using dep dev.zio::zio-blocks-scope:0.0.56
//> using dep dev.zio::zio:2.1.27
//> using dep dev.zio::zio-streams:2.1.27
//> using options -experimental
//> using jvm 21
//> using file Orders.scala
//> using file BlockingPull.scala
//> using file ZioStreams.scala
//> using file ZioResources.scala
//> using file Opened.scala
// Run from this folder:  scala-cli run Step5StreamsZio.scala
// Or from zio-blocks-examples:  sbt "effectIntegration/runMain effectintegration.Step5StreamsZio"

package effectintegration

object Step5StreamsZio extends ZIOAppDefault {
  val placed  = """{"orderId":1,"status":"placed"}"""
  val paid    = """{"orderId":1,"status":"paid"}"""
  val shipped = """{"orderId":1,"status":"shipped"}"""

  val run =
    ZIO.runtime[Any].flatMap { implicit runtime =>
      for {
        events <- ZIO.attemptBlocking(OrderEvents.write(placed, paid, shipped))
        _      <- Console.printLine("-- whole file")
        all    <- ZioStreams.toZStream(OrderEvents.fromFile(events)).runCollect
        _      <- Console.printLine(s"events: ${all.map(_.status)}")
        _      <- Console.printLine("-- first event only")
        first  <- ZioStreams.toZStream(OrderEvents.fromFile(events)).take(1).runCollect
        _      <- Console.printLine(s"events: ${first.map(_.status)}")
        _      <- Console.printLine("-- a malformed line")
        broken <- ZIO.attemptBlocking(OrderEvents.write(placed, "not json", shipped))
        failed <- ZioStreams.toZStream(OrderEvents.fromFile(broken)).runCollect.either
        _      <- Console.printLine(s"result: ${failed.left.map(_.getMessage)}")
        _      <- Console.printLine("-- interruption")
        endless = Stream.fromIterator(Iterator.from(1)).ensuring(println("endless stream closed"))
        fiber  <- ZioStreams.toZStream(endless).runDrain.fork
        _      <- ZIO.sleep(100.millis)
        _      <- fiber.interrupt
        _      <- Console.printLine("-- back-pressure, ZIO Blocks -> ZStream")
        counted = Stream.fromIterable(1 to 1000).map { n => println(s"zio blocks produced $n"); n }
        taken  <- ZioStreams.toZStream(counted, chunkSize = 2).take(3).runCollect
        _      <- Console.printLine(s"zstream took: $taken")
        _      <- Console.printLine("-- back-pressure, ZStream -> ZIO Blocks")
        ticks   = ZStream
                  .iterate(1)(_ + 1)
                  .tap(n => Console.printLine(s"zstream produced $n"))
                  .ensuring(Console.printLine("zstream finalized").orDie)
        back <- ZIO.attemptBlocking(ZioStreams.fromZStream(ticks).take(3).runCollect)
        _    <- Console.printLine(s"zio blocks took: $back")
      } yield ()
    }
}
```

</details>

<details>
  <summary>Runnable version, the cats-effect tab: <code>Step5StreamsCats.scala</code></summary>

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/Step5StreamsCats.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using dep dev.zio::zio-blocks-sql:0.0.56
//> using dep dev.zio::zio-blocks-streams:0.0.56
//> using dep dev.zio::zio-blocks-scope:0.0.56
//> using dep org.typelevel::cats-effect:3.7.1
//> using dep co.fs2::fs2-core:3.14.0
//> using options -experimental
//> using jvm 21
//> using file Orders.scala
//> using file BlockingPull.scala
//> using file CatsStreams.scala
//> using file CatsResources.scala
//> using file Opened.scala
// Run from this folder:  scala-cli run Step5StreamsCats.scala
// Or from zio-blocks-examples:  sbt "effectIntegration/runMain effectintegration.Step5StreamsCats"

package effectintegration

object Step5StreamsCats extends IOApp.Simple {
  val placed  = """{"orderId":1,"status":"placed"}"""
  val paid    = """{"orderId":1,"status":"paid"}"""
  val shipped = """{"orderId":1,"status":"shipped"}"""

  val run =
    Dispatcher.parallel[IO].use { implicit dispatcher =>
      for {
        events <- IO.blocking(OrderEvents.write(placed, paid, shipped))
        _      <- IO.println("-- whole file")
        all    <- CatsStreams.toFs2(OrderEvents.fromFile(events)).compile.toList
        _      <- IO.println(s"events: ${all.map(_.status)}")
        _      <- IO.println("-- first event only")
        first  <- CatsStreams.toFs2(OrderEvents.fromFile(events)).take(1).compile.toList
        _      <- IO.println(s"events: ${first.map(_.status)}")
        _      <- IO.println("-- a malformed line")
        broken <- IO.blocking(OrderEvents.write(placed, "not json", shipped))
        failed <- CatsStreams.toFs2(OrderEvents.fromFile(broken)).compile.toList.attempt
        _      <- IO.println(s"result: ${failed.left.map(_.getMessage)}")
        _      <- IO.println("-- cancellation")
        endless = Stream.fromIterator(Iterator.from(1)).ensuring(println("endless stream closed"))
        fiber  <- CatsStreams.toFs2(endless).compile.drain.start
        _      <- IO.sleep(100.millis)
        _      <- fiber.cancel
        _      <- IO.println("-- back-pressure, ZIO Blocks -> fs2")
        counted = Stream.fromIterable(1 to 1000).map { n => println(s"zio blocks produced $n"); n }
        taken  <- CatsStreams.toFs2(counted, chunkSize = 2).take(3).compile.toList
        _      <- IO.println(s"fs2 took: $taken")
        _      <- IO.println("-- back-pressure, fs2 -> ZIO Blocks")
        ticks   = fs2.Stream
                  .iterate(1)(_ + 1)
                  .covary[IO]
                  .evalTap(n => IO.println(s"fs2 produced $n"))
                  .onFinalize(IO.println("fs2 finalized"))
        back <- IO.blocking(CatsStreams.fromFs2(ticks, buffer = 2).take(3).runCollect)
        _    <- IO.println(s"zio blocks took: $back")
      } yield ()
    }
}
```

</details>

## Putting It Together

The complete service uses each case once. It parses three JSON requests (the third has no total), authorizes each payment through the gateway with a one-second timeout that fails the order instead of returning `None`, stores the order on the blocking pool, and appends it to the archive, which the program opens as a lifted ZIO Blocks resource for its whole run. A request that fails to parse or to authorize is printed as `rejected: …` and does not stop the run. Then it sums the revenue with a stream inside `JdbcTransactor#connect`, and streams a shipment events file through `ZStream` or fs2, keeping only the `shipped` events. Unlike the sections, the gateway answers after 50 ms:

<Tabs groupId="effect-system" defaultValue="zio">
  <TabItem value="zio" label="ZIO">

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/CompleteZio.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using dep dev.zio::zio-blocks-sql:0.0.56
//> using dep dev.zio::zio-blocks-streams:0.0.56
//> using dep dev.zio::zio-blocks-scope:0.0.56
//> using dep dev.zio::zio-blocks-async:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
//> using dep dev.zio::zio:2.1.27
//> using dep dev.zio::zio-streams:2.1.27
//> using options -experimental
//> using jvm 21
//> using file Orders.scala
//> using file OrderArchive.scala
//> using file PaymentGateway.scala
//> using file Cancellable.scala
//> using file Opened.scala
//> using file ZioResources.scala
//> using file ZioAsync.scala
//> using file BlockingPull.scala
//> using file ZioStreams.scala
// Run from this folder:  scala-cli run CompleteZio.scala
// Or from zio-blocks-examples:  sbt "effectIntegration/runMain effectintegration.CompleteZio"

package effectintegration

object CompleteZio extends ZIOAppDefault {
  val gateway = new PaymentGateway(latencyMillis = 50)

  val requests = List(
    """{"id":1,"customer":"Ada","totalCents":4200}""",
    """{"id":2,"customer":"Grace","totalCents":1999}""",
    """{"id":3,"customer":"Linus"}"""
  )

  def placeOrder(transactor: JdbcTransactor, archive: OrderArchive)(body: String): Task[String] =
    for {
      order <- ZIO.fromEither(body.fromJson[Order])
      auth  <- ZioAsync.fromAsync(gateway.authorize(order)).timeoutFail(new Exception("payment timed out"))(1.second)
      _     <- ZIO.attemptBlocking(transactor.transact(OrdersDb.insert(order)))
      _     <- ZIO.attemptBlocking(archive.append(order))
    } yield s"order ${order.id} placed ($auth)"

  val run =
    ZIO.scoped {
      for {
        transactor <- ZIO.attemptBlocking(OrdersDb.open())
        path       <- ZIO.attemptBlocking(Files.createTempFile("archive", ".jsonl"))
        archive    <- ZioResources.fromResource(OrderArchive.resource(path))
        _          <- ZIO.foreachDiscard(requests) { body =>
               placeOrder(transactor, archive)(body).foldZIO(
                 error => Console.printLine(s"rejected: ${error.getMessage}"),
                 placed => Console.printLine(placed)
               )
             }
        revenue <- ZIO.attemptBlocking(transactor.connect(OrdersDb.totals().runFold(0L)(_ + _))).absolve
        _       <- Console.printLine(s"revenue: $revenue cents")
        events  <- ZIO.attemptBlocking(
                    OrderEvents.write(
                      """{"orderId":1,"status":"shipped"}""",
                      """{"orderId":2,"status":"paid"}""",
                      """{"orderId":2,"status":"shipped"}"""
                    )
                  )
        _ <- ZioStreams
               .toZStream(OrderEvents.fromFile(events))
               .filter(_.status == "shipped")
               .foreach(event => Console.printLine(s"shipped: order ${event.orderId}"))
      } yield ()
    }
}
```

  </TabItem>
  <TabItem value="cats" label="cats-effect">

```scala title="zio-blocks-examples/effect-integration/src/main/scala/effectintegration/CompleteCats.scala"
//> using scala 3.9.0
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using dep dev.zio::zio-blocks-sql:0.0.56
//> using dep dev.zio::zio-blocks-streams:0.0.56
//> using dep dev.zio::zio-blocks-scope:0.0.56
//> using dep dev.zio::zio-blocks-async:0.0.56
//> using dep org.xerial:sqlite-jdbc:3.53.4.0
//> using dep org.typelevel::cats-effect:3.7.1
//> using dep co.fs2::fs2-core:3.14.0
//> using options -experimental
//> using jvm 21
//> using file Orders.scala
//> using file OrderArchive.scala
//> using file PaymentGateway.scala
//> using file Cancellable.scala
//> using file Opened.scala
//> using file CatsResources.scala
//> using file CatsAsync.scala
//> using file BlockingPull.scala
//> using file CatsStreams.scala
// Run from this folder:  scala-cli run CompleteCats.scala
// Or from zio-blocks-examples:  sbt "effectIntegration/runMain effectintegration.CompleteCats"

package effectintegration

object CompleteCats extends IOApp.Simple {
  val gateway = new PaymentGateway(latencyMillis = 50)

  val requests = List(
    """{"id":1,"customer":"Ada","totalCents":4200}""",
    """{"id":2,"customer":"Grace","totalCents":1999}""",
    """{"id":3,"customer":"Linus"}"""
  )

  def placeOrder(transactor: JdbcTransactor, archive: OrderArchive)(body: String): IO[String] =
    for {
      order <- IO.fromEither(body.fromJson[Order])
      auth  <- CatsAsync
                .fromAsync(gateway.authorize(order))
                .timeoutTo(1.second, IO.raiseError(new TimeoutException("payment timed out")))
      _ <- IO.blocking(transactor.transact(OrdersDb.insert(order)))
      _ <- IO.blocking(archive.append(order))
    } yield s"order ${order.id} placed ($auth)"

  val resources: Resource[IO, (JdbcTransactor, OrderArchive)] =
    for {
      transactor <- Resource.eval(IO.blocking(OrdersDb.open()))
      path       <- Resource.eval(IO.blocking(Files.createTempFile("archive", ".jsonl")))
      archive    <- CatsResources.fromResource(OrderArchive.resource(path))
    } yield (transactor, archive)

  val run =
    resources.use { case (transactor, archive) =>
      for {
        _ <- requests.traverse_ { body =>
               placeOrder(transactor, archive)(body).attempt.flatMap {
                 case Left(error)   => IO.println(s"rejected: ${error.getMessage}")
                 case Right(placed) => IO.println(placed)
               }
             }
        revenue <- IO.blocking(transactor.connect(OrdersDb.totals().runFold(0L)(_ + _))).rethrow
        _       <- IO.println(s"revenue: $revenue cents")
        events  <- IO.blocking(
                    OrderEvents.write(
                      """{"orderId":1,"status":"shipped"}""",
                      """{"orderId":2,"status":"paid"}""",
                      """{"orderId":2,"status":"shipped"}"""
                    )
                  )
        _ <- CatsStreams
               .toFs2(OrderEvents.fromFile(events))
               .filter(_.status == "shipped")
               .evalMap(event => IO.println(s"shipped: order ${event.orderId}"))
               .compile
               .drain
      } yield ()
    }
}
```

  </TabItem>
</Tabs>

Both programs print the same lines. The archive closes last, when the program's scope or `Resource` ends:

```text
archive opened
order 1 placed (auth-1)
order 2 placed (auth-2)
rejected: missing required field "totalCents" at: .
revenue: 6199 cents
shipped: order 1
shipped: order 2
order-events file closed
archive closed
```

## Running the Examples

All runnable code from this guide is in [`zio-blocks-examples/effect-integration/src/main/scala/effectintegration/`](https://github.com/zio/zio-blocks/tree/main/zio-blocks-examples/effect-integration/src/main/scala/effectintegration). Run any file from that folder with `scala-cli run <File>.scala`, or from `zio-blocks-examples` with `sbt "effectIntegration/runMain effectintegration.<ObjectName>"`.

## Going Further

- [How to Use the sql Module from a ZIO Application](./sql-from-zio.md) covers `TransactorZIO`, its `ZLayer`, and transactions whose body is a ZIO effect.
- [How to Wrap Callback and Future-Based APIs in Async](./wrap-callback-and-future-apis.md) explains the `cancellable` helper and the other ways into `Async`.
- [How to Stream Files and Sockets Without Leaking Resources](./stream-files-and-sockets.md) builds ZIO Blocks streams over files and sockets, which the stream bridge above turns into `ZStream` or fs2 streams.
- [Coming from fs2, ZIO Streams or Ox](./streaming-library-migration.md) and [Coming from cats-effect Resource, ZIO Scope/ZLayer, macwire or Using](./resource-and-di-migration.md) are for moving code off your effect system rather than calling ZIO Blocks from it.

---

## How to Validate Query Parameters and Headers and Answer 400

In this guide, we read the query parameters and headers of a bookstore API as typed values: `limit` and `offset` as `Int`s, an author and year filter as a case class, and an `X-Request-Id` header as a `UUID`. By the end, `GET /books` answers every missing, malformed or out-of-range input with a 400 response whose JSON body names each bad parameter and says what is wrong with it. It never crashes, and it never quietly falls back to a default.

We use the typed accessors of the `http-model-schema` module (`query[T]`, `header[T]` and a derived `QueryCodec`) on top of the small JDK server from [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md). Every section starts that server on a free port, calls it, and prints what came back. The guide covers:

- reading one typed query parameter, required or optional;
- turning every `Missing` and `Malformed` error into one JSON 400 response;
- range checks after decoding;
- decoding a whole filter object with a derived `QueryCodec`;
- typed headers;
- values with spaces and plus signs.

## The Problem

A query parameter arrives as a `String`, or not at all. Each handler has to tell three cases apart: the parameter is missing, it is present but is not a number, or it is a number but out of range. Most handlers get one of these wrong. They crash on `"abc".toInt`, silently use a default when the client sent garbage, or answer a bare 400 that doesn't say which parameter was wrong.

For example, consider `GET /books?limit=20&offset=40&year=2014` handled with nothing but `QueryParams#getFirst`:

```scala

def listBooksByHand(request: Request): Response = {
  val limit: Either[String, Int] = request.queryParams.getFirst("limit") match {
    case None      => Right(20)
    case Some(raw) => raw.toIntOption.toRight(s"limit must be an integer, got '$raw'")
  }
  val offset: Either[String, Int] = request.queryParams.getFirst("offset") match {
    case None      => Right(0)
    case Some(raw) => raw.toIntOption.toRight(s"offset must be an integer, got '$raw'")
  }
  val year: Either[String, Option[Int]] = request.queryParams.getFirst("year") match {
    case None      => Right(None)
    case Some(raw) => raw.toIntOption.map(Some(_)).toRight(s"year must be an integer, got '$raw'")
  }
  (limit, offset, year) match {
    case (Right(l), Right(o), Right(y)) => Response.text(s"limit=$l offset=$o year=$y")
    case _                              =>
      val errors = List(limit, offset, year).collect { case Left(error) => error }
      Response.text(Status.BadRequest, errors.mkString("\n"))
  }
}
```

Every parameter repeats the same lookup, parse and error message, and the final match grows with each new parameter. Range checks, headers and a machine-readable error body would double the code again. A typo in one of the copied error strings, or a forgotten `toIntOption`, is not caught by anything.

In this guide, we let a `Schema` do the parsing, keep the "missing" and "malformed" cases as data, and write the 400 response once.

## Prerequisites

Add the HTTP model, its schema integration and the schema module (for the JSON error body) to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-http-model"        % "0.0.56",
  "dev.zio" %% "zio-blocks-http-model-schema" % "0.0.56",
  "dev.zio" %% "zio-blocks-schema"            % "0.0.56"
)

// Scala 3.9 and newer
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-http-model"        % "0.0.56",
  "dev.zio" %% "zio-blocks-next-http-model-schema" % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema"            % "0.0.56"
)
scalacOptions += "-experimental"
```

The code is the same for every Scala version. The HTTP model is built on the ZIO Blocks streams module, which is compiled for Java 21, so run the program on **JDK 21 or newer**.

You should know how to derive a `Schema` for a case class, as shown in [Your First Schema](./your-first-schema.md), and how the JDK adapter turns a `Request => Response` function into a server, from [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md). We reuse that guide's `JdkHttp` adapter and its `Client` unchanged:

- `JdkHttp.withServer(app) { port => ... }` starts a server for `app` on a free loopback port and stops it when the block returns.
- `Client.call(port, method, target)` sends one request and prints `METHOD target -> status [content type, length]` followed by the body.

Both files are listed in [Running the Examples](#running-the-examples).

We use these imports throughout the guide. The `zio.http.schema._` import adds the typed accessors to `Request`, `QueryParams` and `Headers`:

```scala

```

## The Bookstore Model

The API lists three books. A derived `Schema` gives us the JSON codec for the list:

```scala
final case class Book(id: Int, title: String, authors: List[String], year: Int)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

val books: List[Book] = List(
  Book(1, "Zionomicon", List("John A. De Goes", "Adam Fraser", "Milad Khajavi"), 2024),
  Book(2, "Programming in Scala", List("Martin Odersky", "Lex Spoon", "Bill Venners"), 2008),
  Book(3, "Functional Programming in Scala", List("Paul Chiusano", "Rúnar Bjarnason"), 2014)
)

val bookListJson: JsonCodec[List[Book]] = Schema[List[Book]].jsonCodec
```

## Reading One Typed Query Parameter

`Request#query[T](key)` looks up the first value of `key` and decodes it with the `Schema[T]` in scope. It returns `Either[QueryParamError, T]`: `QueryParamError.Missing(key)` when the parameter is absent, and `QueryParamError.Malformed(key, value, cause)` when the value doesn't parse as a `T`. Both errors have a `message`, so a first version of `GET /books?limit=N` answers 400 with it:

```scala
def listBooks(request: Request): Response =
  request.query[Int]("limit") match {
    case Right(limit) => Response.json(bookListJson.encodeToString(books.take(limit)))
    case Left(error)  => Response.text(Status.BadRequest, error.message)
  }
```

To call it with a good, a missing and a malformed `limit`:

```scala
JdkHttp.withServer(listBooks) { port =>
  Client.call(port, "GET", "/books?limit=1")
  Client.call(port, "GET", "/books")
  Client.call(port, "GET", "/books?limit=abc")
}
// GET /books?limit=1 -> 200 [application/json, 103 bytes]
// [{"id":1,"title":"Zionomicon","authors":["John A. De Goes","Adam Fraser","Milad Khajavi"],"year":2024}]
// GET /books -> 400 [text/plain; charset=UTF-8, 30 bytes]
// Missing query parameter: limit
// GET /books?limit=abc -> 400 [text/plain; charset=UTF-8, 72 bytes]
// Malformed query parameter 'limit' value 'abc': Cannot parse 'abc' as Int
```

`query[T]` decodes `String`, `Int`, `Long`, `Short`, `Byte`, `Boolean`, `Double`, `Float`, `Char`, `BigInt`, `BigDecimal` and `UUID`. `queryAll[T]` decodes every value of a repeated key, such as `?year=2008&year=2014`, into a `Chunk[T]`. The same three methods exist on `QueryParams`, so `request.queryParams.query[Int]("limit")` works too.

## Making a Parameter Optional Without Hiding Mistakes

Most paging parameters are optional: no `limit` means "use 20". The module has `queryOrElse[T](key, default)` for this, but it returns the default whenever decoding fails, not only when the parameter is missing. A client that sends `?limit=abc` never learns about its mistake:

```scala
val typo = Request.get(URL.parse("/books?limit=abc").toOption.get)
```

```scala
println(typo.queryOrElse[Int]("limit", 20))
// 20
```

To fall back only when the parameter is missing, match on `QueryParamError.Missing` and keep `Malformed` as an error:

```scala
def optionalQuery[T: Schema](request: Request, key: String): Either[QueryParamError, Option[T]] =
  request.query[T](key) match {
    case Left(QueryParamError.Missing(_)) => Right(None)
    case other                            => other.map(Some(_))
  }

def listBooksWithDefault(request: Request): Response =
  optionalQuery[Int](request, "limit") match {
    case Right(limit) => Response.json(bookListJson.encodeToString(books.take(limit.getOrElse(20))))
    case Left(error)  => Response.text(Status.BadRequest, error.message)
  }
```

Now a missing `limit` lists all three books, and `limit=abc` is still a 400:

```scala
JdkHttp.withServer(listBooksWithDefault) { port =>
  Client.call(port, "GET", "/books")
  Client.call(port, "GET", "/books?limit=abc")
}
// GET /books -> 200 [application/json, 321 bytes]
// [{"id":1,"title":"Zionomicon","authors":["John A. De Goes","Adam Fraser","Milad Khajavi"],"year":2024},{"id":2,"title":"Programming in Scala","authors":["Martin Odersky","Lex Spoon","Bill Venners"],"year":2008},{"id":3,"title":"Functional Programming in Scala","authors":["Paul Chiusano","Rúnar Bjarnason"],"year":2014}]
// GET /books?limit=abc -> 400 [text/plain; charset=UTF-8, 72 bytes]
// Malformed query parameter 'limit' value 'abc': Cannot parse 'abc' as Int
```

:::warning
`request.query[Option[Int]]("limit")` compiles but does not work: it returns `Malformed(limit, 5, Unsupported schema type for string decoding)` even for `?limit=5`. `query[T]` and `header[T]` decode primitive schemas only, so `Option`, collections and wrapper types fail at runtime. Ask for the primitive type and handle `Missing` as above.
:::

## Answering 400 with a JSON Body That Names Every Problem

A plain-text message is fine for a person, but an API client wants to know which parameter failed so it can highlight the right form field. We describe each failure as a `Problem` (where it was, its name and the reason) and send all of them in one JSON body. The [Your First Schema](./your-first-schema.md) tutorial shows the derived JSON codec we use, and [How to Customize JSON Encoding](./customize-json-encoding.md) shows how to rename the fields if your API style needs it:

```scala
final case class Problem(in: String, name: String, reason: String)

object Problem {
  implicit val schema: Schema[Problem] = Schema.derived

  def query(error: QueryParamError): Problem = error match {
    case QueryParamError.Missing(key)             => Problem("query", key, "is required")
    case QueryParamError.Malformed(key, _, cause) => Problem("query", key, cause)
  }

  def header(error: HeaderError): Problem = error match {
    case HeaderError.Missing(name)             => Problem("header", name, "is required")
    case HeaderError.Malformed(name, _, cause) => Problem("header", name, cause)
  }
}

final case class ErrorBody(problems: List[Problem])

object ErrorBody {
  implicit val schema: Schema[ErrorBody] = Schema.derived

  val json: JsonCodec[ErrorBody] = Schema[ErrorBody].jsonCodec
}

def badRequest(problems: List[Problem]): Response =
  Response.json(Status.BadRequest, ErrorBody.json.encodeToString(ErrorBody(problems)))
```

A for-comprehension over `Either` stops at the first `Left`, so a request with two bad parameters would report only one. To report all of them, we decode each parameter into a `Checked[A]`, an `Either` that carries a list of problems, and combine two results with `both`, which keeps the problems of both sides:

```scala
type Checked[A] = Either[List[Problem], A]

def checkedQuery[T: Schema](request: Request, key: String): Checked[Option[T]] =
  optionalQuery[T](request, key).left.map(error => List(Problem.query(error)))

def both[A, B](a: Checked[A], b: Checked[B]): Checked[(A, B)] =
  (a, b) match {
    case (Right(x), Right(y)) => Right((x, y))
    case _                    => Left(a.swap.getOrElse(Nil) ++ b.swap.getOrElse(Nil))
  }
```

With these, paging with `limit` (default 20) and `offset` (default 0) is two lookups and one `both`:

```scala
final case class Paging(limit: Int, offset: Int)

def paging(request: Request): Checked[Paging] = {
  val limit  = checkedQuery[Int](request, "limit").map(_.getOrElse(20))
  val offset = checkedQuery[Int](request, "offset").map(_.getOrElse(0))
  both(limit, offset).map { case (limit, offset) => Paging(limit, offset) }
}

def listPage(request: Request): Response =
  paging(request) match {
    case Right(page)    => Response.json(bookListJson.encodeToString(books.slice(page.offset, page.offset + page.limit)))
    case Left(problems) => badRequest(problems)
  }
```

A request with two malformed parameters now gets both of them back:

```scala
JdkHttp.withServer(listPage) { port =>
  Client.call(port, "GET", "/books?limit=1&offset=2")
  Client.call(port, "GET", "/books?limit=abc&offset=xyz")
}
// GET /books?limit=1&offset=2 -> 200 [application/json, 111 bytes]
// [{"id":3,"title":"Functional Programming in Scala","authors":["Paul Chiusano","Rúnar Bjarnason"],"year":2014}]
// GET /books?limit=abc&offset=xyz -> 400 [application/json, 149 bytes]
// {"problems":[{"in":"query","name":"limit","reason":"Cannot parse 'abc' as Int"},{"in":"query","name":"offset","reason":"Cannot parse 'xyz' as Int"}]}
```

## Checking Ranges After Decoding

`?limit=500` and `?offset=-1` decode fine as `Int`s, but neither makes sense. The decoders in this module only parse: they don't check the `Validation` constraints a `Schema` can carry, such as `Validation.Numeric.Range` from the [validation reference](../reference/schema/validation.md). A schema for `Int` with a range of 1 to 100 still decodes `?limit=500` to `Right(500)`. Range checks are therefore your own code, run after decoding. `check` turns a predicate into a `Checked` value with a reason the client can read:

```scala
def check[A](name: String, reason: String)(ok: A => Boolean)(value: A): Checked[A] =
  if (ok(value)) Right(value) else Left(List(Problem("query", name, reason)))

def checkedPaging(request: Request): Checked[Paging] = {
  val limit = checkedQuery[Int](request, "limit")
    .map(_.getOrElse(20))
    .flatMap(check[Int]("limit", "must be between 1 and 100")(n => n >= 1 && n <= 100))
  val offset = checkedQuery[Int](request, "offset")
    .map(_.getOrElse(0))
    .flatMap(check[Int]("offset", "must not be negative")(_ >= 0))
  both(limit, offset).map { case (limit, offset) => Paging(limit, offset) }
}

def listCheckedPage(request: Request): Response =
  checkedPaging(request) match {
    case Right(page)    => Response.json(bookListJson.encodeToString(books.slice(page.offset, page.offset + page.limit)))
    case Left(problems) => badRequest(problems)
  }
```

Because `check` runs inside `flatMap`, a malformed value is reported as malformed and only a well-formed value is range-checked:

```scala
JdkHttp.withServer(listCheckedPage) { port =>
  Client.call(port, "GET", "/books?limit=500&offset=-1")
  Client.call(port, "GET", "/books?limit=abc&offset=-1")
}
// GET /books?limit=500&offset=-1 -> 400 [application/json, 144 bytes]
// {"problems":[{"in":"query","name":"limit","reason":"must be between 1 and 100"},{"in":"query","name":"offset","reason":"must not be negative"}]}
// GET /books?limit=abc&offset=-1 -> 400 [application/json, 144 bytes]
// {"problems":[{"in":"query","name":"limit","reason":"Cannot parse 'abc' as Int"},{"in":"query","name":"offset","reason":"must not be negative"}]}
```

## Decoding a Whole Filter with a Derived QueryCodec

When a set of parameters belongs together, such as an optional author and an optional year, a case class describes them better than separate lookups. `Schema#derive(DefaultQueryFormat)` derives a `QueryCodec` for it, and `QueryCodec#decode` reads the whole case class from `QueryParams` in one call. Each field name is a parameter name, and an `Option` field is `None` when its parameter is missing:

```scala
final case class BookFilter(author: Option[String], year: Option[Int])

object BookFilter {
  implicit val schema: Schema[BookFilter] = Schema.derived

  val codec: QueryCodec[BookFilter] = Schema[BookFilter].derive(DefaultQueryFormat)
}

def matches(filter: BookFilter)(book: Book): Boolean =
  filter.author.forall(book.authors.contains) && filter.year.forall(_ == book.year)
```

`decode` returns `Either[SchemaError, BookFilter]`, not a `QueryParamError`. A missing required field is a `SchemaError.MissingField` that carries the field name. A malformed value is a single `SchemaError` whose message names the parameter but has no separate field for it, so we report it under the name of the whole filter:

```scala
def decodedProblems(name: String)(error: SchemaError): List[Problem] =
  error.errors.map {
    case SchemaError.MissingField(_, field) => Problem("query", field, "is required")
    case other                              => Problem("query", name, other.message)
  }

def listFiltered(request: Request): Response =
  BookFilter.codec.decode(request.queryParams) match {
    case Right(filter) => Response.json(bookListJson.encodeToString(books.filter(matches(filter))))
    case Left(error)   => badRequest(decodedProblems("filter")(error))
  }
```

To filter by year, by author and year, and with a malformed year:

```scala
JdkHttp.withServer(listFiltered) { port =>
  Client.call(port, "GET", "/books?year=2014")
  Client.call(port, "GET", "/books?author=Martin%20Odersky&year=2008")
  Client.call(port, "GET", "/books?year=abc")
}
// GET /books?year=2014 -> 200 [application/json, 111 bytes]
// [{"id":3,"title":"Functional Programming in Scala","authors":["Paul Chiusano","Rúnar Bjarnason"],"year":2014}]
// GET /books?author=Martin%20Odersky&year=2008 -> 200 [application/json, 109 bytes]
// [{"id":2,"title":"Programming in Scala","authors":["Martin Odersky","Lex Spoon","Bill Venners"],"year":2008}]
// GET /books?year=abc -> 400 [application/json, 128 bytes]
// {"problems":[{"in":"query","name":"filter","reason":"Malformed query parameter 'year' value 'abc': Cannot parse 'abc' as Int"}]}
```

The codec stops at the first bad field, so `?year=abc` with another bad field still reports one problem.

:::warning
A derived `QueryCodec` ignores default values. `Paging(limit: Int = 20, offset: Int = 0)` does not fall back to `offset = 0` when `offset` is missing; decoding fails instead:

```scala
final case class DefaultedPaging(limit: Int = 20, offset: Int = 0)

object DefaultedPaging {
  implicit val schema: Schema[DefaultedPaging] = Schema.derived

  val codec: QueryCodec[DefaultedPaging] = Schema[DefaultedPaging].derive(DefaultQueryFormat)
}
```

```scala
println(DefaultedPaging.codec.decode(QueryParams.fromEncoded("limit=5&offset=10")))
// Right(DefaultedPaging(5,10))
println(DefaultedPaging.codec.decode(QueryParams.fromEncoded("limit=5")))
// Left(zio.blocks.schema.SchemaError: Missing field 'offset' at: .)
```

Use `Option` fields and apply the defaults yourself, or decode the parameters one by one as in the previous sections.
:::

Use a derived codec when a group of optional parameters maps onto a case class you already have, and one error at a time is enough. Use `query[T]` per parameter when you need defaults, every error at once, or the parameter name as structured data.

## Reading Typed Headers

Headers work the same way, with `HeaderError.Missing(name)` and `HeaderError.Malformed(name, value, cause)`. Header names are case-insensitive, so `X-Request-Id` and `x-request-id` find the same header. The `Client` from the adapter guide sends no custom headers, so we add a small client that sends a `GET` with the headers we pass and prints them:

```scala
object HeaderClient {
  private val client = HttpClient.newBuilder().version(HttpClient.Version.HTTP_1_1).build()

  def get(port: Int, target: String, headers: (String, String)*): Unit = {
    val builder = HttpRequest.newBuilder(URI.create(s"http://127.0.0.1:$port$target")).GET()
    headers.foreach { case (name, value) => builder.header(name, value) }
    val response = client.send(builder.build(), HttpResponse.BodyHandlers.ofString())
    val sent     = headers.map { case (name, value) => s"$name: $value" }.mkString(" [", ", ", "]")
    println(s"GET $target${if (headers.isEmpty) "" else sent} -> ${response.statusCode}")
    if (response.body.nonEmpty) println(response.body)
  }
}
```

An optional `X-Request-Id` must be a `UUID` when it is present. We read it with `Headers#header[T]` and treat `Missing` as `None`, as we did for `limit`:

```scala
def requestId(request: Request): Checked[Option[UUID]] =
  request.headers.header[UUID]("X-Request-Id") match {
    case Right(id)                    => Right(Some(id))
    case Left(HeaderError.Missing(_)) => Right(None)
    case Left(error)                  => Left(List(Problem.header(error)))
  }
```

:::warning
Call `header[T]` on `request.headers`, not on the request. `Request` has its own `header(Header.Codec[A]): Option[A]` method, and a class's own method always wins over an extension method with the same name, so `request.header[UUID]("X-Request-Id")` does not compile. On Scala 3.9 the compiler reports `Found: ("X-Request-Id" : String)` and `Required: zio.http.Header.Codec[java.util.UUID]`. The same applies to `Response`. `request.headerAll[T]` and `request.headerOrElse[T]` have no such clash and work on the request directly.
:::

`headerOrElse[T](name, default)` is safe for a `String`, because a string never fails to decode. It returns the raw value, though, and `Accept-Language` is a list of language tags with weights. For a header the HTTP model knows, pass its typed `Header.Codec` to `header` instead. `request.headers.header(Header.AcceptLanguage)` returns `Either[HeaderError, Header.AcceptLanguage]` with the parsed ranges:

```scala
val german = Request.get(URL.parse("/books").toOption.get).addHeader("Accept-Language", "de-CH, en;q=0.8")
```

```scala
println(german.headerOrElse[String]("Accept-Language", "en"))
// de-CH, en;q=0.8
println(german.headers.header(Header.AcceptLanguage))
// Right(AcceptLanguage(Chunk(LanguageRange(de-CH,1.0),LanguageRange(en,0.8))))
```

The catalog handler uses the first preferred language, or English when the header is missing, and rejects a malformed request id:

```scala
def language(request: Request): String =
  request.headers.header(Header.AcceptLanguage) match {
    case Right(accept) => accept.languages.headOption.fold("en")(_.tag)
    case Left(_)       => "en"
  }

def catalog(request: Request): Response =
  requestId(request) match {
    case Right(id)      => Response.text(s"Catalog in ${language(request)} for request ${id.getOrElse("(none)")}")
    case Left(problems) => badRequest(problems)
  }
```

```scala
JdkHttp.withServer(catalog) { port =>
  HeaderClient.get(port, "/books")
  HeaderClient.get(
    port,
    "/books",
    "X-Request-Id"    -> "3f2504e0-4f89-11d3-9a0c-0305e82c3301",
    "Accept-Language" -> "de-CH, en;q=0.8"
  )
  HeaderClient.get(port, "/books", "X-Request-Id" -> "42")
}
// GET /books -> 200
// Catalog in en for request (none)
// GET /books [X-Request-Id: 3f2504e0-4f89-11d3-9a0c-0305e82c3301, Accept-Language: de-CH, en;q=0.8] -> 200
// Catalog in de-CH for request 3f2504e0-4f89-11d3-9a0c-0305e82c3301
// GET /books [X-Request-Id: 42] -> 400
// {"problems":[{"in":"header","name":"X-Request-Id","reason":"Cannot parse '42' as UUID"}]}
```

Header problems use the same `Problem` type with `in = "header"`, so they can share one 400 response with the query problems, as the complete example below does.

## Values with Spaces and Plus Signs

A value with a space must be encoded in the URL. `QueryParams` decodes `%20` to a space, but it keeps `+` as a plus sign. Browsers encode a space as `+` when they submit an HTML form with `method="get"`, so an author search from such a form arrives as `Martin+Odersky` and matches nothing:

```scala
def byAuthor(request: Request): Response =
  request.query[String]("author") match {
    case Right(author) => Response.json(bookListJson.encodeToString(books.filter(_.authors.contains(author))))
    case Left(error)   => badRequest(List(Problem.query(error)))
  }
```

```scala
JdkHttp.withServer(byAuthor) { port =>
  Client.call(port, "GET", "/books?author=Martin%20Odersky")
  Client.call(port, "GET", "/books?author=Martin+Odersky")
}
// GET /books?author=Martin%20Odersky -> 200 [application/json, 109 bytes]
// [{"id":2,"title":"Programming in Scala","authors":["Martin Odersky","Lex Spoon","Bill Venners"],"year":2008}]
// GET /books?author=Martin+Odersky -> 200 [application/json, 2 bytes]
// []
```

Once the query is a `QueryParams`, `+` and `%2B` both look like `+`, so the handler can't repair the value. The fix has to see the raw query string. `Form.fromString` decodes form data and treats `+` as a space, so this wrapper decodes `URI#getRawQuery` with it and hands the result to the unchanged `JdkHttp.handler`:

```scala
def formQueryHandler(app: Request => Response): HttpHandler = exchange => {
  val rawQuery = Option(exchange.getRequestURI.getRawQuery).getOrElse("")
  val params   = Form.fromString(rawQuery).entries.foldLeft(QueryParams.empty) { case (acc, (key, value)) =>
    acc.add(key, value)
  }
  JdkHttp.handler(request => app(request.updateUrl(_.copy(queryParams = params)))).handle(exchange)
}
```

`JdkHttp.withServer` always installs `JdkHttp.handler`, so we write a variant of it that installs the wrapper:

```scala
def withFormServer[A](app: Request => Response)(use: Int => A): A = {
  val server = HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress, 0), 0)
  server.createContext("/", formQueryHandler(app))
  server.start()
  try use(server.getAddress.getPort)
  finally server.stop(0)
}
```

The same `+` request now finds the book:

```scala
withFormServer(byAuthor) { port =>
  Client.call(port, "GET", "/books?author=Martin+Odersky")
}
// GET /books?author=Martin+Odersky -> 200 [application/json, 109 bytes]
// [{"id":2,"title":"Programming in Scala","authors":["Martin Odersky","Lex Spoon","Bill Venners"],"year":2008}]
```

Use the wrapper only for endpoints that HTML forms submit to. API clients that encode a space as `%20` work with the plain handler.

## Putting It Together

The complete example serves `GET /books` with paging, the derived filter and the optional request id. Every bad parameter and header ends up in one 400 response:

```scala title="http-model-examples/src/main/scala/validatequeryparametersandheaders/CompleteExample.scala"
package validatequeryparametersandheaders

/**
 * How to Validate Query Parameters and Headers and Answer 400 — Complete
 * Example
 *
 * `GET /books` with paging (`limit` 1 to 100, `offset` 0 or more), a filter by
 * author and year decoded with a derived `QueryCodec`, and an optional
 * `X-Request-Id` header. Every bad parameter and header is reported in one JSON
 * 400 response.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * validatequeryparametersandheaders.CompleteExample"
 */
object CompleteExample {
  final case class Paging(limit: Int, offset: Int)

  final case class BookFilter(author: Option[String], year: Option[Int])

  object BookFilter {
    implicit val schema: Schema[BookFilter] = Schema.derived

    val codec: QueryCodec[BookFilter] = Schema[BookFilter].derive(DefaultQueryFormat)
  }

  def paging(request: Request): Checked[Paging] = {
    val limit = checkedQuery[Int](request, "limit")
      .map(_.getOrElse(20))
      .flatMap(check[Int]("limit", "must be between 1 and 100")(n => n >= 1 && n <= 100))
    val offset = checkedQuery[Int](request, "offset")
      .map(_.getOrElse(0))
      .flatMap(check[Int]("offset", "must not be negative")(_ >= 0))
    both(limit, offset).map { case (limit, offset) => Paging(limit, offset) }
  }

  def filter(request: Request): Checked[BookFilter] =
    BookFilter.codec.decode(request.queryParams).left.map(Problem.decoded("filter"))

  def requestId(request: Request): Checked[Option[UUID]] =
    request.headers.header[UUID]("X-Request-Id") match {
      case Right(id)                    => Right(Some(id))
      case Left(HeaderError.Missing(_)) => Right(None)
      case Left(error)                  => Left(List(Problem.header(error)))
    }

  def matches(filter: BookFilter)(book: Book): Boolean =
    filter.author.forall(book.authors.contains) && filter.year.forall(_ == book.year)

  def listBooks(request: Request): Response =
    both(both(paging(request), filter(request)), requestId(request)) match {
      case Right(((page, bookFilter), _)) =>
        val found = books.filter(matches(bookFilter)).slice(page.offset, page.offset + page.limit)
        Response.json(bookListJson.encodeToString(found))
      case Left(problems) => badRequest(problems)
    }

  def app(request: Request): Response =
    (request.method, request.path.segments.toList) match {
      case (Method.GET, List("books")) => listBooks(request)
      case _                           =>
        Response.text(Status.NotFound, s"No route for ${request.method.name} ${request.path.encode}")
    }

  def main(args: Array[String]): Unit =
    JdkHttp.withServer(app) { port =>
      Client.call(port, "GET", "/books?author=Martin%20Odersky")
      Client.call(port, "GET", "/books?limit=1&offset=1")
      Client.call(port, "GET", "/books?limit=0&offset=-1&year=abc")
      HeaderClient.get(port, "/books?year=2024", "X-Request-Id" -> "3f2504e0-4f89-11d3-9a0c-0305e82c3301")
      HeaderClient.get(port, "/books?limit=abc", "X-Request-Id" -> "42")
    }
}
```

## Running the Examples

All code from this guide is available as runnable examples in the `http-model-examples` directory. It is a standalone sbt build that uses the published ZIO Blocks artifacts. To run them, clone the repository and move into that directory:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/http-model-examples
```

The examples share five files: the adapter and the client from the adapter guide (copied unchanged), the header client, the bookstore model, and the helpers this guide builds (`optionalQuery`, `Problem`, `badRequest`, `checkedQuery`, `check` and `both`).

<details>
  <summary>http-model-examples/src/main/scala/validatequeryparametersandheaders/JdkHttp.scala</summary>

```scala title="http-model-examples/src/main/scala/validatequeryparametersandheaders/JdkHttp.scala" showLineNumbers
package validatequeryparametersandheaders

/**
 * How to Serve http-model Requests with the JDK HTTP Server — the adapter
 *
 * Glue between the JDK's built-in `com.sun.net.httpserver.HttpServer` and a
 * plain `Request => Response` function. It is meant for examples, tools and
 * small services, not for production traffic.
 */
object JdkHttp {

  /**
   * Reads the method, URL, headers and body of an exchange into a `Request`.
   * Returns `Left` with a ready-made error response when the method is unknown
   * (501) or the request target is not a valid URL (400).
   */
  def toRequest(exchange: HttpExchange): Either[Response, Request] = {
    val uri    = exchange.getRequestURI
    val target = uri.getRawPath + Option(uri.getRawQuery).fold("")("?" + _)
    for {
      method <- Method.fromString(exchange.getRequestMethod).toRight(Response(Status.NotImplemented))
      url    <- URL.parse(target).left.map(error => Response.text(Status.BadRequest, error))
    } yield {
      val headers = exchange.getRequestHeaders.asScala.foldLeft(Headers.empty) { case (acc, (name, values)) =>
        values.asScala.foldLeft(acc)((withValue, value) => withValue.add(name, value))
      }
      val contentType =
        headers.get(Header.ContentType).map(_.value).getOrElse(ContentType.`application/octet-stream`)
      val body    = Body.fromArray(exchange.getRequestBody.readAllBytes(), contentType)
      val version = Version.fromString(exchange.getProtocol).getOrElse(Version.`HTTP/1.1`)
      Request(method, url, headers, body, version)
    }
  }

  /**
   * Writes the status, headers and body of a `Response` to an exchange. The JDK
   * server computes `Content-Length` itself, so the response's own
   * `content-length` and `transfer-encoding` headers are skipped.
   */
  def writeResponse(exchange: HttpExchange, response: Response): Unit = {
    val out = exchange.getResponseHeaders
    response.headers.toList.foreach { case (name, value) =>
      if (name != "content-length" && name != "transfer-encoding") out.add(name, value)
    }
    val bytes = response.body.toArray
    if (bytes.nonEmpty && !response.headers.has("content-type"))
      out.set("content-type", response.body.contentType.render)
    val code   = response.status.code
    val isHead = exchange.getRequestMethod == "HEAD"
    if (isHead && bytes.nonEmpty) out.set("content-length", bytes.length.toString)
    val noBody = isHead || code == 204 || code == 304 || bytes.isEmpty
    exchange.sendResponseHeaders(code, if (noBody) -1L else bytes.length.toLong)
    if (!noBody) exchange.getResponseBody.write(bytes)
  }

  /**
   * Turns an application into an `HttpHandler`. A `HEAD` request runs the `GET`
   * route and drops the body; a non-fatal exception or `LinkageError` in the
   * application becomes a 500 response; a failure while writing the response is
   * logged and, if no headers have gone out yet, also becomes a 500; the
   * exchange is closed on every path.
   */
  def handler(app: Request => Response): HttpHandler = exchange =>
    try {
      val response =
        try
          toRequest(exchange) match {
            case Left(error)                                     => error
            case Right(request) if request.method == Method.HEAD => app(request.copy(method = Method.GET))
            case Right(request)                                  => app(request)
          }
        catch {
          case error @ (NonFatal(_) | _: LinkageError) =>
            System.err.println(s"Handler failed: $error")
            Response.text(Status.InternalServerError, "Internal Server Error")
        }
      try writeResponse(exchange, response)
      catch {
        case error @ (NonFatal(_) | _: LinkageError) =>
          System.err.println(s"Writing the response failed: $error")
          if (exchange.getResponseCode == -1) {
            exchange.getResponseHeaders.clear()
            exchange.sendResponseHeaders(Status.InternalServerError.code, -1L)
          }
      }
    } finally exchange.close()

  /**
   * Starts a server on the loopback interface. Port 0 asks the operating system
   * for a free port; read it back with `server.getAddress.getPort`.
   */
  def start(app: Request => Response, port: Int = 0): HttpServer = {
    val server = HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress, port), 0)
    server.createContext("/", handler(app))
    server.start()
    server
  }

  /**
   * Starts a server on a free loopback port, passes the port to `use`, and
   * stops the server when `use` returns or throws.
   */
  def withServer[A](app: Request => Response)(use: Int => A): A = {
    val server = start(app)
    try use(server.getAddress.getPort)
    finally server.stop(0)
  }
}
```

</details>

<details>
  <summary>http-model-examples/src/main/scala/validatequeryparametersandheaders/Client.scala</summary>

```scala title="http-model-examples/src/main/scala/validatequeryparametersandheaders/Client.scala" showLineNumbers
package validatequeryparametersandheaders

/**
 * How to Serve http-model Requests with the JDK HTTP Server — a test client
 *
 * Sends one request with the JDK's `java.net.http.HttpClient` and prints the
 * status code, the `Content-Type` and `Content-Length` headers, and the body.
 */
object Client {
  private val client = HttpClient.newBuilder().version(HttpClient.Version.HTTP_1_1).build()

  def call(port: Int, method: String, target: String, body: String = ""): Unit = {
    val builder = HttpRequest.newBuilder(URI.create(s"http://127.0.0.1:$port$target"))
    val request =
      if (body.isEmpty) builder.method(method, HttpRequest.BodyPublishers.noBody()).build()
      else
        builder
          .method(method, HttpRequest.BodyPublishers.ofString(body))
          .header("content-type", "application/json")
          .build()
    val response                     = client.send(request, HttpResponse.BodyHandlers.ofString())
    def header(name: String): String = response.headers.firstValue(name).orElse("none")
    println(s"$method $target -> ${response.statusCode} [${header("content-type")}, ${header("content-length")} bytes]")
    if (response.body.nonEmpty) println(response.body)
  }
}
```

</details>

<details>
  <summary>http-model-examples/src/main/scala/validatequeryparametersandheaders/HeaderClient.scala</summary>

```scala title="http-model-examples/src/main/scala/validatequeryparametersandheaders/HeaderClient.scala" showLineNumbers
package validatequeryparametersandheaders

/**
 * How to Validate Query Parameters and Headers and Answer 400 — a test client
 * that sends headers
 *
 * Sends one `GET` request with the given request headers and prints the status
 * code, the headers that were sent, and the response body.
 */
object HeaderClient {
  private val client = HttpClient.newBuilder().version(HttpClient.Version.HTTP_1_1).build()

  def get(port: Int, target: String, headers: (String, String)*): Unit = {
    val builder = HttpRequest.newBuilder(URI.create(s"http://127.0.0.1:$port$target")).GET()
    headers.foreach { case (name, value) => builder.header(name, value) }
    val response = client.send(builder.build(), HttpResponse.BodyHandlers.ofString())
    val sent     = headers.map { case (name, value) => s"$name: $value" }.mkString(" [", ", ", "]")
    println(s"GET $target${if (headers.isEmpty) "" else sent} -> ${response.statusCode}")
    if (response.body.nonEmpty) println(response.body)
  }
}
```

</details>

<details>
  <summary>http-model-examples/src/main/scala/validatequeryparametersandheaders/Bookstore.scala</summary>

```scala title="http-model-examples/src/main/scala/validatequeryparametersandheaders/Bookstore.scala" showLineNumbers
package validatequeryparametersandheaders

/**
 * How to Validate Query Parameters and Headers and Answer 400 — the bookstore
 *
 * Three books and the JSON codec the examples use to list them.
 */
final case class Book(id: Int, title: String, authors: List[String], year: Int)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

object Bookstore {
  val books: List[Book] = List(
    Book(1, "Zionomicon", List("John A. De Goes", "Adam Fraser", "Milad Khajavi"), 2024),
    Book(2, "Programming in Scala", List("Martin Odersky", "Lex Spoon", "Bill Venners"), 2008),
    Book(3, "Functional Programming in Scala", List("Paul Chiusano", "Rúnar Bjarnason"), 2014)
  )

  val bookListJson: JsonCodec[List[Book]] = Schema[List[Book]].jsonCodec
}
```

</details>

<details>
  <summary>http-model-examples/src/main/scala/validatequeryparametersandheaders/Params.scala</summary>

```scala title="http-model-examples/src/main/scala/validatequeryparametersandheaders/Params.scala" showLineNumbers
package validatequeryparametersandheaders

/**
 * How to Validate Query Parameters and Headers and Answer 400 — shared helpers
 *
 * The helpers the guide builds section by section: an optional query parameter
 * that still rejects malformed values, a `Problem` for every bad parameter or
 * header, and a 400 response that lists them all as JSON.
 */
final case class Problem(in: String, name: String, reason: String)

object Problem {
  implicit val schema: Schema[Problem] = Schema.derived

  def query(error: QueryParamError): Problem = error match {
    case QueryParamError.Missing(key)             => Problem("query", key, "is required")
    case QueryParamError.Malformed(key, _, cause) => Problem("query", key, cause)
  }

  def header(error: HeaderError): Problem = error match {
    case HeaderError.Missing(name)             => Problem("header", name, "is required")
    case HeaderError.Malformed(name, _, cause) => Problem("header", name, cause)
  }

  def decoded(name: String)(error: SchemaError): List[Problem] =
    error.errors.map {
      case SchemaError.MissingField(_, field) => Problem("query", field, "is required")
      case other                              => Problem("query", name, other.message)
    }
}

final case class ErrorBody(problems: List[Problem])

object ErrorBody {
  implicit val schema: Schema[ErrorBody] = Schema.derived

  val json: JsonCodec[ErrorBody] = Schema[ErrorBody].jsonCodec
}

object Params {

  /** A decoded value, or every problem found while decoding it. */
  type Checked[A] = Either[List[Problem], A]

  def optionalQuery[T: Schema](request: Request, key: String): Either[QueryParamError, Option[T]] =
    request.query[T](key) match {
      case Left(QueryParamError.Missing(_)) => Right(None)
      case other                            => other.map(Some(_))
    }

  def badRequest(problems: List[Problem]): Response =
    Response.json(Status.BadRequest, ErrorBody.json.encodeToString(ErrorBody(problems)))

  def checkedQuery[T: Schema](request: Request, key: String): Checked[Option[T]] =
    optionalQuery[T](request, key).left.map(error => List(Problem.query(error)))

  def check[A](name: String, reason: String)(ok: A => Boolean)(value: A): Checked[A] =
    if (ok(value)) Right(value) else Left(List(Problem("query", name, reason)))

  def both[A, B](a: Checked[A], b: Checked[B]): Checked[(A, B)] =
    (a, b) match {
      case (Right(x), Right(y)) => Right((x, y))
      case _                    => Left(a.swap.getOrElse(Nil) ++ b.swap.getOrElse(Nil))
    }
}
```

</details>

The first example reads `?limit=` as a required parameter, then as an optional one:

<details>
  <summary>http-model-examples/src/main/scala/validatequeryparametersandheaders/Step1TypedQueryParameterExample.scala</summary>

```scala title="http-model-examples/src/main/scala/validatequeryparametersandheaders/Step1TypedQueryParameterExample.scala" showLineNumbers
package validatequeryparametersandheaders

/**
 * How to Validate Query Parameters and Headers and Answer 400 — One Typed Query
 * Parameter
 *
 * Reads `?limit=` as an `Int`, first as a required parameter and then as an
 * optional one that falls back to a default only when the parameter is missing,
 * never when it is malformed.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * validatequeryparametersandheaders.Step1TypedQueryParameterExample"
 */
object Step1TypedQueryParameterExample {
  def listBooks(request: Request): Response =
    request.query[Int]("limit") match {
      case Right(limit) => Response.json(bookListJson.encodeToString(books.take(limit)))
      case Left(error)  => Response.text(Status.BadRequest, error.message)
    }

  def listBooksWithDefault(request: Request): Response =
    optionalQuery[Int](request, "limit") match {
      case Right(limit) => Response.json(bookListJson.encodeToString(books.take(limit.getOrElse(20))))
      case Left(error)  => Response.text(Status.BadRequest, error.message)
    }

  def main(args: Array[String]): Unit = {
    JdkHttp.withServer(listBooks) { port =>
      Client.call(port, "GET", "/books?limit=1")
      Client.call(port, "GET", "/books")
      Client.call(port, "GET", "/books?limit=abc")
    }

    val request = Request.get(URL.parse("/books?limit=abc").toOption.get)
    println(request.queryOrElse[Int]("limit", 20))

    JdkHttp.withServer(listBooksWithDefault) { port =>
      Client.call(port, "GET", "/books")
      Client.call(port, "GET", "/books?limit=abc")
    }
  }
}
```

</details>

To run it:

```bash
sbt "runMain validatequeryparametersandheaders.Step1TypedQueryParameterExample"
```

The second example reads `limit` and `offset`, checks their ranges and reports every problem in a JSON 400:

<details>
  <summary>http-model-examples/src/main/scala/validatequeryparametersandheaders/Step2BadRequestExample.scala</summary>

```scala title="http-model-examples/src/main/scala/validatequeryparametersandheaders/Step2BadRequestExample.scala" showLineNumbers
package validatequeryparametersandheaders

/**
 * How to Validate Query Parameters and Headers and Answer 400 — A JSON 400 That
 * Names Every Problem
 *
 * Reads `?limit=` and `?offset=`, checks their ranges after decoding, and
 * answers 400 with a JSON body that lists every bad parameter at once.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * validatequeryparametersandheaders.Step2BadRequestExample"
 */
object Step2BadRequestExample {
  final case class Paging(limit: Int, offset: Int)

  def paging(request: Request): Checked[Paging] = {
    val limit = checkedQuery[Int](request, "limit")
      .map(_.getOrElse(20))
      .flatMap(check[Int]("limit", "must be between 1 and 100")(n => n >= 1 && n <= 100))
    val offset = checkedQuery[Int](request, "offset")
      .map(_.getOrElse(0))
      .flatMap(check[Int]("offset", "must not be negative")(_ >= 0))
    both(limit, offset).map { case (limit, offset) => Paging(limit, offset) }
  }

  def listBooks(request: Request): Response =
    paging(request) match {
      case Right(page)    => Response.json(bookListJson.encodeToString(books.slice(page.offset, page.offset + page.limit)))
      case Left(problems) => badRequest(problems)
    }

  def main(args: Array[String]): Unit =
    JdkHttp.withServer(listBooks) { port =>
      Client.call(port, "GET", "/books?limit=1&offset=2")
      Client.call(port, "GET", "/books?limit=abc&offset=xyz")
      Client.call(port, "GET", "/books?limit=500&offset=-1")
    }
}
```

</details>

To run it:

```bash
sbt "runMain validatequeryparametersandheaders.Step2BadRequestExample"
```

The third example decodes the author and year filter with a derived `QueryCodec`:

<details>
  <summary>http-model-examples/src/main/scala/validatequeryparametersandheaders/Step3QueryCodecExample.scala</summary>

```scala title="http-model-examples/src/main/scala/validatequeryparametersandheaders/Step3QueryCodecExample.scala" showLineNumbers
package validatequeryparametersandheaders

/**
 * How to Validate Query Parameters and Headers and Answer 400 — A Whole Filter
 * from a Derived QueryCodec
 *
 * Decodes `?author=` and `?year=` into a `BookFilter` case class with a
 * `QueryCodec` derived from its schema, and shows that the codec does not use a
 * field's default value when the parameter is missing.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * validatequeryparametersandheaders.Step3QueryCodecExample"
 */
object Step3QueryCodecExample {
  final case class BookFilter(author: Option[String], year: Option[Int])

  object BookFilter {
    implicit val schema: Schema[BookFilter] = Schema.derived

    val codec: QueryCodec[BookFilter] = Schema[BookFilter].derive(DefaultQueryFormat)
  }

  def matches(filter: BookFilter)(book: Book): Boolean =
    filter.author.forall(book.authors.contains) && filter.year.forall(_ == book.year)

  def listBooks(request: Request): Response =
    BookFilter.codec.decode(request.queryParams) match {
      case Right(filter) => Response.json(bookListJson.encodeToString(books.filter(matches(filter))))
      case Left(error)   => badRequest(Problem.decoded("filter")(error))
    }

  final case class DefaultedPaging(limit: Int = 20, offset: Int = 0)

  object DefaultedPaging {
    implicit val schema: Schema[DefaultedPaging] = Schema.derived

    val codec: QueryCodec[DefaultedPaging] = Schema[DefaultedPaging].derive(DefaultQueryFormat)
  }

  def main(args: Array[String]): Unit = {
    JdkHttp.withServer(listBooks) { port =>
      Client.call(port, "GET", "/books?year=2014")
      Client.call(port, "GET", "/books?author=Martin%20Odersky&year=2008")
      Client.call(port, "GET", "/books?year=abc")
    }

    println(DefaultedPaging.codec.decode(QueryParams.fromEncoded("limit=5&offset=10")))
    println(DefaultedPaging.codec.decode(QueryParams.fromEncoded("limit=5")))
  }
}
```

</details>

To run it:

```bash
sbt "runMain validatequeryparametersandheaders.Step3QueryCodecExample"
```

The fourth example reads the `X-Request-Id` and `Accept-Language` headers:

<details>
  <summary>http-model-examples/src/main/scala/validatequeryparametersandheaders/Step4HeadersExample.scala</summary>

```scala title="http-model-examples/src/main/scala/validatequeryparametersandheaders/Step4HeadersExample.scala" showLineNumbers
package validatequeryparametersandheaders

/**
 * How to Validate Query Parameters and Headers and Answer 400 — Typed Headers
 *
 * Reads an optional `X-Request-Id` header as a `UUID`, answering 400 when it is
 * malformed, and the `Accept-Language` header with the typed
 * `Header.AcceptLanguage` codec, falling back to English.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * validatequeryparametersandheaders.Step4HeadersExample"
 */
object Step4HeadersExample {
  def requestId(request: Request): Checked[Option[UUID]] =
    request.headers.header[UUID]("X-Request-Id") match {
      case Right(id)                    => Right(Some(id))
      case Left(HeaderError.Missing(_)) => Right(None)
      case Left(error)                  => Left(List(Problem.header(error)))
    }

  def language(request: Request): String =
    request.headers.header(Header.AcceptLanguage) match {
      case Right(accept) => accept.languages.headOption.fold("en")(_.tag)
      case Left(_)       => "en"
    }

  def catalog(request: Request): Response =
    requestId(request) match {
      case Right(id)      => Response.text(s"Catalog in ${language(request)} for request ${id.getOrElse("(none)")}")
      case Left(problems) => badRequest(problems)
    }

  def main(args: Array[String]): Unit = {
    val request = Request
      .get(URL.parse("/books").toOption.get)
      .addHeader("Accept-Language", "de-CH, en;q=0.8")
    println(request.headerOrElse[String]("Accept-Language", "en"))
    println(request.headers.header(Header.AcceptLanguage))

    JdkHttp.withServer(catalog) { port =>
      HeaderClient.get(port, "/books")
      HeaderClient.get(
        port,
        "/books",
        "X-Request-Id"    -> "3f2504e0-4f89-11d3-9a0c-0305e82c3301",
        "Accept-Language" -> "de-CH, en;q=0.8"
      )
      HeaderClient.get(port, "/books", "X-Request-Id" -> "42")
    }
  }
}
```

</details>

To run it:

```bash
sbt "runMain validatequeryparametersandheaders.Step4HeadersExample"
```

The fifth example shows `%20` and `+` in a query value, and the form-data wrapper:

<details>
  <summary>http-model-examples/src/main/scala/validatequeryparametersandheaders/Step5SpacesExample.scala</summary>

```scala title="http-model-examples/src/main/scala/validatequeryparametersandheaders/Step5SpacesExample.scala" showLineNumbers
package validatequeryparametersandheaders

/**
 * How to Validate Query Parameters and Headers and Answer 400 — Values with
 * Spaces and Plus Signs
 *
 * Shows that `%20` decodes to a space but `+` stays a plus sign, and serves the
 * same handler through a wrapper that decodes the raw query string as form
 * data, so that `+` from an HTML form becomes a space.
 *
 * Run with: cd http-model-examples && sbt "runMain
 * validatequeryparametersandheaders.Step5SpacesExample"
 */
object Step5SpacesExample {
  def byAuthor(request: Request): Response =
    request.query[String]("author") match {
      case Right(author) => Response.json(bookListJson.encodeToString(books.filter(_.authors.contains(author))))
      case Left(error)   => badRequest(List(Problem.query(error)))
    }

  def formQueryHandler(app: Request => Response): HttpHandler = exchange => {
    val rawQuery = Option(exchange.getRequestURI.getRawQuery).getOrElse("")
    val params   = Form.fromString(rawQuery).entries.foldLeft(QueryParams.empty) { case (acc, (key, value)) =>
      acc.add(key, value)
    }
    JdkHttp.handler(request => app(request.updateUrl(_.copy(queryParams = params)))).handle(exchange)
  }

  def withFormServer[A](app: Request => Response)(use: Int => A): A = {
    val server = HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress, 0), 0)
    server.createContext("/", formQueryHandler(app))
    server.start()
    try use(server.getAddress.getPort)
    finally server.stop(0)
  }

  def main(args: Array[String]): Unit = {
    JdkHttp.withServer(byAuthor) { port =>
      Client.call(port, "GET", "/books?author=Martin%20Odersky")
      Client.call(port, "GET", "/books?author=Martin+Odersky")
    }

    withFormServer(byAuthor) { port =>
      Client.call(port, "GET", "/books?author=Martin+Odersky")
    }
  }
}
```

</details>

To run it:

```bash
sbt "runMain validatequeryparametersandheaders.Step5SpacesExample"
```

To run the complete example:

```bash
sbt "runMain validatequeryparametersandheaders.CompleteExample"
```

To compile all the examples without running them:

```bash
sbt compile
```

## Going Further

- **One value at a time**: The [schema integration reference](../reference/http-model/schema.md) lists every accessor on `QueryParams`, `Headers`, `Request` and `Response`, the supported types and the exact error messages.
- **Whole values**: The [codecs reference](../reference/http-model/schema-codecs.md) covers `QueryCodec` and `HeaderCodec` in depth, including repeated parameters as `List` fields, wrapper types and how header names are derived from field names.
- **Custom headers**: Implement `Header.Codec[A]` for a header of your own and pass it to `Headers#header`; the [headers reference](../reference/http-model/headers.md) lists the typed headers the HTTP model already has.
- **Error details**: The [`SchemaError` reference](../reference/schema/schema-error.md) describes the error a derived codec returns.
- **Serving**: [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md) explains the adapter, routing and JSON request bodies.

---

## How to Define Validated Newtypes and Collect Every Validation Error

In this guide, we build the input model for a bookstore API that accepts new book submissions as JSON. Each value that has rules (the ISBN, the title, the price and the page count) gets its own small type, a *newtype*, that can only be created by passing its check. We declare each check once, on the type's `Schema`, so every codec derived from that schema enforces it. By the end, you will have a decoder that turns bad input into a list of every problem, each with the field it belongs to, ready to send back as the body of a `400 Bad Request` response.

We take an incremental approach:

- Wrap each value in a newtype with a private constructor and a smart constructor (a function that checks its input before building the value)
- Attach the smart constructor to the type's `Schema` with `Schema#transform`
- Decode JSON and see the first failure with its field path
- Collect every failure in one pass with `Schema#fromDynamicValue`
- Turn a `SchemaError` into a structured error response
- Reuse the same rules for a second format, TOON

## The Problem

A `Book` that stores its ISBN, title and price as `String` and `BigDecimal` accepts any string and any number. An ISBN of `"123"`, a blank title and a negative price all type-check, so bad values travel deep into the program before anything notices.

The usual fix is a `validate` function, and that is where the trouble starts. Nothing forces anyone to call it, so the HTTP handler calls it, the CSV importer forgets to, and a test fixture builds a `Book` directly. The rules end up copied or skipped, and a function that receives a `Book` cannot tell whether it was checked.

For example, a typical model and its validation look like this:

```scala
final case class Book(isbn: String, title: String, price: BigDecimal, pages: Int)

def validate(book: Book): Either[String, Book] =
  if (book.isbn.length != 13) Left(s"bad ISBN: ${book.isbn}")
  else if (book.title.trim.isEmpty) Left("blank title")
  else if (book.price <= 0) Left(s"bad price: ${book.price}")
  else if (book.pages <= 0) Left(s"bad page count: ${book.pages}")
  else Right(book)

// Compiles and runs: nothing stops us from skipping validate.
val unchecked = Book("123", "   ", BigDecimal(-45), 0)
```

The checks stop at the first problem, report no field path, and live apart from the type they protect.

In this guide, we follow the "parse, don't validate" approach instead: we turn raw input into a type that can only hold valid values, so having a `Book` is proof that it was checked.

## Prerequisites

This guide builds on [Your First Schema](./your-first-schema.md): you should know how to derive a `Schema` for a case class and decode JSON with it. Add the schema module, and the TOON module used in one section, to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-schema"      % "0.0.56",
  "dev.zio" %% "zio-blocks-schema-toon" % "0.0.56"
)

// Scala 3.9 and newer
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-schema"      % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema-toon" % "0.0.56"
)
scalacOptions += "-experimental"
```

On Scala 2.13, also add this compiler option. It makes the generated `apply` and `copy` methods of a case class as private as its constructor, which Scala 3 does by default. Without it, `Isbn("123")` compiles on Scala 2.13 and skips the check:

```scala
scalacOptions += "-Xsource-features:case-apply-copy-access"
```

We use these imports throughout:

```scala

```

## Wrap Each Value in a Validated Newtype

A newtype here is a one-field case class with a private constructor. The only public way to build one is its smart constructor, `parse`, which returns `Either[SchemaError, A]`: `Right` with the value when the input passes the check, or `Left` with a `SchemaError` that explains why it failed. `SchemaError.validationFailed` builds that error from a message.

Parsing can also normalize the input. `Isbn.parse` accepts `978-1-61729-065-7` with hyphens, verifies the 13 digits and the check digit, and stores the bare digits. Each companion object then turns the smart constructor into a `Schema` with `Schema#transform`, which takes a function from the underlying type (`to`) and a function back (`from`):

```scala
final case class Isbn private (value: String)

object Isbn {
  def parse(input: String): Either[SchemaError, Isbn] = {
    val digits = input.filterNot(c => c == '-' || c == ' ')
    if (digits.length != 13 || !digits.forall(_.isDigit))
      Left(SchemaError.validationFailed(s"ISBN must have 13 digits, got '$input'"))
    else {
      val weighted = digits.init.zipWithIndex.map { case (c, i) => c.asDigit * (if (i % 2 == 0) 1 else 3) }
      if ((10 - weighted.sum % 10) % 10 == digits.last.asDigit) Right(new Isbn(digits))
      else Left(SchemaError.validationFailed(s"ISBN check digit is wrong in '$input'"))
    }
  }

  implicit val schema: Schema[Isbn] =
    Schema[String].transform(s => parse(s).fold(e => throw e, identity), _.value)
}

final case class Title private (value: String)

object Title {
  def parse(input: String): Either[SchemaError, Title] = {
    val trimmed = input.trim
    if (trimmed.isEmpty) Left(SchemaError.validationFailed("title must not be blank"))
    else if (trimmed.length > 200)
      Left(SchemaError.validationFailed(s"title must be at most 200 characters, got ${trimmed.length}"))
    else Right(new Title(trimmed))
  }

  implicit val schema: Schema[Title] =
    Schema[String].transform(s => parse(s).fold(e => throw e, identity), _.value)
}

final case class Price private (amount: BigDecimal)

object Price {
  def parse(amount: BigDecimal): Either[SchemaError, Price] =
    if (amount > 0) Right(new Price(amount))
    else Left(SchemaError.validationFailed(s"price must be positive, got $amount"))

  implicit val schema: Schema[Price] =
    Schema[BigDecimal].transform(a => parse(a).fold(e => throw e, identity), _.amount)
}

final case class PageCount private (value: Int)

object PageCount {
  def parse(value: Int): Either[SchemaError, PageCount] =
    if (value > 0) Right(new PageCount(value))
    else Left(SchemaError.validationFailed(s"page count must be positive, got $value"))

  implicit val schema: Schema[PageCount] =
    Schema[Int].transform(n => parse(n).fold(e => throw e, identity), _.value)
}

final case class Book(isbn: Isbn, title: Title, price: Price, pages: PageCount)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}
```

The `to` function of `Schema#transform` returns a plain `B`, not an `Either`, so a failed check has to throw. `SchemaError` is an exception, so `parse(s).fold(e => throw e, identity)` rethrows the `Left` unchanged, and the decoder catches it and returns it as a `Left` with the field path added. The `parse` method stays the single place where the rule lives; the schema only adapts it.

Calling the smart constructor directly shows both outcomes, including the hyphenated ISBN of *Programming in Scala*, Fifth Edition, and the same ISBN with a wrong check digit:

```scala
Isbn.parse("978-0-9971480-0-8")
// res1: Either[SchemaError, Isbn] = Right(Isbn("9780997148008"))
Isbn.parse("978-0-9971480-0-9").left.map(_.message)
// res2: Either[String, Isbn] = Left(
//   "ISBN check digit is wrong in '978-0-9971480-0-9'"
// )
Title.parse("   ").left.map(_.message)
// res3: Either[String, Title] = Left("title must not be blank")
```

`Schema.derived` for `Book` picks up the four implicit schemas from the companion objects. A newtype's schema is a *wrapper* around its underlying type, so JSON shows the bare string or number, not a nested object:

```scala
val fpInScala: Book =
  Book.schema.jsonCodec
    .decode("""{"isbn":"978-1-61729-065-7","title":"Functional Programming in Scala","price":44.99,"pages":320}""")
    .fold(e => throw e, identity)
```

Encoding the decoded book writes the ISBN without hyphens, because `Isbn.parse` stored the normalized digits:

```scala
println(Book.schema.jsonCodec.encodeToString(fpInScala))
// {"isbn":"9781617290657","title":"Functional Programming in Scala","price":44.99,"pages":320}
```

## Reject Bad Input When Decoding JSON

Decoding goes through the same smart constructors, because the derived JSON codec calls each wrapper's `to` function. A valid payload decodes to a `Book`, and an invalid one returns `Left(SchemaError)` instead of throwing:

```scala
println(Book.schema.jsonCodec.decode("""{"isbn":"9781617290657","title":"Functional Programming in Scala","price":44.99,"pages":320}"""))
// Right(Book(Isbn(9781617290657),Title(Functional Programming in Scala),Price(44.99),PageCount(320)))

Book.schema.jsonCodec
  .decode("""{"isbn":"9781617290657","title":"   ","price":44.99,"pages":320}""")
  .left
  .foreach(e => println(e.message))
// title must not be blank at: .title.wrapped
```

The message ends with the path of the field that failed: `.title` for the field, followed by `.wrapped`, the step from a newtype to the value inside it.

The JSON codec stops at the first failure. A *Zionomicon* submission with three bad values reports only the ISBN:

```scala
val zionomiconJson = """{"isbn":"978-ZIO","title":"Zionomicon","price":-45,"pages":0}"""
```

The price and the page count are never checked:

```scala
Book.schema.jsonCodec.decode(zionomiconJson).left.foreach(e => println(e.message))
// ISBN must have 13 digits, got '978-ZIO' at: .isbn.wrapped
```

Failing fast is the right choice inside a service, where any failure means the message is rejected. A client filling in a form needs every problem at once, which the next section provides.

## Collect Every Error in One Pass

`Schema#fromDynamicValue` builds a typed value from a `DynamicValue`, the library's schema-less representation of data. Unlike the JSON codec, it keeps going after a failure and collects every error into one `SchemaError`. To use it on JSON, we parse the text with `Json.parse`, which checks only the JSON syntax, convert the result with `Json#toDynamicValue`, and hand that to the validated schema:

```scala
def decodeAll(json: String): Either[SchemaError, Book] =
  Json.parse(json).flatMap(parsed => Book.schema.fromDynamicValue(parsed.toDynamicValue))
```

The *Zionomicon* submission now reports all three problems:

```scala
decodeAll(zionomiconJson).left.foreach(e => println(e.message))
// ISBN must have 13 digits, got '978-ZIO' at: .isbn.wrapped
// price must be positive, got -45 at: .price.wrapped
// page count must be positive, got 0 at: .pages.wrapped
```

Structural errors are collected together with the newtype checks. A payload with a wrong type for `pages` and no `price` at all reports both, next to the ISBN failure:

```scala
decodeAll("""{"isbn":"978-ZIO","title":"Zionomicon","pages":"many"}""").left.foreach(e => println(e.message))
// ISBN must have 13 digits, got '978-ZIO' at: .isbn.wrapped
// Expected Int at: .pages
// Missing field 'price' at: .
```

A valid payload decodes to the same `Book` as before, and JSON numbers such as `44.99` or `45` become a `BigDecimal` price either way:

```scala
println(decodeAll("""{"isbn":"978-1-61729-065-7","title":"Functional Programming in Scala","price":45,"pages":320}"""))
// Right(Book(Isbn(9781617290657),Title(Functional Programming in Scala),Price(45),PageCount(320)))
```

:::warning
On this route, a field that the JSON leaves out is always an error, even where the JSON codec fills it in. A missing `List` field (which the codec decodes as an empty list), a missing `Option` field (`None`) and a missing field with a default value all fail with `Missing field`. This is a known difference in 0.0.56. If your payloads may omit such fields, send them explicitly or decode with the JSON codec instead.
:::

The same approach works for input that is already a Scala value, such as a row type from a database or a request type with plain fields. Convert it to a `DynamicValue` with its own schema's `Schema#toDynamicValue`, then rebuild it with the validated schema; the field names have to match:

```scala
final case class BookSubmission(isbn: String, title: String, price: BigDecimal, pages: Int)

object BookSubmission {
  implicit val schema: Schema[BookSubmission] = Schema.derived
}

val submission = BookSubmission("978-ZIO", "Zionomicon", BigDecimal(-45), 0)
// submission: BookSubmission = BookSubmission(
//   isbn = "978-ZIO",
//   title = "Zionomicon",
//   price = -45,
//   pages = 0
// )

Book.schema
  .fromDynamicValue(BookSubmission.schema.toDynamicValue(submission))
  .left
  .foreach(e => println(e.message))
// ISBN must have 13 digits, got '978-ZIO' at: .isbn.wrapped
// price must be positive, got -45 at: .price.wrapped
// page count must be positive, got 0 at: .pages.wrapped
```

:::tip
`Into.derived[BookSubmission, Book]` also collects every field error, given an implicit `Into[String, Isbn]` (and so on) built from each `parse` method. Its errors carry the root path `.`, though, with the field name only inside the message text (`converting field BookSubmission.isbn to Book.isbn failed`), so `Schema#fromDynamicValue` is the better choice when you report errors per field.
:::

## Turn the Errors into a 400 Response

A `SchemaError` holds a non-empty list of `SchemaError.Single` values in `SchemaError#errors`. Each one has a `source`, a `DynamicOptic` that describes the path to the failing value, and a `message` that already ends with that path. An API response reads better as a list of objects with the field and the problem kept apart, so we build one with two small helpers.

The field name comes from the `DynamicOptic.Node.Field` steps of the path, which skips the `.wrapped` step. A `SchemaError.MissingField` points at the record that lacks the field, so we add its `fieldName`. The text without the path comes from the specific error type:

```scala
final case class FieldError(field: String, message: String)

object FieldError {
  implicit val schema: Schema[FieldError] = Schema.derived
}

final case class ErrorResponse(status: Int, errors: List[FieldError])

object ErrorResponse {
  implicit val schema: Schema[ErrorResponse] = Schema.derived
}

def fieldOf(error: SchemaError.Single): String = {
  val path = error.source.nodes.collect { case DynamicOptic.Node.Field(name) => name }
  error match {
    case missing: SchemaError.MissingField => (path :+ missing.fieldName).mkString(".")
    case _                                 => path.mkString(".")
  }
}

def detailOf(error: SchemaError.Single): String = error match {
  case e: SchemaError.ConversionFailed    => e.details
  case _: SchemaError.MissingField        => "is required"
  case e: SchemaError.ExpectationMismatch => e.expectation
  case e: SchemaError.Message             => e.details
  case other                              => other.message
}

def toResponse(error: SchemaError): ErrorResponse =
  ErrorResponse(400, error.errors.map(e => FieldError(fieldOf(e), detailOf(e))))
```

The response type has its own derived schema, so the body is encoded with the same JSON codec as everything else:

```scala
decodeAll("""{"isbn":"978-ZIO","title":"Zionomicon","pages":"many"}""").left.foreach { e =>
  println(ErrorResponse.schema.jsonCodec.encodeToString(toResponse(e)))
}
// {"status":400,"errors":[{"field":"isbn","message":"ISBN must have 13 digits, got '978-ZIO'"},{"field":"pages","message":"Expected Int"},{"field":"price","message":"is required"}]}
```

An HTTP handler sends this body with status 400 when `decodeAll` returns `Left`, and continues with a fully checked `Book` when it returns `Right`. The upcoming guide on validating query parameters and headers applies the same idea to the rest of an HTTP request.

## Enforce the Same Rules in Another Format

The checks live in the `Schema`, not in the JSON codec, so any codec derived from `Book.schema` runs them. Deriving a TOON codec with `Schema#derive` and decoding the *Zionomicon* submission rejects the same ISBN, with the same message and path:

```scala

val toonCodec = Book.schema.derive(ToonFormat)
```

The good book round-trips through TOON, and the bad one fails:

```scala
println(toonCodec.encodeToString(fpInScala))
// isbn: "9781617290657"
// title: Functional Programming in Scala
// price: 44.99
// pages: 320

toonCodec
  .decode("isbn: \"978-ZIO\"\ntitle: Zionomicon\nprice: -45\npages: 0")
  .left
  .foreach(e => println(e.message))
// ISBN must have 13 digits, got '978-ZIO' at: .isbn.wrapped
```

Like the JSON codec, the TOON codec stops at the first failure. For every error, decode TOON into `BookSubmission` and rebuild it with `Schema#fromDynamicValue`, as in the previous section.

:::warning
The CSV codec (`zio-blocks-schema-csv`) also runs the newtype checks, but in this release `CsvCodec#decode` throws the `SchemaError` when a check fails instead of returning `Left`. Wrap CSV decoding of validated types in `scala.util.Try` until this is fixed.
:::

## Where the Rules Do Not Appear

The rules live in Scala code, so they don't appear in the JSON Schema contract derived from `Book.schema`. Each newtype shows up as its underlying type:

```scala
println(Book.schema.toJsonSchema)
// {
//   "properties": {
//     "isbn": {
//       "type": "string"
//     },
//     "title": {
//       "type": "string"
//     },
//     "price": {
//       "type": "number"
//     },
//     "pages": {
//       "type": "integer",
//       "maximum": 2147483647,
//       "minimum": -2147483648
//     }
//   },
//   "type": "object",
//   "required": [
//     "isbn",
//     "title",
//     "price",
//     "pages"
//   ],
//   "title": "Book"
// }
```

If other teams need the rules in the contract, add them to a hand-written JSON Schema and load it with `JsonSchema.parse`, as described in [How to Generate JSON Schema from a Type and Validate Payloads Against It](./generate-json-schema-and-validate-payloads.md).

The schema module also has built-in `Validation` constraints, such as `Validation.Numeric.Positive` and `Validation.String.NonEmpty`, attached to primitive types and described in [Validation](../reference/schema/validation.md). In this release only `DynamicSchema#check` enforces them: the derived codecs and `Schema#fromDynamicValue` ignore them, and `Schema#toJsonSchema` does not emit them. Smart constructors are the way to enforce a rule.

## Use Scala 3 Opaque Types

On Scala 3, an opaque type alias is a newtype with no wrapper object at run time: inside its defining scope `Isbn` is a `String`, and outside it is a distinct type. The schema works the same way. Put the smart constructor and a `Schema` built with `Schema#transform` in the companion object, where the compiler finds it for `Schema.derived`, and define `Book` outside the object that declares the opaque type:

```scala
// Scala 3 only
object catalog {
  opaque type Isbn = String

  object Isbn {
    def parse(input: String): Either[SchemaError, Isbn] = {
      val digits = input.filterNot(c => c == '-' || c == ' ')
      if (digits.length == 13 && digits.forall(_.isDigit)) Right(digits)
      else Left(SchemaError.validationFailed(s"ISBN must have 13 digits, got '$input'"))
    }

    given Schema[Isbn] = Schema.string.transform(s => parse(s).fold(e => throw e, identity), identity)

    extension (isbn: Isbn) def value: String = isbn
  }
}

final case class Book(isbn: catalog.Isbn, title: String) derives Schema

println(Schema[Book].jsonCodec.decode("""{"isbn":"978-1-61729-065-7","title":"Functional Programming in Scala"}""").map(_.isbn.value))
// Right(9781617290657)

Schema[Book].jsonCodec
  .decode("""{"isbn":"978-ZIO","title":"Zionomicon"}""")
  .left
  .foreach(e => println(e.message))
// ISBN must have 13 digits, got '978-ZIO' at: .isbn.wrapped

Json
  .parse("""{"isbn":"978-ZIO","title":"Zionomicon"}""")
  .flatMap(json => Schema[Book].fromDynamicValue(json.toDynamicValue))
  .left
  .foreach(e => println(e.message))
// ISBN must have 13 digits, got '978-ZIO' at: .isbn.wrapped
```

Where `Book` is defined matters. Inside `object catalog`, `Isbn` is the same type as `String`, so a `Book` derived there uses the plain `String` schema and accepts any ISBN without calling `parse`. For the same reason, an opaque type needs its own `given Schema`: without one, `Schema.derived` sees through the alias and accepts any string. The given builds on `Schema.string`, not `Schema[String]`: inside `object catalog`, `Schema[String]` can find this given itself, and on Scala 3.3 the program then hangs the first time it uses the schema.

## Putting It Together

The complete example defines the four newtypes and `Book`, decodes a batch of three submissions (*Functional Programming in Scala*, *Zionomicon* and *Programming in Scala*), collects every error, and prints either the decoded book or the `400` response body for each:

```scala title="schema-examples/src/main/scala/validatednewtypes/CompleteExample.scala"
package validatednewtypes

/**
 * Validated Newtypes: Putting It Together, Complete Example
 *
 * Defines the four newtypes and Book, decodes a batch of three submissions with
 * every error collected, and prints either the decoded book or the 400 response
 * body for each.
 *
 * Run with: cd schema-examples && sbt "runMain
 * validatednewtypes.CompleteExample"
 */
object CompleteExample extends App {

  final case class Isbn private (value: String)

  object Isbn {
    def parse(input: String): Either[SchemaError, Isbn] = {
      val digits = input.filterNot(c => c == '-' || c == ' ')
      if (digits.length != 13 || !digits.forall(_.isDigit))
        Left(SchemaError.validationFailed(s"ISBN must have 13 digits, got '$input'"))
      else {
        val weighted = digits.init.zipWithIndex.map { case (c, i) => c.asDigit * (if (i % 2 == 0) 1 else 3) }
        if ((10 - weighted.sum % 10) % 10 == digits.last.asDigit) Right(new Isbn(digits))
        else Left(SchemaError.validationFailed(s"ISBN check digit is wrong in '$input'"))
      }
    }

    implicit val schema: Schema[Isbn] =
      Schema[String].transform(s => parse(s).fold(e => throw e, identity), _.value)
  }

  final case class Title private (value: String)

  object Title {
    def parse(input: String): Either[SchemaError, Title] = {
      val trimmed = input.trim
      if (trimmed.isEmpty) Left(SchemaError.validationFailed("title must not be blank"))
      else if (trimmed.length > 200)
        Left(SchemaError.validationFailed(s"title must be at most 200 characters, got ${trimmed.length}"))
      else Right(new Title(trimmed))
    }

    implicit val schema: Schema[Title] =
      Schema[String].transform(s => parse(s).fold(e => throw e, identity), _.value)
  }

  final case class Price private (amount: BigDecimal)

  object Price {
    def parse(amount: BigDecimal): Either[SchemaError, Price] =
      if (amount > 0) Right(new Price(amount))
      else Left(SchemaError.validationFailed(s"price must be positive, got $amount"))

    implicit val schema: Schema[Price] =
      Schema[BigDecimal].transform(a => parse(a).fold(e => throw e, identity), _.amount)
  }

  final case class PageCount private (value: Int)

  object PageCount {
    def parse(value: Int): Either[SchemaError, PageCount] =
      if (value > 0) Right(new PageCount(value))
      else Left(SchemaError.validationFailed(s"page count must be positive, got $value"))

    implicit val schema: Schema[PageCount] =
      Schema[Int].transform(n => parse(n).fold(e => throw e, identity), _.value)
  }

  final case class Book(isbn: Isbn, title: Title, price: Price, pages: PageCount)

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  def decodeAll(json: String): Either[SchemaError, Book] =
    Json.parse(json).flatMap(parsed => Book.schema.fromDynamicValue(parsed.toDynamicValue))

  final case class FieldError(field: String, message: String)

  object FieldError {
    implicit val schema: Schema[FieldError] = Schema.derived
  }

  final case class ErrorResponse(status: Int, errors: List[FieldError])

  object ErrorResponse {
    implicit val schema: Schema[ErrorResponse] = Schema.derived
  }

  def fieldOf(error: SchemaError.Single): String = {
    val path = error.source.nodes.collect { case DynamicOptic.Node.Field(name) => name }
    error match {
      case missing: SchemaError.MissingField => (path :+ missing.fieldName).mkString(".")
      case _                                 => path.mkString(".")
    }
  }

  def detailOf(error: SchemaError.Single): String = error match {
    case e: SchemaError.ConversionFailed    => e.details
    case _: SchemaError.MissingField        => "is required"
    case e: SchemaError.ExpectationMismatch => e.expectation
    case e: SchemaError.Message             => e.details
    case other                              => other.message
  }

  def toResponse(error: SchemaError): ErrorResponse =
    ErrorResponse(400, error.errors.map(e => FieldError(fieldOf(e), detailOf(e))))

  val submissions: List[String] = List(
    """{"isbn":"978-1-61729-065-7","title":"Functional Programming in Scala","price":44.99,"pages":320}""",
    """{"isbn":"978-ZIO","title":"Zionomicon","price":-45,"pages":0}""",
    """{"isbn":"978-0-9971480-0-9","title":"Programming in Scala","price":49.95,"pages":0}"""
  )

  submissions.foreach { json =>
    decodeAll(json) match {
      case Right(book) => println(s"Accepted: $book")
      case Left(error) => println(ErrorResponse.schema.jsonCodec.encodeToString(toResponse(error)))
    }
  }
}
```

## Running the Examples

All examples in this guide have corresponding runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

The step examples share the newtypes in `BookModel.scala`:

<details>
  <summary>schema-examples/src/main/scala/validatednewtypes/BookModel.scala</summary>

```scala title="schema-examples/src/main/scala/validatednewtypes/BookModel.scala" showLineNumbers
package validatednewtypes

/**
 * Validated Newtypes: shared model
 *
 * Isbn, Title, Price and PageCount are newtypes with private constructors and
 * smart constructors. Each companion turns `parse` into a Schema with
 * `Schema#transform`. Book derives its Schema from the four newtype schemas.
 * This file has no main method; the other examples in this package use it.
 */
final case class Isbn private (value: String)

object Isbn {
  def parse(input: String): Either[SchemaError, Isbn] = {
    val digits = input.filterNot(c => c == '-' || c == ' ')
    if (digits.length != 13 || !digits.forall(_.isDigit))
      Left(SchemaError.validationFailed(s"ISBN must have 13 digits, got '$input'"))
    else {
      val weighted = digits.init.zipWithIndex.map { case (c, i) => c.asDigit * (if (i % 2 == 0) 1 else 3) }
      if ((10 - weighted.sum % 10) % 10 == digits.last.asDigit) Right(new Isbn(digits))
      else Left(SchemaError.validationFailed(s"ISBN check digit is wrong in '$input'"))
    }
  }

  implicit val schema: Schema[Isbn] =
    Schema[String].transform(s => parse(s).fold(e => throw e, identity), _.value)
}

final case class Title private (value: String)

object Title {
  def parse(input: String): Either[SchemaError, Title] = {
    val trimmed = input.trim
    if (trimmed.isEmpty) Left(SchemaError.validationFailed("title must not be blank"))
    else if (trimmed.length > 200)
      Left(SchemaError.validationFailed(s"title must be at most 200 characters, got ${trimmed.length}"))
    else Right(new Title(trimmed))
  }

  implicit val schema: Schema[Title] =
    Schema[String].transform(s => parse(s).fold(e => throw e, identity), _.value)
}

final case class Price private (amount: BigDecimal)

object Price {
  def parse(amount: BigDecimal): Either[SchemaError, Price] =
    if (amount > 0) Right(new Price(amount))
    else Left(SchemaError.validationFailed(s"price must be positive, got $amount"))

  implicit val schema: Schema[Price] =
    Schema[BigDecimal].transform(a => parse(a).fold(e => throw e, identity), _.amount)
}

final case class PageCount private (value: Int)

object PageCount {
  def parse(value: Int): Either[SchemaError, PageCount] =
    if (value > 0) Right(new PageCount(value))
    else Left(SchemaError.validationFailed(s"page count must be positive, got $value"))

  implicit val schema: Schema[PageCount] =
    Schema[Int].transform(n => parse(n).fold(e => throw e, identity), _.value)
}

final case class Book(isbn: Isbn, title: Title, price: Price, pages: PageCount)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}
```

</details>

Run them in order to follow the guide step by step.

### Define the Newtypes

This example calls each smart constructor with good and bad input, then decodes and encodes *Functional Programming in Scala* to show that the newtypes appear in JSON as bare values.

<details>
  <summary>schema-examples/src/main/scala/validatednewtypes/DefineNewtypesExample.scala</summary>

```scala title="schema-examples/src/main/scala/validatednewtypes/DefineNewtypesExample.scala" showLineNumbers
package validatednewtypes

/**
 * Validated Newtypes: Define the Newtypes
 *
 * Calls each smart constructor with good and bad input, then decodes and
 * encodes Functional Programming in Scala to show that the newtypes appear in
 * JSON as bare values.
 *
 * Run with: cd schema-examples && sbt "runMain
 * validatednewtypes.DefineNewtypesExample"
 */
object DefineNewtypesExample extends App {

  println(Isbn.parse("978-0-9971480-0-8"))
  println(Isbn.parse("978-0-9971480-0-9").left.map(_.message))
  println(Title.parse("   ").left.map(_.message))

  val fpInScala: Book =
    Book.schema.jsonCodec
      .decode("""{"isbn":"978-1-61729-065-7","title":"Functional Programming in Scala","price":44.99,"pages":320}""")
      .fold(e => throw e, identity)

  println(fpInScala)
  println(Book.schema.jsonCodec.encodeToString(fpInScala))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain validatednewtypes.DefineNewtypesExample"
```

### Reject Bad JSON

This example decodes a valid payload, a payload with a blank title, and the *Zionomicon* payload with three bad values, which reports only the first.

<details>
  <summary>schema-examples/src/main/scala/validatednewtypes/RejectBadJsonExample.scala</summary>

```scala title="schema-examples/src/main/scala/validatednewtypes/RejectBadJsonExample.scala" showLineNumbers
package validatednewtypes

/**
 * Validated Newtypes: Reject Bad JSON
 *
 * Decodes a valid payload, a payload with a blank title, and the Zionomicon
 * payload with three bad values. The JSON codec stops at the first failure.
 *
 * Run with: cd schema-examples && sbt "runMain
 * validatednewtypes.RejectBadJsonExample"
 */
object RejectBadJsonExample extends App {

  println(
    Book.schema.jsonCodec.decode(
      """{"isbn":"9781617290657","title":"Functional Programming in Scala","price":44.99,"pages":320}"""
    )
  )

  Book.schema.jsonCodec
    .decode("""{"isbn":"9781617290657","title":"   ","price":44.99,"pages":320}""")
    .left
    .foreach(e => println(e.message))

  val zionomiconJson = """{"isbn":"978-ZIO","title":"Zionomicon","price":-45,"pages":0}"""

  Book.schema.jsonCodec.decode(zionomiconJson).left.foreach(e => println(e.message))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain validatednewtypes.RejectBadJsonExample"
```

### Collect Every Error

This example decodes JSON through `Json#toDynamicValue` and `Schema#fromDynamicValue` to collect every error, including a wrong type and a missing field, and validates a `BookSubmission` value the same way.

<details>
  <summary>schema-examples/src/main/scala/validatednewtypes/CollectAllErrorsExample.scala</summary>

```scala title="schema-examples/src/main/scala/validatednewtypes/CollectAllErrorsExample.scala" showLineNumbers
package validatednewtypes

/**
 * Validated Newtypes: Collect Every Error
 *
 * Decodes JSON through Json#toDynamicValue and Schema#fromDynamicValue, which
 * collects every error, including a wrong type and a missing field. A
 * BookSubmission value is validated the same way.
 *
 * Run with: cd schema-examples && sbt "runMain
 * validatednewtypes.CollectAllErrorsExample"
 */
object CollectAllErrorsExample extends App {

  def decodeAll(json: String): Either[SchemaError, Book] =
    Json.parse(json).flatMap(parsed => Book.schema.fromDynamicValue(parsed.toDynamicValue))

  val zionomiconJson = """{"isbn":"978-ZIO","title":"Zionomicon","price":-45,"pages":0}"""

  decodeAll(zionomiconJson).left.foreach(e => println(e.message))

  decodeAll("""{"isbn":"978-ZIO","title":"Zionomicon","pages":"many"}""").left.foreach(e => println(e.message))

  println(
    decodeAll("""{"isbn":"978-1-61729-065-7","title":"Functional Programming in Scala","price":45,"pages":320}""")
  )

  final case class BookSubmission(isbn: String, title: String, price: BigDecimal, pages: Int)

  object BookSubmission {
    implicit val schema: Schema[BookSubmission] = Schema.derived
  }

  val submission = BookSubmission("978-ZIO", "Zionomicon", BigDecimal(-45), 0)

  Book.schema
    .fromDynamicValue(BookSubmission.schema.toDynamicValue(submission))
    .left
    .foreach(e => println(e.message))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain validatednewtypes.CollectAllErrorsExample"
```

### Build the Error Response

This example turns the collected `SchemaError` into an `ErrorResponse` with one `FieldError` per problem and prints it as JSON.

<details>
  <summary>schema-examples/src/main/scala/validatednewtypes/ErrorResponseExample.scala</summary>

```scala title="schema-examples/src/main/scala/validatednewtypes/ErrorResponseExample.scala" showLineNumbers
package validatednewtypes

/**
 * Validated Newtypes: Build the Error Response
 *
 * Turns the collected SchemaError into an ErrorResponse with one FieldError per
 * problem and prints it as JSON.
 *
 * Run with: cd schema-examples && sbt "runMain
 * validatednewtypes.ErrorResponseExample"
 */
object ErrorResponseExample extends App {

  def decodeAll(json: String): Either[SchemaError, Book] =
    Json.parse(json).flatMap(parsed => Book.schema.fromDynamicValue(parsed.toDynamicValue))

  final case class FieldError(field: String, message: String)

  object FieldError {
    implicit val schema: Schema[FieldError] = Schema.derived
  }

  final case class ErrorResponse(status: Int, errors: List[FieldError])

  object ErrorResponse {
    implicit val schema: Schema[ErrorResponse] = Schema.derived
  }

  def fieldOf(error: SchemaError.Single): String = {
    val path = error.source.nodes.collect { case DynamicOptic.Node.Field(name) => name }
    error match {
      case missing: SchemaError.MissingField => (path :+ missing.fieldName).mkString(".")
      case _                                 => path.mkString(".")
    }
  }

  def detailOf(error: SchemaError.Single): String = error match {
    case e: SchemaError.ConversionFailed    => e.details
    case _: SchemaError.MissingField        => "is required"
    case e: SchemaError.ExpectationMismatch => e.expectation
    case e: SchemaError.Message             => e.details
    case other                              => other.message
  }

  def toResponse(error: SchemaError): ErrorResponse =
    ErrorResponse(400, error.errors.map(e => FieldError(fieldOf(e), detailOf(e))))

  decodeAll("""{"isbn":"978-ZIO","title":"Zionomicon","pages":"many"}""").left.foreach { e =>
    println(ErrorResponse.schema.jsonCodec.encodeToString(toResponse(e)))
  }
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain validatednewtypes.ErrorResponseExample"
```

### Decode TOON with the Same Rules

This example derives a TOON codec from `Book.schema`, round-trips a valid book, and rejects the *Zionomicon* submission.

<details>
  <summary>schema-examples/src/main/scala/validatednewtypes/ToonFormatExample.scala</summary>

```scala title="schema-examples/src/main/scala/validatednewtypes/ToonFormatExample.scala" showLineNumbers
package validatednewtypes

/**
 * Validated Newtypes: Decode TOON with the Same Rules
 *
 * Derives a TOON codec from Book.schema, round-trips a valid book and rejects
 * the Zionomicon submission with the same message and path as JSON.
 *
 * Run with: cd schema-examples && sbt "runMain
 * validatednewtypes.ToonFormatExample"
 */
object ToonFormatExample extends App {

  val fpInScala: Book =
    Book.schema.jsonCodec
      .decode("""{"isbn":"978-1-61729-065-7","title":"Functional Programming in Scala","price":44.99,"pages":320}""")
      .fold(e => throw e, identity)

  val toonCodec = Book.schema.derive(ToonFormat)

  println(toonCodec.encodeToString(fpInScala))

  toonCodec
    .decode("isbn: \"978-ZIO\"\ntitle: Zionomicon\nprice: -45\npages: 0")
    .left
    .foreach(e => println(e.message))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain validatednewtypes.ToonFormatExample"
```

### Opaque Types on Scala 3

This example defines `Isbn` as a Scala 3 opaque type with a `given Schema` and decodes a good and a bad payload. It compiles on Scala 3 only.

<details>
  <summary>schema-examples/src/main/scala/validatednewtypes/OpaqueTypesExample.scala</summary>

```scala title="schema-examples/src/main/scala/validatednewtypes/OpaqueTypesExample.scala" showLineNumbers
package validatednewtypes

/**
 * Validated Newtypes: Opaque Types (Scala 3 only)
 *
 * Defines Isbn as a Scala 3 opaque type with a `given Schema` built with
 * Schema#transform, then decodes a good and a bad payload. This file compiles
 * on Scala 3 only.
 *
 * Run with: cd schema-examples && sbt "runMain
 * validatednewtypes.OpaqueTypesExample"
 */
object OpaqueTypesExample extends App {

  object catalog {
    opaque type Isbn = String

    object Isbn {
      def parse(input: String): Either[SchemaError, Isbn] = {
        val digits = input.filterNot(c => c == '-' || c == ' ')
        if (digits.length == 13 && digits.forall(_.isDigit)) Right(digits)
        else Left(SchemaError.validationFailed(s"ISBN must have 13 digits, got '$input'"))
      }

      given Schema[Isbn] = Schema.string.transform(s => parse(s).fold(e => throw e, identity), identity)

      extension (isbn: Isbn) def value: String = isbn
    }
  }

  final case class Book(isbn: catalog.Isbn, title: String) derives Schema

  val good = Schema[Book].jsonCodec.decode("""{"isbn":"978-1-61729-065-7","title":"Functional Programming in Scala"}""")
  println(good.map(b => (b.isbn.value, b.title)))

  val decoded = Schema[Book].jsonCodec.decode("""{"isbn":"978-ZIO","title":"Zionomicon"}""")
  decoded.left.foreach(e => println(e.message))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain validatednewtypes.OpaqueTypesExample"
```

### The Complete Example

This example is the program from [Putting It Together](#putting-it-together).

Run it from the `schema-examples` directory:

```bash
sbt "runMain validatednewtypes.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## Going Further

- [SchemaError](../reference/schema/schema-error.md) covers every error case, how paths are built, and how to combine errors with `SchemaError#++`.
- [DynamicValue](../reference/schema/dynamic-value.md) explains the schema-less representation that `Schema#fromDynamicValue` reads.
- [How to Override the Codec for One Type](./override-codec-for-one-type.md) shows how to change how a type is encoded when a wrapper is not enough.
- [How to Customize JSON Encoding](./customize-json-encoding.md) covers field names, optional fields and strict decoding for the types around your newtypes.
- [How to Use Iron Refined Types and Typed IDs with Schema](./use-iron-refined-types-and-typed-ids.md) writes the same kind of rules as Iron constraint types on Scala 3, and gives every entity's ID its own type with one opaque `Id[+A]`.

---

## How to Wrap Callback and Future-Based APIs in Async

In this guide, we wrap the clients of a small bookstore in `Async`. One client returns a Scala `Future`, one returns a Java `CompletableFuture`, and one reports through a callback and hands back a cancel handle. By the end, you will have a quote pipeline that combines all three in direct style, hands its result back to code that still expects a `Future`, and withdraws the in-flight warehouse request when the `Async` is cancelled.

We take one API shape at a time: a `Future` in and out, then a Java `CompletionStage`, then a JavaScript `Promise`, then callbacks, first with errors and then with cancellation. The guide ends with a table that maps each API shape to its bridge.

## The Problem

Most of the asynchronous code you depend on was written before `Async` existed. Each library picks its own shape: Scala `Future`, Java `CompletionStage`, JavaScript `Promise`, or a plain callback. Gluing these together by hand means converting every result to one shape, and the conversion code is where errors and cancel handles get lost.

For example, here is a stock check that combines a `Future`-based price lookup with a callback-based inventory call, written with nothing but the standard library:

```scala

trait Request { def cancel(): Unit }

def priceAndStock(
  lookupPrice: String => Future[BigDecimal],
  stock: (String, Either[Throwable, Int] => Unit) => Request,
  title: String
)(implicit ec: ExecutionContext): Future[(BigDecimal, Int)] = {
  val stockLevel = Promise[Int]()
  val request    = stock(title, result => stockLevel.complete(result.toTry))
  // `request` goes out of scope here: no caller can ever cancel the call.
  for {
    price <- lookupPrice(title)
    level <- stockLevel.future
  } yield (price, level)
}
```

This works until a caller loses interest in the answer. A `Future` cannot be cancelled, so the `Request` handle has nowhere to go, and the inventory call keeps running after nobody needs it. Every new client adds another hand-written `Promise` adapter.

In this guide, we bring each of these shapes into `Async` with one bridge per shape, and we carry the cancel handle all the way through, so cancelling the `Async` cancels the call.

## Prerequisites

To add the async module to your project, include this dependency in your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %%% "zio-blocks-async" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental"): the same module with the zio-blocks-next- prefix
libraryDependencies += "dev.zio" %%% "zio-blocks-next-async" % "0.0.56"
```

[How to Add ZIO Blocks to Your Build](../getting-started/installation.md) explains the two artifact names and how to choose one per Scala version. Use `%%%` for a cross-built JVM and Scala.js project, and `%%` for a JVM-only one.

This guide assumes you have worked through [Getting Started with Async](./async-getting-started.md), so `Async#map`, `Async#block`, `Async#either`, `Async.async` with `await`, and `Async#start` are familiar. The [Async reference](../reference/async.md) lists every operation with its full signature.

## The Bookstore Clients

We need three clients that know nothing about `Async`. `PriceService` returns a `Future`, `ShippingQuotes` returns a `CompletableFuture` the way a Java SDK would, and `InventoryClient` takes a callback and returns a `Request` that can be cancelled. `InventoryClient` takes its reply delay as a parameter, so we can build a fast warehouse and a slow one. Prices, stock levels and shipping times are fictional sample data.

The clients, together with the imports the rest of the guide uses:

```scala

object PriceService {
  private val prices = Map(
    "Zionomicon"                      -> BigDecimal("49.99"),
    "Programming in Scala"            -> BigDecimal("59.95"),
    "Functional Programming in Scala" -> BigDecimal("44.99")
  )

  def lookup(title: String)(implicit ec: ExecutionContext): Future[BigDecimal] =
    Future(prices.getOrElse(title, throw new NoSuchElementException(s"No price for $title")))
}

object ShippingQuotes {
  def estimateDays(title: String): CompletableFuture[Int] =
    CompletableFuture.supplyAsync { () =>
      if (title.isEmpty) throw new IllegalArgumentException("Title must not be empty")
      if (title.length > 20) 5 else 2
    }
}

final class InventoryClient(replyAfterMillis: Long) {
  private val stockLevels = Map(
    "Zionomicon"                      -> 12,
    "Programming in Scala"            -> 3,
    "Functional Programming in Scala" -> 0
  )
  private val open      = new AtomicInteger(0)
  private val warehouse = Executors.newSingleThreadScheduledExecutor { (r: Runnable) =>
    val t = new Thread(r, "warehouse")
    t.setDaemon(true)
    t
  }

  def openRequests: Int = open.get()

  def stock(title: String)(onDone: Either[Throwable, Int] => Unit): InventoryClient.Request = {
    open.incrementAndGet()
    val reply = warehouse.schedule(
      new Runnable {
        def run(): Unit = {
          open.decrementAndGet()
          onDone(stockLevels.get(title).toRight(new NoSuchElementException(s"Unknown title: $title")))
        }
      },
      replyAfterMillis,
      TimeUnit.MILLISECONDS
    )
    new InventoryClient.Request {
      def cancel(): Unit = if (reply.cancel(false)) open.decrementAndGet()
    }
  }
}

object InventoryClient {
  trait Request {
    def cancel(): Unit
  }
}
```

`InventoryClient#openRequests` counts the questions still waiting for the warehouse. We use it later to check whether a cancelled `Async` left a request behind.

## Bring a Scala `Future` Into `Async`

`Async.fromFuture` turns a `Future[A]` into an `Async[A]`. It needs no `ExecutionContext`: if the future has already completed, the result is a ready value; otherwise the `Async` suspends until the future completes.

To look up a price and use it as an `Async`:

```scala
val price: BigDecimal = Async.fromFuture(PriceService.lookup("Zionomicon")).block
// price: BigDecimal = 49.99
```

A failed `Future` becomes a failed `Async` carrying the same exception, so the usual error operators apply. To observe the failure as data with `Async#either`, and to recover from it with `Async#catchAll`:

```scala
println(Async.fromFuture(PriceService.lookup("Scala with Cats")).either.block)
// Left(java.util.NoSuchElementException: No price for Scala with Cats)

val fallbackPrice: BigDecimal =
  Async
    .fromFuture(PriceService.lookup("Scala with Cats"))
    .catchAll(_ => Async.succeed(BigDecimal(0)))
    .block
// fallbackPrice: BigDecimal = 0
```

## Hand an `Async` Back to Future-Based Code

On the JVM, `Async#toFuture` and `Async#toCompletableFuture` go the other way, for callers that still expect a `Future` or a `CompletableFuture`. Both take an implicit `ExecutionContext` that completes the result; neither blocks a thread while the `Async` is pending.

To hand a formatted price to a Scala caller and a raw price to a Java caller:

```scala
val forScala: Future[String] =
  Async
    .fromFuture(PriceService.lookup("Programming in Scala"))
    .map(p => s"Programming in Scala costs $p")
    .toFuture

val forJava: CompletableFuture[BigDecimal] =
  Async.fromFuture(PriceService.lookup("Functional Programming in Scala")).toCompletableFuture
```

Each caller waits for its result the way it normally would, with `Await.result` for the `Future` and `CompletableFuture#get` for the `CompletableFuture`:

```scala
println(Await.result(forScala, 5.seconds))
// Programming in Scala costs 59.95
println(forJava.get())
// 44.99
```

A failed `Async` produces a failed `Future` (or a `CompletableFuture` completed exceptionally) with the same exception. On Scala.js, `Async#toFuture` works the same way, and `Async#toJsPromise` produces a `js.Promise` without needing an `ExecutionContext`.

## Bring a Java `CompletionStage` Into `Async`

On the JVM, `Async.fromCompletionStage` accepts any `CompletionStage[A]`, which includes every `CompletableFuture[A]`. Java wraps a failure from a stage in a `CompletionException`. `Async.fromCompletionStage` removes that wrapper, so the `Async` fails with the exception the producer threw.

To bring in a shipping estimate, and to see that a failed estimate arrives unwrapped:

```scala
val days: Int = Async.fromCompletionStage(ShippingQuotes.estimateDays("Zionomicon")).block
// days: Int = 2

println(Async.fromCompletionStage(ShippingQuotes.estimateDays("")).either.block)
// Left(java.lang.IllegalArgumentException: Title must not be empty)
```

## Bring a JavaScript `Promise` Into `Async`

On Scala.js, `Async.fromJsPromise` turns a `js.Promise[A]` into an `Async[A]`. The snippet below is for **Scala.js only**: it is not compiled with this page, because the documentation build runs on the JVM.

To bring in a review count from a JavaScript API (`fetchReviewCount` stands for whatever facade you already have):

```scala

def fetchReviewCount(title: String): js.Promise[Int] = ???

val reviews: Async[Int] = Async.fromJsPromise(fetchReviewCount("Zionomicon"))
```

A rejected promise becomes a failed `Async`, and `Async#toJsPromise` hands an `Async` back to JavaScript code.

## Wrap a Callback That Reports Errors

The [callback bridging section of the tutorial](./async-getting-started.md#7-callback-bridging-asyncpromise-and-completer) completes a `Completer` with `Completer#succeed`. Real callbacks also report failures. Call `Completer#fail` on the error branch, and the failure travels through `Async` like any other: `Async#either`, `Async#catchAll`, and a rethrow from `Async#block` all see it.

To wrap `InventoryClient#stock`, whose callback receives an `Either`, the body takes the `Completer` as `c =>` on Scala 2 and `c ?=>` on Scala 3:

<Tabs groupId="scala-version" defaultValue="scala2">
<TabItem value="scala2" label="Scala 2">

```scala
def stockOf(client: InventoryClient, title: String): Async[Int] =
  Async.promise[Int] { c =>
    client.stock(title) {
      case Right(level) => c.succeed(level)
      case Left(error)  => c.fail(error)
    }
    () // the Request handle is dropped here
  }
```

</TabItem>
<TabItem value="scala3" label="Scala 3">

```scala
def stockOf(client: InventoryClient, title: String): Async[Int] =
  Async.promise[Int] { c ?=>
    client.stock(title) {
      case Right(level) => c.succeed(level)
      case Left(error)  => c.fail(error)
    }
    () // the Request handle is dropped here
  }
```

</TabItem>
</Tabs>

To check a title the warehouse knows, and one it does not, we first build a fast warehouse:

```scala
val inventory = new InventoryClient(replyAfterMillis = 10)
```

With a warehouse that replies after 10 milliseconds, one lookup succeeds and the other fails:

```scala
println(stockOf(inventory, "Zionomicon").block)
// 12

println(stockOf(inventory, "Scala with Cats").either.block)
// Left(java.util.NoSuchElementException: Unknown title: Scala with Cats)
```

The `Completer` is one-shot: the first `Completer#succeed` or `Completer#fail` wins, and a callback that fires again is ignored.

## Cancel the Underlying Call When the `Async` Is Cancelled

`stockOf` throws away the `Request` that `InventoryClient#stock` returns, and `Async.promise` has no place to put it. So cancelling the `Async` stops the `Async` from waiting, but the warehouse request stays open.

To see the leftover request, we build a warehouse that takes a minute to reply:

```scala
val slowWarehouse = new InventoryClient(replyAfterMillis = 60000)
```

Starting the lookup and cancelling it straight away leaves the warehouse request open:

```scala
stockOf(slowWarehouse, "Zionomicon").start.cancel()
println(s"Open warehouse requests after cancel: ${slowWarehouse.openRequests}")
// Open warehouse requests after cancel: 1
```

You rarely need to extend `Pollable` directly. It is the low-level extension point for custom suspended computations, and `Async.promise` covers callbacks that cannot be cancelled. A callback with a cancel handle is the case for a custom `Pollable`: when a running `Async` is cancelled, the driver calls `Pollable#cancel` on the pending `Pollable` it is waiting on. The `Completer` that `Async.promise` waits on inherits the default `Pollable#cancel`, which does nothing.

`cancellable` is not part of ZIO Blocks; it is a helper you add to your own code. To build it, keep a `Completer` for the result and forward cancellation to the handle:

```scala
def cancellable[A](register: Completer[A] => (() => Unit)): Async[A] =
  new Pollable[A] {
    private val result     = new Completer[A]
    private val cancelCall = register(result)

    def poll(onComplete: Runnable): Async[A] = {
      val polled = result.poll(onComplete)
      // While pending, answer with this Pollable rather than the Completer, so
      // the driver keeps waiting on us and our cancel() is the one it calls.
      if (polled.asInstanceOf[AnyRef] eq result) this else polled
    }

    override def cancel(): Unit = cancelCall()
  }
```

`register` runs when `cancellable` is called, just as the body of `Async.promise` does. It starts the call, wires the callback to the `Completer`, and returns a function that cancels the call. `Pollable#poll` delegates to `Completer#poll` but answers with `this` while the result is pending. If it answers with the `Completer` instead, the driver waits on the `Completer` from then on, and cancellation reaches the no-op `Completer#cancel` rather than ours.

With the helper in place, a cancellable `stockOf` returns `InventoryClient.Request#cancel` instead of dropping the handle:

```scala
def cancellableStockOf(client: InventoryClient, title: String): Async[Int] =
  cancellable[Int] { c =>
    val request = client.stock(title) {
      case Right(level) => c.succeed(level)
      case Left(error)  => c.fail(error)
    }
    () => request.cancel()
  }

val anotherSlowWarehouse = new InventoryClient(replyAfterMillis = 60000)
```

To repeat the experiment, and to check that the happy path is unchanged:

```scala
cancellableStockOf(anotherSlowWarehouse, "Zionomicon").start.cancel()
println(s"Open warehouse requests after cancel: ${anotherSlowWarehouse.openRequests}")
// Open warehouse requests after cancel: 0

println(cancellableStockOf(inventory, "Programming in Scala").block)
// 3
```

Cancellation reaches the helper through `Async#map` and through `Async.async` blocks too, because each combinator forwards the cancel signal to the computation it is waiting on.

:::warning
`Async.fromFuture`, `Async.fromCompletionStage` and `Async.fromJsPromise` do not cancel their source. Cancelling the `Async` stops it from waiting, but the `Future`, stage or promise keeps running. A `Future` and a `js.Promise` cannot be cancelled at all. For a `CompletableFuture` you want to cancel, use `cancellable`: register `CompletableFuture#whenComplete` to complete the `Completer`, and return `() => future.cancel(true)`. Unlike `Async.fromCompletionStage`, this bridge does not remove a `CompletionException` wrapper for you.
:::

## Putting It Together

The complete example uses the three clients from `Bookstore.scala`, which the Running the Examples section embeds, and quotes a book by combining them in an `Async.async` block: the price from the `Future`-based service, the stock level through the `cancellable` bridge, and the shipping estimate from the `CompletableFuture`. It hands each quote back as a `Future` to a storefront that has not moved to `Async`, recovers from unknown titles, and cancels one quote to show that its warehouse request is withdrawn:

```scala title="async-examples/src/main/scala/zio/blocks/async/wrapcallbackandfutureapis/CompleteExample.scala"
package zio.blocks.async.wrapcallbackandfutureapis

/**
 * Book Quotes: putting it together
 *
 * Combines a `Future`-based price service, a callback-based inventory client
 * (through a cancellable bridge) and a `CompletableFuture`-based shipping SDK
 * in one `Async.async` block, hands the result back to a `Future`-based
 * storefront, and cancels one quote to show its warehouse request is withdrawn.
 *
 * Run with:
 * {{{
 *   cd async-examples && sbt "runMain zio.blocks.async.wrapcallbackandfutureapis.CompleteExample"
 * }}}
 */
object CompleteExample {
  def cancellable[A](register: Completer[A] => (() => Unit)): Async[A] =
    new Pollable[A] {
      private val result     = new Completer[A]
      private val cancelCall = register(result)

      def poll(onComplete: Runnable): Async[A] = {
        val polled = result.poll(onComplete)
        // While pending, answer with this Pollable rather than the Completer, so
        // the driver keeps waiting on us and our cancel() is the one it calls.
        if (polled.asInstanceOf[AnyRef] eq result) this else polled
      }

      override def cancel(): Unit = cancelCall()
    }

  def cancellableStockOf(client: InventoryClient, title: String): Async[Int] =
    cancellable[Int] { c =>
      val request = client.stock(title) {
        case Right(level) => c.succeed(level)
        case Left(error)  => c.fail(error)
      }
      () => request.cancel()
    }

  final case class Quote(title: String, price: BigDecimal, inStock: Int, shippingDays: Int)

  def quote(client: InventoryClient, title: String): Async[Quote] =
    Async.async {
      val price = Async.fromFuture(PriceService.lookup(title)).await
      val stock = cancellableStockOf(client, title).await
      val days  = Async.fromCompletionStage(ShippingQuotes.estimateDays(title)).await
      Quote(title, price, stock, days)
    }

  def storefrontQuote(client: InventoryClient, title: String): Future[String] =
    quote(client, title).map { q =>
      val stock = if (q.inStock == 0) "out of stock" else s"${q.inStock} in stock"
      s"${q.title}: $$${q.price}, $stock, ships in ${q.shippingDays} days"
    }.catchAll(e => Async.succeed(s"$title: unavailable (${e.getMessage})")).toFuture

  def main(args: Array[String]): Unit = {
    val warehouse = new InventoryClient(replyAfterMillis = 10)

    List("Zionomicon", "Programming in Scala", "Functional Programming in Scala", "Scala with Cats").foreach { title =>
      println(Await.result(storefrontQuote(warehouse, title), 5.seconds))
    }

    val slowWarehouse = new InventoryClient(replyAfterMillis = 60000)
    quote(slowWarehouse, "Zionomicon").start.cancel()
    println(s"Open warehouse requests after cancel: ${slowWarehouse.openRequests}")
  }
}
```

## Choosing a Bridge

Each API shape has one bridge into `Async`. The last column says what cancelling the resulting `Async` does to the underlying work:

| API shape                             | Bridge into `Async`                                               | Cancelling the `Async`                       |
|---------------------------------------|-------------------------------------------------------------------|----------------------------------------------|
| Scala `Future[A]`                     | `Async.fromFuture(future)`                                        | Stops waiting; the `Future` runs on          |
| Java `CompletionStage[A]` (JVM)       | `Async.fromCompletionStage(stage)`                                | Stops waiting; the stage runs on             |
| JavaScript `js.Promise[A]` (Scala.js) | `Async.fromJsPromise(promise)`                                    | Stops waiting; the promise runs on           |
| Callback with no cancel handle        | `Async.promise` and `Completer`                                   | Stops waiting; the call runs on              |
| Callback that returns a cancel handle | A `Pollable` that overrides `Pollable#cancel`, like `cancellable` | Cancels the call                             |
| Status you check repeatedly (polling) | A `Pollable` whose `Pollable#poll` checks the status              | Whatever its `Pollable#cancel` override does |

To go back the other way, use `Async#toFuture` and, on the JVM, `Async#toCompletableFuture`, or `Async#toJsPromise` on Scala.js. For the polling shape, the [`Pollable` section of the reference](../reference/async.md#pollable) walks through a job-status example.

## Running the Examples

Every snippet in this guide has a standalone companion file in the `async-examples` build. Clone the repository and move into that build:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/async-examples
```

To check that every example compiles without running any of them:

```bash
sbt compile
```

Each example uses the sample clients from one shared file:

<details>
<summary>Sample Clients</summary>

```scala title="async-examples/src/main/scala/zio/blocks/async/wrapcallbackandfutureapis/Bookstore.scala" showLineNumbers
package zio.blocks.async.wrapcallbackandfutureapis

/**
 * Stand-ins for the third-party clients a bookstore might call. None of them
 * knows about `Async`: one returns a Scala `Future`, one returns a Java
 * `CompletableFuture`, and one reports through a callback and hands back a
 * cancel handle. Prices, stock levels and shipping times are fictional sample
 * data.
 */

/** A Future-based price service. */
object PriceService {
  private val prices = Map(
    "Zionomicon"                      -> BigDecimal("49.99"),
    "Programming in Scala"            -> BigDecimal("59.95"),
    "Functional Programming in Scala" -> BigDecimal("44.99")
  )

  def lookup(title: String)(implicit ec: ExecutionContext): Future[BigDecimal] =
    Future(prices.getOrElse(title, throw new NoSuchElementException(s"No price for $title")))
}

/** A Java-style shipping SDK that returns `CompletableFuture`s. */
object ShippingQuotes {
  def estimateDays(title: String): CompletableFuture[Int] =
    CompletableFuture.supplyAsync { () =>
      if (title.isEmpty) throw new IllegalArgumentException("Title must not be empty")
      if (title.length > 20) 5 else 2
    }
}

/**
 * A callback-based inventory client. `stock` reports through `onDone` once the
 * warehouse replies, after `replyAfterMillis`, and returns a `Request` whose
 * `cancel` withdraws the question before the reply arrives.
 */
final class InventoryClient(replyAfterMillis: Long) {
  private val stockLevels = Map(
    "Zionomicon"                      -> 12,
    "Programming in Scala"            -> 3,
    "Functional Programming in Scala" -> 0
  )
  private val open      = new AtomicInteger(0)
  private val warehouse = Executors.newSingleThreadScheduledExecutor { (r: Runnable) =>
    val t = new Thread(r, "warehouse")
    t.setDaemon(true)
    t
  }

  /** How many questions are still waiting for the warehouse. */
  def openRequests: Int = open.get()

  def stock(title: String)(onDone: Either[Throwable, Int] => Unit): InventoryClient.Request = {
    open.incrementAndGet()
    val reply = warehouse.schedule(
      new Runnable {
        def run(): Unit = {
          open.decrementAndGet()
          onDone(stockLevels.get(title).toRight(new NoSuchElementException(s"Unknown title: $title")))
        }
      },
      replyAfterMillis,
      TimeUnit.MILLISECONDS
    )
    new InventoryClient.Request {
      def cancel(): Unit = if (reply.cancel(false)) open.decrementAndGet()
    }
  }
}

object InventoryClient {
  trait Request {
    def cancel(): Unit
  }
}
```

</details>

<details>
<summary>Bring a Scala Future Into Async</summary>

```scala title="async-examples/src/main/scala/zio/blocks/async/wrapcallbackandfutureapis/Step1FromFutureExample.scala" showLineNumbers
package zio.blocks.async.wrapcallbackandfutureapis

/**
 * Bring a Scala `Future` Into `Async`
 *
 * Demonstrates `Async.fromFuture` on a successful and a failed `Future`,
 * observing the failure with `Async#either` and recovering with
 * `Async#catchAll`.
 *
 * Run with:
 * {{{
 *   cd async-examples && sbt "runMain zio.blocks.async.wrapcallbackandfutureapis.Step1FromFutureExample"
 * }}}
 */
object Step1FromFutureExample {
  def main(args: Array[String]): Unit = {
    val price: BigDecimal = Async.fromFuture(PriceService.lookup("Zionomicon")).block
    println(s"Zionomicon costs $price")

    println(Async.fromFuture(PriceService.lookup("Scala with Cats")).either.block)

    val fallbackPrice: BigDecimal =
      Async
        .fromFuture(PriceService.lookup("Scala with Cats"))
        .catchAll(_ => Async.succeed(BigDecimal(0)))
        .block
    println(s"Fallback price: $fallbackPrice")
  }
}
```

Run it with:

```bash
sbt "runMain zio.blocks.async.wrapcallbackandfutureapis.Step1FromFutureExample"
```

</details>

<details>
<summary>Hand an Async Back to Future-Based Code</summary>

```scala title="async-examples/src/main/scala/zio/blocks/async/wrapcallbackandfutureapis/Step2ToFutureExample.scala" showLineNumbers
package zio.blocks.async.wrapcallbackandfutureapis

/**
 * Hand an `Async` Back to Future-Based Code
 *
 * Demonstrates `Async#toFuture` for Scala callers and
 * `Async#toCompletableFuture` for Java callers.
 *
 * Run with:
 * {{{
 *   cd async-examples && sbt "runMain zio.blocks.async.wrapcallbackandfutureapis.Step2ToFutureExample"
 * }}}
 */
object Step2ToFutureExample {
  def main(args: Array[String]): Unit = {
    val forScala: Future[String] =
      Async
        .fromFuture(PriceService.lookup("Programming in Scala"))
        .map(p => s"Programming in Scala costs $p")
        .toFuture

    println(Await.result(forScala, 5.seconds))

    val forJava: CompletableFuture[BigDecimal] =
      Async.fromFuture(PriceService.lookup("Functional Programming in Scala")).toCompletableFuture

    println(forJava.get())
  }
}
```

Run it with:

```bash
sbt "runMain zio.blocks.async.wrapcallbackandfutureapis.Step2ToFutureExample"
```

</details>

<details>
<summary>Bring a Java CompletionStage Into Async</summary>

```scala title="async-examples/src/main/scala/zio/blocks/async/wrapcallbackandfutureapis/Step3CompletionStageExample.scala" showLineNumbers
package zio.blocks.async.wrapcallbackandfutureapis

/**
 * Bring a Java `CompletionStage` Into `Async`
 *
 * Demonstrates `Async.fromCompletionStage`, which removes the
 * `CompletionException` wrapper so a failed stage arrives as the exception the
 * producer threw.
 *
 * Run with:
 * {{{
 *   cd async-examples && sbt "runMain zio.blocks.async.wrapcallbackandfutureapis.Step3CompletionStageExample"
 * }}}
 */
object Step3CompletionStageExample {
  def main(args: Array[String]): Unit = {
    val days: Int = Async.fromCompletionStage(ShippingQuotes.estimateDays("Zionomicon")).block
    println(s"Zionomicon ships in $days days")

    println(Async.fromCompletionStage(ShippingQuotes.estimateDays("")).either.block)
  }
}
```

Run it with:

```bash
sbt "runMain zio.blocks.async.wrapcallbackandfutureapis.Step3CompletionStageExample"
```

</details>

<details>
<summary>Wrap a Callback That Reports Errors</summary>

```scala title="async-examples/src/main/scala/zio/blocks/async/wrapcallbackandfutureapis/Step4CallbackErrorsExample.scala" showLineNumbers
package zio.blocks.async.wrapcallbackandfutureapis

/**
 * Wrap a Callback That Reports Errors
 *
 * Demonstrates `Async.promise` with a `Completer`: `Completer#succeed` on the
 * success branch and `Completer#fail` on the error branch. The body is a Scala
 * 3 context function (`c ?=>`); on Scala 2 write `c =>`.
 *
 * Run with:
 * {{{
 *   cd async-examples && sbt "runMain zio.blocks.async.wrapcallbackandfutureapis.Step4CallbackErrorsExample"
 * }}}
 */
object Step4CallbackErrorsExample {
  def stockOf(client: InventoryClient, title: String): Async[Int] =
    Async.promise[Int] { c ?=>
      client.stock(title) {
        case Right(level) => c.succeed(level)
        case Left(error)  => c.fail(error)
      }
      () // the Request handle is dropped here
    }

  def main(args: Array[String]): Unit = {
    val inventory = new InventoryClient(replyAfterMillis = 10)

    println(stockOf(inventory, "Zionomicon").block)

    println(stockOf(inventory, "Scala with Cats").either.block)
  }
}
```

Run it with:

```bash
sbt "runMain zio.blocks.async.wrapcallbackandfutureapis.Step4CallbackErrorsExample"
```

</details>

<details>
<summary>Cancel the Underlying Call</summary>

```scala title="async-examples/src/main/scala/zio/blocks/async/wrapcallbackandfutureapis/Step5CancellableCallbackExample.scala" showLineNumbers
package zio.blocks.async.wrapcallbackandfutureapis

/**
 * Cancel the Underlying Call When the `Async` Is Cancelled
 *
 * Shows that a promise-based `stockOf` leaves the warehouse request open after
 * cancellation, then fixes it with a `cancellable` helper: a custom `Pollable`
 * whose `cancel` forwards to the callback API's `Request` handle.
 *
 * Run with:
 * {{{
 *   cd async-examples && sbt "runMain zio.blocks.async.wrapcallbackandfutureapis.Step5CancellableCallbackExample"
 * }}}
 */
object Step5CancellableCallbackExample {
  def stockOf(client: InventoryClient, title: String): Async[Int] =
    Async.promise[Int] { c ?=>
      client.stock(title) {
        case Right(level) => c.succeed(level)
        case Left(error)  => c.fail(error)
      }
      () // the Request handle is dropped here
    }

  def cancellable[A](register: Completer[A] => (() => Unit)): Async[A] =
    new Pollable[A] {
      private val result     = new Completer[A]
      private val cancelCall = register(result)

      def poll(onComplete: Runnable): Async[A] = {
        val polled = result.poll(onComplete)
        // While pending, answer with this Pollable rather than the Completer, so
        // the driver keeps waiting on us and our cancel() is the one it calls.
        if (polled.asInstanceOf[AnyRef] eq result) this else polled
      }

      override def cancel(): Unit = cancelCall()
    }

  def cancellableStockOf(client: InventoryClient, title: String): Async[Int] =
    cancellable[Int] { c =>
      val request = client.stock(title) {
        case Right(level) => c.succeed(level)
        case Left(error)  => c.fail(error)
      }
      () => request.cancel()
    }

  def main(args: Array[String]): Unit = {
    val inventory = new InventoryClient(replyAfterMillis = 10)

    val slowWarehouse = new InventoryClient(replyAfterMillis = 60000)
    stockOf(slowWarehouse, "Zionomicon").start.cancel()
    println(s"Open warehouse requests after cancel: ${slowWarehouse.openRequests}")

    val anotherSlowWarehouse = new InventoryClient(replyAfterMillis = 60000)
    cancellableStockOf(anotherSlowWarehouse, "Zionomicon").start.cancel()
    println(s"Open warehouse requests after cancel: ${anotherSlowWarehouse.openRequests}")

    println(cancellableStockOf(inventory, "Programming in Scala").block)
  }
}
```

Run it with:

```bash
sbt "runMain zio.blocks.async.wrapcallbackandfutureapis.Step5CancellableCallbackExample"
```

</details>

<details>
<summary>Complete Example: Book Quotes</summary>

```scala title="async-examples/src/main/scala/zio/blocks/async/wrapcallbackandfutureapis/CompleteExample.scala" showLineNumbers
package zio.blocks.async.wrapcallbackandfutureapis

/**
 * Book Quotes: putting it together
 *
 * Combines a `Future`-based price service, a callback-based inventory client
 * (through a cancellable bridge) and a `CompletableFuture`-based shipping SDK
 * in one `Async.async` block, hands the result back to a `Future`-based
 * storefront, and cancels one quote to show its warehouse request is withdrawn.
 *
 * Run with:
 * {{{
 *   cd async-examples && sbt "runMain zio.blocks.async.wrapcallbackandfutureapis.CompleteExample"
 * }}}
 */
object CompleteExample {
  def cancellable[A](register: Completer[A] => (() => Unit)): Async[A] =
    new Pollable[A] {
      private val result     = new Completer[A]
      private val cancelCall = register(result)

      def poll(onComplete: Runnable): Async[A] = {
        val polled = result.poll(onComplete)
        // While pending, answer with this Pollable rather than the Completer, so
        // the driver keeps waiting on us and our cancel() is the one it calls.
        if (polled.asInstanceOf[AnyRef] eq result) this else polled
      }

      override def cancel(): Unit = cancelCall()
    }

  def cancellableStockOf(client: InventoryClient, title: String): Async[Int] =
    cancellable[Int] { c =>
      val request = client.stock(title) {
        case Right(level) => c.succeed(level)
        case Left(error)  => c.fail(error)
      }
      () => request.cancel()
    }

  final case class Quote(title: String, price: BigDecimal, inStock: Int, shippingDays: Int)

  def quote(client: InventoryClient, title: String): Async[Quote] =
    Async.async {
      val price = Async.fromFuture(PriceService.lookup(title)).await
      val stock = cancellableStockOf(client, title).await
      val days  = Async.fromCompletionStage(ShippingQuotes.estimateDays(title)).await
      Quote(title, price, stock, days)
    }

  def storefrontQuote(client: InventoryClient, title: String): Future[String] =
    quote(client, title).map { q =>
      val stock = if (q.inStock == 0) "out of stock" else s"${q.inStock} in stock"
      s"${q.title}: $$${q.price}, $stock, ships in ${q.shippingDays} days"
    }.catchAll(e => Async.succeed(s"$title: unavailable (${e.getMessage})")).toFuture

  def main(args: Array[String]): Unit = {
    val warehouse = new InventoryClient(replyAfterMillis = 10)

    List("Zionomicon", "Programming in Scala", "Functional Programming in Scala", "Scala with Cats").foreach { title =>
      println(Await.result(storefrontQuote(warehouse, title), 5.seconds))
    }

    val slowWarehouse = new InventoryClient(replyAfterMillis = 60000)
    quote(slowWarehouse, "Zionomicon").start.cancel()
    println(s"Open warehouse requests after cancel: ${slowWarehouse.openRequests}")
  }
}
```

Run it with:

```bash
sbt "runMain zio.blocks.async.wrapcallbackandfutureapis.CompleteExample"
```

</details>

## Going Further

- The [Async reference](../reference/async.md) covers the [`Pollable` protocol](../reference/async.md#the-pollable-protocol), [`Completer`](../reference/async.md#completer), and [`Running#cancel`](../reference/async.md#runningcancel) in full.
- [Asynchronous Stream Execution](../reference/streams/execution-and-compatibility/async-execution.md) shows how `Async` sources feed streams.
- [Compile-Time Resource Safety with Scope](./compile-time-resource-safety-with-scope.md) ties `Async.Running` handles, which are `AutoCloseable`, to lexical scopes so they are cancelled on exit.
- [How to Use ZIO Blocks in a ZIO or cats-effect Application](./use-zio-blocks-in-zio-or-cats-effect.md) calls `Async` from ZIO or cats-effect, with cancellation carried across.

---

## How to Write a Codec for a New Wire Format

In this guide, we write a codec for a small binary format that ZIO Blocks does not ship, and we package it as a [`BinaryFormat`](../reference/schema/format.md) so that any type with a `Schema` gets it through `Schema#derive`, exactly like `derive(JsonFormat)`. By the end, an IoT gateway's sensor readings encode to a compact, length-prefixed binary message and decode back, a broken message fails with an error that names the field that broke (such as `.readings.at(1).takenAt`), and per-type overrides with `Deriver#withInstance` work as they do for the built-in formats.

The same steps apply to any format you need to integrate: Protobuf, DynamoDB's `AttributeValue`, a message broker's payload, CBOR or an in-house protocol. We build the format one schema node at a time: primitives, records, variants, collections, wrappers, then the format singleton, error paths, recursion and overrides. The guide uses only public API, so the code works from your own module on every Scala version ZIO Blocks supports.

We cover:

- Defining the codec base class
- Encoding primitives, records, variants, sequences and maps
- Handling wrappers, validation and dynamic values
- Packaging the format and checking the round trip
- Reporting errors with paths
- Encoding recursive types
- Overriding one type with `Deriver#withInstance`

## The Problem

A service that speaks a binary protocol needs an encoder and a decoder for every message type, and without a derivation mechanism each pair is written by hand. Here is the hand-written version for two message types of a sensor gateway:

```scala

final case class SensorReading(device: String, takenAt: Instant, celsius: Double)
final case class GatewayBatch(gateway: String, readings: List[SensorReading])

def writeString(s: String, out: ByteBuffer): Unit = {
  val bytes = s.getBytes(UTF_8)
  out.putInt(bytes.length)
  out.put(bytes)
}

def readString(in: ByteBuffer): String = {
  val bytes = new Array[Byte](in.getInt())
  in.get(bytes)
  new String(bytes, UTF_8)
}

def writeReading(r: SensorReading, out: ByteBuffer): Unit = {
  writeString(r.device, out)
  out.putLong(r.takenAt.getEpochSecond)
  out.putInt(r.takenAt.getNano)
  out.putDouble(r.celsius)
}

def readReading(in: ByteBuffer): SensorReading =
  SensorReading(readString(in), Instant.ofEpochSecond(in.getLong(), in.getInt().toLong), in.getDouble())

def writeBatch(b: GatewayBatch, out: ByteBuffer): Unit = {
  writeString(b.gateway, out)
  out.putInt(b.readings.length)
  b.readings.foreach(writeReading(_, out))
}

def readBatch(in: ByteBuffer): GatewayBatch = {
  val gateway = readString(in)
  val count   = in.getInt()
  GatewayBatch(gateway, List.fill(count)(readReading(in)))
}
```

Every new message type needs two more functions, and every new field means editing both of them in the same order; nothing checks that they agree. When a message arrives cut off, `readBatch` throws a `BufferUnderflowException` that says nothing about which reading or which field was missing. A library author is worse off still: a Protobuf or DynamoDB integration has to handle types its users define, which no hand-written function can know about. This guide writes the answer down.

ZIO Blocks already knows the shape of every type that has a `Schema`. A [`Deriver`](../reference/schema/type-class-derivation.md) turns that shape into codecs: we write one function per kind of schema node (primitive, record, variant, sequence, map, wrapper, dynamic value), and derivation combines them for any type.

## Prerequisites

The guide needs only the schema module:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"
```

These imports are used throughout the guide:

```scala

```

You should know how to define a schema with `Schema.derived` ([Your First Schema](./your-first-schema.md)) and how a `Deriver` builds a type class from one ([How to Derive Your Own Type Class from a Schema](./derive-your-own-type-class.md)). That guide derives a type class that only renders values; this one adds what a codec needs on top: building values back from bytes and reporting where decoding failed.

## The Wire Format and the Sensor Model

Our format, which we call the packed format, is tag-free: it writes no field names and no type information, only the values in a fixed order, so both sides must use the same schema. Each kind of schema node is laid out as follows:

| Schema node                          | Bytes on the wire                                     |
| ------------------------------------ | ----------------------------------------------------- |
| `Boolean`                            | One byte, `0` or `1`                                  |
| `Int`, `Long`, `Float`, `Double`     | 4, 8, 4 and 8 bytes, big-endian                       |
| `String`                             | An `Int` byte length, then the UTF-8 bytes            |
| `Instant`                            | A `Long` of epoch seconds, then an `Int` of nanos     |
| Record (case class)                  | Each field in declaration order                       |
| Variant (sealed trait, `Option`)     | An `Int` case index, then the case                    |
| Sequence (`List`, `Vector`, ...)     | An `Int` element count, then each element             |
| Map                                  | An `Int` entry count, then each key and its value     |
| Wrapper (`Schema#transform`)         | The wrapped value                                     |

The messages come from a gateway that collects sensor readings. A `DeviceId` is a validated newtype, a `Measurement` is a sealed trait with a case object, a `Batch` holds a list and a map, and a `Zone` is a recursive tree that describes the site:

```scala
final case class DeviceId(value: String)

object DeviceId {
  implicit val schema: Schema[DeviceId] = Schema[String].transform(
    id =>
      if (id.nonEmpty && id.length <= 16) DeviceId(id)
      else throw SchemaError.validationFailed(s"device id must be 1 to 16 characters, got ${id.length}"),
    _.value
  )
}

sealed trait Measurement
final case class Temperature(celsius: Double) extends Measurement
final case class Humidity(percent: Float)     extends Measurement
case object DoorOpened                        extends Measurement

object Measurement {
  implicit val schema: Schema[Measurement] = Schema.derived
}

final case class Reading(device: DeviceId, takenAt: Instant, measurement: Measurement, battery: Option[Int])

object Reading {
  implicit val schema: Schema[Reading] = Schema.derived
}

final case class Batch(gateway: String, readings: List[Reading], calibration: Map[String, Double])

object Batch {
  implicit val schema: Schema[Batch] = Schema.derived

  val sample: Batch = Batch(
    gateway = "gw-north",
    readings = List(
      Reading(DeviceId("t-17"), Instant.parse("2026-10-29T08:00:00Z"), Temperature(21.5), Some(87)),
      Reading(DeviceId("h-02"), Instant.parse("2026-10-29T08:00:05Z"), Humidity(41.0f), None),
      Reading(DeviceId("d-01"), Instant.parse("2026-10-29T08:00:09Z"), DoorOpened, Some(12))
    ),
    calibration = Map("t-17" -> -0.4, "h-02" -> 1.5)
  )
}

final case class Zone(name: String, devices: Vector[DeviceId], subzones: List[Zone])

object Zone {
  implicit val schema: Schema[Zone] = Schema.derived
}
```

None of these types mentions the packed format. The schemas are all the codec needs.

## Defining the Codec Base Class

Every format's codec extends [`BinaryCodec[A]`](../reference/schema/codec.md), which fixes the contract that `Schema#derive`, `Schema#encode` and `Schema#decode` rely on: `BinaryCodec#encode` writes a value into a `ByteBuffer` the caller provides, and `BinaryCodec#decode` returns `Either[SchemaError, A]`. Our base class adds `PackedCodec#write` and `PackedCodec#read`, which every node's codec implements. `PackedCodec#read` throws on bad input instead of returning `Either`, so a record does not allocate an `Either` per field, and an error travels up through the nested codecs, where each one adds its part of the path. `PackedCodec#decode` catches it once, at the top:

```scala
abstract class PackedCodec[A] extends BinaryCodec[A] {
  def write(value: A, out: ByteBuffer): Unit
  def read(in: ByteBuffer): A

  final def encode(value: A, output: ByteBuffer): Unit = write(value, output)

  final def decode(input: ByteBuffer): Either[SchemaError, A] =
    try Right(read(input))
    catch { case e if NonFatal(e) => Left(PackedCodec.toSchemaError(e)) }

  final def encodeToBuffer(value: A, capacity: Int = 1024): ByteBuffer = {
    val buffer = ByteBuffer.allocate(capacity)
    write(value, buffer)
    buffer.flip()
    buffer
  }
}

object PackedCodec {
  def fail(expectation: String): Nothing = throw SchemaError.expectationMismatch(Nil, expectation)

  def toSchemaError(e: Throwable): SchemaError = e match {
    case error: SchemaError          => error
    case _: BufferUnderflowException => SchemaError.expectationMismatch(Nil, "Unexpected end of input")
    case other                       => SchemaError.expectationMismatch(Nil, String.valueOf(other.getMessage))
  }

  def readCount(in: ByteBuffer): Int = {
    val count = in.getInt()
    if (count < 0) fail(s"Expected a non-negative count, got $count")
    count
  }
}
```

[`SchemaError`](../reference/schema/schema-error.md) is itself an exception, so `PackedCodec.fail` throws one directly. `SchemaError.expectationMismatch(Nil, ...)` creates an error at the root path; the codecs of records, variants and collections add the path segments as the error passes through them. `PackedCodec.toSchemaError` turns the exceptions a `ByteBuffer` throws, such as `BufferUnderflowException` on cut-off input, into a `SchemaError`. `PackedCodec#encodeToBuffer` is a convenience for the examples: it allocates a fixed-size buffer, so a value larger than `capacity` throws `BufferOverflowException`.

## Encoding Primitives

A `Deriver[PackedCodec]` has one method for each kind of schema node. To keep each section to one kind of node, we write the deriver as a chain of traits, one per section, and finish it with an `object` that extends the last trait. In your own module, you can put all seven methods in one object instead.

The primitive codecs read and write fixed-width values and length-prefixed strings:

```scala
object PackedPrimitives {
  val boolean: PackedCodec[Boolean] = new PackedCodec[Boolean] {
    def write(value: Boolean, out: ByteBuffer): Unit = out.put(if (value) 1.toByte else 0.toByte)
    def read(in: ByteBuffer): Boolean                = in.get() match {
      case 0 => false
      case 1 => true
      case b => PackedCodec.fail(s"Expected a boolean byte 0 or 1, got $b")
    }
  }

  val int: PackedCodec[Int] = new PackedCodec[Int] {
    def write(value: Int, out: ByteBuffer): Unit = out.putInt(value)
    def read(in: ByteBuffer): Int                = in.getInt()
  }

  val long: PackedCodec[Long] = new PackedCodec[Long] {
    def write(value: Long, out: ByteBuffer): Unit = out.putLong(value)
    def read(in: ByteBuffer): Long                = in.getLong()
  }

  val float: PackedCodec[Float] = new PackedCodec[Float] {
    def write(value: Float, out: ByteBuffer): Unit = out.putFloat(value)
    def read(in: ByteBuffer): Float                = in.getFloat()
  }

  val double: PackedCodec[Double] = new PackedCodec[Double] {
    def write(value: Double, out: ByteBuffer): Unit = out.putDouble(value)
    def read(in: ByteBuffer): Double                = in.getDouble()
  }

  val string: PackedCodec[String] = new PackedCodec[String] {
    def write(value: String, out: ByteBuffer): Unit = {
      val bytes = value.getBytes(UTF_8)
      out.putInt(bytes.length)
      out.put(bytes)
    }
    def read(in: ByteBuffer): String = {
      val length = PackedCodec.readCount(in)
      if (length > in.remaining) PackedCodec.fail(s"Expected $length bytes of text, got ${in.remaining}")
      val bytes = new Array[Byte](length)
      in.get(bytes)
      new String(bytes, UTF_8)
    }
  }

  val instant: PackedCodec[Instant] = new PackedCodec[Instant] {
    def write(value: Instant, out: ByteBuffer): Unit = {
      out.putLong(value.getEpochSecond)
      out.putInt(value.getNano)
    }
    def read(in: ByteBuffer): Instant = Instant.ofEpochSecond(in.getLong(), in.getInt().toLong)
  }
}
```

`Deriver#derivePrimitive` receives a `PrimitiveType` that says which primitive the node is, and returns its codec wrapped in `Lazy`:

```scala
trait PrimitiveDerivation extends Deriver[PackedCodec] {
  override def derivePrimitive[A](
    primitiveType: PrimitiveType[A],
    typeId: TypeId[A],
    binding: Binding.Primitive[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  ): Lazy[PackedCodec[A]] = {
    val codec = primitiveType match {
      case _: PrimitiveType.Boolean => PackedPrimitives.boolean
      case _: PrimitiveType.Int     => PackedPrimitives.int
      case _: PrimitiveType.Long    => PackedPrimitives.long
      case _: PrimitiveType.Float   => PackedPrimitives.float
      case _: PrimitiveType.Double  => PackedPrimitives.double
      case _: PrimitiveType.String  => PackedPrimitives.string
      case _: PrimitiveType.Instant => PackedPrimitives.instant
      case _                        =>
        throw new UnsupportedOperationException(s"The packed format has no codec for ${typeId.name}")
    }
    Lazy(codec.asInstanceOf[PackedCodec[A]])
  }
}
```

ZIO Blocks has 30 primitive types; the packed format supports the seven our messages use and rejects the rest. Derivation calls `Deriver#derivePrimitive` while `Schema#derive` walks the schema, and we choose the codec before wrapping it in `Lazy`, so a type with an unsupported primitive fails when its codec is derived, not when the first message arrives. This holds for a primitive behind a `Reflect.Deferred` node too, such as a recursive field or a field with a default value: the packed format's `Schema#derive` fails for it as well.

## Encoding Records Field by Field

A record's `Binding.Record` holds a `Constructor`, which builds a value from a `Registers` buffer, and a `Deconstructor`, which writes a value's fields into one. `Registers` stores primitive fields without boxing them, so each field has a fixed slot whose position depends on the types of the fields before it. `Reflect.Record.registers` computes those slots from the field nodes, one `Register` per field, in the layout the binding uses, and `Reflect.Record.usedRegisters` returns the size of the buffer they need.

`Deriver#deriveRecord` receives the fields as `Term`s, each with its name and its node. `Deriver#instance` reads a field's codec from the node's metadata:

```scala
trait RecordDerivation extends PrimitiveDerivation {
  override def deriveRecord[F[_, _], A](
    fields: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Record[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[PackedCodec[A]] = {
    val names         = fields.map(_.name).toArray
    val registers     = Reflect.Record.registers(fields.map(_.value).toArray)
    val usedRegisters = Reflect.Record.usedRegisters(registers)
    Lazy {
      new PackedCodec[A] {
        private lazy val fieldCodecs: Array[PackedCodec[Any]] =
          fields.map(field => D.instance(field.value.metadata).force.asInstanceOf[PackedCodec[Any]]).toArray

        def write(value: A, out: ByteBuffer): Unit = {
          val regs = Registers(usedRegisters)
          binding.deconstructor.deconstruct(regs, RegisterOffset.Zero, value)
          fieldCodecs.indices.foreach { i =>
            fieldCodecs(i).write(registers(i).get(regs, RegisterOffset.Zero), out)
          }
        }

        def read(in: ByteBuffer): A = {
          val regs = Registers(usedRegisters)
          fieldCodecs.indices.foreach { i =>
            val fieldValue =
              try fieldCodecs(i).read(in)
              catch { case e if NonFatal(e) => throw PackedCodec.toSchemaError(e).atField(names(i)) }
            registers(i).set(regs, RegisterOffset.Zero, fieldValue)
          }
          binding.constructor.construct(regs, RegisterOffset.Zero)
        }
      }
    }
  }
}
```

Encoding deconstructs the value into a fresh `Registers` buffer and writes each field with its codec. Decoding reads each field into its register, then calls the constructor once. Two details make the codec correct for every record. The field codecs live in a `lazy val`, so they are looked up on the first `PackedCodec#write` or `PackedCodec#read` and never during derivation; that is what lets a recursive type such as `Zone` derive without looping. And each field read sits in a `try` that rethrows the error with `SchemaError#atField`, which prepends the field's name to the error's path.

:::note
The Avro module's deriver computes register offsets with `Reflect.registerOffset` and `Reflect.typeTag`. Both are `private[blocks]`, so code outside ZIO Blocks cannot call them. If you use the Avro source as a model, use the public `Reflect.Record.registers` and `Reflect.Record.usedRegisters` instead, as above.
:::

## Encoding Variants and Options

A variant's `Binding.Variant` has a `Discriminator` that returns the index of the case a value belongs to. Each case is a `Term` whose node is the case's own record (or wrapper), so its codec already exists. The variant writes the index, then lets the case's codec write the value:

```scala
trait VariantDerivation extends RecordDerivation {
  override def deriveVariant[F[_, _], A](
    cases: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Variant[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[PackedCodec[A]] = {
    val names = cases.map(_.name).toArray
    Lazy {
      new PackedCodec[A] {
        private lazy val caseCodecs: Array[PackedCodec[A]] =
          cases.map(c => D.instance(c.value.metadata).force.asInstanceOf[PackedCodec[A]]).toArray

        def write(value: A, out: ByteBuffer): Unit = {
          val index = binding.discriminator.discriminate(value)
          out.putInt(index)
          caseCodecs(index).write(value, out)
        }

        def read(in: ByteBuffer): A = {
          val index = in.getInt()
          if (index < 0 || index >= caseCodecs.length)
            PackedCodec.fail(s"Expected a case index from 0 to ${caseCodecs.length - 1}, got $index")
          try caseCodecs(index).read(in)
          catch { case e if NonFatal(e) => throw PackedCodec.toSchemaError(e).atCase(names(index)) }
        }
      }
    }
  }
}
```

A case object such as `DoorOpened` is a record with no fields, so it encodes as its index alone. `Option` needs no code of its own: its schema is a variant with the cases `None` (index 0) and `Some` (index 1), so `battery: Option[Int]` encodes as `0`, or as `1` followed by the `Int`. A decode error inside a case gets the case's name added with `SchemaError#atCase`, and an index outside the cases fails before any case is read.

## Encoding Sequences and Maps

A sequence's `Binding.Seq` has a `SeqConstructor`, which builds the collection through a mutable builder, and a `SeqDeconstructor`, which gives its size and an iterator over its elements. One binding serves every collection type, from `List` to `Array`. The array-backed ones need a `ClassTag` of the element type to allocate their array, which `TypeId#classTag` provides:

```scala
trait CollectionDerivation extends VariantDerivation {
  override def deriveSequence[F[_, _], C[_], A](
    element: Reflect[F, A],
    typeId: TypeId[C[A]],
    binding: Binding.Seq[C, A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[C[A]],
    examples: Seq[C[A]]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[PackedCodec[C[A]]] = {
    implicit val elementClassTag: ClassTag[A] = element.typeId.classTag.asInstanceOf[ClassTag[A]]
    D.instance(element.metadata).map { elementCodec =>
      new PackedCodec[C[A]] {
        def write(value: C[A], out: ByteBuffer): Unit = {
          out.putInt(binding.deconstructor.size(value))
          binding.deconstructor.deconstruct(value).foreach(elementCodec.write(_, out))
        }

        def read(in: ByteBuffer): C[A] = {
          val count   = PackedCodec.readCount(in)
          val builder = binding.constructor.newBuilder[A](math.min(count, in.remaining))
          (0 until count).foreach { i =>
            val elem =
              try elementCodec.read(in)
              catch { case e if NonFatal(e) => throw PackedCodec.toSchemaError(e).atIndex(i) }
            binding.constructor.add(builder, elem)
          }
          binding.constructor.result(builder)
        }
      }
    }
  }

  override def deriveMap[F[_, _], M[_, _], K, V](
    key: Reflect[F, K],
    value: Reflect[F, V],
    typeId: TypeId[M[K, V]],
    binding: Binding.Map[M, K, V],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[M[K, V]],
    examples: Seq[M[K, V]]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[PackedCodec[M[K, V]]] =
    D.instance(key.metadata).zip(D.instance(value.metadata)).map { case (keyCodec, valueCodec) =>
      new PackedCodec[M[K, V]] {
        def write(map: M[K, V], out: ByteBuffer): Unit = {
          val entries = binding.deconstructor
          out.putInt(entries.size(map))
          entries.deconstruct(map).foreach { entry =>
            keyCodec.write(entries.getKey(entry), out)
            valueCodec.write(entries.getValue(entry), out)
          }
        }

        def read(in: ByteBuffer): M[K, V] = {
          val count   = PackedCodec.readCount(in)
          val builder = binding.constructor.newObjectBuilder[K, V](math.min(count, in.remaining))
          (0 until count).foreach { i =>
            val k =
              try keyCodec.read(in)
              catch { case e if NonFatal(e) => throw PackedCodec.toSchemaError(e).atIndex(i) }
            val v =
              try valueCodec.read(in)
              catch { case e if NonFatal(e) => throw PackedCodec.toSchemaError(e).atKey(key.toDynamicValue(k)) }
            binding.constructor.addObject(builder, k, v)
          }
          binding.constructor.resultObject(builder)
        }
      }
    }
}
```

Here the child codecs come from `Lazy#map` and `Lazy#zip` instead of a `lazy val`: a `Lazy` is evaluated only when forced and remembers its result, so the element's codec is still looked up on first use. The size hint passed to the builder is capped by the bytes left in the buffer, so a corrupt count cannot make the decoder allocate a huge collection. An element error gets its index with `SchemaError#atIndex`; a map value error gets its key with `SchemaError#atKey`, which takes the key as a `DynamicValue` that `Reflect#toDynamicValue` produces.

## Handling Wrappers and Dynamic Values

A wrapper node comes from `Schema#transform` or a newtype: its `Binding.Wrapper` converts between the wrapper type and the wrapped one. The packed format writes the wrapped value as it is. The last trait in the chain becomes the deriver itself:

```scala
object PackedCodecDeriver extends CollectionDerivation {
  override def deriveWrapper[F[_, _], A, B](
    wrapped: Reflect[F, B],
    typeId: TypeId[A],
    binding: Binding.Wrapper[A, B],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[PackedCodec[A]] =
    D.instance(wrapped.metadata).map { inner =>
      new PackedCodec[A] {
        def write(value: A, out: ByteBuffer): Unit = inner.write(binding.unwrap(value), out)
        def read(in: ByteBuffer): A                = binding.wrap(inner.read(in))
      }
    }

  override def deriveDynamic[F[_, _]](
    binding: Binding.Dynamic,
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[DynamicValue],
    examples: Seq[DynamicValue]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[PackedCodec[DynamicValue]] =
    throw new UnsupportedOperationException("The packed format does not encode DynamicValue")
}
```

The wrapper's codec needs no error handling of its own. `DeviceId`'s schema throws a `SchemaError` from the `Binding.Wrapper#wrap` function when an id is invalid; the exception passes through the wrapper's `PackedCodec#read` unchanged, and the record that holds the `DeviceId` adds the field name to it. A wrap function that throws another exception, such as an `IllegalArgumentException` from `require`, becomes a `SchemaError` with the exception's message in `PackedCodec.toSchemaError`.

`Deriver#deriveDynamic` handles `DynamicValue`, the node for a part of a schema that accepts any value. A tag-free format cannot encode a value whose shape the schema does not fix, so the packed format rejects it; like `Deriver#derivePrimitive`, it runs while `Schema#derive` walks the schema, so the failure comes from `Schema#derive`. A self-describing format would encode the `DynamicValue` instead, as the Avro module does.

## Deriving Codecs and Checking the Round Trip

A format singleton ties the codec type, a MIME type and the deriver together. Extending `BinaryFormat` also brings the implicit that lets `Schema#derive` accept the format:

```scala
object PackedFormat extends BinaryFormat[PackedCodec]("application/x-packed", PackedCodecDeriver)
```

Now every type with a schema gets a packed codec through `Schema#derive`, as with `JsonFormat`. `Schema#getInstance`, `Schema#encode` and `Schema#decode` accept `PackedFormat` too:

```scala
val batchCodec: PackedCodec[Batch] = Batch.schema.derive(PackedFormat)
val batchBytes: ByteBuffer         = batchCodec.encodeToBuffer(Batch.sample)
```

The whole batch, with three readings and two calibration entries, takes 156 bytes, and it decodes back to the value we started with:

```scala
println(s"${PackedFormat.mimeType}: ${batchBytes.remaining} bytes")
// application/x-packed: 156 bytes
println(batchCodec.decode(batchBytes.duplicate()))
// Right(Batch(gw-north,List(Reading(DeviceId(t-17),2026-10-29T08:00:00Z,Temperature(21.5),Some(87)), Reading(DeviceId(h-02),2026-10-29T08:00:05Z,Humidity(41.0),None), Reading(DeviceId(d-01),2026-10-29T08:00:09Z,DoorOpened,Some(12))),Map(t-17 -> -0.4, h-02 -> 1.5)))
```

A round-trip check belongs in the test suite of every format: encode a value, decode it, and compare. It also checks that decoding consumed every byte, which catches an encoder that writes something the decoder never reads:

```scala
def roundTrips[A](codec: PackedCodec[A], value: A): Boolean = {
  val bytes = codec.encodeToBuffer(value)
  codec.decode(bytes) == Right(value) && !bytes.hasRemaining
}

val readingCodec: PackedCodec[Reading] = Reading.schema.derive(PackedFormat)
```

The sample readings cover every case of `Measurement` and both cases of `Option`:

```scala
Batch.sample.readings.foreach { reading =>
  println(s"${reading.measurement} round-trips: ${roundTrips(readingCodec, reading)}")
}
// Temperature(21.5) round-trips: true
// Humidity(41.0) round-trips: true
// DoorOpened round-trips: true
```

<details>
  <summary>The whole format in one file, as the example files use it: <code>PackedFormat.scala</code></summary>

```scala title="schema-examples/src/main/scala/customwireformat/PackedFormat.scala" showLineNumbers
package customwireformat

/**
 * How to Write a Codec for a New Wire Format — The Packed Format
 *
 * The codec base class, the primitive codecs, the deriver (one trait per
 * section, from "Defining the Codec Base Class" to "Handling Wrappers and
 * Dynamic Values") and the format singleton. It has no main method.
 */
abstract class PackedCodec[A] extends BinaryCodec[A] {
  def write(value: A, out: ByteBuffer): Unit
  def read(in: ByteBuffer): A

  final def encode(value: A, output: ByteBuffer): Unit = write(value, output)

  final def decode(input: ByteBuffer): Either[SchemaError, A] =
    try Right(read(input))
    catch { case e if NonFatal(e) => Left(PackedCodec.toSchemaError(e)) }

  final def encodeToBuffer(value: A, capacity: Int = 1024): ByteBuffer = {
    val buffer = ByteBuffer.allocate(capacity)
    write(value, buffer)
    buffer.flip()
    buffer
  }
}

object PackedCodec {
  def fail(expectation: String): Nothing = throw SchemaError.expectationMismatch(Nil, expectation)

  def toSchemaError(e: Throwable): SchemaError = e match {
    case error: SchemaError          => error
    case _: BufferUnderflowException => SchemaError.expectationMismatch(Nil, "Unexpected end of input")
    case other                       => SchemaError.expectationMismatch(Nil, String.valueOf(other.getMessage))
  }

  def readCount(in: ByteBuffer): Int = {
    val count = in.getInt()
    if (count < 0) fail(s"Expected a non-negative count, got $count")
    count
  }
}

object PackedPrimitives {
  val boolean: PackedCodec[Boolean] = new PackedCodec[Boolean] {
    def write(value: Boolean, out: ByteBuffer): Unit = out.put(if (value) 1.toByte else 0.toByte)
    def read(in: ByteBuffer): Boolean                = in.get() match {
      case 0 => false
      case 1 => true
      case b => PackedCodec.fail(s"Expected a boolean byte 0 or 1, got $b")
    }
  }

  val int: PackedCodec[Int] = new PackedCodec[Int] {
    def write(value: Int, out: ByteBuffer): Unit = out.putInt(value)
    def read(in: ByteBuffer): Int                = in.getInt()
  }

  val long: PackedCodec[Long] = new PackedCodec[Long] {
    def write(value: Long, out: ByteBuffer): Unit = out.putLong(value)
    def read(in: ByteBuffer): Long                = in.getLong()
  }

  val float: PackedCodec[Float] = new PackedCodec[Float] {
    def write(value: Float, out: ByteBuffer): Unit = out.putFloat(value)
    def read(in: ByteBuffer): Float                = in.getFloat()
  }

  val double: PackedCodec[Double] = new PackedCodec[Double] {
    def write(value: Double, out: ByteBuffer): Unit = out.putDouble(value)
    def read(in: ByteBuffer): Double                = in.getDouble()
  }

  val string: PackedCodec[String] = new PackedCodec[String] {
    def write(value: String, out: ByteBuffer): Unit = {
      val bytes = value.getBytes(UTF_8)
      out.putInt(bytes.length)
      out.put(bytes)
    }
    def read(in: ByteBuffer): String = {
      val length = PackedCodec.readCount(in)
      if (length > in.remaining) PackedCodec.fail(s"Expected $length bytes of text, got ${in.remaining}")
      val bytes = new Array[Byte](length)
      in.get(bytes)
      new String(bytes, UTF_8)
    }
  }

  val instant: PackedCodec[Instant] = new PackedCodec[Instant] {
    def write(value: Instant, out: ByteBuffer): Unit = {
      out.putLong(value.getEpochSecond)
      out.putInt(value.getNano)
    }
    def read(in: ByteBuffer): Instant = Instant.ofEpochSecond(in.getLong(), in.getInt().toLong)
  }
}

trait PrimitiveDerivation extends Deriver[PackedCodec] {
  override def derivePrimitive[A](
    primitiveType: PrimitiveType[A],
    typeId: TypeId[A],
    binding: Binding.Primitive[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  ): Lazy[PackedCodec[A]] = {
    val codec = primitiveType match {
      case _: PrimitiveType.Boolean => PackedPrimitives.boolean
      case _: PrimitiveType.Int     => PackedPrimitives.int
      case _: PrimitiveType.Long    => PackedPrimitives.long
      case _: PrimitiveType.Float   => PackedPrimitives.float
      case _: PrimitiveType.Double  => PackedPrimitives.double
      case _: PrimitiveType.String  => PackedPrimitives.string
      case _: PrimitiveType.Instant => PackedPrimitives.instant
      case _                        =>
        throw new UnsupportedOperationException(s"The packed format has no codec for ${typeId.name}")
    }
    Lazy(codec.asInstanceOf[PackedCodec[A]])
  }
}

trait RecordDerivation extends PrimitiveDerivation {
  override def deriveRecord[F[_, _], A](
    fields: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Record[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[PackedCodec[A]] = {
    val names         = fields.map(_.name).toArray
    val registers     = Reflect.Record.registers(fields.map(_.value).toArray)
    val usedRegisters = Reflect.Record.usedRegisters(registers)
    Lazy {
      new PackedCodec[A] {
        private lazy val fieldCodecs: Array[PackedCodec[Any]] =
          fields.map(field => D.instance(field.value.metadata).force.asInstanceOf[PackedCodec[Any]]).toArray

        def write(value: A, out: ByteBuffer): Unit = {
          val regs = Registers(usedRegisters)
          binding.deconstructor.deconstruct(regs, RegisterOffset.Zero, value)
          fieldCodecs.indices.foreach { i =>
            fieldCodecs(i).write(registers(i).get(regs, RegisterOffset.Zero), out)
          }
        }

        def read(in: ByteBuffer): A = {
          val regs = Registers(usedRegisters)
          fieldCodecs.indices.foreach { i =>
            val fieldValue =
              try fieldCodecs(i).read(in)
              catch { case e if NonFatal(e) => throw PackedCodec.toSchemaError(e).atField(names(i)) }
            registers(i).set(regs, RegisterOffset.Zero, fieldValue)
          }
          binding.constructor.construct(regs, RegisterOffset.Zero)
        }
      }
    }
  }
}

trait VariantDerivation extends RecordDerivation {
  override def deriveVariant[F[_, _], A](
    cases: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Variant[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[PackedCodec[A]] = {
    val names = cases.map(_.name).toArray
    Lazy {
      new PackedCodec[A] {
        private lazy val caseCodecs: Array[PackedCodec[A]] =
          cases.map(c => D.instance(c.value.metadata).force.asInstanceOf[PackedCodec[A]]).toArray

        def write(value: A, out: ByteBuffer): Unit = {
          val index = binding.discriminator.discriminate(value)
          out.putInt(index)
          caseCodecs(index).write(value, out)
        }

        def read(in: ByteBuffer): A = {
          val index = in.getInt()
          if (index < 0 || index >= caseCodecs.length)
            PackedCodec.fail(s"Expected a case index from 0 to ${caseCodecs.length - 1}, got $index")
          try caseCodecs(index).read(in)
          catch { case e if NonFatal(e) => throw PackedCodec.toSchemaError(e).atCase(names(index)) }
        }
      }
    }
  }
}

trait CollectionDerivation extends VariantDerivation {
  override def deriveSequence[F[_, _], C[_], A](
    element: Reflect[F, A],
    typeId: TypeId[C[A]],
    binding: Binding.Seq[C, A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[C[A]],
    examples: Seq[C[A]]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[PackedCodec[C[A]]] = {
    implicit val elementClassTag: ClassTag[A] = element.typeId.classTag.asInstanceOf[ClassTag[A]]
    D.instance(element.metadata).map { elementCodec =>
      new PackedCodec[C[A]] {
        def write(value: C[A], out: ByteBuffer): Unit = {
          out.putInt(binding.deconstructor.size(value))
          binding.deconstructor.deconstruct(value).foreach(elementCodec.write(_, out))
        }

        def read(in: ByteBuffer): C[A] = {
          val count   = PackedCodec.readCount(in)
          val builder = binding.constructor.newBuilder[A](math.min(count, in.remaining))
          (0 until count).foreach { i =>
            val elem =
              try elementCodec.read(in)
              catch { case e if NonFatal(e) => throw PackedCodec.toSchemaError(e).atIndex(i) }
            binding.constructor.add(builder, elem)
          }
          binding.constructor.result(builder)
        }
      }
    }
  }

  override def deriveMap[F[_, _], M[_, _], K, V](
    key: Reflect[F, K],
    value: Reflect[F, V],
    typeId: TypeId[M[K, V]],
    binding: Binding.Map[M, K, V],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[M[K, V]],
    examples: Seq[M[K, V]]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[PackedCodec[M[K, V]]] =
    D.instance(key.metadata).zip(D.instance(value.metadata)).map { case (keyCodec, valueCodec) =>
      new PackedCodec[M[K, V]] {
        def write(map: M[K, V], out: ByteBuffer): Unit = {
          val entries = binding.deconstructor
          out.putInt(entries.size(map))
          entries.deconstruct(map).foreach { entry =>
            keyCodec.write(entries.getKey(entry), out)
            valueCodec.write(entries.getValue(entry), out)
          }
        }

        def read(in: ByteBuffer): M[K, V] = {
          val count   = PackedCodec.readCount(in)
          val builder = binding.constructor.newObjectBuilder[K, V](math.min(count, in.remaining))
          (0 until count).foreach { i =>
            val k =
              try keyCodec.read(in)
              catch { case e if NonFatal(e) => throw PackedCodec.toSchemaError(e).atIndex(i) }
            val v =
              try valueCodec.read(in)
              catch { case e if NonFatal(e) => throw PackedCodec.toSchemaError(e).atKey(key.toDynamicValue(k)) }
            binding.constructor.addObject(builder, k, v)
          }
          binding.constructor.resultObject(builder)
        }
      }
    }
}

object PackedCodecDeriver extends CollectionDerivation {
  override def deriveWrapper[F[_, _], A, B](
    wrapped: Reflect[F, B],
    typeId: TypeId[A],
    binding: Binding.Wrapper[A, B],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[PackedCodec[A]] =
    D.instance(wrapped.metadata).map { inner =>
      new PackedCodec[A] {
        def write(value: A, out: ByteBuffer): Unit = inner.write(binding.unwrap(value), out)
        def read(in: ByteBuffer): A                = binding.wrap(inner.read(in))
      }
    }

  override def deriveDynamic[F[_, _]](
    binding: Binding.Dynamic,
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[DynamicValue],
    examples: Seq[DynamicValue]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[PackedCodec[DynamicValue]] =
    throw new UnsupportedOperationException("The packed format does not encode DynamicValue")
}

object PackedFormat extends BinaryFormat[PackedCodec]("application/x-packed", PackedCodecDeriver)
```

</details>

<details>
  <summary>Runnable version, with the model and the format from the shared files: <code>Step10RoundTripExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/customwireformat/Step10RoundTripExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using file SensorModel.scala PackedFormat.scala
// Run from this folder: scala-cli run Step10RoundTripExample.scala
// Or from schema-examples: sbt "runMain customwireformat.Step10RoundTripExample"

package customwireformat

/**
 * How to Write a Codec for a New Wire Format — Deriving Codecs and Checking the
 * Round Trip
 *
 * Derives the packed codec of Batch through PackedFormat, encodes the sample
 * batch, decodes it back and checks that every reading round-trips. The model
 * and the format come from the shared files.
 */
object Step10RoundTripExample {
  def main(args: Array[String]): Unit = {
    val batchCodec: PackedCodec[Batch] = Batch.schema.derive(PackedFormat)
    val batchBytes: ByteBuffer         = batchCodec.encodeToBuffer(Batch.sample)

    println(s"${PackedFormat.mimeType}: ${batchBytes.remaining} bytes")
    println(batchCodec.decode(batchBytes.duplicate()))

    def roundTrips[A](codec: PackedCodec[A], value: A): Boolean = {
      val bytes = codec.encodeToBuffer(value)
      codec.decode(bytes) == Right(value) && !bytes.hasRemaining
    }

    val readingCodec: PackedCodec[Reading] = Reading.schema.derive(PackedFormat)
    Batch.sample.readings.foreach { reading =>
      println(s"${reading.measurement} round-trips: ${roundTrips(readingCodec, reading)}")
    }
  }
}
```

</details>

## Reporting Errors with Paths

Every `try` around a child's `PackedCodec#read` adds one segment to the error's path, so a decode error names the exact place where it happens. A message cut off after 70 bytes fails inside the second reading:

```scala
val truncated: ByteBuffer = batchCodec.encodeToBuffer(Batch.sample)
truncated.limit(70)
```

The error says where the bytes ran out:

```scala
println(batchCodec.decode(truncated))
// Left(zio.blocks.schema.SchemaError: Unexpected end of input at: .readings.at(1).takenAt)
```

The encoder does not validate, so a sender built without the `DeviceId` check can send an id that is too long:

```scala
val longIdBatch = Batch.sample.copy(readings = List(Batch.sample.readings.head.copy(device = DeviceId("north-wing-probe-07"))))
```

The receiver's decoder runs the check through `Binding.Wrapper#wrap`, and the record adds the path:

```scala
println(batchCodec.decode(batchCodec.encodeToBuffer(longIdBatch)))
// Left(zio.blocks.schema.SchemaError: device id must be 1 to 16 characters, got 19 at: .readings.at(0).device)
```

In the first reading, the case index of `measurement` starts at byte 20, after the device id (8 bytes) and the timestamp (12 bytes). Overwriting it with an index the variant does not have gives an error at that field:

```scala
val corrupted: ByteBuffer = readingCodec.encodeToBuffer(Batch.sample.readings.head)
corrupted.putInt(20, 7)
```

The variant rejects the index before reading any case:

```scala
println(readingCodec.decode(corrupted))
// Left(zio.blocks.schema.SchemaError: Expected a case index from 0 to 2, got 7 at: .measurement)
```

An unsupported type fails earlier, when its codec is derived. The packed format has no codec for `UUID`, so a firmware update message with a `UUID` field cannot get one:

```scala
final case class FirmwareUpdate(device: DeviceId, release: UUID)

object FirmwareUpdate {
  implicit val schema: Schema[FirmwareUpdate] = Schema.derived
}

println(Try(FirmwareUpdate.schema.derive(PackedFormat)))
// Failure(java.lang.UnsupportedOperationException: The packed format has no codec for UUID)
```

<details>
  <summary>Runnable version, with the codec of <code>Batch</code> from the round-trip section: <code>Step11ErrorsExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/customwireformat/Step11ErrorsExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using file SensorModel.scala PackedFormat.scala
// Run from this folder: scala-cli run Step11ErrorsExample.scala
// Or from schema-examples: sbt "runMain customwireformat.Step11ErrorsExample"

package customwireformat

final case class FirmwareUpdate(device: DeviceId, release: UUID)

object FirmwareUpdate {
  implicit val schema: Schema[FirmwareUpdate] = Schema.derived
}

/**
 * How to Write a Codec for a New Wire Format — Reporting Errors with Paths
 *
 * Decodes a cut-off batch, a batch with an invalid device id and a reading with
 * a corrupted case index, and prints each error with its path. Then derives a
 * codec for a type with a UUID field, which the format rejects.
 */
object Step11ErrorsExample {
  def main(args: Array[String]): Unit = {
    // Setup from section 10: the codec of Batch
    val batchCodec: PackedCodec[Batch] = Batch.schema.derive(PackedFormat)

    val truncated: ByteBuffer = batchCodec.encodeToBuffer(Batch.sample)
    truncated.limit(70)
    println(batchCodec.decode(truncated))

    val longIdBatch =
      Batch.sample.copy(readings = List(Batch.sample.readings.head.copy(device = DeviceId("north-wing-probe-07"))))
    println(batchCodec.decode(batchCodec.encodeToBuffer(longIdBatch)))

    val readingCodec: PackedCodec[Reading] = Reading.schema.derive(PackedFormat)
    val corrupted: ByteBuffer              = readingCodec.encodeToBuffer(Batch.sample.readings.head)
    corrupted.putInt(20, 7)
    println(readingCodec.decode(corrupted))

    println(Try(FirmwareUpdate.schema.derive(PackedFormat)))
  }
}
```

</details>

## Encoding Recursive Types

A `Zone` holds a list of `Zone`s, so its schema refers to itself. The deriver needs no extra code: every codec looks up its children on first use, through a `lazy val` or a `Lazy`, so by the time the `subzones` codec asks for the `Zone` codec, that codec already exists. A site with two floors and a cold room on the first floor round-trips:

```scala
val site = Zone(
  "north-site",
  Vector(DeviceId("gw-north")),
  List(
    Zone("floor-1", Vector(DeviceId("t-17"), DeviceId("h-02")), List(Zone("cold-room", Vector(DeviceId("t-18")), Nil))),
    Zone("floor-2", Vector(DeviceId("d-01")), Nil)
  )
)
val zoneCodec: PackedCodec[Zone] = Zone.schema.derive(PackedFormat)
```

Decoding the encoded site gives back the whole tree:

```scala
println(zoneCodec.decode(zoneCodec.encodeToBuffer(site)))
// Right(Zone(north-site,Vector(DeviceId(gw-north)),List(Zone(floor-1,Vector(DeviceId(t-17), DeviceId(h-02)),List(Zone(cold-room,Vector(DeviceId(t-18)),List()))), Zone(floor-2,Vector(DeviceId(d-01)),List()))))
```

:::warning
Never force a child's instance inside a `derive*` method itself, for example with `D.instance(field.value.metadata).force` outside the codec. For a recursive type, that derives the same type again, without end, until the stack overflows.
:::

<details>
  <summary>Runnable version, with the model and the format from the shared files: <code>Step12RecursiveTypesExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/customwireformat/Step12RecursiveTypesExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using file SensorModel.scala PackedFormat.scala
// Run from this folder: scala-cli run Step12RecursiveTypesExample.scala
// Or from schema-examples: sbt "runMain customwireformat.Step12RecursiveTypesExample"

package customwireformat

/**
 * How to Write a Codec for a New Wire Format — Encoding Recursive Types
 *
 * Derives the packed codec of the recursive Zone tree and round-trips a site
 * with nested zones.
 */
object Step12RecursiveTypesExample {
  def main(args: Array[String]): Unit = {
    val site = Zone(
      "north-site",
      Vector(DeviceId("gw-north")),
      List(
        Zone(
          "floor-1",
          Vector(DeviceId("t-17"), DeviceId("h-02")),
          List(Zone("cold-room", Vector(DeviceId("t-18")), Nil))
        ),
        Zone("floor-2", Vector(DeviceId("d-01")), Nil)
      )
    )
    val zoneCodec: PackedCodec[Zone] = Zone.schema.derive(PackedFormat)

    println(zoneCodec.decode(zoneCodec.encodeToBuffer(site)))
  }
}
```

</details>

## Overriding One Type with `Deriver#withInstance`

Because the format's deriver is an ordinary `Deriver`, users of the format get the same override mechanism as with the built-in formats, without any work from the format's author. Suppose a gateway's firmware sends timestamps as epoch milliseconds. A hand-written `PackedCodec[Instant]` registered with `Deriver#withInstance` replaces the derived one for every `Instant`, at any depth:

```scala
val epochMillis: PackedCodec[Instant] = new PackedCodec[Instant] {
  def write(value: Instant, out: ByteBuffer): Unit = out.putLong(value.toEpochMilli)
  def read(in: ByteBuffer): Instant                = Instant.ofEpochMilli(in.getLong())
}
val millisDeriver: Deriver[PackedCodec]       = PackedFormat.deriver.withInstance[Instant](epochMillis)
val compactReadingCodec: PackedCodec[Reading] = Reading.schema.derive(millisDeriver)

val firstReading = Batch.sample.readings.head
```

Each timestamp now takes 8 bytes instead of 12, and the reading still decodes:

```scala
println(s"default: ${Reading.schema.derive(PackedFormat).encodeToBuffer(firstReading).remaining} bytes")
// default: 40 bytes
println(s"epoch millis: ${compactReadingCodec.encodeToBuffer(firstReading).remaining} bytes")
// epoch millis: 36 bytes
println(compactReadingCodec.decode(compactReadingCodec.encodeToBuffer(firstReading)))
// Right(Reading(DeviceId(t-17),2026-10-29T08:00:00Z,Temperature(21.5),Some(87)))
```

The override drops sub-millisecond precision, which these timestamps do not have. `Schema#derive(PackedFormat)`, `Schema#getInstance`, `Schema#encode` and `Schema#decode` keep using the format's default deriver, so pass the configured deriver to `Schema#derive` wherever the override should apply.

<details>
  <summary>Runnable version, with the model and the format from the shared files: <code>Step13OverrideExample.scala</code></summary>

```scala title="schema-examples/src/main/scala/customwireformat/Step13OverrideExample.scala" showLineNumbers
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using file SensorModel.scala PackedFormat.scala
// Run from this folder: scala-cli run Step13OverrideExample.scala
// Or from schema-examples: sbt "runMain customwireformat.Step13OverrideExample"

package customwireformat

/**
 * How to Write a Codec for a New Wire Format — Overriding One Type with
 * withInstance
 *
 * Replaces the packed codec of Instant with an epoch-milliseconds codec through
 * Deriver#withInstance, and compares the encoded size of one reading.
 */
object Step13OverrideExample {
  def main(args: Array[String]): Unit = {
    val epochMillis: PackedCodec[Instant] = new PackedCodec[Instant] {
      def write(value: Instant, out: ByteBuffer): Unit = out.putLong(value.toEpochMilli)
      def read(in: ByteBuffer): Instant                = Instant.ofEpochMilli(in.getLong())
    }
    val millisDeriver: Deriver[PackedCodec]       = PackedFormat.deriver.withInstance[Instant](epochMillis)
    val compactReadingCodec: PackedCodec[Reading] = Reading.schema.derive(millisDeriver)

    val firstReading = Batch.sample.readings.head
    println(s"default: ${Reading.schema.derive(PackedFormat).encodeToBuffer(firstReading).remaining} bytes")
    println(s"epoch millis: ${compactReadingCodec.encodeToBuffer(firstReading).remaining} bytes")
    println(compactReadingCodec.decode(compactReadingCodec.encodeToBuffer(firstReading)))
  }
}
```

</details>

## Putting It Together

The complete example runs a gateway's batch through the packed format: it derives the codecs, encodes and decodes the batch, rejects a cut-off message with a path, round-trips the zone tree and sends timestamps as epoch milliseconds. It differs from the sections in three ways: it cuts the already encoded batch to 70 bytes with `ByteBuffer#duplicate` instead of encoding it again, it prints only whether the zone tree round-trips instead of the whole tree, and it applies the epoch-milliseconds override to the whole `Batch` instead of one `Reading`. The format and the model come from `PackedFormat.scala` and `SensorModel.scala`:

```scala title="schema-examples/src/main/scala/customwireformat/CompleteExample.scala"
//> using scala 3.9.0
//> using options -experimental
//> using dep dev.zio::zio-blocks-schema:0.0.56
//> using file SensorModel.scala PackedFormat.scala
// Run from this folder: scala-cli run CompleteExample.scala
// Or from schema-examples: sbt "runMain customwireformat.CompleteExample"

package customwireformat

/**
 * How to Write a Codec for a New Wire Format — Complete Example
 *
 * A gateway's batch goes through the packed format: derive the codecs, encode
 * the batch, decode it back, reject a cut-off message with a path, round-trip
 * the recursive zone tree and send timestamps as epoch milliseconds.
 */
object CompleteExample {
  def main(args: Array[String]): Unit = {
    val batchCodec: PackedCodec[Batch] = Batch.schema.derive(PackedFormat)
    val batchBytes: ByteBuffer         = batchCodec.encodeToBuffer(Batch.sample)
    println(s"${PackedFormat.mimeType}: ${batchBytes.remaining} bytes")
    println(batchCodec.decode(batchBytes.duplicate()))

    val truncated = batchBytes.duplicate()
    truncated.limit(70)
    println(batchCodec.decode(truncated))

    val site = Zone(
      "north-site",
      Vector(DeviceId("gw-north")),
      List(
        Zone(
          "floor-1",
          Vector(DeviceId("t-17"), DeviceId("h-02")),
          List(Zone("cold-room", Vector(DeviceId("t-18")), Nil))
        ),
        Zone("floor-2", Vector(DeviceId("d-01")), Nil)
      )
    )
    val zoneCodec: PackedCodec[Zone] = Zone.schema.derive(PackedFormat)
    println(s"zone tree round-trips: ${zoneCodec.decode(zoneCodec.encodeToBuffer(site)) == Right(site)}")

    val epochMillis: PackedCodec[Instant] = new PackedCodec[Instant] {
      def write(value: Instant, out: ByteBuffer): Unit = out.putLong(value.toEpochMilli)
      def read(in: ByteBuffer): Instant                = Instant.ofEpochMilli(in.getLong())
    }
    val millisDeriver: Deriver[PackedCodec]   = PackedFormat.deriver.withInstance[Instant](epochMillis)
    val compactBatchCodec: PackedCodec[Batch] = Batch.schema.derive(millisDeriver)
    println(s"epoch millis: ${compactBatchCodec.encodeToBuffer(Batch.sample).remaining} bytes")
  }
}
```

Running it prints:

```text
application/x-packed: 156 bytes
Right(Batch(gw-north,List(Reading(DeviceId(t-17),2026-10-29T08:00:00Z,Temperature(21.5),Some(87)), Reading(DeviceId(h-02),2026-10-29T08:00:05Z,Humidity(41.0),None), Reading(DeviceId(d-01),2026-10-29T08:00:09Z,DoorOpened,Some(12))),Map(t-17 -> -0.4, h-02 -> 1.5)))
Left(zio.blocks.schema.SchemaError: Unexpected end of input at: .readings.at(1).takenAt)
zone tree round-trips: true
epoch millis: 144 bytes
```

## Running the Examples

All runnable code from this guide is in [`schema-examples/src/main/scala/customwireformat/`](https://github.com/zio/zio-blocks/tree/main/schema-examples/src/main/scala/customwireformat). Run any file from that folder with `scala-cli run <File>.scala`, or from `schema-examples` with `sbt "runMain customwireformat.<ObjectName>"`.

## Going Further

The packed format is deliberately small. A production format usually adds the following, and the built-in formats show how:

- **Schema evolution.** The format writes no field names or tags, so adding, removing or reordering a field breaks every reader that uses the old schema. Avro writes a schema next to the data, and Protobuf numbers its fields; see [Schema Evolution](../reference/schema/schema-evolution/index.md) for what ZIO Blocks offers on the schema side.
- **Modifiers and defaults.** The deriver ignores the `modifiers` and `defaultValue` it receives, so `Modifier.rename` and default values change nothing. A format with field names reads them here, as `JsonCodecDeriver` does.
- **Performance.** Every field passes through `Register[Any]`, which boxes primitive values, and every element of a primitive array is boxed too. The built-in codecs write specialized code for each primitive type and for primitive arrays.
- **Buffer sizing.** `PackedCodec#encodeToBuffer` allocates a fixed capacity. A real format computes the encoded size first or writes to a growing buffer.
- **The remaining primitives and `DynamicValue`.** The format supports seven primitive types and no `DynamicValue`.

The pages below cover the pieces this guide builds on:

- The Avro module is the full-scale version of this guide: [`AvroCodecDeriver.scala`](https://github.com/zio/zio-blocks/blob/main/schema-avro/src/main/scala/zio/blocks/schema/avro/AvroCodecDeriver.scala) handles every primitive type, specializes primitive collections and encodes `DynamicValue`.
- [Type-Class Derivation](../reference/schema/type-class-derivation.md) explains every `Deriver` method and its parameters, and how instance and modifier overrides combine.
- [Register System](../reference/schema/registers.md) and [Binding](../reference/schema/binding.md) document the storage and the constructors that record codecs use.
- [SchemaError](../reference/schema/schema-error.md) lists the error kinds and the `at*` methods that build paths.
- [Codec](../reference/schema/codec.md) and [Format](../reference/schema/format.md) cover using codecs once you have them.
- [How to Override the Codec for One Type Everywhere](./override-codec-for-one-type.md) replaces one type's codec inside an existing format, without writing a new format.

---

## How to Write an OpenAPI Document and Export It as JSON or YAML

In this guide, we will write the OpenAPI 3.1 document for a small bookstore API as a typed Scala value. By the end, you will have one program that builds the document with the `OpenAPI` data model, generates the `Book` schema from the same `Schema` that encodes books as JSON, and prints the document as JSON and as YAML, ready for Swagger UI or a client generator.

The bookstore API has two endpoints: `GET /books?limit=` lists books, and `GET /books/{id}` returns one book, or `404` when there is no such book.

Every request must carry a bearer token. We start with the schemas of the request and response bodies, then describe the endpoints and the security scheme, and finish by exporting the document in both formats.

## The Problem

An OpenAPI document repeats your data types. The `Book` that your service encodes as JSON also appears in the document as a JSON Schema, and the two are written separately. When a field is renamed or becomes optional, the case class changes and the document doesn't. Swagger UI then shows the old shape, and clients generated from the document send or expect JSON that the service no longer accepts.

A hand-written document is also just text. A misspelled keyword such as `"requried"`, a `$ref` that points to a missing component, or a path parameter without `"required": true` is not an error anywhere until a tool reads the file. Here is what such a document often looks like inside a Scala service:

```scala
case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String])

// Kept in sync with Book by hand. Nothing checks that it still matches.
val bookstoreSpec: String =
  """{
    |  "openapi": "3.1.0",
    |  "info": { "title": "Bookstore API", "version": "1.0.0" },
    |  "paths": {
    |    "/books/{id}": {
    |      "get": {
    |        "parameters": [
    |          { "name": "id", "in": "path", "required": true, "schema": { "type": "string" } }
    |        ],
    |        "responses": {
    |          "200": {
    |            "description": "The book",
    |            "content": {
    |              "application/json": { "schema": { "$ref": "#/components/schemas/Book" } }
    |            }
    |          },
    |          "404": { "description": "No book has this ID" }
    |        }
    |      }
    |    }
    |  },
    |  "components": {
    |    "schemas": {
    |      "Book": {
    |        "type": "object",
    |        "properties": {
    |          "title": { "type": "string" },
    |          "firstAuthor": { "type": "string" },
    |          "year": { "type": "integer" },
    |          "isbn": { "type": ["null", "string"] }
    |        },
    |        "required": ["title", "firstAuthor", "year"]
    |      }
    |    }
    |  }
    |}""".stripMargin
```

This string compiles whatever it contains. It covers one endpoint so far, and the `Book` schema inside it is a second copy of the case class above it.

The ZIO Blocks `openapi` module replaces the string with a typed model of the OpenAPI 3.1 document, and generates the schema of each body type from its `Schema`. The compiler checks the structure of the document, and the body schemas follow the case classes automatically.

## Prerequisites

Add the `openapi` module, and the `schema-yaml` module for the YAML export, to your `build.sbt`:

```scala
// Scala 2.13, and Scala 3.3 to 3.8
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-openapi"     % "0.0.56",
  "dev.zio" %% "zio-blocks-schema-yaml" % "0.0.56"
)

// Scala 3.9 and newer
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-blocks-next-openapi"     % "0.0.56",
  "dev.zio" %% "zio-blocks-next-schema-yaml" % "0.0.56"
)
scalacOptions += "-experimental"
```

Both modules are built for the JVM and Scala.js; use `%%%` instead of `%%` in a Scala.js project. The `openapi` module brings in `schema`, so `Schema` and `Json` are available without another dependency.

You should know how to derive a `Schema` for a case class (see [Your First Schema](./your-first-schema.md)). The component schemas in an OpenAPI 3.1 document are JSON Schemas, and they come from the same derivation as in [How to Generate JSON Schema from a Type and Validate Payloads Against It](./generate-json-schema-and-validate-payloads.md).

:::note
You write the paths, operations and responses of the document yourself. The module does not generate a document from endpoint or route definitions; what it generates for you is the JSON Schema of each request and response body.
:::

Throughout the guide we use these imports:

```scala

```

`zio.blocks.docs._` provides the `md"..."` interpolator for descriptions, which OpenAPI treats as Markdown. `OpenAPICodec._` provides `openAPICodec`, the codec that writes the document in the OpenAPI JSON format.

## The Bookstore Types

The API returns books, and an error body when a book is not found. Both are ordinary case classes with a derived `Schema`, the same ones the service would use to encode its JSON responses:

```scala
final case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String])

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

final case class ApiError(message: String)

object ApiError {
  implicit val schema: Schema[ApiError] = Schema.derived
}

val sampleBooks: List[Book] = List(
  Book("Zionomicon", "John A. De Goes", 2024, None),
  Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601")),
  Book("Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
)
```

`sampleBooks` gives the document a realistic example response later on.

## Generating Component Schemas From Your Types

`Schema#toOpenAPISchema` turns a `Schema` into a `SchemaObject`, the OpenAPI type for a schema. A `SchemaObject` wraps the JSON Schema that `Schema#toJsonSchema` derives, so it describes exactly the JSON that the `Book` codec reads and writes:

```scala
println(Schema[Book].toOpenAPISchema.toJson.print(WriterConfig.withIndentionStep2))
// {
//   "properties": {
//     "title": {
//       "type": "string"
//     },
//     "firstAuthor": {
//       "type": "string"
//     },
//     "year": {
//       "type": "integer",
//       "maximum": 2147483647,
//       "minimum": -2147483648
//     },
//     "isbn": {
//       "type": [
//         "null",
//         "string"
//       ]
//     }
//   },
//   "type": "object",
//   "required": [
//     "title",
//     "firstAuthor",
//     "year"
//   ],
//   "title": "Book"
// }
```

The schema follows the codec's rules: `isbn` is an `Option`, so it accepts `null` and is not in `"required"`, and `year` is bounded to the `Int` range. If you add a field to `Book`, the schema changes with it.

A document usually defines each body type once, under `components/schemas`, and points to it with a `$ref` wherever it is used. `Schema#toRefSchema` returns that reference, named after the type, together with the component entry:

```scala
val (bookRef, _)  = Schema[Book].toRefSchema
val (errorRef, _) = Schema[ApiError].toRefSchema
```

To see the reference that `Schema#toRefSchema` returned:

```scala
println(bookRef)
// Ref(Reference(#/components/schemas/Book,None,None))
```

`ReferenceOr[A]` is the type for every place where OpenAPI accepts either a value of `A` or a `$ref` to one: `ReferenceOr.Ref` holds a reference and `ReferenceOr.Value` holds the value itself.

For the `components` section, `OpenAPIGen.schemas` builds the entries for several types at once, keyed by the type's simple name:

```scala
val componentSchemas: Map[String, SchemaObject] = OpenAPIGen.schemas(Schema[Book], Schema[ApiError])
```

To list the component names that were generated:

```scala
println(componentSchemas.keys)
// Set(Book, ApiError)
```

:::warning
`OpenAPIGen.schemas` and `Schema#toRefSchema` name a component after the type's simple name, without its package. Two types with the same simple name in different packages get the same key, and one silently replaces the other. Use `Schema#toRefSchema("CatalogBook")` to choose a name yourself.
:::

`GET /books` returns a list of books. `Schema[List[Book]].toOpenAPISchema` would copy the whole `Book` schema into the response, so we write the small array schema ourselves and point its `items` at the `Book` component. `SchemaObject` takes any JSON Schema as a `Json` value:

```scala
val bookListSchema: ReferenceOr[SchemaObject] = ReferenceOr.Value(
  SchemaObject(
    jsonSchema = Json.Object(
      "type"  -> Json.String("array"),
      "items" -> Json.Object("$ref" -> Json.String("#/components/schemas/Book"))
    )
  )
)
```

## Describing the Endpoints

An endpoint's inputs are `Parameter` values. `limit` is an optional query parameter, and `id` is part of the path. Each parameter's schema comes from a `Schema` as well; `Schema#toInlineSchema` puts the schema in place instead of referencing a component:

```scala
val limitParam: Parameter = Parameter(
  name = "limit",
  in = ParameterLocation.Query,
  description = Some(md"Maximum number of books to return."),
  schema = Some(Schema[Int].toInlineSchema)
)

val idParam: Parameter = Parameter(
  name = "id",
  in = ParameterLocation.Path,
  required = true,
  description = Some(md"Catalog ID of the book."),
  schema = Some(Schema[String].toInlineSchema)
)
```

:::warning
OpenAPI requires every path parameter to be marked as required, but `Parameter` defaults `required` to `false` for every location and does not check it. Pass `required = true` for each `ParameterLocation.Path` parameter yourself.
:::

A `Response` has a Markdown description and a map from media type to `MediaType`, which holds the body's schema and an optional example. `Json.from` encodes the sample books with their `Schema`, so the example in the document is real output of the `Book` codec:

```scala
val booksFound: Response = Response(
  description = md"The books in the catalog.",
  content = ChunkMap(
    "application/json" -> MediaType(schema = Some(bookListSchema), example = Some(Json.from(sampleBooks)))
  )
)

val bookFound: Response = Response(
  description = md"The book.",
  content = ChunkMap("application/json" -> MediaType(schema = Some(bookRef)))
)

val bookNotFound: Response = Response(
  description = md"No book has this ID.",
  content = ChunkMap("application/json" -> MediaType(schema = Some(errorRef)))
)
```

`Paths` maps each path to a `PathItem`, which has one optional `Operation` per HTTP method. Status codes are strings, as in the OpenAPI format. A parameter that every method of a path shares, like `id`, goes on the `PathItem` instead of on each operation:

```scala
val paths: Paths = Paths(
  ChunkMap(
    "/books" -> PathItem(
      get = Some(
        Operation(
          operationId = Some("listBooks"),
          parameters = Chunk(ReferenceOr.Value(limitParam)),
          responses = Responses(ChunkMap("200" -> ReferenceOr.Value(booksFound)))
        )
      )
    ),
    "/books/{id}" -> PathItem(
      parameters = Chunk(ReferenceOr.Value(idParam)),
      get = Some(
        Operation(
          operationId = Some("getBook"),
          responses = Responses(
            ChunkMap(
              "200" -> ReferenceOr.Value(bookFound),
              "404" -> ReferenceOr.Value(bookNotFound)
            )
          )
        )
      )
    )
  )
)
```

The compiler now checks what the hand-written string could not: an `Operation` can only hold `Parameter` and `Response` values, a `ParameterLocation` is one of four cases, and `bookRef` comes from the same `Book` type that the `components` section is built from.

## Requiring a Bearer Token

A security scheme says how clients authenticate. `SecurityScheme.HTTP` with the `bearer` scheme describes an `Authorization: Bearer <token>` header; `bearerFormat` is a hint for readers and tools, here that the token is a JWT. Schemes live in `components`, next to the schemas:

```scala
val components: Components = Components(
  schemas = ChunkMap.from(componentSchemas.map { case (name, schema) => name -> ReferenceOr.Value(schema) }),
  securitySchemes = ChunkMap(
    "bearerAuth" -> ReferenceOr.Value(SecurityScheme.HTTP(scheme = "bearer", bearerFormat = Some("JWT")))
  )
)
```

Defining a scheme does not apply it. A `SecurityRequirement` names the schemes a request must satisfy, each with the list of scopes it needs; bearer tokens have no scopes in OpenAPI, so the list is empty. On the `OpenAPI` value, the requirement applies to every operation, and an `Operation` can override it with its own `security` field:

```scala
val requireToken: SecurityRequirement = SecurityRequirement(ChunkMap("bearerAuth" -> Chunk.empty))
```

The document only declares that a token is required. Issuing and checking tokens is the service's job; [How to Issue and Verify JWTs](./issue-and-verify-jwts.md) shows how to do it with the ZIO Blocks `jwt` module.

## Exporting the Document as JSON

The pieces now fit into one `OpenAPI` value. `openapi` is the version of the OpenAPI format, and `Info` is the API's title and version:

```scala
val api: OpenAPI = OpenAPI(
  openapi = "3.1.0",
  info = Info(title = "Bookstore API", version = "1.0.0"),
  paths = Some(paths),
  components = Some(components),
  security = Chunk(requireToken)
)
```

`openAPICodec.encodeValue` turns the document into a `Json` value in the OpenAPI format, and `Json#print` renders it. `Json#print` alone writes compact JSON on one line; `WriterConfig.withIndentionStep2` indents it for people to read:

```scala
val json: Json = openAPICodec.encodeValue(api)
```

To print the document as indented JSON:

```scala
println(json.print(WriterConfig.withIndentionStep2))
// {
//   "openapi": "3.1.0",
//   "info": {
//     "title": "Bookstore API",
//     "version": "1.0.0"
//   },
//   "paths": {
//     "/books": {
//       "get": {
//         "responses": {
//           "200": {
//             "description": "The books in the catalog.\n\n",
//             "content": {
//               "application/json": {
//                 "schema": {
//                   "type": "array",
//                   "items": {
//                     "$ref": "#/components/schemas/Book"
//                   }
//                 },
//                 "example": [
//                   {
//                     "title": "Zionomicon",
//                     "firstAuthor": "John A. De Goes",
//                     "year": 2024
//                   },
//                   {
//                     "title": "Programming in Scala",
//                     "firstAuthor": "Martin Odersky",
//                     "year": 2008,
//                     "isbn": "9780981531601"
//                   },
//                   {
//                     "title": "Functional Programming in Scala",
//                     "firstAuthor": "Paul Chiusano",
//                     "year": 2014,
//                     "isbn": "9781617290657"
//                   }
//                 ]
//               }
//             }
//           }
//         },
//         "operationId": "listBooks",
//         "parameters": [
//           {
//             "name": "limit",
//             "in": "query",
//             "description": "Maximum number of books to return.\n\n",
//             "schema": {
//               "type": "integer",
//               "maximum": 2147483647,
//               "minimum": -2147483648
//             }
//           }
//         ]
//       }
//     },
//     "/books/{id}": {
//       "get": {
//         "responses": {
//           "200": {
//             "description": "The book.\n\n",
//             "content": {
//               "application/json": {
//                 "schema": {
//                   "$ref": "#/components/schemas/Book"
//                 }
//               }
//             }
//           },
//           "404": {
//             "description": "No book has this ID.\n\n",
//             "content": {
//               "application/json": {
//                 "schema": {
//                   "$ref": "#/components/schemas/ApiError"
//                 }
//               }
//             }
//           }
//         },
//         "operationId": "getBook"
//       },
//       "parameters": [
//         {
//           "name": "id",
//           "in": "path",
//           "description": "Catalog ID of the book.\n\n",
//           "required": true,
//           "schema": {
//             "type": "string"
//           }
//         }
//       ]
//     }
//   },
//   "components": {
//     "schemas": {
//       "Book": {
//         "properties": {
//           "title": {
//             "type": "string"
//           },
//           "firstAuthor": {
//             "type": "string"
//           },
//           "year": {
//             "type": "integer",
//             "maximum": 2147483647,
//             "minimum": -2147483648
//           },
//           "isbn": {
//             "type": [
//               "null",
//               "string"
//             ]
//           }
//         },
//         "type": "object",
//         "required": [
//           "title",
//           "firstAuthor",
//           "year"
//         ],
//         "title": "Book"
//       },
//       "ApiError": {
//         "properties": {
//           "message": {
//             "type": "string"
//           }
//         },
//         "type": "object",
//         "required": [
//           "message"
//         ],
//         "title": "ApiError"
//       }
//     },
//     "securitySchemes": {
//       "bearerAuth": {
//         "type": "http",
//         "scheme": "bearer",
//         "bearerFormat": "JWT"
//       }
//     }
//   },
//   "security": [
//     {
//       "bearerAuth": [
//         
//       ]
//     }
//   ]
// }
```

This is the file that Swagger UI, Redoc or a client generator reads. Three details of the output come from the encoder and the printer:

- Optional fields you didn't set are left out, so the output contains only what the document says. The exceptions are the required ones: an `Operation` always prints `responses` (as `{}` when empty) and a `Response` always prints `description`.
- Descriptions are Markdown, and the renderer ends each paragraph with a blank line, so every description ends with `\n\n`. Markdown readers such as Swagger UI ignore the trailing newlines.
- The empty scope list of `bearerAuth` prints with a line of spaces between its brackets. It is still the empty array `[]` to every JSON reader.

:::warning
Always encode an `OpenAPI` value through `Json`, as above. In this release, `openAPICodec.encodeToString(api)` and `openAPICodec.encode(api)` throw `NotImplementedError`, and `openAPICodec.decode` on a `String` returns a `Left`, because the codec implements only the conversion to and from `Json`.
:::

To write the document to a file, print it and save the text with the JDK:

```scala

Files.writeString(Path.of("openapi.json"), json.print(WriterConfig.withIndentionStep2))
```

Reading a document back works the same way in reverse: `Json.parse` turns the text into `Json`, and `openAPICodec.decode` turns the `Json` into an `OpenAPI` value. Encoding the result again gives exactly the JSON we started from:

```scala
val jsonText: String = json.print(WriterConfig.withIndentionStep2)

val reloaded: Either[SchemaError, OpenAPI] = Json.parse(jsonText).flatMap(openAPICodec.decode)
```

To check that the reloaded document encodes to the same JSON:

```scala
println(reloaded.map(document => openAPICodec.encodeValue(document) == json))
// Right(true)
```

:::note
Compare documents by their JSON, as above, not with `==` on `OpenAPI` values. A description written with `md"..."` and the same description decoded from JSON hold the same text in two different classes, so `reloaded == Right(api)` is `false` even though the documents are the same.
:::

## Exporting the Document as YAML

Many teams keep their OpenAPI documents in YAML. `Yaml.fromJson` from the `schema-yaml` module converts the `Json` value into a YAML tree, and `Yaml#printPretty` renders it as a YAML document:

```scala
val rawYaml: String = Yaml.fromJson(json).printPretty
```

To print the YAML document:

```scala
println(rawYaml)
// ---
// openapi: "3.1.0"
// info:
//   title: Bookstore API
//   version: "1.0.0"
// paths:
//   /books:
//     get:
//       responses:
//         "200":
//           description: "The books in the catalog.\n\n"
//           content:
//             application/json:
//               schema:
//                 type: array
//                 items:
//                   $ref: #/components/schemas/Book
//               example:
//                 - title: Zionomicon
//                   firstAuthor: John A. De Goes
//                   year: 2024
//                 - title: Programming in Scala
//                   firstAuthor: Martin Odersky
//                   year: 2008
//                   isbn: "9780981531601"
//                 - title: Functional Programming in Scala
//                   firstAuthor: Paul Chiusano
//                   year: 2014
//                   isbn: "9781617290657"
//       operationId: listBooks
//       parameters:
//         - name: limit
//           in: query
//           description: "Maximum number of books to return.\n\n"
//           schema:
//             type: integer
//             maximum: 2147483647
//             minimum: -2147483648
//   /books/{id}:
//     get:
//       responses:
//         "200":
//           description: "The book.\n\n"
//           content:
//             application/json:
//               schema:
//                 $ref: #/components/schemas/Book
//         "404":
//           description: "No book has this ID.\n\n"
//           content:
//             application/json:
//               schema:
//                 $ref: #/components/schemas/ApiError
//       operationId: getBook
//     parameters:
//       - name: id
//         in: path
//         description: "Catalog ID of the book.\n\n"
//         required: true
//         schema:
//           type: string
// components:
//   schemas:
//     Book:
//       properties:
//         title:
//           type: string
//         firstAuthor:
//           type: string
//         year:
//           type: integer
//           maximum: 2147483647
//           minimum: -2147483648
//         isbn:
//           type:
//             - "null"
//             - string
//       type: object
//       required:
//         - title
//         - firstAuthor
//         - year
//       title: Book
//     ApiError:
//       properties:
//         message:
//           type: string
//       type: object
//       required:
//         - message
//       title: ApiError
//   securitySchemes:
//     bearerAuth:
//       type: http
//       scheme: bearer
//       bearerFormat: JWT
// security:
//   - bearerAuth: []
```

Look at the `$ref` lines. The YAML writer leaves `#/components/schemas/Book` unquoted, and in YAML a `#` after a space starts a comment, so other YAML readers see `$ref:` with no value. Swagger UI and client generators would lose every reference. Until the writer quotes these values, quote them yourself after printing. Every `$ref` value in the output starts with `#`, so one replacement fixes them all:

```scala
def quoteRefs(yaml: String): String =
  yaml.replaceAll("(\\$ref: )(#.*)", "$1\"$2\"")

val yamlText: String = quoteRefs(rawYaml)
```

To show that every `$ref` value is now quoted:

```scala
yamlText.linesIterator.filter(_.contains("$ref")).foreach(println)
//                   $ref: "#/components/schemas/Book"
//                 $ref: "#/components/schemas/Book"
//                 $ref: "#/components/schemas/ApiError"
```

With the values quoted, `yamlText` is the YAML version of the JSON document above.

## Serving the Document to Swagger UI

Swagger UI is a static web page that renders an OpenAPI document as interactive documentation. It is not part of ZIO Blocks: you run it yourself, for example with the official Docker image, and give it the URL of your document. ZIO Blocks has no HTTP server either, so serve the JSON text from the server your service already uses, usually at `/openapi.json`. With the JDK's built-in server, it looks like this:

```scala

val body: Array[Byte] = json.print.getBytes(UTF_8)

val server: HttpServer = HttpServer.create(new InetSocketAddress(8080), 0)
server.createContext(
  "/openapi.json",
  exchange => {
    exchange.getResponseHeaders.set("Content-Type", "application/json")
    // Swagger UI runs on another origin, so the browser needs permission to fetch the document.
    exchange.getResponseHeaders.set("Access-Control-Allow-Origin", "*")
    exchange.sendResponseHeaders(200, body.length.toLong)
    exchange.getResponseBody.write(body)
    exchange.close()
  }
)
server.start()
```

Then start Swagger UI and point it at the document:

```bash
docker run -p 8081:8080 -e SWAGGER_JSON_URL=http://localhost:8080/openapi.json swaggerapi/swagger-ui
```

Open `http://localhost:8081` to see both endpoints, the `Book` and `ApiError` schemas, and an **Authorize** button for the bearer token.

## What the Module Does Not Do

The module describes and exports a document. It does not generate one from endpoints, and it does not validate one. The types check the structure but not the rules between parts, so a path parameter without `required = true` is accepted:

```scala
println(Parameter(name = "id", in = ParameterLocation.Path).required)
// false
```

The same holds for a `$ref` to a missing component and for a duplicate `operationId`. Run an OpenAPI validator on the exported file in CI to catch these. Only the body schemas come from your types; you describe paths, operations, parameters and responses yourself. To print or parse a document, convert through `Json`, because the codec has no direct text encoding.

## Putting It Together

This example combines everything from the guide in one program: it generates the component schemas, describes both endpoints and the bearer-token requirement, prints the document as JSON and as YAML with quoted references, and checks that the JSON reads back to the same document:

```scala title="openapi-examples/src/main/scala/writeandexportopenapi/CompleteExample.scala"
package writeandexportopenapi

/**
 * How to Write an OpenAPI Document and Export It as JSON or YAML: The Complete
 * Example
 *
 * Builds the component schemas, describes both endpoints and the bearer-token
 * requirement, prints the document as JSON and as YAML with quoted references,
 * and checks that the JSON reads back to the same document.
 *
 * Run with: cd openapi-examples && sbt "runMain
 * writeandexportopenapi.CompleteExample"
 */
object CompleteExample extends App {

  final case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  final case class ApiError(message: String)

  object ApiError {
    implicit val schema: Schema[ApiError] = Schema.derived
  }

  val sampleBooks: List[Book] = List(
    Book("Zionomicon", "John A. De Goes", 2024, None),
    Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601")),
    Book("Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
  )

  val (bookRef, _)  = Schema[Book].toRefSchema
  val (errorRef, _) = Schema[ApiError].toRefSchema

  val componentSchemas: Map[String, SchemaObject] = OpenAPIGen.schemas(Schema[Book], Schema[ApiError])

  val bookListSchema: ReferenceOr[SchemaObject] = ReferenceOr.Value(
    SchemaObject(
      jsonSchema = Json.Object(
        "type"  -> Json.String("array"),
        "items" -> Json.Object("$ref" -> Json.String("#/components/schemas/Book"))
      )
    )
  )

  val limitParam: Parameter = Parameter(
    name = "limit",
    in = ParameterLocation.Query,
    description = Some(md"Maximum number of books to return."),
    schema = Some(Schema[Int].toInlineSchema)
  )

  val idParam: Parameter = Parameter(
    name = "id",
    in = ParameterLocation.Path,
    required = true,
    description = Some(md"Catalog ID of the book."),
    schema = Some(Schema[String].toInlineSchema)
  )

  val booksFound: Response = Response(
    description = md"The books in the catalog.",
    content = ChunkMap(
      "application/json" -> MediaType(schema = Some(bookListSchema), example = Some(Json.from(sampleBooks)))
    )
  )

  val bookFound: Response = Response(
    description = md"The book.",
    content = ChunkMap("application/json" -> MediaType(schema = Some(bookRef)))
  )

  val bookNotFound: Response = Response(
    description = md"No book has this ID.",
    content = ChunkMap("application/json" -> MediaType(schema = Some(errorRef)))
  )

  val paths: Paths = Paths(
    ChunkMap(
      "/books" -> PathItem(
        get = Some(
          Operation(
            operationId = Some("listBooks"),
            parameters = Chunk(ReferenceOr.Value(limitParam)),
            responses = Responses(ChunkMap("200" -> ReferenceOr.Value(booksFound)))
          )
        )
      ),
      "/books/{id}" -> PathItem(
        parameters = Chunk(ReferenceOr.Value(idParam)),
        get = Some(
          Operation(
            operationId = Some("getBook"),
            responses = Responses(
              ChunkMap(
                "200" -> ReferenceOr.Value(bookFound),
                "404" -> ReferenceOr.Value(bookNotFound)
              )
            )
          )
        )
      )
    )
  )

  val components: Components = Components(
    schemas = ChunkMap.from(componentSchemas.map { case (name, schema) => name -> ReferenceOr.Value(schema) }),
    securitySchemes = ChunkMap(
      "bearerAuth" -> ReferenceOr.Value(SecurityScheme.HTTP(scheme = "bearer", bearerFormat = Some("JWT")))
    )
  )

  val requireToken: SecurityRequirement = SecurityRequirement(ChunkMap("bearerAuth" -> Chunk.empty))

  val api: OpenAPI = OpenAPI(
    openapi = "3.1.0",
    info = Info(title = "Bookstore API", version = "1.0.0"),
    paths = Some(paths),
    components = Some(components),
    security = Chunk(requireToken)
  )

  val json: Json = openAPICodec.encodeValue(api)

  println(json.print(WriterConfig.withIndentionStep2))

  val jsonText: String = json.print(WriterConfig.withIndentionStep2)

  val reloaded: Either[SchemaError, OpenAPI] = Json.parse(jsonText).flatMap(openAPICodec.decode)

  println(reloaded.map(document => openAPICodec.encodeValue(document) == json))

  val rawYaml: String = Yaml.fromJson(json).printPretty

  def quoteRefs(yaml: String): String =
    yaml.replaceAll("(\\$ref: )(#.*)", "$1\"$2\"")

  val yamlText: String = quoteRefs(rawYaml)

  println(yamlText)
}
```

## Running the Examples

All examples in this guide have runnable Scala files in the `openapi-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/openapi-examples
```

### Generate Component Schemas

This example prints the `SchemaObject` derived from `Book`, the `$ref` that `Schema#toRefSchema` returns, and the component names from `OpenAPIGen.schemas`.

<details>
  <summary>openapi-examples/src/main/scala/writeandexportopenapi/Step1ComponentSchemasExample.scala</summary>

```scala title="openapi-examples/src/main/scala/writeandexportopenapi/Step1ComponentSchemasExample.scala" showLineNumbers
package writeandexportopenapi

/**
 * How to Write an OpenAPI Document and Export It as JSON or YAML: Step 1 —
 * Generating Component Schemas From Your Types
 *
 * Prints the SchemaObject derived from Book, the $ref that toRefSchema returns,
 * and the component names from OpenAPIGen.schemas.
 *
 * Run with: cd openapi-examples && sbt "runMain
 * writeandexportopenapi.Step1ComponentSchemasExample"
 */
object Step1ComponentSchemasExample extends App {

  final case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  final case class ApiError(message: String)

  object ApiError {
    implicit val schema: Schema[ApiError] = Schema.derived
  }

  val sampleBooks: List[Book] = List(
    Book("Zionomicon", "John A. De Goes", 2024, None),
    Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601")),
    Book("Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
  )

  println(Schema[Book].toOpenAPISchema.toJson.print(WriterConfig.withIndentionStep2))

  val (bookRef, _)  = Schema[Book].toRefSchema
  val (errorRef, _) = Schema[ApiError].toRefSchema

  println(bookRef)

  val componentSchemas: Map[String, SchemaObject] = OpenAPIGen.schemas(Schema[Book], Schema[ApiError])

  println(componentSchemas.keys)
}
```

</details>

Run it from the `openapi-examples` directory:

```bash
sbt "runMain writeandexportopenapi.Step1ComponentSchemasExample"
```

### Describe the Endpoints

This example describes `GET /books` and `GET /books/{id}` with their parameters and responses, and prints the `paths` section of the document.

<details>
  <summary>openapi-examples/src/main/scala/writeandexportopenapi/Step2EndpointsExample.scala</summary>

```scala title="openapi-examples/src/main/scala/writeandexportopenapi/Step2EndpointsExample.scala" showLineNumbers
package writeandexportopenapi

/**
 * How to Write an OpenAPI Document and Export It as JSON or YAML: Step 2 —
 * Describing the Endpoints
 *
 * Describes GET /books and GET /books/{id} with their parameters and responses,
 * and prints the paths section of the document.
 *
 * Run with: cd openapi-examples && sbt "runMain
 * writeandexportopenapi.Step2EndpointsExample"
 */
object Step2EndpointsExample extends App {

  final case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  final case class ApiError(message: String)

  object ApiError {
    implicit val schema: Schema[ApiError] = Schema.derived
  }

  val sampleBooks: List[Book] = List(
    Book("Zionomicon", "John A. De Goes", 2024, None),
    Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601")),
    Book("Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
  )

  val (bookRef, _)  = Schema[Book].toRefSchema
  val (errorRef, _) = Schema[ApiError].toRefSchema

  val componentSchemas: Map[String, SchemaObject] = OpenAPIGen.schemas(Schema[Book], Schema[ApiError])

  val bookListSchema: ReferenceOr[SchemaObject] = ReferenceOr.Value(
    SchemaObject(
      jsonSchema = Json.Object(
        "type"  -> Json.String("array"),
        "items" -> Json.Object("$ref" -> Json.String("#/components/schemas/Book"))
      )
    )
  )

  val limitParam: Parameter = Parameter(
    name = "limit",
    in = ParameterLocation.Query,
    description = Some(md"Maximum number of books to return."),
    schema = Some(Schema[Int].toInlineSchema)
  )

  val idParam: Parameter = Parameter(
    name = "id",
    in = ParameterLocation.Path,
    required = true,
    description = Some(md"Catalog ID of the book."),
    schema = Some(Schema[String].toInlineSchema)
  )

  val booksFound: Response = Response(
    description = md"The books in the catalog.",
    content = ChunkMap(
      "application/json" -> MediaType(schema = Some(bookListSchema), example = Some(Json.from(sampleBooks)))
    )
  )

  val bookFound: Response = Response(
    description = md"The book.",
    content = ChunkMap("application/json" -> MediaType(schema = Some(bookRef)))
  )

  val bookNotFound: Response = Response(
    description = md"No book has this ID.",
    content = ChunkMap("application/json" -> MediaType(schema = Some(errorRef)))
  )

  val paths: Paths = Paths(
    ChunkMap(
      "/books" -> PathItem(
        get = Some(
          Operation(
            operationId = Some("listBooks"),
            parameters = Chunk(ReferenceOr.Value(limitParam)),
            responses = Responses(ChunkMap("200" -> ReferenceOr.Value(booksFound)))
          )
        )
      ),
      "/books/{id}" -> PathItem(
        parameters = Chunk(ReferenceOr.Value(idParam)),
        get = Some(
          Operation(
            operationId = Some("getBook"),
            responses = Responses(
              ChunkMap(
                "200" -> ReferenceOr.Value(bookFound),
                "404" -> ReferenceOr.Value(bookNotFound)
              )
            )
          )
        )
      )
    )
  )

  val paperDoc: OpenAPI = OpenAPI(
    openapi = "3.1.0",
    info = Info(title = "Bookstore API", version = "1.0.0"),
    paths = Some(paths)
  )

  openAPICodec.encodeValue(paperDoc).get("paths").one match {
    case Right(section) => println(section.print(WriterConfig.withIndentionStep2))
    case Left(error)    => println(s"No paths section: $error")
  }
}
```

</details>

Run it from the `openapi-examples` directory:

```bash
sbt "runMain writeandexportopenapi.Step2EndpointsExample"
```

### Require a Bearer Token

This example adds the `bearerAuth` security scheme to `components`, requires it for every operation, and prints both parts of the document.

<details>
  <summary>openapi-examples/src/main/scala/writeandexportopenapi/Step3BearerTokenExample.scala</summary>

```scala title="openapi-examples/src/main/scala/writeandexportopenapi/Step3BearerTokenExample.scala" showLineNumbers
package writeandexportopenapi

/**
 * How to Write an OpenAPI Document and Export It as JSON or YAML: Step 3 —
 * Requiring a Bearer Token
 *
 * Adds the bearerAuth security scheme to components, requires it for every
 * operation, and prints both parts of the document.
 *
 * Run with: cd openapi-examples && sbt "runMain
 * writeandexportopenapi.Step3BearerTokenExample"
 */
object Step3BearerTokenExample extends App {

  final case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  final case class ApiError(message: String)

  object ApiError {
    implicit val schema: Schema[ApiError] = Schema.derived
  }

  val sampleBooks: List[Book] = List(
    Book("Zionomicon", "John A. De Goes", 2024, None),
    Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601")),
    Book("Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
  )

  val (bookRef, _)  = Schema[Book].toRefSchema
  val (errorRef, _) = Schema[ApiError].toRefSchema

  val componentSchemas: Map[String, SchemaObject] = OpenAPIGen.schemas(Schema[Book], Schema[ApiError])

  val bookListSchema: ReferenceOr[SchemaObject] = ReferenceOr.Value(
    SchemaObject(
      jsonSchema = Json.Object(
        "type"  -> Json.String("array"),
        "items" -> Json.Object("$ref" -> Json.String("#/components/schemas/Book"))
      )
    )
  )

  val limitParam: Parameter = Parameter(
    name = "limit",
    in = ParameterLocation.Query,
    description = Some(md"Maximum number of books to return."),
    schema = Some(Schema[Int].toInlineSchema)
  )

  val idParam: Parameter = Parameter(
    name = "id",
    in = ParameterLocation.Path,
    required = true,
    description = Some(md"Catalog ID of the book."),
    schema = Some(Schema[String].toInlineSchema)
  )

  val booksFound: Response = Response(
    description = md"The books in the catalog.",
    content = ChunkMap(
      "application/json" -> MediaType(schema = Some(bookListSchema), example = Some(Json.from(sampleBooks)))
    )
  )

  val bookFound: Response = Response(
    description = md"The book.",
    content = ChunkMap("application/json" -> MediaType(schema = Some(bookRef)))
  )

  val bookNotFound: Response = Response(
    description = md"No book has this ID.",
    content = ChunkMap("application/json" -> MediaType(schema = Some(errorRef)))
  )

  val paths: Paths = Paths(
    ChunkMap(
      "/books" -> PathItem(
        get = Some(
          Operation(
            operationId = Some("listBooks"),
            parameters = Chunk(ReferenceOr.Value(limitParam)),
            responses = Responses(ChunkMap("200" -> ReferenceOr.Value(booksFound)))
          )
        )
      ),
      "/books/{id}" -> PathItem(
        parameters = Chunk(ReferenceOr.Value(idParam)),
        get = Some(
          Operation(
            operationId = Some("getBook"),
            responses = Responses(
              ChunkMap(
                "200" -> ReferenceOr.Value(bookFound),
                "404" -> ReferenceOr.Value(bookNotFound)
              )
            )
          )
        )
      )
    )
  )

  val components: Components = Components(
    schemas = ChunkMap.from(componentSchemas.map { case (name, schema) => name -> ReferenceOr.Value(schema) }),
    securitySchemes = ChunkMap(
      "bearerAuth" -> ReferenceOr.Value(SecurityScheme.HTTP(scheme = "bearer", bearerFormat = Some("JWT")))
    )
  )

  val requireToken: SecurityRequirement = SecurityRequirement(ChunkMap("bearerAuth" -> Chunk.empty))

  val api: OpenAPI = OpenAPI(
    openapi = "3.1.0",
    info = Info(title = "Bookstore API", version = "1.0.0"),
    paths = Some(paths),
    components = Some(components),
    security = Chunk(requireToken)
  )

  val json: Json = openAPICodec.encodeValue(api)

  def show(section: String, selection: zio.blocks.schema.json.JsonSelection): Unit =
    selection.one match {
      case Right(value) => println(s"$section:\n${value.print(WriterConfig.withIndentionStep2)}")
      case Left(error)  => println(s"$section is missing: $error")
    }

  show("components/securitySchemes", json.get("components").get("securitySchemes"))
  show("security", json.get("security"))
}
```

</details>

Run it from the `openapi-examples` directory:

```bash
sbt "runMain writeandexportopenapi.Step3BearerTokenExample"
```

### Export the Document as JSON

This example encodes the whole document with `openAPICodec`, prints it as indented JSON, and reads the text back to check that it gives the same document.

<details>
  <summary>openapi-examples/src/main/scala/writeandexportopenapi/Step4ExportJsonExample.scala</summary>

```scala title="openapi-examples/src/main/scala/writeandexportopenapi/Step4ExportJsonExample.scala" showLineNumbers
package writeandexportopenapi

/**
 * How to Write an OpenAPI Document and Export It as JSON or YAML: Step 4 —
 * Exporting the Document as JSON
 *
 * Encodes the whole document with openAPICodec, prints it as indented JSON, and
 * reads the text back to check that it gives the same document.
 *
 * Run with: cd openapi-examples && sbt "runMain
 * writeandexportopenapi.Step4ExportJsonExample"
 */
object Step4ExportJsonExample extends App {

  final case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  final case class ApiError(message: String)

  object ApiError {
    implicit val schema: Schema[ApiError] = Schema.derived
  }

  val sampleBooks: List[Book] = List(
    Book("Zionomicon", "John A. De Goes", 2024, None),
    Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601")),
    Book("Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
  )

  val (bookRef, _)  = Schema[Book].toRefSchema
  val (errorRef, _) = Schema[ApiError].toRefSchema

  val componentSchemas: Map[String, SchemaObject] = OpenAPIGen.schemas(Schema[Book], Schema[ApiError])

  val bookListSchema: ReferenceOr[SchemaObject] = ReferenceOr.Value(
    SchemaObject(
      jsonSchema = Json.Object(
        "type"  -> Json.String("array"),
        "items" -> Json.Object("$ref" -> Json.String("#/components/schemas/Book"))
      )
    )
  )

  val limitParam: Parameter = Parameter(
    name = "limit",
    in = ParameterLocation.Query,
    description = Some(md"Maximum number of books to return."),
    schema = Some(Schema[Int].toInlineSchema)
  )

  val idParam: Parameter = Parameter(
    name = "id",
    in = ParameterLocation.Path,
    required = true,
    description = Some(md"Catalog ID of the book."),
    schema = Some(Schema[String].toInlineSchema)
  )

  val booksFound: Response = Response(
    description = md"The books in the catalog.",
    content = ChunkMap(
      "application/json" -> MediaType(schema = Some(bookListSchema), example = Some(Json.from(sampleBooks)))
    )
  )

  val bookFound: Response = Response(
    description = md"The book.",
    content = ChunkMap("application/json" -> MediaType(schema = Some(bookRef)))
  )

  val bookNotFound: Response = Response(
    description = md"No book has this ID.",
    content = ChunkMap("application/json" -> MediaType(schema = Some(errorRef)))
  )

  val paths: Paths = Paths(
    ChunkMap(
      "/books" -> PathItem(
        get = Some(
          Operation(
            operationId = Some("listBooks"),
            parameters = Chunk(ReferenceOr.Value(limitParam)),
            responses = Responses(ChunkMap("200" -> ReferenceOr.Value(booksFound)))
          )
        )
      ),
      "/books/{id}" -> PathItem(
        parameters = Chunk(ReferenceOr.Value(idParam)),
        get = Some(
          Operation(
            operationId = Some("getBook"),
            responses = Responses(
              ChunkMap(
                "200" -> ReferenceOr.Value(bookFound),
                "404" -> ReferenceOr.Value(bookNotFound)
              )
            )
          )
        )
      )
    )
  )

  val components: Components = Components(
    schemas = ChunkMap.from(componentSchemas.map { case (name, schema) => name -> ReferenceOr.Value(schema) }),
    securitySchemes = ChunkMap(
      "bearerAuth" -> ReferenceOr.Value(SecurityScheme.HTTP(scheme = "bearer", bearerFormat = Some("JWT")))
    )
  )

  val requireToken: SecurityRequirement = SecurityRequirement(ChunkMap("bearerAuth" -> Chunk.empty))

  val api: OpenAPI = OpenAPI(
    openapi = "3.1.0",
    info = Info(title = "Bookstore API", version = "1.0.0"),
    paths = Some(paths),
    components = Some(components),
    security = Chunk(requireToken)
  )

  val json: Json = openAPICodec.encodeValue(api)

  println(json.print(WriterConfig.withIndentionStep2))

  val jsonText: String = json.print(WriterConfig.withIndentionStep2)

  val reloaded: Either[SchemaError, OpenAPI] = Json.parse(jsonText).flatMap(openAPICodec.decode)

  println(reloaded.map(document => openAPICodec.encodeValue(document) == json))
}
```

</details>

Run it from the `openapi-examples` directory:

```bash
sbt "runMain writeandexportopenapi.Step4ExportJsonExample"
```

### Export the Document as YAML

This example converts the document to YAML with `Yaml.fromJson`, quotes the `$ref` values, and prints the result.

<details>
  <summary>openapi-examples/src/main/scala/writeandexportopenapi/Step5ExportYamlExample.scala</summary>

```scala title="openapi-examples/src/main/scala/writeandexportopenapi/Step5ExportYamlExample.scala" showLineNumbers
package writeandexportopenapi

/**
 * How to Write an OpenAPI Document and Export It as JSON or YAML: Step 5 —
 * Exporting the Document as YAML
 *
 * Converts the document to YAML with Yaml.fromJson, quotes the $ref values, and
 * prints the result.
 *
 * Run with: cd openapi-examples && sbt "runMain
 * writeandexportopenapi.Step5ExportYamlExample"
 */
object Step5ExportYamlExample extends App {

  final case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  final case class ApiError(message: String)

  object ApiError {
    implicit val schema: Schema[ApiError] = Schema.derived
  }

  val sampleBooks: List[Book] = List(
    Book("Zionomicon", "John A. De Goes", 2024, None),
    Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601")),
    Book("Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
  )

  val (bookRef, _)  = Schema[Book].toRefSchema
  val (errorRef, _) = Schema[ApiError].toRefSchema

  val componentSchemas: Map[String, SchemaObject] = OpenAPIGen.schemas(Schema[Book], Schema[ApiError])

  val bookListSchema: ReferenceOr[SchemaObject] = ReferenceOr.Value(
    SchemaObject(
      jsonSchema = Json.Object(
        "type"  -> Json.String("array"),
        "items" -> Json.Object("$ref" -> Json.String("#/components/schemas/Book"))
      )
    )
  )

  val limitParam: Parameter = Parameter(
    name = "limit",
    in = ParameterLocation.Query,
    description = Some(md"Maximum number of books to return."),
    schema = Some(Schema[Int].toInlineSchema)
  )

  val idParam: Parameter = Parameter(
    name = "id",
    in = ParameterLocation.Path,
    required = true,
    description = Some(md"Catalog ID of the book."),
    schema = Some(Schema[String].toInlineSchema)
  )

  val booksFound: Response = Response(
    description = md"The books in the catalog.",
    content = ChunkMap(
      "application/json" -> MediaType(schema = Some(bookListSchema), example = Some(Json.from(sampleBooks)))
    )
  )

  val bookFound: Response = Response(
    description = md"The book.",
    content = ChunkMap("application/json" -> MediaType(schema = Some(bookRef)))
  )

  val bookNotFound: Response = Response(
    description = md"No book has this ID.",
    content = ChunkMap("application/json" -> MediaType(schema = Some(errorRef)))
  )

  val paths: Paths = Paths(
    ChunkMap(
      "/books" -> PathItem(
        get = Some(
          Operation(
            operationId = Some("listBooks"),
            parameters = Chunk(ReferenceOr.Value(limitParam)),
            responses = Responses(ChunkMap("200" -> ReferenceOr.Value(booksFound)))
          )
        )
      ),
      "/books/{id}" -> PathItem(
        parameters = Chunk(ReferenceOr.Value(idParam)),
        get = Some(
          Operation(
            operationId = Some("getBook"),
            responses = Responses(
              ChunkMap(
                "200" -> ReferenceOr.Value(bookFound),
                "404" -> ReferenceOr.Value(bookNotFound)
              )
            )
          )
        )
      )
    )
  )

  val components: Components = Components(
    schemas = ChunkMap.from(componentSchemas.map { case (name, schema) => name -> ReferenceOr.Value(schema) }),
    securitySchemes = ChunkMap(
      "bearerAuth" -> ReferenceOr.Value(SecurityScheme.HTTP(scheme = "bearer", bearerFormat = Some("JWT")))
    )
  )

  val requireToken: SecurityRequirement = SecurityRequirement(ChunkMap("bearerAuth" -> Chunk.empty))

  val api: OpenAPI = OpenAPI(
    openapi = "3.1.0",
    info = Info(title = "Bookstore API", version = "1.0.0"),
    paths = Some(paths),
    components = Some(components),
    security = Chunk(requireToken)
  )

  val json: Json = openAPICodec.encodeValue(api)

  val rawYaml: String = Yaml.fromJson(json).printPretty

  def quoteRefs(yaml: String): String =
    yaml.replaceAll("(\\$ref: )(#.*)", "$1\"$2\"")

  val yamlText: String = quoteRefs(rawYaml)

  println(yamlText)
}
```

</details>

Run it from the `openapi-examples` directory:

```bash
sbt "runMain writeandexportopenapi.Step5ExportYamlExample"
```

### Serve the Document to Swagger UI

This example serves the document at `http://localhost:8080/openapi.json` with the JDK's built-in HTTP server, fetches it once to show that it works, and keeps serving until you press Enter.

<details>
  <summary>openapi-examples/src/main/scala/writeandexportopenapi/Step6ServeSwaggerUiExample.scala</summary>

```scala title="openapi-examples/src/main/scala/writeandexportopenapi/Step6ServeSwaggerUiExample.scala" showLineNumbers
package writeandexportopenapi

/**
 * How to Write an OpenAPI Document and Export It as JSON or YAML: Step 6 —
 * Serving the Document to Swagger UI
 *
 * Serves the document at http://localhost:8080/openapi.json with the JDK's
 * built-in HTTP server, fetches it once to show that it works, and keeps
 * serving until you press Enter.
 *
 * Run with: cd openapi-examples && sbt "runMain
 * writeandexportopenapi.Step6ServeSwaggerUiExample"
 */
object Step6ServeSwaggerUiExample {

  final case class Book(title: String, firstAuthor: String, year: Int, isbn: Option[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  final case class ApiError(message: String)

  object ApiError {
    implicit val schema: Schema[ApiError] = Schema.derived
  }

  val sampleBooks: List[Book] = List(
    Book("Zionomicon", "John A. De Goes", 2024, None),
    Book("Programming in Scala", "Martin Odersky", 2008, Some("9780981531601")),
    Book("Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
  )

  val (bookRef, _)  = Schema[Book].toRefSchema
  val (errorRef, _) = Schema[ApiError].toRefSchema

  val componentSchemas: Map[String, SchemaObject] = OpenAPIGen.schemas(Schema[Book], Schema[ApiError])

  val bookListSchema: ReferenceOr[SchemaObject] = ReferenceOr.Value(
    SchemaObject(
      jsonSchema = Json.Object(
        "type"  -> Json.String("array"),
        "items" -> Json.Object("$ref" -> Json.String("#/components/schemas/Book"))
      )
    )
  )

  val limitParam: Parameter = Parameter(
    name = "limit",
    in = ParameterLocation.Query,
    description = Some(md"Maximum number of books to return."),
    schema = Some(Schema[Int].toInlineSchema)
  )

  val idParam: Parameter = Parameter(
    name = "id",
    in = ParameterLocation.Path,
    required = true,
    description = Some(md"Catalog ID of the book."),
    schema = Some(Schema[String].toInlineSchema)
  )

  val booksFound: Response = Response(
    description = md"The books in the catalog.",
    content = ChunkMap(
      "application/json" -> MediaType(schema = Some(bookListSchema), example = Some(Json.from(sampleBooks)))
    )
  )

  val bookFound: Response = Response(
    description = md"The book.",
    content = ChunkMap("application/json" -> MediaType(schema = Some(bookRef)))
  )

  val bookNotFound: Response = Response(
    description = md"No book has this ID.",
    content = ChunkMap("application/json" -> MediaType(schema = Some(errorRef)))
  )

  val paths: Paths = Paths(
    ChunkMap(
      "/books" -> PathItem(
        get = Some(
          Operation(
            operationId = Some("listBooks"),
            parameters = Chunk(ReferenceOr.Value(limitParam)),
            responses = Responses(ChunkMap("200" -> ReferenceOr.Value(booksFound)))
          )
        )
      ),
      "/books/{id}" -> PathItem(
        parameters = Chunk(ReferenceOr.Value(idParam)),
        get = Some(
          Operation(
            operationId = Some("getBook"),
            responses = Responses(
              ChunkMap(
                "200" -> ReferenceOr.Value(bookFound),
                "404" -> ReferenceOr.Value(bookNotFound)
              )
            )
          )
        )
      )
    )
  )

  val components: Components = Components(
    schemas = ChunkMap.from(componentSchemas.map { case (name, schema) => name -> ReferenceOr.Value(schema) }),
    securitySchemes = ChunkMap(
      "bearerAuth" -> ReferenceOr.Value(SecurityScheme.HTTP(scheme = "bearer", bearerFormat = Some("JWT")))
    )
  )

  val requireToken: SecurityRequirement = SecurityRequirement(ChunkMap("bearerAuth" -> Chunk.empty))

  val api: OpenAPI = OpenAPI(
    openapi = "3.1.0",
    info = Info(title = "Bookstore API", version = "1.0.0"),
    paths = Some(paths),
    components = Some(components),
    security = Chunk(requireToken)
  )

  val json: Json = openAPICodec.encodeValue(api)

  // The handler reads `body` from another thread, so the serving code runs in `main`, not in the
  // initializer of an `App` object, where the request would deadlock on the class initialization.
  def main(args: Array[String]): Unit = {
    val body: Array[Byte] = json.print.getBytes(UTF_8)

    val server: HttpServer = HttpServer.create(new InetSocketAddress(8080), 0)
    server.createContext(
      "/openapi.json",
      exchange => {
        exchange.getResponseHeaders.set("Content-Type", "application/json")
        // Swagger UI runs on another origin, so the browser needs permission to fetch the document.
        exchange.getResponseHeaders.set("Access-Control-Allow-Origin", "*")
        exchange.sendResponseHeaders(200, body.length.toLong)
        exchange.getResponseBody.write(body)
        exchange.close()
      }
    )
    server.start()

    println("Serving the document at http://localhost:8080/openapi.json")

    val client: HttpClient             = HttpClient.newHttpClient()
    val request: HttpRequest           = HttpRequest.newBuilder(URI.create("http://localhost:8080/openapi.json")).build()
    val response: HttpResponse[String] = client.send(request, HttpResponse.BodyHandlers.ofString())
    println(s"GET /openapi.json -> ${response.statusCode()}")
    println(s"Content-Type: ${response.headers().firstValue("Content-Type").orElse("-")}")
    println(s"Body starts with: ${response.body().take(80)}")

    println("Point Swagger UI at the URL above. Press Enter to stop the server.")
    scala.io.StdIn.readLine()
    server.stop(0)
  }
}
```

</details>

Run it from the `openapi-examples` directory, then start Swagger UI as shown in [Serving the Document to Swagger UI](#serving-the-document-to-swagger-ui):

```bash
sbt "runMain writeandexportopenapi.Step6ServeSwaggerUiExample"
```

### The Complete Example

The complete example from "Putting It Together" runs everything in one program:

```bash
sbt "runMain writeandexportopenapi.CompleteExample"
```

### Compile All Examples

To compile every example in the project without running any of them, run this from the `openapi-examples` directory:

```bash
sbt compile
```

## Going Further

- The [OpenAPI reference](../reference/openapi.md) covers every type of the data model, including request bodies, headers, servers, tags and the other security schemes.
- [How to Generate JSON Schema from a Type and Validate Payloads Against It](./generate-json-schema-and-validate-payloads.md) explains the JSON Schema that each component schema contains, and how to validate payloads against it.
- The [YAML format reference](../reference/schema/built-in-codecs/yaml.md) shows the `Yaml` tree and its other uses.

---

## How to Write Joins and Inspect the Generated SQL

In this guide, we query two related tables, `author` and `book`, with the `sql` module's `SqlQuery` builder. By the end, you will have inner and left joins that return each book with its first author and each author with their books, filtered and sorted, plus the exact SQL each join sends to the database, printed at run time and written to `.sql` files at compile time.

We describe the link between the tables once, as a `Rel`, and reuse it in every query:

- Define two tables that reference each other
- Join each book to its first author (many-to-one)
- List each author's books (one-to-many)
- Keep authors who have no books yet (left join)
- Filter and sort a join
- Print the generated SQL for SQLite and PostgreSQL
- Write the SQL to files at compile time for review

## The Problem

A join returns columns from two tables in one row, and your code has to split that row back into two values. Without a query builder, you write the column list, the `ON` condition and the row mapping by hand, and keep all three in sync with the case classes:

```scala

case class Author(authorId: Int, name: String, born: Option[Int])
case class Book(bookId: Int, title: String, year: Int, firstAuthorId: Int)

val connection = DriverManager.getConnection("jdbc:sqlite:library.db")
val statement = connection.prepareStatement(
  """SELECT b.book_id, b.title, b.year, b.first_author_id, a.author_id, a.name, a.born
    |FROM book b JOIN author a ON b.first_author_id = a.author_id""".stripMargin
)
val rs = statement.executeQuery()
val rows = Iterator
  .continually(rs.next())
  .takeWhile(identity)
  .map { _ =>
    val book   = Book(rs.getInt(1), rs.getString(2), rs.getInt(3), rs.getInt(4))
    val born   = rs.getInt(7)
    val author = Author(rs.getInt(5), rs.getString(6), if (rs.wasNull()) None else Some(born))
    (book, author)
  }
  .toList
connection.close()
```

Every field added to `Book` or `Author` means editing the `SELECT` list and shifting every column index after it. A wrong index still compiles and silently puts the wrong value in the wrong field. And the SQL is a string, so the only way to review the joins a program runs is to read every string in the code base.

In this guide, we let `SqlQuery` write the column list and the `ON` condition from the tables' schemas, decode each row into a tuple of case classes, and print the SQL it generates so you can check it.

## Prerequisites

Add the `sql` module and the SQLite JDBC driver to your `build.sbt`:

```scala
// Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-sql" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-sql" % "0.0.56"
scalacOptions += "-experimental"

// The SQLite JDBC driver, for every Scala version
libraryDependencies += "org.xerial" % "sqlite-jdbc" % "3.53.4.0"
```

The `sql` module is published for Scala 3 only. Running queries over JDBC needs the JVM and JDK 21 or newer.

We assume you have connected to a database and run a query with the `sql"..."` interpolator before, as in [Build Your First Database Repository on SQLite](./your-first-sqlite-repository.md). The same queries run on PostgreSQL; [How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md) shows how to build the `JdbcTransactor` for it.

:::note
`SqlQuery` in `zio.blocks.sql.query` is not the query DSL from the [Query DSL with Reified Optics](./query-dsl-reified-optics.md) guides. Those guides build their own SQL generator from `SchemaExpr` optics; this guide uses the join builder that ships with the `sql` module.
:::

Everything in this guide comes from three imports:

```scala

```

## The Library Tables

Our library has authors and books. Each book stores the ID of its first author, so many books can point at one author:

```scala
case class Author(authorId: Int, name: String, born: Option[Int])
object Author {
  given Schema[Author] = Schema.derived
}

case class Book(bookId: Int, title: String, year: Int, firstAuthorId: Int)
object Book {
  given Schema[Book] = Schema.derived
}

val authors = Table.derived[Author]
val books   = Table.derived[Book]
```

`Table.derived` maps each case class to a table named after the type, `author` and `book`, with one snake-case column per field:

```scala
println(s"${authors.name}: ${authors.columns.mkString(", ")}")
// author: author_id, name, born
println(s"${books.name}: ${books.columns.mkString(", ")}")
// book: book_id, title, year, first_author_id
```

:::warning
Give every column a name that no other table in the join uses, as `author_id` and `book_id` do here. Rows decode into tuples by column label, and when two joined tables both have an `id` column, every `id` in the tuple is read from the first one. With `Author(id, ...)` and `Book(id, ...)`, each book comes back paired with an `Author` that carries the book's ID. The SQL itself is correct; only the decoding is wrong.
:::

We create both tables in a temporary SQLite file, so each run starts empty. A file-backed database is needed because every `connect` opens a new connection, and an in-memory SQLite database (`jdbc:sqlite::memory:`) is empty again on every new connection. We write the `CREATE TABLE` statements by hand, because `Table#createTable` doesn't declare a `PRIMARY KEY`:

```scala
val dbFile     = java.nio.file.Files.createTempFile("library", ".db")
val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

transactor.connect {
  sql"""CREATE TABLE author (
    author_id INTEGER PRIMARY KEY,
    name      TEXT NOT NULL,
    born      INTEGER
  )""".update
  sql"""CREATE TABLE book (
    book_id         INTEGER PRIMARY KEY,
    title           TEXT NOT NULL,
    year            INTEGER NOT NULL,
    first_author_id INTEGER NOT NULL REFERENCES author (author_id)
  )""".update
}
```

Then we add four authors and four books. Li Haoyi has no book in our library yet, which matters for the left join later:

```scala
val authorRepo = Repo.derived[Author, Int]
val bookRepo   = Repo.derived[Book, Int]

transactor.connect {
  authorRepo.insert(Author(1, "Martin Odersky", Some(1958)))
  authorRepo.insert(Author(2, "John A. De Goes", None))
  authorRepo.insert(Author(3, "Paul Chiusano", None))
  authorRepo.insert(Author(4, "Li Haoyi", None))

  bookRepo.insert(Book(1, "Zionomicon", 2024, 2))
  bookRepo.insert(Book(2, "Programming in Scala", 2008, 1))
  bookRepo.insert(Book(3, "Functional Programming in Scala", 2014, 3))
  bookRepo.insert(Book(4, "Programming in Scala, Fifth Edition", 2021, 1))
}
```

## Join Each Book to Its First Author

A `Rel` describes how two tables are linked: a foreign-key column in one table equals a key column in the other. `Rel.manyToOne` takes the table holding the foreign key, a selector for that field, the referenced table and a selector for its key:

```scala
val bookAuthor = Rel.manyToOne(books, _.firstAuthorId, authors, _.authorId)
```

The selectors are checked at compile time, so `_.firstAuthorID` doesn't compile, and each field name is turned into its snake-case column name, `first_author_id` and `author_id`.

`SqlQuery.from(books)` starts a query on the `book` table, and `join(bookAuthor)` adds an inner join to `author`. `toFrag` renders the query to a `Frag`, which runs like any `sql"..."` fragment. A join selects every column of both tables, books first, so we decode each row as a `(Book, Author)` tuple:

```scala
val booksWithAuthors = SqlQuery.from(books).join(bookAuthor)
```

```scala
transactor.connect {
  booksWithAuthors.toFrag(SqlDialect.SQLite).query[(Book, Author)].foreach { case (book, author) =>
    println(s"${book.title} (${book.year}) by ${author.name}")
  }
}
// Zionomicon (2024) by John A. De Goes
// Programming in Scala (2008) by Martin Odersky
// Functional Programming in Scala (2014) by Paul Chiusano
// Programming in Scala, Fifth Edition (2021) by Martin Odersky
```

The tuple follows the join order: the table passed to `SqlQuery.from` comes first, then one element per `join`. A tuple whose types don't match that order reads the wrong columns, so keep the two in step.

## List Each Author's Books

The same link, read from the other side, is one-to-many: one author, many books. `Rel.oneToMany` takes the "one" table first:

```scala
val authorBooks = Rel.oneToMany(authors, _.authorId, books, _.firstAuthorId)
```

`Rel.manyToOne` and `Rel.oneToMany` build the same `Rel`: the `ON` condition is "the first table's column equals the second table's column", and each selector must name a field of the table next to it. The constructor's name only documents which way the link reads. In fact `SqlQuery.from(authors).join(bookAuthor)` produces the same rows as the query below.

A join still returns one row per matching pair, so an author with two books comes back twice. We start from `authors` and group the rows by author in Scala:

```scala
val authorsAndBooks = SqlQuery.from(authors).join(authorBooks)
```

```scala
transactor.connect {
  val rows = authorsAndBooks.toFrag(SqlDialect.SQLite).query[(Author, Book)]
  rows.groupMap(_._1)(_._2).toList.sortBy(_._1.name).foreach { case (author, written) =>
    println(s"${author.name}: ${written.sortBy(_.year).map(_.title).mkString(", ")}")
  }
}
// John A. De Goes: Zionomicon
// Martin Odersky: Programming in Scala, Programming in Scala, Fifth Edition
// Paul Chiusano: Functional Programming in Scala
```

Li Haoyi is missing: an inner join only keeps authors that have at least one matching book.

## Keep Authors Who Have No Books

To keep every author, use a left join. `leftJoin` and `joinLeft` are the same method under two names, and so are `join` and `innerJoin`. `join(rel, JoinKind.Left)` picks the kind from a value:

```scala
println(SqlQuery.from(authors).leftJoin(authorBooks) == SqlQuery.from(authors).joinLeft(authorBooks))
// true
println(SqlQuery.from(authors).leftJoin(authorBooks) == SqlQuery.from(authors).join(authorBooks, JoinKind.Left))
// true
```

For an author with no books, the database returns SQL `NULL` in every `book` column. Decoding those `NULL`s as a plain `Book` doesn't fail: SQL `NULL` decodes to `null` for a `String` and to `0` for an `Int`, so Li Haoyi gets a book that doesn't exist:

```scala
val everyAuthor = SqlQuery.from(authors).leftJoin(authorBooks)
```

```scala
transactor.connect {
  everyAuthor.toFrag(SqlDialect.SQLite).query[(Author, Book)].foreach(println)
}
// (Author(1,Martin Odersky,Some(1958)),Book(2,Programming in Scala,2008,1))
// (Author(1,Martin Odersky,Some(1958)),Book(4,Programming in Scala, Fifth Edition,2021,1))
// (Author(2,John A. De Goes,None),Book(1,Zionomicon,2024,2))
// (Author(3,Paul Chiusano,None),Book(3,Functional Programming in Scala,2014,3))
// (Author(4,Li Haoyi,None),Book(0,null,0,0))
```

`Option[Book]` isn't an option either, because `Option` is only supported for single-column types. Instead, give the optional side a row type whose fields are all `Option`s, and turn it into an `Option[Book]` in Scala:

```scala
case class BookColumns(bookId: Option[Int], title: Option[String], year: Option[Int], firstAuthorId: Option[Int]) {
  def toBook: Option[Book] =
    for {
      id     <- bookId
      t      <- title
      y      <- year
      author <- firstAuthorId
    } yield Book(id, t, y, author)
}
object BookColumns {
  given Schema[BookColumns] = Schema.derived
}
```

`BookColumns` has the same fields in the same order as `Book`, so it reads the same columns. Now the missing book is `None`:

```scala
transactor.connect {
  val rows = everyAuthor.toFrag(SqlDialect.SQLite).query[(Author, BookColumns)]
  rows.groupMap(_._1)(_._2.toBook).toList.sortBy(_._1.name).foreach { case (author, written) =>
    val titles = written.flatten.map(_.title)
    println(s"${author.name}: ${if (titles.isEmpty) "no books yet" else titles.mkString(", ")}")
  }
}
// John A. De Goes: Zionomicon
// Li Haoyi: no books yet
// Martin Odersky: Programming in Scala, Programming in Scala, Fifth Edition
// Paul Chiusano: Functional Programming in Scala
```

## Filter and Sort a Join

`where` adds a condition to the query, written as a `sql"..."` fragment; `filter` is the same method. Values you interpolate become bound parameters, as in any `sql"..."` fragment. Inside the fragment, refer to columns through the table aliases the query uses: `t0` is the table passed to `SqlQuery.from`, and `t1`, `t2` and so on are the joined tables in the order you added them. Quote column names with double quotes, as the generated SQL does.

`orderBy` sorts by a column of the `t0` table, ascending unless you pass `SortOrder.Desc`. Here are the books published after 2010, newest first, with their first authors:

```scala
val publishedAfter = 2010

val recentBooks = SqlQuery
  .from(books)
  .join(bookAuthor)
  .where(sql"""t0."year" > $publishedAfter""")
  .orderBy("year", SortOrder.Desc)
```

```scala
transactor.connect {
  recentBooks.toFrag(SqlDialect.SQLite).query[(Book, Author)].foreach { case (book, author) =>
    println(s"${book.year}: ${book.title} by ${author.name}")
  }
}
// 2024: Zionomicon by John A. De Goes
// 2021: Programming in Scala, Fifth Edition by Martin Odersky
// 2014: Functional Programming in Scala by Paul Chiusano
```

A condition on the joined table uses its alias. Each `where` call adds a condition, and the conditions are combined with `AND`:

```scala
val author = "Martin Odersky"

val booksByOdersky = SqlQuery
  .from(books)
  .join(bookAuthor)
  .where(sql"""t1."name" = $author""")
  .where(sql"""t0."year" < ${2020}""")
```

```scala
transactor.connect {
  booksByOdersky.toFrag(SqlDialect.SQLite).query[(Book, Author)].foreach { case (book, _) =>
    println(book.title)
  }
}
// Programming in Scala
```

:::caution
`orderBy` and `groupBy` only reach the `t0` table: they always render `t0."<column>"`, and a name like `t1.name` is rejected with an `IllegalArgumentException` ("Invalid SQL column identifier"). To sort by a column of a joined table, start the query from that table, or sort the decoded rows in Scala. The alias names `t0`, `t1` in a `where` fragment are not checked at all, so a typo there only fails when the database runs the query.
:::

## See the SQL Before It Runs

`SqlQuery#explain(dialect)` returns the SQL on one line, with numbered `?1`, `?2` placeholders, followed by a `-- params:` comment that lists each parameter's position and type, never its value. We render the filtered query for both dialects:

```scala
println(recentBooks.explain(SqlDialect.SQLite))
// SELECT t0."book_id", t0."title", t0."year", t0."first_author_id", t1."author_id", t1."name", t1."born" FROM "book" AS t0 INNER JOIN "author" AS t1 ON t0."first_author_id" = t1."author_id" WHERE t0."year" > ?1 ORDER BY t0."year" DESC
// -- params: 1:Int
println(recentBooks.explain(SqlDialect.PostgreSQL))
// SELECT t0."book_id", t0."title", t0."year", t0."first_author_id", t1."author_id", t1."name", t1."born" FROM "book" AS t0 INNER JOIN "author" AS t1 ON t0."first_author_id" = t1."author_id" WHERE t0."year" > ?1 ORDER BY t0."year" DESC
// -- params: 1:Int
```

The two are identical: both dialects use `?` placeholders and the same quoting, so a `SqlQuery` renders the same SQL for SQLite and PostgreSQL today. What differs between the dialects is the column types in `CREATE TABLE` statements, not the queries.

`explain` is meant for logs and code review. `SqlQuery#sql(dialect)` returns the exact string that `toFrag` sends to the JDBC driver, with plain `?` placeholders:

```scala
println(recentBooks.sql(SqlDialect.SQLite))
// SELECT t0."book_id", t0."title", t0."year", t0."first_author_id", t1."author_id", t1."name", t1."born" FROM "book" AS t0 INNER JOIN "author" AS t1 ON t0."first_author_id" = t1."author_id" WHERE t0."year" > ? ORDER BY t0."year" DESC
println(everyAuthor.sql(SqlDialect.SQLite))
// SELECT t0."author_id", t0."name", t0."born", t1."book_id", t1."title", t1."year", t1."first_author_id" FROM "author" AS t0 LEFT JOIN "book" AS t1 ON t0."author_id" = t1."first_author_id"
```

The output shows the rules every join follows: the columns of each table in schema order, every identifier double-quoted, the `FROM` table aliased as `t0`, each `JOIN` kind spelled out, and each `ON` condition built from the `Rel`.

## Dump the SQL to Files at Compile Time

`explain` needs the program to run. To review every join a code base contains, for example in a pull request or by a DBA who doesn't read Scala, `Dump.dumpQuery` writes a query's SQL to a `.sql` file while the code compiles. The macro only writes when the JVM system property `zib.sql.dumpDir` is set in the JVM that runs the compiler; otherwise each call compiles to nothing.

Dumping happens at compile time, so the code in this section lives in the companion example `JoinsDumpExample.scala` rather than on this page. Two rules decide what the macro can see:

- The query must be written out where `Dump.dumpQuery` can read it: an `inline def`, or the `SqlQuery.from(...)` chain itself. A query stored in a plain `val` or built at run time can't be dumped; the macro warns and writes nothing.
- Likewise, the `Rel` must be built inside the chain, not referenced from a `val`.

```scala
object LibraryQueries {
  inline def booksWithFirstAuthors: SqlQuery[Book] =
    SqlQuery.from(Library.books).join(Rel.manyToOne(Library.books, _.firstAuthorId, Library.authors, _.authorId))
}

object JoinsDumpExample {
  // The val's name becomes the file name
  val booksWithFirstAuthors = Dump.dumpQuery(LibraryQueries.booksWithFirstAuthors)
}
```

Set the property when you start sbt. A running sbt server (`sbt --client`, or the one an IDE starts) keeps the property it was started with, so shut it down first. `clean` makes sure every file is compiled, and so every dump is written:

```bash
sbt -Dzib.sql.dumpDir=target/sql-dumps clean compile
```

`SBT_OPTS="-Dzib.sql.dumpDir=target/sql-dumps"` works too. Passing the property through `scalacOptions` or `javaOptions` does nothing, because those reach the compiler as flags or the forked program at run time, not as properties of the compiler's JVM.

Each `Dump.dumpQuery` call writes one file per dialect, named `<name>_query-<dialect>.sql`, where `<name>` is the `val` the call is assigned to. For the query above, `target/sql-dumps/booksWithFirstAuthors_query-sqlite.sql` contains:

```sql
SELECT t0."book_id", t0."title", t0."year", t0."first_author_id", t1."author_id", t1."name", t1."born" FROM "book" AS t0 INNER JOIN "author" AS t1 ON t0."first_author_id" = t1."author_id"
```

A file is only rewritten when its content changes, so the files can be committed and a pull request that changes a join shows the SQL diff next to the Scala diff. `Dump.dumpTable(table)` does the same for a table's `CREATE TABLE` statement, written to `-<dialect>.sql`.

:::warning
Check the dump files before you rely on them:

- A `where(sql"...")` condition loses its parameter in the dump: `t0."year" > $publishedAfter` is written as `WHERE t0."year" >` with no `?`. Use `explain` for filtered queries.
- A `Dump.dumpQuery` call that isn't assigned to a `val` is named after the enclosing object, as `_local_<Object>___query-<dialect>.sql`, and a second such call in the same object overwrites the first one's file.
- Every `sql"..."` fragment also writes its own file, named after its enclosing method or object, while the property is set.
:::

## Putting It Together

This program builds the library, runs the many-to-one join, the one-to-many and left joins, the filtered query, and prints the generated SQL:

```scala title="sql-examples/src/main/scala/writejoinsandinspectgeneratedsql/CompleteExample.scala"
package writejoinsandinspectgeneratedsql

/**
 * How to Write Joins and Inspect the Generated SQL: Complete Example
 *
 * Builds the library, runs the many-to-one join, the one-to-many and left
 * joins, the filtered query, and prints the generated SQL for both dialects.
 *
 * Run with: cd sql-examples && sbt "runMain
 * writejoinsandinspectgeneratedsql.CompleteExample"
 */
object CompleteExample {
  def main(args: Array[String]): Unit = {
    val transactor = Library.open()

    // Many-to-one: each book with its first author
    val booksWithAuthors = SqlQuery.from(books).join(bookAuthor)

    transactor.connect {
      booksWithAuthors.toFrag(SqlDialect.SQLite).query[(Book, Author)].foreach { case (book, author) =>
        println(s"${book.title} (${book.year}) by ${author.name}")
      }
    }

    // One-to-many: each author's books, grouped in Scala
    val authorsAndBooks = SqlQuery.from(authors).join(authorBooks)

    transactor.connect {
      val rows = authorsAndBooks.toFrag(SqlDialect.SQLite).query[(Author, Book)]
      rows.groupMap(_._1)(_._2).toList.sortBy(_._1.name).foreach { case (author, written) =>
        println(s"${author.name}: ${written.sortBy(_.year).map(_.title).mkString(", ")}")
      }
    }

    // Left join: keep authors without books, decoding the optional side as BookColumns
    val everyAuthor = SqlQuery.from(authors).leftJoin(authorBooks)

    transactor.connect {
      val rows = everyAuthor.toFrag(SqlDialect.SQLite).query[(Author, BookColumns)]
      rows.groupMap(_._1)(_._2.toBook).toList.sortBy(_._1.name).foreach { case (author, written) =>
        val titles = written.flatten.map(_.title)
        println(s"${author.name}: ${if (titles.isEmpty) "no books yet" else titles.mkString(", ")}")
      }
    }

    // Filter and sort
    val publishedAfter = 2010

    val recentBooks = SqlQuery
      .from(books)
      .join(bookAuthor)
      .where(sql"""t0."year" > $publishedAfter""")
      .orderBy("year", SortOrder.Desc)

    transactor.connect {
      recentBooks.toFrag(SqlDialect.SQLite).query[(Book, Author)].foreach { case (book, author) =>
        println(s"${book.year}: ${book.title} by ${author.name}")
      }
    }

    // The generated SQL
    println(recentBooks.explain(SqlDialect.SQLite))
    println(recentBooks.explain(SqlDialect.PostgreSQL))
    println(everyAuthor.sql(SqlDialect.SQLite))
  }
}
```

## Running the Examples

All the code in this guide has runnable companions in the `sql-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/sql-examples
```

Each example creates its own temporary SQLite file, so you can run them in any order. To compile all the examples without running them:

```bash
sbt compile
```

### The Library Tables

The domain, the tables, the `Rel` and the code that creates and fills the database are shared by every example:

<details>
  <summary>sql-examples/src/main/scala/writejoinsandinspectgeneratedsql/Library.scala</summary>

```scala title="sql-examples/src/main/scala/writejoinsandinspectgeneratedsql/Library.scala" showLineNumbers
package writejoinsandinspectgeneratedsql

case class Author(authorId: Int, name: String, born: Option[Int])
object Author {
  given Schema[Author] = Schema.derived
}

case class Book(bookId: Int, title: String, year: Int, firstAuthorId: Int)
object Book {
  given Schema[Book] = Schema.derived
}

case class BookColumns(bookId: Option[Int], title: Option[String], year: Option[Int], firstAuthorId: Option[Int]) {
  def toBook: Option[Book] =
    for {
      id     <- bookId
      t      <- title
      y      <- year
      author <- firstAuthorId
    } yield Book(id, t, y, author)
}
object BookColumns {
  given Schema[BookColumns] = Schema.derived
}

/**
 * How to Write Joins and Inspect the Generated SQL: the library tables
 *
 * Defines the `author` and `book` tables, the `Rel`s that link them, and a
 * `Library.open()` that creates and fills a temporary SQLite database. Shared
 * by every example in this package; it has no `main` of its own.
 */
object Library {
  val authors = Table.derived[Author]
  val books   = Table.derived[Book]

  val bookAuthor  = Rel.manyToOne(books, _.firstAuthorId, authors, _.authorId)
  val authorBooks = Rel.oneToMany(authors, _.authorId, books, _.firstAuthorId)

  /**
   * Creates a temporary SQLite file with both tables, inserts four authors and
   * four books, and returns its transactor.
   */
  def open(): JdbcTransactor = {
    val dbFile = java.nio.file.Files.createTempFile("library", ".db")
    dbFile.toFile.deleteOnExit()
    val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

    transactor.connect {
      sql"""CREATE TABLE author (
        author_id INTEGER PRIMARY KEY,
        name      TEXT NOT NULL,
        born      INTEGER
      )""".update
      sql"""CREATE TABLE book (
        book_id         INTEGER PRIMARY KEY,
        title           TEXT NOT NULL,
        year            INTEGER NOT NULL,
        first_author_id INTEGER NOT NULL REFERENCES author (author_id)
      )""".update
    }

    val authorRepo = Repo.derived[Author, Int]
    val bookRepo   = Repo.derived[Book, Int]

    transactor.connect {
      authorRepo.insert(Author(1, "Martin Odersky", Some(1958)))
      authorRepo.insert(Author(2, "John A. De Goes", None))
      authorRepo.insert(Author(3, "Paul Chiusano", None))
      authorRepo.insert(Author(4, "Li Haoyi", None))

      bookRepo.insert(Book(1, "Zionomicon", 2024, 2))
      bookRepo.insert(Book(2, "Programming in Scala", 2008, 1))
      bookRepo.insert(Book(3, "Functional Programming in Scala", 2014, 3))
      bookRepo.insert(Book(4, "Programming in Scala, Fifth Edition", 2021, 1))
    }

    transactor
  }
}
```

</details>

### Join Each Book to Its First Author

<details>
  <summary>sql-examples/src/main/scala/writejoinsandinspectgeneratedsql/Step1ManyToOneExample.scala</summary>

```scala title="sql-examples/src/main/scala/writejoinsandinspectgeneratedsql/Step1ManyToOneExample.scala" showLineNumbers
package writejoinsandinspectgeneratedsql

/**
 * How to Write Joins and Inspect the Generated SQL: Join Each Book to Its First
 * Author
 *
 * Joins each book to its first author with Rel.manyToOne and decodes rows as
 * (Book, Author) tuples.
 *
 * Run with: cd sql-examples && sbt "runMain
 * writejoinsandinspectgeneratedsql.Step1ManyToOneExample"
 */
object Step1ManyToOneExample {
  def main(args: Array[String]): Unit = {
    val transactor = Library.open()

    val booksWithAuthors = SqlQuery.from(books).join(bookAuthor)

    transactor.connect {
      booksWithAuthors.toFrag(SqlDialect.SQLite).query[(Book, Author)].foreach { case (book, author) =>
        println(s"${book.title} (${book.year}) by ${author.name}")
      }
    }
  }
}
```

</details>

Run it with:

```bash
sbt "runMain writejoinsandinspectgeneratedsql.Step1ManyToOneExample"
```

### List Each Author's Books

<details>
  <summary>sql-examples/src/main/scala/writejoinsandinspectgeneratedsql/Step2OneToManyExample.scala</summary>

```scala title="sql-examples/src/main/scala/writejoinsandinspectgeneratedsql/Step2OneToManyExample.scala" showLineNumbers
package writejoinsandinspectgeneratedsql

/**
 * How to Write Joins and Inspect the Generated SQL: List Each Author's Books
 *
 * Starts from authors, joins with Rel.oneToMany and groups the rows by author
 * in Scala.
 *
 * Run with: cd sql-examples && sbt "runMain
 * writejoinsandinspectgeneratedsql.Step2OneToManyExample"
 */
object Step2OneToManyExample {
  def main(args: Array[String]): Unit = {
    val transactor = Library.open()

    val authorsAndBooks = SqlQuery.from(authors).join(authorBooks)

    transactor.connect {
      val rows = authorsAndBooks.toFrag(SqlDialect.SQLite).query[(Author, Book)]
      rows.groupMap(_._1)(_._2).toList.sortBy(_._1.name).foreach { case (author, written) =>
        println(s"${author.name}: ${written.sortBy(_.year).map(_.title).mkString(", ")}")
      }
    }
  }
}
```

</details>

Run it with:

```bash
sbt "runMain writejoinsandinspectgeneratedsql.Step2OneToManyExample"
```

### Keep Authors Who Have No Books

<details>
  <summary>sql-examples/src/main/scala/writejoinsandinspectgeneratedsql/Step3LeftJoinExample.scala</summary>

```scala title="sql-examples/src/main/scala/writejoinsandinspectgeneratedsql/Step3LeftJoinExample.scala" showLineNumbers
package writejoinsandinspectgeneratedsql

/**
 * How to Write Joins and Inspect the Generated SQL: Keep Authors Who Have No
 * Books
 *
 * Left join: shows the bogus Book decoded from NULL columns, and the
 * BookColumns fix that yields Option[Book].
 *
 * Run with: cd sql-examples && sbt "runMain
 * writejoinsandinspectgeneratedsql.Step3LeftJoinExample"
 */
object Step3LeftJoinExample {
  def main(args: Array[String]): Unit = {
    val transactor = Library.open()

    println(SqlQuery.from(authors).leftJoin(authorBooks) == SqlQuery.from(authors).joinLeft(authorBooks))
    println(SqlQuery.from(authors).leftJoin(authorBooks) == SqlQuery.from(authors).join(authorBooks, JoinKind.Left))

    val everyAuthor = SqlQuery.from(authors).leftJoin(authorBooks)

    transactor.connect {
      everyAuthor.toFrag(SqlDialect.SQLite).query[(Author, Book)].foreach(println)
    }

    transactor.connect {
      val rows = everyAuthor.toFrag(SqlDialect.SQLite).query[(Author, BookColumns)]
      rows.groupMap(_._1)(_._2.toBook).toList.sortBy(_._1.name).foreach { case (author, written) =>
        val titles = written.flatten.map(_.title)
        println(s"${author.name}: ${if (titles.isEmpty) "no books yet" else titles.mkString(", ")}")
      }
    }
  }
}
```

</details>

Run it with:

```bash
sbt "runMain writejoinsandinspectgeneratedsql.Step3LeftJoinExample"
```

### Filter and Sort a Join

<details>
  <summary>sql-examples/src/main/scala/writejoinsandinspectgeneratedsql/Step4FilterSortExample.scala</summary>

```scala title="sql-examples/src/main/scala/writejoinsandinspectgeneratedsql/Step4FilterSortExample.scala" showLineNumbers
package writejoinsandinspectgeneratedsql

/**
 * How to Write Joins and Inspect the Generated SQL: Filter and Sort a Join
 *
 * Adds where conditions using table aliases and sorts with orderBy.
 *
 * Run with: cd sql-examples && sbt "runMain
 * writejoinsandinspectgeneratedsql.Step4FilterSortExample"
 */
object Step4FilterSortExample {
  def main(args: Array[String]): Unit = {
    val transactor = Library.open()

    val publishedAfter = 2010

    val recentBooks = SqlQuery
      .from(books)
      .join(bookAuthor)
      .where(sql"""t0."year" > $publishedAfter""")
      .orderBy("year", SortOrder.Desc)

    transactor.connect {
      recentBooks.toFrag(SqlDialect.SQLite).query[(Book, Author)].foreach { case (book, author) =>
        println(s"${book.year}: ${book.title} by ${author.name}")
      }
    }

    val author = "Martin Odersky"

    val booksByOdersky = SqlQuery
      .from(books)
      .join(bookAuthor)
      .where(sql"""t1."name" = $author""")
      .where(sql"""t0."year" < ${2020}""")

    transactor.connect {
      booksByOdersky.toFrag(SqlDialect.SQLite).query[(Book, Author)].foreach { case (book, _) =>
        println(book.title)
      }
    }
  }
}
```

</details>

Run it with:

```bash
sbt "runMain writejoinsandinspectgeneratedsql.Step4FilterSortExample"
```

### See the SQL Before It Runs

<details>
  <summary>sql-examples/src/main/scala/writejoinsandinspectgeneratedsql/Step5ExplainExample.scala</summary>

```scala title="sql-examples/src/main/scala/writejoinsandinspectgeneratedsql/Step5ExplainExample.scala" showLineNumbers
package writejoinsandinspectgeneratedsql

/**
 * How to Write Joins and Inspect the Generated SQL: See the SQL Before It Runs
 *
 * Prints explain for SQLite and PostgreSQL, and the exact SQL string toFrag
 * sends to the driver.
 *
 * Run with: cd sql-examples && sbt "runMain
 * writejoinsandinspectgeneratedsql.Step5ExplainExample"
 */
object Step5ExplainExample {
  def main(args: Array[String]): Unit = {
    val transactor = Library.open()

    val publishedAfter = 2010

    val recentBooks = SqlQuery
      .from(books)
      .join(bookAuthor)
      .where(sql"""t0."year" > $publishedAfter""")
      .orderBy("year", SortOrder.Desc)

    println(recentBooks.explain(SqlDialect.SQLite))
    println(recentBooks.explain(SqlDialect.PostgreSQL))

    val everyAuthor = SqlQuery.from(authors).leftJoin(authorBooks)

    println(everyAuthor.sql(SqlDialect.SQLite))
  }
}
```

</details>

Run it with:

```bash
sbt "runMain writejoinsandinspectgeneratedsql.Step5ExplainExample"
```

### Dump the SQL to Files at Compile Time

<details>
  <summary>sql-examples/src/main/scala/writejoinsandinspectgeneratedsql/JoinsDumpExample.scala</summary>

```scala title="sql-examples/src/main/scala/writejoinsandinspectgeneratedsql/JoinsDumpExample.scala" showLineNumbers
package writejoinsandinspectgeneratedsql

object LibraryQueries {
  inline def booksWithFirstAuthors: SqlQuery[Book] =
    SqlQuery.from(Library.books).join(Rel.manyToOne(Library.books, _.firstAuthorId, Library.authors, _.authorId))
}

/**
 * How to Write Joins and Inspect the Generated SQL: Dump the SQL to Files at
 * Compile Time
 *
 * `Dump.dumpQuery` writes the query's SQL to a `.sql` file while this file
 * compiles, when the compiler's JVM has the `zib.sql.dumpDir` system property
 * set. `main` prints the files it produced.
 *
 * Run with: cd sql-examples && sbt -Dzib.sql.dumpDir=target/sql-dumps clean
 * compile "runMain writejoinsandinspectgeneratedsql.JoinsDumpExample"
 */
object JoinsDumpExample {
  // The val's name becomes the file name
  val booksWithFirstAuthors = Dump.dumpQuery(LibraryQueries.booksWithFirstAuthors)

  def main(args: Array[String]): Unit = {
    val dir   = java.nio.file.Paths.get("target/sql-dumps")
    val files =
      if (java.nio.file.Files.isDirectory(dir)) {
        val stream = java.nio.file.Files.list(dir)
        try
          stream.toArray.toList
            .map(_.asInstanceOf[java.nio.file.Path])
            .filter(p =>
              p.getFileName.toString.startsWith("booksWithFirstAuthors") && p.getFileName.toString.endsWith(".sql")
            )
            .sortBy(_.getFileName.toString)
        finally stream.close()
      } else Nil

    if (files.isEmpty)
      println("No dump files found. Recompile with: sbt -Dzib.sql.dumpDir=target/sql-dumps clean compile")
    else
      files.foreach { file =>
        println(s"== ${file.getFileName}")
        println(new String(java.nio.file.Files.readAllBytes(file), "UTF-8").trim)
      }
  }
}
```

</details>

Compile with the dump directory set, then run the example to print the files it produced:

```bash
sbt -Dzib.sql.dumpDir=target/sql-dumps clean compile "runMain writejoinsandinspectgeneratedsql.JoinsDumpExample"
```

### The Complete Example

```bash
sbt "runMain writejoinsandinspectgeneratedsql.CompleteExample"
```

## Going Further

- [SqlQuery](../reference/sql/sql-query.md), [Rel](../reference/sql/rel.md) and [Dump](../reference/sql/dump.md) are the reference pages for every builder method, `Rel` constructor and dump rule this guide uses.
- [Frag](../reference/sql/frag.md) documents `Frag#query`, `Frag#queryOne` and the other ways to run the fragment `SqlQuery#toFrag` returns.
- [How to Stream Large Query Results](./stream-large-query-results.md) reads a large join as a stream of row chunks: call `Frag#queryStream` or `Frag#queryChunked` on the fragment `SqlQuery#toFrag` returns, instead of `Frag#query`.
- [DbCodec](../reference/sql/db-codec.md) explains how rows decode into case classes and tuples, and how SQL `NULL` is handled.
- [sql(table) Checked Interpolation](./sql-checked-interpolation.md) checks table and column names in hand-written SQL at compile time, for joins that `SqlQuery` can't express.
- [How to Test Database Code Against SQLite](./test-database-code-against-sqlite.md) gives every test its own database, which suits tests for join queries.
- [SqlLogger](../reference/sql/sql-logger.md) records the SQL of every query a transactor runs, including joins, with its duration and row count.

---

## Render Your First Server-Side HTML Page

Welcome! This tutorial is for Scala developers who want to generate HTML on the server with ZIO Blocks. If you can write a case class, a function and a `List#map`, and you know what an HTML tag looks like, you have everything you need. You don't need any prior experience with templating libraries, web frameworks or ZIO.

## 1. Introduction

By the end of this tutorial, you will understand:

- How a page is built as a tree of `Dom` values and turned into a string with `Dom#render`
- How elements, attributes and Scala collections compose into a full page, and how the compiler checks which children an element accepts
- Why user input is safe to put in a page: text, URLs and embedded JSON are each escaped for the place they land
- How to add a typed style sheet with the `css"..."` interpolator and an embedded JSON state with `ToJs`
- How to query a page with `Dom#select` to check what you built

We learn these concepts by building a detail page for the book *Zionomicon*, one step at a time:

1. Render your first element
2. Compose a page from smaller pieces
3. Show user input safely
4. Style the page with CSS
5. Embed the book as JSON
6. Check the page with selectors

At the end, we save the finished page as `book.html` and open it in a browser. We recommend reading from top to bottom — each section builds on the previous one.

To follow along, add the HTML module to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-html" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-html" % "0.0.56"
```

Scala 3.9 and newer use the `zio-blocks-next-html` artifact name. The HTML module depends on the schema module, so `Schema` is available without another dependency. The code in this tutorial is the same for every Scala version.

Section 7 uses a `Schema`, so you should finish the [Your First Schema](./your-first-schema.md) tutorial first.

## 2. Background: A Page Is a Tree of Values

A web page is a tree. The `<html>` element contains a `<head>` and a `<body>`, the body contains headings and lists, and a list contains items. Many server-side tools build that tree by gluing strings together in a template file. That works until a piece of user input contains a `<` and breaks the page. It also breaks when a typo leaves a tag unclosed, or when a `` ends up inside a ``.

The ZIO Blocks HTML module takes a different approach. Every tag is a Scala function, and calling it returns an immutable `Dom` value. You build the page by calling those functions. Ordinary Scala code (`List#map`, `if`, helper functions) decides what goes in, and the compiler checks what you build. Only at the very end does `Dom#render` turn the tree into a string, and that is where every piece of text is escaped. No HTML is written by hand, so no tag can be left unclosed and no user input can be mistaken for markup.

## 3. Render Your First Element

A `Dom` is one node of an HTML tree: an element such as ``, a piece of text, or a doctype. The single import `zio.blocks.html._` brings in a function for every HTML tag. Calling a tag function with a `String` gives an element with that text inside.

Let's create a heading for our book:

```scala

val heading = h1("Zionomicon")
```

The code above has two parts:

- `import zio.blocks.html._` brings in the whole HTML DSL: tag functions such as `h1`, `div` and `ul`, attribute keys such as `id` and `href`, and the `css"..."` interpolator we use later.
- `h1("Zionomicon")` returns a `Dom.Element` for an `` tag with the text `Zionomicon` as its only child. Nothing is rendered yet; `heading` is a plain value.

To see the HTML, we call `Dom#render` and print the string it returns:

```scala
println(heading.render)
// Zionomicon
```

That's our first piece of HTML. Elements nest by passing one element to another. Let's put the heading and a paragraph inside a ``:

```scala
val summary = div(
  h1("Zionomicon"),
  p("by John A. De Goes, Adam Fraser and Milad Khajavi")
)
```

The code above passes two elements to `div`, which makes them its children in the order we wrote them. Now let's print it twice — once with `Dom#render` and once with `render(indent)`, which puts each child on its own line:

```scala
println(summary.render)
// Zionomiconby John A. De Goes, Adam Fraser and Milad Khajavi

println(summary.render(indent = 2))
// 
//   Zionomicon
//   by John A. De Goes, Adam Fraser and Milad Khajavi
// 
```

The code above prints the same tree in two layouts:

- `summary.render` gives compact HTML with no extra whitespace, which is what we send to a browser.
- `summary.render(indent = 2)` indents each nesting level by two spaces, which is easier to read while we learn. We use it for the rest of this tutorial.

## 4. Compose a Page From Smaller Pieces

A real page needs a `<head>` with metadata, a `<body>` with content, and attributes such as `lang` and `charset`. An attribute is written as `key := value` and passed to the tag function along with its children. Because a page is just a value, we can build it with ordinary Scala functions that take our data and return a `Dom.Element`.

Let's start with the data. We describe a book as a case class and derive a `Schema` for it, exactly as in [Your First Schema](./your-first-schema.md). We don't need the schema until section 7, but deriving it now lets us keep one `Book` type for the whole tutorial:

```scala

final case class Book(title: String, authors: List[String], year: Int)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

val zionomicon = Book("Zionomicon", List("John A. De Goes", "Adam Fraser", "Milad Khajavi"), 2024)

val related = List(
  Book("Programming in Scala", List("Martin Odersky", "Lex Spoon", "Bill Venners"), 2008),
  Book("Functional Programming in Scala", List("Paul Chiusano", "Rúnar Bjarnason"), 2014)
)
```

The code above defines our sample data:

- `Book` holds a title, the list of authors and the year of publication.
- `zionomicon` is the book the page is about.
- `related` is a list of two other Scala books that we show as recommendations.

Now let's write two functions. `bookLine` turns a book into one line of text, and `bookPage` builds the whole page:

```scala
def bookLine(book: Book): String =
  s"${book.title} by ${book.authors.mkString(", ")} (${book.year})"

def bookPage(book: Book, related: List[Book]): Dom.Element =
  html(lang := "en")(
    head(
      meta(charset := "utf-8"),
      title(book.title)
    ),
    body(
      h1(book.title),
      p(bookLine(book)),
      h2("Related books"),
      ul(related.map(relatedBook => li(bookLine(relatedBook))))
    )
  )
```

The code above builds the page shell:

- `html(lang := "en")(...)` creates the `<html>` element with a `lang` attribute, then adds the head and the body as its children.
- `meta(charset := "utf-8")` is a void element — an element that has attributes but never children, such as `<meta>`, `` and ``. Its tag function accepts attributes only.
- `title(book.title)` sets the text shown on the browser tab.
- `related.map(relatedBook => li(bookLine(relatedBook)))` turns the list of books into a list of `` items with ordinary `List#map`.
- `ul(...)` wraps the items in an unordered list.

Let's render the page for Zionomicon:

```scala
println(bookPage(zionomicon, related).render(indent = 2))
// <html lang="en">
//   <head>
//     <meta charset="utf-8"/>
//     <title>Zionomicon</title>
//   </head>
//   <body>
//     Zionomicon
//     Zionomicon by John A. De Goes, Adam Fraser, Milad Khajavi (2024)
//     Related books
//     
//       Programming in Scala by Martin Odersky, Lex Spoon, Bill Venners (2008)
//       Functional Programming in Scala by Paul Chiusano, Rúnar Bjarnason (2014)
//     
//   </body>
// </html>
```

Notice that `<meta charset="utf-8"/>` has no closing tag, and that each related book became one ``. We never wrote a closing tag ourselves, so none can be missing.

The compiler also knows which children each element may have. In HTML, a `` may only contain `` items, so the `ul` function accepts `li` elements and attributes but nothing else. Let's try to put a paragraph directly inside a list:

```scala
ul(p("Zionomicon"))
// error: 
// None of the overloaded alternatives of method ul in trait HtmlElements with types
//  (children: Iterable[zio.blocks.html.Dom.Element.Li]):
//   zio.blocks.html.Dom.Element
//  (effect: zio.blocks.html.Dom.Attribute | zio.blocks.html.Dom.Element.Li, effects
//   : (zio.blocks.html.Dom.Attribute | zio.blocks.html.Dom.Element.Li)*):
//   zio.blocks.html.Dom.Element
//  (): zio.blocks.html.Dom.Element
// match arguments (zio.blocks.html.Dom.Element)
```

The code above does not compile: `p(...)` is not a list item, so the mistake is caught before the page is ever rendered. The error lists what `ul` accepts instead: attributes and `Dom.Element.Li` values, which is the type `li(...)` returns. The same rule applies to tables (`table` takes `tr` rows, `tr` takes `th` and `td` cells) and to drop-down lists (a `<select>` takes `<option>` elements). Void elements are checked the same way, so `meta(p("x"))` does not compile either.

## 5. Show User Input Safely

Pages often show text that users wrote, such as reviews and comments. That text can contain anything, including HTML. If it were pasted into the page as-is, a review containing `<script>` would run its code in every reader's browser. This attack is called cross-site scripting (XSS).

With the HTML module, we don't need to remember to escape anything. A `String` passed as a child is always text, never markup, and `Dom#render` escapes it. Attributes that hold a URL, such as `href` and `src`, get an extra check: a `javascript:`, `vbscript:` or `data:text/html` link is prefixed with `unsafe:`, so the browser no longer treats it as code.

Let's model a review that a user submitted, with a hostile message and a hostile website link:

```scala
final case class Review(reviewer: String, text: String, website: String)

object Review {
  implicit val schema: Schema[Review] = Schema.derived
}

val review = Review(
  reviewer = "mallory",
  text = "Loved it! <script>alert('stolen cookies')</script>",
  website = "javascript:alert('stolen cookies')"
)

def reviewBlock(review: Review): Dom.Element =
  div(
    className := "review",
    p(review.text),
    a(href := review.website, s"Visit ${review.reviewer}'s website")
  )
```

The code above:

- `Review` holds the reviewer's name, the review text and a link to the reviewer's website, all supplied by a user. Like `Book`, it gets a `Schema` that we use in section 7.
- `review.text` contains a `<script>` tag, and `review.website` is a `javascript:` URL — two common XSS attempts.
- `reviewBlock` puts the text in a paragraph and the link in an `` element, exactly as we would for a friendly review. Attributes and children can be mixed in one call; `className := "review"` sets the `class` attribute, which we style in the next section.

Let's render the block:

```scala
println(reviewBlock(review).render(indent = 2))
// 
//   Loved it! &lt;script&gt;alert(&#x27;stolen cookies&#x27;)&lt;/script&gt;
//   Visit mallory&#x27;s website
// 
```

Notice what happened to each piece of user input:

- In the paragraph, `<` and `>` became `&lt;` and `&gt;`, and the quotes became `&#x27;`. The browser shows the review text literally instead of running it.
- In `href`, the link now starts with `unsafe:`, so clicking it does nothing harmful.
- The apostrophe in `mallory's` was escaped too — every piece of text goes through the same rule, so there is no special case to forget.

:::note
Escaping happens in `Dom#render`, not when the value is created. The tree keeps the original text, which is why the same `Dom` value can be rendered many times without ever being escaped twice.
:::

## 6. Style the Page With CSS

A page usually comes with a style sheet. The `css"..."` interpolator builds a `Css` value from CSS text. Values interpolated into it are converted with a `ToCss` instance for their type. That way a length or a color is written by a typed value instead of a hand-written string.

Let's write the style sheet for our book page:

```scala
val pageStyle: Css =
  css"""body { font-family: sans-serif; max-width: ${CssLength(40, "em")}; margin: 0 auto; }
.review { border-left: ${CssLength(4, "px")} solid ${CssColor.Rgb(200, 60, 40)}; padding-left: ${CssLength(1, "em")}; }"""
```

The code above:

- `css"""..."""` turns the text into a `Css` value. The interpolator works like the `s"..."` string interpolator, but each `${...}` must have a `ToCss` instance.
- `CssLength(40, "em")` is a length with a unit. The unit is checked when the value is created, so a typo such as `"emm"` throws an exception instead of producing CSS the browser silently ignores.
- `CssColor.Rgb(200, 60, 40)` is a color written as red, green and blue components.

To put the style sheet into the page, we pass it to the `style` tag function, which creates a `<style>` element:

```scala
println(style(pageStyle).render)
// <style>body { font-family: sans-serif; max-width: 40em; margin: 0 auto; }
// .review { border-left: 4px solid rgb(200,60,40); padding-left: 1em; }</style>
```

The code above prints the `<style>` element. Notice that each typed value was rendered in CSS syntax — `40em`, `4px` and `rgb(200,60,40)` — and that the second rule targets the `review` class we gave the review block in the previous section.

## 7. Embed the Book as JSON

Pages often carry data for JavaScript running in the browser, written as JSON inside a `<script>` element. The type class `ToJs` turns a Scala value into JavaScript source text. For any type that has a `Schema`, the instance `ToJs.fromSchema` is found automatically and encodes the value as JSON, using the same JSON codec as `toJsonString` in [Your First Schema](./your-first-schema.md).

Let's encode the book and wrap the JSON in a `<script>` element:

```scala
val bookData = script(`type` := "application/json", id := "book-data", Js(ToJs[Book].toJs(zionomicon)))
```

The code above:

- `ToJs[Book]` asks the compiler for a `ToJs` instance for `Book`. Because `Book.schema` is in scope, it finds `ToJs.fromSchema`.
- `ToJs#toJs` encodes `zionomicon` as a JSON string.
- `Js(...)` marks that string as JavaScript source for the `script` element. The `script` function only accepts attributes and `Js` values, so plain text can't end up inside a `<script>` by accident.
- `` `type` := "application/json" `` tells the browser the script holds data, not code. `type` is a Scala keyword, so it is written in backticks.
- `id := "book-data"` gives the browser's JavaScript a way to find the element.

Let's render it:

```scala
println(bookData.render)
// <script type="application/json" id="book-data">{"title":"Zionomicon","authors":["John A. De Goes","Adam Fraser","Milad Khajavi"],"year":2024}</script>
```

Text inside `<script>` is not HTML, so it can't be escaped with `&lt;`. JSON that contains `</script>` would end the element early and let the rest run as code. That is why `ToJs.fromSchema` writes every `<` and `>` as a JSON unicode escape. Let's encode our hostile review from section 5 the same way:

```scala
println(ToJs[Review].toJs(review))
// {"reviewer":"mallory","text":"Loved it! \u003cscript\u003ealert('stolen cookies')\u003c/script\u003e","website":"javascript:alert('stolen cookies')"}
```

Notice that `<script>` became `\u003cscript\u003e`. JavaScript reads it back as the original text, but the HTML parser never sees a tag. The same review is now safe in three places: as text, as a link and as JSON, each escaped in the way that place needs.

## 8. Check the Page With Selectors

Because a page is a tree of values, we can query it before we render it. `Dom#select` takes a `CssSelector` and returns a `DomSelection` with every element that matches, much like `document.querySelectorAll` in a browser. This is useful in tests: we can check the structure of a page without comparing long strings.

Let's find all list items on the book page:

```scala
val page = bookPage(zionomicon, related)

val items = page.select(CssSelector.element("li"))
```

The code above:

- `bookPage(zionomicon, related)` builds the page from section 4 again.
- `CssSelector.element("li")` is a selector that matches every `` element, like the CSS selector `li`.
- `page.select(...)` searches the whole tree below `page` and collects the matches.

Now let's count the matches and print their text:

```scala
println(items.length)
// 2

items.texts.foreach(println)
// Programming in Scala by Martin Odersky, Lex Spoon, Bill Venners (2008)
// Functional Programming in Scala by Paul Chiusano, Rúnar Bjarnason (2014)
```

The code above confirms what we expected: `DomSelection#length` says there are two items, one per related book, and `DomSelection#texts` returns the text inside each match. If someone later breaks `bookPage`, a test with these two lines catches it.

## 9. Putting It Together

This example combines all the concepts we've learned. It builds the book page with its style sheet, the hostile review and the embedded JSON. Then it checks the related-book items with a selector and writes the page to `book.html` with a `<!DOCTYPE html>` line in front:

```scala title="html-examples/src/main/scala/yourfirsthtmlpage/CompleteExample.scala"
package yourfirsthtmlpage

/**
 * Render Your First Server-Side HTML Page — Complete Example
 *
 * Builds the book page with its style sheet, a hostile review and embedded
 * JSON, checks the related-book items with a selector and writes the page to
 * book.html.
 *
 * Run with: cd html-examples && sbt "runMain yourfirsthtmlpage.CompleteExample"
 */
object CompleteExample extends App {

  final case class Book(title: String, authors: List[String], year: Int)

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  final case class Review(reviewer: String, text: String, website: String)

  object Review {
    implicit val schema: Schema[Review] = Schema.derived
  }

  val zionomicon = Book("Zionomicon", List("John A. De Goes", "Adam Fraser", "Milad Khajavi"), 2024)

  val related = List(
    Book("Programming in Scala", List("Martin Odersky", "Lex Spoon", "Bill Venners"), 2008),
    Book("Functional Programming in Scala", List("Paul Chiusano", "Rúnar Bjarnason"), 2014)
  )

  val review = Review(
    reviewer = "mallory",
    text = "Loved it! <script>alert('stolen cookies')</script>",
    website = "javascript:alert('stolen cookies')"
  )

  def bookLine(book: Book): String =
    s"${book.title} by ${book.authors.mkString(", ")} (${book.year})"

  def reviewBlock(review: Review): Dom.Element =
    div(
      className := "review",
      p(review.text),
      a(href := review.website, s"Visit ${review.reviewer}'s website")
    )

  val pageStyle: Css =
    css"""body { font-family: sans-serif; max-width: ${CssLength(40, "em")}; margin: 0 auto; }
.review { border-left: ${CssLength(4, "px")} solid ${CssColor.Rgb(200, 60, 40)}; padding-left: ${CssLength(1, "em")}; }"""

  val bookData = script(`type` := "application/json", id := "book-data", Js(ToJs[Book].toJs(zionomicon)))

  def bookPage(book: Book, related: List[Book]): Dom.Element =
    html(lang := "en")(
      head(
        meta(charset := "utf-8"),
        title(book.title),
        style(pageStyle)
      ),
      body(
        h1(book.title),
        p(bookLine(book)),
        h2("Related books"),
        ul(related.map(relatedBook => li(bookLine(relatedBook)))),
        h2("Reviews"),
        reviewBlock(review),
        bookData
      )
    )

  val page = bookPage(zionomicon, related)

  val items = page.select(CssSelector.element("li"))
  println(s"Related books on the page: ${items.length}")

  val document = doctype.render + "\n" + page.render(indent = 2)

  val path = Paths.get("book.html")
  Files.write(path, document.getBytes(StandardCharsets.UTF_8))

  println(s"Wrote ${path.toAbsolutePath}")
  println("Open it in your browser to see the page.")
}
```

A few details are new here:

- `doctype.render` gives `<!DOCTYPE html>`. The doctype is not part of the tree, so we put its string and a line break in front of the rendered page.
- `java.nio.file.Files.write` saves the document. The HTML module only builds strings; saving or sending them is up to you.

Run the example, then open the `book.html` file it creates in your browser. You should see the book title, the authors, the related books, and the review shown as plain text with a harmless link.

## 10. Running the Examples

All examples in this tutorial have corresponding runnable Scala files in the `html-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/html-examples
```

Run them in order to follow the tutorial step by step.

### Render Your First Element

This example renders a heading and a small ``, compact and indented.

<details>
  <summary>html-examples/src/main/scala/yourfirsthtmlpage/Concept1FirstElementExample.scala</summary>

```scala title="html-examples/src/main/scala/yourfirsthtmlpage/Concept1FirstElementExample.scala" showLineNumbers
package yourfirsthtmlpage

/**
 * Render Your First Server-Side HTML Page — Concept 1: Render Your First Element
 *
 * Renders a heading and a small div, compact and indented.
 *
 * Run with: cd html-examples && sbt "runMain yourfirsthtmlpage.Concept1FirstElementExample"
 */
object Concept1FirstElementExample extends App {

  val heading = h1("Zionomicon")

  println(heading.render)

  val summary = div(
    h1("Zionomicon"),
    p("by John A. De Goes, Adam Fraser and Milad Khajavi")
  )

  println(summary.render)

  println(summary.render(indent = 2))
}
```

</details>

Run it from the `html-examples` directory:

```bash
sbt "runMain yourfirsthtmlpage.Concept1FirstElementExample"
```

### Compose a Page From Smaller Pieces

This example builds the page shell with a head, a body and a list of related books generated from a `List[Book]`.

<details>
  <summary>html-examples/src/main/scala/yourfirsthtmlpage/Concept2ComposePageExample.scala</summary>

```scala title="html-examples/src/main/scala/yourfirsthtmlpage/Concept2ComposePageExample.scala" showLineNumbers
package yourfirsthtmlpage

/**
 * Render Your First Server-Side HTML Page — Concept 2: Compose a Page From Smaller Pieces
 *
 * Builds the page shell with a head, a body and a list of related books
 * generated from a List[Book].
 *
 * Run with: cd html-examples && sbt "runMain yourfirsthtmlpage.Concept2ComposePageExample"
 */
object Concept2ComposePageExample extends App {

  final case class Book(title: String, authors: List[String], year: Int)

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  val zionomicon = Book("Zionomicon", List("John A. De Goes", "Adam Fraser", "Milad Khajavi"), 2024)

  val related = List(
    Book("Programming in Scala", List("Martin Odersky", "Lex Spoon", "Bill Venners"), 2008),
    Book("Functional Programming in Scala", List("Paul Chiusano", "Rúnar Bjarnason"), 2014)
  )

  def bookLine(book: Book): String =
    s"${book.title} by ${book.authors.mkString(", ")} (${book.year})"

  def bookPage(book: Book, related: List[Book]): Dom.Element =
    html(lang := "en")(
      head(
        meta(charset := "utf-8"),
        title(book.title)
      ),
      body(
        h1(book.title),
        p(bookLine(book)),
        h2("Related books"),
        ul(related.map(relatedBook => li(bookLine(relatedBook))))
      )
    )

  println(bookPage(zionomicon, related).render(indent = 2))
}
```

</details>

Run it from the `html-examples` directory:

```bash
sbt "runMain yourfirsthtmlpage.Concept2ComposePageExample"
```

### Show User Input Safely

This example renders a hostile review and shows the escaped text and the `unsafe:` link.

<details>
  <summary>html-examples/src/main/scala/yourfirsthtmlpage/Concept3EscapingExample.scala</summary>

```scala title="html-examples/src/main/scala/yourfirsthtmlpage/Concept3EscapingExample.scala" showLineNumbers
package yourfirsthtmlpage

/**
 * Render Your First Server-Side HTML Page — Concept 3: Show User Input Safely
 *
 * Renders a hostile review and shows the escaped text and the unsafe: link.
 *
 * Run with: cd html-examples && sbt "runMain yourfirsthtmlpage.Concept3EscapingExample"
 */
object Concept3EscapingExample extends App {

  final case class Review(reviewer: String, text: String, website: String)

  object Review {
    implicit val schema: Schema[Review] = Schema.derived
  }

  val review = Review(
    reviewer = "mallory",
    text = "Loved it! <script>alert('stolen cookies')</script>",
    website = "javascript:alert('stolen cookies')"
  )

  def reviewBlock(review: Review): Dom.Element =
    div(
      className := "review",
      p(review.text),
      a(href := review.website, s"Visit ${review.reviewer}'s website")
    )

  println(reviewBlock(review).render(indent = 2))
}
```

</details>

Run it from the `html-examples` directory:

```bash
sbt "runMain yourfirsthtmlpage.Concept3EscapingExample"
```

### Style the Page With CSS

This example builds a style sheet with typed lengths and a color and renders it in a `<style>` element.

<details>
  <summary>html-examples/src/main/scala/yourfirsthtmlpage/Concept4CssExample.scala</summary>

```scala title="html-examples/src/main/scala/yourfirsthtmlpage/Concept4CssExample.scala" showLineNumbers
package yourfirsthtmlpage

/**
 * Render Your First Server-Side HTML Page — Concept 4: Style the Page With CSS
 *
 * Builds a style sheet with typed lengths and a color and renders it in a
 * style element.
 *
 * Run with: cd html-examples && sbt "runMain yourfirsthtmlpage.Concept4CssExample"
 */
object Concept4CssExample extends App {

  val pageStyle: Css =
    css"""body { font-family: sans-serif; max-width: ${CssLength(40, "em")}; margin: 0 auto; }
.review { border-left: ${CssLength(4, "px")} solid ${CssColor.Rgb(200, 60, 40)}; padding-left: ${CssLength(1, "em")}; }"""

  println(style(pageStyle).render)
}
```

</details>

Run it from the `html-examples` directory:

```bash
sbt "runMain yourfirsthtmlpage.Concept4CssExample"
```

### Embed the Book as JSON

This example embeds the book as JSON in a `<script>` element and shows how `<` and `>` in a review are escaped.

<details>
  <summary>html-examples/src/main/scala/yourfirsthtmlpage/Concept5JsonStateExample.scala</summary>

```scala title="html-examples/src/main/scala/yourfirsthtmlpage/Concept5JsonStateExample.scala" showLineNumbers
package yourfirsthtmlpage

/**
 * Render Your First Server-Side HTML Page — Concept 5: Embed the Book as JSON
 *
 * Embeds the book as JSON in a script element and shows how < and > in a
 * review are escaped.
 *
 * Run with: cd html-examples && sbt "runMain yourfirsthtmlpage.Concept5JsonStateExample"
 */
object Concept5JsonStateExample extends App {

  final case class Book(title: String, authors: List[String], year: Int)

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  final case class Review(reviewer: String, text: String, website: String)

  object Review {
    implicit val schema: Schema[Review] = Schema.derived
  }

  val zionomicon = Book("Zionomicon", List("John A. De Goes", "Adam Fraser", "Milad Khajavi"), 2024)

  val review = Review(
    reviewer = "mallory",
    text = "Loved it! <script>alert('stolen cookies')</script>",
    website = "javascript:alert('stolen cookies')"
  )

  val bookData = script(`type` := "application/json", id := "book-data", Js(ToJs[Book].toJs(zionomicon)))

  println(bookData.render)

  println(ToJs[Review].toJs(review))
}
```

</details>

Run it from the `html-examples` directory:

```bash
sbt "runMain yourfirsthtmlpage.Concept5JsonStateExample"
```

### Check the Page With Selectors

This example selects the `` items of the book page, counts them and prints their text.

<details>
  <summary>html-examples/src/main/scala/yourfirsthtmlpage/Concept6SelectExample.scala</summary>

```scala title="html-examples/src/main/scala/yourfirsthtmlpage/Concept6SelectExample.scala" showLineNumbers
package yourfirsthtmlpage

/**
 * Render Your First Server-Side HTML Page — Concept 6: Check the Page With Selectors
 *
 * Selects the li items of the book page, counts them and prints their text.
 *
 * Run with: cd html-examples && sbt "runMain yourfirsthtmlpage.Concept6SelectExample"
 */
object Concept6SelectExample extends App {

  final case class Book(title: String, authors: List[String], year: Int)

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  val zionomicon = Book("Zionomicon", List("John A. De Goes", "Adam Fraser", "Milad Khajavi"), 2024)

  val related = List(
    Book("Programming in Scala", List("Martin Odersky", "Lex Spoon", "Bill Venners"), 2008),
    Book("Functional Programming in Scala", List("Paul Chiusano", "Rúnar Bjarnason"), 2014)
  )

  def bookLine(book: Book): String =
    s"${book.title} by ${book.authors.mkString(", ")} (${book.year})"

  def bookPage(book: Book, related: List[Book]): Dom.Element =
    html(lang := "en")(
      head(
        meta(charset := "utf-8"),
        title(book.title)
      ),
      body(
        h1(book.title),
        p(bookLine(book)),
        h2("Related books"),
        ul(related.map(relatedBook => li(bookLine(relatedBook))))
      )
    )

  val page = bookPage(zionomicon, related)

  val items = page.select(CssSelector.element("li"))

  println(items.length)

  items.texts.foreach(println)
}
```

</details>

Run it from the `html-examples` directory:

```bash
sbt "runMain yourfirsthtmlpage.Concept6SelectExample"
```

### The Complete Example

The complete example from "Putting It Together" builds the full page and writes it to `book.html` in the `html-examples` directory:

```bash
sbt "runMain yourfirsthtmlpage.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## 11. What You've Learned

In this tutorial, you learned:

- That a page is a tree of immutable `Dom` values built by calling tag functions, and that `Dom#render` and `render(indent)` turn it into HTML
- How attributes (`key := value`), void elements and Scala collections compose into a full page, and how the compiler rejects children an element can't hold
- That user input is escaped for where it lands: text becomes `&lt;...&gt;`, dangerous URLs get an `unsafe:` prefix, and JSON in a `<script>` gets unicode escapes
- How to write a style sheet with `css"..."` and typed values such as `CssLength` and `CssColor`
- How `ToJs` derives a JSON encoder from a `Schema`, and how `Dom#select` with a `CssSelector` checks a page's structure

You now have a solid foundation in server-side HTML with ZIO Blocks. The next step is to make the page interactive and serve it over HTTP.

## 12. Where to Go Next

- **Want the full HTML API?** Read the [HTML](../reference/html.md) reference page, which covers the `html"..."` template interpolator, the `Css` data type, `Dom#transform`, and every `DomSelection` operation.
- **Ready to make the page interactive?** [Add Interactivity to a Page with HTMX](./add-interactivity-with-htmx.md) turns a book list into a live search with typed `hx-*` attributes; the [HTMX](../reference/htmx/index.md) reference covers every attribute.
- **Want to serve the page over HTTP?** [How to Serve http-model Requests with the JDK HTTP Server](./serve-http-model-requests.md) shows how to hand an http-model `Response` to the server built into the JDK.
- **Prefer reactive signals?** [Datastar](../reference/datastar/index.md) builds on the HTML module with typed `data-*` attributes and server-sent events.
- **Want more from your schemas?** Return to [Your First Schema](./your-first-schema.md) and follow its links to the JSON codec and the other formats one `Schema` can produce.

---

## Your First Schema: Round-Trip a Type Through JSON

Welcome! This tutorial is for Scala developers who are new to ZIO Blocks. If you can write a case class, a companion object and an `Either` pattern match, you have everything you need. You don't need any prior experience with schemas, codecs or JSON libraries to follow along.

## 1. Introduction

By the end of this tutorial, you will understand:

- What a `Schema` is and how to derive one for your own case classes
- How to encode a value to JSON and decode it back into the same value
- How nested case classes, optional fields and lists fit into one schema without extra work
- How to read a `SchemaError` and use its path to fix bad input
- Why one schema can also give you a generic data view and a JSON Schema document

We learn these concepts by building a tiny book catalog, one step at a time:

1. Describe a type with a schema
2. Encode a value to JSON
3. Decode JSON back into a value
4. Grow the model with nested types, optional fields and lists
5. Read and fix a decode error
6. Look at the same data two more ways

We recommend reading from top to bottom — each section builds on the previous one.

To follow along, add the schema module to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56"
scalacOptions += "-experimental"
```

Scala 3.9 and newer use the `zio-blocks-next-schema` artifact name; the code in this tutorial is the same for every Scala version. For scala-cli, Scala.js, or cross-building, see [How to Add ZIO Blocks to Your Build](../getting-started/installation.md).

## 2. Background: One Description, Many Uses

Most Scala programs describe the same data several times. A case class says what a `Book` is, a JSON encoder says how to write it, a JSON decoder says how to read it, and a JSON Schema document tells other teams what to send. Each description is written by hand, and each one can drift away from the others when the case class changes.

ZIO Blocks takes a different approach. You describe the shape of a type once, as a `Schema`. Everything else — the JSON codec, a generic data view, the JSON Schema document, and the other formats ZIO Blocks supports — is derived from that one description. When the case class changes, the schema is derived again at compile time, and every view follows. In this tutorial we watch that happen for JSON.

## 3. Describe a Type with a Schema

A `Schema[A]` is a value that describes the structure of the type `A`: its name, its fields and their types. We rarely write one by hand. Instead, we ask the compiler to derive it with `Schema.derived` and store it as an implicit value in the type's companion object, where the rest of the library can find it.

Let's start with the smallest book we can think of — a title and a publication year:

```scala

case class Book(title: String, year: Int)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}
```

The code above has three parts:

- `import zio.blocks.schema._` brings `Schema` and the extension methods we use later into scope.
- `case class Book(title: String, year: Int)` is an ordinary case class; nothing about it is special to ZIO Blocks.
- `implicit val schema: Schema[Book] = Schema.derived` derives the schema at compile time and stores it where the compiler finds it whenever code asks for a `Schema[Book]`.

A schema is a plain value, so we can print it and look at what the compiler derived:

```scala
println(Book.schema)
// Schema {
//   record Book {
//     title: String
//     year: Int
//   }
// }
```

The code above prints `Book.schema`, the value we derived in the previous step. Notice that the schema records the type as a `record` named `Book` with two fields, in the order we declared them. That's the whole description — and it is all the library needs to produce JSON.

## 4. Encode a Value to JSON

With a schema in scope, every `Book` value gains a `toJsonString` extension method. It asks the schema for its JSON codec and uses it to write the value as a compact JSON string.

Let's create a book and encode it:

```scala
val zionomicon = Book("Zionomicon", 2024)

val zionomiconJson = zionomicon.toJsonString
```

The code above:

- `Book("Zionomicon", 2024)` creates a regular case class value.
- `zionomicon.toJsonString` encodes it with the JSON codec derived from `Book.schema` and returns a `String`.

To see the JSON text, we print it:

```scala
println(zionomiconJson)
// {"title":"Zionomicon","year":2024}
```

The code above prints the string that `toJsonString` returned. Notice the output: each case class field becomes a JSON field with the same name, in declaration order, and there are no extra spaces. We never wrote an encoder — the schema was enough.

## 5. Decode JSON Back Into a Value

Encoding is half of a round trip. To read JSON, we call the `fromJson` extension method on a `String` and tell it which type we expect. Decoding can fail — the text might not be valid JSON or might not match `Book` — so `fromJson` returns an `Either[SchemaError, Book]` instead of throwing.

Let's decode the JSON from the previous step and compare the result with the original value:

```scala
val decoded = zionomiconJson.fromJson[Book]
// decoded: Either[SchemaError, Book] = Right(
//   Book(title = "Zionomicon", year = 2024)
// )

decoded == Right(zionomicon)
// res2: Boolean = true
```

The code above:

- `zionomiconJson.fromJson[Book]` finds `Book.schema`, derives the decoder from it and parses the string.
- `Right(...)` in the output means decoding succeeded and carries the decoded `Book`.
- `decoded == Right(zionomicon)` checks the round trip: the value we get back is equal to the value we started with.

That's a complete JSON round trip: one schema, an encoder and a decoder, and the guarantee that both sides agree because they come from the same description.

## 6. Grow the Model: Nested Types, Optional Fields and Lists

Real data is rarely flat. Let's give each book an author, record the author's birth year only when we know it, and add a list of tags. We start a fresh scope so we can redefine `Book`:

```scala

case class Author(name: String, born: Option[Int])

case class Book(title: String, author: Author, year: Int, tags: List[String])

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}
```

The code above:

- `Author` is a second case class with an optional field, `born: Option[Int]`.
- `Book` now nests an `Author` and holds a `List[String]` of tags.
- `Book.schema` is still one line. `Schema.derived` looks for a schema for each field's type and derives one when it finds none, so `Author`, `Option[Int]` and `List[String]` need no extra code.

Let's encode a book with every field filled in:

```scala
val programmingInScala = Book("Programming in Scala", Author("Martin Odersky", Some(1958)), 2008, List("scala", "fundamentals"))
// programmingInScala: Book = Book(
//   title = "Programming in Scala",
//   author = Author(name = "Martin Odersky", born = Some(1958)),
//   year = 2008,
//   tags = List("scala", "fundamentals")
// )

println(programmingInScala.toJsonString)
// {"title":"Programming in Scala","author":{"name":"Martin Odersky","born":1958},"year":2008,"tags":["scala","fundamentals"]}
```

The code above builds a book with a nested `Author`, `Some(1958)` as the birth year and a two-element tag list, then encodes the whole value in one call. The nested `Author` becomes a nested JSON object, `Some(1958)` is written as the plain number `1958`, and the list becomes a JSON array.

Now let's encode a book where we don't know the author's birth year and have no tags yet:

```scala
val zionomicon = Book("Zionomicon", Author("John A. De Goes", None), 2024, Nil)
// zionomicon: Book = Book(
//   title = "Zionomicon",
//   author = Author(name = "John A. De Goes", born = None),
//   year = 2024,
//   tags = List()
// )

println(zionomicon.toJsonString)
// {"title":"Zionomicon","author":{"name":"John A. De Goes"},"year":2024}
```

The code above leaves the birth year as `None` and passes `Nil`, an empty tag list. Notice what is missing from the output: the JSON codec leaves out a field whose value is `None` and a list that is empty. Decoding works the same way in reverse — a missing optional field becomes `None` and a missing list becomes an empty list:

```scala
"""{"title":"Zionomicon","author":{"name":"John A. De Goes"},"year":2024}""".fromJson[Book]
// res6: Either[SchemaError, Book] = Right(
//   Book(
//     title = "Zionomicon",
//     author = Author(name = "John A. De Goes", born = None),
//     year = 2024,
//     tags = List()
//   )
// )
```

The schema grew with the case classes, and the round trip still holds without a single hand-written codec.

## 7. Read and Fix a Decode Error

Sooner or later, a decoder meets JSON that doesn't fit. Every failure comes back as a `Left` holding a `SchemaError`. Its `SchemaError#message` says what went wrong and where, using a path that starts at `.` for the top-level value.

Let's decode a book whose `year` field is missing:

```scala
val missingYear = """{"title":"Programming in Scala","author":{"name":"Martin Odersky"}}""".fromJson[Book]
// missingYear: Either[SchemaError, Book] = Left(
//   SchemaError(
//     List(
//       ExpectationMismatch(
//         source = DynamicOptic(IndexedSeq()),
//         expectation = "missing required field \"year\""
//       )
//     )
//   )
// )
```

The `Left` holds the error as data: what the decoder expected and the path where it stopped. To read it as one human-readable message, we pattern match on the `Either`:

```scala
missingYear match {
  case Left(error) => println(error.message)
  case Right(book) => println(s"Decoded: $book")
}
// missing required field "year" at: .
```

The code above:

- `"""...""".fromJson[Book]` returns a `Left` because `year` is required — its type is `Int`, not `Option[Int]`.
- `case Left(error) => println(error.message)` prints the human-readable message from the `SchemaError`.
- `at: .` in the message says the problem is in the top-level object itself.

The path becomes really useful when the problem is deep inside the JSON. Let's send the author's name as a number:

```scala
val badName = """{"title":"Programming in Scala","author":{"name":42},"year":2008}""".fromJson[Book]
// badName: Either[SchemaError, Book] = Left(
//   SchemaError(
//     List(
//       ExpectationMismatch(
//         source = DynamicOptic(IndexedSeq(Field("author"), Field("name"))),
//         expectation = "expected '\"'"
//       )
//     )
//   )
// )

badName match {
  case Left(error) => println(error.message)
  case Right(book) => println(s"Decoded: $book")
}
// expected '"' at: .author.name
```

The message says the decoder expected `"`, the opening quote of a JSON string, but found the number `42`. Notice the path after `at:` — it walks from the top-level object into `author` and then into `name`, so we know exactly which value to fix. Let's fix both problems in the input and decode again:

```scala
val fixed = """{"title":"Programming in Scala","author":{"name":"Martin Odersky"},"year":2008}""".fromJson[Book]
// fixed: Either[SchemaError, Book] = Right(
//   Book(
//     title = "Programming in Scala",
//     author = Author(name = "Martin Odersky", born = None),
//     year = 2008,
//     tags = List()
//   )
// )
```

The code above repeats the decode with `name` as a string and `year` present. The result is a `Right`, and the decoded book has `born = None` and an empty tag list, just like in the previous section.

:::note
`fromJson` never throws on bad input. Because the return type is `Either[SchemaError, Book]`, the compiler reminds us to handle the failure case every time we decode.
:::

## 8. Look at the Same Data Two More Ways

The schema gives us more than a JSON codec. To finish, let's look at two more views that come from the same `Book.schema`.

The first is `Schema#toDynamicValue`, which turns a value into a `DynamicValue` — a generic, type-independent tree of records, sequences and primitives. Library code uses it to work with any data without knowing its Scala type. Let's convert `programmingInScala`:

```scala
println(Book.schema.toDynamicValue(programmingInScala))
// {
//   title: "Programming in Scala",
//   author: {
//     name: "Martin Odersky",
//     born: {
//       value: 1958
//     } @ {tag: "Some"}
//   },
//   year: 2008,
//   tags: [
//     "scala",
//     "fundamentals"
//   ]
// }
```

The code above converts the book into a `DynamicValue` and prints that generic tree. Notice that the printed tree follows the structure of `Book`: a record with `title`, `author`, `year` and `tags` fields. One detail differs from the JSON: `born` appears as a value tagged `Some`, because the generic view keeps the `Option` case that the JSON codec flattened into a plain number. The `show` extension method prints the same text in one call, as `println(programmingInScala.show)`.

The second view is `Schema#toJsonSchema`, which describes the JSON format itself as a JSON Schema document that other services and tools can read:

```scala
println(Book.schema.toJsonSchema)
// {
//   "properties": {
//     "title": {
//       "type": "string"
//     },
//     "author": {
//       "properties": {
//         "name": {
//           "type": "string"
//         },
//         "born": {
//           "type": [
//             "null",
//             "integer"
//           ],
//           "maximum": 2147483647,
//           "minimum": -2147483648
//         }
//       },
//       "type": "object",
//       "required": [
//         "name"
//       ],
//       "title": "Author"
//     },
//     "year": {
//       "type": "integer",
//       "maximum": 2147483647,
//       "minimum": -2147483648
//     },
//     "tags": {
//       "items": {
//         "type": "string"
//       },
//       "type": [
//         "array",
//         "null"
//       ],
//       "title": "List[String]"
//     }
//   },
//   "type": "object",
//   "required": [
//     "title",
//     "author",
//     "year"
//   ],
//   "title": "Book"
// }
```

The code above:

- `Book.schema.toJsonSchema` derives a JSON Schema document from the same description that our encoder and decoder use.
- `"required"` lists `title`, `author` and `year` — exactly the fields our decoder rejected when they were missing — while the optional `born` and the `tags` list are not required.

That's the "one description, many uses" idea from the start of this tutorial, made visible: the case class, the JSON codec, the generic view and the JSON Schema document all agree because they come from one schema.

## 9. Putting It Together

This example combines all the concepts we've learned — deriving a schema for nested case classes, encoding to JSON, decoding back, checking the round trip, handling a decode error and printing the generic view:

```scala title="schema-examples/src/main/scala/yourfirstschema/CompleteExample.scala"
package yourfirstschema

/**
 * Your First Schema: Round-Trip a Type Through JSON — Complete Example
 *
 * Derives a schema for nested case classes, encodes to JSON, decodes back,
 * checks the round trip, handles a decode error and prints the generic view.
 *
 * Run with: cd schema-examples && sbt "runMain yourfirstschema.CompleteExample"
 */
object CompleteExample extends App {

  case class Author(name: String, born: Option[Int])

  case class Book(title: String, author: Author, year: Int, tags: List[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  val programmingInScala =
    Book("Programming in Scala", Author("Martin Odersky", Some(1958)), 2008, List("scala", "fundamentals"))

  // Encode to JSON and decode it back
  val programmingInScalaJson = programmingInScala.toJsonString
  println(programmingInScalaJson)

  val decoded = programmingInScalaJson.fromJson[Book]
  println(decoded)
  println(decoded == Right(programmingInScala))

  // Read a decode error
  val missingYear = """{"title":"Programming in Scala","author":{"name":"Martin Odersky"}}""".fromJson[Book]

  missingYear match {
    case Left(error) => println(error.message)
    case Right(book) => println(s"Decoded: $book")
  }

  // The generic view of the same data
  println(programmingInScala.show)
}
```

## 10. Running the Examples

All examples in this tutorial have corresponding runnable Scala files in the `schema-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

Run them in order to follow the tutorial step by step.

### Describe a Type with a Schema

This example derives a schema for the flat `Book` case class and prints it, showing the `record` that the compiler derived.

<details>
  <summary>schema-examples/src/main/scala/yourfirstschema/Concept1DescribeExample.scala</summary>

```scala title="schema-examples/src/main/scala/yourfirstschema/Concept1DescribeExample.scala" showLineNumbers
package yourfirstschema

/**
 * Your First Schema: Round-Trip a Type Through JSON — Concept 1: Describe a
 * Type with a Schema
 *
 * Derives a Schema for a flat Book case class and prints the record the
 * compiler derived.
 *
 * Run with: cd schema-examples && sbt "runMain
 * yourfirstschema.Concept1DescribeExample"
 */
object Concept1DescribeExample extends App {

  case class Book(title: String, year: Int)

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  println(Book.schema)
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain yourfirstschema.Concept1DescribeExample"
```

### Encode a Value to JSON

This example encodes a `Book` to a compact JSON string with `toJsonString`.

<details>
  <summary>schema-examples/src/main/scala/yourfirstschema/Concept2EncodeExample.scala</summary>

```scala title="schema-examples/src/main/scala/yourfirstschema/Concept2EncodeExample.scala" showLineNumbers
package yourfirstschema

/**
 * Your First Schema: Round-Trip a Type Through JSON — Concept 2: Encode a Value
 * to JSON
 *
 * Encodes a Book to a compact JSON string with toJsonString.
 *
 * Run with: cd schema-examples && sbt "runMain
 * yourfirstschema.Concept2EncodeExample"
 */
object Concept2EncodeExample extends App {

  case class Book(title: String, year: Int)

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  val zionomicon = Book("Zionomicon", 2024)

  val zionomiconJson = zionomicon.toJsonString

  println(zionomiconJson)
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain yourfirstschema.Concept2EncodeExample"
```

### Decode JSON Back Into a Value

This example decodes the JSON with `fromJson` and checks that the round trip returns the original value.

<details>
  <summary>schema-examples/src/main/scala/yourfirstschema/Concept3DecodeExample.scala</summary>

```scala title="schema-examples/src/main/scala/yourfirstschema/Concept3DecodeExample.scala" showLineNumbers
package yourfirstschema

/**
 * Your First Schema: Round-Trip a Type Through JSON — Concept 3: Decode JSON
 * Back Into a Value
 *
 * Decodes the JSON with fromJson and checks that the round trip returns the
 * original value.
 *
 * Run with: cd schema-examples && sbt "runMain
 * yourfirstschema.Concept3DecodeExample"
 */
object Concept3DecodeExample extends App {

  case class Book(title: String, year: Int)

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  val zionomicon = Book("Zionomicon", 2024)

  val zionomiconJson = zionomicon.toJsonString

  val decoded = zionomiconJson.fromJson[Book]

  println(decoded)
  println(decoded == Right(zionomicon))
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain yourfirstschema.Concept3DecodeExample"
```

### Grow the Model

This example adds a nested `Author`, an optional birth year and a list of tags, and shows how `None` and empty lists are left out of the JSON and restored on decode.

<details>
  <summary>schema-examples/src/main/scala/yourfirstschema/Concept4NestedTypesExample.scala</summary>

```scala title="schema-examples/src/main/scala/yourfirstschema/Concept4NestedTypesExample.scala" showLineNumbers
package yourfirstschema

/**
 * Your First Schema: Round-Trip a Type Through JSON — Concept 4: Grow the
 * Model: Nested Types, Optional Fields and Lists
 *
 * Adds a nested Author, an optional birth year and a list of tags, and shows
 * that None and empty lists are left out of the JSON and restored on decode.
 *
 * Run with: cd schema-examples && sbt "runMain
 * yourfirstschema.Concept4NestedTypesExample"
 */
object Concept4NestedTypesExample extends App {

  case class Author(name: String, born: Option[Int])

  case class Book(title: String, author: Author, year: Int, tags: List[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  val programmingInScala =
    Book("Programming in Scala", Author("Martin Odersky", Some(1958)), 2008, List("scala", "fundamentals"))

  println(programmingInScala.toJsonString)

  val zionomicon = Book("Zionomicon", Author("John A. De Goes", None), 2024, Nil)

  println(zionomicon.toJsonString)

  println("""{"title":"Zionomicon","author":{"name":"John A. De Goes"},"year":2024}""".fromJson[Book])
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain yourfirstschema.Concept4NestedTypesExample"
```

### Read and Fix a Decode Error

This example decodes JSON with a missing field and a wrongly typed nested field, prints each `SchemaError` message with its path, and then decodes the fixed input.

<details>
  <summary>schema-examples/src/main/scala/yourfirstschema/Concept5DecodeErrorsExample.scala</summary>

```scala title="schema-examples/src/main/scala/yourfirstschema/Concept5DecodeErrorsExample.scala" showLineNumbers
package yourfirstschema

/**
 * Your First Schema: Round-Trip a Type Through JSON — Concept 5: Read and Fix a
 * Decode Error
 *
 * Decodes JSON with a missing field and a wrongly typed nested field, prints
 * each SchemaError message with its path, then decodes the fixed input.
 *
 * Run with: cd schema-examples && sbt "runMain
 * yourfirstschema.Concept5DecodeErrorsExample"
 */
object Concept5DecodeErrorsExample extends App {

  case class Author(name: String, born: Option[Int])

  case class Book(title: String, author: Author, year: Int, tags: List[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  val missingYear = """{"title":"Programming in Scala","author":{"name":"Martin Odersky"}}""".fromJson[Book]

  println(missingYear match {
    case Left(error) => error.message
    case Right(book) => s"Decoded: $book"
  })

  val badName = """{"title":"Programming in Scala","author":{"name":42},"year":2008}""".fromJson[Book]

  println(badName match {
    case Left(error) => error.message
    case Right(book) => s"Decoded: $book"
  })

  val fixed = """{"title":"Programming in Scala","author":{"name":"Martin Odersky"},"year":2008}""".fromJson[Book]

  println(fixed)
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain yourfirstschema.Concept5DecodeErrorsExample"
```

### Look at the Same Data Two More Ways

This example prints the `DynamicValue` view of a book and the JSON Schema document derived from `Book.schema`.

<details>
  <summary>schema-examples/src/main/scala/yourfirstschema/Concept6OtherViewsExample.scala</summary>

```scala title="schema-examples/src/main/scala/yourfirstschema/Concept6OtherViewsExample.scala" showLineNumbers
package yourfirstschema

/**
 * Your First Schema: Round-Trip a Type Through JSON — Concept 6: Look at the
 * Same Data Two More Ways
 *
 * Prints the DynamicValue view of a book and the JSON Schema document derived
 * from Book.schema.
 *
 * Run with: cd schema-examples && sbt "runMain
 * yourfirstschema.Concept6OtherViewsExample"
 */
object Concept6OtherViewsExample extends App {

  case class Author(name: String, born: Option[Int])

  case class Book(title: String, author: Author, year: Int, tags: List[String])

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  val programmingInScala =
    Book("Programming in Scala", Author("Martin Odersky", Some(1958)), 2008, List("scala", "fundamentals"))

  println(Book.schema.toDynamicValue(programmingInScala))

  println(Book.schema.toJsonSchema)
}
```

</details>

Run it from the `schema-examples` directory:

```bash
sbt "runMain yourfirstschema.Concept6OtherViewsExample"
```

### The Complete Example

The complete example from "Putting It Together" runs everything from the tutorial in one program:

```bash
sbt "runMain yourfirstschema.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## 11. What You've Learned

In this tutorial, you learned:

- What a `Schema` is — one description of a type's shape — and how to derive it with `Schema.derived`
- How to encode a value with `toJsonString` and decode it with `fromJson`, and why the round trip holds
- How nested case classes, `Option` fields and `List` fields are derived automatically, and how `None` and empty lists are left out of the JSON
- How to read a `SchemaError` message and use its path to find and fix bad input
- How the same schema also gives you a `DynamicValue` view and a JSON Schema document

You now have a solid foundation in ZIO Blocks schemas. The next step is to tune the JSON format and use the same schema with other formats.

## 12. Where to Go Next

- **Want to see every extension method?** Read [Extension Syntax](../reference/schema/syntax.md), which covers `toJson`, `toJsonBytes`, `show` and more.
- **Ready to customize the JSON?** The [customization guide](./customize-json-encoding.md) walks through it task by task. The [JSON Codec](../reference/schema/built-in-codecs/json/index.md) overview shows how the codec pieces fit together, and [JSON Configuration](../reference/schema/built-in-codecs/json/json-config.md) covers reader and writer settings and field-name mapping with `NameMapper`.
- **Ready to change your types?** Follow [Evolve Your Data](./evolve-your-data.md) to convert between versions of a type with `Into`, `As` and `Migration`.
- **Want to understand errors in depth?** Read [SchemaError](../reference/schema/schema-error.md) for every error case and how errors are combined.
- **Curious about the generic view?** Read [DynamicValue](../reference/schema/dynamic-value.md).
- **Need another format?** Follow [One Type, Many Formats](./one-type-many-formats.md) to derive JSON, YAML, XML, TOON, MessagePack, Avro and CSV codecs from the same `Schema`. The [Built-in Codecs](../reference/schema/built-in-codecs/index.md) overview lists Avro, BSON, CSV, MessagePack, Thrift, TOON, XML and YAML, all derived from the same `Schema`.
- **Need to validate incoming JSON?** Follow [How to Generate JSON Schema from a Type and Validate Payloads Against It](./generate-json-schema-and-validate-payloads.md) to derive a JSON Schema from your `Schema` and check payloads against it.
- **Want the full API?** The [Schema](../reference/schema/schema.md) reference page documents every operation on `Schema`.

---

## Build Your First Database Repository on SQLite

Welcome! This tutorial is for Scala 3 developers who are new to the ZIO Blocks `sql` module. We assume you have finished [Your First Schema](./your-first-schema.md), so you know how to derive a `Schema` for a case class. You also need to know what a database table and a row are, but you don't need any prior experience with JDBC, SQL libraries or transactions to follow along.

## 1. Introduction

By the end of this tutorial, you will understand:

- How a [JdbcTransactor](../reference/sql/transactor.md) opens connections to a SQLite database and gives your code a `DbCon` to work with
- How one `Schema` gives you a [Table](../reference/sql/table.md) with its `CREATE TABLE` statement and a [Repo](../reference/sql/repo.md) with ready-made CRUD operations
- How to write your own SQL with the `sql"..."` interpolator and read the results as case classes
- Why `Frag#queryOne` and `Repo#find` return a `Maybe`, and how to handle the found and the empty case
- How `JdbcTransactor#transact` commits a group of writes together, or rolls all of them back when something fails

We learn these concepts by building a small bookshelf, one step at a time:

1. Connect to SQLite
2. Map a case class to a table
3. Insert rows with a repository
4. Find, update and delete rows
5. Write your own query
6. Group writes in a transaction

We recommend reading from top to bottom — each section builds on the previous one.

To follow along, add the `sql` module and the SQLite JDBC driver to your `build.sbt`:

```scala
// Scala 3.3 LTS
libraryDependencies += "dev.zio" %% "zio-blocks-sql" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-sql" % "0.0.56"

// The SQLite JDBC driver, for every Scala version
libraryDependencies += "org.xerial" % "sqlite-jdbc" % "3.53.4.0"
```

The `sql` module is published for Scala 3 only, and the `JdbcTransactor` we use to talk to the database runs on the JVM. Scala 3.9 and newer use the `zio-blocks-next-sql` artifact name; the code in this tutorial is the same for both.

## 2. Background: One Schema, One Repository

A repository is an object that stores and loads one kind of value — books, users, orders — so the rest of the program never writes SQL for the everyday cases. In many projects, a repository is written by hand: a `CREATE TABLE` statement, an `INSERT` with one placeholder per field, a `SELECT` that reads each column back into the right field, and the same again for `UPDATE` and `DELETE`. Every one of those statements has to change when the case class gains a field.

The ZIO Blocks `sql` module derives all of that from the `Schema` you already know from the previous tutorial. The schema says which fields a `Book` has and what their types are; from it, the module builds the table definition, the code that turns a row into a `Book` and back, and a repository with the standard operations. For anything the repository doesn't cover, you write the SQL yourself and still get the rows back as `Book` values. Finally, a transactor decides when the work talks to the database and whether a group of writes is kept or undone.

## 3. Connect to SQLite

A `JdbcTransactor` is the entry point to a database. It knows how to open a JDBC connection and which `SqlDialect` the database speaks. We create one with `JdbcTransactor.fromUrl`, giving it a JDBC URL and the dialect.

Our bookshelf lives in a SQLite file. SQLite stores a whole database in one file, so there is no server to install:

```scala

val dbFile = java.nio.file.Files.createTempFile("bookshelf", ".db")

val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
```

The code above:

- `import zio.blocks.sql.*` brings in `JdbcTransactor`, `SqlDialect` and every other type we use in this tutorial except `Schema` and `Maybe`, which we import here for later sections. We keep `import zio.blocks.sql.*` after the other `zio.blocks` imports: if your project also uses the `sql-zio` module, it brings a package named `zio` into scope, and a `zio.blocks` import after it fails to compile.
- `Files.createTempFile("bookshelf", ".db")` creates a new, empty file in the system's temporary directory, so each run of this tutorial starts with an empty database.
- `JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)` creates a transactor for that file. It doesn't connect yet; it only remembers how to.

:::caution
Every call to `JdbcTransactor#connect` or `JdbcTransactor#transact` opens a new connection. With the URL `jdbc:sqlite::memory:`, each new connection gets its own empty in-memory database, so a table created in one block is gone in the next. That's why we use a file.
:::

To do some work, we call `JdbcTransactor#connect` and pass it a block of code. The transactor opens a connection, runs the block and closes the connection again. Let's ask SQLite for its version:

```scala
transactor.connect {
  val version = sql"SELECT sqlite_version()".queryOne[String]
  println(s"Connected to SQLite ${version.getOrElse("unknown")}")
}
// Connected to SQLite 3.53.2
```

The code above:

- `transactor.connect { ... }` runs the block with an open connection. Inside the block, a `DbCon` value — the connection plus its dialect — is available as a context parameter, so we never pass it around by hand.
- `sql"SELECT sqlite_version()"` is a piece of SQL; we look at the `sql"..."` interpolator properly in section 7.
- `Frag#queryOne[String]` runs the SQL with the `DbCon` from `JdbcTransactor#connect` and reads the first row as a `String`.
- `version.getOrElse("unknown")` takes the value out of the result, which may be empty; we meet this result type again in section 6.

The printed version comes from the SQLite engine bundled with the JDBC driver. We have a working connection.

## 4. Map a Case Class to a Table

Now let's describe a book. We use the same pattern as in [Your First Schema](./your-first-schema.md): a case class with a derived `Schema` in its companion object. Since the `sql` module is Scala 3 only, we write the schema as a `given`, which is the Scala 3 spelling of the `implicit val` from that tutorial:

```scala
case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String])

object Book {
  given Schema[Book] = Schema.derived
}
```

The code above:

- `id: Long` is the number we use to find a book again later.
- `author` holds each book's first author, which keeps our sample table small.
- `isbn: Option[String]` is optional, because we don't know every book's ISBN.
- `given Schema[Book] = Schema.derived` derives the schema at compile time, exactly as before.

A `Table[A]` describes how values of type `A` are stored: the table's name and one column per field. `Table.derived` builds it from the schema:

```scala
val bookTable = Table.derived[Book]
```

Let's look at the table name and at the `CREATE TABLE` statement the table produces for SQLite:

```scala
println(bookTable.name)
// book
println(bookTable.createTable(SqlDialect.SQLite).sql(SqlDialect.SQLite))
// CREATE TABLE IF NOT EXISTS book (
//   id INTEGER NOT NULL,
//   title TEXT NOT NULL,
//   author TEXT NOT NULL,
//   year INTEGER NOT NULL,
//   isbn TEXT
// )
```

The code above:

- `bookTable.name` is the table name, derived from the type name `Book` in snake case.
- `bookTable.createTable(SqlDialect.SQLite)` builds the `CREATE TABLE` statement as a `Frag`, a piece of SQL that we can run.
- `Frag#sql(SqlDialect.SQLite)` renders that `Frag` as text, so we can read it before we run it.

Notice how each field became a column. `Long` and `Int` map to SQLite's `INTEGER` and `String` maps to `TEXT`. The required fields are `NOT NULL`, and `isbn`, the `Option` field, is the only column that may hold `NULL`. The `IF NOT EXISTS` clause makes the statement safe to run again: when the table is already there, SQLite leaves it alone.

Reading the statement doesn't create the table. Let's run it, then ask SQLite which tables the database now has:

```scala
transactor.connect {
  bookTable.createTable(SqlDialect.SQLite).update
  val tables = sql"SELECT name FROM sqlite_master WHERE type = 'table'".query[String]
  println(tables)
}
// List(book)
```

The code above runs the `CREATE TABLE` statement with `Frag#update`, which executes a `Frag` that changes the database, such as a `CREATE TABLE`, `INSERT` or `DELETE`, and returns the number of affected rows. The query then reads `sqlite_master`, SQLite's own catalog of everything in the database, and `Frag#query[String]` returns all of its matching rows as a `List[String]`.

The list holds our `book` table. The database is ready for data.

## 5. Insert Rows with a Repository

A `Repo[E, ID]` is a repository for entities of type `E` whose ID has type `ID`. `Repo.derived` builds one from the schema, so it already knows the table, the columns and the SQL for each operation:

```scala
val books = Repo.derived[Book, Long]
```

The code above asks for a repository of `Book` values with `Long` IDs. `Repo.derived` finds the ID column on its own: our `Book` has exactly one `Long` field, and it is named `id`.

Let's put some books on the shelf. We create four books, using `None` for the ISBNs we don't know:

```scala
val zionomicon = Book(1L, "Zionomicon", "John A. De Goes", 2024, None)
val programmingInScala = Book(2L, "Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"))
val fpInScala = Book(3L, "Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
val handsOnScala = Book(4L, "Hands-on Scala Programming", "Li Haoyi", 2020, None)
```

Now we insert them, count the rows and list everything on the shelf:

```scala
transactor.connect {
  val inserted = books.insert(zionomicon)
  println(s"Rows inserted: $inserted")

  books.insert(programmingInScala)
  books.insert(fpInScala)
  books.insert(handsOnScala)

  println(s"Books on the shelf: ${books.count}")
  books.all.foreach(println)
}
// Rows inserted: 1
// Books on the shelf: 4
// Book(1,Zionomicon,John A. De Goes,2024,None)
// Book(2,Programming in Scala,Martin Odersky,2008,Some(9780981531601))
// Book(3,Functional Programming in Scala,Paul Chiusano,2014,Some(9781617290657))
// Book(4,Hands-on Scala Programming,Li Haoyi,2020,None)
```

The code above:

- `books.insert(zionomicon)` writes one book and returns the number of rows it inserted.
- `books.count` returns the number of rows in the `book` table as a `Long`.
- `books.all` reads every row back as a `List[Book]`, and `foreach(println)` prints one book per line.

Notice that we never wrote an `INSERT` or a `SELECT`. The repository built them from the schema, and the books came back from the database as ordinary `Book` values, with `None` where the `isbn` column is `NULL`.

## 6. Find, Update and Delete Rows

To read one book, we call `Repo#find` with its ID. A book with that ID may not exist, so `Repo#find` returns a `Maybe[Book]`: either a book or nothing. `Maybe` is the ZIO Blocks version of `Option`, and its `Maybe#fold` method handles both cases — the first argument is used when the value is absent, and the function is used when it is present:

```scala
transactor.connect {
  println(books.find(2L).fold("No book with id 2")(book => s"Found: $book"))
  println(books.find(42L).fold("No book with id 42")(book => s"Found: $book"))
}
// Found: Book(2,Programming in Scala,Martin Odersky,2008,Some(9780981531601))
// No book with id 42
```

The code above finds _Programming in Scala_ with `books.find(2L)`, so `Maybe#fold` calls the function and prints the book. With `books.find(42L)` it finds nothing, so `Maybe#fold` returns its first argument, the "No book" message.

Next, let's change a book. `Repo#update` takes a whole `Book` and overwrites the row with the same ID. We use it to record both authors of _Functional Programming in Scala_:

```scala
transactor.connect {
  val updated = books.update(fpInScala.copy(author = "Paul Chiusano and Rúnar Bjarnason"))
  println(s"Rows updated: $updated")
  println(books.find(3L).fold("No book with id 3")(book => s"Authors: ${book.author}"))
}
// Rows updated: 1
// Authors: Paul Chiusano and Rúnar Bjarnason
```

The code above:

- `fpInScala.copy(author = ...)` makes a new `Book` with the same ID and a new `author`; the case class itself never changes.
- `books.update(...)` writes it to the row with ID `3` and returns the number of rows it changed.
- `books.find(3L)` reads the row again, so we see the new value as the database stores it.

Finally, `Repo#delete` removes the row with a given ID. Let's take _Hands-on Scala Programming_ off the shelf:

```scala
transactor.connect {
  val deleted = books.delete(4L)
  println(s"Rows deleted: $deleted")
  println(s"Books on the shelf: ${books.count}")
}
// Rows deleted: 1
// Books on the shelf: 3
```

The code above:

- `books.delete(4L)` removes the row with ID `4` and returns the number of deleted rows.
- `books.count` returns the number of rows that remain.
- The two `println` calls show the deleted-row count and the new total.

That's the full set of CRUD operations — create, read, update and delete — without a single line of SQL.

## 7. Write Your Own Query

A repository covers the everyday operations, but sooner or later you need a query of your own, such as "every book published before 2010". For that, we write SQL with the `sql"..."` interpolator. Each `$value` inside it becomes a query parameter: the value is sent to the database separately from the SQL text, so it can never change the meaning of the query.

Let's find the books published before a given year:

```scala
val publishedBefore = 2010
// publishedBefore: Int = 2010

transactor.connect {
  val classics = sql"SELECT id, title, author, year, isbn FROM book WHERE year < $publishedBefore".query[Book]
  classics.foreach(println)
}
// Book(2,Programming in Scala,Martin Odersky,2008,Some(9780981531601))
```

The code above builds a `Frag` with `sql"... WHERE year < $publishedBefore"`, whose SQL text is `... WHERE year < ?` and whose parameter is the `Int` value `2010`. Then `Frag#query[Book]` runs it and reads every row into a `Book`, using the same schema as the repository, so the result is a `List[Book]`.

When a query returns at most one row, we use `Frag#queryOne` instead. It returns a `Maybe`, just like `Repo#find`. Let's wrap a title lookup in a method of its own:

```scala
def findByTitle(title: String)(using DbCon): Maybe[Book] =
  sql"SELECT id, title, author, year, isbn FROM book WHERE title = $title".queryOne[Book]
```

The code above:

- `(using DbCon)` says that the method needs a connection context from its caller. We can only call `findByTitle` inside `JdbcTransactor#connect` or `JdbcTransactor#transact`, and the compiler checks that for us.
- `$title` becomes a `String` parameter, just as `$publishedBefore` became an `Int` parameter.
- `import zio.blocks.maybe.Maybe` lets us write the result type in the method signature; we didn't need it before because we never wrote the type down.

Now let's look up one title that is on the shelf and one that isn't:

```scala
transactor.connect {
  println(findByTitle("Zionomicon").fold("Not on the shelf")(book => s"Found: $book"))
  println(findByTitle("Scala Puzzlers").fold("Not on the shelf")(book => s"Found: $book"))
}
// Found: Book(1,Zionomicon,John A. De Goes,2024,None)
// Not on the shelf
```

The code above:

- `findByTitle("Zionomicon")` finds the book, so `Maybe#fold` calls the function and prints it.
- `findByTitle("Scala Puzzlers")` finds nothing, so `Maybe#fold` returns its first argument, "Not on the shelf".
- Both calls run inside one `JdbcTransactor#connect` block, which supplies the `DbCon` that `findByTitle` requires.

Notice that a query of our own and `Repo#find` give us the same kind of result, so the code that handles them looks the same.

## 8. Group Writes in a Transaction

So far, every statement inside `JdbcTransactor#connect` was saved as soon as it ran. Sometimes several writes belong together: either all of them should happen, or none of them. That's what a transaction is for, and `JdbcTransactor#transact` runs a block inside one. When the block finishes normally, `JdbcTransactor#transact` commits the transaction and every write is saved. When the block throws an exception, `JdbcTransactor#transact` rolls the transaction back, undoing every write in the block, and then rethrows the exception to the caller.

Let's make our shelf hold at most three books, and try to put _Hands-on Scala Programming_ back on it:

```scala

Try {
  transactor.transact {
    books.insert(handsOnScala)
    println(s"Inside the transaction: ${books.count} books")
    if (books.count > 3) throw new IllegalStateException("The shelf holds at most 3 books")
  }
} match {
  case Failure(error) => println(s"Transaction failed: ${error.getMessage}")
  case Success(_)     => println("Transaction committed")
}
// Inside the transaction: 4 books
// Transaction failed: The shelf holds at most 3 books

transactor.connect {
  println(s"After the rollback: ${books.count} books")
}
// After the rollback: 3 books
```

The code above:

- `transactor.transact { ... }` opens a connection, starts a transaction and runs the block. The repository methods work here too, because the transaction context is also a `DbCon`.
- `books.insert(handsOnScala)` succeeds, and inside the transaction the count is four.
- `throw new IllegalStateException(...)` stops the block. `JdbcTransactor#transact` rolls the transaction back, so the insert is undone, and then rethrows the exception.
- `Try { ... }` catches the rethrown exception as a `Failure`, and the `match` prints its message.
- The `JdbcTransactor#connect` block afterwards counts the rows in a new connection and finds three books again.

Notice the order of the output: the transaction saw four books, but the shelf ended with three. The insert really happened inside the transaction, and the rollback undid it.

Now let's write a block that succeeds. We swap one book for another — take _Programming in Scala_ off the shelf and put _Hands-on Scala Programming_ on it — so the shelf never holds more than three books:

```scala
transactor.transact {
  books.delete(2L)
  books.insert(handsOnScala)
  println("Swapped Programming in Scala for Hands-on Scala Programming")
}
// Swapped Programming in Scala for Hands-on Scala Programming

transactor.connect {
  books.all.foreach(println)
}
// Book(1,Zionomicon,John A. De Goes,2024,None)
// Book(3,Functional Programming in Scala,Paul Chiusano and Rúnar Bjarnason,2014,Some(9781617290657))
// Book(4,Hands-on Scala Programming,Li Haoyi,2020,None)
```

The code above:

- `books.delete(2L)` removes _Programming in Scala_ and `books.insert(handsOnScala)` puts _Hands-on Scala Programming_ on the shelf.
- The block finishes without an exception, so `JdbcTransactor#transact` commits it and both writes are saved together.
- The `JdbcTransactor#connect` block afterwards lists the shelf, and the swap shows in the output.

If the insert had failed, the delete would have been rolled back too, and _Programming in Scala_ would still be on the shelf.

:::note
The one-argument `JdbcTransactor#transact` keeps the JDBC driver's default isolation level instead of choosing one for you. SQLite supports only the `SERIALIZABLE` level, so on SQLite every transaction behaves as serializable. To ask for a specific level on other databases, see [SQL Transactions](./sql-transactions.md).
:::

## 9. Putting It Together

This example combines all the concepts we've learned — connecting to a SQLite file, creating a table from a schema, CRUD with a derived repository, a hand-written query and a transaction that rolls back:

```scala title="sql-examples/src/main/scala/yourfirstsqliterepository/CompleteExample.scala"
package yourfirstsqliterepository

/**
 * Build Your First Database Repository on SQLite — Complete Example
 *
 * Connects to SQLite, derives a Table and a Repo, runs CRUD operations, a hand-written query and a transaction that rolls back.
 *
 * Run with: cd sql-examples && sbt "runMain
 * yourfirstsqliterepository.CompleteExample"
 */
object CompleteExample extends App {

  case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String])

  object Book {
    given Schema[Book] = Schema.derived
  }

  val dbFile = java.nio.file.Files.createTempFile("bookshelf", ".db")
  dbFile.toFile.deleteOnExit()

  val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

  transactor.connect {
    val version = sql"SELECT sqlite_version()".queryOne[String]
    println(s"Connected to SQLite ${version.getOrElse("unknown")}")
  }

  val bookTable = Table.derived[Book]

  println(bookTable.name)
  println(bookTable.createTable(SqlDialect.SQLite).sql(SqlDialect.SQLite))

  transactor.connect {
    bookTable.createTable(SqlDialect.SQLite).update
    val tables = sql"SELECT name FROM sqlite_master WHERE type = 'table'".query[String]
    println(tables)
  }

  val books = Repo.derived[Book, Long]

  val zionomicon         = Book(1L, "Zionomicon", "John A. De Goes", 2024, None)
  val programmingInScala = Book(2L, "Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"))
  val fpInScala          = Book(3L, "Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
  val handsOnScala       = Book(4L, "Hands-on Scala Programming", "Li Haoyi", 2020, None)

  transactor.connect {
    val inserted = books.insert(zionomicon)
    println(s"Rows inserted: $inserted")

    books.insert(programmingInScala)
    books.insert(fpInScala)
    books.insert(handsOnScala)

    println(s"Books on the shelf: ${books.count}")
    books.all.foreach(println)
  }

  transactor.connect {
    println(books.find(2L).fold("No book with id 2")(book => s"Found: $book"))
    println(books.find(42L).fold("No book with id 42")(book => s"Found: $book"))
  }

  transactor.connect {
    val updated = books.update(fpInScala.copy(author = "Paul Chiusano and Rúnar Bjarnason"))
    println(s"Rows updated: $updated")
    println(books.find(3L).fold("No book with id 3")(book => s"Authors: ${book.author}"))
  }

  transactor.connect {
    val deleted = books.delete(4L)
    println(s"Rows deleted: $deleted")
    println(s"Books on the shelf: ${books.count}")
  }

  val publishedBefore = 2010

  transactor.connect {
    val classics = sql"SELECT id, title, author, year, isbn FROM book WHERE year < $publishedBefore".query[Book]
    classics.foreach(println)
  }

  def findByTitle(title: String)(using DbCon): Maybe[Book] =
    sql"SELECT id, title, author, year, isbn FROM book WHERE title = $title".queryOne[Book]

  transactor.connect {
    println(findByTitle("Zionomicon").fold("Not on the shelf")(book => s"Found: $book"))
    println(findByTitle("Scala Puzzlers").fold("Not on the shelf")(book => s"Found: $book"))
  }

  Try {
    transactor.transact {
      books.insert(handsOnScala)
      println(s"Inside the transaction: ${books.count} books")
      if (books.count > 3) throw new IllegalStateException("The shelf holds at most 3 books")
    }
  } match {
    case Failure(error) => println(s"Transaction failed: ${error.getMessage}")
    case Success(_)     => println("Transaction committed")
  }

  transactor.connect {
    println(s"After the rollback: ${books.count} books")
  }

  transactor.transact {
    books.delete(2L)
    books.insert(handsOnScala)
    println("Swapped Programming in Scala for Hands-on Scala Programming")
  }

  transactor.connect {
    books.all.foreach(println)
  }
}
```

## 10. Running the Examples

All examples in this tutorial have corresponding runnable Scala files in the `sql-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/sql-examples
```

Each example creates its own temporary database file, so you can run them in any order and as often as you like. We recommend running them in order to follow the tutorial step by step.

### Connect to SQLite

This example creates a transactor for a temporary SQLite file and prints the SQLite version.

<details>
  <summary>sql-examples/src/main/scala/yourfirstsqliterepository/Concept1ConnectExample.scala</summary>

```scala title="sql-examples/src/main/scala/yourfirstsqliterepository/Concept1ConnectExample.scala" showLineNumbers
package yourfirstsqliterepository

/**
 * Build Your First Database Repository on SQLite — Concept 1: Connect to SQLite
 *
 * Creates a transactor for a temporary SQLite file and prints the SQLite version.
 *
 * Run with: cd sql-examples && sbt "runMain
 * yourfirstsqliterepository.Concept1ConnectExample"
 */
object Concept1ConnectExample extends App {

  val dbFile = java.nio.file.Files.createTempFile("bookshelf", ".db")
  dbFile.toFile.deleteOnExit()

  val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

  transactor.connect {
    val version = sql"SELECT sqlite_version()".queryOne[String]
    println(s"Connected to SQLite ${version.getOrElse("unknown")}")
  }
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "runMain yourfirstsqliterepository.Concept1ConnectExample"
```

### Map a Case Class to a Table

This example derives a `Table` for `Book`, prints its `CREATE TABLE` statement and creates the table.

<details>
  <summary>sql-examples/src/main/scala/yourfirstsqliterepository/Concept2TableExample.scala</summary>

```scala title="sql-examples/src/main/scala/yourfirstsqliterepository/Concept2TableExample.scala" showLineNumbers
package yourfirstsqliterepository

/**
 * Build Your First Database Repository on SQLite — Concept 2: Map a Case Class to a Table
 *
 * Derives a Table for Book, prints its CREATE TABLE statement and creates the table.
 *
 * Run with: cd sql-examples && sbt "runMain
 * yourfirstsqliterepository.Concept2TableExample"
 */
object Concept2TableExample extends App {

  case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String])

  object Book {
    given Schema[Book] = Schema.derived
  }

  val dbFile = java.nio.file.Files.createTempFile("bookshelf", ".db")
  dbFile.toFile.deleteOnExit()

  val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

  val bookTable = Table.derived[Book]

  println(bookTable.name)
  println(bookTable.createTable(SqlDialect.SQLite).sql(SqlDialect.SQLite))

  transactor.connect {
    bookTable.createTable(SqlDialect.SQLite).update
    val tables = sql"SELECT name FROM sqlite_master WHERE type = 'table'".query[String]
    println(tables)
  }
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "runMain yourfirstsqliterepository.Concept2TableExample"
```

### Insert Rows with a Repository

This example derives a `Repo` for `Book`, inserts four books and lists them.

<details>
  <summary>sql-examples/src/main/scala/yourfirstsqliterepository/Concept3InsertExample.scala</summary>

```scala title="sql-examples/src/main/scala/yourfirstsqliterepository/Concept3InsertExample.scala" showLineNumbers
package yourfirstsqliterepository

/**
 * Build Your First Database Repository on SQLite — Concept 3: Insert Rows with a Repository
 *
 * Derives a Repo for Book, inserts four books and lists them.
 *
 * Run with: cd sql-examples && sbt "runMain
 * yourfirstsqliterepository.Concept3InsertExample"
 */
object Concept3InsertExample extends App {

  case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String])

  object Book {
    given Schema[Book] = Schema.derived
  }

  val dbFile = java.nio.file.Files.createTempFile("bookshelf", ".db")
  dbFile.toFile.deleteOnExit()

  val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

  val bookTable = Table.derived[Book]

  // Setup: create the table
  transactor.connect {
    bookTable.createTable(SqlDialect.SQLite).update
  }

  val books = Repo.derived[Book, Long]

  val zionomicon         = Book(1L, "Zionomicon", "John A. De Goes", 2024, None)
  val programmingInScala = Book(2L, "Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"))
  val fpInScala          = Book(3L, "Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
  val handsOnScala       = Book(4L, "Hands-on Scala Programming", "Li Haoyi", 2020, None)

  transactor.connect {
    val inserted = books.insert(zionomicon)
    println(s"Rows inserted: $inserted")

    books.insert(programmingInScala)
    books.insert(fpInScala)
    books.insert(handsOnScala)

    println(s"Books on the shelf: ${books.count}")
    books.all.foreach(println)
  }
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "runMain yourfirstsqliterepository.Concept3InsertExample"
```

### Find, Update and Delete Rows

This example finds a book that exists and one that doesn't, updates a book's authors and deletes a book.

<details>
  <summary>sql-examples/src/main/scala/yourfirstsqliterepository/Concept4FindUpdateDeleteExample.scala</summary>

```scala title="sql-examples/src/main/scala/yourfirstsqliterepository/Concept4FindUpdateDeleteExample.scala" showLineNumbers
package yourfirstsqliterepository

/**
 * Build Your First Database Repository on SQLite — Concept 4: Find, Update and Delete Rows
 *
 * Finds a book that exists and one that does not, updates a book's authors and deletes a book.
 *
 * Run with: cd sql-examples && sbt "runMain
 * yourfirstsqliterepository.Concept4FindUpdateDeleteExample"
 */
object Concept4FindUpdateDeleteExample extends App {

  case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String])

  object Book {
    given Schema[Book] = Schema.derived
  }

  val dbFile = java.nio.file.Files.createTempFile("bookshelf", ".db")
  dbFile.toFile.deleteOnExit()

  val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

  val bookTable = Table.derived[Book]

  // Setup: create the table
  transactor.connect {
    bookTable.createTable(SqlDialect.SQLite).update
  }

  val books = Repo.derived[Book, Long]

  val zionomicon         = Book(1L, "Zionomicon", "John A. De Goes", 2024, None)
  val programmingInScala = Book(2L, "Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"))
  val fpInScala          = Book(3L, "Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
  val handsOnScala       = Book(4L, "Hands-on Scala Programming", "Li Haoyi", 2020, None)

  // Setup: insert the four books
  transactor.connect {
    books.insert(zionomicon)
    books.insert(programmingInScala)
    books.insert(fpInScala)
    books.insert(handsOnScala)
  }

  transactor.connect {
    println(books.find(2L).fold("No book with id 2")(book => s"Found: $book"))
    println(books.find(42L).fold("No book with id 42")(book => s"Found: $book"))
  }

  transactor.connect {
    val updated = books.update(fpInScala.copy(author = "Paul Chiusano and Rúnar Bjarnason"))
    println(s"Rows updated: $updated")
    println(books.find(3L).fold("No book with id 3")(book => s"Authors: ${book.author}"))
  }

  transactor.connect {
    val deleted = books.delete(4L)
    println(s"Rows deleted: $deleted")
    println(s"Books on the shelf: ${books.count}")
  }
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "runMain yourfirstsqliterepository.Concept4FindUpdateDeleteExample"
```

### Write Your Own Query

This example runs a hand-written query with an `Int` parameter and a `Frag#queryOne` lookup by title, for a title that is on the shelf and one that isn't.

<details>
  <summary>sql-examples/src/main/scala/yourfirstsqliterepository/Concept5QueryExample.scala</summary>

```scala title="sql-examples/src/main/scala/yourfirstsqliterepository/Concept5QueryExample.scala" showLineNumbers
package yourfirstsqliterepository

/**
 * Build Your First Database Repository on SQLite — Concept 5: Write Your Own Query
 *
 * Runs a hand-written query with an Int parameter and a queryOne lookup by title.
 *
 * Run with: cd sql-examples && sbt "runMain
 * yourfirstsqliterepository.Concept5QueryExample"
 */
object Concept5QueryExample extends App {

  case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String])

  object Book {
    given Schema[Book] = Schema.derived
  }

  val dbFile = java.nio.file.Files.createTempFile("bookshelf", ".db")
  dbFile.toFile.deleteOnExit()

  val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

  val bookTable = Table.derived[Book]

  // Setup: create the table
  transactor.connect {
    bookTable.createTable(SqlDialect.SQLite).update
  }

  val books = Repo.derived[Book, Long]

  val zionomicon         = Book(1L, "Zionomicon", "John A. De Goes", 2024, None)
  val programmingInScala = Book(2L, "Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"))
  val fpInScala          = Book(3L, "Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
  val handsOnScala       = Book(4L, "Hands-on Scala Programming", "Li Haoyi", 2020, None)

  // Setup: insert the four books
  transactor.connect {
    books.insert(zionomicon)
    books.insert(programmingInScala)
    books.insert(fpInScala)
    books.insert(handsOnScala)
  }

  // Setup: apply the update and delete from the previous section
  transactor.connect {
    books.update(fpInScala.copy(author = "Paul Chiusano and Rúnar Bjarnason"))
    books.delete(4L)
  }

  val publishedBefore = 2010

  transactor.connect {
    val classics = sql"SELECT id, title, author, year, isbn FROM book WHERE year < $publishedBefore".query[Book]
    classics.foreach(println)
  }

  def findByTitle(title: String)(using DbCon): Maybe[Book] =
    sql"SELECT id, title, author, year, isbn FROM book WHERE title = $title".queryOne[Book]

  transactor.connect {
    println(findByTitle("Zionomicon").fold("Not on the shelf")(book => s"Found: $book"))
    println(findByTitle("Scala Puzzlers").fold("Not on the shelf")(book => s"Found: $book"))
  }
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "runMain yourfirstsqliterepository.Concept5QueryExample"
```

### Group Writes in a Transaction

This example shows a transaction that rolls back after an exception and one that commits two writes together.

<details>
  <summary>sql-examples/src/main/scala/yourfirstsqliterepository/Concept6TransactionExample.scala</summary>

```scala title="sql-examples/src/main/scala/yourfirstsqliterepository/Concept6TransactionExample.scala" showLineNumbers
package yourfirstsqliterepository

/**
 * Build Your First Database Repository on SQLite — Concept 6: Group Writes in a Transaction
 *
 * Shows a transaction that rolls back after an exception and one that commits two writes together.
 *
 * Run with: cd sql-examples && sbt "runMain
 * yourfirstsqliterepository.Concept6TransactionExample"
 */
object Concept6TransactionExample extends App {

  case class Book(id: Long, title: String, author: String, year: Int, isbn: Option[String])

  object Book {
    given Schema[Book] = Schema.derived
  }

  val dbFile = java.nio.file.Files.createTempFile("bookshelf", ".db")
  dbFile.toFile.deleteOnExit()

  val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

  val bookTable = Table.derived[Book]

  // Setup: create the table
  transactor.connect {
    bookTable.createTable(SqlDialect.SQLite).update
  }

  val books = Repo.derived[Book, Long]

  val zionomicon         = Book(1L, "Zionomicon", "John A. De Goes", 2024, None)
  val programmingInScala = Book(2L, "Programming in Scala", "Martin Odersky", 2008, Some("9780981531601"))
  val fpInScala          = Book(3L, "Functional Programming in Scala", "Paul Chiusano", 2014, Some("9781617290657"))
  val handsOnScala       = Book(4L, "Hands-on Scala Programming", "Li Haoyi", 2020, None)

  // Setup: insert the four books
  transactor.connect {
    books.insert(zionomicon)
    books.insert(programmingInScala)
    books.insert(fpInScala)
    books.insert(handsOnScala)
  }

  // Setup: apply the update and delete from the previous section
  transactor.connect {
    books.update(fpInScala.copy(author = "Paul Chiusano and Rúnar Bjarnason"))
    books.delete(4L)
  }

  Try {
    transactor.transact {
      books.insert(handsOnScala)
      println(s"Inside the transaction: ${books.count} books")
      if (books.count > 3) throw new IllegalStateException("The shelf holds at most 3 books")
    }
  } match {
    case Failure(error) => println(s"Transaction failed: ${error.getMessage}")
    case Success(_)     => println("Transaction committed")
  }

  transactor.connect {
    println(s"After the rollback: ${books.count} books")
  }

  transactor.transact {
    books.delete(2L)
    books.insert(handsOnScala)
    println("Swapped Programming in Scala for Hands-on Scala Programming")
  }

  transactor.connect {
    books.all.foreach(println)
  }
}
```

</details>

Run it from the `sql-examples` directory:

```bash
sbt "runMain yourfirstsqliterepository.Concept6TransactionExample"
```

### The Complete Example

The complete example from "Putting It Together" runs everything from the tutorial in one program:

```bash
sbt "runMain yourfirstsqliterepository.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## 11. What You've Learned

In this tutorial, you learned:

- How a `JdbcTransactor` opens a connection for each `JdbcTransactor#connect` or `JdbcTransactor#transact` block, and why a SQLite file keeps your tables between blocks
- How one `Schema` gives you a `Table`, with its `CREATE TABLE` statement, and a `Repo` with `Repo#insert`, `Repo#all`, `Repo#find`, `Repo#update`, `Repo#delete` and `Repo#count`
- How to write your own SQL with `sql"..."`, where every `$value` becomes a query parameter, and read the rows as case classes with `Frag#query` and `Frag#queryOne`
- How `Repo#find` and `Frag#queryOne` return a `Maybe`, and how `Maybe#fold` handles the found and the empty case
- How `JdbcTransactor#transact` commits a group of writes together, or rolls all of them back and rethrows when the block throws

You now have a solid foundation in the ZIO Blocks `sql` module. The next step is to see how transactions and checked queries work in a real application.

## 12. Where to Go Next

- **Ready to control transactions?** The how-to guide [SQL Transactions](./sql-transactions.md) covers isolation levels, read-only transactions and nested transactions with savepoints.
- **Want the compiler to check your SQL?** [sql(table) Checked Interpolation](./sql-checked-interpolation.md) shows how to let the compiler check table and column names in `sql"..."` queries.
- **Want every repository operation?** Read the [Repo](../reference/sql/repo.md) reference, which also covers batch inserts, `insertReturning` and deleting many rows at once.
- **Curious about tables and DDL?** Read [Table](../reference/sql/table.md) for custom table and column names, and [Ddl](../reference/sql/ddl.md) for the statements it generates.
- **Need to stream large results?** The [Frag](../reference/sql/frag.md) reference covers `queryStream` and `queryChunked`.
- **Prefer building queries from Scala expressions?** [Query DSL with Reified Optics — Part 2: SQL Generation](./query-dsl-sql.md) turns typed query expressions into SQL.
- **Moving to PostgreSQL?** [How to Connect to PostgreSQL with a Connection Pool](./connect-to-postgresql-with-a-connection-pool.md) sets up a HikariCP pool and builds the same `Repo` on PostgreSQL.
- **Using ZIO?** [TransactorZIO](../reference/sql/transactor-zio.md) runs the same work as ZIO effects.
- **Testing your database code?** [How to Test Database Code Against SQLite](./test-database-code-against-sqlite.md) gives every test its own SQLite database and rolls back writes.
- **Want the big picture?** The [SQL Module](../reference/sql/index.md) overview links to every `sql` reference page.
- **Ready to put it all together?** [Build a Small JSON Backend from Blocks](./small-json-backend-from-blocks.md) is the capstone tutorial that combines schema, config, scope, SQL and telemetry into one small JSON backend.

---

## Your First Streaming Pipeline: Turn a Sales Log into a Report

Welcome! This tutorial is for Scala developers who are new to ZIO Blocks Streams. If you can write a case class, a function and an `Either` pattern match, you have everything you need. You don't need any prior experience with streaming libraries to follow along.

## 1. Introduction

By the end of this tutorial, you will understand:

- What a `Stream` is, and why nothing happens until you run it
- How to read a file as a stream so that the file is always closed
- How to transform a stream with `Stream#filter`, `Stream#map` and `Stream#grouped`
- How a `Sink` consumes a stream and produces one result
- How a stream fails with a typed error, and how `Stream#catchAll` recovers from it
- How the same stream runs with both a blocking and an asynchronous terminal

We learn these concepts by turning a small bookstore's sales log into a report of copies sold per title, one step at a time:

1. Create and run a stream
2. Read the sales log from a file
3. Keep only the sales with `Stream#filter`
4. Turn each line into a sale with `Stream#map`
5. Batch sales with `Stream#grouped`
6. Add up the sales with a `Sink`
7. Fail on a malformed line
8. Recover with `Stream#catchAll`
9. Run the same pipeline asynchronously

We recommend reading from top to bottom — each section builds on the previous one.

To follow along, add the streams module to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-streams" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-streams" % "0.0.56"
scalacOptions += "-experimental"
```

Scala 3.9 and newer use the `zio-blocks-next-streams` artifact name; the code in this tutorial is the same for every Scala version. On the JVM, the streams module needs JDK 21 or newer. For scala-cli, Scala.js, or cross-building, see [How to Add ZIO Blocks to Your Build](../getting-started/installation.md).

## 2. Background: A Stream Is a Recipe

Picture the sales log of a bookstore that sells Scala books. Every time the till sells a book, it appends one line to a text file: when the sale happened, which title, and how many copies. At the end of the day, the owner wants to know how many copies of each title were sold.

We could read the whole file into a `List` and work on that. That works for a small file, but a busy store's log can be larger than the memory we have, and we must still remember to close the file when we are done — even when something goes wrong halfway through. A **stream** solves both problems. A [`Stream`](../reference/streams/core/stream.md) is a recipe for producing elements one at a time: open the file, read a line, turn it into a sale, hand it on, read the next line. Running the recipe pulls elements through every step, one by one, so the whole file is never in memory at once. And because the stream itself opens the file, it also knows when to close it. Nothing happens while we write the recipe; the work starts only when we run it with a **terminal** operation such as `Stream#runCollect` or `Stream#run`.

## 3. Create and Run a Stream

A `Stream[E, A]` describes a sequence of elements of type `A` that may fail with an error of type `E`. The simplest way to create one is to list its elements with `Stream(...)`.

Let's create a stream of book titles:

```scala

val titles = Stream("Zionomicon", "Programming in Scala", "Functional Programming in Scala")
```

The code above:

- `import zio.blocks.streams._` brings `Stream`, `Sink` and their operations into scope.
- `Stream(...)` creates a `Stream[Nothing, String]`. The error type `Nothing` says this stream can never fail.
- `val titles` stores the recipe; it does not run it.

Creating the stream does not produce anything yet; `titles` is only the recipe. To run it, we call a terminal operation. `Stream#runCollect` runs the stream and collects every element:

```scala
println(titles.runCollect)
// Right(Chunk(Zionomicon,Programming in Scala,Functional Programming in Scala))
```

The code above runs the stream and prints its result. Notice the shape of the output: `Right(...)` appears because every terminal returns an `Either`, where a `Right` means the stream finished without an error and a failure comes back as a `Left`, which we see in section 9. `Chunk(...)` appears because `Stream#runCollect` gathers the elements into a `Chunk`, the immutable sequence type that ZIO Blocks uses throughout, and its `toString` separates the elements with a comma and no space.

## 4. Read the Sales Log from a File

Now let's read real data. First, we write a small sales log to a temporary file, so the example works on any machine:

```scala

val salesLog: Path = Files.createTempFile("sales", ".csv")

Files.writeString(
  salesLog,
  """timestamp,title,quantity
    |2026-10-19T10:15,Zionomicon,2
    |2026-10-19T10:42,Programming in Scala,1
    |2026-10-19T11:05,Functional Programming in Scala,3
    |2026-10-19T12:10,Zionomicon,1
    |""".stripMargin
)
```

The file starts with a header line, followed by one line per sale.

To read it, we need a stream that opens the file, emits its lines, and closes the file afterwards. `Stream.fromAcquireRelease` does exactly that. It takes three parts: `acquire` opens the resource, `release` closes it, and a function receives the open resource and returns the stream that uses it. We print a message in `acquire` and `release` so we can see when they run:

```scala
val salesLines: Stream[Nothing, String] =
  Stream.fromAcquireRelease(
    acquire = {
      println("Opened the sales log")
      Files.newBufferedReader(salesLog)
    },
    release = (reader: BufferedReader) => {
      reader.close()
      println("Closed the sales log")
    }
  )(reader => Stream.fromIterator(reader.lines().iterator().asScala))
```

The code above:

- `acquire` opens the file with `Files.newBufferedReader`. It runs when the stream starts, not when we define `salesLines`.
- `release` closes the reader. The stream calls it exactly once, after the last line, after a failure, or when we stop reading early.
- `reader => Stream.fromIterator(...)` turns the reader's lines into a `Stream[Nothing, String]`. `reader.lines()` is the JDK's way to read a file line by line, and `asScala` converts its Java iterator into a Scala `Iterator`.

Let's run it. This time we use `Stream#runForeach`, a terminal that calls a function for each element as it arrives:

```scala
salesLines.runForeach(line => println(line))
// Opened the sales log
// timestamp,title,quantity
// 2026-10-19T10:15,Zionomicon,2
// 2026-10-19T10:42,Programming in Scala,1
// 2026-10-19T11:05,Functional Programming in Scala,3
// 2026-10-19T12:10,Zionomicon,1
// Closed the sales log
// res2: Either[Nothing, Unit] = Right(())
```

The code above prints each line as the stream reads it. Notice the order of the output: the file is opened first, then every line is printed, then the file is closed. The `Right(())` at the end is the result of `Stream#runForeach` itself: success, with nothing to collect.

The stream closes the file even when we stop early. `Stream#take` stops after a given number of elements. Here we stop after the first two lines:

```scala
salesLines.take(2).runForeach(line => println(line))
// Opened the sales log
// timestamp,title,quantity
// 2026-10-19T10:15,Zionomicon,2
// Closed the sales log
// res3: Either[Nothing, Unit] = Right(())
```

The code above reads only the header and the first sale, and the file is still closed. Notice also that the file was opened a second time: every run of a stream starts the recipe from the beginning.

## 5. Keep Only the Sales with `filter`

The header line is not a sale, so we drop it. `Stream#filter` keeps the elements for which a predicate returns `true` and drops the rest:

```scala
val header = "timestamp,title,quantity"

val saleLines: Stream[Nothing, String] = salesLines.filter(line => line != header)
```

The code above:

- `header` is the first line of the file, the one we want to drop.
- `salesLines.filter(line => line != header)` builds a new stream that passes every line except the header. `salesLines` itself does not change; each operation returns a new stream.
- `saleLines` has the same type as `salesLines`, `Stream[Nothing, String]`, because `Stream#filter` only removes elements.

Let's run it:

```scala
saleLines.runForeach(line => println(line))
// Opened the sales log
// 2026-10-19T10:15,Zionomicon,2
// 2026-10-19T10:42,Programming in Scala,1
// 2026-10-19T11:05,Functional Programming in Scala,3
// 2026-10-19T12:10,Zionomicon,1
// Closed the sales log
// res4: Either[Nothing, Unit] = Right(())
```

The header is gone; only the four sales are left.

## 6. Turn Each Line into a Sale with `map`

Working with comma-separated text is awkward, so let's give each sale a proper type. `Stream#map` applies a function to every element and emits its result:

```scala
final case class Sale(title: String, quantity: Int)

def toSale(line: String): Sale = {
  val fields = line.split(",")
  Sale(fields(1), fields(2).toInt)
}

val sales: Stream[Nothing, Sale] = saleLines.map(toSale)
```

The code above:

- `Sale` is an ordinary case class with the two fields our report needs.
- `toSale` splits a line at the commas and builds a `Sale` from the second and third field. It ignores the timestamp.
- `saleLines.map(toSale)` turns the `Stream[Nothing, String]` into a `Stream[Nothing, Sale]`.

Let's run it:

```scala
sales.runForeach(sale => println(sale))
// Opened the sales log
// Sale(Zionomicon,2)
// Sale(Programming in Scala,1)
// Sale(Functional Programming in Scala,3)
// Sale(Zionomicon,1)
// Closed the sales log
// res5: Either[Nothing, Unit] = Right(())
```

Each line is now a `Sale` value. Notice that `toSale` assumes every line is well formed. We come back to that assumption in section 9.

## 7. Batch Sales with `grouped`

Many systems prefer data in batches rather than one element at a time — a database insert or a call to a remote service is cheaper for ten rows than ten times for one row. `Stream#grouped` collects up to `n` elements into a `Chunk` and emits the chunk:

```scala
sales.grouped(2).runForeach(batch => println(batch))
// Opened the sales log
// Chunk(Sale(Zionomicon,2),Sale(Programming in Scala,1))
// Chunk(Sale(Functional Programming in Scala,3),Sale(Zionomicon,1))
// Closed the sales log
// res6: Either[Nothing, Unit] = Right(())
```

The code above:

- `sales.grouped(2)` turns our `Stream[Nothing, Sale]` into a `Stream[Nothing, Chunk[Sale]]` with two sales per chunk.
- `Stream#runForeach` prints each chunk as it arrives.
- If the number of sales did not divide evenly, the last chunk would be smaller; a stream never drops elements to fill a batch.

## 8. Add Up the Sales with a `Sink`

So far each terminal has decided what to do with the elements. A [`Sink[E, A, Z]`](../reference/streams/core/sink.md) lets us write that decision as a value of its own: it consumes elements of type `A` and produces one result of type `Z`. `Sink.foldLeft` builds a sink from a starting value and a function that combines the result so far with the next element.

Our report counts copies per title. Since our stream now emits batches, the sink adds up one whole batch at a time:

```scala

val copiesPerTitle: Sink[Nothing, Chunk[Sale], Map[String, Int]] =
  Sink.foldLeft(Map.empty[String, Int]) { (totals: Map[String, Int], batch: Chunk[Sale]) =>
    batch.foldLeft(totals) { (acc, sale) =>
      acc.updated(sale.title, acc.getOrElse(sale.title, 0) + sale.quantity)
    }
  }
```

The code above:

- `Sink[Nothing, Chunk[Sale], Map[String, Int]]` reads as: never fails, consumes `Chunk[Sale]` elements, produces a `Map[String, Int]`.
- `Map.empty[String, Int]` is the starting value: no copies of anything.
- For each batch, `batch.foldLeft(totals)` adds every sale's quantity to the running total for its title.

To run a stream into a sink, we pass the sink to the `run` terminal:

```scala
println(sales.grouped(2).run(copiesPerTitle))
// Opened the sales log
// Closed the sales log
// Right(Map(Zionomicon -> 3, Programming in Scala -> 1, Functional Programming in Scala -> 3))
```

The code above reads the file, drops the header, parses the sales, batches them, and adds them up — in a single pass over the file. Zionomicon sold three copies across two sales; the other titles appear once each.

:::tip
A sink is a reusable value, just like a stream. You can run `copiesPerTitle` against any stream of `Chunk[Sale]`, whether it comes from a file, a list in a test, or a network connection.
:::

## 9. Fail on a Malformed Line

Real logs are messy. Suppose the till crashed while it was writing its last sale, leaving a truncated line at the end of the file. Let's append one:

```scala

Files.writeString(salesLog, "2026-10-19T12:31,Zionom", StandardOpenOption.APPEND)
```

Our `toSale` function would throw an exception on this line. Instead, we want the stream to fail with an error value that says which line was bad. `Stream.fail(error)` creates a stream that fails with `error`, and `Stream#flatMap` replaces each element with a whole stream — here a stream of one `Sale`, or a failed one:

```scala
final case class MalformedLine(line: String)

def parse(line: String): Stream[MalformedLine, Sale] =
  line.split(",") match {
    case Array(_, title, quantity) if quantity.toIntOption.isDefined =>
      Stream.succeed(Sale(title, quantity.toInt))
    case _ =>
      Stream.fail(MalformedLine(line))
  }

val parsedSales: Stream[MalformedLine, Sale] = saleLines.flatMap(parse)
```

The code above:

- `MalformedLine` is our error type. It is a plain case class, not an exception.
- `parse` returns `Stream.succeed(...)`, a stream of exactly one `Sale`, when the line has three fields and a numeric quantity. Otherwise, it returns `Stream.fail(MalformedLine(line))`.
- `saleLines.flatMap(parse)` runs `parse` for every line and emits whatever the stream for that line emits. Notice the new type: `Stream[MalformedLine, Sale]`. The error type now tells everyone who uses `parsedSales` that it can fail, and with what.

Let's run the report again on the new stream:

```scala
println(parsedSales.grouped(2).run(copiesPerTitle))
// Opened the sales log
// Closed the sales log
// Left(MalformedLine(2026-10-19T12:31,Zionom))
```

The result is now a `Left` holding the `MalformedLine`, with the exact text of the bad line. Notice that the file was still closed: the stream releases its resource on failure, just as it does on success.

## 10. Recover with `catchAll`

A failed report is not very useful to the store owner. `Stream#catchAll` takes a function from the error to a new stream. When the stream fails, `Stream#catchAll` calls this function and continues with the stream it returns. To skip the truncated line, we log it and continue with `Stream.empty`, a stream with no elements:

```scala
val cleanSales: Stream[Nothing, Sale] =
  parsedSales.catchAll { bad =>
    println(s"Skipping malformed line: ${bad.line}")
    Stream.empty
  }
```

The code above:

- `bad` is the `MalformedLine` value that `parse` passes to `Stream.fail`.
- `Stream.empty` ends the stream cleanly, so the sales read before the bad line are kept.
- The type of `cleanSales` is `Stream[Nothing, Sale]`. We handled the error, so the error type is `Nothing` again, and the compiler knows this stream cannot fail.

Let's run the report one more time:

```scala
println(cleanSales.grouped(2).run(copiesPerTitle))
// Opened the sales log
// Closed the sales log
// Skipping malformed line: 2026-10-19T12:31,Zionom
// Right(Map(Zionomicon -> 3, Programming in Scala -> 1, Functional Programming in Scala -> 3))
```

The report is back to a `Right`, with the same totals as before the crash. Notice the order of the output: the file is closed before `Stream#catchAll` runs, because the failure ends the stream that read the file.

:::caution
A failure ends the stream it happens in. `Stream#catchAll` does not go back and resume reading after the bad line; it continues with the fallback stream instead. Here the truncated line is the last line of the file, so nothing is lost. If a bad line appeared in the middle of the file, the lines after it would not be read.
:::

:::caution
Run a pipeline that uses `Stream#catchAll` from a method such as `def main`, not from the body of an `object ... extends App`. With `Stream#catchAll`, the stream may call your functions on another thread. If they read a `val` of an object that is still being initialized, that thread waits for the initialization to finish while the initialization waits for the stream, and the program hangs. The companion examples use `def main` for this reason.
:::

## 11. Run the Same Pipeline Asynchronously

Every terminal in this tutorial so far — `Stream#runCollect`, `Stream#runForeach` and `Stream#run` — blocks the calling thread until the stream finishes, and exists only on the JVM. Each one has an asynchronous twin ending in `Async`, which also works on Scala.js. `Stream#runAsync` takes the same sink and returns an [`Async[Either[E, Z]]`](../reference/async.md): a value that completes later with the same `Either`:

```scala

val report: Async[Either[Nothing, Map[String, Int]]] = cleanSales.grouped(2).runAsync(copiesPerTitle)
```

The code above:

- `import zio.blocks.async._` brings the `Async` type and its operations into scope.
- `cleanSales.grouped(2).runAsync(copiesPerTitle)` describes the same run as in the previous section, but nothing runs yet: creating `report` does not open the file.
- `report` has the type `Async[Either[Nothing, Map[String, Int]]]`, the same `Either` that the blocking `run` returns, wrapped in `Async`.

To wait for the result at the edge of a JVM program, we call `Async#block`:

```scala
println(report.block)
// Opened the sales log
// Closed the sales log
// Skipping malformed line: 2026-10-19T12:31,Zionom
// Right(Map(Zionomicon -> 3, Programming in Scala -> 1, Functional Programming in Scala -> 3))
```

The code above runs the pipeline and waits for it. The output matches the blocking run line for line. That's the main idea of this tutorial made visible: `cleanSales.grouped(2)` is one description of the work, and the terminal we choose decides how it runs.

:::note
`Async#block` parks the calling thread, so use it only at the edge of a JVM program, such as in `main`. On Scala.js it throws when the value is not yet complete; there you keep the `Async` value and let the host drive it.
:::

## 12. Putting It Together

This example combines all the concepts we've learned — reading the sales log safely, dropping the header with `Stream#filter`, parsing lines into sales with `Stream#flatMap` and `Stream.fail`, recovering from the truncated line with `Stream#catchAll`, batching with `Stream#grouped`, adding up copies per title with a `Sink`, and running the pipeline with both a blocking and an asynchronous terminal:

```scala title="streams-examples/src/main/scala/yourfirststreamingpipeline/CompleteExample.scala"
package yourfirststreamingpipeline

/**
 * Your First Streaming Pipeline: Turn a Sales Log into a Report — Complete
 * Example
 *
 * Runs the whole pipeline with a blocking run and with runAsync(...).block.
 *
 * Run with: cd streams-examples && sbt "runMain
 * yourfirststreamingpipeline.CompleteExample"
 */
object CompleteExample {
  final case class Sale(title: String, quantity: Int)

  def toSale(line: String): Sale = {
    val fields = line.split(",")
    Sale(fields(1), fields(2).toInt)
  }

  final case class MalformedLine(line: String)

  def parse(line: String): Stream[MalformedLine, Sale] =
    line.split(",") match {
      case Array(_, title, quantity) if quantity.toIntOption.isDefined =>
        Stream.succeed(Sale(title, quantity.toInt))
      case _ =>
        Stream.fail(MalformedLine(line))
    }

  def main(args: Array[String]): Unit = {
    val salesLog: Path = Files.createTempFile("sales", ".csv")

    Files.writeString(
      salesLog,
      """timestamp,title,quantity
        |2026-10-19T10:15,Zionomicon,2
        |2026-10-19T10:42,Programming in Scala,1
        |2026-10-19T11:05,Functional Programming in Scala,3
        |2026-10-19T12:10,Zionomicon,1
        |""".stripMargin
    )

    val salesLines: Stream[Nothing, String] =
      Stream.fromAcquireRelease(
        acquire = {
          println("Opened the sales log")
          Files.newBufferedReader(salesLog)
        },
        release = (reader: BufferedReader) => {
          reader.close()
          println("Closed the sales log")
        }
      )(reader => Stream.fromIterator(reader.lines().iterator().asScala))

    val header = "timestamp,title,quantity"

    val saleLines: Stream[Nothing, String] = salesLines.filter(line => line != header)

    val sales: Stream[Nothing, Sale] = saleLines.map(toSale)

    val copiesPerTitle: Sink[Nothing, Chunk[Sale], Map[String, Int]] =
      Sink.foldLeft(Map.empty[String, Int]) { (totals: Map[String, Int], batch: Chunk[Sale]) =>
        batch.foldLeft(totals) { (acc, sale) =>
          acc.updated(sale.title, acc.getOrElse(sale.title, 0) + sale.quantity)
        }
      }

    Files.writeString(salesLog, "2026-10-19T12:31,Zionom", StandardOpenOption.APPEND)

    val parsedSales: Stream[MalformedLine, Sale] = saleLines.flatMap(parse)

    val cleanSales: Stream[Nothing, Sale] =
      parsedSales.catchAll { bad =>
        println(s"Skipping malformed line: ${bad.line}")
        Stream.empty
      }

    println("Blocking run:")
    println(cleanSales.grouped(2).run(copiesPerTitle))

    val report: Async[Either[Nothing, Map[String, Int]]] = cleanSales.grouped(2).runAsync(copiesPerTitle)

    println("Report described; nothing has run yet")
    println("Async run:")
    println(report.block)
  }
}
```

## 13. Running the Examples

All examples in this tutorial have corresponding runnable Scala files in the `streams-examples` project. To get them, clone the repository and change into the examples project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/streams-examples
```

Run them in order to follow the tutorial step by step.

### Create and Run a Stream

This example creates a stream of book titles and runs it with `runCollect`.

<details>
  <summary>streams-examples/src/main/scala/yourfirststreamingpipeline/Concept1CreateAndRunExample.scala</summary>

```scala title="streams-examples/src/main/scala/yourfirststreamingpipeline/Concept1CreateAndRunExample.scala" showLineNumbers
package yourfirststreamingpipeline

/**
 * Your First Streaming Pipeline: Turn a Sales Log into a Report — Concept 1:
 * Create and Run a Stream
 *
 * Creates a stream of book titles and runs it with runCollect.
 *
 * Run with: cd streams-examples && sbt "runMain
 * yourfirststreamingpipeline.Concept1CreateAndRunExample"
 */
object Concept1CreateAndRunExample {
  def main(args: Array[String]): Unit = {
    val titles = Stream("Zionomicon", "Programming in Scala", "Functional Programming in Scala")

    println("Titles stream described; nothing has run yet")
    println(titles.runCollect)
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain yourfirststreamingpipeline.Concept1CreateAndRunExample"
```

### Read the Sales Log from a File

This example reads the sales log with `Stream.fromAcquireRelease` and shows that the file is closed after a full run and after an early stop with `Stream#take`.

<details>
  <summary>streams-examples/src/main/scala/yourfirststreamingpipeline/Concept2ReadFileExample.scala</summary>

```scala title="streams-examples/src/main/scala/yourfirststreamingpipeline/Concept2ReadFileExample.scala" showLineNumbers
package yourfirststreamingpipeline

/**
 * Your First Streaming Pipeline: Turn a Sales Log into a Report — Concept 2:
 * Read the Sales Log from a File
 *
 * Reads the sales log with Stream.fromAcquireRelease and shows that the file is
 * closed after a full run and after an early stop with take.
 *
 * Run with: cd streams-examples && sbt "runMain
 * yourfirststreamingpipeline.Concept2ReadFileExample"
 */
object Concept2ReadFileExample {
  def main(args: Array[String]): Unit = {
    val salesLog: Path = Files.createTempFile("sales", ".csv")

    Files.writeString(
      salesLog,
      """timestamp,title,quantity
        |2026-10-19T10:15,Zionomicon,2
        |2026-10-19T10:42,Programming in Scala,1
        |2026-10-19T11:05,Functional Programming in Scala,3
        |2026-10-19T12:10,Zionomicon,1
        |""".stripMargin
    )

    val salesLines: Stream[Nothing, String] =
      Stream.fromAcquireRelease(
        acquire = {
          println("Opened the sales log")
          Files.newBufferedReader(salesLog)
        },
        release = (reader: BufferedReader) => {
          reader.close()
          println("Closed the sales log")
        }
      )(reader => Stream.fromIterator(reader.lines().iterator().asScala))

    salesLines.runForeach(line => println(line))
    println("--- take(2) ---")
    salesLines.take(2).runForeach(line => println(line))
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain yourfirststreamingpipeline.Concept2ReadFileExample"
```

### Keep Only the Sales with `filter`

This example drops the header line with `Stream#filter`.

<details>
  <summary>streams-examples/src/main/scala/yourfirststreamingpipeline/Concept3FilterExample.scala</summary>

```scala title="streams-examples/src/main/scala/yourfirststreamingpipeline/Concept3FilterExample.scala" showLineNumbers
package yourfirststreamingpipeline

/**
 * Your First Streaming Pipeline: Turn a Sales Log into a Report — Concept 3:
 * Keep Only the Sales with filter
 *
 * Drops the header line with filter.
 *
 * Run with: cd streams-examples && sbt "runMain
 * yourfirststreamingpipeline.Concept3FilterExample"
 */
object Concept3FilterExample {
  def main(args: Array[String]): Unit = {
    val salesLog: Path = Files.createTempFile("sales", ".csv")

    Files.writeString(
      salesLog,
      """timestamp,title,quantity
        |2026-10-19T10:15,Zionomicon,2
        |2026-10-19T10:42,Programming in Scala,1
        |2026-10-19T11:05,Functional Programming in Scala,3
        |2026-10-19T12:10,Zionomicon,1
        |""".stripMargin
    )

    val salesLines: Stream[Nothing, String] =
      Stream.fromAcquireRelease(
        acquire = {
          println("Opened the sales log")
          Files.newBufferedReader(salesLog)
        },
        release = (reader: BufferedReader) => {
          reader.close()
          println("Closed the sales log")
        }
      )(reader => Stream.fromIterator(reader.lines().iterator().asScala))

    val header = "timestamp,title,quantity"

    val saleLines: Stream[Nothing, String] = salesLines.filter(line => line != header)

    saleLines.runForeach(line => println(line))
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain yourfirststreamingpipeline.Concept3FilterExample"
```

### Turn Each Line into a Sale with `map`

This example turns each line into a `Sale` with `Stream#map`.

<details>
  <summary>streams-examples/src/main/scala/yourfirststreamingpipeline/Concept4MapExample.scala</summary>

```scala title="streams-examples/src/main/scala/yourfirststreamingpipeline/Concept4MapExample.scala" showLineNumbers
package yourfirststreamingpipeline

/**
 * Your First Streaming Pipeline: Turn a Sales Log into a Report — Concept 4:
 * Turn Each Line into a Sale with map
 *
 * Turns each line into a Sale with map.
 *
 * Run with: cd streams-examples && sbt "runMain
 * yourfirststreamingpipeline.Concept4MapExample"
 */
object Concept4MapExample {
  final case class Sale(title: String, quantity: Int)

  def toSale(line: String): Sale = {
    val fields = line.split(",")
    Sale(fields(1), fields(2).toInt)
  }

  def main(args: Array[String]): Unit = {
    val salesLog: Path = Files.createTempFile("sales", ".csv")

    Files.writeString(
      salesLog,
      """timestamp,title,quantity
        |2026-10-19T10:15,Zionomicon,2
        |2026-10-19T10:42,Programming in Scala,1
        |2026-10-19T11:05,Functional Programming in Scala,3
        |2026-10-19T12:10,Zionomicon,1
        |""".stripMargin
    )

    val salesLines: Stream[Nothing, String] =
      Stream.fromAcquireRelease(
        acquire = {
          println("Opened the sales log")
          Files.newBufferedReader(salesLog)
        },
        release = (reader: BufferedReader) => {
          reader.close()
          println("Closed the sales log")
        }
      )(reader => Stream.fromIterator(reader.lines().iterator().asScala))

    val header = "timestamp,title,quantity"

    val saleLines: Stream[Nothing, String] = salesLines.filter(line => line != header)

    val sales: Stream[Nothing, Sale] = saleLines.map(toSale)

    sales.runForeach(sale => println(sale))
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain yourfirststreamingpipeline.Concept4MapExample"
```

### Batch Sales with `grouped`

This example batches the sales two at a time with `Stream#grouped`.

<details>
  <summary>streams-examples/src/main/scala/yourfirststreamingpipeline/Concept5GroupedExample.scala</summary>

```scala title="streams-examples/src/main/scala/yourfirststreamingpipeline/Concept5GroupedExample.scala" showLineNumbers
package yourfirststreamingpipeline

/**
 * Your First Streaming Pipeline: Turn a Sales Log into a Report — Concept 5:
 * Batch Sales with grouped
 *
 * Batches the sales two at a time with grouped.
 *
 * Run with: cd streams-examples && sbt "runMain
 * yourfirststreamingpipeline.Concept5GroupedExample"
 */
object Concept5GroupedExample {
  final case class Sale(title: String, quantity: Int)

  def toSale(line: String): Sale = {
    val fields = line.split(",")
    Sale(fields(1), fields(2).toInt)
  }

  def main(args: Array[String]): Unit = {
    val salesLog: Path = Files.createTempFile("sales", ".csv")

    Files.writeString(
      salesLog,
      """timestamp,title,quantity
        |2026-10-19T10:15,Zionomicon,2
        |2026-10-19T10:42,Programming in Scala,1
        |2026-10-19T11:05,Functional Programming in Scala,3
        |2026-10-19T12:10,Zionomicon,1
        |""".stripMargin
    )

    val salesLines: Stream[Nothing, String] =
      Stream.fromAcquireRelease(
        acquire = {
          println("Opened the sales log")
          Files.newBufferedReader(salesLog)
        },
        release = (reader: BufferedReader) => {
          reader.close()
          println("Closed the sales log")
        }
      )(reader => Stream.fromIterator(reader.lines().iterator().asScala))

    val header = "timestamp,title,quantity"

    val saleLines: Stream[Nothing, String] = salesLines.filter(line => line != header)

    val sales: Stream[Nothing, Sale] = saleLines.map(toSale)

    sales.grouped(2).runForeach(batch => println(batch))
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain yourfirststreamingpipeline.Concept5GroupedExample"
```

### Add Up the Sales with a `Sink`

This example adds up copies per title with a `Sink.foldLeft` and runs the stream into it with `run`.

<details>
  <summary>streams-examples/src/main/scala/yourfirststreamingpipeline/Concept6SinkExample.scala</summary>

```scala title="streams-examples/src/main/scala/yourfirststreamingpipeline/Concept6SinkExample.scala" showLineNumbers
package yourfirststreamingpipeline

/**
 * Your First Streaming Pipeline: Turn a Sales Log into a Report — Concept 6:
 * Add Up the Sales with a Sink
 *
 * Adds up copies per title with Sink.foldLeft and runs the stream into it with
 * run.
 *
 * Run with: cd streams-examples && sbt "runMain
 * yourfirststreamingpipeline.Concept6SinkExample"
 */
object Concept6SinkExample {
  final case class Sale(title: String, quantity: Int)

  def toSale(line: String): Sale = {
    val fields = line.split(",")
    Sale(fields(1), fields(2).toInt)
  }

  def main(args: Array[String]): Unit = {
    val salesLog: Path = Files.createTempFile("sales", ".csv")

    Files.writeString(
      salesLog,
      """timestamp,title,quantity
        |2026-10-19T10:15,Zionomicon,2
        |2026-10-19T10:42,Programming in Scala,1
        |2026-10-19T11:05,Functional Programming in Scala,3
        |2026-10-19T12:10,Zionomicon,1
        |""".stripMargin
    )

    val salesLines: Stream[Nothing, String] =
      Stream.fromAcquireRelease(
        acquire = {
          println("Opened the sales log")
          Files.newBufferedReader(salesLog)
        },
        release = (reader: BufferedReader) => {
          reader.close()
          println("Closed the sales log")
        }
      )(reader => Stream.fromIterator(reader.lines().iterator().asScala))

    val header = "timestamp,title,quantity"

    val saleLines: Stream[Nothing, String] = salesLines.filter(line => line != header)

    val sales: Stream[Nothing, Sale] = saleLines.map(toSale)

    val copiesPerTitle: Sink[Nothing, Chunk[Sale], Map[String, Int]] =
      Sink.foldLeft(Map.empty[String, Int]) { (totals: Map[String, Int], batch: Chunk[Sale]) =>
        batch.foldLeft(totals) { (acc, sale) =>
          acc.updated(sale.title, acc.getOrElse(sale.title, 0) + sale.quantity)
        }
      }

    println(sales.grouped(2).run(copiesPerTitle))
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain yourfirststreamingpipeline.Concept6SinkExample"
```

### Fail on a Malformed Line

This example appends a truncated line to the log, parses lines with `Stream#flatMap` and `Stream.fail`, and shows the `Left` that the run returns.

<details>
  <summary>streams-examples/src/main/scala/yourfirststreamingpipeline/Concept7FailExample.scala</summary>

```scala title="streams-examples/src/main/scala/yourfirststreamingpipeline/Concept7FailExample.scala" showLineNumbers
package yourfirststreamingpipeline

/**
 * Your First Streaming Pipeline: Turn a Sales Log into a Report — Concept 7:
 * Fail on a Malformed Line
 *
 * Appends a truncated line to the log, parses lines with flatMap and
 * Stream.fail, and prints the Left that the run returns.
 *
 * Run with: cd streams-examples && sbt "runMain
 * yourfirststreamingpipeline.Concept7FailExample"
 */
object Concept7FailExample {
  final case class Sale(title: String, quantity: Int)

  def toSale(line: String): Sale = {
    val fields = line.split(",")
    Sale(fields(1), fields(2).toInt)
  }

  final case class MalformedLine(line: String)

  def parse(line: String): Stream[MalformedLine, Sale] =
    line.split(",") match {
      case Array(_, title, quantity) if quantity.toIntOption.isDefined =>
        Stream.succeed(Sale(title, quantity.toInt))
      case _ =>
        Stream.fail(MalformedLine(line))
    }

  def main(args: Array[String]): Unit = {
    val salesLog: Path = Files.createTempFile("sales", ".csv")

    Files.writeString(
      salesLog,
      """timestamp,title,quantity
        |2026-10-19T10:15,Zionomicon,2
        |2026-10-19T10:42,Programming in Scala,1
        |2026-10-19T11:05,Functional Programming in Scala,3
        |2026-10-19T12:10,Zionomicon,1
        |""".stripMargin
    )

    val salesLines: Stream[Nothing, String] =
      Stream.fromAcquireRelease(
        acquire = {
          println("Opened the sales log")
          Files.newBufferedReader(salesLog)
        },
        release = (reader: BufferedReader) => {
          reader.close()
          println("Closed the sales log")
        }
      )(reader => Stream.fromIterator(reader.lines().iterator().asScala))

    val header = "timestamp,title,quantity"

    val saleLines: Stream[Nothing, String] = salesLines.filter(line => line != header)

    val sales: Stream[Nothing, Sale] = saleLines.map(toSale)

    val copiesPerTitle: Sink[Nothing, Chunk[Sale], Map[String, Int]] =
      Sink.foldLeft(Map.empty[String, Int]) { (totals: Map[String, Int], batch: Chunk[Sale]) =>
        batch.foldLeft(totals) { (acc, sale) =>
          acc.updated(sale.title, acc.getOrElse(sale.title, 0) + sale.quantity)
        }
      }

    Files.writeString(salesLog, "2026-10-19T12:31,Zionom", StandardOpenOption.APPEND)

    val parsedSales: Stream[MalformedLine, Sale] = saleLines.flatMap(parse)

    println(parsedSales.grouped(2).run(copiesPerTitle))
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain yourfirststreamingpipeline.Concept7FailExample"
```

### Recover with `catchAll`

This example recovers from the truncated line with `Stream#catchAll` and prints the report.

<details>
  <summary>streams-examples/src/main/scala/yourfirststreamingpipeline/Concept8CatchAllExample.scala</summary>

```scala title="streams-examples/src/main/scala/yourfirststreamingpipeline/Concept8CatchAllExample.scala" showLineNumbers
package yourfirststreamingpipeline

/**
 * Your First Streaming Pipeline: Turn a Sales Log into a Report — Concept 8:
 * Recover with catchAll
 *
 * Recovers from the truncated line with catchAll and prints the report.
 *
 * Uses def main rather than extends App: with catchAll the stream may call
 * these functions on another thread, which deadlocks inside an object
 * initializer.
 *
 * Run with: cd streams-examples && sbt "runMain
 * yourfirststreamingpipeline.Concept8CatchAllExample"
 */
object Concept8CatchAllExample {
  final case class Sale(title: String, quantity: Int)

  def toSale(line: String): Sale = {
    val fields = line.split(",")
    Sale(fields(1), fields(2).toInt)
  }

  final case class MalformedLine(line: String)

  def parse(line: String): Stream[MalformedLine, Sale] =
    line.split(",") match {
      case Array(_, title, quantity) if quantity.toIntOption.isDefined =>
        Stream.succeed(Sale(title, quantity.toInt))
      case _ =>
        Stream.fail(MalformedLine(line))
    }

  def main(args: Array[String]): Unit = {
    val salesLog: Path = Files.createTempFile("sales", ".csv")

    Files.writeString(
      salesLog,
      """timestamp,title,quantity
        |2026-10-19T10:15,Zionomicon,2
        |2026-10-19T10:42,Programming in Scala,1
        |2026-10-19T11:05,Functional Programming in Scala,3
        |2026-10-19T12:10,Zionomicon,1
        |""".stripMargin
    )

    val salesLines: Stream[Nothing, String] =
      Stream.fromAcquireRelease(
        acquire = {
          println("Opened the sales log")
          Files.newBufferedReader(salesLog)
        },
        release = (reader: BufferedReader) => {
          reader.close()
          println("Closed the sales log")
        }
      )(reader => Stream.fromIterator(reader.lines().iterator().asScala))

    val header = "timestamp,title,quantity"

    val saleLines: Stream[Nothing, String] = salesLines.filter(line => line != header)

    val sales: Stream[Nothing, Sale] = saleLines.map(toSale)

    val copiesPerTitle: Sink[Nothing, Chunk[Sale], Map[String, Int]] =
      Sink.foldLeft(Map.empty[String, Int]) { (totals: Map[String, Int], batch: Chunk[Sale]) =>
        batch.foldLeft(totals) { (acc, sale) =>
          acc.updated(sale.title, acc.getOrElse(sale.title, 0) + sale.quantity)
        }
      }

    Files.writeString(salesLog, "2026-10-19T12:31,Zionom", StandardOpenOption.APPEND)

    val parsedSales: Stream[MalformedLine, Sale] = saleLines.flatMap(parse)

    val cleanSales: Stream[Nothing, Sale] =
      parsedSales.catchAll { bad =>
        println(s"Skipping malformed line: ${bad.line}")
        Stream.empty
      }

    println(cleanSales.grouped(2).run(copiesPerTitle))
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain yourfirststreamingpipeline.Concept8CatchAllExample"
```

### Run the Same Pipeline Asynchronously

This example runs the pipeline with `runAsync` and waits for the result with `Async#block`.

<details>
  <summary>streams-examples/src/main/scala/yourfirststreamingpipeline/Concept9AsyncExample.scala</summary>

```scala title="streams-examples/src/main/scala/yourfirststreamingpipeline/Concept9AsyncExample.scala" showLineNumbers
package yourfirststreamingpipeline

/**
 * Your First Streaming Pipeline: Turn a Sales Log into a Report — Concept 9:
 * Run the Same Pipeline Asynchronously
 *
 * Runs the pipeline with runAsync and waits for the result with block.
 *
 * Run with: cd streams-examples && sbt "runMain
 * yourfirststreamingpipeline.Concept9AsyncExample"
 */
object Concept9AsyncExample {
  final case class Sale(title: String, quantity: Int)

  def toSale(line: String): Sale = {
    val fields = line.split(",")
    Sale(fields(1), fields(2).toInt)
  }

  final case class MalformedLine(line: String)

  def parse(line: String): Stream[MalformedLine, Sale] =
    line.split(",") match {
      case Array(_, title, quantity) if quantity.toIntOption.isDefined =>
        Stream.succeed(Sale(title, quantity.toInt))
      case _ =>
        Stream.fail(MalformedLine(line))
    }

  def main(args: Array[String]): Unit = {
    val salesLog: Path = Files.createTempFile("sales", ".csv")

    Files.writeString(
      salesLog,
      """timestamp,title,quantity
        |2026-10-19T10:15,Zionomicon,2
        |2026-10-19T10:42,Programming in Scala,1
        |2026-10-19T11:05,Functional Programming in Scala,3
        |2026-10-19T12:10,Zionomicon,1
        |""".stripMargin
    )

    val salesLines: Stream[Nothing, String] =
      Stream.fromAcquireRelease(
        acquire = {
          println("Opened the sales log")
          Files.newBufferedReader(salesLog)
        },
        release = (reader: BufferedReader) => {
          reader.close()
          println("Closed the sales log")
        }
      )(reader => Stream.fromIterator(reader.lines().iterator().asScala))

    val header = "timestamp,title,quantity"

    val saleLines: Stream[Nothing, String] = salesLines.filter(line => line != header)

    val sales: Stream[Nothing, Sale] = saleLines.map(toSale)

    val copiesPerTitle: Sink[Nothing, Chunk[Sale], Map[String, Int]] =
      Sink.foldLeft(Map.empty[String, Int]) { (totals: Map[String, Int], batch: Chunk[Sale]) =>
        batch.foldLeft(totals) { (acc, sale) =>
          acc.updated(sale.title, acc.getOrElse(sale.title, 0) + sale.quantity)
        }
      }

    Files.writeString(salesLog, "2026-10-19T12:31,Zionom", StandardOpenOption.APPEND)

    val parsedSales: Stream[MalformedLine, Sale] = saleLines.flatMap(parse)

    val cleanSales: Stream[Nothing, Sale] =
      parsedSales.catchAll { bad =>
        println(s"Skipping malformed line: ${bad.line}")
        Stream.empty
      }

    val report: Async[Either[Nothing, Map[String, Int]]] = cleanSales.grouped(2).runAsync(copiesPerTitle)

    println("Report described; nothing has run yet")
    println(report.block)
  }
}
```

</details>

Run it from the `streams-examples` directory:

```bash
sbt "runMain yourfirststreamingpipeline.Concept9AsyncExample"
```

### The Complete Example

The complete example from "Putting It Together" runs everything from the tutorial in one program:

```bash
sbt "runMain yourfirststreamingpipeline.CompleteExample"
```

To compile every example at once without running them:

```bash
sbt compile
```

## 14. What You've Learned

In this tutorial, you learned:

- What a `Stream` is — a recipe for producing elements — and why nothing happens until a terminal such as `Stream#runCollect`, `Stream#runForeach` or `Stream#run` runs it
- How to read a file with `Stream.fromAcquireRelease` so that the file is closed after success, failure, or an early stop
- How to transform a stream with `Stream#filter`, `Stream#map` and `Stream#grouped`
- How a `Sink` built with `Sink.foldLeft` consumes a stream and produces one result
- How `Stream.fail` and `Stream#flatMap` turn a bad line into a typed error, and how `Stream#catchAll` recovers from it
- How the same stream runs with the blocking `Stream#run` and the asynchronous `Stream#runAsync`

You now have a solid foundation in ZIO Blocks Streams. The next step is to explore the full set of constructors, operators and sinks.

## 15. Where to Go Next

- **Want the big picture?** The [Streams overview](../reference/streams/index.md) explains the core mental model and how `Stream`, `Sink` and `Pipeline` fit together.
- **Ready to read real files and sockets?** [How to Stream Files and Sockets Without Leaking Resources](./stream-files-and-sockets.md) applies `Stream.fromAcquireRelease` and the NIO constructors to files, channels and sockets, and closes every handle on success, failure and early stop.
- **Coming from another streaming library?** [Coming from fs2, ZIO Streams or Ox](./streaming-library-migration.md) maps fs2, ZIO Streams and Ox code onto ZIO Blocks streams.
- **Want to see every operator?** The [Stream](../reference/streams/core/stream.md) reference page documents every constructor, transformation and terminal, including `Stream.fromAcquireRelease`, `Stream#grouped` and `Stream#catchAll`.
- **Need another way to consume a stream?** The [Sink](../reference/streams/core/sink.md) reference page lists every predefined sink and shows how to transform sinks with `Sink#contramap` and `Sink#map`.
- **Want to reuse a chain of transformations?** Read [Pipeline](../reference/streams/core/pipeline.md), which turns a chain like our `Stream#filter` and `Stream#map` into a value you can apply to any stream.
- **Curious about the asynchronous terminals?** Read [Async Execution](../reference/streams/execution-and-compatibility/async-execution.md), and the [Async](../reference/async.md) reference for the `Async` type that `runAsync` returns.
- **New to `Async` itself?** The [Getting Started with Async](./async-getting-started.md) guide introduces it step by step.

---

## Migrating from ZIO Schema to ZIO Blocks Schema

This guide helps you migrate an application that uses [ZIO Schema](https://github.com/zio/zio-schema) (version 1.x) to [ZIO Blocks Schema](https://github.com/zio/zio-blocks) (the schema module of ZIO Blocks). It covers the conceptual differences between the two libraries, provides a systematic mapping of data types, and shows how to rewrite the most common patterns in the idiomatic ZIO Blocks style.

**What we will cover:**

- Prerequisites and dependency changes
- The core architectural shift from `Schema[A]` as a sealed trait to `Schema[A]` as a thin wrapper over `Reflect[F, A]`
- Migrating schema definitions for primitives, records, enums, collections, optional values, and newtypes
- Replacing the annotation/modifier system
- Adapting codec derivation to the unified `Format + Deriver` model
- Replacing `DynamicValue` usage
- Migrating optics and accessor patterns
- Migrating diff, patch, and schema evolution patterns
- Handling types and features that no longer have a direct analogue

Every ZIO Blocks snippet on this page is compiled and run when the site is built, and the output shown under it is the real output. The ZIO Schema 1.x snippets are shown for comparison only; they are not compiled here.

---

## Prerequisites

Moving to ZIO Blocks Schema changes your build in two places: the dependencies you declare and the packages you import.

### Dependency Changes

Replace the ZIO Schema dependency group with ZIO Blocks Schema.

**Before (ZIO Schema 1.x):**

```scala
libraryDependencies += "dev.zio" %% "zio-schema"            % "1.x.x"
libraryDependencies += "dev.zio" %% "zio-schema-derivation" % "1.x.x"
libraryDependencies += "dev.zio" %% "zio-schema-json"       % "1.x.x"
libraryDependencies += "dev.zio" %% "zio-schema-protobuf"   % "1.x.x"
libraryDependencies += "dev.zio" %% "zio-schema-avro"       % "1.x.x"
```

**After (ZIO Blocks Schema):**

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56"
scalacOptions += "-experimental"
```

The other codec formats are optional modules. On Scala 3.9 and newer, each of them also takes the `zio-blocks-next-` prefix, for example `zio-blocks-next-schema-avro`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema-avro"        % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-thrift"      % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-bson"        % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-messagepack" % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-toon"        % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-yaml"        % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-xml"         % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-csv"         % "0.0.56"
```

For scala-cli, Scala.js, or cross-building, see [How to Add ZIO Blocks to Your Build](../getting-started/installation.md).

Key points:
- JSON codec support is built into `zio-blocks-schema` — no separate JSON module.
- There is no separate `zio-blocks-schema-derivation` dependency; derivation is built in.
- You do not need to add `scala-reflect` for Scala 2 macro derivation: on Scala 2.13, `zio-blocks-schema` brings it in as a transitive dependency.
- ZIO Blocks Schema has **zero runtime dependency on ZIO itself**. You do not need `zio` on your classpath for schema operations.

### Package Rename

All imports change from `zio.schema` to `zio.blocks.schema`.

**Before (ZIO Schema 1.x):**

```scala

```

**After (ZIO Blocks Schema):**

```scala

```

There is no separate annotation package: the `Modifier` annotations live in `zio.blocks.schema`. The codecs of `zio.schema.codec` move to one package per format (`zio.blocks.schema.json`, `zio.blocks.schema.avro`, and so on), and the `MetaSchema` of `zio.schema.meta` is replaced by `DynamicSchema` in `zio.blocks.schema`.

---

## The Core Architecture Shift

The most important thing to understand when migrating is that `Schema[A]` is no longer a sealed trait hierarchy — it is a thin case class.

**Before (ZIO Schema 1.x):** `Schema` is a sealed trait with about 16 kinds of node, and 61 concrete case classes once the arity-specialised `CaseClass0` to `CaseClass22` and `Enum1` to `Enum22` are counted one by one:

```scala
sealed trait Schema[A] {
  def annotations: Chunk[Any]
  def defaultValue: Either[String, A]
  // ...
}
```

**After (ZIO Blocks Schema):** `Schema` is a case class wrapping a `Reflect`:

```scala
final case class Schema[A](reflect: Reflect.Bound[A])
```

The structural description lives in `Reflect[F[_, _], A]`, a sealed trait with eight node types. The `F` type parameter distinguishes a *bound* reflect (with runtime constructors and deconstructors) from an *unbound* one (structural information only):

```
Reflect[F, A]
├── Reflect.Record[F, A]         — case classes and other product types
├── Reflect.Variant[F, A]        — sealed traits, enums, Option, Either
├── Reflect.Sequence[F, A, C[_]] — List, Vector, Set, Chunk, etc.
├── Reflect.Map[F, K, V, M[_,_]] — Map[K, V]
├── Reflect.Primitive[F, A]      — Int, String, UUID, java.time.*, etc.
├── Reflect.Wrapper[F, A, B]     — opaque types and validated newtypes
├── Reflect.Dynamic[F]           — escape hatch for schema-agnostic data
└── Reflect.Deferred[F, A]       — recursive (self-referential) types
```

As a result, you will rarely pattern match on `Schema[A]` directly — instead, you work through `schema.reflect` when you need to inspect structure. The [Reflect reference](../reference/schema/reflect.md) describes each node type.

---

## Migrating Schema Definitions

Most schema definitions migrate by changing one call. This section goes through each kind of type and notes where the underlying model differs.

### Schema Derivation

Automatic derivation changes in name only: `DeriveSchema.gen` becomes `Schema.derived`.

**Before (ZIO Schema 1.x):**

```scala
// The type is inferred from the ascription in Scala 2 and Scala 3;
// DeriveSchema.gen[Person] also works

final case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen
}
```

**After (ZIO Blocks Schema):** the same call in Scala 2 and Scala 3:

```scala

final case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}
```

Printing a schema shows the structure that `Schema.derived` built:

```scala
println(Person.schema)
// Schema {
//   record Person {
//     name: String
//     age: Int
//   }
// }
```

`Schema.derived[A]` works identically in both Scala 2 and Scala 3 in ZIO Blocks Schema, and `Schema.derived` without the type parameter works too when the type is ascribed. There is no separate `DeriveSchema` object to import. For a first walk through derivation and JSON, see [Your First Schema](./your-first-schema.md).

### Primitives

ZIO Blocks Schema has 30 primitive types: `Unit`, `Boolean`, `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `Char`, `String`, `BigInt`, `BigDecimal`, the 16 `java.time` types, `Currency`, and `UUID`. ZIO Schema's 31 standard types cover the same ground with two differences. ZIO Schema's numeric standard types are `java.math.BigInteger` and `java.math.BigDecimal`, with Scala's `BigInt` and `BigDecimal` built on top of them, while ZIO Blocks has the Scala types as primitives. ZIO Schema also has a `Binary` standard type for `Chunk[Byte]`, while in ZIO Blocks a `Chunk[Byte]` is a sequence of `Byte` values.

Implicit schemas are available in the same way.

**Before (ZIO Schema 1.x):**

```scala
val intSchema: Schema[Int]                   = Schema[Int]
val instantSchema: Schema[java.time.Instant] = Schema[java.time.Instant]
```

**After (ZIO Blocks Schema):** identical call sites:

```scala

val intSchema: Schema[Int]                   = Schema[Int]
val instantSchema: Schema[java.time.Instant] = Schema[java.time.Instant]
```

The underlying representation changes: ZIO Schema uses `Schema.Primitive[A](standardType: StandardType[A])`, while ZIO Blocks Schema uses `Reflect.Primitive[F, A](primitiveType: PrimitiveType[A], ...)`. In ZIO Blocks, the primitive type also carries an embedded `Validation[A]` constraint (see the [Migrating Validation](#migrating-validation) section below).

### Records (Case Classes)

**Before (ZIO Schema 1.x):**

```scala

final case class Address(street: String, city: String, postCode: String)
final case class Person(name: String, age: Int, address: Address)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen
}
```

**After (ZIO Blocks Schema):**

```scala

final case class Address(street: String, city: String, postCode: String)
final case class Person(name: String, age: Int, address: Address)

object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}
```

The derived schema includes the nested `Address` record, even though `Address` has no schema of its own:

```scala
println(Person.schema)
// Schema {
//   record Person {
//     name: String
//     age: Int
//     address: record Address {
//       street: String
//       city: String
//       postCode: String
//     }
//   }
// }
```

The derivation call is identical. Internally, ZIO Blocks generates a `Reflect.Record` node, one generic node type for records of every size. ZIO Schema uses the arity-specialised `CaseClass0` to `CaseClass22` instead, and falls back to `GenericRecord` for records with more than 22 fields.

If you were writing schemas manually using `Schema.CaseClass2[...]` or similar, you will need to rewrite those. The equivalent in ZIO Blocks is to write the `Reflect.Record` directly or, preferably, just use `Schema.derived[A]`.

**Before (ZIO Schema 1.x):** manual construction for a 2-field record:

```scala
val personSchema: Schema[Person] =
  Schema.CaseClass2[String, Int, Person](
    id0 = TypeId.fromTypeName("Person"),
    field01 = Schema.Field("name", Schema[String], get0 = _.name, set0 = (p, v) => p.copy(name = v)),
    field02 = Schema.Field("age",  Schema[Int],    get0 = _.age,  set0 = (p, v) => p.copy(age = v)),
    construct0 = Person(_, _)
  )
```

**After (ZIO Blocks Schema):** prefer derivation; no manual `CaseClass*` is required:

```scala
val personSchema: Schema[Person] = Schema.derived[Person]
```

Manual construction is still possible in ZIO Blocks Schema (by assembling a `Reflect.Record` directly), but it is substantially more involved because you must supply a `Binding.Record` with explicit `Constructor[A]` and `Deconstructor[A]` implementations that use the unboxed register system. Automatic derivation is strongly preferred; the [Binding reference](../reference/schema/binding.md) covers the manual route.

### Sealed Traits / Enums (Sum Types)

**Before (ZIO Schema 1.x):**

```scala

sealed trait Shape
case class Circle(radius: Double)                   extends Shape
case class Rectangle(width: Double, height: Double) extends Shape

object Shape {
  implicit val schema: Schema[Shape] = DeriveSchema.gen
}
```

**After (ZIO Blocks Schema):**

```scala

sealed trait Shape
case class Circle(radius: Double)                   extends Shape
case class Rectangle(width: Double, height: Double) extends Shape

object Shape {
  implicit val schema: Schema[Shape] = Schema.derived[Shape]
}
```

The derived schema is a variant with one case per subtype:

```scala
println(Shape.schema)
// Schema {
//   variant Shape {
//     | Circle(radius: Double)
//     | Rectangle(
//         width: Double,
//         height: Double
//       )
//   }
// }
```

Again, the call is identical. Internally, ZIO Blocks generates a `Reflect.Variant` node for any number of cases, where ZIO Schema uses `Enum1` to `Enum22`, and `EnumN` beyond that.

### Optional Values

ZIO Schema has a first-class `Schema.Optional[A]` node. In ZIO Blocks, `Option[A]` is modeled as a `Reflect.Variant` with two cases (`None` and `Some`). From a user perspective this is transparent — implicit schemas for `Option[A]` exist in the same form.

**Before (ZIO Schema 1.x):**

```scala
val optSchema: Schema[Option[String]] = Schema[Option[String]]
```

**After (ZIO Blocks Schema):** identical:

```scala

val optSchema: Schema[Option[String]] = Schema[Option[String]]
```

For value-type `Option` variants (e.g., `Option[Int]`), ZIO Blocks provides specialised implicit instances (`Schema.optionInt`, `Schema.optionLong`, etc.) that avoid boxing. These are resolved automatically by the compiler — no code change required.

:::tip
The internal modeling difference (variant vs. dedicated node) is only relevant if you are pattern-matching on the raw `Schema` or `Reflect` structure. In that case, replace any match on `Schema.Optional(inner, _)` with a check on `Reflect#isOption` and use `Reflect#optionInnerType` to retrieve the inner reflect.

**Before (ZIO Schema 1.x):** pattern matching on `Optional`:

```scala
schema match {
  case Schema.Optional(inner, _) => // use inner
  case _                         => // ...
}
```

**After (ZIO Blocks Schema):** the `Reflect#isOption` predicate and the inner node:

```scala

val r = Schema[Option[String]].reflect
```

The reflect of `Option[String]` is an option, and its inner node is the `String` primitive:

```scala
r.isOption
// res11: Boolean = true
println(r.optionInnerType)
// Some(String)
```
:::

### Either

In ZIO Schema, `Either[A, B]` is a first-class `Schema.Either[A, B]` node. In ZIO Blocks, it is modeled as a two-case `Reflect.Variant`. The implicit schema is provided automatically.

**Before (ZIO Schema 1.x):**

```scala
val eitherSchema: Schema[Either[String, Int]] = Schema.either[String, Int]
```

**After (ZIO Blocks Schema):**

```scala

val eitherSchema: Schema[Either[String, Int]] = Schema[Either[String, Int]]
```

:::warning
ZIO Blocks Schema does not have a `Fallback[A, B]` type. If you were using `Schema.Fallback` for partial decoding, you will need to model that with a custom `Reflect.Variant` or handle it in your codec logic directly.
:::

### Collections

All standard collection types are supported with the same implicit schema pattern.

**Before (ZIO Schema 1.x):**

```scala
Schema[List[String]]
Schema[Vector[Int]]
Schema[Chunk[Double]]
Schema[Set[String]]
Schema[Map[String, Int]]
```

**After (ZIO Blocks Schema):** identical call sites:

```scala

Schema[List[String]]
Schema[Vector[Int]]
Schema[Chunk[Double]] // uses zio.blocks.chunk.Chunk
Schema[Set[String]]
Schema[Map[String, Int]]
```

Note that `Chunk` is now `zio.blocks.chunk.Chunk` (not `zio.Chunk`). This is a zero-dependency replacement with the same API surface for typical usage; see the [Chunk reference](../reference/chunk.md).

ZIO Schema's `NonEmptyChunk`, `NonEmptySet` and `NonEmptyMap` schemas do not have direct equivalents in ZIO Blocks Schema. The recommended approach is to model them as wrapper types.

**Before (ZIO Schema 1.x):** the `NonEmptyChunk` implicit schema:

```scala
val schema: Schema[NonEmptyChunk[String]] = Schema[NonEmptyChunk[String]]
```

**After (ZIO Blocks Schema):** a validated wrapper:

```scala

final case class NonEmptyList[A] private (values: List[A])
object NonEmptyList {
  def apply[A](head: A, tail: A*): NonEmptyList[A] = new NonEmptyList(head :: tail.toList)

  implicit def schema[A](implicit element: Schema[A]): Schema[NonEmptyList[A]] =
    Schema[List[A]].transform(
      to = list =>
        if (list.nonEmpty) new NonEmptyList(list)
        else throw SchemaError.validationFailed("List must not be empty"),
      from = _.values
    )
}
```

Decoding an empty list fails with the wrapper's message, and a non-empty one succeeds:

```scala
println(Schema[NonEmptyList[String]].fromDynamicValue(Schema[List[String]].toDynamicValue(Nil)))
// Left(zio.blocks.schema.SchemaError: List must not be empty at: .wrapped)
println(Schema[NonEmptyList[String]].fromDynamicValue(Schema[List[String]].toDynamicValue(List("sale", "new"))))
// Right(NonEmptyList(List(sale, new)))
```

### Newtypes and Opaque Types

ZIO Schema uses `Schema#transform` and `Schema#transformOrFail` (which wrap a `Transform` node) for both validated newtypes and lossless wrappers.

**Before (ZIO Schema 1.x):**

```scala
implicit val bigDecimalSchema: Schema[BigDecimal] =
  Schema.primitive[java.math.BigDecimal].transform(BigDecimal(_), _.bigDecimal)
```

ZIO Blocks Schema has `Schema#transform(to: A => B, from: B => A)`, which produces a `Reflect.Wrapper` node. The `to` and `from` functions are total but can throw to indicate failure. The method also requires an implicit `TypeId[B]`, which is derived automatically by the macro system for any concrete named type — you will not need to supply it manually for ordinary case classes.

**After (ZIO Blocks Schema):** a validated wrapper and a lossless one:

```scala

case class Email(value: String)
object Email {
  // TypeId[Email] is resolved implicitly from the macro-derived instance
  implicit val schema: Schema[Email] =
    Schema[String].transform(
      to = str =>
        if (str.contains('@')) Email(str)
        else throw SchemaError.validationFailed("Not a valid email address"),
      from = _.value
    )
}

case class UserId(value: Long)
object UserId {
  implicit val schema: Schema[UserId] =
    Schema[Long].transform(UserId(_), _.value)
}
```

A wrapper decodes from the value it wraps, and the validated one rejects bad input with the thrown message:

```scala
println(UserId.schema.toDynamicValue(UserId(42L)))
// 42
println(Email.schema.fromDynamicValue(Schema[String].toDynamicValue("alice@example.com")))
// Right(Email(alice@example.com))
println(Email.schema.fromDynamicValue(Schema[String].toDynamicValue("not-an-email")))
// Left(zio.blocks.schema.SchemaError: Not a valid email address at: .wrapped)
```

:::warning
In ZIO Schema, `Schema#transformOrFail` accepts functions that return `Either[String, B]`. In ZIO Blocks, `Schema#transform` uses total functions that throw on failure — use `throw SchemaError.validationFailed(message)` in the `to` function to signal failure. There is no `transformOrFail` method. To collect every validation error instead of stopping at the first, see [How to Define Validated Newtypes](./validated-newtypes.md).
:::

### Lazy / Recursive Schemas

**Before (ZIO Schema 1.x):**

```scala

case class Tree(value: Int, children: List[Tree])
object Tree {
  implicit lazy val schema: Schema[Tree] = DeriveSchema.gen
  // Or manually with Schema.defer:
  // implicit lazy val schema: Schema[Tree] = Schema.CaseClass2(
  //   ..., field02 = Schema.Field("children", Schema.defer(Schema.list(schema)), ...)
  // )
}
```

**After (ZIO Blocks Schema):**

```scala

case class Tree(value: Int, children: List[Tree])
object Tree {
  implicit val schema: Schema[Tree] = Schema.derived[Tree]
}
```

A tree round-trips through `DynamicValue` with this schema:

```scala
println(Tree.schema.fromDynamicValue(Tree.schema.toDynamicValue(Tree(1, List(Tree(2, Nil), Tree(3, Nil))))))
// Right(Tree(1,List(Tree(2,List()), Tree(3,List()))))
```

Recursive types are handled automatically by the macro. Internally, ZIO Blocks generates a `Reflect.Deferred` node for the recursive reference — you do not need to use `Schema.defer` manually. The `implicit val` (not `lazy val`) is sufficient.

If you were wrapping a recursive reference manually with `Schema.defer(...)`, simply remove that wrapper — recursive references inside `Schema.derived` are handled for you.

---

## Migrating Annotations and Modifiers

ZIO Schema uses an open `Chunk[Any]` annotation system. ZIO Blocks Schema replaces this with a strongly-typed, sealed `Modifier` hierarchy, described on the [Modifier reference page](../reference/schema/modifier.md). Most modifiers are read by the JSON codec, so the examples below encode with the schema's JSON codec, `Schema#jsonCodec` (see [Migrating Codec Derivation](#migrating-codec-derivation)). [How to Customize JSON Encoding](./customize-json-encoding.md) covers the JSON options in depth.

### Transient Fields

**Before (ZIO Schema 1.x):**

```scala

final case class User(name: String, @transientField password: String = "")
object User {
  implicit val schema: Schema[User] = DeriveSchema.gen
}
```

**After (ZIO Blocks Schema):** as in ZIO Schema 1.x, a transient field needs a default value, except that ZIO Blocks also accepts an `Option` or a collection without one. The JSON codec leaves the field out when it encodes, so the decoder fills it in from the default; without one, `Schema.derived` fails to compile:

```scala

final case class User(name: String, @Modifier.transient() password: String = "")
object User {
  implicit val schema: Schema[User] = Schema.derived[User]
}

val userJson: String = User.schema.jsonCodec.encodeToString(User("alice", "s3cr3t"))
```

The encoded JSON has no `password` key, and decoding it gives the default back:

```scala
println(userJson)
// {"name":"alice"}
println(User.schema.jsonCodec.decode(userJson))
// Right(User(alice,))
```

The modifier only affects codecs: `Schema#toDynamicValue` still includes the field. Besides JSON, the TOON, BSON and SQL (`DbCodec`) codecs honour it; the other format modules ignore it. For a field that should be read but never written, use `@Modifier.encodeTransient()` instead.

### Field Renaming

**Before (ZIO Schema 1.x):** the `@fieldName` annotation:

```scala

final case class Product(@fieldName("product_name") name: String, price: Double)
```

**After (ZIO Blocks Schema):** the `@Modifier.rename` annotation:

```scala

final case class Product(@Modifier.rename("product_name") name: String, price: Double)
object Product {
  implicit val schema: Schema[Product] = Schema.derived[Product]
}
```

The JSON codec writes the new name:

```scala
println(Product.schema.jsonCodec.encodeToString(Product("Widget", 9.99)))
// {"product_name":"Widget","price":9.99}
```

### Field Aliases (for Decoding)

**Before (ZIO Schema 1.x):** the `@fieldNameAliases` annotation:

```scala

final case class Config(@fieldNameAliases("max-size", "max_size") maxSize: Int)
```

**After (ZIO Blocks Schema):** the `@Modifier.alias` annotation, one alias per annotation:

```scala

final case class Config(
  @Modifier.alias("max-size")
  @Modifier.alias("max_size")
  maxSize: Int
)
object Config {
  implicit val schema: Schema[Config] = Schema.derived[Config]
}
```

The decoder accepts every alias, as well as the field's own name:

```scala
println(Config.schema.jsonCodec.decode("""{"max-size":10}"""))
// Right(Config(10))
println(Config.schema.jsonCodec.decode("""{"max_size":20}"""))
// Right(Config(20))
println(Config.schema.jsonCodec.decode("""{"maxSize":30}"""))
// Right(Config(30))
```

### Codec-Specific Configuration

ZIO Schema 1.x has no generic key-value annotation: its core annotations, such as `@fieldDefaultValue` and `@optionalField`, have fixed meanings for every codec. ZIO Blocks Schema has one, `@Modifier.config(key, value)`, with keys that follow the convention `format.property`:

```scala

final case class Message(
  @Modifier.config("protobuf.field-id", "1") id: Long,
  @Modifier.config("protobuf.field-id", "2") content: String
)
object Message {
  implicit val schema: Schema[Message] = Schema.derived[Message]
}
```

Among the built-in codecs, only the XML codec (`xml.attribute`, `xml.namespace.uri` and `xml.namespace.prefix`) and the SQL module (`sql.table_name`, `sql.inline_fields` and `sql.inline`) read `Modifier.config` keys; the others ignore them. For any other key, the modifier carries settings for a deriver that you write (see [Migrating Type Class Derivation](#migrating-type-class-derivation)), which finds them on the schema's fields:

```scala
Message.schema.reflect.asRecord.get.fields.foreach { field =>
  println(s"${field.name}: ${field.modifiers}")
}
// id: ArraySeq(config(protobuf.field-id,1))
// content: ArraySeq(config(protobuf.field-id,2))
```

### Discriminator and Case Name Annotations

**Before (ZIO Schema 1.x):**

```scala

@discriminatorName("type")
sealed trait Event
@caseName("user_created")
case class UserCreated(userId: String) extends Event
```

**After (ZIO Blocks Schema):** `Modifier.discriminator` on the sealed trait and `Modifier.rename` on the case:

```scala

@Modifier.discriminator("type")
sealed trait Event
@Modifier.rename("user_created")
case class UserCreated(userId: String) extends Event

object Event {
  implicit val schema: Schema[Event] = Schema.derived[Event]
}
```

The JSON codec writes the case name into the `type` field:

```scala
println(Event.schema.jsonCodec.encodeToString(UserCreated("u-42")))
// {"type":"user_created","userId":"u-42"}
```

### Programmatic Annotation

ZIO Schema allows adding annotations at any time via `schema.annotate(annotation)`. In ZIO Blocks, you add modifiers with `Schema#modifier`.

**Before (ZIO Schema 1.x):**

```scala
val schema2 = schema.annotate(someAnnotation)
```

**After (ZIO Blocks Schema):**

```scala
val schema2 = Event.schema.modifier(Modifier.config("key", "value"))
```

The new schema carries the modifier on its root node:

```scala
println(schema2.reflect.modifiers)
// ArraySeq(discriminator(type), config(key,value))
```

`Schema#modifier` takes a `Modifier.Reflect`, a modifier of the type itself (`Modifier.config`, `Modifier.discriminator`, `Modifier.noExtraFields`, `Modifier.fieldNaming` or `Modifier.caseNaming`). Field-level modifiers such as `Modifier.rename` go on the fields, as annotations.

---

## Migrating Codec Derivation

Codecs are where the two libraries differ most at the call site. This section covers how codecs are obtained, how they read and write data, and what replaces the JSON module and the streaming codecs.

### The Unified Format Model

ZIO Schema has separate codec APIs in each codec sub-module (e.g., `JsonCodec.jsonCodec`, `ProtobufCodec.protobufCodec`). ZIO Blocks Schema introduces a unified `codec.Format` interface that all codec modules implement, and codecs are derived via a consistent call.

**Before (ZIO Schema 1.x):** each codec module has its own factory:

```scala
// JsonCodec.jsonCodec returns a zio.json.JsonCodec (a text codec from the zio-json library)

val jsonCodec = JsonCodec.jsonCodec(Person.schema)

// ProtobufCodec.protobufCodec returns a BinaryCodec[A] (Chunk[Byte] in / out)

val protoCodec: BinaryCodec[Person] = ProtobufCodec.protobufCodec(Person.schema)
```

**After (ZIO Blocks Schema):** all codecs come from `Schema#derive(format)`:

```scala

final case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

val jsonCodec: JsonCodec[Person] = Person.schema.derive(JsonFormat)
val avroCodec: AvroCodec[Person] = Person.schema.derive(AvroFormat)
```

Each call to `Schema#derive` derives a new codec, so keep the result, for example in an implicit `val`. `Schema#getInstance(format)` derives the codec on first use and caches it in the schema, and `Schema#jsonCodec` is the cached JSON codec:

```scala
Person.schema.derive(JsonFormat) eq Person.schema.derive(JsonFormat)
// res39: Boolean = false
Person.schema.getInstance(JsonFormat) eq Person.schema.getInstance(JsonFormat)
// res40: Boolean = true
Person.schema.jsonCodec eq Person.schema.getInstance(JsonFormat)
// res41: Boolean = true
```

### Encoding and Decoding

The codec interface differs significantly between the two libraries.

**Before (ZIO Schema 1.x):**

```scala
// Protobuf (true BinaryCodec: Chunk[Byte] in/out)

val codec: BinaryCodec[Person] = ProtobufCodec.protobufCodec(Person.schema)
val encoded: Chunk[Byte]              = codec.encode(Person("Alice", 30))
val decoded: Either[DecodeError, Person] = codec.decode(encoded)

// JSON (zio-json JsonCodec: String in/out)

val jsonCodec = JsonCodec.jsonCodec(Person.schema)
val json: String                       = jsonCodec.encodeJson(Person("Alice", 30), None).toString
val fromJson: Either[String, Person]   = jsonCodec.decodeJson(json)
```

**After (ZIO Blocks Schema):** `Schema#encode` and `Schema#decode` work with every format. Binary formats, which include JSON, write to and read from a `java.nio.ByteBuffer`; text formats such as CSV use a `java.nio.CharBuffer`:

```scala

val person = Person("Alice", 30)

val buffer = ByteBuffer.allocate(1024)
Person.schema.encode(JsonFormat)(buffer)(person)
buffer.flip()
```

The buffer now holds the JSON bytes, and decoding reads them back:

```scala
println(StandardCharsets.UTF_8.decode(buffer.duplicate()).toString)
// {"name":"Alice","age":30}
println(Person.schema.decode(JsonFormat)(buffer))
// Right(Person(Alice,30))
```

The JSON codec also works with strings and byte arrays directly, which is the closest match to zio-json's `JsonEncoder#encodeJson` and `JsonDecoder#decodeJson`:

```scala
println(jsonCodec.encodeToString(person))
// {"name":"Alice","age":30}
println(jsonCodec.decode("""{"name":"Bob","age":31}"""))
// Right(Person(Bob,31))
println(jsonCodec.decode("""{"name":"Bob"}"""))
// Left(zio.blocks.schema.SchemaError: missing required field "age" at: .)
```

Decoding errors are `SchemaError` values instead of `DecodeError` or `String`. ZIO Blocks codecs do not depend on `zio-json` or any other external codec library.

### JSON Codec

JSON support is built into the core `zio-blocks-schema` module — no separate dependency is needed.

**Before (ZIO Schema 1.x):** a separate `zio-schema-json` module:

```scala
libraryDependencies += "dev.zio" %% "zio-schema-json" % "1.x.x"

val codec = JsonCodec.jsonCodec(Person.schema)  // returns zio.json.JsonCodec
```

**After (ZIO Blocks Schema):** built into `zio-blocks-schema`; no extra dependency:

```scala

val codec = Person.schema.derive(JsonFormat) // returns JsonCodec[Person]
```

To change field naming, discriminators or other JSON settings for a whole codec, derive it from a configured `JsonCodecDeriver`, as [How to Customize JSON Encoding](./customize-json-encoding.md) shows.

### Streaming Codecs

ZIO Schema's streaming codec methods (`streamEncoder`, `streamDecoder`) are `ZPipeline`s that integrate with `ZStream`. ZIO Blocks Schema codecs encode and decode one value at a time, and they do not depend on ZIO's streaming primitives. The JSON codec can read a value from a `java.io.InputStream` and write one to a `java.io.OutputStream` without collecting it into an array first (see [How to Tune JSON Throughput](./tune-json-throughput.md)). If you need a stream of values, wrap the codec in your effect system's streaming abstraction.

---

## Migrating DynamicValue

`DynamicValue` is the untyped value tree in both libraries, but its cases and its API changed. The [DynamicValue reference](../reference/schema/dynamic-value.md) documents the ZIO Blocks version in full.

### Structure Changes

The `DynamicValue` ADT is significantly simplified in ZIO Blocks — from 15 cases down to 6. The key differences:

- ZIO Schema's `DynamicValue.Primitive[A](value: A, standardType: StandardType[A])` stores the raw value and its `StandardType` inline. ZIO Blocks wraps the scalar in a `PrimitiveValue` case class instead.
- `DynamicValue.Record` drops the `TypeId` parameter and uses `Chunk[(String, DynamicValue)]` instead of `ListMap[String, DynamicValue]`.
- `Option`, `Either`, and `Tuple` are no longer dedicated ADT cases — they are represented structurally using `Variant` and `Record`.

| ZIO Schema                                                  | ZIO Blocks Schema                                                   |
|-------------------------------------------------------------|---------------------------------------------------------------------|
| `Primitive[A](value: A, standardType: StandardType[A])`     | `Primitive(value: PrimitiveValue)`                                  |
| `Record(id: TypeId, values: ListMap[String, DynamicValue])` | `Record(fields: Chunk[(String, DynamicValue)])` — no TypeId         |
| `Enumeration(id: TypeId, value: (String, DynamicValue))`    | `Variant(caseNameValue: String, value: DynamicValue)`               |
| `Sequence(values: Chunk[DynamicValue])`                     | `Sequence(elements: Chunk[DynamicValue])`                           |
| `Dictionary(entries: Chunk[(DynamicValue, DynamicValue)])`  | `Map(entries: Chunk[(DynamicValue, DynamicValue)])`                 |
| `SomeValue(value: DynamicValue)`                            | `Variant("Some", Record(Chunk("value" -> ...)))`                    |
| `NoneValue`                                                 | `Variant("None", Record(Chunk.empty))`                              |
| `LeftValue(value: DynamicValue)`                            | `Variant("Left", Record(Chunk("value" -> ...)))`                    |
| `RightValue(value: DynamicValue)`                           | `Variant("Right", Record(Chunk("value" -> ...)))`                   |
| `Tuple(left, right)`                                        | `Record(Chunk("_1" -> left, "_2" -> right))`                        |
| `SetValue(values: Set[DynamicValue])`                       | `Sequence(elements: Chunk[DynamicValue])`                           |
| `BothValue(left, right)`                                    | No direct equivalent (used by `Fallback`, which ZIO Blocks lacks)   |
| `DynamicAst(ast: MetaSchema)`                               | No direct equivalent                                                |
| `Singleton[A](instance: A)`                                 | No direct equivalent                                                |
| `Error(message: String)`                                    | No direct equivalent — use `SchemaError`                            |

The sixth ZIO Blocks case, `DynamicValue.Null`, stands for a missing value and has no ZIO Schema counterpart. To see the `Option`, `Either` and tuple encodings, convert a few values and print their structure:

```scala

val some  = Schema[Option[Int]].toDynamicValue(Some(1))
val none  = Schema[Option[Int]].toDynamicValue(None)
val left  = Schema[Either[String, Int]].toDynamicValue(Left("oops"))
val tuple = Schema.derived[(String, Int)].toDynamicValue(("Alice", 30))
```

Tuples have no implicit schema in ZIO Blocks, so the tuple's schema comes from `Schema.derived`.

The `@ {tag: ...}` suffix marks a variant case in the printed form:

```scala
println(some)
// {
//   value: 1
// } @ {tag: "Some"}
println(none)
// {} @ {tag: "None"}
println(left)
// {
//   value: "oops"
// } @ {tag: "Left"}
println(tuple)
// {
//   _1: "Alice",
//   _2: 30
// }
```

### Primitive Values

In ZIO Schema, primitive values are stored inline in `DynamicValue.Primitive[A](value: A, standardType: StandardType[A])`. There is no separate `PrimitiveValue` type. In ZIO Blocks, a sealed `PrimitiveValue` ADT wraps each primitive.

**Before (ZIO Schema 1.x):** value and `StandardType` are separate constructor arguments:

```scala

val pv: DynamicValue = DynamicValue.Primitive(42, StandardType[Int])
val ps: DynamicValue = DynamicValue.Primitive("hello", StandardType[String])
```

**After (ZIO Blocks Schema):** the value is wrapped in a `PrimitiveValue` case class:

```scala

val pv: DynamicValue = DynamicValue.Primitive(PrimitiveValue.Int(42))
val ps: DynamicValue = DynamicValue.Primitive(PrimitiveValue.String("hello"))
```

The `PrimitiveValue` case names (`Int`, `Long`, `String`, `Boolean`, `Double`, etc.) match the Scala primitive names.

### Converting Between Typed Values and DynamicValue

**Before (ZIO Schema 1.x):**

```scala
// toDynamic is a method on Schema[A], not on the value itself
val dv: DynamicValue         = Person.schema.toDynamic(person)
// toTypedValue requires an implicit Schema[Person] in scope
val back: Either[String, Person] = dv.toTypedValue[Person]
// or, on the schema
val back2: Either[String, Person] = Person.schema.fromDynamic(dv)
```

**After (ZIO Blocks Schema):**

```scala

final case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

val person = Person("Alice", 30)

val dv: DynamicValue                  = Person.schema.toDynamicValue(person)
val back: Either[SchemaError, Person] = Person.schema.fromDynamicValue(dv)
```

The value goes to a `DynamicValue` and comes back unchanged:

```scala
println(dv)
// {
//   name: "Alice",
//   age: 30
// }
println(back)
// Right(Person(Alice,30))
```

Two things change: `Schema#toDynamic` is renamed `Schema#toDynamicValue`, and `DynamicValue#toTypedValue` and `Schema#fromDynamic` are replaced by `Schema#fromDynamicValue`. The error type changes from `String` to `SchemaError`.

### DynamicValue Operations

ZIO Blocks `DynamicValue` has a rich operation API that was absent in ZIO Schema. Where ZIO Schema required you to convert back to a typed value to manipulate data, you can now operate directly on `DynamicValue`:

```scala

val record = DynamicValue.Record(
  Chunk(
    "name" -> DynamicValue.Primitive(PrimitiveValue.String("Alice")),
    "age"  -> DynamicValue.Primitive(PrimitiveValue.Int(30))
  )
)

val other = DynamicValue.Record(
  Chunk(
    "name" -> DynamicValue.Primitive(PrimitiveValue.String("Bob")),
    "age"  -> DynamicValue.Primitive(PrimitiveValue.Int(31))
  )
)
```

`DynamicValue#get` navigates and returns a `DynamicValueSelection`, which supports chaining; `DynamicValueSelection#one` extracts a single value as an `Either[SchemaError, DynamicValue]`. `DynamicValue#set` replaces the value at a path and returns the new `DynamicValue` (it leaves the value unchanged when the path is not found; `DynamicValue#setOrFail` returns an `Either` instead). `DynamicValue#diff` computes a `DynamicPatch` between two values:

```scala
println(record.get("name").one)
// Right("Alice")

println(record.set(DynamicOptic.root.field("name"), DynamicValue.Primitive(PrimitiveValue.String("Bob"))))
// {
//   name: "Bob",
//   age: 30
// }

println(record.diff(other))
// DynamicPatch {
//   .name = "Bob"
//   .age += 1
// }
```

---

## Migrating Schema Introspection

ZIO Schema describes a schema as data with `MetaSchema`. ZIO Blocks has `DynamicSchema` for the same job, and it can be stored, loaded and turned back into a typed schema.

### MetaSchema → DynamicSchema

ZIO Schema has `MetaSchema` (a type-erased structural description of a schema) and `schema.ast` to convert to it. ZIO Blocks uses `DynamicSchema` for the same purpose.

**Before (ZIO Schema 1.x):**

```scala
val meta: MetaSchema = schema.ast
val back: Schema[_]  = meta.toSchema
```

**After (ZIO Blocks Schema):** `Schema#toDynamicSchema` goes one way, and `DynamicSchema#rebind` goes back to a typed schema, taking the bindings (the functions that build and take apart values) from a `BindingResolver`:

```scala

final case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

val dynSchema: DynamicSchema = Person.schema.toDynamicSchema

val resolver             = BindingResolver.empty.bind(Binding.of[Person]) ++ BindingResolver.defaults
val rebound: Schema[Person] = dynSchema.rebind[Person](resolver)
```

`DynamicSchema` wraps a `Reflect[NoBinding, _]` — the full structural description without runtime constructors or deconstructors. Use it for:

- Runtime structural validation of `DynamicValue` instances
- Dynamic schema loading from configuration or network
- Schema inspection without compile-time type information

To check data against the structure, we need one value that matches it and one that misses a field:

```scala

val value = DynamicValue.Record(
  Chunk(
    "name" -> DynamicValue.Primitive(PrimitiveValue.String("Alice")),
    "age"  -> DynamicValue.Primitive(PrimitiveValue.Int(30))
  )
)
val incomplete = DynamicValue.Record(
  Chunk("name" -> DynamicValue.Primitive(PrimitiveValue.String("Alice")))
)
```

`DynamicSchema#check` returns the first mismatch as an `Option[SchemaError]`, and `DynamicSchema#conforms` returns a `Boolean`:

```scala
dynSchema.conforms(value)
// res63: Boolean = true
dynSchema.check(value)
// res64: Option[SchemaError] = None
println(dynSchema.check(incomplete))
// Some(zio.blocks.schema.SchemaError: Missing field 'age' at: .)
```

:::info
ZIO Blocks Schema includes an explicit migration API for evolving values between schema versions. The entry point is [`Migration.newBuilder[A, B]`](../reference/schema/migration.md), which builds a typed `Migration[A, B]` backed by a `DynamicMigration`. In 0.0.56, a `DynamicMigration` has no `Schema`, so it can't be stored or sent.

This is a different model from ZIO Schema's `schema.migrate(newSchema)` and `schema.coerce(newSchema)` methods, which work out the conversion from the two schemas at runtime:

1. Build a migration explicitly with operations like `MigrationBuilder#addField`, `MigrationBuilder#dropField`, `MigrationBuilder#renameField`, `MigrationBuilder#changeFieldType`, and `MigrationBuilder#transformField`.
2. Apply the resulting `Migration[A, B]` to typed values, or inspect the underlying `DynamicMigration`.

Use this when you want structural schema evolution as first-class data rather than implicit derivation. To convert between two versions of a type without writing the steps, derive an `Into[A, B]` instead. [Evolve Your Data](./evolve-your-data.md) walks through both.
:::

### Schema Serialization

ZIO Schema supports serializing a schema itself (via `schema.serializable`, a `Schema[Schema[A]]`). In ZIO Blocks, the serializable form of a schema is its `DynamicSchema`, and `DynamicSchema.schema` is a `Schema[DynamicSchema]`. In 0.0.56, only its Avro codec reads a stored schema back intact: through JSON, YAML, TOON or MessagePack it comes back as a schema that accepts any value (a known bug). The Avro codec comes from the `zio-blocks-schema-avro` module. A loaded `DynamicSchema` checks values directly, or turns into a `Schema[A]` with `DynamicSchema#rebind`:

```scala

val dynamicSchemaCodec  = DynamicSchema.schema.derive(AvroFormat)
val stored: Array[Byte] = dynamicSchemaCodec.encode(dynSchema)
val loaded              = dynamicSchemaCodec.decode(stored)
```

The schema loaded back from Avro bytes accepts the complete value and rejects the incomplete one:

```scala
println(loaded.map(_.conforms(value)))
// Right(true)
println(loaded.map(_.conforms(incomplete)))
// Right(false)
```

The stored form keeps field names, cases, type ids, modifiers, docs and `Validation` constraints, but not the bindings or the default values of case class fields. A schema of a recursive type cannot be stored this way in 0.0.56 (a known bug). [How to Build a Schema Registry](./build-a-schema-registry.md) builds a registry on this API.

---

## Migrating Optics

ZIO Schema uses an `AccessorBuilder` pattern that delegates optic creation to an external `zio-schema-optics` module. ZIO Blocks Schema includes a complete, first-class optics system in the core module.

### Generating Optics

**Before (ZIO Schema 1.x with `zio-schema-optics`):**

```scala

case class Person(name: String, age: Int)
object Person {
  // makeAccessors needs the concrete CaseClass2 type to give typed lenses
  implicit val schema: Schema.CaseClass2[String, Int, Person] =
    DeriveSchema.gen[Person].asInstanceOf[Schema.CaseClass2[String, Int, Person]]
  val (name, age) = schema.makeAccessors(ZioOpticsBuilder)
}
```

**After (ZIO Blocks Schema):** extend `CompanionOptics[Person]` in the companion object and define each optic with its `optic` macro (also available as `$`). It works the same in Scala 2 and Scala 3, and the compiler checks the field path:

```scala

case class Person(name: String, age: Int)
object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived[Person]

  val name: Lens[Person, String] = optic(_.name)
  val age: Lens[Person, Int]     = optic(_.age)
}
```

To get a lens for every field without writing them, extend `DerivedOptics` instead: its `DerivedOptics#optics` member holds one lens per field, named after the field. In 0.0.56 this works on Scala 3 only; on Scala 2, `DerivedOptics#optics` compiles only inside the `zio.blocks.schema` package (a known bug):

```scala
// Scala 3 only in 0.0.56

case class Customer(name: String, email: String)
object Customer extends DerivedOptics {
  implicit val schema: Schema[Customer] = Schema.derived[Customer]
}

val email: Lens[Customer, String] = Customer.optics.email
```

You can also look a lens up by field name at runtime, from the schema's record node. The name and the type are not checked at compile time, so `Reflect.Record#lensByName` returns an `Option`:

```scala
val nameLens: Option[Lens[Person, String]] =
  Person.schema.reflect.asRecord.flatMap(_.lensByName[String]("name"))
```

[Work with Nested Data Using Optics](./nested-data-with-optics.md) teaches optics step by step.

### Using Optics

The four optic types in ZIO Blocks are `Lens`, `Prism`, `Optional`, and `Traversal`. Their usage API is similar to standard optics libraries, with one naming difference: ZIO Blocks uses `Lens#replace` where Monocle uses `Lens#set`.

To read, modify and replace a field with the lenses from the previous section:

```scala
val person = Person("Alice", 30)
// person: Person = Person(name = "Alice", age = 30)

Person.name.get(person)
// res71: String = "Alice"
Person.name.modify(person, _.toUpperCase)
// res72: Person = Person(name = "ALICE", age = 30)
Person.name.replace(person, "Bob")
// res73: Person = Person(name = "Bob", age = 30)
```

For sealed traits, use `Prism`. `Reflect.Variant#prismByName` looks one up by case name:

```scala

sealed trait Shape
case class Circle(radius: Double)          extends Shape
case class Rectangle(w: Double, h: Double) extends Shape

object Shape {
  implicit val schema: Schema[Shape] = Schema.derived[Shape]

  val circlePrism: Prism[Shape, Circle] =
    schema.reflect.asVariant.get.prismByName[Circle]("Circle").get
}
```

`Prism#getOption` returns the case when the value is one, and `Prism#reverseGet` widens a case back to the sealed trait:

```scala
val shape: Shape = Circle(5.0)
// shape: Shape = Circle(5.0)

Shape.circlePrism.getOption(shape)
// res75: Option[Circle] = Some(Circle(5.0))
Shape.circlePrism.getOption(Rectangle(1.0, 2.0))
// res76: Option[Circle] = None
Shape.circlePrism.reverseGet(Circle(3.0))
// res77: Shape = Circle(3.0)
```

The [Optics reference](../reference/schema/optics.md) covers `Optional`, `Traversal` and optic composition.

### Schema Expressions (New in ZIO Blocks)

ZIO Blocks introduces `SchemaExpr[S, A]`, a typed expression language built on top of optics. There is no equivalent in ZIO Schema. These allow you to build inspectable, composable predicates and computations:

```scala

case class Product(name: String, price: Double, inStock: Boolean)
object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived[Product]

  val price: Lens[Product, Double]    = optic(_.price)
  val inStock: Lens[Product, Boolean] = optic(_.inStock)
}

// Build a typed predicate expression
val cheapAndInStock: SchemaExpr[Product, Boolean] =
  (Product.price < 100.0) && (Product.inStock === true)
```

`SchemaExpr#eval` returns `Right` with the results on success, and `Left(OpticCheck)` when a prism in the path does not match:

```scala
cheapAndInStock.eval(Product("Widget", 49.99, inStock = true))
// res79: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(true))
cheapAndInStock.eval(Product("Gadget", 149.99, inStock = true))
// res80: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(false))
```

The [SchemaExpr reference](../reference/schema/schema-expr.md) lists the operators.

---

## Migrating Diff and Patch

Both libraries compute a patch from two values and apply it to a third. The call sites stay the same, while the patch type and the error type change. [How to Diff and Patch Values](./diff-and-patch.md) covers the ZIO Blocks API in depth.

### Diff

ZIO Schema uses `Differ.fromSchema(schema)(a, b)` or the convenience method `schema.diff(a, b)`. ZIO Blocks Schema uses the same convenience method.

**Before (ZIO Schema 1.x):**

```scala
val patch: Patch[Person] = Person.schema.diff(person1, person2)
```

**After (ZIO Blocks Schema):**

```scala

case class Person(name: String, age: Int)
object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived[Person]

  val name: Lens[Person, String] = optic(_.name)
  val age: Lens[Person, Int]     = optic(_.age)
}

val person1 = Person("Alice", 30)
val person2 = Person("Bob", 31)

val patch: Patch[Person] = Person.schema.diff(person1, person2)
```

The printed patch lists one operation per changed field:

```scala
println(patch)
// DynamicPatch {
//   .name = "Bob"
//   .age += 1
// }
```

The call site is identical, but the underlying `Patch` types are different.

### Patch Application

**Before (ZIO Schema 1.x):**

```scala
val result: Either[String, Person] = Person.schema.patch(person, patch)
```

**After (ZIO Blocks Schema):** `Schema#patch`, or equivalently `Patch#apply` with `PatchMode.Strict`:

```scala
Person.schema.patch(person1, patch)
// res83: Either[SchemaError, Person] = Right(Person(name = "Bob", age = 31))
patch.apply(person1, PatchMode.Strict)
// res84: Either[SchemaError, Person] = Right(Person(name = "Bob", age = 31))
```

The error type changes from `String` to `SchemaError`.

### Creating Patches Programmatically

ZIO Schema has no optic-based patch constructors: to build a patch by hand, you assemble the cases of its `Patch` ADT, such as `Patch.Record` and `Patch.Total`. ZIO Blocks Schema provides `Patch` smart constructors that take an optic. `Patch.set` sets a field through a lens, and `Patch#++` composes two patches:

```scala
val renamePatch: Patch[Person] = Patch.set(Person.name, "Bob")
val combined: Patch[Person]    = renamePatch ++ Patch.set(Person.age, 31)
```

Applying the combined patch sets both fields:

```scala
combined(person1, PatchMode.Strict)
// res85: Either[SchemaError, Person] = Right(Person(name = "Bob", age = 31))
```

---

## Migrating Type Class Derivation

Both libraries can derive a custom type class from a schema. The interface keeps its name, `Deriver`, but what it receives changes.

### Before (ZIO Schema 1.x)

ZIO Schema 1.x has `zio.schema.Deriver[F[_]]` in the `zio-schema-derivation` module. A deriver implements `deriveRecord`, `deriveEnum`, `derivePrimitive`, `deriveOption`, `deriveSequence`, `deriveMap` and `deriveTransformedRecord`, receives the `Schema` nodes of the type, and runs with `Derive.derive[F, A](deriver)`. ZIO Schema's own codecs do not use it: each codec module walks the schema with its own code.

### After (ZIO Blocks Schema)

ZIO Blocks Schema's `Deriver[TC]` plays the same role, but it receives `Reflect` nodes and their bindings instead of `Schema` nodes, and you call it with `Schema#derive(deriver)`. The built-in codecs use it too: `JsonFormat`'s deriver is a `Deriver[JsonCodec]`, so a custom codec and a custom type class are written the same way. A deriver implements one method per kind of `Reflect` node:

```scala
trait Deriver[TC[_]] {
  def derivePrimitive[A](primitiveType: PrimitiveType[A], typeId: TypeId[A], binding: Binding.Primitive[A], doc: Doc, modifiers: Seq[Modifier.Reflect], defaultValue: Option[A], examples: Seq[A]): Lazy[TC[A]]
  def deriveRecord[F[_, _], A](fields: IndexedSeq[Term[F, A, _]], typeId: TypeId[A], binding: Binding.Record[A], doc: Doc, modifiers: Seq[Modifier.Reflect], defaultValue: Option[A], examples: Seq[A])(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[TC[A]]
  def deriveVariant[F[_, _], A](cases: IndexedSeq[Term[F, A, _]], typeId: TypeId[A], binding: Binding.Variant[A], doc: Doc, modifiers: Seq[Modifier.Reflect], defaultValue: Option[A], examples: Seq[A])(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[TC[A]]
  def deriveSequence[F[_, _], C[_], A](element: Reflect[F, A], typeId: TypeId[C[A]], binding: Binding.Seq[C, A], doc: Doc, modifiers: Seq[Modifier.Reflect], defaultValue: Option[C[A]], examples: Seq[C[A]])(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[TC[C[A]]]
  def deriveMap[F[_, _], M[_, _], K, V](key: Reflect[F, K], value: Reflect[F, V], typeId: TypeId[M[K, V]], binding: Binding.Map[M, K, V], doc: Doc, modifiers: Seq[Modifier.Reflect], defaultValue: Option[M[K, V]], examples: Seq[M[K, V]])(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[TC[M[K, V]]]
  def deriveDynamic[F[_, _]](binding: Binding.Dynamic, doc: Doc, modifiers: Seq[Modifier.Reflect], defaultValue: Option[DynamicValue], examples: Seq[DynamicValue])(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[TC[DynamicValue]]
  def deriveWrapper[F[_, _], A, B](wrapped: Reflect[F, B], typeId: TypeId[A], binding: Binding.Wrapper[A, B], doc: Doc, modifiers: Seq[Modifier.Reflect], defaultValue: Option[A], examples: Seq[A])(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[TC[A]]
}
```

The companion example `DeriveShowExample` implements all seven methods for a `Show` type class, then derives `Show` for records, variants, recursive types, `DynamicValue` and a wrapper with `Schema#derive(DeriveShow)`. For `Person("Alice", 30)` it prints `Person(name = "Alice", age = 30)`:

```scala title="schema-examples/src/main/scala/typeclassderivation/DeriveShowExample.scala"
package typeclassderivation

/**
 * Implementation of Deriver to derive Show type-class instances for various
 * schema types.
 *
 * Implementation pattern (from SchemaDerivationShowSpec):
 *   - Record/Variant (potentially recursive): pre-compute structural setup
 *     (field names, bindings, reflects) outside the Lazy block; inside Lazy,
 *     use `private lazy val resolvedShows` to defer forcing child instances
 *     until first show() call — this breaks initialization cycles for recursive
 *     types like `sealed trait Expr; case class Add(a: Expr, b: Expr)`.
 *   - Sequence/Map/Wrapper (non-recursive): use `instance(...).map` / `.zip`
 *     monadic composition — no explicit `.force` needed.
 *   - Primitive/Dynamic: simple `Lazy { new Show { ... } }`.
 */
object DeriveShowExample extends App {
  trait Show[A] {
    def show(value: A): String
  }

  object DeriveShow extends Deriver[Show] {

    override def derivePrimitive[A](
      primitiveType: PrimitiveType[A],
      typeId: TypeId[A],
      binding: Binding.Primitive[A],
      doc: Doc,
      modifiers: Seq[Modifier.Reflect],
      defaultValue: Option[A],
      examples: Seq[A]
    ): Lazy[Show[A]] =
      Lazy {
        new Show[A] {
          def show(value: A): String = primitiveType match {
            case _: PrimitiveType.String => "\"" + value + "\""
            case _: PrimitiveType.Char   => "'" + value + "'"
            case _                       => String.valueOf(value)
          }
        }
      }

    override def deriveRecord[F[_, _], A](
      fields: IndexedSeq[Term[F, A, _]],
      typeId: TypeId[A],
      binding: Binding.Record[A],
      doc: Doc,
      modifiers: Seq[Modifier.Reflect],
      defaultValue: Option[A],
      examples: Seq[A]
    )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
      // Pre-compute structural setup outside Lazy — none of this touches Deferred nodes
      val fieldNames = fields.map(_.name)
      // Cast fields to use Binding as F (we are going to create Reflect.Record with Binding as F)
      val recordFields = fields.asInstanceOf[IndexedSeq[Term[Binding, A, _]]]
      // Cast to Binding.Record to access constructor/deconstructor
      val recordBinding = binding
      // Build a Reflect.Record to get access to the computed registers for each field
      val recordReflect = new Reflect.Record[Binding, A](recordFields, typeId, recordBinding, doc, modifiers)
      Lazy {
        new Show[A] {
          // Defer child-instance resolution to first show() call via lazy val.
          // For recursive types (e.g. case class Tree(children: List[Tree])), accessing
          // field.value.metadata during derivation would re-enter a Deferred node that is
          // still initialising, causing an infinite loop. By using a lazy val here we
          // guarantee the access only happens after the framework has finished deriving
          // all instances and every Lazy is fully memoised.
          private lazy val resolvedShows: IndexedSeq[Show[Any]] =
            fields.map(field => D.instance(field.value.metadata).asInstanceOf[Lazy[Show[Any]]].force)
          def show(value: A): String = {
            // Create registers with space for all used registers to hold deconstructed field values
            val registers = Registers(recordReflect.usedRegisters)
            // Deconstruct field values of the record into the registers
            recordBinding.deconstructor.deconstruct(registers, RegisterOffset.Zero, value)
            // Build string representations for all fields
            val fieldStrings = fields.indices.map { i =>
              val fieldValue = recordReflect.registers(i).get(registers, RegisterOffset.Zero)
              s"${fieldNames(i)} = ${resolvedShows(i).show(fieldValue)}"
            }
            s"${typeId.name}(${fieldStrings.mkString(", ")})"
          }
        }
      }
    }

    override def deriveVariant[F[_, _], A](
      cases: IndexedSeq[Term[F, A, _]],
      typeId: TypeId[A],
      binding: Binding.Variant[A],
      doc: Doc,
      modifiers: Seq[Modifier.Reflect],
      defaultValue: Option[A],
      examples: Seq[A]
    )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
      // Collect the Lazy[Show] references for each case outside the Lazy block.
      // Capturing Lazy refs is safe here; we only .force them later inside the lazy val.
      val caseShowLazies = cases.map(c => D.instance(c.value.metadata).asInstanceOf[Lazy[Show[Any]]])

      // Cast binding to Binding.Variant to access discriminator and matchers
      val variantBinding = binding
      Lazy {
        new Show[A] {
          // Force child instances lazily — same recursive-safety rationale as deriveRecord
          private lazy val resolvedShows: IndexedSeq[Show[Any]] = caseShowLazies.map(_.force)
          // Implement show by using discriminator and matchers to find the right case
          // The `value` parameter is of type A (the variant type), e.g. a Shape value
          def show(value: A): String = {
            // Use discriminator to determine which case this value belongs to
            val caseIndex = variantBinding.discriminator.discriminate(value)
            // Use matcher to downcast to the specific case type
            val caseValue = variantBinding.matchers(caseIndex).downcastOrNull(value)
            // Delegate to the case's Show instance — it already knows its own name
            resolvedShows(caseIndex).show(caseValue)
          }
        }
      }
    }

    override def deriveSequence[F[_, _], C[_], A](
      element: Reflect[F, A],
      typeId: TypeId[C[A]],
      binding: Binding.Seq[C, A],
      doc: Doc,
      modifiers: Seq[Modifier.Reflect],
      defaultValue: Option[C[A]],
      examples: Seq[C[A]]
    )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[C[A]]] = {
      // Cast binding to Binding.Seq to access the deconstructor
      val deconstructor = binding.deconstructor
      // Sequences are structurally non-recursive, so we can use monadic .map composition.
      // instance(...).map { elementShow => ... } returns a Lazy that, when forced, builds
      // a Show[C[A]] with elementShow already resolved — no .force needed at show()-time.
      D.instance(element.metadata).map { elementShow =>
        new Show[C[A]] {
          def show(value: C[A]): String = {
            // Use deconstructor to iterate over elements and show each one
            val elements = deconstructor.deconstruct(value).map(elementShow.show)
            s"[${elements.mkString(", ")}]"
          }
        }
      }
    }

    override def deriveMap[F[_, _], M[_, _], K, V](
      key: Reflect[F, K],
      value: Reflect[F, V],
      typeId: TypeId[M[K, V]],
      binding: Binding.Map[M, K, V],
      doc: Doc,
      modifiers: Seq[Modifier.Reflect],
      defaultValue: Option[M[K, V]],
      examples: Seq[M[K, V]]
    )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[M[K, V]]] = {
      // Cast binding to Binding.Map to access the deconstructor
      val deconstructor = binding.deconstructor
      // Maps are non-recursive: use .zip to pair the two child Lazy instances, then .map
      // to build the Show[M[K,V]] with both keyShow and valueShow already resolved.
      D.instance(key.metadata).zip(D.instance(value.metadata)).map { case (keyShow, valueShow) =>
        new Show[M[K, V]] {
          def show(m: M[K, V]): String = {
            // Use deconstructor to iterate over key-value pairs
            val entries = deconstructor
              .deconstruct(m)
              .map { kv =>
                val k = deconstructor.getKey(kv)
                val v = deconstructor.getValue(kv)
                s"${keyShow.show(k)} -> ${valueShow.show(v)}"
              }
              .mkString(", ")
            s"Map($entries)"
          }
        }
      }
    }

    override def deriveDynamic[F[_, _]](
      binding: Binding.Dynamic,
      doc: Doc,
      modifiers: Seq[Modifier.Reflect],
      defaultValue: Option[DynamicValue],
      examples: Seq[DynamicValue]
    )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[DynamicValue]] = Lazy {
      new Show[DynamicValue] {
        def show(value: DynamicValue): String =
          value match {
            case DynamicValue.Primitive(_) =>
              value.toString

            case DynamicValue.Record(fields) =>
              val fieldStrings = fields.map { case (name, v) =>
                s"$name = ${show(v)}"
              }
              s"Record(${fieldStrings.mkString(", ")})"

            case DynamicValue.Variant(caseName, v) =>
              s"$caseName(${show(v)})"

            case DynamicValue.Sequence(elements) =>
              val elemStrings = elements.map(show)
              s"[${elemStrings.mkString(", ")}]"

            case DynamicValue.Map(entries) =>
              val entryStrings = entries.map { case (k, v) =>
                s"${show(k)} -> ${show(v)}"
              }
              s"Map(${entryStrings.mkString(", ")})"
            case Null =>
              "null"
          }
      }
    }

    override def deriveWrapper[F[_, _], A, B](
      wrapped: Reflect[F, B],
      typeId: TypeId[A],
      binding: Binding.Wrapper[A, B],
      doc: Doc,
      modifiers: Seq[Modifier.Reflect],
      defaultValue: Option[A],
      examples: Seq[A]
    )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
      // Cast binding to Binding.Wrapper to access the unwrap function
      val wrapperBinding = binding
      // Wrappers are non-recursive: use .map so wrappedShow is already resolved
      // when show() is called — no .force needed.
      D.instance(wrapped.metadata).map { wrappedShow =>
        new Show[A] {
          def show(value: A): String = {
            // Unwrap the value to access the underlying type B, then delegate to its Show
            val unwrapped = wrapperBinding.unwrap(value)
            s"${typeId.name}(${wrappedShow.show(unwrapped)})"
          }
        }
      }
    }
  }

  // ===========================================
  // Examples
  // ===========================================

  def printHeader(title: String): Unit = {
    println()
    println("=" * 80)
    println(title)
    println("=" * 80)
  }

  /**
   * Example 1: Simple Person Record with two primitive fields
   */
  printHeader("Example 1: Simple Person Record with two primitive fields")

  case class Person(name: String, age: Int)

  object Person {
    implicit val schema: Schema[Person] = Schema.derived[Person]
    implicit val show: Show[Person]     = schema.derive(DeriveShow)
  }

  println(Person.show.show(Person("Alice", 30)))

  /**
   * Example 2: Simple Shape Variant (Circle, Rectangle)
   */
  printHeader("Example 2: Simple Shape Variant (Circle, Rectangle)")
  sealed trait Shape
  case class Circle(radius: Double)                   extends Shape
  case class Rectangle(width: Double, height: Double) extends Shape

  object Shape {
    implicit val schema: Schema[Shape] = Schema.derived[Shape]
    implicit val show: Show[Shape]     = schema.derive(DeriveShow)
  }

  val shape1: Shape = Circle(5.0)
  val shape2: Shape = Rectangle(4.0, 6.0)
  println(Shape.show.show(shape1))
  println(Shape.show.show(shape2))

  /**
   * Example 3: Recursive Tree and Expr
   */
  printHeader("Example 3: Recursive Tree")
  case class Tree(value: Int, children: List[Tree])
  object Tree {
    implicit val schema: Schema[Tree] = Schema.derived[Tree]
    implicit val show: Show[Tree]     = schema.derive(DeriveShow)
  }

  val tree = Tree(1, List(Tree(2, List(Tree(4, Nil))), Tree(3, Nil)))
  println(Tree.show.show(tree))

  /**
   * Example 4: Recursive Sealed Trait (Expr)
   */
  printHeader("Example 4: Recursive Sealed Trait (Expr)")
  sealed trait Expr
  case class Num(n: Int)           extends Expr
  case class Add(a: Expr, b: Expr) extends Expr

  object Expr {
    implicit val schema: Schema[Expr] = Schema.derived[Expr]
    implicit val show: Show[Expr]     = schema.derive(DeriveShow)
  }

  val expr: Expr = Add(Num(1), Add(Num(2), Num(3)))
  println(Expr.show.show(expr))

  /**
   * Example 5: DynamicValue Example
   */
  printHeader("Example 5: DynamicValue Example")
  implicit val dynamicShow: Show[DynamicValue] = Schema.dynamic.derive(DeriveShow)

  val manualRecord = DynamicValue.Record(
    Chunk(
      "id"    -> DynamicValue.Primitive(PrimitiveValue.Int(42)),
      "title" -> DynamicValue.Primitive(PrimitiveValue.String("Hello World")),
      "tags"  -> DynamicValue.Sequence(
        Chunk(
          DynamicValue.Primitive(PrimitiveValue.String("scala")),
          DynamicValue.Primitive(PrimitiveValue.String("zio"))
        )
      )
    )
  )

  println(dynamicShow.show(manualRecord))

  /**
   * Example 6: Simple Email Wrapper Type
   */
  printHeader("Example 6: Simple Email Wrapper Type")
  case class Email(value: String)
  object Email {
    implicit val schema: Schema[Email] = Schema[String].transform(
      Email(_),
      _.value
    )
    implicit val show: Show[Email] = schema.derive(DeriveShow)
  }

  val email = Email("alice@example.com")
  println(s"Email: ${Email.show.show(email)}")

}
```

[How to Derive Your Own Type Class](./derive-your-own-type-class.md) builds a deriver step by step, and the [Type Class Derivation reference](../reference/schema/type-class-derivation.md) documents the interface.

---

## Migrating Validation

Validation moves from an annotation-based combinator library to two smaller mechanisms: constraints on primitive types and checks inside wrapper types.

### Before (ZIO Schema 1.x)

ZIO Schema uses a composable `Validation[A]`, combined with `&&`, `||`, `!`, `allOf`, `anyOf` and `contramap`, and attached to a field with the `@validate(...)` annotation. `Schema.validate` checks the field validations of a value:

```scala

case class User(
  @validate(Validation.greaterThan(0)) age: Int,
  @validate(Validation.minLength(3))   name: String
)
object User {
  implicit val schema: Schema[User] = DeriveSchema.gen
}

Schema.validate(User(-1, "Al"))
// Returns a Chunk[ValidationError] with one error per violation
```

### After (ZIO Blocks Schema)

ZIO Blocks Schema has a simpler, non-composable `Validation[A]` that is embedded inside `PrimitiveType[A]` and checked only by `DynamicSchema#check` (and `DynamicSchema#conforms`), not by `Schema#fromDynamicValue` or the codecs. One primitive carries one constraint, and there are no combinators. A derived schema carries `Validation.None` on every primitive; a constraint comes in when a schema is loaded from JSON Schema with `Schema.fromJsonSchema`, or when you build the primitive node yourself:

```scala

val positive = PrimitiveType.Int(Validation.Numeric.Positive)
val positiveIntSchema: Schema[Int] =
  new Schema(new Reflect.Primitive[Binding, Int](positive, positive.typeId, positive.binding))

val minusOne = DynamicValue.Primitive(PrimitiveValue.Int(-1))
```

`DynamicSchema#check` reports the violated constraint, while `Schema#fromDynamicValue` accepts the same value:

```scala
println(positiveIntSchema.toDynamicSchema.check(minusOne))
// Some(zio.blocks.schema.SchemaError: Validation failed: value must be positive)
println(positiveIntSchema.fromDynamicValue(minusOne))
// Right(-1)
```

For validated types, use `Schema#transform` with a throwing `to` function, which signals failure during `Schema#fromDynamicValue` and decoding:

```scala

final case class Age(value: Int)
object Age {
  implicit val schema: Schema[Age] =
    Schema[Int].transform(
      to = n => if (n > 0) Age(n) else throw SchemaError.validationFailed("Must be positive"),
      from = _.value
    )
}
```

Decoding `-1` now fails with the message thrown in `to`:

```scala
println(Age.schema.fromDynamicValue(DynamicValue.Primitive(PrimitiveValue.Int(-1))))
// Left(zio.blocks.schema.SchemaError: Must be positive at: .wrapped)
```

For validation across several fields, the same rule applies: put the check in the `to` function of a wrapper, or in application-level code that runs after decoding.

:::info
If you rely heavily on ZIO Schema's composable validation (combining validators with `&&`, `||`, `!`, `allOf`, `anyOf` or `contramap`), you will need to implement that logic in the `to` function of a `Schema#transform` wrapper, or in application-level validation code. ZIO Blocks Schema's built-in `Validation` is deliberately simpler: it covers the most common primitive constraints without the complexity of a full combinator library. [How to Define Validated Newtypes](./validated-newtypes.md) shows how to collect every error, and the [Validation reference](../reference/schema/validation.md) lists the built-in constraints.
:::

---

## Migrating the Fail Schema

ZIO Schema provides `Schema.fail[A](message: String)` to represent the absence of schema information.

**Before (ZIO Schema 1.x):**

```scala
val missing: Schema[MyType] = Schema.fail("No schema available for MyType")
```

ZIO Blocks Schema has no equivalent `Fail` schema node. The recommended approach is to leave the implicit schema undefined and let the compiler report the missing instance, or to throw from a type class derivation. If you need a runtime placeholder for data whose shape is not known, use the schema of `DynamicValue`, which accepts any value.

**After (ZIO Blocks Schema):**

```scala

val placeholder: Schema[DynamicValue] = Schema[DynamicValue]
```

---

## Summary of Missing Features

The following ZIO Schema features do not yet have equivalents in ZIO Blocks Schema, or have a different one:

| Feature                                                    | Status                                                                                                                                     |
|------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------|
| `Schema.fail` / fail schemas                               | Not available                                                                                                                              |
| `schema.migrate` / `schema.coerce`                         | Replaced by explicit [`Migration.newBuilder[A, B]`](../reference/schema/migration.md) and `Into[A, B]`                                     |
| `MetaSchema` / schema serialization                        | Replaced by `DynamicSchema`: stored as Avro via `DynamicSchema.schema`, rebound by `DynamicSchema#rebind`; recursive types can't be stored |
| `Fallback[A, B]` schema                                    | Not available                                                                                                                              |
| `NonEmptyChunk` / `NonEmptySet` / `NonEmptyMap` schemas    | Not available — use wrapper types                                                                                                          |
| `Schema.singleton` / singleton schemas                     | Not available                                                                                                                              |
| `DynamicValue.BothValue` / `DynamicValue.DynamicAst`       | Not available                                                                                                                              |
| Composable `Validation` (`&&`, `\|\|`, `!`)                | Not available — use `Schema#transform` with throwing functions                                                                             |
| Streaming codec methods (`streamEncoder`, `streamDecoder`) | Not available — wrap codecs in your effect system                                                                                          |
| ZIO `Chunk` (from `zio-core`)                              | Replaced by `zio.blocks.chunk.Chunk`                                                                                                       |

---

## Running the Examples

All code from this guide is available as runnable examples in the `schema-examples` module, a standalone sbt build that depends on the published ZIO Blocks artifacts.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples
```

**2. Run individual examples with sbt:**

```bash
# Step 1: Schema derivation, primitives, and DynamicValue roundtrip
sbt "runMain ziosschemamigration.Step1SchemaDerivedAndPrimitives"

# Step 2: Modifiers and transform (annotations, newtypes)
sbt "runMain ziosschemamigration.Step2ModifiersAndTransform"

# Step 3: Optics (Lens, Prism) and DynamicSchema validation
sbt "runMain ziosschemamigration.Step3OpticsAndDynamicSchema"

# Step 4: Diff and patch
sbt "runMain ziosschemamigration.Step4DiffAndPatch"

# Complete example: end-to-end e-commerce domain
sbt "runMain ziosschemamigration.CompleteMigrationExample"

# Type class derivation — deriving Show from a Schema
sbt "runMain typeclassderivation.DeriveShowExample"

# Type class derivation — deriving a random generator from a Schema
sbt "runMain typeclassderivation.DeriveGenExample"
```

**3. Or compile all examples at once:**

```bash
sbt compile
```

---

## Going Further

- [Schema Reference](../reference/schema/schema.md) — full `Schema[A]` API
- [Reflect Reference](../reference/schema/reflect.md) — the `Reflect[F, A]` node types
- [Binding Reference](../reference/schema/binding.md) — constructors, deconstructors, and the register system
- [Optics Reference](../reference/schema/optics.md) — `Lens`, `Prism`, `Optional`, `Traversal`
- [Type Class Derivation Reference](../reference/schema/type-class-derivation.md) — implementing `Deriver[TC]`
- [Codec Reference](../reference/schema/codec.md) — the `Format` and `Codec` infrastructure
- [DynamicValue Reference](../reference/schema/dynamic-value.md) — the `DynamicValue` API
- [DynamicSchema Reference](../reference/schema/dynamic-schema.md) — storing, checking and rebinding schemas
- [Validation Reference](../reference/schema/validation.md) — built-in validation constraints
- [Evolve Your Data](./evolve-your-data.md) — `Into`, `As` and `Migration` between versions of a type
- [How to Use Scala 3 Named Tuples with Schema](./use-named-tuples-with-schema.md) — schemas for named tuples

---

## ZIO Blocks

**Modular building blocks for modern Scala applications—no effect system required.**

[![Development](https://img.shields.io/badge/Project%20Stage-Development-green.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-blocks/workflows/CI/badge.svg) [![Maven Central](https://img.shields.io/maven-metadata/v?metadataUrl=https%3A%2F%2Frepo1.maven.org%2Fmaven2%2Fdev%2Fzio%2Fzio-blocks-config_3%2Fmaven-metadata.xml&label=Maven%20Central)](https://central.sonatype.com/artifact/dev.zio/zio-blocks-config_3) [![Sonatype Snapshot](https://img.shields.io/maven-metadata/v?metadataUrl=https%3A%2F%2Fcentral.sonatype.com%2Frepository%2Fmaven-snapshots%2Fdev%2Fzio%2Fzio-blocks-config_3%2Fmaven-metadata.xml&label=Sonatype%20Snapshot)](https://central.sonatype.com/repository/maven-snapshots/dev/zio/zio-blocks-config_3/) [![ZIO Blocks](https://img.shields.io/github/stars/zio/zio-blocks?style=social)](https://github.com/zio/zio-blocks)

## What Is ZIO Blocks?

ZIO Blocks is a **family of type-safe, modular building blocks** for Scala applications. Each block is a standalone library with zero or minimal dependencies, designed to work with *any* Scala stack—ZIO, Cats Effect, Kyo, Ox, Akka, or plain Scala.

The philosophy is simple: **use what you need, nothing more**. Each block is independently useful and designed to compose with other blocks or your existing code.

## Core Principles

- **Zero Lock-In**: No dependency on ZIO, Cats Effect, or any other effect system. Use a block with whatever stack you already have.
- **Modular**: Each block is a separate artifact. Depend on exactly what you need.
- **Cross-Platform**: Most blocks cross-build for JVM and Scala.js, and for Scala 2.13 and 3.x with source compatibility—adopt Scala 3 on your timeline, not ours. The catalog below records the exceptions per block.
- **High Performance**: Implementations that avoid boxing, minimize allocations, and use platform-specific features where they pay off.
- **Type Safety**: Scala's type system carries the correctness guarantees, without runtime overhead.

To call ZIO Blocks from a ZIO or cats-effect application, see [How to Use ZIO Blocks in a ZIO or cats-effect Application](./guides/use-zio-blocks-in-zio-or-cats-effect.md).

## Getting Started

Add a block and use it. Nothing else to wire up—no runtime to install, no effect
type to adopt:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"
```

That line is for Scala 2.13 and Scala 3.3 to 3.8. On Scala 3.9 and newer, use `zio-blocks-next-schema` and add `scalacOptions += "-experimental"`. See [How to Add ZIO Blocks to Your Build](./getting-started/installation.md) for Scala.js, scala-cli, and cross-building.

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val alice = Person("Alice", 30)

// One schema, every format
val jsonStr = alice.toJsonString                      // {"name":"Alice","age":30}
val parsed  = """{"name":"Bob","age":25}""".fromJson[Person]
```

New to ZIO Blocks? [Getting Started](./getting-started/index.md) is a short tour of the core ideas, and the [Your First Schema](./guides/your-first-schema.md) tutorial is the first hands-on lesson.

The four blocks below get a full walkthrough because they have no close
equivalent elsewhere in Scala. Every other block is one row away in the catalog,
and each row links to its own reference page.

## All Blocks

Every block is published under the `dev.zio` organization. Most cross-build for
**Scala 2.13 and 3.x** on both **JVM and Scala.js** with full source compatibility —
adopt Scala 3 on your timeline, not ours. The handful of modules that are narrower
say so in their own row.

### Meta Programming

JSON support is built into `zio-blocks-schema`; the modules below add further formats.

| Block | Artifact | Platform | Scala | Description |
|-------|----------|----------|-------|-------------|
| [Schema](./reference/schema/index.md) | `zio-blocks-schema` | JVM · JS | 2.13 · 3.x | Type-safe schemas with automatic codec, optic, and validator derivation |
| [Avro Codec](./reference/schema/built-in-codecs/avro.md) | `zio-blocks-schema-avro` | JVM | 2.13 · 3.x | Apache Avro binary serialization with automatic schema generation |
| [BSON Codec](./reference/schema/built-in-codecs/bson.md) | `zio-blocks-schema-bson` | JVM | 2.13 · 3.x | MongoDB-compatible BSON serialization with native type support |
| [CSV Codec](./reference/schema/built-in-codecs/csv.md) | `zio-blocks-schema-csv` | JVM · JS | 2.13 · 3.x | RFC 4180-compliant CSV serialization |
| [MessagePack Codec](./reference/schema/built-in-codecs/messagepack.md) | `zio-blocks-schema-messagepack` | JVM · JS | 2.13 · 3.x | Compact binary serialization with optimized streaming |
| [Thrift Codec](./reference/schema/built-in-codecs/thrift.md) | `zio-blocks-schema-thrift` | JVM | 2.13 · 3.x | Apache Thrift binary serialization with TBinaryProtocol |
| [TOON Codec](./reference/schema/built-in-codecs/toon.md) | `zio-blocks-schema-toon` | JVM · JS | 2.13 · 3.x | Token-oriented notation 30–60% smaller than JSON, tuned for LLM prompts |
| [XML Codec](./reference/schema/built-in-codecs/xml.md) | `zio-blocks-schema-xml` | JVM · JS | 2.13 · 3.x | Zero-dependency XML serialization with fluent navigation and patching |
| [YAML Codec](./reference/schema/built-in-codecs/yaml.md) | `zio-blocks-schema-yaml` | JVM · JS | 2.13 · 3.x | Human-readable YAML serialization with JSON interop |
| [TypeId](./reference/typeid.md) | `zio-blocks-typeid` | JVM · JS | 2.13 · 3.x | Compile-time type identity with rich metadata |

### Resource Management

All of these ship in `zio-blocks-scope`.

| Block | Artifact | Platform | Scala | Description |
|-------|----------|----------|-------|-------------|
| [Scope](./reference/resource-management/index.md) | `zio-blocks-scope` | JVM · JS | 2.13 · 3.x | Compile-time safe resource boundaries that keep values from escaping their lifetime |
| [Resource](./reference/resource-management/resource.md) | `zio-blocks-scope` | JVM · JS | 2.13 · 3.x | Lazy recipes that pair acquisition with finalization, composable with `map`, `flatMap`, and `zip` |
| [Unscoped](./reference/resource-management/unscoped.md) | `zio-blocks-scope` | JVM · JS | 2.13 · 3.x | Marker typeclass for plain data that can safely leave a scope |
| [DeferHandle](./reference/resource-management/defer-handle.md) | `zio-blocks-scope` | JVM · JS | 2.13 · 3.x | Handle returned by `Scope.defer` for cancelling a registered finalizer |
| [Finalizer](./reference/resource-management/finalizer.md) | `zio-blocks-scope` | JVM · JS | 2.13 · 3.x | Minimal capability interface for registering cleanup actions |
| [Finalization](./reference/resource-management/finalization.md) | `zio-blocks-scope` | JVM · JS | 2.13 · 3.x | Result of running a scope's finalizers, including any cleanup errors |

### Dependency Injection

| Block | Artifact | Platform | Scala | Description |
|-------|----------|----------|-------|-------------|
| [Wire](./reference/resource-management/wire.md) | `zio-blocks-scope` | JVM · JS | 2.13 · 3.x | Compile-time safe recipes for constructing a service and its dependencies |
| [Context](./reference/context.md) | `zio-blocks-context` | JVM · JS | 2.13 · 3.x | Type-indexed heterogeneous collections |

### Configuration & Feature Flags

| Block | Artifact | Platform | Scala | Description |
|-------|----------|----------|-------|-------------|
| [Configuration](./reference/config/index.md) | `zio-blocks-config` | JVM · JS | 2.13 · 3.x | Typed configuration loading, feature flags, and rollout rules |
| [Config YAML](./reference/config/formats.md) | `zio-blocks-config-yaml` | JVM · JS | 2.13 · 3.x | YAML source adapter for `ConfigSource` |
| [Config JSON](./reference/config/formats.md) | `zio-blocks-config-json` | JVM · JS | 2.13 · 3.x | JSON source adapter for `ConfigSource` |
| [Config HOCON](./reference/config/formats.md) | `zio-blocks-config-hocon` | JVM · JS | 2.13 · 3.x | HOCON source adapter for `ConfigSource` |

### Web & HTTP

| Block | Artifact | Platform | Scala | Description |
|-------|----------|----------|-------|-------------|
| [MediaType](./reference/media-type.md) | `zio-blocks-mediatype` | JVM · JS | 2.13 · 3.x | Type-safe IANA media types with 2,600+ predefined types |
| [HTTP Model](./reference/http-model/index.md) | `zio-blocks-http-model` | JVM · JS | 2.13 · 3.x | Pure HTTP data model with URL parsing, headers, cookies, and forms |
| [HTTP Model Schema](./reference/http-model/schema.md) | `zio-blocks-http-model-schema` | JVM · JS | 2.13 · 3.x | Schema-based typed access to the HTTP model |
| [Endpoint](./reference/endpoint/index.md) | `zio-blocks-endpoint` | JVM · JS | 2.13 · 3.x | Type-safe HTTP endpoint descriptors with composable codecs and typed auth |
| [OpenAPI](./reference/openapi.md) | `zio-blocks-openapi` | JVM · JS | 2.13 · 3.x | Type-safe OpenAPI 3.1 specification generation and rendering |
| [JWT](./reference/jwt.md) | `zio-blocks-jwt` | JVM · JS | 2.13 · 3.x | Zero-dependency JWT signing and verification with HMAC, RSA, ECDSA and EdDSA support |
| [HTML](./reference/html.md) | `zio-blocks-html` | JVM · JS | 2.13 · 3.x | Type-safe HTML templating with XSS protection |
| [Datastar](./reference/datastar/index.md) | `zio-blocks-datastar` | JVM · JS | 3.x | Typed Datastar attribute and signal DSL, plus the SSE events that patch a live page |
| [HTMX](./reference/htmx/index.md) | `zio-blocks-http-htmx` | JVM · JS | 3.x | Typed HTMX DSL for compile-time-checked HTMX attributes |

### Data Types

| Block | Artifact | Platform | Scala | Description |
|-------|----------|----------|-------|-------------|
| [Chunk](./reference/chunk.md) | `zio-blocks-chunk` | JVM · JS | 2.13 · 3.x | High-performance immutable indexed sequences with zero-boxing builders |
| [Maybe](./reference/maybe.md) | `zio-blocks-maybe` | JVM · JS | 2.13 · 3.x | Optional values that store a present reference as is, without a wrapper object (Scala 3) |
| [Combinators](./reference/combinators.md) | `zio-blocks-combinators` | JVM · JS | 2.13 · 3.x | Compile-time composition and decomposition of tuples, eithers, and unions |

### Concurrency

| Block | Artifact | Platform | Scala | Description |
|-------|----------|----------|-------|-------------|
| [Async](./reference/async.md) | `zio-blocks-async` | JVM · JS | 2.13 · 3.x | Zero-allocation asynchronous effect type with direct-style `await` |
| [Mux](./reference/mux.mdx) | `zio-blocks-mux` | JVM · JS | 2.13 · 3.x | Thread-safe multiplexer for HTTP/2, QUIC, and WebSocket-style protocols |
| [RingBuffer](./reference/ringbuffer/index.mdx) | `zio-blocks-ringbuffer` | JVM · JS | 2.13 · 3.x | Lock-free bounded ring buffers (SPSC, SPMC, MPSC, MPMC) |

### Streams

| Block | Artifact | Platform | Scala | Description |
|-------|----------|----------|-------|-------------|
| [Streams](./reference/streams/index.md) | `zio-blocks-streams` | JVM · JS | 2.13 · 3.x | Pull-based streaming with typed errors, zero boxing, and synchronous or asynchronous execution |

### Telemetry

| Block | Artifact | Platform | Scala | Description |
|-------|----------|----------|-------|-------------|
| [Telemetry](./reference/telemetry/index.md) | `zio-blocks-telemetry` | JVM · JS | 2.13 · 3.x | Zero-dependency OpenTelemetry-aligned tracing, logging, and metrics |
| [OTLP Export](./reference/telemetry/otel/index.md) | `zio-blocks-telemetry-otel` | JVM | 2.13 · 3.x | OTLP exporters bridging telemetry signals to an OpenTelemetry collector |

### Persistence

| Block | Artifact | Platform | Scala | Description |
|-------|----------|----------|-------|-------------|
| [SQL Module](./reference/sql/index.md) | `zio-blocks-sql` | JVM · JS | 3.x | Type-safe JDBC wrapper with schema-derived codecs and a CRUD repository |
| [ZIO Integration](./reference/sql-zio.md) | `zio-blocks-sql-zio` | JVM | 3.x | ZIO integration with `ZIO.attemptBlocking` and `ZLayer` |
| [Data Migration](./reference/data-migration.md) | `zio-blocks-data-migration` | JVM · JS | 3.x | Typed, online database schema migrations in three execution models, with no hand-written SQL |
| [Projection](./reference/projection/index.md) | `zio-blocks-projection` | JVM | 3.x | Event-sourced projections with per-entity SQLite storage |

### Tooling & Codegen

| Block | Artifact | Platform | Scala | Description |
|-------|----------|----------|-------|-------------|
| [Code Generation](./reference/codegen/index.md) | `zio-blocks-codegen` | JVM | 2.13 · 3.x | Generic Scala code generation IR and emitter |
| [Docs](./reference/docs.md) | `zio-blocks-markdown` | JVM · JS | 2.13 · 3.x | GitHub Flavored Markdown parsing, rendering, and programmatic construction |
| [Smithy](./reference/smithy.md) | `zio-blocks-smithy` | JVM | 2.13 · 3.x | Smithy IDL parser and AST library for API modeling |

---

## Schema

The Schema block brings dynamic-language productivity to statically-typed Scala. Define your data types once, and derive codecs, validators, optics, and more automatically.

### The Problem

In statically-typed languages, you often maintain separate codec implementations for each data format (JSON, Avro, Protobuf, etc.). Meanwhile, dynamic languages handle data effortlessly:

```javascript
// JavaScript: one line and done
const data = await res.json();
```

In Scala, you'd typically need separate codecs for each format—a significant productivity gap.

### The Solution

ZIO Blocks Schema derives everything from a single schema definition:

```scala
case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Derive codecs for any format:
val jsonCodec    = Schema[Person].derive(JsonFormat)        // JSON
val avroCodec    = Schema[Person].derive(AvroFormat)        // Avro
val toonCodec    = Schema[Person].derive(ToonFormat)        // TOON (LLM-optimized)
val msgpackCodec = Schema[Person].derive(MessagePackFormat) // MessagePack
val thriftCodec  = Schema[Person].derive(ThriftFormat)      // Thrift
```

### Key Features

- **Universal Data Formats**: JSON built in, plus Avro, BSON, CSV, MessagePack, Thrift, TOON, XML, and YAML as separate modules, with Protobuf planned.
- **High Performance**: Register-based design stores primitives directly in byte arrays, enabling zero-allocation serialization.
- **Reflective Optics**: Type-safe lenses, prisms, and traversals with embedded structural metadata.
- **Automatic Derivation**: Derive type class instances for any type with a schema.

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"
```

See the [Meta Programming](#meta-programming) rows above for the optional format modules.

### Example

```scala

case class Address(street: String, city: String)
case class Person(name: String, age: Int, address: Address)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String] = $(_.name)
  val age: Lens[Person, Int] = $(_.age)
  val streetName: Lens[Person, String] = $(_.address.street)
}

val person = Person("Alice", 30, Address("123 Main St", "Springfield"))
val updated = Person.age.replace(person, 31)
```

### Learn More

- [Your First Schema](./guides/your-first-schema.md) — a getting-started tutorial: describe a type once, round-trip it through JSON, and read decode errors
- [Evolve Your Data](./guides/evolve-your-data.md) — a tutorial that validates incoming data with `Into`, converts between versions with `As`, and renames a stored field with a compile-checked `Migration`
- [One Type, Many Formats](./guides/one-type-many-formats.md) — a tutorial that exports one catalog as JSON, YAML, XML, TOON, MessagePack, Avro and CSV from the same schema
- [How to Encode and Decode Records with Avro](./guides/encode-and-decode-records-with-avro.md) — a how-to guide that writes the `.avsc` schema for a sealed trait, streams Avro records, maps each Scala type to its Avro type, and reads the bytes with Apache Avro
- [How to Store Case Classes in MongoDB](./guides/store-case-classes-in-mongodb.md) — a how-to guide that derives a BSON codec, uses `ObjectId` as the `_id`, chooses how sealed traits and options are stored, and inserts and queries tickets with the MongoDB Java driver
- [Schema reference](./reference/schema/index.md) — the full API surface, from `Reflect` and `Binding` through optics, validation, and schema evolution
- [Migrating from ZIO Schema](./guides/zio-schema-migration.md) — a step-by-step port from ZIO Schema 1.x
- [Coming from circe, zio-json or jsoniter-scala](./guides/json-library-migration.md) — move JSON codecs to one derived `Schema`, keeping field names, discriminators and custom formats

---

## Scope

Compile-time verified resource safety for synchronous Scala code. Scope prevents resource leaks at compile time by tagging values with an unnameable type-level identity—values allocated in a scope can only be used within that scope. Child scope values cannot escape to parent scopes, enforced by both the abstract scope-tagged type and the `Unscoped` constraint on `scoped`.

### The Problem

Resource management in Scala is error-prone:

```scala
// Classic try/finally - verbose and easy to get wrong
val db = openDatabase()
try {
  val tx = db.beginTransaction()
  try {
    doWork(tx)
    tx.commit()
  } finally tx.close()  // What if commit() throws?
} finally db.close()

// Using - better, but doesn't prevent returning resources
Using(openDatabase()) { db =>
  db  // Oops! Returned the resource - use after close!
}
```

### The Solution

Scope makes resource leaks a **compile error**, not a runtime bug:

```scala

Scope.global.scoped { scope =>

  val db: $[Database] = allocate(Resource(openDatabase()))

  // Methods are hidden - can't call db.query() directly
  // Must use $ to access:
  val result: String = $(db)(_.query("SELECT 1"))

  // Trying to return `db` would be a compile error!
  result  // Only pure data (String) escapes
}
// db.close() called automatically
```

### Key Features

- **Compile-Time Leak Prevention**: Values of type `scope.$[A]` are opaque and unique to each scope instance. Returning a scoped value from its scope is a type error.
- **Zero Runtime Overhead**: `$[A]` erases to `A` at runtime—zero allocation overhead.
- **Structured Scopes**: Child scopes nest within parents; resources clean up LIFO when scopes exit.
- **Built-in Dependency Injection**: Wire up your application with `Resource.from[T](wires*)` for automatic constructor-based DI.
- **AutoCloseable Integration**: Resources implementing `AutoCloseable` have `close()` registered automatically.
- **Unscoped Constraint**: The `scoped` method requires `Unscoped[A]` evidence on the return type, ensuring only pure data (not resources or closures) can escape.
- **Actionable Runtime Errors**: If a scope reference escapes and is used after closing, `allocate`, `open()`, and `$` throw `IllegalStateException` with a detailed message explaining what went wrong, the common causes, and how to fix it—no silent null returns.

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-scope" % "0.0.56"
```

### Example

```scala

final class Database extends AutoCloseable:
  def query(sql: String): String = s"Result: $sql"
  def close(): Unit = println("Database closed")

Scope.global.scoped { scope =>

  // Allocate returns $[Database] (scoped value)
  val db: $[Database] = allocate(Resource(new Database))

  // Access via $ - result (String) escapes, db does not
  val result: String = $(db)(_.query("SELECT * FROM users"))

  println(result)
}
// Output: Result: SELECT * FROM users
//         Database closed
```

### Learn More

- [Compile-Time Resource Safety with Scope](./guides/compile-time-resource-safety-with-scope.md) — the step-by-step tutorial, from basic resource management through dependency injection
- [How to Fix Common Scope and Wire Compile Errors](./guides/fix-scope-and-wire-compile-errors.md) — every Scope, Wire and `Resource.from` error message, with its cause and its fix
- [Resource Management reference](./reference/resource-management/index.md) — `Scope`, `Resource`, `Wire`, `Unscoped`, and finalization order
- [Coming from cats-effect Resource, ZIO Scope/ZLayer, macwire or Using](./guides/resource-and-di-migration.md) — port `Resource`, `ZLayer`, macwire or `Using` code to `Scope`, `Resource` and `Wire`
- [How to Use ZIO Blocks in a ZIO or cats-effect Application](./guides/use-zio-blocks-in-zio-or-cats-effect.md) — call blocking code, resources, `Async` and streams from ZIO or cats-effect, with cancellation and cleanup carried across

---

## Async

A lightweight, zero-dependency asynchronous effect type. A ready `Async[A]` *is*
an `A`, so synchronous code composed with `map` / `flatMap` allocates nothing on
the happy path while still suspending on genuinely asynchronous work.

### The Problem

Asynchronous Scala forces a choice between two costs. `Future` allocates for
every combinator and needs an `ExecutionContext` threaded everywhere, even when
the value is already available. Full effect systems avoid that but ask you to
adopt a runtime, a set of type classes, and a programming model across your
whole codebase—a heavy price for a library that only occasionally suspends.

### The Solution

`Async[A]` is a value, not a wrapper. When the result is already known, the
representation *is* the result, so composing ready values costs nothing:

```scala

// Constructors collapse to bare values; transformers inline with no allocation
val computed: Int =
  Async.succeed(20).map(_ + 1).flatMap(n => Async.succeed(n * 2)).block
// computed: Int = 42
```

### Key Features

- **Zero Allocation on the Happy Path**: A completed `Async[A]` is represented as the `A` itself; `map` and `flatMap` over ready values allocate nothing.
- **Direct-Style `await`**: `Async.async { ... }` rewrites `.await` calls at compile time into a non-blocking `flatMap` chain—straight-line code, asynchronous execution.
- **No Runtime to Adopt**: No `ExecutionContext` to thread, no type class hierarchy, no effect system dependency.
- **Interop Built In**: Bridges to `Future` and `CompletionStage`, plus `Async.promise` for callback-based APIs.

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-async" % "0.0.56"
```

### Example

Write straight-line asynchronous code with `Async.async` and `.await`, rewritten
at compile time into a non-blocking `flatMap` chain:

```scala

def fetch(id: Int): Async[String] = Async.succeed(s"item-$id")

val program: Async[Int] =
  Async.async {
    val a = fetch(1).await
    val b = fetch(2).await
    (a + b).length
  }
```

### Learn More

- [Getting Started with Async](./guides/async-getting-started.md) — create, compose, and run async effects
- [Async reference](./reference/async.md) — the full API, including `zip`, `catchAll`, `collectAll`, the `Async.promise` callback bridge, and `Future` / `CompletionStage` interop
- [How to Use ZIO Blocks in a ZIO or cats-effect Application](./guides/use-zio-blocks-in-zio-or-cats-effect.md) — call `Async` from ZIO or cats-effect, with cancellation carried across
- [`async-examples`](https://github.com/zio/zio-blocks/blob/main/async-examples/src/main/scala/async/AsyncShowcaseExample.scala) — a single-file order-fulfillment demo (`cd async-examples && sbt run`)

---

## SQL

A thin, type-safe JDBC wrapper that maps Scala case classes to database tables using the same `Schema` you use for JSON and Avro codecs. No ORM runtime, no code generation — just composable SQL fragments, a derived repository abstraction, and a direct ZIO integration.

### The Problem

JDBC is powerful but tedious: manual `ResultSet` traversal, index-based parameter binding, and repetitive CRUD boilerplate make even simple database access error-prone. ORMs solve the boilerplate but add heavy runtimes, hidden queries, and opaque magic.

### The Solution

ZIO Blocks SQL derives everything from a single `Schema[A]`:

```scala
case class User(id: Long, name: String, email: String)
object User:
  given Schema[User] = Schema.derived

// Derive the table, codec, and repository in one line
val repo = Repo.derived[User, Long]

// Use the sql"..." interpolator for custom queries
val frag = sql"SELECT * FROM user WHERE email = ${"alice@example.com"}"
```

### Key Features

- **Schema-derived codecs**: `DbCodec[A]` is auto-derived from `Schema[A]` — column names, types, and nullability come for free.
- **Composable fragments**: The `sql"..."` interpolator creates `Frag` values that compose safely with `++`. SQL injection is structurally impossible.
- **CRUD repository**: `Repo[E, ID]` provides `all`, `find`, `findAll`, `insert`, `insertAll`, `update`, `delete`, `deleteAll`, and `clear` out of the box.
- **DDL generation**: `Table.createTable(dialect)` generates type-accurate `CREATE TABLE IF NOT EXISTS` SQL from the schema.
- **ZIO integration**: `TransactorZIO` lifts blocking JDBC calls into `Task` (or `ZIO`) with proper bracketing and rollback.
- **Effect-system agnostic core**: The `zio-blocks-sql` module has no ZIO dependency — use it with any effect system or plain Scala.

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-sql" % "0.0.56"

// Optional ZIO integration
libraryDependencies += "dev.zio" %% "zio-blocks-sql-zio" % "0.0.56"
```

### Example

```scala

case class Product(id: Long, name: String, price: Double)
object Product:
  given Schema[Product] = Schema.derived
  given DbCodec[Product] = summon[Schema[Product]].deriving(DbCodecDeriver).derive

val repo        = Repo.derived[Product, Long]
val transactor  = TransactorZIO.fromUrl("jdbc:postgresql://localhost/shop", SqlDialect.PostgreSQL)

// Batch insert, then query with a custom filter
val program = transactor.transact:
  repo.insertAll(List(
    Product(1L, "Widget", 9.99),
    Product(2L, "Gadget", 29.99)
  ))
  sql"SELECT * FROM product WHERE price < ${15.0}".query[Product]
```

### Learn More

- [SQL reference](./reference/sql/index.md) — `DbCodec`, `Frag`, `Table`, `Repo`, `Transactor`, dialects, and DDL generation
- [Query DSL guide](./guides/query-dsl-reified-optics.md) — a four-part series building a type-safe query language on reified optics

---

## Compatibility

ZIO Blocks works with any Scala stack:

| Stack | Compatible |
|-------|------------|
| ZIO 2.x | ✅ |
| Cats Effect 3.x | ✅ |
| Kyo | ✅ |
| Ox | ✅ |
| Akka | ✅ |
| Plain Scala | ✅ |

Each block has zero dependencies on effect systems. Use the blocks directly, or integrate them with your effect system of choice.

## Guides

- [How to Add ZIO Blocks to Your Build](./getting-started/installation.md) - Pick the modules you need and add them to an sbt or scala-cli build, for any Scala version and for Scala.js
- [Your First Schema: Round-Trip a Type Through JSON](./guides/your-first-schema.md) - Describe a type once with `Schema`, then encode, decode, and read errors
- [Your First Streaming Pipeline: Turn a Sales Log into a Report](./guides/your-first-streaming-pipeline.md) - Read a sales log with a `Stream`, parse, filter and batch it, sum it with a `Sink`, and recover from a bad line
- [Build Your First Database Repository on SQLite](./guides/your-first-sqlite-repository.md) - Connect to SQLite, derive a table and `Repo` from one schema, write your own queries, and roll back a transaction
- [Instrument a Program with Logs, Traces and Metrics](./guides/instrument-a-program-with-logs-traces-and-metrics.md) - Add structured logs, nested spans, a counter and a histogram to a batch job, switch to JSON logs, and test the spans
- [Build a Small JSON Backend from Blocks](./guides/small-json-backend-from-blocks.md) - Capstone: combine schema, config, scope, SQL, the HTTP model and telemetry into one small JSON backend

These six follow the learning path, from installing the library to a small working backend. The [guides index](https://zioblocks.com/guides/) on the website lists every tutorial, how-to guide, migration guide and deep dive.

## Full API Reference

Every block in the catalog above links to its own reference page. The blocks
large enough to have several pages start from an overview:

- [Schema](./reference/schema/index.md) - core type system, dynamic values, optics, validation, and schema evolution
  - [Built-in Codecs](./reference/schema/built-in-codecs/index.md) - JSON, Avro, BSON, CSV, MessagePack, Thrift, TOON, XML, and YAML
  - [Schema Evolution](./reference/schema/schema-evolution/index.md) - one-way and bidirectional type-safe conversions
- [Telemetry](./reference/telemetry/index.md) - tracing, logging, metrics, and OTLP export
- [SQL](./reference/sql/index.md) - codecs, fragments, tables, repositories, transactors, and dialects
- [Projection](./reference/projection/index.md) - event-sourced read models: `Projection[A]`, `ProjectionEngine`, and per-entity SQLite stores
- [Resource Management](./reference/resource-management/index.md) - `Scope`, `Resource`, `Wire`, `Unscoped`, and finalization
- [Streams](./reference/streams/index.md) - `Stream`, `Pipeline`, `Sink`, and the low-level readers and writers
- [Endpoint](./reference/endpoint/index.md) - endpoint descriptors, HTTP codecs, route patterns, and typed auth
- [HTTP Model](./reference/http-model/index.md) - the pure HTTP data model and its schema-based typed access
- [HTMX](./reference/htmx/index.md) - the typed HTMX attribute DSL
- [Ring Buffer](./reference/ringbuffer/index.mdx) - the SPSC, SPMC, MPSC, and MPMC variants
- [Code Generation](./reference/codegen/index.md) - the Scala code generation IR and emitter

---

## Async

The `async` module provides `Async[A]` — a small asynchronous effect type for Scala 2.13 and Scala 3, targeting both JVM and Scala.js.

An `Async[A]` value is a computation that either yields an `A` or fails with a `Throwable`. Unlike a lazy effect type, building one *runs* it: the synchronous work happens as you construct it, and only a computation that genuinely has to wait for something is left pending. [Evaluation model](#evaluation-model) explains where that line falls.

Conceptually, an `Async[A]` is one of three things — a value that is already available, a failure that has already happened, or a computation that will complete later:

```scala
// The mental model, not the real encoding.
enum Async[+A]:
  case Ready(value: A)                 // already available
  case Failed(cause: Throwable)        // already failed
  case Suspended(source: Pollable[A])  // completes later, via a callback
```

The real definition is a type alias whose runtime representation is `Any`. That is a performance decision, not a modelling one: a ready value *is* its own `Async`, so the common path allocates nothing and boxes nothing. Reach for the three cases above when reasoning about behaviour, and let the encoding stay invisible — no combinator in this module requires you to know it.

Suspension is where the other types enter. A `Pollable[A]` is the extension point that produces a not-yet-available result, [`Completer`](#completer) is the ready-made `Pollable` for bridging callbacks, [`Async.Running`](#asyncrunning) is a `Pollable` you can also cancel, and [`Cancelable`](#cancelable) is that cancellation interface on its own.

The type is aimed at infrastructure-level code: places that need a first-class asynchronous value, on both the JVM and Scala.js, without taking on an ecosystem to get one. Where a fuller effect system offers an environment type, a typed error channel and fibers, `Async` offers a computation, a `Throwable`, and a handle you can cancel — and in exchange stays cheap enough to use on paths that are usually synchronous.

## Installation

Add the module to your build:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-async" % "0.0.56"
```

In a cross-built project, use `%%%` so the same line resolves for both JVM and Scala.js:

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-async" % "0.0.56"
```

The module publishes for JVM and Scala.js, on Scala 2.13 and Scala 3. That single coordinate is all you add: it brings `zio-blocks-combinators` with it, along with `dotty-cps-async` on Scala 3 or `scala-reflect` on Scala 2, both of which power the direct-style rewrite.

## Overview

`Async[A]` is the type you will spend nearly all your time with. Every way of creating a value, every way of transforming one, and every way of running one produces or consumes an `Async[A]`. If you only learn this type, you can already write complete programs — the rest of the module exists to feed values into it or to control one that is already running.

`Completer[A]` is the type you reach for next, and the reason is a problem you have almost certainly hit: some library hands you a result through a callback rather than returning it. `Async.promise` gives you a `Completer`, you pass that to the callback, and you get back an `Async[A]` that completes when the callback fires. From that point on it behaves like any other `Async[A]`, so the callback-based API disappears into ordinary code.

`Async.Running[A]` appears when you start work without waiting for it. Calling `.start` on an `Async[A]` begins the computation immediately and hands you a `Running` as a receipt. Keep it, and you can wait for the result later, run several pieces of work at once and collect them all, or stop the work early.

`Cancelable` is that last ability on its own — a single way to say "stop this." `Async.Running` provides it, and so can anything else you write that needs to be stoppable.

`Pollable[A]` is the one type most programs never touch. It is the extension point for teaching the module about a brand-new source of delayed results — a timer, a socket read, a platform-specific callback. Implementing one makes your source usable anywhere an `Async[A]` is expected. Reach for it only when you are wiring up something genuinely new; for ordinary callback bridging, `Async.promise` and a `Completer` are the right tools.

## Evaluation Model

`Async[A]` is **eager**, not a lazy `IO`. Building one performs its synchronous work straight away — constructing the value *runs* it, up to the first point where it genuinely has to wait:

```scala

// "computing" is printed by this line, not by the one below it.
val fa: Async[Int] = Async.attempt { println("computing"); 42 }
val n: Int         = fa.block  // the value was already there; nothing more runs
```

The same is true across the module. `Async.promise` runs its setup block when you call it, and `Async.succeed(x).map(f)` applies `f` immediately, because `x` is right there. Only a combinator applied to a value that is *already suspended* defers: then the function is kept and runs when a driver settles the value.

A direct-style block splits the same way, at its first genuine wait. Everything above that line runs as you build the value; everything below it is kept for later:

```scala
Async.async {
  val cfg  = loadConfig()             // ┐
  logger.info("starting")             // ├ runs now, as this value is built
  val base = compute(cfg)             // ┘
  val row  = fetchRow(base).await     // the first genuine wait
  transform(row)                      // runs later, when a driver settles it
}
```

An `await` whose value is ready before it is asked does not count as a wait. It hands the value over on the spot and the block carries straight on, so the dividing line is not the first `await` you wrote — it is the first one that has nothing to give yet:

```scala
Async.async {
  val a = Async.succeed(1).await   // already a value: no wait, keep going
  val b = compute(a).await         // also ready: no wait, keep going
  val c = fetchFromNetwork().await // nothing yet — the block stops here
  a + b + c                        // deferred, along with everything below
}
```

The practical consequence is that you cannot find the pause by counting `await`s. A block whose values are all ready pauses nowhere and finishes as you construct it; a block whose *first* `await` is a network call has run none of the lines below it by the time `Async.async { … }` hands you a value.

So where does waiting come from at all? From exactly one place: a [`Pollable`](#pollable) that was asked for its value and answered *not yet*. That is the only thing in the module that can make a computation pending. Everything else already has its answer — `succeed` has a value, `fail` has a cause, `attempt` has run, `map` merely applies a function.

Waiting then spreads in one direction only: to the combinators stacked on top of that pending value, which cannot produce a result until it does.

```scala
val c = new Completer[Int]                    // a Pollable; not completed yet
val a = c.peek.map(_ + 1)                     // above a pending value → deferred
val b = a.flatMap(n => Async.succeed(n * 2))  // still above it → deferred

val d = Async.succeed(1).map(_ + 1)           // no pending value anywhere → already ran
```

`a` and `b` wait only because they sit on top of a `Completer` nobody has completed. `d` shares none of that history, so it is simply the number `2` — the `+ 1` happened as the line was evaluated.

One consequence catches people out. If a function you pass to `map`, `flatMap` or `tap` throws, and the value it is applied to is *ready*, that function runs as you build the value — so the exception escapes at that line rather than becoming a failed `Async` you can `catchAll`. Only `Async.attempt` turns a throw into a `Failure`; the combinators do not:

```scala
Async.succeed(text).map(_.toInt)                          // throws here if text is not a number
Async.succeed(text).flatMap(s => Async.attempt(s.toInt))  // fails as an Async instead
```

The "only one place" part is what distinguishes `Async` from a lazy effect type. `map`, `flatMap` and `zipWith` do not themselves defer anything, and building a chain does not create a plan to be executed later. If you cannot point at a value still waiting to be completed, nothing in your chain is pending: it has all already run.

Two things follow. Any pending computation can be traced to the thing it is waiting on — a `Completer` for a callback bridge, an [`Async.Running`](#asyncrunning) for started work, or your own `Pollable`. And the waiting is temporary and forward-only: when that value is completed, everything above it becomes ready, and nothing below it was ever held up, because that code had already run.

This is a deliberate trade. Eager evaluation is what keeps the ready path allocation-free — no effect tree, no per-step thunk, none of the wrapper objects most effect types build — and that is where the throughput comes from. It is also why `Async[A]` costs close to nothing on a mostly-synchronous path: when there is nothing to wait for, chaining operations onto a value just runs them.

The costs are real too. Building a value has effects, so `Async` is not referentially transparent: you cannot move a construction around, or replace a value with the expression that produced it, and be sure the program still means the same thing.

And letting go of a value does not stop it. With a lazy effect type, discarding an unused effect discards the work with it, because none of it had happened yet. Here the work is already under way:

```scala
val running = Async.start(uploadHugeFile())

// Reassigning or forgetting `running` does not stop the upload. The worker
// keeps going and the bytes keep moving; you have only thrown away your
// ability to watch it or stop it.
```

Stopping requires asking, through [`Cancelable#cancel`](#cancelable), and that reaches less far than you might expect.

The handle is your only route to that request. There is no registry of running computations to consult and no supervisor to ask, so a handle you have dropped cannot be recovered: that work becomes unstoppable for the rest of the process, and it finishes on its own schedule, holding its thread and its socket until it does. Nothing counts how many observers are left, so nothing notices when the last one goes away.

The rule that falls out is to decide at `start` time whether this work might ever need stopping. If it might, give the handle somewhere to live:

```scala

class Uploader {
  private val current = new AtomicReference[Async.Running[Unit]]()

  def begin(): Unit = {
    val previous = current.getAndSet(Async.start(uploadHugeFile()))
    if (previous ne null) previous.cancel()  // stop watching the one displaced
  }

  def abort(): Unit = {
    val running = current.getAndSet(null)
    if (running ne null) running.cancel()    // possible only because it was kept
  }
}
```

Note what `begin` has to do on the way past: replacing a handle means cancelling the one it displaces, or that upload becomes unstoppable while still running. The `AtomicReference` is there because the field is reachable from more than one thread — `abort` may well be called while `begin` is assigning.

A [`Using`](#integration-points) block does the same job when the work is confined to a scope. And if the answer is that it never needs stopping, dropping the handle is fine — that is fire-and-forget, chosen deliberately rather than by accident.

In all of this `Async` sits beside `scala.concurrent.Future`, which is also eager, rather than beside cats-effect `IO` or ZIO.

### Depth Limits on Waiting Values

Call `map` on a value that is still waiting and you get back a wrapper: it remembers the original value and the function you gave it. Ask that wrapper for a result and it has to ask the original first, because until the original produces something there is nothing to apply the function to. That question is an ordinary method call, so the wrapper is left part-way through its own work — holding its place on the call stack — while the value underneath answers.

Stack several and each repeats the pattern. Take `c.peek.map(f).flatMap(g).map(h)`, where `c` is a `Completer` nobody has completed yet. Asking the outermost wrapper sets off a chain of questions inward, and every one of them waits where it stands:

```
map h asks …                  still waiting
 └─ flatMap g asks …          still waiting
     └─ map f asks …          still waiting
         └─ c answers "not yet" — and that travels back out through all three
```

None of them can finish until the innermost one answers, so all of them are held open at the same time. A chain written N deep costs N held-open calls, every single time it is asked.

Most effect systems avoid that with a trampoline: rather than calling its child directly, each step returns a small object meaning *"do this next"* to a loop that keeps running steps until one produces a value. One loop frame serves any depth. The price is paid on every step of every poll — an object allocated to describe the step, and a dispatch through the loop instead of a direct call.

The choice for the general case was between the two, with one refinement described below for chains built in a loop:

| Approach     | Depth                     | Cost per step                                      |
|--------------|---------------------------|----------------------------------------------------|
| Trampoline   | Unlimited                 | An object allocated, and a dispatch, on every poll |
| Direct calls | Limited by the call stack | Nothing                                            |

`Async` takes the second. That is why nothing is allocated while polling, and it is also why recursion through `flatMap` over a waiting value ends in a `StackOverflowError` rather than a slowdown — the ceiling is the bill for the speed, not an oversight. A spine of `map` and `flatMap` wrappers that was built iteratively is the exception: the library polls it in a loop instead of recursing, as the list below explains.

You are unlikely to meet it by hand. A handful of `map` and `flatMap` calls around a network request is nowhere near the limit. It becomes a real risk when the length of the chain is decided by *data* — one `flatMap` per row, per file, per retry — because then the depth is however large the input happens to be, and recursive code that is comfortable in a test can overflow in production on a bigger batch.

Whether you are anywhere near the limit comes down to a single question: is the chain being built on top of a value that is still waiting?

- **On a value that is already there, chains are safe at any length.** Each step runs as you write it and gives back a plain value again, so nothing is left holding anything open. A loop like `var fa = …; while (…) fa = fa.flatMap(g)` stays flat no matter how many times it goes round — the suite takes it to a million.
- **On a value that is still waiting, an iterative chain is safe, and a recursive one is not.** A spine of `map` and `flatMap` wrappers built in a loop (`fa = fa.map(g)`, or `fa = fa.flatMap(g)` where `g` returns a plain or ready value) is interpreted by a loop rather than by nested calls, so a chain of 1,000,000 over a pending `Completer` completes without a `StackOverflowError`. Recursion through `flatMap`, where each continuation builds the next wrapper (`def loop(n) = src.flatMap(_ => loop(n - 1)).map(_ + 1)`), still opens one call per level and runs out of stack somewhere above 100,000 levels on a default JVM. A chain whose continuations return new pending wrappers (`fa = fa.flatMap(_ => src.flatMap(…))` over a still-pending `src`) does not overflow either, but each step walks the chain again, so its time grows with the square of the length: a known bug in 0.0.56.
- **`Async.collectAll` and a `while` loop inside `Async.async` stay safe even when the values are still waiting** — both are tested at 50,000 such steps. `collectAll` is a single step that walks the collection itself, stacking no wrappers at all, and the direct-style loop runs one turn each time it is asked rather than building the whole chain in advance.

So for long, wait-heavy work whose depth comes from recursion, reach for `collectAll` or an `Async.async` `while` loop instead of a hand-built tower of `flatMap`. `Future` never runs into this, but only because it sends every `flatMap` through an `ExecutionContext`; `Async` skips that hop to stay fast and takes the depth limit instead.

### Pending Suspensions Differ by Platform

Say a computation has hit a real wait. When the thing it waits for finally arrives, what makes the rest of the block run? Each platform answers with whatever its own runtime does fastest, so the answers differ:

- **JVM, and Scala.js on Scala 2 or Scala 3 before 3.8** — nothing runs it for you. The value sits there until something asks it for a result: `.block`, `.start`, or an interop converter. Build a value and never drive it, and the code after the wait does not run late — it never runs at all.
- **Scala.js on Scala 3.8 and later** — the block compiles into a real JavaScript async function, and JavaScript already has something whose job is resuming those: the event loop. It is always running, you did not start it, and you cannot opt out of it. So once the awaited value arrives, your code carries on by itself, whether or not anyone is watching.

```scala
val fa = Async.async { record(fetchRow().await) }

// JVM: fetchRow may well finish, but `record` has not run — nothing drove fa.
// Scala.js 3.8+: once fetchRow finishes, `record` runs anyway.
```

You are unlikely ever to see this. Values get built in order to be used, and the moment you `block` on one, `start` it, or hand it to a `Future`, both platforms behave identically and produce the same result. Noticing the difference takes a peculiar shape: build a block, let the thing it waits for arrive, then never drive it — a program that has already gone wrong, since it constructed work and then discarded it.

It is worth documenting because it changes *when side effects happen*. If the code after a wait prints, writes a file, or bumps a counter, Scala.js may do that without you driving anything, while the JVM will not. Cross-platform code that leans on "this has not run yet" is leaning on something true in only one of the two.

## How They Work Together

A computation moves through four phases: construct leaf values, compose them, drive the result, then observe or cancel, as shown in the following flow diagram:

```
┌─ 1. CONSTRUCT — the synchronous part runs now ─────────────────┐
│   Async.succeed(a)       a value you already have              │
│   Async.fail(t)          a failure you already have            │
│   Async.attempt { … }    runs the block, here, on this thread  │
│   Async.promise { … }    runs its setup block now              │
│   new Pollable[A] { … }  the one genuinely deferred leaf       │
└────────────────────────────────────────────────────────────────┘
                          │
                          ▼
        Async[A]  ──  ready, failed, or waiting on something
                          │
                          ▼
┌─ 2. COMPOSE — runs now if ready, defers if not ────────────────┐
│   map    flatMap    zipWith    tap                             │
│   catchAll    ensuring    collectAll                           │
│                                                                │
│   on a ready value the function runs immediately;              │
│   on a pending one it is kept for the driver to run            │
└────────────────────────────────────────────────────────────────┘
                          │
                          ▼
┌─ 3. DRIVE — settle whatever is still pending ──────────────────┐
│   .block             .start            .toFuture               │
│   wait right here    run in the        .toJsPromise            │
│                      background        hand it to the          │
│        │                  │            platform                │
│        ▼                  ▼                                    │
│   the A, or          Async.Running[A]                          │
│   the Throwable      — your receipt                            │
└────────────────────────────────────────────────────────────────┘
                          │
                          ▼
┌─ 4. OBSERVE or CANCEL — using the receipt ─────────────────────┐
│   .block  .flatMap  .zipWith    wait for it, or compose more   │
│   .cancel()                     stop it (via Cancelable)       │
└────────────────────────────────────────────────────────────────┘
```

Three of these types are closely related, and seeing why makes the module much smaller than it first looks. `Pollable[A]` answers one question — *is the result ready yet?* — and anything that can answer it is a `Pollable`:

```
Pollable[A]   —  "a result that is not here yet"
   │
   ├── Completer[A]       you complete it yourself, once, from a callback
   ├── Async.Running[A]   work that is already running; can also be stopped
   └── Failure            a computation that has already failed
```

That shared parent is what lets all three be used interchangeably. Wherever an `Async[A]` is expected, you can supply any of them, and every combinator — `map`, `flatMap`, `zipWith`, and the rest — works on the result without knowing or caring which one it is.

`Failure` is on the list because failing is just another way of being finished — a computation that has failed is not waiting for anything. [`Failure`](#failure) describes what that means for the combinators downstream of it.

You will use `Completer` and `Async.Running` constantly, and `Failure` mostly without naming it. Writing your own `Pollable` is the rare case, reserved for teaching the module about a new source of delayed results.

Two details the diagram leaves out. Composing over a ready value is allocation-free, while composing over a waiting one allocates a `Pollable` that the driver walks poll by poll. And the handle from phase 3 is itself an `Async`, which is what makes phase 4 ordinary composition rather than a separate API.

The following snippet grounds all four phases in an example: two off-thread `Completer` completions are composed with `zipWith` and driven by `block`:

```scala

def delayed[A](value: A, ms: Long): Async[A] = {
  val c = new Completer[A]
  val t = new Thread(new Runnable {
    def run(): Unit = { Thread.sleep(ms); c.succeed(value) }
  })
  t.setDaemon(true)
  t.start()
  c.peek  // the Completer is itself an Async[A]
}

// Phase 1 and 2: construct two off-thread leaves and compose with zipWith
val r: Async[Int] = delayed(3, 30).zipWith(delayed(4, 5))(_ + _)

// Phase 3: drive — parks the calling thread until both off-thread wakers fire
val result: Int = r.block  // => 7
```

A second example shows what `Async.collectAll` guarantees: the results come back in the order you listed the computations, not the order they happened to finish. To make that visible, the delays below are deliberately reversed — the first element takes the longest, the last finishes almost immediately:

```scala

// Completes with `value` after `ms`, on another thread.
def delayed[A](value: A, ms: Long): Async[A] = {
  val c = new Completer[A]
  val t = new Thread(new Runnable {
    def run(): Unit = { Thread.sleep(ms); c.succeed(value) }
  })
  t.setDaemon(true)
  t.start()
  c.peek
}

val ordered: Async[List[Int]] = Async.collectAll(List[Async[Int]](
  delayed(1, 90),  // finishes third
  delayed(2, 45),  // finishes second
  delayed(3, 5)    // finishes first
))

// Completion order is 3, 2, 1 — the list is still 1, 2, 3.
val results: List[Int] = ordered.block  // => List(1, 2, 3)
```

Without that guarantee you would have to tag each computation and re-sort the results yourself. Because `collectAll` keeps the positions, you can zip the output against the input list — or pattern-match on it positionally — and trust that element *n* belongs to computation *n*.

## Operations

Everything you can do with an `Async[A]`: make one, transform it, combine it with another, recover from a failure, and eventually run it. 

One property is worth carrying into the signatures below. `Async[A]` is declared `Async[+A]`, which makes it *covariant*: whenever `B` is a subtype of `A`, an `Async[B]` counts as an `Async[A]`. That matters because `Async.fail` and `Async.never` have no value to offer, so their type is `Async[Nothing]` — and `Nothing` is a subtype of every type in Scala. An `Async[Nothing]` is therefore an `Async[String]`, an `Async[User]`, an `Async` of anything at all:

```scala

case class User(id: Int, name: String)

def fetchUser(id: Int): Async[User] =
  if (id < 0) Async.fail(new IllegalArgumentException("bad id"))  // Async[Nothing]
  else Async.succeed(User(id, "sam"))                             // Async[User]

val forever: Async[User] = Async.never                            // Async[Nothing] fits too
```

Without covariance neither of those would compile against the declared `Async[User]`, and you would be writing `Async.fail[User](…)` or a cast at every failure. This is a property you notice only through the errors it saves you from.

### Creating Values

The companion object provides factories for constructing leaf `Async[A]` values:

```scala
object Async {
  def succeed[A](a: A): Async[A]
  def fail(cause: Throwable): Async[Nothing]
  def attempt[A](body: => A): Async[A]
  def promise[A](body: Completer[A] => Unit): Async[A]  // shape differs on Scala 3; see Completer
  def start[A](body: => A): Async.Running[A]
  val never: Async[Nothing]
  def collectAll[A](as: IterableOnce[Async[A]]): Async[List[A]]
  def async[A](body: A): Async[A]                       // rewritten in place: a macro on Scala 2,
                                                        // a transparent inline def on Scala 3.
                                                        // See the Direct Style pattern

  // JVM only
  def fromFuture[A](future: scala.concurrent.Future[A]): Async[A]
  def fromCompletionStage[A](cs: java.util.concurrent.CompletionStage[A]): Async[A]
}
```

`Async.succeed` lifts a pure, immediately-available value into an `Async[A]`:

```scala

val ready: Async[Int] = Async.succeed(42)
val result: Int = ready.block  // => 42
```

`Async.fail` creates a terminal failure; [`Failure`](#failure) covers how it short-circuits the rest of a chain:

```scala

val boom: Async[Int] = Async.fail(new RuntimeException("boom"))
val result: Int = boom.catchAll(_ => Async.succeed(-1)).block  // => -1
```

`Async.attempt` captures a by-name expression and converts any thrown `Throwable` into a failure:

```scala

val parsed: Async[Int] = Async.attempt("42".toInt)
val bad: Async[Int]    = Async.attempt("nope".toInt)  // => Async.fail(NumberFormatException)
val result: Int        = bad.catchAll(_ => Async.succeed(0)).block  // => 0
```

`Async.never` is a permanently-suspended `Async[Nothing]` — a placeholder where an `Async[A]` is required but no value should ever arrive, and the usual way to test cancellation, as shown under [`Async.Running`](#asyncrunning).

`Async.start(body: => A)` runs a body on a background worker and hands back an `Async.Running[A]`; [Concurrent Fan-Out](#concurrent-fan-out-via-running) covers when to reach for it and the trap to avoid:

```scala

val running: Async.Running[Int] = Async.start { 42 }
val result: Int = running.block  // => 42
```

### Transformation

Pure transformations apply a function to the success value and return a new `Async`:

```scala
implicit class AsyncOps[A](fa: Async[A]) {
  def map[B](f: A => B): Async[B]
  def flatMap[B](f: A => Async[B]): Async[B]
  def as[B](b: B): Async[B]
  def unit: Async[Unit]
}

// flatten is a separate extension, on a nested Async:
implicit class AsyncNestedOps[A](ffa: Async[Async[A]]) {
  def flatten: Async[A]  // collapses one nesting level
}
```

`map` applies a pure function and `flatMap` sequences a dependent second computation:

```scala

val result: Async[String] =
  Async.succeed(21)
    .map(_ * 2)
    .flatMap(n => Async.succeed(s"value: $n"))
val out: String = result.block  // => "value: 42"
```

### Composition

Compositional operators combine independent or dependent `Async` values:

```scala
implicit class AsyncOps[A](fa: Async[A]) {
  def zipWith[B, C](that: Async[B])(f: (A, B) => C): Async[C]
  def zip[B](that: Async[B])(implicit t: Tuples[A, B]): Async[t.Out]  // flattens; see below
  def tap(f: A => Async[Any]): Async[A]
  def ensuring(finalizer: Async[Any]): Async[A]
  def *>[B](that: Async[B]): Async[B]
  def <*[B](that: Async[B]): Async[A]
  def orElse[B](that: => Async[B]): Async[_]  // result type merges A and B via Concat typeclass
}
```

`zipWith` waits for both sides and combines their results; `tap` runs a side-effecting action while passing the original value through. `zip` pairs the two results, and chains of it stay flat rather than nesting — `a zip b zip c` yields `Async[(A, B, C)]`, not `Async[((A, B), C)]` — because it combines through the `Tuples` instances described in the [combinators reference](combinators.md):

```scala

val combined: Async[Int] =
  Async.succeed(3).zipWith(Async.succeed(4))(_ + _)
val tapped: Async[Int] =
  combined.tap(v => Async.attempt(println(s"sum is $v")))
val result: Int = tapped.block  // => 7; "sum is 7" was printed by the line above
```

`*>` and `<*` sequence two effects and discard the left or right result respectively:

```scala

val logged: Async[Int] =
  Async.attempt(println("starting")).*>(Async.succeed(42))
val result: Int = logged.block  // => 42
```

### Error Handling

`Async` represents failure as a `Throwable` and provides dedicated recovery operators:

```scala
implicit class AsyncOps[A](fa: Async[A]) {
  def catchAll[A1 >: A](f: Throwable => Async[A1]): Async[A1]
  def mapError(f: Throwable => Throwable): Async[A]
  def foldCause[B](onFailure: Throwable => B)(onSuccess: A => B): Async[B]
  def either: Async[Either[Throwable, A]]
}
```

`catchAll` recovers from any failure by supplying a replacement `Async[A]`; `either` converts the outcome to an `Either` so the failure surface is visible in the return type:

```scala

val safe: Async[Either[Throwable, Int]] =
  Async.fail(new Exception("oops")).either
val result: Either[Throwable, Int] = safe.block  // => Left(Exception("oops"))
```

`foldCause` handles both the success and failure branches in a single call without allocating a recovery `Async`:

```scala

val message: Async[String] =
  Async.attempt("42".toInt).foldCause(
    (err: Throwable) => s"failed: ${err.getMessage}"
  )(
    (n: Int) => s"parsed: $n"
  )
val result: String = message.block  // => "parsed: 42"
```

### Driving

Driving settles whatever part of an `Async[A]` is still waiting, and delivers the result through one of three mechanisms:

```scala
implicit class AsyncOps[A](fa: Async[A]) {
  def block: A                                        // parks calling thread; re-throws on failure
  def await: A                                        // inside Async.async { } only; the rewrite
                                                      // removes it. Using it elsewhere does not compile
  def start: Async.Running[A]
  def toFuture(implicit ec: scala.concurrent.ExecutionContext): scala.concurrent.Future[A]
  def toCompletableFuture(implicit ec: scala.concurrent.ExecutionContext)
    : java.util.concurrent.CompletableFuture[A]       // JVM only
}
```

`block` parks the calling thread until the computation settles, then returns the value or re-throws the underlying `Throwable`:

```scala

val result: Int = Async.succeed(42).map(_ + 1).block  // => 43
```

Two limits apply. On the JVM, never call `block` from inside a `poll` — you would be putting to sleep the very thread that has to deliver your result, which deadlocks the loop. Keep it at the edge of your program: `main`, a test, the boundary with synchronous code.

On Scala.js there is no thread to park at all. A ready value returns as usual, but a *pending* one gets a single chance to complete synchronously, and if it has not, `block` throws `IllegalStateException`. Scala.js code should reach for `toFuture` or `toJsPromise` and let the event loop deliver the result, or stay inside `Async.async { … }` and use `await`.

`start` hands whatever is still waiting to a *driver* and returns an `Async.Running[A]` immediately, without blocking. The driver is the thing that keeps asking a pending value for its result. On the JVM it is a chain of serialized tasks on `ForkJoinPool.commonPool()`; on Scala.js it is the microtask queue. Neither holds a thread for the duration: a task exits as soon as its current pollable is still pending, and the waker that `poll` registered submits the next task when there is something new to see.

`Async.start(body)` is the other entry point and it is not the same mechanism. It takes a block of ordinary synchronous code rather than an `Async`, so it needs somewhere to run that block: on the JVM a dedicated daemon thread named `zio-blocks-async-eval`, and on Scala.js the next microtask.

```scala

def compute(): Int = 42

val running: Async.Running[Int] = Async.start { compute() }
val result: Int = running.block  // wait here for the result
```

`toFuture` hands off to a `scala.concurrent.Future`, bridging into any code that already expects the standard-library async type:

```scala

val future: scala.concurrent.Future[Int] =
  Async.succeed(99).toFuture
```

### Conditional Execution

`when` and `unless` are package-level functions (brought in by `import zio.blocks.async._`) that conditionally evaluate an `Async[Any]` based on a `Boolean` condition. The unevaluated branch is passed by name so no `Async` is constructed when the condition is false:

```scala

val flag = true
val logged:  Async[Unit] = when(flag)(Async.attempt(println("running")))
val skipped: Async[Unit] = unless(flag)(Async.attempt(println("skipped")))
```

## Common Patterns

The five patterns below address the most frequent tasks: bridging callbacks, writing sequential-looking code, guaranteeing cleanup, sharing in-flight computations, and collecting parallel results.

### Callback Bridge

Callback-based APIs all share one shape. Instead of returning the result, they return `Unit` immediately and call one of two functions you hand them once the work is done:

```scala
// The API you are stuck with. A real one calls back later, from
// another thread; the shape is what matters here.
def legacyApi(onSuccess: String => Unit, onError: Throwable => Unit): Unit =
  onSuccess("done")
```

That signature is the problem. Because `legacyApi` returns `Unit`, there is no value to return from your own function, nothing to pass to another function, and no way to say "do this, then that" — the result only ever appears inside a callback body, so the rest of your program has to be written in there too.

`Async.promise` inverts it. It gives you a `Completer[A]`, which is a value that can be completed later, and hands you back an `Async[A]` representing the eventual result. You pass the completer's two methods where `legacyApi` expects its two functions. The body is written slightly differently on each Scala version — `c =>` on Scala 2, `c ?=>` on Scala 3, for the reason explained under [`Completer`](#completer):

<Tabs groupId="scala-version" defaultValue="scala2">
<TabItem value="scala2" label="Scala 2">

```scala

def legacyApi(onSuccess: String => Unit, onError: Throwable => Unit): Unit =
  onSuccess("done")

val async: Async[String] = Async.promise[String] { c =>
  legacyApi(
    result => c.succeed(result),
    err    => c.fail(err)
  )
}
val result: String = async.block  // waits for the callback; this stub already fired
```

</TabItem>
<TabItem value="scala3" label="Scala 3">

```scala

def legacyApi(onSuccess: String => Unit, onError: Throwable => Unit): Unit =
  onSuccess("done")

val async: Async[String] = Async.promise[String] { c ?=>
  legacyApi(
    result => c.succeed(result),
    err    => c.fail(err)
  )
}
val result: String = async.block  // waits for the callback; this stub already fired
```

</TabItem>
</Tabs>

The two lines inside `legacyApi` are the whole bridge: whichever callback fires, it completes `c`, and completing `c` completes the `Async[String]`. Note what has been gained — `async` is an ordinary value. You can return it, store it, or chain `map` and `flatMap` onto it, and the callback API is no longer visible to anything downstream.

The bridge is also safe against a callback that fires more than once — see [`Completer`](#completer).

### Direct Style

Inside `Async.async { ... }`, use `await` to extract values from `Async` computations in sequential-looking code without explicit `flatMap` chains:

```scala

case class Order(id: Int, userId: Int)
case class User(id: Int, name: String, tier: String)
case class Shipment(orderId: Int, carrier: String)

def fetchOrder(id: Int): Async[Order]      = Async.succeed(Order(id, 1))
def fetchUser(id: Int): Async[User]        = Async.succeed(User(id, "sam", "gold"))
def fulfill(orderId: Int): Async[Shipment] = Async.succeed(Shipment(orderId, "express"))

def fulfillOrGuest(orderId: Int): Async[String] = Async.async {
  val order    = fetchOrder(orderId).catchAll(_ => fetchOrder(9001)).await
  val user     = fetchUser(order.userId)
                   .catchAll(_ => Async.succeed(User(0, "guest", "bronze"))).await
  val shipment = fulfill(order.id).await
  s"shipped ${shipment.orderId} for ${user.name} via ${shipment.carrier}"
}
val result: String = fulfillOrGuest(9001).block
```

Awaits run in source order; a failed `Async[A]` under `await` propagates as `Async.fail`.

Nothing new happens at runtime here. `Async.async` rewrites its body at compile time: the block is split at each `await` and reassembled into the `flatMap` chain you would have written by hand. The example above compiles to roughly this:

```scala
fetchOrder(orderId).catchAll(_ => fetchOrder(9001)).flatMap { order =>
  fetchUser(order.userId)
    .catchAll(_ => Async.succeed(User(0, "guest", "bronze")))
    .flatMap { user =>
      fulfill(order.id).map { shipment =>
        s"shipped ${shipment.orderId} for ${user.name} via ${shipment.carrier}"
      }
    }
}
```

So `await` is not a method that blocks or waits. It is a marker the rewrite removes, and everything after it becomes the continuation that runs once the value arrives. Direct style therefore costs nothing over writing the chain yourself — by the time the code runs, it *is* that chain. Choose whichever reads better.

One consequence is worth remembering: `await` only means something inside an `Async.async` block. Elsewhere there is no rewrite to remove it, and both Scala versions reject it at compile time — Scala 2 through `@compileTimeOnly`, Scala 3 by aborting the macro expansion. You will not ship this mistake.

Within the block, `await` is not restricted to statement position. It also works inside the closures you pass to the strict collections — `List`, `Option`, `Vector`, `Set`, `Map`, `Array`, `Queue`, `ArraySeq` — for `map`, `foreach`, `flatMap`, `filter`, `filterNot`, `collect`, `find`, `exists`, `forall`, `foldLeft`, `foldRight`, `reduce` and `reduceLeft`, and in for-comprehensions over them:

```scala

def fetchName(id: Int): Async[String] = Async.succeed(s"user-$id")

val names: Async[List[String]] = Async.async {
  List(1, 2, 3).map(id => fetchName(id).await)
}
```

Read that list literally; the near neighbours are not all included. `reduceRight` and `reduceOption` are not, and neither is the two-argument `fold`. `takeWhile` and `dropWhile` are, but only over an ordered receiver — `List`, `Vector`, `Queue`, `ArraySeq` or `Array` — because the prefix they compute is meaningless on a `Set` or a `Map`. And two cases surprise people: `Map.filter` with an `await` inside works on Scala 2 only, and a `Map.collect` whose closure yields a pair is unsupported everywhere. Lazy collections are outside the set entirely — force them to a strict collection first.

The semantics are nearly uniform. Each method is lazy and sequential — the closure for element *n+1* runs only after element *n*'s `await` has completed — with one exception: `map` over a `List` or an `Array` applies the closure to every element first, so every call in it happens before the first `await` resolves, and only then awaits the results left to right, failing fast. A `List(a, b, c).map(fetch(_).await)` therefore starts all three fetches before it waits on the first, while `Vector(a, b, c).map(fetch(_).await)` performs them one after another. A failed `await` short-circuits the remaining awaits, and the result keeps the receiver's collection type. Use `Async.collectAll` when you want lazy receivers' calls overlapped.

To see the order, the following program records every call and fails the second one. `List.map` and `Array.map` call the function for all three SKUs, while `Vector.map` stops after the failure:

```scala

val fetchCalls = ListBuffer.empty[String]
// fetchCalls: ListBuffer[String] = ListBuffer("fetch 1", "fetch 2")

def fetchPrice(sku: Int): Async[Int] = {
  fetchCalls += s"fetch $sku"
  if (sku == 2) Async.fail(new IllegalStateException(s"sku $sku: price service down"))
  else Async.succeed(sku * 10)
}

def traceCalls(label: String)(body: => Async[Any]): Unit = {
  fetchCalls.clear()
  val result = body.either.block
  println(s"$label: $result; ${fetchCalls.mkString(", ")}")
}

traceCalls("List.map  ")(Async.async(List(1, 2, 3).map(sku => fetchPrice(sku).await)))
// List.map  : Left(java.lang.IllegalStateException: sku 2: price service down); fetch 1, fetch 2, fetch 3
traceCalls("Array.map ")(Async.async(Array(1, 2, 3).map(sku => fetchPrice(sku).await).toList))
// Array.map : Left(java.lang.IllegalStateException: sku 2: price service down); fetch 1, fetch 2, fetch 3
traceCalls("Vector.map")(Async.async(Vector(1, 2, 3).map(sku => fetchPrice(sku).await)))
// Vector.map: Left(java.lang.IllegalStateException: sku 2: price service down); fetch 1, fetch 2
```

Where a collection method would throw on its own, it still does: `reduce` over an empty receiver fails with `UnsupportedOperationException`, which arrives as an ordinary `Async` failure you can `catchAll`.

The rewrite is performed by `dotty-cps-async` on Scala 3 and by a built-in `scala-reflect` macro on Scala 2.13. Both Scala versions support direct style, and neither asks you to add anything to your build.

Scala.js 3.8 and later takes a hybrid route, decided per call site. An `await` in direct position compiles to JavaScript's own `async`/`await`, which is the fastest path available; an `await` sitting under a lambda, a by-name argument, or a nested method falls back to the `dotty-cps-async` transform, because the native primitive is not legal in those positions. Nothing about this is yours to configure — the wider `Async.async` surface works either way.

### Bracket and Ensuring

Some work has to happen no matter what: closing a file, releasing a connection, deleting a temporary directory. In ordinary code you write that in a `finally` block. `ensuring` is the same idea for `Async`: you attach a cleanup value, and its *outcome* is applied once the computation settles, whether that produced a value or a failure. Note "value", not "thunk" — `ensuring` takes an `Async`, and under the [evaluation model](#evaluation-model) building one runs its synchronous part immediately:

```scala

val result: Async[String] =
  Async.attempt(openResource()).flatMap { res =>
    Async.attempt(res.read()).ensuring(Async.attempt(res.close()))
  }
```

That example is safe only because `Async.attempt(res.read())` is already finished by the time `ensuring` is reached. Written over something that genuinely waits, the same shape closes the resource while the read is still in flight:

```scala
// WRONG: res.close() runs here, as the argument is built —
// not after the read completes.
readAsync(res).ensuring(Async.attempt(res.close()))
```

To defer the effect itself, put it somewhere that is only run when driven — a `flatMap` or `tap` closure, or a finalizer that is genuinely suspended:

```scala
readAsync(res).flatMap(v => Async.attempt(res.close()).as(v))
```

What `ensuring` does guarantee is the rule worth remembering: **the cleanup never changes the answer.** It cannot turn a failure into a success, and it cannot turn a success into a failure.

That last part raises an obvious question — what if the cleanup itself fails? Closing a file can throw too. The answer depends on how the main computation ended, so it is worth seeing both cases:

```scala

// Both fail: reading the resource, and then closing it.
val bothFail: Async[String] =
  Async
    .attempt[String](throw new RuntimeException("read failed"))
    .ensuring(Async.attempt(throw new IllegalStateException("close failed")))

bothFail.either.block match {
  case Left(e) =>
    println(e.getMessage)                     // read failed   <- the original failure
    println(e.getSuppressed()(0).getMessage)  // close failed  <- attached to it
  case Right(_) => ()
}

// Only the cleanup fails.
val readOk: Async[String] =
  Async
    .succeed("contents")
    .ensuring(Async.attempt(throw new IllegalStateException("close failed")))

val value: String = readOk.block  // "contents" — the close failure is gone
```

In the first case the read had already failed, so you get the read's exception — the one that explains what actually went wrong. The close error is not thrown away, though: it is carried along inside that exception, in a list the JVM keeps for exactly this purpose. `getSuppressed` returns that list. Your logging framework almost certainly prints it, usually under a line beginning `Suppressed:`, so both problems end up on the page.

The second case is the one to watch. The read succeeded, so there is no exception to carry the close error, and it is simply dropped — `readOk.block` returns `"contents"` and you never hear that closing failed. If a cleanup error matters to you on the success path, catch it inside the cleanup step itself and log it there:

```scala
Async
  .attempt(res.read())
  .ensuring(Async.attempt(res.close()).catchAll { t =>
    Async.succeed(logger.warn("close failed", t))
  })
```

### Concurrent Fan-Out via Running

Suppose one expensive computation feeds several parts of your program — a report that is both summarised and emailed, say. The obvious approach is to build the work once and use that value in both places — but driving it is what produces the result, so each place that drives it does the work again. You want the work to happen once, in the background, with everyone reading the same outcome.

`Async.start` does that. It hands the body to a background worker and returns immediately with an `Async.Running[A]` — a handle to work already in flight:

```scala

def heavyComputation(): Int = { Thread.sleep(50); 42 }

// Returns straight away; the work proceeds on a background worker.
val running: Async.Running[Int] = Async.start(heavyComputation())

// The handle is itself an Async[Int], so it composes like anything else.
val doubled: Async[Int]    = running.map(_ * 2)
val labelled: Async[String] = running.map(n => s"got $n")

val a: Int    = doubled.block   // 84
val b: String = labelled.block  // "got 42" — heavyComputation ran once, not twice
```

Both consumers see the same settled outcome, because they share one running computation rather than one recipe. `Async.Running[A]` is a subtype of `Async[A]`, so it works with `map`, `flatMap`, and `zipWith` without conversion, and `running.cancel()` stops the driver, which is [less than it sounds](#cancelable).

Sharing the handle is not merely tidier than the alternative — the alternative is unsafe. Driving the same raw `Async` from two places at once is **undefined behaviour**: two `fa.start` calls on the same `fa`, or an `fa.start` racing an `fa.block`. On the JVM that polls the same combinator concurrently, so a function you passed to `map`, `flatMap`, or `tap` may run more than once, and a `collectAll` may read its drain buffer mid-update. Start once, share the `Running`.

Sequential re-use is fine — polling or composing a value again after an earlier drive has settled is well defined. Only *concurrent* driving of the same raw value is not, and single-threaded Scala.js cannot hit it at all.

Take care to start the work the right way round, because the wrong version looks almost identical:

```scala
Async.start(heavyComputation())            // ✅ the worker evaluates it
Async.attempt(heavyComputation()).start    // ❌ already evaluated, on this thread
```

Both lines compile, and both hand you an `Async.Running[Int]`. Only the first one runs anything in the background.

The difference is *when the argument gets evaluated*. Scala normally evaluates an argument before passing it, so in `Async.attempt(heavyComputation())` the computation runs first — on your own thread, right at that line — and `attempt` merely wraps the answer it produced. Tacking `.start` on afterwards cannot un-run it; there is nothing left to move to a worker.

`Async.start` is declared differently. Its parameter is `body: => A`, and that `=>` means "don't evaluate this yet — hand me the code and I will run it when I am ready." It passes the code to a worker thread, which is why the call returns immediately.

The clock shows it plainly:

```scala
Async.start(heavyComputation())            // returns in about 0 ms
Async.attempt(heavyComputation()).start    // returns in about 50 ms — you waited for it
```

So: use `Async.start` for work you want moved off the calling thread. Use `fa.start` when `fa` is an `Async` you have already built and composed and now want driven.

### Batch Collection

Use `Async.collectAll` to sequence a list of `Async` values and gather results into a `List[A]` in input order:

```scala

val batch: Async[List[Int]] = Async.collectAll(List(
  Async.attempt(compute(1)),
  Async.attempt(compute(2)),
  Async.attempt(compute(3))
))
val results: List[Int] = batch.block  // => List(r1, r2, r3) in input order
```

The first failure short-circuits and remaining elements are not driven. Already-ready lists take an optimized path that skips allocating a sequencing continuation.

## Integration Points

You are unlikely to be starting from scratch. Your codebase probably already returns `Future`s, calls a Java library that returns a `CompletionStage`, or talks to a JavaScript API that returns a `Promise`. `Async` is built to sit next to those, so you can adopt it in one part of a program without rewriting everything around it.

Conversions go in both directions, and none of them blocks a thread:

| You have                   | Bring it in with                | You need                     | Hand it out with         |
|----------------------------|---------------------------------|------------------------------|--------------------------|
| `Future[A]`                | `Async.fromFuture(f)`           | `Future[A]`                  | `fa.toFuture`            |
| `CompletionStage[A]` (JVM) | `Async.fromCompletionStage(cs)` | `CompletableFuture[A]` (JVM) | `fa.toCompletableFuture` |
| `js.Promise[A]` (Scala.js) | `Async.fromJsPromise(p)`        | `js.Promise[A]` (Scala.js)   | `fa.toJsPromise`         |

A round trip through `Future` looks like this — take what an existing service hands you, work with it as an `Async`, and give a `Future` back to a caller who still expects one:

```scala

// The service you already have.
def loadUserName(id: Int): Future[String] = Future.successful("sam")

// Bring it in, work with it as an Async, hand a Future back out.
def greet(id: Int): Future[String] =
  Async
    .fromFuture(loadUserName(id))
    .map(name => s"hello, $name")
    .catchAll(_ => Async.succeed("hello, guest"))
    .toFuture
```

Java's `CompletionStage` works the same way:

```scala

def fetchToken(): CompletionStage[String] = CompletableFuture.completedFuture("t-123")

val token: Async[String]                 = Async.fromCompletionStage(fetchToken())
val backToJava: CompletableFuture[String] = token.map(_.toUpperCase).toCompletableFuture
```

On Scala.js the pair is `fromJsPromise` and `toJsPromise`:

```scala

def fetchJson(url: String): js.Promise[String] = js.native

val parsed: Async[String]       = Async.fromJsPromise(fetchJson("/api/config"))
val handedBack: js.Promise[String] = parsed.map(_.trim).toJsPromise
```

Two details are worth knowing before you use them.

Handing a value out to `Future` or `CompletableFuture` needs an `ExecutionContext` in scope, exactly as ordinary `Future` code does — usually `import scala.concurrent.ExecutionContext.Implicits.global`, as above, or whichever one your application already provides. `toJsPromise` needs nothing, because JavaScript has a single built-in event loop to run the callback on.

Failures survive the trip. That takes some care on the Java side: when a `CompletionStage` fails, Java wraps your exception in a `CompletionException` before handing it over. `fromCompletionStage` unwraps it, so the `Async` fails with the exception you actually threw rather than with Java's wrapper — which means `catchAll` sees what you expect.

Two further integration points are worth knowing about:

**Cancelling with `Using`.** `Async.Running` is an `AutoCloseable` — `Cancelable` extends it — so `scala.util.Using` (or Java's try-with-resources) cancels the work automatically when the block ends, the same way it closes a file handle:

```scala

def pollForUpdates(): Nothing = { while (true) Thread.sleep(100); ??? }

Using(Async.start(pollForUpdates())) { running =>
  // Do other work while the poller runs.
  Thread.sleep(500)
} // leaving the block cancels the driver, whether or not the body threw —
  // the loop itself keeps running; see Cancelable
```

The [Scope reference](resource-management/scope.md) covers the wider resource-management model.

**Feeding [streams](streams/core/stream.md).** A callback-based source can be turned into a stream with `Async.promise` and a `Completer`. Because a stream pulls values as it is ready for them, a source that produces faster than the consumer can handle will not overwhelm it.

## Custom Suspension

A result that is not here yet arrives in one of two ways: either something tells you when it is ready, or you have to keep asking. This module has a type for each. `Completer[A]` covers being told, and it is the one you will almost always want. `Pollable[A]` covers having to ask, and exists for the sources that leave you no choice.

### Pollable

Imagine waiting on something that never calls you back — a non-blocking socket that answers "no data yet" when you read it, a hardware timer you have to check, a native handle that reports progress only when asked. There is no callback to hand a `Completer` to. The only way to learn whether the result has arrived is to *ask*, and to keep asking. The module cannot know how to ask your particular source; only your code knows that.

`Pollable[A]` is where you supply that knowledge. It is a single method:

```scala
abstract class Pollable[+A] {
  def poll(onComplete: Runnable): Async[A]
}
```

The driver calls `poll` whenever it gets the chance, and what you return tells it what to do next:

- **Ready?** Return `Async.succeed(a)`. The driver takes the value and stops asking.
- **Not yet?** Return `this` — *and make sure `onComplete` will be run*. The driver does not come back on its own.
- **Failed?** Return `Async.fail(t)`. The driver takes the failure and stops asking, and it travels downstream like any other failure.

`onComplete` is not an optimisation — it is the only thing that gets you polled again. After a `poll` returns `this`, the driver parks and waits for that callback; if nothing ever runs it, a `block` waits forever on the JVM and throws `IllegalStateException` on Scala.js. `Async.never` is precisely a `poll` that returns `this` and never arms it. Either run `onComplete` before returning, or hand it to whatever will know when to check again.

Most programs never need any of this. For a callback-based API, `Async.promise` with a `Completer` is simpler and already correct; for blocking I/O, `Async.attempt` on a worker via `Async.start` fits better. Reach for `Pollable` only when the result genuinely has to be checked rather than delivered.

Here is the smallest thing that behaves like a real suspension — a value that refuses to be ready for its first two visits:

```scala

class Delayed[A](v: A, var ticks: Int) extends Pollable[A] {
  def poll(onComplete: Runnable): Async[A] =
    if (ticks <= 0) Async.succeed(v)
    else { ticks -= 1; onComplete.run(); this }
}

val result: String = new Delayed("done", ticks = 2).block  // => "done"
```

Follow it one visit at a time:

| Visit | `ticks` | What `poll` returns     | What the driver does          |
|-------|--------:|-------------------------|-------------------------------|
| 1st   |       2 | `this`                  | Not ready — asks again        |
| 2nd   |       1 | `this`                  | Not ready — asks again        |
| 3rd   |       0 | `Async.succeed("done")` | Takes the value, stops asking |

Returning `this` means "still me, still waiting." Returning `Async.succeed(v)` means "here it is." And `onComplete.run()` is the nudge that tells the driver to come back soon rather than in its own time. The toy above runs it immediately, which just asks for another visit right away; real code instead hands `onComplete` to whatever it is waiting on — a socket selector, a timer callback — and lets that source run it when something actually happens. The driver can then stay asleep in between, rather than burning a thread asking a question whose answer has not changed. Meanwhile `.block` waits through all three visits and hands you `"done"` at the end.

A real implementation replaces the counter with the actual question — has the socket got bytes, has the timer expired — but the shape does not change.

Here is a case you are likely to meet. A service starts a long job — rendering a report, transcoding a video, restoring an archive — and gives you back a job id. There is no webhook and no callback: the only way to find out whether it has finished is to call `GET /jobs/{id}` and look at the status. That is a `Pollable`:

```scala

sealed trait JobStatus
case object Pending                 extends JobStatus
case class  Done(url: String)       extends JobStatus
case class  Failed(reason: String)  extends JobStatus

// The API you are given: you can ask it, it will never tell you.
def checkJob(id: String): JobStatus = Done("https://example.invalid/report.pdf")

def download(url: String): Unit = ()

// Run `task` once, later, without holding on to a thread in the meantime.
def scheduleIn(ms: Long, task: Runnable): Unit = {
  val t = new Thread(() => { Thread.sleep(ms); task.run() })
  t.setDaemon(true)
  t.start()
}

final class JobPollable(id: String) extends Pollable[String] {
  def poll(onComplete: Runnable): Async[String] =
    checkJob(id) match {
      case Done(url)      => Async.succeed(url)
      case Failed(reason) => Async.fail(new RuntimeException(s"job $id failed: $reason"))
      case Pending        =>
        // Nothing will announce the change, so arrange our own next look.
        scheduleIn(2000, onComplete)
        this
    }
}

// From here on it is an ordinary Async: compose it, start it, cancel it.
val reportUrl: Async[String] = new JobPollable("job-42")
val saved: Async[Unit]       = reportUrl.map(url => download(url))
```

Three things to notice. The status check happens inside `poll`, so it runs only when the driver visits — you are not running a loop of your own. The `Pending` branch schedules the next visit two seconds out, which is what stops this from hammering the service. And a failed job becomes `Async.fail`, so the error travels the same path as every other failure and `catchAll` can recover it.

Note also what is *not* in the example: no blocking wait, no lock, no shared mutable state. Polling puts you in charge of when the check happens, which is the reason to choose it. Because a `Pollable[A]` can be used wherever an `Async[A]` is expected, the value drops straight into any composition and works with every combinator.

### The Pollable Protocol

Everything above is what a `Pollable` looks like from the outside. This is the contract a driver and an implementation hold each other to, and it matters only if you are on one side of it — writing a `poll`, or writing a driver of your own. If you are composing `Async` values and running them with `block`, `start`, or an interop converter, the built-in drivers already honour every rule below and you can skip the section.

**Polling is one-shot.** A driver keeps polling only while `poll` returns a still-pending `Pollable`, and must stop as soon as it returns a terminal result — a raw value, or a failed `Async`. Re-polling a `Pollable` after that is outside the contract: it is not guaranteed to be a pure re-observation, a continuation may run a second time, and diagnostic state may be repeated.

**Identity carries meaning.** A still-pending result is either this pollable itself — the common case — or a *replacement* pollable standing for the rest of the computation, and the caller directs its next poll at whatever came back. Which of the two it is decides what the caller may do next:

| What `poll` returns          | What it means                       | What the driver does next              |
|------------------------------|-------------------------------------|----------------------------------------|
| `this` — the same identity   | Pending; `onComplete` is registered | May wait for that callback             |
| A different `Pollable`       | Synchronous progress, not readiness | Polls the replacement, without waiting |
| A value, or a failed `Async` | Terminal                            | Stops polling                          |

The middle row is the one that trips people up. An identity change is *not* a readiness event, so a caller or a wrapper must never synthesize an `onComplete` call for it — it means "there is more to do right now", and waiting on a callback that nobody will run is how such a wrapper hangs.

**Wakes are permits, not proofs.** Real callbacks may be stale, reentrant, or duplicated. `onComplete` is therefore a coalescible wake permit — "look again" — rather than proof that any particular generation has completed. A driver re-polls and lets `poll` decide; an implementation is free to run `onComplete` more times than strictly necessary without breaking anything, though never fewer.

**Cancellation reaches the leaf.** Alongside `poll`, a `Pollable` may override `cancel()`, which the driver signals on the active pending operation when cancellation wins the race against completion. The default is a no-op, so a source with abortable work has to supply it — and it must be idempotent and non-blocking, because cancellation never waits:

```scala

// `register` stands in for handing the waker to the real source: a socket
// selector, a timer, a third-party library's callback slot.
final class AbortableRead(register: Runnable => Unit) extends Pollable[Array[Byte]] {
  private val aborted = new AtomicBoolean(false)

  def poll(onComplete: Runnable): Async[Array[Byte]] =
    if (aborted.get()) Async.fail(new java.io.IOException("read aborted"))
    else { register(onComplete); this }

  // Idempotent and non-blocking: it records the intent and returns. The
  // driver has already stopped listening by the time this runs.
  override def cancel(): Unit = aborted.set(true)
}
```

The full contract lives in the scaladoc of `Pollable#poll`; [`Cancelable`](#cancelable) covers what cancelling does and does not stop.

### Completer

Polling is the awkward case. Far more often the source does call you back — that is what `Completer[A]` is for, and why you will reach for it and not `Pollable`.

`Completer[A]` is a `Pollable[A]` that is already written: instead of implementing "is it ready?", you hold a value someone else completes exactly once. It is thread-safe, and the first call to `Completer#succeed` or `Completer#fail` wins while every later call does nothing — so a callback that fires twice cannot corrupt the result.

The structural declaration is:

```scala
final class Completer[A] extends Pollable[A] {
  def succeed(a: A): Unit
  def fail(cause: Throwable): Unit
  def peek: Async[A]
  def poll(onComplete: Runnable): Async[A]
}
```

`Async.promise` creates a new `Completer[A]`, passes it to the body, and returns the `Completer` as an `Async[A]` that the driver polls until the callback fires. If the body happens to complete it before returning — a cache hit, a callback that fires inline — the result is a plain ready value: drivers and combinators see no `Pollable`. It is the only place in the module where the *code you write* differs between Scala versions — elsewhere the signatures differ but the call sites are identical:

```scala
// Scala 2 — the completer is an ordinary function parameter
def promise[A](body: Completer[A] => Unit): Async[A]

// Scala 3 — the completer is a context parameter of the body
inline def promise[A](inline body: Completer[A] ?=> Unit): Async[A]
```

The `?=>` on Scala 3 makes the `Completer` a given inside the body rather than a plain argument, which is why the body is written `{ c ?=> ... }` there and `{ c => ... }` on Scala 2. (The two `inline` keywords are what splice the body into the call site instead of allocating a function object — the same technique behind the allocation-free ready path described under [Evaluation Model](#evaluation-model).)

Being a given is not just bookkeeping: it buys you top-level `succeed` and `fail` helpers that find the completer themselves, so on Scala 3 the bridge need not name it at all.

```scala
// Scala 3 only — `succeed` and `fail` take the Completer as a given.
val fetched: Async[Int] = Async.promise[Int] { c ?=>
  legacyLookup(onOk = value => succeed(value), onErr = cause => fail(cause))
}
```

Naming it, as the examples below do, is equally valid and reads better when the callback is registered several lines away from where it fires.

:::warning[Do not write the Scala 2 binder on Scala 3]
On Scala 3, `Async.promise[Int] { (c: Completer[Int]) => ... }` compiles. The typed binder turns the body into an ordinary function value, which the context function discards, so its statements never run. The `Async` never completes, and the only sign is compiler warning E190 at the call site. Write `{ c ?=> ... }` on Scala 3, as in the example below.
:::

<Tabs groupId="scala-version" defaultValue="scala2">
<TabItem value="scala2" label="Scala 2">

```scala

val async: Async[Int] = Async.promise[Int] { c =>
  new Thread(() => { Thread.sleep(20); c.succeed(42) }).start()
}
val result: Int = async.block  // => 42
```

</TabItem>
<TabItem value="scala3" label="Scala 3">

```scala

val async: Async[Int] = Async.promise[Int] { c ?=>
  new Thread(() => { Thread.sleep(20); c.succeed(42) }).start()
}
val result: Int = async.block  // => 42
```

</TabItem>
</Tabs>

Now a case from the JDK rather than a sleeping thread. `AsynchronousFileChannel` reads a file without blocking, and reports the outcome through a `CompletionHandler` with two methods: `completed` when the bytes arrive, `failed` when the read goes wrong. Those two are exactly `succeed` and `fail`, so the bridge is almost mechanical:

```scala

def readChunk(path: Path, size: Int): Async[ByteBuffer] = {
  val completer = new Completer[ByteBuffer]
  val channel   = AsynchronousFileChannel.open(path, StandardOpenOption.READ)
  val buffer    = ByteBuffer.allocate(size)

  channel.read(buffer, 0L, buffer, new CompletionHandler[Integer, ByteBuffer] {
    def completed(bytesRead: Integer, buf: ByteBuffer): Unit = {
      buf.flip()
      completer.succeed(buf)   // the read finished
    }
    def failed(cause: Throwable, buf: ByteBuffer): Unit =
      completer.fail(cause)    // the read went wrong
  })

  completer.peek               // hand the pending result to the caller
}

// An ordinary Async from here on.
val firstBytes: Async[Int] = readChunk(Path.of("data.bin"), 1024).map(_.remaining)
```

This is the same bridge as `Async.promise`, written out by hand: create the `Completer`, give its two methods to the callback, and return `completer.peek` as the `Async[ByteBuffer]` the caller waits on. `Async.promise` packages exactly those three steps, so the same function written with it is shorter:

```scala

def readChunk(path: Path, size: Int): Async[ByteBuffer] =
  Async.promise[ByteBuffer] { c ?=>
    val channel = AsynchronousFileChannel.open(path, StandardOpenOption.READ)
    val buffer  = ByteBuffer.allocate(size)

    channel.read(buffer, 0L, buffer, new CompletionHandler[Integer, ByteBuffer] {
      def completed(bytesRead: Integer, buf: ByteBuffer): Unit = {
        buf.flip()
        c.succeed(buf)
      }
      def failed(cause: Throwable, buf: ByteBuffer): Unit =
        c.fail(cause)
    })
  }
```

The completer is created for you and named `c`, and there is no `peek` at the end — `promise` returns the `Async` itself.

Prefer this version. Write the completer out by hand when the registration does not fit neatly in a single block, when you need to keep the completer around to complete it from elsewhere, or when you want identical source on Scala 2 and Scala 3 — `new Completer[A]` has no context-function syntax to differ over.

`readChunk` returns before a single byte has been read. Nothing blocks, no thread waits, and the caller receives an `Async[ByteBuffer]` that behaves like any other — `map` it, `zipWith` another read, recover it with `catchAll`, or `block` on it at the edge of the program.

The once-only guarantee earns its keep in code like this. You are trusting a third-party library to call your handler correctly; if a buggy or retrying implementation calls `completed` twice, or calls both `completed` and `failed`, the first call still decides the outcome and the rest are ignored. You do not have to defend against it yourself.

`Completer#peek` returns the outcome if the completer is already complete, and the `Completer` itself — still pending — as an `Async[A]` otherwise, bypassing the `Async.promise` body — useful when managing scheduling manually, as shown in the `delayed` helper under [How They Work Together](#how-they-work-together).

## Controlling In-Flight Work

Once `start` has handed you an `Async.Running[A]`, the computation is being driven for you — by the platform driver if it still has waiting to do, and already settled if it does not. These two types are how you keep a grip on it: `Async.Running` is the handle, and `Cancelable` is the ability to stop what it refers to.

### Async.Running

`Async.Running[A]` is the handle returned by `start`. It extends `Pollable[A]`, which makes it an `Async[A]` in its own right, and [`Cancelable`](#cancelable), which is what lets you stop it.

The structural declaration is:

```scala
abstract class Running[+A] extends Pollable[A] with Cancelable
```

Because a `Running` is an `Async[A]`, you can wait for its result with `block`, compose it further with `map` or `flatMap`, or pass it anywhere an `Async[A]` is expected. [Concurrent Fan-Out via Running](#concurrent-fan-out-via-running) walks through that pattern, and why several consumers should share one handle rather than each starting the work themselves.

What you should not do is call `poll` yourself. It is there for drivers, and its contract — stop at a terminal value, never re-poll a settled one — is easy to violate by hand. Drive a `Running` the same way you drive any other `Async`: `block`, `toFuture`, or composition. There is no `isCompleted`; if you want to know whether it has finished without waiting, keep that flag yourself where you complete the work.

Calling `cancel` stops the *driver*, and does nothing if the run has already settled. [`Cancelable`](#cancelable) covers how far that reaches; two consequences belong here:

```scala

val running: Async.Running[Nothing] = Async.never.start
running.cancel()  // the driver stops polling; no value is ever published
```

A cancelled run never settles at all — it does not fail, it simply stops. So anything still holding that handle and calling `block` on it waits forever on the JVM, and gets an `IllegalStateException` on Scala.js. Cancel only when you own every consumer of the handle.

And with `Async.start(body)`, `cancel` stops the driver, not the evaluation of `body`. That block runs to completion regardless — on its own daemon thread on the JVM, on the next microtask on Scala.js — so cancelling that particular `Running` means you have stopped waiting for the result, not that the work behind it has stopped. A suspended `fa.start`, by contrast, does have its active leaf signalled; [`Cancelable`](#cancelable) draws the line.

A `Running` is also an `AutoCloseable`, so [`scala.util.Using`](#integration-points) cancels it on leaving a block.

`block` on a pending handle is available on the JVM only — see [Platform Support](#platform-support).

Attach what you want to observe *before* calling `start`, not after. `start` hands the still-waiting part of the value to a driver, and whatever you composed onto it beforehand is part of what that driver runs:

```scala

// A row that arrives a moment from now, on another thread.
def fetchRow(): Async[String] = Async.promise[String] { c ?=>
  val t = new Thread(() => { Thread.sleep(250); c.succeed("row-1") })
  t.setDaemon(true)
  t.start()
}
def log(msg: String): Unit = ()

// Before start: the tap is part of what the driver runs, and fires when the
// row arrives.
val watched: Async.Running[String] =
  fetchRow().tap(row => Async.attempt(log(s"got $row"))).start

// After start: this builds a *new* Async that nobody is driving. The tap runs
// only if you drive this one too — by blocking on it, or starting it.
val bolted: Async[String] =
  fetchRow().start.tap(row => Async.attempt(log(s"got $row")))
```

The second version is not a compile error and not a lost value, which is what makes it easy to write by mistake: `bolted` is simply a value nobody has driven, so its `tap` has not run and will not until something asks `bolted` for a result. So if you want to time the fetch, log its progress, or react the moment it fails, the observer has to be inside the value you hand to `start`.

`either` and `foldCause` matter more, because they decide whether the run counts as failed. Written `fa.either.start`, where `fa` is the value you are about to start, the run always succeeds, carrying a `Left` or a `Right`. Written `fa.start.either`, the run has already failed; you get your `Either`, but everyone else holding that handle still gets the exception.

All of that assumes there was something to wait for. If the value already holds its answer there is nothing to hand to a driver: `start` wraps it and returns, spawning no worker, and anything you attached ran while you were building the value — see [Evaluation Model](#evaluation-model).

### Running#cancel

A `Running` carries two `cancel` methods. One it inherits from [`Cancelable`](#cancelable) and takes nothing; the other takes a reporter:

```scala
abstract class Running[+A] extends Pollable[A] with Cancelable {
  def cancel(): Unit                                    // inherited from Cancelable
  def cancel(onCleanupFailure: Throwable => Unit): Unit
}
```

They cancel identically. What differs is where a failure thrown by *cleanup* ends up — and cleanup is the one thing cancellation can still fail at. Cancelling signals `cancel()` on the active leaf, and the teardown that follows may be asynchronous and may throw. That failure has nowhere natural to go: the run has been cancelled, so it will never deliver a result, and there is no channel left to carry an exception to whoever was waiting.

So it is reported instead. `cancel()` sends it to the ambient handler for the platform — the calling thread's `UncaughtExceptionHandler` on the JVM, the queue's failure reporter on Scala.js. `cancel(onCleanupFailure)` sends it to your function, which is what you want whenever "a socket refused to close" should reach a log or a metric rather than stderr:

```scala

val running: Async.Running[Nothing] = Async.never.start

running.cancel { cause =>
  System.err.println(s"cleanup after cancellation failed: ${cause.getMessage}")
}
```

Three properties are worth knowing before you rely on it:

- **The reporter is kept only if this cancellation wins.** If the run had already settled, or another `cancel` call claimed it first, your function is dropped and never invoked. A dropped reporter is not an error; it means there was no cancellation cleanup of yours to report on.
- **It is invoked at most once.** When cleanup fails in more than one place, the later causes are attached to the first as suppressed exceptions and the first is what you are handed — so check `getSuppressed` if you are logging the whole picture.
- **It may run on any thread.** Cleanup is driven by whichever party claims it, which is not necessarily the thread that called `cancel`. Keep the reporter short and make sure it cannot throw.

### Cancelable

`Cancelable` is the minimal cancellation interface: one `cancel()` method, safe to call from any thread and safe to call twice.

Be precise about what it stops. Cancellation does two things, and they reach different distances:

- **It stops the driver.** The poll loop halts and publication of a terminal value is suppressed. A cancelled run therefore never delivers at all — it does not fail, it simply stays pending forever, which is why anything still calling `block` on that handle waits indefinitely.
- **It signals the active pending operation.** When cancellation wins the race against completion, the driver calls [`Pollable.cancel()`](#the-pollable-protocol) on whatever leaf the run is currently suspended on. A leaf that owns abortable work — an in-flight socket read, a timer, a registration with a third-party library — implements that hook and gets the chance to tear it down.

The second point is the part to get right, because the default hook is a **no-op**. A leaf that does not override `cancel()` is not aborted by cancelling the run: its socket read stays outstanding, its timer still fires, its `js.Promise` still settles. Whether cancellation reaches the work is a property of the leaf, not of the handle you called `cancel()` on.

:::warning[Bridges do not cancel their source]
A known bug in 0.0.56: the built-in bridges, `Async.fromFuture`, `Async.fromCompletionStage`, `Async.fromJsPromise` and `Async.promise`, leave the default no-op `cancel()` in place. Cancelling an `Async` built on them stops the driver, but the `Future`, `CompletionStage` or callback keeps running. A `CompletableFuture` is not cancelled, and a callback API's cancel handle cannot be reached from the `Completer` that `Async.promise` passes in. To stop the underlying work, keep its own handle and cancel that yourself.
:::

Cancellation is also **cooperative**, never preemptive. It does not wait for an already-running `poll` invocation to return, and it interrupts no thread. Calling `cancel()` publishes the signal and returns; whatever teardown that triggers is driven afterwards, on whichever party wins the claim.

That is why a leaf holding a resource still deserves an explicit finalizer. If a cancelled computation would otherwise leave a socket or a file handle open and you do not control its `Pollable`, close it yourself — pair the cancellation with [`ensuring`](#bracket-and-ensuring), or hold the resource in a `Using` block.

The structural declaration is:

```scala
trait Cancelable extends AutoCloseable {
  def cancel(): Unit
  final def close(): Unit = cancel()
}
```

`Cancelable.noop` is the predefined no-op instance, useful as a placeholder when no real cancellation is needed:

```scala

val c: Cancelable = Cancelable.noop
c.cancel()  // no-op
c.close()   // no-op; delegates to cancel()
```

## AsyncSelector

`AsyncSelector[A]` is a low-level building block, and that is worth saying before anything else: most code should not reach for it. If what you want is "run N of these at a time and give me results as they land", the streams module already provides it — [`mapPar`](streams/core/stream.md#streammappar), [`mergeAll`](streams/core/stream.md#streammergeall), and `mapParAsync` — with backpressure, ordering rules, and resource cleanup you would otherwise write yourself. The selector exists because those operators needed a primitive underneath them, and the streams concurrency engine is its only production consumer.

What it gives you is a **repeatable** multi-way wait. You hand it a fixed number of slots, each holding an independently-pending computation. `select` waits until one of them completes, hands you the winning slot index along with its value, and disarms that slot. `replace` arms the same slot with its next computation, and you select again. That loop is the whole point: a fan-in that runs for the life of a connection, a worker pool that keeps N requests in flight, anything where the same slot is re-used thousands of times.

A hand-rolled race loop can do the first round of that, but not the thousandth. Racing N computations registers a callback on every loser, and the losers survive into the next round, so a fresh callback piles up on each of them every time round. The selector instead keeps **one stable waker per armed slot** for as long as that slot stays armed, no matter how many selections pass over it.

Selection is round-robin rather than first-past-the-post. The scan starts at the slot *after* the previous winner, so a slot that is continuously eligible — armed, and with a completed computation — wins within at most `armedCount` *successful* selections. No slot can be starved by a faster neighbour.

The public surface is small:

```scala
final class AsyncSelector[A] private (slotCount: Int, initial: IndexedSeq[(Int, Async[A])]) extends Cancelable {
  def size: Int
  def replace(index: Int, value: => Async[A]): Unit
  def select: Async[(Int, A)]
  def shutdown: Async[Unit]
  override def cancel(): Unit
}
```

You never construct one directly; `Async.selector` does it:

```scala
def selector[A](inputs: IndexedSeq[Async[A]]): AsyncSelector[A]
```

The `inputs` sequence fixes the slot count for the life of the selector — `size` reports it, and it never changes, disarmed slots included. Here is the full loop:

```scala

val selector: AsyncSelector[Int] =
  Async.selector(Vector(Async.succeed(10), Async.succeed(20), Async.succeed(30)))

// Whichever slot is eligible, scanning from just after the previous winner.
val (firstSlot, firstValue) = selector.select.block

// Re-arm the slot that just won. Until this runs, that slot sits disarmed
// and is skipped by every selection.
selector.replace(firstSlot, Async.succeed(firstValue + 1))

val (secondSlot, secondValue) = selector.select.block

// Cancel every still-armed loser and join all of their cleanup.
selector.shutdown.block
```

Four rules keep that loop honest:

- **A slot can only be armed when it is disarmed.** `replace` on a slot that is still armed throws `IllegalStateException("selector slot N is already armed")`, and `replace` after shutdown throws `IllegalStateException("selector is closed")`. The safe pattern is the one above: re-arm the index you were just handed by `select`, and nothing else.
- **`value` is by-name for a reason.** The thunk is evaluated lazily and outside the selector's own lock, so constructing the next computation cannot deadlock against a concurrent `select`. A thunk that returns `null` is reified as a failed `Async` carrying `NullPointerException("selector input returned null Async")` rather than corrupting the slot.
- **A winner is removed exactly once.** Disarming happens atomically with the win, so the same completion cannot be handed to two selections, and a failure from an input surfaces as a failed `select` rather than poisoning the selector.
- **`shutdown` is where cleanup is joined.** It cancels every armed loser, is idempotent, and — importantly — does not complete until cleanup for every still-armed loser has finished. `cancel()` is the `Cancelable` spelling of the same thing: it starts `shutdown` and returns immediately without waiting. Use `cancel()` when the selector is a resource being closed on the way out of a scope, and `shutdown` when you need to know the cleanup actually finished.

:::warning[`.block` per selection is for the example only]
The snippet above blocks once per round so it reads top to bottom. Real code composes `select` like any other `Async` — `flatMap` it, or `await` it inside `Async.async` — and blocks once at the edge, if at all. Blocking inside a driver deadlocks it; see [Driving](#driving).
:::

## Failure

`Failure` is how a failed `Async` is represented. You never construct one — `Async.fail` and `Completer#fail` produce it, `catchAll` and `either` recover from it — but it explains why a failure travels through a chain untouched: it extends `Pollable[Nothing]`, and `map` and `flatMap` return it unchanged instead of running their functions.

```scala
final class Failure private (val cause: Throwable, private[async] val trusted: Boolean) extends Pollable[Nothing] {

  /** Public construction is deliberately an ordinary, untrusted failure. */
  def this(cause: Throwable) = this(cause, false)

  def poll(onComplete: Runnable): Async[Nothing] = this
}
```

The one-argument constructor is the only one you can call, and it produces an ordinary, untrusted failure. The `trusted` flag is `private[async]`: it marks a failure that entered through one of the library's own internal boundaries — the typed error channel that `zio-blocks-streams` carries in its `Either` — rather than through user code. It changes nothing in the public API: `cause` is the same `Throwable` either way, and every recovery combinator treats both kinds identically. The distinction is visible only to the module's internal fold, which routes a trusted cause down the typed-error path instead of the defect path.

[`block`](#driving) re-throws `cause`; `catchAll` hands your recovery function the original `Throwable`, unwrapped; [`either`](#error-handling) turns it into a `Left` instead.

## Platform Support

The core API behaves identically everywhere by design, and the cross-platform test suite fails if any user-visible core behaviour diverges. What varies is the interop surface, which is deliberately platform-specific, and the two operations that depend on having a thread:

| Feature                      | JVM | Scala.js | How it differs                                                     |
|------------------------------|:---:|:--------:|--------------------------------------------------------------------|
| Constructors and combinators | yes |   yes    | Identical on both                                                  |
| `Async.async` / `await`      | yes |   yes    | Different backend per platform — see [Direct Style](#direct-style) |
| `block` on a pending value   | yes |    no    | Throws on Scala.js: no thread to park                              |
| `fa.start` / `Async.Running` | yes |   yes    | Serialized `ForkJoinPool` tasks on the JVM, microtasks on Scala.js |
| `Async.start(body)`          | yes |   yes    | Daemon thread on the JVM, the next microtask on Scala.js           |
| `Future` interop             | yes |   yes    | Same API on both                                                   |
| `CompletionStage` interop    | yes |    no    | JVM only                                                           |
| `js.Promise` interop         |  no |   yes    | Scala.js only                                                      |

All of it works on Scala 2.13 and Scala 3.

### Where Suspended Work Runs

On the **JVM**, a suspended run is a chain of serialized tasks on `ForkJoinPool.commonPool()`. There is no dedicated worker thread per run and no pool of the module's own: a task exits the moment its current pollable is still pending, and the waker that `poll` registered submits the next task. A run that spends most of its life waiting therefore occupies no thread at all while it waits.

The one exception is `Async.start(body)`, which evaluates an arbitrary synchronous block rather than driving an `Async`. That block gets a thread of its own — a daemon named `zio-blocks-async-eval`.

`block` parks with `LockSupport.park` / `unpark` rather than a monitor, which is a deliberate choice for **virtual threads**: a parked virtual thread (JDK 21+) unmounts its carrier instead of pinning it. The module is Loom-*friendly* in this sense, but it is not Loom-based — it configures no virtual-thread executor, and running on a virtual thread is entirely the caller's decision.

On **Scala.js** there is no thread to hand work to, so scheduling goes through `JSExecutionContext.queue` as microtasks, with a `setTimeout(0)` macrotask escape for work that must yield back to the host event loop. A cap of 1024 consecutive ready resumptions bounds how long a run of already-ready steps can hold the queue before yielding, so a long synchronous chain cannot starve rendering or I/O.

That is also why `block` cannot work there. A pending value gets one chance to complete synchronously inside `poll`; if it has not, `block` throws:

```
java.lang.IllegalStateException: Async.block: suspension did not complete synchronously
and JavaScript cannot block. Drive the Pollable from a non-blocking entry point instead.
```

:::note[There is no blocking-operations API]
The async module has no `attemptBlocking`, no blocking thread pool, and no way to mark an operation as blocking. `block` is the only "blocking" thing in it, and it is a terminal — the point where you leave `Async` and go back to synchronous code, not a place to put a blocking call. Run genuinely blocking work on a thread you control, for example with `Async.start(body)`, and bridge the result back through the resulting `Running`.
:::

## Using Async with Streams

`zio-blocks-streams` depends on this module, so `Async` is not a neighbouring library to streams — it is part of the streams vocabulary. Every cross-platform stream terminal hands you an `Async`, and everything on this page applies to the value you get back.

One convention travels with it. Stream terminals return `Async[Either[E, Z]]`, never `Async[Z]`, because a stream has a typed error channel and `Async` does not. The two are deliberately kept apart:

- The stream's **typed** error `E` stays inside the `Either`. A stream that fails with a typed error still completes its `Async` *successfully*, carrying a `Left`.
- `Async`'s own untyped `Throwable` channel is reserved for **defects** — a callback that threw, a finalizer that failed, a cleanup failure. Those fail the outer `Async` and never appear as a `Left`.

So `catchAll` on a stream terminal recovers bugs, not the errors the stream declares. Those you match on:

```scala

val readings: Stream[String, Int] = Stream(12, 7, 30)

// A terminal is an ordinary Async, so every combinator on this page applies.
val summary: Async[String] =
  readings.runCollectAsync.map(result =>
    result match {
      case Right(values) => s"collected ${values.length} readings"
      case Left(error)   => s"typed error: $error"
    }
  )

// At the edge of a JVM `main`, and nowhere else:
val text: String = summary.block
```

`.block` is the edge-of-the-world move described under [Driving](#driving), and the same two limits hold: never inside a stream callback or a `poll`, and never on a value that may still be pending when you are on Scala.js, where it throws. Cross-platform stream code keeps the `Async` and hands it to the host — `toFuture`, `toJsPromise` — or stays inside `Async.async { … }` and uses `await`.

Cancellation carries across too: cancelling a running stream is the [`Cancelable`](#cancelable) contract on this page, applied to a reader rather than a single leaf.

[Asynchronous Stream Execution](streams/execution-and-compatibility/async-execution.md) is the reference for all of it — the terminal family, the async source constructors and operators, manual pull and reader ownership, and what cancelling a stream cleans up.

## Running the Examples

The `async-examples` module ships `AsyncShowcaseExample`, a single runnable pipeline exercising the whole module: `Completer`-backed callback bridges, the `Async.async` direct-style DSL, `catchAll` recovery, and a final `block`. Its `fulfillOrGuest` function is the one shown under [Direct Style](#direct-style); the file also carries the helpers it calls, which is what makes it runnable as it stands.

To run the full example, clone the repository and execute:

```
cd async-examples && sbt "run"
```

## See Also

- [How to Wrap Callback and Future-Based APIs in Async](../guides/wrap-callback-and-future-apis.md) — bring `Future`, `CompletionStage`, `js.Promise` and callback APIs into `Async`, and cancel the underlying call when the `Async` is cancelled
- [Asynchronous Stream Execution](streams/execution-and-compatibility/async-execution.md) — the `Async` terminal family, async source constructors and operators, reader ownership, and stream cancellation
- [Stream Reference](streams/core/stream.md) — pull-based streaming with resource safety; use `Async.promise` and `Completer` to bridge callback-based push sources into the pull-based stream model
- [Scope Reference](resource-management/scope.md) — compile-time resource safety; `Async.Running` extends `AutoCloseable` and can be used inside `scala.util.Using` or any Scope-managed context for structured cancellation
- [Compile-Time Resource Safety with Scope](../guides/compile-time-resource-safety-with-scope.md) — step-by-step tutorial on resource ownership that applies equally to `Async.Running` handles
- [How to Use ZIO Blocks in a ZIO or cats-effect Application](../guides/use-zio-blocks-in-zio-or-cats-effect.md) — calling `Async` from ZIO or cats-effect, with cancellation carried across

---

## Chunk

`Chunk[A]` is an **immutable, indexed sequence** of elements of type `A`. Unlike `Array`, `Chunk` provides a purely functional interface with optimized performance for high-level operations. It is lazy on expensive operations like repeated concatenation (which use balanced tree structures) while remaining fast on access.

`Chunk[A]`:
- Is purely functional and immutable
- Provides O(1) typical random access (O(log n) worst-case for tree-structured chunks)
- Optimizes concatenation using balanced tree structures (Conc-Trees)
- Automatically specializes primitive types for efficiency without boxing
- Lazily materializes only when necessary to maintain performance

Here is the type signature:

```scala
sealed abstract class Chunk[+A]
  extends IndexedSeq[A]
  with IndexedSeqOps[A, Chunk, Chunk[A]]
  with StrictOptimizedSeqOps[A, Chunk, Chunk[A]]
  with IterableFactoryDefaults[A, Chunk]
  with Serializable
```

## Overview

`Chunk` represents a chunk of values. The implementation is backed by arrays for small chunks but transitions to lazy concatenation trees (`Chunk.Concat`) when building large chunks through repeated concatenation. This design eliminates the O(n²) behavior of naive list concatenation while remaining efficient for both element access and transformation.

`Chunk` has four main internal representations:

```
               ┌─────────────────────────────┐ 
               │           Chunk[A]          │
               └─────────────────────────────┘
                              △
                              │
        ┌──────────────┬──────┴────────┬──────────────┐
        │              │               │              │
   ┌────▼───┐   ┌──────▼──────┐   ┌────▼──────┐   ┌───▼────┐
   │ Empty  │   │ Singleton   │   │Array-     │   │ Concat │
   │        │   │(one element)│   │Backed     │   │(tree)  │
   └────────┘   └─────────────┘   └───────────┘   └────────┘
```

Chunks automatically choose the most efficient representation:
- **Empty**: singleton instance for zero elements
- **Singleton**: single element, no array allocation
- **Array-backed**: standard array for small sequences
- **Concat tree**: balanced binary tree for composite chunks, enabling O(log n) concatenation depth

The implementation is based on [Conc-Trees for Functional and Parallel Programming](http://aleksandar-prokopec.com/resources/docs/lcpc-conc-trees.pdf) by Aleksandar Prokopec and Martin Odersky.

## The Problem

When you work with sequences of data in functional programming, you often need to do two things that seem simple on the surface but are surprisingly tricky to do efficiently at the same time: you need to access elements quickly and merge sequences together without wasting time and memory.

Consider a practical scenario. You're building a data processing pipeline where you collect results from multiple parallel operations. Each operation produces a sequence of values, and you need to combine all of them into one final sequence. With a traditional array, combining sequences requires allocating a new, larger array and copying every single element from the source arrays into it. That's O(n) work just for the merging step. If you're doing this repeatedly—merging results from 10 operations, then 20, then 100—the copying overhead adds up quickly and becomes the bottleneck in your program.

On the other hand, if you use a linked list for efficiency with concatenation, you face a different problem: accessing the millionth element requires traversing through all the previous elements one by one. That's O(n) time for a single lookup. This works fine if you rarely need random access, but in real applications where you're searching, filtering, and transforming sequences, random access happens constantly. It's a painful tradeoff.

There's another subtle but important issue: memory overhead. A linked list that holds a million integers wastes significant memory because each node must store a pointer to the next node in addition to the actual value. An array is compact and efficient, but when you concatenate arrays, you're wasting both time (on copying) and memory (on allocating temporary intermediate arrays).

The real problem isn't just performance in isolation—it's that conventional sequence types force you to choose between mutually incompatible goals. You can optimize for random access (arrays) or for concatenation (lists), but not both efficiently. In a functional programming world where immutability is a core principle, this creates a tension: we want pure, immutable sequences that behave well in parallel processing scenarios where splitting and merging are fundamental operations. Yet the standard approaches either copy too much data or traverse too slowly.

This tension becomes especially acute when you're building systems that process large streams of data, perform data-parallel operations across multiple cores, or need to concatenate sequences as part of their normal operation. Every time you reach for an array and concatenate them, you're paying a hidden tax in copying. Every time you reach for a list, you're paying a hidden tax in traversal. Neither option feels quite right for a modern, functional programming experience.

## The Solution

Chunk solves this problem by combining the strengths of both arrays and balanced trees into a single, cohesive data structure. The key insight is to use different internal representations for different purposes: arrays for small, simple sequences (where the overhead of tree structures would hurt more than help), and balanced tree structures (specifically, Conc-Trees) for composite sequences built through concatenation and transformation.

When you create a small Chunk directly—say, `Chunk(1, 2, 3, 4, 5)`—it's backed by a simple array internally. You get all the benefits: O(1) random access, compact memory, cache-friendly performance. There's no overhead from tree structures or pointers. You're just using an array, exactly as you would expect.

But here's where it gets clever. When you concatenate two Chunks, instead of eagerly copying all the data into a new array, Chunk creates a lightweight tree node that simply links the two chunks together. This is nearly free—just a few pointer assignments. If you concatenate again, you're building a small balanced tree. The beauty is that this tree structure guarantees that no matter how many times you concatenate, random access still works in O(log n) time. The tree stays balanced through careful structural invariants, so the depth never grows too much. For practical sequence sizes you'd work with, O(log n) access is nearly as fast as O(1), with much better memory usage.

Chunk also includes specialized support for incremental building through the ChunkBuilder API. When you're constructing a sequence by repeatedly appending elements, ChunkBuilder appends into one array. `ChunkBuilder#addOne` doubles the array when it is full, so appending one element at a time takes amortized O(1) time, like `ArrayBuffer`, with an occasional copy when the array grows; `ChunkBuilder#addAll` with a `Chunk` grows it to exactly the size it needs. `result()` returns an array-backed chunk.

There's another important optimization: Chunk is designed to handle primitive types like `Int`, `Long`, `Double`, and `Byte` without boxing them. This is a significant performance win in practice. The specialized constructors and accessors ensure that a `Chunk[Int]` really does store integers directly in memory, not wrapped in Java Integer objects.

For functional programming specifically, Chunk embraces immutability completely. When you transform a Chunk, you get a new Chunk. The original remains unchanged. This makes Chunk naturally safe to share across parallel operations without locks or coordination. Different threads can safely read from the same Chunk simultaneously, and transformations produce new, independent Chunks.

The practical result is that Chunk gives you a sequence type that's genuinely efficient across all the operations you actually need: random access is fast, concatenation is fast, transformation is efficient, and memory usage stays reasonable. There's no invisible copying happening in the background, and no painful linear-time traversals. You get array-like performance where it matters most, without sacrificing the ability to build and merge sequences efficiently. For functional programmers and anyone working with immutable data structures, Chunk eliminates the artificial tradeoff between performance and purity.

## Installation

Chunk is available in the core `zio-blocks` library:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-chunk" % "0.0.56"
```

For Scala.js support:

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-chunk" % "0.0.56"
```

Supports Scala 2.13.x and 3.x.

## Factory Methods

Chunk provides comprehensive factory methods for creating instances from various sources. Each factory is optimized for its data source with different performance and safety characteristics. Choose based on your data source and use case.

### Direct Construction

The simplest ways to create chunks from individual elements or single values:

#### `Chunk.apply` — Varargs Constructor

Create a chunk from individual elements using varargs syntax:

```scala
object Chunk {
  def apply[A](as: A*): Chunk[A]
}
```

The simplest way to create chunks from individual elements:

```scala

val numbers = Chunk(1, 2, 3, 4, 5)
// numbers: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5)

val strings = Chunk("alice", "bob", "charlie")
// strings: Chunk[String] = IndexedSeq("alice", "bob", "charlie")

val mixed = Chunk(1, "two", 3.0)
// mixed: Chunk[Int | String | Double] = IndexedSeq(1, "two", 3.0)
```

**Performance:** O(n) — creates array-backed chunk, optimal for small to medium sizes.

#### `Chunk.single` — Single-Element Constructor

Create an efficient single-element chunk:

```scala
object Chunk {
  def single[A](a: A): Chunk[A]
}
```

Convenient for wrapping individual values without array overhead:

```scala

val one = Chunk.single("hello")
// one: Chunk[String] = IndexedSeq("hello")

val singleInt = Chunk.single(42)
// singleInt: Chunk[Int] = IndexedSeq(42)
```

**Performance:** O(1) — specialized for single-element chunks, no array allocation.

#### `Chunk.empty` — Empty Chunk

Create an empty chunk (singleton instance):

```scala
object Chunk {
  def empty[A]: Chunk[A]
}
```

Returns the shared empty chunk singleton:

```scala

val empty = Chunk.empty[Int]
// empty: Chunk[Int] = IndexedSeq()

empty.length
// res3: Int = 0

empty.isEmpty
// res4: Boolean = true
```

**Performance:** O(1) — returns shared singleton, no allocation.

### Generation Methods

Create chunks by generating values using functions:

#### `Chunk.fill` — Repeat Element N Times

Create a chunk by repeating an element n times:

```scala
object Chunk {
  def fill[A](n: Int)(elem: => A): Chunk[A]
}
```

Useful for creating uniform chunks:

```scala

val repeated = Chunk.fill(5)("x")
// repeated: Chunk[String] = IndexedSeq("x", "x", "x", "x", "x")

val zeros = Chunk.fill(3)(0)
// zeros: Chunk[Int] = IndexedSeq(0, 0, 0)

val ones = Chunk.fill(4)(1)
// ones: Chunk[Int] = IndexedSeq(1, 1, 1, 1)
```

**Performance:** O(n) — creates array-backed chunk of size n.

#### `Chunk.iterate` — Repeatedly Apply Function

Create a chunk by repeatedly applying a function starting from an initial value:

```scala
object Chunk {
  def iterate[A](start: A, len: Int)(f: A => A): Chunk[A]
}
```

Generates sequences by function iteration:

```scala

val powers = Chunk.iterate(1, 5)(_ * 2)
// powers: Chunk[Int] = IndexedSeq(1, 2, 4, 8, 16)

val alphabet = Chunk.iterate('a', 3)(c => (c + 1).toChar)
// alphabet: Chunk[Char] = IndexedSeq('a', 'b', 'c')

val decreasing = Chunk.iterate(10, 4)(_ - 1)
// decreasing: Chunk[Int] = IndexedSeq(10, 9, 8, 7)
```

**Performance:** O(n) — function applied n times sequentially.

#### `Chunk.unfold` — Generate from State

Generate a chunk by repeatedly applying a function that returns an optional value:

```scala
object Chunk {
  def unfold[S, A](s: S)(f: S => Option[(A, S)]): Chunk[A]
}
```

Powerful for generating chunks from state transitions:

```scala

// Count from 1 to 5
val counted = Chunk.unfold(1) { n =>
  if (n <= 5) Some((n, n + 1)) else None
}
// counted: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5)

// Generate fibonacci-like sequence
val fibs = Chunk.unfold((1, 1)) { case (a, b) =>
  if (a <= 50) Some((a, (b, a + b))) else None
}
// fibs: Chunk[Int] = IndexedSeq(1, 1, 2, 3, 5, 8, 13, 21, 34)
```

**Performance:** O(n) where n is number of generated elements.

### Collection Conversion Methods

Create chunks from existing Scala collections:

#### `Chunk.fromIterable` — From Scala Collections

Convert any Scala iterable into a chunk:

```scala
object Chunk {
  def fromIterable[A](it: Iterable[A]): Chunk[A]
}
```

Supports all Scala collection types (List, Vector, Set, Seq, etc.). For Chunk and Vector, data may be shared; for others, data is copied:

```scala

val list = List(1, 2, 3)
// list: List[Int] = List(1, 2, 3)
val chunk = Chunk.fromIterable(list)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3)

val vector = Vector("x", "y", "z")
// vector: Vector[String] = Vector("x", "y", "z")
val chunkFromVec = Chunk.fromIterable(vector)
// chunkFromVec: Chunk[String] = IndexedSeq("x", "y", "z")

val set = Set(10, 20, 30)
// set: Set[Int] = Set(10, 20, 30)
val chunkFromSet = Chunk.fromIterable(set)
// chunkFromSet: Chunk[Int] = IndexedSeq(10, 20, 30)
```

**Performance:** O(n) for most types; O(1) for Vector (data sharing).

#### `Chunk.from` — Generic from Iterable (Alias)

Shorter alias for `fromIterable`:

```scala
object Chunk {
  def from[A](it: Iterable[A]): Chunk[A]
}
```

Provides a concise name for generic iterable conversion:

```scala

val list = List(1, 2, 3)
// list: List[Int] = List(1, 2, 3)
val chunk = Chunk.from(list)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3)
```

**Performance:** Same as `fromIterable`.

#### `Chunk.fromIterator` — From Iterator

Consume an iterator and collect all elements into a chunk:

```scala
object Chunk {
  def fromIterator[A](iterator: Iterator[A]): Chunk[A]
}
```

Exhausts the iterator and builds a chunk:

```scala

val iter = Iterator(5, 10, 15)
// iter: Iterator[Int] = empty iterator
val chunk = Chunk.fromIterator(iter)
// chunk: Chunk[Int] = IndexedSeq(5, 10, 15)

// Iterator is now exhausted
val isEmpty = iter.hasNext
// isEmpty: Boolean = false
```

**Performance:** O(n) where n is number of elements in iterator.

#### `Chunk.fromArray` — From Array (Zero-Copy)

Create a chunk backed by an array without copying:

```scala
object Chunk {
  def fromArray[A](array: Array[A]): Chunk[A]
}
```

**Critical Warning:** This is a zero-copy operation that **holds a direct reference to the provided array**. The array **must not be mutated** after creating the chunk. Mutations become visible through the chunk:

```scala

val arr = Array(10, 20, 30)
// arr: Array[Int] = Array(10, 20, 30)
val chunk = Chunk.fromArray(arr)
// chunk: Chunk[Int] = IndexedSeq(10, 20, 30)

chunk
// res12: Chunk[Int] = IndexedSeq(10, 20, 30)
```

Mutating the source array breaks immutability:

```scala

val arr = Array(10, 20, 30)
val chunk = Chunk.fromArray(arr)
arr(0) = 99  // DANGER: mutation visible through chunk!
```

The mutation is visible:

```scala
chunk
// res15: Chunk[Int] = IndexedSeq(99, 20, 30)
```

**Safe Alternative:** Use `fromIterable` which copies data and guarantees immutability:

```scala

val arr = Array(10, 20, 30)
// arr: Array[Int] = Array(10, 20, 30)
val chunk = Chunk.fromIterable(arr.toList)  // Safe copy
// chunk: Chunk[Int] = IndexedSeq(10, 20, 30)

// Now mutations don't affect chunk
```

**Performance:** O(1) zero-copy; O(n) for safe copy alternative.

### Java Interoperability Methods

Create chunks from Java collections and buffers:

#### From `java.nio` Buffers

Create chunks directly from Java NIO buffers (`ByteBuffer`, `CharBuffer`, etc.):

```scala
object Chunk {
  def fromByteBuffer(buffer: ByteBuffer): Chunk[Byte]
  def fromCharBuffer(buffer: CharBuffer): Chunk[Char]
  def fromIntBuffer(buffer: IntBuffer): Chunk[Int]
  def fromLongBuffer(buffer: LongBuffer): Chunk[Long]
  def fromFloatBuffer(buffer: FloatBuffer): Chunk[Float]
  def fromDoubleBuffer(buffer: DoubleBuffer): Chunk[Double]
  def fromShortBuffer(buffer: ShortBuffer): Chunk[Short]
}
```

Working with NIO buffers is seamless:

```scala

val buffer = ByteBuffer.wrap(Array[Byte](1, 2, 3))
// buffer: ByteBuffer = java.nio.HeapByteBuffer[pos=0 lim=3 cap=3]
val chunk = Chunk.fromByteBuffer(buffer)
// chunk: Chunk[Byte] = IndexedSeq(1, 2, 3)
```

**Performance:** O(n) — copies data from buffer into array-backed chunk.

#### `Chunk.fromJavaIterable` — From Java Iterable

Create a chunk from a Java `Iterable`:

```scala
object Chunk {
  def fromJavaIterable[A](iterable: java.lang.Iterable[A]): Chunk[A]
}
```

Interoperate with Java APIs that produce iterables:

```scala

val javaList = Arrays.asList("a", "b", "c")
// javaList: List[String] = [a, b, c]
val chunk = Chunk.fromJavaIterable(javaList)
// chunk: Chunk[String] = IndexedSeq("a", "b", "c")
```

**Performance:** O(n) — iterates and copies elements.

#### `Chunk.fromJavaIterator` — From Java Iterator

Consume a Java `Iterator` and collect its elements into a chunk:

```scala
object Chunk {
  def fromJavaIterator[A](iterator: java.util.Iterator[A]): Chunk[A]
}
```

Building a chunk from a Java iterator:

```scala

val javaIter = Arrays.asList(1, 2, 3).iterator()
// javaIter: Iterator[Int] = java.util.Arrays$ArrayItr@6d4180cb
val chunk = Chunk.fromJavaIterator(javaIter)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3)
```

**Performance:** O(n) — exhausts iterator and builds chunk.

### Builder Methods

`ChunkBuilder[A]` is a mutable builder that accumulates elements incrementally and returns a `Chunk[A]` when complete. Use it when building chunks from elements that arrive one at a time or from streaming sources.

#### `Chunk.newBuilder` — Get a Builder

Obtain a fresh `ChunkBuilder` for incremental construction:

```scala
object Chunk {
  def newBuilder[A]: ChunkBuilder[A]
}
```

A builder integrates with Scala's collection factory pattern:

```scala

val builder = Chunk.newBuilder[Int]
// builder: ChunkBuilder[Int] = ChunkBuilder
builder.addOne(1)
// res21: ChunkBuilder[Int] = ChunkBuilder
builder.addOne(2)
// res22: ChunkBuilder[Int] = ChunkBuilder
val result = builder.result()
// result: Chunk[Int] = IndexedSeq(1, 2)
```

### Using ChunkBuilder for Incremental Construction

`ChunkBuilder[A]` is a mutable builder that accumulates elements and returns a `Chunk[A]`. Use it when building chunks from elements that arrive incrementally—over time, from streaming sources, or when the total size is unknown in advance.

When constructing a chunk from multiple sources, repeatedly calling `++` creates additional tree nodes and increases access depth (O(log n) per operation). Static construction methods like `Chunk.apply` or `Chunk.from` require all elements upfront. `ChunkBuilder` solves this by appending into one array. `ChunkBuilder#addOne` doubles the array when it is full, which yields amortized O(1) append cost, like dynamically sized arrays, with an occasional copy when the array grows; `ChunkBuilder#addAll` with a `Chunk` grows it to exactly the size it needs. `result()` returns an array-backed chunk. It is also the standard Scala mutable builder interface, so it integrates with `scala.collection` builders.

| Scenario                                                                                   | Right choice                                 |
|--------------------------------------------------------------------------------------------|----------------------------------------------|
| Elements available all at once; constructing a static chunk                                | `Chunk.apply(...)` or `Chunk.from(iterable)` |
| Elements arrive incrementally; unknown size in advance; building from a stream or iterator | `ChunkBuilder` with `addOne` / `addAll`      |

The signature of `ChunkBuilder` is:

```scala
object ChunkBuilder {
  def make[A](): ChunkBuilder[A]
  def make[A](capacityHint: Int): ChunkBuilder[A]
}
```

Here's a realistic example: aggregating results from a paginated API that returns chunks of records until a sentinel response:

```scala

case class ApiResponse(records: List[String], hasMore: Boolean)

def fetchAllRecords(): Chunk[String] = {
  @annotation.tailrec
  def loop(response: ApiResponse, builder: ChunkBuilder[String]): Chunk[String] = {
    val builderWithRecords = builder.addAll(response.records.iterator)
    if (response.hasMore) {
      // Simulate fetching next page
      loop(ApiResponse(List("record3", "record4"), false), builderWithRecords)
    } else {
      builderWithRecords.result()
    }
  }
  loop(ApiResponse(List("record1", "record2"), true), ChunkBuilder.make[String](1000))
}
```

In this scenario, the API may return 100 pages before completion. Using `Chunk.apply` would require buffering all responses in memory first. Using naive `++` concatenation repeatedly would create deep tree structures with O(log n) access overhead. `ChunkBuilder` handles pagination efficiently: it keeps one array with amortized O(1) appends and builds no tree.

## Core Operations

Chunk provides a rich set of operations for accessing, transforming, and combining elements. Operations are organized by category:

### Element Access

Chunk provides several operations to inspect and retrieve individual elements efficiently. Whether you need random access by index, efficient access to boundaries, or to check the size, these methods offer fast, predictable performance:

#### `Chunk#apply` — Random Access

Access an element by index in O(log n) time (O(1) for array-backed chunks):

```scala
trait Chunk[+A] {
  def apply(index: Int): A
}
```

Access elements by index in a chunk:

```scala

val chunk = Chunk(10, 20, 30, 40, 50)
```

Accessing by index returns individual elements:

```scala
chunk(0)
// res25: Int = 10
chunk(2)
// res26: Int = 30
chunk(4)
// res27: Int = 50
```

#### `Chunk#head` and `Chunk#last` — First and Last Elements

Access the first or last element:

```scala
trait Chunk[+A] {
  def head: A
  def last: A
}
```

Accessing the first or last element is efficient:

```scala

val chunk = Chunk("a", "b", "c", "d")
```

Getting the first and last elements is immediate:

```scala
chunk.head
// res29: String = "a"
chunk.last
// res30: String = "d"
```

#### `Chunk#length` and `Chunk#size` — Chunk Size

Get the number of elements (O(1) complexity):

```scala
trait Chunk[+A] {
  def length: Int
  def size: Int
}
```

Getting the chunk size is an O(1) operation:

```scala

val chunk = Chunk(1, 2, 3, 4, 5)
```

Both `length` and `size` return the element count:

```scala
chunk.length
// res32: Int = 5
chunk.size
// res33: Int = 5
```

#### `Chunk#headOption` and `Chunk#lastOption` — Safe First and Last

Like `head` and `last`, but return `Option` to safely handle empty chunks:

```scala
trait Chunk[+A] {
  def headOption: Option[A]
  def lastOption: Option[A]
}
```

Accessing the first or last element safely yields an `Option`:

```scala

val chunk = Chunk("a", "b", "c")
val empty = Chunk.empty[Int]
```

`headOption` and `lastOption` provide safe access:

```scala
chunk.headOption
// res35: Option[String] = Some("a")
chunk.lastOption
// res36: Option[String] = Some("c")
empty.headOption
// res37: Option[Int] = None
empty.lastOption
// res38: Option[Int] = None
```

#### `Chunk#indexWhere` — Find Index of Matching Element

Find the index of the first element that matches a predicate:

```scala
trait Chunk[+A] {
  def indexWhere(f: A => Boolean): Int
}
```

Searching for a matching element returns its index or -1 if not found:

```scala

val numbers = Chunk(10, 20, 30, 40)
```

Finding the index of the first even number:

```scala
numbers.indexWhere(_ % 25 == 0)
// res40: Int = -1
numbers.indexWhere(_ % 7 == 0)
// res41: Int = -1
```

#### `Chunk#tail` and `Chunk#init` — Rest and Initial Segments

`tail` returns all elements except the first, `init` returns all elements except the last:

```scala
trait Chunk[+A] {
  def tail: Chunk[A]
  def init: Chunk[A]
}
```

Taking the rest of the chunk after the first element, or all but the last:

```scala

val chunk = Chunk(1, 2, 3, 4)
```

`tail` drops the first element, `init` drops the last:

```scala
chunk.tail
// res43: Chunk[Int] = IndexedSeq(2, 3, 4)
chunk.init
// res44: Chunk[Int] = IndexedSeq(1, 2, 3)
```

### Transformations

Chunk supports functional transformations that let you shape your data in powerful ways. Map applies a function to every element, flatMap chains operations and flattens results, filter keeps only elements that matter, and collect extracts values from nested structures:

#### `Chunk#map` — Transform Elements

Apply a function to each element:

```scala
trait Chunk[+A] {
  def map[B](f: A => B): Chunk[B]
}
```

Mapping a function across elements creates a new chunk:

```scala

val numbers = Chunk(1, 2, 3, 4)
// numbers: Chunk[Int] = IndexedSeq(1, 2, 3, 4)
val doubled = numbers.map(_ * 2)
// doubled: Chunk[Int] = IndexedSeq(2, 4, 6, 8)

val strings = Chunk("hello", "world")
// strings: Chunk[String] = IndexedSeq("hello", "world")
val lengths = strings.map(_.length)
// lengths: Chunk[Int] = IndexedSeq(5, 5)
```

#### `Chunk#flatMap` — Flat Transformation

Map each element to an iterable collection and flatten the result:

```scala
trait Chunk[+A] {
  def flatMap[B](f: A => IterableOnce[B]): Chunk[B]
}
```

Flat-mapping chains transformations and flattens the result. The function can return any `IterableOnce` (such as `Chunk`, `List`, `Vector`, `Array`, etc.):

```scala

val numbers = Chunk(1, 2, 3)
// numbers: Chunk[Int] = IndexedSeq(1, 2, 3)
val expanded = numbers.flatMap(n => Chunk(n, n * 10))
// expanded: Chunk[Int] = IndexedSeq(1, 10, 2, 20, 3, 30)
```

#### `Chunk#filter` — Keep Matching Elements

Keep only elements that match a predicate:

```scala
trait Chunk[+A] {
  def filter(f: A => Boolean): Chunk[A]
}
```

Filtering by a predicate keeps only matching elements:

```scala

val numbers = Chunk(1, 2, 3, 4, 5, 6)
// numbers: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5, 6)
val evens = numbers.filter(_ % 2 == 0)
// evens: Chunk[Int] = IndexedSeq(2, 4, 6)

val longWords = Chunk("a", "hello", "b", "world")
// longWords: Chunk[String] = IndexedSeq("a", "hello", "b", "world")
val filtered = longWords.filter(_.length > 1)
// filtered: Chunk[String] = IndexedSeq("hello", "world")
```

#### `Chunk#collect` — Filter-Map Combined

Apply a partial function, keeping only successful matches:

```scala
trait Chunk[+A] {
  def collect[B](pf: PartialFunction[A, B]): Chunk[B]
}
```

Collecting combines filtering and mapping in one operation:

```scala

val values: Chunk[Any] = Chunk(1, "hello", 2, "world", 3)
// values: Chunk[Any] = IndexedSeq(1, "hello", 2, "world", 3)
val onlyInts = values.collect { case n: Int => n * 10 }
// onlyInts: Chunk[Int] = IndexedSeq(10, 20, 30)
```

#### `Chunk#sorted` — Sort Elements

Sort elements using an ordering:

```scala
trait Chunk[+A] {
  def sorted[A1 >: A](implicit ord: Ordering[A1]): Chunk[A]
}
```

Sorting arranges elements in order:

```scala

val unsorted = Chunk(3, 1, 4, 1, 5, 9, 2, 6)
// unsorted: Chunk[Int] = IndexedSeq(3, 1, 4, 1, 5, 9, 2, 6)
val sorted = unsorted.sorted
// sorted: Chunk[Int] = IndexedSeq(1, 1, 2, 3, 4, 5, 6, 9)

val strings = Chunk("zebra", "apple", "banana")
// strings: Chunk[String] = IndexedSeq("zebra", "apple", "banana")
val sortedStrings = strings.sorted
// sortedStrings: Chunk[String] = IndexedSeq("apple", "banana", "zebra")
```

#### `Chunk#sortBy` — Sort by Key

Sort elements according to a key function:

```scala
trait Chunk[+A] {
  def sortBy[B](f: A => B)(implicit ord: Ordering[B]): Chunk[A]
}
```

Sorting by a key produces a new ordered chunk:

```scala

case class Person(name: String, age: Int)
val people = Chunk(Person("Alice", 32), Person("Bob", 25), Person("Carol", 40))
// people: Chunk[Person] = IndexedSeq(
//   Person(name = "Alice", age = 32),
//   Person(name = "Bob", age = 25),
//   Person(name = "Carol", age = 40)
// )
```

Sorting by age orders the people from youngest to oldest:

```scala
people.sortBy(_.age)
// res51: Chunk[Person] = IndexedSeq(
//   Person(name = "Bob", age = 25),
//   Person(name = "Alice", age = 32),
//   Person(name = "Carol", age = 40)
// )
```

#### `Chunk#collectFirst` — Collect First Matching Partial Function

Apply a partial function to the first matching element and return the result as an option:

```scala
trait Chunk[+A] {
  def collectFirst[B](pf: PartialFunction[A, B]): Option[B]
}
```

Collecting the first match returns an optional result:

```scala

val values = Chunk(1, "two", 3, "four")
```

Extract the first string if present:

```scala
values.collectFirst { case s: String => s }
// res53: Option[String] = Some("two")
```

#### `Chunk#filterNot` — Keep Non-Matching Elements

Keep only elements that do NOT match a predicate:

```scala
trait Chunk[+A] {
  def filterNot(f: A => Boolean): Chunk[A]
}
```

Filtering with a negated predicate removes elements that satisfy the condition:

```scala

val numbers = Chunk(1, 2, 3, 4, 5, 6)
// numbers: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5, 6)
```

Keep only odd numbers:

```scala
numbers.filterNot(_ % 2 == 0)
// res55: Chunk[Int] = IndexedSeq(1, 3, 5)
```

#### `Chunk#distinct` — Remove All Duplicates

Remove duplicate elements, keeping only one occurrence of each unique value:

```scala
trait Chunk[+A] {
  def distinct: Chunk[A]
}
```

Distinct eliminates duplicates based on equality:

```scala

val withDupes = Chunk(1, 2, 3, 2, 4, 1, 5)
```

Keeping only unique values:

```scala
withDupes.distinct
// res57: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5)
```

#### `Chunk#dedupe` — Remove Consecutive Duplicates

Remove only consecutive duplicate elements, preserving the first occurrence in each run:

```scala
trait Chunk[+A] {
  def dedupe: Chunk[A]
}
```

Deduplication differs from `distinct` by only collapsing adjacent duplicates:

```scala

val runs = Chunk(1, 1, 2, 2, 2, 3, 1, 1)
```

Removing consecutive repeats yields:

```scala
runs.dedupe
// res59: Chunk[Int] = IndexedSeq(1, 2, 3, 1)
```

#### `Chunk#flatten` — Flatten Nested Chunks

Flatten a chunk of chunks into a single chunk:

```scala
trait Chunk[+A] {
  def flatten[B](implicit ev: A <:< Chunk[B]): Chunk[B]
}
```

Flattening concatenates nested chunks:

```scala

val nested = Chunk(Chunk(1, 2), Chunk(3), Chunk(4, 5))
// nested: Chunk[Chunk[Int]] = IndexedSeq(
//   IndexedSeq(1, 2),
//   IndexedSeq(3),
//   IndexedSeq(4, 5)
// )
```

A single flattened chunk results:

```scala
nested.flatten
// res61: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5)
```

#### `Chunk#mapAccum` — Stateful Map with Accumulator

Map over elements while accumulating state, returning the final state and the transformed chunk:

```scala
trait Chunk[+A] {
  def mapAccum[S, B](s0: S)(f: (S, A) => (S, B)): (S, Chunk[B])
}
```

Accumulating state during mapping threads an accumulator through:

```scala

val nums = Chunk(10, 20, 30)
```

Starting with sum zero, accumulate running totals:

```scala
val (finalSum, result) = nums.mapAccum(0) { (sum, n) => (sum + n, sum + n) }
// finalSum: Int = 60
// result: Chunk[Int] = IndexedSeq(10, 30, 60)
finalSum
// res63: Int = 60
result
// res64: Chunk[Int] = IndexedSeq(10, 30, 60)
```

#### `Chunk#reverse` — Reverse Order

Reverse the order of elements in the chunk:

```scala
trait Chunk[+A] {
  def reverse: Chunk[A]
}
```

Reversing creates a new chunk with elements in opposite order:

```scala

val original = Chunk(1, 2, 3, 4)
// original: Chunk[Int] = IndexedSeq(1, 2, 3, 4)
```

Reverse produces:

```scala
original.reverse
// res66: Chunk[Int] = IndexedSeq(4, 3, 2, 1)
```

### Combining Chunks

When you need to bring multiple chunks together, Chunk provides efficient operations to merge them. Concatenation uses a balanced tree structure for fast repeated joins, you can append or prepend single elements, and zip operations pair elements from parallel sequences:

#### `Chunk#++(that)` — Concatenation

Combine two chunks. Uses balanced tree structure for efficiency:

```scala
trait Chunk[+A] {
  def ++[A1 >: A](that: Chunk[A1]): Chunk[A1]
}
```

Concatenating two chunks combines them efficiently:

```scala

val chunk1 = Chunk(1, 2, 3)
// chunk1: Chunk[Int] = IndexedSeq(1, 2, 3)
val chunk2 = Chunk(4, 5, 6)
// chunk2: Chunk[Int] = IndexedSeq(4, 5, 6)
val combined = chunk1 ++ chunk2
// combined: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5, 6)

// Efficient even with many concatenations
val many = (1 to 100).foldLeft(Chunk.empty[Int]) { (acc, i) =>
  acc ++ Chunk(i)
}
// many: Chunk[Int] = IndexedSeq(
//   1,
//   2,
//   3,
//   4,
//   5,
//   6,
//   7,
//   8,
//   9,
//   10,
//   11,
//   12,
//   13,
//   14,
//   15,
//   16,
//   17,
//   18,
//   19,
//   20,
//   21,
//   22,
//   23,
//   24,
//   25,
//   26,
//   27,
//   28,
//   29,
//   30,
//   31,
//   32,
//   33,
//   34,
//   35,
//   36,
//   37,
//   38,
//   39,
//   40,
//   41,
//   42,
//   43,
//   44,
//   45,
//   46,
//   47,
//   48,
// ...
```

#### `Chunk#:+(a)` — Append Element

Append a single element to the end:

```scala
trait Chunk[+A] {
  def :+[A1 >: A](a: A1): Chunk[A1]
}
```

Appending a single element creates a new chunk:

```scala

val chunk = Chunk(1, 2, 3)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3)
val appended = chunk :+ 4
// appended: Chunk[Int] = IndexedSeq(1, 2, 3, 4)
```

#### `Chunk#+:(a)` — Prepend Element

Prepend a single element to the beginning:

```scala
trait Chunk[+A] {
  def +:[A1 >: A](a: A1): Chunk[A1]
}
```

Prepending a single element adds it to the front:

```scala

val chunk = Chunk(2, 3, 4)
// chunk: Chunk[Int] = IndexedSeq(2, 3, 4)
val prepended = 1 +: chunk
// prepended: Chunk[Int] = IndexedSeq(1, 2, 3, 4)
```

#### `Chunk#zip` and `Chunk#zipWith` — Combine Parallel Chunks

Combine two chunks element-wise:

```scala
trait Chunk[+A] {
  def zip[B](that: Chunk[B]): Chunk[(A, B)]
  def zipWith[B, C](that: Chunk[B])(f: (A, B) => C): Chunk[C]
}
```

Zipping combines elements from two chunks:

```scala

val numbers = Chunk(1, 2, 3)
// numbers: Chunk[Int] = IndexedSeq(1, 2, 3)
val letters = Chunk("a", "b", "c")
// letters: Chunk[String] = IndexedSeq("a", "b", "c")

val zipped = numbers.zip(letters)
// zipped: Chunk[Tuple2[Int, String]] = IndexedSeq(
//   (1, "a"),
//   (2, "b"),
//   (3, "c")
// )

val combined = numbers.zipWith(letters)((n, l) => s"$l$n")
// combined: Chunk[String] = IndexedSeq("a1", "b2", "c3")
```

#### `Chunk#appended` — Append Element (Method Form)

Append a single element to the end (named method equivalent of `:+`):

```scala
trait Chunk[+A] {
  def appended[A1 >: A](a: A1): Chunk[A1]
}
```

Appending creates a new chunk:

```scala

val chunk = Chunk(1, 2, 3)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3)
val appended = chunk.appended(4)
// appended: Chunk[Int] = IndexedSeq(1, 2, 3, 4)
```

#### `Chunk#prepended` — Prepend Element (Method Form)

Prepend a single element to the beginning (named method equivalent of `+:`):

```scala
trait Chunk[+A] {
  def prepended[A1 >: A](a: A1): Chunk[A1]
}
```

Prepending adds an element to the front:

```scala

val chunk = Chunk(2, 3, 4)
// chunk: Chunk[Int] = IndexedSeq(2, 3, 4)
val prepended = chunk.prepended(1)
// prepended: Chunk[Int] = IndexedSeq(1, 2, 3, 4)
```

#### `Chunk#zipAll` — Zip with Defaults for Unequal Lengths

Combine two chunks element-wise, using default values when one chunk is shorter:

```scala
trait Chunk[+A] {
  def zipAll[B](that: Chunk[B]): Chunk[(Option[A], Option[B])]
}
```

Zipping with defaults ensures both sides always have a value:

```scala

val left = Chunk(1, 2, 3)
val right = Chunk("a", "b")
```

The shorter chunk is padded with `None` values:

```scala
left.zipAll(right)
// res74: Chunk[Tuple2[Option[Int], Option[String]]] = IndexedSeq(
//   (Some(1), Some("a")),
//   (Some(2), Some("b")),
//   (Some(3), None)
// )
```

#### `Chunk#zipAllWith` — Zip with Custom Combiner

Zip two chunks element-wise with a custom combiner function that handles missing elements:

```scala
trait Chunk[+A] {
  def zipAllWith[B, C](that: Chunk[B])(left: A => C, right: B => C)(both: (A, B) => C): Chunk[C]
}
```

Providing separate handlers for left-only, right-only, and both:

```scala

val left = Chunk(1, 2, 3)
// left: Chunk[Int] = IndexedSeq(1, 2, 3)
val right = Chunk("a", "b")
// right: Chunk[String] = IndexedSeq("a", "b")
```

Combine with default for missing elements:

```scala
left.zipAllWith(right)(
  a => s"missing:${a}",      // left only
  b => s"only:${b}"          // right only
)(
  (a, b) => s"$a:$b"         // both present
)
// res76: Chunk[String] = IndexedSeq("1:a", "2:b", "missing:3")
```

#### `Chunk#zipWithIndex` — Zip with Index Starting at Zero

Zip each element with its zero-based index:

```scala
trait Chunk[+A] {
  def zipWithIndex: Chunk[(A, Int)]
}
```

Attaching indices is useful for positional awareness:

```scala

val words = Chunk("alpha", "beta", "gamma")
```

Zipping with index:

```scala
words.zipWithIndex
// res78: Chunk[Tuple2[String, Int]] = IndexedSeq(
//   ("alpha", 0),
//   ("beta", 1),
//   ("gamma", 2)
// )
```

#### `Chunk#updated` — Update Element at Index

Update the element at a given index, returning a new chunk:

```scala
trait Chunk[+A] {
  override def updated[A1 >: A](index: Int, elem: A1): Chunk[A1]
}
```

The `updated` method creates a new chunk with the element at the specified index replaced. The original chunk remains unchanged:

```scala

val chunk = Chunk(1, 2, 3, 4, 5)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5)

val updated = chunk.updated(2, 99)
// updated: Chunk[Int] = IndexedSeq(1, 2, 99, 4, 5)
```

### Slicing and Partitioning

Often you need to work with portions of a chunk rather than the whole. These operations let you keep elements from the ends, skip unwanted portions, extract contiguous ranges by position, and intelligently partition chunks based on predicates or conditions:

#### `Chunk#take` and `Chunk#takeRight` — Take from Ends

Take the first n elements or last n elements:

```scala
trait Chunk[+A] {
  def take(n: Int): Chunk[A]
  def takeRight(n: Int): Chunk[A]
}
```

Taking elements from the beginning or end creates a new chunk:

```scala

val chunk = Chunk(1, 2, 3, 4, 5)
```

Taking from the beginning or end produces new chunks:

```scala
chunk.take(3)
// res81: Chunk[Int] = IndexedSeq(1, 2, 3)
chunk.takeRight(2)
// res82: Chunk[Int] = IndexedSeq(4, 5)
```

#### `Chunk#drop` and `Chunk#dropRight` — Remove from Ends

Remove the first n elements or last n elements:

```scala
trait Chunk[+A] {
  def drop(n: Int): Chunk[A]
  def dropRight(n: Int): Chunk[A]
}
```

Dropping elements removes them from the beginning or end:

```scala

val chunk = Chunk(1, 2, 3, 4, 5)
```

Dropping from beginning or end removes those elements:

```scala
chunk.drop(2)
// res84: Chunk[Int] = IndexedSeq(3, 4, 5)
chunk.dropRight(2)
// res85: Chunk[Int] = IndexedSeq(1, 2, 3)
```

#### `Chunk#slice` — Extract a Range

Extract elements from a start index to an end index:

```scala
trait Chunk[+A] {
  def slice(from: Int, until: Int): Chunk[A]
}
```

Slicing extracts a contiguous range of elements:

```scala

val chunk = Chunk(10, 20, 30, 40, 50, 60)
```

Slicing from a start index to end index extracts the range:

```scala
chunk.slice(1, 4)
// res87: Chunk[Int] = IndexedSeq(20, 30, 40)
chunk.slice(2, 5)
// res88: Chunk[Int] = IndexedSeq(30, 40, 50)
```

#### `Chunk#split` — Split into Equally-Sized Chunks

Split the chunk into N equally-sized chunks. When the chunk size is not evenly divisible, remainder elements are distributed into earlier chunks. If n exceeds the chunk's length, the result contains at most chunk.length single-element chunks.

Note: Passing `0` throws `ArithmeticException` due to division by zero. Negative values are not rejected by the current implementation and may produce unexpected results, so `n` should always be positive in normal use.

The method signature is:

```scala
trait Chunk[+A] {
  def split(n: Int): Chunk[Chunk[A]]
}
```

Splitting divides a chunk into N equal parts:

```scala

val chunk = Chunk(1, 2, 3, 4, 5, 6)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5, 6)
val splitInto2 = chunk.split(2)
// splitInto2: Chunk[Chunk[Int]] = IndexedSeq(
//   IndexedSeq(1, 2, 3),
//   IndexedSeq(4, 5, 6)
// )

val uneven = Chunk(1, 2, 3, 4, 5, 6, 7)
// uneven: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5, 6, 7)
val splitInto3 = uneven.split(3)
// splitInto3: Chunk[Chunk[Int]] = IndexedSeq(
//   IndexedSeq(1, 2, 3),
//   IndexedSeq(4, 5),
//   IndexedSeq(6, 7)
// )
```

#### `Chunk#span` and `Chunk#splitWhere` — Partition by Predicate

`span(f)` splits the chunk into a prefix where the predicate holds, and the remainder. It stops at the first element where the predicate becomes false.

`splitWhere(f)` is similar but has inverted logic: it splits at the first element where the predicate becomes true.

The method signatures are:

```scala
trait Chunk[+A] {
  def span(f: A => Boolean): (Chunk[A], Chunk[A])
  def splitWhere(f: A => Boolean): (Chunk[A], Chunk[A])
}
```

Partitioning by predicate creates two chunks:

```scala

val numbers = Chunk(1, 2, 3, 4, 5, 6)
// numbers: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5, 6)
val (prefix, rest) = numbers.span(_ < 4)
// prefix: Chunk[Int] = IndexedSeq(1, 2, 3)
// rest: Chunk[Int] = IndexedSeq(4, 5, 6)
val (upTo, remaining) = Chunk(1, 2, 5, 3, 4).splitWhere(_ >= 5)
// upTo: Chunk[Int] = IndexedSeq(1, 2)
// remaining: Chunk[Int] = IndexedSeq(5, 3, 4)
```

#### `Chunk#dropWhile` — Drop While Predicate Holds

Drop elements from the beginning while a predicate is true:

```scala
trait Chunk[+A] {
  def dropWhile(f: A => Boolean): Chunk[A]
}
```

Dropping while the predicate holds removes leading elements:

```scala

val numbers = Chunk(1, 2, 3, 4, 5, 6)
```

Drop initial even numbers:

```scala
numbers.dropWhile(_ % 2 == 0)
// res92: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5, 6)
```

#### `Chunk#dropUntil` — Drop Through the First Match

Drop elements up to and including the first one that satisfies the predicate, and keep the rest. When no element satisfies the predicate, the result is empty:

```scala
trait Chunk[+A] {
  def dropUntil(f: A => Boolean): Chunk[A]
}
```

To try it, start with five numbers:

```scala

val numbers = Chunk(1, 2, 3, 4, 5)
// numbers: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5)
```

The first number that is at least 3 is `3`, so `dropUntil` drops `1`, `2` and `3`:

```scala
numbers.dropUntil(_ >= 3)
// res94: Chunk[Int] = IndexedSeq(4, 5)
```

#### `Chunk#takeWhile` — Take While Predicate Holds

Take elements from the beginning while a predicate is true, stopping at the first failure:

```scala
trait Chunk[+A] {
  def takeWhile(f: A => Boolean): Chunk[A]
}
```

Taking while the predicate holds collects a prefix:

```scala

val numbers = Chunk(1, 2, 3, 4, 5, 6)
```

Take the initial odd numbers:

```scala
numbers.takeWhile(_ % 2 != 0)
// res96: Chunk[Int] = IndexedSeq(1)
```

#### `Chunk#splitAt` — Split at Index

Split the chunk into two chunks at a given index: the first contains elements `[0, until)`, the second contains the remainder.

The method signature is:

```scala
trait Chunk[+A] {
  def splitAt(n: Int): (Chunk[A], Chunk[A])
}
```

Splitting creates two chunks from a single index:

```scala

val chunk = Chunk(10, 20, 30, 40, 50)
```

Splitting at index 3:

```scala
chunk.splitAt(3)
// res98: Tuple2[Chunk[Int], Chunk[Int]] = (
//   IndexedSeq(10, 20, 30),
//   IndexedSeq(40, 50)
// )
```

### Querying and Folding

Chunk enables you to ask questions about your data and reduce it to meaningful results. Use fold operations to accumulate values, test predicates to verify conditions hold across elements, and search for specific values that match your criteria:

#### `Chunk#foldLeft` — Left Fold

Process elements left-to-right with an accumulator:

```scala
trait Chunk[+A] {
  def foldLeft[S](s0: S)(f: (S, A) => S): S
}
```

Left-folding accumulates values from left to right:

```scala

val numbers = Chunk(1, 2, 3, 4, 5)
// numbers: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5)

val sum = numbers.foldLeft(0)(_ + _)
// sum: Int = 15

val product = numbers.foldLeft(1)(_ * _)
// product: Int = 120

val concat = Chunk("a", "b", "c").foldLeft("")(_ + _)
// concat: String = "abc"
```

#### `Chunk#foldRight` — Right Fold

Process elements right-to-left with an accumulator:

```scala
trait Chunk[+A] {
  def foldRight[S](s0: S)(f: (A, S) => S): S
}
```

Right-folding accumulates values from right to left:

```scala

val numbers = Chunk(1, 2, 3, 4)
// numbers: Chunk[Int] = IndexedSeq(1, 2, 3, 4)
val result = numbers.foldRight(List[Int]())(_ :: _)
// result: List[Int] = List(1, 2, 3, 4)
```

#### `Chunk#foldWhile` — Fold with Early Exit

Fold left with early termination based on a predicate on the accumulator:

```scala
trait Chunk[+A] {
  def foldWhile[S](s0: S)(pred: S => Boolean)(f: (S, A) => S): S
}
```

`foldWhile` allows you to stop folding when a condition on the accumulator becomes false, avoiding unnecessary processing of remaining elements:

```scala

val numbers = Chunk(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
// numbers: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

// Fold while accumulator is less than 20, adding elements
val result = numbers.foldWhile(0)(acc => acc < 20)((acc, n) => acc + n)
// result: Int = 21
```

#### `Chunk#exists` and `Chunk#forall` — Predicates

Check if any or all elements match a predicate:

```scala
trait Chunk[+A] {
  def exists(f: A => Boolean): Boolean
  def forall(f: A => Boolean): Boolean
}
```

Testing predicates answers questions about elements:

```scala

val numbers = Chunk(2, 4, 6, 8)
val mixed = Chunk(1, 2, 3)
```

Testing existence and universal predicates returns Boolean:

```scala
numbers.exists(_ > 5)
// res103: Boolean = true
numbers.forall(_ % 2 == 0)
// res104: Boolean = true
mixed.forall(_ > 0)
// res105: Boolean = true
mixed.forall(_ % 2 == 0)
// res106: Boolean = false
```

#### `Chunk#find` — First Matching Element

Find the first element matching a predicate:

```scala
trait Chunk[+A] {
  def find(f: A => Boolean): Option[A]
}
```

Finding the first matching element returns an Option:

```scala

val numbers = Chunk(1, 2, 3, 4, 5)
val words = Chunk("apple", "banana", "cherry")
```

Finding returns Some for matches and None for no match:

```scala
numbers.find(_ > 3)
// res108: Option[Int] = Some(4)
numbers.find(_ > 10)
// res109: Option[Int] = None
words.find(_.startsWith("b"))
// res110: Option[String] = Some("banana")
```

#### `Chunk#corresponds` — Check Parallel Correspondence

Check whether corresponding elements of two chunks satisfy a predicate in lockstep:

```scala
trait Chunk[+A] {
  def corresponds[B](that: Chunk[B])(f: (A, B) => Boolean): Boolean
}
```

`corresponds` tests pairwise element compatibility:

```scala

val a = Chunk(1, 2, 3)
val b = Chunk(1, 2, 3)
```

Correspondence holds when all pairs match:

```scala
a.corresponds(b)(_ == _)
// res112: Boolean = true
```

### Grouping

Chunk provides powerful grouping operations to organize elements by keys or into fixed-size partitions. Use `groupBy` to categorize elements into maps of chunks, `groupMap` to combine grouping and mapping in a single pass, and `grouped` to split the chunk into chunks of a specified size.

#### `Chunk#groupBy` — Group Elements by Key

Group elements by a key function, producing a map from keys to chunks:

```scala
trait Chunk[+A] {
  def groupBy[K](f: A => K): Map[K, Chunk[A]]
}
```

Grouping by a key aggregates related elements:

```scala

case class Person(name: String, age: Int)
val people = Chunk(Person("Alice", 32), Person("Bob", 25), Person("Carol", 32))
```

Group people by age:

```scala
people.groupBy(_.age)
// res114: Map[Int, Chunk[Person]] = Map(
//   25 -> IndexedSeq(Person(name = "Bob", age = 25)),
//   32 -> IndexedSeq(Person(name = "Alice", age = 32), Person(name = "Carol", age = 32))
// )
```

#### `Chunk#groupMap` — Group and Map in One Pass

Group elements by a key and simultaneously transform the values:

```scala
trait Chunk[+A] {
  def groupMap[K, V](key: A => K)(f: A => V): Map[K, Chunk[V]]
}
```

Grouping and mapping in a single operation can be more efficient than separate steps:

```scala

val words = Chunk("apple", "banana", "cherry", "avocado")
// words: Chunk[String] = IndexedSeq("apple", "banana", "cherry", "avocado")
```

Group words by their first letter and also uppercase them:

```scala
words.groupMap(_.head)(_.toUpperCase)
// res116: Map[Char, Chunk[String]] = Map(
//   'a' -> IndexedSeq("APPLE", "AVOCADO"),
//   'b' -> IndexedSeq("BANANA"),
//   'c' -> IndexedSeq("CHERRY")
// )
```

#### `Chunk#grouped` — Partition into Fixed-Size Groups

Divide the chunk into groups of a fixed size, returning an iterator over subgroups:

```scala
trait Chunk[+A] {
  def grouped(size: Int): Iterator[Chunk[A]]
}
```

Fixed-size grouping is useful for batch processing:

```scala

val numbers = Chunk(1, 2, 3, 4, 5, 6, 7, 8)
```

Groups of size 3 yield two full groups and a partial:

```scala
numbers.grouped(3).toList
// res118: List[Chunk[Int]] = List(
//   IndexedSeq(1, 2, 3),
//   IndexedSeq(4, 5, 6),
//   IndexedSeq(7, 8)
// )
```

### Equality and Iteration

Chunks support standard equality comparison and side-effect iteration:

#### `Chunk#equals` — Equality Comparison

Compare two chunks for equality:

```scala
trait Chunk[+A] {
  override def equals(that: Any): Boolean
}
```

Chunks are equal if they contain the same elements in the same order. The comparison works correctly regardless of the internal representation (array-backed vs. tree-structured):

```scala

val chunk1 = Chunk(1, 2, 3)
// chunk1: Chunk[Int] = IndexedSeq(1, 2, 3)
val chunk2 = Chunk(1, 2, 3)
// chunk2: Chunk[Int] = IndexedSeq(1, 2, 3)
val chunk3 = Chunk(1, 2, 4)
// chunk3: Chunk[Int] = IndexedSeq(1, 2, 4)

chunk1 == chunk2
// res120: Boolean = true
chunk1 == chunk3
// res121: Boolean = false
```

#### `Chunk#hashCode` — Hash Code Computation

Get the hash code of a chunk:

```scala
trait Chunk[+A] {
  override def hashCode: Int
}
```

Chunks compute their hash code using the MurmurHash3 algorithm applied to all elements, similar to Scala's collection hashing. Equal chunks always have equal hash codes, making chunks suitable for use as keys in hash-based collections:

```scala

val chunk1 = Chunk(1, 2, 3)
// chunk1: Chunk[Int] = IndexedSeq(1, 2, 3)
val chunk2 = Chunk(1, 2, 3)
// chunk2: Chunk[Int] = IndexedSeq(1, 2, 3)

chunk1.hashCode == chunk2.hashCode
// res123: Boolean = true

val chunkSet = Set(chunk1)
// chunkSet: Set[Chunk[Int]] = Set(IndexedSeq(1, 2, 3))
chunkSet.contains(chunk2)
// res124: Boolean = true
```

Chunks can be reliably used in Maps and Sets because equal chunks have equal hash codes.

#### `Chunk#foreach` — Iteration with Side Effects

Perform a side effect for each element (used with `Unit`-returning functions):

```scala
trait Chunk[+A] {
  override def foreach[B](f: A => B): Unit
}
```

`foreach` iterates through all elements and applies a function, ignoring the return value. This is useful for side effects like logging or printing:

```scala

val chunk = Chunk(1, 2, 3)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3)

// Print each element
chunk.foreach(n => println(s"Element: $n"))
// Element: 1
// Element: 2
// Element: 3
```

### Conversion

Sometimes you need to move data from Chunk into other Scala collections or represent it as text. These conversion operations make it easy to export your chunk into arrays, lists, sequences, or render it as a string for logging and display:

#### `Chunk#toArray` — To `Array`

Convert to an array:

```scala
trait Chunk[+A] {
  def toArray[A1 >: A: ClassTag]: Array[A1]
}
```

Converting to an array materializes all elements:

```scala

val chunk = Chunk(1, 2, 3, 4)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3, 4)
val array: Array[Int] = chunk.toArray
// array: Array[Int] = Array(1, 2, 3, 4)
```

#### `Chunk#toList` — To `List`

Convert to a `List`:

```scala
trait Chunk[+A] {
  def toList: List[A]
}
```

Converting to a list produces a sequential data structure:

```scala

val chunk = Chunk("a", "b", "c")
// chunk: Chunk[String] = IndexedSeq("a", "b", "c")
val list = chunk.toList
// list: List[String] = List("a", "b", "c")
```

#### `Chunk#toSeq`, `Chunk#toIterable`, `Chunk#toIndexedSeq` — Standard Collections

Convert to various Scala collection types:

```scala
trait Chunk[+A] {
  def toSeq: Seq[A]
  def toIterable: Iterable[A]
  def toIndexedSeq: IndexedSeq[A]
}
```

Converting to standard Scala collections enables interop:

```scala

val chunk = Chunk(1, 2, 3)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3)

chunk.toSeq
// res130: Chunk[Int] = IndexedSeq(1, 2, 3)
chunk.toIndexedSeq
// res131: IndexedSeq[Int] = IndexedSeq(1, 2, 3)
```

#### `Chunk#toString` — String Representation

Convert to a string:

```scala
trait Chunk[+A] {
  def toString: String
}
```

String representation shows all elements in a compact format:

```scala

val chunk = Chunk(1, 2, 3)
```

String conversion shows the chunk contents:

```scala
chunk.toString
// res133: String = "Chunk(1,2,3)"
```

#### `Chunk#toVector` — To `Vector`

Convert to a `Vector`:

```scala
trait Chunk[+A] {
  def toVector: Vector[A]
}
```

Conversion to `Vector` produces an efficient indexed sequence:

```scala

val chunk = Chunk(1, 2, 3)
```

```scala
chunk.toVector
// res135: Vector[Int] = Vector(1, 2, 3)
```

### Specialized Accessors for Primitive Types

For primitive types, direct accessors avoid boxing:

```scala
trait Chunk[+A] {
  def byte(index: Int)(implicit ev: A <:< Byte): Byte
  def boolean(index: Int)(implicit ev: A <:< Boolean): Boolean
  def char(index: Int)(implicit ev: A <:< Char): Char
  def double(index: Int)(implicit ev: A <:< Double): Double
  def float(index: Int)(implicit ev: A <:< Float): Float
  def int(index: Int)(implicit ev: A <:< Int): Int
  def long(index: Int)(implicit ev: A <:< Long): Long
  def short(index: Int)(implicit ev: A <:< Short): Short
}
```

Accessing primitives without boxing demonstrates zero-overhead specialization:

```scala

val bytes: Chunk[Byte] = Chunk(1.toByte, 2.toByte, 3.toByte)
val ints = Chunk(10, 20, 30)
```

Accessing primitives by specialized methods avoids boxing:

```scala
bytes.byte(0)
// res137: Byte = 1
ints.int(1)
// res138: Int = 20
```

### Iterator Access

Access chunks through iterators for sequential processing or collect the known size of elements:

#### `Chunk#iterator` — Get a Standard Iterator

Get a standard Scala `Iterator` to traverse the chunk sequentially:

```scala
trait Chunk[+A] {
  def iterator: Iterator[A]
}
```

The iterator is useful for sequential processing and integrates with Scala's collection ecosystem:

```scala

val chunk = Chunk(1, 2, 3, 4, 5)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5)
val iter = chunk.iterator
// iter: Iterator[Int] = non-empty iterator

// Process elements sequentially
iter.take(3).toList
// res140: List[Int] = List(1, 2, 3)
```

#### `Chunk#chunkIterator` — Get a Chunk-Specific Iterator

Get a `Chunk.ChunkIterator` for more efficient traversal within the Chunk ecosystem:

```scala
trait Chunk[+A] {
  def chunkIterator: Chunk.ChunkIterator[A]
}
```

`ChunkIterator` is optimized for Chunk operations and can be more efficient than the standard iterator in some cases:

```scala

val chunk = Chunk(10, 20, 30, 40)
// chunk: Chunk[Int] = IndexedSeq(10, 20, 30, 40)
val chunkIter = chunk.chunkIterator
// chunkIter: ChunkIterator[Int] = IndexedSeq(10, 20, 30, 40)
```

#### `Chunk#knownSize` — Get Iterator Size Hint

Get the known size of the chunk for iterator protocol compatibility:

```scala
trait Chunk[+A] {
  def knownSize: Int
}
```

The known size helps collection builders optimize allocations:

```scala

val chunk = Chunk(1, 2, 3)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3)
val size = chunk.knownSize
// size: Int = 3
```

### Array Operations

Perform bulk operations with arrays for interoperability with imperative code:

#### `Chunk#copyToArray` — Copy to Destination Array

Copy elements from the chunk to a destination array at a specified position:

```scala
trait Chunk[+A] {
  def copyToArray[B >: A](dest: Array[B], destPos: Int, length: Int): Int
}
```

This is useful for efficient bulk export of chunk contents to arrays, particularly in interop scenarios. The method returns the number of elements actually copied:

```scala

val chunk = Chunk(1, 2, 3, 4, 5)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5)
val dest = new Array[Int](10)
// dest: Array[Int] = Array(0, 0, 1, 2, 3, 0, 0, 0, 0, 0)

val numCopied = chunk.copyToArray(dest, 2, 3)
// numCopied: Int = 3
dest.take(5)
// res144: Array[Int] = Array(0, 0, 1, 2, 3)
```

### Materialization and Optimization

When you build chunks through many concatenations, they internally form a tree structure for efficiency. However, if you plan to access elements many times, materialization converts that tree into a flat, optimized array-backed representation:

Force the chunk to an array-backed representation, eliminating lazy concatenation trees. Useful before performing many operations:

```scala
trait Chunk[+A] {
  def materialize[A1 >: A]: Chunk[A1]
}
```

Materializing a tree-built chunk converts it to an efficient array-backed form:

```scala

// Build through many concatenations (creates Concat tree)
val built = (1 to 100).foldLeft(Chunk.empty[Int]) { (acc, i) =>
  acc ++ Chunk(i)
}
// built: Chunk[Int] = IndexedSeq(
//   1,
//   2,
//   3,
//   4,
//   5,
//   6,
//   7,
//   8,
//   9,
//   10,
//   11,
//   12,
//   13,
//   14,
//   15,
//   16,
//   17,
//   18,
//   19,
//   20,
//   21,
//   22,
//   23,
//   24,
//   25,
//   26,
//   27,
//   28,
//   29,
//   30,
//   31,
//   32,
//   33,
//   34,
//   35,
//   36,
//   37,
//   38,
//   39,
//   40,
//   41,
//   42,
//   43,
//   44,
//   45,
//   46,
//   47,
//   48,
// ...

// Materialize to array-backed representation
val materialized = built.materialize
// materialized: Chunk[Int] = IndexedSeq(
//   1,
//   2,
//   3,
//   4,
//   5,
//   6,
//   7,
//   8,
//   9,
//   10,
//   11,
//   12,
//   13,
//   14,
//   15,
//   16,
//   17,
//   18,
//   19,
//   20,
//   21,
//   22,
//   23,
//   24,
//   25,
//   26,
//   27,
//   28,
//   29,
//   30,
//   31,
//   32,
//   33,
//   34,
//   35,
//   36,
//   37,
//   38,
//   39,
//   40,
//   41,
//   42,
//   43,
//   44,
//   45,
//   46,
//   47,
//   48,
// ...
// Now faster for repeated access or further operations
```

## Advanced Usage

Advanced use cases include bit-level operations, working with specialized chunk types, and comparing Chunk with other data structures:

### Bit Operations

Chunk provides specialized operations to work at the bit level with numeric types. These are useful when you need to inspect the binary representation of bytes, integers, or longs while respecting endianness preferences:

#### `Chunk#asBitsByte` — Convert to Byte Bits

Convert a chunk of bytes into a chunk of individual bits. This method is only available on `Chunk[Byte]` through an implicit constraint:

```scala
trait Chunk[+A] {
  def asBitsByte(implicit ev: A <:< Byte): Chunk[Boolean]
}
```

Converting bytes to bits reveals the binary representation:

```scala

val bytes = Chunk(1.toByte, 2.toByte)
// bytes: Chunk[Byte] = IndexedSeq(1, 2)
val bits = bytes.asBitsByte
// bits: Chunk[Boolean] = IndexedSeq(
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   true,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   true,
//   false
// )
```

#### `Chunk#asBitsInt` — Convert to Int Bits

Convert a chunk of ints into a chunk of individual bits with a specified endianness. This method is only available on `Chunk[Int]` through an implicit constraint:

```scala
trait Chunk[+A] {
  def asBitsInt(endianness: Chunk.BitChunk.Endianness)(implicit ev: A <:< Int): Chunk[Boolean]
}
```

Converting integers to bits with endianness control provides bit-level access:

```scala

val ints = Chunk(1, 2, 3)
// ints: Chunk[Int] = IndexedSeq(1, 2, 3)
val bits = ints.asBitsInt(Chunk.BitChunk.Endianness.BigEndian)
// bits: Chunk[Boolean] = IndexedSeq(
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   true,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
// ...
```

#### `Chunk#asBitsLong` — Convert to Long Bits

Convert a chunk of longs into a chunk of individual bits with a specified endianness. This method is only available on `Chunk[Long]` through an implicit constraint:

```scala
trait Chunk[+A] {
  def asBitsLong(endianness: Chunk.BitChunk.Endianness)(implicit ev: A <:< Long): Chunk[Boolean]
}
```

Converting longs to bits supports both big-endian and little-endian representations:

```scala

val longs = Chunk(1L, 2L, 3L)
// longs: Chunk[Long] = IndexedSeq(1L, 2L, 3L)
val bits = longs.asBitsLong(Chunk.BitChunk.Endianness.LittleEndian)
// bits: Chunk[Boolean] = IndexedSeq(
//   true,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
//   false,
// ...
```

#### `Chunk#toBinaryString` — Convert Boolean Chunk to Binary String

Convert a chunk of booleans (representing bits) into a binary string representation:

```scala
trait Chunk[+A] {
  def toBinaryString(implicit ev: A <:< Boolean): String
}
```

This operation yields a string of '0' and '1' characters corresponding to the bit values:

```scala

val bits = Chunk(true, false, true, false)
```

The binary string shows the sequence of bits:

```scala
bits.toBinaryString
// res150: String = "1010"
```

#### `Chunk#toPackedByte` — Pack Bits into Bytes

Pack a chunk of booleans (representing individual bits) into a packed byte representation:

```scala
trait Chunk[+A] {
  def toPackedByte(implicit ev: A <:< Boolean): Chunk[Byte]
}
```

This is useful when you need a space-efficient representation of individual bits by packing them eight per byte. The bits are packed from left to right, with the first bit becoming the most significant bit of the first byte:

```scala

val bits = Chunk(true, false, true, false, true, false, true, false)
// bits: Chunk[Boolean] = IndexedSeq(
//   true,
//   false,
//   true,
//   false,
//   true,
//   false,
//   true,
//   false
// )
val packed = bits.toPackedByte
// packed: Chunk[Byte] = IndexedSeq(-86)
```

#### `Chunk#toPackedInt` — Pack Bits into Integers

Pack a chunk of booleans into a packed integer representation with specified endianness:

```scala
trait Chunk[+A] {
  def toPackedInt(endianness: Chunk.BitChunk.Endianness)(implicit ev: A <:< Boolean): Chunk[Int]
}
```

Packing bits into integers is efficient for storing large bit arrays. The endianness parameter controls how bits are ordered within each integer:

```scala

val bits = Chunk(true, false, true, false, true, false, true, false)
// bits: Chunk[Boolean] = IndexedSeq(
//   true,
//   false,
//   true,
//   false,
//   true,
//   false,
//   true,
//   false
// )
val packed = bits.toPackedInt(Chunk.BitChunk.Endianness.BigEndian)
// packed: Chunk[Int] = IndexedSeq(170)
```

#### `Chunk#toPackedLong` — Pack Bits into Longs

Pack a chunk of booleans into a packed long representation with specified endianness:

```scala
trait Chunk[+A] {
  def toPackedLong(endianness: Chunk.BitChunk.Endianness)(implicit ev: A <:< Boolean): Chunk[Long]
}
```

Packing bits into longs is the most space-efficient approach for bit storage, packing 64 bits per long. Choose the endianness that matches your serialization format:

```scala

val bits = Chunk(true, false, true, false, true, false, true, false)
// bits: Chunk[Boolean] = IndexedSeq(
//   true,
//   false,
//   true,
//   false,
//   true,
//   false,
//   true,
//   false
// )
val packed = bits.toPackedLong(Chunk.BitChunk.Endianness.LittleEndian)
// packed: Chunk[Long] = IndexedSeq(6124895493223874560L)
```

#### `Chunk#&` — Bitwise AND

Perform bitwise AND between two boolean chunks:

```scala
trait Chunk[+A] {
  def &(that: Chunk[Boolean])(implicit ev: A <:< Boolean): Chunk.BitChunkByte
}
```

Bitwise AND operates on boolean chunks, returning the packed result as a `Chunk[Boolean]` (its declared type, `Chunk.BitChunkByte`, is internal):

```scala

val bits1 = Chunk(true, true, false, false)
// bits1: Chunk[Boolean] = IndexedSeq(true, true, false, false)
val bits2 = Chunk(true, false, true, false)
// bits2: Chunk[Boolean] = IndexedSeq(true, false, true, false)
val result = bits1 & bits2
// result: BitChunkByte = IndexedSeq(true, false, false, false)
```

#### `Chunk#|` — Bitwise OR

Perform bitwise OR between two boolean chunks:

```scala
trait Chunk[+A] {
  def |(that: Chunk[Boolean])(implicit ev: A <:< Boolean): Chunk.BitChunkByte
}
```

Bitwise OR combines boolean chunks with an OR operation:

```scala

val bits1 = Chunk(true, false, false, false)
// bits1: Chunk[Boolean] = IndexedSeq(true, false, false, false)
val bits2 = Chunk(false, true, false, false)
// bits2: Chunk[Boolean] = IndexedSeq(false, true, false, false)
val result = bits1 | bits2
// result: BitChunkByte = IndexedSeq(true, true, false, false)
```

#### `Chunk#^` — Bitwise XOR

Perform bitwise XOR between two boolean chunks:

```scala
trait Chunk[+A] {
  def ^(that: Chunk[Boolean])(implicit ev: A <:< Boolean): Chunk.BitChunkByte
}
```

Bitwise XOR returns true when bits differ:

```scala

val bits1 = Chunk(true, false, true, false)
// bits1: Chunk[Boolean] = IndexedSeq(true, false, true, false)
val bits2 = Chunk(true, true, false, false)
// bits2: Chunk[Boolean] = IndexedSeq(true, true, false, false)
val result = bits1 ^ bits2
// result: BitChunkByte = IndexedSeq(false, true, true, false)
```

#### `Chunk#negate` — Bitwise NOT

Perform bitwise NOT (negation) on a boolean chunk:

```scala
trait Chunk[+A] {
  def negate(implicit ev: A <:< Boolean): Chunk.BitChunkByte
}
```

Bitwise NOT inverts all bits in the chunk:

```scala

val bits = Chunk(true, false, true, false)
// bits: Chunk[Boolean] = IndexedSeq(true, false, true, false)
val inverted = bits.negate
// inverted: BitChunkByte = IndexedSeq(false, true, false, true)
```

### Text Operations

Chunk makes it convenient to work with text by converting byte or character chunks into readable strings. You can also encode chunks as Base64 for safe transmission or storage in text-based formats:

#### `Chunk#asString` — Convert to String

Convert a chunk of bytes or characters into a string. Supports `Chunk[Byte]`, `Chunk[Char]`, and `Chunk[String]`:

```scala

val chars = Chunk('H', 'i')
val bytes = Chunk(72.toByte, 105.toByte)
```

Converting byte and character chunks to strings:

```scala
val str = chars.asString
// str: String = "Hi"
val strFromBytes = bytes.asString
// strFromBytes: String = "Hi"
```

#### `Chunk#asString(charset)` — Convert to String with Charset

Convert a chunk of bytes into a string using a specified character encoding:

```scala
trait Chunk[+A] {
  def asString(charset: Charset)(implicit ev: A <:< Byte): String
}
```

When working with byte chunks from external sources, you may need to specify the character encoding. Use UTF-8 for most cases, but other encodings are available:

```scala

val utf8Bytes = Chunk(72.toByte, 101.toByte, 108.toByte, 108.toByte, 111.toByte)
// utf8Bytes: Chunk[Byte] = IndexedSeq(72, 101, 108, 108, 111)
val stringUTF8 = utf8Bytes.asString(StandardCharsets.UTF_8)
// stringUTF8: String = "Hello"
```

#### `Chunk#asBase64String` — Encode as Base64

Convert a chunk of bytes or characters to a Base64-encoded string:

```scala

val bytes = Chunk(1.toByte, 2.toByte, 3.toByte)
// bytes: Chunk[Byte] = IndexedSeq(1, 2, 3)
val base64 = bytes.asBase64String
// base64: String = "AQID"
```

### Advanced Transformations

Chunk supports powerful transformation patterns for complex use cases. Collect elements while a condition holds, partition results into success and failure paths using Either, and attach indices to track element positions:

#### `Chunk#collectWhile` — Collect with Early Exit

Apply a partial function while a condition holds, stopping at the first non-match:

```scala

val values: Chunk[Any] = Chunk(1, 2, "x", 3, 4)
// values: Chunk[Any] = IndexedSeq(1, 2, "x", 3, 4)
val collected = values.collectWhile { case n: Int => n * 10 }
// collected: Chunk[Int] = IndexedSeq(10, 20)
```

#### `Chunk#partitionMap` — Partition into Two Chunks by Either Result

Partition elements by applying a total function that returns `Either`:

```scala
trait Chunk[+A] {
  def partitionMap[B, C](f: A => Either[B, C]): (Chunk[B], Chunk[C])
}
```

Partitioning elements by an Either result separates success and failure cases:

```scala

val numbers = Chunk(1, 2, 3, 4, 5)
// numbers: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5)
val (evens, odds) = numbers.partitionMap { n =>
  if (n % 2 == 0) Left(n) else Right(n)
}
// evens: Chunk[Int] = IndexedSeq(2, 4)
// odds: Chunk[Int] = IndexedSeq(1, 3, 5)
```

#### `Chunk#zipWithIndexFrom` — Zip with Index Starting at Custom Value

Zip elements with indices starting from a custom value:

```scala

val chunk = Chunk("a", "b", "c")
```

Zipping with a custom starting index pairs each element:

```scala
chunk.zipWithIndexFrom(10)
// res164: Chunk[Tuple2[String, Int]] = IndexedSeq(
//   ("a", 10),
//   ("b", 11),
//   ("c", 12)
// )
```

### Safe Operations

Chunks can be empty, and these operations help you handle that case gracefully. Rather than throwing exceptions or returning null, these methods provide type-safe fallbacks and alternatives when working with potentially empty chunks:

Return the chunk if non-empty, otherwise return an alternative:

```scala
trait Chunk[+A] {
  def nonEmptyOrElse[B](ifEmpty: => B)(fn: NonEmptyChunk[A] => B): B
}
```

Safely handling empty and non-empty cases provides type-safe alternatives:

```scala

val chunk = Chunk(1, 2, 3)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3)
val nonEmpty = NonEmptyChunk(1, 2, 3)
// nonEmpty: NonEmptyChunk[Int] = NonEmptyChunk(1, 2, 3)
val result = nonEmpty.nonEmptyOrElse(Chunk.empty[Int])(identity)
// result: Chunk[Int] | NonEmptyChunk[Int] = NonEmptyChunk(1, 2, 3)

val empty = Chunk.empty[Int]
// empty: Chunk[Int] = IndexedSeq()
val result2 = empty match {
  case c if c.isEmpty => Chunk(99)
  case c => c
}
// result2: Chunk[Int] = IndexedSeq(99)
```

## NonEmptyChunk

`NonEmptyChunk[A]` is a **type-safe wrapper** around `Chunk[A]` that **guarantees the chunk is non-empty**. It provides the same operations as `Chunk` but with methods like `head` and `last` returning `A` directly instead of `Option[A]` or throwing an exception. This eliminates the need for runtime checks on common operations and makes empty-case handling explicit at the type level.

`NonEmptyChunk[A]`:
- Is a purely functional, immutable sequence of at least one element
- Provides type-safe access to first and last elements
- Supports all Chunk operations while maintaining the non-empty guarantee
- Allows safe reduction operations without requiring a default value
- Integrates seamlessly with Chunk for interoperability

### Construction

Create a `NonEmptyChunk` directly using varargs syntax:

```scala
object NonEmptyChunk {
  def apply[A](a: A, as: A*): NonEmptyChunk[A]
}
```

Creating a non-empty chunk with direct constructor:

```scala

val nonEmpty = NonEmptyChunk(1, 2, 3)
// nonEmpty: NonEmptyChunk[Int] = NonEmptyChunk(1, 2, 3)
```

Create a `NonEmptyChunk` from an existing `Chunk` using `fromChunk`, which returns `Option[NonEmptyChunk[A]]`:

```scala

val chunk = Chunk(1, 2, 3)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3)
val maybeNonEmpty: Option[NonEmptyChunk[Int]] = NonEmptyChunk.fromChunk(chunk)
// maybeNonEmpty: Option[NonEmptyChunk[Int]] = Some(NonEmptyChunk(1, 2, 3))

val empty = Chunk.empty[Int]
// empty: Chunk[Int] = IndexedSeq()
val nothingHere: Option[NonEmptyChunk[Int]] = NonEmptyChunk.fromChunk(empty)
// nothingHere: Option[NonEmptyChunk[Int]] = None
```

Create from a Scala cons list using `fromCons` (requires a non-empty cons list):

```scala

val list = 1 :: 2 :: 3 :: Nil
// list: List[Int] = List(1, 2, 3)
val nonEmpty = list match {
  case cons: scala.collection.immutable.::[Int] => NonEmptyChunk.fromCons(cons)
  case _ => throw new IllegalArgumentException("list must be non-empty")
}
// nonEmpty: NonEmptyChunk[Int] = NonEmptyChunk(1, 2, 3)
```

Create from an iterable with at least one guaranteed element using `fromIterable`:

```scala

val nonEmpty = NonEmptyChunk.fromIterable(5, List(1, 2, 3, 4))
// nonEmpty: NonEmptyChunk[Int] = NonEmptyChunk(5, 1, 2, 3, 4)
```

### Safe Access to Endpoints

Access the first and last elements without risk of exception:

```scala
trait NonEmptyChunk[+A] {
  def head: A
  def last: A
}
```

Safely access endpoints of a guaranteed non-empty chunk:

```scala

val chunk = NonEmptyChunk(10, 20, 30, 40)
// chunk: NonEmptyChunk[Int] = NonEmptyChunk(10, 20, 30, 40)

chunk.head
// res171: Int = 10
chunk.last
// res172: Int = 40
chunk.size
// res173: Int = 4
```

### Reduction and Aggregation

Reduce a non-empty chunk without requiring a starting value, since the chunk is guaranteed to have at least one element:

```scala
trait NonEmptyChunk[+A] {
  def reduce[B >: A](op: (B, B) => B): B
  def reduceMapLeft[B](map: A => B)(reduce: (B, A) => B): B
  def reduceMapRight[B](map: A => B)(reduce: (A, B) => B): B
}
```

Reduction operations always produce a value (no Optional needed):

```scala

val numbers = NonEmptyChunk(1, 2, 3, 4, 5)
// numbers: NonEmptyChunk[Int] = NonEmptyChunk(1, 2, 3, 4, 5)

val sum = numbers.reduce(_ + _)
// sum: Int = 15

val product = numbers.reduceMapLeft[Int](identity)(_ * _)
// product: Int = 120
```

### Transformations

Map and transform elements while preserving the non-empty guarantee:

```scala
trait NonEmptyChunk[+A] {
  def map[B](f: A => B): NonEmptyChunk[B]
  def flatMap[B](f: A => NonEmptyChunk[B]): NonEmptyChunk[B]
  def flatten[B](implicit ev: A <:< NonEmptyChunk[B]): NonEmptyChunk[B]
  def sorted[B >: A](implicit ord: Ordering[B]): NonEmptyChunk[B]
  def sortBy[B](f: A => B)(implicit ord: Ordering[B]): NonEmptyChunk[A]
  def distinct: NonEmptyChunk[A]
  def reverse: NonEmptyChunk[A]
}
```

Transformations maintain the non-empty property for map, flatMap, and other structure-preserving operations:

```scala

val numbers = NonEmptyChunk(3, 1, 4, 1, 5)
// numbers: NonEmptyChunk[Int] = NonEmptyChunk(3, 1, 4, 1, 5)

val doubled = numbers.map(_ * 2)
// doubled: NonEmptyChunk[Int] = NonEmptyChunk(6, 2, 8, 2, 10)

val sorted = numbers.sorted
// sorted: NonEmptyChunk[Int] = NonEmptyChunk(1, 1, 3, 4, 5)
```

### Grouping and Aggregation

Group elements while maintaining non-empty chunks in each group:

```scala
trait NonEmptyChunk[+A] {
  def groupBy[K](f: A => K): Map[K, NonEmptyChunk[A]]
  def groupMap[K, V](key: A => K)(f: A => V): Map[K, NonEmptyChunk[V]]
  def grouped(size: Int): Iterator[NonEmptyChunk[A]]
}
```

Grouping operations guarantee non-empty result chunks:

```scala

case class Person(name: String, age: Int)
val people = NonEmptyChunk(Person("Alice", 30), Person("Bob", 25), Person("Carol", 30))
// people: NonEmptyChunk[Person] = NonEmptyChunk(Person(Alice,30), Person(Bob,25), Person(Carol,30))

people.groupBy(_.age)
// res177: Map[Int, NonEmptyChunk[Person]] = Map(
//   25 -> NonEmptyChunk(Person(Bob,25)),
//   30 -> NonEmptyChunk(Person(Alice,30), Person(Carol,30))
// )
```

### Combining and Concatenating

Extend a non-empty chunk with additional elements or other chunks:

```scala
trait NonEmptyChunk[+A] {
  def appended[A1 >: A](a: A1): NonEmptyChunk[A1]
  def prepended[A1 >: A](a: A1): NonEmptyChunk[A1]
  def ++[A1 >: A](that: Chunk[A1]): NonEmptyChunk[A1]
  def :+[A1 >: A](a: A1): NonEmptyChunk[A1]  // alias for appended
  def +:[A1 >: A](a: A1): NonEmptyChunk[A1]  // alias for prepended
}
```

Concatenation operations preserve the non-empty guarantee:

```scala

val nonEmpty = NonEmptyChunk(1, 2, 3)
// nonEmpty: NonEmptyChunk[Int] = NonEmptyChunk(1, 2, 3)
val chunk = Chunk(4, 5, 6)
// chunk: Chunk[Int] = IndexedSeq(4, 5, 6)

val appended = nonEmpty :+ 4
// appended: NonEmptyChunk[Int] = NonEmptyChunk(1, 2, 3, 4)

val concatenated = nonEmpty ++ chunk
// concatenated: NonEmptyChunk[Int] = NonEmptyChunk(1, 2, 3, 4, 5, 6)

val prepended = 0 +: nonEmpty
// prepended: NonEmptyChunk[Int] = NonEmptyChunk(0, 1, 2, 3)
```

### Conversion to Chunk

Convert a `NonEmptyChunk` back to a regular `Chunk`, losing the non-empty guarantee but gaining compatibility with Chunk APIs:

```scala
trait NonEmptyChunk[+A] {
  def toChunk: Chunk[A]
}
```

Converting back to a `Chunk` for use with generic Chunk operations:

```scala

val nonEmpty = NonEmptyChunk(1, 2, 3)
// nonEmpty: NonEmptyChunk[Int] = NonEmptyChunk(1, 2, 3)
val chunk: Chunk[Int] = nonEmpty.toChunk
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3)

// Now you can use Chunk operations that return Chunk[A] instead of NonEmptyChunk[A]
val filtered: Chunk[Int] = chunk.filter(_ > 1)
// filtered: Chunk[Int] = IndexedSeq(2, 3)
```

### Conversion to Scala Cons List

Convert a `NonEmptyChunk` to a Scala cons list (`::[A]`), maintaining the non-empty guarantee:

```scala
trait NonEmptyChunk[+A] {
  def toCons[A1 >: A]: ::[A1]
}
```

Since `NonEmptyChunk` is guaranteed to be non-empty, conversion to Scala's cons list is safe and always returns a `::` (non-empty list) rather than `Nil`:

```scala

val nonEmpty = NonEmptyChunk(1, 2, 3)
// nonEmpty: NonEmptyChunk[Int] = NonEmptyChunk(1, 2, 3)
val consList: scala.collection.immutable.::[Int] = nonEmpty.toCons
// consList: ::[Int] = List(1, 2, 3)

// Access head and tail like a normal cons cell
consList.head
// res181: Int = 1
consList.tail
// res182: List[Int] = List(2, 3)
```

**Use case:** Interoperability with code expecting Scala cons lists; recursive list processing that relies on the structure being non-empty.

### Integration with Chunk Operations

Many Chunk operations are available on `NonEmptyChunk` and return `NonEmptyChunk` when the structure is guaranteed to remain non-empty:

```scala

val chunk = NonEmptyChunk(1, 2, 3, 4, 5)
// chunk: NonEmptyChunk[Int] = NonEmptyChunk(1, 2, 3, 4, 5)

// These return NonEmptyChunk[A]
val mapped = chunk.map(_ * 2)
// mapped: NonEmptyChunk[Int] = NonEmptyChunk(2, 4, 6, 8, 10)

val sorted = chunk.sorted
// sorted: NonEmptyChunk[Int] = NonEmptyChunk(1, 2, 3, 4, 5)

// These return Chunk[A] (since size might change)
val filtered = chunk.toChunk.filter(_ > 2)
// filtered: Chunk[Int] = IndexedSeq(3, 4, 5)
```

## ChunkMap

`ChunkMap[K, V]` is an order-preserving immutable map backed by parallel chunks. It maintains insertion order during iteration:

The type defines the following members:

```scala
object ChunkMap {
  def empty[K, V]: ChunkMap[K, V]
  def apply[K, V](elems: (K, V)*): ChunkMap[K, V]
  def fromChunk[K, V](chunk: Chunk[(K, V)]): ChunkMap[K, V]
  def fromChunks[K, V](keys: Chunk[K], values: Chunk[V]): ChunkMap[K, V]
  def newBuilder[K, V]: Builder[(K, V), ChunkMap[K, V]]
  class Indexed[K, V](val underlying: ChunkMap[K, V]) { ... }
}
```

We can create a simple map using the `apply` method and then perform basic operations like retrieving a value by key, updating a key with a new value, or removing a key:

```scala

val map = ChunkMap("a" -> 1, "b" -> 2, "c" -> 3)
// map: ChunkMap[String, Int] = Map("a" -> 1, "b" -> 2, "c" -> 3)
map.get("b")
// res185: Option[Int] = Some(2)
map.updated("d", 4)
// res186: ChunkMap[String, Int] = Map("a" -> 1, "b" -> 2, "c" -> 3, "d" -> 4)
map.removed("b")
// res187: ChunkMap[String, Int] = Map("a" -> 1, "c" -> 3)
```

### Creating ChunkMap

`ChunkMap` provides several factory methods for construction from different sources:

The companion object defines these construction methods:

```scala
object ChunkMap {
  def empty[K, V]: ChunkMap[K, V]
  def apply[K, V](elems: (K, V)*): ChunkMap[K, V]
  def fromChunk[K, V](chunk: Chunk[(K, V)]): ChunkMap[K, V]
  def fromChunks[K, V](keys: Chunk[K], values: Chunk[V]): ChunkMap[K, V]
}
```

Create an empty map for a specific key and value type:

```scala

val empty = ChunkMap.empty[String, Int]
// empty: ChunkMap[String, Int] = Map()
```

Construct a map directly from key-value pairs:

```scala
val fromPairs = ChunkMap("x" -> 1, "y" -> 2)
// fromPairs: ChunkMap[String, Int] = Map("x" -> 1, "y" -> 2)
```

Build a map from a `Chunk` containing tuple pairs:

```scala

val fromChunk = ChunkMap.fromChunk(Chunk(("a", 1), ("b", 2)))
// fromChunk: ChunkMap[String, Int] = Map("a" -> 1, "b" -> 2)
```

Construct a map from parallel chunks of keys and values:

```scala

val keys = Chunk("a", "b")
// keys: Chunk[String] = IndexedSeq("a", "b")
val values = Chunk(1, 2)
// values: Chunk[Int] = IndexedSeq(1, 2)
val fromChunks = ChunkMap.fromChunks(keys, values)
// fromChunks: ChunkMap[String, Int] = Map("a" -> 1, "b" -> 2)
```

### Indexed Access

Use positional access to retrieve entries by their insertion order index:

The `ChunkMap` class exposes these methods:

```scala
trait ChunkMap[K, V] {
  def atIndex(idx: Int): (K, V)
  def keyAtIndex(idx: Int): K
  def valueAtIndex(idx: Int): V
  def keysChunk: Chunk[K]
  def valuesChunk: Chunk[V]
}
```

Create a map and experiment with positional access:

```scala

val map = ChunkMap("z" -> 1, "a" -> 2, "m" -> 3)
// map: ChunkMap[String, Int] = Map("z" -> 1, "a" -> 2, "m" -> 3)
```

Retrieve the complete key-value pair at a given index:

```scala
map.atIndex(0)
// res192: Tuple2[String, Int] = ("z", 1)
```

Retrieve just the key at a specific position:

```scala
map.keyAtIndex(1)
// res193: String = "a"
```

Retrieve just the value at a specific position:

```scala
map.valueAtIndex(2)
```

Access the underlying chunks of keys and values:

```scala
val keys: Chunk[String] = map.keysChunk
// keys: Chunk[String] = IndexedSeq("z", "a", "m")
val values: Chunk[Int] = map.valuesChunk
// values: Chunk[Int] = IndexedSeq(1, 2, 3)
```

### Optimized Lookup

The standard `ChunkMap` uses O(n) linear search for key lookups. For frequent lookups, convert to an indexed version that provides O(1) key access:

The indexed wrapper type is nested inside `ChunkMap`:

```scala
trait ChunkMap[K, V] {
  def indexed: ChunkMap.Indexed[K, V]
}
```

Construction of the indexed wrapper builds an internal hash map for constant-time lookups, using extra memory proportional to the number of entries:

```scala

val map = ChunkMap("a" -> 1, "b" -> 2, "c" -> 3)
// map: ChunkMap[String, Int] = Map("a" -> 1, "b" -> 2, "c" -> 3)
val indexed = map.indexed
// indexed: Indexed[String, Int] = Map("a" -> 1, "b" -> 2, "c" -> 3)
indexed.get("b")
// res195: Option[Int] = Some(2)
```

## Comparison with Other Sequence Types

Understanding how Chunk compares to other sequence types helps you choose the right tool for your use case:

### Chunk vs `Array`

| Feature           | Chunk                        | Array                                |
|-------------------|------------------------------|--------------------------------------|
| **Immutability**  | Immutable, purely functional | Mutable, imperative                  |
| **Concatenation** | O(log n) via balanced trees  | O(n) requires copying                |
| **Random Access** | O(1) typical, O(log n) worst | O(1) always                          |
| **Safe API**      | Pure, no side effects        | Low-level, requires careful handling |
| **Boxing**        | Avoids boxing primitives     | Supports native primitives           |
| **Lazy Ops**      | Yes (concatenation deferred) | No (eager)                           |

**Use Chunk when**: Building sequences functionally, using in pure code, performing many concatenations, or sharing immutable data.

**Use `Array` when**: Needing low-level memory control, interfacing with Java, or maximum raw performance is critical.

### Chunk vs `List`

| Feature           | Chunk                        | List                  |
|-------------------|------------------------------|-----------------------|
| **Access**        | O(1) random access           | O(n) linear search    |
| **Prepend**       | O(log n)                     | O(1)                  |
| **Memory**        | Compact arrays               | Linked nodes          |
| **Pattern Match** | Not directly                 | Cons pattern matching |
| **Interop**       | Scala collections compatible | Standard Scala type   |

**Use Chunk when**: Frequent random access or concatenation is important.

**Use List when**: Head/tail pattern matching or traditional functional programming style.

### Chunk vs `Vector`

| Feature            | Chunk                        | Vector                    |
|--------------------|------------------------------|---------------------------|
| **Concatenation**  | O(log n) with rebalancing    | O(log₃₂ n) trie structure |
| **Mutation**       | Immutable, purely functional | Effectively immutable     |
| **Memory**         | Lower overhead for many ops  | Higher memory footprint   |
| **Specialization** | Primitive specialization     | None, uses boxing         |
| **Random Access**  | O(1) typical, O(log n) worst | O(log₃₂ n)                |

**Use Chunk when**: Primitive types matter or concatenation performance is critical.

**Use `Vector` when**: Need guaranteed access performance or already using `Vector` in codebase.

### Additional Core Operations

Chunk provides several additional methods for checking properties, materializing data, and handling non-empty conversions:

#### `Chunk#isEmpty` — Check if Chunk is Empty

Test whether the chunk contains no elements:

```scala
trait Chunk[+A] {
  def isEmpty: Boolean
}
```

Checking emptiness is a fast operation:

```scala

val full = Chunk(1, 2, 3)
// full: Chunk[Int] = IndexedSeq(1, 2, 3)
val empty = Chunk.empty[Int]
// empty: Chunk[Int] = IndexedSeq()

full.isEmpty
// res197: Boolean = false
empty.isEmpty
// res198: Boolean = true
```

**Performance:** O(1) — size is cached.

#### `Chunk#materialize` — Force Full Evaluation

Force the chunk to materialize fully. Useful when you need to ensure lazy operations are evaluated:

```scala
trait Chunk[+A] {
  def materialize[A1 >: A]: Chunk[A1]
}
```

Materializing evaluates all lazy operations:

```scala

val chunk = Chunk(1, 2, 3)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3)
val materialized = chunk.materialize
// materialized: Chunk[Int] = IndexedSeq(1, 2, 3)
```

**Performance:** O(n) — may trigger deferred operations.

#### `Chunk#nonEmptyOrElse` — Safe NonEmptyChunk Conversion

Convert to a `NonEmptyChunk` with a fallback for empty chunks:

```scala
trait Chunk[+A] {
  def nonEmptyOrElse[B](ifEmpty: => B)(fn: NonEmptyChunk[A] => B): B
}
```

Converting safely to non-empty with a fallback:

```scala

val chunk = Chunk(1, 2, 3)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3)
val result = chunk.nonEmptyOrElse("empty")(ne => s"non-empty with ${ne.length} elements")
// result: String = "non-empty with 3 elements"

val empty = Chunk.empty[Int]
// empty: Chunk[Int] = IndexedSeq()
val emptyResult = empty.nonEmptyOrElse("no elements")(ne => "should not see this")
// emptyResult: String = "no elements"
```

**Performance:** O(1) — no iteration needed.

#### `Chunk#updated` — Functional Update at Index

Create a new chunk with an element updated at a specific index:

```scala
trait Chunk[+A] {
  def updated[A1 >: A](index: Int, elem: A1): Chunk[A1]
}
```

Updating at an index returns a new chunk with the element replaced:

```scala

val original = Chunk(10, 20, 30, 40)
// original: Chunk[Int] = IndexedSeq(10, 20, 30, 40)
val updated = original.updated(1, 25)
// updated: Chunk[Int] = IndexedSeq(10, 25, 30, 40)
```

**Performance:** O(log n) for tree-structured chunks; O(n) copy for large arrays.

### Type-Safe Primitive Accessors

Chunk provides type-safe accessors to extract primitive values at a specific index. These methods combine index access with type assertion in a single operation:

#### `Chunk#boolean`, `Chunk#byte`, `Chunk#char`, `Chunk#short`, `Chunk#int`, `Chunk#long`, `Chunk#float`, `Chunk#double` — Extract Primitive by Type and Index

Extract a primitive value of a specific type at a given index:

```scala
trait Chunk[+A] {
  def boolean(index: Int)(implicit ev: A <:< Boolean): Boolean
  def byte(index: Int)(implicit ev: A <:< Byte): Byte
  def char(index: Int)(implicit ev: A <:< Char): Char
  def short(index: Int)(implicit ev: A <:< Short): Short
  def int(index: Int)(implicit ev: A <:< Int): Int
  def long(index: Int)(implicit ev: A <:< Long): Long
  def float(index: Int)(implicit ev: A <:< Float): Float
  def double(index: Int)(implicit ev: A <:< Double): Double
}
```

These methods provide type-safe access to primitive values:

```scala

val boolChunk = Chunk(true, false, true)
// boolChunk: Chunk[Boolean] = IndexedSeq(true, false, true)
val firstBool = boolChunk.boolean(0)
// firstBool: Boolean = true

val intChunk = Chunk(10, 20, 30)
// intChunk: Chunk[Int] = IndexedSeq(10, 20, 30)
val secondInt = intChunk.int(1)
// secondInt: Int = 20

val doubleChunk = Chunk(1.5, 2.5, 3.5)
// doubleChunk: Chunk[Double] = IndexedSeq(1.5, 2.5, 3.5)
val thirdDouble = doubleChunk.double(2)
// thirdDouble: Double = 3.5
```

**Performance:** O(log n) typical; O(1) for array-backed chunks.

### Reduction Methods

Advanced reduction operations for folding with different semantics. These methods are available on `NonEmptyChunk` (which wraps a `Chunk` with the guarantee of at least one element):

#### `NonEmptyChunk#reduce` — Fold Without Initial Value

Reduce a non-empty chunk using a binary function. Since the chunk is guaranteed non-empty, no initial value is needed:

```scala
final class NonEmptyChunk[+A] {
  def reduce[B >: A](op: (B, B) => B): B
}
```

This method is useful when you want to compute an aggregate value without providing an initial state. Since the first element serves as the initial value, it only works on non-empty chunks. 

**Use case:** Computing products, finding max/min, or other associative operations on guaranteed non-empty data.

**Performance:** O(n) — processes all elements sequentially.

#### `NonEmptyChunk#reduceMapLeft` — Reduce with Left Map

Reduce elements using a function that first maps the first element, then reduces with a binary operator. This is useful when the result type differs from the element type:

```scala
final class NonEmptyChunk[+A] {
  def reduceMapLeft[B](map: A => B)(reduce: (B, A) => B): B
}
```

Example usage pattern:

```scala

val nonEmpty = NonEmptyChunk(10, 20, 30, 40)
// nonEmpty: NonEmptyChunk[Int] = NonEmptyChunk(10, 20, 30, 40)
val result: String = nonEmpty.reduceMapLeft(_.toString)((acc, n) => acc + ", " + n)
// result: String = "10, 20, 30, 40"
result
// res204: String = "10, 20, 30, 40"
```

**Use case:** Transforming and aggregating data in a single pass with type conversion.

**Performance:** O(n) — processes all elements with mapping overhead.

#### `NonEmptyChunk#reduceMapRight` — Reduce with Right Map

Reduce elements from right to left, mapping the rightmost element first. This is right-associative:

```scala
final class NonEmptyChunk[+A] {
  def reduceMapRight[B](map: A => B)(reduce: (A, B) => B): B
}
```

Example usage pattern:

```scala

val nonEmpty = NonEmptyChunk(1, 2, 3, 4)
// nonEmpty: NonEmptyChunk[Int] = NonEmptyChunk(1, 2, 3, 4)
val result: String = nonEmpty.reduceMapRight(_.toString)((n, acc) => n.toString + ", " + acc)
// result: String = "1, 2, 3, 4"
result
// res206: String = "1, 2, 3, 4"
```

**Use case:** Right-associative operations like building cons-lists or reverse-order processing.

**Performance:** O(n) — processes all elements right-to-left.

### Additional Utility Methods

Beyond the core operations, `Chunk` provides utility methods for inspecting and extracting underlying data:

## Integration

Chunk integrates deeply with ZIO Blocks' schema system through the [Reflect](./schema/reflect.md) module. When deriving schemas for collection types, `Chunk` is recognized as a key sequence type alongside `List`, `Vector`, and `Set`.

Here's an example of using `Chunk` with schema derivation:

```scala

case class Event(id: Int, tags: Chunk[String])

object Event {
  implicit val schema: Schema[Event] = Schema.derived
}
// Automatically derives Schema[Chunk[String]] for the tags field
```

Chunks work naturally with the [Codec](./schema/codec.md) system for serialization and deserialization:

```scala

val data = Chunk(1, 2, 3)
// data: Chunk[Int] = IndexedSeq(1, 2, 3)

// Encoding to JSON
val json = Json.Array(data.map(i => Json.Number(i)))
// json: Array = Array(IndexedSeq(Number(1), Number(2), Number(3)))

// Decoding from JSON
val decoded: Option[Chunk[Int]] = json match {
  case Json.Array(elements) =>
    Some(Chunk.fromIterable(elements.collect { case Json.Number(n) => n.toInt }))
}
// decoded: Option[Chunk[Int]] = Some(IndexedSeq(1, 2, 3))
```

The [DynamicValue](./schema/dynamic-value.md) system also works with Chunk, allowing schema-driven navigation of chunk data.

See also [How to Avoid Allocation on Hot Paths](../guides/avoid-allocation-on-hot-paths.md) for reading and building chunks without boxing.

## Running the Examples

All code from this guide is available as runnable examples in the appropriate example modules.

To run the examples locally, clone the repository and navigate to the project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

Then build and test with sbt:

```bash
# Compile everything
sbt compile

# Run tests
sbt test

# Specifically test Chunk functionality
sbt chunk/test
```

Chunk examples are integrated throughout the test suites. You can also explore the test file at `chunk/shared/src/test/scala/zio/blocks/chunk/ChunkSpec.scala` to see idiomatic usage patterns.

---

## CaseClass

`CaseClass` represents an immutable case class in the IR. Use it when generating Scala code from data models, APIs, or structured data—it's the most frequently chosen type definition.

## Use Cases

- Modeling product types (records) from OpenAPI schemas, JSON Schema, or data formats
- Defining data transfer objects (DTOs) with named fields
- Creating immutable value types with automatic `equals`, `hashCode`, `copy`

## Construction

Build a case class with a name and field list:

```scala

val person = CaseClass(
  name = "Person",
  fields = List(
    Field("id", TypeRef.Long),
    Field("name", TypeRef.String),
    Field("email", TypeRef.String)
  )
)
```

With optional type parameters:

```scala

val box = CaseClass(
  name = "Box",
  fields = List(Field("value", TypeRef("T"))),
  typeParams = List(TypeParam("T"))
)
```

With derives clauses:

```scala

val user = CaseClass(
  name = "User",
  fields = List(
    Field("id", TypeRef.Long),
    Field("name", TypeRef.String)
  ),
  derives = List("Show", "Eq", "Codec")
)
```

## Key Operations

All `CaseClass` instances support these core operations:

### Accessing Components

Extract parts of a case class:

```scala

val person = CaseClass(
  name = "Person",
  fields = List(
    Field("id", TypeRef.Long),
    Field("name", TypeRef.String),
    Field("email", TypeRef.String)
  )
)
```

Access each component by reading their properties:

```scala

val person = CaseClass(
  name = "Person",
  fields = List(
    Field("id", TypeRef.Long),
    Field("name", TypeRef.String),
    Field("email", TypeRef.String)
  )
)
```

Each component can be accessed and displayed:

```scala

person.name            // "Person"
// res4: String = "Person"
person.fields          // List[Field]
// res5: List[Field] = List(
//   Field(
//     name = "id",
//     typeRef = TypeRef(name = "Long", typeArgs = List()),
//     defaultValue = None,
//     annotations = List(),
//     doc = None
//   ),
//   Field(
//     name = "name",
//     typeRef = TypeRef(name = "String", typeArgs = List()),
//     defaultValue = None,
//     annotations = List(),
//     doc = None
//   ),
//   Field(
//     name = "email",
//     typeRef = TypeRef(name = "String", typeArgs = List()),
//     defaultValue = None,
//     annotations = List(),
//     doc = None
//   )
// )
person.typeParams      // List[TypeParam] (empty if not generic)
// res6: List[TypeParam] = List()
person.derives         // List[String]
// res7: List[String] = List()
person.annotations     // List[Annotation]
// res8: List[Annotation] = List()
person.isValueClass    // Boolean
// res9: Boolean = false
```

### Building with Copy

Use the copy method to modify a case class:

```scala

val person = CaseClass(
  name = "Person",
  fields = List(
    Field("id", TypeRef.Long),
    Field("name", TypeRef.String),
    Field("email", TypeRef.String)
  )
)
```

Create a new version with additional fields:

```scala

val extended = person.copy(
  fields = person.fields :+ Field("phone", TypeRef.String),
  derives = List("Show")
)
// extended: CaseClass = CaseClass(
//   name = "Person",
//   fields = List(
//     Field(
//       name = "id",
//       typeRef = TypeRef(name = "Long", typeArgs = List()),
//       defaultValue = None,
//       annotations = List(),
//       doc = None
//     ),
//     Field(
//       name = "name",
//       typeRef = TypeRef(name = "String", typeArgs = List()),
//       defaultValue = None,
//       annotations = List(),
//       doc = None
//     ),
//     Field(
//       name = "email",
//       typeRef = TypeRef(name = "String", typeArgs = List()),
//       defaultValue = None,
//       annotations = List(),
//       doc = None
//     ),
//     Field(
//       name = "phone",
//       typeRef = TypeRef(name = "String", typeArgs = List()),
//       defaultValue = None,
//       annotations = List(),
//       doc = None
//     )
//   ),
//   typeParams = List(),
//   extendsTypes = List(),
//   derives = List("Show"),
//   annotations = List(),
//   companion = None,
//   doc = None,
//   isValueClass = false
// )
```

### Adding a Companion Object

Case classes can have companion objects with static members:

```scala

val withCompanion = CaseClass(
  name = "Config",
  fields = List(Field("timeout", TypeRef.Long)),
  companion = Some(
    CompanionObject(
      members = List(
        ObjectMember.ValMember("Default", TypeRef("Config"), "Config(5000L)")
      )
    )
  )
)
```

## Examples

Here are concrete examples demonstrating typical `CaseClass` usage patterns:

### Example 1: Simple Case Class

A basic case class with primitive fields:

```scala

val order = CaseClass(
  name = "Order",
  fields = List(
    Field("id", TypeRef.Long),
    Field("total", TypeRef("BigDecimal")),
    Field("status", TypeRef.String)
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.shop"),
  types = List(order)
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res13: String = """package com.shop
// 
// case class Order(
//   id: Long,
//   total: BigDecimal,
//   status: String,
// )
// """
```

### Example 2: Nested Types

A case class with optional and collection fields:

```scala

val user = CaseClass(
  name = "User",
  fields = List(
    Field("id", TypeRef.Long),
    Field("name", TypeRef.String),
    Field("email", TypeRef("Option", List(TypeRef.String))),
    Field("tags", TypeRef("List", List(TypeRef.String)))
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(user)
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res15: String = """package com.example
// 
// case class User(
//   id: Long,
//   name: String,
//   email: Option[String],
//   tags: List[String],
// )
// """
```

### Example 3: Generic Case Class

A polymorphic case class with type parameters:

```scala

val page = CaseClass(
  name = "Page",
  fields = List(
    Field("items", TypeRef("List", List(TypeRef("T")))),
    Field("total", TypeRef.Long),
    Field("pageSize", TypeRef.Int)
  ),
  typeParams = List(TypeParam("T"))
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(page)
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res17: String = """package com.example
// 
// case class Page[T](
//   items: List[T],
//   total: Long,
//   pageSize: Int,
// )
// """
```

### Example 4: Case Class with Derives

Automatic typeclass derivation:

```scala

val product = CaseClass(
  name = "Product",
  fields = List(
    Field("id", TypeRef.String),
    Field("name", TypeRef.String),
    Field("price", TypeRef("java.math.BigDecimal"))
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(product)
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res19: String = """package com.example
// 
// case class Product(
//   id: String,
//   name: String,
//   price: java.math.BigDecimal,
// )
// """
```

### Example 5: Case Class with Companion

A case class with factory methods in the companion:

```scala

val config = CaseClass(
  name = "AppConfig",
  fields = List(
    Field("host", TypeRef.String),
    Field("port", TypeRef.Int)
  ),
  companion = Some(
    CompanionObject(
      members = List(
        ObjectMember.ValMember(
          "Default",
          TypeRef("AppConfig"),
          "AppConfig(\"localhost\", 8080)"
        )
      )
    )
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(config)
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res21: String = """package com.example
// 
// case class AppConfig(
//   host: String,
//   port: Int,
// )
// 
// object AppConfig {
//   val Default: AppConfig = AppConfig("localhost", 8080)
// }
// """
```

---

## EmitterConfig

`EmitterConfig` controls how `ScalaEmitter` formats Scala code. Customize indentation, import sorting, trailing commas, and target Scala version (2 vs 3) by creating a configured instance.

## Use Cases

- Configuring code style (indentation, commas) to match your project conventions
- Targeting Scala 3 features (enums, derives) or Scala 2 fallback syntax
- Controlling import statement ordering
- Emitting code for different codebases with different style preferences

## Configuration Fields

| Field            | Type    | Default | Description                             |
|------------------|---------|---------|-----------------------------------------|
| `indentWidth`    | Int     | 2       | Spaces per indentation level            |
| `sortImports`    | Boolean | true    | Sort import statements alphabetically   |
| `trailingCommas` | Boolean | true    | Add trailing commas in collections      |
| `scala3Syntax`   | Boolean | true    | Target Scala 3 syntax features          |

## Construction

Create a default configuration:

```scala

val default = EmitterConfig()
// indentWidth=2, sortImports=true, trailingCommas=true, scala3Syntax=true
```

With custom settings:

```scala

val custom = EmitterConfig(
  indentWidth = 4,
  sortImports = true,
  trailingCommas = false,
  scala3Syntax = false
)
```

## Key Operations

All `EmitterConfig` instances provide these operations:

### Accessing Configuration

Read configuration values:

```scala

val config = EmitterConfig(indentWidth = 4)

config.indentWidth      // 4
config.sortImports      // true (default)
config.trailingCommas   // true (default)
config.scala3Syntax     // true (default)
```

### Building with Copy

Modify a configuration:

```scala

val base = EmitterConfig()

val scala2Config = base.copy(
  scala3Syntax = false
)

val wideIndentConfig = base.copy(
  indentWidth = 4
)
```

## Examples

Here are practical examples showing different configuration scenarios:

### Example 1: Standard Configuration

The default configuration for most projects:

```scala

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(
    CaseClass("User", List(
      Field("id", TypeRef.Long),
      Field("name", TypeRef.String)
    ))
  )
)

val config = EmitterConfig()
```

Emits:

```scala

ScalaEmitter.emit(file, config)
// res5: String = """package com.example
// 
// case class User(
//   id: Long,
//   name: String,
// )
// """
```

### Example 2: Wide Indentation (4 spaces)

For projects preferring 4-space indentation:

```scala

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(
    SealedTrait(
      "Result",
      cases = List(
        SealedTraitCase.CaseClassCase(
          CaseClass("Success", List(
            Field("value", TypeRef.String)
          ))
        ),
        SealedTraitCase.CaseObjectCase("Failure")
      )
    )
  )
)

val config = EmitterConfig(indentWidth = 4)
```

Emits with 4-space indentation:

```scala

ScalaEmitter.emit(file, config)
// res7: String = """package com.example
// 
// sealed trait Result
// 
// object Result {
//     case class Success(
//         value: String,
//     ) extends Result
//     case object Failure extends Result
// }
// """
```

### Example 3: Scala 2 Compatibility

Target Scala 2 syntax for older codebases:

```scala

val file = ScalaFile(
  packageDecl = PackageDecl("com.legacy"),
  imports = List(
    Import.WildcardImport("scala.collection")
  ),
  types = List(
    Enum(
      name = "Color",
      cases = List(
        EnumCase.SimpleCase("Red"),
        EnumCase.SimpleCase("Blue")
      )
    )
  )
)

val scala2Config = EmitterConfig(
  scala3Syntax = false
)
```

Emits sealed trait syntax (Scala 2 compatible):

```scala

ScalaEmitter.emit(file, scala2Config)
// res9: String = """package com.legacy
// 
// import scala.collection._
// 
// sealed trait Color
// 
// object Color {
//   case object Red extends Color
//   case object Blue extends Color
// }
// """
```

### Example 4: No Trailing Commas

For projects with strict no-trailing-commas rules:

```scala

val file = ScalaFile(
  packageDecl = PackageDecl("com.strict"),
  types = List(
    CaseClass(
      "Options",
      List(
        Field("a", TypeRef.String),
        Field("b", TypeRef.Int),
        Field("c", TypeRef.Boolean)
      )
    )
  )
)

val config = EmitterConfig(
  trailingCommas = false
)
```

Emits without trailing comma on last field:

```scala

ScalaEmitter.emit(file, config)
// res11: String = """package com.strict
// 
// case class Options(
//   a: String,
//   b: Int,
//   c: Boolean
// )
// """
```

### Example 5: Unsorted Imports

For projects that manage imports manually:

```scala

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  imports = List(
    Import.WildcardImport("zio"),
    Import.SingleImport("scala.collection", "Seq"),
    Import.WildcardImport("scala")
  ),
  types = List(
    CaseClass("Data", List(
      Field("items", TypeRef("Seq", List(TypeRef.String)))
    ))
  )
)

val config = EmitterConfig(
  sortImports = false  // Keep imports in declaration order
)
```

Preserves import order:

```scala

ScalaEmitter.emit(file, config)
// res13: String = """package com.example
// 
// import zio.*
// import scala.collection.Seq
// import scala.*
// 
// case class Data(
//   items: Seq[String],
// )
// """
```

### Example 6: Custom Combination

Combining multiple preferences:

```scala

val file = ScalaFile(
  packageDecl = PackageDecl("com.myproject"),
  imports = List(
    Import.WildcardImport("zio")
  ),
  types = List(
    SealedTrait(
      "Response",
      cases = List(
        SealedTraitCase.CaseClassCase(
          CaseClass("Ok", List(
            Field("data", TypeRef.String),
            Field("status", TypeRef.Int)
          ))
        ),
        SealedTraitCase.CaseObjectCase("Error")
      )
    )
  )
)

val config = EmitterConfig(
  indentWidth = 4,
  sortImports = true,
  trailingCommas = false,
  scala3Syntax = false
)
```

Emits:

```scala

ScalaEmitter.emit(file, config)
// res15: String = """package com.myproject
// 
// import zio._
// 
// sealed trait Response
// 
// object Response {
//     case class Ok(
//         data: String,
//         status: Int
//     ) extends Response
//     case object Error extends Response
// }
// """
```

## Cross-Scala Behavior

The `scala3Syntax` field affects code generation:

### Scala 3 Features

When targeting Scala 3, the emitter uses:

- **Enums**: `enum Color { case Red; case Blue }`
- **Derives (case classes only)**: `derives Show, Eq` (sealed traits and enums do not emit derives)
- **Wildcard imports**: `import scala.collection.*`
- **Rename imports**: `import foo.{bar as baz}`
- **Opaque types**: `opaque type UserId = Long`

### Scala 2 Fallback

When targeting Scala 2, the emitter uses:

- **Sealed traits** instead of enums: `sealed trait Color` + `case object Red`
- **Implicit/given** syntax converts to Scala 2 compatible form
- **Wildcard imports**: `import scala.collection._`
- **Rename imports**: `import foo.{bar => baz}`
- **Type aliases** instead of opaque types: `type UserId = Long`

---

## Complete Examples

This page shows complete, runnable examples demonstrating realistic code generation workflows. Each example builds IR models from scratch and emits formatted Scala code.

## Example 1: Simple Domain Model

Build a complete domain model with a case class and sealed trait:

```scala

// Define error types
val error = SealedTrait(
  name = "Error",
  cases = List(
    SealedTraitCase.CaseObjectCase("NotFound"),
    SealedTraitCase.CaseObjectCase("Unauthorized"),
    SealedTraitCase.CaseClassCase(
      CaseClass("ValidationError", List(
        Field("field", TypeRef.String),
        Field("message", TypeRef.String)
      ))
    )
  )
)

// Define a user
val user = CaseClass(
  name = "User",
  fields = List(
    Field("id", TypeRef.Long),
    Field("email", TypeRef.String),
    Field("name", TypeRef.String),
    Field("active", TypeRef.Boolean, defaultValue = Some("true"))
  )
)

// Assemble into a file
val file = ScalaFile(
  packageDecl = PackageDecl("com.example.domain"),
  imports = List(
    Import.WildcardImport("zio")
  ),
  types = List(error, user)
)

val config = EmitterConfig(indentWidth = 2)
```

This generates:

```scala

val code = ScalaEmitter.emit(file, config)
// code: String = """package com.example.domain
// 
// import zio.*
// 
// sealed trait Error
// 
// object Error {
//   case object NotFound extends Error
//   case object Unauthorized extends Error
//   case class ValidationError(
//     field: String,
//     message: String,
//   ) extends Error
// }
// 
// case class User(
//   id: Long,
//   email: String,
//   name: String,
//   active: Boolean = true,
// )
// """
```

## Example 2: Generic Container Types

Build polymorphic types with type parameters:

```scala

// A generic wrapper
val container = CaseClass(
  name = "Container",
  fields = List(
    Field("value", TypeRef("T")),
    Field("metadata", TypeRef("Map", List(TypeRef.String, TypeRef.String)))
  ),
  typeParams = List(TypeParam("T")),
  derives = List("Show")
)

// A generic result type
val result = SealedTrait(
  name = "Result",
  typeParams = List(TypeParam("A")),
  cases = List(
    SealedTraitCase.CaseClassCase(
      CaseClass("Success", List(
        Field("value", TypeRef("A"))
      ), typeParams = List(TypeParam("A")))
    ),
    SealedTraitCase.CaseClassCase(
      CaseClass("Failure", List(
        Field("error", TypeRef.String)
      ))
    )
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example.types"),
  types = List(container, result)
)
```

Generates:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res1: String = """package com.example.types
// 
// case class Container[T](
//   value: T,
//   metadata: Map[String, String],
// ) derives Show
// 
// sealed trait Result[A]
// 
// object Result {
//   case class Success[A](
//     value: A,
//   ) extends Result[A]
//   case class Failure(
//     error: String,
//   ) extends Result[A]
// }
// """
```

## Example 3: API Request/Response Models

Generate HTTP API models from scratch:

```scala

// Request model
val createUserReq = CaseClass(
  name = "CreateUserRequest",
  fields = List(
    Field("email", TypeRef.String),
    Field("name", TypeRef.String),
    Field("password", TypeRef.String)
  )
)

// Response model
val user = CaseClass(
  name = "User",
  fields = List(
    Field("id", TypeRef.Long),
    Field("email", TypeRef.String),
    Field("name", TypeRef.String),
    Field("createdAt", TypeRef.String)
  )
)

// Error responses
val apiError = SealedTrait(
  name = "ApiError",
  cases = List(
    SealedTraitCase.CaseClassCase(
      CaseClass("BadRequest", List(
        Field("message", TypeRef.String)
      ))
    ),
    SealedTraitCase.CaseClassCase(
      CaseClass("Conflict", List(
        Field("resource", TypeRef.String),
        Field("details", TypeRef.String)
      ))
    ),
    SealedTraitCase.CaseObjectCase("InternalServerError")
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example.api.models"),
  types = List(createUserReq, user, apiError)
)
```

Generates:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res3: String = """package com.example.api.models
// 
// case class CreateUserRequest(
//   email: String,
//   name: String,
//   password: String,
// )
// 
// case class User(
//   id: Long,
//   email: String,
//   name: String,
//   createdAt: String,
// )
// 
// sealed trait ApiError
// 
// object ApiError {
//   case class BadRequest(
//     message: String,
//   ) extends ApiError
//   case class Conflict(
//     resource: String,
//     details: String,
//   ) extends ApiError
//   case object InternalServerError extends ApiError
// }
// """
```

## Example 4: Cross-Scala Version Generation

Emit code for both Scala 3 and Scala 2:

```scala

// An enum type
val status = Enum(
  name = "OrderStatus",
  cases = List(
    EnumCase.SimpleCase("Pending"),
    EnumCase.SimpleCase("Shipped"),
    EnumCase.SimpleCase("Delivered"),
    EnumCase.SimpleCase("Cancelled")
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.shop.models"),
  types = List(status)
)

// Generate for Scala 3
val scala3Code = ScalaEmitter.emit(
  file,
  EmitterConfig(scala3Syntax = true)
)

// Generate for Scala 2
val scala2Code = ScalaEmitter.emit(
  file,
  EmitterConfig(scala3Syntax = false)
)
```

Scala 3 output:

```scala

scala3Code
// res5: String = """package com.shop.models
// 
// enum OrderStatus {
//   case Pending, Shipped, Delivered, Cancelled
// }
// """
```

Scala 2 output:

```scala

scala2Code
// res6: String = """package com.shop.models
// 
// sealed trait OrderStatus
// 
// object OrderStatus {
//   case object Pending extends OrderStatus
//   case object Shipped extends OrderStatus
//   case object Delivered extends OrderStatus
//   case object Cancelled extends OrderStatus
// }
// """
```

## Example 5: Object with Static Methods and Values

Generate a companion object with utility methods:

```scala

val config = CaseClass(
  name = "AppConfig",
  fields = List(
    Field("host", TypeRef.String),
    Field("port", TypeRef.Int),
    Field("timeout", TypeRef.Long)
  ),
  companion = Some(
    CompanionObject(
      members = List(
        ObjectMember.ValMember(
          "Default",
          TypeRef("AppConfig"),
          "AppConfig(\"localhost\", 8080, 30000L)"
        ),
        ObjectMember.ValMember(
          "Production",
          TypeRef("AppConfig"),
          "AppConfig(\"api.example.com\", 443, 60000L)"
        )
      )
    )
  ),
  derives = List("Show")
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example.config"),
  types = List(config)
)
```

Generates:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res8: String = """package com.example.config
// 
// case class AppConfig(
//   host: String,
//   port: Int,
//   timeout: Long,
// ) derives Show
// 
// object AppConfig {
//   val Default: AppConfig = AppConfig("localhost", 8080, 30000L)
//   val Production: AppConfig = AppConfig("api.example.com", 443, 60000L)
// }
// """
```

## Example 6: Complex Nested Types

Model deeply nested generic structures:

```scala

// Pagination wrapper
val page = CaseClass(
  name = "Page",
  fields = List(
    Field(
      "items",
      TypeRef("List", List(TypeRef("T")))
    ),
    Field("total", TypeRef.Long),
    Field("page", TypeRef.Int),
    Field("pageSize", TypeRef.Int),
    Field(
      "errors",
      TypeRef("List", List(
        TypeRef("Map", List(
          TypeRef.String,
          TypeRef.String
        ))
      ))
    )
  ),
  typeParams = List(TypeParam("T"))
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example.pagination"),
  imports = List(
    Import.WildcardImport("scala.collection")
  ),
  types = List(page)
)
```

Generates:

```scala

ScalaEmitter.emit(file, EmitterConfig(indentWidth = 2))
// res10: String = """package com.example.pagination
// 
// import scala.collection.*
// 
// case class Page[T](
//   items: List[T],
//   total: Long,
//   page: Int,
//   pageSize: Int,
//   errors: List[Map[String, String]],
// )
// """
```

## Example 7: Building Files Incrementally

Add types to a file step by step:

```scala

// Build types
val user = CaseClass("User", List(Field("id", TypeRef.Long)))
val order = CaseClass("Order", List(
  Field("id", TypeRef.Long),
  Field("userId", TypeRef.Long)
))
val status = Enum(
  "OrderStatus",
  cases = List(
    EnumCase.SimpleCase("Pending"),
    EnumCase.SimpleCase("Complete")
  )
)

// Assemble into a file
val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  imports = List(Import.WildcardImport("zio")),
  types = List(user, order, status)
)
```

Generates:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res12: String = """package com.example
// 
// import zio.*
// 
// case class User(
//   id: Long,
// )
// 
// case class Order(
//   id: Long,
//   userId: Long,
// )
// 
// enum OrderStatus {
//   case Pending, Complete
// }
// """
```

## Example 8: Multi-File Generation

Generate multiple files from a single data model:

```scala

val config = EmitterConfig()

// Models file
val modelsFile = ScalaFile(
  packageDecl = PackageDecl("com.example.models"),
  types = List(
    CaseClass("User", List(
      Field("id", TypeRef.Long),
      Field("name", TypeRef.String)
    )),
    CaseClass("Product", List(
      Field("id", TypeRef.Long),
      Field("price", TypeRef("BigDecimal"))
    ))
  )
)

// Errors file
val errorsFile = ScalaFile(
  packageDecl = PackageDecl("com.example.errors"),
  types = List(
    SealedTrait(
      "DomainError",
      cases = List(
        SealedTraitCase.CaseObjectCase("NotFound"),
        SealedTraitCase.CaseObjectCase("Unauthorized")
      )
    )
  )
)

// Emit both
val modelsCode = ScalaEmitter.emit(modelsFile, config)
val errorsCode = ScalaEmitter.emit(errorsFile, config)
```

Models output:

```scala

modelsCode
// res14: String = """package com.example.models
// 
// case class User(
//   id: Long,
//   name: String,
// )
// 
// case class Product(
//   id: Long,
//   price: BigDecimal,
// )
// """
```

Errors output:

```scala

errorsCode
// res15: String = """package com.example.errors
// 
// sealed trait DomainError
// 
// object DomainError {
//   case object NotFound extends DomainError
//   case object Unauthorized extends DomainError
// }
// """
```

## Key Patterns

These patterns recur across all code generation workflows:

### Building Incrementally

When generating code, you'll often build types incrementally:

```scala

val types = List(
  CaseClass("User", List()),
  CaseClass("Product", List())
)

// Create file with all types
val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = types
)
```

### Reusing Type Definitions

Define common types once and compose them:

```scala

// Common error type
val baseError = SealedTrait(
  "Error",
  cases = List(
    SealedTraitCase.CaseObjectCase("NotFound")
  )
)

// File 1
val file1 = ScalaFile(
  packageDecl = PackageDecl("com.module1"),
  types = List(baseError)
)

// File 2 (same error, different package)
val file2 = ScalaFile(
  packageDecl = PackageDecl("com.module2"),
  types = List(baseError)
)
```

### Configuration Strategies

Choose configuration once and apply to all files:

```scala

// Your project's standard config
val projectConfig = EmitterConfig(
  indentWidth = 2,
  trailingCommas = true,
  sortImports = true,
  scala3Syntax = true
)

// Apply to all generated files
// ScalaEmitter.emit(file1, projectConfig)
// ScalaEmitter.emit(file2, projectConfig)
// ScalaEmitter.emit(file3, projectConfig)
```

## Integration with Generators

In a real code generator, you'd:

1. **Parse** your source format (OpenAPI, Smithy, Protobuf, etc.)
2. **Build IR** models from parsed data
3. **Configure** the emitter once
4. **Emit** files in a loop
5. **Write** strings to disk

[Generate Scala Code from a Smithy Model](../../guides/generate-scala-code-from-smithy.md) is a complete, compiled walk-through of these five steps for Smithy, ending with the generator running as an sbt source generator. [How to Generate Scala Types from an OpenAPI Spec at Build Time](../../guides/generate-scala-types-from-openapi.md) does the same for the component schemas of an OpenAPI document, including `oneOf` with a discriminator.

This separation of concerns (format-specific parsing → generic IR → emission) is the core value of `zio-blocks-codegen`.

---

## Field

`Field` represents a class field (constructor parameter) in the IR. Combine a name with a type reference, optionally adding default values and modifiers.

## Use Cases

- Defining constructor parameters for case classes
- Specifying fields in abstract classes
- Modeling data structure fields with optional defaults

## Construction

Build a field with a name and type:

```scala

val id = Field("id", TypeRef.Long)
val name = Field("name", TypeRef.String)
val email = Field("email", TypeRef.optional(TypeRef.String))
```

With default value:

```scala

val timeout = Field("timeout", TypeRef.Long, defaultValue = Some("5000L"))
val retries = Field("retries", TypeRef.Int, defaultValue = Some("3"))
```

With annotations:

```scala

val annotated = Field(
  "id",
  TypeRef.Long,
  annotations = List(Annotation("id"))
)
```

## Key Operations

All `Field` instances support these operations:

### Accessing Components

Extract parts of a field:

```scala

val field = Field("age", TypeRef.Int, defaultValue = Some("0"))
```

View field properties:

```scala

field.name           // "age"
// res4: String = "age"
field.typeRef        // TypeRef.Int
// res5: TypeRef = TypeRef(name = "Int", typeArgs = List())
field.defaultValue   // Some("0")
// res6: Option[String] = Some("0")
field.annotations    // List[Annotation]
// res7: List[Annotation] = List()
```

### Building with Copy

Modify a field using copy:

```scala

val field = Field("age", TypeRef.Int, defaultValue = Some("0"))
```

Apply modifications:

```scala

val updated = field.copy(
  defaultValue = Some("18"),
  annotations = List(Annotation("min"))
)
// updated: Field = Field(
//   name = "age",
//   typeRef = TypeRef(name = "Int", typeArgs = List()),
//   defaultValue = Some("18"),
//   annotations = List(Annotation(name = "min", args = List())),
//   doc = None
// )
```

## Examples

These examples show practical usage patterns for `Field`:

### Example 1: Simple Fields

Create fields for a case class:

```scala

val order = CaseClass(
  name = "Order",
  fields = List(
    Field("id", TypeRef.Long),
    Field("customerId", TypeRef.String),
    Field("total", TypeRef("BigDecimal")),
    Field("status", TypeRef.String)
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.shop"),
  types = List(order)
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res10: String = """package com.shop
// 
// case class Order(
//   id: Long,
//   customerId: String,
//   total: BigDecimal,
//   status: String,
// )
// """
```

### Example 2: Fields with Defaults

Case class fields with default values:

```scala

val config = CaseClass(
  name = "DatabaseConfig",
  fields = List(
    Field("host", TypeRef.String, defaultValue = Some("\"localhost\"")),
    Field("port", TypeRef.Int, defaultValue = Some("5432")),
    Field("timeout", TypeRef.Long, defaultValue = Some("30000L"))
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(config)
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res12: String = """package com.example
// 
// case class DatabaseConfig(
//   host: String = "localhost",
//   port: Int = 5432,
//   timeout: Long = 30000L,
// )
// """
```

### Example 3: Optional and Collection Fields

Fields with generic types:

```scala

val article = CaseClass(
  name = "Article",
  fields = List(
    Field("title", TypeRef.String),
    Field("content", TypeRef.String),
    Field("author", TypeRef.optional(TypeRef.String)),
    Field("tags", TypeRef.list(TypeRef.String)),
    Field("metadata", TypeRef.map(TypeRef.String, TypeRef.String))
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(article)
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res14: String = """package com.example
// 
// case class Article(
//   title: String,
//   content: String,
//   author: Option[String],
//   tags: List[String],
//   metadata: Map[String, String],
// )
// """
```

### Example 4: Nested Generic Types

Complex field types:

```scala

val response = CaseClass(
  name = "Response",
  fields = List(
    Field(
      "data",
      TypeRef("Option", List(
        TypeRef("List", List(TypeRef.String))
      ))
    ),
    Field(
      "errors",
      TypeRef("List", List(
        TypeRef("Map", List(TypeRef.String, TypeRef.String))
      ))
    )
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(response)
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res16: String = """package com.example
// 
// case class Response(
//   data: Option[List[String]],
//   errors: List[Map[String, String]],
// )
// """
```

### Example 5: Fields with Type Parameters

Fields using generic type variables:

```scala

val page = CaseClass(
  name = "Page",
  fields = List(
    Field("items", TypeRef("List", List(TypeRef("T")))),
    Field("total", TypeRef.Long),
    Field("pageSize", TypeRef.Int)
  ),
  typeParams = List(TypeParam("T"))
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(page)
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res18: String = """package com.example
// 
// case class Page[T](
//   items: List[T],
//   total: Long,
//   pageSize: Int,
// )
// """
```

---

## Code Generation

`zio-blocks-codegen` is a **generic, domain-agnostic Scala code generation library**. It provides an intermediate representation (IR) for building type-safe models of Scala code structures, and a pure emitter that generates well-formatted Scala source files from those models.

Core types: `ScalaFile`, `TypeDefinition`, `CaseClass`, `SealedTrait`, `Enum`, `Field`, `TypeRef`, `Method`, `Annotation`.

Here's the structure of two core types:

```scala
// IR models the structure of Scala code
case class ScalaFile(
  packageDecl: PackageDecl,
  imports: List[Import] = Nil,
  types: List[TypeDefinition] = Nil
)

case class CaseClass(
  name: String,
  fields: List[Field],
  typeParams: List[TypeParam] = Nil,
  derives: List[String] = Nil,
  // ...
)
```

## Introduction

This module is designed to be a **reusable building block** for any tool that needs to generate Scala code—whether from OpenAPI specifications, Smithy models, Protocol Buffers, JSON Schema, or any other source format.

Rather than embedding code generation logic into domain-specific tools, you model your source data in the codegen IR, then emit clean, formatted Scala. This separation enables single source of truth, consistency, and reuse across all generators without cross-coupling.

## Motivation

Codegen IR exists to solve a specific problem: **many generators produce the same output (Scala code) but reinvent the emission logic.** Before `zio-blocks-codegen`, each generator (OpenAPI, Smithy, Protobuf, etc.) had its own IR and emitter, leading to duplication, bugs, and inconsistency. By extracting IR and emitter into `zio-blocks-codegen`, all generators share one implementation.

Before `zio-blocks-codegen`, each generator had its own IR and emitter:

```
OpenAPI → Scala        Smithy → Scala         Protobuf → Scala
    ↓                      ↓                        ↓
[Custom IR]            [Custom IR]             [Custom IR]
    ↓                      ↓                        ↓
[Custom Emitter]       [Custom Emitter]        [Custom Emitter]
    ↓                      ↓                        ↓
Scala Code             Scala Code              Scala Code

❌ Duplication: Same problem solved 3 times with different code
❌ Bugs: Fixes in one place don't help others
❌ Inconsistency: Different styles, formatting, edge cases
```

Now with `zio-blocks-codegen`, all generators converge on a single implementation:

```
OpenAPI → Scala    Smithy → Scala    Protobuf → Scala    JSON Schema → Scala
    ↓                  ↓                   ↓                    ↓
    └──────────────────┬───────────────────┘────────────────────┘
                       ↓
              [zio-blocks-codegen]
                       ↓
            ┌──────────┴──────────┐
            ↓                     ↓
      [IR: Type-safe]     [Emitter: Pure function]
      representation      (no side effects)
      of Scala code               |
            ↓                     ↓
            └──────────┬──────────┘
                       ↓
                  Scala Code

✅ Single source of truth: One well-tested implementation
✅ Consistency: All generators use the same emitter
✅ Reusability: Zero coupling between domain-specific tools
```

## Installation

Add the library to your project:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-codegen" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x

## Overview

The module has two key layers:
1. **IR Layer**: Immutable, strongly-typed models of Scala code structures (files, types, members)
2. **Emit Layer**: Pure functions that convert IR models to formatted Scala source code

### IR Layer (`zio.blocks.codegen.ir`)

The intermediate representation captures the essential structure of Scala code:

- **File structure**: `ScalaFile`, `PackageDecl`, `Import`
- **Type definitions**: `CaseClass`, `SealedTrait`, `Trait`, `AbstractClass`, `Enum`, `ObjectDef`, `OpaqueType`, `Newtype`, `TypeAlias`
- **Members**: `Field`, `Method`, `MethodParam`, `Annotation`, `TypeParam`, `TypeRef`
- **Composition**: Sealed traits like `TypeDefinition`, `SealedTraitCase`, `EnumCase`, `ObjectMember` for modular type safety

Each type is **immutable and strongly typed**. You build the IR by composing these types, then hand off to the emitter.

### Emit Layer (`zio.blocks.codegen.emit`)

The emitter converts IR to Scala source code:

- **`ScalaEmitter`**: Methods to emit any IR node as a formatted string (imports, type definitions, methods, etc.)
- **`EmitterConfig`**: Configurable formatting (indent width, import sorting, trailing commas, Scala 2 vs. Scala 3 syntax)
- **No side effects**: Returns strings; your code writes files (or does anything else)

## How They Work Together

Here's the typical workflow for a code generator:

```
1. Parse source format (OpenAPI, Smithy, etc.)
   ↓
2. Build IR models (ScalaFile, CaseClass, SealedTrait, etc.)
   ↓
3. Call ScalaEmitter.emit(file, config)
   ↓
4. ScalaEmitter returns formatted Scala source as String
   ↓
5. Write string to file (or further process it)
```

Here's an example of building a Scala file with a case class:

```
ScalaFile
├─ packageDecl: PackageDecl("com.example")
├─ imports: [Import.WildcardImport("zio")]
└─ types: [
     CaseClass(
       name = "User",
       fields = [
         Field("id", TypeRef.Long),
         Field("name", TypeRef.String),
         Field("email", TypeRef("Option", List(TypeRef.String)))
       ]
     )
   ]
```

When you call `ScalaEmitter.emit(file, config)`, it walks this tree and produces:

```scala
package com.example

case class User(
  id: Long,
  name: String,
  email: Option[String],
)
```

The architecture flows through three layers:

```
┌──────────────────────────────────────────────────────────┐
│ Your Generator (OpenAPI, Smithy, Protobuf, JSON Schema)  │
│ - Parse input format                                     │
│ - Build IR models (CaseClass, SealedTrait, etc.)         │
└────────────────┬─────────────────────────────────────────┘
                 │ ScalaFile (root IR node)
                 ▼
┌──────────────────────────────────────────────────────────┐
│ zio-blocks-codegen IR Layer                              │
│ - TypeDefinition (sealed trait)                          │
│ - CaseClass, SealedTrait, Enum, ObjectDef, ...           │
│ - Field, Method, TypeRef, Annotation, TypeParam          │
│ - Strongly typed, immutable, composable                  │
└────────────────┬─────────────────────────────────────────┘
                 │ IR models
                 ▼
┌─────────────────────────────────────────────────────────┐
│ zio-blocks-codegen Emit Layer                           │
│ - ScalaEmitter.emit(file, config) → String              │
│ - Supports Scala 3 (enums, derives, * imports)          │
│ - Supports Scala 2 (sealed traits, _ imports, fallback) │
│ - EmitterConfig: indent, imports, commas, version       │
└────────────────┬────────────────────────────────────────┘
                 │ Formatted Scala source
                 ▼
         (Write to file or further process)
```

## Common Patterns

Here are the most common usage patterns:

### Pattern 1: Building a Simple Case Class

Build a case class with fields and derive clauses:

```scala

val user = CaseClass(
  name = "User",
  fields = List(
    Field("id", TypeRef.Long),
    Field("name", TypeRef.String)
  ),
  derives = List("Schema")
)
```

### Pattern 2: Sealed Trait Hierarchies

Model ADTs (algebraic data types) as sealed traits with cases:

```scala

val payment = SealedTrait(
  name = "Payment",
  cases = List(
    SealedTraitCase.CaseClassCase(
      CaseClass(
        "CreditCard",
        List(
          Field("number", TypeRef.String),
          Field("cvv", TypeRef.String)
        )
      )
    ),
    SealedTraitCase.CaseObjectCase("Cash"),
    SealedTraitCase.CaseObjectCase("Check")
  )
)
```

### Pattern 3: Generic Types with Type Parameters

Define polymorphic types:

```scala

val container = CaseClass(
  name = "Container",
  fields = List(
    Field("value", TypeRef("A"))
  ),
  typeParams = List(TypeParam("A"))
)
```

### Pattern 4: Complete File Structure

Assemble a complete Scala file ready for emission:

```scala

val user = CaseClass(
  name = "User",
  fields = List(
    Field("id", TypeRef.Long),
    Field("name", TypeRef.String)
  ),
  derives = List("Schema")
)

val payment = SealedTrait(
  name = "Payment",
  cases = List(
    SealedTraitCase.CaseObjectCase("Cash"),
    SealedTraitCase.CaseObjectCase("Card")
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  imports = List(
    Import.WildcardImport("zio"),
    Import.SingleImport("zio.blocks.schema", "Schema")
  ),
  types = List(user, payment)
)
```

## Cross-Scala Compatibility

The emitter handles both **Scala 3** and **Scala 2** natively:

- **Scala 3 features**: Enums, derives clauses, `*` imports, `as` renames, opaque types
- **Scala 2 fallback**: Sealed traits, `_` imports, `=>` renames, type aliases

You configure the output via `EmitterConfig`:

```scala

val config = EmitterConfig(
  scala3Syntax = true,  // true for Scala 3, false for Scala 2
  indentWidth = 2,
  sortImports = true,
  trailingCommas = true
)
```

## Design Philosophy

Three principles guide codegen IR:

1. **Generic**: No domain-specific logic (OpenAPI-specific stuff stays in `zio-blocks-openapi`)
2. **Pure**: No side effects—just IR models and string emission
3. **Self-contained**: Zero external dependencies, works everywhere

This makes it a safe, reliable foundation for any Scala code generator.

For a complete generator built on this IR, see [Generate Scala Code from a Smithy Model](../../guides/generate-scala-code-from-smithy.md).

---

## ScalaEmitter

`ScalaEmitter` is the core emission engine that converts IR models to formatted Scala source code. It provides the main entry point and methods to emit any IR construct as a properly formatted string.

## Use Cases

- Converting a complete `ScalaFile` IR to a Scala source string
- Emitting individual type definitions as Scala code
- Generating imports, annotations, methods, and type references
- Formatting code with consistent indentation and style

## Main Emission Method

Emit a complete Scala file:

```scala

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(
    CaseClass("User", List(Field("id", TypeRef.Long)))
  )
)

val config = EmitterConfig()
val sourceCode = ScalaEmitter.emit(file, config)
// sourceCode is a String ready to write to a .scala file
```

## Key Operations

All core operations are shown below:

### Emitting Type References

Convert a type reference to Scala syntax:

```scala

// These methods are used internally by emit, but available if you need them:
ScalaEmitter.emitTypeRef(TypeRef.String)
// Returns: "String"

ScalaEmitter.emitTypeRef(TypeRef.list(TypeRef.Int))
// Returns: "List[Int]"

ScalaEmitter.emitTypeRef(TypeRef.map(TypeRef.String, TypeRef.Int))
// Returns: "Map[String, Int]"
```

### Emitting Type Parameters

Emit generic type parameters with bounds:

```scala

val unbounded = TypeParam("T")
// Emits: "T"

val bounded = TypeParam("T", upperBound = Some(TypeRef("Serializable")))
// Emits: "T <: Serializable"

val covariant = TypeParam("T", variance = Variance.Covariant)
// Emits: "+T"
```

### Emitting Annotations

Convert annotations to Scala syntax:

```scala

val deprecated = Annotation("deprecated")
// Emits: "@deprecated"

val withArg = Annotation("Deprecated", args = List("message" -> "\"Use newMethod instead\""))
// Emits: "@Deprecated(message = \"Use newMethod instead\")"
```

## Configuration

The emitter respects `EmitterConfig` settings:

```scala

val file = ScalaFile(PackageDecl("com.example"))

val config = EmitterConfig(
  indentWidth = 2,              // Spaces per indent level
  sortImports = true,           // Sort import statements
  trailingCommas = true,        // Add trailing commas in collections
  scala3Syntax = true           // Scala 3 syntax features
)

ScalaEmitter.emit(file, config)
```

See [EmitterConfig](./emitter-config.md) for all available options.

## Examples

Practical examples demonstrate common usage:

### Example 1: Emit a Complete File

Generate a Scala file with multiple types:

```scala

val file = ScalaFile(
  packageDecl = PackageDecl("com.api"),
  imports = List(
    Import.WildcardImport("zio"),
    Import.SingleImport("scala.collection", "Seq")
  ),
  types = List(
    CaseClass(
      "ApiResponse",
      List(
        Field("status", TypeRef.Int),
        Field("data", TypeRef.optional(TypeRef.String))
      )
    )
  )
)

val config = EmitterConfig(indentWidth = 2)
```

Emits:

```scala

ScalaEmitter.emit(file, config)
// res6: String = """package com.api
// 
// import scala.collection.Seq
// import zio.*
// 
// case class ApiResponse(
//   status: Int,
//   data: Option[String],
// )
// """
```

### Example 2: Cross-Scala Compatibility

Generate code for different Scala versions:

```scala

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(
    Enum(
      name = "Status",
      cases = List(
        EnumCase.SimpleCase("Active"),
        EnumCase.SimpleCase("Inactive")
      )
    )
  )
)

// For Scala 3: emits actual enum
val scala3Config = EmitterConfig(
  scala3Syntax = true
)
val scala3Code = ScalaEmitter.emit(file, scala3Config)

// For Scala 2: emits sealed trait (fallback)
val scala2Config = EmitterConfig(
  scala3Syntax = false
)
val scala2Code = ScalaEmitter.emit(file, scala2Config)
```

Scala 3 emits:

```scala

scala3Code
// res8: String = """package com.example
// 
// enum Status {
//   case Active, Inactive
// }
// """
```

Scala 2 emits:

```scala

scala2Code
// res9: String = """package com.example
// 
// sealed trait Status
// 
// object Status {
//   case object Active extends Status
//   case object Inactive extends Status
// }
// """
```

### Example 3: Sealed Trait with Multiple Cases

Emit a complete ADT:

```scala

val file = ScalaFile(
  packageDecl = PackageDecl("com.errors"),
  types = List(
    SealedTrait(
      name = "DomainError",
      cases = List(
        SealedTraitCase.CaseClassCase(
          CaseClass("ValidationError", List(
            Field("field", TypeRef.String),
            Field("message", TypeRef.String)
          ))
        ),
        SealedTraitCase.CaseClassCase(
          CaseClass("NotFound", List(
            Field("resource", TypeRef.String),
            Field("id", TypeRef.Long)
          ))
        ),
        SealedTraitCase.CaseObjectCase("Unauthorized"),
        SealedTraitCase.CaseObjectCase("InternalError")
      )
    )
  )
)

val config = EmitterConfig(trailingCommas = true)
```

Emits:

```scala

ScalaEmitter.emit(file, config)
// res11: String = """package com.errors
// 
// sealed trait DomainError
// 
// object DomainError {
//   case class ValidationError(
//     field: String,
//     message: String,
//   ) extends DomainError
//   case class NotFound(
//     resource: String,
//     id: Long,
//   ) extends DomainError
//   case object Unauthorized extends DomainError
//   case object InternalError extends DomainError
// }
// """
```

### Example 4: Generic Types

Emit polymorphic types with proper type parameter syntax:

```scala

val file = ScalaFile(
  packageDecl = PackageDecl("com.containers"),
  types = List(
    CaseClass(
      name = "Wrapper",
      fields = List(Field("value", TypeRef("A"))),
      typeParams = List(
        TypeParam("A")
      )
    ),
    CaseClass(
      name = "Pair",
      fields = List(
        Field("left", TypeRef("A")),
        Field("right", TypeRef("B"))
      ),
      typeParams = List(
        TypeParam("A"),
        TypeParam("B")
      )
    )
  )
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res13: String = """package com.containers
// 
// case class Wrapper[A](
//   value: A,
// )
// 
// case class Pair[A, B](
//   left: A,
//   right: B,
// )
// """
```

### Example 5: Formatting Customization

Control code style with configuration:

```scala

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  imports = List(
    Import.WildcardImport("scala.collection"),
    Import.SingleImport("java.time", "Instant")
  ),
  types = List(
    CaseClass(
      name = "Record",
      fields = List(
        Field("a", TypeRef.String),
        Field("b", TypeRef.Int),
        Field("c", TypeRef.Boolean)
      )
    )
  )
)

// Wide indentation, sorted imports, trailing commas
val config = EmitterConfig(
  indentWidth = 4,
  sortImports = true,
  trailingCommas = true
)
```

Emits:

```scala

ScalaEmitter.emit(file, config)
// res15: String = """package com.example
// 
// import java.time.Instant
// import scala.collection.*
// 
// case class Record(
//     a: String,
//     b: Int,
//     c: Boolean,
// )
// """
```

## Design Philosophy

`ScalaEmitter` follows three principles:

1. **Pure**: No side effects. Takes IR + config, returns a string. Your code writes files.
2. **Configurable**: `EmitterConfig` controls formatting. Add options as needed for your generator.
3. **Cross-Scala**: Targets both Scala 3 (enums, derives, `*` imports) and Scala 2 (sealed traits, `_` imports) from the same IR.

---

## ScalaFile

`ScalaFile` is the root IR node representing a complete Scala source file. It holds everything needed to emit a compilable file: the package declaration, imports, and type definitions.

## Use Cases

- As the entry point to `ScalaEmitter.emit(file, config)` to generate source code
- As a container when building complex file structures with multiple types
- As the canonical representation of "what I want to emit as Scala"

## Construction

Build a `ScalaFile` with a package declaration, optional imports, and optional types:

```scala

val simpleFile = ScalaFile(
  packageDecl = PackageDecl("com.example")
)
```

With imports and types:

```scala

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  imports = List(
    Import.WildcardImport("zio")
  ),
  types = List(
    CaseClass(
      name = "User",
      fields = List(Field("id", TypeRef.Long))
    )
  )
)
```

## Key Operations

All core operations are shown below:

### Accessing Components

Extract the parts of a `ScalaFile`:

```scala

// Read-only access to all components
file.packageDecl    // PackageDecl
// res1: PackageDecl = PackageDecl("com.example")
file.imports        // List[Import]
// res2: List[Import] = List(WildcardImport("zio"))
file.types          // List[TypeDefinition]
// res3: List[TypeDefinition] = List(
//   CaseClass(
//     name = "User",
//     fields = List(
//       Field(
//         name = "id",
//         typeRef = TypeRef(name = "Long", typeArgs = List()),
//         defaultValue = None,
//         annotations = List(),
//         doc = None
//       )
//     ),
//     typeParams = List(),
//     extendsTypes = List(),
//     derives = List(),
//     annotations = List(),
//     companion = None,
//     doc = None,
//     isValueClass = false
//   )
// )
```

### Building with Copy

Modify a file by copying with new values:

```scala

val updatedFile = file.copy(
  types = file.types :+ CaseClass(
    name = "Product",
    fields = List(Field("name", TypeRef.String))
  )
)
// updatedFile: ScalaFile = ScalaFile(
//   packageDecl = PackageDecl("com.example"),
//   imports = List(WildcardImport("zio")),
//   types = List(
//     CaseClass(
//       name = "User",
//       fields = List(
//         Field(
//           name = "id",
//           typeRef = TypeRef(name = "Long", typeArgs = List()),
//           defaultValue = None,
//           annotations = List(),
//           doc = None
//         )
//       ),
//       typeParams = List(),
//       extendsTypes = List(),
//       derives = List(),
//       annotations = List(),
//       companion = None,
//       doc = None,
//       isValueClass = false
//     ),
//     CaseClass(
//       name = "Product",
//       fields = List(
//         Field(
//           name = "name",
//           typeRef = TypeRef(name = "String", typeArgs = List()),
//           defaultValue = None,
//           annotations = List(),
//           doc = None
//         )
//       ),
//       typeParams = List(),
//       extendsTypes = List(),
//       derives = List(),
//       annotations = List(),
//       companion = None,
//       doc = None,
//       isValueClass = false
//     )
//   )
// )
```

### Emitting to Source

Generate Scala source from the file:

```scala

val config = EmitterConfig()
val sourceCode = ScalaEmitter.emit(file, config)
// sourceCode is a String ready to write to a .scala file
```

## Examples

Practical examples demonstrate common usage. Each builds a `ScalaFile`, calls `ScalaEmitter.emit()`, and displays the resulting output:

### Example 1: Minimal File

A file with just a package and one case class (all examples use `zio.blocks.codegen.ir._` and `zio.blocks.codegen.emit._` imports):

```scala

val minimal = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(
    CaseClass(
      name = "Empty",
      fields = Nil
    )
  )
)
```

Emits:

```scala

ScalaEmitter.emit(minimal, EmitterConfig())
// res6: String = """package com.example
// 
// case class Empty()
// """
```

### Example 2: File with Multiple Types

A file with a sealed trait and case classes:

```scala

val multiType = ScalaFile(
  packageDecl = PackageDecl("com.payment"),
  imports = List(Import.WildcardImport("zio")),
  types = List(
    SealedTrait(
      name = "PaymentMethod",
      cases = List(
        SealedTraitCase.CaseClassCase(
          CaseClass(
            "Card",
            List(Field("cardNumber", TypeRef.String))
          )
        ),
        SealedTraitCase.CaseObjectCase("Cash")
      )
    )
  )
)
```

Emits:

```scala

ScalaEmitter.emit(multiType, EmitterConfig())
// res8: String = """package com.payment
// 
// import zio.*
// 
// sealed trait PaymentMethod
// 
// object PaymentMethod {
//   case class Card(
//     cardNumber: String,
//   ) extends PaymentMethod
//   case object Cash extends PaymentMethod
// }
// """
```

### Example 3: File with Imports and Type Parameters

A file demonstrating generic types and selective imports:

```scala

val generic = ScalaFile(
  packageDecl = PackageDecl("com.containers"),
  imports = List(
    Import.SingleImport("scala.collection", "Seq"),
    Import.WildcardImport("zio")
  ),
  types = List(
    CaseClass(
      name = "Wrapper",
      fields = List(
        Field("items", TypeRef("Seq", List(TypeRef("T"))))
      ),
      typeParams = List(TypeParam("T"))
    )
  )
)
```

Emits:

```scala

ScalaEmitter.emit(generic, EmitterConfig())
// res10: String = """package com.containers
// 
// import scala.collection.Seq
// import zio.*
// 
// case class Wrapper[T](
//   items: Seq[T],
// )
// """
```

---

## SealedTrait

`SealedTrait` represents a sealed trait in the IR—a sum type (algebraic data type) that enumerates all possible cases. It's essential for modeling discriminated unions and exhaustive pattern matching.

## Use Cases

- Modeling sum types (ADTs) from API responses or domain models
- Creating hierarchies of case classes and case objects
- Defining error types with multiple failure cases
- Implementing discriminated unions with exhaustive matching

## Construction

Build a sealed trait with cases:

```scala

val httpStatus = SealedTrait(
  name = "HttpStatus",
  cases = List(
    SealedTraitCase.CaseObjectCase("Ok"),
    SealedTraitCase.CaseObjectCase("NotFound"),
    SealedTraitCase.CaseObjectCase("ServerError")
  )
)
```

With case classes as cases:

```scala

val result = SealedTrait(
  name = "Result",
  cases = List(
    SealedTraitCase.CaseClassCase(
      CaseClass("Success", List(Field("value", TypeRef.String)))
    ),
    SealedTraitCase.CaseClassCase(
      CaseClass("Failure", List(Field("error", TypeRef.String)))
    )
  )
)
```

With type parameters:

```scala

val option = SealedTrait(
  name = "Option",
  typeParams = List(TypeParam("A")),
  cases = List(
    SealedTraitCase.CaseObjectCase("None"),
    SealedTraitCase.CaseClassCase(
      CaseClass("Some", List(Field("value", TypeRef("A"))), typeParams = List(TypeParam("A")))
    )
  )
)
```

## Key Operations

All core operations are shown below:

### Accessing Components

Extract parts of a sealed trait:

```scala

result.name          // "Result"
// res2: String = "Result"
result.cases         // List[SealedTraitCase]
// res3: List[SealedTraitCase] = List(
//   CaseClassCase(
//     CaseClass(
//       name = "Success",
//       fields = List(
//         Field(
//           name = "value",
//           typeRef = TypeRef(name = "String", typeArgs = List()),
//           defaultValue = None,
//           annotations = List(),
//           doc = None
//         )
//       ),
//       typeParams = List(),
//       extendsTypes = List(),
//       derives = List(),
//       annotations = List(),
//       companion = None,
//       doc = None,
//       isValueClass = false
//     )
//   ),
//   CaseClassCase(
//     CaseClass(
//       name = "Failure",
//       fields = List(
//         Field(
//           name = "error",
//           typeRef = TypeRef(name = "String", typeArgs = List()),
//           defaultValue = None,
//           annotations = List(),
//           doc = None
//         )
//       ),
//       typeParams = List(),
//       extendsTypes = List(),
//       derives = List(),
//       annotations = List(),
//       companion = None,
//       doc = None,
//       isValueClass = false
//     )
//   )
// )
result.typeParams    // List[TypeParam] (empty if not generic)
// res4: List[TypeParam] = List()
result.derives       // List[String]
// res5: List[String] = List()
result.annotations   // List[Annotation]
// res6: List[Annotation] = List()
```

### Working with Cases

Each case is either a `CaseObjectCase` or `CaseClassCase`:

```scala

val caseObj = SealedTraitCase.CaseObjectCase("Unknown")

val caseClass = SealedTraitCase.CaseClassCase(
  CaseClass("Error", List(Field("msg", TypeRef.String)))
)
```

### Building with Copy

Modify a sealed trait:

```scala

val updated = result.copy(
  cases = result.cases :+ SealedTraitCase.CaseObjectCase("Pending")
)
// updated: SealedTrait = SealedTrait(
//   name = "Result",
//   typeParams = List(),
//   extendsTypes = List(),
//   cases = List(
//     CaseClassCase(
//       CaseClass(
//         name = "Success",
//         fields = List(
//           Field(
//             name = "value",
//             typeRef = TypeRef(name = "String", typeArgs = List()),
//             defaultValue = None,
//             annotations = List(),
//             doc = None
//           )
//         ),
//         typeParams = List(),
//         extendsTypes = List(),
//         derives = List(),
//         annotations = List(),
//         companion = None,
//         doc = None,
//         isValueClass = false
//       )
//     ),
//     CaseClassCase(
//       CaseClass(
//         name = "Failure",
//         fields = List(
//           Field(
//             name = "error",
//             typeRef = TypeRef(name = "String", typeArgs = List()),
//             defaultValue = None,
//             annotations = List(),
//             doc = None
//           )
//         ),
//         typeParams = List(),
//         extendsTypes = List(),
//         derives = List(),
//         annotations = List(),
//         companion = None,
//         doc = None,
//         isValueClass = false
//       )
//     ),
//     CaseObjectCase("Pending")
//   ),
// ...
```

## Examples

Practical examples demonstrate common usage:

### Example 1: Simple Enum-Like Sealed Trait

A sealed trait with only case objects:

```scala

val color = SealedTrait(
  name = "Color",
  cases = List(
    SealedTraitCase.CaseObjectCase("Red"),
    SealedTraitCase.CaseObjectCase("Green"),
    SealedTraitCase.CaseObjectCase("Blue")
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.graphics"),
  types = List(color)
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res9: String = """package com.graphics
// 
// sealed trait Color
// 
// object Color {
//   case object Red extends Color
//   case object Green extends Color
//   case object Blue extends Color
// }
// """
```

### Example 2: Mixed Case Objects and Case Classes

A sealed trait with both simple and complex cases:

```scala

val payment = SealedTrait(
  name = "Payment",
  cases = List(
    SealedTraitCase.CaseClassCase(
      CaseClass("CreditCard", List(
        Field("number", TypeRef.String),
        Field("expiry", TypeRef.String)
      ))
    ),
    SealedTraitCase.CaseClassCase(
      CaseClass("BankTransfer", List(
        Field("accountNumber", TypeRef.String)
      ))
    ),
    SealedTraitCase.CaseObjectCase("Cash"),
    SealedTraitCase.CaseObjectCase("Check")
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(payment)
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res11: String = """package com.example
// 
// sealed trait Payment
// 
// object Payment {
//   case class CreditCard(
//     number: String,
//     expiry: String,
//   ) extends Payment
//   case class BankTransfer(
//     accountNumber: String,
//   ) extends Payment
//   case object Cash extends Payment
//   case object Check extends Payment
// }
// """
```

### Example 3: Generic Sealed Trait

A polymorphic sealed trait with type parameters:

```scala

val either = SealedTrait(
  name = "Either",
  typeParams = List(TypeParam("L"), TypeParam("R")),
  cases = List(
    SealedTraitCase.CaseClassCase(
      CaseClass("Left", List(Field("value", TypeRef("L"))), typeParams = List(TypeParam("L")))
    ),
    SealedTraitCase.CaseClassCase(
      CaseClass("Right", List(Field("value", TypeRef("R"))), typeParams = List(TypeParam("R")))
    )
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(either)
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res13: String = """package com.example
// 
// sealed trait Either[L, R]
// 
// object Either {
//   case class Left[L](
//     value: L,
//   ) extends Either[L, R]
//   case class Right[R](
//     value: R,
//   ) extends Either[L, R]
// }
// """
```

### Example 4: Error ADT

A sealed trait for error handling:

```scala

val appError = SealedTrait(
  name = "AppError",
  cases = List(
    SealedTraitCase.CaseClassCase(
      CaseClass("ValidationError", List(
        Field("field", TypeRef.String),
        Field("reason", TypeRef.String)
      ))
    ),
    SealedTraitCase.CaseClassCase(
      CaseClass("NotFound", List(
        Field("id", TypeRef.Long)
      ))
    ),
    SealedTraitCase.CaseObjectCase("Unauthorized"),
    SealedTraitCase.CaseObjectCase("InternalServerError")
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(appError)
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res15: String = """package com.example
// 
// sealed trait AppError
// 
// object AppError {
//   case class ValidationError(
//     field: String,
//     reason: String,
//   ) extends AppError
//   case class NotFound(
//     id: Long,
//   ) extends AppError
//   case object Unauthorized extends AppError
//   case object InternalServerError extends AppError
// }
// """
```

---

## TypeDefinition

`TypeDefinition` is a sealed trait that represents any Scala type definition—case classes, sealed traits, enums, objects, newtypes, type aliases, abstract classes, and regular traits. It's the foundation for modeling Scala's type system in the IR.

## Use Cases

- As the common interface for all compound types when building a `ScalaFile`
- When you need to handle multiple type kinds uniformly (e.g., iterate over all types in a file)
- As a base for specific type definitions like `CaseClass`, `SealedTrait`, `Enum`, etc.

## Variants

`TypeDefinition` has 9 concrete implementations:

| Type            | Purpose                     | Scala Feature                              |
|-----------------|-----------------------------|--------------------------------------------|
| `CaseClass`     | Product types with fields   | Scala 2 & 3                                |
| `SealedTrait`   | Sum types (ADTs) with cases | Scala 2 & 3                                |
| `Trait`         | Open trait definitions      | Scala 2 & 3                                |
| `AbstractClass` | Abstract class definitions  | Scala 2 & 3                                |
| `Enum`          | Scala 3 enum definitions    | Scala 3 only                               |
| `ObjectDef`     | Singleton objects           | Scala 2 & 3                                |
| `OpaqueType`    | Opaque type aliases         | Scala 3 (Scala 2 fallback to `type` alias) |
| `Newtype`       | Zero-cost wrappers          | Scala 2 & 3                                |
| `TypeAlias`     | Type aliases (with optional type parameters) | Scala 2 & 3                |

## Common Interface

Every `TypeDefinition` provides:

```scala
sealed trait TypeDefinition {
  def name: String              // The name of the type
  def annotations: List[Annotation]  // Annotations applied to it
  def doc: Option[String]       // Optional documentation
}
```

## Choosing the Right Variant

Choose the right `TypeDefinition` for your use case:

### Data Structures

**Case Class** — Immutable record with named fields and automatic `equals`, `hashCode`, `copy`:

```scala

val user = CaseClass(
  name = "User",
  fields = List(
    Field("id", TypeRef.Long),
    Field("name", TypeRef.String)
  )
)
```

**Newtype** — Zero-cost wrapper for a single underlying type (e.g., `UserId` wrapping `Long`):

```scala

val userId = Newtype("UserId", TypeRef.Long)
```

**Type Alias** — Synonym for an existing type:

```scala

val stringId = TypeAlias("StringId", typeRef = TypeRef.String)
```

**Type Alias with Type Parameters** — Generic type alias:

```scala

val pair = TypeAlias(
  "Pair",
  typeRef = TypeRef("Tuple2", List(TypeRef("A"), TypeRef("B"))),
  typeParams = List(TypeParam("A"), TypeParam("B"))
)
```

### Sum Types (ADTs)

**Sealed Trait** — Discriminated union type with cases:

```scala

val payment = SealedTrait(
  name = "Payment",
  cases = List(
    SealedTraitCase.CaseClassCase(
      CaseClass("Card", List(Field("number", TypeRef.String)))
    ),
    SealedTraitCase.CaseObjectCase("Cash")
  )
)
```

**Enum** (Scala 3 only) — Enumeration with simple or parameterized cases:

```scala

val color = Enum(
  name = "Color",
  cases = List(
    EnumCase.SimpleCase("Red"),
    EnumCase.SimpleCase("Green"),
    EnumCase.ParameterizedCase("Custom", List(Field("rgb", TypeRef.Int)))
  )
)
```

### Abstractions

**Trait** — Open trait for behavior and contracts:

```scala

val comparable = Trait(
  name = "Comparable",
  members = List(
    ObjectMember.DefMember(
      Method("compare", params = Nil, returnType = TypeRef.Int)
    )
  )
)
```

**Abstract Class** — Mix of fields and methods, but not instantiable:

```scala

val entity = AbstractClass(
  name = "Entity",
  fields = List(Field("id", TypeRef.Long))
)
```

**Object** — Singleton object with static-like members:

```scala

val utils = ObjectDef(
  name = "Utils",
  members = List(
    ObjectMember.ValMember("Version", TypeRef.String, "\"1.0.0\"")
  )
)
```

### Type Aliases (Advanced)

**Opaque Type** (Scala 3) — Type-safe wrapper with an underlying type (distinct from `Newtype`):

```scala

val userId2 = OpaqueType("UserId", underlyingType = TypeRef.Long)
```

## Common Operations

All `TypeDefinition` variants support these core operations:

### Accessing Components

```scala

val cc = CaseClass("User", List(Field("id", TypeRef.Long)))

cc.name            // "User"
cc.annotations     // List[Annotation] (empty by default)
cc.doc             // Option[String] (documentation)
```

### With Annotations

Add annotations (like `@deprecated` or custom annotations):

```scala

val annotated = CaseClass(
  name = "OldAPI",
  fields = Nil,
  annotations = List(
    Annotation("deprecated")
  )
)
```

### With Documentation

Include scaladoc or javadoc comments:

```scala

val documented = CaseClass(
  name = "User",
  fields = List(Field("id", TypeRef.Long)),
  doc = Some("Represents a user in the system")
)
```

### With Derives

Add derives clauses for automatic typeclass derivation:

```scala

val derived = CaseClass(
  name = "Point",
  fields = List(
    Field("x", TypeRef.Int),
    Field("y", TypeRef.Int)
  ),
  derives = List("Show", "Eq")
)
```

## Examples

Practical examples demonstrate common usage:

### Example 1: Combining Types in a File

A file with multiple `TypeDefinition` variants:

```scala

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(
    // Case class
    CaseClass("Person", List(Field("name", TypeRef.String))),
    
    // Sealed trait (sum type)
    SealedTrait(
      "Result",
      typeParams = List(TypeParam("T")),
      cases = List(
        SealedTraitCase.CaseClassCase(
          CaseClass("Success", List(Field("value", TypeRef("T"))), typeParams = List(TypeParam("T")))
        ),
        SealedTraitCase.CaseObjectCase("Failure")
      )
    ),
    
    // Object with static members
    ObjectDef(
      "Config",
      members = List(
        ObjectMember.ValMember("MaxRetries", TypeRef.Int, "3")
      )
    )
  )
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res15: String = """package com.example
// 
// case class Person(
//   name: String,
// )
// 
// sealed trait Result[T]
// 
// object Result {
//   case class Success[T](
//     value: T,
//   ) extends Result[T]
//   case object Failure extends Result[T]
// }
// 
// object Config {
//   val MaxRetries: Int = 3
// }
// """
```

### Example 2: Type Parameters and Generic Types

Defining a polymorphic case class:

```scala

val box = CaseClass(
  name = "Box",
  fields = List(
    Field("value", TypeRef("A"))
  ),
  typeParams = List(
    TypeParam("A")
  )
)

val file = ScalaFile(
  packageDecl = PackageDecl("com.example"),
  types = List(box)
)
```

Emits:

```scala

ScalaEmitter.emit(file, EmitterConfig())
// res17: String = """package com.example
// 
// case class Box[A](
//   value: A,
// )
// """
```

---

## TypeRef

`TypeRef` represents a reference to a Scala type in the IR. It captures both simple types (like `String`, `Int`) and generic types (like `List[String]`, `Map[String, Int]`).

## Use Cases

- Specifying the types of fields in case classes
- Defining return types of methods
- Representing type arguments in generic types
- Building complex nested type expressions

## Construction

Create a simple type reference:

```scala

val stringType = TypeRef("String")
val intType = TypeRef("Int")
val booleanType = TypeRef("Boolean")
```

With type arguments for generics:

```scala

val listOfString = TypeRef("List", List(TypeRef("String")))
val mapStringInt = TypeRef("Map", List(TypeRef("String"), TypeRef("Int")))
val optionalLong = TypeRef("Option", List(TypeRef("Long")))
```

Using factory methods from the companion object:

```scala

val string = TypeRef.String
val int = TypeRef.Int
val optional = TypeRef.optional(TypeRef.String)
val list = TypeRef.list(TypeRef.Int)
val map = TypeRef.map(TypeRef.String, TypeRef.Int)
```

## Key Operations

All core operations are shown below:

### Accessing Components

Extract parts of a type reference:

```scala

val optional = TypeRef("Option", List(TypeRef.String))

optional.name      // "Option"
optional.typeArgs  // List[TypeRef] = List(TypeRef("String"))
```

### Building Nested Types

Compose type references:

```scala

val nestedList = TypeRef("List", List(
  TypeRef("Option", List(TypeRef("String")))
))
// Represents: List[Option[String]]
```

## Primitive Types

`TypeRef` provides factory methods for built-in types:

```scala

TypeRef.Unit        // Unit
TypeRef.Boolean     // Boolean
TypeRef.Byte        // Byte
TypeRef.Short       // Short
TypeRef.Int         // Int
TypeRef.Long        // Long
TypeRef.Float       // Float
TypeRef.Double      // Double
TypeRef.String      // String
TypeRef.BigInt      // BigInt
TypeRef.BigDecimal  // BigDecimal
TypeRef.Nothing     // Nothing
TypeRef.Any         // Any
```

## Generic Type Factories

Convenience methods simplify common generic patterns:

```scala

TypeRef.optional(TypeRef.String)  // Option[String]
TypeRef.list(TypeRef.Int)         // List[Int]
TypeRef.set(TypeRef.String)       // Set[String]
TypeRef.map(TypeRef.String, TypeRef.Int)  // Map[String, Int]
TypeRef.tuple(TypeRef.String, TypeRef.Int)  // Tuple2[String, Int]
```

## Examples

Practical examples demonstrate common usage:

### Example 1: Field Types

Use `TypeRef` to define field types in a case class:

```scala

val user = CaseClass(
  name = "User",
  fields = List(
    Field("id", TypeRef.Long),
    Field("name", TypeRef.String),
    Field("email", TypeRef.optional(TypeRef.String)),
    Field("tags", TypeRef.list(TypeRef.String))
  )
)
```

### Example 2: Generic Type Parameters

Use `TypeRef` with type variable names:

```scala

val container = CaseClass(
  name = "Container",
  fields = List(
    Field("value", TypeRef("T"))
  ),
  typeParams = List(TypeParam("T"))
)
```

Represents: `Container[T]` with a field `value: T`

### Example 3: Complex Nested Generics

Build deeply nested type expressions:

```scala

val complexType = TypeRef("Map", List(
  TypeRef.String,
  TypeRef("List", List(
    TypeRef("Option", List(TypeRef.Int))
  ))
))
// Represents: Map[String, List[Option[Int]]]

val field = Field("data", complexType)
```

### Example 4: Qualified Type Names

Use fully qualified names for non-standard types:

```scala

val bigDecimal = TypeRef("java.math.BigDecimal")
val jsonObject = TypeRef("com.example.json.JsonObject")

val field1 = Field("price", bigDecimal)
val field2 = Field("metadata", jsonObject)
```

### Example 5: Union and Intersection Types (Scala 3)

Represent Scala 3 union and intersection types:

```scala

// Represents: String | Int
val unionType = TypeRef("|", List(TypeRef.String, TypeRef.Int))

// Represents: Serializable & Comparable
val intersectionType = TypeRef("&", List(
  TypeRef("Serializable"),
  TypeRef("Comparable")
))
```

---

## Combinators

The `combinators` module provides compile-time typeclasses for composing and decomposing values in type-safe ways. Each module focuses on a specific domain: tuples, concatenation widening, Either types, and union types.

## Overview

The combinators module consists of four core modules:

- **Tuples** - Tuple composition with automatic flattening and separation
- **Concat** - Union-aware widening for sequential composition: the supertype or `Either` on Scala 2, `L | R` on Scala 3
- **Eithers** - Either canonicalization to left-nested form
- **Unions** - Union type operations (Scala 3 only)

Each module provides:
- A unified typeclass (e.g., `Tuples.Tuples[L, R]`) that provides both `Tuples.Tuples#combine` and `Tuples.Tuples#separate` operations
- A convenience function like `Tuples.combine` (the `Tuples#separate` operation is available on the typeclass instance)

All typeclasses are derived automatically via compile-time resolution and provide zero-cost abstractions.

## Installation

Add the following to your `build.sbt`:

```sbt
libraryDependencies += "dev.zio" %% "zio-blocks-combinators" % "0.0.56"
```

For cross-platform projects (Scala.js):

```sbt
libraryDependencies += "dev.zio" %%% "zio-blocks-combinators" % "0.0.56"
```

Supported platforms:
- **Tuples, Concat, Eithers**: JVM, Scala.js (Scala 2.13 and 3.x)
- **Unions**: JVM, Scala.js (Scala 3 only)

## Motivation

Building type-safe, composable systems requires managing values and types at both runtime and compile time. The combinators module solves three distinct problems that arise in complex Scala applications:

### The Tuple Nesting Problem

When building up results step-by-step—aggregating function parameters, accumulating intermediate results, or constructing compound values—you often end up with deeply nested tuples:

```scala
// Manual nesting is tedious and error-prone
val step1 = (1, "a")
// step1: Tuple2[Int, String] = (1, "a")
val step2 = (step1, true)
// step2: Tuple2[Tuple2[Int, String], Boolean] = ((1, "a"), true)
val step3 = (step2, 3.14)
// step3: Tuple2[Tuple2[Tuple2[Int, String], Boolean], Double] = (
//   ((1, "a"), true),
//   3.14
// )
val step4 = (step3, 'x') 
// step4: Tuple2[Tuple2[Tuple2[Tuple2[Int, String], Boolean], Double], Char] = (
//   (((1, "a"), true), 3.14),
//   'x'
// )
```

This creates two problems:
1. **Ergonomic burden**: Consumers of compound values must destructure deeply nested structures
2. **Inconsistency**: Different code paths produce different tuple shapes, making composition fragile

The `Tuples` combinator automatically flattens these structures, producing clean, predictable tuples at each step.

### The Either Canonicalization Problem

Error handling often involves composing multiple error types through Either chains. Without systematic canonicalization, Either types nest unpredictably:

```scala
// Inconsistent nesting across code paths
val result1: Either[E1, V] = Left(e1)
val result2: Either[E1, Either[E2, V]] = Right(Left(e2))
val result3: Either[Either[E1, E2], V] = Right(Right(v))
// Each path has a different structure!
```

This causes problems when:
1. **Serializing error types** for schemas (each variant has a different shape)
2. **Accumulating errors** (inconsistent nesting makes aggregation complex)
3. **Pattern matching** (must handle multiple nesting patterns)

The `Eithers` combinator canonicalizes all Either types to a uniform left-nested form, ensuring systematic error composition.

### The Scala 3 Union Type Gap

Scala 3 introduces native union types (`A | B`) that are more idiomatic than `Either[A, B]`. However, existing code, libraries, and serialization infrastructure are built around `Either`. When adopting Scala 3, you face a choice:

1. Stick with `Either` for compatibility (missing idiomatic Scala 3 syntax)
2. Switch to union types (breaking compatibility with Either-based code)
3. Maintain two parallel type systems (duplication and cognitive overhead)

The `Unions` combinator bridges this gap, enabling bidirectional conversion between `Either[L, R]` and `L | R` with zero runtime overhead. Use union types idiomatically in your APIs while maintaining Either compatibility at serialization boundaries.

## Quick Example

Here is how to combine multiple values and canonicalize error types:

```scala

// Aggregate three values into a flattened tuple
val username: String = "alice"
// username: String = "alice"
val userId: Int = 42
// userId: Int = 42
val email: String = "alice@example.com"
// email: String = "alice@example.com"
val userTuple: Tuple3[String, Int, String] = Tuples.combine(username, Tuples.combine(userId, email))
// userTuple: Tuple3[String, Int, String] = ("alice", 42, "alice@example.com")

// Canonicalize nested Either types to left-nested form
val validationError: Either[String, Either[String, Boolean]] = Right(Left("invalid email"))
// validationError: Either[String, Either[String, Boolean]] = Right(
//   Left("invalid email")
// )
val canonical      : Either[Either[String, String], Boolean] = Eithers.combine(validationError)
// canonical: Either[Either[String, String], Boolean] = Left(
//   Right("invalid email")
// )
```

The nested `Tuples.combine` call relies on Scala 3 flattening a tuple on the right-hand side. On Scala 2.13 it gives `(String, (Int, String))` and the `Tuple3` annotation does not compile; see [Tuples: Version Differences](#tuples-version-differences).

## Concat

`Concat` is the witness used by APIs such as `Stream.++` / `Stream.concat` to give the result of a concatenation the same shape on both Scala versions. On Scala 3 it derives `Out = L | R` and is identity-like. On Scala 2 it preserves that union behavior without introducing a separate operator.

On Scala 2, its rules are:

- same type => keep that type
- subtype + supertype => keep the supertype
- siblings with a unique meaningful common supertype => keep that supertype (e.g. `Dog` and `Cat` under sealed `Animal` collapse to `Animal`)
- otherwise (no shared meaningful supertype) => widen to `Either[L, R]` (the Scala 2 encoding of `L | R`)

For example, Scala 2 infers witnesses equivalent to these shapes (on Scala 3 the last one is `Concat.WithOut[String, Int, String | Int]`):

```scala
Concat.WithOut[Int, Int, Int]
Concat.WithOut[Dog, Animal, Animal]
Concat.WithOut[Dog, Cat, Animal]
Concat.WithOut[String, Int, Either[String, Int]]
```

A common supertype is "meaningful" when it is something other than the noise types Scala 2's LUB inference produces by default (`Any`, `AnyRef`, `AnyVal`, `Object`, `Product`, `Serializable`, `java.io.Serializable`, `Comparable`). If filtering those parents leaves exactly one candidate, it becomes the result type — and the witness is identity-like, so callers such as `Stream.concat` reuse values bare without wrapping. Zero or multiple meaningful parents fall through to `Either`.

`Concat` is not usually called directly at runtime. It exists mainly so shared APIs can infer the same public result types on both versions: on Scala 2 through the rules above, on Scala 3 through native unions.

## Tuples

The `Tuples` module combines values into flat tuples and separates them back.

### combine

To combine two values into a flattened tuple:

```scala

val result1 = Tuples.combine(1, "hello")                 // (1, "hello")
val result2 = Tuples.combine((1, "hello"), true)         // (1, "hello", true)
val result3 = Tuples.combine((1, "hello"), (true, 3.14)) // (1, "hello", true, 3.14)
```

#### Identity Handling

Unit and EmptyTuple values are automatically eliminated:

```scala

Tuples.combine((), 42)              // 42
Tuples.combine("hello", ())          // "hello"
Tuples.combine(EmptyTuple, "world")  // "world"
```

#### Tuple Flattening

Nested tuples are automatically flattened:

```scala

Tuples.combine((1, "a"), true)          // (1, "a", true)
Tuples.combine((1, "a"), (true, 3.14))  // (1, "a", true, 3.14)
```

### separate

To split a tuple into its init (all but last) and last element, access `Tuples#separate` via the unified typeclass instance:

```scala

val t2 = summon[Tuples.Tuples[Int, String]]
t2.separate((1, "hello")) // (1, "hello")

val t3 = summon[Tuples.Tuples[(Int, String), Boolean]]
t3.separate((1, "hello", true)) // ((1, "hello"), true)

val t4 = summon[Tuples.Tuples[(Int, String, Boolean), Double]]
t4.separate((1, "hello", true, 3.14)) // ((1, "hello", true), 3.14)
```

When building recursive data structures like path codecs, `separate` decomposes combined tuples to process each segment independently:

```scala

// Simulating recursive path encoding: a codec combines left and right path segments
case class PathSegment(name: String, value: String)

def encodePathSegment(combined: (String, String)): PathSegment = {
  val tuples = summon[Tuples.Tuples[String, String]]
  val (left, right) = tuples.separate(combined)
  PathSegment(left, right)
}

// Decompose a 3-element path into segments for recursive encoding
val path: (String, String, String) = ("users", "123", "profile")
val tuples3 = summon[Tuples.Tuples[(String, String), String]]
val (prefix, suffix) = tuples3.separate(path)
// prefix = ("users", "123"), suffix = "profile"
```

## Eithers

The `Eithers` module canonicalizes Either types to left-nested form and separates them.

### combine

To transform an `Either[L, R]` into its left-nested canonical form:

```scala

// Atomic Either - unchanged
Eithers.combine(Left(42): Either[Int, String])
// res5: Either[Int, String] = Left(42)

// Right-nested Either - reassociates to left-nested
val input2 = Right(Right(true)): Either[Int, Either[String, Boolean]]
// input2: Either[Int, Either[String, Boolean]] = Right(Right(true))
Eithers.combine(input2)
// res6: Either[Either[Int, String], Boolean] = Right(true)

// Left(42) becomes Left(Left(42))
val input3 = Left(42): Either[Int, Either[String, Boolean]]
// input3: Either[Int, Either[String, Boolean]] = Left(42)
Eithers.combine(input3)
// res7: Either[Either[Int, String], Boolean] = Left(Left(42))
```

#### Canonical Form

The canonical form is always left-nested:

```
Right-nested input:          Left-nested output:
Either[A, Either[B, C]]  =>  Either[Either[A, B], C]
Either[A, Either[B, Either[C, D]]]  =>  Either[Either[Either[A, B], C], D]
```

This transformation preserves values while reassociating the structure:
- `Left(a)` → `Left(Left(a))`
- `Right(Left(b))` → `Left(Right(b))`
- `Right(Right(c))` → `Right(c)`

### separate

`Eithers#separate` is accessed via the unified typeclass instance and reverses the canonicalization performed by `combine`. Together, they form a round-trip: canonicalizing to left-nested form and then separating back to the original structure:

```scala

val e = summon[Eithers.Eithers[Int, String]]
// e: Eithers[Int, String] {
  type Out >: Either[Int, String] <: Either[Int, String]
} = zio.blocks.combinators.Eithers$Eithers$AtomicInstance@7f41f6e8
val input = Left(42): Either[Int, String]
// input: Either[Int, String] = Left(42)
e.separate(e.combine(input))
// res8: Either[Int, String] = Left(42)
```

Use `separate` to decompose a canonical Either back to its original structure when you need to handle different error types differently:

```scala

sealed trait ValidationError
case class FieldError(field: String) extends ValidationError
case class FormatError(message: String) extends ValidationError

// You have a right-nested Either from multiple validation steps
val input: Either[FieldError, Either[FormatError, String]] = Right(Left(FormatError("invalid date")))

val eithers = summon[Eithers.Eithers[FieldError, Either[FormatError, String]]]
```

Canonicalize to left-nested form for uniform processing, then reverse it to extract the original error types:

```scala
// Original form: Either[FieldError, Either[FormatError, String]]
input
// res9: Either[FieldError, Either[FormatError, String]] = Right(
//   Left(FormatError("invalid date"))
// )

// Canonicalize to left-nested form for uniform processing
val canonicalized = eithers.combine(input)
// canonicalized: Either[Either[FieldError, FormatError], String] = Left(
//   Right(FormatError("invalid date"))
// )

// Reverse canonicalization to extract the original error types
val original = eithers.separate(canonicalized)
// original: Either[FieldError, Either[FormatError, String]] = Right(
//   Left(FormatError("invalid date"))
// )

// Back to original form
original
// res10: Either[FieldError, Either[FormatError, String]] = Right(
//   Left(FormatError("invalid date"))
// )
```

Handle each error type independently:

```scala
// Handle each error type independently
original match {
  case Left(fieldErr: FieldError) => println(s"Field validation failed: ${fieldErr.field}")
  case Right(Left(formatErr: FormatError)) => println(s"Format error: ${formatErr.message}")
  case Right(Right(value)) => println(s"Valid: $value")
}
```

## Unions (Scala 3 Only)

The `Unions` module converts between Either types and Scala 3 union types.

### combine

`Unions.Unions[L, R]` converts an `Either[L, R]` to a union type `L | R`:

```scala

val either1 = Left(42): Either[Int, String]
// either1: Either[Int, String] = Left(42)
Unions.combine(either1)
// res12: Int | String = 42

val either2 = Right("hello"): Either[Int, String]
// either2: Either[Int, String] = Right("hello")
Unions.combine(either2)
// res13: Int | String = "hello"
```

### separate

`Unions#separate` is accessed via the unified typeclass instance and discriminates a union type back to Either:

```scala

val u = summon[Unions.Unions.WithOut[Int, String, Int | String]]
// u: Unions[Int, String] {
  type Out >: Int | String <: Int | String
} = zio.blocks.combinators.Unions$Unions$UnionInstance@ff63408
u.separate(42: Int | String)
// res14: Either[Int, String] = Left(42)
u.separate("hello": Int | String)
// res15: Either[Int, String] = Right("hello")
```
### Same-Type Rejection

Union types collapse same types (`A | A` = `A`), making them ambiguous. The separator rejects overlapping types at compile time:

```scala

// Compile error: Union types must contain unique types
// val u = summon[Unions.Unions.WithOut[Int, Int, Int | Int]]

// Use Either for same-type alternation instead:

val either: Either[Int, Int] = Left(1)  // Distinguishable via Left/Right
```

### Type Erasure Caveat

Union discrimination relies on runtime type tests, which are fragile for erased types:

```scala

// Problematic: List[Int] and List[String] erase to List
val problematicValue: List[Int] | List[String] = List(1, 2, 3)
// Runtime cannot distinguish List[Int] from List[String]

// Safe: Use distinct concrete types
val value: Int | String = 42  // Works reliably
```

## Generic Usage Patterns

The combinators module supports both Scala 2's implicit parameters and Scala 3's context parameters. Here are idiomatic usage patterns for each:

<Tabs groupId="scala-version" defaultValue="scala2">
  <TabItem value="scala2" label="Scala 2">

To combine multiple values using implicit typeclass resolution:

```scala

def combineAll[A, B, C](a: A, b: B, c: C)(
  implicit ab: Tuples.Tuples[A, B],
           abc: Tuples.Tuples[ab.Out, C]
): abc.Out = {
  val step1 = ab.combine(a, b)
  abc.combine(step1, c)
}

val result = combineAll(1, "hello", true)
// result: (Int, String, Boolean)
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3">

To combine multiple values using context parameters:

```scala

def combineAll[A, B, C](a: A, b: B, c: C)(using
  ab: Tuples.Tuples[A, B],
  abc: Tuples.Tuples[ab.Out, C]
): abc.Out =
  val step1 = ab.combine(a, b)
  abc.combine(step1, c)

val result = combineAll(1, "hello", true)
// result: (Int, String, Boolean)
```

  </TabItem>
</Tabs>

### Path-Dependent Types

The `Out` type member is path-dependent:

```scala

def process[L, R](l: L, r: R)(using t: Tuples.Tuples[L, R]): (L, R) =
  t.separate(t.combine(l, r))

val result: (Int, String) = process(1, "hello")
```

## Integration Points

The combinator types integrate with other ZIO Blocks modules through systematic composition:

**Sum Type Encoding**: The `Eithers` canonicalization strategy gives every Either encoding of a sum type the same left-nested form, so code that serializes such types can rely on one shape. The `schema` module does not use it itself.

**Error Handling**: `Eithers` provides a foundation for systematic error composition. Libraries building polymorphic error types can leverage canonicalization to ensure uniform error nesting, preventing subtle bugs from inconsistent Either structure.

**Scala 3 APIs**: The `Unions` type enables idiomatic Scala 3 DSLs and API designs that use native union syntax. Gateway types that convert between union-based and Either-based representations (e.g., for serialization compatibility) can use `Unions` for zero-cost interop.

**Tuple-Based Builders**: The `Tuples` module supports builder patterns and accumulator-based APIs that need to combine heterogeneous values step-by-step. By flattening automatically, it eliminates the ergonomic burden of manual nesting, making fluent builder chains natural.

## Scala 2 vs Scala 3: Compatibility and Differences

The combinators module works across Scala 2.13 and Scala 3.x with **full source compatibility**. Write your code once; it compiles on both versions. However, certain features are version-specific due to language capabilities:

### Tuples: Version Differences

**Scala 2.13 Limitations:**
- Maximum arity of 22 (the standard library tuple limit)
- Tuple flattening only works when the left argument is a tuple (right argument cannot be recursively flattened)
- No `EmptyTuple` type (use `Unit` as identity instead)

**Scala 3.x Enhancements:**
- Unlimited arity (tuples are truly variable-length)
- Recursive flattening on both sides: `Tuples.combine((1, "a"), (true, 3.14))` flattens both tuples into a 4-tuple
- `EmptyTuple` as a first-class type with proper identity semantics

**Example: The difference in practice**

<Tabs groupId="scala-version" defaultValue="scala2">
  <TabItem value="scala2" label="Scala 2.13">

In Scala 2.13, combining two tuples compiles, but only the left tuple is flattened; the right one stays nested (a known limitation of the Scala 2 implementation, not a compile error):

```scala

// compiles; right side not flattened
val result = Tuples.combine((1, 2), (3, 4))  // (1, 2, (3, 4))
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3.x">

In Scala 3.x, recursive flattening on both sides works seamlessly:

```scala

// OK: both sides flattened
val result = Tuples.combine((1, 2), (3, 4))  // (1, 2, 3, 4)
```

  </TabItem>
</Tabs>

### Eithers: Full Cross-Version Support

`Eithers` canonicalization produces the same shapes on Scala 2.13 and 3.x, and is deterministic. One difference remains in how you obtain the instance: on Scala 2.13, `implicitly[Eithers.Eithers[L, R]]` loses the `Out` type member (the macro's declared type is `Eithers[L, R]`), so `separate` is usable only through an implicit parameter, as in the Scaladoc examples. The same applies to `Tuples`, and the `summon[...]` examples on this page are Scala 3 only.

### Unions: Scala 3 Only

`Unions` requires Scala 3 because:
- Union types (`A | B`) are a Scala 3 language feature
- Runtime type tests (via `TypeTest`) are only available in Scala 3
- Scala 2 has no native union syntax

For Scala 2.13 codebases, use `Either` directly or `Eithers` canonicalization instead.

### Feature Matrix

| Feature                        | Scala 2.13 | Scala 3.x | Notes                         |
|--------------------------------|------------|-----------|-------------------------------|
| **Tuples.combine**             | ✅          | ✅         | Left-only flattening in 2.13  |
| **Tuples.separate**            | ✅          | ✅         | Works identically on both     |
| **Eithers.combine**            | ✅          | ✅         | No differences                |
| **Eithers.separate**           | ✅          | ✅         | No differences                |
| **Unions.combine**             | ❌          | ✅         | Requires Scala 3              |
| **Unions.separate**            | ❌          | ✅         | Requires Scala 3              |
| **EmptyTuple as identity**     | ❌          | ✅         | Use `Unit` in Scala 2         |
| **Unlimited tuple arity**      | ❌          | ✅         | Limited to 22 in Scala 2      |
| **Recursive tuple flattening** | ❌          | ✅         | Right side not flattened in 2 |

### Migration Path from Scala 2 to 3

When adopting Scala 3, no changes are required for existing `Tuples` and `Eithers` code. Your code continues to work without modification. However, you can take advantage of new capabilities:

1. **Adopt `EmptyTuple` idiom**: Use `EmptyTuple` instead of `Unit` when combining with `Tuples` in Scala 3 for consistency with modern tuple syntax. Note that `Unit` remains fully supported and valid—`EmptyTuple` is a stylistic enhancement, not a replacement.
2. **Simplify tuple builders**: Leverage recursive flattening on both sides to remove manual nesting. In Scala 3, `Tuples.combine((1, "a"), (true, 3.14))` automatically flattens to `(1, "a", true, 3.14)`.
3. **Adopt `Unions` in Scala 3-only code**: Replace `Either` with union types in new Scala 3-only code for idiomatic syntax using the `Unions` combinator.
4. **Gradual adoption**: Keep `Either` in cross-version modules and use `Unions` in Scala-3-only modules. Convert between them at module boundaries using `Unions.combine` and `Unions.separate` as needed.

## See Also

- **HTTP Model Schema** — When extracting multiple typed query parameters or headers in the HTTP Model schema module, `Eithers` provides systematic composition of error types for uniform error handling.

---

## Concurrency

## Introduction

The blocks in this group deal with work that overlaps in time: computations that wait, streams that share one connection and queues that hand elements between threads. All three run on the JVM and in Scala.js.

## The Blocks

The group has three blocks, one for each of those problems.

| Block                              | Artifact                 | What It Is For                                                                   |
| ---------------------------------- | ------------------------ | -------------------------------------------------------------------------------- |
| [Async](../async.md)               | `zio-blocks-async`       | Zero-allocation asynchronous effect type with direct-style `await`               |
| [Mux](../mux.mdx)                  | `zio-blocks-mux`         | Thread-safe multiplexer for HTTP/2, QUIC, and WebSocket-style protocols          |
| [RingBuffer](../ringbuffer/index.mdx) | `zio-blocks-ringbuffer` | Lock-free bounded ring buffers (SPSC, SPMC, MPSC, MPMC)                          |

## How the Blocks Differ

`Async` is a small asynchronous effect type. Building an `Async` value runs it, and only a computation that has to wait for something is left pending.

`Mux` manages many independent streams over one shared transport, each identified by an ID and each with its own inbound and outbound queues. It targets protocols such as HTTP/2 and QUIC.

`RingBuffer` is a family of fixed-size, lock-free queues with four variants for different producer and consumer patterns, which recycle memory instead of creating garbage.

## Where to Start

Start with [Async](../async.md) to write asynchronous code in a direct style. Read [RingBuffer](../ringbuffer/index.mdx) when threads must exchange elements with little contention, and [Mux](../mux.mdx) when building a protocol implementation over a multiplexed connection.

---

## ConfigDecoder

`ConfigDecoder[A]` reads a value of type `A` out of a `ConfigSource`, accumulating every failure instead of stopping at the first one. Instances are derived from `Schema[A]` by `ConfigDecoderDeriver`, which defines how records, variants, sequences, maps, and primitives map onto dot-separated keys. The trait and its derivation entry point:

```scala
trait ConfigDecoder[A] {
  def decode(source: ConfigSource, prefix: String): Either[::[ConfigError], A]
}

object ConfigDecoder {
  def apply[A](implicit decoder: ConfigDecoder[A]): ConfigDecoder[A]
  def derive[A](implicit schema: Schema[A]): ConfigDecoder[A]
}
```

## Motivation

Writing a config decoder by hand means writing the same code once per field: look up a key, parse the string, handle the missing case, collect the error. For a twenty-field config that is twenty near-identical blocks that drift out of sync with the case class the moment someone adds a field.

`Schema[A]` already knows the field names, the types, and the defaults. `ConfigDecoderDeriver` turns that knowledge into the lookup-and-parse loop, so adding a field to a case class adds a key to the config with no other change. The derivation is a runtime walk over the schema, which means it is also introspectable: the same `Deriver` machinery lets you override a single type's decoding without rewriting the rest.

The `prefix` parameter exists so decoders compose. A record decoder calls its field decoders with `s"$prefix.$fieldName"`, and the recursion bottoms out at primitives that look up exactly one key.

## Deriving a Decoder

`ConfigDecoder.derive` builds a decoder from an implicit `Schema[A]` using the default deriver:

```scala

case class Db(host: String, port: Int)

object Db {
  implicit val schema: Schema[Db] = Schema.derived[Db]
}

val decoder = ConfigDecoder.derive[Db]
```

Decoding with an empty prefix treats field names as top-level keys:

```scala
decoder.decode(ConfigSource.fromMap(Map("host" -> "localhost", "port" -> "5432")), "")
// res0: Either[::[ConfigError], Db] = Right(
//   Db(host = "localhost", port = 5432)
// )
```

Passing a non-empty prefix roots the same decoder under a subtree, which is how one decoder serves both a standalone config file and a section of a larger one:

```scala
decoder.decode(ConfigSource.fromMap(Map("db.host" -> "localhost", "db.port" -> "5432")), "db")
// res1: Either[::[ConfigError], Db] = Right(
//   Db(host = "localhost", port = 5432)
// )
```

`Config.load[A]` is a thin wrapper over these two lines. The difference matters for repeated loads: `Config.load` derives a fresh decoder on every call, while a decoder you hold onto is derived once.

:::tip[Derive once for repeated loads]
Derivation walks the entire schema and allocates a decoder per field. For a config reloaded on a timer, or decoded per request, call `ConfigDecoder.derive[A]` at startup and reuse the instance.
:::

## Mapping Rules

The deriver has one rule per schema shape. Together they determine every key your config file needs, so this section is the reference for "what key does field X read?".

### Records

A record's fields become keys under the record's prefix, joined with dots. Nesting composes: a field whose type is itself a record contributes another path segment:

```scala

case class Db(host: String, port: Int)

object Db {
  implicit val schema: Schema[Db] = Schema.derived[Db]
}

case class Http(host: String, port: Int)

object Http {
  implicit val schema: Schema[Http] = Schema.derived[Http]
}

case class App(db: Db, http: Http)

object App {
  implicit val schema: Schema[App] = Schema.derived[App]
}
```

Four keys, two per nested record, all reachable from one top-level decode:

```scala
Config.load[App](
  ConfigSource.fromMap(
    Map(
      "db.host"   -> "dbhost",
      "db.port"   -> "5432",
      "http.host" -> "0.0.0.0",
      "http.port" -> "8080"
    )
  )
)
// res3: Either[::[ConfigError], App] = Right(
//   App(
//     db = Db(host = "dbhost", port = 5432),
//     http = Http(host = "0.0.0.0", port = 8080)
//   )
// )
```

The deriver uses field names verbatim, in the casing the Scala source declares. To read a differently-cased namespace, wrap the source with `ConfigSource#keyFormat` rather than renaming fields — see [ConfigSource](./config-source.md).

### Primitives

A primitive field looks up exactly one key and parses the string. Parsing rules by type:

| Schema type                                              | Accepted form                                  | Example        |
| -------------------------------------------------------- | ---------------------------------------------- | -------------- |
| `Boolean`                                                | `true`/`false`, `1`/`0`, `yes`/`no`, `on`/`off` (any case) | `on`  |
| `Byte`, `Short`, `Int`, `Long`                            | Decimal integer literal                        | `5432`         |
| `Float`, `Double`                                        | Decimal or scientific literal                  | `0.75`         |
| `Char`                                                   | Exactly one character                          | `x`            |
| `String`                                                 | Any string, taken as-is                        | `localhost`    |
| `BigInt`, `BigDecimal`                                    | Arbitrary-precision numeric literal            | `1e40`         |
| `UUID`                                                   | Canonical 36-character form                    | `f81d4fae-…`   |
| `java.time.Duration`                                     | ISO-8601 duration                              | `PT30S`        |
| `Instant`, `LocalDate`, `LocalDateTime`, `LocalTime`      | ISO-8601                                       | `2026-08-21`   |
| `OffsetDateTime`, `OffsetTime`, `ZonedDateTime`           | ISO-8601 with offset or zone                   | `2026-08-21T00:00:00Z` |
| `ZoneId`, `ZoneOffset`                                   | Zone name or offset                            | `Europe/Berlin` |
| `Period`, `Year`, `YearMonth`, `MonthDay`                 | ISO-8601                                       | `P1M`          |
| `Month`, `DayOfWeek`                                     | Enum name, uppercased before lookup            | `monday`       |
| `Currency`                                               | ISO 4217 code                                  | `EUR`          |
| `Unit`                                                   | Any value; the key must exist                  | `()`           |

A value that fails to parse produces `ConfigError.InvalidValue` naming the path, the offending string, the expected type, and the source:

```scala

case class Db(host: String, port: Int)

object Db {
  implicit val schema: Schema[Db] = Schema.derived[Db]
}
```

The underlying parse exception is retained as the error's `cause`:

```scala
Config.load[Db](ConfigSource.fromMap(Map("host" -> "localhost", "port" -> "not-a-number"), "bad"))
// res5: Either[::[ConfigError], Db] = Left(
//   List(
//     InvalidValue(
//       path = "port",
//       value = "not-a-number",
//       expectedType = "Int",
//       source = "bad",
//       cause = Some(
//         java.lang.NumberFormatException: For input string: "not-a-number"
//       )
//     )
//   )
// )
```

:::warning[Two different duration formats]
Config decoding parses `java.time.Duration` with `Duration.parse`, so it requires ISO-8601 (`PT30S`). Flag readers parse `scala.concurrent.duration.FiniteDuration` with a suffix grammar instead (`30s`). The same string is not valid in both places — see [Flags](./flags.md).
:::

### Optional Fields

A field of type `Option[A]` decodes the inner `A` at the same key the field would otherwise use — there is no extra path segment for the `Option` itself. A missing key yields `None`:

```scala

case class Service(name: String, tag: Option[String])

object Service {
  implicit val schema: Schema[Service] = Schema.derived[Service]
}
```

Supplying the key produces `Some`, and omitting it produces `None`:

```scala
Config.load[Service](ConfigSource.fromMap(Map("name" -> "api", "tag" -> "v2")))
// res7: Either[::[ConfigError], Service] = Right(
//   Service(name = "api", tag = Some("v2"))
// )
Config.load[Service](ConfigSource.fromMap(Map("name" -> "api")))
// res8: Either[::[ConfigError], Service] = Right(
//   Service(name = "api", tag = None)
// )
```

The distinction that matters is *missing* versus *invalid*. An optional field absorbs missing keys only. A key that is present but unparseable still fails the whole decode, because silently returning `None` for a typo would hide the mistake.

### Default Values

A field with a default in the case class falls back to that default when its key is missing, exactly as an `Option` falls back to `None`:

```scala

case class Db(host: String, port: Int = 5432)

object Db {
  implicit val schema: Schema[Db] = Schema.derived[Db]
}
```

Omitting `port` uses the declared default rather than reporting a missing key:

```scala
Config.load[Db](ConfigSource.fromMap(Map("host" -> "localhost")))
// res10: Either[::[ConfigError], Db] = Right(
//   Db(host = "localhost", port = 5432)
// )
```

As with optional fields, the fallback applies only to missing keys. A present-but-invalid value is an error, not a reason to use the default.

### Sequences

A sequence is read in one of two shapes. The indexed shape uses one key per element, numbered from zero, and the deriver probes upward until it finds a gap:

```scala

case class Hosts(items: List[String])

object Hosts {
  implicit val schema: Schema[Hosts] = Schema.derived[Hosts]
}
```

Indexed keys are what the YAML and JSON adapters produce when they flatten an array:

```scala
Config.load[Hosts](
  ConfigSource.fromMap(Map("items.0" -> "alpha", "items.1" -> "beta", "items.2" -> "gamma"))
)
// res12: Either[::[ConfigError], Hosts] = Right(
//   Hosts(List("alpha", "beta", "gamma"))
// )
```

When no indexed key exists, the deriver falls back to reading the prefix itself as a single comma-separated string, which is what an environment variable can realistically hold:

```scala
Config.load[Hosts](ConfigSource.fromMap(Map("items" -> "alpha, beta, gamma")))
// res13: Either[::[ConfigError], Hosts] = Right(
//   Hosts(List("alpha", "beta", "gamma"))
// )
```

Elements are trimmed after splitting, so `"alpha, beta"` and `"alpha,beta"` are equivalent. An element containing a comma cannot be expressed in the flat form; use indexed keys for those.

Probing stops at the first index that has neither a value nor any nested keys beneath it, so `items.0` and `items.2` without `items.1` yields a one-element list rather than an error.

### Maps

A map's keys are discovered by enumeration: the deriver calls `ConfigSource#all` on the prefix and takes the first path segment after it as a map key. Values decode at `prefix.key`:

```scala

case class Limits(counts: Map[String, Int])

object Limits {
  implicit val schema: Schema[Limits] = Schema.derived[Limits]
}
```

Each distinct segment under `counts` becomes one entry:

```scala
Config.load[Limits](ConfigSource.fromMap(Map("counts.read" -> "100", "counts.write" -> "20")))
// res15: Either[::[ConfigError], Limits] = Right(
//   Limits(Map("read" -> 100, "write" -> 20))
// )
```

Because discovery goes through `ConfigSource#all`, a map cannot be decoded from a source that does not enumerate. Map keys themselves are decoded through the key type's decoder, so a `Map[Int, String]` requires segments that parse as integers.

### Sealed Traits

A sealed trait is decoded by reading a discriminator key that names which case to use. The default key is `type`, read at `prefix.type`:

```scala

sealed trait Backend

object Backend {
  case class Postgres(host: String, port: Int) extends Backend
  case class Sqlite(path: String)              extends Backend

  implicit val schema: Schema[Backend] = Schema.derived[Backend]
}

case class Store(backend: Backend)

object Store {
  implicit val schema: Schema[Store] = Schema.derived[Store]
}
```

The discriminator selects the case, and the remaining keys are read at the same prefix as the case's own fields:

```scala
Config.load[Store](
  ConfigSource.fromMap(
    Map("backend.type" -> "Postgres", "backend.host" -> "localhost", "backend.port" -> "5432")
  )
)
// res17: Either[::[ConfigError], Store] = Right(
//   Store(Postgres(host = "localhost", port = 5432))
// )
```

A missing discriminator produces `ConfigError.MissingDiscriminatorKey`, which names the path and the key it expected:

```scala
Config.load[Store](ConfigSource.fromMap(Map("backend.host" -> "localhost")))
// res18: Either[::[ConfigError], Store] = Left(
//   List(MissingDiscriminatorKey(path = "backend", key = "type"))
// )
```

An unrecognized discriminator produces `ConfigError.UnknownDiscriminator`, which lists every accepted value in sorted order — the error tells the reader what to write instead:

```scala
Config.load[Store](ConfigSource.fromMap(Map("backend.type" -> "MySql")))
// res19: Either[::[ConfigError], Store] = Left(
//   List(
//     UnknownDiscriminator(
//       path = "backend.type",
//       found = "MySql",
//       expected = List("Postgres", "Sqlite")
//     )
//   )
// )
```

Case names are matched exactly as the schema reports them, which for a plain sealed trait is the simple class name.

### Wrappers

A wrapper type — one whose schema is a `Binding.Wrapper`, such as a newtype over `String` — decodes its underlying representation and then applies the wrapping function. If wrapping throws, the failure is reported as `ConfigError.InvalidValue` at the wrapper's path, with the thrown exception as the cause. This is how validating newtypes surface their validation failures as ordinary config errors.

### Dynamic Values

A `DynamicValue` field is not parsed at all. A present key becomes a `DynamicValue.Primitive` holding the raw string, and an absent key becomes `DynamicValue.Null` rather than an error. Use it for config sections whose shape is not known at compile time.

## Customizing Derivation

`ConfigDecoderDeriver` is an ordinary `Deriver[ConfigDecoder]`, so the schema layer's override mechanisms apply. Two customizations are specific to config decoding.

### Choosing a Discriminator Key

`ConfigDecoderDeriver#discriminator` returns a new deriver that reads a different discriminator key. Pass the deriver to `Config.load` to use it:

```scala

sealed trait Backend

object Backend {
  case class Postgres(host: String, port: Int) extends Backend
  case class Sqlite(path: String)              extends Backend

  implicit val schema: Schema[Backend] = Schema.derived[Backend]
}

case class Store(backend: Backend)

object Store {
  implicit val schema: Schema[Store] = Schema.derived[Store]
}

val kindDeriver = ConfigDecoderDeriver.discriminator("kind")
```

The same config now selects its case from `backend.kind`:

```scala
Config.load[Store](
  ConfigSource.fromMap(Map("backend.kind" -> "Sqlite", "backend.path" -> "/tmp/db.sqlite")),
  kindDeriver
)
// res21: Either[::[ConfigError], Store] = Right(
//   Store(Sqlite("/tmp/db.sqlite"))
// )
```

`ConfigDecoderDeriver` — the object — is the default instance, equivalent to `new ConfigDecoderDeriver("type")`.

### Overriding a Single Type

To change how one type decodes while leaving everything else alone, build the decoder through the schema's `deriving` API and supply an instance override. This is the standard `Deriver` workflow described in the schema module's derivation documentation; the only config-specific part is that the type class being overridden is `ConfigDecoder`.

## Error Accumulation

Decoding does not stop at the first failure. A record decoder attempts every field, collects the errors, and only then decides the outcome. The exact shape of the `Left` depends on how many errors it finds: a single error is returned on its own, and two or more are wrapped in one `ConfigError.Composite`.

That wrapping happens per record, so a nested config produces nested composites — one per record that had more than one failing field:

```scala

case class Db(host: String, port: Int)

object Db {
  implicit val schema: Schema[Db] = Schema.derived[Db]
}
```

An empty source fails both fields, so both errors arrive together:

```scala
Config.load[Db](ConfigSource.fromMap(Map.empty[String, String], "empty"))
// res23: Either[::[ConfigError], Db] = Left(
//   List(
//     Composite(
//       List(
//         MissingKey(path = "host", source = "empty"),
//         MissingKey(path = "port", source = "empty")
//       )
//     )
//   )
// )
```

`ConfigError.Composite#message` joins the child messages with newlines, and `Config.loadOrThrow` formats them into a numbered report. See [Errors](./errors.md) for the full error model.

## Integration Points

`ConfigDecoder` sits between the schema module and the source layer. It depends on `Schema[A]` and the `Deriver` machinery from `zio-blocks-schema`, reads through `ConfigSource`, and reports failures as `ConfigError`. `Config` is the user-facing wrapper, and `Config.wire` places a derived decoder inside a `zio-blocks-scope` dependency graph.

See [ConfigSource](./config-source.md) for building the sources a decoder reads from, and [Errors](./errors.md) for handling what it returns.

---

## ConfigSource

`ConfigSource` is a flat, string-keyed namespace of configuration values with dot-separated paths. It answers single-key lookups, enumerates keys under a prefix, and pairs every value it returns with a `Provenance` recording where that value came from. Supporting types: `SourceValue`, `Provenance`, `ProvenanceMap`, `KeyMapper`, `KeyFormat`, `Secret`. The reading and composition surface:

```scala
trait ConfigSource extends FlagSource {
  def sourceId: String
  def get(key: String): Maybe[SourceValue[String]]
  def all(prefix: String): Map[String, SourceValue[String]]

  final def orElse(fallback: ConfigSource): ConfigSource
  final def prefix(prefix: String): ConfigSource
  final def keyMapper(mapper: KeyMapper, targetFormat: KeyFormat): ConfigSource
  final def keyFormat(format: KeyFormat): ConfigSource
}
```

## Motivation

Configuration arrives as strings, from places that disagree about structure. Environment variables are flat and uppercase. YAML is nested. HOCON has substitutions. A decoder that had to understand all three would be three decoders.

`ConfigSource` is the one shape they all reduce to: a map from dotted path to string. Nested documents flatten into it, environment variables translate into it, and in-memory maps are already it. Because the shape is uniform, everything built above it — decoding, composition, provenance, key renaming — is written once.

The trait extends `FlagSource`, which is the same idea minus prefix enumeration. That inheritance is what lets a single source object serve both typed configuration and feature flags.

## Construction

Sources come from three places in the core module — a map, the environment, and system properties — plus one constructor per file format.

### From a Map

`ConfigSource.fromMap` wraps a `Map[String, String]`, optionally with an identifier that shows up in provenance and error messages:

```scala

val source = ConfigSource.fromMap(
  Map("db.host" -> "localhost", "db.port" -> "5432"),
  "defaults"
)
```

Looking up a key returns the value wrapped in a `SourceValue`, which carries the provenance alongside the string:

```scala
source.get("db.host")
// res0: Maybe[SourceValue[String]] = SourceValue(
//   value = "localhost",
//   provenance = Resolved(
//     sourceId = "defaults",
//     key = "db.host",
//     rawValue = "localhost"
//   )
// )
```

An absent key yields `Maybe.absent` rather than throwing or returning `null`:

```scala
source.get("db.user")
// res1: Maybe[SourceValue[String]] = zio.blocks.maybe.Absent$@b322d7c
```

`ConfigSource.MapSource` is the underlying case class, so you can pattern match on it or construct it directly when you want the concrete type rather than the trait.

### From the Environment

`EnvSource` reads environment variables. It performs its own key translation: dots become underscores and the result is upper-cased, so `get("db.host")` reads `DB_HOST`. It does not split camelCase, so a `databaseUrl` field needs the mapping described in [Applying a Mapper to a Source](#applying-a-mapper-to-a-source):

```scala

EnvSource.get("db.host")     // reads DB_HOST
EnvSource.all("db")          // every DB_* variable, keys mapped back to dotted form
```

An unset variable is absent. An empty string is present — `FOO=""` resolves to `Some("")`, not a missing key, which matters when you use an empty value to mean "explicitly disabled".

`SysPropSource` reads JVM system properties using the dotted path directly, with no translation.

:::info[Scala.js behavior]
On Scala.js, `EnvSource` reads `process.env` when it is available and returns absent for everything when it is not. `SysPropSource` always returns empty results, even for properties set with `System.setProperty`, which Scala.js emulates: the source never reads them.
:::

### From a File Format

The YAML, JSON, and HOCON adapters each add a constructor to the `ConfigSource` companion via an implicit class, so importing the adapter package makes `ConfigSource.fromYaml`, `ConfigSource.fromJson`, or `ConfigSource.fromHocon` available. Because parsing can fail, those constructors return `Either[ConfigError, ConfigSource]`. See [File Formats](./formats.md).

## Lookup Operations

Two methods make up the reading surface: one resolves a single key, the other enumerates a subtree.

### ConfigSource#get

`ConfigSource#get` takes a full dotted path and returns `Maybe[SourceValue[String]]`. It never partially matches: `get("db")` on a source containing only `db.host` is absent, because `db` itself has no value.

### ConfigSource#all

`ConfigSource#all` returns every entry whose key equals the prefix or begins with the prefix followed by a dot. Passing an empty prefix returns everything:

```scala
source.all("db")
// res3: Map[String, SourceValue[String]] = Map(
//   "db.host" -> SourceValue(
//     value = "localhost",
//     provenance = Resolved(
//       sourceId = "defaults",
//       key = "db.host",
//       rawValue = "localhost"
//     )
//   ),
//   "db.port" -> SourceValue(
//     value = "5432",
//     provenance = Resolved(
//       sourceId = "defaults",
//       key = "db.port",
//       rawValue = "5432"
//     )
//   )
// )
```

The decoder uses `ConfigSource#all` to discover the shape of collections — how many elements a sequence has, which keys a map contains — so a source that cannot enumerate cannot decode those types.

## Composition

Real deployments layer configuration: defaults from a file, overrides from the environment, and a subtree per component. Two operations cover both needs, and both are `final` so every source gets them.

### ConfigSource#orElse

`ConfigSource#orElse` consults the receiver first and falls back to the argument. Provenance records whichever source actually answered, so layering does not obscure the origin:

```scala

val defaults     = ConfigSource.fromMap(Map("host" -> "localhost", "port" -> "5432"), "defaults")
val envOverrides = ConfigSource.fromMap(Map("host" -> "db.prod.internal"), "env")

val layered = defaults.orElse(envOverrides)
```

The receiver wins on conflict, which means the *first* source listed has priority — write the highest-priority source on the left:

```scala
layered.get("host")
// res5: Maybe[SourceValue[String]] = SourceValue(
//   value = "localhost",
//   provenance = Resolved(
//     sourceId = "defaults",
//     key = "host",
//     rawValue = "localhost"
//   )
// )
```

Keys only the fallback provides still resolve, and their provenance names the fallback:

```scala
layered.get("port")
// res6: Maybe[SourceValue[String]] = SourceValue(
//   value = "5432",
//   provenance = Resolved(sourceId = "defaults", key = "port", rawValue = "5432")
// )
```

The composed source's `sourceId` is the two ids joined with a pipe, which is what you see in error messages when a key is missing from both:

```scala
layered.sourceId
// res7: String = "defaults|env"
```

`ConfigSource#all` on a composed source merges both key sets, with the receiver's entries overwriting the fallback's on collision. That merge is why enumeration-driven decoding — sequences and maps — behaves the same on a layered source as on a single one.

### ConfigSource#prefix

`ConfigSource#prefix` re-roots a source so that lookups are relative to a subtree. A source prefixed with `db` turns `get("host")` into `get("db.host")` against the underlying source:

```scala

val full = ConfigSource.fromMap(
  Map("db.host" -> "localhost", "db.port" -> "5432", "http.port" -> "8080"),
  "app"
)

val db = full.prefix("db")
```

Relative lookups resolve against the composed key, so the caller never spells the prefix again:

```scala
db.get("host")
// res9: Maybe[SourceValue[String]] = SourceValue(
//   value = "localhost",
//   provenance = Resolved(
//     sourceId = "app",
//     key = "db.host",
//     rawValue = "localhost"
//   )
// )
```

Enumeration strips the prefix back off the returned keys, which keeps the view consistent — what `ConfigSource#get` resolves is what `ConfigSource#all` reports:

```scala
db.all("")
// res10: Map[String, SourceValue[String]] = Map(
//   "host" -> SourceValue(
//     value = "localhost",
//     provenance = Resolved(
//       sourceId = "app",
//       key = "db.host",
//       rawValue = "localhost"
//     )
//   ),
//   "port" -> SourceValue(
//     value = "5432",
//     provenance = Resolved(sourceId = "app", key = "db.port", rawValue = "5432")
//   )
// )
```

Prefixing preserves `sourceId`, since re-rooting does not change where values come from. This is the mechanism behind `Config.wire[A](prefix)`: one injected source, several independently-rooted config sections.

## Key Mapping

Field names in Scala are camelCase. Environment variables are `UPPER_SNAKE_CASE`. Some YAML files use `kebab-case`. Rather than renaming fields or duplicating keys, a source can translate between the canonical form the decoder asks for and whatever form the source actually uses.

### KeyFormat

`KeyFormat` enumerates the four supported spellings of a key:

| Variant                   | Example        |
| ------------------------- | -------------- |
| `KeyFormat.CamelCase`     | `databaseUrl`  |
| `KeyFormat.SnakeCase`     | `database_url` |
| `KeyFormat.KebabCase`     | `database-url` |
| `KeyFormat.UpperSnakeCase`| `DATABASE_URL` |

### KeyMapper

`KeyMapper` converts in both directions: `KeyMapper#toCanonical` normalizes a source-facing key into lower camelCase, and `KeyMapper#fromCanonical` renders a canonical key into a requested `KeyFormat`:

```scala

val mapper = KeyMapper.default
```

The default mapper treats snake_case and kebab-case as equivalent inputs, collapsing both to camelCase:

```scala
mapper.toCanonical("database_url")
// res12: String = "databaseUrl"
mapper.toCanonical("database-url")
// res13: String = "databaseUrl"
```

Rendering goes the other way, one output per format:

```scala
mapper.fromCanonical("databaseUrl", KeyFormat.UpperSnakeCase)
// res14: String = "DATABASE_URL"
mapper.fromCanonical("databaseUrl", KeyFormat.KebabCase)
// res15: String = "database-url"
```

A key containing neither separator passes through `KeyMapper#toCanonical` unchanged, so already-canonical keys cost nothing.

### Applying a Mapper to a Source

`ConfigSource#keyFormat` wraps a source so that every lookup is rendered into the given format before it reaches the underlying source. Ask for `databaseUrl`, and the wrapped source looks up `DATABASE_URL`:

```scala

val upperSnake = ConfigSource.fromMap(Map("DATABASE_URL" -> "postgres://localhost"), "env-style")
val canonical  = upperSnake.keyFormat(KeyFormat.UpperSnakeCase)
```

The decoder — which only ever asks for camelCase field names — now resolves against an upper-snake namespace:

```scala
canonical.get("databaseUrl")
// res17: Maybe[SourceValue[String]] = SourceValue(
//   value = "postgres://localhost",
//   provenance = Resolved(
//     sourceId = "env-style",
//     key = "DATABASE_URL",
//     rawValue = "postgres://localhost"
//   )
// )
```

`ConfigSource#keyMapper` is the general form, taking an explicit `KeyMapper` as well as the target format, for sources whose naming convention the default mapper does not cover. Both operations compose with `ConfigSource#orElse` and `ConfigSource#prefix`.

:::note[EnvSource and camelCase fields]
`EnvSource` already replaces dots with underscores and upper-cases the result, so dotted keys of lowercase field names need nothing more: `db.host` reads `DB_HOST`. It does not split camelCase, though — a `databaseUrl` field is looked up as `DATABASEURL`. For case classes with camelCase fields, wrap the environment as `EnvSource.keyFormat(KeyFormat.UpperSnakeCase)`, which reads `DATABASE_URL` for `databaseUrl` and `DB_MAX_POOL_SIZE` for `db.maxPoolSize`. `ConfigSource#keyFormat` works the same way for sources that hold environment-style keys without being the environment — a `Map` scraped from an env file, for instance.
:::

## Provenance

Every value a source returns is wrapped with a record of its origin. That record survives composition, prefixing, and key mapping, which is what makes "where did this value come from?" answerable after the fact rather than only at the point of lookup.

### SourceValue and Provenance

`SourceValue` is the pair, and `Provenance` is the origin:

```scala
final case class SourceValue[A](value: A, provenance: Provenance)

sealed trait Provenance {
  def sourceId: String
}

object Provenance {
  final case class Resolved(sourceId: String, key: String, rawValue: Maybe[String]) extends Provenance
  case object Default extends Provenance
}
```

`Provenance.Resolved` names the source that answered, the source-facing key it answered under, and the raw string. The key matters when mapping is involved: a lookup of `databaseUrl` against an upper-snake source records `DATABASE_URL`, telling you the actual variable to change.

`Provenance.Default` marks a value that came from a schema default rather than any source. Its `sourceId` is the constant `"schema-default"`.

### ProvenanceMap

`ProvenanceMap[A]` pairs a decoded value with the source it was decoded from, which allows per-key queries after the load has already succeeded:

```scala

case class Db(host: String, port: Int)

object Db {
  implicit val schema: Schema[Db] = Schema.derived[Db]
}

val source   = ConfigSource.fromMap(Map("host" -> "localhost", "port" -> "5432"), "startup")
val loaded   = Config.loadWithProvenance[Db](source).toOption.get
```

The decoded value is available directly, so a `ProvenanceMap` can be passed around in place of the raw config:

```scala
loaded.value
// res19: Db = Db(host = "localhost", port = 5432)
```

`ProvenanceMap#provenanceOf` looks up a single dotted path and returns absent for keys the source does not have:

```scala
loaded.provenanceOf("port")
// res20: Maybe[Provenance] = Resolved(
//   sourceId = "startup",
//   key = "port",
//   rawValue = "5432"
// )
loaded.provenanceOf("nonexistent")
// res21: Maybe[Provenance] = zio.blocks.maybe.Absent$@b322d7c
```

### Dumping Configuration

`ProvenanceMap#dump` renders every key visible under a prefix as a box-drawn table of key, value, and source. It is meant for a single startup log line that makes a misconfigured deployment obvious:

```scala
println(loaded.dump())
// ┌──────┬───────────┬─────────┐
// │ Key  │ Value     │ Source  │
// ├──────┼───────────┼─────────┤
// │ host │ localhost │ startup │
// │ port │ 5432      │ startup │
// └──────┴───────────┴─────────┘
```

Values are redacted when the key name looks sensitive. The check lowercases the path, normalizes hyphens to underscores, and looks for any of `secret`, `password`, `passwd`, `token`, `apikey`, `api_key`, `accesskey`, `access_key`, `privatekey`, `private_key`, `credential`, or `credentials` as a substring:

```scala

val withSecret = ConfigSource.fromMap(
  Map("db.host" -> "localhost", "db.password" -> "hunter2"),
  "startup"
)
```

The key is still listed — you can see that it was set — but the value is replaced:

```scala
println(ProvenanceMap((), withSecret).dump())
// ┌─────────────┬───────────┬─────────┐
// │ Key         │ Value     │ Source  │
// ├─────────────┼───────────┼─────────┤
// │ db.host     │ localhost │ startup │
// │ db.password │ <secret>  │ startup │
// └─────────────┴───────────┴─────────┘
```

:::warning[Redaction is name-based only]
`ProvenanceMap#dump` redacts by key name, not by type. A secret stored under a key like `db.credentials_blob` is redacted; the same secret under `db.blob` is printed in full, even when the field it decodes into is a `Secret`. `ProvenanceMap#provenanceOf` returns the raw value inside its `Provenance` whatever the key. In 0.0.56 this is a known bug: give secret keys a name that contains one of the words above, and don't log their provenance.
:::

## Secrets

`Secret` is a wrapper whose `toString` is always `<secret>`, so a value inside one cannot leak through string interpolation, logging, or a case class `toString`:

```scala

val token = Secret("s3cr3t-token")
```

Rendering the wrapper reveals nothing, even inside a larger string:

```scala
token.toString
// res26: String = "<secret>"
s"token = $token"
// res27: String = "token = <secret>"
```

`Secret#equals` and `Secret#hashCode` compare the underlying value, so secrets remain usable as map keys and in equality checks:

```scala
token == Secret("s3cr3t-token")
// res28: Boolean = true
```

Reading the value back requires the explicit `Secret.unwrap`, which makes every access point visible in a code search:

```scala
Secret.unwrap(token)
// res29: String = "s3cr3t-token"
```

### Schema Decoding

`Secret` carries its own `Schema[Secret]`, derived by transforming `Schema[String]` rather than reflecting over a case class. That means a `Secret` field decodes like any other primitive wherever `Schema.derived` is used — no per-field annotation or special-casing at the call site:

```scala

case class Db(host: String, password: Secret)

object Db {
  implicit val schema: Schema[Db] = Schema.derived[Db]
}
```

The raw string is read from the source exactly like any other field, then wrapped:

```scala
Config.load[Db](ConfigSource.fromMap(Map("host" -> "localhost", "password" -> "hunter2"), "defaults"))
// res31: Either[::[ConfigError], Db] = Right(
//   Db(host = "localhost", password = <secret>)
// )
```

The decoded `password` is a `Secret`, so the value never appears in a `toString` of the loaded record even though the source held it as a plain string:

```scala
Config.load[Db](ConfigSource.fromMap(Map("host" -> "localhost", "password" -> "hunter2"), "defaults")).map(_.password)
// res32: Either[::[ConfigError], Secret] = Right(<secret>)
```

### Displayable

`Displayable[A]` is the type class behind rendered flag values, with instances for `String`, `Int`, `Long`, `Double`, and `Boolean`, plus a low-priority fallback that calls `toString`. The module provides `Displayable[Secret]` as an implicit in the `zio.blocks.config` package object, so a `StaticFlag[Secret]` renders as `<secret>` in `Flag.dump` output without any per-flag configuration.

To control how a custom type appears in flag dumps, provide your own instance with `Displayable.instance`:

```scala

final case class Port(value: Int)

implicit val portDisplayable: Displayable[Port] = Displayable.instance(p => s"port ${p.value}")
```

## Writing a Custom Source

Implementing `ConfigSource` requires three members: an id, a single-key lookup, and prefix enumeration. Constructing `Provenance.Resolved` yourself is what makes the new source participate in provenance tracking:

```scala

final class UppercaseSource(entries: Map[String, String]) extends ConfigSource {
  val sourceId: String = "uppercase"

  def get(key: String): Maybe[SourceValue[String]] =
    Maybe.fromOption(
      entries.get(key).map(v => SourceValue(v.toUpperCase, Provenance.Resolved(sourceId, key, Maybe.present(v))))
    )

  def all(prefix: String): Map[String, SourceValue[String]] = {
    val dotted = if (prefix.isEmpty) "" else s"$prefix."
    entries.collect {
      case (k, v) if prefix.isEmpty || k == prefix || k.startsWith(dotted) =>
        k -> SourceValue(v.toUpperCase, Provenance.Resolved(sourceId, k, Maybe.present(v)))
    }
  }
}
```

Keep `rawValue` as the original string even when `value` is transformed. Provenance is meant to explain what the source held, not what the source returned.

## Integration Points

`ConfigSource` is the input to every other part of the module: `ConfigDecoder#decode` takes one, `Config.wire` injects one, and `FlagSource.Registry` accepts one because `ConfigSource` extends `FlagSource`. It depends only on `Maybe` from `zio-blocks-maybe`.

See [Config Decoder](./config-decoder.md) for how a source becomes a typed value, [Errors](./errors.md) for what a failed lookup produces, and [File Formats](./formats.md) for the YAML, JSON, and HOCON constructors.

See also the [Build a Typed Configuration Loader](../../guides/typed-config-loader.md) tutorial, which composes sources into a layered chain and prints where each value came from.

---

## Configuration Errors

`ConfigError` is the failure type for every configuration operation. It is a sealed hierarchy divided into four category traits, so code can match on the kind of problem — a missing key, an unparseable value, a source-level failure, a derivation failure — without enumerating every constructor. `ConfigLoadException` wraps a set of errors when a load is allowed to throw. The hierarchy:

```scala
sealed trait ConfigError extends NoStackTrace {
  def message: String
  override def getMessage: String = message
}

sealed trait ConfigParseError      extends ConfigError
sealed trait ConfigValidationError extends ConfigError
sealed trait ConfigSourceError     extends ConfigError
sealed trait ConfigDerivationError extends ConfigError
```

## Motivation

Configuration failures need three properties that a plain exception does not provide.

They need to be **plural**. A config file with four mistakes should report four mistakes, so the person fixing it makes one pass rather than four. That rules out throwing at the first problem, which is why every decoding operation returns `Either[::[ConfigError], A]` — a non-empty list, so a `Left` is guaranteed to say something.

They need to be **cheap**. `ConfigError` extends `NoStackTrace`, so constructing one does not capture a stack trace. Config errors are diagnosed by reading the message, not by reading a trace through the deriver's internals, and a decode that fails on twenty fields should not pay for twenty stack captures.

They need to be **classifiable**. Whether a value was missing, malformed, or rejected by the source changes what the operator should do about it. The four category traits exist so that distinction survives into your error handling.

## Category Traits

Every concrete error extends exactly one category, except `ConfigError.Composite`, which extends `ConfigError` directly because it aggregates errors that may span categories:

| Category                 | Meaning                                        | Constructors                                            |
| ------------------------ | ---------------------------------------------- | ------------------------------------------------------- |
| `ConfigParseError`       | Value present but not convertible               | `InvalidValue`, `ParseError`                             |
| `ConfigValidationError`  | Value parsed but semantically rejected          | *(none built in)*                                        |
| `ConfigSourceError`      | Problem with the source itself                  | `MissingKey`, `DuplicateKey`, `Unauthorized`             |
| `ConfigDerivationError`  | Schema-driven decoding could not proceed        | `UnknownDiscriminator`, `MissingDiscriminatorKey`         |
| *(no category)*          | Aggregate of several errors                     | `Composite`                                              |

:::note[`ConfigValidationError` has no constructors]
The category is declared and documented, but no error in the module currently extends it, and because the trait is sealed it cannot be extended from outside. Matching on it compiles and never matches. Validation failures raised by wrapper types surface as `ConfigError.InvalidValue` instead.
:::

## Error Constructors

Each constructor carries the fields needed to act on the failure without re-reading the config, and renders them into a single-line `message`.

### ConfigError.MissingKey

A required key was not found. Produced by primitive decoding when the key is absent and the field has neither a default nor an `Option` type:

```scala

case class Db(host: String, port: Int)

object Db {
  implicit val schema: Schema[Db] = Schema.derived[Db]
}
```

The error names the dotted path and the source that was searched, which for a composed source is the pipe-joined id of every source in the chain:

```scala
Config.load[Db](ConfigSource.fromMap(Map("port" -> "5432"), "defaults"))
// res0: Either[::[ConfigError], Db] = Left(
//   List(MissingKey(path = "host", source = "defaults"))
// )
```

`MissingKey` is the error that optional fields and defaults absorb. A field of type `Option[A]` becomes `None`, and a field with a declared default takes that default, but only when *every* error from that field is a `MissingKey` — a mix of missing and invalid still fails.

### ConfigError.InvalidValue

A key was present but its value could not be converted to the expected type. Carries the path, the offending string, a human-readable expected type, the source id, and an optional underlying exception:

```scala
Config.load[Db](ConfigSource.fromMap(Map("host" -> "localhost", "port" -> "eight"), "defaults"))
// res1: Either[::[ConfigError], Db] = Left(
//   List(
//     InvalidValue(
//       path = "port",
//       value = "eight",
//       expectedType = "Int",
//       source = "defaults",
//       cause = Some(java.lang.NumberFormatException: For input string: "eight")
//     )
//   )
// )
```

The `expectedType` string is derived from the schema's primitive type name. When a `cause` is present, its message is appended to the rendered message, which is where the underlying `NumberFormatException` text comes from.

`InvalidValue` is also what a failing wrapper type produces: if a newtype's constructor throws while wrapping a successfully-decoded inner value, the thrown exception becomes the `cause`.

`InvalidValue#message` redacts the offending value — printing `<secret>` in its place and dropping the `cause` text — when the key's path looks like it names a secret. It uses the same name-based check as `ProvenanceMap#dump`, so the same limits apply: see [Redaction is name-based only](./config-source.md#dumping-configuration). A key named `password` that fails to parse reports the type mismatch without ever printing what was typed:

```scala

case class Db(password: Int)

object Db {
  implicit val schema: Schema[Db] = Schema.derived[Db]
}
```

```scala
Config.load[Db](ConfigSource.fromMap(Map("password" -> "not-a-number"), "defaults")).left.map(_.head.message)
// res3: Either[String, Db] = Left(
//   "Invalid value '<secret>' for key 'password' (expected Int) in source 'defaults'"
// )
```

Redaction applies to the rendered message only, which is also what `toString` prints. The `value` field of the error still holds the raw string, so read it only where printing it is safe:

```scala
Config.load[Db](ConfigSource.fromMap(Map("password" -> "not-a-number"), "defaults")).left.map(_.head match {
  case e: ConfigError.InvalidValue => e.value
  case other                       => other.message
})
// res4: Either[String, Db] = Left("not-a-number")
```

### ConfigError.ParseError

The raw value could not be decoded at the format level, as distinct from a type mismatch on a well-formed value. Carries the path, source, expected format, and optional cause. The YAML, JSON, and HOCON adapters produce it when the document itself is malformed, and the JVM file loader produces it for a missing file or a rejected include path.

The difference from `InvalidValue` is which layer failed. `InvalidValue` means "this string is not an `Int`"; `ParseError` means "this document is not HOCON", so there is no string to quote.

### ConfigError.DuplicateKey

The same key appears in multiple conflicting sources. Carries the path and the ids of every source that defined it. Nothing in the module produces it — `ConfigSource#orElse` resolves conflicts by precedence rather than reporting them — so it exists for custom sources that want to treat ambiguity as an error rather than resolving it silently.

### ConfigError.Unauthorized

Access to a key was denied by the source. Carries the path and source id. Like `DuplicateKey`, no built-in source produces it; it is the error a custom source backed by a secrets manager should return when a lookup is rejected rather than merely absent, so that "you may not read this" is distinguishable from "this does not exist".

### ConfigError.MissingDiscriminatorKey

A sealed trait was decoded but the key naming which case to use was absent. Carries the record's path and the discriminator key that was expected:

```scala

sealed trait Backend

object Backend {
  case class Postgres(host: String, port: Int) extends Backend
  case class Sqlite(path: String)              extends Backend

  implicit val schema: Schema[Backend] = Schema.derived[Backend]
}
```

The message states both, so the fix is unambiguous:

```scala
Config.load[Backend](ConfigSource.fromMap(Map("host" -> "localhost"), "conf"))
// res6: Either[::[ConfigError], Backend] = Left(
//   List(MissingDiscriminatorKey(path = "", key = "type"))
// )
```

### ConfigError.UnknownDiscriminator

The discriminator key was present but named a case that does not exist. Carries the full key path, the value found, and every accepted value in sorted order:

```scala
Config.load[Backend](ConfigSource.fromMap(Map("type" -> "MySql"), "conf"))
// res7: Either[::[ConfigError], Backend] = Left(
//   List(
//     UnknownDiscriminator(
//       path = "type",
//       found = "MySql",
//       expected = List("Postgres", "Sqlite")
//     )
//   )
// )
```

Listing the expected values in the message is deliberate: a typo in a discriminator is one of the few config mistakes where the error can name the correct answer.

### ConfigError.Composite

Two or more errors accumulated while decoding one record. Carries them as a `::[ConfigError]`, and renders as the child messages joined with newlines:

```scala

case class Db(host: String, port: Int)

object Db {
  implicit val schema: Schema[Db] = Schema.derived[Db]
}

val failed = Config.load[Db](ConfigSource.fromMap(Map.empty[String, String], "empty"))
```

A record with a single failing field returns that error directly, so `Composite` appears only when there is genuinely more than one:

```scala
failed.left.map(_.head.getClass.getSimpleName)
// res9: Either[String, Db] = Left("Composite")
```

Composites nest. Wrapping happens once per record, so a nested config whose inner and outer records both have multiple failures produces a `Composite` containing a `Composite`. Flatten before reporting if you want one list:

```scala

def flatten(error: ConfigError): List[ConfigError] = error match {
  case ConfigError.Composite(errors) => errors.toList.flatMap(flatten)
  case other                         => List(other)
}
```

## Handling Errors

How you consume a `Left` depends on whether the response varies by kind of failure. Matching on categories keeps the handler stable as new constructors are added; matching on constructors gets you the fields.

### Matching by Category

Categories answer "whose fault is this?", which is usually what determines the response — a missing key is an operator problem, a derivation error is a schema problem:

```scala

def describe(error: ConfigError): String = error match {
  case _: ConfigSourceError     => "configuration is incomplete or unreadable"
  case _: ConfigParseError      => "a configured value has the wrong format"
  case _: ConfigDerivationError => "the config shape does not match the schema"
  case _                        => "multiple problems"
}
```

### Matching by Constructor

Matching the concrete type gives access to the fields, which is what you need to build a targeted message or a machine-readable report:

```scala

def remedy(error: ConfigError): String = error match {
  case ConfigError.MissingKey(path, source) =>
    s"set $path in $source"
  case ConfigError.InvalidValue(path, value, expected, _, _) =>
    s"$path is '$value' but must be a $expected"
  case ConfigError.UnknownDiscriminator(path, found, expected) =>
    s"$path is '$found'; use one of ${expected.mkString(", ")}"
  case other =>
    other.message
}
```

Because `ConfigError` extends `NoStackTrace`, and therefore `Throwable`, an individual error can also be thrown or wrapped directly when integrating with code that expects exceptions.

## ConfigLoadException

`Config.loadOrThrow` throws `ConfigLoadException` rather than returning a `Left`. The exception carries both a formatted report for humans and the original error list for programs:

```scala
final class ConfigLoadException(val report: String, val errors: ::[ConfigError])
  extends RuntimeException(report)
```

`report` is a multi-line summary: a count line followed by one indented bullet per top-level error. Catching the exception and reading `errors` recovers the structured form, so throwing does not lose information:

```scala

case class Db(host: String, port: Int)

object Db {
  implicit val schema: Schema[Db] = Schema.derived[Db]
}

try Config.loadOrThrow[Db](ConfigSource.fromMap(Map.empty[String, String], "empty"))
catch {
  case e: ConfigLoadException =>
    println(e.report)
    e.errors.toList.foreach(err => println(err.message))
}
```

Use `Config.loadOrThrow` at the top of `main`, where an unrecoverable config problem should stop startup with a readable message. Prefer `Config.load` anywhere the failure is recoverable — a reload, a per-request decode, or a validation pass that reports rather than aborts.

:::warning[`Config.wire` throws]
`Config.wire` decodes with `Config.loadOrThrow`, so a bad config surfaces as a `ConfigLoadException` while the dependency graph is being allocated rather than as a typed failure. Validate with `Config.load` first if you need to report config problems before touching the graph.
:::

## Flag Exceptions

Flag resolution has a separate hierarchy, `FlagException`, because flags fail at class-load time and cannot return an `Either` from an object initializer. `ConfigError` still appears inside those exceptions as the underlying cause — a flag whose value will not parse carries the same `ConfigError.InvalidValue` a config field would. See [Flags](./flags.md).

## Integration Points

`ConfigError` is produced by `ConfigDecoder`, by every `ConfigSource` constructor that can fail, and by `Rollout` when an expression will not parse. It is consumed by `Config`, which either returns it or formats it into a `ConfigLoadException`, and by `Flag.ReloadResult.Failed`, which carries it out of a failed dynamic-flag reload.

See [Config Decoder](./config-decoder.md) for which rule produces which error, and [Rollout](./rollout.md) for expression-level failures.

---

## Flags

A flag is a named scalar whose value comes from outside the program. `StaticFlag[A]` resolves once at class-load time and never changes; `DynamicFlag[A]` holds a rollout expression that can be updated while the process runs. Both read through `FlagSource`, parse through `Flag.Reader`, and register themselves in a global registry that `Flag.dump` can print. Supporting types: `Flag.Source`, `FlagException`. The three declarations you write against:

```scala
trait FlagSource {
  def sourceId: String
  def get(name: String): Maybe[SourceValue[String]]
  final def orElse(fallback: FlagSource): FlagSource
}

abstract class StaticFlag[A](default: A)(implicit reader: Flag.Reader[A], displayable: Displayable[A])

abstract class DynamicFlag[A](default: A, defaultExpression: String)(implicit reader: Flag.Reader[A])
```

## Motivation

Configuration and flags answer different questions. Configuration describes a deployment: which database, which port, which region. Flags describe a decision: is this feature on, for whom, and at what percentage. The two have different shapes — configuration is a typed record loaded once, a flag is a single value read at the point of use — and different failure modes.

Flags in this module are Scala objects. That choice buys three things. The name is derived from the object's fully-qualified name, so it cannot drift from the code that reads it. The value is a `val`, so reading it is a field access with no lookup cost. And references are compile-checked: a deleted flag is a compile error at every use site, not a silently-missing key.

The cost is that resolution happens during class initialization, which makes ordering matter. A `StaticFlag` touched before its `FlagSource` is registered resolves from the environment or its default and stays that way.

## FlagSource

`FlagSource` is the scalar half of `ConfigSource`: a name-to-string lookup with provenance, without prefix enumeration. Because `ConfigSource extends FlagSource`, any config source is also a flag source.

### Creating a Source

`FlagSource.fromMap` builds an in-memory source, which is what tests and local overrides use:

```scala

val source = FlagSource.fromMap(Map("myapp.poolSize" -> "32"), "local")
```

Lookups return the value with a `Provenance.Resolved` naming the source and the name it answered under:

```scala
source.get("myapp.poolSize")
// res0: Maybe[SourceValue[String]] = SourceValue(
//   value = "32",
//   provenance = Resolved(
//     sourceId = "local",
//     key = "myapp.poolSize",
//     rawValue = "32"
//   )
// )
```

`FlagSource#orElse` layers sources with the receiver taking priority, exactly as on `ConfigSource`.

### The Registry

Flags do not take a source as a constructor argument. They consult `FlagSource.Registry`, a thread-safe global that flags read during initialization:

| Method                          | Effect                                                                 |
| ------------------------------- | ---------------------------------------------------------------------- |
| `Registry.register(source)`     | Adds a source, keyed by `sourceId`; re-registering an id replaces it in place. |
| `Registry.unregister(sourceId)` | Removes a source by id.                                                 |
| `Registry.get(sourceId)`        | Looks up a registered source.                                           |
| `Registry.all`                  | Every source in registration order.                                     |
| `Registry.resolve(name)`        | First source that answers the name wins.                                |
| `Registry.clear()`              | Removes everything; intended for tests.                                 |

Registration order is resolution order, so the first source registered has the highest priority:

```scala

FlagSource.Registry.register(FlagSource.fromMap(Map("myapp.poolSize" -> "64"), "remote"))
FlagSource.Registry.register(FlagSource.fromMap(Map("myapp.poolSize" -> "32"), "fallback"))

FlagSource.Registry.resolve("myapp.poolSize")
```

Re-registering an existing `sourceId` swaps the source without changing its position, so refreshing a source's contents does not reshuffle priorities.

:::warning[Register before first touch]
A `StaticFlag` resolves during its object's initialization, which happens the first time anything references it. Sources registered after that point are never consulted for that flag. Register every source at the very top of `main`, before touching any flag.
:::

## StaticFlag

`StaticFlag[A]` is for values that are fixed for the lifetime of the process: pool sizes, buffer limits, endpoints that cannot change without a restart.

### Defining a Flag

Extend `StaticFlag[A]` from a Scala `object`, passing the default:

```scala

object poolSize extends StaticFlag[Int](10)

val size: Int = poolSize()
```

The flag's `name` is the object's fully-qualified name with `$` separators rewritten to dots, so an object `Limits.poolSize` in package `com.example` is named `com.example.Limits.poolSize`. Nothing declares that name — it follows from where the object lives, which means moving the object renames the flag.

### Resolution Order

Resolution runs once, in a fixed order, and stops at the first source that produces a value:

```
1. FlagSource.Registry.resolve(name)     ← registration order across all sources
2. System property with the flag's name  ← -Dcom.example.Limits.poolSize=32
3. Environment variable                  ← COM_EXAMPLE_LIMITS_POOLSIZE=32
4. The constructor default
```

The environment variable name is the flag name with dots replaced by underscores, uppercased. That translation is fixed and not configurable.

### Members

Each resolved flag exposes what it resolved to and where from:

| Member         | Type          | Meaning                                                   |
| -------------- | ------------- | --------------------------------------------------------- |
| `name`         | `String`      | Derived fully-qualified flag name.                         |
| `value`        | `A`           | The resolved value.                                        |
| `apply()`      | `A`           | Same as `value`; reads as a call at use sites.              |
| `source`       | `Flag.Source` | Which of the four resolution steps answered.                |
| `provenance`   | `Provenance`  | Source id, key, and raw string.                             |
| `displayValue` | `String`      | Rendered through `Displayable[A]`, used by `Flag.dump`.      |

### Failure at Initialization

An object initializer cannot return an `Either`, so a flag that cannot resolve throws. A value that fails to parse throws `ExceptionInInitializerError` wrapping `FlagException.FlagValueParseException`, and a flag whose defining class is not a Scala object throws `FlagException.FlagNameException`.

Failing at class load is the point: a malformed flag stops the process at startup rather than producing a plausible-looking wrong value that surfaces later.

:::note[Secrets in flags]
`Flag.Reader[Secret]` accepts any string, and the module provides `Displayable[Secret]` in the `zio.blocks.config` package object, so a `StaticFlag[Secret]` renders as `<secret>` in dumps without extra configuration. See [ConfigSource](./config-source.md) for `Secret` itself.
:::

## DynamicFlag

`DynamicFlag[A]` is for decisions that change without a restart: enabling a feature for a subset of users, shifting traffic to a new code path, turning off an expensive query under load.

### Defining a Flag

Extend `DynamicFlag[A]` with both a default value and a default rollout expression:

```scala

object newCheckout extends DynamicFlag[Boolean](false, "true@*/50%; false")

val enabled: Boolean = newCheckout("user-1234")
```

The initial expression resolves through the same chain as a static flag's value — registry, then system property, then environment variable, then the constructor default — but what it resolves is the *expression*, not the value. A malformed expression throws `ExceptionInInitializerError` wrapping `FlagException.FlagExpressionParseException`.

### Evaluation

`DynamicFlag#apply` takes a bucketing key and any number of path attributes, evaluates the expression, and tracks a counter for the key. `DynamicFlag#evaluate` does the same without touching counters:

```scala

object newCheckout extends DynamicFlag[Boolean](false, "true@*/prod/*/50%; false")

newCheckout("user-1234", "prod", "eu")           // counted
newCheckout.evaluate("user-1234", "prod", "eu") // not counted
```

The key determines the percentage bucket, and it is also the *first* segment of the path a selector matches against: the attributes are appended to it with slashes, so the call above matches against `user-1234/prod/eu`. A selector must have exactly as many segments as that path, which is why the expression starts with a wildcard. Evaluation details are in [Rollout](./rollout.md).

When a matched value cannot be parsed into `A`, evaluation falls back to the flag's default and increments `DynamicFlag#parseErrorCount`. A non-zero count means the expression contains values the reader rejects — a misconfiguration that would otherwise be invisible, since every call still returns something usable.

### Updating and Reloading

`DynamicFlag#update` replaces the expression from code, returning `Left` if the new expression will not parse and leaving the old one in place:

```scala

object newCheckout extends DynamicFlag[Boolean](false, "false")

newCheckout.update("true@prod/25%; false")
newCheckout.expression
```

An empty or whitespace-only expression is accepted as a no-op and returns `Right(())` without changing anything.

`DynamicFlag#reload` re-reads the expression from `FlagSource.Registry` and reports what happened as a `Flag.ReloadResult`. See [Rollout](./rollout.md) for the result variants and the update history.

## Flag.Reader

`Flag.Reader[A]` parses a raw string into `A` and supplies a type name for error messages. Instances are resolved implicitly by both flag types:

```scala
trait Reader[A] {
  def parse(flagName: String, raw: String): Either[ConfigError, A]
  def typeName: String
}
```

### Built-in Instances

| Type                | Accepted form                                              | Example  |
| ------------------- | ---------------------------------------------------------- | -------- |
| `Int`, `Long`       | Decimal integer literal                                     | `32`     |
| `Double`            | Decimal or scientific literal                               | `0.5`    |
| `Boolean`           | `true`/`false`, `1`/`0`, `yes`/`no`, `on`/`off` (any case)   | `on`     |
| `String`            | Any string, taken as-is                                      | `eu-west` |
| `Secret`            | Any string, wrapped so it cannot be printed                  | `hunter2` |
| `FiniteDuration`    | Digits followed by a unit suffix                              | `30s`    |
| `Seq[A]`            | Comma-separated values of a scalar `A`, each trimmed          | `a, b, c` |

Boolean parsing accepts the same set of spellings as config decoding:

```scala
Flag.Reader.booleanReader.parse("example", "ON")
// res6: Either[ConfigError, Boolean] = Right(true)
Flag.Reader.booleanReader.parse("example", "maybe")
// res7: Either[ConfigError, Boolean] = Left(
//   InvalidValue(
//     path = "example",
//     value = "maybe",
//     expectedType = "Boolean (true/false/1/0/yes/no/on/off)",
//     source = "flag",
//     cause = None
//   )
// )
```

Duration suffixes are `ms`/`millis`/`milliseconds`, `s`/`sec`/`secs`/`seconds`, `m`/`min`/`mins`/`minutes`, `h`/`hour`/`hours`, and `d`/`day`/`days`:

```scala
Flag.Reader.durationReader.parse("example", "500ms")
// res8: Either[ConfigError, FiniteDuration] = Right(500 milliseconds)
Flag.Reader.durationReader.parse("example", "2hours")
// res9: Either[ConfigError, FiniteDuration] = Right(2 hours)
```

The grammar is digits immediately followed by letters, so a space between the number and the unit is rejected:

```scala
Flag.Reader.durationReader.parse("example", "2 hours")
// res10: Either[ConfigError, FiniteDuration] = Left(
//   InvalidValue(
//     path = "example",
//     value = "2 hours",
//     expectedType = "Duration (e.g., 10s, 5m, 1h, 100ms, 2d)",
//     source = "flag",
//     cause = None
//   )
// )
```

:::warning[Flag durations are not config durations]
`Flag.Reader[FiniteDuration]` parses `30s`. Config decoding parses `java.time.Duration` with `Duration.parse`, which requires ISO-8601 (`PT30S`). Neither accepts the other's format — see [Config Decoder](./config-decoder.md).
:::

### Scalar and Sequence Readers

`Flag.Reader.Scalar[A]` marks a reader whose values contain no commas. `Flag.Reader.seqReader` requires that marker, because it splits on commas — a non-scalar element type would make the split ambiguous:

```scala
Flag.Reader.seqReader[Int].parse("example", "1, 2, 3")
// res11: Either[ConfigError, Seq[Int]] = Right(ArraySeq(1, 2, 3))
```

An empty string parses to an empty sequence rather than failing. If any element fails, its error stands for the whole list.

### Custom Readers

`Flag.Reader.apply` builds a reader from a parse function and a type name, and `Flag.Reader.scalar` does the same for comma-free types so that `Seq` of them works:

```scala

final case class Region(code: String)

implicit val regionReader: Flag.Reader.Scalar[Region] =
  Flag.Reader.scalar(
    (flagName, raw) =>
      if (raw.matches("[a-z]{2}-[a-z]+-\\d")) Right(Region(raw))
      else Left(ConfigError.InvalidValue(flagName, raw, "Region (e.g. eu-west-1)", "flag")),
    "Region"
  )
```

Return `ConfigError.InvalidValue` from a failing parse so the resulting message matches every other config failure.

## Flag.Source

`Flag.Source` records which resolution step answers a static flag:

| Variant                            | Meaning                                        |
| ---------------------------------- | ---------------------------------------------- |
| `Flag.Source.FlagSourceValue(id)`  | A registered `FlagSource` answered; `id` names it. |
| `Flag.Source.SystemProperty`       | A JVM system property answered.                 |
| `Flag.Source.EnvironmentVariable`  | An environment variable answered.               |
| `Flag.Source.Default`              | Nothing answered; the constructor default applies. |

Checking for `Flag.Source.Default` at startup is how you detect a flag you meant to configure and did not.

## Diagnostics

Two helpers on `Flag` exist for the two questions that come up when a flag has the wrong value: what did everything resolve to, and did I misspell something?

### Flag.dump

`Flag.dump()` renders every registered flag — static and dynamic — as a table of name, type, value, and source. Static flags show their `displayValue` and provenance source id; dynamic flags show their current expression:

```scala

println(Flag.dump())
```

The registry is populated by construction, so a flag appears only after something has touched it. Print the dump after startup has forced every flag, not before.

### Flag.nearMissWarnings

`Flag.nearMissWarnings` looks for names that resemble a flag but do not match it, which catches the common failures: a case mismatch in a system property, a case mismatch in an environment variable, or a registered source holding a nearly-right key:

```scala

Flag.nearMissWarnings("com.example.Limits.poolSize").foreach(println)
```

Candidates checked include the name lowercased and uppercased, dots swapped for underscores, and underscores swapped for dots. A warning means a source contains something similar — it does not mean the flag is wrong, only that a typo is plausible.

## FlagException

Flag failures are exceptions rather than values because they occur inside object initializers. `FlagException` extends `NoStackTrace`, so constructing one is cheap:

| Exception                                | Thrown when                                                                 |
| ---------------------------------------- | --------------------------------------------------------------------------- |
| `FlagValueParseException`                | A static flag's raw value fails its `Flag.Reader`.                           |
| `FlagExpressionParseException`            | A dynamic flag's initial rollout expression will not parse.                  |
| `FlagNameException`                       | The defining class is not a Scala object, or is a lambda or anonymous class.  |
| `FlagDuplicateNameException`              | A second flag registers under an existing name.                              |

The library creates the `ExceptionInInitializerError` itself, with the `FlagValueParseException` or `FlagExpressionParseException` as its cause, so catch that error and inspect its cause:

```scala

try {
  // Touching a flag object forces its initialization.
  Class.forName("com.example.Limits$poolSize$")
} catch {
  case e: ExceptionInInitializerError =>
    e.getCause match {
      case f: FlagException => println(f.getMessage)
      case other           => throw other
    }
}
```

`FlagDuplicateNameException` means two objects derived the same name, which happens when two flags share a fully-qualified name after `$`-to-dot rewriting. Renaming either object resolves it.

`FlagValueParseException#getMessage` redacts the raw value the same way [`ConfigError.InvalidValue`](./errors.md) does — `<secret>` in its place, and the underlying cause dropped — when the flag's name looks like it names a secret. A flag named `Db.password` that fails to parse does not print the value it was given in its message. The check is by name only, as described in [Redaction is name-based only](./config-source.md#dumping-configuration), so give flags that hold secrets a name that contains one of the listed words.

## Integration Points

Flags depend on `FlagSource` — and therefore accept any `ConfigSource` — on `Flag.Reader` for parsing, on `Displayable` for rendering, and on `ConfigError` for parse failures. `DynamicFlag` additionally depends on `Rollout` for expression evaluation.

See [Rollout](./rollout.md) for the expression language and reload lifecycle, [ConfigSource](./config-source.md) for sources and `Secret`, and [Errors](./errors.md) for the shared error model.

See also the [How to Roll Out a Feature Gradually with Flags](../../guides/roll-out-feature-gradually.md) guide, which ships a feature to a percentage of readers with a `DynamicFlag` and widens it at runtime.

---

## File Formats

Three optional modules add file-format support: `config-yaml`, `config-json`, and `config-hocon`. Each contributes one constructor to the `ConfigSource` companion, parses its format into an in-memory tree, and flattens that tree into the dot-separated namespace every `ConfigSource` exposes. Supporting types: `YamlConfigSource`, `HoconValue`, `HoconError`. One constructor per format:

```scala
// with `import zio.blocks.config.yaml._`
ConfigSource.fromYaml(yaml: String, sourceId: String = "yaml:string"): Either[ConfigError, ConfigSource]

// with `import zio.blocks.config.json._`
ConfigSource.fromJson(json: String, sourceId: String = "json:string"): Either[ConfigError, ConfigSource]

// with `import zio.blocks.config.hocon._`
ConfigSource.fromHocon(hocon: String, sourceId: String = "hocon:string"): Either[ConfigError, ConfigSource]
```

## Motivation

A decoder that understood YAML, JSON, and HOCON natively would be three decoders with three sets of bugs. Flattening avoids that: each format is parsed by code that knows only that format, and the result is the same flat map of dotted keys the core module already handles.

The consequence is that everything above the source layer is format-blind. `ConfigSource#orElse` layers a YAML file under environment variables. `ConfigDecoder` reads a JSON document with the same rules it uses for a `Map`. Provenance reports `yaml:string` where it would otherwise report `env`. Nothing in the decoding path branches on format.

Keeping the adapters in separate artifacts means a service that only reads environment variables does not pull a YAML parser, and adding HOCON support is a build change rather than a code change.

## Common Flattening Rules

All three adapters agree on how a tree becomes flat keys. Learning the rules once tells you what key any document produces:

| Tree shape                | Flattened form                            | Example                                     |
| ------------------------- | ----------------------------------------- | ------------------------------------------- |
| Nested object or mapping  | Keys joined with `.`                       | `db: { host: x }` → `db.host` = `x`          |
| Array or sequence         | Zero-based numeric segment                 | `hosts: [a, b]` → `hosts.0` = `a`, `hosts.1` = `b` |
| Scalar                    | The value as a string                      | `port: 8080` → `port` = `"8080"`             |
| Null                      | **Omitted entirely**                       | `tag: null` → no `tag` key                   |

Two of these have consequences worth stating plainly.

Indexed sequences are exactly the shape `ConfigDecoder` probes for, so a YAML list decodes into a `List` without any extra configuration. See [Config Decoder](./config-decoder.md).

Null omission means an explicit null is indistinguishable from an absent key. A field of type `Option[A]` becomes `None`, a field with a default takes its default, and a required field reports `ConfigError.MissingKey`. There is no way to express "present but null" through a flattened source.

:::warning[Nulls disappear]
Writing `tag: null` to unset a value from a lower-priority layer does not work. `ConfigSource#orElse` sees no key at all, so the fallback still wins. Override with a real value instead.
:::

## YAML

`config-yaml` reads YAML through the `zio-blocks-schema-yaml` reader:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-config-yaml" % "0.0.56"
```

Importing the adapter package makes the constructor available on the `ConfigSource` companion:

```scala

val yamlText =
  """|db:
     |  host: localhost
     |  port: 5432
     |hosts:
     |  - alpha
     |  - beta
     |""".stripMargin
```

Nested mappings become dotted keys and the sequence becomes indexed keys, all as strings:

```scala
ConfigSource.fromYaml(yamlText, "app.yaml")
// res0: Either[ConfigError, ConfigSource] = Right(
//   MapSource(
//     map = Map(
//       "hosts.0" -> "alpha",
//       "hosts.1" -> "beta",
//       "db.port" -> "5432",
//       "db.host" -> "localhost"
//     ),
//     sourceId = "app.yaml"
//   )
// )
```

The resulting source behaves like any other, so decoding a case class needs nothing format-specific:

```scala

case class Db(host: String, port: Int)

object Db {
  implicit val schema: Schema[Db] = Schema.derived[Db]
}
```

Re-rooting with `ConfigSource#prefix` reads the `db` section:

```scala
ConfigSource.fromYaml(yamlText, "app.yaml").map(src => Config.load[Db](src.prefix("db")))
// res1: Either[ConfigError, Either[::[ConfigError], Db]] = Right(
//   Right(Db(host = "localhost", port = 5432))
// )
```

A document that will not parse yields `ConfigError.ParseError` with an empty `path` and `valid YAML` as the expected type, so its message reads `Parse error for key '' (expected valid YAML) in source 'app.yaml'` for a source named `app.yaml`. The error carries no excerpt of the document and no cause, so a secret in a malformed file never reaches a log. `YamlConfigSource.fromString` is the same operation without the syntax import, for code that prefers an explicit call.

Non-scalar mapping keys are skipped rather than rejected, since a dotted key namespace cannot represent them.

## JSON

`config-json` reads JSON through the `zio-blocks-schema-json` parser:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-config-json" % "0.0.56"
```

The import and the constructor mirror the YAML adapter:

```scala

val jsonText =
  """|{
     |  "db": { "host": "localhost", "port": 5432 },
     |  "hosts": ["alpha", "beta"],
     |  "debug": true
     |}
     |""".stripMargin
```

Numbers and booleans are rendered with their `toString`, so every value arrives as a string for the decoder to parse:

```scala
ConfigSource.fromJson(jsonText, "app.json")
// res3: Either[ConfigError, ConfigSource] = Right(
//   MapSource(
//     map = Map(
//       "db.port" -> "5432",
//       "hosts.0" -> "alpha",
//       "hosts.1" -> "beta",
//       "debug" -> "true",
//       "db.host" -> "localhost"
//     ),
//     sourceId = "app.json"
//   )
// )
```

A malformed document yields `ConfigError.ParseError` with `valid JSON` as the expected type and, as with YAML, no excerpt of the input and no cause.

## HOCON

`config-hocon` has its own parser with no external dependency, and it is the only adapter that supports substitutions and includes:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-config-hocon" % "0.0.56"
```

Moving an existing `application.conf` from Typesafe Config, PureConfig or zio-config? [Coming from PureConfig, Typesafe Config or zio-config](../../guides/config-library-migration.md) lists the HOCON features that behave differently and what to change.

:::warning[Known gaps in 0.0.56]
The parser differs from Typesafe Config in these ways, each a known bug in 0.0.56:

- **Environment fallback.** `${VAR}` and `${?VAR}` look only inside the document. `${?VAR}` silently resolves to nothing, and `${VAR}` is a parse error even when the variable is set. Read secrets through an `EnvSource` layered beside the file instead.
- **Array concatenation.** `tags = [scala] [zio]` loads as the empty string `""`, with no error. Write one array per key.
- **Durations and sizes.** Durations decode through Java's ISO-8601 syntax, so `PT10S` works and `10s` is rejected. Size units such as `512M` are not parsed.
- **Includes.** See [Includes](#includes): `fromHocon` drops them.
:::

### Parsing a String

The package object mixes in both the shared and platform-specific syntax, so one import brings in every constructor available on the current platform:

```scala

val hoconText =
  """|db {
     |  host = "localhost"
     |  port = 5432
     |}
     |""".stripMargin
```

Braces nest into dotted keys, and an integral number flattens without a decimal point — `5432`, not `5432.0`:

```scala
ConfigSource.fromHocon(hoconText, "app.conf")
// res5: Either[ConfigError, ConfigSource] = Right(
//   MapSource(
//     map = Map("db.host" -> "localhost", "db.port" -> "5432"),
//     sourceId = "app.conf"
//   )
// )
```

That integral-number handling matters because a value rendered as `5432.0` would not parse as an `Int`. Non-integral values keep their `Double` rendering. The parser never produces an infinite number. A value such as `1e400`, `Infinity` or `-1e400` is kept as the text that was written, as a string.

### Substitutions

Parsing runs in two passes: the text is read into a tree that may contain unresolved `${...}` references, then those references are resolved. Substitution therefore happens **before** flattening, so a substituted value reaches the source layer already expanded:

```scala

val withSubstitution =
  """|base = "/srv/app"
     |paths {
     |  data = ${base}/data
     |  logs = ${base}/logs
     |}
     |""".stripMargin
```

The flattened keys hold expanded strings, and nothing downstream needs to know a substitution was involved:

```scala
ConfigSource.fromHocon(withSubstitution, "app.conf")
// res7: Either[ConfigError, ConfigSource] = Right(
//   MapSource(
//     map = Map(
//       "base" -> "/srv/app",
//       "paths.data" -> "/srv/app/data",
//       "paths.logs" -> "/srv/app/logs"
//     ),
//     sourceId = "app.conf"
//   )
// )
```

The resolution pass detects circular references and reports them as a parse failure rather than looping.

### Includes

`include "file"` directives are resolved through a callback, because the core parser does not assume a filesystem. `HoconParser.parse` accepts a `String => Option[String]` that returns the contents of a named resource, so a caller who uses the parser directly can supply the includes from a source of their choice, such as a test fixture. The library has no classpath loader.

On the JVM, `ConfigSource.fromFile` wires that callback to the filesystem for you.

:::warning[Includes are dropped by `ConfigSource.fromHocon`]
A known bug in 0.0.56: `ConfigSource.fromHocon` passes no callback, so every `include` in a string is silently skipped, and an object whose only content is an include disappears. The public `ConfigSource` API has no way to pass a resolver. Load a file that uses includes with `ConfigSource.fromFile` on the JVM.
:::

### Loading from a File

`ConfigSource.fromFile` is JVM-only and resolves includes relative to the including file's directory:

```scala

ConfigSource.fromFile("conf/application.conf")
```

Three limits guard against a hostile or accidentally-recursive configuration tree:

| Parameter          | Default | Effect                                                                           |
| ------------------ | ------- | -------------------------------------------------------------------------------- |
| `allowedBase`      | `None`  | When set, both the file and every include must resolve inside this directory.      |
| `maxIncludeDepth`  | `10`    | Maximum nesting depth for includes; exceeding it is a parse failure.               |
| `maxFileBytes`     | `1 MiB` | Maximum size of the file and of each include; a larger one is a parse failure. |

`maxFileBytes` has a default only on `ConfigSource.fromFileWithLimits`, which also takes `allowedBase` and `maxIncludeDepth` with defaults. `ConfigSource.fromFile(path, allowedBase, maxIncludeDepth)` always uses the 1 MiB default; the four-argument overload `ConfigSource.fromFile(path, allowedBase, maxIncludeDepth, maxFileBytes)` takes the limit explicitly.

Pass `allowedBase` whenever the path comes from outside the program. Paths are canonicalized before the check, so `../` sequences cannot escape:

```scala

ConfigSource.fromFile(
  path = "conf/application.conf",
  allowedBase = Some(new File("conf"))
)
```

Failures are reported as `ConfigError.ParseError`: a missing file expects `"existing file"`, a path outside `allowedBase` expects `"path inside <base>"`, a file or include over `maxFileBytes` fails with a cause message that starts `File size exceeds limit`, and a malformed document carries the `HoconError` as its cause. The `sourceId` of a file-loaded source is `hocon:` followed by the file's name.

For a walkthrough that loads a split configuration tree on the JVM and runs the same settings in the browser, see [How to Load Configuration Files on the JVM and in the Browser](../../guides/load-configuration-files.md).

### HoconValue

`HoconValue` is the parsed tree, exposed for code that needs the structure rather than a flat source:

```scala
sealed trait HoconValue

object HoconValue {
  final case class Obj(fields: Map[String, HoconValue]) extends HoconValue
  final case class Arr(elements: Seq[HoconValue])       extends HoconValue
  final case class Str(value: String)                   extends HoconValue
  final case class Num(value: Double)                   extends HoconValue
  final case class Bool(value: Boolean)                 extends HoconValue
  case object Null                                      extends HoconValue
}
```

`HoconValue.flatten` performs the flattening the adapter uses, and `HoconValue.deepMerge` merges two trees recursively with the right side winning on conflict — the operation a layered HOCON setup needs before flattening, as opposed to `ConfigSource#orElse`, which layers after.

`HoconError` carries a message with the line and column where parsing failed, and extends `Exception` so it can be thrown or attached as a cause. The line and column are set for lexer errors only: an error found while resolving a substitution reports line 0, column 0. A known bug in 0.0.56: `ConfigSource.fromHocon` reports a syntax error as `Parse error for key '' (expected valid HOCON) in source '<name>'`, with no position. The `HoconError` survives only in the `cause` field of `ConfigError.ParseError`, which neither `message` nor `getCause` shows:

```scala
final case class HoconError(message: String, line: Int, column: Int)
```

## Choosing a Format

The adapters differ in capability, not in how their output is consumed:

| Capability                    | YAML | JSON | HOCON |
| ----------------------------- | ---- | ---- | ----- |
| Nested objects and arrays      | Yes  | Yes  | Yes   |
| Comments                       | Yes  | No   | Yes   |
| Substitutions (`${...}`)       | No   | No   | Yes   |
| Includes                       | No   | No   | Yes   |
| Load from a file               | No   | No   | Yes (JVM) |
| Scala.js                       | Yes  | Yes  | Yes (string only) |

For everything else, prefer the format your deployment already uses. Because all three flatten identically, switching later changes one constructor call.

## Integration Points

Each adapter depends on the core `config` module for `ConfigSource` and `ConfigError`. YAML and JSON additionally depend on `zio-blocks-schema-yaml` and `zio-blocks-schema-json` for parsing; HOCON has no dependency beyond the core module.

Adapters produce nothing but a `ConfigSource`, so see [ConfigSource](./config-source.md) for composition and provenance, and [Config Decoder](./config-decoder.md) for how the flattened keys become typed values.

---

## Configuration & Feature Flags

`zio.blocks.config` loads typed configuration from string-keyed sources, tracks where every resolved value came from, and evaluates feature flags with percentage-based rollouts. The module is synchronous and zero-dependency. Core types: `ConfigSource`, `ConfigDecoder`, `ConfigError`, `Provenance`, `FlagSource`, `StaticFlag`, `DynamicFlag`, `Rollout`. The shapes at the center of the module:

```scala
trait ConfigSource extends FlagSource {
  def sourceId: String
  def get(key: String): Maybe[SourceValue[String]]
  def all(prefix: String): Map[String, SourceValue[String]]
}

trait ConfigDecoder[A] {
  def decode(source: ConfigSource, prefix: String): Either[::[ConfigError], A]
}

final case class SourceValue[A](value: A, provenance: Provenance)
```

## Introduction

Configuration in this module is a two-layer story. The bottom layer is a flat, string-to-string namespace: a `ConfigSource` answers `get("db.host")` with a raw string plus a record of where that string came from. The top layer turns that namespace into typed values: a `ConfigDecoder[A]`, derived from `Schema[A]`, walks your case class and pulls one key per field.

Everything in between — composing sources, renaming keys to match an environment's conventions, accumulating errors instead of failing on the first one, redacting secrets when you print what was loaded — happens without an effect type. There is no `IO` wrapper: loading configuration is a synchronous function call that returns `Either`.

Feature flags reuse the same source abstraction. `FlagSource` is the scalar half of `ConfigSource`, so one source object can answer both typed config lookups and flag lookups.

## Motivation

Most configuration libraries make you choose between two unpleasant options: a stringly-typed map that compiles but fails at runtime, or a typed API bolted to a specific effect system and a specific file format. This module avoids both.

- **Typed without ceremony.** A `Schema[A]` you already have — or that `Schema.derived` writes for you — is all a decoder needs. There is no separate config-description DSL to learn and keep in sync with the case class.
- **Errors accumulate.** A five-field config with three bad fields reports three errors, not the first one. `ConfigError.Composite` carries them all.
- **Provenance is not an afterthought.** Every value arrives paired with the source that produced it, so "why is this port 8080?" has an answer you can print.
- **Format-agnostic.** YAML, JSON, and HOCON adapters all flatten to the same dot-separated namespace, so decoding, composition, and provenance work identically regardless of where the bytes came from.
- **No dependencies, both platforms.** The core module runs on the JVM and Scala.js. Only file-based HOCON loading is JVM-only.

## Installation

The core module provides sources, decoding, and flags:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-config" % "0.0.56"
```

Each file format is a separate artifact, so you depend only on the ones you use:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-config-yaml"  % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-config-json"  % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-config-hocon" % "0.0.56"
```

For Scala.js, use `%%%` instead of `%%`:

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-config" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x.

## Overview

The module divides into three groups of types: the source layer that produces raw strings, the decoding layer that produces typed values, and the flag layer that resolves runtime switches.

### The Source Layer

`ConfigSource` is a flat key-value namespace with dot-separated paths. `ConfigSource.fromMap` builds one from a `Map`, `EnvSource` reads environment variables, and `SysPropSource` reads system properties. Sources compose with `ConfigSource#orElse` and can be re-rooted with `ConfigSource#prefix`. See [Config Source](./config-source.md).

`SourceValue` pairs a raw value with a `Provenance` describing its origin, and `ProvenanceMap` lets you query and print those origins after a successful load. Also on that page: `KeyMapper` and `KeyFormat`, which translate between your canonical camelCase field names and whatever casing the environment actually uses.

### The Decoding Layer

`ConfigDecoder[A]` turns a source into an `A`, accumulating every failure. `ConfigDecoderDeriver` is the `Deriver` that builds those decoders from `Schema[A]`, and it defines the mapping rules: how nested case classes become dotted paths, how sealed traits pick a case from a discriminator key, how sequences and maps are keyed. See [Config Decoder](./config-decoder.md).

`ConfigError` is the failure type, split into four category traits so you can match on the kind of problem rather than the specific constructor. See [Errors](./errors.md).

### The Flag Layer

`StaticFlag[A]` resolves once at class-load time and never changes. `DynamicFlag[A]` holds a rollout expression that can be updated or reloaded while the process runs. Both derive their names from the enclosing Scala object and register themselves globally. See [Flags](./flags.md).

`Rollout` is the expression language behind dynamic flags: a semicolon-separated list of choices, each optionally targeted at a path pattern and a percentage bucket. See [Rollout](./rollout.md).

### The Format Adapters

`config-yaml`, `config-json`, and `config-hocon` each add one `ConfigSource` constructor. They differ in what they support — only HOCON has substitutions and includes — but they agree on the flattening rules. See [File Formats](./formats.md).

## How They Work Together

A typed load moves through four stages. The source produces raw strings, the decoder walks the schema requesting one key per field, errors accumulate rather than short-circuit, and the result is either a fully-built value or every problem found along the way:

```
1. Build a source            ConfigSource.fromMap / fromYaml / EnvSource
2. Compose and re-root       source.orElse(fallback).prefix("app")
3. Derive a decoder          ConfigDecoder.derive[A]  (from Schema[A])
4. Decode                    decoder.decode(source, "")
                                 │
                                 ├─ Right(a)      all fields resolved
                                 └─ Left(errors)  every failure, accumulated
```

The type relationships behind those stages:

```
FlagSource ─────────────────┐  get(name): Maybe[SourceValue[String]]
     ▲                      │
     │ extends              │
ConfigSource ───────────────┤  + all(prefix): Map[String, SourceValue[String]]
     │                      │
     ├─ MapSource           │
     ├─ EnvSource           │  DB_HOST  ← db.host
     ├─ SysPropSource       │
     └─ (yaml/json/hocon)   │  flattened to dotted keys
                            │
                            └─> SourceValue ──> Provenance
                                                  ├─ Resolved(sourceId, key, rawValue)
                                                  └─ Default

Schema[A] ──deriving──> ConfigDecoderDeriver ──> ConfigDecoder[A]
                                                      │
                                    decode(source, prefix)
                                                      │
                                    ┌─────────────────┴──────────────────┐
                                    ▼                                    ▼
                              Right(A)                        Left(::[ConfigError])
                                                                   ├─ MissingKey
                                                                   ├─ InvalidValue
                                                                   ├─ ParseError
                                                                   └─ Composite(all)

StaticFlag[A]   ──resolves once──> Registry → sysprop → env → default
DynamicFlag[A]  ──evaluates──>   Rollout.Choices x bucket → value
```

## Common Patterns

Four shapes cover most usage: a straight typed load, a layered load where environment variables win over file defaults, wiring config into a dependency graph, and loading with provenance so you can explain what happened.

### Loading a Typed Value

`Config.load` derives a decoder and runs it, returning every error it found:

```scala

case class Db(host: String, port: Int)

object Db {
  implicit val schema: Schema[Db] = Schema.derived[Db]
}

val source = ConfigSource.fromMap(Map("host" -> "localhost", "port" -> "5432"), "example")
```

Decoding returns `Either[::[ConfigError], Db]`, where `::` is Scala's non-empty list, so a `Left` always carries at least one error:

```scala
Config.load[Db](source)
// res0: Either[::[ConfigError], Db] = Right(
//   Db(host = "localhost", port = 5432)
// )
```

When a key is missing, the error names both the path and the source that was searched:

```scala
Config.load[Db](ConfigSource.fromMap(Map("host" -> "localhost"), "example"))
// res1: Either[::[ConfigError], Db] = Left(
//   List(MissingKey(path = "port", source = "example"))
// )
```

### Layering Sources

Production configuration usually comes from more than one place: a file provides defaults and the environment overrides them. `ConfigSource#orElse` consults the receiver first and falls back to the argument, and provenance still records which one actually answered:

```scala

case class Db(host: String, port: Int)

object Db {
  implicit val schema: Schema[Db] = Schema.derived[Db]
}

val defaults = ConfigSource.fromMap(Map("host" -> "localhost", "port" -> "5432"), "defaults")
val envOverrides = ConfigSource.fromMap(Map("host" -> "db.prod.internal"), "env")
```

The override wins for `host`, and `port` falls through to the defaults:

```scala
Config.load[Db](envOverrides.orElse(defaults))
// res3: Either[::[ConfigError], Db] = Right(
//   Db(host = "db.prod.internal", port = 5432)
// )
```

### Wiring Config into a Dependency Graph

Rather than decoding at startup and threading the result through constructors, `Config.wire` produces a `Wire.Shared[ConfigSource, A]` so decoding becomes a node in the graph. The `ConfigSource` is the injected input; the typed config is the derived output:

```scala

case class Db(host: String, port: Int)

object Db {
  implicit val schema: Schema[Db]     = Schema.derived[Db]
  implicit val unscoped: Unscoped[Db] = Unscoped.derived[Db]
}

final class DbService(val config: Db)

val source   = ConfigSource.fromMap(Map("host" -> "localhost", "port" -> "5432"), "graph")
def resource = Resource.from[DbService](Wire(source), Config.wire[Db])

Scope.global.scoped { scope =>

  val service = allocate(resource)
  $(service)(_.config)
}
```

`Config.wire[A](prefix)` re-roots the injected source before decoding, which is how you keep several config sections in one graph without giving each its own source.

See also: [How to Feed Configuration into a Wire Dependency Graph](../../guides/feed-config-into-a-wire-graph.md) for a full walkthrough with several services, layered sources and ordered shutdown.

### Loading with Provenance

`Config.loadWithProvenance` returns a `ProvenanceMap[A]` alongside the value, which answers per-key origin questions and renders a table for startup logs:

```scala

case class Db(host: String, port: Int)

object Db {
  implicit val schema: Schema[Db] = Schema.derived[Db]
}

val source = ConfigSource.fromMap(Map("host" -> "localhost", "port" -> "5432"), "startup")
val loaded = Config.loadWithProvenance[Db](source)
```

Querying a single key reports the source id, the source-facing key, and the raw string:

```scala
loaded.map(_.provenanceOf("host"))
// res6: Either[::[ConfigError], Maybe[Provenance]] = Right(
//   Resolved(sourceId = "startup", key = "host", rawValue = "localhost")
// )
```

`ProvenanceMap#dump` renders every visible key as a table, redacting values whose key names look sensitive:

```scala
loaded.map(_.dump()).foreach(println)
// ┌──────┬───────────┬─────────┐
// │ Key  │ Value     │ Source  │
// ├──────┼───────────┼─────────┤
// │ host │ localhost │ startup │
// │ port │ 5432      │ startup │
// └──────┴───────────┴─────────┘
```

## Entry Points

`Config` is the front door for typed loading. Which method you want depends on how you handle failure and whether you need provenance:

| Method                        | Returns                              | Use when                                                        |
| ----------------------------- | ------------------------------------ | --------------------------------------------------------------- |
| `Config.load[A]`              | `Either[::[ConfigError], A]`          | You want to inspect or report errors yourself.                   |
| `Config.loadOrThrow[A]`       | `A`                                  | Failure should abort startup; throws `ConfigLoadException`.       |
| `Config.loadWithProvenance[A]`| `Either[::[ConfigError], ProvenanceMap[A]]` | You need to explain where values came from.               |
| `Config.wire[A]`              | `Wire.Shared[ConfigSource, A]`       | Decoding belongs inside a dependency graph.                      |

`Config.load` derives a fresh decoder on every call. When you load the same type repeatedly — during reloads, or per request — derive once with `ConfigDecoder.derive[A]` and reuse the result.

:::warning[Derivation is not free]
`Schema.derived` runs at compile time, but turning a `Schema[A]` into a `ConfigDecoder[A]` happens at runtime and walks the whole schema. Calling `Config.load[A]` in a hot path re-does that walk every time.
:::

## Integration Points

The module sits on top of three other blocks and hands its output to a fourth:

- **`zio-blocks-schema`** supplies `Schema[A]`, and `ConfigDecoderDeriver` is an ordinary `Deriver[ConfigDecoder]`. Anything the schema layer can describe — records, variants, sequences, maps, wrappers, dynamic values — has a decoding rule.
- **`zio-blocks-maybe`** supplies `Maybe`, the allocation-free option type returned by every source lookup.
- **`zio-blocks-scope`** supplies `Wire` and `Resource`, which is what `Config.wire` produces so config can participate in dependency graphs.
- **`zio-blocks-schema-yaml`** and **`zio-blocks-schema-json`** back the YAML and JSON adapters; the HOCON adapter has its own parser and no external dependency.

Within the module, the dependency direction is one-way: sources know nothing about decoders, decoders know nothing about flags, and flags reuse `FlagSource` without reaching into `ConfigDecoder`.

## Next Steps

Start with [Config Source](./config-source.md) to build and compose sources, then [Config Decoder](./config-decoder.md) for the mapping rules from case classes to keys. [Errors](./errors.md) covers the failure model, [Flags](./flags.md) and [Rollout](./rollout.md) cover runtime switches, and [File Formats](./formats.md) covers YAML, JSON, and HOCON.

See also the [Build a Typed Configuration Loader](../../guides/typed-config-loader.md) tutorial, which layers defaults, a YAML file and environment variables into one typed case class, and the [Coming from PureConfig, Typesafe Config or zio-config](../../guides/config-library-migration.md) guide for porting an existing configuration setup. For runtime switches, see [How to Roll Out a Feature Gradually with Flags](../../guides/roll-out-feature-gradually.md). To load HOCON, JSON and YAML files from disk, split with includes, see [How to Load Configuration Files on the JVM and in the Browser](../../guides/load-configuration-files.md).

---

## Rollout

`Rollout` is the expression language behind dynamic flags. An expression is a semicolon-separated list of choices, each optionally targeted at a slash-separated path pattern and a percentage bucket. `Rollout` parses expressions into `Rollout.Choices`, evaluates them against a path and a bucket, and computes the deterministic bucket for a key. Supporting types: `Rollout.Choice`, `Rollout.Selector`, `Rollout.Segment`, `Flag.ReloadResult`, `DynamicFlag.UpdateRecord`. The parsed form of an expression:

```scala
object Rollout {
  final case class Choices(entries: List[Choice])

  sealed trait Choice
  object Choice {
    final case class Targeted(value: String, selector: Selector) extends Choice
    final case class CatchAll(value: String)                     extends Choice
  }

  final case class Selector(segments: List[Segment], percentage: Maybe[Int])

  sealed trait Segment
  object Segment {
    case object Wildcard                    extends Segment
    final case class Literal(value: String) extends Segment
  }
}
```

## Motivation

A feature flag that is only on or off cannot express the thing teams actually want: on for 10% of users, on for everyone in one region, off everywhere else. Encoding that as three separate booleans means three flags to keep consistent, and encoding it as code means a deploy for every adjustment.

A rollout expression puts the whole decision in one string, which is what makes it deployable through a config source. `"true@*/eu/100%; true@*/50%; false"` reads as three rules in priority order: everyone in `eu`, then half of everyone else, then off. Changing the rollout is changing that string.

The percentage is a function of the key, not a coin flip. The same key always lands in the same bucket, so a user who sees the new checkout flow sees it on every request, and raising 50% to 60% keeps the original 50% inside the group rather than reshuffling everyone.

## Grammar

An expression is a list of choices; a targeted choice pairs a value with a selector; a selector is a path pattern with an optional percentage:

```
expression = choice { ";" choice }
choice     = value "@" selector | value
selector   = path [ "/" percentage ]
path       = segment { "/" segment }
segment    = "*" | literal
percentage = digits "%"
```

Whitespace around choices and segments is trimmed. A choice with no `@` is a catch-all that matches everything.

The percentage is always the final slash-separated component, which is why `prod/50%` means "path `prod`, 50 percent" rather than a two-segment path. A selector consisting only of a percentage is rejected — there is no implicit "match everything" path, so write `*/50%` when you mean half of all keys.

## Evaluation

Evaluation walks the choices left to right and returns the first match. A catch-all matches immediately, so anything after it is unreachable.

### Path Matching

A path matches a selector only when both have the **same number of segments**, compared position by position. `Segment.Wildcard` matches any single segment; `Segment.Literal` requires equality:

```scala

val bucket = Rollout.bucketFor("user-1234")
```

A single-literal selector matches only that exact one-segment path:

```scala
Rollout.select("on@beta; off", "beta", bucket)
// res0: Maybe[String] = "on"
Rollout.select("on@beta; off", "prod", bucket)
// res1: Maybe[String] = "off"
```

A wildcard matches any one segment, but still only one:

```scala
Rollout.select("on@*; off", "anything", bucket)
// res2: Maybe[String] = "on"
Rollout.select("on@*; off", "two/segments", bucket)
// res3: Maybe[String] = "off"
```

Multi-segment patterns mix the two, matching a path of exactly that length:

```scala
Rollout.select("on@*/eu; off", "user-1234/eu", bucket)
// res4: Maybe[String] = "on"
Rollout.select("on@*/eu; off", "user-1234/us", bucket)
// res5: Maybe[String] = "off"
```

:::warning[Segment count is exact]
There is no prefix or suffix matching and no multi-segment wildcard. A selector with two segments never matches a three-segment path. This is the most common cause of a rollout that silently falls through to its catch-all.
:::

### Percentages

The percentage is compared against the bucket, with three values special-cased: an absent percentage always matches a matching path, `0%` never matches, and `100%` always matches. Otherwise the choice matches when `bucket < percentage`:

| Percentage | Matches when the path matches   |
| ---------- | ------------------------------- |
| *(absent)* | Always                          |
| `0%`       | Never                           |
| `100%`     | Always                          |
| `n%`       | `bucket < n`                    |

Each choice is a threshold on the same bucket, not a slice of the traffic. The first matching choice wins, so a percentage is the share of keys it names only when no earlier choice has matched them. A later choice with a higher percentage adds the buckets between the two thresholds. For example, `"a@*/40%; b@*/80%; c"` gives `a` the buckets below 40, `b` the buckets from 40 to 79, and `c` the rest: a 40/40/20 split. A later choice whose percentage is at or below an earlier one never matches: in `"a@*/30%; b@*/20%; c"`, every bucket below 20 already matched `a`, so `b` gets no keys.

### Bucketing

`Rollout.bucketFor` hashes a key with MurmurHash3 and reduces it to `0..99`. The result is stable for a given key across processes and restarts:

```scala
Rollout.bucketFor("user-1234")
// res6: Int = 7
Rollout.bucketFor("user-1234") == Rollout.bucketFor("user-1234")
// res7: Boolean = true
```

Different keys spread across the range, which is what makes a percentage approximate a uniform share:

```scala
(1 to 5).map(i => Rollout.bucketFor(s"user-$i"))
// res8: IndexedSeq[Int] = Vector(64, 97, 86, 68, 0)
```

Stability is the reason to pass a user or session identifier as the key rather than something that changes per call. A random key would re-roll the dice on every evaluation.

## API

Five methods cover parsing, evaluation, and diagnostics. `Rollout.select` is the one-shot form; the others let you parse once and evaluate many times.

### Rollout.select

`Rollout.select` parses and evaluates in one call, returning `Maybe.absent` both when nothing matches and when the expression will not parse:

```scala
Rollout.select("on@prod/50%; off", "prod", 10)
// res9: Maybe[String] = "on"
Rollout.select("@@@not-valid", "prod", 10)
// res10: Maybe[String] = zio.blocks.maybe.Absent$@b322d7c
```

Because a parse failure is indistinguishable from a non-match, use `Rollout.select` only where a malformed expression should behave as "no decision". Parse explicitly when you need to know.

### Rollout.parseChoices

`Rollout.parseChoices` returns the parsed `Choices` or the first `ConfigError` it encountered:

```scala
Rollout.parseChoices("on@prod/50%; off")
// res11: Either[ConfigError, Choices] = Right(
//   Choices(
//     List(
//       Targeted(
//         value = "on",
//         selector = Selector(segments = List(Literal("prod")), percentage = 50)
//       ),
//       CatchAll("off")
//     )
//   )
// )
```

An expression with a malformed choice reports the failure rather than skipping it:

```scala
Rollout.parseChoices("on@prod/150%")
// res12: Either[ConfigError, Choices] = Left(
//   InvalidValue(
//     path = "rollout",
//     value = "prod/150%",
//     expectedType = "percentage <= 100",
//     source = "rollout",
//     cause = None
//   )
// )
```

Parsing is what `DynamicFlag` does at initialization and on every update, so these are the errors a flag rejects.

### Rollout.evaluateIndex

`Rollout.evaluateIndex` evaluates already-parsed choices, which is what the flag does on the hot path — parsing happens once per update, not once per evaluation:

```scala
val choices = Rollout.parseChoices("on@*/eu; off").toOption.get
```

Evaluating the same choices for several paths reuses the parse:

```scala
Rollout.evaluateIndex(choices, "user-1/eu", 42)
// res13: Maybe[String] = "on"
Rollout.evaluateIndex(choices, "user-1/us", 42)
// res14: Maybe[String] = "off"
```

:::warning[Buckets ignore the flag name]
A known bug in 0.0.56: `DynamicFlag` buckets a call by the key alone, so two flags rolled out to the same percentage pick the same users. A user in a low bucket gets every early rollout at once. Salt the key with the flag's purpose, as in `flag(s"recommendations:$id")`, and use the same salt at every call site.
:::

### Rollout.validate

`Rollout.validate` parses an expression and returns warnings about rules that are suspicious rather than invalid. It flags choices rendered unreachable by an earlier catch-all:

```scala
Rollout.validate("off; on@prod")
// res15: Either[ConfigError, List[String]] = Right(
//   List("Choices after index 0 are unreachable (catch-all found)")
// )
```

It also flags a pattern whose percentages sum above 100. That check treats each percentage as a slice of the traffic, which evaluation does not do, so it can warn about a correct split. This 40/40/20 split is valid, and it gets a warning:

```scala
Rollout.validate("a@*/40%; b@*/80%; c")
// res16: Either[ConfigError, List[String]] = Right(
//   List("Cumulative percentage for pattern '*' is 120% (exceeds 100%)")
// )
```

A known bug in 0.0.56: the check cannot see a choice that never matches. The shadowed `b` in `"a@*/30%; b@*/20%; c"` produces no warning, so use the threshold rule above when you review an expression by hand.

A clean expression returns an empty list:

```scala
Rollout.validate("on@*/eu; on@*/50%; off")
// res17: Either[ConfigError, List[String]] = Right(List())
```

Warnings are advisory — `DynamicFlag` does not run them. Call `Rollout.validate` in a test or an admin endpoint that accepts expressions from operators.

### Parse Failures

Every parse failure is a `ConfigError.InvalidValue` with path and source `"rollout"`, and an `expectedType` describing what was wanted:

| Expression        | Rejected because                                     |
| ----------------- | ---------------------------------------------------- |
| `""`              | Empty expression.                                     |
| `"@prod"`         | No value before `@`.                                  |
| `"on@"`           | No selector after `@`.                                |
| `"on@50%"`        | Percentage with no path; write `on@*/50%`.             |
| `"on@prod/"`      | Trailing slash with no percentage.                    |
| `"on@prod/abc%"`  | Non-numeric percentage.                              |
| `"on@prod/150%"`  | Percentage above 100.                                 |

## The Parsed Form

`Rollout.Choices` wraps an ordered `List[Choice]`, and the order is the evaluation order. Matching on the ADT is how an admin tool can render or rewrite an expression rather than treating it as opaque text:

```scala

val parsed = Rollout.parseChoices("on@*/eu/25%; off").toOption.get
```

A `Choice.Targeted` carries its value and selector; a `Choice.CatchAll` carries only a value:

```scala
parsed.entries
// res19: List[Choice] = List(
//   Targeted(
//     value = "on",
//     selector = Selector(
//       segments = List(Wildcard, Literal("eu")),
//       percentage = 25
//     )
//   ),
//   CatchAll("off")
// )
```

`Selector#segments` is the path pattern as a list, and `Selector#percentage` is a `Maybe[Int]` that is absent when the selector had no percentage:

```scala
parsed.entries.collect { case t: Rollout.Choice.Targeted => (t.selector.segments, t.selector.percentage) }
// res20: List[Tuple2[List[Segment], Maybe[Int]]] = List(
//   (List(Wildcard, Literal("eu")), 25)
// )
```

## Reload Lifecycle

A dynamic flag's expression can change while the process runs, either from code or from a source. Both paths record what happened so the change is auditable.

### Flag.ReloadResult

`DynamicFlag#reload` re-reads the expression from `FlagSource.Registry` and reports the outcome:

| Result                              | Meaning                                                       |
| ----------------------------------- | ------------------------------------------------------------- |
| `Flag.ReloadResult.NoSource`        | No registered source provides this flag's name; nothing changed. |
| `Flag.ReloadResult.Unchanged`       | The source's expression is identical to the current one.        |
| `Flag.ReloadResult.Updated(old, new)` | The expression was replaced; both versions are reported.       |
| `Flag.ReloadResult.Failed(error)`   | The source's expression will not parse; the old one is kept.     |

`Failed` keeping the previous expression is deliberate: a bad push to a flag service degrades to "no change", not to the constructor default.

Reloading is manual. Nothing in the module polls, so drive it from whatever scheduler your application already has:

```scala

object newCheckout extends DynamicFlag[Boolean](false, "false")

newCheckout.reload() match {
  case Flag.ReloadResult.Updated(from, to) => println(s"rollout changed: $from -> $to")
  case Flag.ReloadResult.Failed(error)     => println(s"rollout rejected: ${error.message}")
  case Flag.ReloadResult.Unchanged         => ()
  case Flag.ReloadResult.NoSource          => println("no source registered for this flag")
}
```

### DynamicFlag.UpdateRecord

Every successful change — from `DynamicFlag#update` or from a reload — appends a `DynamicFlag.UpdateRecord` holding the old expression, the new one, and a millisecond timestamp:

```scala
final case class UpdateRecord(oldExpression: String, newExpression: String, timestampMillis: Long)
```

`DynamicFlag#updateHistory` returns them most-recent-first, capped at the last ten. Older records are discarded, so the history answers "what changed just now?" rather than serving as an audit log.

### Evaluation Counters

`DynamicFlag#apply` increments a per-key counter, and `DynamicFlag#counters` returns a snapshot as a `Map[String, Long]`. The counters are thread-safe but approximate — they use `LongAdder`, so a snapshot taken during traffic may miss in-flight increments.

Distinct keys are capped at 100. Once that many have been seen, every new key is counted under the single bucket `"other"` instead of getting its own entry. No exception is thrown and existing counters keep working, so a flag keyed by user id degrades to "100 known users plus a bulk count" rather than leaking memory.

`DynamicFlag#evaluate` bypasses counting entirely. Use it on paths where you do not want the bookkeeping, and reserve `DynamicFlag#apply` for the call sites whose distribution you want to observe.

`DynamicFlag#parseErrorCount` counts evaluations where a matched value could not be parsed into the flag's type, causing a fall back to the default. A non-zero count is always a misconfigured expression: the rollout matched, but the value it selected was not readable. Because every call still returns the default, this is otherwise silent.

## Integration Points

`Rollout` depends only on `Maybe` and `ConfigError`. It is used by `DynamicFlag` for initialization, updates, reloads, and every evaluation, and it can be used directly wherever a path-and-percentage decision is needed without a flag object.

See [Flags](./flags.md) for the flag types that drive it, and [Errors](./errors.md) for the error type its parser returns.

See also the [How to Roll Out a Feature Gradually with Flags](../../guides/roll-out-feature-gradually.md) guide, which writes rollout expressions for a percentage and a region and widens them at runtime.

---

## Context

`Context[+R]` is a type-indexed heterogeneous collection that stores values of different types, indexed by their types, with compile-time type safety for lookups. It provides an immutable, cache-aware dependency container where the phantom type `R` (using intersection types) tracks which types are present.

The core type looks like this:

```scala
// Signature (showing public API structure, not actual implementation)
final class Context[+R] {
  def size: Int
  def isEmpty: Boolean
  def nonEmpty: Boolean
  def get[A >: R](implicit ev: IsNominalType[A]): A
  def getOption[A](implicit ev: IsNominalType[A]): Option[A]
  def add[A](a: A)(implicit ev: IsNominalType[A]): Context[R & A]
  def update[A >: R](f: A => A)(implicit ev: IsNominalType[A]): Context[R]
  def ++[R1](that: Context[R1]): Context[R & R1]
  def prune[A >: R](implicit ev: IsNominalIntersection[A]): Context[A]
  override def toString: String
}
```

Key properties: covariant (`+R`), immutable, cached for repeated lookups, supports only nominal types.

## Overview

Context serves as a type-safe registry for heterogeneous dependencies. Here's a quick example:

```scala

case class Config(debug: Boolean)
case class Logger(name: String)
case class Metrics(count: Int)
```

We can create and retrieve values by type:

```scala
val ctx: Context[Config & Logger & Metrics] = Context(
  Config(debug = true),
  Logger("app"),
  Metrics(count = 42)
)
// ctx: Context[Config & Logger & Metrics] = Context(repl.MdocSession.MdocApp.Metrics -> Metrics(42), repl.MdocSession.MdocApp.Logger -> Logger(app), repl.MdocSession.MdocApp.Config -> Config(true))

val config: Config = ctx.get[Config]
// config: Config = Config(true)
val logger: Logger = ctx.get[Logger]
// logger: Logger = Logger("app")
```

This ASCII diagram shows how Context maps types to values:

```
┌─────────────────────────────────┐
│         Context[R]              │
│  (Type-Indexed Store)           │
├─────────────────────────────────┤
│  Config         → Config(true)  │
│  Logger         → Logger("app") │
│  Metrics        → Metrics(42)   │
├─────────────────────────────────┤
│ ✓ Type-safe retrieval           │
│ ✓ No casting                    │
│ ✓ Cache-aware (O(1) repeats)    │
└─────────────────────────────────┘
```

## Motivation

When building modular applications, we often need to pass multiple dependencies around—a database connection, a config object, a logger, and so on. Existing approaches each have limitations:

**`Map[Class[_], Any]`** — no compile-time safety. You must cast the result and remember which keys you registered:

```scala
// Unsafe approach — easy to make mistakes
val deps = scala.collection.mutable.Map[Class[_], Any]()
deps(classOf[Config]) = Config(debug = true)
deps(classOf[Logger]) = Logger("app")

val config = deps(classOf[Config]).asInstanceOf[Config]  // Manual cast
val db = deps(classOf[Database]) // Runtime error if missing
```

**ZIO's `ZEnvironment`** — type-safe but requires the full ZIO effect system:

```scala
// Requires ZIO context

val makeEnv = for {
  config <- ZIO.service[Config]
  logger <- ZIO.service[Logger]
} yield (config, logger)
```

**Context** — combines compile-time type safety with synchronous, pure code:

```scala

case class Config(debug: Boolean)
case class Logger(name: String)

// Type-safe, no effects needed
val ctx = Context(Config(true), Logger("app"))
val config = ctx.get[Config]  // Compile-time proof it exists
```

## Installation

Add the ZIO Blocks Context module to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-context" % "0.0.56"
```

## Construction

Context provides several ways to create instances. Choose the approach that best fits your use case: start empty and add values incrementally, or construct a fully-populated context directly with `apply`.

### Creating Empty Contexts

Use `Context.empty` to create an empty context with no entries:

```scala

val emptyCtx: Context[Any] = Context.empty
// emptyCtx: Context[Any] = Context()
val isEmpty = emptyCtx.isEmpty
// isEmpty: Boolean = true
```

An empty context has type `Context[Any]` and represents no stored dependencies. This is a useful starting point for incremental construction.

### Creating Multi-Value Contexts with `Context.apply`

`Context.apply` is overloaded to accept 1–10 values and returns a context with type `Context[A1 & A2 & ...]`, reflecting all stored types.

#### Single Value

Create a context with one value:

```scala
case class Config(debug: Boolean)
```

With `Config` defined, we can create a single-value context:

```scala
val single: Context[Config] = Context(Config(debug = true))
// single: Context[Config] = Context(repl.MdocSession.MdocApp1.Config -> Config(true))
```

#### Multiple Values

Create a context with multiple values—the type parameter automatically becomes an intersection of all stored types:

```scala
case class Logger(name: String)
```

With `Config` and `Logger` in scope, we can create a multi-value context:

```scala
val multi: Context[Config & Logger] = Context(
  Config(debug = true),
  Logger("myapp")
)
// multi: Context[Config & Logger] = Context(repl.MdocSession.MdocApp1.Logger -> Logger(myapp), repl.MdocSession.MdocApp1.Config -> Config(true))
```

### Building Incrementally with `Context#add`

For contexts that grow over time, use `Context#add` to build incrementally from an empty context. This is useful when dependencies become available at different points in your initialization:

```scala
val ctx = Context.empty
  .add(Config(debug = false))
  .add(Logger("init"))
// ctx: Context[Config & Logger] = Context(repl.MdocSession.MdocApp1.Logger -> Logger(init), repl.MdocSession.MdocApp1.Config -> Config(false))
```

The context accumulates all added entries:

```scala
val size1 = ctx.size
// size1: Int = 2
```

**When to use `Context#add` vs. `Context.apply`:**
- Use `Context.apply` when you know all dependencies upfront and can construct them together
- Use `Context#add` when dependencies are added incrementally or conditionally

## Core Operations

Context supports inspection, retrieval, and modification operations. All methods are type-safe and leverage the phantom type `R` to track what's stored.

### Inspection

The following methods let you check the contents of a context without retrieving specific values:

#### `Context#size`

Returns the number of entries in the context:

```scala
case class Config(debug: Boolean)
case class Logger(name: String)

val ctx = Context(Config(true), Logger("app"))
```

Get the number of entries in the context:

```scala
val sz = ctx.size
// sz: Int = 2
```

#### `Context#isEmpty`

Returns `true` if the context contains no entries:

```scala
case class Config(debug: Boolean)

val empty = Context.empty
val notEmpty = Context(Config(true))
```

Now check the `isEmpty` status of both contexts:

```scala
val e1 = empty.isEmpty
// e1: Boolean = true
val e2 = notEmpty.isEmpty
// e2: Boolean = false
```

#### `Context#nonEmpty`

Returns `true` if the context contains at least one entry (opposite of `isEmpty`):

```scala
case class Config(debug: Boolean)

val empty = Context.empty
val notEmpty = Context(Config(true))
```

Check the `nonEmpty` status of both contexts:

```scala
val e1 = empty.nonEmpty
// e1: Boolean = false
val e2 = notEmpty.nonEmpty
// e2: Boolean = true
```

### Retrieval

The following methods let you retrieve values from a context by type:

#### `Context#get`

Retrieves a value by type. The type bound `A >: R` ensures that a value of type `A` (or a subtype of `A`) is present at compile time:

```scala

case class Config(debug: Boolean)
case class Logger(name: String)
case class Metrics(count: Int)

val ctx = Context(Config(debug = true), Logger("app"), Metrics(100))

// Retrieve by exact type
val config = ctx.get[Config]
```

Or by supertype (subtype matching):

```scala

trait Animal { def sound: String }
case class Dog(name: String) extends Animal {
  def sound = "Woof"
}

val ctxDog = Context(Dog("Buddy"))
val animal = ctxDog.get[Animal]
```

If you attempt to retrieve a type that is not in the context, the code will not compile because the type bound `A >: R` requires it to be present:

```scala
// This is a compile-time error, not a runtime error:
// val metrics: String = ctx.get[String]  // Error: String is not in context type
```

#### `Context#getOption`

Retrieves a value if present, returning `Option[A]`. Unlike `get`, this method does not require the type to be in the context's type parameter. Use it for optional lookups:

```scala
case class Config(debug: Boolean)

val ctx = Context(Config(debug = true))
```

Try to retrieve both an existing type and a missing type:

```scala
val found: Option[Config] = ctx.getOption[Config]
// found: Option[Config] = Some(Config(true))
val missing: Option[String] = ctx.getOption[String]
// missing: Option[String] = None
```

### Modification

All modification methods return a new `Context`—the original remains immutable:

#### `Context#add`

Adds a value to the context, expanding the phantom type by `& A`:

```scala

case class Config(debug: Boolean)
case class Logger(name: String)

val ctx1 = Context(Config(true))
val ctx2 = ctx1.add(Logger("new"))
```

If a value of the same type already exists, it is replaced:

```scala

case class Config(debug: Boolean)
case class Logger(name: String)

val ctx1 = Context(Config(true))
val ctx2 = ctx1.add(Logger("new"))
val ctx3 = ctx2.add(Config(debug = false))
val replaced = ctx3.get[Config]
```

#### `Context#update`

Transforms an existing value if it is present. If the type is not found, the context is returned unchanged:

```scala

case class Metrics(count: Int)

val ctx = Context(Metrics(count = 10))
val updated = ctx.update[Metrics](m => m.copy(count = m.count + 5))
val newCount = updated.get[Metrics].count
```

#### `Context#++ (Union)`

Combines two contexts into a new context containing all entries. When both contexts contain the same type, the value from the right side (second argument) wins:

```scala

case class Config(debug: Boolean)
case class Logger(name: String)
case class Metrics(count: Int)

val left = Context(Config(debug = false), Logger("left"))
val right = Context(Config(debug = true), Metrics(99))
val merged = left ++ right
```

#### `Context#prune`

Narrows a context to contain only specified types. All other entries are discarded:

```scala

case class Config(debug: Boolean)
case class Logger(name: String)
case class Metrics(count: Int)

val full = Context(Config(true), Logger("app"), Metrics(100))
val justConfig = full.prune[Config]
val configSize = justConfig.size
```

#### `Context#toString`

Returns a human-readable representation of the context showing all type-value pairs:

```scala
case class Config(debug: Boolean)
case class Logger(name: String)

val ctx = Context(Config(debug = true), Logger("app"))
```

Convert the context to a human-readable string representation:

```scala
val str = ctx.toString
// str: String = "Context(repl.MdocSession.MdocApp1.Logger -> Logger(app), repl.MdocSession.MdocApp1.Config -> Config(true))"
```

## Covariance

`Context` is covariant in its type parameter, meaning `Context[Dog] <: Context[Animal]` when `Dog <: Animal`. This allows passing a more-specific context to code expecting a more-general one:

```scala

trait Animal { def sound: String }
case class Dog(name: String) extends Animal {
  def sound = "Woof"
}

def processAnimal(ctx: Context[Animal]): String = ctx.get[Animal].sound

val dogCtx = Context(Dog("Buddy"))
val sound = processAnimal(dogCtx)
```

Covariance also applies during retrieval—if you request a supertype, the stored subtype is returned.

## Type Safety: IsNominalType

Context only accepts **nominal types**—concrete classes, case classes, traits, and objects. The compiler automatically derives `IsNominalType[A]` for allowed types and rejects unsupported kinds:

**Supported:**
- Classes: `case class Config(...)`
- Traits: `trait Logger`
- Objects: `object Registry`
- Applied types: `List[Int]`, `Map[String, Int]`
- Enums (Scala 3): `enum Color { case Red, Green, Blue }`
- Opaque types (Scala 3): `opaque type UserId = Long`

**Not supported (compile error):**
- Intersection types: `A & B` (use the context type parameter instead)
- Union types: `A | B`
- Structural types: `{ def foo: Int }`
- Type parameters and abstract type members (require `IsNominalType[A]` from the caller instead)

Attempting to store an unsupported type:

```scala
// This fails at compile time:
// Context.empty.add(null: (String & Int))  // Error: unsupported type
```

## Performance

**Caching**: When a value is retrieved via `get` or `getOption`, the result is cached. Repeated lookups for the same type return the cached value in O(1) time without traversing the entries again.

**Subtype matching**: Supertype lookups scan the entry list linearly to find a compatible subtype. After the first lookup, the result is cached, so subsequent requests for that supertype are O(1).

**Platform optimizations**: The JVM implementation uses `ConcurrentHashMap` for thread-safe caching. The JavaScript platform uses a simpler in-memory mutable hash map for efficient lookups.

## Comparing Approaches

Here is a comparison of Context with related alternatives:

| Feature             | `Map[Class[_], Any]` | `ZEnvironment`   | `Context` |
|---------------------|----------------------|------------------|-----------|
| Type-safe retrieval | ✗ (cast required)    | ✓                | ✓         |
| Compile-time proof  | ✗                    | ✓                | ✓         |
| Effect-free         | ✓                    | ✗ (requires ZIO) | ✓         |
| Immutable           | ✓                    | ✓                | ✓         |
| Cached lookups      | ✗                    | ✓                | ✓         |
| Supertype matching  | ✗                    | ✓                | ✓         |

## Integration with Wire and Scope

`Context` is the dependency carrier in ZIO Blocks' Wire-based dependency injection system. A `Wire[-In, +Out]` describes how to build an output given input dependencies, and contexts supply those dependencies. Wire and Scope work together to provide type-safe, compile-checked dependency injection:

```scala

case class Config(debug: Boolean)
case class Logger(name: String)
case class Service(config: Config, logger: Logger)

// Derive a wire for Service from its constructor. The inputs it needs, Config & Logger, are inferred.
val buildService = Wire.shared[Service]
// buildService: Shared[Config & Logger, Service] = Shared(
//   repl.MdocSession$MdocApp11$$Lambda/0x000000004a4430c0@34e60a0f
// )

// Create a context with the required dependencies
val deps = Context(Config(debug = true), Logger("app"))
// deps: Context[Config & Logger] = Context(repl.MdocSession.MdocApp11.Logger -> Logger(app), repl.MdocSession.MdocApp11.Config -> Config(true))

// Create a scope and instantiate the service inside it
Scope.global.scoped { scope =>
  val service: Service = buildService.make(scope, deps)
  println(service)
}
// Service(Config(true),Logger(app))
```

`Wire.shared[Service]` and `Wire.unique[Service]` are the entry points. The `make(scope, context)` method is on the wire they return, not on the `Wire` companion.

## Running the Examples

All code from this guide is available as runnable examples in the `schema-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

**Context construction: creating contexts with apply, empty.add, and inspecting size/isEmpty/nonEmpty**

```scala title="schema-examples/src/main/scala/context/ContextConstructionExample.scala" 
package context

// Context.empty creates an empty, type-safe dependency container.
// Use Context.apply(...) to construct contexts with 1–10 values.
// The phantom type parameter tracks which types are present.
object ContextConstructionExample extends App {

  case class Config(debug: Boolean)
  case class Logger(name: String)
  case class Metrics(count: Int)

  // Start with an empty context.
  show(Context.empty.size)
  show(Context.empty.isEmpty)

  // Create a context with one value.
  val ctx1 = Context(Config(debug = true))
  show(ctx1.size)
  show(ctx1.nonEmpty)

  // Create a context with multiple values (up to 10 supported).
  val ctx2 = Context(
    Config(debug = false),
    Logger("myapp")
  )
  show(ctx2.size)
  show(ctx2.isEmpty)

  // Create a larger context.
  val ctx3 = Context(
    Config(debug = true),
    Logger("prod"),
    Metrics(count = 100)
  )
  show(ctx3.size)

  // Build incrementally from empty using add.
  val ctxBuilt = Context.empty
    .add(Config(debug = false))
    .add(Logger("init"))
    .add(Metrics(count = 0))
  show(ctxBuilt.size)
  show(ctxBuilt.nonEmpty)

  // toString shows the context contents in a readable format.
  show(ctx3.toString)
}
```

([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/context/ContextConstructionExample.scala))

```bash
cd schema-examples && sbt "runMain context.ContextConstructionExample"
```

**Context retrieval: using get, supertype lookups, and getOption for safe access**

```scala title="schema-examples/src/main/scala/context/ContextRetrievalExample.scala" 
package context

// Context#get[A] retrieves a value by type with compile-time proof of existence.
// Context#getOption[A] retrieves a value if present, returning None if missing.
// Supertype matching allows retrieving a subtype using a more general supertype.
object ContextRetrievalExample extends App {

  case class Config(debug: Boolean)
  case class Logger(name: String)
  case class Metrics(count: Int)

  trait Animal { def sound: String }
  case class Dog(name: String) extends Animal {
    def sound = "Woof"
  }

  // Create a context with multiple values.
  val ctx = Context(
    Config(debug = true),
    Logger("app"),
    Metrics(count = 42)
  )

  // Retrieve by exact type.
  val config: Config = ctx.get[Config]
  show(config)

  val logger: Logger = ctx.get[Logger]
  show(logger)

  val metrics: Metrics = ctx.get[Metrics]
  show(metrics)

  // getOption returns Some if the type is present.
  val configOpt: Option[Config] = ctx.getOption[Config]
  show(configOpt)

  // getOption returns None if the type is missing (safe for optional lookups).
  val missingOpt: Option[String] = ctx.getOption[String]
  show(missingOpt)

  // Supertype matching: retrieve a subtype using a supertype.
  val dogCtx         = Context(Dog("Buddy"))
  val animal: Animal = dogCtx.get[Animal]
  show(animal.sound)

  // getOption also supports supertype matching.
  val animalOpt: Option[Animal] = dogCtx.getOption[Animal]
  show(animalOpt.map(_.sound))

  // Cache efficiency: repeated lookups return the cached instance.
  val first  = ctx.get[Logger]
  val second = ctx.get[Logger]
  show(first eq second)
}
```

([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/context/ContextRetrievalExample.scala))

```bash
cd schema-examples && sbt "runMain context.ContextRetrievalExample"
```

**Context modification: adding values, updating existing ones, merging contexts, and pruning types**

```scala title="schema-examples/src/main/scala/context/ContextModificationExample.scala" 
package context

// Context is immutable; modification methods return new contexts.
// Context#add expands the context with a new value (or replaces if type exists).
// Context#update transforms a value if present.
// Context#++ merges two contexts (right side wins on conflict).
// Context#prune narrows to specific types.
object ContextModificationExample extends App {

  case class Config(debug: Boolean)
  case class Logger(name: String)
  case class Metrics(count: Int)

  // Start with a simple context.
  val ctx1 = Context(Config(debug = false))
  show(ctx1.size)

  // add expands the context with a new value.
  val ctx2 = ctx1.add(Logger("app"))
  show(ctx2.size)
  show(ctx2.get[Config])
  show(ctx2.get[Logger])

  // add replaces if the type already exists.
  val ctx3 = ctx2.add(Config(debug = true))
  show(ctx3.size)
  show(ctx3.get[Config].debug)

  // update transforms a value if present.
  val ctx4    = ctx3.add(Metrics(count = 100))
  val updated = ctx4.update[Metrics](m => m.copy(count = m.count + 50))
  show(updated.get[Metrics].count)

  // update also works on existing types.
  val configUpdated = updated.update[Config](c => c.copy(debug = false))
  show(configUpdated.get[Config])

  // ++ merges two contexts; right side wins on conflict.
  val left   = Context(Config(debug = false), Logger("left"))
  val right  = Context(Config(debug = true), Metrics(count = 99))
  val merged = left ++ right
  show(merged.size)
  show(merged.get[Config].debug)
  show(merged.get[Logger].name)
  show(merged.get[Metrics].count)

  // prune narrows a context to specific types.
  val full = Context(Config(true), Logger("app"), Metrics(100))
  show(full.size)
  val justConfig = full.prune[Config]
  show(justConfig.size)
  show(justConfig.getOption[Logger])

  // Chaining modifications.
  val chain = Context.empty
    .add(Config(debug = true))
    .add(Logger("chain"))
    .add(Metrics(count = 0))
    .update[Metrics](m => m.copy(count = 42))
  show(chain.size)
  show(chain.get[Metrics].count)
}
```

([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/context/ContextModificationExample.scala))

```bash
cd schema-examples && sbt "runMain context.ContextModificationExample"
```

## See Also

- [Telemetry Reference](./telemetry/index.md) — Its `OtelContext` bridge snapshots the active `SpanContext` into a `Context[R & OtelContext]`
- [Telemetry Guide](../guides/telemetry-guide.md) — Architecture and patterns for the telemetry module, including where `Context` fits in

---

## Data Migration

The `zio.blocks.data.migration` module provides three execution models for evolving database schemas online. It builds on [`Migration[A, B]`](./schema/migration.md) for the typed row transformation and [`Repo[E, ID]`](./sql/index.md) for database access. The row conversion is a typed `Migration`, not hand-written SQL, XML or Liquibase-style migration files; the schema changes themselves (DDL) are SQL statements you run, for example with `TinyMigrator`.

## Overview

Data migration in ZIO Blocks is split into three execution tiers, each suited to a different scale of change:

| Model       | Class           | When to use                                                            |
| ----------- | --------------- | ---------------------------------------------------------------------- |
| **A-Tiny**  | `TinyMigrator`  | DDL-only changes at startup (add column, rename table)                 |
| **A-Small** | `SmallMigrator` | Queue-based batch backfill for moderate tables                         |
| **B-Large** | `LargeMigrator` | Incremental worker with pause/resume and safe cutover for large tables |

All three share the same building blocks:

- `Migration[A, B]` from `zio.blocks.schema.migration` defines the typed transformation
- `Repo[E, ID]` from `zio.blocks.sql` provides database access for source and target
- `TargetStrategy` controls whether rows are updated in place or written to a shadow table
- `QueueTable` tracks dirty keys that need processing

The module is Scala 3 only.

## Execution Models

The examples on this page move a `users` table from `UserV1` rows to `UserV2` rows, which add an `age` column. Both repositories name the same table, `users`: the migrators read each row through the first and write it through the second, and for a shadow table they derive the shadow table's name from the second:

```scala

case class UserV1(id: Long, name: String)
case class UserV2(id: Long, name: String, age: Int)

object UserV1 { implicit val schema: Schema[UserV1] = Schema.derived }
object UserV2 { implicit val schema: Schema[UserV2] = Schema.derived }

val usersV1 = Repo.derived[UserV1, Long]("users", "id", _.id)
val usersV2 = Repo.derived[UserV2, Long]("users", "id", _.id)

val addAge: Migration[UserV1, UserV2] =
  Migration.newBuilder[UserV1, UserV2].addField(_.age, SchemaExpr.literal(0)).build
```

We keep `import zio.blocks.sql._` after the other `zio.blocks` imports: if your project also uses the `sql-zio` module, it brings a package named `zio` into scope, and a `zio.blocks` import after it fails to compile.

### A-Tiny: `TinyMigrator`

`TinyMigrator` runs DDL at application startup, in one transaction through `Transactor#transact`. Extend it, put the statements in `TinyMigrator#run`, and call `TinyMigrator#migrate`:

```scala
class AddAgeColumn(transactor: Transactor) extends TinyMigrator(transactor) {
  def run()(using DbTx): Unit =
    Frag.literal("ALTER TABLE users ADD COLUMN age INTEGER NOT NULL DEFAULT 0").update
}

def addAgeColumn(transactor: Transactor): Unit = AddAgeColumn(transactor).migrate()
```

Tiny migrations are appropriate when the schema change is fast, non-blocking, and can run before the application starts serving requests. They convert no rows.

### A-Small: `SmallMigrator`

`SmallMigrator` is a queue-based batch worker for moderate-sized tables. It reads dirty keys from a `QueueTable`, fetches the source rows, applies the `Migration[A, B]`, and writes the results to the target.

The lifecycle has three steps:

1. `SmallMigrator#init` prepares the target table: it creates the shadow table for `TargetStrategy.ShadowTable`, and installs capture triggers when `captureTriggers = true`. It doesn't create the queue table; `QueueTable.create` does.
2. `SmallMigrator#processBatch` claims a batch of keys and converts their rows in one transaction.
3. `SmallMigrator#complete` finalizes the migration (swaps the shadow table in, if there is one). It refuses to run while keys are still queued.

`SmallMigrator#processBatch` returns the number of rows it wrote, which is `0` for a batch whose rows were all deleted before it ran, even while more keys wait behind it. So we loop until `QueueTable.pending` finds the queue empty, not until a batch returns `0`. The following in-place backfill runs after `AddAgeColumn`: it queues every existing row, converts the rows, and completes:

```scala
def backfillInPlace()(using transactor: Transactor, dialect: Dialect): Unit = {
  QueueTable.create[Long]("users_age_q", transactor)
  transactor.transact(QueueTable.enqueue("users_age_q", usersV1.all.map(_.id)))

  val migrator = SmallMigrator[UserV1, UserV2, Long, Long](
    repoV1 = usersV1,
    repoV2 = usersV2,
    migration = addAge,
    queueTable = "users_age_q",
    batchSize = 500,
    target = TargetStrategy.InPlace
  )

  migrator.init()
  while (transactor.connect(QueueTable.pending("users_age_q")) > 0) migrator.processBatch()
  migrator.complete()
}
```

While an in-place backfill runs, the application queues the key of every row it writes with `QueueTable.enqueue`, in the same transaction as the write. Capture triggers can't do it here, because they would also queue the migrator's own writes (see [Queue Primitives](#queue-primitives)).

`SmallMigrator` supports both `TargetStrategy.InPlace` and `TargetStrategy.ShadowTable`.

### B-Large: `LargeMigrator`

`LargeMigrator` is an incremental worker designed for large tables where a full backfill would take too long. It adds `LargeMigrator#run` with `LargeMigrator#pause` and `LargeMigrator#resume`, concurrent workers on PostgreSQL via `SKIP LOCKED`, and a cutover protocol that converts every key still queued before the tables are swapped.

The lifecycle has five steps:

1. `LargeMigrator#init` prepares the target table and, with `captureTriggers = true`, installs the capture triggers.
2. `LargeMigrator#run` converts batches while the application keeps writing, until a batch writes no rows.
3. `LargeMigrator#fence` moves the migrator into its cutover state. It doesn't block writes: the application stops writing to the source table before calling it.
4. `LargeMigrator#drain` converts batches until a count finds the queue empty.
5. `LargeMigrator#complete` swaps the shadow table in and finalizes the migration.

The cutover is the last three steps. In 0.0.56, `LargeMigrator#complete` doesn't check the queue (a known bug; `SmallMigrator#complete` does), so we check `LargeMigrator#pendingCount` right before it:

```scala
def cutOver(migrator: LargeMigrator[UserV1, UserV2, Long, Long]): Unit = {
  // Every writer to `users` has stopped before this point
  migrator.fence()
  migrator.drain()
  require(migrator.pendingCount == 0, "a write reached the queue after drain(); stop all writers first")
  migrator.complete()
}
```

`LargeMigrator` is typically used with `TargetStrategy.ShadowTable` and capture triggers. [Full Example](#full-example) puts every step into one program for PostgreSQL.

## Target Strategies

`TargetStrategy` controls where migrated rows are written. It has two cases.

`TargetStrategy.InPlace` updates rows in the table that both repositories name. The table needs the new columns before the first batch, for example from a `TinyMigrator`, so it suits changes where the old and new columns can live side by side for a while.

`TargetStrategy.ShadowTable(suffix)` writes migrated rows into `_<suffix>`, where `` is the V2 repository's table, and at completion renames `` to `_old_<suffix>` and the shadow table to ``. `init()` creates the shadow table with `CREATE TABLE IF NOT EXISTS _<suffix> (LIKE  INCLUDING ALL)`, a copy of the table's current columns, not of the V2 columns. When the migration changes columns, create the shadow table yourself with the new columns before `init()`; `IF NOT EXISTS` then leaves it alone. Shadow tables need PostgreSQL.

With `usersV2` as the V2 repository, the two strategies look like this:

```scala
val inPlace: TargetStrategy = TargetStrategy.InPlace
val shadow: TargetStrategy  = TargetStrategy.ShadowTable("v2") // shadow table: users_v2
```

## Completion Protocol (B-Large)

`LargeMigrator` keeps its cutover state in memory and moves through four states:

```
Initialized → Fenced → Drained → Completed
```

A new migrator starts in `Initialized`. `fence()`, `drain()` and `complete()` each move it to the next state; `init()` leaves the state alone and records that the migrator is prepared, which `fence()` requires.

### `LargeMigrator#init` (Initialized)

Prepares the target table. For `TargetStrategy.ShadowTable`, it creates the shadow table, as described in [Target Strategies](#target-strategies), and with `captureTriggers = true` it installs the capture triggers. The source table continues to accept writes normally. `init()` doesn't create the queue table, and it doesn't queue the rows that already exist: the triggers see only the writes made after `init()`. Create the queue with `QueueTable.create` before `init()`, and queue the existing keys after it, so that no write falls between the two.

### `LargeMigrator#fence` (Fenced)

Moves the migrator into its cutover state. It changes nothing in the database and blocks no write: the application **must** stop writing to the source table before calling `fence()`. A write that still arrives is queued by its trigger, and `drain()` converts it if it lands before `drain()` returns.

### `LargeMigrator#drain` (Drained)

Converts batches until a `COUNT` of the queue table finds it empty, then returns the number of rows it wrote. If batches come back empty while keys stay queued for 100 rounds in a row, for example because another worker holds their rows locked, `drain()` throws an `IllegalStateException`. A call to `pause()` makes `drain()` return early, and the migrator stays fenced.

### `LargeMigrator#complete` (Completed)

For `TargetStrategy.ShadowTable`, renames the source table to `_old_<suffix>` and the shadow table to `` in one transaction; for `TargetStrategy.InPlace`, it changes no table. In 0.0.56, `complete()` doesn't check the queue (a known bug), so a write that lands after `drain()` returns is missing from the new table: check `LargeMigrator#pendingCount` first. The migrator leaves the old table, the queue table and the trigger function behind; drop them after verifying the result.

The protocol converts every key that is in the queue when `drain()` makes its final count. Rows that existed before `init()` are converted only if their keys were queued, and writes that land after `drain()` returns are not converted. The queue table survives a crash, so a restarted worker continues from the keys still queued. The cutover state does not: a new `LargeMigrator` starts in `Initialized` and needs its own `init()`, `fence()`, `drain()` and `complete()`. `init()` is safe to run again, because the shadow table's DDL uses `CREATE TABLE IF NOT EXISTS` and the triggers use `CREATE OR REPLACE TRIGGER`.

## ID Type Flexibility

`LargeMigrator` and `SmallMigrator` accept separate `ID1` and `ID2` type parameters for the source and target repos, but construction requires evidence that they are the same type (`ID1 =:= ID2`), as the examples above do with `SmallMigrator[UserV1, UserV2, Long, Long]`.

The queue table stores V1 IDs (`ID1`). The `ID1` type must have a `DbCodec` instance available as a given. The equality constraint keeps delete propagation type-safe: when a source row disappears between dequeue and read, the same key is reinterpreted for the target repo via the type evidence rather than a cast. Migrations that change the primary key type are out of scope for the built-in migrators.

## Queue Primitives

`QueueTable` is the dirty-key tracking mechanism shared by `SmallMigrator` and `LargeMigrator`. The queue table has three columns:

| Column    | Type                        | Description                                                                   |
| --------- | --------------------------- | ----------------------------------------------------------------------------- |
| `id`      | `TEXT NOT NULL PRIMARY KEY` | The primary key of the affected source row                                    |
| `op`      | `TEXT NOT NULL DEFAULT 'I'` | The operation type: `'I'` (insert), `'U'` (update), or `'D'` (delete)         |
| `payload` | `TEXT`                      | JSON-serialized V1 row data, captured only for `'D'` operations on PostgreSQL |

Queue entries are coalesced by primary key. Keys enter the queue in two ways. The application can call `QueueTable.enqueue` for every row it writes, or for a batch of rows to backfill. Alternatively, pass `captureTriggers = true` when constructing `SmallMigrator` or `LargeMigrator`: `init()` then installs triggers via `QueueTable.installTriggers` so every source INSERT/UPDATE/DELETE upserts the affected key in the writer's own transaction. Trigger installation is idempotent (PostgreSQL requires PG 14+ for `CREATE OR REPLACE TRIGGER`). Do not enable capture triggers when the migrator writes to the same physical table it captures from: the worker's own writes would re-enqueue processed keys forever.

For `INSERT` and `UPDATE`, only the key and operation type are stored. For `DELETE`, PostgreSQL additionally captures the full row as JSON via `row_to_json(OLD)::text` into the `payload` column; the payload is reserved for future delete-recovery and is not read by the current workers. SQLite does not support `row_to_json`, so its delete entries carry no payload.

| Operation                    | Description                                                 |
| ---------------------------- | ----------------------------------------------------------- |
| `QueueTable.create`          | Creates the queue table (id, op, payload columns)           |
| `QueueTable.enqueue`         | Inserts the IDs into the queue, skipping IDs already queued |
| `QueueTable.dequeue`         | Claims and removes a batch of IDs, returning `List[ID]`     |
| `QueueTable.pending`         | Returns the number of unprocessed keys                      |
| `QueueTable.installTriggers` | Installs capture triggers on the source table               |

PostgreSQL uses `FOR UPDATE SKIP LOCKED` for concurrent worker claims. SQLite uses a single consumer with `BEGIN IMMEDIATE` and a busy timeout. Queue IDs are stored as TEXT, so batches are claimed in lexicographic order (`'10'` before `'9'`); this affects claim order only, never correctness.

A worker claims the dirty key and, in one transaction, looks up the source row by ID, applies the `Migration[A, B]`, and writes the result to the target. If the source row is missing at processing time (deleted between enqueue and dequeue), the corresponding target row is deleted — `findAll` returns only rows that still exist.

## Database Support

| Feature                                        | PostgreSQL         | SQLite              |
| ---------------------------------------------- | ------------------ | ------------------- |
| `TinyMigrator`                                 | Yes                | Yes                 |
| `SmallMigrator`                                | Yes                | Yes                 |
| `LargeMigrator`                                | Yes (multi-worker) | Yes (single-worker) |
| `SKIP LOCKED`                                  | Yes                | No                  |
| `ShadowTable` (CREATE TABLE LIKE, atomic swap) | Yes                | No                  |
| `InPlace` updates                              | Yes                | Yes                 |

PostgreSQL supports the full feature set including concurrent workers, shadow table creation via `CREATE TABLE LIKE`, and atomic DDL swap. SQLite supports queue primitives and in-place updates but does not support shadow tables or concurrent workers.

## Contextual Requirements

`TinyMigrator` takes a `Transactor` as a constructor parameter. `SmallMigrator` and `LargeMigrator` take four context parameters after their arguments: the `Transactor`, a `DbCodec[ID1]` for the queue table's key column, the `Dialect` whose SQL they generate, and the evidence that `ID1 =:= ID2`:

```scala
final class LargeMigrator[A, B, ID1, ID2](
  repoV1: Repo[A, ID1],
  repoV2: Repo[B, ID2],
  migration: Migration[A, B],
  queueTable: String,
  batchSize: Int,
  target: TargetStrategy,
  captureTriggers: Boolean = false
)(using transactor: Transactor, codecId: DbCodec[ID1], dialect: Dialect, ev: ID1 =:= ID2)
```

`SmallMigrator` has the same parameters.

## Full Example

The following program migrates the live `users` table on PostgreSQL to the `UserV2` shape through a shadow table, while the application keeps writing until the cutover. It creates the shadow table with the new columns and the queue table first, installs the capture triggers with `init()`, queues the rows that already exist, copies them with `run()`, and cuts over once the application's writers have stopped:

```scala

case class UserV1(id: Long, name: String)
case class UserV2(id: Long, name: String, age: Int)

object UserV1 { implicit val schema: Schema[UserV1] = Schema.derived }
object UserV2 { implicit val schema: Schema[UserV2] = Schema.derived }

// Both repositories name the live table
val usersV1 = Repo.derived[UserV1, Long]("users", "id", _.id)
val usersV2 = Repo.derived[UserV2, Long]("users", "id", _.id)

val addAge: Migration[UserV1, UserV2] =
  Migration.newBuilder[UserV1, UserV2].addField(_.age, SchemaExpr.literal(0)).build

def migrateUsers(stopWriters: () => Unit): Unit = {
  given transactor: Transactor =
    JdbcTransactor.fromUrl("jdbc:postgresql://localhost:5432/shop", "shop", "shop", SqlDialect.PostgreSQL)
  given Dialect = Dialect.Postgres

  // 1. The shadow table, with the V2 columns, and the queue table
  transactor.transact {
    Frag
      .literal("CREATE TABLE users_v2 (id BIGINT PRIMARY KEY, name TEXT NOT NULL, age INTEGER NOT NULL)")
      .update
  }
  QueueTable.create[Long]("users_age_q", transactor)

  val migrator = LargeMigrator[UserV1, UserV2, Long, Long](
    repoV1 = usersV1,
    repoV2 = usersV2,
    migration = addAge,
    queueTable = "users_age_q",
    batchSize = 500,
    target = TargetStrategy.ShadowTable("v2"),
    captureTriggers = true
  )

  // 2. Install the triggers, then queue the rows that already exist
  migrator.init()
  transactor.transact {
    Frag.literal("INSERT INTO users_age_q (id) SELECT CAST(id AS TEXT) FROM users ON CONFLICT DO NOTHING").update
  }

  // 3. Copy while the application writes; the triggers queue its writes
  migrator.run()

  // 4. Cut over: stop the writers, fence, drain, check the queue, swap
  stopWriters()
  migrator.fence()
  migrator.drain()
  require(migrator.pendingCount == 0, "a write reached the queue after drain(); stop all writers first")
  migrator.complete()
}
```

After `complete()`, `users` holds `UserV2` rows and the old rows are in `users_old_v2`. The queue stores keys as text, hence the `CAST`, and `ON CONFLICT DO NOTHING` skips a key that a trigger queued first. [How to Migrate Table Rows to a New Version Online](../guides/migrate-table-rows-online.md) runs this flow against PostgreSQL 16 with a writer thread, shows its output, and drops the old table, the queue table and the trigger function afterwards.

## Failure Policy

When converting a row fails, the migrator throws a `RuntimeException` (`Migration failed: ...`); a database error propagates as its own exception. Either way, the batch's transaction rolls back and every key of the batch stays in the queue, including the keys of rows that converted fine. No data is written for that batch, and no automatic retry or dead-letter configuration is applied: fix the cause, for example with a corrected migration in a new migrator, and process the queue again. The guide's [Recover from a Failed Batch and Restart](../guides/migrate-table-rows-online.md#recover-from-a-failed-batch-and-restart) section shows this.

## Architecture Decisions

For a detailed record of architecture decisions, see the [ADR](../adr/2026-07-18-data-migration.md).

## See Also

- [How to Migrate Table Rows to a New Version Online](../guides/migrate-table-rows-online.md) — a how-to guide that moves a table to a new row shape while the application keeps running.
- [Projection](./projection/index.md) — event-sourced read models that complement schema-migration workflows: while `DataMigration` evolves the write-side schema, `ProjectionEngine` rebuilds read-side views from the event log without touching source tables.
- [SQL Module](./sql/index.md) — the `Transactor`, `Repo`, and `Table` types shared by both the data-migration and projection modules.

---

## Data Types

## Introduction

The blocks in this group are small, general-purpose building blocks. Two of them offer alternatives to everyday standard-library types that allocate less, and the third checks the composition of tuples, eithers and unions at compile time.

## The Blocks

The group has three blocks.

| Block                          | Artifact                   | What It Is For                                                                                  |
| ------------------------------ | -------------------------- | ----------------------------------------------------------------------------------------------- |
| [Chunk](../chunk.md)           | `zio-blocks-chunk`         | High-performance immutable indexed sequences with zero-boxing builders                          |
| [Maybe](../maybe.md)           | `zio-blocks-maybe`         | Optional values that store a present reference as is, without a wrapper object (Scala 3)        |
| [Combinators](../combinators.md) | `zio-blocks-combinators` | Compile-time composition and decomposition of tuples, eithers, and unions                       |

## How the Blocks Differ

`Chunk` is an immutable, indexed sequence with a functional interface. It stays fast for element access and is lazy for expensive operations such as repeated concatenation.

`Maybe` is a low-allocation alternative to `Option`. On Scala 3 it is an opaque type, so a present reference is stored as it is rather than inside a wrapper object, and only `null` and nested `Maybe` values are wrapped in `Present`. On Scala 2.13 it is a sealed trait and allocates like `Option`.

`Combinators` provides compile-time typeclasses for composing and decomposing tuples, `Either` types and union types, which is useful for code that must work with many shapes of the same operation.

## Where to Start

Start with [Chunk](../chunk.md) for sequences. Use [Maybe](../maybe.md) where allocation matters and the code is on Scala 3, and [Combinators](../combinators.md) when writing generic code over tuples or eithers.

---

## Datastar Attributes

`DatastarAttributes` supplies 27 `data*` helpers that make rendered HTML reactive. Each returns either a finished `Dom.Attribute` or a `DatastarAttrKey` awaiting a value, and `ToDatastarExpr` decides what may be assigned — rejecting raw `String` at compile time. The package object extends this trait, so one wildcard import brings every helper into scope. The two types every helper funnels through:

```scala
final class DatastarAttrKey(val name: String) {
  def :=[T](value: T)(implicit toDatastarExpr: ToDatastarExpr[T]): Dom.Attribute
}

trait ToDatastarExpr[-A] {
  def toDatastarExpr(a: A): String
}
```

## Motivation

Datastar's whole interface is attribute names and expression strings. `data-text="$count"` displays a signal; `data-show="$count > 0"` conditionally renders; `data-class:active="$selected"` toggles a class. Getting one character wrong produces an attribute the framework ignores, with no error anywhere.

The failure that costs the most time is subtler than a typo. Attribute values are *expressions*, evaluated in the browser — `"$count"` reads a signal, but `"count"` is a string literal that happens to look right. Interpolating a Scala `String` into an expression position produces exactly that: valid HTML, ignored semantics, a blank element.

These helpers make the attribute name a method call and the value type-checked. There is no attribute name to misspell, and `ToDatastarExpr` is deliberately unavailable for `String`, so the one mistake that fails silently in the browser fails loudly at compile time instead.

## Quick Showcase

Attributes compose with the HTML DSL like any other:

```scala

val open = Signal[Boolean]("open")

val panel = div(
  dataSignals(open := false),
  button(dataOn.click := js"$open = !$open", "toggle"),
  div(dataShow := open, dataClass("visible") := open, "panel contents")
)
```

Rendering produces the attribute names and expressions Datastar expects:

```scala
panel.renderMinified
// res0: String = "<button data-on:click=\"$open = !$open\">toggle</button>panel contents"
```

## Attribute Naming

Two shapes appear in the rendered output, and which one a helper produces tells you whether it takes a key.

| Helper form | Renders as | Example |
| ----------- | ---------- | ------- |
| No key | `data-<name>` | `dataText` → `data-text` |
| Keyed | `data-<name>:<key>` | `dataClass("active")` → `data-class:active` |
| Own attribute | `data-<multi-word-name>` | `dataOnIntersect` → `data-on-intersect` |
| Modified | `…__<modifier>` | `dataOn.click.once` → `data-on:click__once` |

Keys derived from signal names are kebab-cased, so a `Signal[Int]("itemCount")` used as a `dataComputed` key becomes `data-computed:item-count`.

The third row is the one that breaks the pattern. A DOM event is a *key* on `data-on`, giving `data-on:click`, but the non-DOM triggers are attribute names in their own right — `data-on-intersect`, `data-on-interval`, `data-on-signal-patch` — with hyphens and no key. Their modifiers still attach with `__`.

## Declaring State

Two helpers put signals on the page.

### `dataSignals` — initial values

`dataSignals` has three forms. Given one or more `SignalUpdate`s it renders the whole object as the attribute value:

```scala

val price    = Signal[Double]("price")
val quantity = Signal[Int]("quantity")
```

Several updates become a single `data-signals` attribute:

```scala
div(dataSignals(price := 9.99, quantity := 1)).renderMinified
// res2: String = ""
```

Given a single `Signal`, it returns a `DataSignalsBuilder` for the keyed form — `data-signals:<name>` — which sets one signal rather than an object:

```scala
div(dataSignals(price) := js"42.0").renderMinified
// res3: String = ""
```

The builder also carries the case modifiers described below. The third form, bare `dataSignals`, is a `DatastarAttrKey` for assigning a raw expression.

### `dataComputed` — derived signals

`dataComputed(signal)` declares a signal whose value is an expression over others, recomputed by the browser whenever an input changes:

```scala
div(dataComputed(Signal[Double]("total")) := js"$price * $quantity").renderMinified
// res4: String = ""
```

The signal name becomes the attribute key, kebab-cased.

## Displaying and Binding

Three helpers read a signal and change what the element shows, without any handler being involved.

### `dataText` and `dataShow`

`dataText` sets an element's text content from an expression, and `dataShow` controls its visibility:

```scala

val count = Signal[Int]("count")
```

Both take an expression, so a signal or a `js"..."` both work:

```scala
span(dataText := count).renderMinified
// res6: String = ""
div(dataShow := js"$count > 0", "non-empty").renderMinified
// res7: String = "non-empty"
```

### `dataBind` — two-way input binding

`dataBind(signal)` binds a form control to a signal in both directions, and needs no value:

```scala
input(dataBind(count)).renderMinified
// res8: String = "<input data-bind:count/>"
```

The rendered attribute is `data-bind:<name>` with no value, which is how Datastar recognizes the binding form.

## Styling

Three helpers follow the same keyed-or-bare pattern: a key selects what to modify, and the expression decides when.

| Helper | Keyed form | Purpose |
| ------ | ---------- | ------- |
| `dataClass(name)` | `data-class:<name>` | Toggle one class |
| `dataClass` | `data-class` | Object of class names to conditions |
| `dataStyle(name)` | `data-style:<name>` | Set one CSS property |
| `dataStyle` | `data-style` | Object of properties to values |
| `dataAttr(name)` | `data-attr:<name>` | Set one HTML attribute |
| `dataAttr` | `data-attr` | Object of attributes to values |

The keyed form is the common one, toggling a single class from a condition:

```scala

val selected = Signal[Boolean]("selected")
```

Keyed and bare forms render differently, and the bare form takes an object expression:

```scala
div(dataClass("active") := selected).renderMinified
// res10: String = ""
div(dataStyle("color") := js"$selected ? 'red' : 'gray'").renderMinified
// res11: String = ""
```

## Effects and References

`dataEffect` runs an expression whenever its dependencies change, `dataIndicator(signal)` sets a signal while a request is in flight, and `dataRef(name)` exposes the element to expressions by name:

```scala

val loading = Signal[Boolean]("loading")
```

`dataIndicator` and `dataRef` are complete attributes rather than keys, since their argument is the whole content:

```scala
button(dataIndicator(loading), "save").renderMinified
// res13: String = "<button data-indicator:loading>save</button>"
div(dataRef("panel")).renderMinified
// res14: String = ""
```

## Morph Control

When the server patches elements, Datastar morphs the existing DOM rather than replacing it. Four helpers control what that morph may touch — all of them complete attributes taking no expression:

| Helper | Effect |
| ------ | ------ |
| `dataIgnore` | Datastar ignores this element and its subtree entirely |
| `dataIgnoreSelf` | Ignores this element but still processes its children |
| `dataIgnoreMorph` | Preserves this element across morphs |
| `dataPreserveAttr(attrs*)` | Preserves the named attributes across morphs |

Preserving an attribute matters for state the server does not know about — a scroll position, an open `details`, a user-resized width:

```scala

```

The first three are boolean attributes; the fourth names what to keep:

```scala
div(dataIgnore).renderMinified
// res16: String = ""
div(dataPreserveAttr("open", "style")).renderMinified
// res17: String = ""
```

## Other Attributes

`dataJsonSignals` renders the full signal state as JSON, which is useful for debugging a page's reactive state, and `dataOnSignalPatchFilter` narrows which signal patches a `data-on-signal-patch` handler responds to. Both are `DatastarAttrKey`s taking an expression.

The trigger helpers — `dataOn`, `dataInit`, `dataOnIntersect`, `dataOnInterval`, `dataOnSignalPatch` — are covered in [Event Handlers](./events.md).

## ToDatastarExpr

`ToDatastarExpr[A]` is the type class deciding what may be assigned with `:=`. It is contravariant, and instances derive from `ToJs[A]`, so anything the HTML module can render as JavaScript works here.

### What Is Accepted

An instance exists for any `A` with a `ToJs[A]` — which covers `Js` values from the `js"..."` interpolator, `Signal[A]`, `SignalUpdate[A]`, and `DatastarRef`:

```scala

val count = Signal[Int]("count")
```

Each renders to its expression form:

```scala
span(dataText := count).renderMinified
// res19: String = ""
span(dataText := js"$count + 1").renderMinified
// res20: String = ""
```

### Why `String` Is Rejected

There is deliberately **no** usable instance for `String`. The mechanism is a private `NotString` witness with two conflicting instances for `String`, which makes the implicit search ambiguous, plus an `@implicitAmbiguous` annotation supplying the message:

```scala
@implicitAmbiguous(
  "Raw String values are not allowed in Datastar expression positions. " +
  "Use js\"...\" for Datastar expressions or typed Signal/SignalUpdate values."
)
private sealed trait NotString[A]
```

The result is that `dataText := "count"` does not compile, and the error tells you to use `js"..."` instead.

:::tip[This is the module's most valuable guard]
Assigning a raw `String` is the one Datastar mistake that produces valid HTML and no error at runtime — the attribute renders, the browser reads it as a literal, and the element silently shows nothing useful. Making it a compile error is worth the unusual implicit machinery.
:::

`@implicitNotFound` covers the other case: a type with no `ToJs` instance at all gets a message naming the type and pointing at the same two options.

## DatastarAttrKey

`DatastarAttrKey` is the low-level escape hatch — a raw attribute name plus `:=`. Every keyed helper returns one, and you can construct one for an attribute the module has no helper for yet.

`new DatastarAttrKey(name)` makes a key for an attribute that has no helper. The helpers `dataAttr`, `dataClass`, `dataStyle`, `dataText`, `dataShow`, `dataEffect`, `dataSignals`, `dataJsonSignals`, and `dataOnSignalPatchFilter` all have bare forms:

```scala

val theme = Signal[String]("theme")
```

The bare form takes an object expression covering several keys at once:

```scala
div(dataStyle := js"{color: $theme}").renderMinified
// res22: String = ""
```

:::warning[`dataAttr` collides with the HTML module]
`zio.blocks.html` also defines `dataAttr`, for plain static `data-*` attributes — `dataAttr("id") := "42"` renders `data-id="42"`. Datastar's renders `data-attr:id="<expression>"`, a reactive binding. With both packages wildcard-imported, the bare name is ambiguous and will not compile.

Qualify the one you mean, `zio.http.datastar.dataAttr` for the reactive form, or import selectively. The ambiguity error is the good case: it stops you silently getting a static attribute where you wanted a binding.
:::

## Integration Points

Every helper returns a `Dom.Attribute` from [HTML](../html.md), so Datastar attributes are indistinguishable from `id` or `class` at the point of use and compose in the same element constructors. `ToDatastarExpr` derives from that module's `ToJs`, and the `js"..."` interpolator producing most expression values is also its.

Values assigned here are usually [Signals](./signals.md), and the triggers that pair with these attributes are in [Event Handlers](./events.md). What the server sends back to change them is in [Server-Sent Events](./sse.md).

---

## Event Handlers

`dataOn` opens the event side of the attribute DSL: sixteen predefined DOM events, fourteen chainable modifiers, and four case modifiers, all rendering into a single `data-on:<event>__<modifiers>` attribute. Four sibling builders cover triggers that are not DOM events — intersection, interval, signal patches, and initialization. Core types: `DataOn`, `PartialDataOn`, `EventModifier`, `CaseModifier`, `DataOnIntersect`, `DataOnInterval`, `DataOnSignalPatch`, `DataInit`. The shape of the builder:

```scala
final class DataOn private (name: String, modifiers: Maybe[EventModifier], caseModifier: CaseModifier) {
  def debounce(millis: Long): DataOn
  def once: DataOn
  def :=[T](value: T)(implicit toDatastarExpr: ToDatastarExpr[T]): Dom.Attribute
}
```

## Motivation

An event handler in Datastar is an attribute whose name encodes both the event and its options. `data-on:input__debounce.300ms` debounces; `data-on:click__once__prevent` fires once and calls `preventDefault`. The name is structured, order-insensitive between modifiers, and entirely stringly-typed on the wire.

Written by hand that is a lot of punctuation to get right, and a modifier that Datastar does not recognize is silently ignored — the handler still fires, just without the debouncing you thought you had. A search box that issues a request per keystroke looks like it works, right up until it doesn't.

`dataOn` turns the whole name into a method chain. The event is a method, each modifier is a method, and the attribute name is assembled for you. A modifier that does not exist is a compile error, and the rendered name is correct by construction.

## Quick Showcase

Chain modifiers before assigning the handler:

```scala

val term = Signal[String]("term")

val box = input(
  dataBind(term),
  dataOn.input.debounce(300) := js"@get('/search')"
)
```

The modifier becomes part of the attribute name:

```scala
box.renderMinified
// res0: String = "<input data-bind:term data-on:input__debounce.300ms=\"@get(&#x27;/search&#x27;)\"/>"
```

## Predefined Events

Sixteen events have dedicated methods, each returning a `DataOn` ready for modifiers or assignment:

| Category | Methods |
| -------- | ------- |
| Pointer | `dataOn.click`, `dataOn.mouseover`, `dataOn.mouseout`, `dataOn.mouseenter`, `dataOn.mouseleave` |
| Keyboard | `dataOn.keydown`, `dataOn.keyup`, `dataOn.keypress` |
| Form | `dataOn.submit`, `dataOn.input`, `dataOn.change`, `dataOn.focus`, `dataOn.blur` |
| Window | `dataOn.scroll`, `dataOn.resize`, `dataOn.load` |

Each renders as `data-on:<event>`:

```scala

```

The method name is the event name — `dataOn.click`, `dataOn.submit`, `dataOn.input` — so nothing needs looking up:

```scala
button(dataOn.click := js"@post('/save')").renderMinified
// res2: String = "<button data-on:click=\"@post(&#x27;/save&#x27;)\"></button>"
form(dataOn.submit := js"@post('/submit')").renderMinified
// res3: String = "<form data-on:submit=\"@post(&#x27;/submit&#x27;)\"></form>"
```

### `dataOn.apply` — any other event

For an event without a dedicated method, `dataOn(name)` takes it as a string. The name is validated, so a structurally invalid event name fails rather than rendering a broken attribute:

```scala
div(dataOn("animationend") := js"@get('/done')").renderMinified
// res4: String = ""
```

`dataOn` on its own is a `PartialDataOn` — the builder that the event methods live on — so `dataOn` alone is not a complete attribute.

## Event Modifiers

Fourteen modifiers chain before `:=`, each contributing a `__` suffix.

| Method | Renders | Effect |
| ------ | ------- | ------ |
| `debounce(ms)` | `__debounce.<ms>ms` | Fire after quiet period |
| `debounceLeading(ms)` | `__debounce.<ms>ms.leading` | Fire immediately, then suppress |
| `throttle(ms)` | `__throttle.<ms>ms` | At most once per interval |
| `throttleLeading(ms)` | `__throttle.<ms>ms.leading` | Leading-edge throttle |
| `delay(ms)` | `__delay.<ms>ms` | Wait before firing |
| `DataOn#once` | `__once` | Fire at most once |
| `DataOn#passive` | `__passive` | Passive listener |
| `DataOn#capture` | `__capture` | Capture phase |
| `DataOn#stop` | `__stop` | `stopPropagation` |
| `DataOn#prevent` | `__prevent` | `preventDefault` |
| `DataOn#outside` | `__outside` | Fire on events outside the element |
| `DataOn#window` | `__window` | Listen on the window object |
| `DataOn#document` | `__document` | Listen on the document object |
| `DataOn#viewTransition` | `__viewTransition` | Wrap the resulting patch in a view transition |

Modifiers combine, and the rendered name carries each one:

```scala

```

A click that fires once and suppresses the default action needs two, `DataOn#once` and `DataOn#prevent`:

```scala
button(dataOn.click.once.prevent := js"@post('/subscribe')").renderMinified
// res6: String = "<button data-on:click__once__prevent=\"@post(&#x27;/subscribe&#x27;)\"></button>"
```

`DataOn#outside` is the one worth knowing about: it inverts the target, firing when the event happens anywhere *but* this element — which is how a dropdown closes when you click away:

```scala
div(dataOn.click.outside := js"$$open = false").renderMinified
// res7: String = ""
```

Repeating a timing or target modifier does not stack: the chain is normalized to the last effective value before the name is rendered, so chaining `DataOn#debounce` twice keeps only the second value.

Modifiers are represented by the `EventModifier` ADT — `Debounce`, `Throttle`, `Delay`, `Once`, `Passive`, `Capture`, `Stop`, `Prevent`, `Outside`, `Window`, `Document`, `ViewTransition`, and `And` for combination — but the builder methods are the intended interface.

Each trigger has its own modifier ADT, and they are not interchangeable: `EventModifier` for `dataOn`, `IntersectModifier` for `dataOnIntersect` (which adds `Half`, `Full`, `Exit`, and `Threshold`), `OnIntervalModifier` for `dataOnInterval` (`Duration`, `ViewTransition`), `OnSignalPatchModifier` for `dataOnSignalPatch` (`Delay`, `Debounce`, `Throttle`), and `InitModifier` for `dataInit` (`Delay`, `ViewTransition`). Every one of them has an `And` variant, which is how a chain of builder calls accumulates. You never need to construct these directly — the builder methods do it — but they are what a modifier chain is made of.

## Case Modifiers

`CaseModifier` controls how the event name is cased in the rendered attribute. Its four variants — `Camel`, `Kebab`, `Snake`, and `Pascal` — are selected by the same-named builder methods `DataOn#camel`, `DataOn#kebab`, `DataOn#snake`, and `DataOn#pascal`. The suffix appears only when the requested case differs from the builder's default, which for `dataOn` is kebab:

```scala

```

A custom event whose real name is camelCase needs the modifier. Note what it actually does: the attribute key stays kebab-cased either way, and the modifier is what tells Datastar to convert it back:

```scala
div(dataOn("myCustomEvent").camel := js"@get('/handle')").renderMinified
// res9: String = ""
div(dataOn("myCustomEvent") := js"@get('/handle')").renderMinified
// res10: String = ""
```

So `DataOn#camel` does not change the rendered key — it appends `__case.camel` alongside it. The default differs per builder: `dataOn` defaults to kebab, while `dataSignals(signal)` defaults to camel, and a modifier matching the default is omitted. That is why the same case call is a no-op on one and meaningful on the other.

Note also that the non-DOM triggers render as their own hyphenated attribute names — `data-on-intersect`, not `data-on:intersect` — since the trigger is the attribute rather than a key on it.

## Non-DOM Triggers

Four builders fire on something other than a DOM event. Each has its own modifier set and its own `:=`.

### `dataOnIntersect` — element enters the viewport

Fires when the element becomes visible, which is the basis for infinite scroll and lazy loading:

| Modifier | Effect |
| -------- | ------ |
| `DataOnIntersect#once` | Fire only the first time |
| `half` | Require 50% visibility |
| `full` | Require 100% visibility |
| `exit` | Fire on leaving rather than entering |
| `threshold(pct)` | Require `pct` percent visibility, as a whole number such as `25` |
| `delay(ms)`, `debounce(ms)`, `throttle(ms)` | Timing control |
| `DataOnIntersect#viewTransition` | Wrap the patch in a view transition |

A sentinel element at the end of a list is the canonical use:

```scala

```

Loading the next page once, when the sentinel is fully visible:

```scala
div(dataOnIntersect.once.full := js"@get('/page/2')").renderMinified
// res12: String = ""
```

### `dataOnInterval` — fire on a timer

Polls on an interval, with `DataOnInterval#duration` setting the period, `DataOnInterval#durationLeading` firing immediately as well, and `DataOnInterval#viewTransition` wrapping the resulting patch:

```scala
div(dataOnInterval.duration(5000) := js"@get('/status')").renderMinified
// res13: String = ""
```

Without a `DataOnInterval#duration`, the attribute renders bare and Datastar applies its default interval.

### `dataOnSignalPatch` — react to signal changes

Fires when signals are patched, with `DataOnSignalPatch#delay`, `DataOnSignalPatch#debounce`, and `DataOnSignalPatch#throttle` for timing. Pair it with `dataOnSignalPatchFilter` to narrow which signals count:

```scala
div(dataOnSignalPatch.debounce(200) := js"@get('/recalculate')").renderMinified
// res14: String = ""
```

### `dataInit` — fire once on load

Runs when Datastar first processes the element, with `DataInit#delay` and `DataInit#viewTransition`:

```scala
div(dataInit := js"@get('/bootstrap')").renderMinified
// res15: String = ""
```

This is the hook for fetching initial content that is too expensive to render server-side on first paint.

## Choosing a Trigger

| You want to react to | Use |
| -------------------- | --- |
| A user interaction | `dataOn.<event>` |
| An event with no dedicated method | `dataOn("name")` |
| The element scrolling into view | `dataOnIntersect` |
| The passage of time | `dataOnInterval` |
| Signal state changing | `dataOnSignalPatch` |
| The page loading | `dataInit` |

## Integration Points

Every builder here ends in `:=`, which goes through `ToDatastarExpr` and returns a `Dom.Attribute` — see [Attributes](./attributes.md) for that type class and why raw `String` values are rejected. Handler expressions typically read and write [Signals](./signals.md) and call Datastar's `@get`/`@post` actions, which the server answers with [Server-Sent Events](./sse.md).

The `js"..."` interpolator producing those expressions, and the `Dom` types the attributes attach to, come from [HTML](../html.md).

---

## Datastar

`zio.http.datastar` builds [Datastar](https://data-star.dev) hypermedia applications: it renders the `data-*` attributes that make a page reactive, and it produces the SSE events that patch that page from the server. Core types: `Signal`, `SignalUpdate`, `DatastarEvent`, `DataOn`, `DatastarAttrKey`, `ToDatastarExpr`, `ElementPatchMode`. The two halves of the module:

```scala
final class Signal[A] private (val name: String) {
  def :=(value: A)(implicit schema: Schema[A]): SignalUpdate[A]
  def ref: DatastarRef
}

sealed trait DatastarEvent {
  def renderSSE: String
}
```

## Introduction

Datastar puts application state in the browser as named *signals*, and drives changes to it declaratively. A page says what it reacts to — "when this button is clicked, POST to `/increment`" — using `data-*` attributes, and the server replies with events that patch elements or signals in place. No client-side framework code, no JSON API to design, no separate view model.

This module is the Scala side of that contract. It gives you a typed way to write the attributes, a typed way to name and update signals, and a builder for each kind of SSE event, so both directions of the exchange are checked by the compiler rather than assembled from strings.

Two things it deliberately does not do: it does not ship a server, and it does not ship the Datastar JavaScript. It produces `Dom` attributes and SSE payload strings, which you serve with whatever HTTP layer you already have.

## Motivation

The natural way to write Datastar from a server language is string interpolation — `s"data-on-click=\"@post('/increment')\""` — and it goes wrong in the usual ways. A misspelled signal name fails silently in the browser. An attribute name typo produces an attribute Datastar ignores. A value that should be a Datastar expression gets a Scala `String`, which renders as a literal instead of an expression, and the page just does nothing.

This module closes each of those:

- **Signal names are validated, literals at compile time.** `Signal[Int]("count")` checks the name during compilation; `Signal.dynamic` defers the same check to runtime for names you compute.
- **Raw strings are rejected in expression positions.** `ToDatastarExpr` is deliberately ambiguous for `String`, so passing one is a compile error that names the fix. Expressions come from the `js"..."` interpolator or from typed signals.
- **Attributes are `Dom.Attribute` values**, so they compose with the rest of the HTML DSL and cannot be misplaced into text content. The helpers also settle the naming convention for you: plain attributes render with a hyphen (`data-text`), and keyed ones with a colon (`data-on:click`, `data-class:active`, `data-computed:total`).
- **SSE events are builders**, so each event kind offers the options it accepts and rendering emits the exact protocol field names. The one exception is `executeScript`, which returns an element-patch builder and so exposes `selector` and `mode` that do not apply to it.

## Installation

Add the module to your build:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-datastar" % "0.0.56"
```

On Scala 3.9 or newer, use the `zio-blocks-next-` prefix and enable experimental features. The `zio-blocks-next-*` artifacts are compiled with `-experimental`, so code that calls them must be too:

```scala
scalaVersion := "3.9.0"

libraryDependencies += "dev.zio" %% "zio-blocks-next-datastar" % "0.0.56"

scalacOptions += "-experimental"
```

For Scala.js, use `%%%` instead of `%%`:

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-datastar" % "0.0.56"
```

Supported Scala versions: Scala 3.x only. The module is not published for Scala 2.13. Scala 3.3 to 3.8 use `zio-blocks-datastar`, and Scala 3.9 and newer use `zio-blocks-next-datastar`, as described in [How to Add ZIO Blocks to Your Build](../../getting-started/installation.md).

:::note[The package is `zio.http.datastar`]
Despite living under `zio/blocks/datastar` in the source tree, the package is `zio.http.datastar`. The package object extends `DatastarAttributes`, so a single wildcard import brings every `data*` helper into scope.
:::

## Overview

The module divides along the direction data flows.

### Signals — state that lives in the browser

`Signal[A]` is a named, typed handle on a client-side signal. `Signal#:=` pairs it with a value to produce a `SignalUpdate[A]`, serialized through the type's JSON codec. `Signal#ref` produces a `DatastarRef`, the `$name` form used inside expressions. See [Signals](./signals.md).

### Attributes — declaring reactivity on the page

`DatastarAttributes` supplies 27 `data*` helpers — `dataText`, `dataShow`, `dataBind`, `dataClass`, `dataComputed`, and the rest — each returning either a `Dom.Attribute` or a `DatastarAttrKey` awaiting a value. `ToDatastarExpr` governs what may be assigned. See [Attributes](./attributes.md).

### Events — reacting to the user

`dataOn` opens the event side: sixteen predefined events, fourteen chainable modifiers for debouncing, throttling, and propagation control, and four specialized triggers for intersection, interval, signal-patch, and init. See [Event Handlers](./events.md).

### SSE — patching the page from the server

`DatastarEvent` builds the four server-to-browser events: patch elements, patch signals, execute a script, and remove elements. `ElementPatchMode` chooses where content lands, and `EventType` names the protocol event. See [Server-Sent Events](./sse.md).

## How They Work Together

A Datastar interaction is a loop, and the module sits on both ends of it:

```
1. Server renders HTML          html DSL + data* attributes
2. Browser becomes reactive     Datastar reads data-* and wires up signals
3. User acts                    an event matching a data-on:* fires
4. Browser issues a request     @post('/increment') from the expression
5. Server responds with SSE     DatastarEvent.patchSignals / patchElements
6. Browser applies the patch    signals update, elements morph in place
                                └─> back to 3
```

The types involved at each end:

```
BROWSER SIDE (rendered into HTML)

  Signal[A] ──ref──> DatastarRef ──> "$count"   used inside js"..." expressions
      │
      └─:=─> SignalUpdate[A] ──> {"count": 42}  JSON via Schema[A].jsonCodec

  dataOn.click ──> DataOn ──modifiers──> DataOn ──:=──> Dom.Attribute
                              │                          data-on:click__debounce.300ms
                              ├─ EventModifier            (14: debounce, throttle,
                              │                            once, passive, stop, …)
                              └─ CaseModifier             (__case.camel | kebab | snake | pascal)

  dataText / dataShow / dataClass(…) ──> DatastarAttrKey ──:=──> Dom.Attribute
                                                    ▲
                                          ToDatastarExpr guards the value:
                                          js"…" and Signal ok, raw String rejected

SERVER SIDE (rendered into an SSE stream)

  DatastarEvent.patchElements(dom) ──> PatchElementsBuilder ──renderSSE──> String
        │                                    ├─ selector(CssSelector)
        │                                    ├─ mode(ElementPatchMode)
        │                                    ├─ viewTransition / namespace
        │                                    └─ eventId / retry
        ├─ patchSignals(updates*) ──> PatchSignalsBuilder    ├─ onlyIfMissing
        ├─ executeScript(js) ──────> PatchElementsBuilder
        └─ removeElements(sel) ────> RemoveElementsBuilder

  renderSSE delegates to zio.http.ServerSentEvent, so the result is
  standard SSE with a Datastar-specific event name and data body.
```

## Common Patterns

Four shapes cover most Datastar work.

### A Counter, End to End

The smallest complete loop: a signal, an attribute that displays it, a button that asks the server to change it, and an SSE event that does:

```scala

val count = Signal[Int]("count")

val page = div(
  dataSignals(count := 0),
  span(dataText := count),
  button(dataOn.click := js"@post('/increment')", "increment")
)
```

Rendering gives ordinary HTML that Datastar can read:

```scala
page.renderMinified
// res0: String = "<button data-on:click=\"@post(&#x27;/increment&#x27;)\">increment</button>"
```

The handler for `/increment` replies with a signal patch rather than a JSON body:

```scala
DatastarEvent.patchSignals(count := 1).renderSSE
// res1: String = """event: datastar-patch-signals
// data: signals {"count":1}
// 
// """
```

### Deriving State Instead of Storing It

`dataComputed` defines a signal whose value is an expression over other signals, so the derived value never has to be kept in sync:

```scala

val price    = Signal[Double]("price")
val quantity = Signal[Int]("quantity")
val total    = Signal[Double]("total")

val form = div(
  dataSignals(price := 9.99, quantity := 1),
  dataComputed(total) := js"$$price * $$quantity",
  span(dataText := total)
)
```

The computed attribute carries the expression, and the browser recomputes it whenever either input changes:

```scala
form.renderMinified
// res3: String = ""
```

### Rate-Limiting a Chatty Event

Modifiers chain on `dataOn` before the handler is assigned, which is how a keystroke-driven search avoids one request per character:

```scala

val term = Signal[String]("term")

val search = input(
  dataBind(term),
  dataOn.input.debounce(300) := js"@get('/search')"
)
```

The modifier becomes part of the attribute name, so the browser applies it without any JavaScript of yours:

```scala
search.renderMinified
// res5: String = "<input data-bind:term data-on:input__debounce.300ms=\"@get(&#x27;/search&#x27;)\"/>"
```

### Patching a Fragment Rather Than the Page

`DatastarEvent.patchElements` with a selector and a mode replaces part of the page, leaving the rest — and its signal state — untouched:

```scala

val row = tr(td("Widget"), td("in stock"))

val event = DatastarEvent
  .patchElements(row)
  .selector(CssSelector.id("inventory"))
  .mode(ElementPatchMode.Append)
  .renderSSE
```

The rendered event names the selector and mode as protocol fields:

```scala
event
// res7: String = """event: datastar-patch-elements
// data: selector #inventory
// data: mode append
// data: elements Widgetin stock
// 
// """
```

## Integration Points

The module is a thin typed layer over three other blocks, and adds no dependency of its own:

- **`zio-blocks-html`** supplies `Dom`, `Dom.Attribute`, `Js`, `CssSelector`, and the `ToJs` type class. Every attribute helper returns a `Dom.Attribute`, so Datastar attributes compose with the HTML DSL exactly like `id` or `class` — see [HTML](../html.md).
- **`zio-blocks-schema`** supplies the JSON codec behind `Signal#:=`. A signal of type `A` needs a `Schema[A]`, and the serialized form is whatever that schema's JSON codec produces — see [Schema](../schema/index.md).
- **`zio-blocks-http-model`** supplies `ServerSentEvent`, which `DatastarEvent#renderSSE` delegates to for the wire format. The Datastar-specific part is the event name and the structured `data:` body — see [ServerSentEvent](../http-model/server-sent-event.md).

Within the module, the dependency direction is one-way: attributes and events both consume `Signal`, and neither knows about the other. Nothing in the SSE layer reads the attribute DSL.

## Next Steps

Start with [Signals](./signals.md), which every other page builds on, then [Attributes](./attributes.md) for the declarative surface and [Event Handlers](./events.md) for triggers. [Server-Sent Events](./sse.md) covers the server half. To build a counter and a live dashboard step by step, follow [Build a Live Counter and Dashboard with Datastar](../../guides/build-a-live-dashboard-with-datastar.md).

---

## Signal

`Signal[A]` is a named, typed handle on a piece of state that lives in the browser. `Signal#:=` pairs it with a value to produce a `SignalUpdate[A]`, serialized through the type's JSON codec, and `Signal#ref` produces the `$name` reference form used inside Datastar expressions. Names are validated — literals during compilation. The three types:

```scala
final class Signal[A] private (val name: String) {
  def :=(value: A)(implicit schema: Schema[A]): SignalUpdate[A]
  def ref: DatastarRef
}

final class SignalUpdate[A] private (val name: String, val serialized: String)

final class DatastarRef private (val signalName: String) {
  val value: String   // "$" + signalName
}
```

## Motivation

Datastar identifies state by name, as a string, on both sides of the wire. The browser reads `data-signals="{count: 0}"`, an expression refers to `$count`, and the server patches it with `signals {"count": 42}`. Three places, one name, and nothing checks that they agree.

They disagree in ways that produce no error. A signal referenced as `$cont` is simply undefined, so the expression evaluates to nothing and the element renders blank. A patch sent for `"counter"` when the page declared `"count"` creates a second signal nobody reads. Both look like the feature is broken rather than the name being wrong.

`Signal[A]` makes the name a value declared once, so all three uses are the same Scala identifier. Declaring it also validates it — and the type parameter decides how values serialize, so a `Signal[Int]` cannot be patched with a string.

## Quick Showcase

Declare a signal, then use the same value for the attribute, the expression, and the patch:

```scala

val count = Signal[Int]("count")
```

The three forms it produces — an initial value, a reference, and a patch:

```scala
(count := 0).serialized
// res0: String = "0"
count.ref.value
// res1: String = "$count"
DatastarEvent.patchSignals(count := 42).renderSSE
// res2: String = """event: datastar-patch-signals
// data: signals {"count":42}
// 
// """
```

## Construction

Two constructors. They differ in when a *literal* name is checked; for a computed name both check at runtime.

### `Signal.apply` — checked at compile time

`Signal[A]("name")` validates a literal name during compilation, so a malformed name is a build failure rather than a silently dead expression:

```scala

val count   = Signal[Int]("count")
val nested  = Signal[String]("user.profile.name")
```

Dotted names are how Datastar expresses nested signal namespaces, and each segment is validated separately:

```scala
count.name
// res4: String = "count"
nested.name
// res5: String = "user.profile.name"
```

### `Signal.dynamic` — checked at runtime

`Signal.apply` also accepts a computed name — its macro falls back to a runtime check when the argument is not a literal — so `Signal.dynamic` is not strictly required. Prefer it anyway for computed names, because it states that intent at the call site. Either way the check throws `IllegalArgumentException`:

```scala
object Signal {
  def dynamic[A](name: String): Signal[A]
}
```

Use it for names derived from data — a row id, a tenant prefix — and validate before constructing if the source is untrusted:

```scala

def rowSignal(id: Long): Signal[Boolean] = Signal.dynamic[Boolean](s"row$id.selected")
```

A well-formed computed name behaves exactly like a literal one:

```scala
rowSignal(17).name
// res7: String = "row17.selected"
rowSignal(17).ref.value
// res8: String = "$row17.selected"
```

A malformed one fails at the point of construction rather than in the browser:

```scala
scala.util.Try(Signal.dynamic[Int]("has spaces")).failed.map(_.getMessage)
// res9: Try[String] = Success(
//   "Invalid Datastar signal name 'has spaces'. Signal names must be dot-separated JavaScript identifiers and must not contain '__'."
// )
```

### Name Rules

A valid name is one or more dot-separated segments, each a Java identifier, and must not contain `__`:

| Name | Valid | Why |
| ---- | ----- | --- |
| `count` | yes | single identifier |
| `user.profile.name` | yes | dot-separated identifiers |
| `row17.selected` | yes | digits are identifier parts, just not the first character |
| `row.17.selected` | **no** | the segment `17` starts with a digit |
| `has spaces` | no | space is not an identifier character |
| `count__case` | no | `__` is reserved for Datastar modifier syntax |
| `` (empty) | no | at least one segment is required |
| `a..b` | no | empty segment |

Two of these catch people out. The `__` restriction exists because Datastar uses a double underscore to separate an attribute name from its modifiers, so a signal containing one would be parsed as a modifier suffix. And every segment must *start* with an identifier-start character, which rules out a purely numeric segment — the natural `s"row.$id.selected"` for a per-row signal is rejected, and `s"row$id.selected"` is the form that works.

## Producing Values

A signal on its own is just a name. Two methods turn it into something renderable.

### `Signal#:=` — pair a value with the name

`Signal#:=` requires a `Schema[A]` and serializes through that schema's JSON codec, producing a `SignalUpdate[A]`:

```scala
final class Signal[A] {
  def :=(value: A)(implicit schema: Schema[A]): SignalUpdate[A]
}
```

The update holds the name and the already-serialized JSON:

```scala

final case class Point(x: Int, y: Int)

object Point {
  implicit val schema: zio.blocks.schema.Schema[Point] = zio.blocks.schema.Schema.derived[Point]
}

val origin = Signal[Point]("origin")
```

Because serialization goes through the schema, a case class signal works without extra ceremony:

```scala
(origin := Point(3, 4)).serialized
// res11: String = "{\"x\":3,\"y\":4}"
```

:::warning[`serialized` is inserted verbatim]
`SignalUpdate#serialized` is spliced directly into the rendered expression or SSE payload. It is produced by the schema codec, so it is valid JSON — but if you construct a payload by other means (`DatastarEvent.patchSignalsRaw`), you are responsible for the JSON being well formed.
:::

### `Signal#ref` — refer to the signal in an expression

`Signal#ref` returns a `DatastarRef`, whose `value` is the name prefixed with `$` — the form Datastar expressions use to read a signal:

```scala

val price    = Signal[Double]("price")
val quantity = Signal[Int]("quantity")
```

A reference renders as `$name`, and both `Signal` and `DatastarRef` have `ToJs` instances so either can be interpolated into a `js"..."` expression:

```scala
price.ref.value
// res13: String = "$price"
js"$price * $quantity".value
// res14: String = "$price * $quantity"
```

Interpolating the signal itself is equivalent to interpolating its ref, which is why most code never mentions `DatastarRef` explicitly.

## SignalUpdate

`SignalUpdate[A]` carries a name and its serialized value. Beyond being consumed by `DatastarEvent.patchSignals` and the `dataSignals` attribute, it renders as a JavaScript object literal.

### `SignalUpdate.objectExpression`

`SignalUpdate.objectExpression` renders one or more updates as a single object expression, which is what the `data-signals` attribute contains:

```scala
object SignalUpdate {
  def objectExpression(update: SignalUpdate[_], updates: SignalUpdate[_]*): String
}
```

Several updates become one object, with each key quoted:

```scala

val price    = Signal[Double]("price")
val quantity = Signal[Int]("quantity")
```

Keys are escaped, so a dotted signal name stays a single key rather than becoming a nested path:

```scala
SignalUpdate.objectExpression(price := 9.99, quantity := 2)
// res16: String = "{\"price\": 9.99, \"quantity\": 2}"
```

An implicit `ToJs[SignalUpdate[A]]` renders a single update the same way, so an update can be interpolated into an expression directly.

## Integration Points

`Signal` sits at the centre of the module: [Attributes](./attributes.md) accept signals as values and as keys (`dataBind`, `dataComputed`, `dataIndicator`), [Event Handlers](./events.md) reference them inside expressions, and [Server-Sent Events](./sse.md) patch them by name.

Outside the module, `Signal#:=` depends on `Schema[A]` and its JSON codec from [Schema](../schema/index.md), and the `ToJs` instances come from [HTML](../html.md), which is what lets a signal be interpolated into `js"..."`.

---

## Server-Sent Events

`DatastarEvent` builds the events a server sends to patch a live page: replace elements, patch signals, execute a script, or remove elements. Each constructor returns a builder for its event kind, and `DatastarEvent#renderSSE` produces the wire format. `DatastarEvent.executeScript` is the exception noted below: it reuses the element-patch builder, so it offers two options that do not apply to it. Supporting types: `PatchElementsBuilder`, `PatchSignalsBuilder`, `RemoveElementsBuilder`, `ElementPatchMode`, `EventType`. The constructors and the one method they all end in:

```scala
sealed trait DatastarEvent {
  def renderSSE: String
}

object DatastarEvent {
  def patchElements(elements: Dom): PatchElementsBuilder
  def patchSignals(first: SignalUpdate[_], rest: SignalUpdate[_]*): PatchSignalsBuilder
  def patchSignalsRaw(json: String): PatchSignalsBuilder
  def executeScript(code: Js): PatchElementsBuilder
  def removeElements(selector: CssSelector): RemoveElementsBuilder
}
```

## Motivation

The Datastar response protocol is SSE with structure inside the `data:` field. A patch looks like this on the wire:

```
event: datastar-patch-elements
data: selector #results
data: mode append
data: elements Widget

```

Three things make that easy to get wrong by hand. The event name must match the protocol exactly. The `data:` lines are a small keyed format, not free text, and the keys differ per event kind. And the terminating blank line is required — omit it and the browser waits indefinitely for an event it already has.

`DatastarEvent` removes all three concerns. Each constructor knows its event name, each builder method maps to one protocol field, and `DatastarEvent#renderSSE` delegates the SSE envelope to `ServerSentEvent`, which supplies the terminator.

## Quick Showcase

Build an event and render it:

```scala

val count = Signal[Int]("count")
```

A signal patch is one line of data:

```scala
DatastarEvent.patchSignals(count := 42).renderSSE
// res0: String = """event: datastar-patch-signals
// data: signals {"count":42}
// 
// """
```

An element patch carries its target and mode:

```scala
DatastarEvent
  .patchElements(li("Widget"))
  .selector(CssSelector.id("results"))
  .mode(ElementPatchMode.Append)
  .renderSSE
// res1: String = """event: datastar-patch-elements
// data: selector #results
// data: mode append
// data: elements Widget
// 
// """
```

## Patching Elements

`DatastarEvent.patchElements` takes a `Dom` and returns a `PatchElementsBuilder`. Its options map one-to-one onto protocol fields:

| Method | Field | Default |
| ------ | ----- | ------- |
| `selector(CssSelector)` | `selector` | omitted — patch by element id |
| `mode(ElementPatchMode)` | `mode` | `Outer`, which is omitted |
| `viewTransition` | `useViewTransition true` | omitted |
| `namespace(String)` | `namespace` | omitted |
| `eventId(String)` | SSE `id:` | omitted |
| `retry(Long)` | SSE `retry:` | omitted |

Every option is omitted when unset, so the minimal event is just the elements:

```scala

val row = li("Widget")
```

With no selector, Datastar matches on the element's own id, and `mode` is absent because `Outer` is the default:

```scala
DatastarEvent.patchElements(row).renderSSE
// res3: String = """event: datastar-patch-elements
// data: elements Widget
// 
// """
```

Adding options adds exactly the corresponding lines:

```scala
DatastarEvent.patchElements(row).selector(CssSelector.id("list")).viewTransition.eventId("evt-1").renderSSE
// res4: String = """event: datastar-patch-elements
// id: evt-1
// data: selector #list
// data: useViewTransition true
// data: elements Widget
// 
// """
```

:::note[`mode` is omitted when it is `Outer`]
`ElementPatchMode.Outer` is the protocol default, so setting it explicitly produces no `mode` line. That is intentional and keeps the payload minimal; do not read the absence of a `mode` line as "no mode".
:::

### ElementPatchMode

Eight modes decide where the content lands relative to the target. Each renders as its lowercase name:

| Mode | Effect |
| ---- | ------ |
| `Outer` | Morph the target element into the new content *(default)* |
| `Inner` | Morph the target's children |
| `Replace` | Replace the target element without morphing |
| `Prepend` | Insert before the target's existing children |
| `Append` | Insert after the target's existing children |
| `Before` | Insert immediately before the target |
| `After` | Insert immediately after the target |
| `Remove` | Remove the target |

`Append` is the one to reach for when adding to a list without re-rendering it:

```scala
ElementPatchMode.Append.render
// res5: String = "append"
ElementPatchMode.Inner.render
// res6: String = "inner"
```

## Patching Signals

`DatastarEvent.patchSignals` takes one or more `SignalUpdate`s and renders them as a single JSON object under the `signals` key:

```scala

val price    = Signal[Double]("price")
val quantity = Signal[Int]("quantity")
```

Multiple updates go in one event rather than one event each:

```scala
DatastarEvent.patchSignals(price := 19.99, quantity := 3).renderSSE
// res8: String = """event: datastar-patch-signals
// data: signals {"price":19.99,"quantity":3}
// 
// """
```

`PatchSignalsBuilder#onlyIfMissing` adds `onlyIfMissing true`, which tells Datastar to set the signal only when it is not already present — the way to supply a default without clobbering client state:

```scala
DatastarEvent.patchSignals(price := 0.0).onlyIfMissing.renderSSE
// res9: String = """event: datastar-patch-signals
// data: onlyIfMissing true
// data: signals {"price":0.0}
// 
// """
```

### `DatastarEvent.patchSignalsRaw` — pre-serialized JSON

When the JSON already exists — from a cache, or a shape no `Schema` describes — `DatastarEvent.patchSignalsRaw` takes it verbatim:

```scala
DatastarEvent.patchSignalsRaw("""{"count":7}""").renderSSE
// res10: String = """event: datastar-patch-signals
// data: signals {"count":7}
// 
// """
```

:::warning[`patchSignalsRaw` does not validate]
The string is inserted into the payload unchanged. Malformed JSON produces an event the browser silently discards, and there is no server-side error. Prefer `DatastarEvent.patchSignals` with typed updates, which cannot produce invalid JSON.
:::

## Executing Scripts

`DatastarEvent.executeScript` takes a `Js` and returns a `PatchElementsBuilder` — because on the wire it *is* an element patch, appending a `<script>` element that the browser executes:

```scala

```

The rendered event shows the mechanism rather than hiding it:

```scala
DatastarEvent.executeScript(js"console.log('done')").renderSSE
// res12: String = """event: datastar-patch-elements
// data: selector body
// data: mode append
// data: elements <script data-effect="el.remove()">console.log('done')</script>
// 
// """
```

The script is appended to `body` and carries `data-effect="el.remove()"`, so it deletes itself after running and leaves no residue in the DOM.

Because the return type is `PatchElementsBuilder`, the element-patch options are all available — including `selector` and `mode`, which would override that targeting and rarely make sense here. Treat the builder as offering `PatchElementsBuilder#eventId` and `PatchElementsBuilder#retry`.

## Removing Elements

`DatastarEvent.removeElements` takes a `CssSelector` and returns a `RemoveElementsBuilder` with a deliberately narrow surface — `viewTransition`, `namespace`, `RemoveElementsBuilder#eventId`, and `RemoveElementsBuilder#retry`:

```scala
DatastarEvent.removeElements(CssSelector.id("banner")).renderSSE
// res13: String = """event: datastar-patch-elements
// data: selector #banner
// data: mode remove
// data: elements 
// 
// """
```

There is no `mode` or `selector` method, because the selector is the argument and the mode is fixed to `remove` with an empty element body. That is the builder pattern doing its job: an option that would be meaningless is not offered.

## EventType

`EventType` names the protocol event in the SSE `event:` field. Two values, and you rarely name them directly since each constructor selects the right one:

| Value | Renders |
| ----- | ------- |
| `EventType.PatchElements` | `datastar-patch-elements` |
| `EventType.PatchSignals` | `datastar-patch-signals` |

Both render the protocol string rather than the Scala name:

```scala
EventType.PatchElements.render
// res14: String = "datastar-patch-elements"
EventType.PatchSignals.render
// res15: String = "datastar-patch-signals"
```

Both element patches and script execution use `PatchElements`, and removal does too — the distinction between them is in the `data:` body, not the event name.

## Streaming Several Events

`DatastarEvent#renderSSE` produces one complete event including its terminating blank line, so a stream is a concatenation. Nothing in the module manages the stream itself; you write the strings to whatever response body your HTTP layer uses:

```scala

val progress = Signal[Int]("progress")

val stream: String =
  (1 to 3).map(step => DatastarEvent.patchSignals(progress := step * 33).renderSSE).mkString
```

Each event is self-delimiting, so concatenating what `DatastarEvent#renderSSE` returns is valid SSE:

```scala
stream
// res17: String = """event: datastar-patch-signals
// data: signals {"progress":33}
// 
// event: datastar-patch-signals
// data: signals {"progress":66}
// 
// event: datastar-patch-signals
// data: signals {"progress":99}
// 
// """
```

Set `PatchElementsBuilder#eventId` when the client should be able to resume with `Last-Event-ID`, and `PatchElementsBuilder#retry` to control the reconnection delay — both are ordinary SSE fields handled by `ServerSentEvent`.

## Integration Points

`DatastarEvent#renderSSE` delegates to `ServerSentEvent` from `zio-blocks-http-model`, which supplies the SSE envelope, the field ordering, and the terminating blank line — see [ServerSentEvent](../http-model/server-sent-event.md). The Datastar-specific part is the event name and the keyed `data:` body.

Element patches carry `Dom` values and `CssSelector` targets from [HTML](../html.md), and signal patches carry [SignalUpdate](./signals.md) values whose JSON comes from the type's `Schema`. What the patched page does with the result is determined by the [Attributes](./attributes.md) and [Event Handlers](./events.md) rendered into it.

---

## Dependency Injection

## Introduction

A service usually needs other services to work, and something has to decide how they are built and handed over. The blocks in this group describe that wiring with types, so that a missing dependency, or one of the wrong type, is a compile error rather than a failure at startup.

## The Blocks

The group has two blocks, which work at different points in an application's life.

| Block                                          | Artifact               | What It Is For                                                                    |
| ---------------------------------------------- | ---------------------- | --------------------------------------------------------------------------------- |
| [Wire](../resource-management/wire.md)         | `zio-blocks-scope`     | Compile-time safe recipes for constructing a service and its dependencies         |
| [Context](../context.md)                       | `zio-blocks-context`   | Type-indexed heterogeneous collections                                            |

## How the Blocks Differ

`Wire` is a recipe: it describes how to construct a service and what that service needs, and the compiler validates the whole dependency graph. It ships in the same artifact as the resource-management types, because a wire builds services that may hold resources, and the [Resource Management](../resource-management/index.md) page shows how wires, resources and scopes fit together.

`Context` is a container: an immutable collection that stores values of different types, indexed by their types. A phantom type parameter tracks which types are present, so looking up a value that is not there does not compile.

## Where to Start

Start with [Wire](../resource-management/wire.md) to assemble an application from its parts, and read [Context](../context.md) when a program needs to carry a set of services around by type.

---

## ZIO Blocks Docs (Markdown)

ZIO Blocks Markdown is a **pure, zero-dependency GitHub Flavored Markdown library** providing an immutable ADT for markdown documents, a strict parser with error handling, multiple renderers (GFM markdown, HTML, terminal), and a compile-time validated string interpolator. Core types: `Doc`, `Block`, `Inline`, `Parser`, `Renderer`, `ToMarkdown`.

Here are the core type definitions:

```scala

final case class Doc(blocks: Chunk[Block], metadata: Map[String, String] = Map.empty)

sealed trait Block extends Product with Serializable
final case class Paragraph(content: Chunk[Inline]) extends Block
final case class Heading(level: HeadingLevel, content: Chunk[Inline]) extends Block
final case class CodeBlock(info: Option[String], code: String) extends Block

sealed trait Inline extends Product with Serializable
final case class Text(value: String) extends Inline
final case class Code(value: String) extends Inline
final case class Link(text: Chunk[Inline], url: String, title: Option[String]) extends Inline
```

## Motivation

Markdown is the de facto standard for documentation, READMEs, and formatted text in software development. However, parsing and generating markdown at runtime often requires hand-crafted parsers or fragile string concatenation. ZIO Blocks Markdown provides:

- **Type-safe ADT**: Every markdown element is a Scala case class—no stringly-typed HTML generation
- **Strict parsing**: Rejects invalid markdown with precise error locations (line, column)
- **Round-trip semantics**: Parse-render-parse cycles preserve document structure
- **Normalized equality**: Two documents are equal if their normalized forms are equal (adjacent text nodes merged, empty blocks removed)
- **Multiple output formats**: Render to GFM markdown, HTML (full document or fragment), or colorized terminal
- **Compile-time validation**: The `md"..."` string interpolator validates markdown syntax at compile time, catching errors before runtime
- **Type class system**: `ToMarkdown` lets you interpolate custom types into markdown documents

## Installation

Add the dependency to your `build.sbt`:

```sbt
libraryDependencies += "dev.zio" %% "zio-blocks-markdown" % "0.0.56"
```

For Scala.js, use the cross-build syntax:

```sbt
libraryDependencies += "dev.zio" %%% "zio-blocks-markdown" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x

## How They Work Together

Markdown documents flow through a **parsing → normalization → rendering pipeline**:

```
Markdown String ──> Parser ──> Doc (blocks) ──> Renderer ──> Markdown/HTML/Terminal
                        │           │
                    ParseError   ToMarkdown
                                (type class)

Doc = Chunk[Block]
Block = Paragraph | Heading | CodeBlock | List | Table | ... (contains Inlines)
Inline = Text | Code | Emphasis | Strong | Link | Image | ... (leaf nodes)
```

**Typical workflow:**

1. **Parse markdown** — `Parser.parse("# Hello")` returns `Either[ParseError, Doc]`
2. **Build programmatically** — construct `Doc` with `Heading`, `Paragraph`, `BulletList`, etc.
3. **Normalize** — `doc.normalize` merges adjacent text, removes empty blocks
4. **Render to desired format** — GFM (`Renderer.render`), HTML (`HtmlRenderer.render`), terminal (`TerminalRenderer.render`)
5. **Use interpolator** — `md"# Title with $interpolation"` validates markdown at compile time

Here's a complete example of composing a document:

```scala

val doc = Doc(Chunk(
  Heading(HeadingLevel.H1, Chunk(Text("My Document"))),
  Paragraph(Chunk(Text("This is a paragraph with "), Strong(Chunk(Text("bold"))), Text(" text."))),
  CodeBlock(Some("scala"), "val x = 42"),
  BulletList(Chunk(
    ListItem(Chunk(Paragraph(Chunk(Text("Item 1")))), None),
    ListItem(Chunk(Paragraph(Chunk(Text("Item 2")))), None)
  ), tight = true)
))
```

Rendering to GFM markdown:

```scala

Renderer.render(doc)
// res0: String = """# My Document
// This is a paragraph with **bold** text.
// 
// ```scala
// val x = 42
// ```
// - Item 1
// - Item 2
// """
```

Rendering to HTML:

```scala

HtmlRenderer.render(doc)
// res1: String = "<!DOCTYPE html><html><head></head><body>My DocumentThis is a paragraph with bold text.<pre><code class=\"language-scala\">val x = 42</code></pre>Item 1Item 2</body></html>"
```

Rendering to terminal:

```scala

TerminalRenderer.render(doc)
// res2: String = """My Document
// 
// This is a paragraph with bold text.
// 
// val x = 42
// 
// • Item 1
// • Item 2
// """
```

Parsing markdown strings to create documents:

```scala

val markdown = """# My Document

This is a paragraph with **bold** text.

- Item 1
- Item 2
"""

Parser.parse(markdown) match {
  case Right(parsedDoc) => println(s"Successfully parsed ${parsedDoc.blocks.size} blocks")
  case Left(err) => println(s"Parse error: ${err.message}")
}
```

Round-trip verification (parse → render → parse preserves structure):

```scala

val input = "# Hello\n\nWorld"
// input: String = """# Hello
// 
// World"""
val parsed1 = Parser.parse(input).toOption.get
// parsed1: Doc = Doc(
//   blocks = IndexedSeq(
//     Heading(level = H1, content = IndexedSeq(Text("Hello"))),
//     Paragraph(IndexedSeq(Text("World")))
//   ),
//   metadata = Map()
// )
val rendered = Renderer.render(parsed1)
// rendered: String = """# Hello
// World
// 
// """
val parsed2 = Parser.parse(rendered).toOption.get
// parsed2: Doc = Doc(
//   blocks = IndexedSeq(
//     Heading(level = H1, content = IndexedSeq(Text("Hello"))),
//     Paragraph(IndexedSeq(Text("World")))
//   ),
//   metadata = Map()
// )
parsed1 == parsed2  // Equal after normalization
// res5: Boolean = true
```

## Common Patterns

The markdown module provides several patterns for working with documents and types. Here are the most common usage scenarios:

### String Interpolation with Compile-Time Validation

The `md"..."` interpolator validates markdown at compile time and supports interpolation of any type with a `ToMarkdown` instance:

```scala

val name = "Alice"
val count = 42
```

Now interpolate these values into markdown:

```scala

md"# Welcome $name\nYou have $count items."
// res6: Doc = Doc(
//   blocks = IndexedSeq(
//     Heading(
//       level = H1,
//       content = IndexedSeq(Text("Welcome Alice\\nYou have 42 items."))
//     )
//   ),
//   metadata = Map()
// )
```

### Task Lists (GFM Feature)

Task list items use `ListItem` with `checked: Option[Boolean]`:

```scala

val tasks = BulletList(Chunk(
  ListItem(Chunk(Paragraph(Chunk(Text("Buy groceries")))), Some(true)),
  ListItem(Chunk(Paragraph(Chunk(Text("Write docs")))), Some(false)),
  ListItem(Chunk(Paragraph(Chunk(Text("Call Alice")))), None)  // No checkbox
), tight = true)
```

Rendering the task list:

```scala

Renderer.render(Doc(Chunk(tasks)))
// res7: String = """- [x] Buy groceries
// - [ ] Write docs
// - Call Alice
// """
```

### Tables with Alignment

Tables require a header row, alignment specification, and data rows:

```scala

val table = Table(
  header = TableRow(Chunk(
    Chunk(Text("Name")),
    Chunk(Text("Age")),
    Chunk(Text("City"))
  )),
  alignments = Chunk(Alignment.Left, Alignment.Right, Alignment.Center),
  rows = Chunk(
    TableRow(Chunk(
      Chunk(Text("Alice")),
      Chunk(Text("30")),
      Chunk(Text("NYC"))
    )),
    TableRow(Chunk(
      Chunk(Text("Bob")),
      Chunk(Text("25")),
      Chunk(Text("LA"))
    ))
  )
)
```

Rendering the table to markdown:

```scala

Renderer.render(Doc(Chunk(table)))
// res8: String = """| Name | Age | City |
// |:---|---:|:--:|
// | Alice | 30 | NYC |
// | Bob | 25 | LA |
// """
```

### Parsing with Error Handling

Parser returns `Either[ParseError, Doc]` with precise location information:

```scala

Parser.parse("# Hello\n[invalid link](") match {
  case Right(doc) => println("Parsed successfully")
  case Left(err) => println(s"Error at line ${err.line}, column ${err.column}: ${err.message}")
}
```

### Round-Trip Semantics

Parse-render-parse cycles preserve document meaning (normalized forms are equal):

```scala

val input = "# Hello\n\nWorld"
val parsed1 = Parser.parse(input).toOption.get
val rendered = Renderer.render(parsed1)
val parsed2 = Parser.parse(rendered).toOption.get
assert(parsed1 == parsed2)  // Equal after normalization
```

### Custom Type Interpolation

Implement `ToMarkdown` for your types to enable interpolation:

```scala

case class User(name: String, role: String)

implicit val userToMarkdown: ToMarkdown[User] = { user =>
  Strong(Chunk(Text(user.name), Text(s" – ${user.role}")))
}

val user = User("Alice", "Engineer")
val doc = md"# Team\n$user"
```

Additional use cases for custom type interpolation include building documentation programmatically with rich data types, generating reports with structured business objects, and creating templates that mix markdown formatting with domain-specific data. For example, you could create instances for your API models to automatically format them as markdown tables or code examples, making it easy to generate consistent documentation from live data structures.

## Integration Points

The markdown module integrates seamlessly with other ZIO Blocks components to provide a cohesive ecosystem for working with structured data and documentation.

**Schema Integration**: The `schema` module can derive codecs for markdown documents, enabling serialization and deserialization of `Doc` and related types. This allows you to persist markdown structures to various formats (JSON, MessagePack, BSON, etc.) while maintaining type safety and schema validation.

**Chunk Integration**: Core data structures like `Doc`, `Block`, and `Inline` use `Chunk[T]` throughout for efficient immutable sequences. This provides O(1) concatenation, memory efficiency, and seamless interoperability with other ZIO libraries that also rely on chunks for data streaming and collection manipulation.

**HTTP Integration**: Markdown documents can be served directly as documentation endpoints in HTTP servers. Combined with the multiple renderer options (GFM, HTML, terminal), you can build documentation APIs that serve content in multiple formats based on client preferences, making it trivial to expose living documentation alongside your application.

---

## Doc

A complete GitHub Flavored Markdown document.

### Definition

Here is the `Doc` type definition:

```scala

final case class Doc(blocks: Chunk[Block], metadata: Map[String, String] = Map.empty)
```

A parsed or constructed markdown document. The `metadata` field is reserved for future use and defaults to an empty map.

### Construction

Create an empty document:

```scala

val empty = Doc.empty
```

Construct a document from blocks:

```scala

val doc = Doc(Chunk(
  Heading(HeadingLevel.H1, Chunk(Text("Title"))),
  Paragraph(Chunk(Text("Content")))
))
```

Parse a markdown string to create a document:

```scala

Parser.parse("# Title\n\nContent") match {
  case Right(doc) => // use doc
  case Left(err) => // handle error
}
```

### Core Operations

Merge documents using concatenation with `Doc#++`:

```scala

val doc1 = Doc(Chunk(Heading(HeadingLevel.H1, Chunk(Text("Part 1")))))
val doc2 = Doc(Chunk(Paragraph(Chunk(Text("Part 2")))))
val combined = doc1 ++ doc2
```

Render a document to GFM markdown:

```scala

val doc = Doc(Chunk(Heading(HeadingLevel.H1, Chunk(Text("Hello")))))
// doc: Doc = Doc(
//   blocks = IndexedSeq(Heading(level = H1, content = IndexedSeq(Text("Hello")))),
//   metadata = Map()
// )
val markdown: String = doc.toString  // Calls Renderer.render internally
// markdown: String = """# Hello
// """
```

Render to HTML with full document structure including DOCTYPE (returns complete HTML5 document with `<!DOCTYPE html>` wrapper):

```scala

val doc = Doc(Chunk(Heading(HeadingLevel.H1, Chunk(Text("Hello")))))
// doc: Doc = Doc(
//   blocks = IndexedSeq(Heading(level = H1, content = IndexedSeq(Text("Hello")))),
//   metadata = Map()
// )
val html = doc.toHtml
// html: String = "<!DOCTYPE html><html><head></head><body>Hello</body></html>"
```

Render to HTML fragment containing only the content (returns just the rendered HTML blocks without wrapper tags):

```scala

val doc = Doc(Chunk(Heading(HeadingLevel.H1, Chunk(Text("Hello")))))
// doc: Doc = Doc(
//   blocks = IndexedSeq(Heading(level = H1, content = IndexedSeq(Text("Hello")))),
//   metadata = Map()
// )
val fragment = doc.toHtmlFragment
// fragment: String = "Hello"
```

Render to colorized terminal output (returns ANSI-colored string suitable for terminal display):

```scala

val doc = Doc(Chunk(Heading(HeadingLevel.H1, Chunk(Text("Hello")))))
// doc: Doc = Doc(
//   blocks = IndexedSeq(Heading(level = H1, content = IndexedSeq(Text("Hello")))),
//   metadata = Map()
// )
val terminal = doc.toTerminal
// terminal: String = """Hello
// 
// """
```

Canonicalize document structure with `Doc#normalize` to merge adjacent text nodes and remove empty blocks:

```scala

val doc = Doc(Chunk(
  Paragraph(Chunk(Text("A"), Text("B"))),  // Adjacent text nodes
  Paragraph(Chunk()),                      // Empty paragraph
  Paragraph(Chunk(Text("C")))
))
```

Call `Doc#normalize` to see the result:

```scala

val normalized = doc.normalize
// normalized: Doc = Doc(
//   blocks = IndexedSeq(
//     Paragraph(IndexedSeq(Text("AB"))),
//     Paragraph(IndexedSeq(Text("C")))
//   ),
//   metadata = Map()
// )
```

### Equality and Hashing

Two documents are equal if their **normalized forms** are equal. This means:

```scala

val doc1 = Doc(Chunk(Paragraph(Chunk(Text("Hello"), Text(" "), Text("World")))))
val doc2 = Doc(Chunk(Paragraph(Chunk(Text("Hello World")))))
assert(doc1 == doc2)  // Equal after normalization
```

Hash code computes from the normalized form for consistency with `equals`.

---

## Block (Sealed Trait)

A block-level markdown element:

```scala

sealed trait Block extends Product with Serializable
```

Block is a sealed trait with the following concrete subtypes.

### Paragraph

A paragraph containing inline content:

```scala

final case class Paragraph(content: Chunk[Inline]) extends Block
```

Here's an example of creating a paragraph with mixed content:

```scala

val para = Paragraph(Chunk(
  Text("Hello "),
  Strong(Chunk(Text("world")))
))
```

### Heading

An ATX-style heading (# to ######) with a level and inline content:

```scala

final case class Heading(level: HeadingLevel, content: Chunk[Inline]) extends Block
```

Here's an example of creating headings with different levels:

```scala

val h1 = Heading(HeadingLevel.H1, Chunk(Text("Title")))
val h3 = Heading(HeadingLevel.H3, Chunk(Text("Subsection")))
```

### CodeBlock

A fenced code block with optional language/info string:

```scala

final case class CodeBlock(info: Option[String], code: String) extends Block
```

Here are examples of creating code blocks with and without language specification:

```scala

// Scala code block
val scalaBlock = CodeBlock(Some("scala"), "val x = 42\nprintln(x)")

// No language specified
val plainBlock = CodeBlock(None, "some code")
```

### ThematicBreak

A thematic break (horizontal rule):

```scala

case object ThematicBreak extends Block
```

Create a thematic break:

```scala

val break = ThematicBreak
```

**Renders as:** `---\n` (or `***` or `___`)

### BlockQuote

A block quote containing nested blocks:

```scala

final case class BlockQuote(content: Chunk[Block]) extends Block
```

Here's an example of creating a block quote:

```scala

val quote = BlockQuote(Chunk(
  Paragraph(Chunk(Text("This is a famous quote.")))
))
```

### BulletList

An unordered list with bullet markers (-, *, +):

```scala

final case class BulletList(items: Chunk[ListItem], tight: Boolean) extends Block
```

The `tight` parameter controls spacing: `true` removes blank lines between items for compact rendering. Here's an example:

```scala

val list = BulletList(Chunk(
  ListItem(Chunk(Paragraph(Chunk(Text("Item 1")))), None),
  ListItem(Chunk(Paragraph(Chunk(Text("Item 2")))), None)
), tight = true)
```

### OrderedList

An ordered list with numeric markers (1., 2., etc.):

```scala

final case class OrderedList(start: Int, items: Chunk[ListItem], tight: Boolean) extends Block
```

The `start` parameter specifies the starting number (typically 1). Here's an example:

```scala

val list = OrderedList(
  start = 1,
  items = Chunk(
    ListItem(Chunk(Paragraph(Chunk(Text("First")))), None),
    ListItem(Chunk(Paragraph(Chunk(Text("Second")))), None)
  ),
  tight = true
)
```

### ListItem

A list item, optionally a task list item:

```scala

final case class ListItem(content: Chunk[Block], checked: Option[Boolean]) extends Block
```

The `checked` parameter: `Some(true)` renders as `[x]`, `Some(false)` renders as `[ ]`, `None` for regular list items. Here are examples of each type:

```scala

// Regular list item
val item = ListItem(Chunk(Paragraph(Chunk(Text("Task")))), None)

// Completed task
val completed = ListItem(Chunk(Paragraph(Chunk(Text("Done")))), Some(true))

// Incomplete task
val incomplete = ListItem(Chunk(Paragraph(Chunk(Text("TODO")))), Some(false))
```

### HtmlBlock

Raw HTML block content:

```scala

final case class HtmlBlock(content: String) extends Block
```

Here's an example of creating an HTML block:

```scala

val html = HtmlBlock("Custom HTML")
```

### Table

A GitHub Flavored Markdown table with aligned columns:

```scala

final case class Table(header: TableRow, alignments: Chunk[Alignment], rows: Chunk[TableRow]) extends Block
```

Here's an example of creating a table:

```scala

val table = Table(
  header = TableRow(Chunk(Chunk(Text("Name")), Chunk(Text("Age")))),
  alignments = Chunk(Alignment.Left, Alignment.Right),
  rows = Chunk(
    TableRow(Chunk(Chunk(Text("Alice")), Chunk(Text("30")))),
    TableRow(Chunk(Chunk(Text("Bob")), Chunk(Text("25"))))
  )
)
```

---

## Inline (Sealed Trait)

An inline-level markdown element:

```scala

sealed trait Inline extends Product with Serializable
```

Inline is a sealed trait with concrete subtypes, defining both object-level and top-level forms for API compatibility.

### Text

Plain text content:

```scala

final case class Text(value: String) extends Inline
```

Here's an example of creating text:

```scala

val text = Text("Hello world")
```

### Code

Inline code span (backtick-delimited):

```scala

final case class Code(value: String) extends Inline
```

Here's an example of creating inline code:

```scala

val code = Code("val x = 42")
```

### Emphasis

Emphasized (italic) text:

```scala

final case class Emphasis(content: Chunk[Inline]) extends Inline
```

Here's an example of creating emphasized text:

```scala

val emphasis = Emphasis(Chunk(Text("italic")))
// Renders as: *italic*
```

### Strong

Strong (bold) text:

```scala

final case class Strong(content: Chunk[Inline]) extends Inline
```

Here's an example of creating strong text:

```scala

val strong = Strong(Chunk(Text("bold")))
// Renders as: **bold**
```

### Strikethrough

Strikethrough text (GFM feature):

```scala

final case class Strikethrough(content: Chunk[Inline]) extends Inline
```

Here's an example of creating strikethrough text:

```scala

val struck = Strikethrough(Chunk(Text("deprecated")))
// Renders as: ~~deprecated~~
```

### Link

A hyperlink:

```scala

final case class Link(text: Chunk[Inline], url: String, title: Option[String]) extends Inline
```

The `title` parameter is optional link title text. Here are examples of creating links:

```scala

// Simple link
val link = Link(Chunk(Text("Click here")), "https://example.com", None)

// Link with title
val titled = Link(Chunk(Text("Docs")), "/docs", Some("Documentation"))
```

### Image

An image reference:

```scala

final case class Image(alt: String, url: String, title: Option[String]) extends Inline
```

Here are examples of creating images:

```scala

val img = Image(alt = "Logo", url = "/logo.png", None)
val imgWithTitle = Image(alt = "Icon", url = "/icon.svg", Some("App Icon"))
```

### HtmlInline

Raw HTML inline content:

```scala

final case class HtmlInline(content: String) extends Inline
```

Here's an example of creating HTML inline content:

```scala

val html = HtmlInline("custom")
```

### SoftBreak

A soft line break (single newline, rendered as space or newline depending on context):

```scala

case object SoftBreak extends Inline
```

### HardBreak

A hard line break (two spaces or backslash before newline):

```scala

case object HardBreak extends Inline
```

### Autolink

An autolink (URL or email in angle brackets):

```scala

final case class Autolink(url: String, isEmail: Boolean) extends Inline
```

Here are examples of creating autolinks:

```scala

val urlLink = Autolink("https://example.com", isEmail = false)
val emailLink = Autolink("user@example.com", isEmail = true)
// Renders as: <https://example.com> and <user@example.com>
```

---

## HeadingLevel

Heading levels from H1 to H6:

```scala

sealed abstract class HeadingLevel(val value: Int) extends Product with Serializable

object HeadingLevel {
  case object H1 extends HeadingLevel(1)
  case object H2 extends HeadingLevel(2)
  case object H3 extends HeadingLevel(3)
  case object H4 extends HeadingLevel(4)
  case object H5 extends HeadingLevel(5)
  case object H6 extends HeadingLevel(6)
}
```

### Safe Construction

Construct a heading level from an integer, returning an `Option`:

```scala

HeadingLevel.fromInt(2) == Some(HeadingLevel.H2)
HeadingLevel.fromInt(7) == None  // Out of range
```

Construct a heading level unsafely, throwing if input is invalid:

```scala

HeadingLevel.unsafeFromInt(3)  // HeadingLevel.H3
HeadingLevel.unsafeFromInt(7)  // Throws IllegalArgumentException
```

### Accessing Value

Access the numeric value of a heading level:

```scala

HeadingLevel.H1.value == 1
```

---

## Alignment

Table column alignment specification:

```scala

sealed trait Alignment extends Product with Serializable

object Alignment {
  case object Left   extends Alignment
  case object Right  extends Alignment
  case object Center extends Alignment
  case object None   extends Alignment
}
```

### Predefined Alignments

Use predefined alignment values for table columns:

```scala

Alignment.Left    // Renders as :---
Alignment.Right   // Renders as ---:
Alignment.Center  // Renders as :---:
Alignment.None    // Renders as ---
```

### Use in Tables

Create a table with specific column alignments:

```scala

val header = TableRow(Chunk(
  Chunk(Text("Name")),
  Chunk(Text("Age")),
  Chunk(Text("City"))
))
val rows = Chunk(
  TableRow(Chunk(
    Chunk(Text("Alice")),
    Chunk(Text("30")),
    Chunk(Text("NYC"))
  )),
  TableRow(Chunk(
    Chunk(Text("Bob")),
    Chunk(Text("25")),
    Chunk(Text("LA"))
  ))
)
val alignments = Chunk(Alignment.Left, Alignment.Center, Alignment.Right)
val table = Table(header, alignments, rows)
val doc = Doc(Chunk(table))
```

If we render this document, the table will have left-aligned "Name", center-aligned "Age", and right-aligned "City":

```scala

Renderer.render(doc)
// res45: String = """| Name | Age | City |
// |:---|:--:|---:|
// | Alice | 30 | NYC |
// | Bob | 25 | LA |
// """
```

---

## TableRow

A single row in a table, containing cells as inline content:

```scala

final case class TableRow(cells: Chunk[Chunk[Inline]])
```

Each cell is a `Chunk[Inline]`, allowing formatted content (text, links, emphasis, etc.). Here's an example:

```scala

val row = TableRow(Chunk(
  Chunk(Text("Alice")),
  Chunk(Strong(Chunk(Text("30")))),
  Chunk(Link(Chunk(Text("NYC")), "/cities/nyc", None))
))
```

---

## Parser

A singleton object providing strict GitHub Flavored Markdown parsing with position-aware error reporting.

### Parsing

Use the main entry point to parse markdown:

```scala

Parser.parse(input: String): Either[ParseError, Doc]
```

Returns `Right(doc)` on success or `Left(error)` on parse failure. Here's an example:

```scala

val result = Parser.parse("# Hello\n\nWorld")
result match {
  case Right(doc) => println(s"Parsed ${doc.blocks.size} blocks")
  case Left(err) => println(s"Parse error at line ${err.line}: ${err.message}")
}
```

### Supported Features

- ATX headings (# to ######)
- Fenced code blocks (``` or ~~~)
- Thematic breaks (---, ***, ___)
- Block quotes (> prefix)
- Bullet lists (-, *, +)
- Ordered lists (1., 2., etc.)
- Task lists (- [ ] and - [x])
- Tables with alignment (GFM)
- Inline formatting (emphasis, strong, strikethrough)
- Links and images with optional titles
- Autolinks (`<url>` or `<email>`)
- HTML blocks and inline HTML
- Soft and hard line breaks

### Unsupported Features

- YAML frontmatter (causes parse error)
- Setext headings (use ATX style with # instead)
- Indented code blocks (use fenced with ``` instead)
- Link reference definitions

---

## ParseError

Error information from parsing, with precise location data:

```scala

final case class ParseError(
  message: String,    // Human-readable error description
  line: Int,          // 1-based line number
  column: Int,        // 1-based column number
  input: String       // The line that caused the error
)
```

Here's an example of creating a `ParseError`:

```scala

val err = ParseError("Unexpected token", line = 5, column = 12, input = "[invalid markdown]")
```

Get the string representation of the error:

```scala

err.toString
// res48: String = """ParseError at line 5, column 12: Unexpected token
//   [invalid markdown]"""
```

---

## Renderer

A singleton object that renders markdown documents back to GitHub Flavored Markdown string format:

```scala

object Renderer
```

### Rendering

Render an entire document to GFM markdown:

```scala

Renderer.render(doc: Doc): String
```

Here's an example of rendering a document:

```scala

val doc = Doc(Chunk(
  Heading(HeadingLevel.H1, Chunk(Text("Title"))),
  Paragraph(Chunk(Text("Content")))
))
val rendered = Renderer.render(doc)
```

The result:

```scala

rendered
// res50: String = """# Title
// Content
// 
// """
```

Render individual blocks:

```scala

Renderer.renderBlock(block: Block): String
```

Render inline content:

```scala

Renderer.renderInlines(inlines: Chunk[Inline]): String
Renderer.renderInline(inline: Inline): String
```

### Normalization During Rendering

The renderer does not normalize; use `doc.normalize` before rendering if normalization is needed.

### Round-Trip Semantics

Re-parse rendered output to verify round-trip semantics:

```scala

val original = Doc(Chunk(Heading(HeadingLevel.H1, Chunk(Text("Title")))))
val markdown = Renderer.render(original)
val reparsed = Parser.parse(markdown).toOption.get
assert(original == reparsed)  // Equal after normalization
```

---

## HtmlRenderer

A singleton object that renders markdown documents to HTML5:

```scala

object HtmlRenderer
```

### Full Document Rendering

Render a document as complete HTML with DOCTYPE and html wrapper tags:

```scala

HtmlRenderer.render(doc: Doc): String
```

Here's an example of rendering a complete HTML document:

```scala

val doc = Doc(Chunk(Heading(HeadingLevel.H1, Chunk(Text("Title")))))
val html = HtmlRenderer.render(doc)
```

The result:

```scala

html
// res54: String = "<!DOCTYPE html><html><head></head><body>Title</body></html>"
```

### Fragment Rendering

Render content-only HTML without wrapper tags:

```scala

HtmlRenderer.renderFragment(doc: Doc): String
```

Here's an example of rendering an HTML fragment:

```scala

val fragment = HtmlRenderer.renderFragment(doc)
```

Use fragments to embed markdown content into existing HTML templates.

### HTML Feature Mapping

- Headings → `` through ``
- Paragraphs → ``
- Code blocks → `<pre><code class="language-...">` (language from info string)
- Thematic breaks → ``
- Block quotes → `<blockquote>`
- Lists → `` and `` with ``
- Task lists → `` with checkbox-like rendering
- Tables → ``, ``, `` with alignment classes
- Inline code → `<code>`
- Emphasis → ``
- Strong → ``
- Strikethrough → `<del>` or `<s>`
- Links → ``
- Images → ``
- HTML blocks and inlines → pass through as-is

---

## TerminalRenderer

A singleton object that renders markdown to ANSI-colored terminal output optimized for console display:

```scala

object TerminalRenderer
```

### Rendering

Render a document to ANSI-colored terminal output:
```scala

TerminalRenderer.render(doc: Doc): String
```

Here's an example of rendering to terminal:
```scala

val doc = Doc(Chunk(Heading(HeadingLevel.H1, Chunk(Text("Title")))))
val terminal = TerminalRenderer.render(doc)
// Returns ANSI-colored string: Title\n\n
```

### Color Scheme

Headings use different colors by level:
- H1 → Red
- H2 → Yellow
- H3 → Green
- H4 → Cyan
- H5 → Blue
- H6 → Magenta

Inline elements use ANSI styles:
- Strong/Bold → ANSI bold
- Emphasis/Italic → ANSI italic
- Code → Gray background
- Links → Underlined
- Strikethrough → ANSI strikethrough style

The output is designed to be readable on both light and dark terminal backgrounds.

---

## ToMarkdown

Type class for converting Scala values to markdown inline elements, enabling interpolation in the `md"..."` string interpolator.

### Definition

Define the `ToMarkdown` type class:

```scala

trait ToMarkdown[-A] {
  def toMarkdown(a: A): Inline
}
```

The type parameter is contravariant (`-A`), allowing subtypes to use supertype instances.

### Summon an Instance

Summon a ToMarkdown instance:

```scala

ToMarkdown[String]  // Implicitly summons the ToMarkdown[String] instance
```

### Built-In Instances

Convert primitive types to plain text:

```scala

ToMarkdown[String]    // a: String => Text(a)
ToMarkdown[Int]       // a: Int => Text(a.toString)
ToMarkdown[Long]      // a: Long => Text(a.toString)
ToMarkdown[Double]    // a: Double => Text(a.toString)
ToMarkdown[Boolean]   // a: Boolean => Text(a.toString)
```

Pass inline elements through unchanged:

```scala

ToMarkdown[Inline]    // a: Inline => a (identity)
```

Convert collections to comma-separated text:

```scala

ToMarkdown[List[A]]    // as: List[A] => Text(as.map(...).mkString(", "))
ToMarkdown[Chunk[A]]   // as: Chunk[A] => Text(as.map(...).mkString(", "))
ToMarkdown[Vector[A]]  // as: Vector[A] => Text(as.map(...).mkString(", "))
ToMarkdown[Seq[A]]     // as: Seq[A] => Text(as.map(...).mkString(", "))
```

For collection instances, each element is converted using its `ToMarkdown[A]` instance, then joined with ", ".

Render blocks to markdown text:

```scala

ToMarkdown[Block]     // b: Block => Text(Renderer.render(Doc(Chunk(b))).trim)
```

### Custom Implementations

Define custom `ToMarkdown` instances for your types:

```scala

case class Person(name: String, age: Int)

implicit val personToMarkdown: ToMarkdown[Person] = { person =>
  Strong(Chunk(Text(person.name)))
}

val p = Person("Alice", 30)
val doc = md"## User: $p"  // Interpolates as strong text "Alice"
```

---

## String Interpolator (`md"..."`)

The `md` string interpolator validates markdown at compile time and supports runtime interpolation of values.

### Compile-Time Validation

Markdown syntax inside `md"..."` is validated at compile time:

```scala

val doc = md"# Valid heading"  // Compiles

val invalid = md"# [unclosed link]("  // Compile error: Invalid markdown
```

### Runtime Interpolation

Values are interpolated and converted to inline markdown using `ToMarkdown`:

```scala

val name = "Alice"
val count = 42
val doc = md"""
# Welcome $name

You have $count items.
"""
// Equivalent to parsing: "# Welcome Alice\n\nYou have 42 items."
```

### Custom Type Interpolation

Any type with a `ToMarkdown` instance can be interpolated:

```scala

case class Tag(label: String)
implicit val tagToMarkdown: ToMarkdown[Tag] = tag =>
  Code(tag.label)

val tag = Tag("important")
val doc = md"Please review: $tag"  // Renders tag as inline code
```

### Interpolation Mechanics

The interpolator:
1. Collects string parts and interpolated values (as `Inline` elements via `ToMarkdown`)
2. Combines parts and rendered inlines into a markdown string
3. Parses the combined string at runtime
4. Returns a `Doc` or throws `IllegalArgumentException` if parsing fails

---

## AuthType

`AuthType` is a sealed trait that describes an HTTP authentication scheme as a first-class type parameter on `Endpoint`. Each `AuthType` variant carries an associated type `ClientRequirement` — the type of credential the client must provide — and a codec that extracts it from the request. Its definition is:

```scala
sealed trait AuthType {
  type ClientRequirement
  def codec: HttpCodec[CodecKind.Request, ClientRequirement]
  def unauthorizedStatus: Status
}
```

## Motivation

Authentication requirements are often encoded informally — a comment in the handler, a middleware convention, or a bare string header check. `AuthType` makes the auth requirement part of the endpoint's static type. A bearer-secured endpoint has type `Endpoint[..., AuthType.Bearer]`, which means:

- The `auth.codec` field is typed as `HttpCodec[CodecKind.Request, zio.http.Header.Authorization.Bearer]`, not `HttpCodec[..., String]`.
- Interpreters (server, client, OpenAPI) can inspect the auth type without stringly-typed reflection.
- Composing auth types with `|` produces a union that the compiler verifies is discriminated.

## Built-in Variants

Each built-in variant maps to a specific HTTP authorization scheme. Use the one that matches your API's authentication model.

### `AuthType.None`

The default auth type — no authentication required. Its `ClientRequirement` is `Unit` and its codec is `HttpCodec.Empty`:

```scala

val publicEndpoint = Endpoint(Method.GET / "health")
```

### `AuthType.Basic`

HTTP Basic authentication. The `ClientRequirement` is `zio.http.Header.Authorization.Basic`:

```scala

val basicEndpoint = Endpoint(Method.GET / "admin")
  .auth(AuthType.Basic)

val codec = basicEndpoint.auth.codec
```

### `AuthType.Bearer`

Bearer token authentication (OAuth 2.0 / JWT). The `ClientRequirement` is `zio.http.Header.Authorization.Bearer`:

```scala

val bearerEndpoint = Endpoint(Method.GET / "me")
  .auth(AuthType.Bearer)
```

### `AuthType.Digest`

HTTP Digest authentication. The `ClientRequirement` is `zio.http.Header.Authorization.Digest`:

```scala

val digestEndpoint = Endpoint(Method.GET / "secure")
  .auth(AuthType.Digest)
```

### `AuthType.Custom`

For authentication schemes not covered by the built-in variants, `Custom` wraps any `HttpCodec[CodecKind.Request, ClientReq]`:

```scala

final case class ApiKey(value: String)

val apiKeyCodec = HttpCodec.requestHeader("X-Api-Key", Schema.string.transform[ApiKey](ApiKey(_), _.value))
val apiKeyAuth  = AuthType.Custom(apiKeyCodec)
val keyEndpoint = Endpoint(Method.GET / "data").auth(apiKeyAuth)
```

## Composition

`AuthType` values compose in two ways: `|` builds an OR alternative that accepts either scheme, and `Scoped` attaches scope requirements to an existing auth type.

### `AuthType#|` — OR composition

Two `AuthType` values can be combined with `|` to accept either scheme. The result is an `AuthType.Or` whose `ClientRequirement` is the union of both:

```scala

val flexEndpoint = Endpoint(Method.GET / "resource")
  .auth(AuthType.Basic | AuthType.Bearer)
```

The `|` operator automatically computes the combined `ClientRequirement` type as a union of both requirements. The codec of the resulting `Or` is a `HttpCodec.Fallback` — it tries the left scheme first and falls back to the right.

### `AuthType.Scoped`

To attach OAuth scope requirements to a bearer auth, wrap it in `AuthType.Scoped`:

```scala

val scopedEndpoint = Endpoint(Method.GET / "admin")
  .auth(AuthType.Scoped(AuthType.Bearer, List("admin:read", "admin:write")))
```

`Scoped` does not change the codec — it carries the scopes as metadata for interpreters that perform scope-level authorization checks.

## Unauthorized Status

`AuthType#unauthorizedStatus` is `Status.NotFound` by default (to avoid leaking endpoint existence). Nothing in the module returns a status itself: a server built on the endpoint should answer with this value when the client does not meet the auth requirement. To change it, use `Endpoint#unauthorizedStatus` or call `AuthType#withUnauthorizedStatus` directly:

```scala

val endpoint = Endpoint(Method.GET / "me")
  .auth(AuthType.Bearer)
  .unauthorizedStatus(Status.Unauthorized)
```

`Or` composition preserves `AuthType#unauthorizedStatus` — it uses the left auth type's status.

## See Also

- [How to Protect an Endpoint with JWT Bearer Tokens](../../guides/protect-endpoints-with-jwt.md): `AuthType.Bearer` only describes the requirement; this guide verifies the JWT in the handler and answers 401 or 403.
- [How to Protect Pages with Basic Auth or a Session Cookie](../../guides/protect-pages-with-basic-auth-or-a-session-cookie.md): `AuthType.Basic` only describes the requirement; this guide checks the Basic credentials in the handler and answers 401 with a `WWW-Authenticate` challenge.

---

## Bulk Endpoint Creation

## Bulk endpoint creation with `endpoints { ... }`

The `endpoints` macro (Scala 3 only; it is not in `zio-blocks-endpoint` 0.0.56, so add `zio-blocks-next-endpoint` and `-experimental` as described in [the endpoint overview](./index.md)) lets you define multiple `Endpoint` values in a block and access them by name on the returned `NamedTuple`. Member names are either explicit `val` names or auto-generated from the `RoutePattern.render` string (method prefix + path template). Prefix grouping via `/` (`"api" / endpoints { ... }`, `PathCodec.int("id") / endpoints { ... }`) is available via the default `import zio.blocks.endpoint.*` — no extra import is needed. All examples below assume `import zio.blocks.endpoint.*` and `scalacOptions += "-experimental"`.

String prefixes such as `"api"` are auto-converted to a literal `PathCodec[Unit]` via a `Conversion[String, PathCodec[Unit]]` provided by `zio.blocks.endpoint.*` — there is no String-specific `/` operator; constant prefixes auto-convert and use the same grouping `/` as capturing prefixes.

> **Inline-only:** prefix grouping (`prefix / endpoints { ... }`) requires an inline `endpoints { ... }` block. Binding the group to a value first (`val g = endpoints { ... }; "api" / g`) is not supported — the macro must see the block literal to compose prefixes.

```scala

val api = "api" / endpoints {
  val customer = Endpoint(Method.GET / "customers")
  Endpoint(Method.GET / "health")
}
```

Member access is static (zero runtime cost):

```scala

val api = "api" / endpoints {
  val customer = Endpoint(Method.GET / "customers")
  Endpoint(Method.GET / "health")
}
val c: Endpoint[Unit, Unit, Unit, Unit, AuthType.None.type] = api.customer
val h: Endpoint[Unit, Unit, Unit, Unit, AuthType.None.type] = api.`GET /health`
```

Auto-naming follows `RoutePattern.render` for the shapes it supports: a method followed by one literal or one capture, or by one literal and one capture:

- `GET /user/{userId}` for path variables (RFC 6570 `{var}`)
- `GET#|POST /orders` for multi-method
- `...` for trailing segments
- `.unused` renders as `{name}` when it is the only segment after the method

Other shapes stop the compile with "cannot auto-name this endpoint; assign it to a val": two literals (`GET / "notes" / "latest"`), two captures, a literal after a capture, `.unused` after a literal, a `~` segment anywhere, and a route that starts from a `RoutePattern` value. This is a known bug in 0.0.56. Assign such an endpoint to a `val` and the `val` name becomes the member name.

Constant-prefix nesting bakes the prefix into each child's `RoutePattern` at the description level; grouping nodes have no path themselves:

```scala

val nested = "api" / endpoints {
  "v1" / endpoints {
    val users = Endpoint(Method.GET / "users")
  }
}
val u = nested.v1.users // route.render == "GET /api/v1/users"
```

Path-variable prefixes are implemented the same way. A capturing prefix contributes its segment to every child's path, while children keep their relative auto-names:

```scala

val byId = PathCodec.int("id") / endpoints {
  val get = Endpoint(Method.GET / "orders")
  Endpoint(Method.DELETE / "orders") // auto-named `DELETE /orders`
}

val getOrder: Endpoint[Int, Unit, Unit, Unit, AuthType.None.type] = byId.get
val delOrder = byId.`DELETE /orders`
```

Both children carry the captured segment: `byId.get.route.render == "GET /{id}/orders"` and the auto-named member renders `"DELETE /{id}/orders"`. Static types are preserved — `byId.get` is an `Endpoint[Int, Unit, Unit, Unit, AuthType.None.type]`, so the captured `id` will be delivered to handlers when routes are created on the zio-http side. A child with its own path variables composes positionally: under `int("id")`, `Endpoint(Method.GET / "orders" / PathCodec.int("orderId"))` renders as `GET /{id}/orders/{orderId}` and carries `(Int, Int)`:

```scala

val ordersById = PathCodec.int("id") / endpoints {
  val o = Endpoint(Method.GET / "orders" / PathCodec.int("orderId"))
}

val lookup: Endpoint[(Int, Int), Unit, Unit, Unit, AuthType.None.type] = ordersById.o
// lookup.route.render == "GET /{id}/orders/{orderId}"
```

Variable prefixes should be bound to an explicit `val` — the val names the subgroup whose members you access through it (`byId.get`, `ordersById.o`). Constant-prefix and capturing-prefix subgroups can be freely nested inside each other: `"api" / endpoints { PathCodec.int("id") / endpoints { ... } }` composes both prefixes into every leaf route at compile time.

The returned type is a Scala 3 `NamedTuple` — static member access, erased at runtime. The DSL is Scala 3 only; see [the endpoint overview](./index.md) for the artifact, the Scala version and the `-experimental` rule.

---

## Endpoint

`Endpoint[PathInput, Input, Err, Output, Auth]` is the top-level descriptor for an HTTP endpoint. It holds a typed route, three independent codec channels (request input, error output, success output), an authentication type, and documentation metadata. `Endpoint` is pure data — it carries no server or client logic and imposes no effect type. Its full shape is:

```scala
final case class Endpoint[PathInput, Input, Err, Output, Auth <: AuthType](
  route: RoutePattern[PathInput],
  input: HttpCodec[CodecKind.Request, Input],
  error: HttpCodec[CodecKind.Response, Err],
  output: HttpCodec[CodecKind.Response, Output],
  auth: Auth,
  doc: Doc
)
```

## Motivation

An endpoint descriptor separates the **shape** of an HTTP surface from its **execution**. A single `Endpoint` value can be interpreted by a server to generate routes, by a client generator to produce typed API calls, or by an OpenAPI renderer to produce specification documents. This means the endpoint definition is the single source of truth — change it once and every interpreter updates.

The five type parameters track everything the compiler needs to enforce consistency across the whole stack:

| Parameter   | Meaning                                                    |
|-------------|------------------------------------------------------------|
| `PathInput` | Type of values extracted from path segments                |
| `Input`     | Aggregate type of all request inputs (query, header, body) |
| `Err`       | Aggregate type of all error response shapes                |
| `Output`    | Aggregate type of all success response shapes              |
| `Auth`      | Authentication scheme, carries the client requirement      |

## Construction

We create an `Endpoint` from a `RoutePattern` using `Endpoint.apply`:

```scala

val ep = Endpoint(Method.GET / "users" / PathCodec.int("id"))
```

The route can also be built from a separate `RoutePattern` value:

```scala

val route = Method.POST / "orders" / PathCodec.uuid("orderId")
val ep    = Endpoint(route)
```

The initial `Endpoint` starts with `Unit` for all three codec channels and `AuthType.None` for auth, so further builder calls always widen the types additively.

## Request Input Builders

Every `Endpoint#in`, `Endpoint#query`, and `Endpoint#header` call adds another input component to the endpoint, widening the `Input` type parameter.

### Body input

To add a request body typed by a `Schema`, use `Endpoint#in`:

```scala

val ep = Endpoint(Method.POST / "users")
  .in(Schema.string)
```

To specify the content type explicitly, pass a `MediaType` as well:

```scala

val ep = Endpoint(Method.POST / "users")
  .in(MediaTypes.application.`json`, Schema.string)
```

To add a raw `HttpCodec.Body` node directly, use `Endpoint#in`:

```scala

val ep = Endpoint(Method.POST / "users")
  .in(HttpCodec.requestBody(Schema.string))
```

### Query parameters

To add a query parameter by name and schema, use `Endpoint#query`:

```scala

val ep = Endpoint(Method.GET / "users")
  .query("page", Schema.int)
  .query("limit", Schema.int)
```

To add a pre-built `HttpCodec.Query` node, use `Endpoint#query`:

```scala

val pageCodec = HttpCodec.query("page", Schema.int)
val ep        = Endpoint(Method.GET / "users").query(pageCodec)
```

### Request headers

To add a request header by name and schema, use `Endpoint#header`:

```scala

val ep = Endpoint(Method.GET / "users")
  .header("X-Trace-Id", Schema.string)
```

To add a pre-built `HttpCodec.Header` node, use `Endpoint#header`:

```scala

val traceCodec = HttpCodec.requestHeader("X-Trace-Id", Schema.string)
val ep         = Endpoint(Method.GET / "users").header(traceCodec)
```

## Success Output Builders

Every `Endpoint#out` and `Endpoint#outHeader` call adds a success response alternative or header component, widening `Output`.

### Response body

To add a 200 OK response body, use `Endpoint#out`:

```scala

val ep = Endpoint(Method.GET / "users")
  .out(Schema.string)
```

To specify a non-200 status code, use `Endpoint#out` with a `Status`:

```scala

val ep = Endpoint(Method.POST / "users")
  .out(Status.Created, Schema.int)
```

To add content-type negotiation, pass a `MediaType`:

```scala

val ep = Endpoint(Method.GET / "users")
  .out(MediaTypes.application.`json`, Schema.string)
  .out(Status.Created, MediaTypes.text.`plain`, Schema.int)
```

Multiple `Endpoint#out` calls produce alternatives. In 0.0.56 the first `Endpoint#out` call gives `Either[A, Unit]`, not `A`: the empty `Unit` output stays as an alternative (a known bug). A second call gives `Either[Either[B, A], Unit]`, and each further alternative adds a nested `Either` layer.

### Response headers

To add a typed response header, use `Endpoint#outHeader`:

```scala

val ep = Endpoint(Method.GET / "users")
  .out(Schema.string)
  .outHeader("X-Total-Count", Schema.int)
```

## Error Output Builders

Error channels work like success channels but populate the `Err` type parameter. Two variants exist: `Endpoint#outError` (cross-version) and `Endpoint#orOutError` (Scala 3 unions).

### `Endpoint#outError` — cross-version additive errors

To add an error response with a status code and body schema, use `Endpoint#outError`:

```scala

val ep = Endpoint(Method.GET / "users")
  .out(Schema.string)
  .outError(Status.NotFound, Schema.string)
  .outError(Status.BadRequest, Schema.string)
```

Each `Endpoint#outError` call widens `Err` by one nested `Either` layer. As with `Endpoint#out`, the first call gives `Either[A, Unit]` in 0.0.56 (a known bug), and the second gives `Either[Either[B, A], Unit]`.

### `Endpoint#orOutError` — Scala 3 union errors

On Scala 3, `Endpoint#orOutError` accumulates error types as a native union type instead of nested `Either`s. The first call replaces the initial `Unit` error outright; subsequent calls build a `Fallback` codec backed by `Unions` derivation:

```scala

val ep = Endpoint(Method.GET / "users")
  .orOutError(Status.NotFound, Schema.string)
  .orOutError(Status.Conflict, Schema.int)

val typed: Endpoint[Unit, Unit, String | Int, Unit, AuthType.None.type] = ep
```

The compiler rejects overlapping union members — two `Endpoint#orOutError` calls both using `Schema.string` produce a compile error because `String | String` is not a valid discriminated union.

## Authentication

To attach an authentication scheme, use `Endpoint#auth`:

```scala

val secured = Endpoint(Method.GET / "me")
  .auth(AuthType.Bearer)
```

The `Auth` type parameter carries the `ClientRequirement` associated type, so a bearer-secured endpoint exposes `auth.codec` typed as `HttpCodec[CodecKind.Request, zio.http.Header.Authorization.Bearer]`. See [AuthType](./auth-type.md) for all variants.

To override the HTTP status the server sends when the client does not meet the auth requirement, use `Endpoint#unauthorizedStatus`:

```scala

val secured = Endpoint(Method.GET / "me")
  .auth(AuthType.Bearer)
  .unauthorizedStatus(Status.Unauthorized)
```

## Documentation

To attach a `Doc` value to the endpoint as a whole, use `Endpoint#doc`:

```scala

val ep = Endpoint(Method.GET / "users")
  .doc(Doc.empty)
```

Documentation attached here flows through to OpenAPI generation and any other documentation interpreters.

---

## HttpCodec

`HttpCodec[K, A]` is a composable, typed descriptor for HTTP request and response parts. The phantom type parameter `K` (either `CodecKind.Request` or `CodecKind.Response`) tracks which direction the codec belongs to, so the compiler prevents mixing request-side codecs (query, request header, request body) with response-side codecs (status, response header, response body). The trait signature is:

```scala
sealed trait HttpCodec[+K <: CodecKind, A]
```

## Motivation

HTTP surfaces have two directions — request and response — and each direction has several distinct parts. Without static direction tracking, it is easy to accidentally pass a response status codec where a request header codec is expected, or combine a query parameter with a response body.

`HttpCodec` makes that class of mistake a compile error. The phantom type `K` carries direction at the type level, so `HttpCodec[CodecKind.Request, A]` and `HttpCodec[CodecKind.Response, A]` are incompatible types. All combinators (`++`, `|`) preserve this constraint: combining two request codecs yields a request codec, and combining request with response is a type error.

## `CodecKind`

`CodecKind` is a phantom type hierarchy with two sealed subtypes:

```scala
sealed trait CodecKind

object CodecKind {
  sealed trait Request  extends CodecKind  // query, request header, request body
  sealed trait Response extends CodecKind  // status, response header, response body
}
```

These are never instantiated — they exist only to parameterize `HttpCodec[K, A]` at the type level.

## Structure

`HttpCodec` is an ADT with seven node types:

| Node          | Kind      | Carries                                              |
| ------------- | --------- | ---------------------------------------------------- |
| `Empty`       | both      | No data — neutral element for `++`                  |
| `Combine`     | both      | Two codecs composed sequentially with `++`           |
| `Fallback`    | both      | Two codecs composed as alternatives with `&#124;`    |
| `Query`       | `Request` | Named query parameter with `Schema[A]`               |
| `Header`      | both      | Named HTTP header with `Schema[A]` (request or response) |
| `Body`        | both      | Request or response body with `Schema[A]`            |
| `StatusCodec` | `Response`| HTTP status code                                     |

## Construction

Smart constructors on the `HttpCodec` companion build each atom type. Choose the constructor that matches the HTTP part you are describing.

### Query parameters

To describe a named query parameter, use `HttpCodec.query`:

```scala

val limitCodec: HttpCodec.Query[Int] = HttpCodec.query("limit", Schema.int)
```

Optional fields on `Query` include `default`, `doc`, `examples`, and `deprecated`. To create a query codec with a default value:

```scala

val pageCodec = HttpCodec.query("page", Schema.int, default = Some(1))
```

### Request headers

To describe a request header by name and schema, use `HttpCodec.requestHeader`:

```scala

val traceHeader: HttpCodec.Header[CodecKind.Request, String] =
  HttpCodec.requestHeader("X-Trace-Id", Schema.string)
```

To use a zio-http typed header instance (which provides its own name and parse/render logic), pass the typed header directly:

```scala

val authHeader: HttpCodec.Header[CodecKind.Request, Header.Authorization] =
  HttpCodec.requestHeader(Header.Authorization)
```

### Response headers

To describe a response header, use `HttpCodec.responseHeader`:

```scala

val totalCount: HttpCodec.Header[CodecKind.Response, Int] =
  HttpCodec.responseHeader("X-Total-Count", Schema.int)
```

### Request body

To describe a request body, use `HttpCodec.requestBody`:

```scala

val body: HttpCodec[CodecKind.Request, String] =
  HttpCodec.requestBody(Schema.string)
```

To restrict the accepted content types, pass a `Chunk[MediaType]`:

```scala

val jsonBody: HttpCodec[CodecKind.Request, String] =
  HttpCodec.requestBody(Schema.string, mediaTypes = Chunk.single(MediaTypes.application.`json`))
```

### Response body

To describe a response body, use `HttpCodec.responseBody`:

```scala

val body: HttpCodec[CodecKind.Response, String] =
  HttpCodec.responseBody(Schema.string)
```

### Status codes

To describe a required HTTP status code, use `HttpCodec.status`:

```scala

val created: HttpCodec[CodecKind.Response, Unit] = HttpCodec.status(Status.Created)
```

Predefined status constants are available directly on `HttpCodec`:

```scala

val ok           = HttpCodec.Ok
val created      = HttpCodec.Created
val notFound     = HttpCodec.NotFound
val badRequest   = HttpCodec.BadRequest
val unauthorized = HttpCodec.Unauthorized
```

For any other status code, use `HttpCodec.CustomStatus(code)`.

## Composition

Two operators combine `HttpCodec` values: `++` sequences parts within the same direction, while `|` creates alternatives for content negotiation or multi-status responses.

### Sequential composition with `++`

`++` combines two codecs of the same direction into a single codec whose type is the product of both. The result type is automatically flattened, eliminating `Unit` components and nested tuples:

```scala

val queryAndHeader: HttpCodec[CodecKind.Request, (String, Int)] =
  HttpCodec.query("name", Schema.string) ++ HttpCodec.query("age", Schema.int)
```

The compiler rejects mixing directions — combining a request codec with a response codec is a type error:

```scala

// This would be a compile error:
// HttpCodec.query("name", Schema.string) ++ HttpCodec.status(Status.Ok)
```

### Alternative composition with `|`

`|` combines two codecs as alternatives. The result type is automatically computed as a nested `Either`:

```scala

val okOrCreated =
  (HttpCodec.responseBody(Schema.string) ++ HttpCodec.Ok) |
  (HttpCodec.responseBody(Schema.int) ++ HttpCodec.Created)
```

## Authentication Codecs

Pre-built request codecs for common authorization header schemes are available on `HttpCodec`:

```scala

val basic: HttpCodec[CodecKind.Request, Header.Authorization.Basic]   = HttpCodec.basicAuth
val bearer: HttpCodec[CodecKind.Request, Header.Authorization.Bearer] = HttpCodec.bearerAuth
val digest: HttpCodec[CodecKind.Request, Header.Authorization.Digest] = HttpCodec.digestAuth
val proxy: HttpCodec[CodecKind.Request, Header.ProxyAuthorization]    = HttpCodec.proxyAuthorization
```

These codecs use `Schema.transform` internally to parse the raw `Authorization` header string into the typed zio-http auth model, surfacing a `SchemaError` if the scheme does not match. The message names the class of the parsed header, for example `... but found Bearer`, or `Unparsed` for an unknown scheme.

## Metadata Fields

Every atom node (`Query`, `Header`, `Body`, `StatusCodec`) carries optional metadata that documentation renderers and OpenAPI generators consume:

| Field        | Type              | Purpose                                        |
| ------------ | ----------------- | ---------------------------------------------- |
| `doc`        | `Doc`             | Free-text description for OpenAPI output       |
| `examples`   | `Chunk[(String, A)]` | Example values for the OpenAPI spec         |
| `deprecated` | `Option[Doc]`     | Marks the field as deprecated with a message   |
| `default`    | `Option[A]`       | Default value (query and header only)          |

To create a query codec with documentation and an example:

```scala

val limitCodec = HttpCodec.query(
  name     = "limit",
  schema   = Schema.int,
  default  = Some(20),
  doc      = Doc.empty,
  examples = Chunk("default" -> 20, "max" -> 100)
)
```

---

## Endpoint(Endpoint)

`zio-blocks-endpoint` is a **pure, type-safe HTTP endpoint descriptor** for building clients, servers, and API documentation from a single source of truth. It provides composable types that describe every part of an HTTP surface — routes, query parameters, headers, request bodies, response bodies, error shapes, and authentication — without committing to any particular server or client implementation.

Core types: `Endpoint`, `HttpCodec`, `RoutePattern`, `PathCodec`, `SegmentCodec`, `AuthType`, `RouteTree`. The top-level descriptor holds all of them together:

```scala
final case class Endpoint[PathInput, Input, Err, Output, Auth <: AuthType](
  route: RoutePattern[PathInput],
  input: HttpCodec[CodecKind.Request, Input],
  error: HttpCodec[CodecKind.Response, Err],
  output: HttpCodec[CodecKind.Response, Output],
  auth: Auth,
  doc: Doc
)
```

## Introduction

`zio-blocks-endpoint` separates the **description** of an HTTP surface from its **interpretation**. An `Endpoint` value is plain data — it can be handed to a ZIO HTTP server to generate routes, to a client generator to produce typed API calls, or to an OpenAPI renderer to produce specification documents. None of that interpretation code lives here; this module only describes what an endpoint looks like.

The DSL is designed to stay close to zio-http where that improves ergonomics, while adding precise types for error shapes, authentication, and content negotiation that zio-http does not encode directly.

## Motivation

Without a typed endpoint descriptor, HTTP surface definitions are scattered: routes in one place, request validation in another, error handling in a third. Adding a new endpoint means updating multiple layers by hand and hoping they stay consistent.

`zio-blocks-endpoint` solves this by encoding the full shape of an HTTP endpoint — including error variants, auth requirements, content types, and documentation — into a single composable value:

- **One source of truth**: change the endpoint descriptor and every interpreter (server, client, OpenAPI) updates automatically.
- **Type-safe error channels**: error types are encoded in the `Err` type parameter, not buried in `Either` chains or thrown exceptions.
- **Direction-checked codecs**: `HttpCodec[CodecKind.Request, A]` and `HttpCodec[CodecKind.Response, A]` are distinct types; the compiler prevents accidentally using a response codec where a request codec is expected.
- **Compile-time path validation**: path segment combinations (like `string ~ string`) that would be ambiguous to parse are rejected by the Scala 3 macro in `SegmentCodec` before the code compiles.

## Installation

The endpoint module is a cross-platform library (JVM + Scala.js). Add the dependency to your build definition:

**JVM (Scala 3.x):**
```scala
libraryDependencies += "dev.zio" %% "zio-blocks-endpoint" % "0.0.56"
```

**Scala.js (Scala 3.x):**
```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-endpoint" % "0.0.56"
```

**For Scala 3.10+**, use `zio-blocks-next-endpoint` and add `scalacOptions += "-experimental"` (see [How to Add ZIO Blocks to Your Build](../../getting-started/installation.md)):
```scala
libraryDependencies += "dev.zio" %% "zio-blocks-next-endpoint" % "0.0.56"  // JVM
libraryDependencies += "dev.zio" %%% "zio-blocks-next-endpoint" % "0.0.56" // Scala.js
```

Supported Scala versions: 3.x (Scala 3 only — the endpoint module uses Scala 3-only DSL and macro code).

## Overview

These seven types form the complete endpoint DSL:

**[Endpoint](./endpoint.md)** is the top-level descriptor. It holds a route, three codec channels (input, error, output), an auth type, and documentation. Endpoint is pure data with no server or client logic.

**[HttpCodec](./http-codec.md)** is a composable typed descriptor for HTTP request and response parts. Query parameters, headers, bodies, and status codes are all `HttpCodec` nodes, combined with `++` (sequential) or `|` (alternative).

**[RoutePattern](./route-pattern.md)** pairs an HTTP method with a typed path pattern. The primary syntax is `Method.GET / "users" / PathCodec.int("id")`.

**[PathCodec](./path-codec.md)** is a composable path descriptor. Segments are combined with `/`, and literal alternatives with `orElse`. It supports bidirectional path conversion via `decode` and `format`.

**[SegmentCodec](./segment-codec.md)** describes a single URL path segment. It supports typed segment kinds (`SegmentCodec.bool`, `SegmentCodec.int`, `SegmentCodec.long`, `SegmentCodec.string`, `SegmentCodec.uuid`) and intra-segment composition via `~`, with ambiguous combinations rejected at compile time.

**[AuthType](./auth-type.md)** describes an authentication scheme as a first-class type parameter. Built-in variants include `None`, `Basic`, `Bearer`, and `Digest`; custom schemes and `Or` combinations are also supported.

**[RouteTree](./route-tree.md)** is a routing trie keyed by HTTP method and path. The trie matches literals first, then dynamic segments in priority order. Server-side interpreters use it to build efficient route dispatch tables.

## How They Work Together

A typical endpoint definition flows like this:

```
1. Define a RoutePattern       Method.GET / "users" / PathCodec.int("id")
2. Create an Endpoint          Endpoint(route)
3. Describe request input      .query("verbose", Schema.boolean)
                               .header("X-Trace", Schema.string)
                               .in(Schema.string)
4. Describe success output     .out(Schema.string)
                               .out(Status.Created, Schema.int)
5. Describe error output       .outError(Status.NotFound, Schema.string)
                               .orOutError(Status.Conflict, Schema.int)   // Scala 3 unions
6. Set authentication          .auth(AuthType.Bearer)
7. Add documentation           .doc(Doc.paragraph("Returns a user by ID"))
```

The full type-level view:

```
RoutePattern[PathInput]
  └─ method: Method (GET, POST, ...)
  └─ pathCodec: PathCodec[PathInput]
       └─ Segment(SegmentCodec[A])  ──  literal / int / string / uuid / bool / long / trailing
       └─ Concat(left, right)       ──  left ++ right
       └─ Transform(codec, f, g)    ──  bidirectional type mapping

Endpoint[PathInput, Input, Err, Output, Auth]
  ├─ route:  RoutePattern[PathInput]
  ├─ input:  HttpCodec[Request,  Input]   ──  Query | Header | Body  (combined with ++)
  ├─ error:  HttpCodec[Response, Err]     ──  Body + Status          (alternatives with |)
  ├─ output: HttpCodec[Response, Output]  ──  Body + Status          (alternatives with |)
  └─ auth:   Auth <: AuthType             ──  None | Basic | Bearer | Digest | Custom | Or
```

The phantom type `CodecKind` (either `Request` or `Response`) on `HttpCodec` means the compiler rejects mixing the two directions, even before a server interprets the value.

## Common Patterns

Several composition patterns appear regularly when building endpoints.

**Single success response:** Use `Endpoint#out` for a 200 OK response with a body:

```scala

val getUser = Endpoint(Method.GET / "users" / PathCodec.int("id"))
  .out(Schema.string)
```

**Multiple success variants:** Chain additional `Endpoint#out` calls to add alternatives. The output type widens to an `Either`-based union:

```scala

val createOrUpdate = Endpoint(Method.POST / "users")
  .in(Schema.string)
  .out(Status.Created, Schema.int)
  .out(Status.Ok, Schema.string)
```

**Scala 3 union errors:** Use `Endpoint#orOutError` to accumulate error types as a Scala 3 union rather than nested `Either`s:

```scala

val withUnionErrors = Endpoint(Method.GET / "users")
  .orOutError(Status.NotFound, Schema.string)
  .orOutError(Status.Conflict, Schema.int)

val typed: Endpoint[Unit, Unit, String | Int, Unit, AuthType.None.type] = withUnionErrors
```

**Path prefixing with `RoutePattern#nest`:** Use `RoutePattern#nest` to prepend a version prefix to an existing pattern without rewriting it:

```scala

val route  = Method.GET / "users" / PathCodec.int("id")
val versioned = route.nest(PathCodec("/api/v1"))
```

**Auth composition with `|`:** Combine auth types when an endpoint accepts multiple schemes:

```scala

val flexAuth = Endpoint(Method.GET / "me")
  .auth(AuthType.Basic | AuthType.Bearer)
```

## Integration Points

The endpoint types integrate with each other and with the broader ZIO Blocks ecosystem:

```
Endpoint
  ├─ uses RoutePattern      for routing lookup
  ├─ uses HttpCodec         for all three channels (input, error, output)
  ├─ uses AuthType          to carry the typed auth requirement
  └─ uses Doc               from zio-blocks-docs for API documentation

HttpCodec
  ├─ uses Schema            from zio-blocks-schema for body and header serialization
  ├─ uses MediaType         from zio-blocks-mediatype for content negotiation
  └─ uses Doc               for per-field documentation and examples

RoutePattern
  └─ uses PathCodec         for typed path composition

PathCodec
  └─ uses SegmentCodec      for individual segment descriptors

RouteTree (server-side only)
  └─ uses RoutePattern      to build the routing trie
  └─ uses SegmentSubtree    for per-level trie nodes
```

Cross-module: `zio-blocks-openapi` consumes `Endpoint` values to generate OpenAPI 3.1 specifications. `zio-blocks-schema` provides the `Schema[A]` instances that `HttpCodec.Body` uses for serialization.

## Running the Examples

All code from this section is available as runnable examples in the `endpoint-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

### Basic Endpoint Definition

Constructs `Endpoint` values from `RoutePattern` and chains request body, query parameters, headers, success outputs, response headers, and typed error variants using the builder DSL.

```scala title="endpoint-examples/src/main/scala/endpointexamples/BasicEndpointDefinition.scala" 
package endpointexamples

/**
 * Endpoint — Basic Endpoint Definition
 *
 * Demonstrates constructing an `Endpoint` from a `RoutePattern` and attaching
 * request body, query parameters, headers, success outputs, and error outputs
 * using the builder DSL.
 *
 * Run with: sbt "endpoint-examples/runMain
 * endpointexamples.BasicEndpointDefinition"
 */
@main def BasicEndpointDefinition(): Unit = {

  // Simplest endpoint: GET /health with a string response
  val health = Endpoint(Method.GET / "health")
    .out(Schema.string)

  println(s"Health route: ${health.route.render}")

  // POST with a typed request body and a 201 Created response
  val createUser = Endpoint(Method.POST / "users")
    .in(Schema.string)
    .out(Status.Created, Schema.int)

  println(s"Create user route: ${createUser.route.render}")

  // GET with query parameters and a request header
  val listUsers = Endpoint(Method.GET / "users")
    .query("page", Schema.int)
    .query("limit", Schema.int)
    .header("X-Trace-Id", Schema.string)
    .out(Schema.string)

  println(s"List users route: ${listUsers.route.render}")

  // GET with a dynamic path segment and typed error variants
  val getUser = Endpoint(Method.GET / "users" / PathCodec.int("id"))
    .out(Schema.string)
    .outError(Status.NotFound, Schema.string)
    .outError(Status.BadRequest, Schema.string)

  println(s"Get user route: ${getUser.route.render}")

  // Response header on the success channel
  val withRespHeader = Endpoint(Method.GET / "users")
    .out(Schema.string)
    .outHeader("X-Total-Count", Schema.int)

  println(s"With response header route: ${withRespHeader.route.render}")

  println("BasicEndpointDefinition complete")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/endpoint-examples/src/main/scala/endpointexamples/BasicEndpointDefinition.scala))

```bash
cd endpoint-examples && sbt "runMain endpointexamples.BasicEndpointDefinition"
```

### HttpCodec Smart Constructors and Composition

Builds `HttpCodec` atoms for query parameters, request headers, response headers, request bodies, response bodies, and status codes. Shows sequential composition with `++` and alternative composition with `|`.

```scala title="endpoint-examples/src/main/scala/endpointexamples/HttpCodecConstruction.scala" 
package endpointexamples

/**
 * HttpCodec — Smart Constructors and Composition
 *
 * Demonstrates building `HttpCodec` atoms (query parameters, headers, bodies,
 * status codes) and composing them with `++` (sequential) and `|`
 * (alternative).
 *
 * Run with: sbt "endpoint-examples/runMain
 * endpointexamples.HttpCodecConstruction"
 */
@main def HttpCodecConstruction(): Unit = {

  // --- Smart constructors ---

  // Query parameter with an optional default value
  val pageCodec  = HttpCodec.query("page", Schema.int, default = Some(1))
  val limitCodec = HttpCodec.query("limit", Schema.int)

  println(s"Query 'page'  default: ${pageCodec.default}")
  println(s"Query 'limit' default: ${limitCodec.default}")

  // Request header by name and schema
  val traceHeader = HttpCodec.requestHeader("X-Trace-Id", Schema.string)
  println(s"Request header name: ${traceHeader.name}")

  // Response header
  val totalCount = HttpCodec.responseHeader("X-Total-Count", Schema.int)
  println(s"Response header name: ${totalCount.name}")

  // Request body restricted to JSON
  val jsonBody =
    HttpCodec.requestBody(Schema.string, mediaTypes = Chunk.single(MediaTypes.application.`json`))
  println(s"Request body codec node: ${jsonBody.getClass.getSimpleName}")

  // Response body
  val respBody = HttpCodec.responseBody(Schema.string)
  println(s"Response body codec node: ${respBody.getClass.getSimpleName}")

  // Status code atoms — predefined constants
  println(s"Ok status:       ${HttpCodec.Ok}")
  println(s"Created status:  ${HttpCodec.Created}")
  println(s"NotFound status: ${HttpCodec.NotFound}")

  // --- Sequential composition with ++ ---
  // Combines two request-side codecs into a single codec whose value is a tuple
  val nameAndAgeQuery: HttpCodec[CodecKind.Request, (String, Int)] =
    HttpCodec.query("name", Schema.string) ++ HttpCodec.query("age", Schema.int)

  println(s"Sequential codec: ${nameAndAgeQuery.getClass.getSimpleName}")

  // --- Alternative composition with | ---
  // Builds a fallback: try the left codec, then the right
  val okOrCreated =
    (HttpCodec.responseBody(Schema.string) ++ HttpCodec.Ok) |
      (HttpCodec.responseBody(Schema.int) ++ HttpCodec.Created)

  println(s"Alternative codec: ${okOrCreated.getClass.getSimpleName}")

  // --- Metadata: doc, examples, default ---
  val richQuery = HttpCodec.query(
    name = "limit",
    schema = Schema.int,
    default = Some(20),
    doc = Doc.empty,
    examples = Chunk("default" -> 20, "max" -> 100)
  )
  println(s"Rich query default: ${richQuery.default}")
  println(s"Rich query examples count: ${richQuery.examples.length}")

  // --- Auth codecs ---
  val bearerCodec = HttpCodec.bearerAuth
  val basicCodec  = HttpCodec.basicAuth
  val digestCodec = HttpCodec.digestAuth
  println(s"Bearer codec: ${bearerCodec.getClass.getSimpleName}")
  println(s"Basic codec:  ${basicCodec.getClass.getSimpleName}")
  println(s"Digest codec: ${digestCodec.getClass.getSimpleName}")

  println("HttpCodecConstruction complete")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/endpoint-examples/src/main/scala/endpointexamples/HttpCodecConstruction.scala))

```bash
cd endpoint-examples && sbt "runMain endpointexamples.HttpCodecConstruction"
```

### PathCodec and SegmentCodec

Demonstrates all `SegmentCodec` kinds, intra-segment composition with `~` for patterns like `v42`, bidirectional `PathCodec` decode and format, `RoutePattern` matching, `nest` for version prefixes, and `transform`/`transformOrFail` for domain type mapping.

```scala title="endpoint-examples/src/main/scala/endpointexamples/PathAndSegmentCodecs.scala" 
package endpointexamples

/**
 * PathCodec and SegmentCodec — Typed Path Construction and Decoding
 *
 * Demonstrates `SegmentCodec` kinds, intra-segment composition with `~`,
 * `PathCodec` construction, bidirectional decode/format, `RoutePattern`
 * matching, nesting, and type transformations.
 *
 * Run with: sbt "endpoint-examples/runMain
 * endpointexamples.PathAndSegmentCodecs"
 */
@main def PathAndSegmentCodecs(): Unit = {

  // --- SegmentCodec kinds ---
  val intSeg   = SegmentCodec.int("id")
  val strSeg   = SegmentCodec.string("slug")
  val uuidSeg  = SegmentCodec.uuid("id")
  val boolSeg  = SegmentCodec.bool("flag")
  val longSeg  = SegmentCodec.long("id")
  val trailing = SegmentCodec.Trailing

  println(s"Segment kinds: int=${intSeg.render()}, string=${strSeg.render()}, uuid=${uuidSeg.render()}")
  println(s"  bool=${boolSeg.render()}, long=${longSeg.render()}, trailing=${trailing.render()}")

  // Intra-segment composition: single path segment containing a literal prefix and an integer
  // "v42" decodes to 42 and formats 42 back to "v42"
  val versionSeg: SegmentCodec[Int] =
    SegmentCodec.literal("v") ~ SegmentCodec.int("major")

  println(s"Version segment renders as: ${versionSeg.render()}")
  val formattedVersion: Path = versionSeg.format(3)
  println(s"Version 3 formats to: $formattedVersion")

  // --- PathCodec construction ---
  val usersPath   = PathCodec.literal("users") / PathCodec.int("id")
  val apiPath     = PathCodec("/api/v1/users")
  val versionPath = PathCodec(versionSeg)
  println(s"versionPath renders as: ${versionPath.render}")

  println(s"usersPath renders as: ${usersPath.render}")
  println(s"apiPath renders as:   ${apiPath.render}")

  // Bidirectional: decode a Path to a typed value
  val decoded: Either[String, Int] = PathCodec.int("id").decode(Path("/42"))
  println(s"Decoded /42: $decoded")

  // Bidirectional: format a typed value back to a Path
  val uuidValue = java.util.UUID.fromString("550e8400-e29b-41d4-a716-446655440000")
  val formatted = PathCodec.uuid("id").format(uuidValue)
  println(s"Formatted UUID: $formatted")

  // Literal alternatives: match either /users or /members
  val eitherPath: PathCodec[Unit] =
    PathCodec.literal("users").orElse(PathCodec.literal("members"))

  println(s"orElse matches /users: ${eitherPath.matches(Path("/users"))}")
  println(s"orElse matches /members: ${eitherPath.matches(Path("/members"))}")

  // --- RoutePattern construction and operations ---
  val route = Method.GET / "users" / PathCodec.int("id")

  // Decode: extract a typed value from a method + path
  val routeDecoded: Either[String, Int] = route.decode(Method.GET, Path("/users/42"))
  println(s"Route decoded: $routeDecoded")

  // Encode: rebuild (Method, Path) from a typed value
  val routeEncoded: Either[String, (Method, Path)] = route.encode(42)
  println(s"Route encoded: $routeEncoded")

  // Render: human-readable string matching OpenAPI path parameter convention
  println(s"Route rendered: ${route.render}")

  // Nest: prepend a version prefix to an existing pattern
  val versioned = route.nest(PathCodec("/api/v1"))
  println(s"Versioned route: ${versioned.render}")

  // --- Type transformations ---
  final case class UserId(value: Int)

  val userIdCodec: PathCodec[UserId] =
    PathCodec.int("id").transform(UserId(_), _.value)

  val decodedUserId = userIdCodec.decode(Path("/99"))
  println(s"UserId decoded: $decodedUserId")

  // transformOrFail: reject non-positive segment values at parse time
  val positiveInt: PathCodec[Int] =
    PathCodec
      .int("count")
      .transformOrFail(
        n => if (n > 0) Right(n) else Left(s"Expected positive, got $n"),
        (n: Int) => Right(n)
      )

  println(s"Positive decode 5:  ${positiveInt.decode(Path("/5"))}")
  println(s"Positive decode -1: ${positiveInt.decode(Path("/-1"))}")

  println("PathAndSegmentCodecs complete")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/endpoint-examples/src/main/scala/endpointexamples/PathAndSegmentCodecs.scala))

```bash
cd endpoint-examples && sbt "runMain endpointexamples.PathAndSegmentCodecs"
```

### AuthType Patterns

Shows all built-in `AuthType` variants (None, Basic, Bearer, Digest), a custom API-key variant, OR composition with `|`, scoped bearer tokens for OAuth scope metadata, and overriding the default unauthorized status.

```scala title="endpoint-examples/src/main/scala/endpointexamples/AuthTypePatterns.scala" 
package endpointexamples

/**
 * AuthType — Authentication Scheme Variants and Composition
 *
 * Demonstrates all built-in `AuthType` variants (None, Basic, Bearer, Digest),
 * the Custom variant for API keys, OR composition with `|`, scoped bearer
 * tokens, and overriding the default unauthorized status.
 *
 * Run with: sbt "endpoint-examples/runMain endpointexamples.AuthTypePatterns"
 */
@main def AuthTypePatterns(): Unit = {

  // --- AuthType.None (default) ---
  // No authentication required; every new Endpoint starts with None
  val publicEndpoint = Endpoint(Method.GET / "health")
    .out(Schema.string)

  println(s"Public endpoint auth: ${publicEndpoint.auth}")

  // --- AuthType.Basic ---
  val basicEndpoint = Endpoint(Method.GET / "admin")
    .auth(AuthType.Basic)

  println(s"Basic auth unauth status: ${basicEndpoint.auth.unauthorizedStatus}")

  // --- AuthType.Bearer ---
  val bearerEndpoint = Endpoint(Method.GET / "me")
    .auth(AuthType.Bearer)

  println(s"Bearer auth unauth status: ${bearerEndpoint.auth.unauthorizedStatus}")

  // --- AuthType.Digest ---
  val digestEndpoint = Endpoint(Method.GET / "secure")
    .auth(AuthType.Digest)

  println(s"Digest auth unauth status: ${digestEndpoint.auth.unauthorizedStatus}")

  // --- AuthType.Custom ---
  // Wrap any HttpCodec for schemes not covered by the built-in variants
  val apiKeyCodec = HttpCodec.requestHeader("X-Api-Key", Schema.string)
  val apiKeyAuth  = AuthType.Custom(apiKeyCodec)
  val keyEndpoint = Endpoint(Method.GET / "data").auth(apiKeyAuth)

  println(s"Custom auth unauth status: ${keyEndpoint.auth.unauthorizedStatus}")

  // --- OR composition: accept either scheme ---
  // The codec tries the left scheme first and falls back to the right
  val flexEndpoint = Endpoint(Method.GET / "resource")
    .auth(AuthType.Basic | AuthType.Bearer)

  println(s"Flex auth unauth status: ${flexEndpoint.auth.unauthorizedStatus}")

  // --- Scoped bearer: attach OAuth scope metadata ---
  val scopedEndpoint = Endpoint(Method.GET / "admin")
    .auth(AuthType.Scoped(AuthType.Bearer, List("admin:read", "admin:write")))

  println(s"Scoped auth unauth status: ${scopedEndpoint.auth.unauthorizedStatus}")

  // --- Override the default unauthorized status ---
  // Default is Status.NotFound (to avoid leaking endpoint existence);
  // override to Status.Unauthorized when endpoint existence is public knowledge
  val strictEndpoint = Endpoint(Method.GET / "me")
    .auth(AuthType.Bearer)
    .unauthorizedStatus(Status.Unauthorized)

  println(s"Strict unauth status: ${strictEndpoint.auth.unauthorizedStatus}")

  println("AuthTypePatterns complete")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/endpoint-examples/src/main/scala/endpointexamples/AuthTypePatterns.scala))

```bash
cd endpoint-examples && sbt "runMain endpointexamples.AuthTypePatterns"
```

### Complete REST API

Assembles a full users CRUD API combining `Endpoint`, `RoutePattern`, `PathCodec`, `SegmentCodec`, `HttpCodec`, `AuthType`, and `RouteTree`. Demonstrates versioned routes via `nest`, Scala 3 union error types via `orOutError`, and `RouteTree` lookup priority for efficient O(depth) dispatch.

```scala title="endpoint-examples/src/main/scala/endpointexamples/CompleteApiDefinition.scala" 
package endpointexamples

/**
 * Complete REST API — All Endpoint Types Working Together
 *
 * Shows a full users CRUD API built from `Endpoint`, `RoutePattern`,
 * `PathCodec`, `SegmentCodec`, `HttpCodec`, `AuthType`, and `RouteTree`.
 * Demonstrates versioned routes via `nest`, Scala 3 union error types via
 * `orOutError`, and `RouteTree` lookup priority.
 *
 * Run with: sbt "endpoint-examples/runMain
 * endpointexamples.CompleteApiDefinition"
 */
@main def CompleteApiDefinition(): Unit = {

  // --- Domain type with a typed path codec ---
  final case class UserId(value: Int)

  val userIdPath: PathCodec[UserId] =
    PathCodec.int("id").transform(UserId(_), _.value)

  // --- Endpoint definitions ---

  // GET /users?page=&limit=  — paginated list, bearer-secured
  val listUsers = Endpoint(Method.GET / "users")
    .query("page", Schema.int)
    .query("limit", Schema.int)
    .out(Schema.string)
    .outError(Status.BadRequest, Schema.string)
    .auth(AuthType.Bearer)

  // GET /users/{id}  — fetch a single user, bearer-secured
  val getUser = Endpoint(Method.GET / "users" / userIdPath)
    .out(Schema.string)
    .outError(Status.NotFound, Schema.string)
    .auth(AuthType.Bearer)

  // POST /users  — create a user, 201 on success, bearer-secured
  val createUser = Endpoint(Method.POST / "users")
    .in(Schema.string)
    .out(Status.Created, Schema.int)
    .outError(Status.BadRequest, Schema.string)
    .outError(Status.Conflict, Schema.string)
    .auth(AuthType.Bearer)

  // DELETE /users/{id}  — remove a user, bearer-secured
  val deleteUser = Endpoint(Method.DELETE / "users" / PathCodec.int("id"))
    .out(Status.NoContent, Schema.unit)
    .outError(Status.NotFound, Schema.string)
    .auth(AuthType.Bearer)

  // GET /health  — public health check, no auth required
  val health = Endpoint(Method.GET / "health")
    .out(Schema.string)

  // --- Union error types (Scala 3 only) ---
  // orOutError accumulates error types as a native union instead of nested Eithers;
  // the first call sets Err directly, subsequent calls widen to a union
  val withUnionErrors = Endpoint(Method.GET / "items" / PathCodec.int("id"))
    .orOutError(Status.NotFound, Schema.string)
    .orOutError(Status.Conflict, Schema.int)

  val _: Endpoint[Int, Unit, String | Int, Unit, AuthType.None.type] = withUnionErrors

  // --- Versioned routes via nest ---
  val v1ListUsers = listUsers.route.nest(PathCodec("/api/v1"))
  val v2ListUsers = listUsers.route.nest(PathCodec("/api/v2"))

  println("Endpoint routes:")
  println(s"  ${listUsers.route.render}")
  println(s"  ${getUser.route.render}")
  println(s"  ${createUser.route.render}")
  println(s"  ${deleteUser.route.render}")
  println(s"  ${health.route.render}")
  println(s"  v1: ${v1ListUsers.render}")
  println(s"  v2: ${v2ListUsers.render}")

  // --- RouteTree: O(depth) routing trie ---
  // Literals are matched first; dynamic segments follow priority ordering
  // (int > long > uuid > bool > string > combined > trailing)
  val tree = RouteTree
    .empty[String]
    .add(Method.GET / "users", "list-users")
    .add(Method.GET / "users" / PathCodec.int("id"), "get-user")
    .add(Method.POST / "users", "create-user")
    .add(Method.DELETE / "users" / PathCodec.int("id"), "delete-user")
    .add(Method.GET / "health", "health")

  println("\nRouteTree lookups:")
  println(s"  GET    /users       → ${tree.get(Method.GET, Path("/users"))}")
  println(s"  GET    /users/42    → ${tree.get(Method.GET, Path("/users/42"))}")
  println(s"  POST   /users       → ${tree.get(Method.POST, Path("/users"))}")
  println(s"  DELETE /users/7     → ${tree.get(Method.DELETE, Path("/users/7"))}")
  println(s"  GET    /health      → ${tree.get(Method.GET, Path("/health"))}")
  // HEAD falls back to GET per HTTP spec
  println(s"  HEAD   /users       → ${tree.get(Method.HEAD, Path("/users"))}")
  // Unregistered path returns None
  println(s"  GET    /notfound    → ${tree.get(Method.GET, Path("/notfound"))}")

  // --- RouteTree merge: right-hand side wins on conflict ---
  val treeA  = RouteTree.empty[String].add(Method.GET / "users", "users-v1")
  val treeB  = RouteTree.empty[String].add(Method.GET / "users", "users-v2")
  val merged = treeA.merge(treeB)
  println(s"\nMerged GET /users → ${merged.get(Method.GET, Path("/users"))}")

  // --- RoutePattern decode and encode ---
  val route   = Method.GET / "users" / PathCodec.int("id")
  val decoded = route.decode(Method.GET, Path("/users/99"))
  val encoded = route.encode(99)
  println(s"\nRoute decode /users/99 → $decoded")
  println(s"Route encode 99        → $encoded")

  println("\nCompleteApiDefinition complete")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/endpoint-examples/src/main/scala/endpointexamples/CompleteApiDefinition.scala))

```bash
cd endpoint-examples && sbt "runMain endpointexamples.CompleteApiDefinition"
```

**3. Or compile all examples at once:**

```bash
cd endpoint-examples && sbt "compile"
```

---

## PathCodec

`PathCodec[A]` is a composable descriptor for URL path structures. It holds a tree of segment codecs connected by concatenation and fallback nodes, and provides bidirectional path conversion: `PathCodec#decode` extracts a typed value from a `Path`, returning `Either[String, A]` (the typed value or error), and `PathCodec#format` formats a typed value back to a `Path`, returning `Either[String, Path]` (the path or error). Its definition begins:

```scala
sealed trait PathCodec[A]
```

## Motivation

URL paths need both matching and generation. A routing library that only matches paths requires a separate URL-builder for links and redirects, leading to duplication and drift. `PathCodec` is bidirectional: every codec that can decode `/users/42` into `42: Int` can also format `42` back into `/users/42`. This makes it safe to use the same path definition for routing, link generation, and OpenAPI path parameter documentation.

## Structure

`PathCodec` is an ADT with four node types:

| Node        | Meaning                                                        |
| ----------- | -------------------------------------------------------------- |
| `Segment`   | A single path segment, described by a `SegmentCodec[A]`        |
| `Concat`    | Two path codecs composed sequentially with `/` or `++`         |
| `Transform` | Bidirectional type mapping over an existing codec              |
| `Fallback`  | Two literal alternatives (applies only with `orElse`)          |

## Construction

We build a `PathCodec` from smart constructors that produce typed segment nodes, from a path string, or by wrapping a `SegmentCodec` directly.

### Predefined segment constructors

The most common path building blocks are available as smart constructors on the companion:

```scala

val literalUsers: PathCodec[Unit]          = PathCodec.literal("users")
val intId: PathCodec[Int]                  = PathCodec.int("id")
val longId: PathCodec[Long]               = PathCodec.long("id")
val stringSlug: PathCodec[String]         = PathCodec.string("slug")
val uuidId: PathCodec[java.util.UUID]     = PathCodec.uuid("id")
val boolFlag: PathCodec[Boolean]          = PathCodec.bool("enabled")
val rest: PathCodec[zio.http.Path]        = PathCodec.trailing
```

`PathCodec.literal` validates the value — it rejects empty strings and strings containing `/` or characters requiring URL encoding.

### From a string

To build a codec from a slash-separated string of literal segments, use `PathCodec.apply`:

```scala

val path: PathCodec[Unit] = PathCodec("/api/v1/users")
```

This is equivalent to concatenating `PathCodec.literal` for each segment.

### From a `SegmentCodec`

To wrap a custom `SegmentCodec[A]` into a `PathCodec[A]`, use `PathCodec.apply(segment)`:

```scala

val combined = PathCodec(SegmentCodec.literal("v") ~ SegmentCodec.int("version"))
```

There is also an implicit conversion from `SegmentCodec[A]` to `PathCodec[A]` and from `String` to `PathCodec[Unit]`, so both can appear directly in `/` expressions.

## Marking a capture as unused with `.unused`

Sometimes a route needs to capture a segment for matching or formatting, but a downstream handler intentionally does not consume that variable. `PathCodec` instances expose `.unused`, which converts the value type to `Unit` while keeping the same path shape:

```scala

val userId: PathCodec[Int] = PathCodec.int("id")
val ignoredUserId: PathCodec[Unit] = PathCodec.int("id").unused
```

`.unused` converts the path codec's value type to `Unit`: the route still matches the same shape and renders `{name}` placeholders, but decoding yields `Unit` instead of the captured value, so no handler parameter is required. `format` on an unused path fails because there is no value to encode back into the ignored segment.

`PathCodec.unused` composes normally with `/` and is unaffected by `transform` / `transformOrFail` lifting.

## Composition

Path codecs compose in two ways: sequential concatenation with `/` or `++`, and literal alternatives with `orElse`.

### Sequential composition with `/` and `++`

`/` and `++` are equivalent: both concatenate two path codecs. The result type is flattened automatically (so `Unit / Int` gives `Int`, not `(Unit, Int)`), eliminating `Unit` components:

```scala

val route: PathCodec[Int] = PathCodec.literal("users") / PathCodec.int("id")
```

In the context of a `RoutePattern`, the same operator works directly:

```scala

val pattern = Method.GET / "users" / PathCodec.int("id") / "posts"
```

### Literal alternatives with `orElse`

To match either of two literal segments, use `orElse`:

```scala

val either: PathCodec[Unit] =
  PathCodec.literal("users").orElse(PathCodec.literal("members"))
```

`orElse` is for literal alternatives only. Both branches must be `PathCodec[Unit]` with no captured path variables, and `PathCodec#alternatives` still validates at runtime that the branches are genuinely literal-only. In practice, use `orElse` only with `PathCodec.literal(...)` or string-literal path codecs.

## Decoding and Formatting

`PathCodec` is bidirectional: `PathCodec#decode` turns a runtime `Path` into a typed value, and `PathCodec#format` turns a typed value back into a `Path`.

### `PathCodec#decode`

To extract a typed value from a runtime `Path`:

```scala

val codec = PathCodec.int("id")

val result: Either[String, Int] = codec.decode(Path("/42"))
```

`PathCodec#decode` returns `Left(message)` when no segment matches or a segment cannot be parsed.

### `PathCodec#format`

To turn a typed value into a `Path`:

```scala

val codec = PathCodec.uuid("id")

val path: Either[String, Path] =
  codec.format(java.util.UUID.fromString("550e8400-e29b-41d4-a716-446655440000"))
```

### `PathCodec#matches`

To test whether a `Path` matches without extracting a value:

```scala

val codec   = PathCodec.literal("users")
val matched = codec.matches(Path("/users"))
```

## Type Transformations

Use these methods to map the typed value that `PathCodec` decodes or encodes without changing the underlying path structure.

### `PathCodec#transform`

To map the decoded value to a different type without changing the path structure, use `PathCodec#transform`. Both directions must be total:

```scala

final case class UserId(value: Int)

val userIdCodec: PathCodec[UserId] =
  PathCodec.int("id").transform(UserId(_), _.value)
```

### `PathCodec#transformOrFail`

When decoding or encoding can fail, use `PathCodec#transformOrFail`. A `Left` from the decode function causes the path not to match:

```scala

val nonNegativeInt: PathCodec[Int] =
  PathCodec.int("count").transformOrFail(
    n => if (n >= 0) Right(n) else Left(s"Expected non-negative, got $n"),
    (n: Int) => Right(n)
  )
```

## Rendering

`PathCodec#render` produces a human-readable path string. Dynamic segments appear as `{name}` by default:

```scala

val codec = PathCodec.literal("users") / PathCodec.int("id")
val rendered: String = codec.render
```

To use a different prefix/suffix for dynamic segments (e.g., `:id` for Express-style paths), call the lower-level `PathCodec.render(codec, prefix = ":", suffix = "")`.

---

## RoutePattern

`RoutePattern[A]` pairs an HTTP method with a typed path pattern. It is the primary routing descriptor in `zio-blocks-endpoint`: every `Endpoint` carries a `RoutePattern` that determines which HTTP method and URL path it matches. Its shape is:

```scala
final case class RoutePattern[A](
  method: Method,
  pathCodec: PathCodec[A],
  doc: Doc = Doc.empty
)
```

## Motivation

HTTP routing requires matching both a method (GET, POST, …) and a path (`/users/42`). `RoutePattern` holds both in a single typed value. The type parameter `A` is the type of values extracted from the dynamic path segments — `Unit` for fully-literal paths, `Int` for a single integer segment, `(String, UUID)` for two dynamic segments, and so on.

Using a typed route pattern means route construction and path extraction are verified at compile time: the type of the extracted path value is always consistent with the path codec definition.

## Construction

Several construction forms exist. The method-first syntax is the most common and most readable; the others cover less typical use cases.

### Method-first syntax (recommended)

The primary syntax uses the `Method` extension method `/` to produce a `RoutePattern` directly:

```scala

val getUsers    = Method.GET / "users"
val postUser    = Method.POST / "users"
val deleteOrder = Method.DELETE / "orders" / PathCodec.uuid("orderId")
```

This is the recommended construction style: it reads like the route itself and keeps the method close to the path.

### Constant constructors

Pre-built method-only patterns are available as constants on the `RoutePattern` companion:

```scala

val get     = RoutePattern.GET
val post    = RoutePattern.POST
val put     = RoutePattern.PUT
val delete  = RoutePattern.DELETE
val patch   = RoutePattern.PATCH
val head    = RoutePattern.HEAD
val options = RoutePattern.OPTIONS
```

These are equivalent to `RoutePattern(Method.GET)` etc., with an empty path codec. Additional constants like `CONNECT` and `TRACE` are also available for the complete set of HTTP methods.

### From a `Path` value

To construct a pattern from a runtime `Path` (all literal segments), use `RoutePattern.apply(method, path)`:

```scala

val route = RoutePattern(Method.GET, Path("/users/active"))
```

### Catch-all trailing patterns

To match any path suffix, use `RoutePattern.any`:

```scala

val catchAll: RoutePattern[Path]   = RoutePattern.any
val getAny: RoutePattern[Path]     = RoutePattern.any(Method.GET)
```

A trailing catch-all captures a `zio.http.Path` runtime value without declaring any named path variables.

## Path Composition with `/`

To append additional `PathCodec` segments to a `RoutePattern`, use the `/` method:

```scala

val route = Method.GET / "users" / PathCodec.int("id") / "posts"
```

Each `/` call produces a new `RoutePattern` with a widened type. The type is automatically flattened, eliminating `Unit` components: `Unit / Int / Unit` becomes `Int`, not `((Unit, Int), Unit)`.

## Decoding and Encoding

`RoutePattern` is bidirectional: `RoutePattern#decode` validates a live request against the pattern, and `RoutePattern#encode` rebuilds a request pair from a typed path value.

### `RoutePattern#decode`

To check whether a method and path match, and extract the typed path value:

```scala

val route = Method.GET / "users" / PathCodec.int("id")

val result: Either[String, Int] = route.decode(Method.GET, Path("/users/42"))
```

`RoutePattern#decode` returns `Left` if the method does not match or any segment fails to parse, and `Right(value)` with the extracted path value otherwise. HEAD requests automatically fall back to GET for compatibility.

### `RoutePattern#encode`

To turn a typed value back into a `(Method, Path)` pair:

```scala

val route = Method.POST / "orders" / PathCodec.uuid("orderId")

val result: Either[String, (Method, Path)] = route.encode(java.util.UUID.fromString("550e8400-e29b-41d4-a716-446655440000"))
```

### `RoutePattern#matches`

To test membership without extracting the value:

```scala

val route   = Method.GET / "users"
val matches = route.matches(Method.GET, Path("/users"))
```

## Structural Operations

Three structural operations transform an existing `RoutePattern` without building a new one from scratch: `RoutePattern#alternatives`, `RoutePattern#nest`, and `RoutePattern#render`.

### Alternatives

`RoutePattern#alternatives` expands `Method.ANY` and `Method.Methods` into a flat list of single-method patterns. `RouteTree` calls this before inserting into the trie:

```scala

val anyGet: RoutePattern[?] = RoutePattern.any(Method.GET)
val expanded = anyGet.alternatives
```

### Nesting

`RoutePattern#nest` prepends a literal path prefix without modifying the route's type or dynamic segments:

```scala

val route   = Method.GET / "users" / PathCodec.int("id")
val versioned = route.nest(PathCodec("/api/v1"))
```

This is useful for adding a version prefix to a group of existing routes without rewriting each one.

### Rendering

`RoutePattern#render` produces a human-readable string representation, useful for logging and OpenAPI path generation:

```scala

val route = Method.GET / "users" / PathCodec.int("id")
val rendered: String = route.render
```

Dynamic segments are rendered as `{name}` by default, matching OpenAPI path parameter convention.

---

## RouteTree

`RouteTree[A]` is a routing trie keyed by HTTP method and path. It maps `(Method, Path)` pairs to values of type `A`, performing prefix-tree lookup with segment-priority ordering. Server-side interpreters primarily use it to build efficient route dispatch tables from a collection of `RoutePattern` values. Its structure is:

```scala
final case class RouteTree[A](
  roots: Map[Method, SegmentSubtree[A]]
)
```

`SegmentSubtree[A]` is a single level of the trie, holding literal-keyed branches and priority-ordered dynamic segment branches:

```scala
final case class SegmentSubtree[A](
  literals: Map[String, SegmentSubtree[A]],
  others: ListMap[SegmentCodec.Key, (SegmentCodec[_], SegmentSubtree[A])],
  value: Option[A]
)
```

## Motivation

Routing a `(Method, Path)` pair to a handler requires checking many patterns efficiently. A flat scan through all routes is O(n) and scales poorly. `RouteTree` uses a prefix trie keyed on path segments, so matching is O(depth) — proportional to the number of segments in the path, not the number of registered routes.

Within each trie level, literal segments are stored in a `Map[String, ...]` for O(1) lookup, while dynamic segments are stored in a `ListMap` ordered by match priority. The priority order (literal → int → long → uuid → bool → string → combined → trailing) ensures that more specific segments win ambiguous matches.

## Building a `RouteTree`

Start from an empty tree and add patterns with `RouteTree#add`:

```scala

val tree = RouteTree.empty[String]
  .add(Method.GET / "users", "list-users")
  .add(Method.GET / "users" / PathCodec.int("id"), "get-user")
  .add(Method.POST / "users", "create-user")
```

`RouteTree#add` calls `RoutePattern#alternatives` internally to expand `Method.ANY` and `Method.Methods` into individual method entries before inserting into the trie.

## Lookup

`RouteTree#get` looks up a `(Method, Path)` pair and returns the associated value if a match is found:

```scala

val tree = RouteTree.empty[String]
  .add(Method.GET / "users" / PathCodec.int("id"), "get-user")

val result: Option[String] = tree.get(Method.GET, Path("/users/42"))
```

A HEAD request with no matching HEAD route falls back to the GET subtree, matching the HTTP specification. The fallback also applies when a HEAD route exists but none of them matches the path.

## Merging

`RouteTree#merge` combines two trees. On conflicts, the right-hand-side value takes precedence:

```scala

val treeA = RouteTree.empty[String].add(Method.GET / "users", "users-v1")
val treeB = RouteTree.empty[String].add(Method.GET / "users", "users-v2")

val merged = treeA.merge(treeB)
// GET /users → "users-v2"
```

## Mapping

`RouteTree#map` transforms all values in the trie while preserving its structure:

```scala

val tree: RouteTree[String] = RouteTree.empty[String]
  .add(Method.GET / "users", "get-users")

val lengths: RouteTree[Int] = tree.map(_.length)
```

## Match Priority

Within a single trie level, `SegmentSubtree` tries literal branches first, then dynamic branches in priority order. Given a path like `/users/active`:

1. `literals.get("active")` is tried first — if an exact literal route for `"active"` exists, it matches.
2. Otherwise, dynamic segment codecs are tried in order: `Int` (fails, `"active"` is not numeric), `Long` (fails), `UUID` (fails), `Bool` (fails), `String` (succeeds).

This means `/users/42` matches an `Int` route even if a `String` route is also registered, because integers are higher priority than strings.

## `SegmentSubtree`

`SegmentSubtree` is the internal per-level trie node. It is not typically constructed directly — `RouteTree#add` builds it internally. The two key fields are:

- **`literals`**: a `Map[String, SegmentSubtree[A]]` for O(1) exact-match lookups.
- **`others`**: a `ListMap[SegmentCodec.Key, (SegmentCodec[_], SegmentSubtree[A])]` ordered by priority for dynamic segment matching.

The `value: Option[A]` field holds the registered value when a complete path terminates at this node. Trailing segments have special handling: if a `Trailing` codec is registered and the current index is past the end of the path segments, the lookup returns its subtree value.

---

## SegmentCodec

`SegmentCodec[A]` describes a single URL path segment. It supports basic typed segment kinds — `SegmentCodec.bool`, `SegmentCodec.int`, `SegmentCodec.long`, `SegmentCodec.string`, `SegmentCodec.uuid`, and `SegmentCodec.literal` — as well as intra-segment composition via `~`, which combines multiple typed parts within a single path segment (for example, `v42` as a literal prefix followed by an integer). Ambiguous adjacencies (`string ~ string`, numeric ~ numeric in any `Int`/`Long` order, anything involving `Trailing`) are rejected at runtime with `IllegalArgumentException`. Combine representation first, then transform: `transform` / `transformOrFail` lift a segment into a `PathCodec`, ending intra-segment composition. The core type-level shape is:

```scala
sealed trait SegmentCodec[A]
```

(The trait also includes methods for `~` composition, `transform` lifting, documentation, examples, formatting, and rendering.)

## Motivation

Standard routing libraries treat path segments as plain strings, deferring all parsing to handler code. `SegmentCodec` encodes the type of each segment statically, so the compiler knows whether a segment produces an `Int`, a `UUID`, or a custom domain type. This enables:

- **Type-safe path building**: `PathCodec.int("id")` produces a `PathCodec[Int]`, not `PathCodec[String]`.
- **Bidirectional conversion**: every `SegmentCodec` can both decode a string into `A` and format an `A` back to a string.
- **Fail-fast combination validation**: the `~` operator rejects ambiguous adjacencies (`string ~ string`, numeric ~ numeric) at composition time, so bad splits surface immediately instead of misrouting.

## Segment Kinds

Each segment kind maps to a specific URL token type. Literal segments match a fixed string; the others capture and parse dynamic values.

### Literal segments

A `Literal` segment matches exactly one fixed string value. Use `SegmentCodec.literal` with a compile-time string constant:

```scala

val usersLit: SegmentCodec.Literal = SegmentCodec.literal("users")
```

`SegmentCodec.literal` is a macro: it validates that the value is a non-empty, URL-safe, single-segment string at compile time and rejects `""`, `"foo/bar"`, or strings requiring percent-encoding.

### Typed dynamic segments

Standard typed segment constructors each accept a name string for documentation and OpenAPI path parameter labels:

```scala

val boolSeg: SegmentCodec[Boolean]        = SegmentCodec.bool("flag")
val intSeg: SegmentCodec[Int]             = SegmentCodec.int("id")
val longSeg: SegmentCodec[Long]           = SegmentCodec.long("id")
val stringSeg: SegmentCodec[String]       = SegmentCodec.string("slug")
val uuidSeg: SegmentCodec[java.util.UUID] = SegmentCodec.uuid("id")
```

If a route should capture a segment for matching or formatting but the handler intentionally does not consume the variable, the leaf dynamic segment codecs expose `.unused`:

```scala

val requiredId = SegmentCodec.int("id")
val ignoredId  = SegmentCodec.int("id").unused
```

`.unused` keeps decoding, formatting, rendering, and composition identical. The only difference is at the type level: downstream tooling (handler macros, static checks) sees the variable as intentionally unused and does not require the handler to bind it.

The ordering of match priority in the routing trie follows the kind: `Literal` matches first, then `Int`, `Long`, `UUID`, `Bool`, `String`, `Combined`, and `Trailing` last.

### Trailing segment

`SegmentCodec.Trailing` captures all remaining path segments as a `zio.http.Path`. Use it for wildcard routes; it always has the lowest priority in the trie:

```scala

val rest: SegmentCodec[Path] = SegmentCodec.Trailing
```

## Intra-Segment Composition with `~`

The `~` operator combines two `SegmentCodec` values into a `Combined` codec that matches a single path segment containing both parts consecutively. This is useful for version strings like `v42` or prefixed identifiers like `usr-550e8400`:

```scala

val versionSeg: SegmentCodec[Int] =
  SegmentCodec.literal("v") ~ SegmentCodec.int("major")
```

The type is automatically flattened (eliminating `Unit` from the literal), so the resulting codec decodes `"v42"` into `42` and formats `42` back to `"v42"`.

### Runtime adjacency validation

The `~` operator checks the physical adjacency at composition time. Two categories of combination are always rejected with `IllegalArgumentException`:

- **Two string segments**: `SegmentCodec.string("a") ~ SegmentCodec.string("b")` — both are unbounded greedy matchers; the parser cannot know where one ends and the other begins.
- **Two numeric segments**: `SegmentCodec.int("a") ~ SegmentCodec.int("b")` — numeric segments are also ambiguously bounded (any `Int`/`Long` order, including through a flattened combined tail).

Anything involving `Trailing` is likewise rejected. Combinations that are safe compose normally:

```scala

// Safe: literal delimiter separates two dynamic segments
val versionMajorMinor =
  SegmentCodec.literal("v") ~ SegmentCodec.int("major") ~
  SegmentCodec.literal("x") ~ SegmentCodec.int("minor")

// Safe: UUID in the middle, bounded by strings on both sides
val prefixedUuid =
  SegmentCodec.string("prefix") ~ SegmentCodec.uuid("id") ~ SegmentCodec.string("suffix")
```

Attempting an ambiguous combination like `string ~ string` throws `IllegalArgumentException` at composition time, not a silent misroute.

## Type Transformations

Combine representation first, then transform: `transform` / `transformOrFail` map the decoded segment value into a domain type and lift the segment into a `PathCodec`, ending intra-segment composition (a transformed codec no longer offers `~`; compose with `/` instead).

### `SegmentCodec#transform`

To map the decoded segment value to a different type, use `SegmentCodec#transform`. Both the decode and encode functions must be total:

```scala

final case class UserId(value: java.util.UUID)

val userIdCodec: PathCodec[UserId] =
  SegmentCodec.uuid("id").transform(UserId(_), _.value)
```

### `SegmentCodec#transformOrFail`

When decoding can fail, use `SegmentCodec#transformOrFail`. A `Left` result causes segment matching to fail for that candidate:

```scala

final case class PositiveInt(value: Int)

val positiveIntCodec: PathCodec[PositiveInt] =
  SegmentCodec.int("count").transformOrFail(
    n => if (n > 0) Right(PositiveInt(n)) else Left(s"Expected positive, got $n"),
    p => Right(p.value)
  )
```

## Formatting and Rendering

These methods convert a typed value back to a string for use in URL construction and documentation output.

### `SegmentCodec#format`

To format a typed value into a single-segment `Path`:

```scala

val seg  = SegmentCodec.int("id")
val path: Path = seg.format(42)
```

### `SegmentCodec#render`

To produce the human-readable representation of the segment for use in OpenAPI path parameters and documentation:

```scala

val intSeg = SegmentCodec.int("id")
val rendered: String = intSeg.render()
```

By default, `SegmentCodec.render` produces `/{name}` for dynamic segments and `/{value}` for literal segments (including the leading slash in both cases). This leading slash is part of the segment's representation, not added by path composition. To customize the format, pass prefix and suffix arguments: `intSeg.render(":", "")` produces `/:id`. Both dynamic and literal segments always include this leading slash in their rendered form.

## Priority Ordering

When `RouteTree` builds a routing trie, it uses `SegmentCodec.Kind` ordering to resolve ambiguous matches. Literals always win; within dynamic segments, more specific types take priority:

| Priority | Kind       |
| -------- | ---------- |
| 1        | `Literal`  |
| 2        | `Int`      |
| 3        | `Long`     |
| 4        | `UUID`     |
| 5        | `Bool`     |
| 6        | `String`   |
| 7        | `Combined` |
| 8        | `Trailing` |

This ordering ensures that `/users/42` matches an `Int` route before a `String` route, and `/users/active` matches a `String` route because `"active"` is not a valid integer.

---

## HTML

`zio-blocks-html` is a **type-safe HTML templating library** providing immutable data structures and a fluent DSL for building HTML, CSS, and JavaScript. It offers compile-time safety, automatic XSS protection, and zero-dependency simplicity across Scala 2.13, 3.x, and both JVM and Scala.js platforms.

Core types: `Dom` (HTML tree), `CssSelector` (CSS queries), `DomSelection` (DOM navigation), `Css` (stylesheets), `Js` (JavaScript expressions).

The module is structured around these core types:

```

// Dom variants
sealed trait Dom
case class Dom.Text(content: String) extends Dom
sealed trait Dom.Element extends Dom
case class Dom.Element.Generic(tag: String, attributes: Chunk[Dom.Attribute], children: Chunk[Dom]) extends Dom.Element
case class Dom.Element.Script(attributes: Chunk[Dom.Attribute], children: Chunk[Dom]) extends Dom.Element
case class Dom.Element.Style(attributes: Chunk[Dom.Attribute], children: Chunk[Dom]) extends Dom.Element
sealed trait Dom.Element.Void extends Dom // void elements (self-closing, no children)
case class Dom.Element.VoidGeneric(tag: String, attributes: Chunk[Dom.Attribute]) extends Dom.Element.Void

// Sealed content-model markers (implemented by dedicated element classes)
sealed trait Dom.Element.Li          extends Dom.Element // , child of ul/ol
sealed trait Dom.Element.Cell        extends Dom.Element //  or , child of tr
sealed trait Dom.Element.Th          extends Dom.Element.Cell
sealed trait Dom.Element.Td          extends Dom.Element.Cell
sealed trait Dom.Element.Tr          extends Dom.Element // , child of table
sealed trait Dom.Element.SelectChild extends Dom.Element // <option> or <optgroup>, child of select
sealed trait Dom.Element.Opt         extends Dom.Element.SelectChild
sealed trait Dom.Element.Optgroup    extends Dom.Element.SelectChild

// Attribute variants
sealed trait Dom.Attribute
case class Dom.Attribute.KeyValue(name: String, value: Dom.AttributeValue) extends Dom.Attribute
case class Dom.Attribute.AppendValue(name: String, value: Dom.AttributeValue, separator: Dom.AttributeSeparator) extends Dom.Attribute
case class Dom.Attribute.BooleanAttribute(name: String, enabled: Boolean) extends Dom.Attribute

// CssSelector variants (additional types omitted for brevity)
sealed trait CssSelector
case class CssSelector.Element(tag: String) extends CssSelector
case class CssSelector.Class(name: String) extends CssSelector

// Css variants
sealed trait Css
case class Css.Rule(selector: CssSelector, declarations: Chunk[Css.Declaration]) extends Css
final case class Css.Declaration private (property: String, value: String) // not a Css; built through Css.Declaration.apply
```

## Motivation

Why use `zio-blocks-html`?

String concatenation for HTML is error-prone:
- No type safety — typos in tag names or attributes only surface at runtime
- Manual escaping is tedious and easy to forget
- Tightly couples HTML structure to data serialization

Template engines with runtime parsing add overhead and complexity:
- Parsing validation on every render
- Separate template syntax to learn
- Difficult to compose programmatically

`zio-blocks-html` provides:
- **Type-safe construction** via DSL functions (`div(id := "main", p("Hello"))`)
- **Automatic XSS protection** through position-aware typeclasses and HTML escaping
- **Scala 3 compile-time optimizations** for string interpolators (constant folding, macro validation)
- **Zero dependencies** — pure Scala, works with any HTTP framework
- **Cross-platform** — identical API on JVM and Scala.js
- **Structured querying** — CSS selectors for DOM navigation and testing

## Installation

Add to `build.sbt`:

```
libraryDependencies += "dev.zio" %% "zio-blocks-html" % "0.0.56"
```

For Scala.js projects, use `%%%`:

```
libraryDependencies += "dev.zio" %%% "zio-blocks-html" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x

:::note
The `html` module depends directly on `zio-blocks-schema` and `zio-blocks-chunk`. The schema dependency enables `ToJs` auto-derivation via JSON encoding — if your type has a `Schema` instance, it automatically becomes usable in `js"..."` expressions.
:::

## Overview: How They Work Together

The module is organized around five core subsystems that compose together:

```
┌─────────────────────────────────────────────────────┐
│  DSL Functions (div, p, span, ...)                  │
│  + Attribute Builders (id :=, className +=)         │
│  └─> Produces: Generic / Void element nodes         │
│                                                      │
├─────────────────────────────────────────────────────┤
│  String Interpolators (html"", css"", js"")          │
│  └─> Position-aware typeclasses                      │
│      └─> html"" → Dom.Element (attribute escaping)   │
│      └─> css"" → Css (CSS value types)               │
│      └─> js"" → Js (JavaScript with </script> guard) │
│                                                      │
├─────────────────────────────────────────────────────┤
│  Dom ADT (sealed trait Dom)                          │
│  ├─> Dom.Text (HTML-escaped text content)            │
│  ├─> Dom.Element (Generic, Script, Style)            │
│  ├─> Dom.Element.Void (void/self-closing elements)   │
│  ├─> Dom.Doctype (<!DOCTYPE html>)                   │
│  └─> Dom.Empty (renders to nothing)                  │
│                                                      │
├─────────────────────────────────────────────────────┤
│  Rendering                                           │
│  ├─> dom.render → minified HTML String               │
│  ├─> dom.render(indent=2) → pretty-printed String    │
│  └─> Used in HTTP responses, testing                 │
│                                                      │
├─────────────────────────────────────────────────────┤
│  CSS Selectors + DOM Querying                        │
│  ├─> CssSelector ADT (Element, Class, Id, etc.)      │
│  ├─> Fluent combinators (>, >>, +, ~, &, |)          │
│  ├─> DomSelection API (select, filter, extract)      │
│  └─> Used for testing, transformation, navigation    │
└─────────────────────────────────────────────────────┘
```

**Typical workflow:**

1. **Build** — use DSL (`div(className := "card", ...)`) or interpolators (`html"$content"`)
2. **Compose** — nest elements and use typeclasses to extend for custom types
3. **Query** — use CSS selectors (`page.select(div.hover).texts`) for testing or transformation
4. **Render** — call `Dom#render` or `Dom#render(indent)` for HTTP response or testing
5. **Style** — use `css"..."` and `Css.Rule` to define or embed stylesheets

## The Dom ADT

The `Dom` sealed trait is the core data model. Everything in the module works with or produces `Dom` nodes.

### The Node Types

A `Dom` tree is composed of seven node types, described below: `Text`, `Generic`, `Script`, `Style`, `Doctype`, `Empty` and `Dom.Element.Void` (a node type of its own, described in [Dom.Element.Void](#domelementvoid)):

#### Dom.Text

A text node containing string content. Text is automatically HTML-escaped when rendered to prevent XSS injection. Escaping happens at render time (not construction time), so `Dom.Text` stores the raw string.

To create text nodes, use the DSL (strings are converted via `ToModifier[String]`) or explicitly. Call `Dom#render` to convert a text node to an HTML string:

```scala

val text = Dom.Text("Hello, world!")
// text: Text = Text("Hello, world!")
text.render
// res0: String = "Hello, world!"
```

#### Dom.Element.Generic

A standard HTML element. The tag is the element name, attributes is a `Chunk` of key-value pairs and modifiers, and children is a `Chunk` of DOM nodes. Text content in children is HTML-escaped during rendering.

The DSL functions (`div`, `p`, `span`, etc.) construct these:

```scala

val elem = div(id := "main", p("Content"))
```

#### Dom.Element.Script

A specialized script element with `tag = "script"`. Unlike `Generic`, Script renders its text children **without HTML escaping**, allowing inline JavaScript to be emitted as-is. This is the mechanism that enables safe `js"..."` interpolation — the interpolator escapes the JavaScript, but the Script element does not double-escape.

Use `script().inlineJs(js"...")` or `script().externalJs(url)`:

```scala

val inlineScript = script().inlineJs(js"console.log('Hello');")
val externalScript = script().externalJs("/app.js")
```

#### Dom.Element.Style

A specialized style element with `tag = "style"`. Like Script, Style renders text children **without escaping**, allowing raw CSS to be emitted. Use `style().inlineCss(css"...")`:

```scala

val inlineStyle = style().inlineCss(css"body { margin: 0; }")
```

#### Dom.Doctype

A DOCTYPE declaration node. Renders as `<!DOCTYPE value>`. The singleton `doctype` value renders as `<!DOCTYPE html>`:

```scala

doctype.render
// res4: String = "<!DOCTYPE html>"
```

#### Dom.Empty

A no-op node that renders to empty string. Useful for conditional rendering (e.g., `if (condition) element else Dom.Empty`).

### Attributes and Values

Elements carry a `Chunk[Dom.Attribute]` where each attribute is one of:

- **`Dom.Attribute.KeyValue(name: String, value: Dom.AttributeValue)`** — A standard `name="value"` attribute. The value can be a string, multi-value, boolean, or JavaScript expression.

- **`Dom.Attribute.AppendValue(name: String, value: Dom.AttributeValue, separator: Dom.AttributeSeparator)`** — Declares a value to be concatenated with any existing value for the same attribute. Provides accumulation of multi-valued attributes like `class` and `rel`. The separator appears between values (Space, Comma, Semicolon, or Custom).

- **`Dom.Attribute.BooleanAttribute(name: String, enabled: Boolean)`** — A standalone attribute (like `disabled`, `checked`, `required`) that renders as just the name when enabled, or is omitted when disabled.

:::tip
The DSL provides `id := value` (`:=`) for single-valued attributes and `className += "class"` (`+=`) for appending to multi-valued attributes. Both are easier to use than constructing `Dom.Attribute` directly.
:::

### Tree Traversal Operations

`Dom` provides four pure tree transformation methods:

**`Dom#collect(pf: PartialFunction[Dom, Dom]): List[Dom]`** — Applies a partial function to every node in the tree (depth-first) and collects the results. Useful for extracting or transforming specific nodes:

```scala

val tree = div(p("A"), span("B"), p("C"))
val paragraphs = tree.collect { case el: Dom.Element if el.tag == "p" => el }
```

**`Dom#filter(predicate: Dom => Boolean): Dom`** — Removes any node for which the predicate returns false. A failing child is removed from its parent together with its subtree; when the root itself fails, the result is `Dom.Empty`. Matching elements have their children recursively filtered:

```scala

val tree = div(p("A"), span("Keep"), p("B"), span("Also keep"))
val filtered = tree.filter {
  case el: Dom.Element => el.tag == "div" || el.tag == "span"
  case _ => true
}
```

**`Dom#find(predicate: Dom => Boolean): Option[Dom]`** — Returns the first node (depth-first) matching the predicate, or `None`:

```scala

val tree = div(p("First"), span("Second"), p("Third"))
tree.find { case el: Dom.Element => el.tag == "p"; case _ => false }
```

**`Dom#transform(f: Dom => Dom): Dom`** — Applies a transformation function to every node in pre-order. Each node receives the transformation first, and if it is an Element, its children are recursed on the transformed node, so a transformation that changes the child list affects what gets recursed into:

```scala

val tree = div(h3("Old"), p("Content"))
val upgraded = tree.transform {
  case el: Dom.Element.Generic if el.tag == "h3" => el.copy(tag = "h2")
  case other => other
}
```

## The HTML DSL

The DSL provides functions for all HTML5 elements, allowing fluent construction with a modifier pattern.

### Element Functions

All standard HTML elements are available as lowercase functions (or backtick-quoted if they shadow Scala keywords):

```scala

// Basic elements
val paragraph = p("Hello, world!")
val heading = h1("Welcome")
val container = div("Content here")

// Void elements (auto self-closing)
val image = img(src := "photo.jpg", alt := "A photo")
val lineBreak = br
val horizontalRule = hr

// Keyword-shadowed elements (use backticks)
val objElement = `object`("data.bin")
```

### Tag Names That Clash With Your Imports

`import zio.blocks.html._` brings a member for every tag, so some tag names collide with names from other imports or with your own helpers. The compiler reports these clashes as ambiguous references or as type errors that point somewhere else, so it helps to know them in advance:

- **`span` and `scala.concurrent.duration._`.** A wildcard import of `scala.concurrent.duration._` also brings an object named `span` into scope, so `span("1")` fails with "Reference to span is ambiguous". Import only the extension you need, such as `scala.concurrent.duration.DurationInt`, and keep the DSL's `span`.
- **`main` inside an object that defines `def main(args: Array[String])`.** A call to `main(id := "main")(...)` resolves to your method, and the error names `Array[String]`, not the element. Qualify the element as `zio.blocks.html.main(...)`, or rename the import.
- **Tag names as helper names.** Names such as `search`, `title`, `select`, `label`, `header`, `option`, `data`, `time` and `output` clash with helpers of the same name, such as a `def search(query: String)` that you import with a wildcard. Import your own helpers by name, or rename them.

Import only the duration extension next to the DSL:

```scala

val delay  = 300.millis
val banner = span(id := "banner")("Saved")
```

Qualify the element inside an object with its own `main` method:

```scala

object Layout {
  def page: Dom.Element = html(body(zio.blocks.html.main(id := "main")("content")))

  def main(args: Array[String]): Unit = println(page.render)
}
```

### Setting Attributes

Attributes are set using the `:=` operator on `AttributeKey` builders:

```scala

val link = a(
  href := "https://example.com",
  target := "_blank",
  titleAttr := "Visit Example",
  "Visit Example"
)
```

### Multi-Valued Attributes: Override vs Accumulate

For attributes that can have multiple values (`class`, `rel`), you can override or accumulate:

- **`:=` (override)** — Replaces any earlier `:=` value, so the last `:=` wins. Values added with `+=` are always appended, so `className += "sale", className := "card"` renders `class="card sale"`:

```scala

val div1 = div(className := "a", className := "b")
```

- **`+=` (append)** — Concatenates with the previous value using a separator (space for `class`):

```scala

val div2 = div(className += "card", className += "active")
```

- **Mixed** — Set a base value, then append:

```scala

val div3 = div(className := "base").when(true)(className += "extra")
```

### Boolean Attributes

Use `BooleanAttribute.:=` to conditionally include attributes like `disabled`, `required`, or `checked`:

```scala

val isDisabled = true
val submitBtn = button(disabled := isDisabled, "Submit")
```

### Data and ARIA Attributes

Use `dataAttr(name)` and `aria(name)` builders for HTML5 data attributes and ARIA accessibility attributes:

```scala

val userDiv = div(
  dataAttr("user-id") := "42",
  dataAttr("action") := "edit",
  "User Card"
)

val closeBtn = button(
  aria("label") := "Close",
  aria("expanded") := "false",
  "×"
)
```

For custom attributes not provided by the DSL, use the generic `attr(name)` builder:

```scala

val alpineDiv = div(attr("x-data") := "{open: false}", "Alpine.js component")
val htmxDiv = div(attr("hx-get") := "/api/data", "HTMX target")
```

### Programmatic Multi-Valued Attributes

For programmatic construction of multi-valued attributes (when the DSL `+=` operator is not available), use `Dom.multiAttr(name)` or `Dom.multiAttr(name, separator)`:

```scala

// Create a custom multi-valued attribute builder (alternative to DSL's className)
val classBuilder = Dom.multiAttr("class")
val div1 = div(classBuilder := "base", classBuilder += "active")

// Create a multi-valued attribute with explicit separator
val rel = Dom.multiAttr("rel", Dom.AttributeSeparator.Space)
val link = a(rel := "prev", rel += "first", href := "/previous")
```

The `MultiAttributeKey` class handles accumulation of values with configurable separators (Space, Comma, Semicolon, or Custom).

For constructing multi-valued attributes directly from collections (without a builder chain), use the `Iterable[String]` overload:

```scala

// Directly create a multi-valued attribute from a collection
val customClasses = Dom.multiAttr("class", List("card", "active", "large"))
val div1 = div(customClasses)
```

This approach is useful when programmatically building multi-valued attributes outside the DSL's builder pattern.

### Children

Children can be strings, elements, or collections. The DSL uses the `ToModifier` typeclass to convert values into DOM nodes:

```scala

// String children are converted to Dom.Text
val simple = p("Plain text")

// Element children are nested
val nested = div(p("First"), p("Second"))

// Lists of children — elements append directly, no wrapper
val items = List("Apple", "Banana", "Cherry")
val listEl = ul(items.map(item => li(item)))
```

### Conditional Rendering

Use `when(condition)` to apply modifiers conditionally:

```scala

val isHighlighted = true
val box = div(
  className := "box"
).when(isHighlighted)(
  className += "highlighted",
  titleAttr := "This is highlighted"
)
```

Use `whenSome(option)` to apply modifiers based on an `Option`:

```scala

val maybeTitle: Option[String] = Some("Important")
val card = div(
  className := "card",
  p("Content")
).whenSome(maybeTitle)(t => Seq(
  titleAttr := t,
  className += "has-title"
))
```

### DomModifier and the Conversion Type Classes

Everything passed to an element factory becomes a `DomModifier` — a deferred description of one modification, collected during construction and applied in a batch. The ADT has four cases:

```scala
sealed trait DomModifier {
  def applyTo(element: Dom.Element): Dom.Element
}

object DomModifier {
  final case class AddAttr(attr: Dom.Attribute)            extends DomModifier
  final case class AddChild(child: Dom)                    extends DomModifier
  final case class AddChildren(children: Chunk[Dom])       extends DomModifier
  final case class AddEffects(effects: Chunk[DomModifier]) extends DomModifier
}
```

`AddEffects` is what makes the ADT recursive, so a group of modifications can be passed as one value and applied as a unit.

`DomModifier#applyTo` applies a single effect and returns the modified element. It is the direct path, and the one to avoid in a loop:

```scala

val withId = DomModifier.AddAttr(id := "main").applyTo(div)
```

Each call produces a fresh element, so applying effects one at a time allocates an intermediate per effect:

```scala
withId.render
// res24: String = ""
```

:::note[Prefer the factory for multiple effects]
A factory call such as `div(a, b, c)` applies every effect in one pass. Calling `DomModifier#applyTo` repeatedly rebuilds the element each time. Use `DomModifier#applyTo` for a single effect computed at runtime, and the factory syntax for everything else.
:::

`DomModifierConversions` supplies the implicit that turns any `A` with a `ToModifier[A]` instance into a `DomModifier`, which is why strings, elements, and collections can be passed to a factory interchangeably.

#### ToDom

`ToDom[A]` converts a value into a `Dom` node. It is contravariant, and the instance set covers `Dom` itself, `Dom.Element`, `Dom.Text`, `String`, `Int`, `Long`, `Double`, `Boolean`, plus derived instances for `Option` and `Iterable`.

The two derived instances have behaviour worth knowing, because both can produce a node you did not write. `None` becomes `Dom.Empty`, which renders as nothing:

```scala

val absent  = ToDom[Option[String]].toDom(None)
val present = ToDom[Option[String]].toDom(Some("hello"))
```

That is what lets an optional value be dropped into a template without a conditional:

```scala
absent.render
// res26: String = ""
present.render
// res27: String = "hello"
```

`Iterable` is the surprising one. An empty collection becomes `Dom.Empty` and a single-element collection becomes that element, but **two or more elements are wrapped in a ``**:

```scala
ToDom[Iterable[String]].toDom(Nil).render
// res28: String = ""
ToDom[Iterable[String]].toDom(List("a")).render
// res29: String = "a"
ToDom[Iterable[String]].toDom(List("a", "b")).render
// res30: String = "ab"
```

:::warning[Collections of two or more get a wrapper element]
`ToDom` has to return a single `Dom`, so a multi-element `Iterable` is wrapped in a ``. That inserts an element into your markup which is invalid inside ``, ``, and `<select>`, and which affects CSS descendant selectors. To splice a collection without a wrapper, pass it to a factory — `ul(items.map(li(_)))` uses the collection overload and adds the children directly.
:::

#### ToText

`ToText[A]` converts a value to a plain `String` rather than a DOM node. Instances cover `String`, `Int`, `Long`, `Double`, `Float`, `Boolean`, `Char`, `Byte`, `Short`, `BigInt`, and `BigDecimal`:

```scala
ToText[Int].toText(42)
// res31: String = "42"
ToText[BigDecimal].toText(BigDecimal("1.50"))
// res32: String = "1.50"
```

There is deliberately no `ToText[Dom]`: rendering an element to text is `Dom#render`, whereas `ToText` is for values that are already scalar.

### Dom.Element.Void

Void HTML elements (`br`, `hr`, `img`, `input`, `meta`, `link`, `area`, `base`, `col`, `embed`, `param`, `source`, `track`, `wbr`) are a **separate type** from `Dom.Element` — they extend `Dom.Element.Void` which extends `Dom` directly (not `Dom.Element`). This means they **structurally cannot have children** — passing a modifier like a string or another element to a void element factory is a **compile-time error**:

```scala

img("oops")             // ❌ does not compile — Void elements don't accept children
input(div("child"))     // ❌ does not compile — Void elements don't accept children
// error: 
// Conflicting definitions:
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 17 and
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 20
// 
// error: 
// Conflicting definitions:
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 20 and
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 26
// 
// error: 
// Conflicting definitions:
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 26 and
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 29
// 
// error: 
// Conflicting definitions:
// val div1: zio.blocks.html.Dom.Element in class MdocApp0 at line 64 and
// val div1: zio.blocks.html.Dom.Element in class MdocApp0 at line 88
// 
// error: 
// Conflicting definitions:
// val link: zio.blocks.html.Dom.Element in class MdocApp0 at line 57 and
// val link: zio.blocks.html.Dom.Element in class MdocApp0 at line 90
// 
// error: 
// Conflicting definitions:
// val div1: zio.blocks.html.Dom.Element in class MdocApp0 at line 88 and
// val div1: zio.blocks.html.Dom.Element in class MdocApp0 at line 93
// 
// error: 
// Found:    ("oops" : String)
// Required: zio.blocks.html.Dom.Attribute
// error: 
// Found:    zio.blocks.html.Dom.Element
// Required: zio.blocks.html.Dom.Attribute
```

Void elements only accept `Dom.Attribute` arguments:

```scala

val image = img(src := "photo.jpg", alt := "A photo")   // ✅ compiles — only attributes
val lineBreak = br                                        // ✅ compiles — no args
val textField = input(`type` := "text", placeholder := "Enter text")  // ✅ compiles
```

Void elements render as self-closing tags:

```scala

val voidElements = div(
  br,
  hr,
  img(src := "photo.jpg", alt := "A photo"),
  input(`type` := "text", placeholder := "Enter text")
)
```

## Typed Content Models

Several container elements enforce the HTML content model of their children at
compile time. Instead of accepting arbitrary modifiers, these factories only
accept attributes plus children whose type matches the content model:

| Factory | Accepted element children | Marker type |
|---|---|---|
| `ul(...)`, `ol(...)` | `` | `Dom.Element.Li` |
| `tr(...)` | ``, `` | `Dom.Element.Cell` (`Th \| Td`) |
| `table(...)` | `` | `Dom.Element.Tr` |
| `select(...)` | `<option>`, `<optgroup>` | `Dom.Element.SelectChild` (`Opt \| Optgroup`) |
| `optgroup(...)` | `<option>` | `Dom.Element.Opt` |

Invalid children are rejected by the compiler:

```scala

ul(div("not a list item"))   // ❌ does not compile — ul only accepts Li children
tr(p("not a cell"))          // ❌ does not compile — tr only accepts Th/Td cells
select(div("not an option")) // ❌ does not compile — select only accepts option/optgroup
// error: 
// Conflicting definitions:
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 17 and
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 20
// 
// error: 
// Conflicting definitions:
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 20 and
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 26
// 
// error: 
// Conflicting definitions:
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 26 and
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 29
// 
// error: 
// Conflicting definitions:
// val div1: zio.blocks.html.Dom.Element in class MdocApp0 at line 64 and
// val div1: zio.blocks.html.Dom.Element in class MdocApp0 at line 88
// 
// error: 
// Conflicting definitions:
// val link: zio.blocks.html.Dom.Element in class MdocApp0 at line 57 and
// val link: zio.blocks.html.Dom.Element in class MdocApp0 at line 90
// 
// error: 
// Conflicting definitions:
// val div1: zio.blocks.html.Dom.Element in class MdocApp0 at line 88 and
// val div1: zio.blocks.html.Dom.Element in class MdocApp0 at line 93
// 
// error: 
// None of the overloaded alternatives of method ul in trait HtmlElements with types
//  (children: Iterable[zio.blocks.html.Dom.Element.Li]):
//   zio.blocks.html.Dom.Element
//  (effect: zio.blocks.html.Dom.Attribute | zio.blocks.html.Dom.Element.Li, effects
//   : (zio.blocks.html.Dom.Attribute | zio.blocks.html.Dom.Element.Li)*):
//   zio.blocks.html.Dom.Element
//  (): zio.blocks.html.Dom.Element
// match arguments (zio.blocks.html.Dom.Element)
// error: 
// None of the overloaded alternatives of method tr in trait HtmlElements with types
//  (children: Iterable[zio.blocks.html.Dom.Element.Cell]):
//   zio.blocks.html.Dom.Element.Tr
//  (effect: zio.blocks.html.Dom.Attribute | zio.blocks.html.Dom.Element.Cell,
//   effects: (zio.blocks.html.Dom.Attribute | zio.blocks.html.Dom.Element.Cell)*)
//   : zio.blocks.html.Dom.Element.Tr
//  (): zio.blocks.html.Dom.Element.Tr
// match arguments (zio.blocks.html.Dom.Element)
// error:
// None of the overloaded alternatives of method select in trait HtmlElements with types
//  (children: Iterable[zio.blocks.html.Dom.Element.SelectChild]):
//   zio.blocks.html.Dom.Element
//  (
//   effect: zio.blocks.html.Dom.Attribute |
//     zio.blocks.html.Dom.Element.SelectChild,
// effects: (zio.blocks.html.Dom.Attribute |
//   zio.blocks.html.Dom.Element.SelectChild)*): zio.blocks.html.Dom.Element
//  (): zio.blocks.html.Dom.Element
// match arguments (zio.blocks.html.Dom.Element)
// select(div("not an option")) // ❌ does not compile — select only accepts option/optgroup
// ^^^^^^
```

Attributes are accepted alongside the constrained children:

```scala

val list = ul(id := "menu", className := "nav", li("Home"), li("About"))
val row = tr(className := "header-row", th("Name"), td("Value"))
val picker = select(id := "color", name := "color", option("Red"), optgroup(option("Light"), option("Dark")))
```

Each factory has three forms: empty (`ul()`), mixed attributes and children
(`ul(id := "x", li("a"))`), and collections of children
(`ul(items.map(item => li(item)))`).

The typed factories produce elements carrying a sealed marker type (`Li`,
`Cell`, `Tr`, `SelectChild`, ...). The markers are sealed — their
implementations live inside the module — so a value typed `Dom.Element.Li` is
guaranteed to render as ``; the DSL factories are the intended
construction path. Elements with permissive content models (`li`, `th`, `td`,
`option`) keep accepting any flow content through the regular modifier API.

:::note
`caption`, `colgroup`, `thead`, `tbody`, and `tfoot` do not have typed markers
yet, so tables containing them cannot be built with the `table(...)`
factory. Compose those tables with the `html""` interpolator or generic
elements.
:::

### How the Constraint Is Encoded

The guarantee is the same on both Scala versions, but the signature achieving it is not. Scala 3 uses a union type directly; Scala 2 has no unions, so each constrained factory takes a sealed argument trait with implicit conversions into it:

<Tabs groupId="scala-version" defaultValue="scala2">
  <TabItem value="scala2" label="Scala 2">

```scala
sealed trait ListArg
object ListArg {
  final case class Attribute(value: Dom.Attribute) extends ListArg
  final case class Item(value: Dom.Element.Li)     extends ListArg

  implicit def fromAttribute(attribute: Dom.Attribute): ListArg = Attribute(attribute)
  implicit def fromLi(item: Dom.Element.Li): ListArg            = Item(item)
}

def ul(effect: ListArg, effects: ListArg*): Dom.Element
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3">

```scala
def ul(
  effect: Dom.Attribute | Dom.Element.Li,
  effects: (Dom.Attribute | Dom.Element.Li)*
): Dom.Element
```

  </TabItem>
</Tabs>

Call sites are identical — `ul(id := "menu", li("Home"))` compiles unchanged on both — because the Scala 2 conversions are implicit. The encoding only becomes visible when you name the argument type, which is why these traits exist at all:

| Factory                | Scala 2 argument type | Scala 3 union                                  |
| ---------------------- | --------------------- | ---------------------------------------------- |
| `ul(...)`, `ol(...)`   | `ListArg`             | `Dom.Attribute \| Dom.Element.Li`               |
| `tr(...)`              | `CellArg`             | `Dom.Attribute \| Dom.Element.Cell`             |
| `table(...)`           | `RowArg`              | `Dom.Attribute \| Dom.Element.Tr`               |
| `select(...)`          | `SelectArg`           | `Dom.Attribute \| Dom.Element.SelectChild`      |
| `optgroup(...)`        | `OptgroupArg`         | `Dom.Attribute \| Dom.Element.Opt`              |
| `script(...)`          | `ScriptArg`           | `Dom.Attribute \| Js`                           |
| `style(...)`           | `StyleArg`            | `Dom.Attribute \| Css`                          |

`ScriptArg` and `StyleArg` follow the same shape: they are the argument types of `script(...)` and `style(...)`.

Writing a function that forwards to one of these factories is where the difference surfaces. On Scala 2 the parameter type is the argument trait; on Scala 3 it is the union:

<Tabs groupId="scala-version" defaultValue="scala2">
  <TabItem value="scala2" label="Scala 2">

```scala
def menu(items: ListArg*): Dom.Element = ul(items.head, items.tail: _*)
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3">

```scala
def menu(items: (Dom.Attribute | Dom.Element.Li)*): Dom.Element =
  ul(items.head, items.tail*)
```

  </TabItem>
</Tabs>

Taking `Iterable[Dom.Element.Li]` avoids the difference entirely, since `ul`, `ol`, `tr`, `table`, `optgroup` and `select` each have a collection overload that is spelled the same on both versions (`script` and `style` have none).

### Concrete Element Classes

The marker traits are implemented by case classes named after their tag: `LiElement`, `ThElement`, `TdElement`, `TrElement`, `OptElement`, and `OptgroupElement`. Each holds `attributes` and `children` and hard-codes its `tag`, which is what makes a value typed `Dom.Element.Li` guaranteed to render as ``.

You do not normally name them — the factories return the marker trait, not the implementation — but they appear in pattern matches over a DOM tree and in `toString` output. Because equality is structural, matching on `Dom.Element.Generic` will *not* catch a factory-produced `li`, even though the two are equal:

```scala

val item = li("Home")
```

The factory returns the marker type, and the runtime class is the implementation behind it:

```scala
item.getClass.getSimpleName
// res38: String = "LiElement"
```

Matching against `Generic` at that static type does not merely fail — it does not compile. `Dom.Element.Li` and `Generic` are disjoint, so the compiler reports the case as unreachable:

```scala
item match {
  case _: Dom.Element.Generic => "matched Generic"
  case _                      => "no match"
}
// error: 
// Conflicting definitions:
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 17 and
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 20
// 
// error: 
// Conflicting definitions:
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 20 and
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 26
// 
// error: 
// Conflicting definitions:
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 26 and
// val tree: zio.blocks.html.Dom.Element in object MdocApp at line 29
// 
// error: 
// Conflicting definitions:
// val div1: zio.blocks.html.Dom.Element in class MdocApp0 at line 64 and
// val div1: zio.blocks.html.Dom.Element in class MdocApp0 at line 88
// 
// error: 
// Conflicting definitions:
// val link: zio.blocks.html.Dom.Element in class MdocApp0 at line 57 and
// val link: zio.blocks.html.Dom.Element in class MdocApp0 at line 90
// 
// error: 
// Conflicting definitions:
// val div1: zio.blocks.html.Dom.Element in class MdocApp0 at line 88 and
// val div1: zio.blocks.html.Dom.Element in class MdocApp0 at line 93
//
```

Widening to `Dom.Element` makes the match legal, and then `Generic` simply does not match:

```scala
val widened: Dom.Element = item
// widened: Element = LiElement(
//   attributes = IndexedSeq(),
//   children = IndexedSeq(Text("Home"))
// )
widened match {
  case _: Dom.Element.Generic => "matched Generic"
  case _: Dom.Element.Li      => "matched Li"
  case _                      => "no match"
}
// res40: String = "matched Li"
```

:::note[Structural equality, disjoint types]
`li("Home") == Generic("li", ...)` is `true`, but `LiElement` is not a `Generic` and never matches one. Equality is structural and crosses classes; the types do not. Match on the marker trait, or on `Dom.Element` plus `tag`, rather than on a concrete class.
:::

### Equality Across Element Classes

Element equality is structural (tag + attributes + children + text-escaping
semantics), independent of the concrete class. A factory-produced `li(...)`
equals the structurally identical `Generic("li", ...)`, and vice versa; equal
elements also have equal hash codes:

```scala

val fromFactory = li("a")
val generic = Dom.Element.Generic("li", Chunk.empty, Chunk(Dom.Text("a")))
fromFactory == generic  // true — structural equality across classes
```

Elements that render differently are never equal, even with identical fields:
`script`/`style` render their children **unescaped** while `Generic` escapes
them, so a `Script` never equals an equally-shaped `Generic("script", ...)` —
equal elements always render identically.

## String Interpolators

The module provides four string interpolators: `html""`, `css""`, `js""`, and `selector""`. All offer compile-time safety (on Scala 3) and automatic escaping appropriate to their context.

### `html""` Interpolator

The `html""` interpolator is **position-aware**: it detects whether each interpolated argument is in an attribute-value position or content position, then summons the appropriate typeclass.

The `html""` interpolator determines position based on the preceding string:

**Position detection:**
- **Attribute position**: if the preceding string part ends with `=`, `='`, or `="`, the next argument is treated as an attribute value and uses `ToAttrValue[A]`
- **Content position**: otherwise, the argument is treated as element content and uses `ToElements[A]`

Position-aware interpolation enables safe attribute values and content:

```scala

val name = "Alice"
val age = 30

// name is in content position → ToElements[String]
// age is in attribute position (the whole value) → ToAttrValue[Int]
val elem = html"""User: $name"""
```

**Single-root requirement**: The `html""` interpolator requires a **single root element**. Multiple top-level nodes cause an exception at runtime (Scala 2) or compile error for static templates (Scala 3). Whitespace around the root counts too: a template that starts or ends with a newline has three top-level nodes, so begin the template with the root tag and end it right after the closing tag. A `$` inside a quoted attribute value, such as `id="user-$age"`, is not supported in 0.0.56 (a known bug); interpolate the whole value instead, as in `id=$age`:

```scala

// This compiles (single root)
val page = html"AB"
```

**XSS Protection:**

Content interpolated into `html""` is stored as `Dom.Text` nodes, which are HTML-escaped during `Dom#render` to prevent XSS:

```scala

val userInput = "<script>alert('XSS')</script>"
val safe = html"$userInput"
```

:::danger
The `html""` interpolator requires a **single root element**. On Scala 3, pure-static templates with multiple roots fail at compile time. On Scala 2 or for dynamic templates, the error occurs at evaluation time. Always wrap multiple elements in a container (e.g., ``).
:::

### `css""` Interpolator

The `css""` interpolator returns a `Css` value with `ToCss` typeclass dispatch for interpolated parts:

```scala

val color = "blue"
val size = 16

val styles = css"color: $color; font-size: ${size}px;"
```

Use `CssLength` and `CssColor` types for type-safe CSS values:

```scala

val width = css"width: ${CssLength(300.0, "px")};"
val background = css"background: ${CssColor.Hex.unsafe("ff0000")};"
```

:::note
`CssColor.Hex` returns `Option[CssColor]` because hex validation may fail. Use `CssColor.Hex.apply()` (or just `CssColor.Hex(...)`) for safe validation, or `CssColor.Hex.unsafe()` for known-valid hex strings.
:::

### `js""` Interpolator

The `js""` interpolator returns a `Js` value. Strings are automatically quoted and escaped; numerics and booleans are rendered unquoted:

```scala

val message = "Hello, world!"
val count = 42

val code = js"console.log($message); alert($count);"
```

:::warning
Never interpolate untrusted user input directly into `js""`. The interpolator escapes string values, but the `Js` value is rendered without additional escaping in script elements. Always use `ToJs[String]` (which automatically quotes and escapes) for any untrusted input.
:::

The interpolator renders a `String` value as a quoted JavaScript literal, writing `<` and `>` as Unicode escapes:

```scala

val userInput = "if (x < y) alert('<script>');"
// userInput: String = "if (x < y) alert('<script>');"
val code = js"let check = $userInput"
// code: Js = Js(
//   "let check = \"if (x \\u003c y) alert(\\'\\u003cscript\\u003e\\');\""
// )

println(code.value)
// let check = "if (x \u003c y) alert(\'\u003cscript\u003e\');"
```

### `selector""` Interpolator

The `selector""` interpolator returns a `CssSelector`:

```scala

val className = "active"
val selector = selector".$className"
```

## CSS Selectors and the DSL

The `CssSelector` ADT provides a fluent DSL for building CSS selectors with combinator methods.

### Basic Selectors

Create selectors for elements, classes, IDs, or universal matches:

```scala

val divSel = CssSelector.Element("div")
val classSel = CssSelector.Class("container")
val idSel = CssSelector.Id("header")
val universal = CssSelector.Universal
```

### Combinators

All HTML elements (`div`, `span`, `p`, etc.) implement `CssSelectable`, so you can use them directly as selectors:

```scala

// Element selectors via DSL elements
val divSel = div.selector
val childSel = div > span              // div > span
val descendantSel = div >> span        // div span (descendant)
val adjacentSel = div + span           // div + span
val siblingSel = div ~ span           // div ~ span
val andSel = div & CssSelector.Class("active")  // div.active
val orSel = div | span                 // div, span
```

### Pseudo-Classes and Pseudo-Elements

Pseudo-classes match elements by their state, and pseudo-elements create dynamic content:

```scala

val hoverSel = a.hover              // a:hover
val firstChild = li().firstChild      // li:first-child
val nthChild = tr().nthChild(2)       // tr:nth-child(2)
val before = div.before             // div::before
val after = span.after              // span::after
```

### Attribute Selectors

Select elements by their attribute values using built-in matchers:

```scala

val input = CssSelector.Element("input")

val hasType = input.withAttribute("type")              // input[type]
val exactType = input.withAttribute("type", "text")    // input[type="text"]
val containsClass = input.withAttributeContaining("class", "btn")  // input[class*="btn"]
val startsWithHref = a.withAttributeStarting("href", "https")      // a[href^="https"]
val endsWithPng = img.withAttributeEnding("src", ".png")           // img[src$=".png"]
```

### The Selector ADT

The DSL operators are constructors: each one builds a `CssSelector` node, and the result is an ordinary sealed ADT you can pattern match on. That matters when a selector arrives from elsewhere — a stylesheet you are rewriting, a rule you are inspecting — rather than being written inline.

Every operator and its node:

| Expression                              | Node                                    | Renders as        |
| --------------------------------------- | --------------------------------------- | ----------------- |
| `CssSelector.Element("div")`            | `Element(tag)`                           | `div`             |
| `CssSelector.Class("active")`           | `Class(name)`                            | `.active`         |
| `CssSelector.Id("header")`              | `Id(name)`                               | `#header`         |
| `CssSelector.Universal`                 | `Universal`                              | `*`               |
| `div > span`                            | `Child(parent, child)`                   | `div > span`      |
| `div >> span`                           | `Descendant(parent, descendant)`         | `div span`        |
| `div + span`                            | `AdjacentSibling(previous, next)`        | `div + span`      |
| `div ~ span`                            | `GeneralSibling(previous, sibling)`      | `div ~ span`      |
| `div & CssSelector.Class("active")`     | `And(left, right)`                       | `div.active`      |
| `div \| span`                           | `Or(left, right)`                        | `div, span`       |
| `a.hover`                               | `PseudoClass(inner, pseudoClass)`        | `a:hover`         |
| `div.before`                            | `PseudoElement(inner, pseudoElement)`    | `div::before`     |
| `input.withAttribute("type")`           | `Attribute(inner, attribute, matcher)`   | `input[type]`     |
| `div.not(CssSelector.Class("hidden"))`  | `Not(inner, negated)`                    | `div:not(.hidden)`|
| `CssSelector.Raw("...")`                | `Raw(value)`                             | verbatim          |

Because the nodes are recursive, a selector built with operators nests rather than flattening:

```scala

val selector = div >> span & CssSelector.Class("active")
```

Matching on the outermost node tells you which operator produced it:

```scala
selector.render
// res54: String = "div span.active"
selector match {
  case CssSelector.And(left, right) => s"And(${left.render}, ${right.render})"
  case other                        => other.getClass.getSimpleName
}
// res55: String = "And(div span, .active)"
```

`CssSelector.Raw` is the escape hatch for syntax the ADT does not model. It renders verbatim, so nothing validates it:

```scala
CssSelector.Raw("div:has(> img)").render
// res56: String = "div:has(> img)"
```

#### Attribute Matchers

`CssSelector.Attribute` carries an `Option[AttributeMatch]`. `None` tests for presence alone; the six matchers correspond to the CSS attribute operators:

| Builder                                    | Matcher                        | Renders as              |
| ------------------------------------------ | ------------------------------ | ----------------------- |
| `withAttribute(name)`                      | `None`                          | `[name]`                |
| `withAttribute(name, value)`               | `Exact(value)`                  | `[name="value"]`        |
| `withAttributeContaining(name, value)`     | `Contains(value)`               | `[name*="value"]`       |
| `withAttributeStarting(name, value)`       | `StartsWith(value)`             | `[name^="value"]`       |
| `withAttributeEnding(name, value)`         | `EndsWith(value)`               | `[name$="value"]`       |
| `withAttributeWord(name, value)`           | `WhitespaceContains(value)`     | `[name~="value"]`       |
| `withAttributePrefix(name, value)`         | `HyphenPrefix(value)`           | `[name\|="value"]`      |

Every matcher has a builder, and building a `CssSelector.Attribute` node by hand gives the same selector:

```scala
div.withAttributeWord("class", "card").render
// res57: String = "div[class~=\"card\"]"
div.withAttributePrefix("lang", "en").render
// res58: String = "div[lang|=\"en\"]"
CssSelector.Attribute(
  CssSelector.Element("div"),
  "class",
  Some(CssSelector.AttributeMatch.WhitespaceContains("card"))
) == div.withAttributeWord("class", "card")
// res59: Boolean = true
```

`WhitespaceContains` matches one word in a space-separated list, which is the correct matcher for a single class among several; `HyphenPrefix` matches a value or that value followed by a hyphen, which is what `lang` attributes need.

## DOM Querying with DomSelection

The `DomSelection` API lets you query and navigate DOM trees using CSS selectors. It is useful for testing, transforming templates, and extracting information.

### Selecting Elements

Call `Dom#select(selector)` to query the tree:

```scala

val page = div(
  header(nav(a(href := "/", "Home"), a(href := "/about", "About"))),
  main(p(className += "intro", "Hello"), p("World")),
  footer(p("© 2026"))
)

// Query by tag
val paragraphs = page.select(CssSelector.Element("p"))
paragraphs.length

// Query by class
val intros = page.select(CssSelector.Class("intro"))
intros.texts
```

### Navigation

The `DomSelection` API returned by `Dom#select` provides navigation methods like `.children` and `.first` to traverse the selection results:

```scala

val page = div(
  nav(a(href := "/", "Home"), a(href := "/about", "About")),
  main(p("Content"))
)

// Direct children
val navLinks = page.select(CssSelector.Element("nav")).children
```

### Extraction

The `DomSelection` API provides methods like `.attrs` and `.texts` to extract attribute values and text content from selected elements:

```scala

val page = div(
  a(href := "/home", "Home"),
  a(href := "/about", "About"),
  a(href := "/contact", "Contact")
)

val hrefs = page.select(CssSelector.Element("a")).attrs("href")
```

### Filtering

The `DomSelection` API provides `.filter` and `.withClass` methods to filter selections by predicate or attribute:

```scala

val page = div(
  p(className += "visible", "A"),
  p(className += "hidden", "B"),
  p("C")
)

// Filter by predicate
val visible = page.select(CssSelector.Element("p")).filter { 
  case el: Dom.Element => el.attributes.exists {
    case attr: Dom.Attribute.KeyValue if attr.name == "class" => true
    case _ => false
  }
  case _ => false
}

// Filter by class
val withClass = page.select(CssSelector.Element("p")).withClass("visible")
```

### Modifying Selections

:::warning
`DomSelection` returns new copies with modifications; the original DOM tree remains unchanged. The `DomSelection` API provides functional transformations. To modify the original DOM tree, use `Dom#transform` with a tree-rewriting function, or rebuild the tree from scratch using the DSL.
:::

The `DomSelection` API provides methods like `.modifyAll`, `.replaceAll`, and `.removeAll` to transform or replace selected nodes:

```scala

val page = div(
  p("Old 1"),
  p("Old 2"),
  span("Keep")
)

// Transform all Element nodes in the selection; non-Element nodes pass through unchanged
val modifiedSelection = page.select(CssSelector.Element("p")).modifyAll {
  case el: Dom.Element.Generic if el.tag == "p" => el.copy(tag = "div")
  case other => other
}

// Replace all selected nodes (returns DomSelection of replacement nodes)
val replacedSelection = page.select(CssSelector.Element("p")).replaceAll(p("New"))

// Remove all selected nodes (returns empty DomSelection)
val removedSelection = page.select(CssSelector.Element("p")).removeAll
```

:::note
Pseudo-class selectors (`:hover`, `:focus`, etc.) match elements structurally by their underlying element selector only — they cannot detect browser interaction state in a static DOM tree. `div.hover` matches the same elements as `CssSelector.Element("div")`.

Adjacent sibling (`+`) and general sibling (`~`) selectors are supported in CSS output but not in DOM querying — `DomSelection.select` with these combinators returns empty results.
:::

## The CSS ADT

The `Css` ADT represents structured stylesheets as a typed data structure, separate from strings.

All `Css` subtypes support `.render()` for minified output and `.render(indent: Int)` for indented pretty-printing:
- `Css.Rule` — single CSS rule with selector and declarations
- `Css.Sheet` — collection of rules
- `Css.Raw` — raw CSS string
- `Css.Comment` — CSS comment

### Typed Values: Lengths and Colors

A `Css.Declaration` takes its value as a `String`, so nothing stops `"10pixels"` or `"#gggggg"` from reaching a stylesheet. `CssLength` and `CssColor` are the typed alternatives: each validates on construction and renders itself, so an invalid value cannot be built rather than being caught downstream.

`CssLength` pairs a numeric value with a unit, and rejects units outside the CSS set — `px`, `em`, `rem`, `%`, `vh`, `vw`, `ch`, `ex`, `vmin`, `vmax`, `cm`, `mm`, `in`, `pt`, `pc`:

```scala

```

The second import is what brings the numeric extension methods into scope. They live in `object CssLength` rather than the package, so `import zio.blocks.html._` alone gives you the `CssLength` constructor but not `300.px`.

Writing the constructor directly works, but the extension methods on `Int` and `Double` are shorter and produce the same value:

```scala
CssLength(300.0, "px").render
// res65: String = "300px"
300.px.render
// res66: String = "300px"
1.5.rem.render
// res67: String = "1.5rem"
```

`CssLengthIntOps` and `CssLengthDoubleOps` provide `px`, `em`, `rem`, `pct`, `vh`, and `vw` on numeric literals. `pct` is spelled out because `%` is not a legal method name:

```scala
50.pct.render
// res68: String = "50%"
100.vh.render
// res69: String = "100vh"
```

A whole-number `Double` renders without its decimal point, so `2.0.em` and `2.em` produce identical CSS:

```scala
2.0.em.render
// res70: String = "2em"
2.em.render
// res71: String = "2em"
```

`CssColor` is a sealed ADT with five cases, covering the notations CSS accepts:

| Case                   | Constructor                       | Renders as              |
| ---------------------- | --------------------------------- | ----------------------- |
| `CssColor.Hex`         | `Hex(value)` / `Hex.unsafe(value)` | `#ff0000`               |
| `CssColor.Rgb`         | `Rgb(r, g, b)`                     | `rgb(255,0,0)`          |
| `CssColor.Rgba`        | `Rgba(r, g, b, a)`                 | `rgba(255,0,0,0.5)`     |
| `CssColor.Hsl`         | `Hsl(h, s, l)`                     | `hsl(0,100%,50%)`       |
| `CssColor.Named`       | `Named(name)`                      | `red`                   |

The numeric cases are plain case classes, so they need no validation step:

```scala
CssColor.Rgb(255, 0, 0).render
// res72: String = "rgb(255,0,0)"
CssColor.Rgba(255, 0, 0, 0.5).render
// res73: String = "rgba(255,0,0,0.5)"
CssColor.Hsl(0, 100, 50).render
// res74: String = "hsl(0,100%,50%)"
```

`Hsl` renders its saturation and lightness with percent signs while taking them as bare `Int`s, so pass `100` rather than `1.0` for full saturation.

`CssColor.Hex` and `CssColor.Named` are the two validated cases, and both return an `Option` from `CssColor.Hex.apply` rather than throwing:

```scala
CssColor.Hex("ff0000")
// res75: Option[CssColor] = Some(Hex("ff0000"))
CssColor.Hex("gggggg")
// res76: Option[CssColor] = None
```

A leading `#` is optional and the value is lowercased, so the same colour written four ways yields one representation:

```scala
CssColor.Hex("#FF0000").map(_.render)
// res77: Option[String] = Some("#ff0000")
```

`CssColor.Hex.unsafe` skips validation for literals you know are well-formed. It returns a `CssColor` directly rather than an `Option`, which is what makes it convenient inside an interpolator — and what makes it unsuitable for anything user-supplied:

```scala
CssColor.Hex.unsafe("ff0000").render
// res78: String = "#ff0000"
```

:::warning[`unsafe` renders whatever it is given]
`CssColor.Hex.unsafe` performs no checking, so `Hex.unsafe("not-a-color")` renders as `#not-a-color` and produces a stylesheet the browser silently ignores. Use it only for compile-time literals; route anything dynamic through `CssColor.Hex.apply` and handle the `None`.
:::

### Declarations and Rules

A `Css.Declaration` is a property-value pair:

```scala

val marginDecl = Css.Declaration("margin", "10px")
val colorDecl = Css.Declaration("color", "blue")

val rule = Css.Rule(
  CssSelector.Element("p"),
  Chunk(marginDecl, colorDecl)
)

rule.render
```

### Stylesheets

A `Css.Sheet` is a collection of rules:

```scala

val bodyRule = Css.Rule(
  CssSelector.Element("body"),
  Chunk(
    Css.Declaration("margin", "0"),
    Css.Declaration("font-family", "sans-serif")
  )
)

val stylesheet = Css.Sheet(Chunk(bodyRule))

stylesheet.render(indent = 2)
```

### Embedding Stylesheets

Embed stylesheets in HTML via the `style()` element:

```scala

val page = html(
  head(
    style().inlineCss(
      Css.Sheet(Chunk(
        Css.Rule(
          CssSelector.Element("body"),
          Chunk(Css.Declaration("background", "#f0f0f0"))
        )
      ))
    )
  ),
  body(p("Styled"))
)

page.render
```

### Raw CSS

For CSS features not expressible via the ADT (media queries, keyframes, at-rules), use `Css.Raw`:

```scala

val mediaQuery = Css.Raw("""
  |@media (max-width: 600px) {
  |  body { font-size: 14px; }
  |}
""".stripMargin)

println(mediaQuery.render)
```

:::tip
Prefer `Css.Rule` and `Css.Sheet` over `Css.Raw` when possible — structured CSS enables future optimization. Declaration values are not checked, so validate values that come from users. Use `Css.Raw` only for trusted, hardcoded CSS.
:::

### CSS Comments

Add comments to stylesheets using `Css.Comment`:

```scala

val stylesheet = Css.Sheet(Chunk(
  Css.Comment("Mobile-first responsive design"),
  Css.Rule(
    CssSelector.Element("body"),
    Chunk(Css.Declaration("font-size", "16px"))
  ),
  Css.Comment("Tablet and desktop breakpoints"),
  Css.Raw("@media (min-width: 768px) { body { font-size: 18px; } }")
))

stylesheet.render(indent = 2)
```

## Rendering

All `Dom` and `Css` values support multiple rendering modes.

### Minified Rendering

`Dom#render` produces compact HTML with no extra whitespace. Use `Dom#renderMinified` as an explicit alias for the same operation:

```scala

val page = div(h1("Title"), p("Content"))
page.render
```

### Pretty-Printed Rendering

`render(indent: Int)` produces indented, readable output:

```scala

val page = div(h1("Title"), p("Content"))
page.render(indent = 2)
```

### Performance Notes

- `Dom#render` uses a pre-allocated `StringBuilder` and while-loop rendering — zero allocations for iteration
- Indentation strings get cached in an array of pre-built space strings (up to 128 characters) to prevent repeated string allocation
- Void elements automatically self-close with no children
- Script and Style elements render their children without escaping (with `</` → `<\/` protection for scripts)

## Security: XSS Protection

The module provides multiple layers of automatic XSS protection:

### HTML Text Escaping

All `Dom.Text` nodes are HTML-escaped during rendering:
- `&` → `&amp;`
- `<` → `&lt;`
- `>` → `&gt;`
- `"` → `&quot;`
- `'` → `&#x27;`

Untrusted content is always escaped to prevent XSS:

```scala

val userInput = "<script>alert('XSS')</script>"
val safe = div(p(userInput))

safe.render
```

### JavaScript String Escaping

`ToJs[String]` renders a string as a quoted JavaScript literal. The characters `<`, `>` and `&` are written as `\u003c`, `\u003e` and `\u0026`, quotes and backslashes get a backslash, and line breaks are written as escapes. For example:

```scala

val userInput = "</script><script>alert('XSS');</script>"
// userInput: String = "</script><script>alert('XSS');</script>"
val code = js"let payload = $userInput"
// code: Js = Js(
//   "let payload = \"\\u003c/script\\u003e\\u003cscript\\u003ealert(\\'XSS\\');\\u003c/script\\u003e\""
// )

println(code.value)
// let payload = "\u003c/script\u003e\u003cscript\u003ealert(\'XSS\');\u003c/script\u003e"
```

### URL Sanitization

Attributes named `href`, `src`, `action`, or `formaction` are checked for dangerous schemes at render time:
- `javascript:`, `vbscript:`, `data:text/html` → prefixed with `unsafe:`

Dangerous URLs are automatically sanitized in HTML output:

```scala

val dangerous = a(href := "javascript:alert('XSS')", "Click me")
```

### No Raw HTML Escape Hatch

The module intentionally provides no `Dom.Raw` type for embedding arbitrary HTML. This prevents XSS by ensuring all dynamic content is either constructed via the DSL or interpolated through context-aware interpolators.

## Common Patterns

The module supports several architectural patterns for code organization and reuse:

### Building Reusable Components

Define functions that return `Dom.Element` to create reusable components:

```scala

def card(title: String, content: String): Dom.Element =
  div(
    className := "card",
    h2(title),
    p(content)
  )

val page = div(
  card("Card 1", "Content A"),
  card("Card 2", "Content B")
)
```

### Conditional Rendering

Use `when` and `whenSome` for conditional modifiers:

```scala

def userCard(name: String, isAdmin: Boolean): Dom.Element =
  div(
    className := "user-card"
  ).when(isAdmin)(
    className += "admin",
    span(className := "badge", "Admin")
  )
```

### Rendering Collections

Map over collections to create child elements:

```scala

def userList(users: List[String]): Dom.Element =
  ul(users.map(user => li(user)))

val page = userList(List("Alice", "Bob", "Charlie"))
```

### Template Composition

Combine interpolators and DSL for flexibility:

```scala

val pageTitle = "My Page"
// pageTitle: String = "My Page"
val pageContent = "Welcome to my site"
// pageContent: String = "Welcome to my site"

val page = html"""<html>
  <head><title>$pageTitle</title></head>
  <body>
    ${div(p(pageContent))}
  </body>
</html>"""
// page: Dom = Generic(
//   tag = "html",
//   attributes = IndexedSeq(),
//   children = IndexedSeq(
//     Text(
//       """
//       """
//     ),
//     Generic(
//       tag = "head",
//       attributes = IndexedSeq(),
//       children = IndexedSeq(
//         Generic(
//           tag = "title",
//           attributes = IndexedSeq(),
//           children = IndexedSeq(Text("My Page"))
//         )
//       )
//     ),
//     Text(
//       """
//       """
//     ),
//     Generic(
//       tag = "body",
//       attributes = IndexedSeq(),
//       children = IndexedSeq(
//         Text(
//           """
//         """
//         ),
//         Generic(
//           tag = "div",
//           attributes = IndexedSeq(),
//           children = IndexedSeq(
//             Generic(
//               tag = "p",
//               attributes = IndexedSeq(),
//               children = IndexedSeq(Text("Welcome to my site"))
//             )
//           )
//         ),
//         Text(
//           """
//       """
//         )
//       )
//     ),
//     Text(
// ...

println(page.render)
// <html>
//       <head><title>My Page</title></head>
//       <body>
//         Welcome to my site
//       </body>
//     </html>
```

### Querying for Tests

Use `DomSelection` to write structural assertions:

```scala

val page = div(
  ul(
    li(a(href := "/home", "Home")),
    li(a(href := "/about", "About"))
  )
)

// Test: check number of links
val links = page.select(CssSelector.Element("a"))
assert(links.length == 2)

// Test: check specific href
val aboutLink = page.select(a.withAttribute("href", "/about"))
assert(aboutLink.texts.contains("About"))
```

## Complete Example: A Dashboard Page

Here is a complete, self-contained example building a full dashboard page with navigation, metadata, styling, and content sections:

```scala

val userName = "Alice"
val items = List("Dashboard", "Settings", "Logout")

val pieces: Chunk[Dom] = Chunk(
  doctype,
  html(
    head(
      meta(charset := "utf-8"),
      meta(name := "viewport", content := "width=device-width, initial-scale=1.0"),
      title("My App"),
      link(rel := "stylesheet", href := "/style.css"),
      style().inlineCss(
        Css.Sheet(Chunk(
          Css.Rule(CssSelector.Element("body"), Chunk(
            Css.Declaration("margin", "0"),
            Css.Declaration("font-family", "sans-serif")
          ))
        ))
      )
    ),
    body(
      header(
        nav(items.map(item => a(href := "#", item)))
      ),
      main(
        h1(s"Welcome, $userName!"),
        p("This is your dashboard.")
      ),
      footer(
        p("© 2026 My App")
      ),
      script().inlineJs(js"console.log('Page loaded');")
    )
  )
)

pieces.map(_.render(indent = 2)).mkString("\n")
```

## See also

- [Render Your First Server-Side HTML Page](../guides/your-first-html-page.md) — a hands-on tutorial that builds a book page as a `Dom` tree, step by step, and saves it as an HTML file.

---

## Attribute Values and Infrastructure

This section documents supporting attribute value types and infrastructure for the HTMX DSL. These types handle specialized data encoding, configuration, and type-class infrastructure that enables extensibility.

## Attribute Value Types

These types enable specialized data encoding and configuration:

### HxVals — Custom Values

`HxVals` represents the `hx-vals` attribute, allowing you to send custom data alongside form parameters. It accepts both schema-backed values (automatically JSON-encoded) and raw JSON:

```scala

// From a schema-backed value
case class Extra(userId: Int, role: String)
HxVals.from(Extra(123, "admin"))

// From raw JSON
HxVals.json(Json.Object(Chunk("userId" -> Json.Number(123))))
```

Use `HxVals.from[T](value)` with an implicit `Schema[T]` to automatically encode any data type to JSON. First, set up a schema for your data type:

```scala

case class RequestContext(userId: Int, sessionId: String)
object RequestContext {
  implicit val schema: Schema[RequestContext] = Schema.derived
}
```

Then use it in your form:

```scala

form(
  hxPost := "/api/action",
  hxVals := HxVals.from(RequestContext(42, "abc123")),
  button("Submit")
)
```

Or use `HxVals.json(json)` when you already have JSON. Set up the imports first:

```scala

```

Then construct the JSON:

```scala

form(
  hxPost := "/api/action",
  hxVals := HxVals.json(Json.Object(Chunk("priority" -> Json.String("high")))),
  button("Submit")
)
```

### HxHeadersValue — Custom Headers

`HxHeadersValue` represents the `hx-headers` attribute, sending custom HTTP headers with HTMX requests. Like `HxVals`, it accepts schema-backed values or raw JSON:

```scala

// From raw JSON object
val example1 = HxHeadersValue.json(Json.Object(Chunk(
  "X-Custom-Header" -> Json.String("value"),
  "X-Request-ID" -> Json.String("12345")
)))

// From a schema-backed value

case class Headers(traceId: String)
object Headers {
  implicit val schema: Schema[Headers] = Schema.derived
}
val example2 = HxHeadersValue.from(Headers("xyz789"))
```

Add custom headers to all requests within an element:

```scala

div(
  hxHeaders := HxHeadersValue.json(Json.Object(Chunk(
    "X-API-Key" -> Json.String("secret123"),
    "X-Client-Version" -> Json.String("2.0")
  ))),
  button(hxPost := "/action", "Request (with custom headers)")
)
```

### HxSwapOob — Out-of-Bounds Swaps

`HxSwapOob` represents the `hx-swap-oob` attribute, enabling out-of-bounds swaps that update content outside the primary request target. This allows a single response to update multiple DOM regions simultaneously.

Use `HxSwapOob(true)` for the boolean flag. `HxSwapOob(false)` does not turn the out-of-band swap off: htmx still swaps out of band (a known bug in 0.0.56):

```scala

div(
  id := "status-badge",
  hxSwapOob := HxSwapOob(true),
  "Ready"
)
```

Or use `HxSwapOob.using(swap)` to specify a swap strategy:

```scala

div(
  id := "notification",
  hxSwapOob := HxSwapOob.using(HxSwap.BeforeEnd),
  span("New notification")
)
```

Combine with a target selector:

```scala

div(
  hxSwapOob := HxSwapOob.using(
    HxSwap.InnerHTML,
    HxTarget.css("#status")
  ),
  "Updated Status"
)
```

### HxExtensions — HTMX Extensions

`HxExtensions` represents the `hx-ext` attribute, enabling HTMX extensions. Pass extension names as comma-separated values:

```scala

// Enable multiple extensions
HxExtensions("json-enc", "class-tools")
```

Use in elements to opt in to extensions:

```scala

form(
  hxPost := "/api/submit",
  hxExt := HxExtensions("json-enc", "debug"),
  button("Submit")
)
```

The `hxExt` attribute key provides a convenience builder to construct extensions:

```scala

form(
  hxExt("json-enc", "class-tools"),
  button("Submit")
)
```

### HxAttributeNames — Attribute Disinheritance

`HxAttributeNames` represents the `hx-disinherit` attribute, preventing child elements from inheriting specific HTMX attributes from their ancestors. Pass attribute names as space-separated values:

```scala

// Prevent inheriting these attributes
HxAttributeNames("hx-trigger", "hx-swap")
```

Use to override inherited attributes:

```scala

div(
  hxPost := "/parent-action",
  hxTrigger := HxTrigger.click,
  button(
    hxDisinherit := HxAttributeNames("hx-post", "hx-trigger"),
    hxPost := "/child-action",
    hxTrigger := HxTrigger.submit,
    "Child (overrides parent)"
  )
)
```

The `hxDisinherit` attribute key provides a convenience builder:

```scala

div(
  hxPost := "/parent-action",
  button(
    hxDisinherit("hx-post", "hx-trigger"),
    hxPost := "/child-action",
    "Child"
  )
)
```

## Infrastructure Types

These types provide the internal machinery for the HTMX DSL:

### HtmxAttrKey — Typed Attribute Keys

`HtmxAttrKey[-A]` is the typed attribute key binding an attribute name to its expected value type. Unlike plain HTML attributes, HTMX attributes carry compile-time type information, preventing type mismatches.

The `HtmxAttributes` mixin provides predefined keys like `hxPost`, `hxSwap`, `hxTarget`, etc.; use these rather than constructing `HtmxAttrKey` directly. Each key enforces the correct value type:

```scala

// hxSwap expects HxSwap; this compiles
div(hxSwap := HxSwap.InnerHTML)

// hxTarget expects HxTarget; this compiles
div(hxTarget := HxTarget.This)

// hxPost expects UrlLike (String | Path | URL); this compiles
div(hxPost := "/api/endpoint")
```

### ToHtmxValue — Type Class for Rendering

`ToHtmxValue[-A]` is the type class that renders HTMX domain values to attribute strings. It's automatically derived for types like `HxSwap`, `HxTarget`, `HxTrigger`, and basic primitives.

The module provides instances for common types out of the box:

- **Primitives:** `String`, `Boolean`, `Int`, `Long`, `Double` render to their string representations
- **Selectors:** `CssSelector` renders via its `render` method
- **URLs:** `Path` and `URL` render to encoded strings
- **HTMX domain types:** `HxSwap`, `HxTarget`, `HxTrigger`, `HxParams`, etc. render via their `render` methods
- **Schema-backed types:** `Schema[A]` instances enable `HxVals.from()` to encode any data to JSON

#### The Closed Set of Keys

The type class is for rendering the built-in value types. Each HTMX attribute key is created with a fixed value type, and `:=` takes exactly that type, so defining a `ToHtmxValue` instance for your own type does not make it assignable to any key. To render a domain value, convert it to a `String` (or to the key's value type) yourself, or call `ToHtmxValue#toHtmxValue` by hand:

```scala

val rendered: String = ToHtmxValue[String].toHtmxValue("hello")
```

#### Implicit Resolution

When you assign a value to an HTMX attribute, the type class is summoned implicitly:

```scala

// Implicit ToHtmxValue[HxSwap] is summoned
div(hxSwap := HxSwap.InnerHTML)

// Implicit ToHtmxValue[String] is summoned
div(hxPost := "/endpoint")
```

## Integration

These types work together to provide a complete, extensible HTMX DSL. For example, a form might use multiple attribute value types:

```scala

form(
  hxPost := "/api/submit",
  hxTrigger := HxTrigger.submit,
  hxTarget := HxTarget.closest("form"),
  hxSwap := HxSwap.InnerHTML.settle(250.millis),
  hxParams := HxParams.only("name", "email"),
  hxEncoding := HxEncoding.Multipart,
  hxHeaders := HxHeadersValue.json(Json.Object(Chunk(
    "X-Request-ID" -> Json.String("12345")
  ))),
  hxVals := HxVals.from(RequestContext(42, "session-abc")),
  button("Submit")
)
```

Each attribute key is type-safe, and the `ToHtmxValue` type class ensures values render correctly to HTMX syntax.

---

## HxEncoding

`HxEncoding` represents the `hx-encoding` attribute, controlling how form data is encoded when sent in an HTMX request. The primary use case is file uploads, which require `multipart/form-data` encoding instead of the default URL-encoded form submission.

Use `HxEncoding.Multipart` to enable multipart form data encoding for requests that include file fields. Here is the core pattern:

```scala

// Multipart encoding (for file uploads)
HxEncoding.Multipart
```

## Encoding Strategies

**`HxEncoding.Multipart`** sets the `hx-encoding` attribute to `multipart/form-data`, enabling file uploads and other multipart data:

```scala

form(
  hxPost := "/upload",
  hxEncoding := HxEncoding.Multipart,
  input(name := "file", `type` := "file"),
  input(name := "description"),
  button("Upload")
)
```

When you omit `hxEncoding`, HTMX uses the default encoding (URL-encoded form data), so there is no DSL value for the default case. This keeps the DSL focused on the special cases that deviate from the default.

## Common Patterns

Multipart encoding enables file uploads in HTMX forms. Here are practical usage patterns:

### File Upload with Metadata

Send a file along with additional form fields:

```scala

form(
  hxPost := "/api/upload-document",
  hxEncoding := HxEncoding.Multipart,
  hxTarget := HxTarget.css("#upload-status"),
  hxSwap := HxSwap.InnerHTML,
  fieldset(
    legend("Upload Document"),
    input(
      name := "file",
      `type` := "file",
      accept := ".pdf,.doc,.docx"
    ),
    input(
      name := "title",
      placeholder := "Document Title"
    ),
    textarea(name := "description", "Description"),
    button("Upload")
  )
)
```

### Profile Picture Upload

Update a profile picture with multipart encoding:

```scala

form(
  hxPost := "/api/profile-picture",
  hxEncoding := HxEncoding.Multipart,
  hxTarget := HxTarget.css("#profile-pic"),
  hxSwap := HxSwap.OuterHTML,
  input(
    name := "image",
    `type` := "file",
    accept := "image/*"
  ),
  button("Update Picture")
)
```

## Integration with Other Request Control Attributes

`HxEncoding` works alongside `HxParams` to control both how data is encoded and which fields are sent:

```scala

form(
  hxPost := "/upload",
  hxEncoding := HxEncoding.Multipart,
  hxParams := HxParams.only("file", "title"),  // only send these fields
  input(name := "file", `type` := "file"),
  input(name := "title"),
  input(name := "csrf_token"),  // excluded by hxParams
  button("Upload")
)
```

The `ToHtmxValue[HxEncoding]` instance renders automatically, so `HxEncoding` values work seamlessly with the `hxEncoding` attribute key.

---

## HxParams

`HxParams` represents the `hx-params` attribute, controlling which form parameters are included in the HTMX request. Instead of sending all form fields, you can explicitly allow or forbid specific parameters through compile-safe domain types.

Use `HxParams.All` to include all parameters, `HxParams.None` to include none, or `HxParams.only()` and `HxParams.not()` to allow/forbid specific names. Here are the core patterns:

```scala

// Include all parameters
HxParams.All

// Include no parameters
HxParams.None

// Include only specific parameters
HxParams.only("query", "page")

// Exclude specific parameters
HxParams.not("csrf_token", "session")
```

## Strategies

**`HxParams.All`** includes all form parameters in the request (the default HTMX behavior):

```scala

input(
  hxPost := "/search",
  hxParams := HxParams.All
)
```

**`HxParams.None`** excludes all form parameters. The request body is empty:

```scala

button(
  hxPost := "/ping",
  hxParams := HxParams.None,
  "Ping Server"
)
```

**`HxParams.only(first, rest*)`** includes only the listed parameters by name:

```scala

input(
  hxPost := "/search",
  hxParams := HxParams.only("query", "page", "limit"),
  placeholder := "Search..."
)
```

**`HxParams.not(first, rest*)`** excludes the listed parameters, sending all others:

```scala

form(
  hxPost := "/submit",
  hxParams := HxParams.not("csrf_token", "_method"),
  input(name := "username"),
  input(name := "csrf_token"),
  button("Submit")
)
```

## Rendering & Parsing

**`render: String`** produces the literal `hx-params` attribute value string:

```scala

HxParams.All.render                           // "*"
HxParams.None.render                          // "none"
HxParams.only("query", "page").render         // "query,page"
HxParams.not("csrf_token", "session").render  // "not csrf_token,session"
```

**`HxParams.parse(value: String): Either[String, HxParams]`** parses a rendered string back into a typed `HxParams`:

```scala

HxParams.parse("*")                    // Right(HxParams.All)
HxParams.parse("none")                 // Right(HxParams.None)
HxParams.parse("query,page")           // Right(HxParams.only("query", "page"))
HxParams.parse("not csrf,session")     // Right(HxParams.not("csrf", "session"))
```

Parse failures return a descriptive `Left`:

```scala

HxParams.parse("")                     // Left("HTMX params list must be non-empty")
HxParams.parse("query,,page")          // Left("HTMX params list cannot contain empty names")
```

## Common Patterns

`HxParams` is useful when you need fine-grained control over which form fields are sent to the server. Here are representative usage patterns:

### Search Without CSRF Token

Include only the search query parameter, excluding the CSRF token typically added by form handlers:

```scala

input(
  name := "query",
  hxPost := "/search",
  hxParams := HxParams.only("query"),
  placeholder := "Search..."
)
```

### Selective Parameter Submission

When a form has many fields but an HTMX request should only send a few:

```scala

form(
  input(name := "firstName"),
  input(name := "lastName"),
  input(
    name := "email",
    hxPost := "/validate-email",
    hxParams := HxParams.only("email"),
    hxTarget := HxTarget.next(".error"),
    placeholder := "Email"
  ),
  div("Error message")
)
```

### Exclude Framework Parameters

Prevent sending framework-specific parameters (like Rails' `_method`, CSRF tokens) to the HTMX endpoint:

```scala

form(
  hxPost := "/api/submit",
  hxParams := HxParams.not("_method", "authenticity_token", "utf8"),
  button("Submit")
)
```

### No Parameter Request

Send a request without any form data, useful for server-side-only actions:

```scala

button(
  hxPost := "/logout",
  hxParams := HxParams.None,
  "Logout"
)
```

## Integration with Request Control

`HxParams` works with other request-control attributes like `hxEncoding` and `hxSync`:

```scala

form(
  hxPost := "/upload",
  hxParams := HxParams.only("file", "description"),
  hxEncoding := HxEncoding.Multipart,
  input(name := "file", `type` := "file"),
  input(name := "description"),
  button("Upload")
)
```

The `ToHtmxValue[HxParams]` instance renders automatically, so `HxParams` values work seamlessly with the `hxParams` attribute key.

---

## HxSwap

`HxSwap` represents the `hx-swap` attribute, controlling how HTMX replaces DOM content after a successful response. It combines a base strategy (innerHTML, outerHTML, etc.) with optional modifiers that refine timing, animation, scrolling, and focus behavior.

Start from one of the predefined strategies, then call modifier methods to customize timing, animation, and scroll behavior. All modifiers are optional, and calling the same modifier twice replaces the earlier value rather than stacking. Here are the core patterns:

```scala

// Base strategy only
HxSwap.InnerHTML

// With timing modifiers
HxSwap.OuterHTML.swap(1.second).settle(500.millis)

// With animation and scroll
HxSwap.BeforeEnd.transition.scroll(HxSwap.ScrollPosition.Top)

// Full pipeline
HxSwap.InnerHTML
  .swap(500.millis)
  .settle(250.millis)
  .transition
  .scroll(HxSwap.ScrollPosition.Bottom)
  .show(HxSwap.ShowPosition.Top)
  .focusScroll(true)
```

## Swap Strategies

Core HTMX strategies define where and how content replaces the DOM:

- `HxSwap.InnerHTML` — Replaces the inner content of the target element. This is the default HTMX behavior.
- `HxSwap.OuterHTML` — Replaces the target element itself, including its opening and closing tags.
- `HxSwap.TextContent` — Replaces only the text content, leaving the element structure intact.
- `HxSwap.BeforeBegin` — Inserts the response as a sibling before the target element.
- `HxSwap.AfterBegin` — Inserts the response as the first child of the target element.
- `HxSwap.BeforeEnd` — Inserts the response as the last child of the target element.
- `HxSwap.AfterEnd` — Inserts the response as a sibling after the target element.
- `HxSwap.Delete` — Removes the target element; response content is discarded.
- `HxSwap.NoneSwap` — Prevents any content from being swapped into the DOM; the response is ignored.

All strategies are available as immutable `HxSwap` values, ready for modifier chaining:

```scala

div(hxSwap := HxSwap.InnerHTML)
div(hxSwap := HxSwap.OuterHTML)
div(hxSwap := HxSwap.BeforeBegin)
```

## Timing Modifiers

Control when and how long the swap operation takes:

**`swap(duration: FiniteDuration)`** adds a `swap:` delay before the swap begins. Useful for coordinating with request latency or preparing the DOM:

```scala

// Wait 500ms after response before swapping
div(hxSwap := HxSwap.InnerHTML.swap(500.millis))
```

**`settle(duration: FiniteDuration)`** adds a `settle:` delay after the swap before settling (CSS transitions, settling messages). This allows animations to run:

```scala

// Swap immediately, but wait 250ms for CSS to settle
div(hxSwap := HxSwap.InnerHTML.settle(250.millis))
```

Both `swap()` and `settle()` accept any `FiniteDuration` that renders to a valid HTMX duration (milliseconds: `ms`, seconds: `s`, or bare integers treated as milliseconds). Calling either method twice replaces the first value:

```scala

val swap1 = HxSwap.InnerHTML.swap(1.second)
val swap2 = swap1.swap(500.millis)  // replaces 1.second with 500.millis
```

## Animation & Transition

**`transition: HxSwap`** enables the `transition:true` modifier, allowing CSS transitions to run during the swap. Useful with timing modifiers to coordinate animation:

```scala

// Enable transitions and give them 300ms to complete
div(hxSwap := HxSwap.InnerHTML.transition.settle(300.millis))
```

## Scroll Behavior

Control where the page scrolls after a swap:

**`scroll(position: HxSwap.ScrollPosition)`** adds a `scroll:` modifier. Valid positions are `HxSwap.ScrollPosition.Top` and `HxSwap.ScrollPosition.Bottom`:

```scala

// Scroll to top after swapping
div(hxSwap := HxSwap.InnerHTML.scroll(HxSwap.ScrollPosition.Top))

// Scroll to bottom (useful for chat/log appends)
div(hxSwap := HxSwap.BeforeEnd.scroll(HxSwap.ScrollPosition.Bottom))
```

**`show(position: HxSwap.ShowPosition)`** adds a `show:` modifier to control where newly swapped content becomes visible. Valid positions are `HxSwap.ShowPosition.Top` and `HxSwap.ShowPosition.Bottom`:

```scala

// Show the top of the newly swapped content
div(hxSwap := HxSwap.InnerHTML.show(HxSwap.ShowPosition.Top))
```

## Focus & Title Management

**`focusScroll(enabled: Boolean)`** adds `focusScroll:true` or `focusScroll:false`, which is meant to control whether the browser scrolls to a focused input after the swap. It does not control whether focus moves. htmx 2 spells the modifier `focus-scroll:` and ignores the `focusScroll:` token this method renders (a known bug in 0.0.56):

```scala

// Enable focus scroll (default HTMX behavior)
div(hxSwap := HxSwap.InnerHTML.focusScroll(true))

// Disable automatic focus movement
div(hxSwap := HxSwap.InnerHTML.focusScroll(false))
```

**`ignoreTitle: HxSwap`** enables the `ignoreTitle:true` modifier, preventing HTMX from updating the document title if the response contains an `<title>` tag:

```scala

// Don't update the page title even if response has one
div(hxSwap := HxSwap.InnerHTML.ignoreTitle)
```

## Rendering & Parsing

**`render: String`** produces the literal `hx-swap` attribute value string:

```scala

val swap = HxSwap.InnerHTML.swap(1.second).settle(500.millis).transition
swap.render  // "innerHTML swap:1s settle:500ms transition:true"
```

**`HxSwap.parse(value: String): Either[String, HxSwap]`** decodes a rendered HTMX swap string into a typed `HxSwap` value. The parser accepts the same format that `render` produces:

```scala

val parsed = HxSwap.parse("outerHTML swap:2s settle:250ms transition:true scroll:bottom")
// Right(HxSwap.OuterHTML.swap(2 seconds).settle(250 millis).transition.scroll(ScrollPosition.Bottom))
```

Parse failures return a descriptive `Left` with the error reason:

```scala

HxSwap.parse("")              // Left("Empty HTMX swap value")
HxSwap.parse("bogus")         // Left("Unknown HTMX swap strategy: bogus")
HxSwap.parse("innerHTML swap:later")  // Left("Unsupported HTMX duration: later")
HxSwap.parse("innerHTML scroll:middle")  // Left("Invalid HTMX scroll position: middle")
```

## Common Patterns

Effective swap strategies enhance user experience through proper timing and animation. Here are practical usage patterns:

### Fade-In New Content

Combine `HxSwap#transition` with `HxSwap#settle` delay to fade in new elements:

```scala

div(
  hxGet := "/new-items",
  hxSwap := HxSwap.BeforeEnd.transition.settle(300.millis)
)
```

### Scroll New Content Into View

Use `HxSwap#scroll` with `BeforeEnd` or `AfterEnd` to append content and scroll to it:

```scala

div(
  id := "messages",
  hxGet := "/new-messages",
  hxTrigger := HxTrigger.every(2.seconds),
  hxSwap := HxSwap.BeforeEnd.scroll(HxSwap.ScrollPosition.Bottom)
)
```

### Replace and Focus

Use `InnerHTML` or `OuterHTML` with `HxSwap#focusScroll` to replace content and move focus:

```scala

form(
  hxPost := "/submit",
  hxSwap := HxSwap.OuterHTML.focusScroll(true),
  button("Submit")
)
```

### Preserve Animations While Settling

Increase the `HxSwap#settle` delay to let CSS animations complete before HTMX considers the swap done:

```scala

// CSS animations run for 500ms; settle after they complete
div(
  hxPost := "/update",
  hxSwap := HxSwap.InnerHTML.transition.settle(500.millis)
)
```

## Integration with Other Module Types

`HxSwap` is most often paired with `HxTrigger` (when to request) and `HxTarget` (where to apply the swap):

```scala

button(
  hxPost := "/api/action",
  hxTrigger := HxTrigger.click.delay(100.millis),
  hxTarget := HxTarget.closest("form"),
  hxSwap := HxSwap.InnerHTML.settle(250.millis),
  "Perform Action"
)
```

The `ToHtmxValue[HxSwap]` type class renders automatically, enabling you to assign `HxSwap` values directly using the `:=` operator on the `hxSwap` attribute key.

---

## HxSync

`HxSync` represents the `hx-sync` attribute, coordinating multiple HTMX requests by specifying how new requests interact with pending or running requests on a target element. This prevents race conditions and ensures predictable behavior when multiple events fire in quick succession.

Create an `HxSync` by pairing an `HxTarget` (which element to synchronize) with a `HxSyncStrategy` (how to handle conflicts):

```scala

// Wait for this element's request to finish before starting a new one
HxSync(HxTarget.This, HxSyncStrategy.Queue)

// Replace a pending request with a new one
HxSync(HxTarget.closest("form"), HxSyncStrategy.Replace)

// Drop new requests while one is in flight
HxSync(HxTarget.This, HxSyncStrategy.Drop)

// Abort the current request and start the new one immediately
HxSync(HxTarget.This, HxSyncStrategy.Abort)
```

## Sync Strategies

**`HxSyncStrategy.Queue`** queues new requests, executing them after the current request completes. This is the default HTMX behavior when `hx-sync` is not specified. Useful for ensuring requests are processed in order:

```scala

input(
  hxPost := "/search",
  hxSync := HxSync(HxTarget.This, HxSyncStrategy.Queue),
  placeholder := "Search..."
)
```

**`HxSyncStrategy.Replace`** cancels any pending request and immediately sends the new one. Useful for searches and filters where the latest value is all that matters:

```scala

input(
  hxPost := "/search",
  hxSync := HxSync(HxTarget.This, HxSyncStrategy.Replace),
  placeholder := "Live Search (only latest request sent)"
)
```

**`HxSyncStrategy.Drop`** discards new requests while one is in flight. Only the first request in a burst is sent; others are ignored. Useful for expensive operations that shouldn't be duplicated:

```scala

button(
  hxPost := "/expensive-operation",
  hxSync := HxSync(HxTarget.This, HxSyncStrategy.Drop),
  "Process (only first request sent)"
)
```

**`HxSyncStrategy.Abort`** cancels any in-flight request and immediately sends the new one. The old request's response is discarded. Useful for time-sensitive operations:

```scala

button(
  hxPost := "/time-sensitive-action",
  hxSync := HxSync(HxTarget.This, HxSyncStrategy.Abort),
  "Execute (cancels previous)"
)
```

## Target Specification

The first parameter of `HxSync` is an `HxTarget` specifying which element to synchronize. This allows coordinating requests across multiple elements:

**Synchronize the current element:**

Use `HxTarget.This` to synchronize the element that initiates the request:

```scala

input(
  hxPost := "/search",
  hxSync := HxSync(HxTarget.This, HxSyncStrategy.Replace),
  placeholder := "Search..."
)
```

**Synchronize a parent form:**

Synchronize across all fields in a form using `HxTarget.closest()`:

```scala

input(
  hxPost := "/validate",
  hxSync := HxSync(HxTarget.closest("form"), HxSyncStrategy.Queue),
  placeholder := "Email"
)
```

**Synchronize multiple fields at once:**

Apply the same sync strategy to multiple fields within a form:

```scala

form(
  input(
    name := "firstName",
    hxPost := "/validate",
    hxSync := HxSync(HxTarget.closest("form"), HxSyncStrategy.Queue)
  ),
  input(
    name := "email",
    hxPost := "/validate",
    hxSync := HxSync(HxTarget.closest("form"), HxSyncStrategy.Queue)
  )
)
```

## Rendering & Parsing

**`render: String`** produces the literal `hx-sync` attribute value string:

```scala

HxSync(HxTarget.This, HxSyncStrategy.Queue).render     // "this:queue"
HxSync(HxTarget.closest("form"), HxSyncStrategy.Replace).render  // "closest form:replace"
```

**`HxSync.parse(value: String): Either[String, HxSync]`** parses a rendered string back into a typed `HxSync`:

```scala

HxSync.parse("this:queue")             // Right(HxSync(HxTarget.This, HxSyncStrategy.Queue))
HxSync.parse("closest form:replace")   // Right(HxSync(HxTarget.closest("form"), HxSyncStrategy.Replace))
```

## Common Patterns

Synchronization strategies prevent race conditions in interactive forms. Here are practical usage patterns:

### Search with Last-Value-Wins

Use `Replace` to ensure only the most recent search is sent when typing quickly:

```scala

input(
  hxPost := "/search",
  hxTrigger := HxTrigger.input.delay(300.millis),
  hxSync := HxSync(HxTarget.This, HxSyncStrategy.Replace),
  hxTarget := HxTarget.css("#results"),
  placeholder := "Search (only latest request sent)"
)
```

### Ordered Form Validation

Use `Queue` to validate form fields in the order they were changed, preventing race conditions:

```scala

form(
  input(
    name := "username",
    hxPost := "/validate/username",
    hxSync := HxSync(HxTarget.closest("form"), HxSyncStrategy.Queue)
  ),
  input(
    name := "email",
    hxPost := "/validate/email",
    hxSync := HxSync(HxTarget.closest("form"), HxSyncStrategy.Queue)
  )
)
```

### Prevent Double-Submit

Use `Drop` to prevent duplicate submissions when users click a button repeatedly:

```scala

form(
  button(
    hxPost := "/submit",
    hxSync := HxSync(HxTarget.This, HxSyncStrategy.Drop),
    "Submit (first click only)"
  )
)
```

### Abort Slow Requests

Use `Abort` for requests that shouldn't queue up, replacing old requests immediately:

```scala

button(
  hxPost := "/fetch-latest",
  hxSync := HxSync(HxTarget.This, HxSyncStrategy.Abort),
  "Fetch Latest (aborts previous)"
)
```

## Integration with Other Attributes

`HxSync` works alongside trigger and timing attributes to coordinate complex request patterns:

```scala

input(
  hxPost := "/search",
  hxTrigger := HxTrigger.input.delay(500.millis).changed,
  hxSync := HxSync(HxTarget.This, HxSyncStrategy.Replace),
  hxParams := HxParams.only("query"),
  hxTarget := HxTarget.css("#results"),
  placeholder := "Search"
)
```

The `ToHtmxValue[HxSync]` instance renders automatically, so `HxSync` values work seamlessly with the `hxSync` attribute key.

---

## HxTarget

`HxTarget` represents the `hx-target` attribute and related selector-based attributes, declaring where HTMX applies the swap. It supports DOM traversal patterns like "closest ancestor," "first descendant," "next sibling," and raw CSS selectors, eliminating stringly mistakes in selector construction.

Use `HxTarget.This` for the current element, or call traversal methods like `closest()`, `find()`, `next()`, and `previous()` for common patterns. For arbitrary CSS selectors, use `css()`. Here are the core patterns:

```scala

// Current element
HxTarget.This

// Closest ancestor matching selector
HxTarget.closest("form")

// First descendant matching selector
HxTarget.find(".results")

// Next sibling
HxTarget.next

// Next sibling matching selector
HxTarget.next(".item")

// Raw CSS selector
HxTarget.css("#results")
```

## Common Targets

**`HxTarget.This`** targets the element itself (the `this` keyword in HTMX). Backtick-friendly and identifier-friendly aliases are available:

```scala

button(
  hxPost := "/api/action",
  hxTarget := HxTarget.This,
  "Update Me"
)
```

```scala

HxTarget.`this`  // backtick-friendly
HxTarget.this_   // identifier-friendly (underscore avoids Scala keyword)
```

## DOM Traversal Patterns

**`closest(selector: String)`** traverses up the DOM tree to find the closest ancestor matching the selector:

```scala

// Update the closest form ancestor
input(
  hxPost := "/validate",
  hxTarget := HxTarget.closest("form")
)

// Update the closest .card ancestor
button(
  hxGet := "/refresh",
  hxTarget := HxTarget.closest(".card")
)
```

**`find(selector: String)`** finds the first descendant matching the selector:

```scala

// Update the first .results div inside this element
div(
  hxGet := "/search-results",
  hxTarget := HxTarget.find(".results")
)
```

**`next`** targets the next sibling element without filtering:

```scala

// Update the next sibling
button(
  hxPost := "/action",
  hxTarget := HxTarget.next,
  "Go"
)
```

**`next(selector: String)`** targets the next element matching the selector. htmx scans the whole document in order, starting after the current element, so the match need not be a sibling (the bare `next` form is the sibling):

```scala

// Update the next .row sibling
div(
  hxGet := "/row-data",
  hxTarget := HxTarget.next(".row")
)
```

**`HxTarget.previous`** targets the previous sibling element:

```scala

button(
  hxPost := "/action",
  hxTarget := HxTarget.previous
)
```

**`previous(selector: String)`** targets the previous element matching the selector. htmx scans the whole document backwards from the current element, so the match need not be a sibling (the bare `previous` form is the sibling):

```scala

div(
  hxGet := "/prev-data",
  hxTarget := HxTarget.previous(".section")
)
```

## CSS Selectors

**`css(selector: String)`** uses a raw CSS selector string. Validation ensures the selector is non-empty but otherwise accepts any string:

```scala

div(
  hxGet := "/data",
  hxTarget := HxTarget.css("#results")
)

// Complex CSS selector
div(
  hxPost := "/search",
  hxTarget := HxTarget.css("div.container > .results:first-child")
)
```

**`css(selector: CssSelector)`** accepts a typed `CssSelector` from `zio.blocks.html`, providing type-safe selector construction:

```scala

div(
  hxGet := "/update",
  hxTarget := HxTarget.css(selector".results")
)
```

## Rendering & Parsing

**`render: String`** produces the literal HTMX target selector string:

```scala

HxTarget.This.render                  // "this"
HxTarget.closest("form").render       // "closest form"
HxTarget.find(".results").render      // "find .results"
HxTarget.next.render                  // "next"
HxTarget.next(".item").render         // "next .item"
HxTarget.previous(".row").render      // "previous .row"
HxTarget.css("#results").render       // "#results"
```

**`HxTarget.parse(value: String): Either[String, HxTarget]`** parses a rendered selector string back into a typed `HxTarget`:

```scala

HxTarget.parse("this")                // Right(HxTarget.This)
HxTarget.parse("closest form")        // Right(HxTarget.closest("form"))
HxTarget.parse("find .results")       // Right(HxTarget.find(".results"))
HxTarget.parse("next")                // Right(HxTarget.next)
HxTarget.parse("next .item")          // Right(HxTarget.next(".item"))
HxTarget.parse("#results")            // Right(HxTarget.css("#results"))
```

Parse failures return a descriptive `Left`:

```scala

HxTarget.parse("")                    // Left("HTMX target selector must be non-empty")
HxTarget.parse("   ")                 // Left("HTMX target selector must be non-empty")
```

## Common Patterns

Selecting the right target element is essential for HTMX interactions. Here are practical patterns for common scenarios:

### Update a Form Container

When a button inside a form fires a request, target the form to replace all its content:

```scala

form(
  id := "search-form",
  input(placeholder := "Search..."),
  button(
    hxPost := "/search",
    hxTarget := HxTarget.closest("form"),
    "Search"
  )
)
```

### Update a Results Section

Target a specific div outside the trigger element:

```scala

div(
  input(
    hxPost := "/search",
    hxTarget := HxTarget.find(".results"),
    placeholder := "Search..."
  ),
  div(id := "results", "Results here")
)
```

### Append to a List

Use `AfterBegin` or `BeforeEnd` with a target on the list container:

```scala

ul(
  id := "items",
  li(
    button(
      hxPost := "/add-item",
      hxTarget := HxTarget.closest("ul"),
      hxSwap := HxSwap.BeforeEnd,
      "Add Item"
    )
  ),
  li("Item 1")
)
```

### Target the Next Row in a Table

Each row has an "edit" button that updates the next row:

```scala

tr(
  td("Data 1"),
  td(button(
    hxGet := "/edit",
    hxTarget := HxTarget.next("tr"),
    "Edit"
  ))
)
```

## Integration with Other Attributes

Multiple HTMX attributes accept `HxTarget` values:

- `hxTarget` — Where to apply the swap
- `hxInclude` — Which elements to include in the request (uses similar selector patterns)
- `hxSync` — Which element to synchronize with

All accept the same `HxTarget` type:

```scala

form(
  hxPost := "/submit",
  hxTarget := HxTarget.closest("form"),
  hxInclude := HxTarget.closest("form"),
  hxSync := HxSync(HxTarget.This, HxSyncStrategy.Queue),
  button("Submit")
)
```

The `ToHtmxValue[HxTarget]` instance handles rendering automatically, so `HxTarget` values work seamlessly with all attribute keys that accept selectors.

---

## HxTrigger

`HxTrigger` represents the `hx-trigger` attribute, declaring which event fires an HTMX request. It combines an event name with optional modifiers that refine timing (delay, throttle), state (once, changed), source (from), queue strategy, and filtering. `HxTriggerSet` composes multiple triggers in a comma-separated list for complex event handling.

Start from a predefined trigger like `HxTrigger.click` or construct one with `HxTrigger("eventName")`. Add modifiers by chaining methods. Modifiers within the same group (e.g., two `delay` calls) replace earlier values; unrelated modifiers accumulate. Here are the core patterns:

```scala

// Predefined trigger
HxTrigger.click

// Custom event name
HxTrigger("myEvent")

// With modifiers
HxTrigger.input.delay(500.millis).changed

// Multiple triggers
HxTrigger(HxTrigger.click, HxTrigger.load)

// Polling
HxTrigger.every(2.seconds)
```

## Predefined Triggers

Common HTMX events are available as case objects:

- `HxTrigger.click` — Fires on element click.
- `HxTrigger.submit` — Fires on form submission.
- `HxTrigger.load` — Fires when the element is first loaded and added to the DOM.
- `HxTrigger.change` — Fires when an input/select/textarea value changes.
- `HxTrigger.input` — Fires on each keystroke in an input/textarea.
- `HxTrigger.revealed` — Fires when the element scrolls into view (requires Intersection Observer).
- `HxTrigger.intersect` — Fires when the element enters the intersection observer threshold.

All predefined triggers are ready for modifier chaining:

```scala

input(hxPost := "/search", hxTrigger := HxTrigger.input)
form(hxPost := "/submit", hxTrigger := HxTrigger.submit)
div(hxGet := "/status", hxTrigger := HxTrigger.load)
```

## Custom Events

Construct a trigger from an arbitrary event name using `HxTrigger("eventName")`:

```scala

// Custom event (must be fired by JavaScript)
div(hxPost := "/api/action", hxTrigger := HxTrigger("custom-event"))
```

The module validates that the event name is non-empty but otherwise accepts any string. Custom events must be fired by your JavaScript code or other HTMX event sources.

## Timing Modifiers

Control when the request fires relative to the triggering event:

**`delay(duration: FiniteDuration)`** adds a `delay:` modifier, waiting before firing the request. Useful for debouncing high-frequency events like typing:

```scala

// Wait 500ms after typing stops before firing
input(
  hxPost := "/search",
  hxTrigger := HxTrigger.input.delay(500.millis)
)
```

**`throttle(duration: FiniteDuration)`** adds a `throttle:` modifier, firing at most once per duration. Useful for rate-limiting requests on frequent events:

```scala

// Fire at most once per second
input(
  hxPost := "/search",
  hxTrigger := HxTrigger.input.throttle(1.second)
)
```

Both modifiers accept any `FiniteDuration`. Calling `delay()` or `throttle()` twice replaces the earlier value:

```scala

val trigger1 = HxTrigger.input.delay(1.second)
val trigger2 = trigger1.delay(500.millis)  // replaces 1.second
```

## State Modifiers

Refine which events fire the request based on element state:

**`once: HxTrigger`** adds the `once` modifier, firing the request exactly once and then disabling the trigger:

```scala

// Load data only on first page visit
div(hxGet := "/initial-data", hxTrigger := HxTrigger.load.once)
```

**`changed: HxTrigger`** adds the `changed` modifier, firing only when the element's value actually changes (not on every event):

```scala

// Fire only if the value is different from the last value
input(
  hxPost := "/validate",
  hxTrigger := HxTrigger.input.changed
)
```

Combine these with timing modifiers for fine-grained control:

```scala

HxTrigger.input.delay(500.millis).changed
```

## Source & Target Control

Direct where the request originates and how it affects other elements:

**`from(selector: String)` or `from(target: HxTarget)`** adds a `from:` modifier, listening for the trigger on a different element. The request still fires on the original element, but it listens for the trigger event on the specified source:

```scala

// Button fires a request, but listens for clicks on the input
input(id := "search")
button(
  "Go",
  hxPost := "/search",
  hxTrigger := HxTrigger.click.from("#search")
)
```

**`target(selector: String)`** adds a `target:` modifier, restricting the trigger to events from specific descendant elements. Useful for event delegation:

```scala

// Only fire on clicks within .clickable items
div(
  hxPost := "/item-selected",
  hxTrigger := HxTrigger.click.target(".clickable"),
  div(className := "clickable", "Item 1"),
  div(className := "clickable", "Item 2")
)
```

## Queue Strategy

Control how requests queue when multiple triggers fire in quick succession:

**`queue(strategy: HxTrigger.QueueStrategy)`** adds a `queue:` modifier. Valid strategies are:

- `HxTrigger.QueueStrategy.First` — Queue the first request only; discard later ones until it completes.
- `HxTrigger.QueueStrategy.Last` — Keep the most recent request; discard earlier queued ones.
- `HxTrigger.QueueStrategy.All` — Queue all requests and fire them in order (default HTMX behavior).
- `HxTrigger.QueueStrategy.None` — Abort pending requests and fire the new one immediately.

Here are examples using different queue strategies:

```scala

// Only fire the most recent request
input(
  hxPost := "/search",
  hxTrigger := HxTrigger.input.delay(500.millis).queue(HxTrigger.QueueStrategy.Last)
)

// Fire only the first request in a burst
button(
  hxPost := "/submit",
  hxTrigger := HxTrigger.click.queue(HxTrigger.QueueStrategy.First)
)
```

## Intersection Observer

**`threshold(value: Double)`** adds a `threshold:` modifier for Intersection Observer-based triggers (e.g., `HxTrigger.intersect`). The value is a fraction between 0.0 and 1.0:

```scala

// Load when 50% of the element is visible
div(
  hxGet := "/load-more",
  hxTrigger := HxTrigger.intersect.threshold(0.5)
)
```

**`root(selector: String)`** adds a `root:` modifier, specifying the container for Intersection Observer calculations:

```scala

// Observe intersection within a scrollable container
div(
  id := "scrollable-list",
  div(
    hxGet := "/item",
    hxTrigger := HxTrigger.intersect.root(".scrollable-list")
  )
)
```

## JavaScript Filtering

**`filter(expression: Js)`** adds a JavaScript expression that must return true for the request to fire. The `Js` type is intentionally raw—do not build it from unsanitized user input:

```scala

// Only POST if input has 3+ characters
input(
  hxPost := "/search",
  hxTrigger := HxTrigger.input.filter(Js("event.target.value.length > 2"))
)
```

The filter expression has access to the native JavaScript `event` object and the element's JavaScript context.

## Request Consumption

**`consume: HxTrigger`** adds the `consume` modifier, preventing the triggering event from bubbling to parent handlers:

```scala

// Click on button fires request and stops event propagation
button(
  hxPost := "/action",
  hxTrigger := HxTrigger.click.consume
)
```

## Polling

**`HxTrigger.every(duration: FiniteDuration)`** creates a special polling trigger that fires every N milliseconds/seconds:

```scala

// Poll every 3 seconds
div(
  hxGet := "/status",
  hxTrigger := HxTrigger.every(3.seconds)
)
```

The duration can be any `FiniteDuration`. Polling triggers can be paused and resumed by HTMX via the `htmx:beforeRequest` and `htmx:afterRequest` events.

## Rendering & Parsing

**`render: String`** produces the literal `hx-trigger` attribute value string:

```scala

val trigger = HxTrigger.input.delay(500.millis).changed
trigger.render  // "input delay:500ms changed"
```

## Multiple Triggers (HxTriggerSet)

Use the `HxTrigger` constructor to combine multiple triggers in a comma-separated list:

```scala

// Fire on click or every 5 seconds
val triggers = HxTrigger(
  HxTrigger.click,
  HxTrigger.every(5.seconds)
)
triggers.render  // "click, every 5s"
```

Assign `HxTriggerSet` directly to the `hxTrigger` attribute:

```scala

div(
  hxGet := "/update",
  hxTrigger := HxTrigger(
    HxTrigger.click,
    HxTrigger.load,
    HxTrigger.every(10.seconds)
  )
)
```

## Common Patterns

`HxTrigger` enables sophisticated event handling through modifiers. Here are practical usage patterns:

### Debounced Search

Delay firing to let the user finish typing, and only fire if the value changed:

```scala

input(
  placeholder := "Search...",
  hxPost := "/api/search",
  hxTrigger := HxTrigger.input.delay(500.millis).changed
)
```

### Rate-Limited Live Updates

Throttle rapid events to avoid overwhelming the server:

```scala

div(
  hxPost := "/analytics",
  hxTrigger := HxTrigger.input.throttle(1.second)
)
```

### Load Once

Fire the request exactly once when the page loads:

```scala

div(
  hxGet := "/welcome-message",
  hxTrigger := HxTrigger.load.once
)
```

### Lazy Loading with Intersection Observer

Load content when it scrolls into view:

```scala

// Load when 10% of the image is visible
img(
  src := "/placeholder.jpg",
  hxGet := "/lazy-image",
  hxSwap := HxSwap.OuterHTML,
  hxTrigger := HxTrigger.intersect.threshold(0.1)
)
```

### Dual Triggers: Click or Automatic

Fire on click or every 30 seconds:

```scala

button(
  hxPost := "/refresh",
  hxTrigger := HxTrigger(
    HxTrigger.click,
    HxTrigger.every(30.seconds)
  ),
  "Refresh Now"
)
```

### Event Delegation

Parent element listens for clicks on child items:

```scala

ul(
  hxPost := "/item-selected",
  hxTrigger := HxTrigger.click.target("li"),
  li("Item 1"),
  li("Item 2"),
  li("Item 3")
)
```

## Integration with Other Module Types

`HxTrigger` pairs with `HxTarget` (where to swap) and `HxSwap` (how to swap):

```scala

button(
  hxPost := "/api/action",
  hxTrigger := HxTrigger.click.delay(100.millis),
  hxTarget := HxTarget.closest("form"),
  hxSwap := HxSwap.InnerHTML.settle(250.millis),
  "Submit"
)
```

The `ToHtmxValue` type class handles both `HxTrigger` (single trigger) and `HxTriggerSet` (multiple triggers), so you can assign either directly to the `hxTrigger` attribute key.

---

## HxUrlUpdate

`HxUrlUpdate` represents the `hx-push-url` and `hx-replace-url` attributes, controlling whether and how the browser's URL bar updates after an HTMX request. You can enable/disable URL updates as booleans or specify a custom URL to push/replace.

Use `HxUrlUpdate.Enabled` or `HxUrlUpdate(true)` to update the URL with the response URL, or pass a custom URL string/Path/URL to update to a different address. Use `HxUrlUpdate.Disabled` or `HxUrlUpdate(false)` to prevent updates. Here are the core patterns:

```scala

// Enable URL update (default)
HxUrlUpdate.Enabled

// Disable URL update
HxUrlUpdate.Disabled

// Custom URL string
HxUrlUpdate("/custom-url")

// Custom URL using Path
HxUrlUpdate(Path("/custom-url"))

// Boolean shorthand
HxUrlUpdate(true)
HxUrlUpdate(false)
```

## Strategies

**`HxUrlUpdate.Enabled`** updates the browser URL with the response URL. This is the default behavior when `hxPushUrl` or `hxReplaceUrl` are set:

```scala

a(
  href := "/page-2",
  hxPushUrl := HxUrlUpdate.Enabled,
  "Page 2"
)
```

**`HxUrlUpdate.Disabled`** prevents the URL from updating, even though the page content changes:

```scala

form(
  hxPost := "/search",
  hxPushUrl := HxUrlUpdate.Disabled,
  "Search (URL won't change)"
)
```

**`HxUrlUpdate(customUrl: String)`** updates the URL to a specific custom address instead of the response URL:

```scala

form(
  hxPost := "/api/search-internal",
  hxPushUrl := HxUrlUpdate("/results"),
  "Search (URL becomes /results)"
)
```

**`HxUrlUpdate(path: Path)` or `HxUrlUpdate(url: URL)`** accepts typed paths and URLs, which are then encoded to strings:

```scala

form(
  hxPost := "/api/search",
  hxPushUrl := HxUrlUpdate(Path("/search-results")),
  button("Search")
)
```

## Boolean Constructors

Convenient overloads let you use booleans directly:

```scala

form(
  hxPost := "/search",
  hxPushUrl := HxUrlUpdate(true),  // same as HxUrlUpdate.Enabled
  button("Search")
)

button(
  hxPost := "/action",
  hxPushUrl := HxUrlUpdate(false), // same as HxUrlUpdate.Disabled
  "Action (URL unchanged)"
)
```

## Rendering & Parsing

**`render: String`** produces the literal attribute value string:

```scala

HxUrlUpdate.Enabled.render                    // "true"
HxUrlUpdate.Disabled.render                   // "false"
HxUrlUpdate("/results").render                // "/results"
```

**`HxUrlUpdate.parse(value: String): Either[String, HxUrlUpdate]`** parses a rendered string back into a typed `HxUrlUpdate`:

```scala

HxUrlUpdate.parse("true")                     // Right(HxUrlUpdate.Enabled)
HxUrlUpdate.parse("false")                    // Right(HxUrlUpdate.Disabled)
HxUrlUpdate.parse("/custom")                  // Right(HxUrlUpdate("/custom"))
```

## Common Patterns

URL updates enable seamless integration with browser history and bookmarking. Here are representative usage patterns:

### Progressive Enhancement with URL Updates

Keep the URL in sync with SPA-like navigation while using server-side rendering:

```scala

nav(
  a(
    href := "/users",
    hxPushUrl := HxUrlUpdate.Enabled,
    hxTarget := HxTarget.css("#content"),
    "Users"
  ),
  a(
    href := "/settings",
    hxPushUrl := HxUrlUpdate.Enabled,
    hxTarget := HxTarget.css("#content"),
    "Settings"
  )
)
```

### API-Only Requests Without URL Change

Make internal API calls that don't affect the URL:

```scala

button(
  hxPost := "/api/save-draft",
  hxPushUrl := HxUrlUpdate.Disabled,
  "Save Draft"
)
```

### Redirect to a Different URL

Use an internal API endpoint but show a different user-friendly URL:

```scala

form(
  hxPost := "/api/checkout-internal",
  hxPushUrl := HxUrlUpdate("/order-confirmation"),
  hxTarget := HxTarget.css("#main"),
  button("Complete Order")
)
```

### Search with Query Parameter URL

Push a URL with query parameters to reflect the search state:

```scala

input(
  name := "query",
  hxPost := "/search",
  hxPushUrl := HxUrlUpdate("/search?query=..."),  // placeholder; dynamically set by server
  placeholder := "Search..."
)
```

## Replace vs Push

Use `hxReplaceUrl` instead of `hxPushUrl` to replace the current history entry rather than adding a new one:

```scala

// This request replaces the current history entry
form(
  hxPost := "/search",
  hxReplaceUrl := HxUrlUpdate("/search-results"),
  button("Search")
)
```

Both attributes accept the same `HxUrlUpdate` type. Use `replace` for searches, filters, and state changes that shouldn't create browser history entries. Use `push` for navigation to new pages.

## Integration with Other Attributes

Combine `HxUrlUpdate` with `hxPushUrl` and `hxReplaceUrl` to control history and URL bar updates:

```scala

form(
  hxPost := "/api/search",
  hxTarget := HxTarget.css("#results"),
  hxSwap := HxSwap.InnerHTML,
  hxReplaceUrl := HxUrlUpdate("/search?q=example"),
  input(placeholder := "Search...")
)
```

The `ToHtmxValue[HxUrlUpdate]` instance renders automatically, so `HxUrlUpdate` values work seamlessly with both attribute keys.

---

## HTMX

`zio.http.htmx` is a **typed HTMX DSL** for building safe, compile-time HTMX attribute declarations within `zio.blocks.html`. It provides immutable types representing HTMX events, swap strategies, target selectors, and request modifiers, eliminating stringly-typed misuse through rich domain types while maintaining explicit string surfaces for URLs and raw JavaScript where needed.

Core types: `HxTrigger`, `HxSwap`, `HxTarget`, `HxParams`, `HxUrlUpdate`, `HxEncoding`, `HxSync`, `HtmxAttrKey`, `ToHtmxValue`.

Here are the core patterns of typed HTMX construction:

```scala

// Compile-safe event triggers with modifiers
div(hxPost := "/search", hxTrigger := HxTrigger.input.delay(500.millis))

// Typed swap strategies with multiple modifiers
div(hxSwap := HxSwap.InnerHTML.swap(1.second).transition)

// Type-safe target selection
button(hxTarget := HxTarget.closest("form"))

// Request control through domain types
form(hxParams := HxParams.only("query", "page"))
```

## Introduction

The `htmx` module eliminates accidental stringly mistakes in HTMX attribute construction. Rather than writing raw strings like `"innerHTML swap:1s settle:500ms transition:true"`, you compose domain values through a type-safe DSL that guides you toward correct HTMX syntax at compile time.

Most attributes are narrower than plain HTML strings. `hxSwap` accepts `HxSwap`, `hxTarget` accepts `HxTarget`, and selector-only attributes accept `CssSelector`. When a string surface is genuinely necessary—for URLs, custom JavaScript filters, or dynamic values—the DSL provides explicit entry points.

## Motivation

HTMX introduces many new attributes and modifier combinations. Writing these as raw strings is error-prone:
- Typos in strategy names (`"innterHTML"`, `"delya:1s"`) silently fail at runtime
- Mixing unrelated modifiers in the same attribute (`"innerHTML queue:first threshold:0.5"`) compiles but confuses intent
- URLs and JavaScript filters have no type guidance, leading to unsafe assumptions

The typed HTMX DSL catches these mistakes at compile time:
- Strategy names are exhaustive case objects: `HxSwap.InnerHTML`, `HxSwap.AfterBegin`, etc.
- Modifier methods enforce correct grouping: repeated calls within the same modifier group (`swap()` or `settle()`) replace earlier values, while different groups (`swap`, `settle`, `transition`) can be combined
- Type-safe attributes like `hxTarget` prevent passing raw strings where structured selectors belong
- The `ToHtmxValue` type class renders the built-in value types to HTMX syntax

## Installation

Add the HTMX module to your project dependencies:

```scala
// JVM
libraryDependencies += "dev.zio" %% "zio-blocks-http-htmx" % "0.0.56"

// Scala.js
libraryDependencies += "dev.zio" %%% "zio-blocks-http-htmx" % "0.0.56"
```

On Scala 3.9 or newer, use the `zio-blocks-next-` prefix and enable experimental features:

```scala
scalaVersion := "3.9.0"

libraryDependencies += "dev.zio" %% "zio-blocks-next-http-htmx" % "0.0.56"

scalacOptions += "-experimental"
```

Supported Scala versions: Scala 3.x. The module is cross-compiled for JVM and Scala.js.

### Which htmx Version This Reference Targets

The attribute syntax in this reference targets **htmx 2.x**. The examples and browser checks pin `htmx@2.0.11`:

```html
<script src="https://cdn.jsdelivr.net/npm/htmx.org@2.0.11/dist/htmx.min.js"></script>
```

htmx 4 parses several of these attributes differently, so a value that works in htmx 2 may do nothing in htmx 4. Pin the htmx version in your page the same way.

## Overview

Each type in the module addresses a specific HTMX concern:

**Event Triggering:** `HxTrigger` declares which event fires a request (click, input, load, custom). Modifiers add timing (`HxTrigger#delay`, `HxTrigger#throttle`), use `HxTrigger#filter` for filtering, source control (`from`), and queue strategy (`queue`). `HxTriggerSet` composes multiple triggers in a comma-separated list for more complex interactions.

**Swap Strategies:** `HxSwap` selects how the response content replaces the DOM (innerHTML, outerHTML, beforeBegin, etc.). Modifiers control timing with `HxSwap#swap` and `HxSwap#settle`, animation (`transition`), scrolling (`scroll`, `show`), and focus behavior (`focusScroll`, `ignoreTitle`).

**Target Selection:** `HxTarget` selects where swap happens (current element, closest ancestor, next sibling, custom CSS selector). Variants support common patterns like `HxTarget.This`, `HxTarget.closest(selector)`, and `HxTarget.next(selector)`.

**Request Control:** `HxParams` filters which form fields are submitted (all, none, only listed, all except listed). `HxUrlUpdate` controls whether the URL bar updates after the request. `HxEncoding` and `HxSync` handle multi-step request concerns.

**Infrastructure:** `HtmxAttrKey` is the typed attribute key binding a name to a value type. `ToHtmxValue` is the type class rendering domain values to HTMX attribute strings, for the built-in value types. The set of attribute keys is closed, so a custom instance does not make a new type assignable to a key.

## How They Work Together

A typical HTMX request flow combines multiple types: trigger defines when, swap defines what, target defines where, and request control refines how. Here is the overall flow:

```
User Action
    ↓
HxTrigger (when: click, input, load, every N seconds, etc.)
    ├─ Modifiers: delay, throttle, filter, from, queue
    ↓
Request Sent
    ├─ URL: hxPost, hxGet, hxPut, hxPatch, hxDelete
    ├─ Parameters: HxParams (all, none, only, not)
    ├─ Encoding: HxEncoding (multipart override; default is URL-encoded)
    └─ Synchronization: HxSync (queue strategy, abort behavior)
    ↓
Response Received
    ↓
HxTarget (where: this, closest, find, next, previous, css selector)
    ↓
HxSwap (how: innerHTML, outerHTML, beforeBegin, etc.)
    ├─ Modifiers: swap delay, settle delay, transition, scroll, show
    └─ Focus: ignoreTitle, focusScroll
    ↓
DOM Updated & Settled
```

**Example workflow:** When a user types in a search input, fire a POST request to `/api/search` after a 500ms delay. Send only the query and page parameters. Replace the results section (the closest parent with CSS class `results`) with innerHTML strategy, wait 250ms for CSS transitions, then scroll the results into view:

```scala

input(
  placeholder := "Search...",
  hxPost := "/api/search",
  hxTrigger := HxTrigger.input.delay(500.millis),
  hxParams := HxParams.only("query", "page"),
  hxTarget := HxTarget.closest(".results"),
  hxSwap := HxSwap.InnerHTML.settle(250.millis).scroll(HxSwap.ScrollPosition.Top)
)
```

This example shows how types guide each concern: `HxTrigger.input` is compile-checked (not a typo), `HxTrigger#delay` is a method not a string, `HxParams.only()` is exhaustively typed, `HxTarget.closest()` takes a string but validates non-emptiness, and `HxSwap` chains modifiers with type safety.

## Common Patterns

The HTMX DSL supports several common interaction patterns. Here are representative examples:

### Pattern 1: Progressive Enhancement with Boost

Use `hxBoost := true` to transform regular links and form submissions into HTMX requests, maintaining server-side rendering and graceful degradation:

```scala

// Entire nav boosted—clicks are HTMX requests, but work without JS
nav(
  hxBoost := true,
  ul(
    li(a(href := "/", "Home")),
    li(a(href := "/products", "Products")),
    li(a(href := "/contact", "Contact"))
  )
)
```

### Pattern 2: Polling and Periodic Updates

Use `HxTrigger.every()` with a duration to poll an endpoint at regular intervals:

```scala

// Poll every 2 seconds for new notifications
div(
  id := "notifications",
  hxGet := "/api/notifications",
  hxTrigger := HxTrigger.every(2.seconds),
  hxSwap := HxSwap.InnerHTML
)
```

### Pattern 3: Chained Modifiers for Complex Interactions

Combine multiple trigger modifiers to refine when and how a request fires:

```scala

// Fire on input, throttle to once per second, only if value changed
input(
  hxPost := "/search",
  hxTrigger := HxTrigger.input.throttle(1.second).changed,
  hxSwap := HxSwap.InnerHTML
)
```

Modifier methods return updated `HxTrigger` instances, so you can chain: `HxTrigger.click.delay(100.millis).once` creates a single-fire click handler with a delay.

### Pattern 4: Out-of-Band Swaps

Update multiple DOM regions with a single response using `hxSwapOob` (out-of-band swap):

```scala

// Main content updates inline; status badge updates separately
div(
  hxPost := "/api/action",
  hxSwap := HxSwap.InnerHTML,
  button("Submit"),
  div(id := "status", hxSwapOob := HxSwapOob.using(HxSwap.InnerHTML), "Ready")
)
```

The response contains both the new main content and a separate element targeted by `hxSwapOob`, allowing one request to update multiple areas.

### Pattern 5: Conditional Rendering with JavaScript Filters

Use `HxTrigger#filter` with the `Js` type to add a JavaScript condition that gates the request:

```scala

input(
  hxPost := "/search",
  hxTrigger := HxTrigger.input.filter(Js("event.target.value.length > 2")),
  placeholder := "Only POST if search has 3+ characters"
)
```

The `Js` type is intentionally raw—do not build it from unsanitized user input.

## Integration Points

**With `zio.blocks.html`:** All HTMX attributes integrate seamlessly into the HTML DSL. `HtmxAttributes` is mixed into the `zio.http.htmx` package object, making attributes like `hxPost`, `hxTrigger`, and `hxSwap` directly available when you `import zio.http.htmx._`.

**With `zio.http` URL/Path types:** URL-bearing attributes accept `URL` and `Path` types from `zio.http`, which are then rendered to valid URL strings. The `ToHtmxValue` type class provides encoding for these types.

**With `zio.blocks.schema` for JSON encoding:** For domain values, use `HxVals.from(...)` and `HxHeadersValue.from(...)` to encode values via their `Schema` to JSON. This allows schema-backed types to be automatically JSON-encoded in HTMX attributes.

**With CSS selectors:** Attributes that accept selectors (hxSelect, hxSelectOob, hxDisabledElt, hxIndicator) accept `CssSelector` from `zio.blocks.html`, providing type-safe selector construction. `hxTarget` takes an `HxTarget` instead.

**With JavaScript:** Attributes that accept raw JavaScript expressions (`hxOn:*`, `hxTrigger.filter()`) accept the `Js` type, making it explicit that you're writing unescaped JavaScript code. This prevents accidental XSS while allowing intentional dynamic behavior.

**With headers:** The [`zio.http.htmx.headers`](./response-headers.md) submodule provides typed HTMX request/response headers (HX-Request, HX-Trigger, HX-Redirect, etc.), letting you inspect and build headers with the same type safety as attributes.

**In a tutorial:** [Add Interactivity to a Page with HTMX](../../guides/add-interactivity-with-htmx.md) builds a live book search with these attributes, step by step.

**Custom types:** A `ToHtmxValue[MyType]` instance does not make `MyType` usable in an attribute: each key takes a fixed value type. Convert a domain value to the key's value type (usually a `String`) before assigning it.

## Running the Examples

All code from this guide is available as runnable examples in the `zio-blocks-htmx-examples` module.

**1. Clone the repository and navigate to the project:**

Start by cloning the repository and entering the project directory:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

### Basic Usage

Demonstrates fundamental HTMX attribute construction: triggering requests (hxPost, hxGet), swapping strategies (InnerHTML, OuterHTML), and target selection (This, closest, find). Shows how the typed DSL ensures correct HTMX syntax at compile time. Here is the source code:

```scala title="zio-blocks-htmx-examples/src/main/scala/zioBlocksHtmx/BasicUsage.scala" 
package zioBlocksHtmx

/**
 * HTMX DSL — Basic Usage
 *
 * Demonstrates fundamental HTMX attribute construction: triggering requests
 * (hxPost, hxGet), swapping strategies (InnerHTML, OuterHTML), and target
 * selection (This, closest, find). Shows how the typed DSL ensures correct HTMX
 * syntax at compile time.
 *
 * Run with: sbt "zio-blocks-htmx-examples/runMain zioBlocksHtmx.BasicUsage"
 */
object BasicUsage {
  def main(args: Array[String]): Unit = {
    println("=== HTMX Basic Usage Examples ===\n")

    // Example 1: Simple click trigger with POST
    println("1. Click trigger with POST request:")
    val clickButton = button(
      hxPost    := "/api/action",
      hxTrigger := HxTrigger.click,
      "Click me"
    )
    println(s"  Rendered: $clickButton\n")

    // Example 2: Input with GET request
    println("2. Input with debounced GET request:")
    val searchInput = input(
      `type`      := "text",
      placeholder := "Search...",
      hxGet       := "/api/search",
      hxTrigger   := HxTrigger.input.delay(500.millis),
      hxTarget    := HxTarget.next("div")
    )
    println(s"  Rendered: $searchInput\n")

    // Example 3: Different swap strategies
    println("3. Swap strategies:")
    val innerHTMLDiv = div(
      id     := "content",
      hxGet  := "/api/content",
      hxSwap := HxSwap.InnerHTML,
      "Click to load inner content"
    )
    val outerHTMLDiv = div(
      id     := "card",
      hxPost := "/api/replace",
      hxSwap := HxSwap.OuterHTML,
      "This entire div will be replaced"
    )
    val appendDiv = div(
      id     := "messages",
      hxGet  := "/api/new-message",
      hxSwap := HxSwap.BeforeEnd.scroll(HxSwap.ScrollPosition.Bottom),
      "New messages will be appended"
    )
    println(s"  InnerHTML: ${innerHTMLDiv}")
    println(s"  OuterHTML: ${outerHTMLDiv}")
    println(s"  Append with scroll: ${appendDiv}\n")

    // Example 4: Target selection patterns
    println("4. Target selection:")
    val targetThis = button(
      hxPost   := "/api/toggle",
      hxTarget := HxTarget.This,
      "Toggle this button"
    )
    val targetClosest = button(
      hxPost   := "/api/validate",
      hxTarget := HxTarget.closest("form"),
      "Validate form"
    )
    val targetFind = div(
      hxGet    := "/api/update",
      hxTarget := HxTarget.find(".result"),
      "Content with .result child"
    )
    println(s"  This: $targetThis")
    println(s"  Closest: $targetClosest")
    println(s"  Find: $targetFind\n")

    // Example 5: Form submission with parameters
    println("5. Form submission with selective parameters:")
    val searchForm = form(
      hxPost    := "/api/search",
      hxTrigger := HxTrigger.submit,
      hxParams  := HxParams.only("query", "page"),
      hxSwap    := HxSwap.InnerHTML,
      input(`type`  := "text", name   := "query", placeholder := "Query"),
      input(`type`  := "hidden", name := "page", value        := "1"),
      input(`type`  := "hidden", name := "unused", value      := "ignored"),
      button(`type` := "submit", "Search")
    )
    println(s"  Form with selective params: $searchForm\n")

    // Example 6: Load event with once modifier
    println("6. Load event with once modifier:")
    val initialLoadDiv = div(
      id        := "initial",
      hxGet     := "/api/bootstrap-data",
      hxTrigger := HxTrigger.load.once,
      hxSwap    := HxSwap.InnerHTML,
      "Loading initial data..."
    )
    println(s"  Load once: $initialLoadDiv\n")

    // Example 7: Change event on select
    println("7. Change event on select element:")
    val selectDropdown = select(
      name      := "category",
      hxPost    := "/api/category-changed",
      hxTrigger := HxTrigger.change,
      hxTarget  := HxTarget.next("div"),
      option(value := "all", "All Categories"),
      option(value := "news", "News"),
      option(value := "updates", "Updates")
    )
    println(s"  Select with change: $selectDropdown\n")

    println("✓ Basic usage examples complete")
  }
}
```

([source](https://github.com/zio/zio-blocks/blob/main/zio-blocks-htmx-examples/src/main/scala/zioBlocksHtmx/BasicUsage.scala))

Run this example with the following command:

```bash
cd zio-blocks-htmx-examples && sbt "runMain zioBlocksHtmx.BasicUsage"
```

### Advanced Patterns

Demonstrates complex HTMX interactions: combining multiple triggers, chaining modifiers, controlling request queuing, animation with transitions, and JavaScript-based filtering. Shows how modifiers compose to create sophisticated client-side behaviors. Here is the source code:

```scala title="zio-blocks-htmx-examples/src/main/scala/zioBlocksHtmx/AdvancedPatterns.scala" 
package zioBlocksHtmx

/**
 * HTMX DSL — Advanced Patterns
 *
 * Demonstrates complex HTMX interactions: combining multiple triggers, chaining
 * modifiers, controlling request queuing, animation with transitions, and
 * JavaScript-based filtering. Shows how modifiers compose to create
 * sophisticated client-side behaviors.
 *
 * Run with: sbt "zio-blocks-htmx-examples/runMain
 * zioBlocksHtmx.AdvancedPatterns"
 */
object AdvancedPatterns {
  def main(args: Array[String]): Unit = {
    println("=== HTMX Advanced Patterns ===\n")

    // Example 1: Debounced search with state modifier
    println("1. Debounced search with value-changed detection:")
    val debouncedSearch = input(
      `type`      := "text",
      name        := "query",
      placeholder := "Type to search...",
      hxPost      := "/api/search",
      hxTrigger   := HxTrigger.input.delay(500.millis).changed,
      hxParams    := HxParams.only("query"),
      hxTarget    := HxTarget.next("div"),
      hxSwap      := HxSwap.InnerHTML
    )
    println(s"  Trigger: ${HxTrigger.input.delay(500.millis).changed.render}")
    println(s"  Element: $debouncedSearch\n")

    // Example 2: Rate-limited input with throttle
    println("2. Rate-limited input events (at most 1 request per second):")
    val throttledStatus = input(
      id          := "status",
      `type`      := "text",
      hxPost      := "/api/status",
      hxTrigger   := HxTrigger.input.throttle(1.second),
      hxSwap      := HxSwap.OuterHTML,
      placeholder := "Type slowly..."
    )
    println(s"  Trigger: ${HxTrigger.input.throttle(1.second).render}")
    println(s"  Element: $throttledStatus\n")

    // Example 3: Multiple triggers with polling and user action
    println("3. Dual triggers: user click OR automatic polling:")
    val autoRefresh = div(
      id        := "data",
      hxGet     := "/api/data",
      hxTrigger := HxTrigger(
        HxTrigger.click,
        HxTrigger.every(30.seconds)
      ),
      hxSwap := HxSwap.InnerHTML.transition.settle(300.millis),
      "Data refreshes on click or every 30s"
    )
    println(s"  Triggers: ${HxTrigger(HxTrigger.click, HxTrigger.every(30.seconds)).render}")
    println(s"  Element: $autoRefresh\n")

    // Example 4: Request queuing strategy
    println("4. Queue strategy: keep only most recent request:")
    val queuedInput = input(
      `type`      := "text",
      placeholder := "Fast typing...",
      hxPost      := "/api/validate",
      hxTrigger   := HxTrigger.input.delay(300.millis).queue(HxTrigger.QueueStrategy.Last),
      hxTarget    := HxTarget.next("span"),
      hxSwap      := HxSwap.InnerHTML
    )
    println(s"  Trigger: ${HxTrigger.input.delay(300.millis).queue(HxTrigger.QueueStrategy.Last).render}")
    println(s"  Element: $queuedInput\n")

    // Example 5: Animation with timing modifiers
    println("5. CSS transition with settle delay:")
    val animatedSwap = button(
      hxPost    := "/api/action",
      hxTrigger := HxTrigger.click,
      hxSwap    := HxSwap.InnerHTML.transition.settle(400.millis),
      hxTarget  := HxTarget.This,
      "Click for animated response"
    )
    println(s"  Swap: ${HxSwap.InnerHTML.transition.settle(400.millis).render}")
    println(s"  Element: $animatedSwap\n")

    // Example 6: Event delegation with target modifier
    println("6. Event delegation: parent listens to child clicks:")
    val eventDelegation = ul(
      id        := "items",
      hxPost    := "/api/item-clicked",
      hxTrigger := HxTrigger.click.target("li"),
      hxSwap    := HxSwap.InnerHTML,
      li("Item 1"),
      li("Item 2"),
      li("Item 3")
    )
    println(s"""  Trigger: ${HxTrigger.click.target("li").render}""")
    println(s"  Element: $eventDelegation\n")

    // Example 7: JavaScript filtering
    println("7. JavaScript filter: only trigger if condition met:")
    val filteredTrigger = input(
      `type`      := "text",
      placeholder := "Min 3 characters...",
      hxPost      := "/api/search",
      hxTrigger   := HxTrigger.input.filter(Js("event.target.value.length > 2")),
      hxTarget    := HxTarget.next("div"),
      hxSwap      := HxSwap.InnerHTML
    )
    println(s"""  Trigger: ${HxTrigger.input.filter(Js("event.target.value.length > 2")).render}""")
    println(s"  Element: $filteredTrigger\n")

    // Example 8: Source control with from modifier
    println("8. Source control: button listens to input events:")
    val sourceControl = div(
      input(
        id          := "query-input",
        `type`      := "text",
        placeholder := "Type here..."
      ),
      button(
        id        := "search-btn",
        hxPost    := "/api/search",
        hxTrigger := HxTrigger.click.from("#query-input"),
        hxTarget  := HxTarget.next("div"),
        "Search"
      )
    )
    println(s"""  Button trigger: ${HxTrigger.click.from("#query-input").render}""")
    println(s"  Element: $sourceControl\n")

    // Example 9: Intersection observer with threshold
    println("9. Lazy loading with intersection observer:")
    val lazyLoad = img(
      src       := "/placeholder.jpg",
      hxGet     := "/api/lazy-image",
      hxTrigger := HxTrigger.intersect.threshold(0.5),
      hxSwap    := HxSwap.OuterHTML,
      alt       := "Lazy loaded image"
    )
    println(s"  Trigger: ${HxTrigger.intersect.threshold(0.5).render}")
    println(s"  Element: $lazyLoad\n")

    // Example 10: Complex modifier chain
    println("10. Complex modifier chain - multiple modifiers:")
    val complexChain = input(
      `type`      := "text",
      name        := "search",
      placeholder := "Advanced search",
      hxPost      := "/api/search",
      hxTrigger   := HxTrigger.input
        .delay(500.millis)
        .throttle(1.second)
        .changed
        .filter(Js("event.target.value.trim().length > 0")),
      hxParams := HxParams.only("search"),
      hxTarget := HxTarget.closest(".search-results"),
      hxSwap   := HxSwap.InnerHTML.transition.settle(250.millis)
    )
    println(
      s"""  Complex trigger: ${HxTrigger.input
          .delay(500.millis)
          .throttle(1.second)
          .changed
          .filter(Js("event.target.value.trim().length > 0"))
          .render}"""
    )
    println(s"  Swap: ${HxSwap.InnerHTML.transition.settle(250.millis).render}")
    println(s"  Element: $complexChain\n")

    // Example 11: Scroll positioning after swap
    println("11. Scroll to bottom after appending messages:")
    val messageList = div(
      id        := "messages",
      hxGet     := "/api/messages",
      hxTrigger := HxTrigger.every(2.seconds),
      hxSwap    := HxSwap.BeforeEnd.scroll(HxSwap.ScrollPosition.Bottom).show(HxSwap.ShowPosition.Bottom),
      "Messages will be appended and scrolled into view"
    )
    println(s"  Swap: ${HxSwap.BeforeEnd.scroll(HxSwap.ScrollPosition.Bottom).show(HxSwap.ShowPosition.Bottom).render}")
    println(s"  Element: $messageList\n")

    println("✓ Advanced pattern examples complete")
  }
}
```

([source](https://github.com/zio/zio-blocks/blob/main/zio-blocks-htmx-examples/src/main/scala/zioBlocksHtmx/AdvancedPatterns.scala))

Run this example with the following command:

```bash
cd zio-blocks-htmx-examples && sbt "runMain zioBlocksHtmx.AdvancedPatterns"
```

### Complete Example

A realistic e-commerce search and filtering interface combining multiple HTMX attributes: debounced search input, live category filtering, paginated results, out-of-band notifications, and dynamic UI updates. Demonstrates how types compose to create a type-safe, interactive UI. Here is the source code:

```scala title="zio-blocks-htmx-examples/src/main/scala/zioBlocksHtmx/CompleteExample.scala" 
package zioBlocksHtmx

/**
 * HTMX DSL — Complete Realistic Example
 *
 * A real-world e-commerce search and filtering interface combining multiple
 * HTMX attributes: debounced search input, live category filtering, paginated
 * results, out-of-band notifications, and dynamic UI updates. Demonstrates how
 * types compose to create a type-safe, interactive UI.
 *
 * Run with: sbt "zio-blocks-htmx-examples/runMain
 * zioBlocksHtmx.CompleteExample"
 */
object CompleteExample {
  def main(args: Array[String]): Unit = {
    println("=== HTMX Complete Realistic Example ===\n")

    println("Building a type-safe e-commerce search interface...\n")

    // Part 1: Search input with debounce and parameter filtering
    val searchInput = input(
      `type`      := "text",
      name        := "query",
      id          := "search-query",
      placeholder := "Search products...",
      hxPost      := "/api/search",
      hxTrigger   := HxTrigger.input
        .delay(500.millis)
        .changed,
      hxParams := HxParams.only("query", "category", "page"),
      hxTarget := HxTarget.find(".results-container"),
      hxSwap   := HxSwap.InnerHTML.transition.settle(250.millis)
    )

    println("1. Search Input:")
    println(s"   Placeholder: Search products")
    println(s"   Trigger: input with 500ms delay + changed")
    println(s"   Params: query, category, page (others excluded)")
    println(s"   Swap: InnerHTML with transition and 250ms settle")
    println(s"   Rendered: $searchInput\n")

    // Part 2: Category filter with instant response
    val categoryFilter = select(
      name      := "category",
      id        := "category-filter",
      hxPost    := "/api/search",
      hxTrigger := HxTrigger.change,
      hxTarget  := HxTarget.find(".results-container"),
      hxSwap    := HxSwap.InnerHTML.transition,
      option(value := "all", selected, "All Categories"),
      option(value := "electronics", "Electronics"),
      option(value := "books", "Books"),
      option(value := "clothing", "Clothing")
    )

    println("2. Category Filter:")
    println(s"   Trigger: change event (no delay)")
    println(s"   Target: .results-container")
    println(s"   Rendered: $categoryFilter\n")

    // Part 3: Results container with dynamic content
    val resultsContainer = div(
      id        := "results",
      `class`   := "results-container",
      hxGet     := "/api/search",
      hxTrigger := HxTrigger.load,
      hxTarget  := HxTarget.This,
      hxSwap    := HxSwap.InnerHTML,
      div("Loading results...")
    )

    println("3. Results Container:")
    println(s"   Trigger: load (fetches initial results)")
    println(s"   Swap: InnerHTML")
    println(s"   Rendered: $resultsContainer\n")

    // Part 4: Pagination buttons
    val prevButton = button(
      id        := "btn-prev",
      hxPost    := "/api/search",
      hxTrigger := HxTrigger.click,
      hxParams  := HxParams.only("query", "category", "page"),
      hxTarget  := HxTarget.find(".results-container"),
      hxSwap    := HxSwap.InnerHTML.scroll(HxSwap.ScrollPosition.Top),
      "← Previous"
    )

    val nextButton = button(
      id        := "btn-next",
      hxPost    := "/api/search",
      hxTrigger := HxTrigger.click,
      hxParams  := HxParams.only("query", "category", "page"),
      hxTarget  := HxTarget.find(".results-container"),
      hxSwap    := HxSwap.InnerHTML.scroll(HxSwap.ScrollPosition.Top),
      "Next →"
    )

    println("4. Pagination Buttons:")
    println(s"   Behavior: POST request, replace results, scroll to top")
    println(s"   Prev: $prevButton")
    println(s"   Next: $nextButton\n")

    // Part 5: Status badge with out-of-band updates
    val statusBadge = div(
      id        := "search-status",
      hxSwapOob := HxSwapOob.using(HxSwap.InnerHTML),
      "Ready"
    )

    println("5. Status Badge (Out-of-Band):")
    println(s"   Updates separately from main results")
    println(s"   Server response can include status updates")
    println(s"   Rendered: $statusBadge\n")

    // Part 6: Complete search form integrating all parts
    val wholeForm = div(
      id := "search-interface",
      h2("Product Search"),
      div(
        `class` := "search-controls",
        div(
          label(`for` := "search-query", "Search:"),
          searchInput
        ),
        div(
          label(`for` := "category-filter", "Category:"),
          categoryFilter
        )
      ),
      resultsContainer,
      div(
        `class` := "pagination-controls",
        prevButton,
        span(id := "page-info", "Page 1"),
        nextButton
      ),
      statusBadge
    )

    println("6. Complete Search Form:")
    println(s"   Integrates search input, category filter, results, pagination")
    println(s"   Form structure: $wholeForm\n")

    // Part 7: Advanced example - filtering based on price range
    val priceRangeInput = input(
      `type`    := "range",
      name      := "max-price",
      id        := "price-slider",
      min       := "0",
      max       := "1000",
      value     := "1000",
      hxPost    := "/api/search",
      hxTrigger := HxTrigger.change.throttle(500.millis),
      hxParams  := HxParams.only("query", "category", "max-price"),
      hxTarget  := HxTarget.find(".results-container"),
      hxSwap    := HxSwap.InnerHTML.transition
    )

    println("7. Advanced - Price Range Filter:")
    println(s"   Trigger: change with 500ms throttle")
    println(s"   Prevents excessive requests while dragging slider")
    println(s"   Rendered: $priceRangeInput\n")

    // Part 8: Bonus - polling for real-time updates
    val liveUpdatesDiv = div(
      id        := "live-updates",
      hxGet     := "/api/trending",
      hxTrigger := HxTrigger.every(10.seconds),
      hxSwap    := HxSwap.InnerHTML,
      "Trending products..."
    )

    println("8. Bonus - Live Updates Panel:")
    println(s"   Polls /api/trending every 10 seconds")
    println(s"   Keeps trending products fresh without user action")
    println(s"   Rendered: $liveUpdatesDiv\n")

    println("=== Type Safety in Action ===")
    println("All HTMX attributes are compile-time checked:")
    println("✓ HxTrigger validates event names and modifiers")
    println("✓ HxSwap ensures correct strategy and modifier combinations")
    println("✓ HxParams prevents typos in parameter names")
    println("✓ HxTarget validates selector syntax")
    println("✓ No raw strings = no HTMX syntax errors at runtime")
    println("\n✓ Complete realistic example built successfully")
  }
}
```

([source](https://github.com/zio/zio-blocks/blob/main/zio-blocks-htmx-examples/src/main/scala/zioBlocksHtmx/CompleteExample.scala))

Run this example with the following command:

```bash
cd zio-blocks-htmx-examples && sbt "runMain zioBlocksHtmx.CompleteExample"
```

**3. Or compile all examples at once:**

To compile all example sources without running them, use:

```bash
cd zio-blocks-htmx-examples && sbt "compile"
```

---

## Request and Response Headers

`zio.http.htmx.headers` provides typed HTTP headers for the request/response side of an HTMX exchange: the headers HTMX sends with every request, and the headers a handler sets to steer the client afterward. Each header follows the same `Header` / `Header.Typed` pattern that `zio.http` applies everywhere else, so reading one from a `Request` or `Response` returns a parsed domain value instead of a raw string, and several response headers reuse the same `HxSwap`, `HxTarget`, and `HxUrlUpdate` types that back the attribute DSL, keeping the two sides of an interaction consistent.

Here are the core patterns for reading and writing HTMX headers:

```scala

// Reading a request header HTMX sent
def handle(request: Request): Boolean =
  request.header(HxRequest).exists(_.enabled)

// Setting response headers that steer the client afterward
val response = Response.ok.addHeaders(
  Headers(HxRefresh.name -> "true", HxTriggerHeader.name -> "orderPlaced")
)
```

## Overview

The module splits into two directions and reuses supporting types from the attribute DSL for structured values:

| Direction | Type                     | Wire name                   | Shape                                             |
| --------- | ------------------------ | ---------------------------- | -------------------------------------------------- |
| Request   | `HxRequest`               | `hx-request`                 | `Boolean`, always `true` on an HTMX request         |
| Request   | `HxBoosted`                | `hx-boosted`                 | `Boolean`, set when the request came from `hxBoost` |
| Request   | `HxHistoryRestoreRequest`  | `hx-history-restore-request` | `Boolean`                                           |
| Request   | `HxCurrentUrl`             | `hx-current-url`             | non-empty `String` or `URL`                         |
| Request   | `HxTargetId`               | `hx-target`                  | non-empty `String` (target element id)              |
| Request   | `HxTriggerId`              | `hx-trigger`                 | non-empty `String` (triggering element id)          |
| Request   | `HxTriggerName`            | `hx-trigger-name`            | non-empty `String`                                  |
| Request   | `HxPrompt`                 | `hx-prompt`                  | `String`, trimmed                                   |
| Response  | `HxRedirect`               | `hx-redirect`                | non-empty `String` or `URL`                         |
| Response  | `HxRefresh`                | `hx-refresh`                 | `Boolean`                                           |
| Response  | `HxPushUrl`                | `hx-push-url`                | `HxUrlUpdate`                                       |
| Response  | `HxReplaceUrl`             | `hx-replace-url`             | `HxUrlUpdate`                                       |
| Response  | `HxReswap`                 | `hx-reswap`                  | `HxSwap`                                            |
| Response  | `HxRetarget`               | `hx-retarget`                | `CssSelector`                                       |
| Response  | `HxReselect`               | `hx-reselect`                | `CssSelector`                                       |
| Response  | `HxTriggerHeader`          | `hx-trigger`                 | `HxEventPayload`                                    |
| Response  | `HxTriggerAfterSettle`     | `hx-trigger-after-settle`    | `HxEventPayload`                                    |
| Response  | `HxTriggerAfterSwap`       | `hx-trigger-after-swap`      | `HxEventPayload`                                    |
| Response  | `HxLocation`               | `hx-location`                | `Json.Object`, built from path/target/swap          |

`HxTriggerId` (request) and `HxTriggerHeader` (response) share the wire name `hx-trigger` but read in opposite directions—one reports which element fired the request, the other tells the client which events to dispatch after the response arrives.

## Reading Request Headers

These headers describe the HTMX request itself: whether it came from HTMX at all, which element and value triggered it, and what the browser's current URL was. All of them parse from plain strings with no structured payload.

### Presence Flags

`HxRequest`, `HxBoosted`, and `HxHistoryRestoreRequest` are boolean headers—present with value `"true"` or `"false"`, parsed case-insensitively. Checking `HxRequest` tells you whether a request came from HTMX at all, which is the first step when a route answers with a fragment for htmx and a full page for the browser. It is not enough on its own, because htmx also sends `HX-Request: true` in two cases that need the full page:

- a link inside an `hx-boost` element sends `HX-Boosted: true`, and htmx swaps the response into `<body>` and takes the page title from it;
- a Back-button request that misses htmx's history cache sends `HX-History-Restore-Request: true`, and htmx swaps the response into `<body>` as well.

```scala

val boosted = Request.get(URL.root).addHeaders(Headers(HxRequest.name -> "true", HxBoosted.name -> "true"))
```

Reading each flag back through `Request#header` parses the stored string into the typed value:

```scala
boosted.header(HxRequest)
// res1: Option[HxRequest] = Some(HxRequest(true))
boosted.header(HxBoosted)
// res2: Option[HxBoosted] = Some(HxBoosted(true))
```

A route that answers a fragment only when `HX-Request` is `true` must also check the other two flags:

```scala
def wantsFragment(request: Request): Boolean =
  request.header(HxRequest).exists(_.enabled) &&
    !request.header(HxBoosted).exists(_.enabled) &&
    !request.header(HxHistoryRestoreRequest).exists(_.enabled)
```

The [Add Interactivity to a Page with HTMX](../../guides/add-interactivity-with-htmx.md) tutorial checks `HxRequest` and `HxHistoryRestoreRequest` only, which is enough for a page without `hxBoost`. [How to Return HTMX Fragments, Full Pages and Response Headers](../../guides/return-htmx-fragments-and-response-headers.md) shows the full check in a browser.

### Identifiers and Text Values

`HxTargetId`, `HxTriggerId`, and `HxTriggerName` carry the id or name of the element involved in the request; `HxCurrentUrl` carries the browser's current URL. All four trim their value and reject blank input except `HxPrompt`, which passes through any trimmed string including an empty one:

```scala

val request = Request
  .get(URL.root)
  .addHeaders(Headers(HxTriggerName.name -> "search", HxCurrentUrl.name -> "https://zio.dev/blocks"))
```

Reading them back gives the trimmed values as typed headers:

```scala
request.header(HxTriggerName)
// res4: Option[HxTriggerName] = Some(HxTriggerName("search"))
request.header(HxCurrentUrl)
// res5: Option[HxCurrentUrl] = Some(HxCurrentUrl("https://zio.dev/blocks"))
```

`HxCurrentUrl` also accepts a `zio.http.URL` directly at construction time, encoding it to a string: `HxCurrentUrl(URL.parse("https://zio.dev/htmx").toOption.get)`.

## Setting Response Headers

Response headers tell HTMX what to do with a response: redirect, refresh, change what gets swapped or where, fire client-side events, or update the URL bar. Redirect, location and refresh act before any swap happens. The after-swap and after-settle triggers fire from the swap. Several reuse types already documented for the attribute DSL, so the same modifiers and constructors apply on both sides of the exchange.

### Redirecting and Refreshing

`HxRedirect` sends the browser to a new URL with a full page load, and `HxRefresh` forces a full page reload of the current URL:

```scala

val redirect = Response.ok.addHeaders(Headers(HxRedirect.name -> "/login", HxRefresh.name -> "true"))
```

Reading them back through `Response#header` gives the parsed values:

```scala
redirect.header(HxRedirect)
// res7: Option[HxRedirect] = Some(HxRedirect("/login"))
redirect.header(HxRefresh)
// res8: Option[HxRefresh] = Some(HxRefresh(true))
```

`HxRedirect` also has a `URL` constructor—`HxRedirect(URL.parse("https://zio.dev/login").toOption.get)`—that encodes the URL before storing it.

### Controlling the URL Bar

`HxPushUrl` and `HxReplaceUrl` decide whether the browser's URL bar updates after a swap, using the same `HxUrlUpdate` type as the `hx-push-url` attribute: `HxUrlUpdate.Enabled` or `HxUrlUpdate.Disabled` for a plain boolean, or a `String`/`Path`/`URL` to push a specific address:

```scala

HxPushUrl(HxUrlUpdate.Enabled)
HxPushUrl(HxUrlUpdate(Path("/orders/42")))
```

### Overriding the Swap Strategy

`HxReswap` lets a response override the `hx-swap` strategy the requesting element declared, using the identical `HxSwap` DSL—including its timing and animation modifiers—documented in [HxSwap](./hx-swap.md):

```scala

val header = HxReswap(HxSwap.InnerHTML.swap(1.second).settle(250.millis))
```

Rendering reproduces the raw `hx-swap` syntax, and parsing that string round-trips it back to the same header:

```scala
HxReswap.render(header)
// res11: String = "innerHTML swap:1s settle:250ms"
HxReswap.parse(HxReswap.render(header)) == Right(header)
// res12: Boolean = true
```

### Retargeting and Reselecting

`HxRetarget` overrides where the response swaps in, and `HxReselect` overrides which part of the response fragment the client applies—both take a `CssSelector` from `zio.blocks.html`, the same type `hxTarget` and `hxSelect` accept:

```scala

HxRetarget(CssSelector.id("result"))
HxReselect(CssSelector.raw(".items > li"))
```

### Triggering Client-Side Events

`HxTriggerHeader`, `HxTriggerAfterSettle`, and `HxTriggerAfterSwap` fire HTMX events on the client—before, after settle, and after swap respectively. Each carries an `HxEventPayload`: either a plain event name or a JSON value forwarded to listeners as `event.detail`:

```scala

val plain = HxTriggerAfterSwap(HxEventPayload.Event("orderPlaced"))
val withDetail = HxTriggerAfterSwap(HxEventPayload.JsonValue(Json.Object("orderId" -> Json.Number(42))))
```

Rendering shows the two payload shapes side by side—a bare event name and a JSON object:

```scala
HxTriggerAfterSwap.render(plain)
// res15: String = "orderPlaced"
HxTriggerAfterSwap.render(withDetail)
// res16: String = "{\"orderId\":42}"
```

`HxEventPayload.parse` decides which shape it saw by inspecting the value: a leading `{` or `[` parses as JSON, anything else is treated as a plain event name, and a blank value is rejected.

:::note[Why `HxTriggerHeader`, not `HxTrigger`]
The response header is named `HxTriggerHeader` rather than `HxTrigger` to avoid colliding with the attribute-DSL type documented in [HxTrigger](./hx-trigger.md)—`zio.http.htmx.HxTrigger` declares which client event fires a request, while `zio.http.htmx.headers.HxTriggerHeader` tells the client which events to dispatch after a response. Both share the wire name `hx-trigger` but never appear in the same import.
:::

### Redirecting Without a Full Navigation

`HxLocation` triggers a client-side navigation to a new path without a full page load, optionally specifying where to swap the result and how—reusing `HxTarget` and `HxSwap` from the attribute DSL:

```scala

val location = HxLocation("/next", target = Some(HxTarget.css("#main")), swap = Some(HxSwap.AfterEnd))
```

htmx finds the `target` of an `HX-Location` header with `document.querySelector`, so give it a CSS selector such as `HxTarget.css("#main")`. The relative forms, `HxTarget.closest`, `find`, `next`, `previous` and `this`, cannot be resolved there: htmx fires `htmx:targetError` and does nothing, and the path is never requested. `HxLocation` still accepts them, which is a known bug in 0.0.56.

Rendering it produces the JSON object HTMX expects on the wire:

```scala
HxLocation.render(location)
// res18: String = "{\"path\":\"/next\",\"target\":\"#main\",\"swap\":\"afterend\"}"
```

`HxLocation` also has overloads accepting a `zio.http.Path` or `zio.http.URL` for the first argument, encoding it the same way before building the JSON object. Parsing rejects anything that isn't a JSON object, since the wire format is always `{"path": ..., "target": ..., "swap": ...}` with `target` and `swap` omitted when absent.

## Reading and Writing Through Request and Response

Every header in this module is a `Header.Typed[H]`, so it works with the same `Request#header` / `Response#header` / `addHeader` / `addHeaders` methods `zio.http` provides everywhere else—there is no HTMX-specific accessor. Combining several response headers in one call is the common case, for example after processing a form submission:

```scala

val afterSubmit = Response.ok
  .addHeader(HxRedirect(URL.parse("/orders").toOption.get))
  .addHeader(HxTriggerHeader(HxEventPayload.Event("orderPlaced")))
```

Choose the trigger header with care when the same response also carries `HX-Redirect`, `HX-Location` or `HX-Refresh`. htmx 2 handles those three before it swaps anything: it navigates or reloads and stops, so the response never reaches the swap. `HX-Trigger` still fires, but `HX-Trigger-After-Swap` and `HX-Trigger-After-Settle` never do in that case. Use `HxTriggerAfterSettle` only with a response that is swapped into the page.

Because parsing failures return `Left` rather than throwing, `Request#header`/`Response#header` return `None` for a header that is present but malformed, exactly as they do for any other typed header in `zio.http`—see [Header](../http-model/headers.md) for the full read/write surface these types plug into.

## Integration Points

**With the attribute DSL:** `HxReswap`, `HxLocation`, `HxRetarget`, and `HxReselect` accept the exact same `HxSwap`, `HxTarget`, and `CssSelector` values that `hxSwap`, `hxTarget`, and `hxSelect` attributes accept, so a value built for one side works unchanged on the other.

**With `zio.blocks.schema.json`:** `HxEventPayload.JsonValue` wraps a `zio.blocks.schema.json.Json` value, letting event payloads carry structured data without a separate serialization step.

**In a tutorial:** [Add Interactivity to a Page with HTMX](../../guides/add-interactivity-with-htmx.md) reads `HxRequest` and `HxHistoryRestoreRequest` to answer one URL with a full page or a fragment.

**In a how-to guide:** [How to Return HTMX Fragments, Full Pages and Response Headers](../../guides/return-htmx-fragments-and-response-headers.md) shows one URL answering with a full page or a fragment, out-of-band swaps, `HX-Trigger`, `HX-Location`, `HX-Redirect` and `HX-Refresh` after a form post, `HX-Retarget` and `HX-Reswap` for validation errors, and `HX-Replace-Url`.

**With `zio.http`:** every type here is a `Header`/`Header.Typed[H]`, the same type class the built-in HTTP headers use, so `Headers`, `Request`, and `Response` treat HTMX headers no differently than `Content-Length` or `Cache-Control`.

---

## Header

`Header` is the typed model of a single HTTP header field. Each of the 75 built-in headers is a case class or sealed ADT paired with a companion that knows the header's wire name and how to parse and render it. `Header.Codec[A]` is the type class carrying that knowledge, and `Headers` uses it to decode raw strings on demand and cache the result. The lead-in to every typed read:

```scala
trait Header {
  def headerName: String
  def renderedValue: String
}

object Header {
  trait Codec[A] {
    def name: String
    def parse(value: String): Either[String, A]
    def render(value: A): String
  }

  trait Typed[H <: Header] extends Codec[H]

  case class Custom(headerName: String, rawValue: String) extends Header
}
```

## Motivation

An HTTP message is a bag of strings, and treating it that way pushes the same three mistakes into every handler. `headers.rawGet("content-length").map(_.toInt)` throws on a malformed value. `headers.rawGet("Content-Length")` works only because someone remembered the casing rules. `cache-control: max-age=600, no-store` has to be split, trimmed, and matched by hand at each call site.

A typed header replaces all three with one lookup. `Header.ContentLength` knows its wire name is `content-length`, that the value is a `Long`, and that a non-numeric value is a parse failure rather than an exception. Reading it is `headers.get(Header.ContentLength)`, and the result is an `Option[ContentLength]` with a `length: Long` inside.

Parsing stays lazy because most handlers read two or three headers out of fifteen. `Headers` keeps the raw strings and only parses an entry when a typed read asks for it, caching the parsed value per entry so a header read twice is parsed once.

## Quick Showcase

Setting a typed header renders it; reading it back parses it:

```scala

val headers = Headers.empty
  .add(Header.ContentLength(1024))
  .add(Header.CacheControl.MaxAge(600))
  .add("x-request-id", "abc-123")
```

Rendering happened at `Headers#add`, so the collection holds wire-format strings:

```scala
headers.toList
// res0: List[Tuple2[String, String]] = List(
//   ("content-length", "1024"),
//   ("cache-control", "max-age=600"),
//   ("x-request-id", "abc-123")
// )
```

Typed reads give back the domain value, and untyped reads give back the string:

```scala
headers.get(Header.ContentLength)
// res1: Option[ContentLength] = Some(ContentLength(1024L))
headers.get(Header.CacheControl)
// res2: Option[CacheControl] = Some(MaxAge(600L))
headers.rawGet("x-request-id")
// res3: Option[String] = Some("abc-123")
```

Lookups are case-insensitive in both directions, because names are lowercased on the way in:

```scala
headers.rawGet("X-Request-ID")
// res4: Option[String] = Some("abc-123")
headers.has("Content-Length")
// res5: Boolean = true
```

## The Codec Type Class

`Header.Codec[A]` is what every typed read and write goes through. It pairs a wire name with a parse and a render:

```scala
trait Codec[A] {
  def name: String
  def parse(value: String): Either[String, A]
  def render(value: A): String
}
```

`Header.Typed[H <: Header]` narrows that to codecs whose domain value is itself a `Header`, which is what all 75 built-ins are. Each header's companion object *is* its codec — `Header.ContentLength` refers to the companion when used as a codec and to the case class when used as a type:

```scala

```

The codec's name is the canonical lowercase wire name, and parsing reports a message rather than throwing:

```scala
Header.ContentLength.name
// res7: String = "content-length"
Header.ContentLength.parse("1024")
// res8: Either[String, ContentLength] = Right(ContentLength(1024L))
Header.ContentLength.parse("not-a-number")
// res9: Either[String, ContentLength] = Left(
//   "Invalid content-length: not-a-number"
// )
```

Rendering is the inverse, and every built-in round-trips:

```scala
Header.ContentLength.render(Header.ContentLength(1024))
// res10: String = "1024"
```

Because `Header` itself carries `headerName` and `renderedValue`, an instance knows how to write itself without the codec being named again:

```scala
Header.CacheControl.MaxAge(600).headerName
// res11: String = "cache-control"
Header.CacheControl.MaxAge(600).renderedValue
// res12: String = "max-age=600"
```

## Reading Headers

Six methods read from a `Headers` collection: three typed, three raw. Which you want depends on whether you need the domain value or the exact bytes.

### `Headers#get` — first parseable match

`Headers#get` scans for the first entry whose name matches the codec and returns the parsed value:

```scala
final class Headers {
  def get[A](headerCodec: Header.Codec[A]): Option[A]
}
```

A present, well-formed header parses; an absent one is `None`:

```scala

val headers = Headers("content-length" -> "2048", "accept" -> "application/json")
```

The typed result carries the domain value, not the string:

```scala
headers.get(Header.ContentLength)
// res14: Option[ContentLength] = Some(ContentLength(2048L))
headers.get(Header.ETag)
// res15: Option[ETag] = None
```

:::warning[Unparseable headers are skipped, not reported]
`Headers#get` treats a parse failure exactly like a name mismatch: it discards the error and keeps scanning. A malformed header therefore reads as `None` — or as the *next* entry with the same name that does parse. There is no method that surfaces the parse error from a collection read.
:::

That behaviour is worth seeing, because it is the one place a typed read can mislead you:

```scala

val malformed = Headers("content-length" -> "huge")
val shadowing = Headers("content-length" -> "huge", "content-length" -> "512")
```

A single bad value is indistinguishable from an absent header, and a bad value followed by a good one silently yields the good one:

```scala
malformed.get(Header.ContentLength)
// res17: Option[ContentLength] = None
shadowing.get(Header.ContentLength)
// res18: Option[ContentLength] = Some(ContentLength(512L))
```

To tell "missing" from "malformed", read the raw value and parse it explicitly:

```scala
malformed.rawGet("content-length").map(Header.ContentLength.parse)
// res19: Option[Either[String, ContentLength]] = Some(
//   Left("Invalid content-length: huge")
// )
```

### `Headers#getAll` — every parseable match

`Headers#getAll` returns all matching entries in header order, skipping the ones that fail to parse:

```scala
final class Headers {
  def getAll[A](headerCodec: Header.Codec[A]): Chunk[A]
}
```

Multi-value headers are the normal case for `accept-encoding`, `via`, and `set-cookie`:

```scala

val multi = Headers(
  "content-length" -> "100",
  "content-length" -> "oops",
  "content-length" -> "300"
)
```

Two entries parse and the middle one is dropped, so the typed result is shorter than the raw one:

```scala
multi.getAll(Header.ContentLength)
// res21: Chunk[ContentLength] = IndexedSeq(
//   ContentLength(100L),
//   ContentLength(300L)
// )
multi.rawGetAll("content-length").length
// res22: Int = 3
```

How many entries survive depends on how strict the individual codec is, and they vary. `Header.ContentLength` rejects anything non-numeric, while some codecs accept almost any input:

```scala
Header.AcceptEncoding.parse("gzip")
// res23: Either[String, AcceptEncoding] = Right(GZip(None))
Header.AcceptEncoding.parse("!!!")
// res24: Either[String, AcceptEncoding] = Right(GZip(None))
```

:::warning[`AcceptEncoding` falls back to `GZip` on unknown values]
`Header.AcceptEncoding` matches a fixed set of encoding names and returns `GZip` for anything it does not recognize, rather than reporting a parse failure. A request with `accept-encoding: bogus` — or a typo like `identityy` — reads as if the client asked for gzip. `Header.AcceptEncoding.parse` rejects only values with no non-empty comma-separated part, so almost any string succeeds. A server choosing a response encoding from this header should read the raw value instead of trusting the parsed variant. Tracked as [zio/zio-blocks#1618](https://github.com/zio/zio-blocks/issues/1618).
:::

### `Headers#getLast` — last parseable match

`Headers#getLast` is `Headers#getAll` keeping only the final element, which is what you want when a later header is meant to override an earlier one:

```scala
final class Headers {
  def getLast[H <: Header](headerType: Header.Typed[H]): Option[H]
}
```

Note the tighter bound: `Headers#getLast` takes a `Header.Typed[H]`, so it works with the built-ins but not with a bare `Header.Codec[A]` over a non-`Header` type:

```scala

val overridden = Headers("content-length" -> "100", "content-length" -> "200")
```

First and last differ, and each method says which it takes:

```scala
overridden.get(Header.ContentLength)
// res26: Option[ContentLength] = Some(ContentLength(100L))
overridden.getLast(Header.ContentLength)
// res27: Option[ContentLength] = Some(ContentLength(200L))
```

### Raw Access

Three methods bypass parsing entirely and hand back the stored string. Use them for headers with no typed model, for pass-through proxying, and for telling a malformed value from an absent one:

| Method                    | Returns             | Picks                        |
| ------------------------- | ------------------- | ---------------------------- |
| `Headers#rawGet`          | `Option[String]`    | First entry with that name    |
| `Headers#rawGetLast`      | `Option[String]`    | Last entry with that name     |
| `Headers#rawGetAll`       | `Chunk[String]`     | Every entry, in header order  |

All three validate the name before scanning and are case-insensitive:

```scala

val headers = Headers("x-trace" -> "a", "x-trace" -> "b")
```

Raw reads never fail on content, only on a structurally invalid name:

```scala
headers.rawGet("X-Trace")
// res29: Option[String] = Some("a")
headers.rawGetLast("x-trace")
// res30: Option[String] = Some("b")
headers.rawGetAll("x-trace")
// res31: Chunk[String] = IndexedSeq("a", "b")
```

`Headers#has` and its alias `Headers#contains` test for presence without reading a value:

```scala
headers.has("x-trace")
// res32: Boolean = true
headers.contains("x-missing")
// res33: Boolean = false
```

## The Parse Cache

`Headers` stores three parallel arrays: lowercased names, raw values, and a lazily-filled slot for the parsed value of each entry. A typed read fills that slot; a second read of the same header reuses it.

The cache is keyed by *codec identity*, compared by reference. Two codecs sharing a wire name therefore never read each other's cached values, which is what keeps a custom `Header.Codec[MyType]` named `content-length` from colliding with `Header.ContentLength`.

Three consequences are worth knowing:

- **The cache is dropped by `Headers#add`.** Appending builds fresh arrays and does not carry parsed values across, so a read-then-add-then-read sequence parses twice. Read after you finish assembling, not between additions.
- **Duplicate parses are possible under concurrency.** Filling a slot is not synchronized, so two threads reading the same header on the same instance may both parse it. Both produce equal values, so the race is benign — but it is a race, and a codec with side effects would run twice.
- **Failures are not cached.** An entry that fails to parse leaves its slot empty, so every read retries the failing parse.

## Writing Headers

Writes come in typed and raw forms, and the distinction that matters is append versus replace.

### Appending and Replacing

`Headers#add` appends, keeping any existing header of the same name. `Headers#set` replaces every entry with that name:

```scala
final class Headers {
  def add(header: Header): Headers
  def add(name: String, value: String): Headers
  def set(header: Header): Headers
  def set(name: String, value: String): Headers
  def remove(name: String): Headers
}
```

Starting from a collection that already has a value, the two diverge:

```scala

val base = Headers.empty.add(Header.ContentLength(100))
```

`Headers#add` produces two entries, while `Headers#set` produces one:

```scala
base.add(Header.ContentLength(200)).toList
// res35: List[Tuple2[String, String]] = List(
//   ("content-length", "100"),
//   ("content-length", "200")
// )
base.set(Header.ContentLength(200)).toList
// res36: List[Tuple2[String, String]] = List(("content-length", "200"))
```

`Headers#remove` drops every entry with the given name, and `Headers#++` concatenates without deduplicating — the result can hold duplicates from both sides:

```scala
base.remove("content-length").toList
// res37: List[Tuple2[String, String]] = List()
(base ++ Headers("content-length" -> "300")).toList
// res38: List[Tuple2[String, String]] = List(
//   ("content-length", "100"),
//   ("content-length", "300")
// )
```

Every method returns a new `Headers`; the header names and values are never changed in place. The only in-place write is the parse cache slot that a typed read fills (see [The Parse Cache](#the-parse-cache)).

### `Headers.apply` and `Headers.empty`

The companion builds a collection from name-value pairs, and `Headers.empty` is the identity for `Headers#++`:

```scala
object Headers {
  def apply(pairs: (String, String)*): Headers
  val empty: Headers
}
```

Pairs are validated and lowercased as they are added:

```scala

```

Both entry points produce the same shape, so a builder chain can start from either:

```scala
Headers("Content-Type" -> "application/json").toList
// res40: List[Tuple2[String, String]] = List(
//   ("content-type", "application/json")
// )
Headers.empty.size
// res41: Int = 0
```

### `HeadersBuilder` — batch construction

Building with `Headers#add` allocates fresh arrays per call, which is wasteful when adding many headers at once. `HeadersBuilder` accumulates into a growable buffer and copies once:

```scala
final class HeadersBuilder {
  def add(name: String, value: String): Unit
  def reset(): Unit
  def build(): Headers
}

object HeadersBuilder {
  def make(initialCapacity: Int = 8): HeadersBuilder
}
```

The builder is mutable and single-threaded by design, and `HeadersBuilder#build` snapshots it:

```scala

val builder = HeadersBuilder.make(4)
builder.add("content-type", "application/json")
builder.add(Header.ContentLength(64).headerName, Header.ContentLength(64).renderedValue)
```

The built collection is an ordinary immutable `Headers`:

```scala
builder.build().toList
// res45: List[Tuple2[String, String]] = List(
//   ("content-type", "application/json"),
//   ("content-length", "64")
// )
```

`HeadersBuilder#reset` clears the buffer for reuse without reallocating, which is why the capacity argument exists. The requested capacity is raised to a minimum of four, and the buffer doubles when it fills.

## Validation and Injection Safety

Header names and values are validated on every write, and the value rule exists for a security reason rather than a formatting one.

A name must be a non-empty HTTP token: ASCII letters, digits, or one of `!#$%&'*+-.^_`|~`. A value may contain anything **except** carriage return or line feed. Rejecting CR and LF is what prevents response splitting and header injection — a value carrying `\r\n` would otherwise terminate the header and let an attacker append headers or a body of their own choosing.

Two public methods expose the checks as values, for validating input before you try to use it:

```scala
object Headers {
  def validateName(name: String): Either[String, Unit]
  def validateValue(value: String): Either[String, Unit]
}
```

Both report a message rather than throwing:

```scala

```

A structurally invalid name and a CR/LF-bearing value are each rejected with a reason:

```scala
Headers.validateName("x-trace")
// res47: Either[String, Unit] = Right(())
Headers.validateName("x trace")
// res48: Either[String, Unit] = Left("Invalid header name: x trace")
Headers.validateValue("ok")
// res49: Either[String, Unit] = Right(())
Headers.validateValue("evil\r\nSet-Cookie: admin=true")
// res50: Either[String, Unit] = Left("Header value cannot contain CR or LF")
```

:::danger[The mutating methods throw]
`Headers#add`, `Headers#set`, `Headers#remove`, `Headers#has`, the `rawGet*` family, and `HeadersBuilder#add` all enforce the same invariants by throwing `IllegalArgumentException`. When a name or value comes from outside your program — a proxied request, a user-supplied field — check it with `Headers.validateName` or `Headers.validateValue` first, or the throw becomes your error handling.
:::

Note that the raw *read* methods validate their name argument too, so `headers.rawGet(userSuppliedName)` can throw even though it only reads.

## Equality and Rendering

`Headers#equals` compares `Headers#toList`, so equality is order-sensitive and name-case-insensitive — names were lowercased on the way in, but two collections holding the same headers in a different order are not equal. `Headers#hashCode` is derived from the same list, so it agrees.

`Headers#toString` renders every name and raw value:

```scala

val withAuth = Headers("authorization" -> "Bearer sk-live-1234567890")
```

The value appears in full, which matters wherever a `Headers` reaches a log:

```scala
withAuth.toString
// res52: String = "Headers(authorization: Bearer sk-live-1234567890)"
```

:::warning[`toString` prints credentials]
There is no redaction. `authorization`, `cookie`, and `set-cookie` values are rendered verbatim, and anything that logs a `Request` or `Response` logs its headers. Strip or replace sensitive entries with `Headers#remove` or `Headers#set` before logging.
:::

`Headers#toList` and `Headers#toChunk` expose the same pairs for iteration, differing only in collection type.

## The Header Catalog

All 75 built-in headers, grouped the way the module's own test suites group them. Every entry's companion object is its `Header.Typed` codec, and the wire name column is exactly what `Codec#name` returns.

### Authentication

| Type                               | Wire name              | Shape                                          |
| ---------------------------------- | ---------------------- | ---------------------------------------------- |
| `Header.Authorization`             | `authorization`        | ADT: `Basic`, `Bearer`, `Digest`, `Unparsed`    |
| `Header.ProxyAuthorization`        | `proxy-authorization`  | ADT: `Basic`, `Bearer`, `Digest`, `Unparsed`    |
| `Header.WWWAuthenticate`           | `www-authenticate`     | `scheme: String`, `params: Map[String, String]` |
| `Header.ProxyAuthenticate`         | `proxy-authenticate`   | `scheme: String`, `params: Map[String, String]` |

`Unparsed` is the escape hatch: a scheme the ADT does not model is preserved rather than rejected, so an unknown authentication scheme survives a parse-render round trip.

[How to Protect Pages with Basic Auth or a Session Cookie](../../guides/protect-pages-with-basic-auth-or-a-session-cookie.md) reads `Header.Authorization.Basic` from requests and answers with a `Header.WWWAuthenticate` challenge.

### Content

| Type                               | Wire name                   | Shape                                                      |
| ---------------------------------- | --------------------------- | ---------------------------------------------------------- |
| `Header.ContentType`               | `content-type`              | `value: ContentType`                                        |
| `Header.ContentLength`             | `content-length`            | `length: Long`                                              |
| `Header.ContentEncoding`           | `content-encoding`          | ADT: `GZip`, `Deflate`, `Br`, `Compress`, `Identity`, `Multiple` |
| `Header.ContentDisposition`        | `content-disposition`       | ADT: `Attachment`, `Inline`, `FormData`                       |
| `Header.ContentLanguage`           | `content-language`          | `language: String`                                           |
| `Header.ContentLocation`           | `content-location`          | `location: String`                                           |
| `Header.ContentRange`              | `content-range`             | `unit: String`, `range: Option[(Long, Long)]`, `size: Option[Long]` |
| `Header.ContentSecurityPolicy`     | `content-security-policy`   | `directives: String`                                         |
| `Header.ContentTransferEncoding`   | `content-transfer-encoding` | ADT: `SevenBit`, `EightBit`, `Binary`, `QuotedPrintable`, `Base64` |
| `Header.ContentMd5`                | `content-md5`               | `value: String`                                              |
| `Header.ContentBase`               | `content-base`              | `uri: String`                                                |

`Multiple` on `ContentEncoding` carries a `Chunk` of the others, which is how a comma-separated list becomes one value rather than several entries.

### Caching

| Type                        | Wire name             | Shape                                    |
| --------------------------- | --------------------- | ---------------------------------------- |
| `Header.CacheControl`       | `cache-control`       | ADT, 17 variants — see below              |
| `Header.ETag`               | `etag`                | `tag: String`, `weak: Boolean`            |
| `Header.IfMatch`            | `if-match`            | ADT: `Any`, `ETags`                       |
| `Header.IfNoneMatch`        | `if-none-match`       | ADT: `Any`, `ETags`                       |
| `Header.IfModifiedSince`    | `if-modified-since`   | `date: String`                            |
| `Header.IfUnmodifiedSince`  | `if-unmodified-since` | `date: String`                            |
| `Header.IfRange`            | `if-range`            | `value: String`                           |
| `Header.Expires`            | `expires`             | `date: String`                            |
| `Header.Age`                | `age`                 | `seconds: Long`                           |
| `Header.LastModified`       | `last-modified`       | `date: String`                            |
| `Header.Pragma`             | `pragma`              | `directives: String`                      |
| `Header.Vary`               | `vary`                | ADT: `Any`, `Headers`                     |

`CacheControl` is the largest ADT in the module. Ten variants are flag directives with no argument — `NoCache`, `NoStore`, `NoTransform`, `Public`, `Private`, `MustRevalidate`, `ProxyRevalidate`, `Immutable`, `OnlyIfCached`, `MustUnderstand` — and six take a duration: `MaxAge`, `SMaxAge`, `MinFresh`, `StaleWhileRevalidate`, and `StaleIfError` each hold a `Long`, while `MaxStale` holds an `Option[Long]` because `max-stale` is valid with or without a value. `Multiple` wraps a `Chunk[CacheControl]` for comma-separated directive lists.

Parsing splits on `=` to decide between the two families, so an unknown directive name and a non-numeric duration produce different messages:

```scala

```

Both failures name what went wrong, which is what makes them useful in a 400 response:

```scala
Header.CacheControl.parse("max-age=600")
// res54: Either[String, CacheControl] = Right(MaxAge(600L))
Header.CacheControl.parse("max-stale")
// res55: Either[String, CacheControl] = Right(MaxStale(None))
Header.CacheControl.parse("nonsense")
// res56: Either[String, CacheControl] = Left(
//   "Unknown cache-control directive: nonsense"
// )
Header.CacheControl.parse("max-age=soon")
// res57: Either[String, CacheControl] = Left(
//   "Invalid number in cache-control directive: soon"
// )
```

### Content Negotiation

| Type                      | Wire name          | Shape                                                            |
| ------------------------- | ------------------ | ---------------------------------------------------------------- |
| `Header.Accept`           | `accept`           | `mediaRanges: Chunk[Accept.MediaRange]`                           |
| `Header.AcceptEncoding`   | `accept-encoding`  | ADT: `GZip`, `Deflate`, `Br`, `Compress`, `Identity`, `Any`, `Multiple` |
| `Header.AcceptLanguage`   | `accept-language`  | `languages: Chunk[AcceptLanguage.LanguageRange]`                  |
| `Header.AcceptRanges`     | `accept-ranges`    | ADT: `Bytes`, `None_`                                             |
| `Header.AcceptPatch`      | `accept-patch`     | `mediaTypes: Chunk[MediaType]`                                    |

`Accept.MediaRange` and `AcceptLanguage.LanguageRange` are the per-entry types that carry a quality weight, which is what makes these headers a `Chunk` rather than a single value. `None_` on `AcceptRanges` is spelled with a trailing underscore because `None` is taken.

### CORS

| Type                                     | Wire name                          | Shape                             |
| ---------------------------------------- | ---------------------------------- | --------------------------------- |
| `Header.AccessControlAllowOrigin`        | `access-control-allow-origin`       | ADT: `All`, `Specific`             |
| `Header.AccessControlAllowMethods`       | `access-control-allow-methods`      | `methods: Chunk[Method]`           |
| `Header.AccessControlAllowHeaders`       | `access-control-allow-headers`      | `headers: Chunk[String]`           |
| `Header.AccessControlAllowCredentials`   | `access-control-allow-credentials`  | `allow: Boolean`                   |
| `Header.AccessControlExposeHeaders`      | `access-control-expose-headers`     | `headers: Chunk[String]`           |
| `Header.AccessControlMaxAge`             | `access-control-max-age`            | `seconds: Long`                    |
| `Header.AccessControlRequestHeaders`     | `access-control-request-headers`    | `headers: Chunk[String]`           |
| `Header.AccessControlRequestMethod`      | `access-control-request-method`     | `method: Method`                   |
| `Header.Origin`                          | `origin`                           | ADT: `Null_`, `Value`              |

`AccessControlAllowMethods` and `AccessControlRequestMethod` hold `Method` values rather than strings, so an unrecognized verb fails at parse time instead of reaching your CORS logic.

### Routing and Identity

| Type                    | Wire name       | Shape                                |
| ----------------------- | --------------- | ------------------------------------ |
| `Header.Host`           | `host`          | `host: String`, `port: Option[Int]`   |
| `Header.Location`       | `location`      | `uri: String`                         |
| `Header.Referer`        | `referer`       | `uri: String`                         |
| `Header.Via`            | `via`           | `entries: Chunk[String]`              |
| `Header.Forwarded`      | `forwarded`     | `params: String`                      |
| `Header.MaxForwards`    | `max-forwards`  | `count: Int`                          |
| `Header.From`           | `from`          | `email: String`                       |
| `Header.UserAgent`      | `user-agent`    | `product: String`                     |
| `Header.Server`         | `server`        | `product: String`                     |
| `Header.Date`           | `date`          | `value: String`                       |
| `Header.Link`           | `link`          | `value: String`                       |
| `Header.RetryAfter`     | `retry-after`   | `value: String`                       |
| `Header.Allow`          | `allow`         | `methods: Chunk[Method]`              |
| `Header.Expect`         | `expect`        | `value: String`                       |
| `Header.Range`          | `range`         | `unit: String`, `ranges: String`      |

`Host` splits the port out as an `Option[Int]`, so `example.com` and `example.com:8080` both parse and render back to their original form.

### Cookies

| Type                       | Wire name    | Shape           |
| -------------------------- | ------------ | --------------- |
| `Header.CookieHeader`      | `cookie`     | `value: String`  |
| `Header.SetCookieHeader`   | `set-cookie` | `value: String`  |

Both model the header as an unparsed string; structured cookie handling lives in the separate `Cookie` type documented in [the HTTP model](./model.md). Two aliases exist on the companion for readability at call sites — `Header.Cookie` is `Header.CookieHeader` and `Header.SetCookie` is `Header.SetCookieHeader`, both typed as the codec rather than the case class.

### Connection Management

| Type                       | Wire name            | Shape                                                                  |
| -------------------------- | -------------------- | ---------------------------------------------------------------------- |
| `Header.Connection`        | `connection`         | ADT: `Close`, `KeepAlive`, `Other`                                       |
| `Header.Upgrade`           | `upgrade`            | `protocol: String`                                                      |
| `Header.Te`                | `te`                 | `value: String`                                                         |
| `Header.Trailer`           | `trailer`            | `headers: Chunk[String]`                                                |
| `Header.TransferEncoding`  | `transfer-encoding`  | ADT: `Chunked`, `Compress`, `Deflate`, `GZip`, `Identity`, `Multiple`    |

### Security

| Type                               | Wire name                    | Shape                                              |
| ---------------------------------- | ---------------------------- | -------------------------------------------------- |
| `Header.XFrameOptions`             | `x-frame-options`            | ADT: `Deny`, `SameOrigin`                           |
| `Header.XRequestedWith`            | `x-requested-with`           | `value: String`                                     |
| `Header.DNT`                       | `dnt`                        | ADT: `TrackingAllowed`, `TrackingNotAllowed`, `Unset` |
| `Header.UpgradeInsecureRequests`   | `upgrade-insecure-requests`  | `upgrade: Boolean`                                  |
| `Header.ClearSiteData`             | `clear-site-data`            | `directives: Chunk[String]`                         |

### WebSocket

| Type                              | Wire name                  | Shape                       |
| --------------------------------- | -------------------------- | --------------------------- |
| `Header.SecWebSocketAccept`       | `sec-websocket-accept`      | `value: String`              |
| `Header.SecWebSocketExtensions`   | `sec-websocket-extensions`  | `value: String`              |
| `Header.SecWebSocketKey`          | `sec-websocket-key`         | `value: String`              |
| `Header.SecWebSocketLocation`     | `sec-websocket-location`    | `value: String`              |
| `Header.SecWebSocketOrigin`       | `sec-websocket-origin`      | `value: String`              |
| `Header.SecWebSocketProtocol`     | `sec-websocket-protocol`    | `protocols: Chunk[String]`   |
| `Header.SecWebSocketVersion`      | `sec-websocket-version`     | `version: String`            |

## Headers Without a Typed Model

Two mechanisms cover headers the catalog does not include, and the choice between them is whether you want a value or a type.

### `Header.Custom` — a name and a string

`Header.Custom` is a `Header` whose name and value you supply directly. Use it to write an arbitrary header through the same API as a built-in:

```scala

val custom = Header.Custom("x-tenant-id", "acme-42")
```

It renders exactly what it holds, with no parsing on either side:

```scala
Headers.empty.add(custom).toList
// res59: List[Tuple2[String, String]] = List(("x-tenant-id", "acme-42"))
```

`Header.Custom` has no codec, so it cannot be passed to `Headers#get`. Reading it back means `Headers#rawGet`.

### A Custom `Header.Codec`

Implementing `Header.Codec[A]` gives a header both a domain type and typed reads. Nothing requires `A` to extend `Header` unless you also want `Headers#getLast`:

```scala

final case class TenantId(value: String)

val tenantIdCodec: Header.Codec[TenantId] = new Header.Codec[TenantId] {
  def name: String = "x-tenant-id"

  def parse(value: String): Either[String, TenantId] =
    if (value.isEmpty) Left("x-tenant-id must not be empty")
    else Right(TenantId(value))

  def render(value: TenantId): String = value.value
}
```

The custom codec now works with the typed read path, cache included:

```scala
val headers = Headers("x-tenant-id" -> "acme-42")
// headers: Headers = Headers(x-tenant-id: acme-42)
headers.get(tenantIdCodec)
// res61: Option[TenantId] = Some(TenantId("acme-42"))
```

A rejected value is skipped like any other parse failure, so the collection read is `None` rather than the error message:

```scala
Headers("x-tenant-id" -> "").get(tenantIdCodec)
// res62: Option[TenantId] = None
tenantIdCodec.parse("")
// res63: Either[String, TenantId] = Left("x-tenant-id must not be empty")
```

Return a message describing what was expected. Parse errors reach users through whatever 400 response your handler builds, and the codec's message is the only description available.

:::tip[Hold the codec as a `val`]
The parse cache compares codec identity by reference. A codec constructed inline on every request is a different instance each time, so its cached values are never reused. Define it once as a `val` or an `object`.
:::

## Integration Points

`Header` and `Headers` are used by `Request` and `Response`, which each hold a `Headers` alongside their status or method, URL, and body — see [the HTTP model](./model.md). `Header.ContentType` wraps the `ContentType` type that `Body` also carries, and the media-type values inside it come from `zio-blocks-mediatype`.

Several headers hold values from elsewhere in the module rather than strings: `Allow`, `AccessControlAllowMethods`, and `AccessControlRequestMethod` hold `Method`, and `Chunk` is the sequence type throughout.

For deriving header codecs from a `Schema[A]` instead of writing them by hand, see [HTTP model schema](./schema.md), which builds `HeaderCodec` and `QueryCodec` instances from schemas. `ServerSentEvent` does not use `Headers` or `Response`: it renders a `String`, which you put in a `Response` body with a `text/event-stream` content type yourself; see [Server-Sent Events](./server-sent-event.md) and the guide [How to Send Server-Sent Events with http-model](../../guides/send-server-sent-events.md).

---

## HTTP Model


`zio-blocks-http-model` is a **pure, zero-dependency HTTP data model** for building HTTP clients and servers. It provides immutable types representing all HTTP concepts: requests, responses, headers, URLs, paths, query parameters, methods, status codes, versions, cookies, and forms. The module separates protocol concerns (representing HTTP messages) from effect concerns (actually sending/receiving them), enabling portable, testable HTTP code across any Scala application.

## Core Capabilities

The HTTP model is organized into two complementary modules:

### [HTTP Model](./model.md) — Pure Data Types

The foundation of the HTTP model: immutable data types representing HTTP requests, responses, and all associated primitives. No effects, no I/O, no coupling to specific runtimes.

**Key types:** `Request`, `Response`, `URL`, `Headers`, `Body`, `Method`, `Status`, `Version`, `Scheme`, `Path`, `QueryParams`, `ContentType`, `RequestCookie`, `ResponseCookie`, `Form`.

**Use this when:** Building request/response data structures, parsing HTTP primitives, serializing messages for storage or caching, or sharing HTTP types across effect systems.

### [Schema-Based Typed Access](./schema.md) — Type-Safe Extraction

Extension methods that add **type-safe, validated extraction** of query parameters and headers to the core HTTP model. Automatically decode string values to typed objects using schema-based decoding with comprehensive error reporting.

**Key features:** Typed `QueryParams#query[T]`, `Headers#header[T]`, `Request#query[T]` methods with schema-based decoding, automatic error reporting for missing or malformed values, support for 11 primitive types and custom types via `Schema[T]`.

**Use this when:** You need to extract and validate query parameters or headers with type safety, decode values to domain types, or provide clear error messages for malformed input.

## How They Work Together

1. **Build pure HTTP data** using the core HTTP model types
2. **Add type-safe extraction** with schema-based extension methods
3. **Hand off to an HTTP client/server library** (ZIO HTTP, Akka, Play, etc.) for the actual I/O work

See [How to Serve http-model Requests with the JDK HTTP Server](../../guides/serve-http-model-requests.md) for a worked hand-off to the server built into the JDK.

This separation keeps your domain logic portable and testable while maintaining full expressiveness for HTTP interactions.

## Getting Started

Start with the [HTTP Model](./model.md) to understand the core data types and how they compose. [Header](./headers.md) covers the typed header model — the 75 built-in headers, the codec type class behind them, and the parse cache — and [ServerSentEvent](./server-sent-event.md) covers the SSE envelope and its wire format. Then explore [Schema-Based Typed Access](./schema.md) to learn how to safely extract and validate HTTP parameters and headers.

---

**Modules:**
- [`zio-blocks-http-model`](./model.md) — Core immutable data types for HTTP
- [`zio-blocks-http-model`](./headers.md) — Typed header model and the `Header.Codec` type class
- [`zio-blocks-http-model`](./server-sent-event.md) — Server-Sent Event envelope and payload encoders
- [`zio-blocks-http-model-schema`](./schema.md) — Schema-based typed extraction for query parameters and headers

---

## HTTP Model(Http-model)

`zio-blocks-http-model` is a **runtime-independent HTTP data model** for building HTTP clients and servers. It provides immutable types representing all HTTP concepts: requests, responses, headers, URLs, paths, query parameters, methods, status codes, versions, cookies, and forms.

Core types: `Request`, `Response`, `URL`, `Headers`, `Body`, `Method`, `Status`, `Version`, `Scheme`, `Path`, `QueryParams`, `ContentType`, `RequestCookie`, `ResponseCookie`, `Form`.

Here is a high-level overview of the core types and their relationships:

```scala
// Core request/response types
final case class Request(method: Method, url: URL, headers: Headers, body: Body, version: Version)
final case class Response(status: Status, headers: Headers, body: Body, version: Version)

// URL structure
final case class URL(scheme: Option[Scheme], host: Option[String], port: Option[Int],
                     path: Path, queryParams: QueryParams, fragment: Option[String])
final case class Path(segments: Chunk[String], hasLeadingSlash: Boolean, trailingSlash: Boolean)

// HTTP primitives
sealed abstract class Method(val name: String, val ordinal: Int)
opaque type Status = Int
sealed abstract class Version(val major: Int, val minor: Int)
sealed trait Scheme

// Headers and body
final class Headers(...)
sealed trait Header
final class Body(val stream: Stream[Nothing, Byte], val contentType: ContentType)
```

## Motivation

Building modern distributed systems requires HTTP clients and servers, but most HTTP libraries bake effects (I/O, streaming, async) directly into their data types. This creates coupling problems when sharing types across layers or testing without pulling in entire runtimes.

### The Problem: Protocol, Effects, and Coupling

Imagine building a distributed system where you need an HTTP client to call external APIs and an HTTP server to handle incoming requests. Your first instinct is to reach for a popular HTTP library. But here's the trouble: most HTTP libraries bake "effects" (I/O operations, streaming, async) directly into their data structures.

This creates a coupling problem:

**Scenario 1: Sharing Types Across Layers**
You want your client request logic (building a request to send) to use the same types as your server request handling (receiving and parsing a request). But your HTTP library makes this difficult — the `Request` type is tied to a particular runtime, file-stream implementation, or async framework. Sharing becomes messy.

**Scenario 2: Testing Without an HTTP Runtime**
You're writing unit tests for your request-building logic. You want to serialize a request to JSON for snapshots, or cache requests for debugging. But your `Request` type requires pulling in an HTTP runtime or other baggage you don't need in tests. A simple unit test becomes a production-grade setup.

**Scenario 3: Lock-In**
You've built your entire API client around ZIO's HTTP library, but your team decides to use Akka for one microservice. Now your request/response types aren't portable — they're coupled to ZIO. Refactoring is painful.

### The Solution: Runtime-Independent HTTP Data

`zio-blocks-http-model` separates **protocol concerns** (representing HTTP messages) from **effect concerns** (actually sending/receiving them). It provides:

- **Immutable message values** — `Request`, `Response`, `URL`, and `Headers` are immutable data. `Body` wraps a pull-based byte stream; materializing the complete body is represented by the lightweight `Async` abstraction.

- **No dependency on an HTTP runtime** — Not coupled to ZIO HTTP, Akka, or a server/client implementation. The module depends on ZIO Blocks primitives such as Chunk, MediaType, and Stream.

- **Incremental, lazy parsing** — Headers are parsed on first access and cached as an optimization. You pay only for what you use.

- **Efficient, composable** — Headers and QueryParams use parallel array-backed collections for fast lookups. Types compose naturally without forcing a single architectural path.

This separation is powerful: you can build, manipulate, serialize, and test HTTP messages using pure data, then hand them off to any HTTP client/server library (ZIO, Akka, Play, etc.) for the actual I/O work. Your domain logic stays portable and testable. For a worked example, see [How to Serve http-model Requests with the JDK HTTP Server](../../guides/serve-http-model-requests.md).

## Installation

Add the following to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-http-model" % "0.0.56"
```

For cross-platform projects (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-http-model" % "0.0.56"
```

Supported Scala versions: Scala 2.13 and Scala 3.x. On Scala 3.9 and newer the artifact is `zio-blocks-next-http-model` (see [How to Add ZIO Blocks to Your Build](../../getting-started/installation.md)). The artifacts are cross-platform for JVM and Scala.js.

## How They Work Together

The HTTP model consists of **request/response messages** at the center, composed of smaller types that handle specific concerns:

```
Request ────────────────────────────────────────────────────
  ├─ method: Method (HTTP verb: GET, POST, PUT, DELETE, etc.)
  ├─ url: URL (target endpoint)
  │   ├─ scheme: Option[Scheme] (HTTP, HTTPS, WS, WSS, Custom)
  │   ├─ host: Option[String] (domain name)
  │   ├─ port: Option[Int] (port number)
  │   ├─ path: Path (URL path segments)
  │   ├─ queryParams: QueryParams (query string parameters)
  │   └─ fragment: Option[String] (anchor)
  ├─ headers: Headers (HTTP headers, lazily parsed for typed access)
  │   └─ Header (individual headers: Authorization, Content-Type, etc.)
  ├─ body: Body (message content)
  │   └─ contentType: ContentType (media type, charset, boundary)
  └─ version: Version (HTTP/1.0, HTTP/1.1, HTTP/2.0, HTTP/3.0)

Response ────────────────────────────────────────────────────
  ├─ status: Status (HTTP status code: 200, 404, 500, etc.)
  ├─ headers: Headers (same as Request)
  ├─ body: Body (response content)
  └─ version: Version (HTTP protocol version)
```

These types don't exist in isolation — they work together as you build requests and handle responses. The hierarchy above shows the structure, but understanding the actual workflow reveals why each piece is important. Here is the typical flow when using the HTTP model in a client or server:

1. **Build URL** — Parse or construct a URL with path segments and query parameters
2. **Create Request** — Combine method, URL, headers, and body into a Request
3. **Send** — Transmit the request (handled by HTTP client/server library, not http-model)
4. **Receive Response** — Parse incoming Response with status, headers, and body
5. **Access Data** — Extract typed headers, cookies, content type from Headers
6. **Parse Content** — Body contains raw bytes; higher-level libraries deserialize based on content type

## Common Patterns

### Building Requests with Headers and Query Parameters

Create a request with headers and query parameters:

```scala

val url = URL.parse("https://api.example.com/users?role=admin").toOption.get
val req = Request
  .get(url)
  .addHeader("authorization", "Bearer token123")
  .addHeader("content-type", "application/json")
```

### Extracting and Manipulating Headers

Access headers by name or retrieve all headers:

```scala

// Accessing headers in a request
val url = URL.parse("https://example.com").toOption.get
val req = Request.get(url).addHeader("content-type", "application/json")

val allHeaders = req.headers.toList                // All headers as List[(String, String)]
val headerCount = allHeaders.length                // Count headers
val contentType = req.headers.rawGet("content-type") // Get raw string header value (case-insensitive lookup)
```

Custom typed headers do not need their own `Header` subtype. Define a `Header.Codec[A]` and reuse it with `Headers`, `Request`, and `Response`:

```scala

object TraceIdHeader extends Header.Codec[String] {
  def name: String = "x-trace-id"
  def parse(value: String): Either[String, String] =
    if (value.startsWith("trace-")) Right(value) else Left("trace id must start with trace-")
  def render(value: String): String = value
}

val request = Request
  .get(URL.parse("https://example.com").toOption.get)
  .addHeader("x-trace-id", "trace-123")

val traceId = request.header(TraceIdHeader)
```

### Creating Responses with Status and Content

Build a JSON response:

```scala

val body = Body.fromString("""{"message":"ok"}""", Charset.UTF8)
val response = Response(
  status = Status.Ok,
  body = body,
  headers = Headers("content-type" -> "application/json"),
  version = Version.`HTTP/1.1`
)
```

### URL Manipulation

Parse URLs and extract components:

```scala

val url = URL.parse("https://example.com/api/v1/users?page=1&limit=10").toOption.get

url.scheme                           // Some(Scheme.HTTPS)
url.host                             // Some("example.com")
url.port                             // None (defaults to 443 for HTTPS)
url.path.segments                    // Chunk("api", "v1", "users")
url.queryParams.getFirst("page")     // Some("1") (first value for key)
url.??("filter", "active")           // Add parameter using ?? operator
```

### Form Data Submission

Submit a form with key-value pairs:

```scala

val url = URL.parse("https://example.com/login").toOption.get
val form = Form("username" -> "alice", "password" -> "secret123", "remember" -> "true")
val body = Body.fromString(form.encode, Charset.UTF8)
val req = Request.post(url, body)
  .addHeader("content-type", "application/x-www-form-urlencoded")
```

### Cookies in Requests and Responses

Send cookies to server and receive cookies from server:

```scala

val url = URL.parse("https://example.com").toOption.get

// Request: send cookies to server
val req = Request.get(url)
  .addHeader("cookie", "sessionId=abc123; userId=456")

// Response: server sets cookies for client to store
val response = Response.ok
  .addHeader("set-cookie", "sessionId=abc123; Max-Age=3600; Secure; HttpOnly")
```

---

## Method

`Method` represents the HTTP verb (request method) as sealed case objects:

```scala
sealed abstract class Method(val name: String, val ordinal: Int)
```

### Predefined Methods

The nine standard HTTP methods are predefined as case objects:

```scala

Method.GET      // Retrieve a resource
Method.POST     // Create a new resource
Method.PUT      // Replace a resource entirely
Method.DELETE   // Remove a resource
Method.PATCH    // Partially update a resource
Method.HEAD     // Retrieve headers only (no body)
Method.OPTIONS  // Describe communication options
Method.TRACE    // Echo the request back (for debugging)
Method.CONNECT  // Establish a tunnel (for proxies)
```

### Parsing Methods

Parse HTTP method strings to `Method` objects:

```scala

Method.fromString("GET")    // Some(Method.GET)
Method.fromString("POST")   // Some(Method.POST)
Method.fromString("CUSTOM") // None (HTTP allows custom methods, but not predefined)
```

### Method Properties

Access properties of a `Method` object:

```scala

val m = Method.GET
m.name       // "GET"
m.ordinal    // 0 to 8 for the standard methods, -1 for Method.ANY and Method.Methods
m.toString   // "GET"
```

---

## Status

`Status` wraps an `Int` HTTP status code with no allocation, and provides predefined constants for the common codes (1xx–5xx). A few codes have no constant, such as 208, 226, 423, 424, 506, 508 and 510; `Status(code).text` is `Unknown` for them.

On Scala 3, `Status` is an opaque type alias for `Int`. On Scala 2.13, it is a value class, and `Status#toString` gives `Status(200)` instead of `200`:

```scala
opaque type Status = Int                              // Scala 3
final case class Status(code: Int) extends AnyVal     // Scala 2.13
```

### Predefined Status Codes

Status codes organized by category:

```scala

// 1xx Informational
Status.Continue           // 100
Status.SwitchingProtocols // 101

// 2xx Success
Status.Ok                 // 200
Status.Created            // 201
Status.Accepted           // 202
Status.NoContent          // 204

// 3xx Redirection
Status.MovedPermanently   // 301
Status.Found              // 302 (temporary redirect)
Status.SeeOther           // 303
Status.NotModified        // 304 (response not modified since condition)
Status.TemporaryRedirect  // 307
Status.PermanentRedirect  // 308

// 4xx Client Errors
Status.BadRequest         // 400 (malformed request)
Status.Unauthorized       // 401 (authentication required)
Status.Forbidden          // 403 (authenticated but not authorized)
Status.NotFound           // 404 (resource not found)
Status.MethodNotAllowed   // 405
Status.Conflict           // 409
Status.Gone               // 410 (resource permanently removed)
Status.UnprocessableEntity // 422
Status.TooManyRequests    // 429 (rate limited)

// 5xx Server Errors
Status.InternalServerError // 500 (generic server error)
Status.BadGateway          // 502
Status.ServiceUnavailable  // 503 (server temporarily unavailable)
Status.GatewayTimeout      // 504
```

### Creating Custom Status Codes

Create arbitrary status codes:

```scala

val custom = Status(418)          // I'm a teapot (Easter egg)
val ok     = Status.fromInt(200)  // Parse from Int, returns Status
```

### Status Code Operations

Query status code properties:

```scala

val status = Status.Ok

status.code              // 200 (the underlying Int)
status.text              // "OK" (human-readable)
status.isSuccess         // true (2xx status codes)
status.isInformational   // false (1xx codes)
status.isRedirection     // false (3xx codes)
status.isClientError     // false (4xx codes)
status.isServerError     // false (5xx codes)
status.isError           // false (true for 4xx or 5xx)
```

---

## Version

`Version` represents HTTP protocol versions:

```scala
sealed abstract class Version(val major: Int, val minor: Int)
```

### Predefined Versions

The four standard HTTP versions are predefined:

```scala

Version.`HTTP/1.0`  // HTTP/1.0 (1996)
Version.`HTTP/1.1`  // HTTP/1.1 (1997, persistent connections)
Version.`HTTP/2.0`  // HTTP/2.0 (2015, multiplexing)
Version.`HTTP/3.0`  // HTTP/3.0 (2022, QUIC-based)
```

### Parsing and Rendering

Parse version strings and render to strings:

```scala

Version.fromString("HTTP/1.1") // Some(Version.`HTTP/1.1`)
Version.fromString("HTTP/2")   // Some(Version.`HTTP/2.0`) (suffix optional)
Version.fromString("HTTP/3")   // Some(Version.`HTTP/3.0`)

Version.render(Version.`HTTP/1.1`) // "HTTP/1.1"
Version.`HTTP/2.0`.text            // "HTTP/2.0"
```

### Version Properties

Access version components:

```scala

val v = Version.`HTTP/2.0`
v.major  // 2 (major version number)
v.minor  // 0 (minor version number)
```

---

## Scheme

`Scheme` represents URL schemes with support for HTTP, HTTPS, WebSocket protocols, and custom schemes:

```scala
sealed trait Scheme {
  def text: String
  def defaultPort: Option[Int]
  def isSecure: Boolean
  def isWebSocket: Boolean
}
```

### Predefined Schemes

Standard schemes for HTTP, HTTPS, and WebSocket:

```scala

Scheme.HTTP   // http://, default port 80 (unencrypted)
Scheme.HTTPS  // https://, default port 443 (encrypted)
Scheme.WS     // ws://, default port 80 (WebSocket, unencrypted)
Scheme.WSS    // wss://, default port 443 (WebSocket, encrypted)
```

### Scheme Properties

```scala

Scheme.HTTPS.isSecure      // true
Scheme.HTTP.isSecure       // false
Scheme.WS.isWebSocket      // true
Scheme.HTTP.isWebSocket    // false

Scheme.HTTP.defaultPort    // Some(80)
Scheme.HTTPS.defaultPort   // Some(443)
Scheme.WS.defaultPort      // Some(80)
Scheme.WSS.defaultPort     // Some(443)

Scheme.HTTPS.text          // "https"
```

### Custom Schemes

Create custom schemes (e.g., FTP):

```scala

val ftp = Scheme.Custom("ftp")
val text = ftp.text              // "ftp"
val isSecure = ftp.isSecure      // false (custom schemes assumed insecure)
```

---

## URL

`URL` represents a complete Uniform Resource Locator with scheme, host, port, path, query parameters, and fragment:

```scala
final case class URL(
  scheme: Option[Scheme],
  host: Option[String],
  port: Option[Int],
  path: Path,
  queryParams: QueryParams,
  fragment: Option[String]
)
```

### Parsing URLs

Parse URL strings into `URL` objects:

```scala

URL.parse("https://example.com/api/users?id=123&role=admin")
// Returns: Either[String, URL]

URL.parse("https://api.example.com:8080/v1/users?page=1#results")
// Parses: scheme, host, port (explicit), path, query params, fragment
```

### Constructing URLs

Build URLs by parsing strings:

```scala

// Simplest approach: parse a URL string
val url1 = URL.parse("https://api.example.com/users/123?filter=active&sort=name").toOption.get

// Modify an existing URL by adding query parameters
val url2 = url1.??("page", "1")
```

### URL Operations

Access URL components and modify them:

```scala

val url = URL.parse("https://example.com:8080/api/v1/users?page=1#section").toOption.get

val scheme = url.scheme                        // Some(Scheme.HTTPS)
val host = url.host                            // Some("example.com")
val port = url.port                            // Some(8080)
val pathSegments = url.path.segments           // Chunk("api", "v1", "users")
val pageParam = url.queryParams.getFirst("page") // Some("1") (first value for key)
val fragment = url.fragment                    // Some("section")

val urlWithSort = url.??("sort", "name")      // Add query parameter using ?? operator
val urlWithNewFragment = url.copy(fragment = Some("top")) // Set fragment
```

---

## Path

`Path` represents the URL path (e.g., `/api/v1/users`) stored as segments for efficient manipulation:

```scala
final case class Path(
  segments: Chunk[String],
  hasLeadingSlash: Boolean,
  trailingSlash: Boolean
)
```

### Creating Paths

Create paths from segments or parse from strings:

```scala

val path1 = Path.apply("/api/v1/users")    // Parse from decoded path string
val path2 = Path.fromEncoded("/api/v1/caf%C3%A9") // Split at "/", then decode each segment
val root = Path.root                       // Empty path "/"
```

### Path Operations

Access path segments and build modified paths:

```scala

val path = Path.apply("/api/v1/users")

val segments = path.segments              // Chunk("api", "v1", "users")
val hasLeadingSlash = path.hasLeadingSlash // true
val hasTrailingSlash = path.trailingSlash  // false
val encoded = path.encode                 // "/api/v1/users" (encoded string)
```

---

## QueryParams

`QueryParams` is an immutable multi-map for URL query parameters, optimized for performance. Keys are compared exactly, so `QueryParams("Page" -> "1").get("page")` is `None`:

```scala
final class QueryParams private[http] (...)
```

### Creating QueryParams

Create query parameters from key-value pairs:

```scala

val params1 = QueryParams("page" -> "1", "limit" -> "10")
val params2 = QueryParams.empty
val params3 = QueryParams.fromEncoded("page=1&limit=10") // Parse from encoded query string
```

### QueryParams Operations

Access and modify query parameters:

```scala

val params = QueryParams("page" -> "1", "role" -> "admin", "role" -> "user")

val withSort = params.add("sort", "name")   // Add parameter
val withoutPage = params.remove("page")     // Remove all values for key
val encoded = params.encode                 // "page=1&role=admin&role=user"
```

### Multi-Value Parameters

`QueryParams` supports multiple values for the same key:

```scala

val params = QueryParams("id" -> "1", "id" -> "2", "id" -> "3")
val firstId = params.getFirst("id")   // Some("1") (first value)
val allIds = params.get("id")         // Some(Chunk("1", "2", "3"))
```

---

## Headers

`Headers` is an immutable, case-insensitive collection of HTTP header fields. It stores names pre-lowercased and raw values as strings, allows several entries with the same name, and parses a value into a typed `Header` only when a typed read asks for it:

```scala
final class Headers private[http] (...) {
  def size: Int
  def get[A](headerCodec: Header.Codec[A]): Option[A]
  def rawGet(name: String): Option[String]
  def add(header: Header): Headers
  def set(header: Header): Headers
  def remove(name: String): Headers
}
```

This section covers the collection itself. The 75 built-in typed headers, the codec type class behind them, the parse cache, and the validation rules all live on the [Header](./headers.md) page.

### Creating Headers

`Headers.apply` takes name-value pairs, and `Headers.empty` is the starting point for a builder chain:

```scala

val headers = Headers("content-type" -> "application/json", "cache-control" -> "no-cache")
```

Names are lowercased as they are stored, so the collection reports them in canonical form regardless of how they were written:

```scala
Headers("Content-Type" -> "application/json").toList
// res27: List[Tuple2[String, String]] = List(
//   ("content-type", "application/json")
// )
```

### Reading Raw Values

Three methods read the stored string without parsing, differing in which entry they pick when a name repeats:

```scala
headers.rawGet("Content-Type")
// res28: Option[String] = Some("application/json")
headers.rawGetAll("cache-control")
// res29: Chunk[String] = IndexedSeq("no-cache")
headers.has("content-type")
// res30: Boolean = true
```

Lookups are case-insensitive, and `Headers#contains` is an alias for `Headers#has`. For typed reads — `Headers#get`, `Headers#getAll`, and `Headers#getLast`, which decode into a `Header` — see [Header](./headers.md).

### Modifying Headers

`Headers#add` appends and `Headers#set` replaces every entry with the same name, which is the distinction that matters for multi-value headers:

```scala
val twice   = headers.add("set-cookie", "a=1").add("set-cookie", "b=2")
val replaced = twice.set("set-cookie", "c=3")
```

Appending keeps both cookies while setting collapses them to one:

```scala
twice.rawGetAll("set-cookie")
// res31: Chunk[String] = IndexedSeq("a=1", "b=2")
replaced.rawGetAll("set-cookie")
// res32: Chunk[String] = IndexedSeq("c=3")
```

`Headers#remove` drops every entry with a name, `Headers#++` concatenates two collections without deduplicating, and `Headers#toList` and `Headers#toChunk` expose the pairs for iteration. Every operation returns a new `Headers`.

:::warning[Names and values are validated by throwing]
A header name must be an HTTP token and a value must not contain CR or LF — the latter is what prevents response splitting. The mutating methods enforce both with `IllegalArgumentException`, so check untrusted input with `Headers.validateName` or `Headers.validateValue` first. See [Header](./headers.md) for the rules and the safe pre-checks.
:::

---

## Body

`Body` wraps a `Stream[Nothing, Byte]` with a content type:

```scala
final class Body private (
  val stream: Stream[Nothing, Byte],
  val contentType: ContentType
)
```

### Creating Bodies

`Body.empty` provides an empty body with default `application/octet-stream` content type:

```scala

val empty = Body.empty
// Body(stream = Stream.fromChunk(Chunk.empty[Byte]), contentType = application/octet-stream)
```

`Body.fromString` creates a body with `text/plain` content type:

```scala

val fromString = Body.fromString("Hello, World!", Charset.UTF8)
// Content-Type: text/plain; charset=UTF-8
```

`Body.fromArray` creates a body with default `application/octet-stream` content type without copying the supplied array. Later mutations to the array are visible through the body:

```scala

val fromBytes = Body.fromArray(Array[Byte](1, 2, 3))
// Content-Type: application/octet-stream
```

`Body.fromChunk` creates a body from a `Chunk[Byte]` with optional content type:

```scala

val chunk = Chunk[Byte](1, 2, 3, 4, 5)
val body = Body.fromChunk(chunk)
// Content-Type: application/octet-stream (default)

val jsonBody = Body.fromChunk(chunk, ContentType(MediaTypes.application.`json`))
// Content-Type: application/json
```

`Body.fromStream` creates a body from a `Stream[Nothing, Byte]` with optional content type:

```scala

val stream = Stream.fromChunk(Chunk[Byte](1, 2, 3))
val body = Body.fromStream(stream, ContentType(MediaTypes.application.`octet-stream`))
// Content-Type: application/octet-stream
```

### Reading Bodies

`Body` provides access to the stream, content type, and convenience methods:

```scala

val body = Body.fromString("Hello!", Charset.UTF8)

body.length           // Some(6)
body.isEmpty          // false
body.nonEmpty         // true
body.toStream         // Stream[Nothing, Byte]
body.contentType      // ContentType(text/plain; charset=UTF-8)

val utf8: zio.blocks.async.Async[String] = body.asStringAsync()
val ascii: zio.blocks.async.Async[String] = body.asStringAsync(Charset.ASCII)
val bytes: zio.blocks.async.Async[Array[Byte]] = body.toArrayAsync
val chunk: zio.blocks.async.Async[zio.blocks.chunk.Chunk[Byte]] = body.toChunkAsync
val fromContentType: zio.blocks.async.Async[String] = body.asStringFromContentTypeAsync
val text: zio.blocks.async.Async[String] = body.textAsync
```

The five asynchronous materializers—`toChunkAsync`, `toArrayAsync`, `asStringAsync`, `asStringFromContentTypeAsync`, and the `textAsync` alias—are defined in shared sources and are available on both JVM and Scala.js. `asStringFromContentTypeAsync` and `textAsync` use the charset declared by the content type, falling back to UTF-8.

`Body` retains synchronous twins with the corresponding result types:

```scala

val body = Body.fromString("Hello!", Charset.UTF8)

val chunk: Chunk[Byte] = body.toChunk
val bytes: Array[Byte] = body.toArray
val utf8: String = body.asString()
val ascii: String = body.asString(Charset.ASCII)
val fromContentType: String = body.asStringFromContentType
val text: String = body.text
```

The synchronous methods remain available on both platforms for compatibility and for streams that complete synchronously. Prefer the asynchronous methods whenever a body can suspend; Scala.js cannot block while waiting for pending asynchronous work.

---

## ContentType

`ContentType` represents the MIME type, character encoding, and multipart boundary:

```scala
final case class ContentType(
  mediaType: MediaType,
  boundary: Option[Boundary] = None,
  charset: Option[Charset] = None
)
```

### Setting Content Types

Content types are specified as header values in requests and responses:

```scala

// Common content type headers
val jsonRequest = Request.post(
  URL.parse("https://example.com/api").toOption.get,
  Body.fromString("""{"key":"value"}""", Charset.UTF8)
).addHeader("content-type", "application/json")

val htmlResponse = Response.ok
  .addHeader("content-type", "text/html; charset=utf-8")

val xmlRequest = Request.post(
  URL.parse("https://example.com/xml").toOption.get,
  Body.fromString("<root/>", Charset.UTF8)
).addHeader("content-type", "application/xml")

val formRequest = Request.post(
  URL.parse("https://example.com/form").toOption.get,
  Body.fromString("username=alice&password=secret", Charset.UTF8)
).addHeader("content-type", "application/x-www-form-urlencoded")

val imageResponse = Response.ok
  .addHeader("content-type", "image/png")
```

---

## Charset

`Charset` represents character encoding for text content:

```scala
sealed abstract class Charset(val name: String)
```

### Predefined Charsets

Standard character encodings:

```scala

Charset.UTF8      // UTF-8 (Unicode, variable-length encoding)
Charset.ASCII     // ASCII (7-bit encoding)
Charset.ISO_8859_1 // ISO-8859-1 (Latin-1)
Charset.UTF16     // UTF-16 (with BOM detection)
Charset.UTF16BE   // UTF-16 Big-Endian
Charset.UTF16LE   // UTF-16 Little-Endian
```

### Charset Properties

```scala

Charset.UTF8.name   // "UTF-8"
Charset.ISO_8859_1.name // "ISO-8859-1"
```

---

## RequestCookie

`RequestCookie` represents a simple cookie sent from client to server in the Cookie header:

```scala
final case class RequestCookie(name: String, value: String)
```

### Creating Request Cookies

```scala

val cookie = RequestCookie("sessionId", "abc123xyz")
val name = cookie.name       // "sessionId"
val value = cookie.value     // "abc123xyz"
```

### Sending Cookies

Include cookies in requests, either as a raw `cookie` header or from `RequestCookie` values, which `Cookie.renderRequest` renders as a header value and `Request#addCookie` adds one at a time:

```scala

val url = URL.parse("https://example.com").toOption.get
val req = Request.get(url)
  .addHeader("cookie", "sessionId=abc123; userId=456")

// Or build from RequestCookie objects
val cookies = Chunk(
  RequestCookie("sessionId", "abc123"),
  RequestCookie("userId", "456")
)
val header = Cookie.renderRequest(cookies) // "sessionId=abc123; userId=456"
val req2   = cookies.foldLeft(Request.get(url))(_.addCookie(_))
```

---

## ResponseCookie

`ResponseCookie` represents a cookie set by server for client storage, with full RFC 6265 attributes:

```scala
final case class ResponseCookie(
  name: String,
  value: String,
  expires: Option[String] = None,
  domain: Option[String] = None,
  path: Option[Path] = None,
  maxAge: Option[Long] = None,
  isSecure: Boolean = false,
  isHttpOnly: Boolean = false,
  sameSite: Option[SameSite] = None,
  isPartitioned: Boolean = false,
  priority: Option[CookiePriority] = None
)
```

### Creating Response Cookies

Create cookies with full control over attributes:

```scala

val sessionCookie = ResponseCookie(
  name = "sessionId",
  value = "abc123xyz",
  path = Some(Path.apply("/")),  // Apply to root path
  maxAge = Some(3600),           // 1 hour (in seconds)
  isSecure = true,               // HTTPS only (prevents transmission over HTTP)
  isHttpOnly = true,             // No JavaScript access (prevents XSS theft)
  sameSite = Some(SameSite.Strict), // Prevent CSRF attacks
  priority = Some(CookiePriority.High)
)
```

### Setting Cookies in Responses

Include cookies in responses by adding the Set-Cookie header:

```scala

val response = Response(Status.Ok)
  .addHeader("set-cookie", "sessionId=abc123; Max-Age=3600; Secure; HttpOnly; SameSite=Strict")
```

### SameSite Attribute

`SameSite` controls whether the browser sends the cookie on cross-site requests. `ResponseCookie` sets no `SameSite` attribute by default (`sameSite = None`), and the browser then applies its own default, so set it explicitly:

```scala

val strict = SameSite.Strict  // Only same-site requests (safest)
val lax = SameSite.Lax        // Also cross-site top-level GET navigations (links), not cross-site form posts
val none = SameSite.None_     // Cross-site allowed; render requires Secure flag
```

For a complete login flow, with a session cookie set by `Response#addCookie`, read back with `Request#cookies` and expired on logout, see [How to Protect Pages with Basic Auth or a Session Cookie](../../guides/protect-pages-with-basic-auth-or-a-session-cookie.md).

---

## Form

`Form` represents `application/x-www-form-urlencoded` form data as key-value pairs. Spaces are encoded as `+`, and `+` decodes back to a space.

```scala
final case class Form(entries: Chunk[(String, String)])
```

### Creating Forms

Create forms with key-value pairs:

```scala

val form = Form(
  "username" -> "alice",
  "email" -> "alice@example.com",
  "subscribe" -> "true"
)
```

### Submitting Forms

Send forms as request body:

```scala

val form = Form("username" -> "alice", "password" -> "secret")
val body = Body.fromString(form.encode, Charset.UTF8)

val url = URL.parse("https://example.com/login").toOption.get
val request = Request.post(url, body)
  .addHeader("content-type", "application/x-www-form-urlencoded")
```

### Form Operations

Access and encode form data:

```scala

val form = Form("key1" -> "value1", "key2" -> "value2")

val entries = form.entries      // Chunk[(String, String)] of all entries
val encoded = form.encode       // "key1=value1&key2=value2" (form-url-encoded)
```

---

## FormField

`FormField` represents individual fields in multipart form data (used for file uploads):

```scala
sealed trait FormField

case class FormField.Simple(name: String, value: String)
case class FormField.Text(name: String, filename: Option[String], value: String, contentType: Option[ContentType])
case class FormField.Binary(name: String, filename: Option[String], data: Chunk[Byte], contentType: Option[ContentType])
```

### Multipart Form Fields

Create fields for multipart form submissions:

```scala

val textField = FormField.Simple("username", "alice")

val imageBytes = Chunk.fromArray("...image data...".getBytes)
val fileField = FormField.Binary(
  name = "avatar",
  filename = Some("profile.png"),
  data = imageBytes,
  contentType = zio.http.ContentType.parse("image/png").toOption.get
)
```

---

## Request

`Request` is the core type representing an HTTP request message with method, URL, headers, body, and version:

```scala
final case class Request(
  method: Method,
  url: URL,
  headers: Headers,
  body: Body,
  version: Version
)
```

### Creating Requests

Create requests using convenience methods:

```scala

val url = URL.parse("https://api.example.com/users").toOption.get
val body = Body.fromString("""{"name":"Alice"}""", Charset.UTF8)

// Using convenience methods (recommended)
val getReq = Request.get(url)
val postReq = Request.post(url, body)
val putReq = Request.put(url, body)
val deleteReq = Request.delete(url)
val patchReq = Request.patch(url, body)
val headReq = Request.head(url)
val optionsReq = Request.options(url)
```

### Request Operations

Access and modify request components:

```scala

val url = URL.parse("https://api.example.com/users").toOption.get
val req = Request.get(url)

val method = req.method              // Method.GET
val reqUrl = req.url                 // URL
val headers = req.headers            // Headers collection
val body = req.body                  // Body
val version = req.version            // Version

val withAuth = req.addHeader("authorization", "Bearer token")  // Add header
val transformed = req.updateHeaders(_.add("x-custom", "value")) // Transform headers
```

---

## Response

`Response` is the core type representing an HTTP response message with status, headers, body, and version:

```scala
final case class Response(
  status: Status,
  headers: Headers,
  body: Body,
  version: Version
)
```

### Creating Responses

Create responses with status, headers, and body:

```scala

Response(
  status = Status.Ok,
  headers = Headers("content-type" -> "application/json"),
  body = Body.fromString("""{"message":"ok"}""", Charset.UTF8),
  version = Version.`HTTP/1.1`
)
```

### Response Convenience Constructors

Quick constructors for common responses:

```scala

Response.ok                                              // 200 OK response value
Response.ok.addHeader("content-type", "application/json") // With headers
Response(Status.NotFound)                              // 404 Not Found
Response(Status.InternalServerError)                   // 500 Internal Server Error
```

### Response Operations

Access and modify response components:

```scala

val body = Body.fromString("""{"data":"example"}""", Charset.UTF8)
val resp = Response.ok.addHeader("content-type", "application/json")

resp.status                     // Status.Ok
resp.headers                    // Headers
resp.body                       // Body
resp.version                    // Version

val modified = resp.copy(status = Status.Created) // Change status code
modified.addHeader("cache-control", "no-cache")  // Add header
```

---

## Integration Points

The HTTP model types integrate with each other in a natural composition hierarchy:

- **Request & Response** are the top-level message types, containing all other types
- **URL** decomposes into Scheme, Host, Port, Path, QueryParams, and Fragment
- **Headers** is a collection of individual Header objects (generic strings and typed headers)
- **Body** carries ContentType metadata describing its content
- **Method & Status** are enumerations representing HTTP verbs and response codes
- **Version** describes the HTTP protocol version being used
- **Cookies** (Request & Response variants) appear as header values in Cookie and Set-Cookie headers
- **Form** is serialized as a Body with `application/x-www-form-urlencoded` content type

## Advanced Usage

### Building a Complete HTTP Exchange

Here's how to build a full request and response for a real scenario:

```scala

// Build request
val url = URL.parse("https://api.example.com/users").toOption.get
val requestBody = Body.fromString("""{"name":"Alice","age":30}""", Charset.UTF8)

val request = Request(
  method = Method.POST,
  url = url,
  headers = Headers(
    "content-type" -> "application/json",
    "authorization" -> "Bearer abc123",
    "user-agent" -> "MyClient/1.0"
  ),
  body = requestBody,
  version = Version.`HTTP/1.1`
)

// Build response
val responseBody = Body.fromString("""{"id":123,"name":"Alice","age":30}""", Charset.UTF8)

val response = Response(
  status = Status.Created,
  headers = Headers(
    "content-type" -> "application/json",
    "location" -> "/users/123"
  ),
  body = responseBody,
  version = Version.`HTTP/1.1`
)
```

### URL Building with Fluent API

Compose complex URLs using operators:

```scala

val url = URL.parse("https://api.example.com").toOption.get

val extended = (url / "v1" / "users" / "123") ?? ("include", "profile") ?? ("include", "posts")

extended.encode
// "https://api.example.com/v1/users/123?include=profile&include=posts"
```

### Cookie Management

Parse and render cookies in requests and responses:

```scala

// Parse cookies from request header
val cookieHeader = "session=abc; theme=dark"
val requestCookies = Cookie.parseRequest(cookieHeader)

// Create response with Set-Cookie headers
val sessionCookie = ResponseCookie(
  name = "session",
  value = "xyz123",
  path = Some(Path("/")),
  maxAge = Some(3600),
  isSecure = true,
  isHttpOnly = true,
  sameSite = Some(SameSite.Strict)
)

val response = Response(
  status = Status.Ok,
  headers = Headers(
    "set-cookie" -> Cookie.renderResponse(sessionCookie)
  ),
  body = Body.empty
)
```

### Form Submission

Build and submit HTML forms:

```scala

val form = Form(
  "username" -> "alice",
  "password" -> "secret",
  "remember" -> "true"
)

val formBody = Body.fromString(form.encode, Charset.UTF8)

val request = Request(
  method = Method.POST,
  url = URL.parse("/login").toOption.get,
  headers = Headers(
    "content-type" -> "application/x-www-form-urlencoded"
  ),
  body = formBody,
  version = Version.`HTTP/1.1`
)
```

## Design Principles

### Single Encoding Contract

`Path` and `QueryParams` store decoded values internally. Encoding happens only at output boundaries:

- `Path.fromEncoded(s)` decodes, stores decoded segments
- `Path.encode` encodes segments for transmission
- `QueryParams.fromEncoded(s)` decodes, stores decoded key-value pairs
- `QueryParams.encode` encodes for transmission

This eliminates double-encoding bugs and clarifies responsibilities.

### Lazy Header Parsing

Imagine you're building a high-traffic microservice that receives thousands of HTTP requests per second. Each request arrives with 20–30 headers: `content-type`, `authorization`, `cache-control`, `etag`, `x-request-id`, `x-trace-id`, custom headers your team added, and more.

Here's the problem: Your handler only *actually needs* three of those headers:

```scala

def describe(request: Request): String = {
  val authToken   = request.headers.get(Header.Authorization)
  val contentType = request.headers.get(Header.ContentType)
  val requestId   = request.headers.rawGet("x-request-id")

  // The other 17+ headers? Never touched.
  s"$authToken $contentType $requestId"
}
```

If the `Headers` class eagerly parsed all 20+ headers the moment the request arrived, you'd be wasting CPU time parsing headers you don't care about. With 1,000 requests/second, that's parsing 20,000 unnecessary headers per second.

**http-model's solution: Parse headers only when you ask for them.**

Internally, `Headers` stores header names and values as raw strings. When you call `Headers#get(Header.Authorization)` for the *first time*, it parses that specific header value into a typed structure (extracting charset, splitting directives, validating format, etc.), then *caches* the result. The second time you ask for the same header, it returns the cached parsed value instantly — no re-parsing.

```scala

val headers = Headers(
  "content-type"  -> "application/json; charset=utf-8",
  "authorization" -> "Bearer abc123xyz",
  "cache-control" -> "no-cache, max-age=3600"
  // ... 17 more headers
)

// First access: parses the "content-type" string and caches the parsed Header.ContentType
val ct1 = headers.get(Header.ContentType)

// Second access: returns the cached parsed result (no parsing)
val ct2 = headers.get(Header.ContentType)

// Other headers never accessed? Never parsed.
```

This design shines in three ways:

**Performance**: Only pay the cost for headers you actually use. With 20 headers and using 3, that's an ~85% reduction in parsing work. At scale (thousands of requests/second), this savings compounds dramatically.

**Robustness**: Custom or unknown headers don't cause parsing failures. If a header can't be parsed, it stays as a raw string, and your code can still access it as a plain value without crashing the handler.

**Simplicity**: Your code is clean — you just ask for headers by name, and http-model handles parsing transparently. No manual string manipulation or error handling on your end.

### Stream-Backed Bodies

Let's say you're downloading a 500MB video file over HTTP. Should your `Body` object represent that as:

**Option A: A single `Chunk[Byte]` with all 500MB in memory?**

```scala
val body = Body.fromChunk(Chunk[Byte](/* 500MB of bytes */))
// Everything loaded into RAM at once
```

**Option B: A Stream that yields bytes incrementally as they arrive?**

```scala
val byteStream: Stream[Nothing, Byte] = /* yields bytes as they download */
val body = Body.fromStream(byteStream)
// Only a small buffer in RAM; the rest comes from the network
```

Most HTTP libraries choose Option B for large files — streaming makes sense when you want to process data *as it arrives* without loading everything into memory first.

**http-model chooses a stream-backed body with runtime-independent asynchronous materialization.**

#### The Streaming Trade-off

Streaming saves memory and allows processing to begin before the complete payload arrives, but collecting the complete body may need to wait for asynchronous input. `Body` exposes that operation as `Async`, rather than coupling the HTTP model to ZIO, Cats Effect, Scala Futures, or a particular HTTP runtime.

#### http-model's Choice: Stream-Backed Bodies

http-model wraps a `Stream[Nothing, Byte]` — a pull-based stream with cross-platform asynchronous materialization and no required effect runtime:

```scala
final class Body private (
  val stream: Stream[Nothing, Byte],
  val contentType: ContentType
)
```

Create bodies from chunks, arrays, strings, or streams:

```scala
val body = Body.fromChunk(
  Chunk.fromArray(myBytes),
  ContentType.`application/json`
)

// Accessing the data on JVM and Scala.js:
val chunk: Async[Chunk[Byte]] = body.toChunkAsync
val bytes: Async[Array[Byte]] = body.toArrayAsync
val decoded: Async[String] = body.asStringAsync(Charset.UTF8)
val decodedFromContentType: Async[String] = body.asStringFromContentTypeAsync
val text: Async[String] = body.textAsync
val len: Option[Long] = body.length
val raw: Stream[Nothing, Byte] = body.toStream
```

For chunk-backed bodies, materialization reuses the known chunk and `length` returns `Some(n)`. For a stream without known metadata, materialization consumes the stream and `length` returns `None`. The shared `Body.scala` defines both the five `Async` methods and their synchronous compatibility twins.

```
Your Application Code
        ↓
     (uses)
        ↓
┌─────────────────────┐
│   HTTP Client       │ (ZIO HTTP, Akka HTTP, etc.)
│   (does streaming)  │
└─────────────────────┘
        ↓
    (wraps/unwraps)
        ↓
┌─────────────────────┐
│  http-model Body    │ (stream-backed, async materialization)
│  (pull-based I/O)   │
└─────────────────────┘
```

Body remains independent of any HTTP or effect runtime, but materializing an opaque stream can be asynchronous. Prefer `toChunkAsync`, `toArrayAsync`, `asStringAsync`, `asStringFromContentTypeAsync`, or `textAsync` for streams that can suspend. The synchronous twins remain available on both platforms for compatibility and synchronously completing streams.

## Running the Examples

All code from this guide is available as runnable examples in the `http-model-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

### Basic HTTP Request/Response

Demonstrates creating HTTP requests and responses with URLs, methods, headers, and bodies. Shows how `Request`, `Response`, `Method`, `URL`, `Headers`, and `Body` types work together.

```bash
cd http-model-examples && sbt "runMain httpmodel.BasicHttpRequest"
```

### Headers and Query Parameters

Shows how to work with headers and query parameters in URLs and requests. Demonstrates how `Headers`, `QueryParams`, and `URL` types compose for extracting and manipulating HTTP metadata.

```bash
cd http-model-examples && sbt "runMain httpmodel.HeadersAndQueryParams"
```

### Form Submission and Cookies

Demonstrates handling form data and cookies using `Request`, `Response`, `Form`, and cookie types. Shows realistic form submission scenarios with proper content-type headers and cookie management.

```bash
cd http-model-examples && sbt "runMain httpmodel.FormAndCookies"
```

### Complete HTTP Exchange

Shows a realistic HTTP exchange scenario: creating a request with multiple headers and query parameters, sending it, and receiving a response with status codes and headers. Demonstrates all core types working together in a practical scenario.

```bash
cd http-model-examples && sbt "runMain httpmodel.CompleteHttpExchange"
```

---

## HeaderCodec and QueryCodec

`HeaderCodec[A]` and `QueryCodec[A]` encode a whole value of type `A` into `Headers` or `QueryParams` and decode it back. Both are derived from `Schema[A]` through a format — `DefaultHeaderFormat` or `DefaultQueryFormat` — which pairs a MIME type with the `Deriver` that builds instances. The types involved:

```scala
abstract class HeaderCodec[A] extends Codec[Headers, HeadersBuilder, A] {
  def encodeToHeaders(value: A): Headers
}

abstract class QueryCodec[A] extends Codec[QueryParams, QueryParamsBuilder, A] {
  def encodeToQueryParams(value: A): QueryParams
}

abstract class HeaderFormat[TC[A] <: HeaderCodec[A]](val mimeType: String, val deriver: Deriver[TC])
abstract class QueryFormat[TC[A] <: QueryCodec[A]](val mimeType: String, val deriver: Deriver[TC])
```

## Motivation

The extension classes on [the schema page](./schema.md) read one field at a time: `request.query[Int]("page")` pulls a single parameter and gives you an `Either`. That is the right shape when a handler wants two or three values out of a request.

It is the wrong shape when the request *is* a value. A search endpoint taking eight query parameters becomes eight extractions, eight error branches, and a constructor call that has to be kept in step with all of them. Adding a parameter means touching four places.

A codec collapses that to one call. Define the case class, derive a `QueryCodec` from its schema, and `QueryCodec#decode` gives you `Either[SchemaError, SearchRequest]` for the whole thing. Adding a field to the case class adds a parameter, with no other change — the same bargain `Schema` makes everywhere else in the library.

The two directions are symmetric, which matters for clients as much as servers: the same codec that decodes an incoming request encodes an outgoing one.

## Quick Showcase

Derive a codec from a schema, then round-trip a value through it:

```scala

final case class Search(page: Int, term: String, active: Boolean)

object Search {
  implicit val schema: Schema[Search] = Schema.derived[Search]
}

val codec = Schema[Search].derive(DefaultQueryFormat)
```

Encoding produces one parameter per field, named after the field:

```scala
codec.encodeToQueryParams(Search(2, "boots", active = true))
// res0: QueryParams = QueryParams((page,2), (term,boots), (active,true))
```

Decoding is the inverse, and reports a `SchemaError` rather than throwing:

```scala
codec.decode(codec.encodeToQueryParams(Search(2, "boots", active = true)))
// res1: Either[SchemaError, Search] = Right(
//   Search(page = 2, term = "boots", active = true)
// )
```

## Deriving a Codec

Both codecs are derived the same way: call `Schema#derive` with the format object. The implicit `Derivable` that each format supplies is what lets a format stand in for its deriver:

```scala
trait Schema[A] {
  def derive[D, TC[_]](d: D)(implicit ev: Derivable[D, TC]): TC[A]
  def deriving[TC[_]](deriver: Deriver[TC]): DerivationBuilder[TC, A]
}
```

`DefaultQueryFormat` produces a `QueryCodec`, and `DefaultHeaderFormat` produces a `HeaderCodec`:

```scala

final case class Trace(traceId: String, apiKey: String)

object Trace {
  implicit val schema: Schema[Trace] = Schema.derived[Trace]
}

val headerCodec = Schema[Trace].derive(DefaultHeaderFormat)
```

The derived codec carries the format's MIME type, which is what an endpoint description uses to advertise the wire shape:

```scala
DefaultHeaderFormat.mimeType
// res3: String = "application/http-headers"
DefaultQueryFormat.mimeType
// res4: String = "application/x-www-form-urlencoded"
```

## Encoding and Decoding

Each codec has a convenience encoder that returns the finished collection, plus the `Codec#encode` and `Codec#decode` methods it inherits.

### `HeaderCodec#encodeToHeaders` and `QueryCodec#encodeToQueryParams`

These build and return the collection directly, which is what application code normally wants:

```scala
abstract class HeaderCodec[A] {
  def encodeToHeaders(value: A): Headers
}
```

Encoding a record yields one entry per field:

```scala
headerCodec.encodeToHeaders(Trace("trace-1", "secret"))
// res5: Headers = Headers(trace-id: trace-1, api-key: secret)
```

:::warning[The builder is a reused thread-local]
`HeaderCodec#encodeToHeaders` and `QueryCodec#encodeToQueryParams` borrow a per-thread builder, reset it, fill it, and snapshot it. The returned collection is safe to keep, but the builder is not reentrant: if a custom codec's `Codec#encode` calls `HeaderCodec#encodeToHeaders` again on the same thread, the inner call resets the buffer the outer one was filling. Inside a custom `Codec#encode`, write to the `output` builder you were handed rather than calling the convenience encoder.
:::

### `Codec#encode` and `Codec#decode`

`Codec#encode` writes into a builder you supply, which is how nested and composed codecs avoid intermediate collections. `Codec#decode` reads a whole collection:

```scala
abstract class QueryCodec[A] {
  def encode(value: A, output: QueryParamsBuilder): Unit
  def decode(input: QueryParams): Either[SchemaError, A]
}
```

A decode failure names the field and what was expected:

```scala

final case class Search(page: Int, term: String)

object Search {
  implicit val schema: Schema[Search] = Schema.derived[Search]
}

val codec = Schema[Search].derive(DefaultQueryFormat)
```

A missing parameter and a malformed one are both reported as a `SchemaError`, not an exception:

```scala
codec.decode(QueryParams("term" -> "boots"))
// res7: Either[SchemaError, Search] = Left(
//   SchemaError(
//     List(MissingField(source = DynamicOptic(ArraySeq()), fieldName = "page"))
//   )
// )
codec.decode(QueryParams("page" -> "not-a-number", "term" -> "boots"))
// res8: Either[SchemaError, Search] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(ArraySeq()),
//         details = "Malformed query parameter 'page' value 'not-a-number': Cannot parse 'not-a-number' as Int",
//         cause = None
//       )
//     )
//   )
// )
```

## Field Naming

The two codecs disagree about names, and this is the single most important thing to know about them.

| | Field `traceId` becomes | Decoding |
| ------------- | ----------------------- | ---------------- |
| `QueryCodec`  | `traceId` — verbatim     | exact match       |
| `HeaderCodec` | `trace-id` — kebab-case  | case-insensitive  |

`QueryCodec` uses the Scala field name unchanged, because query parameters are conventionally camelCase or snake_case and the schema's name is as good a guess as any.

`HeaderCodec` rewrites camelCase into kebab-case, because that is the conventional shape of an HTTP header name. Two steps are involved: the deriver emits `TRACE-ID`, uppercasing every character and inserting a hyphen before each interior capital, and then `Headers` lowercases every name as it stores it. The second step makes the first unobservable, so what you get back is `trace-id`:

```scala

final case class Meta(traceId: String, bigIntValue: BigInt, uuidValue: java.util.UUID)

object Meta {
  implicit val schema: Schema[Meta] = Schema.derived[Meta]
}

val codec = Schema[Meta].derive(DefaultHeaderFormat)
val uuid  = java.util.UUID.fromString("123e4567-e89b-12d3-a456-426614174000")
```

Every camelCase boundary becomes a hyphen, so `bigIntValue` becomes three segments rather than two:

```scala
codec.encodeToHeaders(Meta("trace-1", BigInt(42), uuid)).toList.map(_._1)
// res10: List[String] = List("trace-id", "big-int-value", "uuid-value")
```

Decoding ignores case, so a client that sends `trace-id` or `TRACE-ID` still decodes:

```scala
codec.decode(Headers("trace-id" -> "t", "big-int-value" -> "42", "UUID-VALUE" -> uuid.toString))
// res11: Either[SchemaError, Meta] = Right(
//   Meta(
//     traceId = "t",
//     bigIntValue = 42,
//     uuidValue = 123e4567-e89b-12d3-a456-426614174000
//   )
// )
```

:::warning[Names are not configurable]
Neither codec offers a name mapper. The transformation is fixed, so a header or parameter whose wire name does not follow the convention cannot be reached by renaming the field — you need a custom codec instance for that type, or the field-at-a-time extraction on [the schema page](./schema.md).
:::

## Supported Field Shapes

Four shapes are supported per field, and they compose one level deep.

### Primitives

Each primitive field is one entry, rendered as its string form:

| Scala type | Example wire value |
| --- | --- |
| `String` | `alice` |
| `Boolean` | `true` |
| `Byte`, `Short`, `Int`, `Long` | `42` |
| `Float`, `Double` | `12.34` |
| `BigInt`, `BigDecimal` | `1234567890123456789` |
| `Char` | `z` — exactly one character |
| `UUID` | `123e4567-e89b-12d3-a456-426614174000` |

A `Char` field with more than one character fails to decode, and the message says so:

```scala

val charCodec = Schema[Char].derive(DefaultQueryFormat)
```

The error names the expectation rather than the underlying exception:

```scala
charCodec.decode(QueryParams("value" -> "too-long"))
// res13: Either[SchemaError, Char] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(ArraySeq()),
//         details = "Malformed query parameter 'value' value 'too-long': Expected single character but got 'too-long'",
//         cause = None
//       )
//     )
//   )
// )
```

### Optional Fields

An `Option[A]` field omits its key entirely when `None`, and an absent key decodes back to `None`:

```scala

final case class Filters(page: Option[Int], term: Option[String])

object Filters {
  implicit val schema: Schema[Filters] = Schema.derived[Filters]
}

val codec = Schema[Filters].derive(DefaultQueryFormat)
```

Encoding drops the absent field rather than emitting an empty value:

```scala
codec.encodeToQueryParams(Filters(Some(7), None))
// res15: QueryParams = QueryParams((page,7))
```

An empty collection decodes to all-`None`, so a request with no parameters is valid rather than an error:

```scala
codec.decode(QueryParams.empty)
// res16: Either[SchemaError, Filters] = Right(
//   Filters(page = None, term = None)
// )
```

### Sequences

A `List`, `Chunk`, or other sequence field becomes **repeated entries under one key**, not a delimited single value:

```scala

final case class Selection(tags: List[String], ids: Chunk[Int])

object Selection {
  implicit val schema: Schema[Selection] = Schema.derived[Selection]
}

val codec = Schema[Selection].derive(DefaultQueryFormat)
```

Two tags and three ids produce five parameters:

```scala
codec.encodeToQueryParams(Selection(List("a", "b"), Chunk(1, 2, 3)))
// res18: QueryParams = QueryParams((tags,a), (tags,b), (ids,1), (ids,2), (ids,3))
```

Decoding collects them back into the declared collection type, preserving order:

```scala
codec.decode(codec.encodeToQueryParams(Selection(List("a", "b"), Chunk(1, 2, 3))))
// res19: Either[SchemaError, Selection] = Right(
//   Selection(tags = List("a", "b"), ids = IndexedSeq(1, 2, 3))
// )
```

Because there is no delimiter, a value containing a comma round-trips correctly — the repeated-key encoding has no escaping problem to solve.

### Wrapper Types

A newtype built with `Schema#transform` encodes as its underlying value, so the wrapper is invisible on the wire:

```scala

final case class UserId(value: String)

object UserId {
  implicit val schema: Schema[UserId] = Schema[String].transform(UserId(_), _.value)
}

final case class Owned(id: UserId)

object Owned {
  implicit val schema: Schema[Owned] = Schema.derived[Owned]
}

val codec = Schema[Owned].derive(DefaultQueryFormat)
```

The parameter holds the wrapped string, with no trace of the wrapper:

```scala
codec.encodeToQueryParams(Owned(UserId("user-1")))
// res21: QueryParams = QueryParams((id,user-1))
```

If the wrapping function throws — a validating newtype rejecting its input — the failure surfaces as a `SchemaError` from `QueryCodec#decode` rather than as an exception.

## Top-Level Codecs

A schema that is not a record also derives, and the resulting codec uses the single key `value`:

```scala

val intCodec  = Schema[Int].derive(DefaultQueryFormat)
val listCodec = Schema[List[Int]].derive(DefaultQueryFormat)
```

A top-level primitive is one parameter, and a top-level sequence is repeated parameters, both under `value`:

```scala
intCodec.encodeToQueryParams(42)
// res23: QueryParams = QueryParams((value,42))
listCodec.encodeToQueryParams(List(1, 2))
// res24: QueryParams = QueryParams((value,1), (value,2))
```

`HeaderCodec` applies its naming rule here too, so the single-word name comes back as `value` and is matched case-insensitively on the way in:

```scala
val headerIntCodec = Schema[Int].derive(DefaultHeaderFormat)
// headerIntCodec: HeaderCodec[Int] = zio.http.schema.HeaderCodecDeriver$$anon$3@69853922
headerIntCodec.encodeToHeaders(42).toList
// res25: List[Tuple2[String, String]] = List(("value", "42"))
headerIntCodec.decode(Headers("value" -> "42"))
// res26: Either[SchemaError, Int] = Right(42)
```

## Unsupported Shapes

Three shapes have no encoding, and one of them is a trap.

**Nested records** are rejected during derivation, because a flat namespace of names has no way to express a field whose value is itself a record. Use a flat case class, or extract the nested part separately.

**Top-level `Option`, `Map`, and `DynamicValue`** derive without complaint and then **throw when you encode**:

```scala

val optionCodec = Schema[Option[Int]].derive(DefaultQueryFormat)
```

Derivation succeeds, so nothing warns you at the point where the mistake was made:

```scala
scala.util.Try(optionCodec.encodeToQueryParams(Some(1))).isFailure
// res28: Boolean = true
```

:::danger[Failure is deferred to the first encode]
`Schema[Option[A]]`, `Schema[Map[K, V]]`, and `Schema[DynamicValue]` all produce a codec that throws on use. Because derivation is where the error belongs and is not where it appears, a codec built at startup can look healthy until the first request that encodes through it. Derive and exercise these codecs in a test rather than trusting that construction succeeded.
:::

A `Map` field inside a record is likewise unsupported. Model the entries you expect as fields, or take the raw collection and read it with the field-at-a-time API.

## Formats

A format is the pairing of a MIME type with a `Deriver`. `HeaderFormat` and `QueryFormat` are the base classes, and each provides an implicit `Derivable` so that `Schema#derive` accepts the format object directly:

```scala
abstract class HeaderFormat[TC[A] <: HeaderCodec[A]](val mimeType: String, val deriver: Deriver[TC]) {
  implicit def derivable: Derivable[this.type, TC]
}
```

The two built-in formats are the only ones needed for the default wire shapes:

| Format | MIME type | Deriver |
| --- | --- | --- |
| `DefaultHeaderFormat` | `application/http-headers` | `HeaderCodecDeriver` |
| `DefaultQueryFormat` | `application/x-www-form-urlencoded` | `QueryCodecDeriver` |

Both are `case object`s extending a sealed abstract class, so they are singletons and can be matched on.

### Defining a Custom Format

Subclass `QueryFormat` or `HeaderFormat` when you want a different MIME type reported for the same derivation strategy, or a different deriver entirely:

```scala

case object FormUrlEncoded extends QueryFormat[QueryCodec]("application/x-www-form-urlencoded; charset=utf-8", QueryCodecDeriver)
```

The `TC` parameter is the codec type the deriver produces, bounded by `QueryCodec` so that the convenience encoder stays available on whatever it derives.

## Overriding a Single Type

`HeaderCodecDeriver` and `QueryCodecDeriver` are ordinary `Deriver` instances, so the schema module's override mechanism applies. Supply a hand-written codec for one type and let derivation handle the rest:

```scala

val prefixedInt = new QueryCodec[Int] {
  def encode(value: Int, output: QueryParamsBuilder): Unit =
    output.add("value", s"custom-$value")

  def decode(input: QueryParams): Either[SchemaError, Int] =
    input.getFirst("value") match {
      case Some(s) if s.startsWith("custom-") => Right(s.stripPrefix("custom-").toInt)
      case other                              => Left(SchemaError(s"Expected custom- prefix, got: $other"))
    }
}

val codec = Schema[Int].deriving(QueryCodecDeriver).instance(TypeId.int, prefixedInt).derive
```

The override applies when the overridden type is the whole schema, so the custom rendering is used on both sides. A record's fields of that type ignore the override in 0.0.56, a known bug:

```scala
codec.encodeToQueryParams(42)
// res31: QueryParams = QueryParams((value,custom-42))
codec.decode(codec.encodeToQueryParams(42))
// res32: Either[SchemaError, Int] = Right(42)
```

Note that `Codec#encode` writes into the supplied builder rather than returning a collection — that is the method to implement, and the convenience encoder is derived from it.

## Choosing Between the Two APIs

Both this page's codecs and [the schema page](./schema.md)'s extension classes read the same collections. They differ in granularity:

| | Codecs | Extension classes |
| --- | --- | --- |
| Unit of work | Whole value | One field |
| Result | `Either[SchemaError, A]` | `Either[QueryParamError, T]` / `Either[HeaderError, T]` |
| Encoding | Yes, symmetric with decoding | Decoding only |
| Names | Fixed by the naming rule | You pass the name |
| Nested records | Unsupported | Not applicable |
| Best for | A request or response that *is* a value | Pulling two or three values out |

Use a codec when the collection maps onto a type you already have. Use the extension classes when it does not, when you need a name the codec cannot produce, or when you want per-field error handling.

## Integration Points

`HeaderCodec` and `QueryCodec` extend `Codec[Out, Builder, A]` from `zio-blocks-schema`, so they participate in the same derivation machinery as every other codec in the library — the `Deriver`, `Derivable`, and instance-override APIs are the schema module's, not this module's.

On the HTTP side they produce and consume `Headers`, `HeadersBuilder`, `QueryParams`, and `QueryParamsBuilder` from `zio-blocks-http-model`; see [the HTTP model](./model.md) for those collections and [Header](./headers.md) for the typed single-header model, which is a different mechanism from the whole-record codecs here.

Errors are `SchemaError` from the schema module rather than this module's `HeaderError` and `QueryParamError`, which belong to the field-at-a-time API on [the schema page](./schema.md).

For a worked example that decodes a filter object with a derived `QueryCodec` and turns its `SchemaError` into a 400 response, see [How to Validate Query Parameters and Headers and Answer 400](../../guides/validate-query-parameters-and-headers.md).

---

## Schema-Based Typed Access

`zio-blocks-http-model-schema` adds **type-safe, validated extraction** of query parameters and headers to the core HTTP model. It provides extension methods on `QueryParams`, `Headers`, `Request`, and `Response` that automatically decode string values to typed objects using schema-based decoding with comprehensive error reporting.

Core features are built on **extension methods** — `QueryParamsSchemaOps`, `HeadersSchemaOps`, `RequestSchemaOps`, `ResponseSchemaOps` — which add typed, schema-based extraction to query parameters and headers. Complemented by error types `QueryParamError` and `HeaderError`, the module provides automatic decoding for 12 primitive types (see [Supported Types](#supported-types)).

This page covers extraction one field at a time. The module also derives whole-value codecs — a `QueryCodec[A]` or `HeaderCodec[A]` that encodes and decodes an entire case class in one call — which are documented in [Codecs](./schema-codecs.md). A known bug in 0.0.56: a derived codec ignores field default values, so a field such as `limit: Int = 20` is still required. Model optional parameters with `Option` instead.

## Motivation

Building HTTP handlers often requires extracting and validating query parameters or headers — "get the `userId` query parameter as a `UUID`." Without schema-based extraction, this becomes tedious and error-prone:

```scala

// Manual extraction (error-prone, repetitive)
val params = QueryParams("userId" -> "550e8400-e29b-41d4-a716-446655440000")
val userIdStr = params.getFirst("userId")
val userId = userIdStr match {
  case None => Left("Missing userId")
  case Some(s) =>
    try Right(java.util.UUID.fromString(s))
    catch { case e: IllegalArgumentException => Left(s"Invalid UUID format: ${e.getMessage}") }
}
```

Every parameter requires 8+ lines of boilerplate with manual exception handling, error message creation, and type-specific parsing. UUID parsing alone involves `IllegalArgumentException` handling; multiply this across dozens of handlers extracting `UUID`, `Int`, `Boolean` parameters, and you have duplicated extraction logic everywhere — inconsistent error messages, risk of forgotten error handling, and no compile-time guarantees on correctness.

The solution is to use schema-based extraction for clean, declarative code:

```scala

val params = QueryParams("userId" -> "550e8400-e29b-41d4-a716-446655440000")
val userId = params.query[java.util.UUID]("userId")  // 1 line, automatic UUID parsing + errors
```

`zio-blocks-http-model-schema` separates **extraction logic from business logic**. The module achieves this through:

- **Automatic decoding** — Pass a `Schema[T]`, get `Either[Error, T]` back. Works for 12 primitive types out of the box. Other types compile but do not decode in 0.0.56; see [Supported Types](#supported-types).
- **Explicit error handling** — `Either` forces error handling. `QueryParamError` and `HeaderError` distinguish "missing" from "malformed" cases.
- **Composable** — Works directly on `QueryParams`, `Headers`, `Request`, `Response` with zero configuration.
- **Zero-dependency** — Pure extraction layer; doesn't pull in ZIO, async runtimes, or HTTP client libraries.

This keeps HTTP request handling clean, testable, and portable across different effect systems.

## Installation

Add the following to your `build.sbt`:

```
libraryDependencies += "dev.zio" %% "zio-blocks-http-model-schema" % "0.0.56"
```

For cross-platform projects (Scala.js):

```
libraryDependencies += "dev.zio" %%% "zio-blocks-http-model-schema" % "0.0.56"
```

Supported Scala versions: Scala 2.13 and Scala 3.x. Scala 2.13 and Scala 3.3 to 3.8 use `zio-blocks-http-model-schema`. On Scala 3.9 and newer, use `zio-blocks-next-http-model-schema` and add `scalacOptions += "-experimental"`, because the `zio-blocks-next-*` artifacts are compiled with `-experimental` (see [How to Add ZIO Blocks to Your Build](../../getting-started/installation.md)). It brings in `zio-blocks-http-model` and `zio-blocks-schema` transitively.

## How They Work Together

To understand how the module works, we add a **schema-based extraction layer** on top of core HTTP model types:

```
HTTP Model Types (from zio-blocks-http-model)
    ├─ QueryParams: raw string key-value pairs
    ├─ Headers: raw string header name-value pairs
    ├─ Request: contains queryParams and headers
    └─ Response: contains headers

Schema-Based Extraction (this module)
    ├─ QueryParamsSchemaOps.query[T](key) ─────┐
    ├─ HeadersSchemaOps.header[T](name) ───────┼──> StringDecoder.decode(raw, Schema[T])
    ├─ RequestSchemaOps.query[T](key) ─────────┤   ├─> Right(typedValue)
    ├─ request.headers.header[T](name) ────────┤   └─> Left(error)
    └─ response.headers.header[T](name) ───────┘

Typical Workflow:
1. Parse URL or receive Request (core HTTP model)
2. Extract queryParams or headers (access raw strings)
3. Use schema methods to decode to typed values (this module)
4. Handle Either[Error, T] in business logic
```

## Quick Showcase

Setting up and extracting query parameters with type safety:

```scala

val url = URL.parse("/api/users?page=2&limit=50&sort=name").toOption.get
// url: URL = URL(
//   scheme = None,
//   host = None,
//   port = None,
//   path = Path(
//     segments = IndexedSeq("api", "users"),
//     hasLeadingSlash = true,
//     trailingSlash = false
//   ),
//   queryParams = QueryParams((page,2), (limit,50), (sort,name)),
//   fragment = None
// )
val request = Request.get(url)
// request: Request = Request(
//   method = GET,
//   url = URL(
//     scheme = None,
//     host = None,
//     port = None,
//     path = Path(
//       segments = IndexedSeq("api", "users"),
//       hasLeadingSlash = true,
//       trailingSlash = false
//     ),
//     queryParams = QueryParams((page,2), (limit,50), (sort,name)),
//     fragment = None
//   ),
//   headers = Headers(),
//   body = Body(length=0, contentType=ContentType(MediaType(application,octet-stream,true,true,List(bin, dms, lrf, mar, so, dist, distz, pkg, bpk, dump, elc, deploy, exe, dll, deb, dmg, iso, img, msi, msp, msm, buffer),Map(),Map()),None,None)),
//   version = HTTP/1.1
// )

// Extract query parameters
val pageResult = request.query[Int]("page")
// pageResult: Either[QueryParamError, Int] = Right(2)
val limitResult = request.query[Int]("limit")
// limitResult: Either[QueryParamError, Int] = Right(50)
val sortResult = request.query[String]("sort")
// sortResult: Either[QueryParamError, String] = Right("name")

// Results are properly typed and decoded
(pageResult, limitResult, sortResult)
// res4: Tuple3[Either[QueryParamError, Int], Either[QueryParamError, Int], Either[QueryParamError, String]] = (
//   Right(2),
//   Right(50),
//   Right("name")
// )

// Handle errors with pattern matching
pageResult match {
  case Right(page) => s"Page: $page"
  case Left(QueryParamError.Missing(key)) => s"Missing $key"
  case Left(QueryParamError.Malformed(key, value, cause)) => s"Bad $key: $cause"
}
// res5: String = "Page: 2"
```

## Extension Classes

### QueryParamsSchemaOps

Extension methods for `QueryParams` to extract and decode query parameters with type safety.

#### `QueryParams#query[T]`

Extract a single query parameter value and decode it to type `T`.

**Signature:** `query[T](key: String): Either[QueryParamError, T]`

Returns `Right(value)` if parameter exists and decoding succeeds. Returns `Left(QueryParamError.Missing(key))` if parameter is missing. Returns `Left(QueryParamError.Malformed(...))` if parameter exists but decoding fails. Decoding works for primitive schemas only; see [Supported Types](#supported-types).

When a query parameter is required, use `query[T]` and handle the error:

```scala

val url = URL.parse("/search?q=zio").toOption.get
// url: URL = URL(
//   scheme = None,
//   host = None,
//   port = None,
//   path = Path(
//     segments = IndexedSeq("search"),
//     hasLeadingSlash = true,
//     trailingSlash = false
//   ),
//   queryParams = QueryParams((q,zio)),
//   fragment = None
// )
val params = url.queryParams
// params: QueryParams = QueryParams((q,zio))

params.query[String]("q") match {
  case Right(q) => s"Search for: $q"
  case Left(error) => s"Error: ${error.message}"
}
// res7: String = "Search for: zio"
```

#### `QueryParams#queryAll[T]`

Extract all values for a query parameter key and decode them to type `T`.

**Signature:** `queryAll[T](key: String): Either[QueryParamError, Chunk[T]]`

Returns `Right(chunk)` with all decoded values if parameter exists and all values decode successfully. Returns `Left(QueryParamError.Missing(key))` if no values exist for the key. Returns `Left(QueryParamError.Malformed(...))` if any value fails to decode.

**Pattern: Extract Multiple Values for Same Parameter**

When a query parameter appears multiple times (e.g., `?tag=scala&tag=fp`), use `queryAll[T]`:

```scala

val url = URL.parse("/search?tag=scala&tag=functional&tag=zio").toOption.get
// url: URL = URL(
//   scheme = None,
//   host = None,
//   port = None,
//   path = Path(
//     segments = IndexedSeq("search"),
//     hasLeadingSlash = true,
//     trailingSlash = false
//   ),
//   queryParams = QueryParams((tag,scala), (tag,functional), (tag,zio)),
//   fragment = None
// )
val params = url.queryParams
// params: QueryParams = QueryParams((tag,scala), (tag,functional), (tag,zio))
```

Extract all values for a multi-valued parameter:

```scala
params.queryAll[String]("tag") match {
  case Right(tags) => s"Tags: ${tags.toList}"
  case Left(error) => s"Error: ${error.message}"
}
// res9: String = "Tags: List(scala, functional, zio)"
```

**Short-circuit behavior:** Decoding stops at the first malformed value; only the first error is reported.

#### `QueryParams#queryOrElse[T]`

Extract a query parameter with a default fallback.

**Signature:** `queryOrElse[T](key: String, default: => T): T`

Returns the decoded value if parameter exists and decodes successfully. Returns `default` if parameter is missing or decoding fails. A malformed value is ignored silently, so `queryOrElse[Int]("limit", 20)` returns `20` for `?limit=abc` and the client's mistake is never reported. Use `query[T]` and fall back only on `QueryParamError.Missing` when a malformed value must be an error.

**Pattern: Extract with Default Fallback**

When a query parameter is optional with a sensible default, use `queryOrElse`:

```scala

val url = URL.parse("/api/items?page=2").toOption.get
// url: URL = URL(
//   scheme = None,
//   host = None,
//   port = None,
//   path = Path(
//     segments = IndexedSeq("api", "items"),
//     hasLeadingSlash = true,
//     trailingSlash = false
//   ),
//   queryParams = QueryParams((page,2)),
//   fragment = None
// )
val params = url.queryParams
// params: QueryParams = QueryParams((page,2))
```

Extract with fallback defaults:

```scala
val page = params.queryOrElse[Int]("page", 1)
// page: Int = 2
val limit = params.queryOrElse[Int]("limit", 20)
// limit: Int = 20
(page, limit)
// res11: Tuple2[Int, Int] = (2, 20)
```

### HeadersSchemaOps

Extension methods for `Headers` to extract and decode header values with type safety. Uses `rawGet`/`rawGetAll` internally for raw string header access. The methods mirror those of `QueryParamsSchemaOps`, with `name` in place of `key`.

#### `Headers#header[T]`

Extract a single header value and decode it to type `T`.

**Signature:** `header[T](name: String): Either[HeaderError, T]`

Header name matching is **case-insensitive** (HTTP spec). Returns `Right(value)` on success, `Left(HeaderError.Missing(name))` if header not found, or `Left(HeaderError.Malformed(...))` if decoding fails.

Here's how to use `header[T]`:

```scala

val headers = Headers("x-user-id" -> "42", "x-api-version" -> "2")
// headers: Headers = Headers(x-user-id: 42, x-api-version: 2)
```

Calling `header[T]` returns an `Either` with the decoded value or an error:

```scala
headers.header[Int]("x-user-id")
// res13: Either[HeaderError, Int] = Right(42)
```

Header names are **case-insensitive**:

```scala
headers.header[Int]("X-User-ID")
// res14: Either[HeaderError, Int] = Right(42)
```

Missing headers produce a `Missing` error:

```scala
headers.header[Int]("x-missing")
// res15: Either[HeaderError, Int] = Left(Missing("x-missing"))
```

You can also decode with a custom `Header.Codec[A]` when the header type belongs to your domain instead of the core HTTP model:

```scala

object TraceIdHeader extends Header.Codec[String] {
  def name: String = "x-trace-id"
  def parse(value: String): Either[String, String] =
    if (value.startsWith("trace-")) Right(value) else Left("trace id must start with trace-")
  def render(value: String): String = value
}

val customHeaders = Headers("x-trace-id" -> "trace-123")
// customHeaders: Headers = Headers(x-trace-id: trace-123)
customHeaders.header(TraceIdHeader)
// res16: Either[HeaderError, String] = Right("trace-123")
```

#### `Headers#headerAll[T]`

Extract all values for a header name and decode them to type `T`.

**Signature:** `headerAll[T](name: String): Either[HeaderError, Chunk[T]]`

HTTP allows multiple headers with the same name; this method collects and decodes all of them. Returns `Right(chunk)` with all decoded values, `Left(HeaderError.Missing(name))` if no headers exist for the name, or `Left(HeaderError.Malformed(...))` if any value fails to decode.

Here's how to extract multiple headers:

```scala

val headers = Headers("x-tag" -> "scala", "x-tag" -> "functional", "x-tag" -> "zio")
// headers: Headers = Headers(x-tag: scala, x-tag: functional, x-tag: zio)
```

Extract all values for a header:

```scala
headers.headerAll[String]("x-tag")
// res18: Either[HeaderError, Chunk[String]] = Right(
//   IndexedSeq("scala", "functional", "zio")
// )
```

Missing headers return a `Missing` error:

```scala
headers.headerAll[String]("x-missing")
// res19: Either[HeaderError, Chunk[String]] = Left(Missing("x-missing"))
```

The codec-based overload works for repeated headers as well and reports the first malformed value as `HeaderError.Malformed`:

```scala

object TraceIdHeader extends Header.Codec[String] {
  def name: String = "x-trace-id"
  def parse(value: String): Either[String, String] =
    if (value.startsWith("trace-")) Right(value) else Left("trace id must start with trace-")
  def render(value: String): String = value
}

val repeatedHeaders = Headers("x-trace-id" -> "trace-1", "x-trace-id" -> "trace-2")
// repeatedHeaders: Headers = Headers(x-trace-id: trace-1, x-trace-id: trace-2)
repeatedHeaders.headerAll(TraceIdHeader)
// res20: Either[HeaderError, Chunk[String]] = Right(
//   IndexedSeq("trace-1", "trace-2")
// )
```

#### `Headers#headerOrElse[T]`

Extract a header with a default fallback. Like `queryOrElse`, it returns `default` for a malformed value as well as a missing one, without reporting the malformed value.

**Signature:** `headerOrElse[T](name: String, default: => T): T`

Use `headerOrElse[T]` when a header is optional with a sensible default:

```scala

val headers = Headers("x-count" -> "5")
// headers: Headers = Headers(x-count: 5)
```

When the header exists, it's decoded and returned:

```scala
headers.headerOrElse[Int]("x-count", 0)
// res22: Int = 5
```

When missing, the default is used:

```scala
headers.headerOrElse[Int]("x-missing", 0)
// res23: Int = 0
```

### RequestSchemaOps

Extension methods for `Request` to extract query parameters and headers using the same schema-based API.

Exposes the query methods from `QueryParamsSchemaOps` and the header methods from `HeadersSchemaOps` on `Request`. The query methods and `headerAll`/`headerOrElse` work as they do on `QueryParams` and `Headers`; `header[T]` does not (see the caution below).

:::caution[Known bug in 0.0.56]
`request.header[T](name)` does not compile. `Request` already has a `header` method that takes a `Header.Codec`, and Scala uses that member instead of the extension, so the call fails with "Required: zio.http.Header.Codec[T]". Call the typed accessor on the headers instead: `request.headers.header[T](name)`. `request.headerAll[T]` and `request.headerOrElse[T]` do not clash and work on `Request`. The examples below use `request.headers`.
:::

Query parameters and headers are extracted identically; use `request.query[T]` for a query parameter and `request.headers.header[T]` or `request.headers.headerAll[T]` for a header:

```scala

val request = Request.get(URL.parse("/").toOption.get)
  .addHeader("x-user-id", "42")
  .addHeader("x-api-version", "2")
// request: Request = Request(
//   method = GET,
//   url = URL(
//     scheme = None,
//     host = None,
//     port = None,
//     path = Path(
//       segments = IndexedSeq(),
//       hasLeadingSlash = true,
//       trailingSlash = false
//     ),
//     queryParams = QueryParams(),
//     fragment = None
//   ),
//   headers = Headers(x-user-id: 42, x-api-version: 2),
//   body = Body(length=0, contentType=ContentType(MediaType(application,octet-stream,true,true,List(bin, dms, lrf, mar, so, dist, distz, pkg, bpk, dump, elc, deploy, exe, dll, deb, dmg, iso, img, msi, msp, msm, buffer),Map(),Map()),None,None)),
//   version = HTTP/1.1
// )
```

Extract headers from the request using the headers API:

```scala
val userId = request.headers.header[Int]("x-user-id")
// userId: Either[HeaderError, Int] = Right(42)
val apiVersion = request.headers.headerOrElse[Int]("x-api-version", 1)
// apiVersion: Int = 2
(userId, apiVersion)
// res25: Tuple2[Either[HeaderError, Int], Int] = (Right(42), 2)
```

### ResponseSchemaOps

Extension methods for `Response` to extract headers using the schema-based API.

Exposes the header methods from `HeadersSchemaOps` on `Response`, except that `header[T]` does not compile (see below). Note: `Response` does not have `query*` methods (responses don't have query parameters). As with `Request`, `response.header[T](name)` does not compile in 0.0.56 (the same known bug), so use `response.headers.header[T](name)`.

`Response` provides the same header extraction methods through its headers:

```scala

val response = Response.ok
  .addHeader("x-request-id", "req-12345")
  .addHeader("x-ratelimit-remaining", "99")
// response: Response = Response(
//   status = 200,
//   headers = Headers(x-request-id: req-12345, x-ratelimit-remaining: 99),
//   body = Body(length=0, contentType=ContentType(MediaType(application,octet-stream,true,true,List(bin, dms, lrf, mar, so, dist, distz, pkg, bpk, dump, elc, deploy, exe, dll, deb, dmg, iso, img, msi, msp, msm, buffer),Map(),Map()),None,None)),
//   version = HTTP/1.1
// )
```

Extract a header from the response using the headers API:

```scala
response.headers.header[String]("x-request-id")
// res27: Either[HeaderError, String] = Right("req-12345")
```

Use a default if the header is missing:

```scala
response.headers.headerOrElse[Int]("x-ratelimit-remaining", 100)
// res28: Int = 99
```

## Composing Multiple Extractions

Extract multiple parameters or headers in a single operation using `Either`'s monadic operations:

```scala

val request = Request.get(URL.parse("/api/posts?userId=5&page=2").toOption.get)
// request: Request = Request(
//   method = GET,
//   url = URL(
//     scheme = None,
//     host = None,
//     port = None,
//     path = Path(
//       segments = IndexedSeq("api", "posts"),
//       hasLeadingSlash = true,
//       trailingSlash = false
//     ),
//     queryParams = QueryParams((userId,5), (page,2)),
//     fragment = None
//   ),
//   headers = Headers(),
//   body = Body(length=0, contentType=ContentType(MediaType(application,octet-stream,true,true,List(bin, dms, lrf, mar, so, dist, distz, pkg, bpk, dump, elc, deploy, exe, dll, deb, dmg, iso, img, msi, msp, msm, buffer),Map(),Map()),None,None)),
//   version = HTTP/1.1
// )
```

Combine multiple extractions with a for-comprehension:

```scala
val result = for {
  userId <- request.query[Int]("userId")
  page <- request.query[Int]("page")
} yield (userId, page)
// result: Either[QueryParamError, Tuple2[Int, Int]] = Right((5, 2))
```

Handle the combined result:

```scala
result match {
  case Right((userId, page)) => s"User $userId, page $page"
  case Left(error) => s"Extraction failed: ${error.message}"
}
// res30: String = "User 5, page 2"
```

The for-comprehension short-circuits on the first error, so only the first error is reported if any extraction fails. This pattern is useful when you need multiple parameters to be present and valid before proceeding with business logic.

## Error Handling

The module provides two error types for explicit error handling: `QueryParamError` and `HeaderError`.

### QueryParamError

Error type for query parameter extraction failures:

```scala
sealed trait QueryParamError extends Product with Serializable {
  def message: String
}

object QueryParamError {
  final case class Missing(key: String) extends QueryParamError {
    def message: String = s"Missing query parameter: $key"
  }
  final case class Malformed(key: String, value: String, cause: String) extends QueryParamError {
    def message: String = s"Malformed query parameter '$key' value '$value': $cause"
  }
}
```

**Variants:**

- **`Missing(key)`** — Query parameter with name `key` is not present in the parameters
  - Example: `QueryParamError.Missing("page")` when accessing a non-existent parameter
  - Message: `"Missing query parameter: page"`

- **`Malformed(key, value, cause)`** — Query parameter with name `key` is present but decoding the `value` to the requested type fails
  - Example: `QueryParamError.Malformed("age", "abc", "Cannot parse 'abc' as Int")` when `age=abc` but `Int` was requested
  - Message: `"Malformed query parameter 'age' value 'abc': Cannot parse 'abc' as Int"`

**Accessing error messages:**

All `QueryParamError` subtypes have a `message` property for user-friendly error reporting:

```scala

val error: QueryParamError = QueryParamError.Malformed("page", "invalid", "Cannot parse 'invalid' as Int")
// error: QueryParamError = Malformed(
//   key = "page",
//   value = "invalid",
//   cause = "Cannot parse 'invalid' as Int"
// )
```

The message provides detailed error information:

```scala
error.message
// res33: String = "Malformed query parameter 'page' value 'invalid': Cannot parse 'invalid' as Int"
```

### HeaderError

Error type for header extraction failures. Structurally identical to `QueryParamError`, with `name` replacing `key` and header-specific message prefixes:

```scala
sealed trait HeaderError extends Product with Serializable {
  def message: String
}

object HeaderError {
  final case class Missing(name: String) extends HeaderError {
    def message: String = s"Missing header: $name"
  }
  final case class Malformed(name: String, value: String, cause: String) extends HeaderError {
    def message: String = s"Malformed header '$name' value '$value': $cause"
  }
}
```

**Handling patterns:**

Pattern-match on error type to distinguish "missing" from "malformed":

```scala

val request = Request.get(URL.parse("/").toOption.get)
  .addHeader("x-token", "invalid-token")
// request: Request = Request(
//   method = GET,
//   url = URL(
//     scheme = None,
//     host = None,
//     port = None,
//     path = Path(
//       segments = IndexedSeq(),
//       hasLeadingSlash = true,
//       trailingSlash = false
//     ),
//     queryParams = QueryParams(),
//     fragment = None
//   ),
//   headers = Headers(x-token: invalid-token),
//   body = Body(length=0, contentType=ContentType(MediaType(application,octet-stream,true,true,List(bin, dms, lrf, mar, so, dist, distz, pkg, bpk, dump, elc, deploy, exe, dll, deb, dmg, iso, img, msi, msp, msm, buffer),Map(),Map()),None,None)),
//   version = HTTP/1.1
// )
```

When you extract a header with the wrong type, you get a `Malformed` error:

```scala
request.headers.header[Int]("x-token") match {
  case Right(token) => s"Token: $token"
  case Left(HeaderError.Missing(name)) => s"Missing required header: $name"
  case Left(HeaderError.Malformed(name, value, cause)) => s"Bad header: $cause"
}
// res35: String = "Bad header: Cannot parse 'invalid-token' as Int"
```

## Supported Types

The decoders in this module handle primitive schemas. `query[T]` and `header[T]` accept any `Schema[T]` at compile time, but in 0.0.56 a type that is not a primitive, such as `Option[T]`, a `List[T]` or a wrapper built with `Schema#transform`, always returns `Malformed` with the message "Unsupported schema type for string decoding", even for a valid value such as `?limit=5`. This is a known bug. Decode the primitive and convert it yourself, or derive a [`QueryCodec` or `HeaderCodec`](./schema-codecs.md), which handles `Option` fields and wrappers.

The decoders also parse only. They do not check the `Validation` a schema can carry, so a `Schema[Int]` with a range of 1 to 100 still decodes `?limit=500` to `500`. Check ranges after decoding. This is a known bug in 0.0.56, and the same applies to `queryOrElse` and `headerOrElse`.

Built-in primitive support includes:

### Primitives

- **`String`** — No decoding, raw string value
- **`Int`** — Parsed via `String#toInt`, error on invalid format
- **`Long`** — Parsed via `String#toLong`, error on invalid format
- **`Boolean`** — Parsed via `String#toBoolean` (case-insensitive; accepts "true"/"True"/"TRUE" → true and "false"/"False"/"FALSE" → false; any other value produces a Malformed error)
- **`Double`** — Parsed via `String#toDouble`, error on invalid format
- **`Float`** — Parsed via `String#toFloat`, error on invalid format
- **`Short`** — Parsed via `String#toShort`, error on invalid format
- **`Byte`** — Parsed via `String#toByte`, error on invalid format
- **`Char`** — Parses single character; returns a `Left` with error message `"Expected single character but got 'value'"` if string length ≠ 1 (differs from standard error pattern)

### Big Numbers

- **`BigInt`** — Parsed via `scala.BigInt(string)`, error on invalid format
- **`BigDecimal`** — Parsed via `scala.BigDecimal(string)`, error on invalid format

### UUID

- **`java.util.UUID`** — Parsed via `java.util.UUID.fromString(string)`, error on invalid format. On the JVM, `java.util.UUID.fromString` also accepts shortened groups such as `1-2-3-4-5`

### Error Messages

Most decoding errors follow the pattern: `"Cannot parse 'value' as TypeName"`. Example error messages:

Here are common error message formats:

```
Cannot parse 'abc' as Int
Cannot parse 'notaboolean' as Boolean
Cannot parse 'not-a-uuid' as UUID
Cannot parse '12.34.56' as BigDecimal
```

**Exception:** `Char` parsing uses a different error message format:

```
Expected single character but got 'multichar'
Expected single character but got ''
```

### Custom Types

In 0.0.56 only the primitive types listed above decode through `query[T]` and `header[T]`. A custom type, including one built with `Schema#transform`, returns `Malformed` (see [Supported Types](#supported-types)). To read a custom type, decode the primitive and convert it yourself, or derive a whole-value codec instead — see [Codecs](./schema-codecs.md), which maps a case class onto a whole collection of parameters or headers rather than a single value.

## See Also

- [How to Validate Query Parameters and Headers and Answer 400](../../guides/validate-query-parameters-and-headers.md) — A worked bookstore API that reads typed query parameters and headers, checks ranges, and answers bad input with a JSON 400 that names every problem
- [Codecs](./schema-codecs.md) — `QueryCodec` and `HeaderCodec`, derived from a schema, for encoding and decoding a whole value rather than one field
- [Combinators](../combinators.md) — Systematically compose and canonicalize Either types for uniform error handling across multiple query parameter or header extractions. The `Eithers` combinator canonicalizes nested Either types, making error accumulation consistent.

---

## ServerSentEvent

`ServerSentEvent[A]` is an immutable envelope for one Server-Sent Event: a payload of type `A` plus the three optional SSE metadata fields. `SseDataEncoder[A]` turns the payload into the `data:` lines of the wire format. The type is covariant in `A`, and the metadata fields are `Maybe` rather than `Option` to stay allocation-free when absent:

```scala
final class ServerSentEvent[+A] private (
  val data: A,
  val eventType: Maybe[String],
  val eventId: Maybe[String],
  val retryMillis: Maybe[Long]
)

trait SseDataEncoder[-A] {
  def lines(value: A): Chunk[String]
}
```

## Motivation

The SSE wire format is small enough to hand-write and fiddly enough to get wrong. Fields are `name: value` lines, the event ends with a blank line, and a multi-line payload has to become several `data:` lines rather than one line containing newlines. Emit a payload with an embedded `\n` as a single `data:` line and the stream desynchronizes — the client reads the remainder as a new field, or as the end of the event.

`ServerSentEvent` owns that formatting. The envelope holds the metadata, an `SseDataEncoder` decides how the payload becomes lines, and `ServerSentEvent#render` assembles them in the order the specification requires. Splitting a multi-line string is the encoder's job, so a payload containing newlines is correct by construction rather than by remembering.

Separating the encoder from the envelope is what lets the payload be typed. `ServerSentEvent[String]` and `ServerSentEvent[Chunk[String]]` work out of the box, and any other type works as soon as it has an encoder — the envelope never needs to change.

## Quick Showcase

An event is a payload plus optional metadata, and rendering produces the wire format:

```scala

val event = ServerSentEvent("hello", "greeting").id("42").retry(3000)
```

Rendering emits the metadata fields, then the data lines, then the blank line that terminates the event:

```scala
print(event.render)
// event: greeting
// id: 42
// retry: 3000
// data: hello
//
```

## Construction

Three entry points cover the cases: payload only, payload with an event name, and payload with any combination of metadata.

### `ServerSentEvent.apply` — payload, optionally named

The one-argument form creates an event with no metadata at all, and the two-argument form sets the `event:` field:

```scala
object ServerSentEvent {
  def apply[A](data: A): ServerSentEvent[A]
  def apply[A](data: A, event: String): ServerSentEvent[A]
}
```

A bare event renders as a single `data:` line followed by the blank line:

```scala

val bare  = ServerSentEvent("tick")
val named = ServerSentEvent("tick", "heartbeat")
```

Naming the event adds one line above the data:

```scala
print(bare.render)
// data: tick
// 
print(named.render)
// event: heartbeat
// data: tick
//
```

### `ServerSentEvent.fromOptions` — all metadata at once

`ServerSentEvent.fromOptions` takes the three metadata fields as `Option`, each defaulting to `None`, which suits code that already has them in optional form:

```scala
object ServerSentEvent {
  def fromOptions[A](
    data: A,
    event: Option[String] = None,
    id: Option[String] = None,
    retry: Option[Long] = None
  ): ServerSentEvent[A]
}
```

Supplying a subset by name leaves the rest absent:

```scala

val event = ServerSentEvent.fromOptions("payload", id = Some("7"), retry = Some(1000))
```

Only the fields you set appear on the wire:

```scala
print(event.render)
// id: 7
// retry: 1000
// data: payload
//
```

This is the only constructor that takes `Option`. The accessors return `Maybe`, so a round trip through `ServerSentEvent.fromOptions` converts between the two.

## Setting Metadata

Five methods produce a modified copy. Each returns a new envelope; nothing mutates.

| Method                        | Effect                                    |
| ----------------------------- | ----------------------------------------- |
| `ServerSentEvent#event`       | Sets the `event:` field.                   |
| `ServerSentEvent#clearEvent`  | Removes the `event:` field.                |
| `ServerSentEvent#id`          | Sets the `id:` field.                      |
| `ServerSentEvent#retry`       | Sets the `retry:` field, in milliseconds.  |
| `ServerSentEvent#clearRetry`  | Removes the `retry:` field.                |

They chain, since each returns a `ServerSentEvent[A]`:

```scala

val event = ServerSentEvent("payload").event("update").id("100").retry(5000)
```

Clearing a field drops its line without disturbing the others:

```scala
print(event.clearRetry.render)
// event: update
// id: 100
// data: payload
// 
print(event.clearEvent.clearRetry.render)
// id: 100
// data: payload
//
```

:::note[There is no `clearId`]
`ServerSentEvent#clearEvent` and `ServerSentEvent#clearRetry` exist, but the `id:` field has no clearing method. To produce an event without an id, build a fresh envelope or use `ServerSentEvent.fromOptions` with `id = None`.
:::

The accessors expose what is set, as `Maybe`:

```scala
event.data
// res9: String = "payload"
event.eventType
// res10: Maybe[String] = "update"
event.eventId
// res11: Maybe[String] = "100"
event.retryMillis
// res12: Maybe[Long] = 5000L
```

## Validation

Two invariants are enforced at construction, and both throw rather than returning an error, because an invalid event cannot be rendered safely.

The `event:` and `id:` fields must not contain a carriage return or line feed. The reason is the same as for header values: a newline inside a field would terminate that field early and let the remainder be read as a new field or a new event, desynchronizing the stream.

The `retry:` value must be non-negative, since it is a reconnection delay in milliseconds.

Both checks are visible from a bare import:

```scala

```

Both rejections are `IllegalArgumentException` with a message naming the field:

```scala
scala.util.Try(ServerSentEvent("data", "bad\nevent")).failed.map(_.getMessage)
// res14: Try[String] = Success(
//   "SSE event must not contain CR or LF characters"
// )
scala.util.Try(ServerSentEvent("data").retry(-1)).failed.map(_.getMessage)
// res15: Try[String] = Success("SSE retry must be non-negative")
```

Validation applies to the metadata only. The **payload** is never validated, because embedded newlines in the payload are legitimate — the encoder splits them into separate `data:` lines.

## Rendering

`ServerSentEvent#render` needs an `SseDataEncoder` for the payload type and produces the complete wire representation, terminating blank line included:

```scala
final class ServerSentEvent[+A] {
  def render(implicit encoder: SseDataEncoder[A]): String
}
```

Fields are emitted in a fixed order — `event:`, then `id:`, then `retry:`, then the `data:` lines — with absent fields omitted entirely. A payload that produces no lines still emits one empty `data:` line, so an event is never rendered without a data field:

```scala

val empty = ServerSentEvent(Chunk.empty[String])
```

The result is a single valueless data line and the terminator:

```scala
empty.render
// res17: String = """data: 
// 
// """
```

## SseDataEncoder

`SseDataEncoder[A]` maps a payload to its `data:` lines. It is contravariant in `A`, so an encoder for a supertype serves every subtype:

```scala
trait SseDataEncoder[-A] {
  def lines(value: A): Chunk[String]
}

object SseDataEncoder {
  def apply[A](implicit encoder: SseDataEncoder[A]): SseDataEncoder[A]
  implicit val string: SseDataEncoder[String]
  implicit val stringChunk: SseDataEncoder[Chunk[String]]
}
```

### Built-in Instances

`SseDataEncoder.string` splits a `String` on line breaks, so a multi-line payload becomes one `data:` line per line. `\n`, `\r`, and `\r\n` all count as a single break:

```scala

val multiline = ServerSentEvent("first line\nsecond line\nthird line")
```

Three lines in the payload become three `data:` lines, which is what the SSE specification requires:

```scala
print(multiline.render)
// data: first line
// data: second line
// data: third line
//
```

`SseDataEncoder.stringChunk` treats each element as a line and splits each element as well, so a chunk whose elements themselves contain newlines still flattens correctly:

```scala

val chunked = ServerSentEvent(Chunk("alpha", "beta\ngamma"))
```

The two elements yield three lines:

```scala
print(chunked.render)
// data: alpha
// data: beta
// data: gamma
//
```

An empty `Chunk` renders as one empty `data:` line rather than none, matching the single-`String` behaviour for an empty payload.

### Custom Instances

An encoder for your own type is one method. Returning several lines is how a structured payload spans multiple `data:` fields:

```scala

final case class Progress(step: Int, total: Int, note: String)

implicit val progressEncoder: SseDataEncoder[Progress] =
  new SseDataEncoder[Progress] {
    def lines(value: Progress): Chunk[String] =
      Chunk(s"step=${value.step}/${value.total}", s"note=${value.note}")
  }
```

With the instance in implicit scope, the payload type flows through the envelope unchanged:

```scala
print(ServerSentEvent(Progress(3, 10, "compiling"), "progress").render)
// event: progress
// data: step=3/10
// data: note=compiling
//
```

:::warning[Split lines yourself, or don't produce them]
`ServerSentEvent#render` prefixes each line the encoder returns with `data: ` and does not inspect it. An encoder that returns a string containing `\n` therefore emits a malformed event. Either split within `SseDataEncoder#lines`, or delegate to `SseDataEncoder.string` for the parts that might contain breaks.
:::

For a JSON payload, encode to a `String` with the JSON codec and reuse `SseDataEncoder.string`, which handles any newlines the rendered document contains.

## Equality and Rendering as Text

`ServerSentEvent#equals` compares the payload and all three metadata fields, so two envelopes are equal when their payloads are equal (`==`) and their metadata is equal. Two envelopes can render the same text and still be unequal, such as `ServerSentEvent("a")` and `ServerSentEvent(Chunk("a"))`. `ServerSentEvent#hashCode` agrees with it.

`ServerSentEvent#toString` is a diagnostic rendering, not the wire format — absent fields appear as `null` and no `data:` prefixes are added:

```scala

val event = ServerSentEvent("payload").id("9")
```

Use `ServerSentEvent#render` for anything that goes over the wire and `ServerSentEvent#toString` only for logs:

```scala
event.toString
// res25: String = "ServerSentEvent(data=payload, event=null, id=9, retry=null)"
```

## Integration Points

An SSE response is an ordinary `Response` whose body streams rendered events with a `text/event-stream` content type, so `ServerSentEvent` composes with the rest of [the HTTP model](./model.md) rather than replacing any of it. Setting that content type is a `Header.ContentType` — see [Header](./headers.md).

The type depends on `Chunk` for the line sequence and on `Maybe` for its optional fields, both from the core blocks: see [Chunk](../chunk.md) and [Maybe](../maybe.md).

See also: [How to Send Server-Sent Events with http-model](../../guides/send-server-sent-events.md) builds and serves an SSE response end to end.

---

## Reference

Each block has its own reference pages, which describe what its data types are for, how to construct them, and the operations they offer. Find a block by name, or jump to an area. The [overview](../index.md) compares every block in one table, and the [API reference](pathname:///api/) lists every type and method.

<ReferenceDirectory />

---

## JWT

`zio-blocks-jwt` is a zero-dependency, cross-platform (JVM + Scala.js) JWT library for Scala 2.13 and Scala 3.

## Getting Started

Add the dependency to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-jwt" % "0.0.56"

// Scala 3.9 and newer (also needs scalacOptions += "-experimental")
libraryDependencies += "dev.zio" %% "zio-blocks-next-jwt" % "0.0.56"
```

HMAC algorithms (`HS256`, `HS384`, `HS512`) work out of the box with the built-in pure Scala backend. For RSA, RSA-PSS, ECDSA and EdDSA, install a platform backend **once** at application startup:

```scala

JvmJwtCryptoBackend.install()
```

For Scala.js (Node.js) use `JsJwtCryptoBackend.install()` instead (available on the JS platform with `jwtJS`).

The installed backend is global. To use a different backend for one call, pass it in `JwtSignOptions(backend = Some(...))` or `JwtDecodeOptions(backend = Some(...))`.

## Signing

```scala

val key    = "0123456789ABCDEF0123456789ABCDEF".getBytes("UTF-8") // 32 bytes for HS256 (JWA minimum)
val claims = JwtClaims(sub = Some("user-123"), iss = Some("my-app"))
val token: Either[JwtError, String] = Jwt.sign(claims, key, Algorithm.HS256)
```

## Decoding and Verifying

```scala

val key    = "0123456789ABCDEF0123456789ABCDEF".getBytes("UTF-8")
val claims = JwtClaims(sub = Some("user-123"), iss = Some("my-app"))
val token  = Jwt.sign(claims, key, Algorithm.HS256).getOrElse("")
val result: Either[JwtError, JwtClaims] =
  Jwt.decode(token, key, Algorithm.HS256, clockSkewSeconds = 30L, issuer = Some("my-app"))
```

`Jwt.decode` checks the token in this order: size limits, the three-segment structure, the header (including its `alg`), that the header's `alg` equals the `alg` argument, the signature, and only then the claims (`exp`, `nbf`, `iss`, `aud`). A tampered payload is therefore reported as `InvalidSignature`, never as a claim error.

Time checks:

- `exp` and `nbf` are only checked when present. A token without `exp` never expires; check `claims.exp.isDefined` yourself if your service requires it.
- A token is expired when `now - clockSkewSeconds > exp`, and not yet valid when `now + clockSkewSeconds < nbf`.
- `now` is `System.currentTimeMillis() / 1000` unless `JwtDecodeOptions(nowSeconds = Some(t))` fixes it, which is useful in tests.

`Jwt.decodeWithoutVerification(token)` returns the header and claims **without checking the signature or any claim**. Use it only to inspect a token; never base an access decision on its output. `Jwt.decodeUnsafe` is an older name for the same operation.

## Claims

`JwtClaims` models the RFC 7519 registered claims plus arbitrary extra claims:

| Field   | Type                                  | RFC claim |
|---------|---------------------------------------|-----------|
| `iss`   | `Option[String]`                      | Issuer    |
| `sub`   | `Option[String]`                      | Subject   |
| `aud`   | `Option[JwtAudience]`                 | Audience  |
| `exp`   | `Option[Long]`                        | Expiration (Unix seconds) |
| `nbf`   | `Option[Long]`                        | Not Before (Unix seconds) |
| `iat`   | `Option[Long]`                        | Issued At (Unix seconds)  |
| `jti`   | `Option[String]`                      | JWT ID    |
| `extra` | `Map[String, JwtValue]`                 | Custom claims, preserving JSON type |

`exp`/`nbf`/`iat` are `NumericDate` per RFC 7519 §2: JSON numbers (integer, fractional, exponent) truncated toward zero to seconds, and rejected with `InvalidClaim` if negative or out of `Long` range.

### Extra claims

`JwtValue` is the public JSON ADT for `extra` (and for `JwtClaims` round-trip):

```scala

val claims = JwtClaims(
  sub   = Some("user-123"),
  extra = Map(
    "role"   -> JwtValue.Str("admin"),
    "level"  -> JwtValue.Num("3"),
    "active" -> JwtValue.Bool(true),
    "meta"   -> JwtValue.Null,
    "tags"   -> JwtValue.Arr(Chunk(JwtValue.Str("a"), JwtValue.Num("1"))),
    "profile"-> JwtValue.Obj(Map("age" -> JwtValue.Num("30"), "nested" -> JwtValue.Obj(Map("k" -> JwtValue.Str("v")))))
  )
)
```

Objects and arrays nest up to `JwtLimits.maxDepth` (default 32) and keep their values, except that a repeated field name keeps its last value. Do not put registered claim names (`iss`, `sub`, `aud`, `exp`, `nbf`, `iat`, `jti`) in `extra`; use the dedicated fields.

### Audience

Per RFC 7519 §4.1.3, `aud` may be a single string or an array. A JSON `null` `aud` in a token is treated as absent. Arrays must contain only strings; an array with any non-string element is rejected with `InvalidClaim("aud", "array must contain only strings")`:

```scala

val c1 = JwtClaims(aud = Some(JwtAudience.Single("my-service")))
val c2 = JwtClaims(aud = Some(JwtAudience.Multiple(Chunk("service-a", "service-b"))))
val c3 = JwtClaims(aud = None) // absent
```

When `expectedAudience` is set in `JwtDecodeOptions`, `aud` must be present and equal (or, for an array, contain) the expected value. A missing `aud` yields `MissingClaim("aud")`, and a mismatch yields `InvalidClaim("aud", "expected <expected> but found <actual>")`. When `expectedAudience` is not set, `aud` is only parsed, never checked.

### Limits and Options

`JwtLimits` bounds the token, each segment, the JSON text, the nesting depth, the fields per object and the elements per array; the following example sets each limit and passes them to decoding and signing:

```scala

val limits = JwtLimits(maxTokenChars = 8192, maxSegmentChars = 4096, maxJsonChars = 8192, maxDepth = 32, maxFields = 512, maxArrayElements = 512)
val decodeOpts = JwtDecodeOptions(
  issuer = Some("my-app"),
  expectedAudience = Some("my-service"),
  clockSkewSeconds = 30L,
  limits = limits,
  nowSeconds = Some(1700000000L)
)
val signOpts = JwtSignOptions(limits = limits)
val key = "0123456789ABCDEF0123456789ABCDEF".getBytes("UTF-8")
val claims = JwtClaims(sub = Some("user-123"))
val token = Jwt.sign(claims, key, Algorithm.HS256, signOpts).getOrElse("")
Jwt.sign(claims, key, Algorithm.HS256, signOpts)
Jwt.decode(token, key, Algorithm.HS256, decodeOpts)
```

On decode, the size limits run before any parsing, and the header's JSON limits (`maxDepth`, `maxFields`, `maxArrayElements`) run before the signature is checked. The payload is parsed only after the signature verifies.

`JwtHeader` also carries `kid` and `typ` (default `"JWT"`):

```scala

val hdr = JwtHeader(Algorithm.HS256, typ = "JWT", kid = Some("key-1"))
val key = "0123456789ABCDEF0123456789ABCDEF".getBytes("UTF-8")
val claims = JwtClaims(sub = Some("user-123"))
Jwt.sign(claims, key, Algorithm.HS256, hdr)
```

`kid` is written and parsed but never used to select a key, and `typ` is not checked on decode.

## Supported Algorithms and Key Strength

| Algorithm | JVM | JS (Node.js) | Pure Scala | Key minima / curve |
|-----------|-----|--------------|-----------|--------------------|
| HS256     | ✓   | ✓            | ✓         | 32 bytes |
| HS384     | ✓   | ✓            | ✓         | 48 bytes |
| HS512     | ✓   | ✓            | ✓         | 64 bytes |
| RS256     | ✓   | ✓            | —         | RSA ≥2048 bits (checked on JVM only) |
| RS384     | ✓   | ✓            | —         | RSA ≥2048 bits (checked on JVM only) |
| RS512     | ✓   | ✓            | —         | RSA ≥2048 bits (checked on JVM only) |
| PS256     | ✓   | —            | —         | RSA ≥2048 bits, PSS |
| PS384     | ✓   | —            | —         | RSA ≥2048 bits, PSS |
| PS512     | ✓   | —            | —         | RSA ≥2048 bits, PSS |
| ES256     | ✓   | ✓            | —         | P-256 (`prime256v1`, checked on JVM only) |
| ES384     | ✓   | ✓            | —         | P-384 (`secp384r1`, checked on JVM only) |
| ES512     | ✓   | ✓            | —         | P-521 (`secp521r1`, 66-byte components, checked on JVM only) |
| EdDSA     | ✓   | —            | —         | Ed25519 |

HMAC keys shorter than the minimum length for their algorithm are rejected with `InvalidKey` (not `UnsupportedAlgorithm` or `InvalidToken`) on every backend. On the JVM, RSA keys under 2048 bits and a P-256, P-384 or P-521 key used with the wrong `ES*` algorithm (e.g. `ES256` with `secp384r1`) are also rejected with `InvalidKey`; the Node.js backend does not check RSA key size or EC curves.

Query what a backend supports at runtime:

```scala

JvmJwtCryptoBackend.supportedAlgorithms // Set[Algorithm]
// res7: Set[Algorithm] = Set(
//   RS256,
//   PS384,
//   RS512,
//   PS512,
//   HS384,
//   ES256,
//   PS256,
//   EdDSA,
//   ES512,
//   HS512,
//   RS384,
//   ES384,
//   HS256
// )
```

`UnsupportedAlgorithm` is returned when the backend does not support the algorithm: any asymmetric algorithm when no platform backend is installed, `PS*`/`EdDSA` on JS (Node), or a token whose `alg` header is not a known algorithm (such as `FOO` or `none`).

In a browser or other JavaScript runtime without Node's `crypto` module, `JsJwtCryptoBackend` returns `UnsupportedAlgorithm` for `RS*`/`ES*`; HMAC algorithms still work.

### Key Formats

Every `key` argument is an `Array[Byte]`:

| Algorithm family     | Signing key (`Jwt.sign`)        | Verification key (`Jwt.decode`)              |
|----------------------|---------------------------------|----------------------------------------------|
| `HS*`                | Raw secret bytes                | The same secret bytes                        |
| `RS*`, `PS*`         | RSA private key, PKCS#8 DER     | RSA public key, X.509 SubjectPublicKeyInfo DER |
| `ES*`                | EC private key, PKCS#8 DER      | EC public key, X.509 SubjectPublicKeyInfo DER  |
| `EdDSA`              | Ed25519 private key, PKCS#8 DER | Ed25519 public key, X.509 SubjectPublicKeyInfo DER |

On the JVM, `java.security.PrivateKey.getEncoded` returns PKCS#8 DER and `PublicKey.getEncoded` returns X.509 DER. The module does not parse PEM or JWK: for a PEM file, drop the `-----BEGIN ...-----`/`-----END ...-----` lines and Base64-decode the rest. Only `BEGIN PRIVATE KEY` and `BEGIN PUBLIC KEY` PEM files contain these formats; convert PKCS#1 or SEC1 keys first. A key in the wrong format is rejected with `InvalidKey`.

## Error Handling

All errors are represented as `JwtError` subtypes (no stack traces). Limits and validation map to distinct cases:

| Error                | Meaning                                      |
|----------------------|----------------------------------------------|
| `InvalidToken`       | Malformed structure (not three segments, empty segment), invalid Base64URL, or invalid JSON |
| `ExpiredToken`       | `exp` is in the past (with `clockSkewSeconds` and `nowSeconds`) |
| `NotYetValid`        | `nbf` is in the future                       |
| `InvalidSignature`   | Signature did not verify (after `alg` check, before claims parsing) |
| `UnsupportedAlgorithm` | Backend does not support the algorithm, or the `alg` header is unknown (`FOO`, `none`) |
| `MissingClaim`       | Required claim absent (`alg`, `iss` when `issuer` is set, `aud` when `expectedAudience` is set) |
| `InvalidClaim`       | A claim has the wrong type (`iss`/`sub` not a string, `exp` not a number, `aud` array with non-strings), or `iss`/`aud` does not match `issuer`/`expectedAudience` |
| `AlgorithmMismatch`  | On `decode`, the token's `alg` header differs from the `alg` argument; on `sign`, `header.alg` differs from the `alg` argument |
| `InvalidKey`         | Key bytes are malformed, in the wrong format, or too weak (short HMAC secret; on JVM, RSA under 2048 bits or a P-256/P-384/P-521 key with the wrong `ES*` algorithm) |
| `TokenTooLarge`      | Compact token > `maxTokenChars` |
| `SegmentTooLarge`    | A segment > `maxSegmentChars` |
| `JsonTooLarge`       | Decoded JSON > `maxJsonChars` |
| `TooDeep`            | JSON depth > `maxDepth` |
| `TooManyFields`      | Object fields > `maxFields` |
| `TooManyElements`    | Array elements > `maxArrayElements` |

## Base64URL Encoding

`Base64Url` is a public object that encodes/decodes with no padding (`=`), per RFC 7515 §2 and RFC 4648 §5. Tokens containing `=` are rejected with `InvalidToken`; length `mod 4 == 1` and non-Base64Url chars (`+`, `/`, `!`) are rejected; trailing bits for 2- or 3-char tails must be zero (e.g. `AB` / `ABC` are rejected).

## Limitations

- No JWK or JWKS support, and no key lookup by `kid`.
- No JWE: tokens are signed, not encrypted, so anyone holding a token can read its claims.
- The module is new and has no binary-compatibility (MiMa) baseline yet; expect API changes.

## See Also

- [How to Issue and Verify JWTs](../guides/issue-and-verify-jwts.md): sign and verify with HMAC, RSA and EdDSA keys, fix the clock in tests, load PEM keys, and reject hostile tokens.
- [How to Protect an Endpoint with JWT Bearer Tokens](../guides/protect-endpoints-with-jwt.md): read the bearer token from a request, verify it, and answer 401 or 403 correctly.

---

## Maybe

`Maybe[A]` is a **low-allocation alternative to `Option[A]`** that uses a top-level `Absent` sentinel object to represent the absence of a value. On Scala 3, it is an opaque type alias for `A | Absent.type | Present[A]`, where `Present[A]` is a public wrapper allocated whenever a present value would otherwise be indistinguishable from absence — nested `Maybe`s (`Maybe.present(Maybe.absent)` → `Present(Absent)`) and `null` values (`Maybe.present(null)` → `Present(null)`). On Scala 2.13, it is a sealed trait (`Present[A]` | `Absent`). Core types: `Maybe[A]`, `Present[A]`.

Here's the type definition and basic construction:

```scala
// Scala 3
final class Present[+A](val value: A) // manual companion: apply + unapply that matches both raw values and wrappers
object Absent
opaque type Maybe[+A] = A | Absent.type | Present[A]

val present: Maybe[Int] = Maybe.present(42)
val absent: Maybe[Int]  = Maybe.absent
```

## Allocation Profile

On Scala 3, the encoding stores a present value as the value itself whenever it can. `Maybe[A]` erases to `Object`, though, so on the JVM a primitive is stored as its box, and the box is what may allocate. In the first rows below, `tick` stands for any reference value, such as a `String` or a case-class instance:

| Case                        | Example                                 | Allocations                                                |
| --------------------------- | --------------------------------------- | ---------------------------------------------------------- |
| Flat present, reference     | `Maybe.present(tick)`                   | **0** (the reference itself)                               |
| Flat apply, reference       | `Maybe(tick)`                           | **0** (the reference itself)                               |
| Flat fromOption, reference  | `Maybe.fromOption(Some(tick))`          | **0** beyond the caller's `Some`                           |
| Flat present, primitive     | `Maybe.present(1000L)`                  | **1** box (`java.lang.Long`), unless the JVM caches it     |
| Flat present, cached box    | `Maybe.present(42)`                     | **0** (the JVM's cached `Integer` for `42`)                |
| Absent                      | `Maybe.absent`                          | **0** (`Absent` singleton)                                 |
| Present-of-absent           | `Maybe.present(Maybe.absent)`           | **1** (`Present(Absent)`)                                  |
| Present null                | `Maybe.fromOption(Some(null))`          | **1** (`Present(null)`)                                    |

The JVM caches the boxes of `Boolean`, of `Byte`, `Short`, `Int` and `Long` values from -128 to 127, and of `Char` values up to 127, so those primitives reuse a box. Every other `Int` or `Long`, and every `Float` and `Double`, gets a new box each time: writing `Maybe(price)` into a `Maybe[Long]` field allocates 24 bytes per write, against 40 for `Some(price)` in an `Option[Long]` field. [How to Avoid Allocation on Hot Paths](../guides/avoid-allocation-on-hot-paths.md) measures both.

The `Present[A]` wrapper is allocated **only** for present-of-absent cases: wrapping a nested `Maybe` that is itself absent (`Present(Absent)`) or already a `Present` wrapper, and wrapping a `null` value (`Present(null)`). All other flat cases store the reference raw; for a reference type, that means zero allocation.

### Nesting Depth

Nested `Maybe` values reduce wrapper depth by 1 compared to `Option`:

```scala

// Option: Some(None) = 2 wrappers
val optNested: Option[Option[Int]] = Some(None)

// Maybe: Present(Absent) = 1 wrapper (Present) + Absent singleton
val maybeNested: Maybe[Maybe[Int]] = Maybe.present(Maybe.absent[Int])
// maybeNested matches Present(Absent), not Absent

// Flattening removes the Present wrapper
val flat: Maybe[Int] = maybeNested.flatten
// flat is absent (Absent)
```

See also [How to Avoid Allocation on Hot Paths](../guides/avoid-allocation-on-hot-paths.md) for swapping `Option` for `Maybe` in a hot loop and measuring the effect.

## Motivation

When working with optional values, you face a choice: `Option[A]` provides type safety and functional composition but allocates a wrapper object for every value. `Maybe[A]` provides an alternative with different trade-offs depending on your Scala version.

**On Scala 3:** `Maybe[A]` eliminates allocation overhead by leveraging union types and a top-level `Absent` sentinel. The type is an opaque alias for `A | Absent.type | Present[A]`, where `Present[A]` is a public wrapper allocated whenever a present value would otherwise be indistinguishable from absence. Flat reference values (non-nested, non-null) are stored raw with zero allocation, while a primitive is stored as its JVM box, which allocates unless the JVM caches it; only present-of-absent values, a present `null` and an already-present `Maybe` allocate a `Present` wrapper (`Maybe.present(Maybe.absent)` → `Present(Absent)`, `Maybe.present(null)` → `Present(null)`, `Maybe.present(Present(x))` → `Present(Present(x))`). This gives you a dedicated API (`map`, `flatMap`, `filter`, etc.) with minimal runtime overhead. `case Absent` is a stable-identifier pattern that works directly.

**On Scala 2.13:** `Maybe[A]` is implemented as a sealed trait (`Present[A]` | `Absent`). Present values allocate a wrapper, so the allocation savings versus `Option` are less pronounced. However, the unified API and interoperability benefits still apply.

### Why Maybe over Option?

- **No wrapper (flat case)**: Non-nested `Maybe` values are either the raw value itself or the `Absent` singleton, so a reference costs nothing; a primitive still costs its box
- **Sound nesting**: Nested `Maybe[Maybe[A]]` is now sound via `Present[A]`, without requiring a compile-time guard
- **Familiar API**: All your favorite `Option` combinators (`map`, `flatMap`, `fold`, etc.)
- **Type safety**: The opaque type prevents accidentally mixing nullable and non-nullable values
- **Interoperable**: Seamless conversion to/from `Option` with `toOption` and `Maybe#fromOption`

## Cross-Version Parity

The API surface is consistent across Scala 2.13 and Scala 3, but the underlying encoding differs:

| Aspect            | Scala 3                                                                    | Scala 2.13                            |
| ----------------- | -------------------------------------------------------------------------- | ------------------------------------- |
| Encoding          | `opaque type Maybe[+A] = A \| Absent.type \| Present[A]`                   | Sealed trait `Present[A]` \| `Absent` |
| Flat allocation   | **0** for a reference or `Absent`; a primitive keeps its JVM box           | **1** (wrapper object)                |
| Nested allocation | **1** (`Present(Absent)` or `Present(Present(x))`)                             | **1** (wrapper object)                |
| `Present` type    | Public `final class Present[+A]` with manual companion (`apply`/`unapply`) | `MaybeValue.Present[A]` (sealed)      |
| Nesting guard     | **Removed** (nesting now sound via `Present`)                              | N/A (never existed)                   |

### Behavior Differences

- **`Maybe.present(null)`**: On Scala 3, produces `Present(null)`, a present `null`. On Scala 2.13, produces `MaybeValue.Present(null)`, also a present `null`. Both are distinguishable from `Maybe.absent`, and `isPresent` is `true` and `isAbsent` is `false` for them.
- **`Maybe.fromOption(Some(null))`**: Routes through `present`, so same behavior as above.
- **`Maybe(Maybe.absent)`**: absent on Scala 3, present on Scala 2.
- **`toString` and `==`**: `Maybe.present(8080)` prints `8080` on Scala 3 and equals `8080` only when compared as `Any` (a direct `Maybe.present(8080) == 8080` does not compile there); on Scala 2 it prints `Present(8080)`. `Maybe.absent` prints `zio.blocks.maybe.Absent$@<hash>` on Scala 3 and `Absent` on Scala 2.
- **Pattern matching**: On Scala 3, `Present(v)` matches both present shapes (a raw value and a `Present(...)` wrapper); absent matches `case Absent` (stable-identifier pattern, no `case _` required inside the package; external two-case matches also compile without exhaustivity warning). On Scala 2.13, `MaybeValue.Present(v)` / `MaybeValue.Absent` are the compiler-checked native patterns.

## Pattern Matching

On Scala 3, a `Maybe[A]` value can take three runtime shapes. The `Present` companion's `unapply` collapses the two present shapes into one pattern; absent is now a real top-level singleton object:

| Shape | Pattern | Meaning |
|-------|---------|---------|
| `Absent` object | `case Absent` | absent (stable-identifier pattern) |
| `Present(v)` wrapper | `case Present(v)` | present-of-absent (a nested `Maybe`) |
| raw `v` | `case Present(v)` | present, stored without a `Present` wrapper |

(Note: the `Present` companion's `unapply` collapses both present shapes — one `Some` allocation per present match.)

```scala

val maybe: Maybe[Int] = Maybe.present(42)

val description: String = maybe match {
  case Present(v) => s"present ($v)"
  case Absent     => "absent"
}
```

`case Absent` is a stable-identifier pattern that works because `Absent` is a plain top-level object (not a case object). The two-case match `case Present(v); case Absent` compiles without exhaustivity warning even from an external package — verified under this build's `-Xfatal-warnings` settings (`WildcardImportSpec`). Note that this is weaker than the sealed-hierarchy guarantee on Scala 2.13 below: exhaustivity here depends on the compiler decomposing the opaque union, so prefer `fold` when you need a guarantee that is independent of compiler behavior. `Present(v)` matches both a raw value and a `Present(...)` wrapper, at the cost of one `Some` allocation per present match (inherent to the `Option`-returning extractor protocol).

For production code, prefer `fold`, which is exhaustive, warning-free, and zero-allocation:

```scala

val maybe: Maybe[Int] = Maybe.present(42)
val description: String = maybe.fold("absent")(v => s"present ($v)")
```

Scala 2.13 parity: on Scala 2.13, absent is the non-null case object `MaybeValue.Absent`, so it **can** be matched explicitly — the sealed `MaybeValue` trait (`Present` | `Absent`) gives compiler-checked exhaustivity:

```scala

val maybe: Maybe[Int] = Maybe.present(42)
val description: String = maybe match {
  case MaybeValue.Present(v) => s"present ($v)"
  case MaybeValue.Absent     => "absent"
}
```

So on Scala 2.13 the sealed `MaybeValue` form is the compiler-checked native idiom; on Scala 3 the opaque union with `Absent` object enables `case Absent` while keeping the wrapper-free encoding.

## Installation

Add the `zio-blocks-maybe` module to your build:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-maybe" % "0.0.56"
```

For Scala.js:

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-maybe" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x

## Overview

`Maybe[A]` provides a complete set of operations for working with optional values:

- **Constructors**: `Maybe.apply`, `Maybe.present`, `Maybe.absent`, `Maybe.fromOption`
- **Predicates**: `Maybe#isAbsent`, `isPresent`, `isEmpty`, `isDefined`, `nonEmpty`
- **Access**: `get`, `Maybe#getOrElse`, `orElse`, `orNull`
- **Transformations**: `map`, `flatMap`, `flatten`
- **Filtering**: `filter`, `filterNot`, `collect`
- **Logical**: `exists`, `forall`, `contains`
- **Conversions**: `toOption`, `toList`, `toSeq`, `iterator`, `toRight`, `toLeft`
- **Composition**: `Maybe#zip`, `unzip`, `unzip3`, `fold`, `foreach`

## How It Works

The workflow is straightforward: create a `Maybe` from a value or `None`, transform and filter it using functional operations, extract the result with safe accessors, and fall back to defaults when needed:

```
Value ─→ Maybe.apply (wrap) ─→ map / flatMap (transform) 
  ↓                                    ↓
[null]  ←─────── isAbsent (test) ← filter / collect (refine)
  ↓                                    ↓
absent ─→ orElse (fallback) ─→ get / getOrElse (extract)
```

### Typical Data Flow

1. **Create** a `Maybe` using `Maybe.apply`, `Maybe.present`, `Maybe.fromOption`, or explicitly with `Maybe.absent` for no value
2. **Transform** values using `map` or `flatMap` — operations on absent values short-circuit and remain absent
3. **Filter** with `filter` or `collect` to refine the value or produce absence based on a predicate
4. **Compose** with other `Maybe` values using `Maybe#zip` or flatMap chains to build complex workflows
5. **Extract** the result using `get` (throws on absence), `Maybe#getOrElse` (provides a default), `toOption` (convert to `Option`), or `fold` (handle both branches)

## Common Patterns

Here are key patterns for working effectively with `Maybe`:

### Present and Absent States

Every `Maybe` is either present (holds a value, possibly `null`) or absent (the `Absent` singleton). Test the state with predicates:

```scala

val value: Maybe[Int] = Maybe.present(42)

if (value.isPresent) {
  println(value.get) // Safe: we know it's present
} else {
  println("No value")
}
```

### Safe Extraction with Defaults

Use `Maybe#getOrElse` to provide a fallback when the value is absent, avoiding the exception risk of `get`:

```scala

val userId: Maybe[String] = Maybe.absent
val name = userId.getOrElse("anonymous")
```

### Chaining Transformations

Combine `map` and `flatMap` to thread operations through optional values. Absence propagates automatically:

```scala

val userId: Maybe[Int] = Maybe.present(123)

val greeting = userId
  .map(id => s"User $id")
  .map(msg => s"Hello, $msg!")

println(greeting) // Maybe[String] containing "Hello, User 123!"
```

### Filtering with Predicates

Use `filter` to keep a value only if it satisfies a condition, or `collect` with a partial function for both filtering and transformation:

```scala

val age: Maybe[Int] = Maybe.present(25)

val isAdult = age.filter(_ >= 18) // Maybe[Int] containing 25
val isChild = age.filter(_ < 18)  // Maybe[Int] absent
```

### Combining Multiple Maybes

Use `Maybe#zip` to combine two `Maybe` values into a tuple, or chain multiple operations with `flatMap`:

```scala

val firstName: Maybe[String] = Maybe.present("Alice")
val lastName: Maybe[String]  = Maybe.present("Smith")

val fullName = firstName.zip(lastName).map { case (f, l) => s"$f $l" }
```

### Converting to and from Option

Interoperate seamlessly with `Option` using `toOption` and `Maybe#fromOption`:

```scala

val opt: Option[String] = Some("value")
val m: Maybe[String]    = Maybe.fromOption(opt)
val backToOpt           = m.toOption
```

### Handling Errors with Either

Convert a `Maybe` to an `Either` to propagate errors in fail-fast computations:

```scala

val value: Maybe[Int] = Maybe.present(10)

val result: Either[String, Int] = value.toRight("Value not found")
```

## Integration Points

`Maybe[A]` integrates with the broader Scala and ZIO ecosystem:

- **Option interoperability**: Convert bidirectionally with `toOption` and `Maybe#fromOption`; on Scala 3 a `Conversion[Option[A], Maybe[A]]` is provided for implicit conversions; on Scala 2 the conversion is `Maybe.optionToMaybe`, which needs `import Maybe._`
- **Schema support**: Schema codecs use private unsafe methods (`unsafeIsAbsent`, `unsafeGet`, `unsafeWrap`) for efficient serialization and deserialization
- **For-comprehensions**: Supports `withFilter` for guard clauses in for-expressions
- **Collections**: Convert to `List`, `Seq`, or `Iterator` for bulk operations
- **Pattern matching**: Works naturally in match expressions, though testing with predicates is more common

### Example: For-Comprehension with Guards

Combine multiple `Maybe` values with filter guards:

```scala

val maybeX: Maybe[Int] = Maybe.present(5)
val maybeY: Maybe[Int] = Maybe.present(10)

val result = for {
  x <- maybeX
  y <- maybeY
  if x + y > 10
} yield x + y

println(result) // Maybe[Int] containing 15
```

## Operations Reference

All `Maybe` operations are organized by category. Each subsection documents a group of related methods with examples.

### Constructors

Create `Maybe` values using these factory methods:

#### Maybe.apply

Wraps a value in `Maybe`, treating `null` as `Maybe.absent`:

```scala

val present: Maybe[Int]    = Maybe(42)                         // Maybe[Int] containing 42
val absent: Maybe[String]  = Maybe(null.asInstanceOf[String])  // Maybe.absent (the Absent object)
```

> **Note:** `Maybe.apply` collapses `null` to `Maybe.absent`. Use `Maybe.present` when you need to preserve present-ness even for `null` values (e.g., in nested `Maybe`s).

#### Maybe.present

Explicitly wraps a value, preserving present-ness even for `null`:

```scala

val value: Maybe[Int] = Maybe.present(100)

// Nested case: present of an absent Maybe produces Present(Absent), not absence
val nested: Maybe[Maybe[Int]] = Maybe.present(Maybe.absent[Int])
// nested is Present(Absent), distinguishable from Maybe.absent
```

> **Note:** Unlike `Maybe.apply`, `Maybe.present` preserves present-ness even for `null`. A non-null value is returned as-is, with no wrapper (a primitive keeps its JVM box). A `null` value is wrapped in `Present(null)`, which is distinguishable from `Maybe.absent` (the `Absent` object). This is what makes nested `Maybe`s sound.

#### Maybe.absent

Creates an absent value for any type:

```scala

val empty: Maybe[String] = Maybe.absent
```

#### Maybe.empty

Alias for `Maybe.absent`:

```scala

val empty: Maybe[Double] = Maybe.empty
```

#### Maybe.fromOption

Converts an `Option` to `Maybe`:

```scala

val fromSome = Maybe.fromOption(Some(5))  // Maybe[Int] containing 5
val fromNone = Maybe.fromOption(None)     // Maybe[Nothing] absent
```

### State Testing

Test whether a `Maybe` is present or absent:

#### isPresent, isDefined, nonEmpty

All three are equivalent—return true if the value is not the `Absent` object (a present `null` counts as present):

```scala

val value: Maybe[Int] = Maybe.present(42)
println(value.isPresent)  // true
println(value.isDefined)  // true
println(value.nonEmpty)   // true
```

#### isAbsent, isEmpty

Both return true if the value is the `Absent` object (a present `null` is not absent):

```scala

val empty: Maybe[Int] = Maybe.absent
println(empty.isAbsent)  // true
println(empty.isEmpty)   // true
```

### Access

Retrieve the value or provide a fallback:

#### get

Unwraps the value, throwing `NoSuchElementException` if absent:

```scala

val value: Maybe[Int] = Maybe.present(10)
println(value.get)  // 10

try {
  Maybe.absent[Int].get
} catch {
  case e: NoSuchElementException => println(e.getMessage)  // "Maybe.absent.get"
}
```

#### getOrElse

Returns the value if present, or evaluates the default:

```scala

val value: Maybe[Int]    = Maybe.absent
val withDefault: Int     = value.getOrElse(99)
println(withDefault)  // 99
```

#### orElse

Returns the current `Maybe` if present, or another `Maybe`:

```scala

val first: Maybe[Int]  = Maybe.absent
val second: Maybe[Int] = Maybe.present(42)
val result = first.orElse(second)
println(result.get)  // 42
```

#### orNull

Converts to nullable, returning the value or `null`:

```scala

val value: Maybe[String] = Maybe.absent
val nullable: String = value.orNull
println(nullable)  // null
```

### Transformations

Transform values or short-circuit on absence:

#### map

Applies a function if present, remains absent otherwise:

```scala

val value: Maybe[Int]      = Maybe.present(5)
val doubled: Maybe[Int]    = value.map(_ * 2)
val absent: Maybe[String]  = Maybe.absent.map(_ => "never runs")
println(doubled.get)  // 10
println(absent.isAbsent)  // true
```

#### flatMap

Chains operations that return `Maybe`, flattening the result:

```scala

def safeDivide(a: Int, b: Int): Maybe[Int] =
  if (b == 0) Maybe.absent else Maybe(a / b)

val result = Maybe.present(10).flatMap(safeDivide(_, 2))
println(result.get)  // 5

val divideByZero = Maybe.present(10).flatMap(safeDivide(_, 0))
println(divideByZero.isAbsent)  // true
```

#### flatten

Unwraps a nested `Maybe`:

```scala

val nested: Maybe[Maybe[Int]] = Maybe.fromOption(Some(Maybe.fromOption(Some(42))))
val flat: Maybe[Int]          = nested.flatten
println(flat.get)  // 42

// Nested present-of-absent flattens to absent
val nestedAbsent: Maybe[Maybe[Int]] = Maybe.present(Maybe.absent[Int])
val flatAbsent: Maybe[Int]          = nestedAbsent.flatten
println(flatAbsent.isAbsent)  // true
```

### Filtering

Keep or discard values based on predicates:

#### filter

Keeps the value only if the predicate is true:

```scala

val value: Maybe[Int] = Maybe.present(10)
val even = value.filter(_ % 2 == 0)  // Present: 10
val odd  = value.filter(_ % 2 != 0)  // Absent
println(even.get)  // 10
println(odd.isAbsent)  // true
```

#### filterNot

Inverse of `filter`—keeps the value if the predicate is false:

```scala

val value: Maybe[Int]   = Maybe.present(5)
val notOdd: Maybe[Int]  = value.filterNot(_ % 2 != 0)
println(notOdd.isAbsent)  // true
```

#### collect

Combines filtering with transformation using a partial function:

```scala

val value: Maybe[Int] = Maybe.present(8)
val result = value.collect { case n if n % 2 == 0 => s"even: $n" }
println(result.get)  // "even: 8"

val odd: Maybe[Int]     = Maybe.present(7)
val noMatch: Maybe[String] = odd.collect { case n if n % 2 == 0 => s"even: $n" }
println(noMatch.isAbsent)  // true
```

### Logical Predicates

Test conditions without extracting the value:

#### contains

Returns true if the value equals the given element:

```scala

val value: Maybe[Int] = Maybe.present(42)
println(value.contains(42))  // true
println(value.contains(41))  // false
println(Maybe.absent[Int].contains(42))  // false
```

#### exists

Returns true if the value satisfies the predicate:

```scala

val value: Maybe[Int] = Maybe.present(10)
println(value.exists(_ > 5))   // true
println(value.exists(_ > 20))  // false
println(Maybe.absent[Int].exists(_ => true))  // false
```

#### forall

Returns true if the value satisfies the predicate, or if absent (vacuous truth):

```scala

val value: Maybe[Int] = Maybe.present(10)
println(value.forall(_ > 5))   // true
println(value.forall(_ > 20))  // false
println(Maybe.absent[Int].forall(_ => false))  // true (absent = vacuously true)
```

### Iteration and Conversion

Convert `Maybe` to other types or iterate its contents:

#### foreach

Executes a side effect if present:

```scala

val value: Maybe[Int] = Maybe.present(42)
value.foreach(x => println(s"Value: $x"))

Maybe.absent[Int].foreach(_ => println("not called"))
```

#### toOption

Converts to `Option[A]`:

```scala

val value: Maybe[Int]     = Maybe.present(7)
val some: Option[Int]     = value.toOption
val noneVal: Option[Int]  = Maybe.absent[Int].toOption
println(some)  // Some(7)
println(noneVal)  // None
```

#### toList

Converts to `List[A]`:

```scala

val value: Maybe[Int]   = Maybe.present(5)
val list: List[Int]     = value.toList
val emptyList: List[Int] = Maybe.absent[Int].toList
println(list)  // List(5)
println(emptyList)  // List()
```

#### toSeq

Converts to `Seq[A]`:

```scala

val value: Maybe[String] = Maybe.present("hello")
val seq: Seq[String]     = value.toSeq
println(seq)  // List(hello)
```

#### iterator

Creates an iterator over the value:

```scala

val value: Maybe[Int] = Maybe.present(42)
val it = value.iterator
println(it.toList)  // List(42)

Maybe.absent[Int].iterator.toList  // List()
```

### Either Conversion

Convert `Maybe` to `Either` for error handling:

#### toRight

Converts to `Either[X, A]` with a left error value:

```scala

val value: Maybe[Int] = Maybe.present(10)
val right: Either[String, Int] = value.toRight("not found")
println(right)  // Right(10)

val absent: Maybe[Int]         = Maybe.absent
val left: Either[String, Int]  = absent.toRight("not found")
println(left)  // Left("not found")
```

#### toLeft

Converts to `Either[A, X]` with a right success value:

```scala

val value: Maybe[String] = Maybe.present("error")
val left: Either[String, Unit] = value.toLeft(())
println(left)  // Left("error")

val absent: Maybe[String]      = Maybe.absent
val right: Either[String, Unit] = absent.toLeft(())
println(right)  // Right(())
```

### Composition

Combine multiple `Maybe` values:

#### zip

Combines two `Maybe` values into a tuple:

```scala

val x: Maybe[Int]    = Maybe.present(1)
val y: Maybe[String] = Maybe.present("a")
val tuple = x.zip(y)
println(tuple.get)  // (1, "a")

val absent: Maybe[String] = Maybe.absent
val zipped = x.zip(absent)
println(zipped.isAbsent)  // true
```

#### unzip

Splits a `Maybe[(A, B)]` into a tuple of `Maybe[A]` and `Maybe[B]`:

```scala

val tuple: Maybe[(Int, String)] = Maybe.present((42, "answer"))
val (x, y) = tuple.unzip
println(x.get)  // 42
println(y.get)  // "answer"

val absent: Maybe[(Int, String)] = Maybe.absent
val (ax, ay) = absent.unzip
println(ax.isAbsent && ay.isAbsent)  // true
```

#### unzip3

Splits a `Maybe[(A, B, C)]` into three `Maybe` values:

```scala

val triple: Maybe[(Int, String, Double)] = Maybe.present((1, "one", 1.0))
val (a, b, c) = triple.unzip3
println(a.get)  // 1
println(b.get)  // "one"
println(c.get)  // 1.0
```

### Folding

Reduce a `Maybe` to a value by handling both branches:

#### fold

Applies one of two functions based on presence:

```scala

val value: Maybe[Int] = Maybe.present(10)
val result = value.fold(-1)((x: Int) => x * 2)
println(result)  // 20

val absent: Maybe[Int] = Maybe.absent
val fallback = absent.fold(-1)((x: Int) => x * 2)
println(fallback)  // -1
```

### For-Comprehensions

Use `withFilter` to support guard clauses:

#### withFilter

Enables guarded for-expressions:

```scala

val x: Maybe[Int] = Maybe.present(5)
val y: Maybe[Int] = Maybe.present(15)

val result = for {
  a <- x
  b <- y
  if a + b > 15
} yield a + b

println(result.get)  // 20
```

## Running Examples

End-to-end workflows combining multiple `Maybe` operations:

### Example: Parsing and Transforming User Input

Build a pipeline that parses user input, validates it, and transforms the result:

```scala

case class User(id: Int, name: String, age: Int)

def parseId(input: String): Maybe[Int] =
  Maybe.fromOption(input.toIntOption)

def validateAge(age: Int): Maybe[Int] =
  Maybe.absent[Int].orElse(
    if (age >= 0 && age <= 150) Maybe.present(age) else Maybe.absent
  )

val input = "42"
val processedAge = parseId(input)
  .flatMap(id => Maybe.present(User(id, "Alice", 30)))
  .map(_.age)
  .flatMap(validateAge)

println(processedAge.getOrElse(-1))  // 30
```

### Example: Chaining Optional Database Results

Work with nullable database results, filtering and transforming as needed:

```scala

case class Product(id: Int, name: String, price: Double, inStock: Boolean)

def findProduct(id: Int): Maybe[Product] =
  if (id > 0) Maybe.present(Product(id, s"Product $id", 99.99, true))
  else Maybe.absent

def applyDiscount(product: Product, percent: Int): Maybe[Product] =
  if (percent >= 0 && percent <= 100)
    Maybe.present(product.copy(price = product.price * (1 - percent / 100.0)))
  else Maybe.absent

val discountedProduct = findProduct(123)
  .filter(_.inStock)
  .flatMap(applyDiscount(_, 20))
  .map(p => s"${p.name} now costs $$${p.price}")

println(discountedProduct.getOrElse("Product not available"))
```

### Example: Combining Multiple Maybes with Fallbacks

Handle scenarios where multiple optional values need to be combined:

```scala

case class Config(host: Maybe[String], port: Maybe[Int], timeout: Maybe[Long])

def buildConnection(config: Config): String = {
  val host = config.host.getOrElse("localhost")
  val port = config.port.getOrElse(8080)
  val timeout = config.timeout.getOrElse(5000L)
  s"Connection to $host:$port with timeout $timeout ms"
}

val config = Config(
  host = Maybe.present("api.example.com"),
  port = Maybe.absent,
  timeout = Maybe.present(10000L)
)

println(buildConnection(config))
```

### Example: Error Handling with Either Conversion

Convert `Maybe` to `Either` for composable error handling in result types:

```scala

case class Request(id: Maybe[String], method: Maybe[String])

def validateRequest(req: Request): Either[String, (String, String)] = {
  for {
    id <- req.id.toRight("Missing request ID")
    method <- req.method.toRight("Missing HTTP method")
  } yield (id, method)
}

val validReq = Request(
  id = Maybe.present("req-123"),
  method = Maybe.present("GET")
)

val invalidReq = Request(
  id = Maybe.absent,
  method = Maybe.present("POST")
)

println(validateRequest(validReq))  // Right((req-123, GET))
println(validateRequest(invalidReq))  // Left(Missing request ID)
```

### Example: Building a Computation Pipeline

Chain multiple transformations with fallback at each step:

```scala

def getUser(id: Int): Maybe[String] =
  if (id > 0) Maybe.present(s"user_$id") else Maybe.absent

def getUserEmail(username: String): Maybe[String] =
  if (username.nonEmpty) Maybe.present(s"$username@example.com") else Maybe.absent

def getUserProfile(id: Int): Maybe[String] = {
  val username = getUser(id)
  username
    .flatMap(getUserEmail)
    .map(email => s"Profile for $email")
    .orElse(Maybe.present("Guest user"))
}

println(getUserProfile(42))  // Profile for user_42@example.com
println(getUserProfile(-1))  // Guest user
```

---

## MediaType

`MediaType` is a **type-safe representation of IANA media types** (also known as MIME types). It captures structured metadata about content types including compressibility, binary/text classification, and associated file extensions. 

ZIO Blocks MediaType is designed to be a comprehensive and efficient implementation for handling media types in Scala applications, especially those involving HTTP content negotiation, file handling, and data serialization.

`MediaType`:

- is a zero-dependency data type in the `zio-blocks-mediatype` module
- ships with 2,600+ predefined IANA media types auto-generated from the [mime-db](https://github.com/jshttp/mime-db) database
- provides a `mediaType"..."` string interpolator with compile-time validation
- supports wildcard and case-insensitive matching
- is cross-platform (JVM and Scala.js) and cross-version (Scala 2.13 and 3.x)

```scala
final case class MediaType(
  mainType: String,
  subType: String,
  compressible: Boolean = false,
  binary: Boolean = false,
  fileExtensions: List[String] = Nil,
  extensions: Map[String, String] = Map.empty,
  parameters: Map[String, String] = Map.empty
)
```

## Motivation

Media types are fundamental to content negotiation in HTTP, file type detection, and serialization format selection. Working with raw strings like `"application/json"` is error-prone — typos go undetected, metadata (is it compressible? binary? what file extensions?) must be tracked separately, and matching logic must account for wildcards and case insensitivity.

`MediaType` solves these problems by providing:

- **Structured representation** — main type, subtype, and parameters as distinct fields
- **Rich metadata** — compressibility, binary/text classification, and file extensions baked in
- **Compile-time safety** — the `mediaType"..."` interpolator catches malformed types at compile time
- **Correct matching** — wildcard and case-insensitive matching with parameter awareness

```text
                    MediaType
           ┌──────────┼──────────┐
        mainType    subType   parameters
           │          │          │
     "application"  "json"   Map("charset" -> "utf-8")
           │
     ┌─────┴──────────────────────────────────┐
     │  compressible = true                   │
     │  binary       = false                  │
     │  fileExtensions = List("json", "map")  │
     └────────────────────────────────────────┘
```

A quick example:

```scala

// Compile-time validated media type
val json = mediaType"application/json"

// Look up by file extension
val detected = MediaType.forFileExtension("png")
// Some(MediaType("image", "png", binary = true, ...))

// Wildcard matching
val textAny = mediaType"text/*"
val html    = mediaType"text/html"
textAny.matches(html) // true
```

## Installation

Add the following to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-mediatype" % "0.0.56"
```

For cross-platform projects (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-mediatype" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x.

## Creating Instances

### Direct Construction

We can create a `MediaType` by specifying the main type and subtype directly. All other fields have sensible defaults:

```scala

// Minimal — just mainType and subType
val plain = MediaType("text", "plain")
// compressible = false, binary = false, fileExtensions = Nil, ...

// With all fields
val json = MediaType(
  mainType       = "application",
  subType        = "json",
  compressible   = true,
  binary         = false,
  fileExtensions = List("json", "map"),
  extensions     = Map("source" -> "iana"),
  parameters     = Map("charset" -> "utf-8")
)
```

### Parsing from a String

`MediaType.parse` parses a standard media type string in the format `mainType/subType[; key=value]*`:

```scala

// Simple type
val json: Either[String, MediaType] = MediaType.parse("application/json")
// Right(MediaType("application", "json", compressible = true, ...))

// With parameters
val html = MediaType.parse("text/html; charset=utf-8")
// Right(MediaType("text", "html", ..., parameters = Map("charset" -> "utf-8")))

// Predefined instances are reused — parse returns the same object
val parsed = MediaType.parse("application/json").toOption.get
parsed eq MediaTypes.application.`json` // true (reference equality)

// Invalid input returns Left with an error message
MediaType.parse("")               // Left("Invalid media type: cannot be empty")
MediaType.parse("applicationjson") // Left("Invalid media type: must contain '/' separator")
MediaType.parse("/json")          // Left("Invalid media type: main type cannot be empty")
MediaType.parse("application/")   // Left("Invalid media type: subtype cannot be empty")
```

### Unsafe Parsing

When we are certain the input is valid, `unsafeFromString` returns a `MediaType` directly or throws an `IllegalArgumentException`:

```scala

val json = MediaType.unsafeFromString("application/json")
// MediaType("application", "json", compressible = true, ...)

// Throws IllegalArgumentException for invalid input:
// MediaType.unsafeFromString("invalid")
```

### String Interpolator

The `mediaType"..."` interpolator validates the media type at compile time. Invalid types produce compile errors, not runtime failures:

```scala

val json     = mediaType"application/json"
val htmlUtf8 = mediaType"text/html; charset=utf-8"
val wildcard = mediaType"*/*"
val vendor   = mediaType"text/vnd.api+json"
```

The interpolator reuses predefined instances when available (reference equality with `MediaTypes` constants). It does not support variable interpolation — only literal strings are accepted.

:::note
The `mediaType` interpolator is available after importing `zio.blocks.mediatype._`. Invalid inputs produce clear compile-time errors:

```text
mediaType""                → "Invalid media type: cannot be empty"
mediaType"applicationjson" → "Invalid media type: must contain '/' separator"
mediaType"/json"           → "Invalid media type: main type cannot be empty"
mediaType"application/"    → "Invalid media type: subtype cannot be empty"
```
:::

### File Extension Lookup

`MediaType.forFileExtension` finds a `MediaType` by its associated file extension. The lookup is case-insensitive and strips a leading `.` if present:

```scala

MediaType.forFileExtension("json")  // Some(MediaType("application", "json", ...))
MediaType.forFileExtension(".html") // Some(MediaType("text", "html", ...))
MediaType.forFileExtension("PNG")   // Some(MediaType("image", "png", ...))
MediaType.forFileExtension("jpg")   // Some(MediaType("image", "jpeg", ...))
MediaType.forFileExtension("pdf")   // Some(MediaType("application", "pdf", ...))

MediaType.forFileExtension("xyz123") // None (unknown extension)
MediaType.forFileExtension("")       // None
MediaType.forFileExtension(".")      // None
```

## Predefined Media Types

The `MediaTypes` object contains 2,600+ predefined media type constants auto-generated from the [jshttp/mime-db](https://github.com/jshttp/mime-db) database. Types are organized by main type category:

| Category     | Object                    | Examples                             |
|--------------|---------------------------|--------------------------------------|
| Application  | `MediaTypes.application`  | `json`, `pdf`, `xml`, `octet-stream` |
| Audio        | `MediaTypes.audio`        | `mpeg`, `ogg`, `wav`                 |
| Chemical     | `MediaTypes.chemical`     | `x-cif`, `x-pdb`                     |
| Font         | `MediaTypes.font`         | `woff`, `woff2`, `otf`               |
| Image        | `MediaTypes.image`        | `png`, `jpeg`, `gif`, `svg+xml`      |
| Message      | `MediaTypes.message`      | `rfc822`, `partial`                  |
| Model        | `MediaTypes.model`        | `gltf+json`, `stl`                   |
| Multipart    | `MediaTypes.multipart`    | `form-data`, `mixed`                 |
| Text         | `MediaTypes.text`         | `html`, `css`, `plain`, `csv`        |
| Video        | `MediaTypes.video`        | `mp4`, `webm`, `ogg`                 |
| X-Conference | `MediaTypes.x_conference` | `x-cooltalk`                         |
| X-Shader     | `MediaTypes.x_shader`     | `x-vertex`, `x-fragment`             |
| Wildcard     | `MediaTypes.any`          | `*/*`                                |

### Accessing Predefined Types

Since many subtype names contain special characters (hyphens, dots, plus signs), predefined constants use backtick identifiers:

```scala

// Common application types
val json      = MediaTypes.application.`json`
val pdf       = MediaTypes.application.`pdf`
val xmlType   = MediaTypes.application.`xml`

// Common text types
val html      = MediaTypes.text.`html`
val css       = MediaTypes.text.`css`
val plain     = MediaTypes.text.`plain`

// Common image types
val png       = MediaTypes.image.`png`
val jpeg      = MediaTypes.image.`jpeg`

// Wildcard (matches any type)
val any       = MediaTypes.any
```

### Listing All Types

Each category object has an `all` field returning a `List[MediaType]` of all types in that category. The top-level `allMediaTypes` aggregates every category:

```scala

// All types in a category
val appTypes: List[zio.blocks.mediatype.MediaType] = MediaTypes.application.all

// All 2,600+ predefined types
val everything: List[zio.blocks.mediatype.MediaType] = MediaTypes.allMediaTypes
```

### Predefined Metadata

Each predefined instance comes with rich metadata from the IANA registry:

```scala

val json = MediaTypes.application.`json`
json.compressible   // true
json.binary         // false
json.fileExtensions // List("json", "map")

val jpeg = MediaTypes.image.`jpeg`
jpeg.compressible   // false
jpeg.binary         // true
jpeg.fileExtensions // List("jpg", "jpeg", "jpe")

val html = MediaTypes.text.`html`
html.compressible   // true
html.binary         // false
html.fileExtensions // List("html", "htm", "shtml")
```

## Core Operations

### `fullType`

Returns the complete media type string by combining `mainType` and `subType` with a `/` separator:

```scala

val mt = MediaType("application", "json")
mt.fullType // "application/json"

MediaType("*", "*").fullType  // "*/*"
MediaType("text", "*").fullType // "text/*"
```

### `matches`

Compares two `MediaType` values with support for wildcards, case insensitivity, and parameter matching:

```scala
final case class MediaType(...) {
  def matches(other: MediaType, ignoreParameters: Boolean = false): Boolean
}
```

The matching rules are:

1. **Wildcard matching** — `"*"` in either `mainType` or `subType` matches any value
2. **Case-insensitive** — `"APPLICATION/JSON"` matches `"application/json"`
3. **Parameter subset** — when `ignoreParameters = false` (the default), all parameters in `this` must exist in `other` with matching values (case-insensitive). Extra parameters in `other` are allowed.

```scala

val json    = mediaType"application/json"
val textAll = mediaType"text/*"
val html    = mediaType"text/html"
val any     = mediaType"*/*"

// Exact match
json.matches(json) // true

// Wildcard matching
any.matches(json)      // true — */* matches anything
textAll.matches(html)  // true — text/* matches text/html
textAll.matches(json)  // false — text/* does not match application/json

// Case-insensitive
MediaType("APPLICATION", "JSON").matches(json) // true

// Parameter matching
val htmlUtf8  = MediaType("text", "html", parameters = Map("charset" -> "utf-8"))
val htmlLatin = MediaType("text", "html", parameters = Map("charset" -> "iso-8859-1"))
val htmlFull  = MediaType("text", "html", parameters = Map("charset" -> "utf-8", "boundary" -> "xxx"))

htmlUtf8.matches(htmlLatin)                          // false — charset mismatch
htmlUtf8.matches(htmlLatin, ignoreParameters = true) // true  — parameters ignored
htmlUtf8.matches(htmlFull)                           // true  — subset match (charset matches)
```

### `forFileExtension`

Looks up a `MediaType` by file extension. The lookup is case-insensitive and strips a leading `.` if present:

```scala
object MediaType {
  def forFileExtension(ext: String): Option[MediaType]
}
```

```scala

MediaType.forFileExtension("json")  // Some(MediaType("application", "json", ...))
MediaType.forFileExtension(".html") // Some(MediaType("text", "html", ...))
MediaType.forFileExtension("PNG")   // Some(MediaType("image", "png", ...))
MediaType.forFileExtension("")      // None
MediaType.forFileExtension("xyz")   // None
```

:::tip
`forFileExtension` gives priority to `text/*` types when an extension maps to multiple types. For example, `"js"` maps to `text/javascript` rather than `application/javascript`.
:::

### `parse`

Parses a media type string into a `MediaType`, returning `Left` with an error message for invalid input:

```scala
object MediaType {
  def parse(s: String): Either[String, MediaType]
}
```

When the parsed type matches a predefined instance, that instance is returned (preserving reference equality and all metadata). Parameters from the input string are merged into the result:

```scala

// Returns predefined instance with full metadata
val json = MediaType.parse("application/json")
// Right(MediaType("application", "json", compressible=true, binary=false, ...))

// Parameters are parsed and attached
val result = MediaType.parse("multipart/form-data; boundary=abc; charset=utf-8")
// Right(MediaType(..., parameters = Map("boundary" -> "abc", "charset" -> "utf-8")))

// Unknown types get a fresh instance
val custom = MediaType.parse("custom/x-my-format")
// Right(MediaType("custom", "x-my-format"))

// Malformed parameters (no "=") are silently ignored
val partial = MediaType.parse("text/html; charset=utf-8; malformed")
// Right(MediaType(..., parameters = Map("charset" -> "utf-8")))
```

### `unsafeFromString`

Like `parse`, but throws `IllegalArgumentException` instead of returning `Left`:

```scala
object MediaType {
  def unsafeFromString(s: String): MediaType
}
```

For valid input, it returns the corresponding `MediaType` (reusing predefined instances when possible). For invalid input, it throws an exception with a descriptive message:

```scala

val json = MediaType.unsafeFromString("application/json")

// Throws IllegalArgumentException:
// MediaType.unsafeFromString("not-a-media-type")
```

## Advanced Usage

### Content Negotiation

We can use `matches` with wildcard types to implement HTTP-style content negotiation:

```scala

def negotiate(
  accept: List[MediaType],
  available: List[MediaType]
): Option[MediaType] =
  available.find(avail => accept.exists(_.matches(avail, ignoreParameters = true)))

val accept    = List(mediaType"text/*", mediaType"application/json")
val available = List(mediaType"application/xml", mediaType"application/json", mediaType"text/html")

negotiate(accept, available)
// Some(MediaType("application", "json", ...))
```

### File Type Detection

Combine `forFileExtension` with file path processing to detect content types:

```scala

def detectContentType(filename: String): Option[MediaType] = {
  val ext = filename.lastIndexOf('.') match {
    case -1 => ""
    case i  => filename.substring(i + 1)
  }
  MediaType.forFileExtension(ext)
}

detectContentType("report.pdf")   // Some(MediaType("application", "pdf", ...))
detectContentType("photo.jpg")    // Some(MediaType("image", "jpeg", ...))
detectContentType("data.json")    // Some(MediaType("application", "json", ...))
detectContentType("Makefile")     // None
```

---

## Meta Programming

## Introduction

Scala erases most type information at runtime, so a `List[Int]` and a `List[String]` look the same to the JVM. The blocks in this group turn type information into ordinary values that a program can inspect, store and act on. This lets a library derive serializers, validators and accessors from the shape of a type instead of requiring them to be written by hand.

## The Blocks

The group has two blocks, and [Schema](../schema/index.md) is the one most programs start with.

| Block                          | Artifact              | What It Is For                                                                           |
| ------------------------------ | --------------------- | ---------------------------------------------------------------------------------------- |
| [Schema](../schema/index.md)   | `zio-blocks-schema`   | Type-safe schemas with automatic codec, optic, and validator derivation                  |
| [TypeId](../typeid.md)         | `zio-blocks-typeid`   | Compile-time type identity with rich metadata                                            |

## How the Blocks Differ

`Schema` describes the structure of a data type, and one schema drives JSON support, validation and optics. Further formats such as Avro, BSON, CSV, MessagePack, Thrift, TOON, XML and YAML are separate modules, each with its own artifact (for example `zio-blocks-schema-avro`). The [overview](../../index.md) lists them with their platforms, and the [Schema](../schema/index.md) page documents them.

`TypeId` answers a narrower question: which type is this? It captures names, type parameters, parent types and annotations as data, which suits registries, code generation and dispatching on a type that would otherwise be erased.

## Where to Start

Start with [Schema](../schema/index.md) to define a data type once and derive codecs from it. Reach for [TypeId](../typeid.md) when a program needs a type's identity as a value rather than its structure. The [overview](../../index.md) compares every block in one table.

---

## Mux

`zio-blocks-mux` is a **high-performance multiplexer for ID-multiplexed protocols** (HTTP/2, QUIC, WebSockets with multiplexing, and other stream-based transports). It manages multiple concurrent independent streams over a shared transport, each identified by a unique ID, with separate inbound/outbound message queues and automatic state machine lifecycle management.

Core types: `Mux`, `MuxStream`, `MuxError`.

Create a mux and exchange messages:

```scala

val mux = Mux[Int, String, String](100)
val streamOrError = mux.open(1)
val stream = streamOrError match {
  case s: MuxStream[Int, String, String] => s
  case _: MuxError => sys.error("Failed to open stream")
}
stream.send("hello")
```

## Introduction

Multiplexing allows a single transport connection (TCP, QUIC, WebSocket) to carry multiple independent logical streams simultaneously. Each stream has its own ID, message queues, and lifecycle independent from others. The `Mux` primitive handles all the bookkeeping: stream creation, capacity enforcement, message queuing with backpressure, and graceful shutdown semantics.

## Motivation

Without multiplexing, protocols must open a new connection per concurrent operation (HTTP/1.1 with keep-alive), which is expensive and scales poorly. With multiplexing (HTTP/2, QUIC), one connection carries many streams, reducing connection overhead and latency while maintaining logical independence.

`Mux` provides:
- **Thread-safe stream registry** — `open`, `get`, `cancel`, `closeAll` and `activeCount` are safe to call from any thread. On the JVM, `open`, `cancel` and `closeAll` take a `ReentrantLock`, while `get` and `activeCount` read a `ConcurrentHashMap` without it; on Scala.js, which runs one thread, the registry is a plain `HashMap` with no locking
- **Lock-free per-stream queues** — `send()` and `offerInbound()` use lock-free ring buffers on JVM for high-throughput messaging
- **Separate inbound/outbound queues** — independent message directions, two-way communication
- **Automatic state machine** — stream lifecycle (OPEN → HALF_CLOSED_LOCAL/REMOTE → CLOSED) with proper half-close semantics
- **Backpressure** — per-stream and mux-level capacity limits to prevent unbounded buffering
- **Graceful shutdown** — `closeAll` atomically closes all streams with a terminal error
- **Thread-safe per-stream operations** — `send` and `offerInbound` are multi-thread safe; `receive` and `takeOutbound` follow single-consumer contract

## Installation

Add the dependency to your build:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-mux" % "0.0.56"
```

For Scala.js:

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-mux" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x

:::tip[Getting Started]
New to Mux? Check out the [Getting Started with Mux](../guides/getting-started-with-mux.md) tutorial for a comprehensive step-by-step guide that teaches you the core concepts, common patterns, and best practices. The tutorial is designed for newcomers and includes runnable examples.
:::

## Overview

**IMPORTANT: Error Handling with Union Types and Either**

In Scala 3, methods return union types (e.g., `Option[Out] | MuxError`). In Scala 2, they return `Either[MuxError, Option[Out]]`. You must use pattern matching to safely handle all branches:
- Success cases: `Some(msg)` or `None` (in Scala 3) / `Right(Some(msg))` or `Right(None)` (in Scala 2)
- Error cases: `MuxError` (in Scala 3) / `Left(error)` (in Scala 2)

Using higher-order functions like `.forEach()` or `.map()` without pattern matching will silently ignore error conditions and lose error information. Always explicitly pattern-match on all branches to handle both success and failure paths correctly.

The multiplexer has three core concepts:

**`Mux[Id, In, Out]`** is the entry point. You create it with a fixed capacity (maximum concurrent streams), then open, retrieve, cancel, or close streams. It enforces capacity limits and maintains a registry of active streams.

**`MuxStream[Id, In, Out]`** represents a single logical stream within the mux. It has two independent message queues: one for messages you send (outbound, drained by the protocol), and one for messages delivered to you (inbound, enqueued by the protocol). It also manages a state machine that enforces correct sequencing of sends and receives as the stream progresses through OPEN, HALF_CLOSED_LOCAL, HALF_CLOSED_REMOTE, and CLOSED states.

**`MuxError`** is a sealed trait representing all failure cases: `StreamClosed` (attempting operations on a closed stream), `CapacityExceeded` (too many concurrent streams), `QueueFull` (per-stream message queue exhausted), `Cancelled` (a reason you pass to `cancel` or `closeAll`; the library never creates it), `MuxClosed` (mux itself is closed), and `ProtocolError` (duplicate ID, null message).

## How They Work Together

The typical flow is:

**1. Create a mux** with a fixed capacity:

```scala

val mux = Mux[Int, String, String](100)
```

**2. Open a stream** (allocates a slot in the mux's capacity):

```scala

val mux = Mux[Int, String, String](100)
val streamOrError = mux.open(1)
val stream = streamOrError match {
  case s: MuxStream[Int, String, String] => s
  case error: MuxError => throw new RuntimeException(s"Failed: $error")
}
```

**3. Exchange messages** via the stream's two-way queue:
   - **Your side sends messages** via `stream.send(msg)`, placing them in the outbound queue.
   - **The protocol drains** those messages via `stream.takeOutbound()` and transmits them over the shared transport.
   - **The protocol receives** messages from the peer and delivers them via `stream.offerInbound(msg)`, placing them in the inbound queue.
   - **Your side receives messages** via `stream.receive()`, reading from the inbound queue.

**4. Signal end-of-stream** when either side is done sending:
   - Your side calls `stream.halfClose()` to signal you're done sending (local close).
   - The protocol calls `stream.signalRemoteClose()` when the peer signals end-of-stream (remote close).
   - Once both sides close, the stream transitions to CLOSED.

**5. Close or cancel** the stream to release capacity:
   - Call `stream.close()` to forcibly close a single stream.
   - Call `mux.cancel(id, reason)` to cancel a stream externally (e.g., protocol error).
   - Call `mux.closeAll(reason)` to atomically close all active streams and reject new opens.

This design mirrors **HTTP/2 stream lifecycle**: each stream is independent, supports half-closed states for proper shutdown, and the mux enforces capacity limits and graceful shutdown semantics.

The architecture shows how application code, mux streams, and protocol layers interact:

```
┌─────────────────────────────────────────────────────────┐
│ Your Application                                        │
└──────────────────────┬──────────────────────────────────┘
                       │
         send(msg)     │      receive(msg)
              ────────┬┴────────
                      │
        ┌─────────────▼──────────────┐
        │   MuxStream (Stream 1)     │
        │ ┌─────────────────────────┐│
        │ │ Outbound Queue          ││
        │ │ (messages to peer)      ││
        │ └─────────────┬───────────┘│
        │               │            │
        │ ┌─────────────┴───────────┐│
        │ │ Inbound Queue           ││
        │ │ (messages from peer)    ││
        │ └─────────────┬───────────┘│
        └───────────────┼────────────┘
                        │
         takeOutbound() │      offerInbound()
              ────────┬─┴──────────
                      │
┌─────────────────────▼──────────────────────────────────┐
│ Protocol Layer (HTTP/2 framing, etc.)                  │
└─────────────────────┬──────────────────────────────────┘
                      │
          Shared Transport (TCP, QUIC, etc.)
```

The mux holds multiple streams in a concurrent map. Each stream can be accessed independently:

<Tabs groupId="scala-version" defaultValue="scala3">
  <TabItem value="scala2" label="Scala 2">

```scala

val mux = Mux[Int, String, String](100)
val s1: Either[MuxError, MuxStream[Int, String, String]] = mux.open(1)
val stream1 = s1 match {
  case Right(s) => s
  case Left(error) => throw new RuntimeException(s"Failed: $error")
}

val s2: Either[MuxError, MuxStream[Int, String, String]] = mux.open(2)
val stream2 = s2 match {
  case Right(s) => s
  case Left(error) => throw new RuntimeException(s"Failed: $error")
}

stream1.send("hello")
stream2.send("world")
mux.get(1).foreach { stream =>
  stream.receive() match {
    case Right(Some(msg)) => println(s"Received: $msg")
    case Right(None) => println("No message yet")
    case Left(error) => println(s"Error: $error")
  }
}
mux.get(2).foreach { stream =>
  stream.receive() match {
    case Right(Some(msg)) => println(s"Received: $msg")
    case Right(None) => println("No message yet")
    case Left(error) => println(s"Error: $error")
  }
}
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3">

```scala

val mux = Mux[Int, String, String](100)

val s1: MuxStream[Int, String, String] | MuxError = mux.open(1)
val stream1 = s1 match {
  case s: MuxStream[Int, String, String] => s
  case error: MuxError => throw new RuntimeException(s"Failed: $error")
}

val s2: MuxStream[Int, String, String] | MuxError = mux.open(2)
val stream2 = s2 match {
  case s: MuxStream[Int, String, String] => s
  case error: MuxError => throw new RuntimeException(s"Failed: $error")
}

stream1.send("hello")
stream2.send("world")
mux.get(1).foreach { stream =>
  stream.receive() match {
    case Some(msg) => println(s"Received: $msg")
    case None => println("No message yet")
    case error: MuxError => println(s"Error: $error")
  }
}
mux.get(2).foreach { stream =>
  stream.receive() match {
    case Some(msg) => println(s"Received: $msg")
    case None => println("No message yet")
    case error: MuxError => println(s"Error: $error")
  }
}
```

  </TabItem>
</Tabs>

## Diagram

To see the mux data flow in action, use this interactive explorer. It shows three concurrent streams sharing the same mux. Select a stream tab, then click actions in either zone to watch messages move through the queues and observe how the state machine responds.

- **Application code** (top) — calls `send()` to enqueue outbound messages and `receive()` to dequeue inbound messages.
- **MuxStream** (middle) — holds the outbound and inbound ring-buffer queues and tracks stream state (`OPEN`, `HALF_CLOSED_LOCAL`, `HALF_CLOSED_REMOTE`, `CLOSED`).
- **Protocol layer** (bottom) — calls `takeOutbound()` to drain messages for transmission and `offerInbound()` to deliver messages arriving from the peer.

Use the **Lifecycle** controls to drive the state machine: `halfClose()` signals your side is done sending; `signalRemoteClose()` signals the peer is done; `close()` forces immediate closure. The event log shows the exact return values the API would produce, including `QueueFull` and `StreamClosed` errors.

<MuxDataFlow />

## Common Patterns

**Capacity Management**

Mux enforces a fixed capacity (e.g., 100 concurrent streams). When the limit is reached, `open` returns `CapacityExceeded`. Close or cancel streams to free capacity:

```scala

val mux = Mux[Int, String, String](5)
for (i <- 1 to 5) {
  mux.open(i) match {
    case _: MuxStream[Int, String, String] => ()
    case e: MuxError                       => println(s"Failed to open stream $i: $e")
  }
}
val result = mux.open(6) // returns CapacityExceeded — mux is full

mux.cancel(1, MuxError.Cancelled(1, "freed"))
val newStream = mux.open(6) // succeeds now that stream 1 was freed
```

**Half-Close Shutdown**

Streams support half-closed states (like HTTP/2). Your side calls `halfClose()` to signal it's done sending; the protocol calls `signalRemoteClose()` when the peer closes their sending side:

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case _: MuxError => sys.error("open failed")
}

stream.halfClose()
stream.receive()
stream.signalRemoteClose()
val sendResult = stream.send("fail")
```

**Backpressure and Queue Management**

Each stream has separate inbound and outbound queues (capacity 256 per queue). If the protocol producer is faster than the consumer, the queue fills and `offerInbound` or `send` returns `QueueFull`:

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case _: MuxError => sys.error("open failed")
}

for (i <- 1 to 256) {
  stream.send(s"msg-$i") match {
    case ()          => ()
    case e: MuxError => sys.error(s"send failed: $e")
  }
}
val result = stream.send("overflow")
stream.takeOutbound()
val recovered = stream.send("now-ok")
```

**Graceful Shutdown**

Call `closeAll` to atomically close all streams and prevent new opens. After closure, `receive()` / `takeOutbound()` return the terminal error once their queues are drained:

```scala

val mux = Mux[Int, String, String](100)
for (i <- 1 to 10) {
  mux.open(i) match {
    case _: MuxStream[?, ?, ?] => ()
    case e: MuxError           => sys.error(s"Failed to open stream $i: $e")
  }
}

mux.closeAll(MuxError.MuxClosed)

mux.activeCount
val newOpen = mux.open(11)
```

## Integration Points

`Mux` is designed as a protocol-independent multiplexing layer. It integrates with:

- **Transport protocols**: HTTP/2, QUIC, multiplexed WebSockets — any ID-based multiplexed protocol
- **Message types**: Generic over message types (`In`, `Out`) — use your protocol's frame/message types
- **Stream IDs**: Generic over ID type — use the protocol's stream ID type (Int for HTTP/2, Long for QUIC, etc.)
- **Error handling**: Terminal errors from protocol errors or local cancellation become available to user code via `receive()` and `takeOutbound()`

The mux does not depend on external modules except `zio-blocks-ringbuffer` for its lock-free ring buffer queues. It is pure and zero-dependency beyond that.

---

## MuxError

`MuxError` is a sealed trait representing all failure conditions in mux operations.

<Tabs groupId="scala-version" defaultValue="scala2">
  <TabItem value="scala2" label="Scala 2">

Here are the error type definitions for both Scala versions:

```scala
sealed trait MuxError

object MuxError {
  final case class StreamClosed(id: Any) extends MuxError
  final case class CapacityExceeded(limit: Int) extends MuxError
  final case class QueueFull(queueCapacity: Int) extends MuxError
  final case class Cancelled(id: Any, reason: String) extends MuxError
  case object MuxClosed extends MuxError
  final case class ProtocolError(message: String) extends MuxError
}
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3">

Scala 3 uses the same structure with union return types in method signatures:

```scala
sealed trait MuxError

object MuxError {
  final case class StreamClosed(id: Any) extends MuxError
  final case class CapacityExceeded(limit: Int) extends MuxError
  final case class QueueFull(queueCapacity: Int) extends MuxError
  final case class Cancelled(id: Any, reason: String) extends MuxError
  case object MuxClosed extends MuxError
  final case class ProtocolError(message: String) extends MuxError
}
```

  </TabItem>
</Tabs>

**Variants**

**`StreamClosed(id: Any)`** — Returned when attempting to send or receive on a stream that is already closed. The `id` field is typed as `Any` to keep `MuxError` non-generic; you can pattern-match on it or cast it at the use site if you need the concrete type.

**`CapacityExceeded(limit: Int)`** — Returned by `open` when the number of active streams has reached the mux's capacity limit. The `limit` field shows the configured capacity. Close or cancel other streams to free capacity.

**`QueueFull(queueCapacity: Int)`** — Returned by `send` (outbound queue full) or `offerInbound` (inbound queue full) when the per-stream message queue has exhausted its capacity (typically 256). Drain the queue (by calling `takeOutbound()` or `receive()`) to resume sending/receiving.

**`Cancelled(id: Any, reason: String)`** — A reason you can pass to `mux.cancel(id, reason)` or `closeAll`; the library never creates a `Cancelled` itself. `receive()` returns the reason you passed, whatever its case, so a `Cancelled` appears only when you pass one.

**`MuxClosed`** — Set when the mux itself is closed via `closeAll`. After this, `open` returns this error and all active streams transition to CLOSED.

**`ProtocolError(message: String)`** — Returned when the protocol contract is violated (duplicate stream ID, or a `null` message). These indicate bugs in the protocol implementation or incorrect API usage.

---

## Mux

`Mux[Id, In, Out]` is the entry point for multiplexed stream coordination. It manages a registry of active streams, enforces capacity limits, and provides operations to open, retrieve, cancel, and close streams.

**Factory**

To create a multiplexer, use the factory constructor with a fixed capacity:

```scala

val mux = Mux[Int, String, String](100)
```

The capacity must be positive; zero or negative capacity throws `IllegalArgumentException`. Capacity is fixed at construction and never changes.

**Core Operations**

**`def open(id: Id): MuxStream[Id, In, Out] | MuxError`** (Scala 3) / **`def open(id: Id): Either[MuxError, MuxStream[Id, In, Out]]`** (Scala 2) — Open a new stream with the given ID. Transitions the stream from IDLE to OPEN state. Returns the new stream on success, or an error if:
- Mux is closed (returns `MuxClosed`)
- Stream ID already exists (returns `ProtocolError`)
- Capacity exceeded (returns `CapacityExceeded`)

Opening a stream with error handling:

<Tabs groupId="scala-version" defaultValue="scala3">
  <TabItem value="scala2" label="Scala 2">

```scala

val mux = Mux[Int, String, String](100)

mux.open(1) match {
  case Right(stream) => println(s"Stream ${stream.id} opened")
  case Left(error) => println(s"Failed: $error")
}
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3">

```scala

val mux = Mux[Int, String, String](100)

mux.open(1) match {
  case stream: MuxStream[Int, String, String] => println(s"Stream ${stream.id} opened")
  case error: MuxError => println(s"Failed: $error")
}
```

  </TabItem>
</Tabs>

**`def get(id: Id): Option[MuxStream[Id, In, Out]]`** — Retrieve an existing stream by ID. Returns `Some(stream)` if the stream is open, `None` otherwise. This is a non-blocking lookup and does not modify any state.

Looking up a stream:

```scala

val mux = Mux[Int, String, String](100)
val result = mux.open(1)
val retrieved = mux.get(1)
```

**`def cancel(id: Id, reason: MuxError): Unit`** — Cancel a stream by ID, removing it from the mux and setting a terminal error. The stream transitions to CLOSED and any pending `receive()` calls on that stream return the terminal error. Cancelling a non-existent stream is a no-op.

Cancelling a stream:

```scala

val mux = Mux[Int, String, String](100)
mux.open(1)
mux.cancel(1, MuxError.Cancelled(1, "peer error"))
val retrieved = mux.get(1)
```

**`def closeAll(reason: MuxError): Unit`** — Close all active streams atomically with a terminal error. Transitions all streams to CLOSED, clears the stream registry, and sets the mux to a closed state. After `closeAll`, `open` returns `MuxClosed`.

Closing all streams:

```scala

val mux = Mux[Int, String, String](100)
for (i <- 1 to 10) {
  mux.open(i) match {
    case _: MuxStream[?, ?, ?] => ()
    case e: MuxError           => sys.error(s"Failed to open stream $i: $e")
  }
}
mux.closeAll(MuxError.MuxClosed)
val count = mux.activeCount
```

**`def activeCount: Int`** — Return the current number of active (open) streams. This is a point-in-time snapshot; the count may change immediately after if other threads open or close streams.

Checking the number of active streams:

```scala

val mux = Mux[Int, String, String](100)
mux.open(1)
mux.open(2)
val count = mux.activeCount
```

---

## MuxStream

`MuxStream[Id, In, Out]` represents a single logical stream within a mux. It has two independent message queues (inbound and outbound), a state machine, and operations to send/receive messages and manage lifecycle.

**Stream State Machine**

A stream progresses through four states:

- **OPEN** — Initial state after `open`. Both sides can send and receive.
- **HALF_CLOSED_LOCAL** — Local side calls `halfClose()`. Local side cannot send; remote side can still send (which transitions to CLOSED via `signalRemoteClose()`).
- **HALF_CLOSED_REMOTE** — Remote side calls `signalRemoteClose()`. Remote side cannot send; local side can still send (which transitions to CLOSED via `halfClose()`).
- **CLOSED** — Both sides closed or stream was forcibly closed. No operations allowed except state queries.

Attempting to send on a HALF_CLOSED_LOCAL or CLOSED stream returns `StreamClosed`. Attempting to offer inbound on a HALF_CLOSED_REMOTE or CLOSED stream returns `StreamClosed`. Attempting to receive on a fully CLOSED stream returns the terminal error after the queue drains.

The state transitions form a finite state machine:

```
          ┌─────────────────────────────────────────────┐
          │                   OPEN                      │
          │ (both sides can send and receive)           │
          └─┬─────────────────────────────────────────┬─┘
            │                                         │
            │ halfClose()                 signalRemoteClose()
            │                                         │
      ┌─────┴──────────────────┐         ┌────────────┴──────────┐
      │ HALF_CLOSED_LOCAL      │         │ HALF_CLOSED_REMOTE    │
      │ (local: done sending)  │         │ (remote: done sending)│
      │ (remote: can still rx) │         │ (local: can still tx) │
      └─┬──────────────────────┘         └──────────┬────────────┘
        │                                           │
        │ signalRemoteClose()          halfClose()  │
        │                                           │
        └──────────────┬────────────────────────────┘
                       │
                   ┌───┴────┐
                   │ CLOSED │
                   └────────┘
```

**Query Operations**

**`def id: Id`** — Return the stream's unique identifier within the mux.

**`def isClosed: Boolean`** — Return true if the stream is in CLOSED state, false otherwise.

**`def isHalfClosed: Boolean`** — Return true if the stream is in HALF_CLOSED_LOCAL or HALF_CLOSED_REMOTE state, false otherwise.

**Message Operations**

**`def send(msg: In): Unit | MuxError`** (Scala 3) / **`def send(msg: In): Either[MuxError, Unit]`** (Scala 2) — Send a message on this stream. Places the message into the outbound queue for the protocol to drain. The protocol is responsible for transmitting the message over the shared transport.

Returns `Unit` on success, or an error if:
- Stream is closed (returns `StreamClosed`)
- Stream is in HALF_CLOSED_LOCAL state (returns `StreamClosed`)
- Message is null (returns `ProtocolError`)
- Outbound queue is full (returns `QueueFull`)

Sending a message on a stream:

<Tabs groupId="scala-version" defaultValue="scala3">
  <TabItem value="scala2" label="Scala 2">

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case Right(s) => s
  case Left(_) => sys.error("open failed")
}

val sendResult = stream.send("hello")
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3">

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case _: MuxError => sys.error("open failed")
}

val sendResult = stream.send("hello")
```

  </TabItem>
</Tabs>

**`def receive(): Option[Out] | MuxError`** (Scala 3) / **`def receive(): Either[MuxError, Option[Out]]`** (Scala 2) — Receive a message from the inbound queue. This is non-blocking: it returns immediately with whatever is available.

Returns:
- `Some(msg)` if a message is available in the inbound queue
- `None` if the queue is empty (no message yet, but stream is still open)
- A `MuxError` if the stream is closed (queue has drained and a terminal error is set)

Use this in a polling loop or with a reactor to wait for messages:

<Tabs groupId="scala-version" defaultValue="scala3">
  <TabItem value="scala2" label="Scala 2">

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case Right(s) => s
  case Left(_) => sys.error("open failed")
}

val result = stream.receive()
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3">

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case _: MuxError => sys.error("open failed")
}

val result = stream.receive()
```

  </TabItem>
</Tabs>

**`def takeOutbound(): Option[In] | MuxError`** (Scala 3) / **`def takeOutbound(): Either[MuxError, Option[In]]`** (Scala 2) — Take the next message from the outbound queue. Called by the protocol to drain messages sent via `send()` and transmit them over the shared transport. Non-blocking: returns immediately.

Returns:
- `Some(msg)` if a message is available in the outbound queue
- `None` if the queue is empty (no outbound message yet, but stream is still open)
- A `MuxError` if the stream is closed

Draining messages from the outbound queue:

<Tabs groupId="scala-version" defaultValue="scala3">
  <TabItem value="scala2" label="Scala 2">

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case Right(s) => s
  case Left(_) => sys.error("open failed")
}

stream.send("outgoing")
val outbound = stream.takeOutbound()
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3">

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case _: MuxError => sys.error("open failed")
}

stream.send("outgoing")
val outbound = stream.takeOutbound()
```

  </TabItem>
</Tabs>

**`def offerInbound(msg: Out): Unit | MuxError`** (Scala 3) / **`def offerInbound(msg: Out): Either[MuxError, Unit]`** (Scala 2) — Deliver a message to this stream's inbound queue. Called by the protocol when a message arrives from the peer.

Returns `Unit` on success, or an error if:
- Stream is closed (returns `StreamClosed`)
- Stream is in HALF_CLOSED_REMOTE state (returns `StreamClosed`)
- Message is null (returns `ProtocolError`)
- Inbound queue is full (returns `QueueFull`)

Delivering a message to the inbound queue:

<Tabs groupId="scala-version" defaultValue="scala3">
  <TabItem value="scala2" label="Scala 2">

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case Right(s) => s
  case Left(_) => sys.error("open failed")
}

stream.offerInbound("response from peer")
val received = stream.receive()
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3">

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case _: MuxError => sys.error("open failed")
}

stream.offerInbound("response from peer")
val received = stream.receive()
```

  </TabItem>
</Tabs>

**Lifecycle Operations**

**`def halfClose(): Unit`** — Signal that the local side is done sending. Transitions the stream based on the current state:
- OPEN → HALF_CLOSED_LOCAL (local side cannot send; remote can still receive pending messages)
- HALF_CLOSED_REMOTE → CLOSED (both sides are now closed)

After `halfClose()`, `send()` returns `StreamClosed`. You can still call `receive()` to drain any buffered inbound messages.

Signalling local half-close:

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case _: MuxError => sys.error("open failed")
}

stream.halfClose()
val isHc = stream.isHalfClosed
val sendAfter = stream.send("fail")
val recAfter = stream.receive()
```

**`def signalRemoteClose(): Unit`** — Signal that the remote side is done sending. Called by the protocol when the peer signals END_STREAM. Transitions the stream based on the current state:
- OPEN → HALF_CLOSED_REMOTE (remote side cannot send; local can still receive pending messages)
- HALF_CLOSED_LOCAL → CLOSED (both sides are now closed)

After `signalRemoteClose()`, `offerInbound()` returns `StreamClosed`. You can still call `receive()` to drain any buffered inbound messages. Once they are drained, `receive()` keeps returning `None`, not an error, until the local side also half-closes, so `receive()` alone does not tell the application that the peer has finished. A protocol that needs to tell it delivers an end marker as the last inbound message, as [How to Run a Mux over a Real Socket](../guides/run-a-mux-over-a-socket.md) does.

Signalling remote half-close:

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case _: MuxError => sys.error("open failed")
}

stream.signalRemoteClose()
val isHc = stream.isHalfClosed
val offerAfter = stream.offerInbound("fail")
```

**`def close(): Unit`** — Forcibly close this stream immediately, transitioning it to CLOSED state and removing it from the mux. Any pending `receive()` calls on this stream will return the terminal error after the queue drains.

Forcibly closing a stream:

```scala

val mux = Mux[Int, String, String](100)
val stream = mux.open(1) match {
  case s: MuxStream[Int, String, String] => s
  case _: MuxError => sys.error("open failed")
}

stream.close()
val isClosed = stream.isClosed
val inMux = mux.get(1)
```

**Thread Safety**

- `send()` and `offerInbound()` are **multi-thread safe**: multiple threads can safely call these concurrently on the same stream.
- `receive()` and `takeOutbound()` are **single-consumer**: on each stream, at most one thread at a time may call `receive()`, and at most one thread at a time may call `takeOutbound()`. The two can be different threads, which is the usual layout: the application calls `receive()` and the protocol's writer calls `takeOutbound()`. Two threads calling the same one concurrently corrupt the ring buffer: messages are delivered twice or lost, and the queue can stop accepting new messages.
- State queries (`id`, `isClosed`, `isHalfClosed`) are always safe to call from any thread.

## Performance

- JVM: `MpscRingBuffer` for inbound and outbound, lock-free and zero-alloc for multi-producer writes
- JVM: `VarHandle` CAS for stream state transitions
- JS: `ArrayDeque` fallback
- No `synchronized`: the registry's lock is a `ReentrantLock`, and the per-stream queues and state transitions take no lock, so a virtual thread never pins its carrier thread inside `Mux`

## See Also

- [How to Run a Mux over a Real Socket](../guides/run-a-mux-over-a-socket.md) -- wires a `Mux` to a TCP connection: a frame format, a reader loop and a writer loop, half-close, back-pressure, resets and shutdown.
- [Streams](streams/index.md) -- the pull-based `Stream`, `Reader`, `Sink`, and `Writer` module. `Mux` is not built on it, but the two are the same concurrency-infrastructure family, and a stream per multiplexed channel is the natural pairing when you are coordinating many keyed streams over one transport.
- [Async Execution](streams/execution-and-compatibility/async-execution.md) -- how a stream runs without blocking, which is what you want on each side of a `Mux` channel.

---

## OpenAPI

`zio-blocks-openapi` is a **complete, type-safe OpenAPI 3.1 data model** for building API documentation programmatically. It provides immutable case classes and sealed traits representing every OpenAPI concept—operations, parameters, security schemes, and components—enabling you to construct OpenAPI documents in compile-time-safe Scala and export them as JSON for consumption by tools like Swagger UI, Redoc, and API validators.

Core types: `OpenAPI`, `Info`, `Paths`, `PathItem`, `Operation`, `Parameter`, `RequestBody`, `Response`, `Components`, `SchemaObject`, `SecurityScheme`, `ReferenceOr`.

```scala
final case class OpenAPI(
  openapi: String,
  info: Info,
  jsonSchemaDialect: Option[String] = None,
  servers: Chunk[Server] = Chunk.empty,
  paths: Option[Paths] = None,
  webhooks: ChunkMap[String, ReferenceOr[PathItem]] = ChunkMap.empty,
  components: Option[Components] = None,
  security: Chunk[SecurityRequirement] = Chunk.empty,
  tags: Chunk[Tag] = Chunk.empty,
  externalDocs: Option[ExternalDocumentation] = None,
  extensions: ChunkMap[String, Json] = ChunkMap.empty
)
```

See also: [How to Write an OpenAPI Document and Export It as JSON or YAML](../guides/write-and-export-openapi.md) walks through a complete document, from component schemas to JSON and YAML output. For the opposite direction, [How to Generate Scala Types from an OpenAPI Spec at Build Time](../guides/generate-scala-types-from-openapi.md) parses a document and generates Scala types from its component schemas.

## Introduction

OpenAPI documents are the **lingua franca for API specifications**. They define request/response contracts, authentication methods, and data schemas in a standardized JSON or YAML format that external tools consume. Building these documents manually in JSON is error-prone; maintaining them as your API evolves is tedious.

The OpenAPI module bridges the gap by letting you author API specs as Scala code—leveraging the type system for compile-time correctness—then export to standard JSON that any OpenAPI tool understands. You get type safety during authoring plus interoperability with the entire OpenAPI ecosystem.

## Installation

```scala
// Scala 2.13, and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-openapi" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-openapi" % "0.0.56"
scalacOptions += "-experimental"
```

The module brings in `zio-blocks-schema`, which provides `Schema[A]` integration. For Scala.js, use `%%%` instead of `%%`. To export documents as YAML, also add `zio-blocks-schema-yaml` (`zio-blocks-next-schema-yaml` on Scala 3.9 and newer).

## How They Work Together

The OpenAPI module follows a clear workflow:

**1. Define your data types** using ZIO Blocks `Schema`:

```scala

case class User(id: Int, name: String, email: String)
object User {
  implicit val schema: Schema[User] = Schema.derived
}

case class ErrorResponse(code: Int, message: String)
object ErrorResponse {
  implicit val schema: Schema[ErrorResponse] = Schema.derived
}
```

**2. Create an OpenAPI document** by composing types:

```scala
val api = OpenAPI(
  openapi = "3.1.0",
  info = Info(
    title = "User API",
    version = "1.0.0",
    description = Some(md"API for managing users")
  ),
  paths = Some(Paths(ChunkMap(
    "/users" -> PathItem(
      get = Some(Operation(
        summary = Some(md"List all users"),
        description = Some(md"Returns a paginated list of users"),
        responses = Responses(ChunkMap(
          "200" -> ReferenceOr.Value(Response(
            description = md"Successful response",
            content = ChunkMap(
              "application/json" -> MediaType(
                schema = Some(ReferenceOr.Value(
                  Schema[List[User]].toOpenAPISchema
                ))
              )
            )
          ))
        ))
      ))
    ),
    "/users/{id}" -> PathItem(
      get = Some(Operation(
        summary = Some(md"Get a user by ID"),
        parameters = Chunk(
          ReferenceOr.Value(Parameter(
            name = "id",
            in = ParameterLocation.Path,
            required = true,
            schema = Some(ReferenceOr.Value(
              Schema[Int].toOpenAPISchema
            ))
          ))
        ),
        responses = Responses(ChunkMap(
          "200" -> ReferenceOr.Value(Response(
            description = md"User found",
            content = ChunkMap(
              "application/json" -> MediaType(
                schema = Some(ReferenceOr.Value(
                  Schema[User].toOpenAPISchema
                ))
              )
            )
          )),
          "404" -> ReferenceOr.Value(Response(
            description = md"User not found",
            content = ChunkMap(
              "application/json" -> MediaType(
                schema = Some(ReferenceOr.Value(
                  Schema[ErrorResponse].toOpenAPISchema
                ))
              )
            )
          ))
        ))
      ))
    )
  ))),
  components = Some(Components(
    schemas = ChunkMap(
      Schema[User].toRefSchema._2._1 -> ReferenceOr.Value(Schema[User].toRefSchema._2._2),
      Schema[ErrorResponse].toRefSchema._2._1 -> ReferenceOr.Value(Schema[ErrorResponse].toRefSchema._2._2)
    )
  ))
)
```

**3. Serialize to JSON** for tools to consume:

```scala

val json = openAPICodec.encodeValue(api)
```

**4. Render or serve** the JSON (e.g., to Swagger UI):

```scala

val jsonString = Json.jsonCodec.encodeToString(json, WriterConfig.withIndentionStep2)
// jsonString: String = """{
//   "openapi": "3.1.0",
//   "info": {
//     "title": "User API",
//     "version": "1.0.0",
//     "description": "API for managing users\n\n"
//   },
//   "paths": {
//     "/users": {
//       "get": {
//         "responses": {
//           "200": {
//             "description": "Successful response\n\n",
//             "content": {
//               "application/json": {
//                 "schema": {
//                   "items": {
//                     "properties": {
//                       "id": {
//                         "type": "integer",
//                         "maximum": 2147483647,
//                         "minimum": -2147483648
//                       },
//                       "name": {
//                         "type": "string"
//                       },
//                       "email": {
//                         "type": "string"
//                       }
//                     },
//                     "type": "object",
//                     "required": [
//                       "id",
//                       "name",
//                       "email"
//                     ],
//                     "title": "User"
//                   },
//                   "type": [
//                     "array",
//                     "null"
//                   ],
//                   "title": "List[User]"
//                 }
//               }
//             }
//           }
//         },
//         "summary": "List all users\n\n",
// ...
```

### Type Relationships Diagram

```
OpenAPI (root document)
├─ info: Info (metadata)
├─ servers: Chunk[Server]
├─ paths: Option[Paths] (map of path strings to PathItem)
│  └─ PathItem
│     ├─ get: Operation
│     ├─ post: Operation
│     ├─ put: Operation
│     └─ ... (other HTTP methods)
│        ├─ parameters: Chunk[ReferenceOr[Parameter]]
│        ├─ requestBody: Option[ReferenceOr[RequestBody]]
│        │  └─ content: ChunkMap[String, MediaType]
│        │     └─ schema: ReferenceOr[SchemaObject]
│        └─ responses: Responses
│           └─ ChunkMap[statusCode, ReferenceOr[Response]]
│              └─ content: ChunkMap[String, MediaType]
│                 └─ schema: ReferenceOr[SchemaObject]
├─ components: Option[Components]
│  ├─ schemas: ChunkMap[String, ReferenceOr[SchemaObject]]
│  ├─ responses: ChunkMap[String, ReferenceOr[Response]]
│  ├─ parameters: ChunkMap[String, ReferenceOr[Parameter]]
│  └─ securitySchemes: ChunkMap[String, ReferenceOr[SecurityScheme]]
└─ security: Chunk[SecurityRequirement]
```

## Common Patterns

### Building Reusable Schema Components

Avoid duplicating schema definitions by moving them to `components.schemas`:

```scala

case class User(id: Int, name: String, email: String)
object User {
  implicit val schema: Schema[User] = Schema.derived
}

val userSchemaComponent = Schema[User].toRefSchema
// Returns: (ReferenceOr.Ref(...), ("User", SchemaObject(...)))
// Use the ref in operations, store the component in components.schemas
```

### Using `ReferenceOr` for Inline vs. Referenced Schemas

`ReferenceOr[A]` is a sealed trait with two cases:

- **`ReferenceOr.Ref`**: Points to a schema in `#/components/schemas/<name>`
- **`ReferenceOr.Value`**: Inline schema definition

Prefer `Ref` for reusable schemas; use `Value` for simple, one-off schemas:

```scala

// Reusable: use Ref
val userRef = ReferenceOr.Ref(Reference(`$ref` = "#/components/schemas/User"))

// One-off: use Value
val simpleString = ReferenceOr.Value(Schema[String].toOpenAPISchema)
```

### Security Schemes

Define authentication methods in `components.securitySchemes`:

```scala

val apiKeyScheme = SecurityScheme.APIKey(
  name = "X-API-Key",
  in = APIKeyLocation.Header,
  description = Some(md"API key for authentication")
)

val oauthScheme = SecurityScheme.OAuth2(
  flows = OAuthFlows(
    authorizationCode = Some(OAuthFlow(
      authorizationUrl = Some("https://example.com/oauth/authorize"),
      tokenUrl = Some("https://example.com/oauth/token"),
      scopes = ChunkMap("read" -> "Read access", "write" -> "Write access")
    ))
  ),
  description = Some(md"OAuth 2.0 authorization")
)

val components = Components(
  securitySchemes = ChunkMap(
    "api_key" -> ReferenceOr.Value(apiKeyScheme),
    "oauth2" -> ReferenceOr.Value(oauthScheme)
  )
)
```

### Path Parameters vs. Query Parameters

Distinguish parameter locations using `ParameterLocation`:

```scala

val pathParam = Parameter(
  name = "id",
  in = ParameterLocation.Path,
  required = true,
  schema = Some(ReferenceOr.Value(Schema[Int].toOpenAPISchema))
)

val queryParam = Parameter(
  name = "limit",
  in = ParameterLocation.Query,
  required = false,
  schema = Some(ReferenceOr.Value(Schema[Int].toOpenAPISchema))
)

val headerParam = Parameter(
  name = "X-Custom-Header",
  in = ParameterLocation.Header,
  required = false,
  schema = Some(ReferenceOr.Value(Schema[String].toOpenAPISchema))
)
```

## Integration Points

The OpenAPI module integrates tightly with other ZIO Blocks components:

- **Schema Integration**: All OpenAPI types have `Schema.derived` instances, enabling round-trip serialization via `DynamicValue`. Use `Schema[A].toOpenAPISchema` to convert any schema to an OpenAPI component.
- **Markdown Support**: Description fields use the in-house `Doc` type, which supports CommonMark rendering. The encoder writes each description as Markdown text, which ends with a blank line (`\n\n`).
- **JSON AST**: All codecs operate on the `Json` AST from `zio-blocks-schema`, not external JSON libraries. Encode with `openAPICodec.encodeValue` and print the `Json`; `openAPICodec.encodeToString` and `encode` throw `NotImplementedError` in this release. To render as YAML, convert the `Json` with `Yaml.fromJson` from `zio-blocks-schema-yaml`, as shown in [How to Write an OpenAPI Document and Export It as JSON or YAML](../guides/write-and-export-openapi.md).

---

## OpenAPI

`OpenAPI` is the root document object representing a complete OpenAPI 3.1 specification.

### Definition

Every OpenAPI document requires:
- **`openapi`**: Version string (typically `"3.1.0"`)
- **`info`**: Metadata about the API (`Info`)

Optional top-level fields include:
- **`servers`**: Server definitions for the API (`Chunk[Server]`)
- **`paths`**: Map of endpoint paths to operations (`Paths`)
- **`components`**: Reusable schemas, responses, parameters, and other components (`Components`)
- **`security`**: Security requirements applied to the API (`Chunk[SecurityRequirement]`)

Response definitions are modeled per `Operation`, not as a top-level field on `OpenAPI`.

### Creating an OpenAPI Document

To construct an `OpenAPI` document:

```scala

val minimalApi = OpenAPI(
  openapi = "3.1.0",
  info = Info(title = "My API", version = "1.0.0")
)
```

Add paths, operations, and components as shown in the "How They Work Together" section above.

### Serialization

Encode an `OpenAPI` document to `Json` AST:

```scala

val myApi = OpenAPI(openapi = "3.1.0", info = Info(title = "My API", version = "1.0.0"))
val encoded: Json = openAPICodec.encodeValue(myApi)
```

Decode from `Json` AST back to an `OpenAPI` instance:

```scala

val myApi = OpenAPI(openapi = "3.1.0", info = Info(title = "My API", version = "1.0.0"))
val encoded: Json = openAPICodec.encodeValue(myApi)
val decoded: OpenAPI = openAPICodec.decodeValue(encoded)
```

---

## Info

`Info` contains metadata about the API: title, version, contact, and license.

### Definition

Required fields:
- **`title`**: API name (e.g., `"User API"`)
- **`version`**: API version (e.g., `"1.0.0"`)

Optional fields:
- **`description`**: Markdown-formatted description (`Doc`)
- **`termsOfService`**: Terms of service URL
- **`contact`**: Contact information (`Contact`)
- **`license`**: License information (`License`)

### Creating Info

```scala

val info = Info(
  title = "Pet Store API",
  version = "3.0.0",
  description = Some(md"API for managing a pet store"),
  contact = Some(Contact(
    name = Some("API Support"),
    url = Some("https://example.com/support"),
    email = Some("support@example.com")
  )),
  license = Some(License(
    name = "Apache 2.0",
    identifier = Some("Apache-2.0")
  ))
)
```

---

## Paths & PathItem

`Paths` represents the collection of URL paths and their operations. `PathItem` groups HTTP methods (GET, POST, PUT, etc.) on a single path.

### Definition

`Paths` is a wrapper case class with two fields:
- **`paths`**: `ChunkMap[String, PathItem]`, where keys are path strings (e.g., `"/users/{id}"`)
- **`extensions`**: `ChunkMap[String, Json]`, for OpenAPI specification extensions

`PathItem` contains optional fields for each HTTP method:
- **`get`, `post`, `put`, `delete`, `patch`, `head`, `options`, `trace`**: `Operation` instances
- **`parameters`**: Path-level parameters shared by all methods on this path
- **`servers`**: Optional server overrides for this path

### Creating Path Items

To define a path with multiple operations:

```scala

val userPaths = Paths(ChunkMap(
  "/users" -> PathItem(
    get = Some(Operation(
      summary = Some(md"List users"),
      responses = Responses(ChunkMap(
        "200" -> ReferenceOr.Value(Response(
          description = md"User list",
          content = ChunkMap(
            "application/json" -> MediaType(schema = None)
          )
        ))
      ))
    )),
    post = Some(Operation(
      summary = Some(md"Create user"),
      requestBody = Some(ReferenceOr.Value(RequestBody(
        description = Some(md"User data"),
        content = ChunkMap(
          "application/json" -> MediaType(schema = None)
        ),
        required = true
      ))),
      responses = Responses(ChunkMap(
        "201" -> ReferenceOr.Value(Response(
          description = md"User created",
          content = ChunkMap(
            "application/json" -> MediaType(schema = None)
          )
        ))
      ))
    ))
  ),
  "/users/{id}" -> PathItem(
    parameters = Chunk(
      ReferenceOr.Value(Parameter(
        name = "id",
        in = ParameterLocation.Path,
        required = true,
        schema = Some(ReferenceOr.Value(Schema[String].toOpenAPISchema))
      ))
    ),
    get = Some(Operation(
      summary = Some(md"Get user by ID"),
      responses = Responses(ChunkMap(
        "200" -> ReferenceOr.Value(Response(
          description = md"User found",
          content = ChunkMap(
            "application/json" -> MediaType(schema = None)
          )
        )),
        "404" -> ReferenceOr.Value(Response(
          description = md"User not found",
          content = ChunkMap(
            "application/json" -> MediaType(schema = None)
          )
        ))
      ))
    ))
  )
))
```

---

## Operation

`Operation` represents a single HTTP operation (GET, POST, etc.) on a path.

### Definition

Key fields:
- **`responses`**: Map of status codes to response definitions. OpenAPI requires it; the constructor defaults it to an empty `Responses()`, which is encoded as `{}`
- **`operationId`**: Unique operation identifier
- **`summary`**: Short description
- **`description`**: Detailed markdown description
- **`parameters`**: Path, query, header, and cookie parameters
- **`requestBody`**: Request payload definition
- **`deprecated`**: Whether the operation is deprecated
- **`tags`**: Group operations in documentation (e.g., `"users"`, `"products"`)

### Defining an Operation

With summary, description, and parameters:

```scala

val getUser = Operation(
  tags = Chunk("users"),
  summary = Some(md"Retrieve user"),
  description = Some(md"Fetches a single user by ID"),
  operationId = Some("getUserById"),
  parameters = Chunk(
    ReferenceOr.Value(Parameter(
      name = "id",
      in = ParameterLocation.Path,
      required = true,
      schema = Some(ReferenceOr.Value(Schema[String].toOpenAPISchema)),
      description = Some(md"User ID")
    ))
  ),
  responses = Responses(ChunkMap(
    "200" -> ReferenceOr.Value(Response(
      description = md"User found",
      content = ChunkMap(
        "application/json" -> MediaType(schema = None)
      )
    )),
    "404" -> ReferenceOr.Value(Response(
      description = md"User not found",
      content = ChunkMap(
        "application/json" -> MediaType(schema = None)
      )
    ))
  ))
)
```

---

## Parameter

`Parameter` represents query, path, header, or cookie parameters in a request.

### Definition

Required fields:
- **`name`**: Parameter name (e.g., `"id"`, `"limit"`)
- **`in`**: Location—`Path`, `Query`, `Header`, or `Cookie` (`ParameterLocation`)

Optional fields:
- **`schema`**: Data type of the parameter (`Option[ReferenceOr[SchemaObject]]`, default: `None`)
- **`description`**: Markdown description
- **`required`**: Whether the parameter is mandatory (default: `false`, also for path parameters, which OpenAPI requires to be `true`)
- **`deprecated`**: Whether the parameter is deprecated
- **`allowEmptyValue`**: Whether empty string values are allowed

### Creating Parameters

Path parameter (required):

```scala

val idPathParam = Parameter(
  name = "id",
  in = ParameterLocation.Path,
  required = true,
  schema = Some(ReferenceOr.Value(Schema[String].toOpenAPISchema)),
  description = Some(md"User identifier")
)
```

Query parameter (optional with default):

```scala

val limitQueryParam = Parameter(
  name = "limit",
  in = ParameterLocation.Query,
  required = false,
  schema = Some(ReferenceOr.Value(Schema[Int].toOpenAPISchema)),
  description = Some(md"Maximum number of results (default: 20)")
)
```

Header parameter:

```scala

val authHeaderParam = Parameter(
  name = "X-API-Key",
  in = ParameterLocation.Header,
  required = true,
  schema = Some(ReferenceOr.Value(Schema[String].toOpenAPISchema)),
  description = Some(md"API key for authentication")
)
```

---

## RequestBody & Response

`RequestBody` defines the structure of a request payload. `Response` defines the structure and status of a response.

### RequestBody Definition

Key fields:
- **`content`**: Map of MIME types to `MediaType` definitions
- **`description`**: Optional markdown description
- **`required`**: Whether the request body is mandatory (default: `false`)

### Creating a RequestBody

```scala

case class User(name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val createUserBody = RequestBody(
  description = Some(md"User data to create"),
  content = ChunkMap(
    "application/json" -> MediaType(
      schema = Some(ReferenceOr.Value(Schema[User].toOpenAPISchema)),
      example = Some(Json.Object(Chunk(
        "name" -> Json.String("John Doe"),
        "email" -> Json.String("john@example.com")
      )))
    )
  ),
  required = true
)
```

### Response Definition

Key fields:
- **`description`**: Required. Markdown description of the response
- **`content`**: Map of MIME types to `MediaType` definitions
- **`headers`**: Optional response headers
- **`links`**: Optional links to related operations

### Creating a Response

```scala

case class User2(id: Int, name: String, email: String)
object User2 { implicit val schema: Schema[User2] = Schema.derived }
case class ErrorResponse2(code: Int, message: String)
object ErrorResponse2 { implicit val schema: Schema[ErrorResponse2] = Schema.derived }

val successResponse = Response(
  description = md"User successfully created",
  content = ChunkMap(
    "application/json" -> MediaType(
      schema = Some(ReferenceOr.Value(Schema[User2].toOpenAPISchema))
    )
  )
)

val errorResponse = Response(
  description = md"Request validation failed",
  content = ChunkMap(
    "application/json" -> MediaType(
      schema = Some(ReferenceOr.Value(Schema[ErrorResponse2].toOpenAPISchema))
    )
  )
)
```

### Responses

`Responses` is a map of HTTP status codes to `ReferenceOr[Response]`:

```scala

val ok = Response(description = md"Created", content = ChunkMap())
val err = Response(description = md"Bad request", content = ChunkMap())

val responses = Responses(ChunkMap(
  "201" -> ReferenceOr.Value(ok),
  "400" -> ReferenceOr.Value(err),
  "401" -> ReferenceOr.Value(Response(
    description = md"Unauthorized",
    content = ChunkMap()
  )),
  "500" -> ReferenceOr.Value(Response(
    description = md"Internal server error",
    content = ChunkMap()
  ))
))
```

---

## MediaType

`MediaType` specifies the schema and encoding for a particular MIME type in a request or response.

### Definition

Key fields:
- **`schema`**: Data type for this MIME type (`ReferenceOr[SchemaObject]`)
- **`example`**: Example value as `Json`
- **`encoding`**: Encoding rules for multipart form data
- **`extensions`**: Custom vendor extensions (`x-*` fields)

### Creating MediaType

With schema and example:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val jsonMedia = MediaType(
  schema = Some(ReferenceOr.Value(Schema[User].toOpenAPISchema)),
  example = Some(Json.Object(
    "id" -> Json.Number(1),
    "name" -> Json.String("Alice"),
    "email" -> Json.String("alice@example.com")
  ))
)
```

For form data:

```scala

val formMedia = MediaType(
  schema = Some(ReferenceOr.Value(Schema[Map[String, String]].toOpenAPISchema)),
  encoding = ChunkMap(
    "file" -> Encoding(
      contentType = Some("application/octet-stream")
    )
  )
)
```

---

## Components

`Components` stores reusable schema and security definitions referenced throughout the document.

### Definition

Key fields:
- **`schemas`**: Reusable schema objects (`ChunkMap[String, ReferenceOr[SchemaObject]]`)
- **`responses`**: Reusable response definitions
- **`parameters`**: Reusable parameter definitions
- **`securitySchemes`**: Authentication method definitions
- **`examples`**, **`requestBodies`**, **`headers`**, **`links`**, **`callbacks`**: Additional reusable components

### Creating Components

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }
case class ErrorResponse(code: Int, message: String)
object ErrorResponse { implicit val schema: Schema[ErrorResponse] = Schema.derived }

val components = Components(
  schemas = ChunkMap(
    Schema[User].toRefSchema._2._1 -> ReferenceOr.Value(Schema[User].toRefSchema._2._2),
    Schema[ErrorResponse].toRefSchema._2._1 -> ReferenceOr.Value(Schema[ErrorResponse].toRefSchema._2._2)
  ),
  parameters = ChunkMap(
    "id" -> ReferenceOr.Value(Parameter(
      name = "id",
      in = ParameterLocation.Path,
      required = true,
      schema = Some(ReferenceOr.Value(Schema[String].toOpenAPISchema))
    ))
  ),
  responses = ChunkMap(
    "NotFound" -> ReferenceOr.Value(Response(
      description = md"Resource not found",
      content = ChunkMap(
        "application/json" -> MediaType(
          schema = Some(ReferenceOr.Value(
            Schema[ErrorResponse].toOpenAPISchema
          ))
        )
      )
    )),
    "Unauthorized" -> ReferenceOr.Value(Response(
      description = md"Unauthorized access",
      content = ChunkMap()
    ))
  ),
  securitySchemes = ChunkMap(
    "api_key" -> ReferenceOr.Value(SecurityScheme.APIKey(
      name = "X-API-Key",
      in = APIKeyLocation.Header,
      description = Some(md"API key header")
    ))
  )
)
```

---

## SchemaObject

`SchemaObject` wraps a JSON Schema 2020-12 definition with OpenAPI-specific extensions like discriminator, XML metadata, and examples.

### Definition

`SchemaObject` contains:
- **`jsonSchema`**: Raw JSON Schema 2020-12 as `Json` AST
- **`discriminator`**: Polymorphism discriminator for oneOf/anyOf
- **`xml`**: XML serialization metadata
- **`example`**: Example value for documentation
- **`extensions`**: Custom `x-*` fields

### Creating SchemaObject

Directly from a `Schema[A]`:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val userSchema = Schema[User].toOpenAPISchema
// Returns a SchemaObject with the User type's JSON Schema
```

Or with additional OpenAPI metadata:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val enrichedSchema = SchemaObject(
  jsonSchema = Schema[User].toJsonSchema.toJson,
  discriminator = None,
  xml = Some(XML(
    name = Some("user"),
    namespace = None,
    prefix = None,
    attribute = false,
    wrapped = false
  )),
  example = Some(Json.Object(
    "id" -> Json.Number(1),
    "name" -> Json.String("John")
  )),
  extensions = ChunkMap(
    "x-generated" -> Json.String("true"),
    "x-version" -> Json.String("1.0.0")
  )
)
```

### Converting Schemas to SchemaObject

Use the `SchemaOps` extension methods on any `Schema[A]`:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val schemaObj: SchemaObject = Schema[User].toOpenAPISchema
val (ref, component) = Schema[User].toRefSchema
// ref = ReferenceOr.Ref pointing to #/components/schemas/User
// component = ("User", SchemaObject(...))
```

---

## ReferenceOr

`ReferenceOr[A]` is a sealed trait representing the OpenAPI pattern of choosing between a `$ref` and an inline value.

### Definition

Two cases:
- **`ReferenceOr.Ref`**: Points to a definition at `#/components/<type>/<name>`
- **`ReferenceOr.Value`**: Inline definition without reference

### Using ReferenceOr

Prefer `Ref` for reusable components:

```scala

val userRef = ReferenceOr.Ref(Reference(`$ref` = "#/components/schemas/User"))

val responseRef = ReferenceOr.Ref(Reference(
  `$ref` = "#/components/responses/NotFound"
))
```

Use `Value` for inline, one-off definitions:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val inlineUser = ReferenceOr.Value(Schema[User].toOpenAPISchema)

val inlineError = ReferenceOr.Value(Response(
  description = md"Quick error",
  content = ChunkMap()
))
```

### Pattern Matching on ReferenceOr

```scala

def describeRef[A](ref: ReferenceOr[A]): String = ref match {
  case ReferenceOr.Ref(r)   => s"Reference to ${r.`$ref`}"
  case ReferenceOr.Value(_) => "Inline value"
}
```

---

## SecurityScheme

`SecurityScheme` is a sealed trait representing different authentication methods. Variants include API Key, HTTP Basic/Bearer, OAuth 2.0, OpenID Connect, and Mutual TLS.

### Definition

Sealed trait variants:
- **`APIKey`**: API key in header, query, or cookie
- **`HTTP`**: HTTP authentication (Basic, Bearer, etc.)
- **`OAuth2`**: OAuth 2.0 authorization flows
- **`OpenIdConnect`**: OpenID Connect discovery
- **`MutualTLS`**: Mutual TLS certificate-based

### Creating Security Schemes

API Key authentication:

```scala

val apiKeySecurity = SecurityScheme.APIKey(
  name = "X-API-Key",
  in = APIKeyLocation.Header,
  description = Some(md"API key required in header")
)
```

HTTP Bearer token:

```scala

val bearerSecurity = SecurityScheme.HTTP(
  scheme = "bearer",
  bearerFormat = Some("JWT"),
  description = Some(md"JWT bearer token")
)
```

OAuth 2.0:

```scala

val oauthSecurity = SecurityScheme.OAuth2(
  flows = OAuthFlows(
    authorizationCode = Some(OAuthFlow(
      authorizationUrl = Some("https://example.com/oauth/authorize"),
      tokenUrl = Some("https://example.com/oauth/token"),
      scopes = ChunkMap(
        "read:users" -> "Read user data",
        "write:users" -> "Modify user data"
      )
    ))
  ),
  description = Some(md"OAuth 2.0 authorization")
)
```

:::note
The `OAuthFlows` type supports multiple flow types: `implicit`, `password`, `clientCredentials`, and `authorizationCode`. Choose the flow that matches your OAuth 2.0 configuration.
:::

OpenID Connect:

```scala

val oidcSecurity = SecurityScheme.OpenIdConnect(
  openIdConnectUrl = "https://example.com/.well-known/openid-configuration",
  description = Some(md"OpenID Connect discovery")
)
```

---

## Discriminator

`Discriminator` specifies how to distinguish between different variants in a polymorphic schema (using `oneOf` or `anyOf`).

### Definition

Key fields:
- **`propertyName`**: Field name used to discriminate (e.g., `"type"`, `"kind"`)
- **`mapping`**: Optional explicit mapping of discriminator values to schema references

### Creating a Discriminator

Simple discriminator by property name:

```scala

val discriminator = Discriminator(propertyName = "type")
```

With explicit value-to-schema mapping:

```scala

val mappedDiscriminator = Discriminator(
  propertyName = "kind",
  mapping = ChunkMap(
    "user" -> "#/components/schemas/User",
    "admin" -> "#/components/schemas/Admin",
    "guest" -> "#/components/schemas/Guest"
  )
)
```

---

## Server & ServerVariable

`Server` specifies base URLs and server-specific variables. `ServerVariable` allows parameterization of server URLs.

### Definition

`Server` contains:
- **`url`**: Server URL (may contain variable placeholders like `{base_path}`)
- **`description`**: Optional description
- **`variables`**: Map of variable names to `ServerVariable`

`ServerVariable` contains:
- **`enum`**: List of allowed values
- **`default`**: Default value
- **`description`**: Description of the variable

### Creating Servers

Single static server:

```scala

val server = Server(
  url = "https://api.example.com",
  description = Some(md"Production API")
)
```

Server with variables:

```scala

val variableServer = Server(
  url = "https://{host}:{port}/{basePath}",
  description = Some(md"Development API with variables"),
  variables = ChunkMap(
    "host" -> ServerVariable(
      default = "localhost",
      `enum` = Chunk("localhost", "staging.example.com", "api.example.com"),
      description = Some(md"API host")
    ),
    "port" -> ServerVariable(
      default = "8080",
      `enum` = Chunk("8080", "443"),
      description = Some(md"Port number")
    ),
    "basePath" -> ServerVariable(
      default = "v1",
      `enum` = Chunk("v1", "v2"),
      description = Some(md"API version path")
    )
  )
)
```

:::note
`ServerVariable` contains enumerable values for each variable. The field is named to avoid the reserved `enum` keyword in Scala.
:::

---

## Tag

`Tag` groups related operations under a heading in generated documentation.

### Definition

Key fields:
- **`name`**: Tag identifier (e.g., `"users"`, `"products"`)
- **`description`**: Markdown description of the tag
- **`externalDocs`**: Link to external documentation

### Creating Tags

```scala

val userTag = Tag(
  name = "users",
  description = Some(md"User management operations")
)

val productsTag = Tag(
  name = "products",
  description = Some(md"Product catalog operations"),
  externalDocs = Some(ExternalDocumentation(
    url = "https://docs.example.com/products",
    description = Some(md"Full product API documentation")
  ))
)
```

---

## Common Extension Fields

Most types have an `extensions` field for custom `x-*` vendor-specific metadata; `Reference`, `XML`, `Discriminator` and `SecurityRequirement` do not. These extensions are preserved during encoding/decoding:

```scala

val operationWithExtensions = Operation(
  summary = Some(md"Get user"),
  responses = Responses(ChunkMap(
    "200" -> ReferenceOr.Value(Response(
      description = md"User found",
      content = ChunkMap()
    ))
  )),
  extensions = ChunkMap(
    "x-internal" -> Json.Boolean(true),
    "x-rate-limit" -> Json.Number(100),
    "x-deprecated-at" -> Json.String("2024-01-01")
  )
)
```

---

## Round-Tripping with Schema

All OpenAPI types have `Schema.derived` instances, enabling serialization through `DynamicValue`:

```scala

val myApi = OpenAPI(openapi = "3.1.0", info = Info(title = "My API", version = "1.0.0"))
val openAPISchema = Schema[OpenAPI]

val apiDynamic = openAPISchema.toDynamicValue(myApi)

val apiRestored = openAPISchema.fromDynamicValue(apiDynamic)
```

This enables integration with other ZIO Blocks modules that work with `DynamicValue`.

---

## Persistence

## Introduction

The blocks in this group store and evolve application data. They work from the data types the rest of the program already uses: the SQL module derives its codecs and repositories from a schema, and migrations are typed conversions between two such types. All four blocks require Scala 3.

## The Blocks

The group has four blocks.

| Block                                    | Artifact                      | What It Is For                                                                                    |
| ---------------------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------- |
| [SQL Module](../sql/index.md)            | `zio-blocks-sql`              | Type-safe JDBC wrapper with schema-derived codecs and a CRUD repository                           |
| [ZIO Integration](../sql-zio.md)         | `zio-blocks-sql-zio`          | ZIO integration with `ZIO.attemptBlocking` and `ZLayer`                                           |
| [Data Migration](../data-migration.md)   | `zio-blocks-data-migration`   | Typed, online database schema migrations in three execution models, with no hand-written SQL      |
| [Projection](../projection/index.md)     | `zio-blocks-projection`       | Event-sourced projections with per-entity SQLite storage                                          |

## How the Blocks Fit Together

The [SQL Module](../sql/index.md) is the foundation. Codecs map Scala types to database columns, fragments carry parameterized SQL, and repositories expose CRUD operations derived at compile time from a schema. [ZIO Integration](../sql-zio.md) is an adapter for that module: it lets a ZIO application use the same fragments, repositories and tables, and runs the JDBC calls off ZIO's main thread pool.

[Data Migration](../data-migration.md) builds on the SQL module's repositories and on [Migration](../schema/migration.md) for the typed row conversion, and it offers three execution models for evolving a schema online. [Projection](../projection/index.md) takes a different approach to storage: it turns an append-only event log into queryable read models that are described declaratively.

## Where to Start

Start with the [SQL Module](../sql/index.md). Add [ZIO Integration](../sql-zio.md) when the application uses ZIO, [Data Migration](../data-migration.md) when a schema has to change under load, and [Projection](../projection/index.md) when the source of truth is an event log.

---

## EventStore[E]

`EventStore[E]` is the write side of the module: an append-only log for one event type `E`. It assigns a monotonic sequence number to each event, streams history on demand, and broadcasts live events to a `Hub`:

```scala

trait EventStore[E] {
  def append(entityId: String, event: E): Task[Long]
  def readFrom(afterSeq: Long, tags: Set[String] = Set.empty): ZStream[Any, Throwable, EventEnvelope[E]]
  def readAll(tags: Set[String] = Set.empty): ZStream[Any, Throwable, EventEnvelope[E]]
  def subscribe: Hub[EventEnvelope[E]]
}
```

The examples on this page share the setup below: an event type with two cases, a helper that gets the `Transactor` of a database file, and an effect that opens a store with [`SQLiteEventStore.make`](#creating-a-store--make):

```scala

sealed trait UserEvent
object UserEvent {
  case class UserCreated(name: String, email: String) extends UserEvent
  case class EmailChanged(newEmail: String)            extends UserEvent
  implicit val schema: Schema[UserEvent] = Schema.derived[UserEvent]
}

// The Transactor of a database file, from a cache that is closed with the Scope
def transactor(file: String) = TransactorCache.make().flatMap(_.get(file))

// The store kept in events/users.db
val openStore = transactor("events/users.db").flatMap(tx => SQLiteEventStore.make[UserEvent](tx))
```

## Appending Events — `append`

```scala
def append(entityId: String, event: E): Task[Long]
```

`append` records `event` under the entity `entityId` and returns the sequence number the store gave it. Sequence numbers start at 1 and only grow, so they also give the order of the events. The event is saved with its tag, the case name or its `@eventTag` number, and the current time. Once it is stored, its envelope is published to the store's [hub](#following-new-events--subscribe):

```scala
val appended = openStore.flatMap { store =>
  for {
    a <- store.append("user-1", UserCreated("Alice", "alice@example.com"))
    b <- store.append("user-2", UserCreated("Bob", "bob@example.com"))
    c <- store.append("user-1", EmailChanged("alice@zio.dev"))
  } yield List(a, b, c) // List(1, 2, 3) on a new store
}
```

## Reading Events — `readFrom` and `readAll`

```scala
def readFrom(afterSeq: Long, tags: Set[String] = Set.empty): ZStream[Any, Throwable, EventEnvelope[E]]
def readAll(tags: Set[String] = Set.empty): ZStream[Any, Throwable, EventEnvelope[E]]
```

`readFrom` streams the stored events whose sequence number is strictly greater than `afterSeq`, in order, each in its [`EventEnvelope`](#eventenvelope). `readAll` is `readFrom(0L, tags)`, so it streams every event. Both read from the database as the stream is pulled, rather than loading the whole log first. The engine uses `readFrom` to catch up from its saved position, and a schema rebuild uses `readAll` to replay everything.

`tags` keeps only the events with one of the given tags, and an empty set, the default, keeps all of them. Which stored tags a read can return is described under [Which Stored Events Are Read](./migration.md#which-stored-events-are-read).

After the three appends above, these reads return:

```scala
val reads = openStore.flatMap { store =>
  for {
    all     <- store.readAll().runCollect                    // events 1, 2 and 3
    after1  <- store.readFrom(1L).runCollect                 // events 2 and 3
    changed <- store.readAll(Set("EmailChanged")).runCollect // event 3 only
  } yield (all, after1, changed)
}
```

## Following New Events — `subscribe`

```scala
def subscribe: Hub[EventEnvelope[E]]
```

`subscribe` returns the store's hub, the ZIO `Hub` that every appended envelope is published to. Despite its name it does not subscribe by itself: call `subscribe` on the hub, inside a `Scope`, to get a queue of the envelopes appended from then on. Events stored earlier are not in that queue; read them with `readFrom`. The engine does both: it catches up first, then follows the hub:

```scala
val next = openStore.flatMap { store =>
  for {
    queue <- store.subscribe.subscribe
    _     <- store.append("user-3", UserCreated("Charlie", "charlie@example.com"))
    env   <- queue.take // event 4, the UserCreated of user-3
  } yield env
}
```

## EventEnvelope

Every event in the store is wrapped in an `EventEnvelope`:

```scala

final case class EventEnvelope[+E](
  seq: Long,       // monotonic sequence number; used as the watermark
  tag: String,     // event type identifier (class name or @eventTag number)
  event: E,        // the original event value
  timestamp: java.time.Instant,
  entityId: String // the entity the event was recorded under
)
```

The engine saves `seq` as its watermark after each event it applies, and resumes after it on restart.

## SQLiteEventStore

`SQLiteEventStore[E]` is the implementation of `EventStore[E]` that keeps the events in a SQLite file. Each event is a row of the `events` table, whose columns are `seq`, `tag`, `payload`, `ts` and `entityId`: `seq` is the row's primary key and gives the sequence number, and `payload` is the event encoded as JSON. An index on `tag` and `seq` serves filtered reads.

Its constructor is private. Every factory below takes a `Transactor` for the database file, usually from a [`TransactorCache`](./projection-engine.md#connection-management), and creates the table and its index if they are missing.

### Creating a Store — `make`

```scala
object SQLiteEventStore {
  def make[E: Schema](transactor: Transactor): Task[SQLiteEventStore[E]]
  def make[E: Schema](transactor: Transactor, migration: Migration[E, E]): Task[SQLiteEventStore[E]]
}
```

`make` is the usual way to create a store. It creates a bounded hub of 4,096 envelopes for the store to publish to, and works out the store's tags from the event type with `TagResolver`. Pass a `migration` when a case of the event type was renamed; see [Renaming an Event Case](./migration.md#renaming-an-event-case). When the store starts, it logs a warning for the tags in the database that it cannot explain.

```scala
// UserCreated used to be called UserRegistered
val migration = Migration
  .newBuilder[UserEvent, UserEvent]
  .renameCase("UserRegistered", "UserCreated")
  .build
```

The store is then created with the migration:

```scala
val renamed = transactor("events/renamed.db").flatMap(tx => SQLiteEventStore.make[UserEvent](tx, migration))
```

### The Store's Tags — `tagInfo`

```scala
val tagInfo: TagInfo
```

`tagInfo` is the [`TagInfo`](./migration.md#taginfo-and-tagresolver) the store was created with: the tags the event type has now, the old tags that were renamed to them, and the migration for old events. `make` works it out with `TagResolver.resolve`, so the `tagInfo` of the two stores above can be computed without a store:

```scala
TagResolver.resolve[UserEvent].currentTags
// res4: Set[String] = Set("UserCreated", "EmailChanged")
TagResolver.resolve[UserEvent].aliases
// res5: Map[String, String] = Map()
TagResolver.resolve[UserEvent](migration).aliases
// res6: Map[String, String] = Map("UserRegistered" -> "UserCreated")
TagResolver.resolve[UserEvent](migration).allTags
// res7: Set[String] = Set("UserCreated", "EmailChanged", "UserRegistered")
```

### Bringing Your Own TagInfo — `makeWithResolver`

```scala
object SQLiteEventStore {
  def makeWithResolver[E: Schema](transactor: Transactor, tagInfo: TagInfo): Task[SQLiteEventStore[E]]
}
```

`makeWithResolver` takes a `TagInfo` you built yourself, for example with `TagResolver.resolve`, instead of working it out. `make` is a shortcut for it:

```scala
val info = TagResolver.resolve[UserEvent](migration)
```

The store then has `info` as its `tagInfo`:

```scala
val resolved = transactor("events/resolved.db").flatMap(tx => SQLiteEventStore.makeWithResolver[UserEvent](tx, info))
```

### Choosing the Hub — `makeWithHub`

```scala
object SQLiteEventStore {
  def makeWithHub[E: Schema](transactor: Transactor, hub: Hub[EventEnvelope[E]]): Task[SQLiteEventStore[E]]
  def makeWithHub[E: Schema](transactor: Transactor, hub: Hub[EventEnvelope[E]], migration: Migration[E, E]): Task[SQLiteEventStore[E]]
  def makeWithHub[E: Schema](transactor: Transactor, hub: Hub[EventEnvelope[E]], tagInfo: TagInfo): Task[SQLiteEventStore[E]]
}
```

`makeWithHub` uses the hub you give it instead of creating one, for example to choose its capacity or its strategy when it is full. The three overloads differ in how the tags are worked out, the same way as `make` with and without a migration, and `makeWithResolver`:

```scala
// The Transactor of a file, and a new sliding hub of 16 envelopes
def txAndHub(file: String) = transactor(file).zip(Hub.sliding[EventEnvelope[UserEvent]](16))

val withHub = txAndHub("events/hub.db").flatMap { case (tx, hub) =>
  SQLiteEventStore.makeWithHub[UserEvent](tx, hub)
}
val withHubAndMigration = txAndHub("events/hub-migration.db").flatMap { case (tx, hub) =>
  SQLiteEventStore.makeWithHub[UserEvent](tx, hub, migration)
}
val withHubAndInfo = txAndHub("events/hub-info.db").flatMap { case (tx, hub) =>
  SQLiteEventStore.makeWithHub[UserEvent](tx, hub, info)
}
```

Each store publishes to the hub it was given: its `subscribe` returns that hub.

### Blocking Construction — `makeBlocking` and `makeBlockingWithResolver`

```scala
object SQLiteEventStore {
  def makeBlocking[E: Schema](transactor: Transactor, hub: Hub[EventEnvelope[E]]): SQLiteEventStore[E]
  def makeBlocking[E: Schema](transactor: Transactor, hub: Hub[EventEnvelope[E]], migration: Migration[E, E]): SQLiteEventStore[E]
  def makeBlockingWithResolver[E: Schema](transactor: Transactor, hub: Hub[EventEnvelope[E]], tagInfo: TagInfo): SQLiteEventStore[E]
}
```

These return the store directly instead of a `Task`, for tests that need to build one outside an effect. They work on the calling thread, and unlike the factories above they do not check the tags in the database. As with `makeWithHub`, you pass the hub:

```scala
val blocking = for {
  hub <- Hub.bounded[EventEnvelope[UserEvent]](16)
  tx1 <- transactor("events/blocking.db")
  tx2 <- transactor("events/blocking-migration.db")
  tx3 <- transactor("events/blocking-info.db")
} yield (
  SQLiteEventStore.makeBlocking[UserEvent](tx1, hub),
  SQLiteEventStore.makeBlocking[UserEvent](tx2, hub, migration),
  SQLiteEventStore.makeBlockingWithResolver[UserEvent](tx3, hub, info)
)
```

## See Also

- [Projection module overview](./index.md) — how the event store, the projections, the engine and the projection stores fit together.
- [Projection[A]](./projection.md) — the projections that read the event store.
- [Migration](./migration.md) — renamed event cases, `@eventTag`, and changed event fields.
- [ProjectionEngine](./projection-engine.md) — the runtime that replays and follows an event store.

---

## Projection

The projection module turns an append-only event log into queryable read models with a purely declarative API. You describe — once, in a builder chain — which events a model listens to, how they route to entity stores, and what each kind of event does to the model. An engine then reads those descriptions and handles replay, watermark tracking, schema-change detection, and live streaming automatically. 

## Event Sourcing Basics

Most applications store the **current state** of things: a row in a table that is updated in place. When an order is shipped, the `status` column changes from `placed` to `shipped`, and the earlier value is gone.

**Event sourcing** stores something else: the **sequence of facts that happened**. Nothing is updated in place. Each fact is an **event**, and it is appended to a log:

```text
1. OrderPlaced   order-1, alice, keyboard, 80
2. OrderPlaced   order-2, alice, monitor, 250
3. OrderShipped  order-1
```

The current state is not stored. It is worked out by replaying the events in order: after event 3, `order-1` is shipped and `order-2` is still placed.

A few properties follow from this:

- **Events are facts about the past.** They are only ever appended, never edited or deleted, and their order is kept. The log is the source of truth.
- **State can be recomputed.** Replaying the same events always gives the same state, so a view of the data can be thrown away and rebuilt, or built in a different shape later.
- **History comes for free.** You can answer "how did we get here?", not only "where are we now?".
- **Events are permanent.** Once written they stay as they are, so renaming an event or changing its shape needs care. The [Migration](./migration.md#changing-events) page covers this.

### Reading the current state

Replaying the whole log to answer every question would be slow. Event sourcing therefore usually separates the two sides of an application, often called **CQRS** (command query responsibility segregation):

- the **write side** appends events to the log;
- the **read side** answers queries from **read models**, which are tables or documents shaped for the questions you ask, such as "all orders of a customer" or "how many orders are pending".

A **projection** is what keeps a read model up to date. It listens to the events as they are appended and applies each one to the read model:

```text
append events  -->  event log  -->  projection  -->  read model  -->  queries
  (write side)                                                         (read side)
```

Two consequences are worth knowing up front:

- **Reads can lag a little.** A read model is updated after an event is appended, not at the same instant, so a query right after a write may not see it yet. This is called eventual consistency.
- **A read model is disposable.** If its shape changes, or it is lost, it is rebuilt by replaying the log through the projection.

This library is the projection part: you declare how events turn into read models, and an engine runs the declaration and keeps SQLite files up to date.

## Motivation

A projection keeps a read model up to date as events arrive, so queries such as "what is Alice's current email?" or "how many orders were placed today?" never replay history. Written by hand, that code tends to go wrong in the same few ways:

- **Routing is scattered.** The knowledge that an `OrderPlaced` event must update the order, the customer's totals, and a daily counter lives in several handlers, so it is easy to miss one.
- **Restarts are fragile.** The code has to catch up on events it missed while it was down, then switch to live events, without skipping or double-applying any.
- **Event shapes change.** Once an event or a read model gains a field, old data no longer matches the code, and the read model has to be rebuilt by hand.
- **Tests need infrastructure.** Verifying a handler usually means standing up a database and a stream.

ZIO Blocks Projection addresses these by making a projection a declaration rather than a hand-rolled consumer. You state which event types a projection listens to, which entity each event belongs to, and how an event changes that entity. The engine then does the rest:

- it reads historical events to catch up, then follows live events. It saves its position after every event it applies, so a restart resumes where it left off;
- it routes each event to the entity (or entities) that should see it, following the routing declared once on the projection (`routeToSelf`, `routedBy`, `routeToAll`);
- it detects a changed schema and rebuilds the read model, with a shortcut for added fields and migrations to map old event tags to new ones;
- it stores each projection in its own SQLite file, so read models are isolated, individually backed up, and queryable offline. Unrelated projections never wait for each other's locks or write-ahead log, as they would with one shared database.

Because the same declaration runs against an in-memory store, you can test a projection by appending events and querying the result, with no database. The [Quick Start](#quick-start) shows the whole loop in one short program.

Projection is a good fit when you already treat events as the source of truth and want fast, local, query-friendly views of them. It is not a general query engine over arbitrary data, and the read models it maintains are derived state: they can always be discarded and rebuilt from the events.

## Glossary

The terms below are used throughout the pages of this module. The first list gives the general event sourcing terms, which apply to any event-sourced system. The second list names the types of this library. Most of them are explained in more detail in the sections that follow and on the type pages.

### Event-sourcing Terms

- **Event**: A fact that happened, such as `OrderPlaced`. Events are only appended to the log and are never changed.
- **Event log (event store)**: The append-only record of all events, in order. It is the source of truth.
- **Entity (aggregate)**: The thing events are about, such as a user, an order or a repository. Events are recorded under the ID of an entity.
- **Sequence number**: The position of an event in the log. A consumer saves the last one it processed, so it can resume after it.
- **Replay**: Applying the events of the log in order, to work out a state or to rebuild a view.
- **Read model**: A table or document shaped for queries, such as "all orders of a customer". It is derived from the events and can be rebuilt.
- **Projection**: The rules, and the process running them, that keep a read model up to date by applying each event as it arrives.
- **CQRS**: Command query responsibility segregation: the write side appends events, and the read side answers queries from read models.
- **Eventual consistency**: A read model is updated shortly after an event is appended, so a query made right after a write may not see it yet.
- **Event versioning (schema evolution)**: Changing the name or shape of an event or a read model after data exists. Old events stay as they are and are read by the new code, and a read model is rebuilt from them. See [Migration](./migration.md).

```text
event store  --(engine reads events and runs handlers)-->  projection store
(the log)                                                  (the finished rows)
```

### Types in This Library

The types below are the ones you meet when you use the library, grouped by the part of the picture above they belong to. Events are written to an event store. A projection says how those events become rows, and its handlers do the work for each event. Sources and routing decide which events reach which rows. The engine runs everything and keeps the rows in stores. Schemas keep the old events and rows working when their shape changes.

#### Events

- `EventStore`: The event log, as a trait. It appends events, reads the history, and lets a consumer follow new events.
- `SQLiteEventStore`: An `EventStore` kept in a SQLite file.
- `EventEnvelope`: An event together with the metadata the store records about it: its sequence number, its tag, its timestamp and the entity ID it was recorded under.
- `eventTag`: The annotation `@eventTag(n)`, which gives an event case a stable numeric tag. See [Numeric Tags](./migration.md#numeric-tags--eventtag).

#### Projections

- `Projection[A]`: The declaration of how events become the read model `A`: which sources to listen to, how to route their events, and what each event does. It does nothing until an engine runs it.
- `Handler[E, A]`: The function that turns an event `E` into a `ProjectionAction[A]`. It only describes the change.
- `ProjectionContext`: The metadata a handler receives together with an event, such as the entity ID.
- `ProjectionAction`: What a handler returns: an insert, an upsert, an update, a delete, a truncate or nothing.
- `FieldUpdate`: A change to one field of a row, used by an update action, such as setting a value or incrementing a counter. See [Aggregating Fields](./projection.md#aggregating-fields).
- `ProjectionScope`: How the data of a `Projection` is split across `ProjectionStore`s. It is worked out from how the projection is built and is not set directly. It is one of three: one row for each entity ID (the default), one store for each routing key, called a shard, or one shared store for the whole projection. See [Projection Scope](./projection.md#projection-scope).

#### Sources and routing

- `SourceBinding`: A named source of events, given to `Projection.from`, together with its routing and its handlers. The name must match the key of an `EventStore` given to the engine.
- `RoutingMode`: How the events of a source are directed: to the entity they were recorded under, to one shared store, or to a store chosen by a key that is read from the event.

#### Stores and the engine

- `ProjectionStore[A]`: Where the rows of a read model are kept.
- `SQLiteProjectionStore`: A `ProjectionStore` kept in a SQLite file.
- `InMemoryProjectionStore`: A `ProjectionStore` kept in an in-memory map, used for tests.
- `ProjectionEngine`: The runtime that reads events from `EventStore`s, runs the handlers and writes the results to `ProjectionStore`s.
- `TestEngine`: A `ProjectionEngine` with in-memory stores, for tests and examples.
- `TransactorCache`: A cache of `Transactor`s, one for each database file, shared by the stores and the engine. A `Transactor` opens a new connection for each operation, so no connection stays open. See [TransactorCache](./projection-engine.md#connection-management).
- **WAL (write-ahead log)**: A way for SQLite to make writes safe. A write is first appended to a separate `.db-wal` file, and it is copied into the main `.db` file later. Readers can keep reading while a write happens, and a crash does not lose the writes in the log. There is still only one writer at a time for each file. The `TransactorCache` turns it on for every file it opens, so you will see `-wal` and `-shm` files next to the `.db` files.

#### Schemas

- `EntityPath`: Says which folder the files of a projection go in and which field of the row is its ID. See [Entity Paths](./projection.md#entity-paths).
- `SchemaHash`: A fingerprint of a read model's schema. When it changes, the engine rebuilds the `ProjectionStore`.
- `SchemaEvolution`: Rebuilds a `ProjectionStore` by replaying the events. See [Full Rebuild](./migration.md#full-rebuild).
- `TagInfo`: A cheat sheet of the tags of an event type: which tags are current, and which old tags were renamed to them. It lets an `EventStore` find the events that were saved under an old name. See [Renaming an Event Case](./migration.md#renaming-an-event-case).
- `TagResolver`: Builds a `TagInfo` from an event type and, if a case was renamed, a `Migration`.

## Installation

The core module is sufficient to compile projections and run tests:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-projection" % "0.0.56"
```

For durable SQLite persistence on the JVM, add the JDBC driver:

```scala
libraryDependencies += "org.xerial" % "sqlite-jdbc" % "3.53.4.0"
```

The library does not bring `sqlite-jdbc` with it, so add it to your own build to keep events and read models in SQLite files. The SQLite stores open their files through a [`TransactorCache`](./projection-engine.md#connection-management), which loads the driver by name when it first opens a file. Without the driver, that first file fails to open with `ClassNotFoundException: org.sqlite.JDBC`. For a `SQLiteEventStore` that happens when you create it. For the projection stores that the engine creates, it happens when a store is first used, because the engine does not open their files when it is created.

## Overview

Two types carry the whole lifecycle:

- [`Projection[A]`](./projection.md) is the descriptor: a pure, immutable value that names the read model, lists its sources, and records a handler per event type. It holds no fibers.
- [`ProjectionEngine`](./projection-engine.md) is the runtime: it reads `Projection[A]` values, connects each one to an `EventStore[E]`, and drives events through the right handler into a `ProjectionStore[A]`. It does not inspect the handler functions. `EventStore[E]` is the write-side contract, the log that events are appended to, and [`ProjectionStore[A]`](./projection-store.md) is the read-side contract, the rows of a read model, as described under [Event Sourcing Basics](#event-sourcing-basics).

## How They Work Together

A projection goes through nine steps, from the declaration in your code to a row you can query:

1. **Declare the projection.** Call `Projection[A](name)` for a read model with one row per entity, or `Projection.global[A](name)` for one shared row of totals. The result is an empty, immutable description: it knows the shape of a row and where its files go, and nothing else yet.
2. **Say where the events come from.** Call `from(sourceName)` to name the event store the projection reads. A projection can read several, one `from` call each: everything that follows a `from` call, the routing and the handlers up to the next `from`, belongs to that source, which is how one read model can combine facts from more than one feed, such as a daily stats row that counts both new users and new repositories (see [`from`](./projection.md#defining-sources)).
3. **Say how the events are routed.** Right after `from`, choose one of three:
   - **Per entity:** `routeToSelf` sends each event to the row of the entity it was recorded under.
   - **By a key from the event:** `routedBy[E](_.field)` groups events by a value read from the event.
   - **One shared store:** `routeToAll` sends them all to one store.

   Which one fits, and where each keeps its files, is described under [Projection Scope](./projection.md#projection-scope).
4. **Add a handler for each event type.** Call `on[E]` for an event type, then one of the handler verbs, which says what that event does to the row:
   - **Insert and delete:** the event adds a new row or removes one. `insert(...)` builds the row from the event; `delete` removes the row of the entity the event belongs to.
   - **Field updates:** the event changes one column of an existing row, for example `status` to `"shipped"`. `update(_.field)(...)` picks the column by field, checked by the compiler; `updateWithField("column", ...)` and `updateField("column")(...)` pick it by name.
   - **Aggregation:** the event makes the same change every time, such as adding one to a counter: `aggregate(FieldUpdate.Increment("order_count", 1L))` on `OrderPlaced` adds one to `order_count` for every order. `aggregate(Chunk(...))` makes several changes at once, and `aggregateField(...)` is another name for the single form.
   - **Custom actions:** the handler looks at the event and decides what to do: insert, update, or nothing. `custom(...)` returns that decision as a `ProjectionAction`.
5. **Open the event stores.** Each source name given to `from` needs an `EventStore`, which holds the events of one event type. On the JVM, open a SQLite file through a `TransactorCache` and wrap it: `cache.get("events/users.db")` gives a connection to the file, and `SQLiteEventStore.make[E](transactor)` the store for the event type `E`.
6. **Create an engine.** Give the projections to a `ProjectionEngine`, together with the event stores they read. 
7. **Append events.** Call `eventStore.append(entityId, event)`. The store gives the event the next sequence number, wraps it in an `EventEnvelope` with that number, its tag, the time and the entity ID, and publishes it to the engine.
8. **The engine does the rest.** It first catches up on the events it has not seen yet, then follows new ones as they arrive. For each event it builds a `ProjectionContext` from the envelope, asks the projection for the handler's action with `handle`, applies that action to the store chosen by the routing, and saves the sequence number as its watermark, so that a restart continues from there.
9. **Query.** Call `engine.query(projection, entityId)` to read the row back from the store.

The diagram shows how the types hand the data along:

```
┌─────────────────────┐              ┌──────────────────────┐
│ Application         │─ append() ──►│ EventStore[E]        │
└─────────────────────┘              └──────────┬───────────┘
                                                │ readFrom / subscribe
                                                │ EventEnvelope[E]
                                                ▼
┌─────────────────────┐        ┌────────────────────────────┐
│ Projection[A]       │        │    ProjectionEngine        │
│ (descriptor)        │◄───────│                            │
│                     │handle()│                            │
│                     │───────►│  ProjectionAction[A]       │
└─────────────────────┘        └─────────────┬──────────────┘
                                             │ insert / upsert /
                                             │ updateFields / delete
                                             ▼
                               ┌─────────────────────────────┐
                               │    ProjectionStore[A]       │
                               │    (SQLite or InMemory)     │
                               └─────────────────────────────┘
                                             ▲
                               engine.query(projection, id)
```

## Quick Start

A projection is defined once, then run by an engine. The example below defines the events and the read model, declares the projection, and runs it on SQLite files, the way an application does.

Open the event store and the projection store yourself, through a `TransactorCache`, and give them to `ProjectionEngine.makeWithStores`. The stores do not create folders, so the program creates `events/` and `users/` first. It needs `sqlite-jdbc` on the runtime classpath, see [Installation](#installation). The program is `Example1QuickStart` in the projection examples:

```scala title="projection-examples/src/main/scala/projection/examples/Example1QuickStart.scala"
/*
 * Example 1 of 13: the quick start.
 *
 * Start here. Next: Example2PerEntity.
 *
 * The program of the Quick Start in the documentation.
 *
 * It defines the events and the read model, declares a projection, and runs it
 * on SQLite files, the way an application does. Two kinds of event are recorded
 * for each user: `UserCreated` inserts the user's row, and `EmailChanged`
 * updates one column of it, so the read model follows the events as they come.
 *
 * The events go to `events/users.db` and the read model to
 * `users/user_profiles.db`, both relative to the working directory. The stores do
 * not create folders, so the program creates the two folders first. The files
 * stay on disk after the program ends, so running it again continues after the
 * last event it processed. The program appends its events only when the event
 * store is empty, so a second run appends nothing and reads the same rows back
 * from the files.
 *
 * Run it from the repository root with:
 *   sbt "projection-examples/runMain projection.examples.Example1QuickStart"
 */

package projection.examples

object Example1QuickStart extends ZIOAppDefault {

  // The events. They share a sealed trait, so that one event store holds both kinds.
  sealed trait UserEvent
  object UserEvent {
    case class UserCreated(name: String, email: String) extends UserEvent
    object UserCreated {
      implicit val schema: Schema[UserCreated] = Schema.derived[UserCreated]
    }
    case class EmailChanged(newEmail: String) extends UserEvent
    object EmailChanged {
      implicit val schema: Schema[EmailChanged] = Schema.derived[EmailChanged]
    }
    implicit val schema: Schema[UserEvent] = Schema.derived[UserEvent]
  }

  // The read model. The ID field is `userId`, so the derived folder is `users`, the same folder the store is opened in below.
  case class UserProfile(@Modifier.id userId: String, name: String, email: String)
  object UserProfile {
    implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
    implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
  }

  // The projection: what each event does to the row of the user it was recorded under
  val projection: Projection[UserProfile] = Projection[UserProfile]("user_profiles")
    .from("users")
    .routeToSelf
    .on[UserCreated]
    .insert((e, ctx) => UserProfile(ctx.entityId, e.name, e.email))
    .on[EmailChanged]
    .updateWithField("email", (e, _) => e.newEmail)

  // Wire the projection to SQLite files
  val program: ZIO[Scope, Throwable, List[(String, Option[UserProfile])]] = for {
    _        <- ZIO.attemptBlocking(List("events", "users").foreach(d => Files.createDirectories(Paths.get(d))))
    cache    <- TransactorCache.make()
    tx       <- cache.get("events/users.db")
    users    <- SQLiteEventStore.make[UserEvent](tx)
    profiles <- SQLiteProjectionStore.make[UserProfile]("users/user_profiles.db", cache)
    engine   <- ProjectionEngine.makeWithStores(
                specs = List(projection),
                stores = Map("user_profiles" -> profiles), // projection name -> its store
                eventStores = Map("users" -> users), // source name given to `from` -> its event store
                cache = cache
              )
    _ <- engine.start
    // The events are appended on the first run only; a later run finds them in events/users.db
    stored <- users.readAll().runCount
    _      <- ZIO.when(stored == 0) {
           users.append("user-1", UserCreated("Alice", "alice@example.com")) *>
             users.append("user-2", UserCreated("Bob", "bob@example.com")) *>
             users.append("user-3", UserCreated("Charlie", "charlie@example.com")) *>
             users.append("user-1", EmailChanged("alice@zio.dev"))
         }
    // The engine applies the events in the background, in order, so once the last one shows all of them have been applied
    _    <- engine.query(projection, "user-1").repeatUntil(_.exists(_.email == "alice@zio.dev"))
    rows <- ZIO.foreach(List("user-1", "user-2", "user-3"))(id => engine.query(projection, id).map(id -> _))
  } yield rows

  def run: ZIO[Any, Throwable, Unit] =
    ZIO.scoped(program).flatMap { rows =>
      ZIO.foreachDiscard(rows) { case (id, row) =>
        Console.printLine(s"$id: ${row.map(u => s"${u.name} <${u.email}>").getOrElse("not found")}")
      }
    }
}
```

The files stay on disk after the program ends. When a program starts again, the events are still in `events/users.db`, and the engine continues after the last event it applied. That is why the program appends its events only when the event store is empty: run it a second time, and it reads the same rows back from the files. The files are plain SQLite databases, so `sqlite3 users/user_profiles.db` shows the rows while the program is stopped.

## See Also

The [SQL Module](../sql/index.md) supplies the `Transactor`, `DbCon`, and `SqlNameMapper` types that back the SQLite implementations of `EventStore[E]` and `ProjectionStore[A]`. The [Data Migration](../data-migration.md) reference covers typed online schema migrations that pair naturally with projection-backed read models during schema evolution.

---

## Migrating Read Models and Events

Code changes after data exists. In an event-sourced application, two kinds of stored data can fall out of step with the code:

- the rows of a read model, the `A` of a `Projection[A]`, kept in a projection store;
- the events, kept in an event store.

They are handled in opposite ways. The rows are derived from the events, so they can be thrown away and rebuilt by replaying the events. The events are the source of truth and are never rewritten, so new code has to read old events as they were stored.

Find your change:

- You added, removed, renamed or changed a field of the row type: [Full Rebuild](#full-rebuild), or [Adding Fields Without a Rebuild](#adding-fields-without-a-rebuild--registermigration) when you only add fields.
- You changed what a handler does: [Rebuilding on Demand](#rebuilding-on-demand).
- You renamed the row class or the projection: [Renaming the Row Class or the Projection](#renaming-the-row-class-or-the-projection).
- You renamed a case of the event type: [Renaming an Event Case](#renaming-an-event-case).
- You added, removed or changed a field of an event: [Changing the Fields of an Event](#changing-the-fields-of-an-event).

This page is about the data a projection keeps. For the `Migration` type itself, see [Migration](../schema/migration.md); for migrating a relational database while it runs, see [Data Migration](../data-migration.md).

## Changing the Read Model

The rows of a projection are kept in a table named after the row class, in snake_case (`UserProfile` becomes `user_profile`), with one column for each field. On every `start`, the engine checks whether that table still matches the row type, so a change is handled when the new version of the application starts.

### How a Change Is Detected — `SchemaHash`

```scala
object SchemaHash {
  def compute[A: Schema]: String
}
```

`compute` returns a fingerprint of the shape of `A`: a SHA-256 hash, as 64 hex characters. The store keeps the fingerprint of the rows it holds next to the watermark, the sequence number of the last event applied, in a table called `_projection_meta`. On `start`, the engine compares the stored fingerprint with the current one:

- No fingerprint is stored, as in a new store: it stores the current one.
- The fingerprints are the same: nothing happens.
- They differ: the rows are out of date. The engine takes the [shortcut](#adding-fields-without-a-rebuild--registermigration) if one is registered and fits, and otherwise [rebuilds](#full-rebuild) the rows. With [`lazyRebuild`](#rebuilding-later--lazyrebuild) on, it only marks the projection for a later rebuild.

The fingerprint changes when you:

- rename the row class;
- add, remove, rename or reorder a field;
- change the type of a field, for example from `Int` to `Long`;
- change the fields of a case class used directly as the type of a field.

It does not change when you:

- change the fields of a case class held in an `Option`, a `List` or another collection;
- add a case to a sealed trait used as the type of a field;
- move `@Modifier.id` to another field;
- change a handler.

For those, see [Rebuilding on Demand](#rebuilding-on-demand).

### Full Rebuild

A full rebuild throws the rows away and computes them again from the stored events:

1. The table is dropped and created again, with the columns of the current row type.
2. The watermark is set back to 0.
3. Every event of the projection's event store is read, from the first one, and applied through the handlers.
4. The watermark and the new fingerprint are saved.
5. The engine then goes on as usual: it catches up on newer events and follows new ones.

`start` waits for the rebuild, so with a long history the application takes longer to start. In return, old rows get real values for new fields, computed from the events. An event whose change cannot be applied to the table is logged and skipped. An event that can no longer be decoded stops the rebuild; see [Changing the Fields of an Event](#changing-the-fields-of-an-event).

The example below runs two versions of an application on the same files. The first keeps `UserProfile(id, name)`; the second adds `email`, so its `start` rebuilds the table, and the old users get their email from the stored events:

```scala title="projection-examples/src/main/scala/projection/examples/Example8SchemaEvolution.scala"
/*
 * Example 8 of 13: schema evolution.
 *
 * Builds on Example2PerEntity. Next: Example9MigrationShortcut.
 *
 * When the read model's class changes, such as `UserProfile` gaining an `email`
 * field, the data already stored no longer matches. The engine notices this when
 * it starts, by comparing a hash of the schema with the one saved in the store,
 * and rebuilds the store by replaying the events.
 *
 * This example runs on real SQLite files and uses two engines in a row, like an
 * application that is stopped and started again with a newer version. The events
 * stay in `.projection-examples/example8/events/users.db`, and the read model in
 * `.projection-examples/example8/profiles/userProfiles.db`.
 *   1. The first engine runs the projection with `V1.UserProfile` (id, name).
 *   2. The second engine runs it with `V2.UserProfile` (id, name, email). Both
 *      classes are called `UserProfile`, so they use the same table, and the
 *      projection keeps its name.
 *
 * It shows:
 *   - `SchemaHash.compute`, and that two versions of a type hash differently
 *   - the second engine detecting the changed schema in `start` and rebuilding
 *     the table from the stored events, so the old rows gain their `email`
 *   - that the rebuilt projection keeps processing new events
 *
 * Run it from the repository root with:
 *   sbt "projection-examples/runMain projection.examples.Example8SchemaEvolution"
 */

package projection.examples

object Example8SchemaEvolution extends ZIOAppDefault {

  private val profilesFolder: String = Sqlite.path("example8/profiles")

  object V1 {
    case class UserProfile(@Modifier.id id: String, name: String)
    object UserProfile {
      implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
      implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile](profilesFolder)
    }
  }

  object V2 {
    case class UserProfile(@Modifier.id id: String, name: String, email: String)
    object UserProfile {
      implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
      implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile](profilesFolder)
    }
  }

  private def show(profile: Option[V2.UserProfile]): String =
    profile.map(u => s"${u.name} <${u.email}>").getOrElse("not found")

  def demo: ZIO[Any, Throwable, Unit] =
    Console.printLine("\n--- Example 8: Schema Evolution (UserProfile V1 -> V2) ---").orDie *>
      ZIO.scoped {
        for {
          _ <- Sqlite.folders(profilesFolder, Sqlite.path("example8/events"))

          hashV1 = SchemaHash.compute[V1.UserProfile]
          hashV2 = SchemaHash.compute[V2.UserProfile]
          _     <- Console.printLine(s"  SchemaHash V1: ${hashV1.take(16)}...").orDie
          _     <- Console.printLine(s"  SchemaHash V2: ${hashV2.take(16)}...").orDie
          _     <- Console.printLine(s"  Hashes differ: ${hashV1 != hashV2}").orDie

          projectionV1 = Projection[V1.UserProfile]("userProfiles")
                           .from("users")
                           .routeToSelf
                           .on[UserCreated]
                           .insert((e, ctx) => V1.UserProfile(ctx.entityId, e.name))

          projectionV2 = Projection[V2.UserProfile]("userProfiles")
                           .from("users")
                           .routeToSelf
                           .on[UserCreated]
                           .insert((e, ctx) => V2.UserProfile(ctx.entityId, e.name, e.email))

          // 1. The first version of the application.
          _ <- ZIO.scoped {
                 for {
                   cache  <- TransactorCache.make()
                   users  <- Sqlite.eventStore[UserCreated](cache, Sqlite.path("example8/events/users.db"))
                   engine <- ProjectionEngine.makeWithStoresVarargs(
                               cache,
                               Map[String, EventStore[?]]("users" -> users)
                             )(projectionV1)
                   _ <- engine.start

                   _ <- users.append("user-1", UserCreated("Alice", "alice@example.com"))
                   _ <- users.append("user-2", UserCreated("Bob", "bob@example.com"))

                   u2 <- pollUntil(engine.query(projectionV1, "user-2"))
                   u1 <- engine.query(projectionV1, "user-1")
                   _  <- Console.printLine(s"  V1 user-1: ${u1.map(_.name).getOrElse("not found")}").orDie
                   _  <- Console.printLine(s"  V1 user-2: ${u2.map(_.name).getOrElse("not found")}").orDie
                 } yield ()
               }

          _ <- Console.printLine("  --- restart with V2 ---").orDie

          // 2. The second version of the application, on the same files.
          _ <- ZIO.scoped {
                 for {
                   cache  <- TransactorCache.make()
                   users  <- Sqlite.eventStore[UserCreated](cache, Sqlite.path("example8/events/users.db"))
                   engine <- ProjectionEngine.makeWithStoresVarargs(
                               cache,
                               Map[String, EventStore[?]]("users" -> users)
                             )(projectionV2)
                   // `start` finds the changed schema hash and rebuilds the table from the stored events.
                   _ <- engine.start

                   u1 <- engine.query(projectionV2, "user-1")
                   u2 <- engine.query(projectionV2, "user-2")
                   _  <- Console.printLine(s"  After rebuild, user-1: ${show(u1)}").orDie
                   _  <- Console.printLine(s"  After rebuild, user-2: ${show(u2)}").orDie

                   _  <- users.append("user-3", UserCreated("Carol", "carol@example.com"))
                   u3 <- pollUntil(engine.query(projectionV2, "user-3"))
                   _  <- Console.printLine(s"  New event, user-3:    ${show(u3)}").orDie
                 } yield ()
               }

          _ <- Console.printLine("  ✓ Schema Evolution: hash check + rebuild from stored events keeps the data").orDie
        } yield ()
      }

  def run: ZIO[Any, Throwable, Unit] = demo
}
```

It prints `After rebuild, user-1: Alice <alice@example.com>`.

### Adding Fields Without a Rebuild — `registerMigration`

When the only change is new fields, the engine can add the columns instead of replaying the events. Describe the change with a `Migration` from the old row type to the new one, and register it on the engine:

```scala
final class ProjectionEngine {
  def registerMigration[Old, New](spec: Projection[New], migration: Migration[Old, New]): Task[Unit]
}
```

Build the migration with `Migration.newBuilder[Old, New]`, one `addField` for each new field, and `build`; see [Migration](../schema/migration.md). The rules:

- Register it before `start`, which is when the fingerprints are compared.
- The engine keeps it under the projection's name, so registering another one replaces it.
- It is used only when every step of the migration adds a top-level field. Any other step, such as a rename, means a full rebuild.
- Each new field becomes a column added with `ALTER TABLE ... ADD COLUMN`. The fingerprint is saved, the watermark is kept, and no event is replayed.

The old rows get no value in the new columns: the default given to `addField` is not written. They read back as `null` for a `String`, `None` for an `Option` and `0` for an `Int`, whatever the default was. So use the shortcut for fields that old rows can do without, such as an `Option`, and let the engine rebuild otherwise.

:::caution
The engine does not check that a registered migration describes the change it finds. While a migration made only of `addField` steps is registered, the engine uses it for any change of the fingerprint: it adds the migration's columns, saves the new fingerprint and does not rebuild, even when other fields changed too. The table is then left with missing columns. Register a migration only in the version of the application that makes that change, and remove it afterwards.
:::

The example below runs the two versions of the full rebuild example again, with a migration registered for the second version. The old users come back with an empty email:

```scala title="projection-examples/src/main/scala/projection/examples/Example9MigrationShortcut.scala"
/*
 * Example 9 of 13: the migration shortcut.
 *
 * Read Example8SchemaEvolution first. It is the same story, with one change: the
 * second engine is told how the read model changed. Next: Example10RebuildOnDemand.
 *
 * When the read model gains a field, such as `UserProfile` gaining an `email`,
 * the engine normally rebuilds the whole table by replaying every event. If the
 * change is only added fields, a `Migration` registered with
 * `engine.registerMigration` lets the engine add the columns with
 * `ALTER TABLE ADD COLUMN` instead, and no event is replayed.
 *
 * This example runs on real SQLite files and uses two engines in a row. The
 * events stay in `.projection-examples/example9/events/users.db`, and the read
 * model in `.projection-examples/example9/profiles/userProfiles.db`.
 *   1. The first engine runs the projection with `V1.UserProfile` (id, name).
 *   2. The second engine runs it with `V2.UserProfile` (id, name, email), after
 *      registering a migration that adds `email` with the default `""`.
 *
 * It shows:
 *   - registering the migration before `start`, so the engine can use it
 *   - the old rows being kept without a rebuild. Their new `email` column is
 *     empty (`null`), not `""` and not the real address: the shortcut only adds
 *     the column and does not replay the events. A rebuild, as in
 *     Example8SchemaEvolution, would fill it from the stored events.
 *   - that the migrated projection keeps processing new events
 *
 * Run it from the repository root with:
 *   sbt "projection-examples/runMain projection.examples.Example9MigrationShortcut"
 */

package projection.examples

object Example9MigrationShortcut extends ZIOAppDefault {

  private val profilesFolder: String = Sqlite.path("example9/profiles")

  object V1 {
    case class UserProfile(@Modifier.id id: String, name: String)
    object UserProfile {
      implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
      implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile](profilesFolder)
    }
  }

  object V2 {
    case class UserProfile(@Modifier.id id: String, name: String, email: String)
    object UserProfile {
      implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
      implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile](profilesFolder)
    }
  }

  private def show(profile: Option[V2.UserProfile]): String =
    profile.map(u => s"${u.name} <${u.email}>").getOrElse("not found")

  def demo: ZIO[Any, Throwable, Unit] =
    Console.printLine("\n--- Example 9: Migration Shortcut (UserProfile V1 -> V2) ---").orDie *>
      ZIO.scoped {
        for {
          _ <- Sqlite.folders(profilesFolder, Sqlite.path("example9/events"))

          hashV1 = SchemaHash.compute[V1.UserProfile]
          hashV2 = SchemaHash.compute[V2.UserProfile]
          _     <- Console.printLine(s"  SchemaHash V1: ${hashV1.take(16)}...").orDie
          _     <- Console.printLine(s"  SchemaHash V2: ${hashV2.take(16)}...").orDie
          _     <- Console.printLine(s"  Hashes differ: ${hashV1 != hashV2}").orDie

          projectionV1 = Projection[V1.UserProfile]("userProfiles")
                           .from("users")
                           .routeToSelf
                           .on[UserCreated]
                           .insert((e, ctx) => V1.UserProfile(ctx.entityId, e.name))

          projectionV2 = Projection[V2.UserProfile]("userProfiles")
                           .from("users")
                           .routeToSelf
                           .on[UserCreated]
                           .insert((e, ctx) => V2.UserProfile(ctx.entityId, e.name, e.email))

          // Describes the change from V1 to V2: the new field `email` starts as "".
          migration = Migration
                        .newBuilder[V1.UserProfile, V2.UserProfile]
                        .addField(_.email, SchemaExpr.literal(""))
                        .build

          // 1. The first version of the application.
          _ <- ZIO.scoped {
                 for {
                   cache  <- TransactorCache.make()
                   users  <- Sqlite.eventStore[UserCreated](cache, Sqlite.path("example9/events/users.db"))
                   engine <- ProjectionEngine.makeWithStoresVarargs(
                               cache,
                               Map[String, EventStore[?]]("users" -> users)
                             )(projectionV1)
                   _ <- engine.start

                   _ <- users.append("user-1", UserCreated("Alice", "alice@example.com"))
                   _ <- users.append("user-2", UserCreated("Bob", "bob@example.com"))

                   u2 <- pollUntil(engine.query(projectionV1, "user-2"))
                   u1 <- engine.query(projectionV1, "user-1")
                   _  <- Console.printLine(s"  V1 user-1: ${u1.map(_.name).getOrElse("not found")}").orDie
                   _  <- Console.printLine(s"  V1 user-2: ${u2.map(_.name).getOrElse("not found")}").orDie
                 } yield ()
               }

          _ <- Console.printLine("  --- restart with V2 ---").orDie

          // 2. The second version of the application, on the same files.
          _ <- ZIO.scoped {
                 for {
                   cache  <- TransactorCache.make()
                   users  <- Sqlite.eventStore[UserCreated](cache, Sqlite.path("example9/events/users.db"))
                   engine <- ProjectionEngine.makeWithStoresVarargs(
                               cache,
                               Map[String, EventStore[?]]("users" -> users)
                             )(projectionV2)
                   // The migration is registered before `start`, which is where the changed schema hash is found.
                   // Because it only adds a field, the engine adds the column and does not replay the events.
                   _ <- engine.registerMigration(projectionV2, migration)
                   _ <- engine.start

                   u1 <- engine.query(projectionV2, "user-1")
                   u2 <- engine.query(projectionV2, "user-2")
                   _  <- Console.printLine(s"  After migration, user-1: ${show(u1)}").orDie
                   _  <- Console.printLine(s"  After migration, user-2: ${show(u2)}").orDie

                   _  <- users.append("user-3", UserCreated("Carol", "carol@example.com"))
                   u3 <- pollUntil(engine.query(projectionV2, "user-3"))
                   _  <- Console.printLine(s"  New event, user-3:    ${show(u3)}").orDie
                 } yield ()
               }

          _ <- Console
                 .printLine("  ✓ Migration shortcut: the column was added, old rows kept, their new field is empty")
                 .orDie
        } yield ()
      }

  def run: ZIO[Any, Throwable, Unit] = demo
}
```

It prints `After migration, user-1: Alice <null>`.

### Rebuilding Later — `lazyRebuild`

```scala
final case class ProjectionEngineConfig(
  // ...
  rebuildParallelism: Int = 4,
  lazyRebuild: Boolean = false,
  evolution: SchemaEvolutionConfig = SchemaEvolutionConfig.default
)
```

By default, `start` rebuilds a changed projection before it returns. With `lazyRebuild = true`, it only marks the projection as pending, so the application starts at once:

- The first `query` of the projection waits while it is rebuilt, or while the shortcut runs.
- A background fiber rebuilds the pending projections one after the other, in groups of `rebuildParallelism`.
- `queryByName` does not trigger the rebuild.

Until then, the engine keeps applying events to the old table, and the events that no longer fit it fail and are logged. The rebuild applies them again from the stored events. The `evolution` settings are not read by the engine; see [`SchemaEvolution`](#schemaevolution). The other settings are described under [Configuration](./projection-engine.md#configuration).

### Rebuilding on Demand

Some changes need a rebuild that the engine cannot detect: a handler that now computes something else, or a change the fingerprint does not see. There are two ways to rebuild:

- Delete the projection's store file while the application is stopped: `<folder>/<name>.db`, with its `-wal` and `-shm` files. For a cross-entity projection, delete its `<name>_shard_*.db` files too, and for a global projection the file is `global/<name>.db`. When the application starts, the empty store has no fingerprint and a watermark of 0, so the engine applies every stored event again as it catches up.
- In code, call `SchemaEvolution.rebuildWithHash` on the store before the engine starts. It empties the store and applies every stored event again, with the current handlers.

The example below runs three versions of an application on the same files. The second changes the handler to store the names in upper case, and its old rows keep their names, because nothing triggers a rebuild. The third rebuilds on demand, before the engine starts:

```scala title="projection-examples/src/main/scala/projection/examples/Example10RebuildOnDemand.scala"
/*
 * Example 10 of 13: rebuilding on demand.
 *
 * Builds on Example8SchemaEvolution. Read it first. Next: Example11TagRename.
 *
 * The engine rebuilds a projection by itself only when the shape of its row type
 * changes. A handler that now computes something else does not change that
 * shape, so the rows already stored keep the old results. To recompute them,
 * rebuild the projection yourself: `SchemaEvolution.rebuildWithHash` empties the
 * projection's store and applies every stored event again through the current
 * handlers.
 *
 * This example runs on real SQLite files: the events are kept in
 * `.projection-examples/example10/events/users.db`, and the read model in
 * `.projection-examples/example10/profiles/user_profiles.db`. It runs three
 * versions of an application in a row, on the same files:
 *   1. The first version stores the users' names as they are.
 *   2. The second changes the handler to store the names in upper case. The old
 *      rows keep their names, because nothing triggers a rebuild.
 *   3. The third has the same handler as the second, and calls
 *      `rebuildWithHash` before the engine starts. Now the old rows are in upper
 *      case too.
 *
 * It shows:
 *   - that changing a handler does not rebuild the stored rows
 *   - rebuilding on demand with `SchemaEvolution.rebuildWithHash`, before
 *     `start`, with the stores created for `makeWithStores`
 *
 * Run it from the repository root with:
 *   sbt "projection-examples/runMain projection.examples.Example10RebuildOnDemand"
 */

package projection.examples

object Example10RebuildOnDemand extends ZIOAppDefault {

  private val profilesFolder: String = Sqlite.path("example10/profiles")
  private val profilesFile: String   = s"$profilesFolder/user_profiles.db"
  private val eventsFile: String     = Sqlite.path("example10/events/users.db")

  case class UserProfile(@Modifier.id userId: String, name: String)
  object UserProfile {
    implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
    implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile](profilesFolder)
  }

  // The first version stores the name as it is.
  val asIs: Projection[UserProfile] = Projection[UserProfile]("user_profiles")
    .from("users")
    .routeToSelf
    .on[UserCreated]
    .insert((e, ctx) => UserProfile(ctx.entityId, e.name))

  // A later version stores the name in upper case. The row type is the same, so its fingerprint is the same too.
  val upperCase: Projection[UserProfile] = Projection[UserProfile]("user_profiles")
    .from("users")
    .routeToSelf
    .on[UserCreated]
    .insert((e, ctx) => UserProfile(ctx.entityId, e.name.toUpperCase))

  private def name(profile: Option[UserProfile]): String = profile.map(_.name).getOrElse("not found")

  def demo: ZIO[Any, Throwable, Unit] =
    Console.printLine("\n--- Example 10: Rebuilding on Demand (a changed handler) ---").orDie *>
      ZIO.scoped {
        for {
          _ <- Sqlite.folders(profilesFolder, Sqlite.path("example10/events"))

          // 1. The first version of the application.
          _ <- ZIO.scoped {
                 for {
                   cache  <- TransactorCache.make()
                   users  <- Sqlite.eventStore[UserCreated](cache, eventsFile)
                   engine <-
                     ProjectionEngine.makeWithStoresVarargs(cache, Map[String, EventStore[?]]("users" -> users))(asIs)
                   _  <- engine.start
                   _  <- users.append("user-1", UserCreated("Alice", "alice@example.com"))
                   _  <- users.append("user-2", UserCreated("Bob", "bob@example.com"))
                   u2 <- pollUntil(engine.query(asIs, "user-2"))
                   u1 <- engine.query(asIs, "user-1")
                   _  <- Console.printLine(s"  V1 user-1: ${name(u1)}").orDie
                   _  <- Console.printLine(s"  V1 user-2: ${name(u2)}").orDie
                 } yield ()
               }

          _ <- Console.printLine("  --- restart with the upper-case handler ---").orDie

          // 2. The handler changed but the row type did not, so `start` finds the same fingerprint and does not rebuild.
          _ <- ZIO.scoped {
                 for {
                   cache  <- TransactorCache.make()
                   users  <- Sqlite.eventStore[UserCreated](cache, eventsFile)
                   engine <- ProjectionEngine.makeWithStoresVarargs(
                               cache,
                               Map[String, EventStore[?]]("users" -> users)
                             )(upperCase)
                   _  <- engine.start
                   u1 <- engine.query(upperCase, "user-1")
                   _  <- Console.printLine(s"  V2 user-1: ${name(u1)} (not rebuilt)").orDie
                   _  <- users.append("user-3", UserCreated("Carol", "carol@example.com"))
                   u3 <- pollUntil(engine.query(upperCase, "user-3"))
                   _  <- Console.printLine(s"  V2 new event, user-3: ${name(u3)}").orDie
                 } yield ()
               }

          _ <- Console.printLine("  --- restart and rebuild on demand ---").orDie

          // 3. The same handler, with the rows rebuilt before the engine starts.
          _ <- ZIO.scoped {
                 for {
                   cache    <- TransactorCache.make()
                   tx       <- cache.get(eventsFile)
                   users    <- SQLiteEventStore.make[UserCreated](tx)
                   profiles <- SQLiteProjectionStore.make[UserProfile](profilesFile, cache)
                   // Empties the store and applies every stored event again, with the current handler.
                   _      <- SchemaEvolution.rebuildWithHash(profiles, users, upperCase)
                   engine <- ProjectionEngine.makeWithStores(
                               specs = List(upperCase),
                               stores = Map("user_profiles" -> profiles),
                               eventStores = Map("users" -> users),
                               cache = cache
                             )
                   _    <- engine.start
                   rows <-
                     ZIO.foreach(List("user-1", "user-2", "user-3"))(id => engine.query(upperCase, id).map(id -> _))
                   _ <- ZIO.foreachDiscard(rows) { case (id, row) =>
                          Console.printLine(s"  After rebuildWithHash, $id: ${name(row)}").orDie
                        }
                 } yield ()
               }

          _ <-
            Console
              .printLine("  ✓ Rebuild on demand: rebuildWithHash applied every stored event again with the new handler")
              .orDie
        } yield ()
      }

  def run: ZIO[Any, Throwable, Unit] = demo
}
```

It prints `V2 user-1: Alice (not rebuilt)`, then `After rebuildWithHash, user-1: ALICE`.

Neither way works for a projection that reads several sources; see [Limitations](#limitations).

### Renaming the Row Class or the Projection

The name of the row class is also the name of its table. When you rename the class, the engine finds no table and no fingerprint under the new name: it creates an empty table and fills it by applying every stored event, as for a new projection. The old table stays in the file, unused.

To keep the table, give the renamed class its old table name with `@Modifier.config("sql.table_name", ...)`. The class name is part of the fingerprint, so the engine then rebuilds that table as for any other change:

```scala

// Renamed from UserProfile; the annotation keeps the table `user_profile`
@Modifier.config("sql.table_name", "user_profile")
case class Profile(@Modifier.id userId: String, name: String)
object Profile {
  implicit val schema: Schema[Profile]         = Schema.derived[Profile]
  implicit val entityPath: EntityPath[Profile] = EntityPath.derived[Profile]
}
```

Renaming the projection, or moving it to another folder with its `EntityPath`, gives it a new file: the engine builds it from scratch from the stored events, and the old file stays. A migration registered with `registerMigration` is looked up under the new name.

## Changing Events

Events are never rewritten. Each stored event is a row of the event store's `events` table, with a sequence number, a tag and a payload. The tag says which case of the event type the event is, and the payload is the event as JSON, keyed by the case name:

```text
seq | tag          | payload
1   | UserLoggedIn | {"UserLoggedIn":{"userId":"user-1"}}
```

When events are read back, to catch up or to rebuild, the store keeps only the rows whose tag it knows, and decodes each payload with the current event type.

### Which Stored Events Are Read

The store knows the tags the event type has now, and the old names of renamed cases that you declared with a migration. Every read keeps only the events with those tags, including catching up and rebuilding. An event under any other tag, for example of a case that was removed, or renamed without a migration, is skipped without an error. When the store is created, it logs a warning that lists the tags it found in the database and cannot explain:

```text
Unknown event tags detected: Closed, Opened not in known tags Activated
```

So removing a case from the event type hides its stored events from every read, including a full rebuild.

### Renaming an Event Case

A renamed case keeps its old name in the stored events, in their tag and in their JSON. To go on reading them, describe the rename with a `Migration` of the event type to itself, and create the event store with it:

```scala
object SQLiteEventStore {
  def make[E: Schema](transactor: Transactor, migration: Migration[E, E]): Task[SQLiteEventStore[E]]
}
```

The other event store factories take a migration the same way; see [EventStore](./event-store.md#creating-a-store--make).

```scala

sealed trait LoginEvent
object LoginEvent {
  case class UserAuthenticated(userId: String) extends LoginEvent // was UserLoggedIn
  implicit val schema: Schema[LoginEvent] = Schema.derived[LoginEvent]
}

val renames: Migration[LoginEvent, LoginEvent] =
  Migration.newBuilder[LoginEvent, LoginEvent].renameCase("UserLoggedIn", "UserAuthenticated").build

def openEvents(cache: TransactorCache): Task[SQLiteEventStore[LoginEvent]] =
  cache.get("events/logins.db").flatMap(tx => SQLiteEventStore.make[LoginEvent](tx, renames))
```

The store then:

- reads the events saved under the old name too, also when a read asks only for the new tag;
- decodes them as the renamed case;
- reports them under the new tag, so the handlers only need `on[UserAuthenticated]`;
- saves new events under the new name.

The stored events themselves are not changed. The rules:

- Only `renameCase` steps make old names readable. Do not rely on other steps of the migration to change stored events.
- If a case was renamed more than once, list every rename: `UserLoggedIn` to `UserSignedIn`, then `UserSignedIn` to `UserAuthenticated`, makes both old names read as `UserAuthenticated`.
- Keep the renames for as long as events with the old names exist.

The example below appends an event with the old version of the event type, then reads it back through a store created with the rename. It also prints the aliases the store works out:

```scala title="projection-examples/src/main/scala/projection/examples/Example11TagRename.scala"
/*
 * Example 11 of 13: renaming an event.
 *
 * Builds on Example8SchemaEvolution. Next: Example12EventFieldChanges.
 *
 * Event tags are stored as strings, so renaming a case, here `UserLoggedIn` to
 * `UserAuthenticated`, leaves old events in the store under the old tag. A
 * `Migration` that renames the case lets `TagResolver` work out the alias, so old
 * events can still be recognised and read.
 *
 * This example runs on a real SQLite event store,
 * `.projection-examples/example11/events/logins.db`, with two versions of the event type in
 * a row. Both are called `LoginEvent`, so they use the same table.
 *   1. `V1.LoginEvent` has the case `UserLoggedIn`. An event is appended with it.
 *   2. `V2.LoginEvent` has the case `UserAuthenticated`. The store is opened again
 *      with a migration that renames the case, and the old event is read back
 *      under its new name.
 *
 * It shows:
 *   - `TagResolver.resolve` building a `TagInfo` from a migration, and its
 *     helpers: `currentTags`, `aliases`, `allTags`, `expandRequested`, `isOldTag`,
 *     `currentTagFor` and `normalize`
 *   - a chain of renames (A to B to C), which resolves straight to the final name
 *   - `SQLiteEventStore.make` with a migration, so that `readAll` returns an old
 *     event with the current tag and decoded as the new case
 *
 * Run it from the repository root with:
 *   sbt "projection-examples/runMain projection.examples.Example11TagRename"
 */

package projection.examples

object Example11TagRename extends ZIOAppDefault {

  object V1 {
    sealed trait LoginEvent
    object LoginEvent {
      case class UserLoggedIn(userId: String, timestamp: String) extends LoginEvent
      object UserLoggedIn {
        implicit val schema: Schema[UserLoggedIn] = Schema.derived[UserLoggedIn]
      }
      implicit val schema: Schema[LoginEvent] = Schema.derived[LoginEvent]
    }
  }

  object V2 {
    sealed trait LoginEvent
    object LoginEvent {
      case class UserAuthenticated(userId: String, timestamp: String) extends LoginEvent
      object UserAuthenticated {
        implicit val schema: Schema[UserAuthenticated] = Schema.derived[UserAuthenticated]
      }
      implicit val schema: Schema[LoginEvent] = Schema.derived[LoginEvent]
    }
  }

  def demo: ZIO[Any, Throwable, Unit] =
    Console.printLine("\n--- Example 11: Tag Rename (UserLoggedIn -> UserAuthenticated) ---").orDie *>
      ZIO.scoped {
        for {
          _ <- Sqlite.folders(Sqlite.path("example11/events"))

          migration = Migration
                        .newBuilder[V2.LoginEvent, V2.LoginEvent]
                        .renameCase("UserLoggedIn", "UserAuthenticated")
                        .build

          tagInfo = TagResolver.resolve[V2.LoginEvent](migration)

          _ <- Console.printLine(s"  Current tags: ${tagInfo.currentTags.mkString(", ")}").orDie
          _ <- Console.printLine(s"  Aliases:      ${tagInfo.aliases}").orDie
          _ <- Console.printLine(s"  All tags:     ${tagInfo.allTags.mkString(", ")}").orDie

          expanded = tagInfo.expandRequested(Set("UserAuthenticated"))
          _       <- Console.printLine(s"  expandRequested(UserAuthenticated) -> ${expanded.mkString(", ")}").orDie

          _ <- Console.printLine(s"  isOldTag('UserLoggedIn'):      ${tagInfo.isOldTag("UserLoggedIn")}").orDie
          _ <- Console.printLine(s"  isOldTag('UserAuthenticated'): ${tagInfo.isOldTag("UserAuthenticated")}").orDie
          _ <- Console.printLine(s"  currentTagFor('UserLoggedIn'): ${tagInfo.currentTagFor("UserLoggedIn")}").orDie
          _ <- Console.printLine(s"  normalize('UserLoggedIn'):      ${tagInfo.normalize("UserLoggedIn")}").orDie
          _ <- Console.printLine(s"  normalize('UserAuthenticated'): ${tagInfo.normalize("UserAuthenticated")}").orDie

          migration2 = Migration
                         .newBuilder[V2.LoginEvent, V2.LoginEvent]
                         .renameCase("UserLoggedIn", "UserSignedIn")
                         .renameCase("UserSignedIn", "UserAuthenticated")
                         .build
          tagInfo2 = TagResolver.resolve[V2.LoginEvent](migration2)
          _       <- Console.printLine(s"  Transitive chain: ${tagInfo2.aliases}").orDie

          // 1. The first version writes an event under the tag `UserLoggedIn`.
          _ <- ZIO.scoped {
                 for {
                   cache  <- TransactorCache.make()
                   tx     <- cache.get(Sqlite.path("example11/events/logins.db"))
                   events <- SQLiteEventStore.make[V1.LoginEvent](tx)
                   _      <- events.append("user-1", V1.LoginEvent.UserLoggedIn("user-1", "10:00"))
                 } yield ()
               }

          // 2. The second version opens the same file with the migration, and also appends an event.
          _ <- ZIO.scoped {
                 for {
                   cache  <- TransactorCache.make()
                   tx     <- cache.get(Sqlite.path("example11/events/logins.db"))
                   events <- SQLiteEventStore.make[V2.LoginEvent](tx, migration)
                   _      <- events.append("user-2", V2.LoginEvent.UserAuthenticated("user-2", "11:00"))
                   all    <- events.readAll().runCollect
                   _      <- ZIO.foreachDiscard(all) { env =>
                          Console.printLine(s"  seq ${env.seq}: tag ${env.tag}, ${env.entityId}, ${env.event}").orDie
                        }
                   byNewTag <- events.readAll(Set("UserAuthenticated")).runCollect
                   _        <- Console.printLine(s"  readAll(UserAuthenticated) returns ${byNewTag.size} events").orDie
                 } yield ()
               }

          _ <- Console.printLine("  ✓ Tag Rename: migration aliases ensure old events are readable").orDie
        } yield ()
      }

  def run: ZIO[Any, Throwable, Unit] = demo
}
```

It prints `seq 1: tag UserAuthenticated, user-1, UserAuthenticated(user-1,10:00)` for the event stored as `UserLoggedIn`.

### Numeric Tags — `@eventTag`

```scala
final class eventTag(val value: Int) extends StaticAnnotation
```

By default, the tag of an event is the name of its case. `@eventTag(n)` on a case makes the tag the number `n` instead. Only the tag changes: the payload stays keyed by the case name. The numbers are not checked for uniqueness.

Renaming a numbered case needs care. The number still matches, but the stored payloads carry the old case name, so they no longer decode, and the read fails. `renameCase` does not help, because it works on tags, not on payloads. Instead, keep the old name readable with [`@Modifier.alias`](../schema/modifier.md#alias) on the renamed case:

```scala

sealed trait LoginEvent
object LoginEvent {
  // Renamed from UserLoggedIn, which was stored with the tag 1
  @eventTag(1)
  @Modifier.alias("UserLoggedIn")
  case class UserAuthenticated(userId: String) extends LoginEvent

  @eventTag(2)
  case class UserLoggedOut(userId: String) extends LoginEvent

  implicit val schema: Schema[LoginEvent] = Schema.derived[LoginEvent]
}
```

Removing `@eventTag` from a case changes its tag back to the case name, so the events stored under the number are no longer read; see [Which Stored Events Are Read](#which-stored-events-are-read).

### Changing the Fields of an Event

Stored payloads are not migrated. An old event is decoded with the current case class, and the JSON codec decides what it becomes:

- **Added field:** it takes the field's default value, `None` for an `Option`, or an empty collection. A field with none of these makes the event unreadable, with `missing required field`.
- **Removed field:** its stored value is ignored.
- **Renamed field:** the new name counts as missing, unless the field keeps its old name as an alias: `@Modifier.alias("owner") holder: String`.
- **Changed type:** the event is unreadable unless the stored value still fits the new type, as a `String` fits an `Option[String]`.

:::caution
A field that holds its default value is not written to the payload. If you change the default later, the old events that had the default read back with the new one.
:::

So add a field as an `Option` or with a default, and never change a default or a type: add a new field or a new case instead.

The example below stores two events with the first version of an event type, then reads them back with two newer versions:

```scala title="projection-examples/src/main/scala/projection/examples/Example12EventFieldChanges.scala"
/*
 * Example 12 of 13: changing the fields of an event.
 *
 * Builds on Example11TagRename. Read it first. Next: Example13EngineCatchUp.
 *
 * Stored events are never rewritten. When a case of the event type gains, loses
 * or changes a field, the events already stored keep their old JSON, and the
 * event store decodes that JSON with the current case class. No migration is
 * applied: the JSON codec decides what an old event becomes.
 *   - a field that was removed is ignored
 *   - a field that was added takes its default value, `None` for an `Option`, or
 *     an empty collection
 *   - a field that was added with none of these makes the event fail to decode
 *
 * This example stores two `AccountOpened` events with the first version of the
 * event type, in `.projection-examples/example12/events/accounts.db`, then reads
 * them back with two newer versions:
 *   1. V1 has `owner` and `plan`.
 *   2. V2 drops `plan` and adds `currency` with a default, `referrer` as an
 *      `Option` and `tags` as a `List`. The old events read back with `EUR`,
 *      `None` and an empty list.
 *   3. V3 adds `country` with no default. Reading fails with
 *      `missing required field "country"`.
 *
 * It shows:
 *   - that old events are decoded with the current case class, without a
 *     migration
 *   - which added fields are filled in, and which make an old event unreadable
 *
 * Run it from the repository root with:
 *   sbt "projection-examples/runMain projection.examples.Example12EventFieldChanges"
 */

package projection.examples

object Example12EventFieldChanges extends ZIOAppDefault {

  object V1 {
    sealed trait AccountEvent
    object AccountEvent {
      case class AccountOpened(owner: String, plan: String) extends AccountEvent
      implicit val schema: Schema[AccountEvent] = Schema.derived[AccountEvent]
    }
  }

  object V2 {
    sealed trait AccountEvent
    object AccountEvent {
      case class AccountOpened(owner: String, currency: String = "EUR", referrer: Option[String], tags: List[String])
          extends AccountEvent
      implicit val schema: Schema[AccountEvent] = Schema.derived[AccountEvent]
    }
  }

  object V3 {
    sealed trait AccountEvent
    object AccountEvent {
      case class AccountOpened(owner: String, country: String) extends AccountEvent
      implicit val schema: Schema[AccountEvent] = Schema.derived[AccountEvent]
    }
  }

  private val eventsFile: String = Sqlite.path("example12/events/accounts.db")

  // Opens the event store with the given version of the event type and reads every event.
  private def readAll[E: Schema]: Task[Chunk[EventEnvelope[E]]] =
    ZIO.scoped {
      for {
        cache  <- TransactorCache.make()
        events <- Sqlite.eventStore[E](cache, eventsFile)
        all    <- events.readAll().runCollect
      } yield all
    }

  def demo: ZIO[Any, Throwable, Unit] =
    Console.printLine("\n--- Example 12: Changing the Fields of an Event (AccountOpened V1 -> V2 -> V3) ---").orDie *>
      ZIO.scoped {
        for {
          _ <- Sqlite.folders(Sqlite.path("example12/events"))

          // 1. The first version stores two events.
          _ <- ZIO.scoped {
                 for {
                   cache <- TransactorCache.make()
                   v1    <- Sqlite.eventStore[V1.AccountEvent](cache, eventsFile)
                   _     <- v1.append("acc-1", V1.AccountEvent.AccountOpened("alice", "basic"))
                   _     <- v1.append("acc-2", V1.AccountEvent.AccountOpened("bob", "pro"))
                 } yield ()
               }

          // 2. A newer version reads them: `plan` is ignored, and the new fields are filled in.
          v2 <- readAll[V2.AccountEvent]
          _  <- ZIO.foreachDiscard(v2)(e => Console.printLine(s"  V2 reads seq ${e.seq}: ${e.event}").orDie)

          // 3. A version with a new required field cannot read them.
          v3 <- readAll[V3.AccountEvent].either
          _  <- v3 match {
                 case Right(events) => Console.printLine(s"  V3 reads ${events.size} events").orDie
                 case Left(error)   => Console.printLine(s"  V3 fails: ${error.getMessage}").orDie
               }

          _ <- Console
                 .printLine("  ✓ Event fields: old events are read with the current case class, without a migration")
                 .orDie
        } yield ()
      }

  def run: ZIO[Any, Throwable, Unit] = demo
}
```

It prints `V2 reads seq 1: AccountOpened(alice,EUR,None,List())`, and for the third version `missing required field "country"`.

An event that cannot be decoded fails the whole read:

- While catching up, the engine stops reading that source and logs `bind failed`. It does not follow the source's new events either, until the application is restarted with code that can read the event.
- During a full rebuild, the replay stops at that event and is logged. The rows hold only the events before it, yet the new fingerprint is saved and the watermark stays at 0. When the application next starts with fixed code, the engine catches up from the first event on top of those rows, so the events the rebuild had applied are applied twice. After fixing the code, [rebuild on demand](#rebuilding-on-demand) instead.

## Limitations

These cases are not handled yet:

- **Several sources.** A projection that reads several event stores keeps one watermark for all of them, although each store numbers its events from 1. An event appended to one source while the application is stopped can fall below the watermark set by another source, and is then never applied. A full rebuild replays only the first source, and the events of the others below the watermark are lost; deleting the store file does not help.
- **Cross-entity projections.** A full rebuild writes every row to the main file and leaves the shard files as they were. Later events for those rows fail with `no column named ...`, and queries return the stale rows of the main file. To rebuild such a projection, stop the application and delete the main file and its shard files.
- **Engines without event stores.** An engine created with `make` or `makeWithConfig` has no event stores, so it cannot rebuild: it logs `no EventStore for source ...`, keeps the old fingerprint, and queries fail on the changed table.
- **`TestEngine`.** It starts the engine inside `make`, on new in-memory stores, so a test can neither store an old fingerprint first nor register a migration before `start`. Test a migration with two engines in a row on SQLite files, as the examples on this page do.

## Advanced: Running the Steps Yourself

:::note[Advanced]
You do not need this section to migrate a projection: the engine runs these steps for you. It documents the building blocks, for tests and for custom runners.
:::

### `SchemaEvolution`

```scala
object SchemaEvolution {
  def needsRebuild[A: Schema](store: ProjectionStore[A]): Task[Boolean]
  def rebuild[A](store: ProjectionStore[A], eventStore: EventStore[_], spec: Projection[A], currentHash: String): Task[Unit]
  def rebuildWithHash[A: Schema: EntityPath](store: ProjectionStore[A], eventStore: EventStore[_], spec: Projection[A]): Task[Unit]
  def isSimpleAddFieldMigration(migration: Migration[?, ?]): Boolean
  def tryMigrationShortcut[A: Schema: EntityPath](
    store: ProjectionStore[A],
    migrationOpt: Option[Migration[?, A]],
    currentHash: String
  ): Task[Boolean]
  def checkAndRebuild[A: Schema: EntityPath](
    store: ProjectionStore[A],
    eventStore: EventStore[_],
    spec: Projection[A],
    config: SchemaEvolutionConfig = SchemaEvolutionConfig.default,
    migrationOpt: Option[Migration[?, A]] = None
  ): Task[Boolean]
}
```

- `needsRebuild` is `true` when a fingerprint is stored and differs from the current one. It is `false` for a new store.
- `rebuild` does the [full rebuild](#full-rebuild) and saves the fingerprint you pass; `rebuildWithHash` computes the fingerprint from `A`.
- `isSimpleAddFieldMigration` says whether a migration qualifies for the shortcut: it is not empty, and every step adds a top-level field. It is useful in a test.
- `tryMigrationShortcut` runs the shortcut and returns `true`, or returns `false` when there is no migration or it does not qualify.
- `checkAndRebuild` compares the fingerprints, then runs the shortcut or a rebuild, like the engine does on `start`. It returns `true` when it changed the store.

`SchemaEvolutionConfig` configures these helpers, not the engine:

```scala
final case class SchemaEvolutionConfig(
  rebuildParallelism: Int = 4,
  lazyRebuild: Boolean = false,
  enableMigrationShortcut: Boolean = true
)
```

- `enableMigrationShortcut = false` makes `checkAndRebuild` always rebuild.
- `rebuildParallelism` is how many projections `bulkRebuildPending` rebuilds at the same time.
- `lazyRebuild` is not read.

The other members of `SchemaEvolution`, `tryMigrationShortcutForSpec`, `registerMigration`, `lazyRebuildIfNeeded` and `bulkRebuildPending`, are the engine's own building blocks and are not meant to be called directly. A custom store takes part in a rebuild and in the shortcut through `recreateTable` and `addColumn`; see [ProjectionStore](./projection-store.md).

### `TagInfo` and `TagResolver`

The event store gathers what it knows about tags in a `TagInfo`, built by `TagResolver` from the event type and the migration:

```scala
final case class TagInfo(
  aliases: Map[String, String],
  allTags: Set[String],
  currentTags: Set[String],
  migration: DynamicMigration
) {
  def isOldTag(tag: String): Boolean
  def currentTagFor(oldTag: String): Option[String]
  def normalize(tag: String): String
  def migrateValue(dv: DynamicValue): Either[SchemaError, DynamicValue]
  def expandRequested(requested: Set[String]): Set[String]
  def unknownTags(distinctDbTags: Set[String]): Set[String]
}
```

- `aliases` maps each old tag to the current tag it became, with chains of renames followed.
- `currentTags` are the tags of the event type now, and `allTags` adds the old ones.
- `migration` is the migration the store was given.
- `isOldTag` and `currentTagFor` look a tag up in `aliases`. `normalize` returns the current tag for an old one, and any other tag unchanged.
- `expandRequested` widens the tags that a read asks for with their old names. An empty request means all the known tags.
- `unknownTags` returns the tags found in the database that are neither current nor old; they make the startup warning.
- `migrateValue` runs the migration on a decoded value. The store uses it only as a fallback, when a renamed event does not decode after its old case name was replaced.

```scala
object TagResolver {
  def resolve[E: Schema]: TagInfo
  def resolve[E: Schema](migration: Migration[E, E]): TagInfo
  def resolveOpt[E](opt: Option[Migration[E, E]])(using schema: Schema[E]): TagInfo
  def resolveImplicit[E: Schema](implicit migration: Migration[E, E] = null): TagInfo
  inline def resolveAuto[E](using schema: Schema[E]): TagInfo
  def currentTags[E](using schema: Schema[E]): Set[String]
  def allTags[E: Schema](migration: Migration[E, E] = null): Set[String]
  def aliases[E: Schema](migration: Migration[E, E] = null): Map[String, String]
  def aliasMap[E](migration: Migration[E, E]): Map[String, String]
  def aliasMapFromDynamic(dm: DynamicMigration): Map[String, String]
  def unknownTags(tagInfo: TagInfo, distinctDbTags: Set[String]): Set[String]
}
```

- `resolve` builds the `TagInfo` of an event type, with the aliases of a migration when you give one; `SQLiteEventStore.make` uses it. `resolveOpt` takes the migration as an `Option`, and `resolveImplicit` and `resolveAuto` use a migration found in implicit scope, if there is one.
- `currentTags`, `allTags` and `aliases` each return one part of the `TagInfo`. `aliasMap` gives the aliases of a migration with chains followed, and `aliasMapFromDynamic` the renames of a `DynamicMigration`, without following chains.
- `unknownTags` is the same as `TagInfo#unknownTags`.

The `TagResolver[E]` trait is a view of one `TagInfo`: implement `tagInfo`, and `aliases`, `allTags` and `currentTags` come from it. For how the event store uses its `TagInfo`, see [EventStore](./event-store.md#the-stores-tags--taginfo).

## See Also

- [ProjectionEngine](./projection-engine.md) — `start`, `query` and the engine's configuration.
- [EventStore[E]](./event-store.md) — the event store and its factories.
- [ProjectionStore[A]](./projection-store.md) — the store hooks `recreateTable` and `addColumn`.
- [Testing](./testing.md) — the in-memory helpers.
- [Migration](../schema/migration.md) — building a `Migration`: `addField`, `renameCase` and the other steps.
- [Modifier](../schema/modifier.md) — `alias`, `rename` and `config`.

---

## ProjectionEngine

`ProjectionEngine` is the **runtime of the projection module**. A [`Projection[A]`](./projection.md) only describes a read model: which sources it reads, how events are routed, and what each event does to a row. The engine is what runs those descriptions. It reads events from [`EventStore`](./event-store.md)s, runs the matching handler for each event, applies the resulting action to a [`ProjectionStore`](./projection-store.md), and answers queries from the stores. Its fundamental operations are `ProjectionEngine#start` and `ProjectionEngine#query`.

`ProjectionEngine`:

- **Resumes where it stopped.** After every event it applies, the engine saves the event's sequence number as the projection's **watermark**, the position of the last event applied. After a restart it continues with the events after the watermark.
- **Catches up before it returns.** `start` first applies the stored events after the watermark, a step called **catch-up**, and returns only when every source has finished it. Each source then gets a **live fiber**, a background ZIO fiber that applies the new events as they are appended.
- **Keeps each source in order.** The events of one source are processed one at a time, in the order they were appended. Separate sources and separate projections run on separate fibers, with no order between them.
- **Is eventually consistent.** Events appended after `start` are applied in the background, so a query right after an append may not see the change yet.
- **Runs inside a `Scope`.** A ZIO `Scope` owns resources and background fibers, and releases them when it closes. The fibers of the engine live until the `Scope` given to `start` closes. There is no stop method.

This is the API that applications run. Tests usually use [`TestEngine`](./testing.md#testengine) instead, which wires an engine on in-memory stores. Add the module and the `sqlite-jdbc` driver as described under [Installation](./index.md#installation). The module is built for the JVM only, and for Scala 3 only (3.3.7 and 3.9.0).

The class has a private constructor, so engines are created with the factories on its companion object. Its public members are:

```scala
final class ProjectionEngine private[projection] (
  // the projections, their stores, the event stores and the cache are private
  val config: ProjectionEngineConfig // the configuration the engine was created with
  // ...
) {
  def transactorCache: TransactorCache // the cache the engine opens SQLite files with
  def registerMigration[Old, New](spec: Projection[New], migration: Migration[Old, New]): Task[Unit]
  def start: ZIO[Scope, Nothing, Unit]
  def query[A](spec: Projection[A], entityId: String): Task[Option[A]]
  def queryByName[A](specName: String, entityId: String): Task[Option[A]]
  def getLastProcessedSeq[A](spec: Projection[A]): Task[Long]
  def storesMap: Map[String, ProjectionStore[_]] // the primary store of each projection, by name; no shards
  def eventStoresMap: Map[String, EventStore[_]] // the event stores, by source name
}
```

`Migration` is `zio.blocks.schema.migration.Migration`. The class has no type parameter: each method that needs a row type takes it as its own type parameter.

## Usage

The program below defines an event, a read model and a projection. It opens an event store and a projection store on SQLite files, creates the engine with `ProjectionEngine.makeWithStores`, and appends two events before it starts the engine. Because `start` returns only after catch-up, the queries right after it already see both rows, and the watermark is the sequence number of the second event:

```scala title="projection-examples/src/main/scala/projection/examples/Example13EngineCatchUp.scala"
/*
 * Example 13 of 13: catch-up in `start`.
 *
 * Read Example1QuickStart first.
 *
 * The program of the Usage section of the `ProjectionEngine` page in the
 * documentation.
 *
 * `ProjectionEngine#start` first applies the events that are already stored, a
 * step called catch-up, and returns only when it is done. Only then does the
 * engine follow new events, on a background fiber. So the events appended here
 * before `start` can be read right after it, without polling. After each event
 * it applies, the engine saves the event's sequence number in the projection's
 * store. That number, the watermark, is where it resumes on the next start.
 *
 * It shows:
 *   - opening an event store and a projection store, and giving them to
 *     `ProjectionEngine.makeWithStores`
 *   - events appended before `start`, applied by catch-up and read right after
 *     it
 *   - the watermark, read with `getLastProcessedSeq`
 *
 * This example runs on real SQLite files: the events are kept in
 * `.projection-examples/example13/events/users.db`, and the projection in
 * `.projection-examples/example13/users/user_profiles.db`.
 *
 * Run it from the repository root with:
 *   sbt "projection-examples/runMain projection.examples.Example13EngineCatchUp"
 */

package projection.examples

object Example13EngineCatchUp extends ZIOAppDefault {

  // The event. The event store holds this one type.
  case class UserCreated(name: String, email: String)
  object UserCreated {
    implicit val schema: Schema[UserCreated] = Schema.derived[UserCreated]
  }

  // The read model, one row per user. `@Modifier.id` marks the ID field, and `EntityPath.derived` finds it.
  case class UserProfile(@Modifier.id userId: String, name: String, email: String)
  object UserProfile {
    implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
    implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
  }

  // The projection: each `UserCreated` inserts the row of the user it was recorded under
  val projection: Projection[UserProfile] = Projection[UserProfile]("user_profiles")
    .from("users")
    .routeToSelf
    .on[UserCreated]
    .insert((e, ctx) => UserProfile(ctx.entityId, e.name, e.email))

  private def show(row: Option[UserProfile]): String =
    row.map(u => s"${u.name} <${u.email}>").getOrElse("not found")

  def demo: ZIO[Any, Throwable, Unit] =
    Console.printLine("\n--- Example 13: Catch-up in start ---").orDie *>
      ZIO.scoped {
        for {
          _        <- Sqlite.folders(Sqlite.path("example13/events"), Sqlite.path("example13/users"))
          cache    <- TransactorCache.make()
          tx       <- cache.get(Sqlite.path("example13/events/users.db"))
          users    <- SQLiteEventStore.make[UserCreated](tx)
          profiles <- SQLiteProjectionStore.make[UserProfile](Sqlite.path("example13/users/user_profiles.db"), cache)
          engine   <- ProjectionEngine.makeWithStores(
                      specs = List(projection),
                      stores = Map("user_profiles" -> profiles), // projection name -> its primary store
                      eventStores = Map("users" -> users), // source name given to `from` -> its event store
                      cache = cache
                    )
          // Appended before `start`, so catch-up applies them
          _ <- users.append("user-1", UserCreated("Alice", "alice@example.com"))
          _ <- users.append("user-2", UserCreated("Bob", "bob@example.com"))
          // `start` returns only after catch-up, so both rows are there right away, with no polling
          _     <- engine.start
          alice <- engine.query(projection, "user-1")
          bob   <- engine.query(projection, "user-2")
          seq   <- engine.getLastProcessedSeq(projection)
          _     <- Console.printLine(s"  user-1: ${show(alice)}").orDie
          _     <- Console.printLine(s"  user-2: ${show(bob)}").orDie
          _     <- Console.printLine(s"  Watermark: $seq").orDie
        } yield ()
      }

  def run: ZIO[Any, Throwable, Unit] = demo
}
```

It prints `user-1: Alice <alice@example.com>`, `user-2: Bob <bob@example.com>` and `Watermark: 2`. When `ZIO.scoped` ends, its `Scope` closes, so the engine's live fiber stops and the cache closes. The files stay on disk, so an engine started on them again would continue after event 2.

## Creating an Engine

The companion object has four factories. They differ in what they create for you:

- **`ProjectionEngine.makeWithStores`** takes everything: the primary store of each projection, the event stores, and a `TransactorCache`. It needs no `Scope`, and an invalid projection name or folder is a **defect**: the effect dies, instead of failing with an error that `catchAll` can handle.
- **`ProjectionEngine.makeWithStoresVarargs`** takes the event stores and a `TransactorCache`, and creates a SQLite primary store for each projection. It needs no `Scope`, and an invalid name or folder is a **typed failure**: the task fails with an `IllegalArgumentException`, which `catchAll` can handle.
- **`ProjectionEngine.make`** and **`ProjectionEngine.makeWithConfig`** take only the projections. They create the primary stores and a `TransactorCache` that is closed with the enclosing `Scope`, so they need one. An invalid name or folder is a typed failure. They give the engine no event stores, so it has nothing to read.

The **primary store** of a projection is the store the engine reads and writes for it, and where it keeps its watermark. None of the factories opens a file: a store opens its file the first time it is used, so a missing driver or folder shows up then.

### From Your Own Stores — `makeWithStores`

`ProjectionEngine.makeWithStores` builds an engine from stores you created yourself. It is the factory the [Usage](#usage) example uses, and the one `TestEngine` uses for its in-memory stores:

```scala
object ProjectionEngine {
  def makeWithStores(
    specs: List[Projection[_]],
    stores: Map[String, ProjectionStore[_]],
    eventStores: Map[String, EventStore[_]],
    cache: TransactorCache,
    config: ProjectionEngineConfig = ProjectionEngineConfig.default
  ): UIO[ProjectionEngine]
}
```

- `stores` is keyed by **projection name**, the name given to `Projection[A](name)`.
- `eventStores` is keyed by **source name**, the name given to `from`. How a source finds its store is described under [`start`](#starting-the-engine--start).
- `stores` must have an entry for every projection. This is not checked when the engine is created: a missing entry makes `start` or `query` die with a `NoSuchElementException`.
- The names and folders of the projections are checked with [`validateName` and `validateBasePath`](#companion-helpers--validatename-validatebasepath-and-boundedhub). An invalid one is a defect, which is why the result is a `UIO`.
- `cache` also opens the SQLite files of the shards of cross-entity projections. A **shard** is the store of one routing key, and a **routing key** is the value that `routedBy` reads from an event to choose the shard; see [CrossEntity](./projection.md#crossentity-one-drawer-per-group). The engine never closes the cache: close it yourself, or create it with `TransactorCache.make` inside a `Scope`.

### Creating the Primary Stores — `makeWithStoresVarargs`

`ProjectionEngine.makeWithStoresVarargs` takes the event stores, like `makeWithStores`, but creates a `SQLiteProjectionStore` for each projection itself. The projections go in a second parameter list:

```scala
object ProjectionEngine {
  def makeWithStoresVarargs(
    cache: TransactorCache,
    eventStores: Map[String, EventStore[_]],
    config: ProjectionEngineConfig = ProjectionEngineConfig.default
  )(specs: Projection[_]*): Task[ProjectionEngine]
}
```

Each primary store is the file `<basePath>/<name>.db`, where `basePath` comes from the projection's [`EntityPath`](./projection.md#entity-paths), or `global/<name>.db` for a global projection. The paths are relative to the working directory, and SQLite does not create folders, so the folders must exist before the stores are used.

The snippet below runs the `projection` from [Usage](#usage) and a second projection of the same row type, each reading its own event store. Both primary stores land in `users/`, the folder derived from `UserProfile`, so the program creates `events/` and `users/` first:

```scala

val admins: Projection[UserProfile] = Projection[UserProfile]("admin_profiles")
  .from("admins")
  .routeToSelf
  .on[UserCreated]
  .insert((e, ctx) => UserProfile(ctx.entityId, e.name, e.email))

val twoSources: ZIO[Scope, Throwable, ProjectionEngine] = for {
  _      <- ZIO.attemptBlocking(List("events", "users").foreach(d => Files.createDirectories(Paths.get(d))))
  cache  <- TransactorCache.make()
  users  <- cache.get("events/users.db").flatMap(tx => SQLiteEventStore.make[UserCreated](tx))
  admin  <- cache.get("events/admins.db").flatMap(tx => SQLiteEventStore.make[UserCreated](tx))
  engine <- ProjectionEngine.makeWithStoresVarargs(
              cache,
              Map[String, EventStore[?]]("users" -> users, "admins" -> admin)
            )(projection, admins)
  _      <- engine.start
} yield engine
```

The engine reads `users/user_profiles.db` and `users/admin_profiles.db`. Like `makeWithStores`, the factory registers no finalizer, so the cache here is closed by the `Scope` that `TransactorCache.make` runs in.

### Without Event Stores — `make` and `makeWithConfig`

`ProjectionEngine.make` and `ProjectionEngine.makeWithConfig` take only the projections. `make` uses `ProjectionEngineConfig.default`, and `makeWithConfig` takes the settings described under [Configuration](#configuration):

```scala
object ProjectionEngine {
  def make(specs: Projection[_]*): ZIO[Scope, Throwable, ProjectionEngine]
  def makeWithConfig(config: ProjectionEngineConfig, specs: Projection[_]*): ZIO[Scope, Throwable, ProjectionEngine]
}
```

They check the names and folders, create a `TransactorCache` with `TransactorCache.make()`, which is closed when the enclosing `Scope` closes, and create the primary stores at the same paths as `makeWithStoresVarargs`:

```scala
val withoutEvents: ZIO[Scope, Throwable, Set[String]] =
  ProjectionEngine
    .makeWithConfig(ProjectionEngineConfig(batchSize = 500), projection)
    .map(_.storesMap.keySet)
```

:::caution[No events to read]
An engine created with `make` or `makeWithConfig` has no event stores, and there is no method to add them later. Its `start` logs `no EventStore for source '...' - skipping` for every source, so the engine neither catches up nor follows new events. It cannot rebuild a store after a schema change either: it keeps the old fingerprint. See [Limitations](./migration.md#limitations). To run projections on SQLite files, use `makeWithStores` or `makeWithStoresVarargs`.
:::

### Companion Helpers — `validateName`, `validateBasePath` and `boundedHub`

The companion object also has the checks the factories run, and a helper for building a hub:

```scala
object ProjectionEngine {
  def validateName(name: String): Unit
  def validateBasePath(path: String): Unit
  def boundedHub[E](capacity: Int = 4096): UIO[Hub[EventEnvelope[E]]]
}
```

- `validateName` throws an `IllegalArgumentException` unless the name is non-empty and made only of letters, digits, `_` and `-`. A projection name is also a file name, which is why it is checked.
- `validateBasePath` throws an `IllegalArgumentException` for a folder path that could leave the working directory: an empty path, a backslash, `..`, `//`, or a leading `/`.
- The factories run both on each projection: `validateName` on its name, and `validateBasePath` on the folder of its `EntityPath` and on the folder of a `@path` annotation on the row type.
- `boundedHub` creates a bounded `Hub` for live events. A ZIO `Hub` hands each message published to it to every subscriber, and an event store publishes each new event to its hub. When a bounded hub is full, publishing waits, so a slow consumer slows the producer down instead of losing events. The engine does not call it.

A hub from `boundedHub` can be given to an event store, as described under [Choosing the Hub](./event-store.md#choosing-the-hub--makewithhub):

```scala
val smallHub: ZIO[Scope, Throwable, SQLiteEventStore[UserCreated]] = for {
  cache <- TransactorCache.make()
  tx    <- cache.get("events/users.db")
  hub   <- ProjectionEngine.boundedHub[UserCreated](capacity = 256)
  users <- SQLiteEventStore.makeWithHub[UserCreated](tx, hub)
} yield users
```

## Core Operations

An engine is started once, then queried; the operations below cover starting it, reading rows back and registering a migration.

### Starting the Engine — `start`

`ProjectionEngine#start` runs the projections and returns once they have caught up:

```scala
final class ProjectionEngine {
  def start: ZIO[Scope, Nothing, Unit]
}
```

It runs these steps in order:

1. **Schema check.** For each projection, it compares the fingerprint of the row type stored in the primary store with the current one, and brings a stale store up to date, as described under [Schema Changes](#schema-changes).
2. **Event store lookup.** For each source of each projection, it looks for the event store under the source's name, then under `"_default"`, and then takes the first store in the map. So a misspelt source name does not fail: it reads another store. With no event stores at all, it logs a warning and skips the source.
3. **Catch-up.** It reads the watermark from the primary store and applies the stored events after it, read with `EventStore#readFrom` in batches of up to `batchSize`.
4. **Live fiber.** It subscribes to the event store's `Hub`, which hands every new event to each subscriber, and forks the live fiber, which applies those events as they arrive.
5. **Pre-warm.** With `lazyRebuild` on, it forks one more fiber that rebuilds the pending projections in the background.

Steps 2 to 4 run for one source after another, not in parallel. `start` returns when every source has caught up and its live fiber is running. A source whose catch-up fails is logged with `bind failed` and skipped, and it gets no live fiber. With `lazyRebuild`, `start` does not wait for the pre-warm.

`start` has no error type: a failing source or rebuild is logged, not returned. Its fibers are forked in the `Scope` it runs in, so closing that `Scope` stops the engine. Wrap the program in `ZIO.scoped`, as [Usage](#usage) does, or run it in a `ZIOAppDefault`, which provides a `Scope` for the whole application.

:::note[Failed events]
The engine logs an event that fails and goes on with the next one. An event fails when:

- its handler throws;
- no handler matches it, which is logged at debug level only;
- the store rejects the action.

The watermark moves only after an event succeeds. So the next event that succeeds moves it past the failed one, and the failed event is never retried.
:::

:::caution[Lost and doubled events]
Nothing connects catch-up to the live fiber, and the watermark is saved in a separate write after each action. So:

- An event appended while `start` is catching up is not applied. Once a later event is applied, not even a restart applies it.
- An event is applied twice when catch-up reads it but its publish to the hub is delayed until after the subscription, for example by a full hub, or when the process stops between applying it and saving the watermark.

Avoid appending events while the engine starts, and prefer actions that give the same row when applied twice, such as `Upsert` or `FieldUpdate.Set`.
:::

### Reading Rows — `query`, `queryByName` and `getLastProcessedSeq`

Three methods read from the stores of a projection:

```scala
final class ProjectionEngine {
  def query[A](spec: Projection[A], entityId: String): Task[Option[A]]
  def queryByName[A](specName: String, entityId: String): Task[Option[A]]
  def getLastProcessedSeq[A](spec: Projection[A]): Task[Long]
}
```

- `query` returns the row with the ID `entityId`, or `None`. If the projection is waiting for a [lazy rebuild](./migration.md#rebuilding-later--lazyrebuild), the query first runs the rebuild and waits for it, and a failed rebuild fails the query. Without an event store, the projection is just unmarked, with no rebuild. The query then reads the primary store.
- For a cross-entity projection only, when the primary store has no such row, `query` searches the shards this engine has opened, in no particular order, and returns the first match. The engine opens a shard when it routes an event to it, and a read error in a shard counts as no match.
- `queryByName` reads only the primary store of the projection named `specName`, and returns `None` for an unknown name. It neither runs a pending rebuild nor searches shards, and the type `A` is not checked: the caller chooses it.
- `getLastProcessedSeq` returns the watermark. It is kept in the primary store, also for a cross-entity projection, and `0` means that no event was applied yet.

Events appended after `start` are applied by the live fibers, so a query right after an append can still return `None`. Poll until the row appears. This function appends an event and asks again every 20 milliseconds until the new row is there:

```scala
def appendAndRead(engine: ProjectionEngine, users: EventStore[UserCreated]): Task[Option[UserProfile]] =
  users.append("user-3", UserCreated("Carol", "carol@example.com")) *>
    engine.query(projection, "user-3").delay(20.millis).repeatUntil(_.isDefined)
```

### Registering a Migration — `registerMigration`

`ProjectionEngine#registerMigration` tells the engine how the row type of a projection changed, so that it can add columns instead of rebuilding:

```scala
final class ProjectionEngine {
  def registerMigration[Old, New](spec: Projection[New], migration: Migration[Old, New]): Task[Unit]
}
```

Register it before `start`, which is where the schema check runs. The migration is kept under the projection's name, so registering another one replaces it. When it is used, and what the old rows get, is described under [Adding Fields Without a Rebuild](./migration.md#adding-fields-without-a-rebuild--registermigration).

## Configuration

`ProjectionEngineConfig` holds the settings of an engine, and `ProjectionEngineConfig.default` is the value with every default:

```scala
final case class ProjectionEngineConfig(
  batchSize: Int = 100,
  batchTimeout: zio.Duration = 50.millis,
  ringCapacity: Int = 4096,
  rebuildParallelism: Int = 4,
  lazyRebuild: Boolean = false,
  evolution: SchemaEvolutionConfig = SchemaEvolutionConfig.default
)

object ProjectionEngineConfig {
  val default: ProjectionEngineConfig = ProjectionEngineConfig()
}
```

- `batchSize` is the most events that catch-up reads together. Each event of a batch is still applied, and its watermark saved, on its own, so the size does not change how much is applied again after a crash. Live events are processed one at a time.
- `batchTimeout` is how long catch-up waits for a batch to fill before it applies what it has.
- `ringCapacity` is not read by the engine. Only `TestEngine.makeWithConfig` reads it, as the capacity of the hub of each in-memory source. A `SQLiteEventStore` created with `make` has its own hub of 4,096 events; to choose another, see [Choosing the Hub](./event-store.md#choosing-the-hub--makewithhub).
- `rebuildParallelism` is the size of the groups in which the pre-warm fiber rebuilds pending projections. The groups run one after another, and so do the projections in a group, so it does not make rebuilds run in parallel. It is read only when `lazyRebuild` is on.
- `lazyRebuild` makes `start` mark a projection whose schema changed instead of rebuilding it; see [Rebuilding Later](./migration.md#rebuilding-later--lazyrebuild).
- `evolution` is not read by the engine. In particular, its `enableMigrationShortcut` is ignored: the engine always tries the shortcut when a migration is registered.

A configuration is an ordinary case class value, built with named arguments or copied from the default:

```scala
val catchUpInLargeBatches = ProjectionEngineConfig(batchSize = 1000, batchTimeout = 200.millis)
val rebuildLater          = ProjectionEngineConfig.default.copy(lazyRebuild = true)
```

## Schema Changes

On every `start`, before catch-up, the engine checks whether the stored rows still match the row type. It compares two **fingerprints**, hashes of the shape of the row type: the one saved in the primary store, and the one of the current row type. See [How a Change Is Detected](./migration.md#how-a-change-is-detected--schemahash). What it does next depends on the result:

- **No saved fingerprint**, because the store is new: it saves the current one.
- **Same fingerprint:** nothing.
- **Different fingerprint:** it brings the rows up to date in one of two ways:
  - with the [add-column shortcut](./migration.md#adding-fields-without-a-rebuild--registermigration), if a migration registered for the projection only adds fields;
  - otherwise with a [full rebuild](./migration.md#full-rebuild) from the projection's event store.
- **Different fingerprint, with [`lazyRebuild`](./migration.md#rebuilding-later--lazyrebuild) on:** it only marks the projection. The first `query`, or the pre-warm fiber, rebuilds it later.

Two limits apply; see [Limitations](./migration.md#limitations):

- A rebuild replays only one event store: the one `start` uses for the projection's first source.
- An engine without event stores, made with `make` or `makeWithConfig`, changes nothing and keeps the old fingerprint.

## Connection Management

The SQLite stores and the engine open their database files through a `TransactorCache`. It keeps one `Transactor`, the [SQL module](../sql/index.md)'s handle on a database file, for each file path:

```scala
object TransactorCache {
  def make(maxSize: Int = 256): ZIO[Scope, Nothing, TransactorCache]
  def makeUnscoped(maxSize: Int = 256): UIO[TransactorCache]
  def live(maxSize: Int = 256): ZLayer[Scope, Nothing, TransactorCache]
  def liveConfig(config: TransactorCacheConfig): ZLayer[Scope, Nothing, TransactorCache]
}

trait TransactorCache {
  def get(path: String): Task[Transactor]
  def evict(path: String): Task[Unit]
  def close: Task[Unit]
  def size: Task[Int]
}
```

- **Share one cache.** Create it once and give it to the projection stores and the engine. Open each event store on the `Transactor` that `get` returns for its file. The [Usage](#usage) example does this.
- **Closing it.** `make` closes the cache when its `Scope` closes. `makeUnscoped` creates one that you close yourself with `close`. `live` and `liveConfig` provide one as a `ZLayer`; `liveConfig` takes the size as a `TransactorCacheConfig(maxSize)`.
- **Its size.** `maxSize`, 256 by default, is how many file paths the cache keeps. When it is full, it drops the least recently used path, and the next `get` for that file sets it up again. A `maxSize` of 0 or less means no limit. `evict` drops one path, and `size` returns how many paths it keeps.

## Testing

[`TestEngine`](./testing.md#testengine) creates an `InMemoryProjectionStore` for each projection and an in-memory event store for each source name, then calls `ProjectionEngine.makeWithStores` and `start` for you. The engine it wraps is available as its `engine` field, so a test can call the operations on this page with no files and no `sqlite-jdbc`. Because `start` runs inside `TestEngine.make`, a test cannot register a migration first; see [Limitations](./migration.md#limitations).

## See Also

- [Projection[A]](./projection.md) — the descriptor the engine runs: its sources, routing, handlers and actions.
- [EventStore[E]](./event-store.md) — where the engine reads events from, during catch-up and live.
- [ProjectionStore[A]](./projection-store.md) — where the engine writes the rows and keeps the watermark.
- [Migration](./migration.md) — what `start` does when the row type or the events change.
- [Testing](./testing.md) — `TestEngine`, which runs the engine on in-memory stores.
- [SQL Module](../sql/index.md) — the `Transactor` behind the SQLite stores.

---

## ProjectionStore[A]

`ProjectionStore[A]` is where the rows of one read model `A` are kept. Next to the rows, it keeps the watermark (the number of the last event applied) and the schema hash (a fingerprint of the row type):

```scala
trait ProjectionStore[A] {
  // Rows
  def insert(a: A): Task[Unit]
  def upsert(a: A): Task[Unit]
  def updateFields(entityId: String, updates: Chunk[FieldUpdate]): Task[Unit]
  def delete(entityId: String): Task[Unit]
  def truncate: Task[Unit]
  def findById(entityId: String): Task[Option[A]]

  // Watermark and schema hash
  def getLastProcessedSeq: Task[Long]
  def updateLastProcessedSeq(seq: Long): Task[Unit]
  def getSchemaHash: Task[Option[String]]
  def updateSchemaHash(hash: String): Task[Unit]

  // Used when the row type changes
  def recreateTable(): Task[Unit] = truncate
  def addColumn(columnName: String, sqlType: String): Task[Unit] = ZIO.unit
}
```

:::note
You rarely call these methods yourself: the engine does. You create a store to pass to `ProjectionEngine.makeWithStores`, or implement the trait for another database.
:::

## Motivation

Events record everything that happened, but they answer questions such as "what is Alice's email now?" badly: you would have to replay all of her events every time. A read model keeps the current answer instead, one row per user, and updates it as events arrive.

`ProjectionStore[A]` is where those rows live. Next to the rows it keeps the watermark, so that the engine knows where to continue after a restart, and a fingerprint of the row type, so that the engine notices when the row class changes. Handlers never write SQL: the engine turns their actions into calls on the store. The same interface works on a SQLite file in production and on an in-memory map in tests.

## Usage

There are two stores:

- `SQLiteProjectionStore` keeps the rows in a SQLite file. Create one with `SQLiteProjectionStore.make[A](path, cache)` to pass to `ProjectionEngine.makeWithStores`, as the [ProjectionEngine example](./projection-engine.md#usage) does. The folder of `path` must exist.
- `InMemoryProjectionStore` keeps the rows in memory, for tests; see [Testing](./testing.md#inmemoryprojectionstore).

The other engine factories create their SQLite stores themselves.

:::note[Writing your own store]
`recreateTable` and `addColumn` have defaults that only empty the rows and do nothing. That is enough for a store without tables, such as the in-memory one. A store with tables must override both, or a changed row type will not fit its table; see [Changing the Read Model](./migration.md#changing-the-read-model).
:::

## See Also

- [Projection module overview](./index.md) — how the event store, the projections, the engine and the projection stores fit together.
- [Projection[A]](./projection.md) — the handlers whose actions the engine applies to a store, and [Aggregating Fields](./projection.md#aggregating-fields) for `FieldUpdate`.
- [ProjectionEngine](./projection-engine.md) — the runtime that calls the store, and creates the SQLite stores for most factories.
- [Testing](./testing.md) — `InMemoryProjectionStore` and the other in-memory helpers.
- [Migration](./migration.md) — the full rebuild and the add-column shortcut, which call `recreateTable` and `addColumn`.

---

## Projection[A]

`Projection[A]` is the declarative description of a read model `A` built from events. It encodes which event sources to listen to, how to route each event to a specific entity or shard, and what each event type does to the read model's shape. A `Projection` is pure data — it carries no effects and makes no I/O calls; a `ProjectionEngine` reads it and drives the actual event processing.

## Creating a Projection

The `Projection` companion object provides three factory methods. Each returns a new projection with no sources yet, ready for the builder chain:

```scala
object Projection {
  def apply[A: Schema: EntityPath](name: String): Projection[A]
  def global[A: Schema](name: String): Projection[A]
  def apply[A](name: String, entityPath: EntityPath[A])(using schema: Schema[A]): Projection[A]
}
```

- `Projection[A](name)` keeps one row for each entity ID, in the folder of its [`EntityPath[A]`](#entity-paths). It needs a [`Schema[A]`](../schema/schema.md) and an `EntityPath[A]` in implicit scope.
- `Projection.global[A](name)` keeps every row in one shared store, `global/<name>.db`, which suits totals and counters. It needs only a `Schema[A]`. With no `EntityPath`, the store looks for an ID column called `id`, so name the ID field `id`.
- `Projection[A](name, entityPath)` is the first one with the `EntityPath[A]` passed explicitly, for example to keep a projection of the same type in another folder.

The `name` is also the file name of the store, so it may contain only letters, digits, `_` and `-`:

```scala

case class UserProfile(@Modifier.id userId: String, name: String)
object UserProfile {
  implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
  implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
}

case class DailyStats(@Modifier.id id: String, userCount: Long)
object DailyStats {
  implicit val schema: Schema[DailyStats] = Schema.derived[DailyStats]
}

val profiles = Projection[UserProfile]("user_profiles")                                       // users/user_profiles.db
val stats    = Projection.global[DailyStats]("daily_stats")                                   // global/daily_stats.db
val archived = Projection[UserProfile]("old_profiles", EntityPath.derived[UserProfile]("archive")) // archive/old_profiles.db
```

## Entity Paths

`EntityPath[A]` describes where a projection's SQLite files live and which field of `A` holds the entity identifier. The ID field is marked with `@Modifier.id`, from `zio.blocks.schema.Modifier`. An entity path is required for per-entity and cross-entity projections; global projections do not use it, see [Projection Scope](#projection-scope).

```scala
trait EntityPath[A] {
  def basePath: String       // folder name, e.g. "users"
  def entityIdField: String  // field name, e.g. "userId"
}
```

The engine uses `basePath` to construct the path to each SQLite file and `entityIdField` to locate the primary-key column in `A`'s schema. Both fields are exposed as read-only members and are set at construction time.

### Manual Construction

`EntityPath.apply[A]` accepts explicit strings for both fields, giving full control over the folder name and the id field without relying on derivation:

```scala

case class MyEntity(@Modifier.id id: String, value: Int)
object MyEntity {
  implicit val schema: Schema[MyEntity]         = Schema.derived[MyEntity]
  implicit val entityPath: EntityPath[MyEntity] = EntityPath[MyEntity]("my_entities", "id")
}
```

### Derived Construction

`EntityPath.derived[A]` inspects `Schema[A]` and applies a four-step algorithm to produce both values automatically:

1. Find the id field: prefer the field annotated `@Modifier.id`, then fall back to any field named `"id"`. If there is neither, `derived` fails with `Entity <type> must have @Modifier.id field or field named 'id'`. A type that is not a case class fails too.
2. Use that field's name as `entityIdField`.
3. Strip the `"Id"` suffix (case-insensitive) from the field name, and a `_` left in front of it, so `user_id` gives `user` as well. A field named exactly `id` is left alone.
4. Convert the result to `snake_case` and pluralize it to obtain `basePath`. The plural follows English for common endings: `box` becomes `boxes` and `category` becomes `categories`.

The table below shows the outcome for three representative types:

| Type                                                            | Id Field   | Derived `basePath` |
|-----------------------------------------------------------------|------------|--------------------|
| `case class UserAccount(@Modifier.id userId: String, ...)`      | `userId`   | `"users"`          |
| `case class OrderRecord(@Modifier.id orderId: String, ...)`     | `orderId`  | `"orders"`         |
| `case class Item(@Modifier.id itemId: String, ...)`             | `itemId`   | `"items"`          |

If the ID field is exactly `id`, `basePath` is `ids`. Prefer a name such as `userId`, or override the folder as shown in the next two subsections.

A concrete derivation looks like this:

```scala

case class UserProfile(@Modifier.id userId: String, name: String, email: String)
object UserProfile {
  implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
  implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
}
// basePath = "users", entityIdField = "userId"
```

### Overriding the Folder Name

`EntityPath.derived[A](pathOverride)` keeps the standard id-field derivation but substitutes an explicit `basePath`, which is useful when the conventional pluralized name conflicts with an existing schema:

```scala
implicit val ep: EntityPath[OrderRecord] = EntityPath.derived[OrderRecord]("custom_orders")
// basePath      = "custom_orders"
// entityIdField = "orderId"   (still derived normally)
```

The path can have several folders, such as `"example2/orders"`.

### The `@path` Annotation

Placing `@path("folder")` on a case class type overrides the derived folder name at the declaration site rather than at each call to `derived`:

```scala

@path("custom_users")
case class UserProfile(@Modifier.id id: String, name: String)
object UserProfile {
  implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
  implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
}
// basePath = "custom_users" (the annotation overrides the derived "ids"), entityIdField = "id"
```

The annotation takes precedence over the derivation algorithm, but not over an explicit `pathOverride` argument — when `pathOverride` is provided to `EntityPath.derived[A](pathOverride)`, it is used unconditionally and the annotation is ignored.

## Sources and Routing

A projection says which event sources it reads with `from`, and how the events of each source are routed with one of the routing methods that follow it.

### Defining Sources

```scala
final class Projection[A] {
  def from(sourceName: String): FromBuilder[A]
}
```

`from` adds a source to the projection: the event store that the engine is given under the name `sourceName`. It returns a `FromBuilder[A]`, on which you call one of the [routing methods](#defining-routes). The `on[E]` handlers after it belong to that source, up to the next `from`, so a projection that reads several sources calls `from` once for each:

```scala

case class RepoCreated(ownerId: String, repoName: String)
object RepoCreated {
  implicit val schema: Schema[RepoCreated] = Schema.derived[RepoCreated]
}

case class IssueOpened(ownerId: String, title: String)
object IssueOpened {
  implicit val schema: Schema[IssueOpened] = Schema.derived[IssueOpened]
}

case class OwnerActivity(@Modifier.id id: String, ownerId: String, kind: String, text: String)
object OwnerActivity {
  implicit val schema: Schema[OwnerActivity]         = Schema.derived[OwnerActivity]
  implicit val entityPath: EntityPath[OwnerActivity] = EntityPath.derived[OwnerActivity]
}

val activity = Projection[OwnerActivity]("ownerActivity")
  .from("repos")                       // first source
  .routedBy[RepoCreated](_.ownerId)
  .on[RepoCreated]
  .insert((e, ctx) => OwnerActivity(ctx.entityId, e.ownerId, "repo", e.repoName))
  .from("issues")                      // second source
  .routedBy[IssueOpened](_.ownerId)
  .on[IssueOpened]
  .insert((e, ctx) => OwnerActivity(ctx.entityId, e.ownerId, "issue", e.title))

// activity.sourceNames == List("repos", "issues")
// activity.scope       == ProjectionScope.CrossEntity(...)
```

The engine then needs an event store for each name, here `"repos"` and `"issues"`. A projection with no `from` reads one source named `"_default"`, see [Default Source](#default-source).

### Defining Routes

After `from`, call one of the routing methods to finish the source binding. Each method returns a `Projection[A]` with the source added, so you can continue the builder chain with `on[E]` handlers or another `from`. The routing methods are:

```scala
final class FromBuilder[A] {
  def routeToSelf: Projection[A]
  def routeToAll: Projection[A]
  def routedBy[E](selector: E => String): Projection[A]
  def routedByField[E](selector: E => Any): Projection[A]   // inline macro
}
```

Each routing method finishes the `from` call and returns the projection. Only `routedBy` and `routedByField` choose where events go: they split the projection's rows by a key read from the event. `routeToSelf` and `routeToAll` choose no key: in a projection without `routedBy`, every event goes to the projection's one store and changes the row with the ID it was recorded under.

#### `routeToSelf`

`routeToSelf` is for a projection that keeps one row for each entity: each event changes the row of the entity it was recorded under. It is what a projection with no `from` uses:

```scala
Projection[UserProfile]("userProfiles")
  .from("users").routeToSelf
  .on[UserCreated].insert((e, ctx) => UserProfile(ctx.entityId, e.name, e.email))
```

#### `routeToAll`

`routeToAll` is for a source whose events all feed shared rows, typically in a projection made with `Projection.global[A]`. It does not make the projection global, which only `Projection.global` does, and it does not merge rows: to count everything in one row, record the events under one ID, such as a date:

```scala
Projection.global[DailyStats]("dailyStats")
  .from("users").routeToAll
  .on[UserCreated].aggregate(FieldUpdate.Increment("user_count", 1L))
```

#### `routedBy`

`routedBy[E]` reads a key from each event with `selector`. Events with the same key go to the same store, called a shard, `<basePath>/<name>_shard_<key>.db`, even when they were recorded under different entities. Within a shard, the row is still the one with the event's entity ID. Using it on any source makes the projection [`CrossEntity`](#projection-scope):

```scala
Projection[CustomerOrder]("customer_orders")
  .from("orders").routedBy[OrderEvent](_.customerId)
  .on[OrderPlaced].insert((e, ctx) => CustomerOrder(ctx.entityId, e.customerId, e.product))
```

#### `routedByField`

`routedByField[E]` is `routedBy` for a selector that returns any type, such as an `Int` or a `UUID`: the value is turned into the key with `toString`. `routedByField[E](_.customerId)` is the same as `routedBy[E](e => e.customerId.toString)`:

```scala
Projection[CustomerOrder]("customer_orders")
  .from("orders").routedByField[OrderEvent](_.customerId)
  .on[OrderPlaced].insert((e, ctx) => CustomerOrder(ctx.entityId, e.customerId, e.product))
```

The full example below reads two sources: it routes the orders by customer with `routedBy`, and feeds a global row of shop totals with `routeToAll`:

```scala title="projection-examples/src/main/scala/projection/examples/Example3CrossEntity.scala"
/*
 * Example 3 of 13: a cross-entity projection.
 *
 * Read Example2PerEntity first. Next: Example4InsertAndDelete.
 *
 * In Example 2 every event belongs to the entity it was recorded under. Here the
 * events are recorded per order (`order-1`, `order-2`, ...), but they are viewed
 * per customer, and the customer is a field inside the event. `routedBy` tells the
 * projection to read that key from each event and send the event to the store for
 * that key, called a shard. So one customer's orders are kept together, apart from
 * everyone else's.
 *
 * Two kinds of event are involved, `OrderPlaced` and `OrderShipped`. Both carry the
 * `customerId`, so both are routed to the same shard. That is what lets
 * `OrderShipped` update a row that `OrderPlaced` created earlier: the row has the
 * ID of the order, and it is in the shard of its customer.
 *
 * A second projection, `ShopStats`, shows the other side: one global row with the
 * numbers for the whole shop (customers, orders, shipped, pending, revenue). It
 * is a global projection, so it ignores the customer and counts everything. Its
 * counters update the row whose ID the event was recorded under, and the orders
 * are recorded under their own IDs, so each event is also written to a second
 * feed, "shop", under the single ID "shop". That feed is what the stats read.
 *
 * This example runs on real SQLite files: the events are kept in
 * `.projection-examples/example3/events/orders.db`, and the projection in
 * `.projection-examples/example3/orders/`, where each customer gets their own file next to
 * the main one, for example `customer_orders_shard_alice.db`.
 *
 * It shows:
 *   - `routedBy`, which makes the projection's scope `CrossEntity`
 *   - the routing key chooses the shard, while the row keeps the ID of the order
 *   - several event types converging on the same projection
 *   - an update landing on the right row, because routing is the same for every
 *     event of a customer
 *   - an engine on SQLite files, and the shard files that appear
 *   - reading a row back by ID, which looks in the main store and then in the
 *     shards. The engine has no call that lists the rows of a shard
 *   - a global projection with several counters, some of them going up and down
 *     (`pending`), fed by three kinds of event
 *
 * Run it from the repository root with:
 *   sbt "projection-examples/runMain projection.examples.Example3CrossEntity"
 */

package projection.examples

object Example3CrossEntity extends ZIOAppDefault {

  sealed trait OrderEvent { def customerId: String }
  object OrderEvent       {
    case class OrderPlaced(customerId: String, product: String, total: Long) extends OrderEvent
    object OrderPlaced {
      implicit val schema: Schema[OrderPlaced] = Schema.derived[OrderPlaced]
    }
    case class OrderShipped(customerId: String) extends OrderEvent
    object OrderShipped {
      implicit val schema: Schema[OrderShipped] = Schema.derived[OrderShipped]
    }
    case class CustomerRegistered(customerId: String, name: String) extends OrderEvent
    object CustomerRegistered {
      implicit val schema: Schema[CustomerRegistered] = Schema.derived[CustomerRegistered]
    }
    implicit val schema: Schema[OrderEvent] = Schema.derived[OrderEvent]
  }

  private val ordersFolder: String = Sqlite.path("example3/orders")

  // Not called `Order`: the table is named after the class, and `order` is an SQL keyword.
  case class CustomerOrder(@Modifier.id id: String, customerId: String, product: String, total: Long, status: String)
  object CustomerOrder {
    implicit val schema: Schema[CustomerOrder] = Schema.derived[CustomerOrder]
    // The folder is chosen explicitly. Derived from the `id` field, it would be "ids".
    implicit val entityPath: EntityPath[CustomerOrder] = EntityPath.derived[CustomerOrder](ordersFolder)
  }

  // The key is always "shop". Its field is called `id` because a global projection has no `EntityPath`, so its
  // SQLite store looks for a column with that name.
  case class ShopStats(
    @Modifier.id id: String,
    customers: Long,
    orders: Long,
    shipped: Long,
    pending: Long,
    revenue: Long
  )
  object ShopStats {
    implicit val schema: Schema[ShopStats]         = Schema.derived[ShopStats]
    implicit val entityPath: EntityPath[ShopStats] = EntityPath.derived[ShopStats]
  }

  private def show(order: Option[CustomerOrder]): String =
    order.map(o => s"${o.customerId}, ${o.product}, ${o.total}, ${o.status}").getOrElse("not found")

  // `CrossEntity` holds the routing function, which prints as an unreadable lambda, so only the kind is printed.
  private def scopeName(scope: ProjectionScope): String = scope match {
    case ProjectionScope.PerEntity      => "PerEntity"
    case ProjectionScope.CrossEntity(_) => "CrossEntity"
    case ProjectionScope.Global         => "Global"
  }

  def demo: ZIO[Any, Throwable, Unit] =
    Console.printLine("\n--- Example 3: CrossEntity Projection (orders routed by customerId) ---").orDie *>
      ZIO.scoped {
        for {
          _ <- Sqlite.globalFolder("shop_stats") *> Sqlite.folders(ordersFolder, Sqlite.path("example3/events"))

          projection = Projection[CustomerOrder]("customer_orders")
                         .from("orders")
                         .routedBy[OrderEvent](_.customerId)
                         .on[OrderPlaced]
                         .insert((e, ctx) => CustomerOrder(ctx.entityId, e.customerId, e.product, e.total, "placed"))
                         .on[OrderShipped]
                         .updateWithField("status", (_, _) => "shipped")

          // One row for the whole shop. `pending` goes up when an order is placed and down when it is shipped.
          stats = Projection
                    .global[ShopStats]("shop_stats")
                    .from("shop")
                    .routeToAll
                    .on[CustomerRegistered]
                    .aggregate(FieldUpdate.Increment("customers", 1L))
                    .on[OrderPlaced]
                    .custom { (e, _) =>
                      ProjectionAction.Update(
                        Chunk(
                          FieldUpdate.Increment("orders", 1L),
                          FieldUpdate.Increment("pending", 1L),
                          FieldUpdate.Increment("revenue", e.total)
                        )
                      )
                    }
                    .on[OrderShipped]
                    .aggregate(Chunk(FieldUpdate.Increment("shipped", 1L), FieldUpdate.Decrement("pending", 1L)))

          _ <-
            Console.printLine(s"  Scope: ${scopeName(projection.scope)}, stats scope: ${scopeName(stats.scope)}").orDie

          cache  <- TransactorCache.make()
          orders <- Sqlite.eventStore[OrderEvent](cache, Sqlite.path("example3/events/orders.db"))
          shop   <- Sqlite.eventStore[OrderEvent](cache, Sqlite.path("example3/events/shop.db"))
          engine <- ProjectionEngine.makeWithStoresVarargs(
                      cache,
                      Map[String, EventStore[?]]("orders" -> orders, "shop" -> shop)
                    )(projection, stats)
          _ <- engine.start

          // Order events are recorded under the ID of the order, and carry the customer inside. Each one is also
          // written to the "shop" feed under the single ID "shop", which is what the stats read.
          record = (orderId: String, event: OrderEvent) => orders.append(orderId, event) *> shop.append("shop", event)

          _ <- ZIO.foreachDiscard(List("alice", "bob", "charlie", "dave")) { customer =>
                 shop.append("shop", CustomerRegistered(customer, customer.capitalize))
               }
          _ <- record("order-1", OrderPlaced("alice", "keyboard", 80))
          _ <- record("order-2", OrderPlaced("alice", "monitor", 250))
          _ <- record("order-3", OrderPlaced("bob", "mouse", 25))
          _ <- record("order-1", OrderShipped("alice"))
          _ <- record("order-3", OrderShipped("bob"))
          _ <- record("order-4", OrderPlaced("charlie", "cable", 9))

          // Events are applied in order, so once order-4 exists every earlier event has been applied.
          order4 <- pollUntil(engine.query(projection, "order-4"))
          order1 <- engine.query(projection, "order-1")
          order2 <- engine.query(projection, "order-2")
          order3 <- engine.query(projection, "order-3")

          // The stats are done when all four customers, four orders and two shipments have been counted.
          totals <- pollUntil(
                      engine
                        .query(stats, "shop")
                        .map(_.filter(t => t.customers == 4 && t.orders == 4 && t.shipped == 2)),
                      5.seconds
                    )

          files <- Sqlite.dbFiles(ordersFolder)
          _     <- Console.printLine(s"  Files in $ordersFolder/: ${files.mkString(", ")}").orDie
          _     <- Console.printLine(s"  order-1: ${show(order1)}").orDie
          _     <- Console.printLine(s"  order-2: ${show(order2)}").orDie
          _     <- Console.printLine(s"  order-3: ${show(order3)}").orDie
          _     <- Console.printLine(s"  order-4: ${show(order4)}").orDie
          _     <- totals match {
                 case Some(t) =>
                   Console
                     .printLine(
                       s"  Stats: customers ${t.customers}, orders ${t.orders}, shipped ${t.shipped}, " +
                         s"pending ${t.pending}, revenue ${t.revenue}"
                     )
                     .orDie
                 case None =>
                   Console.printLine("  ⚠ Stats not yet ready (timeout)").orDie
               }
          _ <- Console.printLine("  ✓ CrossEntity: orders kept per customer in a shard, one global row of stats").orDie
        } yield ()
      }

  def run: ZIO[Any, Throwable, Unit] = demo
}
```

## Defining Handlers

`on[E]` opens a handler for events of type `E` and returns a `HandlerBuilder[E, A]`. Each handler verb on the builder closes it and returns a new `Projection[A]` with the handler added to the most recent source binding:

```scala
final class Projection[A] {
  def on[E: Schema: ClassTag]: HandlerBuilder[E, A]
}

final class HandlerBuilder[E, A] {
  def insert(f: (E, ProjectionContext) => A): Projection[A]
  def delete: Projection[A]
  def custom(f: (E, ProjectionContext) => ProjectionAction[A]): Projection[A]
  def aggregate(update: FieldUpdate): Projection[A]
  def aggregate(updates: Chunk[FieldUpdate]): Projection[A]
  def aggregateField(update: FieldUpdate): Projection[A]
  def updateWithField(fieldName: String, f: (E, ProjectionContext) => Any): Projection[A]
  def updateField(fieldName: String)(f: (E, ProjectionContext) => Any): Projection[A]
  def update(selector: A => Any)(f: (E, ProjectionContext) => Any): Projection[A] // inline macro
}
```

When an event arrives, the projection looks for a handler by the event's class. A handler matches any event that is an instance of its event type, so a handler for a sealed trait also matches each of its cases. If more than one handler matches, the first one added wins. The engine searches the handlers of all the sources together, in the order the sources were added, and it does not narrow the search to the source the event came from. So when two sources have a handler for the same event type, the handler of the source added first handles the events of both.

Handlers describe changes as `ProjectionAction` values and cannot perform effects. The engine applies each returned action to the projection store; the handler only decides which action to produce. This separation makes event replay deterministic: running the same sequence of events through the same handlers always yields the same read-model state.

### ProjectionContext

Most handler functions receive a `ProjectionContext` alongside the event, providing the metadata the engine attached when it processed the event envelope:

```scala
final case class ProjectionContext(
  entityId:       String,
  timestamp:      java.time.Instant,
  seq:            Long,
  sourceEntityId: Option[String]
)
```

`entityId` is the entity the event was recorded under and determines the routing target for `RouteToSelf`. `timestamp` is the wall-clock time at which the event was appended. `seq` is the monotonic sequence number that the engine uses as a resume watermark after a restart. The engine always sets `sourceEntityId` to `Some(entityId)`, the event's own entity ID, also in a cross-entity projection.

### Inserting and Deleting Rows

`insert` builds a complete row from the event and context and produces `ProjectionAction.Insert`, creating a new entry in the projection store:

```scala
Projection[UserProfile]("userProfiles")
  .from("users").routeToSelf
  .on[UserCreated]
  .insert((e, ctx) => UserProfile(ctx.entityId, e.name, e.email))
  .on[UserDeleted]
  .delete
```

`delete` removes the entity's row unconditionally, producing `ProjectionAction.Delete`. It takes no function argument because the target row is always identified by `ctx.entityId`.

The full example below creates three users, deletes one of them, and reads the rows back. `UserCreated` inserts a `UserProfile` and `UserDeleted` removes it:

```scala title="projection-examples/src/main/scala/projection/examples/Example4InsertAndDelete.scala"
/*
 * Example 4 of 13: inserting and deleting rows.
 *
 * Builds on Example2PerEntity. Read it first. Next: Example5FieldUpdates.
 *
 * Two handlers add and remove whole rows:
 *   - `insert` builds a complete row from the event and the `ProjectionContext`
 *     and adds it to the store
 *   - `delete` removes the row of the entity the event was recorded under. It
 *     takes no function, because the row is always the one with `ctx.entityId`
 *
 * Here `UserCreated` inserts a `UserProfile` and `UserDeleted` removes it, so a
 * user's row exists from the moment the user is created until the user is
 * deleted.
 *
 * It shows:
 *   - `insert`, which builds the row from the event and `ctx.entityId`
 *   - `delete`, which removes the row for the entity of the event
 *   - that a deleted row is gone: querying it returns `None`
 *   - running it in memory with `TestEngine`
 *
 * Run it from the repository root with:
 *   sbt "projection-examples/runMain projection.examples.Example4InsertAndDelete"
 */

package projection.examples

object Example4InsertAndDelete extends ZIOAppDefault {

  case class UserDeleted()
  object UserDeleted {
    implicit val schema: Schema[UserDeleted] = Schema.derived[UserDeleted]
  }

  case class UserProfile(@Modifier.id id: String, name: String, email: String)
  object UserProfile {
    implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
    implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
  }

  private def show(profile: Option[UserProfile]): String =
    profile.map(p => s"${p.name} <${p.email}>").getOrElse("no row")

  def demo: ZIO[Any, Throwable, Unit] =
    Console.printLine("\n--- Example 4: Insert and Delete (UserProfile rows) ---").orDie *>
      ZIO.scoped {
        for {
          projection = Projection[UserProfile]("user_profiles")
                         .from("users")
                         .routeToSelf
                         .on[UserCreated]
                         .insert((e, ctx) => UserProfile(ctx.entityId, e.name, e.email))
                         .on[UserDeleted]
                         .delete

          engine <- TestEngine.make(projection)

          _ <- engine.append("user-1", UserCreated("Alice", "alice@example.com"))
          _ <- engine.append("user-2", UserCreated("Bob", "bob@example.com"))
          _ <- engine.append("user-3", UserCreated("Charlie", "charlie@example.com"))
          _ <- engine.append("user-2", UserDeleted())

          // A deleted row cannot be waited for, so the last event is a new user. Events are applied in order, so
          // once user-4 exists the deletion has been applied.
          _ <- engine.append("user-4", UserCreated("Dave", "dave@example.com"))
          _ <- pollUntil(engine.query(projection, "user-4"))

          u1 <- engine.query(projection, "user-1")
          u2 <- engine.query(projection, "user-2")
          u3 <- engine.query(projection, "user-3")

          _ <- Console.printLine(s"  user-1: ${show(u1)}").orDie
          _ <- Console.printLine(s"  user-2: ${show(u2)}").orDie
          _ <- Console.printLine(s"  user-3: ${show(u3)}").orDie
          _ <- Console.printLine("  ✓ Insert and delete: user-2 was inserted, then its row was removed").orDie
        } yield ()
      }

  def run: ZIO[Any, Throwable, Unit] = demo
}
```

Running it prints one line for each user:

- `user-1`: `Alice <alice@example.com>`. The row was inserted and never deleted.
- `user-2`: `no row`. The row was inserted, then removed by `delete`, so the query returns `None`.
- `user-3`: `Charlie <charlie@example.com>`. The deletion of `user-2` did not touch it, because each `delete` removes only the row of the entity the event was recorded under.

### Field Updates

`updateWithField` (and its alias `updateField`) sets a named column to a value computed from the event. The column name is passed as a raw string and must already be in `snake_case` to match the generated schema:

```scala
  .on[OrderShipped]
  .updateWithField("status", (_, _) => "shipped")
```

`update` is the compile-time macro variant. A selector such as `_.firstName` is resolved to the column name at compile time, and the macro converts the Scala field name to `snake_case` automatically — a mistyped field name is a compile error rather than a runtime failure:

```scala
  .on[UserRenamed]
  .update(_.firstName)((e, _) => e.newName)
  // _.firstName resolved to column "first_name" at compile time
```

The full example below registers a user and then changes one column at a time, using each of the three ways to name the column. `UserRenamed` uses the `update` macro, `EmailChanged` uses `updateWithField` with the column name as a string, and `UserVerified` uses `updateField`. Each handler changes only its own column:

```scala title="projection-examples/src/main/scala/projection/examples/Example5FieldUpdates.scala"
/*
 * Example 5 of 13: field updates.
 *
 * Builds on Example2PerEntity. Read it first. Next: Example6GlobalAggregate.
 *
 * `insert` writes a whole row. When an event changes only one column, a field
 * update sets that column and leaves the rest of the row as it is. There are
 * three ways to name the column:
 *   - `update(_.firstName)`, which picks the field with a selector. The name is
 *     checked by the compiler, and the macro turns it into the column name
 *     `first_name`, so a typo does not compile
 *   - `updateWithField("email", ...)`, which takes the column name as a string.
 *     It must already be the column name, so `snake_case`, and it is not checked
 *     until the event is applied
 *   - `updateField("status")(...)`, which is the same as `updateWithField` with
 *     the function in a second parameter list
 *
 * Each handler computes the new value from the event, and may also use the
 * `ProjectionContext`.
 *
 * It shows:
 *   - the three ways to update one column
 *   - an update changing one column and keeping the others
 *   - running it in memory with `TestEngine`
 *
 * Run it from the repository root with:
 *   sbt "projection-examples/runMain projection.examples.Example5FieldUpdates"
 */

package projection.examples

object Example5FieldUpdates extends ZIOAppDefault {

  case class UserRegistered(firstName: String, email: String)
  object UserRegistered {
    implicit val schema: Schema[UserRegistered] = Schema.derived[UserRegistered]
  }

  case class UserRenamed(newName: String)
  object UserRenamed {
    implicit val schema: Schema[UserRenamed] = Schema.derived[UserRenamed]
  }

  case class EmailChanged(newEmail: String)
  object EmailChanged {
    implicit val schema: Schema[EmailChanged] = Schema.derived[EmailChanged]
  }

  case class UserVerified()
  object UserVerified {
    implicit val schema: Schema[UserVerified] = Schema.derived[UserVerified]
  }

  case class UserProfile(@Modifier.id id: String, firstName: String, email: String, status: String)
  object UserProfile {
    implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
    implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
  }

  private def show(profile: Option[UserProfile]): String =
    profile.map(p => s"${p.firstName} <${p.email}>, ${p.status}").getOrElse("not found")

  def demo: ZIO[Any, Throwable, Unit] =
    Console.printLine("\n--- Example 5: Field Updates (update, updateWithField, updateField) ---").orDie *>
      ZIO.scoped {
        for {
          projection = Projection[UserProfile]("user_profiles")
                         .from("users")
                         .routeToSelf
                         .on[UserRegistered]
                         .insert((e, ctx) => UserProfile(ctx.entityId, e.firstName, e.email, "pending"))
                         .on[UserRenamed]
                         .update(_.firstName)((e, _) => e.newName)
                         .on[EmailChanged]
                         .updateWithField("email", (e, _) => e.newEmail)
                         .on[UserVerified]
                         .updateField("status")((_, _) => "verified")

          engine <- TestEngine.make(projection)

          _ <- engine.append("user-1", UserRegistered("Alice", "alice@example.com"))
          _ <- engine.append("user-2", UserRegistered("Bob", "bob@example.com"))
          _ <- engine.append("user-1", UserRenamed("Alicia"))
          _ <- engine.append("user-1", EmailChanged("alicia@example.com"))
          _ <- engine.append("user-1", UserVerified())
          _ <- engine.append("user-2", UserRenamed("Robert"))

          // Events are applied in order, so once the last one is visible every earlier event has been applied.
          u2 <- pollUntil(engine.query(projection, "user-2").map(_.filter(_.firstName == "Robert")))
          u1 <- engine.query(projection, "user-1")

          _ <- Console.printLine(s"  user-1: ${show(u1)}").orDie
          _ <- Console.printLine(s"  user-2: ${show(u2)}").orDie
          _ <- Console.printLine("  ✓ Field updates: each event changed one column and kept the others").orDie
        } yield ()
      }

  def run: ZIO[Any, Throwable, Unit] = demo
}
```

Running it prints one line for each user:

- `user-1`: `Alicia <alicia@example.com>, verified`. The name, the email and the status were each changed by a separate event.
- `user-2`: `Robert <bob@example.com>, pending`. Only the name changed, so the email and the status are as they were registered.

### Aggregating Fields

`aggregate` and `aggregateField` apply one or more `FieldUpdate` operations, producing `ProjectionAction.Update`. This rewrites individual columns rather than the entire row, and numeric operations such as `Increment` are executed in SQL without reading the row first. The available `FieldUpdate` operations are `Set`, `Increment`, `Decrement`, `Max`, and `Min`. `aggregateField` is an alias for the single-update overload of `aggregate`:

```scala
Projection.global[DailyStats]("dailyStats")
  .from("users").routeToAll
  .on[UserCreated]
  .aggregate(FieldUpdate.Increment("user_count", 1L))
  .from("repos").routeToAll
  .on[RepoCreated]
  .aggregate(Chunk(
    FieldUpdate.Increment("repo_count", 1L),
    FieldUpdate.Max("peak", 100L)
  ))
```

To change several columns for one event, pass a `Chunk` of updates, as the `RepoCreated` handler above does. The SQLite store applies them in one transaction.

The `FieldUpdate` values are fixed when the projection is declared, so they cannot use data from the event. When an update needs the event's data, for example adding an order's total to `revenue`, build the update inside [`custom`](#custom-actions) with `ProjectionAction.Update`.

Each operation is one change to a column, run as SQL:

| Operation | SQL | Effect |
| --- | --- | --- |
| `Set(field, value)` | `SET col = ?` | Replace the value |
| `Increment(field, by)` | `SET col = COALESCE(col, 0) + ?` | Add to a counter |
| `Decrement(field, by)` | `SET col = COALESCE(col, 0) - ?` | Subtract from a counter |
| `Max(field, value)` | `SET col = MAX(COALESCE(col, ?), ?)` | Keep the highest value |
| `Min(field, value)` | `SET col = MIN(COALESCE(col, ?), ?)` | Keep the lowest value |

Before it applies the updates, the store runs `INSERT OR IGNORE`, which creates the row with the default value of each column if it is missing. Incrementing a counter of a row that does not exist yet therefore gives `1`, and `COALESCE` treats a missing value as `0`. The updates of one event run in one transaction, and the arithmetic happens inside SQL, so several fibers incrementing the same counter produce the correct total without lost updates.

A field can also be named with a selector instead of a string. `FieldUpdate.increment[DailyStats](_.userCount)` picks the field at compile time and maps it to the column `user_count`, as `update` does, so a typo does not compile. The factories are `setValue`, `increment`, `decrement`, `maxValue` and `minValue`.

The full example below counts users and repositories into one `DailyStats` row. Each source is bound with `routeToAll`, so the events of both sources land in the same row, and each handler adds one to its own counter with `FieldUpdate.Increment`. The row is created on the first event, with the other counters at zero:

```scala title="projection-examples/src/main/scala/projection/examples/Example6GlobalAggregate.scala"
/*
 * Example 6 of 13: a global aggregate projection.
 *
 * Read Examples 2 and 3 first. Next: Example7CustomActions.
 *
 * A global projection keeps one shared store for the whole projection, which is
 * the place for totals and counters. Here `UserCreated` events from the "users"
 * source and `RepoCreated` events from the "repos" source both feed a single
 * `DailyStats` row.
 *
 * This example runs on real SQLite files: the events are kept in
 * `.projection-examples/example6/events/users.db` and `.projection-examples/example6/events/repos.db`,
 * and the projection in `global/dailyStats.db`. The engine always keeps the store
 * of a global projection in a folder called `global`, so it is not inside
 * `.projection-examples`.
 *
 * It shows:
 *   - `Projection.global`, with `routeToAll` on each source
 *   - `aggregate` with `FieldUpdate.Increment`, which adds to a counter in SQL
 *     without reading the row first
 *   - one projection listening to more than one source, each with its own
 *     event store
 *
 * Run it from the repository root with:
 *   sbt "projection-examples/runMain projection.examples.Example6GlobalAggregate"
 */

package projection.examples

object Example6GlobalAggregate extends ZIOAppDefault {

  // The key is the date. Its field is called `id` because a global projection has no `EntityPath`, so its
  // SQLite store looks for a column with that name.
  case class DailyStats(@Modifier.id id: String, userCount: Int, repoCount: Int)
  object DailyStats {
    implicit val schema: Schema[DailyStats]         = Schema.derived[DailyStats]
    implicit val entityPath: EntityPath[DailyStats] = EntityPath.derived[DailyStats]
  }

  def demo: ZIO[Any, Throwable, Unit] =
    Console.printLine("\n--- Example 6: Global Aggregate (DailyStats from users + repos) ---").orDie *>
      ZIO.scoped {
        for {
          _ <- Sqlite.globalFolder("dailyStats") *> Sqlite.folders(Sqlite.path("example6/events"))

          dateKey = "2025-08-26"

          projection = Projection
                         .global[DailyStats]("dailyStats")
                         .from("users")
                         .routeToAll
                         .on[UserCreated]
                         .aggregate(FieldUpdate.Increment("user_count", 1L))
                         .from("repos")
                         .routeToAll
                         .on[RepoCreated]
                         .aggregate(FieldUpdate.Increment("repo_count", 1L))

          _ <- Console.printLine(s"  Scope: ${projection.scope}").orDie

          cache  <- TransactorCache.make()
          users  <- Sqlite.eventStore[UserCreated](cache, Sqlite.path("example6/events/users.db"))
          repos  <- Sqlite.eventStore[RepoCreated](cache, Sqlite.path("example6/events/repos.db"))
          engine <- ProjectionEngine.makeWithStoresVarargs(
                      cache,
                      Map[String, EventStore[?]]("users" -> users, "repos" -> repos)
                    )(projection)
          _ <- engine.start

          _ <- ZIO.foreachDiscard(1 to 5)(i => users.append(dateKey, UserCreated(s"user-$i", s"u$i@ex.com")))
          _ <- ZIO.foreachDiscard(1 to 3)(i => repos.append(dateKey, RepoCreated("any", s"repo-$i")))

          stats <- pollUntil(
                     engine.query(projection, dateKey).map(_.filter(s => s.userCount == 5 && s.repoCount == 3)),
                     5.seconds
                   )

          files <- Sqlite.dbFiles("global")
          _     <- Console.printLine(s"  Files in global/: ${files.mkString(", ")}").orDie
          _     <- stats match {
                 case Some(s) =>
                   Console.printLine(s"  Date: $dateKey, Users: ${s.userCount}, Repos: ${s.repoCount}").orDie
                 case None =>
                   Console.printLine("  ⚠ Aggregate not yet ready (timeout)").orDie
               }
          _ <- Console.printLine("  ✓ Global: events from multiple sources converge to single aggregate row").orDie
        } yield ()
      }

  def run: ZIO[Any, Throwable, Unit] = demo
}
```

Running it appends five `UserCreated` and three `RepoCreated` events and reads back one row with `Users: 5, Repos: 3`.

### Custom Actions

`custom` accepts any `ProjectionAction[A]`, covering cases the other verbs do not — for example, `Upsert` (insert or update), `Noop` (no operation), or `Truncate` (clear all rows). Its signature is `custom(f: (E, ProjectionContext) => ProjectionAction[A]): Projection[A]`. Use it when the action depends on the event: the handler looks at the event and returns the action to apply.

The example below handles two events for a user profile, and each handler picks between actions:

- `ProfileImported` carries a `draft` flag. A draft is ignored with `Noop`, so no row is written. Anything else is written with `Upsert`, which creates the row or replaces it if it exists. That makes importing the same user twice safe, where `Insert` would not be.
- `AccountClosed` carries an `eraseData` flag. When it is set, the row is removed with `Delete`. Otherwise `Update` changes only the `status` column and keeps the rest of the row.

```scala title="projection-examples/src/main/scala/projection/examples/Example7CustomActions.scala"
/*
 * Example 7 of 13: custom actions.
 *
 * Builds on Example2PerEntity. Read it first. Next: Example8SchemaEvolution.
 *
 * `insert`, `delete` and the field updates each produce one fixed action. When
 * what should happen depends on the event, `custom` lets the handler look at the
 * event and return the action itself. This example uses four of the actions a
 * handler can return: `Upsert`, `Noop`, `Delete` and `Update`.
 *
 *   - `ProfileImported` has a `draft` flag. A draft is ignored with `Noop`, so no
 *     row is written. Anything else is written with `Upsert`, which creates the
 *     row or replaces it if it exists, so importing the same user twice is safe.
 *   - `AccountClosed` has an `eraseData` flag. When it is set, the row is removed
 *     with `Delete`. Otherwise `Update` changes only the `status` column and keeps
 *     the rest of the row.
 *
 * It shows:
 *   - `custom`, which returns a `ProjectionAction` chosen from the event
 *   - `Upsert` replacing a row, `Noop` writing nothing, `Delete` removing a row
 *   - `Update` with a `FieldUpdate.Set`, changing one column of an existing row
 *   - running it in memory with `TestEngine`
 *
 * Run it from the repository root with:
 *   sbt "projection-examples/runMain projection.examples.Example7CustomActions"
 */

package projection.examples

object Example7CustomActions extends ZIOAppDefault {

  case class ProfileImported(name: String, email: String, draft: Boolean)
  object ProfileImported {
    implicit val schema: Schema[ProfileImported] = Schema.derived[ProfileImported]
  }

  case class AccountClosed(eraseData: Boolean)
  object AccountClosed {
    implicit val schema: Schema[AccountClosed] = Schema.derived[AccountClosed]
  }

  case class UserProfile(@Modifier.id id: String, name: String, email: String, status: String)
  object UserProfile {
    implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
    implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
  }

  private def show(profile: Option[UserProfile]): String =
    profile.map(p => s"${p.name} <${p.email}>, ${p.status}").getOrElse("no row")

  def demo: ZIO[Any, Throwable, Unit] =
    Console.printLine("\n--- Example 7: Custom Actions (Upsert, Noop, Delete, Update) ---").orDie *>
      ZIO.scoped {
        for {
          projection = Projection[UserProfile]("user_profiles")
                         .from("users")
                         .routeToSelf
                         .on[ProfileImported]
                         .custom { (e, ctx) =>
                           if (e.draft) ProjectionAction.Noop
                           else ProjectionAction.Upsert(UserProfile(ctx.entityId, e.name, e.email, "active"))
                         }
                         .on[AccountClosed]
                         .custom { (e, _) =>
                           if (e.eraseData) ProjectionAction.Delete
                           else ProjectionAction.Update(Chunk(FieldUpdate.Set("status", "closed")))
                         }

          engine <- TestEngine.make(projection)

          _ <- engine.append("user-1", ProfileImported("Alice", "alice@example.com", draft = false))
          _ <- engine.append("user-1", ProfileImported("Alice B.", "alice@example.com", draft = false))
          _ <- engine.append("user-2", ProfileImported("Bob", "bob@example.com", draft = true))
          _ <- engine.append("user-3", ProfileImported("Carol", "carol@example.com", draft = false))
          _ <- engine.append("user-3", AccountClosed(eraseData = false))
          _ <- engine.append("user-4", ProfileImported("Dave", "dave@example.com", draft = false))
          _ <- engine.append("user-4", AccountClosed(eraseData = true))
          // Two of the users end up without a row, so there is nothing to wait for on them. Events are applied in
          // order, so once this last user exists every earlier event has been applied.
          _ <- engine.append("user-5", ProfileImported("Erin", "erin@example.com", draft = false))

          _  <- pollUntil(engine.query(projection, "user-5"))
          u1 <- engine.query(projection, "user-1")
          u2 <- engine.query(projection, "user-2")
          u3 <- engine.query(projection, "user-3")
          u4 <- engine.query(projection, "user-4")

          _ <- Console.printLine(s"  user-1: ${show(u1)}").orDie
          _ <- Console.printLine(s"  user-2: ${show(u2)}").orDie
          _ <- Console.printLine(s"  user-3: ${show(u3)}").orDie
          _ <- Console.printLine(s"  user-4: ${show(u4)}").orDie
          _ <- Console.printLine("  ✓ Custom: the handler chose Upsert, Noop, Update or Delete for each event").orDie
        } yield ()
      }

  def run: ZIO[Any, Throwable, Unit] = demo
}
```

Running it prints one line for each user:

- `user-1`: `Alice B. <alice@example.com>, active`. The second import replaced the first.
- `user-2`: `no row`. The draft was a `Noop`, so no row was ever written.
- `user-3`: `Carol <carol@example.com>, closed`. Only `status` changed.
- `user-4`: `no row`. The row was created, then removed by `Delete`.

### Default Source

Most projections read one event feed and keep each entity's row under its own ID. Writing `.from("some-name").routeToSelf` for that would mean inventing a name for a feed that has only one meaning, so the library lets you skip it. When you add the first handler to a projection that has no source binding yet, with `on[E]` and no `from()` before it, the library creates the binding for you:

- The source is named `"_default"`.
- The routing is `RouteToSelf`, so each event goes to the row of the entity it was recorded under (`ctx.entityId`).
- The handler is put in that binding.

```scala
Projection[UserProfile]("userProfiles")
  .on[UserCreated]   // implicitly creates "_default" binding with RouteToSelf
  .insert((e, ctx) => UserProfile(ctx.entityId, e.name, e.email))
```

This is the same projection as the one below, with the binding written out:

```scala
Projection[UserProfile]("userProfiles")
  .from("_default").routeToSelf
  .on[UserCreated]
  .insert((e, ctx) => UserProfile(ctx.entityId, e.name, e.email))
```

The shortcut is purely a convenience. It gives nothing that `.from("_default").routeToSelf` does not, and nothing requires you to use it.

The engine looks each source name up in the event stores it was given. If no store has that name, it uses the store under `"_default"`, and if there is none, the first store in the map, so a misspelt name reads another store instead of failing. With a single event store, every source reads that store.

What the shortcut does and does not do:
- Only the first handler creates the binding. A second `on[E]` attaches to the binding that already exists, whether it is `"_default"` or one made by an earlier `from()`.
- A `from()` after the handlers adds a second binding. The projection then has two sources, `"_default"` and the new one, and needs an event store for each.
- It can only give `RouteToSelf`. For any other routing, write `from()` yourself.

## Projection Scope

A scope is the way the rows of a projection are organized. To see what that means, picture a shop that writes a note every time something happens: "Alice placed order 7", "order 7 shipped". These notes are the events. To answer questions quickly ("is order 7 shipped?", "how many orders so far?"), someone files each note into a cabinet as it arrives. The filed result is the read model, and the scope is how that cabinet is arranged.

`ProjectionScope` is the type that describes it. It answers two questions: which `ProjectionStore` an event is applied to, and which file that store lives in.

```scala
sealed trait ProjectionScope
object ProjectionScope {
  case object PerEntity                              extends ProjectionScope
  case object Global                                 extends ProjectionScope
  case class  CrossEntity(routedBy: Any => String)   extends ProjectionScope
}
```

You never set a scope directly. It is worked out from how the projection is built, and `projection.scope` returns it:

1. A projection made with `Projection.global` has the scope `Global`.
2. Otherwise, if any source uses `routedBy` or `routedByField`, it has the scope `CrossEntity`. The key function of the first such source is kept in the `CrossEntity` case.
3. Otherwise it has the scope `PerEntity`, which is the default.

Note that `routeToAll` alone does not make a projection global. It only sends the events of its source to the same store. Only `Projection.global` gives the scope `Global` and puts the store in the `global/` folder.

The three scopes keep their files in different places:

| Scope                   | File Location                              |
|-------------------------|--------------------------------------------|
| `PerEntity` (default)   | `<basePath>/<name>.db`                     |
| `CrossEntity(routedBy)` | `<basePath>/<name>_shard_<key>.db` per key |
| `Global`                | `global/<name>.db`                         |

```scala

case class UserRegistered(name: String)
object UserRegistered {
  implicit val schema: Schema[UserRegistered] = Schema.derived[UserRegistered]
}

case class OrderPlaced(customerId: String, total: Long)
object OrderPlaced {
  implicit val schema: Schema[OrderPlaced] = Schema.derived[OrderPlaced]
}

case class UserProfile(@Modifier.id id: String, name: String)
object UserProfile {
  implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
  implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
}

case class CustomerOrder(@Modifier.id id: String, customerId: String, total: Long)
object CustomerOrder {
  implicit val schema: Schema[CustomerOrder]         = Schema.derived[CustomerOrder]
  implicit val entityPath: EntityPath[CustomerOrder] = EntityPath.derived[CustomerOrder]
}

case class OrderStats(@Modifier.id id: String, orders: Long)
object OrderStats {
  implicit val schema: Schema[OrderStats] = Schema.derived[OrderStats]
}

// PerEntity: one row for each entity ID.
val profiles = Projection[UserProfile]("userProfiles")
  .from("users")
  .routeToSelf
  .on[UserRegistered]
  .insert((e, ctx) => UserProfile(ctx.entityId, e.name))

// CrossEntity: events are grouped by a key read from the event.
val ordersByCustomer = Projection[CustomerOrder]("ordersByCustomer")
  .from("orders")
  .routedBy[OrderPlaced](_.customerId)
  .on[OrderPlaced]
  .insert((e, ctx) => CustomerOrder(ctx.entityId, e.customerId, e.total))

// Global: one shared store, here with a counter.
val stats = Projection
  .global[OrderStats]("orderStats")
  .from("orders")
  .routeToAll
  .on[OrderPlaced]
  .aggregate(FieldUpdate.Increment("orders", 1L))

// profiles.scope         == ProjectionScope.PerEntity
// ordersByCustomer.scope == ProjectionScope.CrossEntity(...)
// stats.scope            == ProjectionScope.Global
```

### PerEntity: one card per thing

Each order, user or product gets its own card, and a new note about order 7 updates the card of order 7 only. This scope answers the question "what is the state of this one thing?", for example "alice, keyboard, shipped".

Every entity has its own row, and the events recorded under an entity ID update that entity's row. The routing mode of the source is [`routeToSelf`](#routetoself), which files a note under the thing it was recorded under, and a handler reads that ID as `ProjectionContext#entityId`. The store is the file `<basePath>/<name>.db`, for example `users/userProfiles.db`. It is the right scope for state that belongs to one thing and is looked up by its own ID, such as a user's profile or the status of an order.

### CrossEntity: one drawer per group

Here the cabinet has a drawer for each customer. The note says "order 7 placed, customer Alice", so it is filed in Alice's drawer and not in a drawer for order 7. Inside the drawer each order still has its own card. This scope answers the question "what belongs together under this customer?".

Events are grouped by a key that is read from the event, not by the ID they were recorded under. You say which part of the event picks the drawer with the routing mode [`routedBy`](#routedby), for example `routedBy[OrderPlaced](_.customerId)`. Any source that uses `routedBy` makes the projection `CrossEntity`. Each routing key gets its own store, called a shard, next to the main one: `<basePath>/<name>_shard_<key>.db`. With SQLite each shard is its own file, such as `customer_orders_shard_alice.db`. In the file name the key is cut to 64 characters, and any character other than letters, digits, `_` and `-` is replaced by `_`. If the key cannot be read from an event, the engine uses the entity ID.

The routing key only chooses the shard. A row keeps its own ID, so two rows in the same shard still need different IDs. To read a row you give its ID: the engine looks in the main store first, and then in the shards it has opened since it was created. It opens a shard when it routes the first event to it. It uses the scope to know that there are shards to look in; it does not use the key to pick one.

It is the right scope for views that group things by a related entity, such as the orders of a customer or the repositories of an owner. See [Routing Methods](#defining-routes) for a full example.

### Global: one scoreboard on the wall

There is no card for each thing and no drawer for each group. Every note, whoever it is about, changes the numbers on one board: orders so far, shipped, pending. This scope answers the question "how many in total?".

All events go to one shared store, `global/<name>.db`, whichever entity or key they belong to, and from as many sources as the projection reads. The routing mode of its sources is [`routeToAll`](#routetoall), which does not split the events by entity or by key. It does not make a projection global by itself, only `Projection.global` does. It is the right scope for totals and counters, such as the number of orders or the revenue of a shop. See [Aggregating Fields](#aggregating-fields) for a full example.

Two details matter in practice:

- A counter update is applied to the row whose ID the event was recorded under. To keep one row of totals, record all the events under one fixed ID, for example a date or `"shop"`. Otherwise each event updates a different row.
- A global projection has no `EntityPath`, so its store looks for an ID column called `id`. Name the ID field of the row `id`.

## Inspecting a Projection

Several read-only members expose a completed projection's internal structure. These are primarily useful for debugging, validation, and building custom runners or middleware layers.

### Validation

`validate()` checks the projection for structural problems and returns a list of warning strings, empty when the projection is well-formed:

```scala
val warnings = proj.validate()
// Possible warnings (with <name> replaced by the projection's name):
//   "Projection <name> has no bindings"
//   "Projection <name> has no handlers"
//   "Projection <name> has binding with empty sourceName"
//   "Handler for <tag> has null class"
```

Running `validate()` before passing a projection to an engine helps catch misconfigured projections early, before any I/O occurs.

### Looking Up a Handler and a Routing Key

:::note[Advanced]
You do not need this section to define or run a projection. The engine calls `handle` and `routingKey` for you, so a normal application never calls them. They are described here to show how the engine decides what an event does and where it goes, and for anyone writing a test or a custom runner.
:::

Two methods show what the projection would do with an event, without an engine. They are what the engine calls for every event: the first finds the handler, and the second finds the key that decides which store gets the event.

```scala
final class Projection[A] {
  def handle(event: Any, ctx: ProjectionContext, source: Option[String] = None): Option[ProjectionAction[A]]
  def routingKey(event: Any, ctx: ProjectionContext, sourceName: String): Option[String]
}
```

- `handle` returns the `ProjectionAction` of the first handler that matches the event, or `None` if no handler matches. With `source = Some(name)` it searches only the handlers of that source. With `None`, which is what the engine uses, it searches all handlers, in the order the sources were added. It does not apply the action to a store, so it is a way to unit-test a handler without an engine.
- `routingKey` returns the key for an event in the named source: `ctx.entityId` for `routeToSelf`, the extracted key for `routedBy`, and `None` for `routeToAll` or when there is no source with that name. See [Binding Event Sources](#sources-and-routing).

The snippet below shows both methods on a projection with an `"orders"` source that is routed by customer. It is a fragment: `OrderPlaced`, `OrderShipped` and `Refunded` are event case classes, and `ctx` is a `ProjectionContext` for an event recorded under `"order-1"`:

```scala
val projection: Projection[CustomerOrder] =
  Projection[CustomerOrder]("customer_orders")
    .from("orders").routedBy[OrderPlaced](_.customerId)
    .on[OrderPlaced].insert((e, ctx) => CustomerOrder(ctx.entityId, e.customerId, e.product, "placed"))
    .on[OrderShipped].updateWithField("status", (_, _) => "shipped")

// Which action would each event produce?
projection.handle(OrderPlaced("alice", "keyboard"), ctx)
// Some(Insert(CustomerOrder(order-1,alice,keyboard,placed)))

projection.handle(OrderShipped("alice"), ctx)
// Some(Update(Chunk(Set(status,shipped))))

projection.handle(Refunded("alice"), ctx)
// None: no handler matches this event type

projection.handle(OrderShipped("alice"), ctx, Some("other"))
// None: there is no source named "other"

// Which key picks the shard of the event?
projection.routingKey(OrderPlaced("alice", "keyboard"), ctx, "orders")
// Some(alice): the customer id, because the source is routed by customer

projection.routingKey(OrderPlaced("alice", "keyboard"), ctx, "other")
// None: there is no source named "other"
```

Nothing was written anywhere by these calls. `handle` only says that the event would insert a row, or set `status`, and `routingKey` says it would go to the shard of `alice`.

## See Also

- [Projection module overview](./index.md) — module introduction, key concepts, and integration points with `zio-blocks-schema`, `zio-blocks-sql`, and ZIO Streams.
- [ProjectionEngine](./projection-engine.md) — the runtime that executes `Projection[A]` descriptors: event dispatch, watermark tracking, schema-change detection, and the SQLite / in-memory store implementations.
- [Migration](./migration.md) — renamed event cases, `@eventTag`, changed event fields, and what happens when the row type changes.

---

## Testing

The projection module ships with in-memory implementations for unit tests, in the package `zio.blocks.projection.testing`. No SQLite is required, and nothing is written to disk. `TestEngine` wires a whole engine in one call, `InMemoryProjectionStore` is the store it uses, `TestContext` builds the `ProjectionContext` a handler receives, and `TestProjectionEngine` runs a handler on its own, without an engine.

## TestEngine

`TestEngine` replaces the full JVM wiring for unit and integration tests. It creates an `InMemoryProjectionStore[A]` per projection, an in-memory `EventStore` per source name, starts the underlying `ProjectionEngine`, and manages the whole lifecycle in a single `Scope`:

```scala
object TestEngine {
  def make(projections: Projection[_]*): ZIO[Scope, Nothing, TestEngine]
  def makeWithConfig(config: ProjectionEngineConfig, projections: Projection[_]*): ZIO[Scope, Nothing, TestEngine]
  def apply(projections: Projection[_]*): ZIO[Scope, Nothing, TestEngine] // same as make
}

final class TestEngine private (
  val projections: List[Projection[_]],
  val stores: Map[String, ProjectionStore[_]],
  val engine: ProjectionEngine
) {
  def append[E: Schema](entityId: String, event: E): Task[Unit]
  def append[E: Schema](entityId: String, event: E, source: String): Task[Unit]
  def query[A](projection: Projection[A], id: String): Task[Option[A]]
  def queryByName[A](specName: String, id: String): Task[Option[A]]
  def underlying: ProjectionEngine // same as engine
}
```

`make` uses the default `ProjectionEngineConfig`; `makeWithConfig` takes one of your own, and its `ringCapacity` sets the size of each source's `Hub`. The engine runs until the enclosing `Scope` closes.

```scala

case class UserProfile(@Modifier.id id: String, name: String, email: String)
object UserProfile {
  implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
  implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
}
case class UserCreated(name: String, email: String)
object UserCreated {
  implicit val schema: Schema[UserCreated] = Schema.derived[UserCreated]
}

val projection = Projection[UserProfile]("userProfiles")
  .from("users").routeToSelf
  .on[UserCreated]
  .insert((e, ctx) => UserProfile(ctx.entityId, e.name, e.email))

val test: ZIO[Scope, Throwable, Unit] = for {
  engine <- TestEngine.make(projection)
  _      <- engine.append("user-1", UserCreated("Alice", "alice@example.com"))
  result <- engine.query(projection, "user-1")
  // result == Some(UserProfile("user-1", "Alice", "alice@example.com"))
} yield ()
```

The two-argument `append` picks the source for you. With a single source the event goes to it. With several sources it goes to every source whose bindings declare a handler for the event's runtime class, or to all sources if none does. To direct an event to one source by name, use the three-argument `append(entityId, event, source)`; it fails if there is no source with that name.

`query` reads a row back by projection and ID, and `queryByName` does the same by the projection's name. The underlying `ProjectionEngine` is available as `engine.engine`, and the in-memory store of each projection as `engine.stores`, keyed by projection name.

:::caution
Processing is asynchronous. The engine applies events on background fibers, so a query issued immediately after `append` may return `None`. In tests, poll with a retry loop until the expected row appears. The projection examples do this with a small `pollUntil` helper, in `projection-examples/src/main/scala/projection/examples/Polling.scala`.
:::

## InMemoryProjectionStore

A `ProjectionStore[A]` backed by a `Ref[Map[String, A]]`. It supports every store operation: insert, upsert, `updateFields`, delete, truncate, the watermark and the schema hash. A `FieldUpdate` is applied by decoding the stored row into a `DynamicValue`, changing the field, and encoding it back. For rows without `Option` fields and with valid column names, counters behave as they do in SQLite. In 0.0.56 the in-memory store differs in these cases, which are known bugs:

- A row type with an `Option` field: a missing row cannot be created, a `Set` of that field fails, and an `Increment` of a `None` field does nothing.
- An unknown column name: SQLite fails, while the in-memory store ignores it.
- A `Set` with a value of the wrong type: SQLite stores it and reads back `0` for an `Int` column, while the in-memory store fails with `Expected Int`.

```scala
object InMemoryProjectionStore {
  def make[A: Schema: EntityPath]: Task[InMemoryProjectionStore[A]]
  def makeWithRefs[A: Schema: EntityPath](
    map: Map[String, A] = Map.empty,
    seq: Long = 0L,
    hash: Option[String] = None
  ): Task[InMemoryProjectionStore[A]]
}
```

`make` creates an empty store. `makeWithRefs` creates one with rows, a watermark and a schema hash already in it, for a test that starts from a known state.

```scala

case class UserProfile(@Modifier.id id: String, name: String, email: String)
object UserProfile {
  implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
  implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
}

val test: ZIO[Any, Throwable, Unit] = for {
  store <- InMemoryProjectionStore.make[UserProfile]
  _     <- store.insert(UserProfile("u1", "Alice", "alice@example.com"))
  alice <- store.findById("u1")
  // alice == Some(UserProfile("u1", "Alice", "alice@example.com"))
} yield ()
```

## TestContext

A factory for the `ProjectionContext` values that handlers receive, so a handler can be called directly in a test:

```scala
object TestContext {
  def make(
    entityId: String = "test-1",
    seq: Long = 0L,
    timestamp: Instant = Instant.now(),
    sourceEntityId: Option[String] = None
  ): ProjectionContext
  def makeWithSource(entityId: String, sourceEntityId: String): ProjectionContext
  def withSeq(entityId: String, seq: Long): ProjectionContext
  def withTimestamp(entityId: String, timestamp: Instant): ProjectionContext
}
```

```scala

val ctx1 = TestContext.make(entityId = "user-1")
// ProjectionContext("user-1", now, 0, None)

val ctx2 = TestContext.makeWithSource(entityId = "user-1", sourceEntityId = "src-1")
// ProjectionContext("user-1", now, 0, Some("src-1"))

val ctx3 = TestContext.withSeq(entityId = "user-1", seq = 42L)
// ProjectionContext("user-1", now, 42, None)

val ctx4 = TestContext.withTimestamp(entityId = "user-1", timestamp = Instant.EPOCH)
// ProjectionContext("user-1", 1970-01-01T00:00:00Z, 0, None)
```

## TestProjectionEngine

`TestProjectionEngine` runs handlers without an engine, a hub or a scope. It calls the projection's handler for an event and, if you give it a store, applies the returned action to that store at once. Because nothing happens in the background, there is nothing to wait for, which makes it the simplest way to test what a handler does:

```scala
object TestProjectionEngine {
  def make: TestProjectionEngine
}

class TestProjectionEngine {
  def processEvent[E, A](spec: Projection[A], event: E, ctx: ProjectionContext): Task[ProjectionAction[A]]
  def processEvents[E, A](spec: Projection[A], events: List[(E, ProjectionContext)]): Task[Unit]
  def processEvents[E, A](spec: Projection[A], events: List[(E, ProjectionContext)], store: ProjectionStore[A]): Task[Unit]
  def processEventsWithActions[E, A](spec: Projection[A], events: List[(E, ProjectionContext)]): Task[List[ProjectionAction[A]]]
  def applyAction[A](store: ProjectionStore[A], action: ProjectionAction[A], ctx: ProjectionContext): Task[Unit]
  def processAndApply[E, A](spec: Projection[A], event: E, ctx: ProjectionContext, store: ProjectionStore[A]): Task[ProjectionAction[A]]
}
```

- `processEvent` returns the action the handler produces, or `Noop` when no handler matches. It does not touch a store.
- `processAndApply` does the same and then applies the action to the store, returning the action. `processEvents` with a store does this for a list of events, and `processEventsWithActions` collects the actions of a list without applying them.
- `applyAction` applies one action to a store the way the engine would: `Insert` and `Upsert` write the row, `Update` changes fields of the row with `ctx.entityId`, `Delete` removes that row, `Truncate` empties the store, and `Noop` does nothing.

```scala

case class UserProfile(@Modifier.id id: String, name: String, email: String)
object UserProfile {
  implicit val schema: Schema[UserProfile]         = Schema.derived[UserProfile]
  implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
}
case class UserCreated(name: String, email: String)
object UserCreated {
  implicit val schema: Schema[UserCreated] = Schema.derived[UserCreated]
}

val projection = Projection[UserProfile]("userProfiles")
  .from("users").routeToSelf
  .on[UserCreated]
  .insert((e, ctx) => UserProfile(ctx.entityId, e.name, e.email))

val test: ZIO[Any, Throwable, Unit] = for {
  store  <- InMemoryProjectionStore.make[UserProfile]
  runner  = TestProjectionEngine.make
  action <- runner.processAndApply(projection, UserCreated("Alice", "alice@example.com"), TestContext.make("user-1"), store)
  // action == ProjectionAction.Insert(UserProfile("user-1", "Alice", "alice@example.com"))
  alice  <- store.findById("user-1")
  // alice == Some(UserProfile("user-1", "Alice", "alice@example.com"))
} yield ()
```

## See Also

- [Projection module overview](./index.md) — how `Projection[A]`, `ProjectionEngine`, the stores and the test helpers fit together.
- [Projection[A]](./projection.md) — the descriptor the helpers run: sources, routing, handlers and `ProjectionAction`.
- [ProjectionEngine](./projection-engine.md) — the runtime that `TestEngine` wraps, and the SQLite stores used outside tests.

---

## DeferHandle

`DeferHandle` is a handle returned by `Scope.defer` that allows cancelling a registered finalizer before the scope closes:

```scala
abstract class DeferHandle {
  def cancel(): Unit
}
```

When `Scope#defer(cleanup)` is called, the cleanup action is registered and a `DeferHandle` is returned. This handle can be used to remove that finalizer early, preventing it from running when the scope closes. This is useful when a resource is explicitly released before the scope ends, and running the finalizer again would be unnecessary or harmful.

## Construction

`DeferHandle` is not instantiated directly. Instead, it is created by calling `Scope#defer` with a cleanup action:

```scala

trait Scope {
  def defer(cleanup: => Unit): DeferHandle
}
```

The following example demonstrates creating a `DeferHandle`:

```scala

Scope.global.scoped { scope =>

  val handle = defer {
    println("This cleanup will run when scope closes, unless cancelled")
  }

  // The handle can now be used to cancel the cleanup
  handle.cancel()
}
```

## Core Operations

The `DeferHandle#cancel` method removes the registered finalizer so it will not run when the scope closes:

```scala
trait DeferHandle {
  def cancel(): Unit
}
```

This method is:

- **Thread-safe**: Can be called from any thread without synchronization
- **Idempotent**: Calling it multiple times has the same effect as calling once

If the scope has already closed (and the finalizer has already run or been discarded), calling `DeferHandle#cancel` is a no-op. In the following example, we register a cleanup action, then cancel it before the scope closes:

```scala

Scope.global.scoped { scope =>

  val buffer = allocate(ByteArrayOutputStream())
  val closeHandle = defer {
    println("Auto-closing buffer")
    $(buffer)(_.close())
  }

  // Manually close the buffer
  $(buffer)(_.close())

  // Cancel the automatic finalizer since we already closed it
  closeHandle.cancel()
}
```

## Use Cases

`DeferHandle` is useful in several common scenarios:

### Preventing Duplicate Cleanup

When a resource is explicitly released before the scope ends, cancel the automatic finalizer to avoid duplicate cleanup:

```scala

val result = Scope.global.scoped { scope =>

  val buffer = allocate(ByteArrayOutputStream())

  val finalizeHandle = defer {
    println(s"Finalizer running, buffer closing")
    $(buffer)(_.close())
  }

  // Explicit cleanup
  $(buffer) { buf =>
    buf.write("data".getBytes)
    println(s"Manual use: buffer has ${buf.size()} bytes")
    buf.close()
  }

  // Cancel the automatic finalizer
  finalizeHandle.cancel()
  
  "done"
}
```

### Conditional Cleanup

Cancel finalizers based on runtime conditions:

```scala

def acquireResource(shouldCleanup: Boolean) = Scope.global.scoped { scope =>

  val resource = "important resource"
  val handle = scope.defer {
    println("Cleaning up resource")
  }

  if (!shouldCleanup) {
    handle.cancel()
    println("Cleanup disabled")
  }

  resource
}

acquireResource(shouldCleanup = false)
acquireResource(shouldCleanup = true)
```

### Transferring Ownership

When transferring a resource to external management, cancel its finalizer so the external system can control cleanup:

```scala

val result = Scope.global.scoped { scope =>

  val stream = allocate(ByteArrayInputStream("data".getBytes))
  val handle = defer {
    println("Scope finalizer would close stream")
    $(stream)(_.close())
  }

  // Transfer ownership to external manager
  // (In real code, this might pass to a thread pool or async framework)
  handle.cancel()  // Let the manager handle cleanup
  
  "transferred"
}
```

## Noop Handle

When `defer()` is called on an already-closed scope, a no-op handle is returned:

```scala

Scope.global.scoped { scope =>

  val handle = scope.defer {
    println("This will run when scope closes")
  }

  // Subsequent calls to cancel() remove the finalizer
  handle.cancel()

  println("Finalizer has been cancelled")
}
```

## Thread Safety

`DeferHandle` is thread-safe. Multiple threads can call `cancel()` on the same handle without external synchronization:

```scala

val result = Scope.global.scoped { scope =>

  val handle = defer {
    println("Finalizer")
  }

  // Use a thread pool to simulate concurrent access
  val executor = Executors.newFixedThreadPool(5)
  val latch = new CountDownLatch(5)

  (1 to 5).foreach { i =>
    executor.submit(new Runnable {
      def run(): Unit = {
        handle.cancel()
        println(s"Thread $i cancelled")
        latch.countDown()
      }
    })
  }

  // Wait for all threads to finish
  latch.await()
  executor.shutdown()
  
  "completed"
}
```

## See Also

- [`Scope#defer`](./scope.md#finalizer) — the method that returns a `DeferHandle`
- [`Finalizer`](./finalizer.md) — the trait defining `Finalizer#defer`
- [`Finalization`](./finalization.md) — the result of running all finalizers

## Integration

`DeferHandle` is part of ZIO Blocks' resource management system. It works directly with:

- **[`Scope`](./scope.md)** — The primary way to create a `DeferHandle` is via `Scope#defer`. A scope manages multiple finalizers and runs them all when the scope closes. `DeferHandle` allows selective cancellation of individual finalizers before that happens.

- **[`Finalizer`](./finalizer.md)** — `Finalizer` defines the `Finalizer#defer` operation that returns a `DeferHandle`. It abstracts the concept of registering cleanup actions.

- **[`Finalization`](./finalization.md)** — When a scope closes, it runs all registered finalizers. A cancelled `DeferHandle` removes its associated finalizer from this process.

Together, these types form the foundation of compile-time resource safety in ZIO Blocks, allowing you to manage resource lifecycles with certainty that cleanup will occur exactly when needed.

---

## Finalization

`Finalization` is the result of running all finalizers in a scope, collecting any errors that occurred during cleanup:

```scala

final class Finalization(val errors: Chunk[Throwable]) {
  def isEmpty: Boolean
  def nonEmpty: Boolean
  def orThrow(): Unit
  def suppress(initial: Throwable): Throwable
}
```

When a scope closes, each registered finalizer runs in LIFO order. If any finalizer throws an exception, that error is caught and collected into a `Finalization`. This type ensures that all finalizers run even if some fail, and allows the caller to decide how to handle accumulated errors.

`Finalization` collects errors from finalizers in a `Chunk[Throwable]`. The first error in the chunk corresponds to the head of the chunk (the first finalizer that failed in LIFO execution order).

## Core Methods

The following four methods allow you to inspect and handle errors from finalization:

### `Finalization#isEmpty`

Returns `true` if no finalizer errors were collected:

```scala

final class Finalization(val errors: Chunk[Throwable]) {
  def isEmpty: Boolean
}
```

Here's an example of checking if finalization succeeded:

```scala

Scope.global.scoped { scope =>

  $(open()) { openScope =>

    defer {
      println("Cleanup")
    }
    val fin = openScope.close()
    if (fin.isEmpty) println("No errors") else println("Errors occurred")
  }
}
```

### `Finalization#nonEmpty`

Returns `true` if at least one finalizer error was collected:

```scala

final class Finalization(val errors: Chunk[Throwable]) {
  def nonEmpty: Boolean
}
```

Here's an example of checking for errors:

```scala

Scope.global.scoped { scope =>

  $(open()) { openScope =>

    defer {
      throw new Exception("Cleanup failed")
    }
    val fin = openScope.close()
    if (fin.nonEmpty) {
      println(s"Errors occurred: ${fin.errors.length}")
    }
  }
}
```

### `Finalization#orThrow()`

Throws the first collected error with all remaining errors added as suppressed exceptions. Does nothing if there are no errors:

```scala

final class Finalization(val errors: Chunk[Throwable]) {
  def orThrow(): Unit
}
```

The first error corresponds to the head of the chunk (the first finalizer that failed in LIFO execution order). Remaining errors are attached as suppressed exceptions using `addSuppressed()`. Here's an example:

```scala

Scope.global.scoped { scope =>

  $(open()) { openScope =>
    defer { throw Exception("Error 2") }
    defer { throw Exception("Error 1") }
    val fin = openScope.close()
    
    try {
      fin.orThrow()
    } catch {
      case e: Exception =>
        println(s"Primary: ${e.getMessage}")
        e.getSuppressed.foreach(s => println(s"Suppressed: ${s.getMessage}"))
    }
  }
}
```

### `Finalization#suppress(initial)`

Adds all collected finalizer errors as suppressed exceptions to `initial` and returns it. If there are no errors, `initial` is returned unchanged:

```scala

final class Finalization(val errors: Chunk[Throwable]) {
  def suppress(initial: Throwable): Throwable
}
```

This is useful when you want to preserve the original error context while attaching cleanup errors. Here's an example:

```scala

Scope.global.scoped { scope =>

  val initialError = Exception("Original error")

  $(open()) { openScope =>
    defer { throw Exception("Cleanup error") }
    val fin = openScope.close()
    
    val combined = fin.suppress(initialError)
    println(s"Primary: ${combined.getMessage}")
    combined.getSuppressed.foreach(s => println(s"Suppressed: ${s.getMessage}"))
  }
}
```

## Error Ordering

Errors in the finalization are ordered by when finalizers ran (in LIFO sequence):

```scala

Scope.global.scoped { scope =>

  $(open()) { openScope =>
    // Registered first, runs last (LIFO)
    defer { throw Exception("Error 1") }

    // Registered second, runs second
    defer { throw Exception("Error 2") }

    // Registered third, runs first
    defer { throw Exception("Error 3") }

    val fin = openScope.close()
    
    // Errors list order: [Error 3, Error 2, Error 1]
    println(s"First error: ${fin.errors.head.getMessage}")
    println(s"Total errors: ${fin.errors.length}")
  }
}
```

## Use Cases

Here are common scenarios where finalization handling is useful:

### Conditional Error Handling

Check if errors occurred and handle them appropriately:

```scala

Scope.global.scoped { scope =>

  $(open()) { openScope =>
    defer {
      println("Cleanup complete")
    }
    val fin = openScope.close()
    
    if (fin.nonEmpty) {
      fin.orThrow()
    } else {
      println("No errors during finalization")
    }
  }
}
```

### Combining Multiple Error Sources

Attach cleanup errors to an existing error:

```scala

def doWork(): Unit = {
  Scope.global.scoped { scope =>

    try {
      throw Exception("Work failed")
    } catch {
      case workError: Exception =>
        $(open()) { openScope =>
          defer { throw Exception("Cleanup failed") }
          val fin = openScope.close()
          val combined = fin.suppress(workError)
          throw combined
        }
    }
    ()  // Return unit on normal path
  }
}

try {
  doWork()
} catch {
  case e: Exception =>
    println(s"Work: ${e.getMessage}")
    e.getSuppressed.foreach(s => println(s"During cleanup: ${s.getMessage}"))
}
```

### Logging All Cleanup Errors

Inspect and log all errors without stopping execution:

```scala

Scope.global.scoped { scope =>

  $(open()) { openScope =>
    defer { throw Exception("Error 1") }
    defer { throw Exception("Error 2") }
    val fin = openScope.close()
    
    if (fin.nonEmpty) {
      println(s"Finalization collected ${fin.errors.length} errors:")
      fin.errors.foreach(e => println(s"  - ${e.getMessage}"))
    }
  }
}
```

## Relationship to Scope

`Finalization` is returned by:

- `OpenScope#close()` — when explicitly closing a child scope made with `open()`

## See Also

- [`Scope#defer`](./scope.md) — registers finalizers that produce errors
- [`DeferHandle`](./defer-handle.md) — handle for cancelling finalizers
- [`Finalizer`](./finalizer.md) — the trait for registering cleanup actions

---

## Finalizer

`Finalizer` is a minimal capability interface for registering cleanup actions. It exposes only the `Finalizer#defer` method, preventing code from accessing scope internals like resource allocation or closing.

The structural definition:

```scala
trait Finalizer {
  def defer(f: => Unit): DeferHandle
}
```

This trait serves as a boundary between scope management internals and user code that only needs to register cleanup actions. By exposing only `defer`, code can safely request cleanup registration without requiring full scope access.

## Motivation / Use Case

`Finalizer` integrates with `Scope` to enable resource management patterns:

```scala

def openConnection(url: String)(implicit fin: Finalizer): String = {
  fin.defer {
    println(s"Closing connection to $url")
  }
  s"Connected to $url"
}

Scope.global.scoped { scope =>

  openConnection("https://example.com")
  // Connection closes when scope exits
  ()
}
```

By decoupling code that needs cleanup registration from code that manages the complete scope lifecycle, `Finalizer` allows functions to safely register finalizers without requiring full scope access.

## Construction / Creating Instances

`Finalizer` is not typically constructed directly. Instead, it is obtained through a scope:

### From a `Scope`

Any `Scope` instance can be used as a `Finalizer` since `Scope extends Finalizer`:

```scala

Scope.global.scoped { scope =>

  // scope is both a Scope and a Finalizer
  val handle = scope.defer {
    println("Cleanup")
  }
  ()  // Return unit
}
```

### As a Context Bound

Functions can request a `Finalizer` via `implicit` parameter, enabling decoupled cleanup registration:

```scala

def setupResource(name: String)(implicit fin: Finalizer): String = {
  fin.defer {
    println(s"Closing $name")
  }
  name
}
```

## Core Operations

The `Finalizer` interface provides a single core operation for registering cleanup handlers:

### `Finalizer#defer`

Registers a finalizer (cleanup action) to run when the scope closes. The cleanup action runs in LIFO order along with other finalizers registered on the same scope. Returns a `DeferHandle` that can cancel the registration before the scope closes:

```scala

Scope.global.scoped { scope =>

  val handle1 = scope.defer {
    println("Cleanup 1")
  }

  val handle2 = scope.defer {
    println("Cleanup 2")
  }

  // Finalizers run in LIFO: Cleanup 2, then Cleanup 1
  // Can cancel before scope closes
  handle1.cancel()
  // Now only Cleanup 2 runs
}
```

### Package-Level `defer` Helper

A package-level convenience function allows writing `defer { cleanup }` when a `Finalizer` is in scope. The signature is:

```scala
def defer(finalizer: => Unit)(implicit fin: Finalizer): DeferHandle
```

This removes the need to write `fin.defer { cleanup }`. Here's the convenience function in use:

```scala

def setupWithCleanup()(implicit fin: Finalizer) = {
  defer {
    println("Cleanup")
  }
}

Scope.global.scoped { scope =>

  setupWithCleanup()
  // Cleanup prints when scope closes
  ()  // Return unit (which is Unscoped)
}
```

## Integration

`Finalizer` is a supertrait of `Scope`. The structural definition shows this relationship:

```scala
sealed abstract class Scope extends Finalizer with ScopeVersionSpecific
```

This means any `Scope` instance can be used where a `Finalizer` is expected. However, the converse is not true—a `Finalizer` reference does not provide `Scope#allocate`, `Scope#$`, or other scope operations.

## Finalization Order

Finalizers registered with `Finalizer#defer` run in **LIFO order** (last registered runs first) when the scope closes. This ensures that resources acquired in order can be cleaned up in reverse order:

```scala

Scope.global.scoped { scope =>

  scope.defer { println("First registered, runs last") }
  scope.defer { println("Second registered, runs first") }
  // Output on scope close:
  // Second registered, runs first
  // First registered, runs last
  ()  // Return unit (which is Unscoped)
}
```

## See Also

- [`Scope`](./scope.md) — the full scope lifecycle management
- [`DeferHandle`](./defer-handle.md) — the handle returned by `Finalizer#defer` for cancellation
- [`Finalization`](./finalization.md) — the result of running all finalizers

---

## Resource Management

## Introduction

Resource management and dependency injection are fundamental to building reliable, maintainable applications. ZIO Blocks provides three complementary types that work together to eliminate common lifetime bugs while enabling powerful composition patterns: **Scope** provides compile-time safe resource boundaries, **Resource** encapsulates acquisition and cleanup with automatic finalization, and **Wire** describes dependency graphs with type-safe construction recipes. Together, they form a cohesive system for managing object lifecycles, preventing resource leaks, and building dependency-injected architectures.

## Overview

These three types solve the fundamental problem of managing resources and dependencies in concurrent, long-lived applications:

**[Scope](./scope.md)** is the foundation — it provides a compile-time safe boundary that prevents resources from escaping their intended lifetime. Using path-dependent types, Scope ensures that values allocated in one scope cannot accidentally be used in another scope, catching lifetime violations at compile time rather than causing runtime bugs.

**[Resource](./resource.md)** builds on Scope to describe how to acquire and finalize resources. Rather than executing immediately, a Resource is a lazy recipe that composes naturally with `map`, `flatMap`, and `zip`. When allocated within a scope, finalizers run automatically in LIFO order, ensuring cleanup happens even when errors occur.

**[Wire](./wire.md)** brings it all together by describing how to construct services and their dependencies. The Wire macro automatically handles dependency resolution and AutoCloseable registration, letting you declaratively specify a dependency graph that the compiler validates. Cycle detection is not working yet: in 0.0.56 a dependency cycle crashes the macro (a known bug, #1767).

### How They Work Together

The typical flow is:

1. **Define** dependencies using `Wire.shared[T]` or `Wire.unique[T]` — the macro inspects constructor parameters and generates a wire
2. **Compose** wires together using `Resource.from[App](wire1, wire2, ...)` — builds the dependency graph
3. **Allocate** within a scope using `scope.allocate(resource)` — acquires resources and registers finalizers
4. **Use** scoped values via the `$` accessor — compile-time ensures they can't escape
5. **Cleanup** happens automatically when the scope exits — finalizers run in reverse order (LIFO)

### Common Patterns

**Shared Singletons** — Use `Wire.shared[T]` for expensive resources (database connections, thread pools) that should be created once and reused across the application.

**Per-Request Instances** — Use `Wire.unique[T]` for request-scoped state that should be fresh for each request or operation.

**Manual Construction** — Use `Wire.Shared.fromFunction` or `Wire.Unique.fromFunction` when macro derivation doesn't fit your use case (custom initialization logic, special parameters).

**Resource Composition** — Use `Resource.map`, `Resource.flatMap`, and `Resource.zip` to build complex dependency chains where later resources depend on earlier ones.

### Integration Points

- **Wire** uses **Resource** to manage lifecycles of constructed services
- **Resource** uses **Scope** for finalization and scoped value boundaries
- Both **Wire** and **Resource** produce values that are usable only within a **Scope** context

## See Also

- [Compile-Time Resource Safety with Scope](../../guides/compile-time-resource-safety-with-scope.md) — a step-by-step tutorial covering `Scope`, `Resource` and `Wire`
- [Coming from cats-effect Resource, ZIO Scope/ZLayer, macwire or Using](../../guides/resource-and-di-migration.md) — maps each of those libraries onto `Scope`, `Resource` and `Wire`
- [How to Use ZIO Blocks in a ZIO or cats-effect Application](../../guides/use-zio-blocks-in-zio-or-cats-effect.md) — using `Scope` and `Resource` from ZIO or cats-effect, with cleanup carried across

---

## Resource

`Resource[A]` is a **lazy recipe for managing resource lifecycles**, encapsulating both acquisition and finalization tied to a `Scope`. Resources describe *what* to do, not *when* — creation only happens when a resource is passed to `scope.allocate()`. They compose naturally with `map`, `flatMap`, and `zip` to build complex dependency graphs with automatic cleanup in LIFO order.

```scala
sealed trait Resource[+A] {
  def map[B](f: A => B): Resource[B]
  def flatMap[B](f: A => Resource[B]): Resource[B]
  def zip[B](that: Resource[B]): Resource[(A, B)]
}
```

Key properties:
- **Lazy**: Resources don't acquire anything until allocated via `scope.allocate()`
- **Covariant**: `Resource[Dog]` is a subtype of `Resource[Animal]` when `Dog <: Animal`
- **Composable**: `map`, `flatMap`, and `zip` combine resources into larger structures
- **Two strategies**: Shared (memoized with reference counting) and Unique (fresh per allocation)
- **Auto-cleanup**: Finalizers run automatically in LIFO order when scopes close

## Motivation

Without Resources, managing complex initialization and cleanup is tedious and error-prone. Resources eliminate manual bookkeeping by tying value lifecycles to Scopes and registering finalizers automatically.

**Benefits:**
- Automatic cleanup even on exceptions
- LIFO finalization order (inner resources close before outer ones)
- Compositional: build complex dependency graphs declaratively
- Type-safe: compiler ensures you have dependencies available
- Works seamlessly with `Wire` for constructor-based dependency injection

## Installation

Add the ZIO Blocks Scope module to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-scope" % "0.0.56"
```

For cross-platform (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-scope" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x.

## Shared and Unique Resources

Resource offers two distinct types for managing instance lifecycles: **Shared** for singleton-like resources that are allocated once and reused across multiple scopes, and **Unique** for fresh resources created on each allocation. Understanding when to use each type is critical for building efficient and correct resource-management architectures.

### Comparison

| Aspect             | Shared                                             | Unique                                    |
|--------------------|----------------------------------------------------|-------------------------------------------|
| **Creation**       | `Resource.shared(f)`                               | `Resource.unique(f)` or `Resource(value)` |
| **Memoization**    | Yes, with reference counting                       | No, fresh per allocation                  |
| **When to use**    | Expensive resources (DB connections, thread pools) | Per-request state, isolated handlers      |
| **Instance reuse** | Same instance across all allocations               | New instance per allocation               |
| **Finalization**   | Runs when last reference released                  | Runs immediately when scope closes        |
| **Thread safety**  | Thread-safe atomic (CAS) reference counting        | Per-scope, no global coordination needed  |

### Shared Resources

**Shared resources are memoized**: the first allocation initializes the instance; subsequent allocations return the same reference with automatic reference counting. Finalizers run only when the last reference is released. Use shared resources for **expensive, globally singular components** — database connection pools, thread pools, logging systems, caches, and other heavyweight infrastructure that should exist exactly once for the lifetime of the application.

The canonical example is a **database connection pool**. Building a fresh pool for each service layer is wasteful and defeats pooling's purpose. Instead, wrap the pool in a shared resource: both the user service and order service allocate the same pool instance, with the system automatically tracking references and closing the pool only when the last service releases it.

Here's what shared acquisition looks like:

```scala
object Resource {
  def shared[A](f: Scope => A): Resource[A]
}
```

When you allocate a shared resource, reference counting ensures cleanup happens exactly once:

```scala

class DatabasePool extends AutoCloseable {
  def close(): Unit = println("Pool closed (after all services released)")
}

val poolResource = Resource.shared { scope =>
  println("Creating database pool (first allocation only)")
  new DatabasePool
}

// Allocate in ServiceA
Scope.global.scoped { scopeA =>

  val poolA: $[DatabasePool] = poolResource.allocate
  println("ServiceA allocated pool")

  // Allocate in ServiceB within a nested scope
  scopeA.scoped { scopeB =>

    val poolB: $[DatabasePool] = poolResource.allocate
    println("ServiceB allocated same pool instance (reference count incremented)")

    // Both services share the same underlying pool instance
    println("Both ServiceA and ServiceB are using the same pool instance")
  }
  println("ServiceB released, but pool stays open for ServiceA (ref count -= 1)")
}
println("All services released, pool closed (ref count == 0)")
```

### Unique Resources

**Unique resources create fresh instances each time**: every allocation produces a new value. Finalizers run when their owning scope closes, independent of other allocations. Use unique resources for **per-request state, isolated services, and stateful handlers** — anything that should never be shared because it encapsulates request-specific or scope-specific data.

A typical scenario is **per-request caches**: each API request gets its own cache instance to avoid one request polluting another's cached data. Similarly, stateful handlers (parser state machines, transaction contexts, event buffers) need isolation to prevent cross-contamination.

Here's what unique acquisition looks like:

```scala
object Resource {
  def unique[A](f: Scope => A): Resource[A]
}
```

When you allocate a unique resource, each allocation is independent:

```scala

class RequestCache extends AutoCloseable {
  def close(): Unit = println("Request cache closed")
}

val cacheResource = Resource.unique { scope =>
  println("Creating new request cache")
  new RequestCache
}

// Two allocations in the same scope produce different instances
Scope.global.scoped { scope =>

  println("Creating first cache...")
  val cache1: $[RequestCache] = cacheResource.allocate

  println("Creating second cache...")
  val cache2: $[RequestCache] = cacheResource.allocate

  println("Both caches are independent instances")
}
```

### Diamond Dependency Pattern

A classic architecture uses shared resources to solve **diamond dependencies**, where multiple services depend on the same expensive component. Consider an e-commerce application where both `ProductService` and `OrderService` depend on `Logger`:

```
       CachingApp
        /      \
       /        \
  ProductService  OrderService
       \        /
        \      /
         Logger
```

Without shared resources, each service would receive a different `Logger` instance. With `Resource.shared`, both services automatically receive the same singleton instance:

```scala

class Logger extends AutoCloseable {
  def log(msg: String): Unit = println(s"LOG: $msg")
  def close(): Unit = println("Logger closed")
}

class ProductService(val logger: Logger) {
  def findProduct(id: String): String = {
    logger.log(s"Finding product $id")
    s"Product-$id"
  }
}

class OrderService(val logger: Logger) {
  def createOrder(productId: String): String = {
    logger.log(s"Creating order for $productId")
    s"ORD-123"
  }
}

class CachingApp(val productService: ProductService, val orderService: OrderService) extends AutoCloseable {
  def close(): Unit = println("App closed")
}

Scope.global.scoped { scope =>

  // Use Wire.shared[Logger] to ensure only one instance is created
  val app: $[CachingApp] = allocate(
    Resource.from[CachingApp](
      Wire.shared[Logger]  // Both services get the same Logger instance
    )
  )

  $(app) { a =>
    println(s"ProductService and OrderService share Logger? ${a.productService.logger eq a.orderService.logger}")
    a.productService.findProduct("P001")
    a.orderService.createOrder("P001")
  }
}
```

In this pattern, `Resource.shared` (via `Wire.shared`) guarantees a single `Logger` instance across all services, eliminating duplication while maintaining proper cleanup semantics.

### Multiple Shared Dependencies

Most applications need more than one shared resource. Here's a realistic example where `CachingApp` depends on both a shared `Logger` and a shared `MetricsCollector`. Both resources are singletons that all services share:

```
          CachingApp
           /      \
          /        \
         /          \
    ProductService  OrderService
     /  \            /  \
    /    \          /    \
   Logger Metrics  Logger Metrics
    \    /          \    /
     \  /            \  /
      \/              \/
   (shared singleton instances)
```

Here's the implementation:

```scala

class Logger extends AutoCloseable {
  def log(msg: String): Unit = println(s"[LOG] $msg")
  def close(): Unit = println("[Logger] Closed")
}

class MetricsCollector extends AutoCloseable {
  private var eventCount = 0
  def recordEvent(name: String): Unit = {
    eventCount += 1
    println(s"[METRICS] Event: $name (total: $eventCount)")
  }
  def close(): Unit = println(s"[MetricsCollector] Closed after $eventCount events")
}

class ProductService(val logger: Logger, val metrics: MetricsCollector) {
  def findProduct(id: String): String = {
    logger.log(s"Finding product $id")
    metrics.recordEvent("product.find")
    s"Product-$id"
  }
}

class OrderService(val logger: Logger, val metrics: MetricsCollector) {
  def createOrder(productId: String): String = {
    logger.log(s"Creating order for $productId")
    metrics.recordEvent("order.create")
    s"ORD-123"
  }
}

class CachingApp(
  val productService: ProductService,
  val orderService: OrderService
) extends AutoCloseable {
  def close(): Unit = println("[CachingApp] Closed")
}

Scope.global.scoped { scope =>

  // Both Logger and MetricsCollector are shared (singleton instances)
  val app: $[CachingApp] = allocate(
    Resource.from[CachingApp](
      Wire.shared[Logger],              // One Logger for all services
      Wire.shared[MetricsCollector]     // One MetricsCollector for all services
    )
  )

  $(app) { a =>
    println(s"ProductService and OrderService share Logger? ${a.productService.logger eq a.orderService.logger}")
    println(s"ProductService and OrderService share Metrics? ${a.productService.metrics eq a.orderService.metrics}")

    a.productService.findProduct("P001")
    a.orderService.createOrder("P001")
  }
}
```

When this scope closes, both the `Logger` and `MetricsCollector` are finalized exactly once, regardless of how many services reference them. The macro automatically builds a dependency graph, verifies all wires, and ensures LIFO cleanup order.

## Construction

Resources can be created in several ways: from values, from explicit acquire/release pairs, from `AutoCloseable` types, from custom functions, or derived automatically from a type's constructor using the `Resource.from` macros.

### `Resource.apply` — Wrap a Value

Wraps a by-name value as a resource. If the value implements `AutoCloseable`, its `close()` method is automatically registered as a finalizer:

```scala
object Resource {
  def apply[A](value: => A): Resource[A]
}
```

Here's how to use it:

```scala

case class Config(debug: Boolean)

class Database(val name: String) extends AutoCloseable {
  def close(): Unit = println(s"Closing database $name")
}

val configResource = Resource(Config(debug = true))
val dbResource = Resource(new Database("mydb"))
```

### `Resource.acquireRelease` — Explicit Lifecycle

Creates a resource with separate acquire and release functions. The acquire thunk runs during allocation; the release function is registered as a finalizer:

```scala
object Resource {
  def acquireRelease[A](acquire: => A)(release: A => Unit): Resource[A]
}
```

Here's an example with file handling:

```scala

val fileResource = Resource.acquireRelease {
  new FileInputStream("data.txt")
} { stream =>
  stream.close()
}
```

### `Resource.fromAutoCloseable` — Type-Safe Wrapping

Creates a resource specifically for `AutoCloseable` subtypes. This is a compile-time verified alternative to `Resource(value)` when you know the value is closeable:

```scala
object Resource {
  def fromAutoCloseable[A <: AutoCloseable](value: => A): Resource[A]
}
```

Here's an example with a stream:

```scala

val streamResource = Resource.fromAutoCloseable {
  new BufferedInputStream(new FileInputStream("data.bin"))
}
```

### `Resource.shared` — Memoized with Reference Counting

Creates a shared resource that memoizes its value across multiple allocations using reference counting. The first allocation initializes the value using an `OpenScope` parented to `Scope.global`; subsequent allocations increment the reference count and return the same instance. Each scope that receives the shared value registers a finalizer that decrements the count. When the count reaches zero, the shared scope closes automatically. This mechanism is **thread-safe**: the state is an `AtomicReference` updated with compare-and-swap, so concurrent allocations share one instance and one reference count. It is not lock-free: while the first caller runs the initializer, every other caller waits for it.

Under the hood, a shared resource progresses through four states: (1) **Uninitialized** — the resource has never been allocated and the first call triggers initialization; (2) **Pending** — initialization is in progress and other threads wait via spin-yield for the value to become available; (3) **Created** — the value is fully initialized and ready, each allocation increments the reference count, and each scope registers a finalizer to decrement it; (4) **Destroyed** — the reference count reached zero, the resource was cleaned up, and further allocations will fail. This state machine ensures that concurrent allocations share one value, and that cleanup happens once, when the last reference is released. If the initializer throws, the open scope is closed, the exception reaches the caller that ran it, the state returns to Uninitialized, and the next allocation runs the initializer again.

Use shared resources for **expensive, singleton-like components** (database connection pools, thread pools, logging systems, caches) that should exist exactly once for the lifetime of the application, even when multiple services depend on them.

Here's the signature:

```scala
object Resource {
  def shared[A](f: Scope => A): Resource[A]
}
```

Here's a realistic example showing reference counting and automatic cleanup:

```scala

class ExpensiveComponent extends AutoCloseable {
  println("ExpensiveComponent initialized (expensive operation)")
  def close(): Unit = println("ExpensiveComponent cleaned up (last reference released)")
}

val sharedResource = Resource.shared { scope =>
  new ExpensiveComponent()
}

// Multiple services allocating the same shared resource
Scope.global.scoped { scope =>

  // First allocation initializes the component
  println("ServiceA allocating...")
  val componentA: $[ExpensiveComponent] = sharedResource.allocate

  // ServiceB in a nested scope receives the same instance
  scope.scoped { innerScope =>

    println("ServiceB allocating (same instance, ref count += 1)...")
    val componentB: $[ExpensiveComponent] = sharedResource.allocate
    println("Both services have the same instance")
  }

  println("ServiceB released, but component stays alive (ref count -= 1)")
}

println("ServiceA released, component cleaned up (ref count == 0)")
```

### `Resource.unique` — Fresh Instances

Creates a unique resource that produces a fresh instance each time it's allocated. Use for per-request state or resources that should never be shared:

```scala
object Resource {
  def unique[A](f: Scope => A): Resource[A]
}
```

Here's an example showing fresh instances:

```scala

var counter = 0

val uniqueResource = Resource.unique[Int] { _ =>
  counter += 1
  counter
}
```

### `Resource.from[T]` — Derive from Constructor

Derives a `Resource[T]` from `T`'s primary constructor, requiring no external dependencies.
If `T` extends `AutoCloseable` and its constructor takes no `Finalizer` or `Scope` parameter,
`Resource.from` registers its `close()` method as a finalizer. A known bug in 0.0.56: when the
constructor does take a `Finalizer` or `Scope`, `close()` is not registered. Use
`Resource.from[T]()` (with empty parentheses), which registers it, or register the cleanup
inside the constructor with `defer`. The signature is:

```scala
object Resource {
  def from[T]: Resource[T]
}
```

To derive a resource for a type that has no constructor dependencies:

```scala

class MetricsCollector extends AutoCloseable {
  def record(event: String): Unit = println(s"Recording: $event")
  def close(): Unit               = println("MetricsCollector closed")
}

def metricsResource = Resource.from[MetricsCollector]
```

Internally, the macro inspects `T`'s constructor at compile time, verifies that no external
dependencies are needed, and synthesizes a `Resource.shared` that instantiates `T`, passing
the scope to any `Finalizer` or `Scope` parameter, and, when `T` extends `AutoCloseable` and
takes no such parameter, registers `close()` as a scope finalizer. Any attempt
to use `Resource.from[T]` on a type with unsatisfied constructor parameters results in a
compile-time error.

### `Resource.from[T](wires: Wire[?, ?]*)` — Derive with Dependency Overrides

Derives a `Resource[T]` from `T`'s constructor, with `Wire` values provided as dependency
overrides. Any dependency not covered by an explicit wire is auto-derived if the macro can
construct it; otherwise a compile-time error is produced:

```scala
object Resource {
  def from[T](wires: Wire[?, ?]*): Resource[T]
}
```

To derive a resource for a type whose dependencies are provided via wires:

```scala

case class Config(host: String, port: Int)

class Logger(config: Config) extends AutoCloseable {
  def log(msg: String): Unit = println(s"[$msg] ${config.host}:${config.port}")
  def close(): Unit          = log("Logger shutting down")
}

class Service(logger: Logger) extends AutoCloseable {
  def run(): Unit  = logger.log("Service running")
  def close(): Unit = logger.log("Service shutting down")
}

def serviceResource = Resource.from[Service](
  Wire(Config("localhost", 8080))
)
```

Internally, the macro builds a complete dependency graph at compile time: it inspects each
provided wire's input and output types, identifies all constructor parameters of `T`, and
auto-derives wires for any remaining dependencies. It then topologically sorts all wires to
determine acquisition order and composes them into a single `Resource` whose finalizers run
in LIFO order — inner dependencies close before outer ones.

## Core Operations

Resources support transformation and composition through `map`, `flatMap`, and `zip`.

### `Resource#map` — Transform the Value

Transforms the value produced by a resource without affecting finalization. The transformation function is applied after the resource is acquired:

```scala
trait Resource[+A] {
  def map[B](f: A => B): Resource[B]
}
```

Here's a usage example:

```scala

val portResource = Resource(8080)
val urlResource = portResource.map(port => s"http://localhost:$port")
```

### `Resource#flatMap` — Sequence Resources

Sequences two resources, using the result of the first to create the second. Both sets of finalizers are registered and run in LIFO order (inner before outer):

```scala
trait Resource[+A] {
  def flatMap[B](f: A => Resource[B]): Resource[B]
}
```

Here's an example with dependent resources:

```scala

case class DbConfig(url: String)

class Database(config: DbConfig) extends AutoCloseable {
  def query(sql: String): String = s"Result from ${config.url}: $sql"
  def close(): Unit = println("Database closed")
}

val configResource = Resource(DbConfig("jdbc:postgres://localhost"))
val dbResource = configResource.flatMap { config =>
  Resource.fromAutoCloseable(new Database(config))
}
```

### `Resource#zip` — Combine Resources

Combines two resources into a single resource that produces a tuple of both values. Both resources are acquired and both sets of finalizers are registered:

```scala
trait Resource[+A] {
  def zip[B](that: Resource[B]): Resource[(A, B)]
}
```

Here's an example combining multiple resources:

```scala

case class DbConfig(url: String)

class Database(config: DbConfig) extends AutoCloseable {
  def close(): Unit = println("Database closed")
}

class Cache extends AutoCloseable {
  def close(): Unit = println("Cache closed")
}

val dbResource = Resource.fromAutoCloseable(new Database(DbConfig("jdbc:postgres://localhost")))
val cacheResource = Resource.fromAutoCloseable(new Cache())
val combined = dbResource.zip(cacheResource)
```

### `Resource#allocate` — Acquire Within a Scope

Allocates a `Resource[A]` within the current `Scope`, returning a scoped `$[A]`. This is syntax sugar for `scope.allocate(resource)`, available via `import scope._` inside a `scoped` block:

```scala
implicit class ResourceOps[A](private val r: Resource[A]) {
  def allocate: $[A]
}
```

### `Resource#use` — Acquire, Use, and Finalize

For top-level application code, `use` provides a lighter API than spelling out
`Scope.global.scoped` and `allocate` directly:

```scala
trait Resource[+A] {
  def use[B](f: A => B)(implicit ev: Unscoped[B]): B
}
```

Example:

```scala

class HttpServer extends AutoCloseable {
  def start(): Unit = println("server started")
  def close(): Unit = println("server stopped")
}

val serverResource = Resource.fromAutoCloseable(new HttpServer)

serverResource.use(_.start())
```

`use` still creates a global scope under the hood, acquires the resource,
runs the callback, and guarantees finalization when the callback returns.

To allocate a resource and use its value inside a scope:

```scala

class Database extends AutoCloseable {
  def query(sql: String): String = s"Result: $sql"
  def close(): Unit              = println("Database closed")
}

Scope.global.scoped { implicit scope =>

  val db: $[Database] = Resource.fromAutoCloseable(new Database).allocate
  $(db)(_.query("SELECT 1"))
}
```

### `$[Resource[A]]#allocate` — Allocate a Scoped Resource

Allocates a `$[Resource[A]]` — a Resource that is itself a scoped value — returning `$[A]`. Use this when a method on a scoped object returns a Resource and you need to immediately acquire it while keeping the result scoped.

**What is a scoped resource?** A scoped value `$[A]` represents an `A` that is valid only while a scope is alive. A **scoped resource** `$[Resource[A]]` is a Resource object that exists *inside* that scope. When you call a method like `$(pool)(_.lease())` that returns a Resource, the result is typed as `$[Resource[Connection]]` — the Resource itself is scoped. The `.allocate` extension method unwraps this scoped resource and acquires it, returning the acquired value as a new scoped value `$[A]`.

**Motivation:** This pattern appears frequently in resource factories. A scoped object (like a database pool) has methods that produce new Resources. Without this extension, you'd need to unwrap the `$[Resource[A]]` from the `$` context, allocate it separately, and re-wrap the result. The `.allocate` method chains naturally, letting you write `$(pool)(_.lease()).allocate` instead of dealing with intermediate unwrapping.

The implicit class:

```scala
implicit class ScopedResourceOps[A](private val sr: $[Resource[A]]) {
  def allocate: $[A]
}
```

Here's a realistic example where a database pool (a scoped object) has a method that returns a Resource for individual connections:

```scala

class Connection extends AutoCloseable {
  def query(sql: String): String = s"Result: $sql"
  def close(): Unit              = println("Connection closed")
}

class Pool extends AutoCloseable {
  def lease(): Resource[Connection] = Resource.fromAutoCloseable(new Connection)
  def close(): Unit                 = println("Pool closed")
}

Scope.global.scoped { implicit scope =>

  val pool: $[Pool]           = Resource.fromAutoCloseable(new Pool).allocate
  val conn: $[Connection]     = $(pool)(_.lease()).allocate
  $(conn)(_.query("SELECT 1"))
}
```

## Integration

Resource is a core abstraction in ZIO Blocks' resource management ecosystem:

- **[`Scope`](./scope.md)** — Resources require a `Scope` for allocation. The `Scope` manages the lifetime of acquired resources and automatically runs finalizers when the scope closes.
- **[`Wire`](./wire.md)** — The `Resource.from[T](wires)` macro builds dependency graphs using `Wire` values, enabling constructor-based dependency injection with automatic resource management.
- **[`Finalizer`](./finalizer.md)** — Resources register their cleanup logic with `Finalizer` objects, which execute in LIFO order when scopes close.
- **[How to Fix Common Scope and Wire Compile Errors](../../guides/fix-scope-and-wire-compile-errors.md)** — The compile-time errors that `Resource.from` and the Scope macros report, and how to fix each one.

For example, `Resource.from[T](wires*)` uses `Wire` values to construct instances with their dependencies, and registers `close()` for each `AutoCloseable` class the macro derives (an explicit wire registers its own cleanup):

```scala

final case class Config(host: String, port: Int)

final class Logger(config: Config) extends AutoCloseable {
  def log(msg: String): Unit = println(s"[${config.host}:${config.port}] $msg")
  def close(): Unit          = log("Logger shutting down")
}

final class Service(logger: Logger) extends AutoCloseable {
  def run(): Unit   = logger.log("Service running")
  def close(): Unit = logger.log("Service shutting down")
}

def serviceResource = Resource.from[Service](
  Wire(Config("localhost", 8080))
)
```

This builds a complete dependency graph: `Config` → `Logger` → `Service`, with all finalizers managed by the containing `Scope`.

## Running the Examples

All code from this guide is available as runnable examples in the `scope-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

### Basic Lifecycle Management with Temporary Files

This example demonstrates creating and automatically cleaning up temporary files using Resource's lifecycle management. It shows how Resource ensures files are closed even if exceptions occur:

```scala title="scope-examples/src/main/scala/scope/examples/TempFileHandlingExample.scala" 
package scope.examples

/**
 * Demonstrates `scope.defer(...)` for registering manual cleanup actions.
 *
 * This example shows how to create temporary files during processing and ensure
 * they are deleted when the scope exits—even if processing fails. Deferred
 * cleanup actions run in LIFO (last-in-first-out) order.
 */

/** Represents a temporary file with basic read/write operations. */
case class TempFile(path: String) {
  private var content: String = ""

  def write(data: String): Unit = content = data
  def read(): String            = content
  def delete(): Boolean         = { println(s"  Deleting: $path"); true }
}

/** Result of processing temporary files. */
case class ProcessingResult(processedCount: Int, totalBytes: Long, errors: List[String])

/** Processes a list of temporary files and aggregates results. */
object FileProcessor {
  def process(files: List[TempFile]): ProcessingResult = {
    val totalBytes = files.map(_.read().length.toLong).sum
    ProcessingResult(processedCount = files.size, totalBytes = totalBytes, errors = Nil)
  }
}

@main def tempFileHandlingExample(): Unit = {
  println("=== Temp File Handling Example ===\n")
  println("Demonstrating scope.defer() for manual cleanup registration.\n")

  val result = Scope.global.scoped { scope =>
    // Create temp files and register cleanup via defer.
    // Cleanup runs in LIFO order: file3, file2, file1.

    val file1 = createTempFile(scope, "/tmp/data-001.tmp", "First file content")
    val file2 = createTempFile(scope, "/tmp/data-002.tmp", "Second file - more data here")
    val file3 = createTempFile(scope, "/tmp/data-003.tmp", "Third file with the most content of all")

    println("\nProcessing files...")
    val processingResult = FileProcessor.process(List(file1, file2, file3))
    println(s"Processed ${processingResult.processedCount} files, ${processingResult.totalBytes} bytes\n")

    println("Exiting scope - cleanup runs in LIFO order:")
    processingResult
  }

  println(s"\nFinal result: $result")
}

/**
 * Creates a temporary file and registers its cleanup with the scope.
 *
 * The cleanup action is registered via `defer(...)`, ensuring the file is
 * deleted when the scope closes—regardless of whether processing succeeds.
 *
 * @param s
 *   the scope to register cleanup with
 * @param path
 *   the file path
 * @param content
 *   initial content to write
 * @return
 *   the created TempFile
 */
private def createTempFile(s: Scope, path: String, content: String): TempFile = {
  val file = TempFile(path)
  file.write(content)
  println(s"Created: $path (${content.length} bytes)")

  // Register cleanup - will run when scope exits, in LIFO order
  s.defer {
    file.delete()
  }

  file
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/TempFileHandlingExample.scala))

Run this example with:

```bash
cd scope-examples && sbt "runMain scope.examples.tempFileHandlingExample"
```

### Acquiring and Releasing Database Connections

This example demonstrates the acquire-release pattern using Resource to manage database connections. It shows proper connection initialization and guaranteed cleanup:

```scala title="scope-examples/src/main/scala/scope/examples/DatabaseConnectionExample.scala" 
package scope.examples

/**
 * Configuration for database connection.
 *
 * @param host
 *   the database server hostname
 * @param port
 *   the database server port
 * @param database
 *   the database name to connect to
 */
final case class DbConfig(host: String, port: Int, database: String) {
  def connectionUrl: String = s"jdbc:postgresql://$host:$port/$database"
}

/**
 * Represents the result of a database query.
 *
 * @param rows
 *   the result set as a list of row maps
 */
final case class QueryResult(rows: List[Map[String, String]]) {
  def isEmpty: Boolean = rows.isEmpty
  def size: Int        = rows.size
}

/**
 * Simulates a database connection with lifecycle management.
 *
 * This class demonstrates how AutoCloseable resources integrate with ZIO Blocks
 * Scope. When allocated via `allocate(Resource(...))`, the `close()` method is
 * automatically registered as a finalizer.
 *
 * @param config
 *   the database configuration
 */
final class Database(config: DbConfig) extends AutoCloseable {
  private var connected = false

  def connect(): Unit = {
    println(s"[Database] Connecting to ${config.connectionUrl}...")
    connected = true
    println(s"[Database] Connected successfully")
  }

  def query(sql: String): QueryResult = {
    require(connected, "Database not connected")
    println(s"[Database] Executing: $sql")
    sql match {
      case s if s.contains("users") =>
        QueryResult(
          List(
            Map("id" -> "1", "name" -> "Alice"),
            Map("id" -> "2", "name" -> "Bob")
          )
        )
      case s if s.contains("orders") =>
        QueryResult(
          List(
            Map("order_id" -> "101", "user_id" -> "1", "total" -> "99.99"),
            Map("order_id" -> "102", "user_id" -> "2", "total" -> "149.50")
          )
        )
      case _ =>
        QueryResult(List(Map("result" -> "OK")))
    }
  }

  override def close(): Unit = {
    println(s"[Database] Closing connection to ${config.connectionUrl}")
    connected = false
  }
}

/**
 * Demonstrates basic resource lifecycle management with ZIO Blocks Scope.
 *
 * This example shows:
 *   - Allocating an AutoCloseable resource with automatic cleanup
 *   - Using `$(value)(f)` to access scoped values and execute queries
 *   - LIFO finalizer ordering (last allocated = first closed)
 *
 * When the scope exits, all registered finalizers run in reverse order,
 * ensuring proper cleanup even if exceptions occur.
 */
@main def runDatabaseExample(): Unit = {
  println("=== Database Connection Example ===\n")

  val config = DbConfig("localhost", 5432, "myapp")

  Scope.global.scoped { scope =>

    println("[Scope] Entering scoped region\n")

    // Allocate the database resource. Because Database extends AutoCloseable,
    // its close() method is automatically registered as a finalizer.
    val db: $[Database] = allocate(Resource {
      val database = new Database(config)
      database.connect()
      database
    })

    // Use $(value)(f) to access the scoped value and execute queries.
    $(db) { database =>
      val users = database.query("SELECT * FROM users")
      println(s"[Result] Found ${users.size} users: ${users.rows.map(_("name")).mkString(", ")}\n")

      val orders = database.query("SELECT * FROM orders WHERE status = 'pending'")
      println(s"[Result] Found ${orders.size} orders\n")

      val health = database.query("SELECT 1 AS health_check")
      println(s"[Result] Health check: ${health.rows.head("result")}\n")
    }

    println("[Scope] Exiting scoped region - finalizers will run in LIFO order")
  }

  println("\n=== Example Complete ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/DatabaseConnectionExample.scala))

Run this example with:

```bash
cd scope-examples && sbt "runMain scope.examples.runDatabaseExample"
```

### Shared Resources with Memoization and Reference Counting

This example demonstrates Resource.shared to create a singleton logger instance that is automatically cleaned up only when the last service releases it. Shows reference counting in action:

```scala title="scope-examples/src/main/scala/scope/examples/CachingSharedLoggerExample.scala" 
package scope.examples

/**
 * Demonstrates `Wire.shared` vs `Wire.unique` and diamond dependency patterns.
 *
 * Two services (ProductService, OrderService) share one Logger instance
 * (diamond pattern), but each gets its own unique Cache instance. This shows
 * how shared wires provide singleton behavior while unique wires create fresh
 * instances per injection site.
 *
 * Key concepts:
 *   - `Wire.shared[T]`: Single instance shared across all dependents (memoized)
 *   - `Wire.unique[T]`: Fresh instance created for each dependent
 *   - Diamond dependency: Multiple services depend on the same shared resource
 *   - Reference counting: Shared resources track usage and clean up when last
 *     user closes
 */
object CachingSharedLoggerExample {

  /** Tracks instantiation counts for demonstration purposes. */
  val loggerInstances = new AtomicInteger(0)
  val cacheInstances  = new AtomicInteger(0)

  /**
   * A shared logger that tracks instantiations and provides logging methods.
   * Implements AutoCloseable for proper resource cleanup.
   */
  class Logger extends AutoCloseable {
    val instanceId: Int = loggerInstances.incrementAndGet()
    println(s"  [Logger#$instanceId] Created")

    def info(msg: String): Unit  = println(s"  [Logger#$instanceId] INFO: $msg")
    def debug(msg: String): Unit = println(s"  [Logger#$instanceId] DEBUG: $msg")
    def close(): Unit            = println(s"  [Logger#$instanceId] Closed")
  }

  /**
   * A unique cache per service. Each service gets its own isolated cache
   * instance. Implements AutoCloseable for proper resource cleanup. Note: No
   * constructor params so it can be auto-wired with Wire.unique.
   */
  class Cache extends AutoCloseable {
    val instanceId: Int                    = cacheInstances.incrementAndGet()
    private var store: Map[String, String] = Map.empty
    println(s"  [Cache#$instanceId] Created")

    def get(key: String): Option[String]      = store.get(key)
    def put(key: String, value: String): Unit = store = store.updated(key, value)
    def close(): Unit                         = println(s"  [Cache#$instanceId] Closed")
  }

  /** Product service with its own cache but sharing the logger. */
  class ProductService(val logger: Logger, val cache: Cache) {
    println(s"  [ProductService] Created with Logger#${logger.instanceId} and Cache#${cache.instanceId}")

    def findProduct(id: String): String =
      cache.get(id) match {
        case Some(product) =>
          logger.debug(s"Cache hit for product $id")
          product
        case None =>
          logger.info(s"Loading product $id from database")
          val product = s"Product-$id"
          cache.put(id, product)
          product
      }
  }

  /**
   * Order service with its own cache but sharing the same logger as
   * ProductService.
   */
  class OrderService(val logger: Logger, val cache: Cache) {
    println(s"  [OrderService] Created with Logger#${logger.instanceId} and Cache#${cache.instanceId}")

    def createOrder(productId: String): String = {
      val orderId = s"ORD-${System.currentTimeMillis() % 10000}"
      cache.put(orderId, productId)
      logger.info(s"Created order $orderId for product $productId")
      orderId
    }
  }

  /** Top-level application combining both services. */
  class CachingApp(val productService: ProductService, val orderService: OrderService) extends AutoCloseable {
    def run(): Unit = {
      productService.logger.info("=== Application Started ===")
      val product = productService.findProduct("P001")
      orderService.createOrder(product)
      productService.findProduct("P001") // cache hit
    }
    def close(): Unit = println("  [CachingApp] Closed")
  }

  @main def runCachingExample(): Unit = {
    println("\n╔════════════════════════════════════════════════════════════════╗")
    println("║  Wire.shared vs Wire.unique - Diamond Dependency Example       ║")
    println("╚════════════════════════════════════════════════════════════════╝\n")

    println("Creating wires...")
    println("  - Logger: Wire.shared (singleton across all services)")
    println("  - Cache:  Wire.unique (fresh instance per service)\n")

    println("─── Resource Acquisition ───")
    Scope.global.scoped { scope =>

      val app: $[CachingApp] = allocate(
        Resource.from[CachingApp](
          Wire.shared[Logger],
          Wire.unique[Cache]
        )
      )

      println("\n─── Verification ───")
      println(s"  Logger instances created: ${loggerInstances.get()} (expected: 1)")
      println(s"  Cache instances created:  ${cacheInstances.get()} (expected: 2)")
      $(app) { a =>
        println(s"  ProductService.logger eq OrderService.logger: ${a.productService.logger eq a.orderService.logger}")
        println(s"  ProductService.cache  eq OrderService.cache:  ${a.productService.cache eq a.orderService.cache}")

        println("\n─── Running Application ───")
        a.run()
      }

      println("\n─── Scope Closing (LIFO cleanup) ───")
    }

    println("\n─── Summary ───")
    println(s"  Final Logger count: ${loggerInstances.get()} (shared = 1 instance)")
    println(s"  Final Cache count:  ${cacheInstances.get()} (unique = 2 instances)")
    println("\nDiamond pattern verified: both services received the same Logger instance.")
  }
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/CachingSharedLoggerExample.scala))

Run this example with:

```bash
cd scope-examples && sbt "runMain scope.examples.runCachingExample"
```

### Managing Shared Expensive Resources

This example demonstrates using Resource.shared for a database connection pool—an expensive resource that should exist exactly once. Shows how multiple services safely share the same pool instance with automatic cleanup:

```scala title="scope-examples/src/main/scala/scope/examples/ConnectionPoolExample.scala" 
package scope.examples

/**
 * Demonstrates `Resource.Shared` with reference counting and nested resource
 * acquisition.
 *
 * This example shows a realistic connection pool pattern where:
 *   - The pool itself is a shared resource (created once, ref-counted)
 *   - Individual connections are resources that must be allocated in a scope
 *   - `pool.acquire` returns `Resource[PooledConnection]`, forcing proper
 *     scoping
 *
 * This pattern is common for database pools, HTTP client pools, and thread
 * pools.
 */

/** Configuration for the connection pool. */
final case class PoolConfig(maxConnections: Int, timeout: Long)

/**
 * A connection retrieved from the pool.
 *
 * Connections are resources - they must be released back to the pool when done.
 * This is enforced by making `acquire` return a `Resource[PooledConnection]`.
 */
final class PooledConnection(val id: Int, pool: ConnectionPool) extends AutoCloseable {
  println(s"    [Conn#$id] Acquired from pool")

  def execute(sql: String): String = {
    println(s"    [Conn#$id] Executing: $sql")
    s"Result from connection $id"
  }

  override def close(): Unit =
    pool.release(this)
}

/**
 * A connection pool that manages pooled connections.
 *
 * Key design: `acquire` returns `Resource[PooledConnection]`, not a raw
 * connection. This forces callers to allocate the connection in a scope,
 * ensuring proper release even if exceptions occur.
 */
final class ConnectionPool(config: PoolConfig) extends AutoCloseable {
  private val nextId  = new AtomicInteger(0)
  private val active  = new AtomicInteger(0)
  private val _closed = new AtomicInteger(0)

  println(s"  [Pool] Created with max ${config.maxConnections} connections")

  /**
   * Acquires a connection from the pool.
   *
   * Returns a `Resource[PooledConnection]` that must be allocated in a scope.
   * The connection is automatically released when the scope exits.
   */
  def acquire: Resource[PooledConnection] = Resource.acquireRelease {
    if (_closed.get() > 0) throw new IllegalStateException("Pool is closed")
    if (active.get() >= config.maxConnections)
      throw new IllegalStateException(s"Pool exhausted (max: ${config.maxConnections})")

    val id   = nextId.incrementAndGet()
    val conn = new PooledConnection(id, this)
    active.incrementAndGet()
    println(s"  [Pool] Active connections: ${active.get()}/${config.maxConnections}")
    conn
  } { conn =>
    conn.close()
  }

  private[examples] def release(conn: PooledConnection): Unit = {
    val count = active.decrementAndGet()
    println(s"    [Conn#${conn.id}] Released back to pool (active: $count)")
  }

  def activeConnections: Int = active.get()

  override def close(): Unit =
    if (_closed.compareAndSet(0, 1)) {
      println(s"  [Pool] *** POOL CLOSED *** (served ${nextId.get()} total connections)")
    }
}

@main def connectionPoolExample(): Unit = {
  println("=== Connection Pool with Resource-based Acquire ===\n")

  val poolConfig = PoolConfig(maxConnections = 3, timeout = 5000L)

  val poolResource: Resource[ConnectionPool] =
    Resource.fromAutoCloseable(new ConnectionPool(poolConfig))

  Scope.global.scoped { appScope =>

    println("[App] Allocating pool\n")
    val pool: $[ConnectionPool] = poolResource.allocate

    println("--- ServiceA doing work (connection scoped to this block) ---")
    appScope.scoped { workScope =>

      val p: $[ConnectionPool]   = lower(pool)
      val c: $[PooledConnection] = $(p)(_.acquire).allocate
      val result                 = $(c)(_.execute("SELECT * FROM service_a_table"))
      println(s"  [ServiceA] Got: $result")
    }
    println()

    println("--- ServiceB doing work ---")
    appScope.scoped { workScope =>

      val p: $[ConnectionPool]   = lower(pool)
      val c: $[PooledConnection] = $(p)(_.acquire).allocate
      val result                 = $(c)(_.execute("SELECT * FROM service_b_table"))
      println(s"  [ServiceB] Got: $result")
    }
    println()

    println("--- Multiple connections in same scope ---")
    appScope.scoped { workScope =>

      val p: $[ConnectionPool]   = lower(pool)
      val a: $[PooledConnection] = $(p)(_.acquire).allocate
      val b: $[PooledConnection] = $(p)(_.acquire).allocate
      val aId                    = $(a)(_.id)
      val bId                    = $(b)(_.id)
      println(s"  [Parallel] Using connections $aId and $bId")
      $(a)(_.execute("UPDATE table_a SET x = 1"))
      $(b)(_.execute("UPDATE table_b SET y = 2"))
      ()
    }
    println()

    println("[App] All work complete, exiting app scope...")
  }

  println("\n=== Example Complete ===")
  println("\nKey insight: pool.acquire returns Resource[PooledConnection],")
  println("forcing proper scoped allocation and automatic release.")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/ConnectionPoolExample.scala))

Run this example with:

```bash
cd scope-examples && sbt "runMain scope.examples.connectionPoolExample"
```

### Transactional Resource Management

This example demonstrates combining Resource with transaction boundaries. Shows how to manage resources (connections, transactions) that must be coordinated across scopes with proper rollback on failure:

```scala title="scope-examples/src/main/scala/scope/examples/TransactionBoundaryExample.scala" 
package scope.examples

/**
 * Transaction Boundary Example
 *
 * Demonstrates nested scopes and resource-returning methods for database
 * transaction management.
 *
 * Key patterns shown:
 *   - '''Resource-returning methods''': `beginTransaction` returns
 *     `Resource[DbTransaction]`
 *   - '''Nested scopes''': Transactions live in child scopes of the connection
 *   - '''Automatic cleanup''': Uncommitted transactions auto-rollback on scope
 *     exit
 *   - '''LIFO ordering''': Transaction closes before connection
 */
object TransactionBoundaryExample {

  /** Simulates a database connection that can create transactions. */
  class DbConnection(val id: String) extends AutoCloseable {
    println(s"  [DbConnection $id] Opened")

    /**
     * Begins a new transaction.
     *
     * Returns a `Resource[DbTransaction]` that must be allocated in a scope.
     * This ensures the transaction is always properly closed (with rollback if
     * not committed) when the scope exits.
     */
    def beginTransaction(txId: String): Resource[DbTransaction] =
      Resource.acquireRelease {
        new DbTransaction(this, txId)
      } { tx =>
        tx.close()
      }

    def close(): Unit =
      println(s"  [DbConnection $id] Closed")
  }

  /** Simulates an active database transaction. */
  class DbTransaction(val conn: DbConnection, val id: String) extends AutoCloseable {
    private var committed  = false
    private var rolledBack = false
    println(s"    [Tx $id] Started on connection ${conn.id}")

    def execute(sql: String): Int = {
      require(!committed && !rolledBack, s"Transaction $id already completed")
      println(s"    [Tx $id] Execute: $sql")
      sql.hashCode.abs % 100 + 1
    }

    def commit(): Unit = {
      require(!committed && !rolledBack, s"Transaction $id already completed")
      committed = true
      println(s"    [Tx $id] Committed")
    }

    def rollback(): Unit =
      if (!committed && !rolledBack) {
        rolledBack = true
        println(s"    [Tx $id] Rolled back")
      }

    def close(): Unit = {
      if (!committed && !rolledBack) {
        println(s"    [Tx $id] Auto-rollback (not committed)")
        rollback()
      }
      println(s"    [Tx $id] Closed")
    }
  }

  /** Result of transaction operations. */
  case class TxResult(success: Boolean, affectedRows: Int) derives Unscoped

  @main def runTransactionBoundaryExample(): Unit = {
    println("=== Transaction Boundary Example ===\n")
    println("Demonstrating Resource-returning beginTransaction method\n")

    Scope.global.scoped { connScope =>

      // Allocate the connection in the outer scope
      val conn: $[DbConnection] = Resource.fromAutoCloseable(new DbConnection("db-001")).allocate
      println()

      // Transaction 1: Successful insert
      println("--- Transaction 1: Insert user ---")
      val result1: TxResult =
        connScope.scoped { txScope =>

          val c: $[DbConnection]   = lower(conn)
          val tx: $[DbTransaction] = $(c)(_.beginTransaction("tx-001")).allocate
          val rows                 = $(tx)(_.execute("INSERT INTO users VALUES (1, 'Alice')"))
          $(tx)(_.commit())
          TxResult(success = true, affectedRows = rows)
        }
      println(s"  Result: $result1\n")

      // Transaction 2: Transfer funds (multiple operations)
      println("--- Transaction 2: Transfer funds ---")
      val result2: TxResult =
        connScope.scoped { txScope =>

          val c: $[DbConnection]   = lower(conn)
          val tx: $[DbTransaction] = $(c)(_.beginTransaction("tx-002")).allocate
          val rows1                = $(tx)(_.execute("UPDATE accounts SET balance = balance - 100 WHERE id = 1"))
          val rows2                = $(tx)(_.execute("UPDATE accounts SET balance = balance + 100 WHERE id = 2"))
          $(tx)(_.commit())
          TxResult(success = true, affectedRows = rows1 + rows2)
        }
      println(s"  Result: $result2\n")

      // Transaction 3: Demonstrates auto-rollback on scope exit without commit
      println("--- Transaction 3: Auto-rollback (no explicit commit) ---")
      val result3: TxResult =
        connScope.scoped { txScope =>

          val c: $[DbConnection]   = lower(conn)
          val tx: $[DbTransaction] = $(c)(_.beginTransaction("tx-003")).allocate
          $(tx)(_.execute("DELETE FROM audit_log"))
          println("    [App] Not committing - scope exit will trigger auto-rollback...")
          TxResult(success = false, affectedRows = 0)
        }
      println(s"  Result: $result3\n")

      println("--- All transactions complete, connection still open ---")
      println("--- Exiting connection scope ---")
    }

    println("\n=== Example complete ===")
    println("\nKey insight: beginTransaction() returns Resource[DbTransaction],")
    println("forcing proper scoped allocation and automatic cleanup.")
  }
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/TransactionBoundaryExample.scala))

Run this example with:

```bash
cd scope-examples && sbt "runMain scope.examples.runTransactionBoundaryExample"
```

### Multi-Layer Service Construction

This example demonstrates Resource.from macro to automatically build a complex dependency graph with multiple services. Shows automatic wiring of constructor dependencies and cleanup in correct LIFO order:

```scala title="scope-examples/src/main/scala/scope/examples/LayeredWebServiceExample.scala" 
package scope.examples

/**
 * Demonstrates auto-wiring a layered web service using
 * `Resource.from[T](wires*)`.
 *
 * The macro automatically derives wires for concrete classes (Database,
 * UserRepository, UserController) while requiring only the leaf config value to
 * be provided explicitly. Resources are cleaned up in LIFO order when the scope
 * closes.
 *
 * Layer hierarchy:
 * {{{
 *   AppConfig (leaf value via Wire)
 *       ↓
 *   Database (auto-wired, AutoCloseable)
 *       ↓
 *   UserRepository (auto-wired)
 *       ↓
 *   UserController (auto-wired, AutoCloseable)
 * }}}
 */

/** Application configuration - the leaf dependency provided via Wire(value). */
case class WebAppConfig(dbUrl: String, serverPort: Int)

/** Domain model for users. */
case class User(id: Long, name: String, email: String)

/** Database layer - acquires a connection and releases it on close. */
class WebDatabase(config: WebAppConfig) extends AutoCloseable {
  println(s"  [WebDatabase] Connecting to ${config.dbUrl}")

  def execute(sql: String): Int = {
    println(s"  [WebDatabase] Executing: $sql")
    1
  }

  def close(): Unit = println("  [WebDatabase] Connection closed")
}

/** Repository layer - provides data access using the database. */
class UserRepository(db: WebDatabase) {
  println("  [UserRepository] Initialized")

  private var nextId = 1L

  def findById(id: Long): Option[User] = {
    db.execute(s"SELECT * FROM users WHERE id = $id")
    if (id > 0) Some(User(id, "Alice", "alice@example.com")) else None
  }

  def save(user: User): Long = {
    db.execute(s"INSERT INTO users VALUES (${user.id}, '${user.name}', '${user.email}')")
    val id = nextId
    nextId += 1
    id
  }
}

/** Controller layer - handles HTTP requests using the repository. */
class UserController(repo: UserRepository) extends AutoCloseable {
  println("  [UserController] Ready to serve requests")

  def getUser(id: Long): String =
    repo.findById(id).map(u => s"User(${u.id}, ${u.name})").getOrElse("Not found")

  def createUser(name: String, email: String): String = {
    val id = repo.save(User(0, name, email))
    s"Created user with id=$id"
  }

  def close(): Unit = println("  [UserController] Shutting down")
}

/**
 * Entry point demonstrating the auto-wiring feature.
 *
 * Only `Wire(config)` is provided; the macro derives wires for Database,
 * UserRepository, and UserController from their constructors.
 */
@main def layeredWebServiceExample(): Unit = {
  val config = WebAppConfig(dbUrl = "jdbc:postgresql://localhost:5432/mydb", serverPort = 8080)

  println("=== Constructing layers (order: config → database → repository → controller) ===")

  // Resource.from auto-wires the entire dependency graph
  val controllerResource: Resource[UserController] = Resource.from[UserController](
    Wire(config)
  )

  // Allocate within a scoped block; cleanup runs on scope exit
  Scope.global.scoped { scope =>

    val controller: $[UserController] = allocate(controllerResource)

    println("\n=== Handling requests ===")
    println(s"  GET /users/1  → ${$(controller)(_.getUser(1))}")
    println(s"  POST /users   → ${$(controller)(_.createUser("Bob", "bob@example.com"))}")

    println("\n=== Scope closing (LIFO cleanup: controller → database) ===")
  }

  println("=== Done ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/LayeredWebServiceExample.scala))

```bash
cd scope-examples && sbt "runMain scope.examples.layeredWebServiceExample"
```

---

## Scope

`Scope` is a **compile-time safe resource lifecycle manager** that tags allocated values with a scope-specific type, preventing use-after-close at compile time. Each scope instance has a distinct `$[A]` type that is unique to that scope, making values from different scopes structurally incompatible. The `$` operator macro and `Unscoped` typeclass create multiple layers of compile-time protection, eliminating an entire class of lifetime bugs without runtime overhead.

`Scope`:
- Prevents resource leaks and use-after-close via compile-time type checking
- Allocates resources eagerly and runs finalizers deterministically in LIFO order
- Is purely synchronous with zero runtime overhead (scoped values erase to underlying types)

Here's the interface definition:

```scala
sealed abstract class Scope {
  type $[+A]

  def scoped[A](f: Scope.Child[this.type] => A)(using Unscoped[A]): A  // Scala 2: implicit Unscoped[A]
  def allocate[A](resource: Resource[A]): $[A]
  def allocate[A <: AutoCloseable](value: => A): $[A]
  def open(): $[Scope.OpenScope]
  def defer(f: => Unit): DeferHandle
  def lower[A](value: parent.$[A]): $[A]
  def isClosed: Boolean
  def isOwner: Boolean
}
```

## Motivation

Most resource bugs in Scala are "escape" bugs—scenarios where a resource is used outside of its intended lifetime, leading to undefined behavior, crashes, or data corruption:

- **Storing in fields:** You open a database connection and store it in a field, intending to close it in a finalizer. But if the finalizer runs before you're truly done with the connection, or if you forget to close it, the connection is silently used after closure.
- **Capturing in closures:** You create a file handle and pass it to an async framework via a callback. The callback might be invoked long after your scope has closed and the file has been released, causing the program to crash or silently read/write corrupted data.
- **Passing to untrusted code:** You pass a resource to a library function that might store a reference and use it later, outside your scope. You have no way to know when it's safe to close.
- **Mixing lifetimes:** In large codebases, it becomes unclear which scope owns which resource. A developer might use a resource in the wrong scope, or two scopes might try to close the same resource.

Scope addresses these with a *tight* design. Each design choice solves a specific problem and works together with the others:

1. **Compile-time leak prevention via type tagging** — Every scope has its own `$[A]` type, combined with the `$` macro that restricts how you can use values and the `Unscoped` typeclass that marks safe return types. Together, these prevent returning resources from their scope at compile time. No runtime wrapper objects needed.

2. **Zero runtime overhead** — Scoped values erase to the underlying type `A` at runtime (via casts). There's no boxing, no extra objects, no GC pressure. The compile-time safety is "free."

3. **Eager allocation** — Resources are acquired immediately when you call `allocate`, not deferred to some later point. This makes lifetimes predictable and your code matches your mental model.

4. **Deterministic, LIFO finalization** — Finalizers are guaranteed to run in reverse order of allocation when a scope closes. If acquisition order implies dependencies (common in resource hierarchies), cleanup order is automatically correct. Exceptions in finalizers are collected rather than stopping cleanup.

5. **Structured scopes with parent-child relationships** — Scopes form a hierarchy; children always close before parents. The `lower` operator lets you safely use parent-scoped values in children, since parent will outlive child.

If you've used `try/finally`, `Using`, or ZIO's `Scope`, this is the same problem space—but optimized for **synchronous code** with **compile-time boundaries**.

## Installation

Add the following dependency to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-scope" % "0.0.56"
```

Supported Scala versions: **2.13.x** and **3.x**.

## Quickstart

Here's a minimal example showing resource allocation, usage, and cleanup. This example introduces a canonical `Database` stub that we'll reuse throughout this guide:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

val out: String =
  Scope.global.scoped { scope =>

    val db: $[Database] =
      Resource.fromAutoCloseable(new Database).allocate

    // Safe access: the lambda parameter can only be used as a receiver
    $(db)(_.query("SELECT 1"))
  }

println(out)
```

What's happening in this code:

**Allocating resources in a scope.** When you call `Resource.fromAutoCloseable(new Database).allocate`, you're acquiring a database connection. The `allocate` method returns a **scoped value** of type `scope.$[Database]`—notice the `$` wrapper. This type is unique to the `scope` instance. You can import the scope to use the short form `$[Database]`.

**The `$` operator restricts access.** You cannot call `db.query(...)` directly on `$[Database]` because the methods are hidden at the type level. Instead, you use the `$` access operator: `$(db)(f)`, which takes a lambda. The lambda's parameter must be used only as a receiver (for method/field access), preventing accidental capture or escape.

**Safe return from scoped.** The `scoped` block returns a plain `String` (the result of `_.query("SELECT 1")`). This is safe because `String` is marked as `Unscoped`—a typeclass that says "this type is pure data, safe to leave a scope." If you tried to return `db` instead, the compiler would error.

**LIFO cleanup.** When the `scoped` block exits (normally or via exception), all finalizers run in reverse order. The database's `close()` method is registered automatically because `Database` extends `AutoCloseable`. So cleanup happens at the right time, in the right order, even if an exception occurred.

## Safety Model

Scope's compile-time safety comes from *three reinforcing layers* that work together to prevent resource leaks.

1. **Type identity per scope.** Every scope has a distinct `$[A]` type. This makes values from different scopes **structurally incompatible** at compile time, so you cannot accidentally use a resource in the wrong scope without an explicit conversion (`lower` for parent → child). For example, `scope1.$[Database]` and `scope2.$[Database]` are different types—the compiler refuses to mix them:

```scala
// does not compile:
Scope.global.scoped { scope1 =>

  val db1 = allocate(new Database)

  scope1.scoped { scope2 =>

    val x: scope2.$[Database] = db1  // Error: type mismatch
    // scope2.$[Database] is not compatible with scope1.$[Database]
  }
}
```

To safely use a parent scope's resource in a child scope, use `lower`:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

Scope.global.scoped { outer =>

  val db = allocate(new Database)

  outer.scoped { inner =>

    val dbInChild = inner.lower(db)  // ✓ Correct: retags for child scope
    $(dbInChild)(_.query("SELECT 1"))
  }
  // db is still alive here after child closes
}
```

2. **Controlled access via the `$` macro.** The `$` operator only allows using an unwrapped value as a **method/field receiver**. This prevents returning the resource, storing it in a local val/var, passing it as an argument to a function, or capturing it in a closure. The `$` macro also requires a **lambda literal** (not a method reference or variable):

```scala
// does not compile:
val f: Database => String = _.query("x")
(scope $ db)(f) // Error: "$ requires a lambda literal ..."
```

A lambda literal is an anonymous function written directly in code (e.g., `_.query("x")` or `x => x + 1`). The macro inspects the actual code you pass, so you must pass the lambda directly: `$(db)(_.query("x"))` compiles, but storing it in a variable first defeats this check. Without this restriction, you could smuggle the resource out indirectly via a stored function:

```scala
// hypothetical: if the macro didn't require a lambda literal
var leaked: Database = null

val f: Database => String = { db =>
  leaked = db  // Store the database somewhere the macro can't see
  db.query("x")
}

$(db)(f)  // Macro sees the call but can't detect the smuggling above

// After the scope closes, the resource is still accessible:
leaked.query("SELECT *")  // Use-after-close bug!
```

By requiring a lambda literal, the macro can analyze the actual code syntax. It rejects any attempt to store or capture the parameter, making smuggling impossible.

3. **Scope boundary enforcement via `Unscoped`.** A `scoped { ... }` block can only return values with an `Unscoped` instance (pure data). Resources and closures cannot escape the scope boundary at compile time. For example, trying to return a resource directly fails:

```scala
// does not compile:
Scope.global.scoped { scope =>

  val db = allocate(new Database)
  db  // Error: No given instance of Unscoped[$[Database]]
}
```

Closures over resources are also rejected:

```scala
// does not compile:
Scope.global.scoped { scope =>

  val db = allocate(new Database)
  () => db.query("SELECT 1")  // Error: No given instance of Unscoped[() => String]
  // (the closure captures db)
}
```

Only types with an `Unscoped` instance can cross the scope boundary—typically pure data:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

Scope.global.scoped { scope =>

  val db = allocate(new Database)
  $(db)(_.query("SELECT 1"))  // ✓ Correct: returns String, which is Unscoped
}
```

## Construction

### `Scope.global` — The Root Scope

`Scope.global` is the predefined root scope instance. It exists for the lifetime of your application and is the entry point for all scope-based resource management.

In `Scope.global`, the `$[A]` type is an identity type (i.e., `$[A] = A`). Finalizers registered in the global scope run on JVM shutdown via a shutdown hook. On Scala.js, global finalizers are not automatically invoked.

Use `Scope.global` to access the root scope:

```scala

val result: String = Scope.global.scoped { scope =>

  "no resources allocated"
}
```

### `Scope#scoped` — Create and Enter a Child Scope

`scoped` creates a new child scope with lexical lifetime. All resources allocated within the lambda are automatically cleaned up (LIFO) when the lambda exits, whether normally or via exception:

The lambda receives the child scope as a parameter. You can import its members to use the short form `$[A]` instead of `scope.$[A]`:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("database closed")
}

Scope.global.scoped { scope =>

  val db: $[Database] =
    Resource.fromAutoCloseable(new Database).allocate

  // Use the database within the scope
  val result = $(db)(_.query("SELECT 1"))
  result
  // db is automatically closed here (scope exits)
}
```

### `Scope#open` — Create an Unowned Child Scope

`open()` creates a child scope you explicitly close, returning an `OpenScope` handle. Unlike `scoped { }`, this allows non-lexical lifetime management:

The child scope is **unowned** (usable from any thread) but remains **linked to the parent** (if the parent closes, the child's finalizers also run). You must call `close()` to detach and finalize immediately.

This is useful for resource pools, lazy initialization, or service factories where you need to decouple resource acquisition from cleanup. Unlike `scoped { }`, which ties lifetime to a lexical block, `open()` lets you keep resources alive across function boundaries and explicit time boundaries.

Here's a practical application initialization pattern:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

// Application initialization: open resources early, return handle for later cleanup
val appResources = Scope.global.open()
val db = appResources.scope.allocate(Resource.fromAutoCloseable(new Database))

try {
  // Use database from anywhere in the application
  val result = appResources.scope.scoped { scope =>

    // Can create child scopes and use parent resources with lower()
    val dbInChild = scope.lower(db)
    $(dbInChild)(_.query("SELECT 1"))
  }

  println(s"Query result: $result")

  // ... rest of application code ...

} finally {
  // Application shutdown: explicit cleanup (decoupled from creation)
  appResources.close().orThrow()
}
```

## Core Operations

### `Scope#allocate` — Acquire a Resource

Allocates a `Resource[A]` in this scope, acquiring the underlying value immediately and registering its finalizer:

```scala
sealed abstract class Scope {
  def allocate[A](resource: Resource[A]): $[A]
  def allocate[A <: AutoCloseable](value: => A): $[A]
}
```

The first overload accepts any `Resource`. The second is a convenience for `AutoCloseable` values—their `close()` method is automatically registered as a finalizer.

If the scope is already closed, `allocate` throws `IllegalStateException`. Otherwise, the resource is acquired eagerly and its finalizer is registered to run LIFO when the scope closes:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

Scope.global.scoped { scope =>

  // Using Resource factory
  val db1: $[Database] =
    Resource.fromAutoCloseable(new Database).allocate

  // Using AutoCloseable overload (convenience)
  val db2: $[Database] = allocate(new Database)

  // Both are equivalent; use whichever is more readable
  ()
}
```

### `$` — Access a Scoped Value

The `$` operator safely accesses a scoped value by enforcing it is only used as a method/field receiver, preventing accidental capture or escape:

**Single value:** Use infix or unqualified syntax:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

Scope.global.scoped { scope =>

  val db: $[Database] = allocate(new Database)

  // Infix syntax
  val result1 = (scope $ db)(_.query("SELECT 1"))

  // Unqualified after `import scope._`
  val result2 = $(db)(_.query("SELECT 2"))

  result1 + result2
}
```

**Multiple values:** Use unqualified syntax only:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

final class Cache extends AutoCloseable {
  def key(): String = "cache_key"
  def close(): Unit = ()
}

Scope.global.scoped { scope =>

  val db: $[Database] = allocate(new Database)
  val cache: $[Cache] = allocate(new Cache)

  // Multiple values: each parameter may only be a receiver
  val result = $(db, cache)((d, c) => d.query(c.key()))
  result
}
```

The `$` macro enforces receiver-only rules at compile time:
- ✓ Allowed: `d.method()`, `d.method(c.key())` (method calls, field access)
- ✗ Rejected: `store(d)`, `() => d.method()`, `d` (returned), `{ val x = d; 1 }` (binding)

If a result type is `Unscoped[B]` (pure data), `$` auto-unwraps it to `B`. Otherwise, it returns `scope.$[B]`.

### `Scope#lower` — Use a Parent Value in a Child Scope

`lower` retagges a parent-scoped value into a child scope. This is safe because a parent scope always outlives its children:

This is useful when a child scope needs access to resources allocated in its parent:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

Scope.global.scoped { outer =>

  val db: $[Database] = allocate(new Database)

  // Create an inner scope that needs the database
  outer.scoped { inner =>

    // Retag the parent's database into the child
    val dbInChild: inner.$[Database] = inner.lower(db)

    // Now use it in the child
    $(dbInChild)(_.query("child query"))
  }
  // When inner exits, its finalizers run
  // When outer exits, db's finalizers run (still alive for the outer scope)
}
```

### `Finalizer#defer` — Register a Manual Finalizer

`defer` registers a cleanup action to run when the scope closes. It returns a `DeferHandle` that can cancel the registration:

```scala
trait Finalizer {
  def defer(f: => Unit): DeferHandle
}
```

`defer` is useful for resources that are not wrapped in `Resource`, or when you need explicit control over finalization. Here's a practical example—managing a temporary file and a logger that don't implement `AutoCloseable`:

```scala

// A logger that needs manual cleanup but doesn't implement AutoCloseable
class Logger {
  def log(msg: String): Unit = println(s"[LOG] $msg")
  def close(): Unit = println("Logger closed")
}

Scope.global.scoped { scope =>

  // Create a temporary file (not AutoCloseable from standard library)
  val tempFile = Files.createTempFile("app", ".tmp")
  defer {
    Files.deleteIfExists(tempFile)
    println(s"Temp file deleted: $tempFile")
  }

  // Create a logger (not AutoCloseable)
  val logger = new Logger
  val loggerHandle = defer(logger.close())

  // Use both resources
  logger.log("Processing file: " + tempFile)
  Files.write(tempFile, "data".getBytes())

  // If needed, cancel the finalizer and clean up manually
  val data = Files.readAllBytes(tempFile)
  logger.log(s"Read ${data.length} bytes")

  // loggerHandle.cancel()  // Would prevent auto-cleanup
}
// When the scope exits: logger closes, then temp file is deleted (LIFO order)
```

If the scope is already closed, `defer` is silently ignored (no-op). The finalizer is guaranteed to run in LIFO order with other finalizers when the scope closes.

### `Scope#isClosed` — Check If Closed

Returns whether this scope's finalizers have already run:

```scala
sealed abstract class Scope {
  def isClosed: Boolean
}
```

Once `isClosed` returns `true`, subsequent calls to `allocate`, `open`, or `$` throw `IllegalStateException`. This is checked to prevent use-after-close bugs.

Here's a practical example—a resource manager that guards against using a closed scope:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

// A service that holds and manages a scope
class DatabaseService {
  private val serviceScope = Scope.global.open()

  // Initialize database once at startup
  private val db = {
    try {
      serviceScope.scope.allocate(Resource.fromAutoCloseable(new Database))
    } catch {
      case e: IllegalStateException =>
        serviceScope.close().orThrow()
        throw e
    }
  }

  def isAvailable: Boolean = !serviceScope.scope.isClosed

  def execute(query: String): Either[String, String] = {
    if (serviceScope.scope.isClosed) {
      Left("Database service has been shut down")
    } else {
      try {
        Right(serviceScope.scope.scoped { scope =>

          val dbInChild = scope.lower(db)
          $(dbInChild)(_.query(query))
        })
      } catch {
        case e: IllegalStateException => Left(s"Service error: ${e.getMessage}")
      }
    }
  }

  def shutdown(): Unit = {
    if (!serviceScope.scope.isClosed) {
      serviceScope.close().orThrow()
      println("Service shutdown complete")
    }
  }
}

// Usage
val service = new DatabaseService
println(s"Service available: ${service.isAvailable}")

val result1 = service.execute("SELECT 1")
println(s"Query result: $result1")

service.shutdown()

// Attempting to use after shutdown is now safe
val result2 = service.execute("SELECT 2")
println(s"Query after shutdown: $result2")
```

On the JVM, `Scope.global.isClosed` returns `false` until its shutdown hook runs. On Scala.js it never closes, so `isClosed` is always `false` and finalizers registered on `Scope.global` never run. Child scopes created with `scoped { }` are closed when the block exits, while those created with `open()` remain open until you call `close()`.

### `Scope#isOwner` — Check Thread Ownership

Returns whether the calling thread is the owner of this scope:

```scala
sealed abstract class Scope {
  def isOwner: Boolean
}
```

Ownership is used to detect cross-thread scope misuse. Thread ownership rules:
- `Scope.global`: always returns `true` (any thread may use it)
- Child scopes created via `scoped { }`: returns `true` only on the thread that entered the block
- Child scopes created via `open()`: always returns `true` (unowned, usable cross-thread)

These rules describe the JVM. On Scala.js, `isOwner` returns `true` for every scope, because there is only one thread.

On the JVM, calling `scoped { }` on a scope you don't own throws `IllegalStateException` at runtime; on Scala.js the check never fires. This JVM example shows `isOwner` on the owning thread and on another thread:

```scala

Scope.global.scoped { scope =>
  // On the thread that entered scoped, isOwner is true
  assert(scope.isOwner)

  // On a different thread, isOwner returns false
  val thread = new Thread {
    override def run(): Unit = {
      assert(!scope.isOwner)
    }
  }
  thread.start()
  thread.join()
}
```

**Example 1: Thread-owned scope (scoped) — fails on worker thread**

Thread-owned scopes cannot be used to create child scopes from a different thread:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

val executor = Executors.newFixedThreadPool(1)

try {
  Scope.global.scoped { scope =>

    val db = allocate(new Database)

    // Try to create a child scope from a different thread
    val future = executor.submit { () =>
      try {
        scope.scoped { childScope =>

          val dbInChild = childScope.lower(db)
          $(dbInChild)(_.query("SELECT 1"))
        }
      } catch {
        case e: IllegalStateException => s"Error: ${e.getMessage}"
      }
    }
    println(future.get())
  }
} finally {
  executor.shutdown()
}

// Example Output:
// Error: Cannot create child scope: current thread 'pool-1-thread-1' does not own this scope (owner: 'main')
// db closed
```

**Example 2: Unowned scope (open) — works across threads**

Open scopes are unowned and usable from any thread:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

val executor = Executors.newFixedThreadPool(1)

try {
  val poolScope = Scope.global.open()
  val db = poolScope.scope.allocate(Resource.fromAutoCloseable(new Database))

  // Use the resource from a worker thread
  val future = executor.submit { () =>
    poolScope.scope.scoped { scope =>

      val dbInChild = scope.lower(db)
      $(dbInChild)(_.query("SELECT 1"))
    }
  }
  println(future.get())

  poolScope.close().orThrow()
} finally {
  executor.shutdown()
}

// Output:
// result: SELECT 1
// db closed
```

The key difference: `scoped { }` creates **owned** scopes (tied to the entering thread), while `open()` creates **unowned** scopes (usable from any thread). Choose based on whether your resources need to cross thread boundaries.

## Returning Unscoped Data from a Scope

A `scoped { }` block can only return values that have an `Unscoped` instance—that is, pure data types with no embedded resources or cleanup logic. This restriction prevents resource leaks: you cannot accidentally return a resource that would be cleaned up before you could use it.

For built-in types like `String`, `Int`, or `List[String]`, `Unscoped` instances exist automatically. However, when your custom type contains a field whose type has no predefined `Unscoped` instance (such as `java.util.Date`, a legacy Java type that is pure data but not automatically recognized), automatic derivation won't work. In such cases, you must provide an `Unscoped` instance explicitly, asserting that your type holds only pure data:

```scala

// java.util.Date has no predefined Unscoped instance, so Unscoped.derived
// won't work here — we must provide the instance explicitly
case class QueryResult(rows: List[String], count: Int, executedAt: Date)

object QueryResult {
  implicit val unscoped: Unscoped[QueryResult] = new Unscoped[QueryResult] {}
}

Scope.global.scoped { scope =>

  // ... acquire database ...
  QueryResult(List("a", "b"), 2, new Date())  // Returns safely
}
```

**Only add `Unscoped` for pure data types.** Never add it for types that hold resources (connections, streams, file handles). If you encounter the compile error [`No given instance of Unscoped[MyType]`](#no-given-instance-of-unscopedmytype--escaping-a-scope), see the compile errors section for how to fix it. For the complete API and examples, see the [Unscoped reference](./unscoped.md).

## Lexical vs Explicit Scopes

The Scope API provides two primary patterns for managing resource lifetimes. Choose `Scope#scoped` if you can write both the code that acquires and the code that releases the resource in the same expression; choose `Scope#open()` if the resource lifetime must outlive the function that creates it. Most user code should prefer `scoped` for automatic cleanup and thread safety—use `open()` only when you need manual lifetime control, such as in connection pools or DI containers.

### Lexical Scopes with `Scope#scoped`

Use `Scope#scoped` when the resource lifetime is lexically bounded. Lexical scopes are thread-owned by default, preventing accidental cross-thread access and providing automatic cleanup even on exception. This makes them safe and composable: you can nest `scoped` blocks to express hierarchical resource dependencies, and the code structure naturally matches the resource lifetime.

Here's a basic pattern showing how to acquire and use a resource within a single scope:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

Scope.global.scoped { scope =>

  val db = allocate(new Database)
  $(db)(_.query("SELECT * FROM users"))
  // db closes when scope exits
}
```

The only trade-off is that you must know the scope's lifetime upfront and cannot easily extend resource lifetime across function boundaries without returning resources themselves. For details on different allocation approaches (with `Resource.fromAutoCloseable()` or directly with `AutoCloseable`), see [Core Operations — allocate](#scopeallocate--acquire-a-resource).

#### Nesting for hierarchical resources

When resources depend on each other, nest `scoped` blocks to express the hierarchy. Parent scopes always outlive their children, so you can safely use parent resources in child scopes:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

final class Connection extends AutoCloseable {
  def close(): Unit = println("connection closed")
}

Scope.global.scoped { outerScope =>

  val db = allocate(new Database)

  // Child scope for connection
  outerScope.scoped { innerScope =>

    val conn = allocate(new Connection)
    // Use conn and db here
    // conn closes first (LIFO)
  }

  // Can still use db here
  // db closes when outerScope exits
}
```

### Explicit Scopes with `Scope#open`

Use `Scope#open()` when the resource lifetime is not lexically bounded. Open scopes are unowned (usable from any thread), which makes them suitable for patterns like connection pools, resource caches, and DI containers where resources must outlive the function that creates them. This pattern gives you full control over resource acquisition and release timing.

The trade-off is that you accept full responsibility for cleanup: forgetting to call `close()` leaves resources open, and any exception during cleanup must be explicitly handled. Here's the key pattern—returning an `OpenScope` handle from a function:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

def acquireDatabase(): Scope.OpenScope = {
  val os = Scope.global.open()
  val _ = os.scope.allocate(Resource.fromAutoCloseable(new Database))
  os
}

val handle = acquireDatabase()
try {
  // Use handle.scope as needed
  ()
} finally {
  handle.close().orThrow()
}
```

## Dependency Injection

`Scope` integrates seamlessly with `Wire` and `Resource.from` for automatic dependency injection. `Wire` describes a recipe for constructing a service and its dependencies, while `Scope` manages the resource lifetime. Together they eliminate manual dependency passing and ensure proper cleanup in LIFO order.

Here's an example using `Wire` and `Resource.from` within a scope:

```scala

final class Config(val dbUrl: String)

final class Database(config: Config) extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println(s"db closed (${config.dbUrl})")
}

final class UserService(db: Database) {
  def getUser(id: Int): String = s"user $id from ${db.query("SELECT * FROM users")}"
}

final class App(service: UserService) {
  def run(): Unit = println(service.getUser(1))
}

// Wire describes the dependency graph: App -> UserService -> Database -> Config
// Resource.from uses the Wire to automatically construct the entire graph
val config = Config("jdbc:postgres://localhost/db")
def appResource = Resource.from[App](
  Wire(config)
)

appResource.use(_.run())
// All resources (Database, App) clean up automatically in reverse order
```

For more details on `Wire` sharing strategies, resource composition, and advanced DI patterns, see the [Wire reference](./wire.md) and [Resource reference](./resource.md).

## Best Practices

### Entry point pattern — use `Resource#use` for app startup, `Scope.global.scoped` for manual scope control

For application entrypoints, prefer the lighter `Resource#use` API when you
have a root resource:

```scala

class App {
  def run(): Unit = println("running")
}

val appResource: Resource[App] = Resource(new App)

appResource.use(_.run())
```

Reach for `Scope.global.scoped` directly when you need manual control over
multiple allocations, child scopes, lowering, or explicit `$` access.

That lower-level form still looks like this:

```scala

object MyApp {
  def main(args: Array[String]): Unit = {
    Scope.global.scoped { scope =>

      // All resources acquired here
      // Automatic cleanup when main exits
    }
  }
}
```

This is your "outer boundary" for resource safety. Everything inside is protected.

### Composition — use `Resource` builders before allocation

Build resource acquisition/release logic outside the scope, then `Scope#allocate` once inside. This separates *construction* (how) from *allocation* (when), making code testable and reusable.

**Key combinators:**

- **`.map(f)`** — Transform a resource's value
- **`.flatMap(f)`** — Chain resources where the second depends on the first
- **`.zip(other)`** — Combine two independent resources

**Example: Using `.zip()` to combine independent resources:**

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

final class Cache extends AutoCloseable {
  def get(key: String): Option[String] = None
  def close(): Unit = println("cache closed")
}

// Compose outside scope — reusable across multiple applications
val dbResource = Resource.fromAutoCloseable(new Database)
val cacheResource = Resource.fromAutoCloseable(new Cache)
val appResources = dbResource.zip(cacheResource)

// Allocate once inside scope
Scope.global.scoped { scope =>

  val both = allocate(appResources)
  // Use both — cleanup happens in LIFO order (cache first, then db)
}
```

**Example: Using `.flatMap()` for dependent resources:**

```scala

final class Config(val host: String, val port: Int)

final class Database(val config: Config) extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

// Config resource must be acquired first, then database
val configResource = Resource(new Config("localhost", 5432))
val dbResource = configResource.flatMap { cfg =>
  Resource.fromAutoCloseable(new Database(cfg))
}

// Allocate the dependent chain
Scope.global.scoped { scope =>

  val db = allocate(dbResource)
  // db was initialized with config; cleanup happens in reverse order
}
```

To learn more about building and composing resources, see the [Resource reference](./resource.md).

## Runtime Errors

For every error the Scope and Wire macros report, see [How to Fix Common Scope and Wire Compile Errors](../../guides/fix-scope-and-wire-compile-errors.md).

Runtime errors occur when you violate scope rules at runtime—typically by accessing resources after the scope has already cleaned them up, or by mixing scopes across threads.

### `IllegalStateException` — accessing a closed scope

This error occurs when you attempt to acquire resources (via `allocate`, `open`, or the `$` operator) on a scope that has already closed. Every `scoped { }` block cleans up its resources as soon as the block exits, so any attempt to use the scope after that point fails. `defer` does not throw on a closed scope: it returns a no-op `DeferHandle` and ignores the finalizer.

The error message starts with one of these lines, depending on the operation:
```
Cannot allocate resource: scope is already closed.
Cannot open child scope: scope is already closed.
Cannot access scoped value: scope is already closed.
```

This usually happens when you:
- Store a scope in a field or closure and try to use it later after the enclosing `scoped` block has exited
- Accidentally pass a scope to an async operation that runs after cleanup

To avoid this, keep the scope's lifetime clear: allocate resources, use them, then let them clean up when the scope exits. If you need resources to survive longer, use `Scope.global.open()` to get a handle you can manage manually. You can also check `scope.isClosed` before attempting operations as a defensive check.

### `IllegalStateException` — cross-thread scope usage

Scopes created by `scoped { }` are owned by the thread that entered the block. Only `scoped` checks the owner: calling `scoped` on a scope from a different thread fails. `allocate`, `open`, `defer` and `$` do not check the owner, so a closure that runs on another thread can still allocate into a scope that is open. Keep allocation on the owning thread.

The error message for a `scoped` call from another thread is typically:
```
Cannot create child scope: current thread '...' does not own this scope (owner: '...')
```

This happens when you try to:
- Pass a scope to another thread and use it there
- Share a scope across multiple threads that call `scoped { }` on it

To fix this, use thread-unowned scopes when you need to share across threads. Instead of `scope.scoped { }` (which creates a thread-owned child), use `Scope.global.open()` or the scope's `open()` method directly to get an `OpenScope` handle. These unowned scopes can be safely passed and used from any thread, though you're responsible for manual cleanup via the returned handle.

## Compile Errors

For a task-oriented walkthrough of every error the Scope and Wire macros report, see [How to Fix Common Scope and Wire Compile Errors](../../guides/fix-scope-and-wire-compile-errors.md).

The following compile errors occur when `Scope` type rules are violated. All examples below use this scoping pattern (see [Quickstart](#quickstart) for full context):

```scala
Scope.global.scoped { scope =>

  val db: $[Database] = allocate(new Database)
  // ... usage or error ...
}
```

### `No given instance of Unscoped[MyType]` — escaping a scope

This error occurs when the type you return from a `scoped { }` block has no `Unscoped` instance. See [Scope boundary enforcement via `Unscoped`](#safety-model) for the full explanation of why this restriction exists.

If you write:
```scala
db  // ERROR: No given instance of Unscoped[$[Database]]
```

The compiler rejects this because `db` is a scoped resource (type `$[Database]`), not safe data. Even though you're inside the `scoped` block, the type system prevents you from returning it because it would be useless outside the scope (the resource would already be cleaned up).

To fix this, you have two options:

**Option 1: Extract data from the resource before returning**

Call a method on the resource to get pure data (strings, numbers, etc.) that are naturally `Unscoped`:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

Scope.global.scoped { scope =>

  val db = allocate(new Database)
  $(db)(_.query("data"))  // ✓ Correct: Returns String, which is Unscoped
}
```

The `String` returned by `query()` is pure data with no cleanup logic, so it can safely escape the scope.

**Option 2: Implement `Unscoped` for your custom types**

If you create custom types that hold only pure data, add an `Unscoped` instance so they can escape scopes. See the [Unscoped reference](./unscoped.md) for details and examples.

### `Scoped values may only be used as a method receiver` — macro violation

**What this means:** A scoped value (the parameter inside a `$(value)` lambda) can only be used as the receiver of a method call—the object you call `.method()` on. It cannot be passed to other functions, stored in variables, or captured in nested lambdas. This restriction prevents the resource from leaking out of its scope and being used after cleanup.

**When you hit this error:**

The macro detects several violations:

- **Passing as an argument:**
  ```scala
  $(db)(d => store(d))    // ERROR: cannot pass scoped value to a function
  $(db)(d => println(d))  // ERROR: cannot pass to println
  ```

- **Storing in a variable:**
  ```scala
  $(db)(d => {
    val conn = d           // ERROR: cannot bind to val/var
    conn.query()
  })
  ```

- **Returning the value itself:**
  ```scala
  $(db)(d => d)           // ERROR: must call a method, not return bare reference
  ```

- **Capturing in a nested lambda or closure:**
  ```scala
  $(db)(d =>
    () => d.query()       // ERROR: cannot capture in nested lambda
  )
  ```

**What works — calling methods on the parameter:**

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

Scope.global.scoped { scope =>

  val db = allocate(new Database)
  $(db)(d => d.query("SELECT * FROM users"))  // ✓ Method call on receiver
  $(db)(_.query("data"))                      // ✓ Using underscore shorthand
  $(db)(d => d.query("x").length)             // ✓ Chain method calls
  ()
}
```

If you need to transform or extract data from a resource before using it elsewhere, call a method to extract what you need:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

Scope.global.scoped { scope =>

  val db = allocate(new Database)
  $(db)(d => d.query("SELECT COUNT(*)"))  // ✓ Returns String (pure data)
  // The returned String can now be passed to other functions
}
```

**Why this restriction exists:** Scoped values are bound to a specific cleanup phase. Allowing them to escape (via arguments or closures) would let them be used after cleanup, causing crashes or data corruption. By restricting usage to method calls only, the macro ensures the resource never leaves its scope.

## Integration

Scope integrates seamlessly with ZIO Blocks' other data types for building complex resource management systems.

### Resource

`Scope` manages the lifecycle of `Resource[A]` values through the `allocate` method. A `Resource` describes how to acquire and clean up a value; `Scope` executes that plan and tracks finalizers. For comprehensive information on constructing, composing, and sharing resources, see the [Resource reference](./resource.md).

Key integration points:
- Use `Resource[A].allocate` to acquire within a scope
- Compose resources with `map`, `flatMap`, `zip`, and other combinators before allocating
- Use `Resource.shared` for multiple-use resources within a scope

### Finalizer

`Scope` extends `Finalizer`, the interface for registering cleanup actions. The `defer` method registers a finalizer that runs when the scope closes. For information on the `DeferHandle` and cancellation, see the [Finalizer reference](./finalizer.md).

Key integration points:
- `scope.defer(f)` registers a cleanup action
- `DeferHandle.cancel()` prevents a finalizer from running
- Finalizers run in LIFO order regardless of whether registered via `allocate` or `defer`

### Wire + Resource.from

For dependency injection patterns, Scope works naturally with [`Wire`](./wire.md) and [`Resource.from`](./resource.md) to build layered service architectures. Allocate resources in a parent scope, then use `lower` to pass them to child scopes as needed.

## Running the Examples

All code from this guide is available as runnable examples in the `scope-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

### Basic Database Connection Lifecycle Management

This example demonstrates how to allocate a database connection within a scope, ensure proper cleanup, and handle the connection's lifecycle safely.

```scala title="scope-examples/src/main/scala/scope/examples/DatabaseConnectionExample.scala" 
package scope.examples

/**
 * Configuration for database connection.
 *
 * @param host
 *   the database server hostname
 * @param port
 *   the database server port
 * @param database
 *   the database name to connect to
 */
final case class DbConfig(host: String, port: Int, database: String) {
  def connectionUrl: String = s"jdbc:postgresql://$host:$port/$database"
}

/**
 * Represents the result of a database query.
 *
 * @param rows
 *   the result set as a list of row maps
 */
final case class QueryResult(rows: List[Map[String, String]]) {
  def isEmpty: Boolean = rows.isEmpty
  def size: Int        = rows.size
}

/**
 * Simulates a database connection with lifecycle management.
 *
 * This class demonstrates how AutoCloseable resources integrate with ZIO Blocks
 * Scope. When allocated via `allocate(Resource(...))`, the `close()` method is
 * automatically registered as a finalizer.
 *
 * @param config
 *   the database configuration
 */
final class Database(config: DbConfig) extends AutoCloseable {
  private var connected = false

  def connect(): Unit = {
    println(s"[Database] Connecting to ${config.connectionUrl}...")
    connected = true
    println(s"[Database] Connected successfully")
  }

  def query(sql: String): QueryResult = {
    require(connected, "Database not connected")
    println(s"[Database] Executing: $sql")
    sql match {
      case s if s.contains("users") =>
        QueryResult(
          List(
            Map("id" -> "1", "name" -> "Alice"),
            Map("id" -> "2", "name" -> "Bob")
          )
        )
      case s if s.contains("orders") =>
        QueryResult(
          List(
            Map("order_id" -> "101", "user_id" -> "1", "total" -> "99.99"),
            Map("order_id" -> "102", "user_id" -> "2", "total" -> "149.50")
          )
        )
      case _ =>
        QueryResult(List(Map("result" -> "OK")))
    }
  }

  override def close(): Unit = {
    println(s"[Database] Closing connection to ${config.connectionUrl}")
    connected = false
  }
}

/**
 * Demonstrates basic resource lifecycle management with ZIO Blocks Scope.
 *
 * This example shows:
 *   - Allocating an AutoCloseable resource with automatic cleanup
 *   - Using `$(value)(f)` to access scoped values and execute queries
 *   - LIFO finalizer ordering (last allocated = first closed)
 *
 * When the scope exits, all registered finalizers run in reverse order,
 * ensuring proper cleanup even if exceptions occur.
 */
@main def runDatabaseExample(): Unit = {
  println("=== Database Connection Example ===\n")

  val config = DbConfig("localhost", 5432, "myapp")

  Scope.global.scoped { scope =>

    println("[Scope] Entering scoped region\n")

    // Allocate the database resource. Because Database extends AutoCloseable,
    // its close() method is automatically registered as a finalizer.
    val db: $[Database] = allocate(Resource {
      val database = new Database(config)
      database.connect()
      database
    })

    // Use $(value)(f) to access the scoped value and execute queries.
    $(db) { database =>
      val users = database.query("SELECT * FROM users")
      println(s"[Result] Found ${users.size} users: ${users.rows.map(_("name")).mkString(", ")}\n")

      val orders = database.query("SELECT * FROM orders WHERE status = 'pending'")
      println(s"[Result] Found ${orders.size} orders\n")

      val health = database.query("SELECT 1 AS health_check")
      println(s"[Result] Health check: ${health.rows.head("result")}\n")
    }

    println("[Scope] Exiting scoped region - finalizers will run in LIFO order")
  }

  println("\n=== Example Complete ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/DatabaseConnectionExample.scala))

```bash
cd scope-examples && sbt "runMain runDatabaseExample"
```

### Managing a Connection Pool with Multiple Allocations

This example demonstrates allocating multiple connections from a pool within the same scope and ensuring all are cleaned up correctly.

```scala title="scope-examples/src/main/scala/scope/examples/ConnectionPoolExample.scala" 
package scope.examples

/**
 * Demonstrates `Resource.Shared` with reference counting and nested resource
 * acquisition.
 *
 * This example shows a realistic connection pool pattern where:
 *   - The pool itself is a shared resource (created once, ref-counted)
 *   - Individual connections are resources that must be allocated in a scope
 *   - `pool.acquire` returns `Resource[PooledConnection]`, forcing proper
 *     scoping
 *
 * This pattern is common for database pools, HTTP client pools, and thread
 * pools.
 */

/** Configuration for the connection pool. */
final case class PoolConfig(maxConnections: Int, timeout: Long)

/**
 * A connection retrieved from the pool.
 *
 * Connections are resources - they must be released back to the pool when done.
 * This is enforced by making `acquire` return a `Resource[PooledConnection]`.
 */
final class PooledConnection(val id: Int, pool: ConnectionPool) extends AutoCloseable {
  println(s"    [Conn#$id] Acquired from pool")

  def execute(sql: String): String = {
    println(s"    [Conn#$id] Executing: $sql")
    s"Result from connection $id"
  }

  override def close(): Unit =
    pool.release(this)
}

/**
 * A connection pool that manages pooled connections.
 *
 * Key design: `acquire` returns `Resource[PooledConnection]`, not a raw
 * connection. This forces callers to allocate the connection in a scope,
 * ensuring proper release even if exceptions occur.
 */
final class ConnectionPool(config: PoolConfig) extends AutoCloseable {
  private val nextId  = new AtomicInteger(0)
  private val active  = new AtomicInteger(0)
  private val _closed = new AtomicInteger(0)

  println(s"  [Pool] Created with max ${config.maxConnections} connections")

  /**
   * Acquires a connection from the pool.
   *
   * Returns a `Resource[PooledConnection]` that must be allocated in a scope.
   * The connection is automatically released when the scope exits.
   */
  def acquire: Resource[PooledConnection] = Resource.acquireRelease {
    if (_closed.get() > 0) throw new IllegalStateException("Pool is closed")
    if (active.get() >= config.maxConnections)
      throw new IllegalStateException(s"Pool exhausted (max: ${config.maxConnections})")

    val id   = nextId.incrementAndGet()
    val conn = new PooledConnection(id, this)
    active.incrementAndGet()
    println(s"  [Pool] Active connections: ${active.get()}/${config.maxConnections}")
    conn
  } { conn =>
    conn.close()
  }

  private[examples] def release(conn: PooledConnection): Unit = {
    val count = active.decrementAndGet()
    println(s"    [Conn#${conn.id}] Released back to pool (active: $count)")
  }

  def activeConnections: Int = active.get()

  override def close(): Unit =
    if (_closed.compareAndSet(0, 1)) {
      println(s"  [Pool] *** POOL CLOSED *** (served ${nextId.get()} total connections)")
    }
}

@main def connectionPoolExample(): Unit = {
  println("=== Connection Pool with Resource-based Acquire ===\n")

  val poolConfig = PoolConfig(maxConnections = 3, timeout = 5000L)

  val poolResource: Resource[ConnectionPool] =
    Resource.fromAutoCloseable(new ConnectionPool(poolConfig))

  Scope.global.scoped { appScope =>

    println("[App] Allocating pool\n")
    val pool: $[ConnectionPool] = poolResource.allocate

    println("--- ServiceA doing work (connection scoped to this block) ---")
    appScope.scoped { workScope =>

      val p: $[ConnectionPool]   = lower(pool)
      val c: $[PooledConnection] = $(p)(_.acquire).allocate
      val result                 = $(c)(_.execute("SELECT * FROM service_a_table"))
      println(s"  [ServiceA] Got: $result")
    }
    println()

    println("--- ServiceB doing work ---")
    appScope.scoped { workScope =>

      val p: $[ConnectionPool]   = lower(pool)
      val c: $[PooledConnection] = $(p)(_.acquire).allocate
      val result                 = $(c)(_.execute("SELECT * FROM service_b_table"))
      println(s"  [ServiceB] Got: $result")
    }
    println()

    println("--- Multiple connections in same scope ---")
    appScope.scoped { workScope =>

      val p: $[ConnectionPool]   = lower(pool)
      val a: $[PooledConnection] = $(p)(_.acquire).allocate
      val b: $[PooledConnection] = $(p)(_.acquire).allocate
      val aId                    = $(a)(_.id)
      val bId                    = $(b)(_.id)
      println(s"  [Parallel] Using connections $aId and $bId")
      $(a)(_.execute("UPDATE table_a SET x = 1"))
      $(b)(_.execute("UPDATE table_b SET y = 2"))
      ()
    }
    println()

    println("[App] All work complete, exiting app scope...")
  }

  println("\n=== Example Complete ===")
  println("\nKey insight: pool.acquire returns Resource[PooledConnection],")
  println("forcing proper scoped allocation and automatic release.")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/ConnectionPoolExample.scala))

```bash
cd scope-examples && sbt "runMain scope.examples.connectionPoolExample"
```

### Handling Temporary File Resources with Automatic Cleanup

This example shows how to allocate temporary file resources and ensure they are automatically cleaned up when the scope closes, even if errors occur.

```scala title="scope-examples/src/main/scala/scope/examples/TempFileHandlingExample.scala" 
package scope.examples

/**
 * Demonstrates `scope.defer(...)` for registering manual cleanup actions.
 *
 * This example shows how to create temporary files during processing and ensure
 * they are deleted when the scope exits—even if processing fails. Deferred
 * cleanup actions run in LIFO (last-in-first-out) order.
 */

/** Represents a temporary file with basic read/write operations. */
case class TempFile(path: String) {
  private var content: String = ""

  def write(data: String): Unit = content = data
  def read(): String            = content
  def delete(): Boolean         = { println(s"  Deleting: $path"); true }
}

/** Result of processing temporary files. */
case class ProcessingResult(processedCount: Int, totalBytes: Long, errors: List[String])

/** Processes a list of temporary files and aggregates results. */
object FileProcessor {
  def process(files: List[TempFile]): ProcessingResult = {
    val totalBytes = files.map(_.read().length.toLong).sum
    ProcessingResult(processedCount = files.size, totalBytes = totalBytes, errors = Nil)
  }
}

@main def tempFileHandlingExample(): Unit = {
  println("=== Temp File Handling Example ===\n")
  println("Demonstrating scope.defer() for manual cleanup registration.\n")

  val result = Scope.global.scoped { scope =>
    // Create temp files and register cleanup via defer.
    // Cleanup runs in LIFO order: file3, file2, file1.

    val file1 = createTempFile(scope, "/tmp/data-001.tmp", "First file content")
    val file2 = createTempFile(scope, "/tmp/data-002.tmp", "Second file - more data here")
    val file3 = createTempFile(scope, "/tmp/data-003.tmp", "Third file with the most content of all")

    println("\nProcessing files...")
    val processingResult = FileProcessor.process(List(file1, file2, file3))
    println(s"Processed ${processingResult.processedCount} files, ${processingResult.totalBytes} bytes\n")

    println("Exiting scope - cleanup runs in LIFO order:")
    processingResult
  }

  println(s"\nFinal result: $result")
}

/**
 * Creates a temporary file and registers its cleanup with the scope.
 *
 * The cleanup action is registered via `defer(...)`, ensuring the file is
 * deleted when the scope closes—regardless of whether processing succeeds.
 *
 * @param s
 *   the scope to register cleanup with
 * @param path
 *   the file path
 * @param content
 *   initial content to write
 * @return
 *   the created TempFile
 */
private def createTempFile(s: Scope, path: String, content: String): TempFile = {
  val file = TempFile(path)
  file.write(content)
  println(s"Created: $path (${content.length} bytes)")

  // Register cleanup - will run when scope exits, in LIFO order
  s.defer {
    file.delete()
  }

  file
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/TempFileHandlingExample.scala))

```bash
cd scope-examples && sbt "runMain scope.examples.tempFileHandlingExample"
```

### Managing Database Transactions with Commit/Rollback Semantics

This example demonstrates managing database transactions within a scope, showing how to handle commit and rollback operations correctly.

```scala title="scope-examples/src/main/scala/scope/examples/TransactionBoundaryExample.scala" 
package scope.examples

/**
 * Transaction Boundary Example
 *
 * Demonstrates nested scopes and resource-returning methods for database
 * transaction management.
 *
 * Key patterns shown:
 *   - '''Resource-returning methods''': `beginTransaction` returns
 *     `Resource[DbTransaction]`
 *   - '''Nested scopes''': Transactions live in child scopes of the connection
 *   - '''Automatic cleanup''': Uncommitted transactions auto-rollback on scope
 *     exit
 *   - '''LIFO ordering''': Transaction closes before connection
 */
object TransactionBoundaryExample {

  /** Simulates a database connection that can create transactions. */
  class DbConnection(val id: String) extends AutoCloseable {
    println(s"  [DbConnection $id] Opened")

    /**
     * Begins a new transaction.
     *
     * Returns a `Resource[DbTransaction]` that must be allocated in a scope.
     * This ensures the transaction is always properly closed (with rollback if
     * not committed) when the scope exits.
     */
    def beginTransaction(txId: String): Resource[DbTransaction] =
      Resource.acquireRelease {
        new DbTransaction(this, txId)
      } { tx =>
        tx.close()
      }

    def close(): Unit =
      println(s"  [DbConnection $id] Closed")
  }

  /** Simulates an active database transaction. */
  class DbTransaction(val conn: DbConnection, val id: String) extends AutoCloseable {
    private var committed  = false
    private var rolledBack = false
    println(s"    [Tx $id] Started on connection ${conn.id}")

    def execute(sql: String): Int = {
      require(!committed && !rolledBack, s"Transaction $id already completed")
      println(s"    [Tx $id] Execute: $sql")
      sql.hashCode.abs % 100 + 1
    }

    def commit(): Unit = {
      require(!committed && !rolledBack, s"Transaction $id already completed")
      committed = true
      println(s"    [Tx $id] Committed")
    }

    def rollback(): Unit =
      if (!committed && !rolledBack) {
        rolledBack = true
        println(s"    [Tx $id] Rolled back")
      }

    def close(): Unit = {
      if (!committed && !rolledBack) {
        println(s"    [Tx $id] Auto-rollback (not committed)")
        rollback()
      }
      println(s"    [Tx $id] Closed")
    }
  }

  /** Result of transaction operations. */
  case class TxResult(success: Boolean, affectedRows: Int) derives Unscoped

  @main def runTransactionBoundaryExample(): Unit = {
    println("=== Transaction Boundary Example ===\n")
    println("Demonstrating Resource-returning beginTransaction method\n")

    Scope.global.scoped { connScope =>

      // Allocate the connection in the outer scope
      val conn: $[DbConnection] = Resource.fromAutoCloseable(new DbConnection("db-001")).allocate
      println()

      // Transaction 1: Successful insert
      println("--- Transaction 1: Insert user ---")
      val result1: TxResult =
        connScope.scoped { txScope =>

          val c: $[DbConnection]   = lower(conn)
          val tx: $[DbTransaction] = $(c)(_.beginTransaction("tx-001")).allocate
          val rows                 = $(tx)(_.execute("INSERT INTO users VALUES (1, 'Alice')"))
          $(tx)(_.commit())
          TxResult(success = true, affectedRows = rows)
        }
      println(s"  Result: $result1\n")

      // Transaction 2: Transfer funds (multiple operations)
      println("--- Transaction 2: Transfer funds ---")
      val result2: TxResult =
        connScope.scoped { txScope =>

          val c: $[DbConnection]   = lower(conn)
          val tx: $[DbTransaction] = $(c)(_.beginTransaction("tx-002")).allocate
          val rows1                = $(tx)(_.execute("UPDATE accounts SET balance = balance - 100 WHERE id = 1"))
          val rows2                = $(tx)(_.execute("UPDATE accounts SET balance = balance + 100 WHERE id = 2"))
          $(tx)(_.commit())
          TxResult(success = true, affectedRows = rows1 + rows2)
        }
      println(s"  Result: $result2\n")

      // Transaction 3: Demonstrates auto-rollback on scope exit without commit
      println("--- Transaction 3: Auto-rollback (no explicit commit) ---")
      val result3: TxResult =
        connScope.scoped { txScope =>

          val c: $[DbConnection]   = lower(conn)
          val tx: $[DbTransaction] = $(c)(_.beginTransaction("tx-003")).allocate
          $(tx)(_.execute("DELETE FROM audit_log"))
          println("    [App] Not committing - scope exit will trigger auto-rollback...")
          TxResult(success = false, affectedRows = 0)
        }
      println(s"  Result: $result3\n")

      println("--- All transactions complete, connection still open ---")
      println("--- Exiting connection scope ---")
    }

    println("\n=== Example complete ===")
    println("\nKey insight: beginTransaction() returns Resource[DbTransaction],")
    println("forcing proper scoped allocation and automatic cleanup.")
  }
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/TransactionBoundaryExample.scala))

```bash
cd scope-examples && sbt "runMain scope.examples.runTransactionBoundaryExample"
```

### Implementing an HTTP Client Pipeline with Request/Response Interceptors

This example shows how to build an HTTP client pipeline with interceptors for logging, authentication, and error handling, all managed within a scope.

```scala title="scope-examples/src/main/scala/scope/examples/HttpClientPipelineExample.scala" 
package scope.examples

/**
 * HTTP Client Pipeline Example
 *
 * Demonstrates using scoped values with the `$` operator for safe resource
 * access. Operations are eager with the new opaque type API.
 */

/** API configuration containing base URL and authentication credentials. */
final case class ApiConfig(baseUrl: String, apiKey: String)

/** Parsed JSON data as a simple key-value store. */
final case class ParsedData(values: Map[String, String]) derives Unscoped

/** HTTP response containing status, body, and headers. */
final case class HttpResponse(statusCode: Int, body: String, headers: Map[String, String])

/** Stateless JSON parser that converts raw JSON strings to structured data. */
object JsonParser {
  def parse(json: String): ParsedData = {
    println(s"  [JsonParser] Parsing ${json.take(50)}...")
    val entries = json.stripPrefix("{").stripSuffix("}").split(",").map(_.trim).filter(_.nonEmpty)
    val values  = entries.flatMap { entry =>
      entry.split(":").map(_.trim.stripPrefix("\"").stripSuffix("\"")) match {
        case Array(k, v) => Some(k -> v)
        case _           => None
      }
    }.toMap
    ParsedData(values)
  }
}

/**
 * HTTP client that manages a connection to an API server.
 *
 * Implements `AutoCloseable` so the scope automatically registers cleanup.
 */
final class HttpClient(config: ApiConfig) extends AutoCloseable {
  println(s"  [HttpClient] Opening connection to ${config.baseUrl}")

  def get(path: String): HttpResponse = {
    println(s"  [HttpClient] GET $path")
    HttpResponse(200, s"""{"path":"$path","data":"sample"}""", Map("X-Api-Key" -> config.apiKey))
  }

  def post(path: String, body: String): HttpResponse = {
    println(s"  [HttpClient] POST $path with body: $body")
    HttpResponse(201, s"""{"created":true,"echo":"$body"}""", Map("Content-Type" -> "application/json"))
  }

  override def close(): Unit =
    println(s"  [HttpClient] Closing connection to ${config.baseUrl}")
}

/**
 * Demonstrates using scoped values with the `$` operator.
 *
 * Key concepts:
 *   - `allocate` returns `$[A]` (scoped value)
 *   - `$(scopedValue)(f)` applies a function to the underlying value
 *   - `$` auto-unwraps to pure data when the return type is `Unscoped`
 *   - Operations are eager (zero-cost wrapper)
 */
@main def httpClientPipelineExample(): Unit = {
  println("=== HTTP Client Pipeline Example ===\n")
  val config = ApiConfig("https://api.example.com", "secret-key-123")

  Scope.global.scoped { scope =>

    // Step 1: Allocate the HTTP client (automatically cleaned up when scope closes)
    val client: $[HttpClient] = allocate(Resource[HttpClient](new HttpClient(config)))

    // Step 2: Use the client to fetch and parse data
    println("Executing requests...\n")

    // Fetch and parse users
    println("--- Fetching: users ---")
    val users: ParsedData = $(client) { c =>
      val response = c.get("/users")
      JsonParser.parse(response.body)
    }

    // Fetch and parse orders
    println("\n--- Fetching: orders ---")
    val orders: ParsedData = $(client) { c =>
      val response = c.get("/orders")
      JsonParser.parse(response.body)
    }

    // Post analytics event
    println("\n--- Posting: analytics ---")
    val analytics: ParsedData = $(client) { c =>
      val response = c.post("/analytics", """{"event":"fetch_complete"}""")
      JsonParser.parse(response.body)
    }

    // Step 3: Access all results
    println(s"\n=== Users Result ===")
    println(s"Users data: ${users.values}")
    println(s"\n=== Orders Result ===")
    println(s"Orders data: ${orders.values}")
    println(s"\n=== Analytics Result ===")
    println(s"Analytics: ${analytics.values}")
  }

  println("\n[Scope closed - HttpClient was automatically cleaned up]")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/HttpClientPipelineExample.scala))

```bash
cd scope-examples && sbt "runMain scope.examples.httpClientPipelineExample"
```

### Managing a Shared, Cached Logger Across Multiple Services

This example demonstrates allocating a logger once at the top level and sharing it across multiple services, ensuring it is properly closed when the application shuts down.

```scala title="scope-examples/src/main/scala/scope/examples/CachingSharedLoggerExample.scala" 
package scope.examples

/**
 * Demonstrates `Wire.shared` vs `Wire.unique` and diamond dependency patterns.
 *
 * Two services (ProductService, OrderService) share one Logger instance
 * (diamond pattern), but each gets its own unique Cache instance. This shows
 * how shared wires provide singleton behavior while unique wires create fresh
 * instances per injection site.
 *
 * Key concepts:
 *   - `Wire.shared[T]`: Single instance shared across all dependents (memoized)
 *   - `Wire.unique[T]`: Fresh instance created for each dependent
 *   - Diamond dependency: Multiple services depend on the same shared resource
 *   - Reference counting: Shared resources track usage and clean up when last
 *     user closes
 */
object CachingSharedLoggerExample {

  /** Tracks instantiation counts for demonstration purposes. */
  val loggerInstances = new AtomicInteger(0)
  val cacheInstances  = new AtomicInteger(0)

  /**
   * A shared logger that tracks instantiations and provides logging methods.
   * Implements AutoCloseable for proper resource cleanup.
   */
  class Logger extends AutoCloseable {
    val instanceId: Int = loggerInstances.incrementAndGet()
    println(s"  [Logger#$instanceId] Created")

    def info(msg: String): Unit  = println(s"  [Logger#$instanceId] INFO: $msg")
    def debug(msg: String): Unit = println(s"  [Logger#$instanceId] DEBUG: $msg")
    def close(): Unit            = println(s"  [Logger#$instanceId] Closed")
  }

  /**
   * A unique cache per service. Each service gets its own isolated cache
   * instance. Implements AutoCloseable for proper resource cleanup. Note: No
   * constructor params so it can be auto-wired with Wire.unique.
   */
  class Cache extends AutoCloseable {
    val instanceId: Int                    = cacheInstances.incrementAndGet()
    private var store: Map[String, String] = Map.empty
    println(s"  [Cache#$instanceId] Created")

    def get(key: String): Option[String]      = store.get(key)
    def put(key: String, value: String): Unit = store = store.updated(key, value)
    def close(): Unit                         = println(s"  [Cache#$instanceId] Closed")
  }

  /** Product service with its own cache but sharing the logger. */
  class ProductService(val logger: Logger, val cache: Cache) {
    println(s"  [ProductService] Created with Logger#${logger.instanceId} and Cache#${cache.instanceId}")

    def findProduct(id: String): String =
      cache.get(id) match {
        case Some(product) =>
          logger.debug(s"Cache hit for product $id")
          product
        case None =>
          logger.info(s"Loading product $id from database")
          val product = s"Product-$id"
          cache.put(id, product)
          product
      }
  }

  /**
   * Order service with its own cache but sharing the same logger as
   * ProductService.
   */
  class OrderService(val logger: Logger, val cache: Cache) {
    println(s"  [OrderService] Created with Logger#${logger.instanceId} and Cache#${cache.instanceId}")

    def createOrder(productId: String): String = {
      val orderId = s"ORD-${System.currentTimeMillis() % 10000}"
      cache.put(orderId, productId)
      logger.info(s"Created order $orderId for product $productId")
      orderId
    }
  }

  /** Top-level application combining both services. */
  class CachingApp(val productService: ProductService, val orderService: OrderService) extends AutoCloseable {
    def run(): Unit = {
      productService.logger.info("=== Application Started ===")
      val product = productService.findProduct("P001")
      orderService.createOrder(product)
      productService.findProduct("P001") // cache hit
    }
    def close(): Unit = println("  [CachingApp] Closed")
  }

  @main def runCachingExample(): Unit = {
    println("\n╔════════════════════════════════════════════════════════════════╗")
    println("║  Wire.shared vs Wire.unique - Diamond Dependency Example       ║")
    println("╚════════════════════════════════════════════════════════════════╝\n")

    println("Creating wires...")
    println("  - Logger: Wire.shared (singleton across all services)")
    println("  - Cache:  Wire.unique (fresh instance per service)\n")

    println("─── Resource Acquisition ───")
    Scope.global.scoped { scope =>

      val app: $[CachingApp] = allocate(
        Resource.from[CachingApp](
          Wire.shared[Logger],
          Wire.unique[Cache]
        )
      )

      println("\n─── Verification ───")
      println(s"  Logger instances created: ${loggerInstances.get()} (expected: 1)")
      println(s"  Cache instances created:  ${cacheInstances.get()} (expected: 2)")
      $(app) { a =>
        println(s"  ProductService.logger eq OrderService.logger: ${a.productService.logger eq a.orderService.logger}")
        println(s"  ProductService.cache  eq OrderService.cache:  ${a.productService.cache eq a.orderService.cache}")

        println("\n─── Running Application ───")
        a.run()
      }

      println("\n─── Scope Closing (LIFO cleanup) ───")
    }

    println("\n─── Summary ───")
    println(s"  Final Logger count: ${loggerInstances.get()} (shared = 1 instance)")
    println(s"  Final Cache count:  ${cacheInstances.get()} (unique = 2 instances)")
    println("\nDiamond pattern verified: both services received the same Logger instance.")
  }
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/CachingSharedLoggerExample.scala))

```bash
cd scope-examples && sbt "runMain scope.examples.runCachingExample"
```

### Building a Layered Web Service with Dependency Injection

This example shows how to build a multi-layered web service using Scope for dependency injection, allocating services at different layers and passing them down through child scopes.

```scala title="scope-examples/src/main/scala/scope/examples/LayeredWebServiceExample.scala" 
package scope.examples

/**
 * Demonstrates auto-wiring a layered web service using
 * `Resource.from[T](wires*)`.
 *
 * The macro automatically derives wires for concrete classes (Database,
 * UserRepository, UserController) while requiring only the leaf config value to
 * be provided explicitly. Resources are cleaned up in LIFO order when the scope
 * closes.
 *
 * Layer hierarchy:
 * {{{
 *   AppConfig (leaf value via Wire)
 *       ↓
 *   Database (auto-wired, AutoCloseable)
 *       ↓
 *   UserRepository (auto-wired)
 *       ↓
 *   UserController (auto-wired, AutoCloseable)
 * }}}
 */

/** Application configuration - the leaf dependency provided via Wire(value). */
case class WebAppConfig(dbUrl: String, serverPort: Int)

/** Domain model for users. */
case class User(id: Long, name: String, email: String)

/** Database layer - acquires a connection and releases it on close. */
class WebDatabase(config: WebAppConfig) extends AutoCloseable {
  println(s"  [WebDatabase] Connecting to ${config.dbUrl}")

  def execute(sql: String): Int = {
    println(s"  [WebDatabase] Executing: $sql")
    1
  }

  def close(): Unit = println("  [WebDatabase] Connection closed")
}

/** Repository layer - provides data access using the database. */
class UserRepository(db: WebDatabase) {
  println("  [UserRepository] Initialized")

  private var nextId = 1L

  def findById(id: Long): Option[User] = {
    db.execute(s"SELECT * FROM users WHERE id = $id")
    if (id > 0) Some(User(id, "Alice", "alice@example.com")) else None
  }

  def save(user: User): Long = {
    db.execute(s"INSERT INTO users VALUES (${user.id}, '${user.name}', '${user.email}')")
    val id = nextId
    nextId += 1
    id
  }
}

/** Controller layer - handles HTTP requests using the repository. */
class UserController(repo: UserRepository) extends AutoCloseable {
  println("  [UserController] Ready to serve requests")

  def getUser(id: Long): String =
    repo.findById(id).map(u => s"User(${u.id}, ${u.name})").getOrElse("Not found")

  def createUser(name: String, email: String): String = {
    val id = repo.save(User(0, name, email))
    s"Created user with id=$id"
  }

  def close(): Unit = println("  [UserController] Shutting down")
}

/**
 * Entry point demonstrating the auto-wiring feature.
 *
 * Only `Wire(config)` is provided; the macro derives wires for Database,
 * UserRepository, and UserController from their constructors.
 */
@main def layeredWebServiceExample(): Unit = {
  val config = WebAppConfig(dbUrl = "jdbc:postgresql://localhost:5432/mydb", serverPort = 8080)

  println("=== Constructing layers (order: config → database → repository → controller) ===")

  // Resource.from auto-wires the entire dependency graph
  val controllerResource: Resource[UserController] = Resource.from[UserController](
    Wire(config)
  )

  // Allocate within a scoped block; cleanup runs on scope exit
  Scope.global.scoped { scope =>

    val controller: $[UserController] = allocate(controllerResource)

    println("\n=== Handling requests ===")
    println(s"  GET /users/1  → ${$(controller)(_.getUser(1))}")
    println(s"  POST /users   → ${$(controller)(_.createUser("Bob", "bob@example.com"))}")

    println("\n=== Scope closing (LIFO cleanup: controller → database) ===")
  }

  println("=== Done ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/LayeredWebServiceExample.scala))

```bash
cd scope-examples && sbt "runMain scope.examples.layeredWebServiceExample"
```

### Reading Configuration from a File with Scope Management

This example demonstrates loading configuration from a file within a scope, ensuring the file handle is properly closed when no longer needed.

```scala title="scope-examples/src/main/scala/scope/examples/ConfigReaderExample.scala" 
package scope.examples

/**
 * Demonstrates the `Unscoped` marker trait behavior.
 *
 * ==Key Concepts==
 *
 *   - '''`Unscoped`''' marks pure data types that can safely escape a scope
 *   - Pure data escapes freely; resources remain scope-bound
 *
 * ==Example Scenario==
 *
 * A configuration reader produces `ConfigData` (pure, escapable data), while a
 * secret store holds resources that must remain scoped to prevent leakage.
 */

// ---------------------------------------------------------------------------
// Domain Types
// ---------------------------------------------------------------------------

/**
 * Pure configuration data that can safely escape any scope.
 *
 * By deriving `Unscoped`, we declare this type contains no resources. When
 * returned from `scoped { ... }`, the raw `ConfigData` is returned.
 */
case class ConfigData(
  appName: String,
  version: String,
  settings: Map[String, String]
) derives Unscoped

/**
 * Reads configuration files from disk.
 *
 * This is a resource (holds file handles, caches) and must be closed.
 */
class ConfigReader extends AutoCloseable {
  private var closed = false

  def readConfig(@annotation.unused path: String): ConfigData = {
    require(!closed, "ConfigReader is closed")
    ConfigData(
      appName = "MyApplication",
      version = "1.0.0",
      settings = Map(
        "database.host" -> "localhost",
        "database.port" -> "5432",
        "log.level"     -> "INFO"
      )
    )
  }

  override def close(): Unit = {
    closed = true
    println("  [ConfigReader] Closed.")
  }
}

/**
 * Manages access to application secrets.
 *
 * This resource maintains connections and caches; it should NOT have an
 * `Unscoped` instance. It cannot escape the scope.
 */
class SecretStore extends AutoCloseable {
  private var closed = false

  def getSecret(key: String): String = {
    require(!closed, "SecretStore is closed")
    s"secret-value-for-$key"
  }

  override def close(): Unit = {
    closed = true
    println("  [SecretStore] Closed.")
  }
}

// ---------------------------------------------------------------------------
// Main Example
// ---------------------------------------------------------------------------

@main def runConfigReaderExample(): Unit = {
  println("=== Unscoped Example ===\n")

  // ConfigData is Unscoped, so it escapes the scope as raw ConfigData
  val escapedConfig: ConfigData = Scope.global.scoped { scope =>

    val reader: $[ConfigReader] = allocate(Resource(new ConfigReader))

    // $(reader)(f) auto-unwraps to ConfigData (Unscoped)
    $(reader)(_.readConfig("/etc/app/config.json"))
  }

  println("Escaped config (used outside scope):")
  println(s"  App: ${escapedConfig.appName} v${escapedConfig.version}")
  escapedConfig.settings.foreach { case (k, v) => println(s"    $k = $v") }
  println()

  println("SecretStore stays scoped:")
  Scope.global.scoped { scope =>

    val secrets: $[SecretStore] = allocate(Resource(new SecretStore))

    $(secrets) { s =>
      val dbPassword = s.getSecret("database.password")
      println(s"  Retrieved secret: $dbPassword")
    }
    ()
  }
  println("\n=== Example Complete ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/ConfigReaderExample.scala))

```bash
cd scope-examples && sbt "runMain scope.examples.runConfigReaderExample"
```

### Implementing a Plugin Architecture with Automatic Resource Discovery

This example shows how to build a plugin system that discovers and loads plugins dynamically, managing their lifecycle with scopes.

```scala title="scope-examples/src/main/scala/scope/examples/PluginArchitectureExample.scala" 
package scope.examples

/**
 * Plugin Architecture Example — Trait Injection via Subtype Wires
 *
 * Demonstrates how abstract trait dependencies are resolved via concrete
 * implementation wires using subtype resolution. This pattern enables:
 *   - Clean interface/implementation separation
 *   - Easy swapping of implementations (e.g., Stripe vs PayPal)
 *   - Compile-time verified dependency graphs
 */

/** Configuration for payment gateway connections. */
final case class GatewayConfig(apiKey: String, merchantId: String)

/** Result of a payment operation. */
final case class PaymentResult(transactionId: String, success: Boolean, message: String)

/**
 * Abstract payment gateway interface.
 *
 * Services depend on this trait, not concrete implementations.
 */
trait PaymentGateway {
  def charge(amount: BigDecimal, currency: String): PaymentResult
  def refund(transactionId: String): PaymentResult
}

/**
 * Stripe implementation of [[PaymentGateway]].
 *
 * When wired via `Wire.shared[StripeGateway]`, this satisfies any
 * `PaymentGateway` dependency through subtype resolution.
 */
final class StripeGateway(config: GatewayConfig) extends PaymentGateway with AutoCloseable {
  println(s"[Stripe] Connected with merchant ${config.merchantId}")

  def charge(amount: BigDecimal, currency: String): PaymentResult = {
    val txId = s"stripe_${System.nanoTime()}"
    PaymentResult(txId, success = true, s"Charged $currency $amount via Stripe")
  }

  def refund(transactionId: String): PaymentResult =
    PaymentResult(transactionId, success = true, s"Refunded $transactionId via Stripe")

  def close(): Unit = println("[Stripe] Connection closed")
}

/** PayPal implementation — demonstrates swappability. */
final class PayPalGateway(config: GatewayConfig) extends PaymentGateway with AutoCloseable {
  println(s"[PayPal] Connected with merchant ${config.merchantId}")

  def charge(amount: BigDecimal, currency: String): PaymentResult = {
    val txId = s"paypal_${System.nanoTime()}"
    PaymentResult(txId, success = true, s"Charged $currency $amount via PayPal")
  }

  def refund(transactionId: String): PaymentResult =
    PaymentResult(transactionId, success = true, s"Refunded $transactionId via PayPal")

  def close(): Unit = println("[PayPal] Connection closed")
}

/**
 * Checkout service that depends on the abstract [[PaymentGateway]] trait.
 *
 * This service is unaware of which gateway implementation is injected.
 */
final class CheckoutService(gateway: PaymentGateway) extends AutoCloseable {
  def processOrder(orderId: String, amount: BigDecimal): PaymentResult = {
    println(s"[Checkout] Processing order $orderId")
    gateway.charge(amount, "USD")
  }

  def close(): Unit = println("[Checkout] Service shutdown")
}

@main def pluginArchitectureExample(): Unit = {
  val config = GatewayConfig(apiKey = "sk_test_xxx", merchantId = "acme_corp")

  println("=== Using Stripe Gateway ===")
  val stripeResource: Resource[CheckoutService] = Resource.from[CheckoutService](
    Wire(config),
    Wire.shared[StripeGateway] // Satisfies PaymentGateway via subtyping
  )

  Scope.global.scoped { scope =>

    val checkout: $[CheckoutService] = allocate(stripeResource)
    $(checkout) { c =>
      val result = c.processOrder("ORD-001", BigDecimal("99.99"))
      println(s"Result: ${result.message}")
    }
    ()
  }

  println("\n=== Using PayPal Gateway ===")
  val paypalResource: Resource[CheckoutService] = Resource.from[CheckoutService](
    Wire(config),
    Wire.shared[PayPalGateway] // Swap to PayPal — no other changes needed
  )

  Scope.global.scoped { scope =>

    val checkout: $[CheckoutService] = allocate(paypalResource)
    $(checkout) { c =>
      val result = c.processOrder("ORD-002", BigDecimal("149.99"))
      println(s"Result: ${result.message}")
    }
    ()
  }
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/PluginArchitectureExample.scala))

```bash
cd scope-examples && sbt "runMain scope.examples.pluginArchitectureExample"
```

### Demonstrating Thread Ownership Enforcement in Scope Hierarchies

This example demonstrates how Scope enforces thread ownership, preventing cross-thread scope misuse and illustrating the difference between owned and unowned scopes.

```scala title="scope-examples/src/main/scala/scope/examples/ThreadOwnershipExample.scala" 
package scope.examples

/**
 * Simulates a stateful resource that tracks which thread owns it.
 *
 * @param name
 *   the resource name
 */
final class ThreadAwareResource(val name: String) extends AutoCloseable {
  private val createdThread = Thread.currentThread()

  def getInfo: String = {
    val currentThread = Thread.currentThread()
    val owner         = createdThread.getName
    val current       = currentThread.getName
    if (createdThread eq currentThread) {
      s"[$name] Safe: owned by '$owner', accessed by '$current' (same thread)"
    } else {
      s"[$name] WARNING: owned by '$owner', accessed by '$current' (different thread!)"
    }
  }

  override def close(): Unit =
    println(s"[$name] Closing resource (was created by ${createdThread.getName})")
}

/**
 * Demonstrates thread ownership enforcement in ZIO Blocks Scope.
 *
 * This example shows:
 *   - Scope.global: `isOwner` always true; any thread can create children from
 *     it
 *   - Scope.Child: captures the creating thread; `isOwner` checks
 *     `Thread.currentThread() eq owner`
 *   - Scope.open(): creates an unowned child scope; `isOwner` always true from
 *     any thread
 *   - Calling `scoped` on a Scope.Child from a different thread throws
 *     IllegalStateException
 *
 * Thread ownership prevents accidentally passing a scope to another thread and
 * using it there, which would violate structured concurrency guarantees.
 */
@main def runThreadOwnershipExample(): Unit = {
  println("=== Thread Ownership Example ===\n")

  // === Part 1: Single-thread usage (CORRECT) ===
  println("--- Part 1: Single-thread usage (correct) ---\n")

  Scope.global.scoped { scope =>
    val currentThread = Thread.currentThread().getName
    println(s"[Main] Entered scope on thread: $currentThread\n")

    // Scope.Child is owned by the current thread (main)
    scope.scoped { child =>

      println(s"[Main] Created child scope on thread: $currentThread")
      println(s"[Main] Child scope isOwner: ${child.isOwner} (true only for the creating thread)")

      val res: $[ThreadAwareResource] =
        allocate(Resource(new ThreadAwareResource("SingleThreadResource")))

      $(res) { r =>
        println(s"[Main] ${r.getInfo}\n")
      }
    }

    println(s"[Main] Child scope closed, finalizers ran\n")
  }

  // === Part 2: Demonstrating Scope.open() for cross-thread usage ===
  println("--- Part 2: Unowned scope via open() (for cross-thread) ---\n")

  Scope.global.scoped { scope =>

    val mainThread = Thread.currentThread().getName
    println(s"[Main] On thread: $mainThread\n")

    // open() creates an unowned scope that any thread can use
    $(open()) { handle =>
      val childScope = handle.scope
      println(s"[Main] Created unowned scope via open()")
      println(s"[Main] Unowned scope isOwner: ${childScope.isOwner} (true from any thread)\n")

      // Now we can use this scope from a different thread
      val executor = Executors.newSingleThreadExecutor { r =>
        val t = new Thread(r)
        t.setName("worker-thread")
        t
      }

      try {
        val latch = new CountDownLatch(1)

        executor.execute { () =>
          try {
            val workerThread = Thread.currentThread().getName
            println(s"[Worker] On thread: $workerThread\n")

            // Using the unowned scope from a different thread - this works!
            childScope.scoped { workerChild =>

              println(s"[Worker] Created child of unowned scope")

              val res: $[ThreadAwareResource] =
                allocate(Resource(new ThreadAwareResource("CrossThreadResource")))

              $(res) { r =>
                println(s"[Worker] ${r.getInfo}\n")
              }

              println("[Worker] Worker scope closed")
            }
          } finally {
            latch.countDown()
          }
        }

        // Wait for worker thread to finish
        latch.await()
        println()
      } finally {
        executor.shutdown()
        // Clean up the open scope and propagate any finalizer failures
        handle.close().orThrow()
        println("[Main] Unowned scope closed\n")
      }
    }
  }

  // === Part 3: Explanation of ownership violation (what would fail) ===
  println("--- Part 3: Thread ownership violation (explanation) ---\n")
  println("""
If you tried to pass a Scope.Child to another thread and call scoped on it,
you would get an IllegalStateException:

  Scope.global.scoped { scope =>

    // This scope is owned by the main thread
    val executor = Executors.newSingleThreadExecutor()

    executor.execute { () =>
      // This would throw: Cannot create child scope: current thread does not own this scope.
      scope.scoped { child => ... }  // WRONG: scope is owned by main thread
    }
  }

Solution: Use scope.open() instead, which creates an unowned scope that
any thread can use. See Part 2 above for the correct pattern.
""")

  println("=== Example Complete ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/ThreadOwnershipExample.scala))

```bash
cd scope-examples && sbt "runMain runThreadOwnershipExample"
```

### Detecting and Demonstrating Circular Dependency Scenarios

This example shows a circular dependency between two services, which the Wire macro should reject at compile time. It does not do that in 0.0.56: a cycle makes the `Resource.from` macro crash instead of printing a cycle diagram (a known bug, tracked as #1767). The commented-out code is the shape to avoid, not a working compile-error demo.

```scala title="scope-examples/src/main/scala/scope/examples/CircularDependencyDemoExample.scala" 
package scope.examples

/**
 * Demonstrates how to break a circular dependency in ZIO Blocks Scope.
 *
 * The `Resource.from[T]` macro analyzes the dependency graph at compile time.
 * It is meant to reject circular dependencies with an error that shows the
 * cycle path, but in 0.0.56 a cycle makes the macro crash instead (a known bug,
 * #1767), so the cycle shown below is a shape to avoid.
 *
 * ==The Problem==
 * Circular dependencies (A → B → A) cannot be resolved by constructor injection
 * because neither service can be instantiated without the other already
 * existing.
 *
 * ==The Solution==
 * Break the cycle by introducing an interface (trait) that one service depends
 * on, allowing the implementation to be provided separately. This is a standard
 * Dependency Inversion Principle pattern.
 */

// ─────────────────────────────────────────────────────────────────────────────
// PROBLEMATIC: Circular Dependency (do not compile)
// ─────────────────────────────────────────────────────────────────────────────

// These classes form a cycle: ServiceA → ServiceB → ServiceA
// Uncommenting the Resource.from call below crashes the macro in 0.0.56 (#1767);
// it does not print a cycle diagram.

// class ServiceA(b: ServiceB) {
//   def greet(): String = s"A says hello, B says: ${b.respond()}"
// }
//
// class ServiceB(a: ServiceA) {
//   def respond(): String = s"B responds, A type: ${a.getClass.getSimpleName}"
// }
//
// Attempting to wire this should fail at compile time:
// val circularResource = Resource.from[ServiceA]()
//
// Intended compile error (not produced in 0.0.56):
// ┌────────────────────────────┐
// │                            ▼
//     ServiceA ──► ServiceB
// ▲                            │
// └────────────────────────────┘
//
// Break the cycle by:
//   • Introducing an interface/trait
//   • Using lazy initialization
//   • Restructuring dependencies

// ─────────────────────────────────────────────────────────────────────────────
// SOLUTION: Break the cycle with an interface
// ─────────────────────────────────────────────────────────────────────────────

/** Interface that ServiceA depends on, breaking the compile-time cycle. */
trait ServiceBApi {
  def respond(): String
}

/** Concrete implementation of ServiceA that depends only on the interface. */
class ServiceAImpl(b: ServiceBApi) {
  def greet(): String = s"A says hello, B says: ${b.respond()}"
}

/** Concrete implementation of ServiceB without any dependency on A. */
class ServiceBImpl extends ServiceBApi {
  override def respond(): String = "B responds successfully"
}

/** Application that composes both services. */
class Application(a: ServiceAImpl, @annotation.unused b: ServiceBApi) {
  def run(): String = a.greet()
}

/**
 * Demonstrates the working (non-circular) pattern.
 *
 * The dependency graph is now: Application → ServiceAImpl → ServiceBApi ↘
 * ServiceBApi
 *
 * ServiceBImpl provides ServiceBApi, and there is no cycle.
 */
@main def circularDependencyDemoExample(): Unit = {
  println("=== Circular Dependency Demo ===\n")
  println("Demonstrating how to break a dependency cycle.\n")

  Scope.global.scoped { scope =>

    println("Creating application with proper dependency structure...")

    // Wire.shared[ServiceBImpl] provides both ServiceBImpl and ServiceBApi (via subtyping)
    val app: $[Application] = allocate(
      Resource.from[Application](
        Wire.shared[ServiceBImpl]
      )
    )

    println(s"Result: ${$(app)(_.run())}")
    println("\nNo cycles in the dependency graph - application wired successfully.")
  }

  println("\n─── Key Takeaways ───")
  println("• Resource.from[T] should reject cycles at compile time (not yet in 0.0.56)")
  println("• Break cycles by introducing interfaces/traits")
  println("• The Dependency Inversion Principle resolves most cycles")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/CircularDependencyDemoExample.scala))

```bash
cd scope-examples && sbt "runMain scope.examples.circularDependencyDemoExample"
```

### Using Scope with Legacy Libraries that Don't Support Managed Resources

This example demonstrates how to integrate Scope with legacy libraries that don't natively support resource management, using wrapper resources and the `leak` escape hatch when necessary.

```scala title="scope-examples/src/main/scala/scope/examples/LegacyLibraryInteropExample.scala" 
package scope.examples

/**
 * Demonstrates `leak(scopedValue)` for third-party library interop.
 *
 * Sometimes you must pass scoped resources to legacy or third-party libraries
 * that require raw types. The `leak` escape hatch extracts the underlying
 * value, bypassing compile-time safety. Use sparingly—you assume responsibility
 * for ensuring the resource outlives its usage.
 */

// ---------------------------------------------------------------------------
// Fake classes simulating a legacy networking library
// ---------------------------------------------------------------------------

/** Configuration for establishing a socket connection. */
case class SocketConfig(host: String, port: Int)

/**
 * A managed socket that implements AutoCloseable.
 *
 * In a real scenario, this would wrap an actual network socket.
 */
class ManagedSocket(val config: SocketConfig) extends AutoCloseable {
  private var closed = false

  def send(data: Array[Byte]): Unit =
    if (closed) throw new IllegalStateException("Socket closed")
    else println(s"  [Socket] Sent ${data.length} bytes to ${config.host}:${config.port}")

  def receive(): Array[Byte] =
    if (closed) throw new IllegalStateException("Socket closed")
    else {
      println(s"  [Socket] Received response from ${config.host}:${config.port}")
      "ACK".getBytes
    }

  override def close(): Unit = {
    closed = true
    println(s"  [Socket] Connection to ${config.host}:${config.port} closed")
  }
}

/**
 * Simulates a third-party protocol handler that cannot be modified.
 *
 * This legacy library requires a raw `ManagedSocket` and does not understand
 * scoped types. This is the typical scenario where `leak` becomes necessary.
 */
object LegacyProtocolHandler {

  /**
   * Handles a connection using a proprietary protocol.
   *
   * @param socket
   *   the raw socket—must remain open for the duration of this call
   */
  def handleConnection(socket: ManagedSocket): Unit = {
    println("  [Legacy] Starting proprietary protocol handshake...")
    socket.send("HELLO".getBytes)
    val response = socket.receive()
    println(s"  [Legacy] Handshake complete: ${new String(response)}")
  }
}

// ---------------------------------------------------------------------------
// Example entry point
// ---------------------------------------------------------------------------

@main def legacyLibraryInteropExample(): Unit = {
  println("=== Legacy Library Interop Example ===\n")
  println("Demonstrating leak() for passing scoped resources to third-party code.\n")

  Scope.global.scoped { scope =>

    // Allocate the socket as a scoped resource.
    // The socket is tagged with the scope's type, preventing accidental escape.
    val scopedSocket: $[ManagedSocket] = allocate(
      Resource.fromAutoCloseable(new ManagedSocket(SocketConfig("api.example.com", 443)))
    )
    println("Allocated scoped socket.\n")

    // -------------------------------------------------------------------------
    // WARNING: leak() bypasses compile-time safety guarantees!
    //
    // After calling leak(), the compiler cannot prevent you from:
    //   - Storing the socket in a field that outlives the scope
    //   - Passing it to code that might cache or close it unexpectedly
    //   - Using it after the scope has closed
    //
    // Only use leak() when:
    //   1. The third-party API genuinely cannot accept scoped types
    //   2. You can guarantee the scope outlives all usage of the leaked value
    //   3. The third-party code won't cache or transfer ownership
    // -------------------------------------------------------------------------

    // WARNING: leak() bypasses compile-time safety — use only for third-party interop.
    // This intentionally escapes the scoped type and will emit a compiler warning.
    @nowarn("msg=.*leaked.*|.*leak.*")
    val rawSocket: ManagedSocket = leak(scopedSocket)

    println("Passing raw socket to legacy protocol handler:")
    LegacyProtocolHandler.handleConnection(rawSocket)

    println("\nScope exiting - socket will be closed automatically:")
  }

  println("\nExample complete. The socket was safely closed when the scope exited.")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/LegacyLibraryInteropExample.scala))

```bash
cd scope-examples && sbt "runMain scope.examples.legacyLibraryInteropExample"
```

### Integration Testing with Automatic Setup and Teardown

This example shows how to use Scope to manage test fixtures and resources in integration tests, ensuring automatic cleanup between test runs and proper resource finalization.

```scala title="scope-examples/src/main/scala/scope/examples/IntegrationTestHarnessExample.scala" 
package scope.examples

/**
 * Demonstrates combining DI-wired services with manually-managed test fixtures.
 *
 * This example shows a realistic integration test setup where:
 *   - Application services are wired via [[Resource.from]] (DI approach)
 *   - Test fixtures are managed with [[Resource.acquireRelease]] (manual
 *     approach)
 *   - Both resource types are properly cleaned up in LIFO order
 */
object IntegrationTestHarnessExample {

  // --- Test Infrastructure ---

  /** Configuration for the test environment. */
  case class TestConfig(dbUrl: String, serverPort: Int)

  /** Test database with lifecycle hooks for setup/teardown and data seeding. */
  class TestDatabase(val config: TestConfig) extends AutoCloseable {
    private var data = Map.empty[String, Any]

    def setup(): Unit                         = println(s"  [DB] Initialized at ${config.dbUrl}")
    def teardown(): Unit                      = { data = Map.empty; println("  [DB] Data cleared") }
    def seed(newData: Map[String, Any]): Unit = {
      data = newData; println(s"  [DB] Seeded with ${newData.size} entries")
    }
    def query(key: String): Option[Any] = data.get(key)
    def close(): Unit                   = println("  [DB] Connection closed")
  }

  /** Test HTTP server that can be started and stopped. */
  class TestServer(val config: TestConfig) extends AutoCloseable {
    val baseUrl: String = s"http://localhost:${config.serverPort}"

    def start(): Unit = println(s"  [Server] Started at $baseUrl")
    def stop(): Unit  = println("  [Server] Stopped")
    def close(): Unit = println("  [Server] Resources released")
  }

  /** Aggregates test fixtures for convenient access during tests. */
  case class TestFixture(db: TestDatabase, server: TestServer)

  // --- Application Under Test ---

  /** The application being tested; requires a database connection. */
  class AppUnderTest(val db: TestDatabase) extends AutoCloseable {
    def handleRequest(req: String): String = db.query(req).map(_.toString).getOrElse("Not found")
    def close(): Unit                      = println("  [App] Shutdown complete")
  }

  // --- Resource Definitions ---

  /**
   * Creates a manually-managed test fixture using [[Resource.acquireRelease]].
   *
   * This approach gives explicit control over setup and teardown phases, which
   * is typical for test fixtures that need initialization beyond construction.
   */
  def testFixtureResource(config: TestConfig): Resource[TestFixture] =
    Resource.acquireRelease {
      println("  [Fixture] Acquiring test fixture...")
      val db     = new TestDatabase(config)
      val server = new TestServer(config)
      db.setup()
      server.start()
      TestFixture(db, server)
    } { fixture =>
      println("  [Fixture] Releasing test fixture...")
      fixture.server.stop()
      fixture.db.teardown()
      fixture.db.close()
      fixture.server.close()
    }

  /**
   * Runs the integration test harness example.
   *
   * Demonstrates:
   *   1. Manual fixture via [[Resource.acquireRelease]] for test infrastructure
   *   2. DI-wired application via [[Resource.from]] consuming the fixture
   *   3. Proper cleanup ordering: app closes before fixtures
   */
  def run(): Unit = {
    println("=== Integration Test Harness Example ===\n")

    val config = TestConfig("jdbc:h2:mem:test", 8080)

    // Combine manual fixtures with DI-wired application
    val testHarnessResource: Resource[(TestFixture, AppUnderTest)] =
      testFixtureResource(config).flatMap { fixture =>
        // Seed test data
        fixture.db.seed(Map("user:1" -> "Alice", "user:2" -> "Bob"))

        // Wire the app using DI, injecting the fixture's database
        val appWire     = Wire.shared[AppUnderTest]
        val dbWire      = Wire(fixture.db)
        val appResource = Resource.from[AppUnderTest](appWire, dbWire)

        appResource.map(app => (fixture, app))
      }

    // Run in a scoped block - all resources cleaned up on exit
    Scope.global.scoped { scope =>

      println("Allocating resources...")
      val harness: $[(TestFixture, AppUnderTest)] = allocate(testHarnessResource)
      println()

      // Run test scenarios - access the tuple via $
      println("Running test scenarios:")
      $(harness) { h =>
        println(s"  GET user:1 -> ${h._2.handleRequest("user:1")}")
        println(s"  GET user:2 -> ${h._2.handleRequest("user:2")}")
        println(s"  GET user:3 -> ${h._2.handleRequest("user:3")}")
        println(s"  Server URL: ${h._1.server.baseUrl}")
      }
      println()

      println("Scope closing, releasing resources in LIFO order...")
    }

    println("\n=== Example Complete ===")
  }
}

@main def runIntegrationTestHarness(): Unit = IntegrationTestHarnessExample.run()
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/IntegrationTestHarnessExample.scala))

```bash
cd scope-examples && sbt "runMain scope.examples.IntegrationTestHarnessExample"
```

---

## Unscoped

`Unscoped[A]` is a marker typeclass for types that can safely escape a scope without tracking. Types with an `Unscoped` instance are considered "safe data"—they don't hold resources and can be freely extracted from a scope. Here's the definition:

```scala
trait Unscoped[A]
```

The `Unscoped` typeclass distinguishes between two categories of types:

1. **Unscoped types** (have an instance): Primitives, strings, collections, value types, and pure data. These can leave a scope without risk.
2. **Scoped types** (no instance): Resources like streams, connections, handles. These must remain tracked within a scope.

When the `$` operator is used to access a scoped value, if the result type has an `Unscoped` instance, it returns the value directly (unwrapped). Otherwise, it returns the value still wrapped in `$`.

## Motivation / Use Case

The exact problem: A `scoped` block automatically closes all resources when it exits. If you accidentally returned a resource (like a database connection or file handle) from the block, it would be closed—but you might try to use it later, causing a **use-after-close** crash. Here's an example (this would fail without `Unscoped`):

```scala

final class Database {
  def query(sql: String) = s"result: $sql"
}

// Without Unscoped constraint, this compiles (BAD):
// val db: Database = Scope.global.scoped { scope =>
//   val db = allocate(Resource(new Database()))
//   db  // BUG: returns the resource itself, not data extracted from it
// }
// db.query("SELECT 1")  // CRASH: use-after-close (scope already closed it)
```

The solution: `Unscoped` makes this a **compile error** instead of a runtime bug. When a `scoped` block returns a value, that value's type must have an `Unscoped` instance—meaning the type checker verifies you're only extracting *computed results* (like `Int`, `String`, or aggregate data), not resources themselves.

You can still extract computed results by using the `$` operator to unwrap scoped values *within* the scope:

```scala

Scope.global.scoped { scope =>

  val intValue = allocate(Resource(42))
  // Extract the Int value (not the Resource), computed inside the scope
  val n: Int = $(intValue)(x => x + 1)

  val text = allocate(Resource("hello"))
  // Extract the String value (not the Resource), computed inside the scope
  val s: String = $(text)(x => x.toUpperCase)

  (n, s)  // Tuple of pure data: safe to return
}
```

## Returning Unscoped Data from Scopes

Extract computed results that don't hold resources:

```scala

case class ProcessingResult(count: Int, elapsed: FiniteDuration)

object ProcessingResult {
  implicit val unscoped: Unscoped[ProcessingResult] = new Unscoped[ProcessingResult] {}
}

def processData(): ProcessingResult = Scope.global.scoped { scope =>

  val startTime = java.time.Instant.now()
  val input = allocate(Resource(Seq(1, 2, 3, 4, 5)))
  val count = $(input)(_.length)

  val endTime = java.time.Instant.now()
  val elapsed = java.time.Duration.between(startTime, endTime).toNanos

  ProcessingResult(count, FiniteDuration(elapsed, java.util.concurrent.TimeUnit.NANOSECONDS))
}

val result = processData()
println(result)
```

Only create instances for **pure data types** that don't hold resources. Never create instances for types that contain connections, streams, handles, or any resource-like fields.

## Predefined Instances

All built-in instances follow a simple principle: **if a type cannot hold resources, it gets an `Unscoped` instance**. Collections inherit this property from their elements — `List[Int]` is unscoped because `Int` is unscoped.

**Primitive and atomic values** (cannot hold resources by nature):
- `Int`, `Long`, `Short`, `Byte`, `Char`, `Boolean`, `Float`, `Double`, `Unit`
- `String`, `BigInt`, `BigDecimal`
- `java.util.UUID`

**Collections with conditional instances** (safe when elements/entries are unscoped):
- Sequences: `Array[A]`, `List[A]`, `Vector[A]`, `Seq[A]`, `IndexedSeq[A]`, `Iterable[A]`
- Sets: `Set[A]`
- Maps: `Map[K, V]` (when both `K` and `V` are unscoped)
- Wrappers: `Option[A]`, `Either[A, B]`, `Tuple2[A, B]` through `Tuple4[A, B, C, D]`
- ZIO types: `zio.blocks.chunk.Chunk[A]`

**Standard library time types** (immutable, cannot hold resources):
- `java.time.Instant`, `LocalDate`, `LocalTime`, `LocalDateTime`, `ZonedDateTime`, `OffsetDateTime`
- `java.time.Duration`, `Period`, `ZoneId`, `ZoneOffset`
- `scala.concurrent.duration.Duration`, `FiniteDuration`

All other types (resources, handles, connections) must be manually defined if needed.

**Scala 3 derives instances for you.** A case class or enum whose fields are all `Unscoped` gets an `Unscoped` instance automatically, so you rarely need to write one. On Scala 2, ask for an instance with `Unscoped.derived[A]` or write one by hand.

:::warning[A closed `AutoCloseable` can escape on Scala 3]
A known bug in 0.0.56: the automatic derivation does not check whether the class extends `AutoCloseable`. For example, `case class Conn(url: String) extends AutoCloseable` is `Unscoped` on Scala 3, so `Resource.fromAutoCloseable(Conn("x")).use(c => c)` compiles and returns a resource that is already closed. Return only computed results, never the resource itself, and check your own types for this.
:::

## Thread Safety

`Unscoped` instances themselves are immutable and thread-safe. However, the types they mark must be truly immutable for safe concurrent use. For example, `Array[Int]` is mutable—if shared across threads without synchronization, it could cause data races.

## Integration

- [`Scope.$`](./scope.md) — the operator that uses `Unscoped`
- [`Resource`](./resource.md) — types that provide `Unscoped` may be wrapped in resources
- [`Scope`](./scope.md) — manages the lifecycle of resources
- [How to Fix Common Scope and Wire Compile Errors](../../guides/fix-scope-and-wire-compile-errors.md) — fixing the `Unscoped` and other Scope compile errors

---

## Wire

`Wire[-In, +Out]` is a **compile-time safe recipe for constructing a service and its dependencies**. Wires describe how to construct an `Out` value given access to its dependencies via a `Context[In]` and a `Scope` for finalization:

```scala
sealed trait Wire[-In, +Out] {
  def isShared: Boolean
  def isUnique: Boolean = !isShared

  def shared: Wire.Shared[In, Out]
  def unique: Wire.Unique[In, Out]

  def toResource(deps: Context[In]): Resource[Out]
}
```

The type parameters are:
- **`In` (contravariant)**: the dependencies required to construct the service
- **`Out` (covariant)**: the service produced

Wires are the building blocks of dependency injection in Scope. They form the foundation for constructor-based dependency injection via `Resource.from[T](wires*)`.

## Overview

A `Wire` built with `Wire.shared` or `Wire.unique` is a **lazy recipe**, not an execution. It holds a construction function `(Scope, Context[In]) => Out` and a **sharing strategy** — either `Shared` (reference-counted via `Resource.shared`) or `Unique` (fresh instance per allocation). Such a Wire does nothing until you convert it to a `Resource` and allocate it within a scope. `Wire(value)` is the exception: a known bug in 0.0.56 evaluates `value` when the Wire is created, not when it is allocated.

Here's the typical flow:

```
Wire (recipe)
  ↓
Resource (lazy, composable)
  ↓
scope.allocate(...)
  ↓
$[T] (scoped value in scope)
```

## Motivation

Without Wires, building a multi-layer application requires manual dependency passing:

```scala

final case class Config(dbUrl: String)

final class Database(config: Config) extends AutoCloseable {
  def close(): Unit = println(s"closing connection to ${config.dbUrl}")
}

final class UserService(db: Database) {
  def getUser(id: Int): String = s"user $id"
}

final class App(service: UserService) {
  def run(): Unit = println(service.getUser(1))
}

// Manual wiring:
Scope.global.scoped { scope =>
  val config = Config("jdbc:postgres://localhost/db")
  val db = new Database(config)
  scope.defer(db.close())
  val service = new UserService(db)
  new App(service).run()
}
```

Manual wiring must build every object by hand and register each `close()` itself with `Scope#defer`. A value allocated through `Resource#allocate` is scoped and cannot be passed as a constructor argument, so manual wiring either builds plain values and defers each `close()`, or composes `Resource`s with `Resource#map` and `Resource#flatMap`.

With `Wire` + `Resource.from`, the macro handles the dependency graph:

```scala
Scope.global.scoped { scope =>

  val app = Resource.from[App](
    Wire(Config("jdbc:postgres://localhost/db"))
  ).allocate
  $(app)(_.run())
}
```

:::note
Write a `Resource.from` as a `def`, not a `val`. A known bug in 0.0.56: a `Resource.from[T]` stored in a `val` works in its first scope and then fails on a second use with `IllegalStateException: Cannot allocate from a destroyed shared resource`.
:::

**Benefits:**
- **Compile-time graph validation** — missing dependencies and duplicate providers are reported at compile time; a dependency cycle is not (see the note below)
- **Automatic finalization** — `AutoCloseable` resources are finalized in LIFO order
- **Sharing control** — choose which services are singletons (shared) vs fresh per allocation (unique)
- **Type-safe construction** — no stringly-typed dependency resolution

## Installation

Add the following dependency to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-scope" % "0.0.56"
```

For cross-platform (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-scope" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x.

## Construction

`Wire` provides multiple ways to create instances, ranging from automatic macro-based derivation to manual construction for custom logic. Here are the main construction methods:

### Wire.shared[T] — derive a shared wire

The `Wire.shared[T]` macro inspects `T`'s primary constructor and generates a shared wire that reuses the same instance across dependents:

```scala

final case class Config(debug: Boolean)

final class Database(config: Config) extends AutoCloseable {
  def query(sql: String): String = s"[db] $sql"
  def close(): Unit = println("database closed")
}

val wire: Wire.Shared[Config, Database] = Wire.shared[Database]

Scope.global.scoped { scope =>

  val config = Config(debug = true)
  val deps = Context[Config](config)
  val db = allocate(wire.toResource(deps))
  $(db)(_.query("SELECT 1"))
}
```

### Wire.unique[T] — derive a unique wire

Like `Wire.shared[T]`, but its resource is unique: each allocation builds a fresh instance. Use for request-scoped or per-call services. A known bug in 0.0.56: `Resource.from` does not always respect a wire's sharing. On Scala 3 it treats every explicit wire as shared, so a unique wire gives one instance to every dependent. On Scala 2 it reads the wire's static type, so a shared wire typed as plain `Wire` gives each dependent its own instance; keep shared wires typed as `Wire.Shared`. The example below uses `Wire#toResource`, where each allocation builds a fresh instance on both versions:

```scala

final class RequestHandler {
  val id = scala.util.Random.nextInt()
}

val wire: Wire.Unique[Any, RequestHandler] = Wire.unique[RequestHandler]
// wire: Unique[Any, RequestHandler] = Unique(
//   repl.MdocSession$MdocApp$$Lambda/0x0000000049cc73f0@62b97a4
// )

Scope.global.scoped { scope =>

  val deps = Context.empty
  val resource = wire.toResource(deps)

  val h1 = allocate(resource)
  val h2 = allocate(resource)

  $(h1)(_.id) != $(h2)(_.id)
}
// res3: Boolean = true
```

### Wire.apply[T] — lift a pre-existing value

Creates a shared wire that injects a value you already have. If the value is `AutoCloseable`, its `close()` method is automatically registered as a finalizer:

```scala

final case class Config(dbUrl: String)

val config = Config("jdbc:postgres://localhost/db")
val wire: Wire.Shared[Any, Config] = Wire(config)

Scope.global.scoped { scope =>

  val cfg = allocate(wire.toResource(Context.empty))
  $(cfg)(_.dbUrl)
}
```

### Wire.Shared.fromFunction — manual shared wire construction

Use this for custom construction logic when macro derivation doesn't fit. Unlike `Wire.shared[T]`, `Wire.Shared.fromFunction` does not register `close()` for an `AutoCloseable` result (a known bug in 0.0.56). Register it inside the function with `scope.defer`, on the `Scope` the function receives:

```scala

final case class Config(timeout: Int)

final class Client(config: Config) extends AutoCloseable {
  def call(): String = s"calling with timeout=${config.timeout}"
  def close(): Unit  = println("client closed")
}

val wire: Wire.Shared[Config, Client] =
  Wire.Shared.fromFunction { (scope, ctx) =>
    val config = ctx.get[Config]
    val client = new Client(config)
    scope.defer(client.close())
    client
  }

Scope.global.scoped { scope =>

  val config = Config(30)
  val deps = Context[Config](config)
  val client = allocate(wire.toResource(deps))
  $(client)(_.call())
}
```

### Wire.Unique.fromFunction — manual unique wire construction

Like `Wire.Shared.fromFunction`, but for unique wires. `Wire.Unique.fromFunction` does not register `close()` for an `AutoCloseable` result either (a known bug in 0.0.56), so register it inside the function with `scope.defer`. The example below returns a plain class, which needs no cleanup:

```scala

final class RequestContext {
  val id = scala.util.Random.nextInt()
}

val wire: Wire.Unique[Any, RequestContext] =
  Wire.Unique.fromFunction { (_, _) =>
    new RequestContext
  }
// wire: Unique[Any, RequestContext] = Unique(
//   repl.MdocSession$MdocApp$$Lambda/0x0000000049ca8438@9ca1730
// )

Scope.global.scoped { scope =>

  val deps = Context.empty
  val resource = wire.toResource(deps)

  val r1 = allocate(resource)
  val r2 = allocate(resource)

  $(r1)(_.id) != $(r2)(_.id)
}
// res6: Boolean = true
```

## Shared vs Unique

The fundamental difference is **reuse semantics**:

| Aspect | Shared | Unique |
|--------|--------|--------|
| **Construction** | `Wire.shared[T]` macro or `Wire.Shared.fromFunction` | `Wire.unique[T]` macro or `Wire.Unique.fromFunction` |
| **Resource type** | `Resource.shared[T]` (reference-counted) | `Resource.unique[T]` (fresh per call) |
| **When to use** | Singletons, expensive resources (connections, thread pools) | Request scoped, stateful per-call (request handlers) |
| **Instance reuse** | Same instance across entire dependency graph | New instance per allocation |
| **Finalization** | Runs when last referencing scope closes | Runs when each scope closes |

A known bug in 0.0.56 breaks this table inside `Resource.from`: on Scala 3 an explicit unique wire is shared across the graph, and on Scala 2 a shared wire whose static type is plain `Wire` gives each dependent its own instance. `Wire#toResource` follows the table on both versions.

In the diamond pattern (where `App` depends on both `UserService` and `OrderService`, both of which depend on `Database`), a shared wire ensures `Database` is constructed once and both services receive the same instance:

```scala

final class Database {
  val id = scala.util.Random.nextInt()
}

final class UserService(db: Database) {
  def getDbId(): Int = db.id
}

final class OrderService(db: Database) {
  def getDbId(): Int = db.id
}

final class App(userService: UserService, orderService: OrderService) {
  def check(): Boolean = {
    // With shared Database, these should be equal
    userService.getDbId() == orderService.getDbId()
  }
}

// Using shared wires for all dependencies
def resource = Resource.from[App](
  Wire.shared[Database],
  Wire.shared[UserService],
  Wire.shared[OrderService]
)

Scope.global.scoped { scope =>

  val app = resource.allocate
  $(app)(_.check())
}
// res7: Boolean = true
```

## Core Operations

The `Wire` interface provides a set of operations for inspecting and transforming wires:

### `Wire#isShared` and `Wire#isUnique`

Check the sharing strategy of a wire:

```scala
trait Wire[-In, +Out] {
  def isShared: Boolean
  def isUnique: Boolean = !isShared
}
```

Here's how to use these methods:

```scala

final class Greeter

val sharedWire = Wire.shared[Greeter]
val uniqueWire = Wire.unique[Greeter]
```

Each wire reports its strategy:

```scala
println(s"sharedWire.isShared: ${sharedWire.isShared}")
// sharedWire.isShared: true
println(s"sharedWire.isUnique: ${sharedWire.isUnique}")
// sharedWire.isUnique: false
println(s"uniqueWire.isShared: ${uniqueWire.isShared}")
// uniqueWire.isShared: false
println(s"uniqueWire.isUnique: ${uniqueWire.isUnique}")
// uniqueWire.isUnique: true
```

### `Wire#shared` and `Wire#unique` — convert between strategies

Convert a wire to the opposite strategy:

```scala
trait Wire[-In, +Out] {
  def shared: Wire.Shared[In, Out]
  def unique: Wire.Unique[In, Out]
}
```

To convert between sharing strategies:

```scala
val original = Wire.shared[Greeter]

val converted: Wire.Unique[Any, Greeter] = original.unique

// Converting back returns a new shared wire
val backToShared: Wire.Shared[Any, Greeter] = converted.shared
```

The original stays shared, the converted wire is unique, and converting back yields a shared wire again:

```scala
println(s"original.isShared: ${original.isShared}")
// original.isShared: true
println(s"converted.isUnique: ${converted.isUnique}")
// converted.isUnique: true
println(s"backToShared.isShared: ${backToShared.isShared}")
// backToShared.isShared: true
```

Calling `shared` on an already-shared wire returns `this` (identity); likewise `unique` on a unique wire returns `this`.

### `Wire#toResource` — convert a wire to a Resource

Converts the wire to a lazy `Resource` by providing the dependency context:

```scala
trait Wire[-In, +Out] {
  def toResource(deps: Context[In]): Resource[Out]
}
```

`Config` takes a `String`, so the wire needs a `Context[String]`:

```scala

final case class Config(value: String)

val wire = Wire.shared[Config]
val deps = Context[String]("hello")

val resource: Resource[Config] = wire.toResource(deps)

Scope.global.scoped { scope =>

  val cfg = allocate(resource)
  $(cfg)(_.value)
}
```

### `Wire#make` — construct directly from a wire

Directly construct a value without going through `Wire#toResource`. This is a low-level operation; prefer `allocate(wire.toResource(...))` for safety:

```scala
trait Wire.Shared[-In, +Out] {
  def make(scope: Scope, context: Context[In]): Out
}
```

Here's how to use this method:

```scala

final class Service {
  def getName: String = "service"
}

val wire = Wire.shared[Service]

Scope.global.scoped { scope =>

  val service = wire.make(scope, Context.empty)
  println(service.getName)
}
```

## Macro Derivation

When you call `Wire.shared[T]` or `Wire.unique[T]`, the macro performs these checks:

1. **Is `T` a class?** — traits and abstract classes are rejected; only concrete classes can be auto-wired.
2. **Does `T` have a primary constructor?** — the macro inspects constructor parameters to determine dependencies.
3. **Is each parameter either a dependency type or a special injected type?** — parameters of type `Scope` or `Finalizer` are recognized and injected automatically; others are looked up in the context.
4. **Does `T` extend `AutoCloseable`?** — if yes, `close()` is automatically registered as a finalizer in the scope.

Example with all three features:

```scala

final case class Config(dbUrl: String)

final class Logger(implicit finalizer: Finalizer) {
  def log(msg: String): Unit = println(msg)
}

final class Database(config: Config)(implicit scope: Scope) extends AutoCloseable {
  def connect(): Unit = {
    scope.defer(println("database connection closed"))
    println(s"connecting to ${config.dbUrl}")
  }

  def query(sql: String): String = s"result: $sql"

  def close(): Unit = println("database closed")
}

final class Service(db: Database, logger: Logger) {
  def run(): Unit = {
    logger.log(db.query("SELECT 1"))
  }
}

// Macro handles Finalizer injection, Scope injection, and AutoCloseable registration
val wire = Wire.shared[Service]
```

### What happens with subtype conflicts

If a constructor has dependencies of related types (e.g., both `FileInputStream` and `InputStream`), the macro rejects the wire because `Context` is type-indexed and cannot reliably disambiguate. For example:

```scala

final class Copier(input: InputStream, fileInput: FileInputStream)

// does not compile: the macro reports "Dependency type conflict in Copier"
// and "java.io.FileInputStream is a subtype of java.io.InputStream."
val copierWire = Wire.shared[Copier]
```

To fix this, wrap one of the types to make it distinct:

```scala

final case class FileInputWrapper(value: FileInputStream)

final class Copier(input: InputStream, fileInput: FileInputWrapper)

val copierWire = Wire.shared[Copier]
```

## Integration with Resource.from

`Wire` is designed for use with `Resource.from[T](wires*)`, which performs whole-graph dependency injection:

```scala

final case class AppConfig(dbUrl: String)

final class Database(config: AppConfig) extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = ()
}

final class Repository(db: Database) {
  def query(): String = db.query("SELECT *")
}

final class Service(repo: Repository) extends AutoCloseable {
  def run(): String = repo.query()
  def close(): Unit = ()
}

final class App(service: Service) {
  def run(): String = service.run()
}

// Provide only the leaf dependency; Resource.from derives the rest
def appResource: Resource[App] = Resource.from[App](
  Wire(AppConfig("jdbc:postgres://localhost/db"))
)

appResource.use(_.run())
```

When `Resource.from` composes wires, it respects the sharing strategy:
- **Shared wires** → reference-counted (single instance in the graph)
- **Unique wires** → fresh per allocation

A known bug in 0.0.56: on Scala 3, an explicit unique wire is also built once and shared across the graph. Scala 2 follows the wire's static type, so a shared wire stored as a plain `Wire[In, Out]` gives each dependent its own instance.

The macro reports missing dependencies and duplicate providers at compile time. For what each error looks like and how to fix it, see [How to Fix Common Scope and Wire Compile Errors](../../guides/fix-scope-and-wire-compile-errors.md).

A known bug in 0.0.56: a dependency cycle does not produce a cycle error. The `Resource.from` macro crashes instead (#1767), so do not rely on cycle detection yet.

The `AppConfig` leaf above is hardcoded, which is rarely what a real graph wants. `Config.wire[A]` produces a `Wire.Shared[ConfigSource, A]` that decodes `A` from an injected `ConfigSource`, so configuration becomes a node in the graph rather than something resolved before it — see [Config](../config/index.md). For a complete walkthrough, see [How to Feed Configuration into a Wire Dependency Graph](../../guides/feed-config-into-a-wire-graph.md).

## Comparison with Alternatives

| Feature | Wire | Manual Passing | Service Locator |
|---------|------|----------------|-----------------|
| **Type safety** | ✓ (compile-time validation) | ✓ (implicit) | ✗ (string keys) |
| **Cycle detection** | ✗ (a cycle crashes the macro in 0.0.56) | ✗ | ✗ (runtime) |
| **Sharing semantics** | ✓ (configurable) | Manual | ✓ (singleton pattern) |
| **Finalization** | ✓ (LIFO, automatic) | Manual | Manual |
| **Performance** | ~0 overhead (macro-generated) | ~0 overhead | ~1 allocation overhead |

Coming from `ZLayer` or macwire? [Coming from cats-effect Resource, ZIO Scope/ZLayer, macwire or Using](../../guides/resource-and-di-migration.md) maps `ZLayer.make`, `ZLayer.fromFunction`, `wire[A]` and `autowire` onto `Resource.from` and `Wire`.

## Running the Examples

All code from this guide is available as runnable examples in the `scope-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

Basic wire construction demonstrates how to create and use `Wire` for dependency injection. View the example source code:

```scala title="scope-examples/src/main/scala/wire/WireBasicExample.scala" 
package wire

/**
 * Demonstrates basic Wire construction patterns:
 *   - Wire.shared[T] macro derivation
 *   - Wire.apply(value) to lift a value
 *   - Converting wires to Resources
 *   - Using wires in a dependency graph
 */

final case class DbConfig(host: String, port: Int) {
  def url: String = s"jdbc:postgresql://$host:$port/db"
}

final class Database(config: DbConfig) extends AutoCloseable {
  println(s"[Database] Connecting to ${config.url}")

  def query(sql: String): String = {
    println(s"[Database] Executing: $sql")
    "result"
  }

  def close(): Unit =
    println("[Database] Connection closed")
}

final class UserService(db: Database) {
  println("[UserService] Initialized")

  def getUser(id: Int): String = {
    db.query(s"SELECT * FROM users WHERE id = $id")
    s"User(id=$id, name=Alice)"
  }
}

final class BasicApp(service: UserService) {
  def run(): Unit = {
    val user = service.getUser(1)
    println(s"[App] Got: $user")
  }
}

@main def wireBasicExample(): Unit = {
  println("=== Wire Basic Construction Example ===\n")

  // Create the dependency leaf (config) using Wire.apply
  val configWire: Wire.Shared[Any, DbConfig] =
    Wire(DbConfig("localhost", 5432))

  // Derive wires for Database and UserService using the macro
  val dbWire: Wire.Shared[DbConfig, Database] =
    Wire.shared[Database]

  val serviceWire: Wire.Shared[Database, UserService] =
    Wire.shared[UserService]

  val appWire: Wire.Shared[UserService, BasicApp] =
    Wire.shared[BasicApp]

  println("[Setup] Created all wires\n")

  // Use Resource.from to automatically compose the dependency graph
  val appResource: Resource[BasicApp] = Resource.from[BasicApp](
    configWire,
    dbWire,
    serviceWire,
    appWire
  )

  println("[Setup] Composed resource graph\n")

  // Allocate within a scope
  Scope.global.scoped { scope =>

    println("[Scope] Entering scoped region\n")

    val app: $[BasicApp] = allocate(appResource)

    println("\n[App] Running application")
    $(app)(_.run())

    println("\n[Scope] Exiting scoped region - finalizers will run")
  }

  println("\n=== Example Complete ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/wire/WireBasicExample.scala))

Run this example with:

```bash
cd scope-examples && sbt "runMain wire.wireBasicExample"
```

Comparing shared vs unique semantics shows how shared wires reuse the same instance across dependents, while unique wires create fresh instances. View the example:

```scala title="scope-examples/src/main/scala/wire/WireSharedUniqueExample.scala" 
package wire

/**
 * Demonstrates the semantic difference between shared and unique wires:
 *   - Shared wires: same instance across dependents (reference-counted)
 *   - Unique wires: fresh instance per allocation
 *
 * Uses a counter to track how many times each service is instantiated.
 */

final class Counter {
  private val count = new AtomicInteger(0)

  def next(): Int = count.incrementAndGet()

  def value: Int = count.get()
}

final class ServiceA(counter: Counter) {
  val id = counter.next()
  println(s"[ServiceA] Initialized with counter id=$id")
}

final class ServiceB(counter: Counter) {
  val id = counter.next()
  println(s"[ServiceB] Initialized with counter id=$id")
}

final class SharedDependencyApp(a: ServiceA, b: ServiceB) {
  def checkSharing(): Boolean = {
    // If Counter is shared, both services got the same instance
    val aCountId = a.id
    val bCountId = b.id
    println(s"[SharedDependencyApp] ServiceA counter id=$aCountId, ServiceB counter id=$bCountId")
    aCountId != bCountId && a.id < b.id // Sequential IDs from same Counter
  }
}

final class UniqueDependencyApp(a: ServiceA, b: ServiceB) {
  def checkUniqueness(): Boolean = {
    // If Counter is unique, services got different instances (different starting IDs)
    val aCountId = a.id
    val bCountId = b.id
    println(s"[UniqueDependencyApp] ServiceA counter id=$aCountId, ServiceB counter id=$bCountId")
    aCountId != bCountId // Different Counter instances
  }
}

@main def wireSharedUniqueExample(): Unit = {
  println("=== Wire Shared vs Unique Example ===\n")

  println("--- Test 1: Shared Counter (diamond pattern) ---\n")

  // Using a SHARED Counter: both ServiceA and ServiceB share the same Counter instance
  val sharedResource: Resource[SharedDependencyApp] = Resource.from[SharedDependencyApp](
    Wire.shared[Counter] // Shared: one instance across the graph
  )

  Scope.global.scoped { scope =>

    println("[Scope] Entering scoped region\n")

    val app      = allocate(sharedResource)
    val isShared = $(app)(_.checkSharing())

    println(s"\n[Result] Counter was shared: $isShared\n")
  }

  println("\n--- Test 2: Unique Counter ---\n")

  // Using a UNIQUE Counter: each dependency gets a fresh Counter instance
  val uniqueResource: Resource[UniqueDependencyApp] = Resource.from[UniqueDependencyApp](
    Wire.unique[Counter] // Unique: fresh instance per dependency
  )

  Scope.global.scoped { scope =>

    println("[Scope] Entering scoped region\n")

    val app      = allocate(uniqueResource)
    val isUnique = $(app)(_.checkUniqueness())

    println(s"\n[Result] Counters were unique: $isUnique\n")
  }

  println("\n=== Example Complete ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/wire/WireSharedUniqueExample.scala))

Run this example with:

```bash
cd scope-examples && sbt "runMain wire.wireSharedUniqueExample"
```

Manual wire construction demonstrates how to use `fromFunction` for custom construction logic. View the example:

```scala title="scope-examples/src/main/scala/wire/WireFromFunctionExample.scala" 
package wire

/**
 * Demonstrates manual wire construction using fromFunction:
 *   - Wire.Shared.fromFunction for custom shared wire logic
 *   - Wire.Unique.fromFunction for custom unique wire logic
 *   - Using manual wires when macro derivation doesn't fit
 *
 * This is useful for complex initialization, conditional logic, or when you
 * need control over which dependencies to use.
 */

final case class ApiKey(value: String)

final case class HttpConfig(baseUrl: String, timeout: Int)

/**
 * A custom HTTP client that uses an API key and timeout from config.
 * Demonstrates a scenario where we want custom construction logic rather than
 * simple parameter passing.
 */
final class HttpClient(config: HttpConfig, apiKey: ApiKey) extends AutoCloseable {
  println(
    s"[HttpClient] Created with baseUrl=${config.baseUrl}, timeout=${config.timeout}ms, apiKey=${apiKey.value}"
  )

  def get(path: String): String = {
    println(s"[HttpClient] GET $path with api key")
    "response"
  }

  def close(): Unit =
    println("[HttpClient] Connection pool closed")
}

/**
 * A custom authenticator that needs the API key. Demonstrates context
 * extraction in a manual wire.
 */
final class Authenticator(apiKey: ApiKey) {
  println(s"[Authenticator] Using API key: ${apiKey.value}")

  def authenticate(): Boolean = {
    println("[Authenticator] Validating API key...")
    true
  }
}

final class ManualWireApp(client: HttpClient, auth: Authenticator) {
  def run(): Unit =
    if (auth.authenticate()) {
      val response = client.get("/api/users")
      println(s"[App] Got response: $response")
    }
}

@main def wireFromFunctionExample(): Unit = {
  println("=== Wire fromFunction (Manual Construction) Example ===\n")

  // Manually create wires using fromFunction for custom logic
  // This gives full control when macro derivation isn't suitable

  val httpClientWire: Wire.Shared[HttpConfig & ApiKey, HttpClient] =
    Wire.Shared.fromFunction { (scope, ctx) =>
      // Extract both dependencies from context
      val config = ctx.get[HttpConfig]
      val apiKey = ctx.get[ApiKey]

      // Custom initialization logic
      println("[Manual] Custom HttpClient construction")
      val client = new HttpClient(config, apiKey)

      // Register custom cleanup if needed (in addition to AutoCloseable)
      scope.defer(println("[Manual] HttpClient cleanup deferred"))

      client
    }

  val authenticatorWire: Wire.Shared[ApiKey, Authenticator] =
    Wire.Shared.fromFunction { (scope, ctx) =>
      val apiKey = ctx.get[ApiKey]

      // Custom initialization logic
      println("[Manual] Custom Authenticator construction")
      val auth = new Authenticator(apiKey)

      scope.defer(println("[Manual] Authenticator cleanup deferred"))

      auth
    }

  // Provide leaf dependencies
  val configWire = Wire(HttpConfig("https://api.example.com", 30000))
  val apiKeyWire = Wire(ApiKey("secret-key-12345"))

  // Compose into the app resource
  val appResource: Resource[ManualWireApp] = Resource.from[ManualWireApp](
    configWire,
    apiKeyWire,
    httpClientWire,
    authenticatorWire
  )

  println("[Setup] Created manual wires\n")

  // Allocate and run
  Scope.global.scoped { scope =>

    println("[Scope] Entering scoped region\n")

    val app = allocate(appResource)

    println("\n[App] Running application")
    $(app)(_.run())

    println("\n[Scope] Exiting scoped region - finalizers will run")
  }

  println("\n=== Example Complete ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/wire/WireFromFunctionExample.scala))

Run this example with:

```bash
cd scope-examples && sbt "runMain wire.wireFromFunctionExample"
```

---

## Advanced Topics

## Thread Safety and Correctness

Ring buffers are **lock-free** but must be used correctly:

- **Wrong thread access causes undefined behavior**: Using `SpscRingBuffer` from multiple producer threads produces data races, silent data loss, or crashes. Always use the implementation matching your thread pattern.
- **`SpscRingBuffer#offer` and `SpscRingBuffer#take` thread contract**: The producer thread must be the sole caller of `SpscRingBuffer#offer`; the consumer thread must be the sole caller of `SpscRingBuffer#take`. They may be the same physical thread (as in single-threaded environments like Scala.js or unit tests) or different threads.
- **State queries are approximate**: Under concurrency, `SpscRingBuffer#size`, `SpscRingBuffer#isEmpty`, and `SpscRingBuffer#isFull` may stay stale by the time they return. Do not rely on them for exact synchronization — use `SpscRingBuffer#offer`'s return value for backpressure instead.
- **Null elements are forbidden**: All implementations reject `null` with `NullPointerException`. If you need to store nullable values, wrap them in `Option` or another container.

:::warning
**Critical:** Ring buffers do not enforce thread-safety at runtime. Using the wrong implementation for your thread pattern or calling methods from the wrong thread **does not throw an exception** — it silently corrupts data. Test thoroughly and document your threading contract.
:::

## Advanced Usage: Cache-Line Padding

Ring buffers use **cache-line padding** to prevent false sharing between producer and consumer indices on modern CPUs. The padding is transparent to users but enables dramatically lower latency on multi-core systems.

Each implementation pads its internal index fields (producer index, consumer index) to occupy a full cache line (128 bytes on Apple Silicon, 64 bytes on most other architectures). This ensures that when one thread reads its index, it does not invalidate the cache line holding the other thread's index, eliminating costly cache-coherency traffic.

This optimization is automatic and requires no configuration. Ring buffers are inherently more efficient than comparable Scala and Java implementations because of this padding.

## Designing With Ring Buffers

Common patterns for using ring buffers include:

### Pattern: Producer-Consumer Pipeline

Ring buffers form the backbone of producer-consumer pipelines where one or more producers generate work and one or more consumers process it:

```
┌──────────┐       ┌──────────────┐       ┌──────────┐
│Producer 1├──────>│ RingBuffer   │<──────┤Consumer 1│
│Producer 2├──────>│ (MPMC, cap=N)│<──────┤Consumer 2│
└──────────┘       └──────────────┘       └──────────┘
```

In this pattern:
- Producers call `MpmcRingBuffer#offer` and handle backpressure if `false` is returned (e.g., retry, queue internally, apply rate limiting).
- Consumers call `MpmcRingBuffer#take` in a tight loop, checking for `null` to detect empty buffers.
- Ring buffer capacity bounds memory and provides natural backpressure.

### Pattern: Batch Processing

For workloads where producers batch elements together, use `SpscRingBuffer#fill` (SPSC) or call `SpscRingBuffer#offer` in a loop:

```
Producer fills batch of N items
         ↓
   Ring Buffer (growing)
         ↓
Consumer drain()s batch of M items
         ↓
     Process batch
```

Batching reduces per-element synchronization costs.

### Pattern: Work Stealing with Multiple Consumers (MPMC)

When multiple workers consume from the same queue, use `MpmcRingBuffer`. Each worker calls `MpmcRingBuffer#take` to grab the next item atomically:

```scala

case class Task(id: Int, work: String)

val queue = MpmcRingBuffer[Task](256)

def worker(): Unit = {
  while (true) {
    val task = queue.take()
    if (task ne null) {
      println(s"Processing: ${task.work}")
    }
  }
}
```

The CAS loop in `MpmcRingBuffer#take` ensures no two workers grab the same task.

## Performance Characteristics

All ring buffer implementations provide O(1) time complexity for `offer`, `take`, `size`, `isEmpty`, and `isFull` operations.

- **SPSC** (FastFlow) — Fastest: avoids volatile reads on the fast path, minimal cache traffic
- **SPMC** — Fast: producer uses index-based checking; consumers CAS on a shared index
- **MPSC** — Fast: producers CAS on a shared index with a cached limit; consumer uses FastFlow relaxed-poll
- **MPMC** — Slightly slower: uses sequence buffer stamps for coordination; all indices use CAS

Actual performance depends on:
- **CPU cache architecture** — 64-byte vs 128-byte cache lines affect padding efficiency
- **Contention level** — high contention increases CAS failure rates and retries
- **Element size** — larger elements may affect cache locality
- **Platform** — JVM JIT warmup, Scala.js compiled code, GraalVM-generated native image

Micro-benchmark your specific workload if latency is critical.

## Integration with Other ZIO Blocks Types

Ring buffers are standalone data structures and do not depend on other ZIO Blocks types. However, they integrate well with:

- **Threading models**: Ring buffers work on raw JVM threads, virtual threads (Loom), or platform-specific threads. Pair with `ZIO.fork` or `Thread` as needed.
- **Reactive streams**: Ring buffers can back reactive sources, where producers feed a `Source` and consumers pull from it. The ring buffer provides natural backpressure via `offer`'s return value.
- **Event loops**: In game engines or event-driven systems, ring buffers connect event producers (input, network) to event dispatchers (main loop) with predictable latency.

---

## RingBuffer

Ring buffers are **fixed-size, lock-free queues** for efficiently exchanging elements between producer and consumer threads with minimal contention and cache-line effects. They use a circular array to recycle memory, eliminating garbage collection pressure from transient allocations. The `zio.blocks.ringbuffer` module provides four specialized implementations tuned for different producer/consumer thread patterns:

```scala
final class SpscRingBuffer[A <: AnyRef](val capacity: Int) {
  def offer(a: A): Boolean
  def take(): A
  def size: Int
  def isEmpty: Boolean
  def isFull: Boolean
  def drain(consumer: A => Unit, limit: Int): Int
  def fill(supplier: () => A, limit: Int): Int
}

final class SpmcRingBuffer[A <: AnyRef](val capacity: Int) {
  def offer(a: A): Boolean
  def take(): A
  def size: Int
  def isEmpty: Boolean
  def isFull: Boolean
}

final class MpscRingBuffer[A <: AnyRef](val capacity: Int) {
  def offer(a: A): Boolean
  def take(): A
  def size: Int
  def isEmpty: Boolean
  def isFull: Boolean
  def drain(consumer: A => Unit, limit: Int): Int
}

final class MpmcRingBuffer[A <: AnyRef](val capacity: Int) {
  def offer(a: A): Boolean
  def take(): A
  def size: Int
  def isEmpty: Boolean
  def isFull: Boolean
}
```

## Motivation

Building low-latency systems — trading platforms, game engines, real-time event processors — requires careful control over memory allocation and CPU cache behavior. Standard JVM collections like `LinkedList` or `ArrayDeque` are convenient but have a cost: every enqueue/dequeue pair allocates a node, triggering garbage collection pauses that can destroy millisecond-scale latencies.

A naive approach is to pre-allocate a large `Array[A]` and manually manage head/tail indices. This code shows a simplified single-threaded version:

```scala
class SimpleRingBuffer {
  // WARNING: Mutable state (vars) demonstrated here to show the problem;
  // this is NOT how ring buffers actually solve concurrency.
  private var head = 0
  private var tail = 0
  private val array = new Array[String](1024)

  def offer(x: String): Boolean = {
    if (tail - head >= array.length) false  // full
    else {
      array(tail % array.length) = x
      tail += 1
      true
    }
  }

  def take(): String = {
    if (head == tail) null  // empty
    else {
      val x = array(head % array.length)
      head += 1
      x
    }
  }
}
```

This works for single-threaded code, but introduces a critical problem under concurrency: both threads read and write `head` and `tail` without synchronization, leading to lost updates, stale reads, and silent data corruption. Adding `synchronized` blocks solves the data race but reintroduces contention and latency pauses.

Ring buffers solve both problems with **lock-free algorithms** and **cache-line padding**. A ring buffer provides:
- **No garbage collection** — reuses the same array forever
- **Lock-free access** — uses atomic compare-and-swap (CAS) for coordination, avoiding mutex contention
- **Predictable latency** — no surprise GC pauses or lock waits
- **Thread-specialized variants** — choose SPSC, MPSC, SPMC, or MPMC based on your thread pattern

This module provides four implementations tuned for maximum throughput and minimal latency across all producer/consumer combinations.

## Overview

Ring buffers are high-performance data structures for:

- **Event queues** in IO, networking, and game engines where throughput and latency matter
- **Thread-safe work queues** with bounded capacity to backpressure senders
- **Inter-thread communication** between producers and consumers with predictable latency
- **Concurrent batch processing** where producers fill elements and consumers drain them

## Why Ring Buffers

Ring buffers excel when you need:

- **Ultra-low latency** — lock-free algorithms and cache-line padding eliminate pauses from contention
- **Predictable throughput** — no GC overhead since the same memory array is reused forever
- **Bounded resources** — fixed capacity prevents runaway memory growth and enables backpressure
- **High concurrency** — multiple implementations optimized for different thread patterns (SPSC, MPMC, etc.)

How do ring buffers compare to other queue and collection types?

### Comparison with Java and Scala Alternatives

| Property                | RingBuffer (ZIO Blocks)                | `java.util.Queue` (ConcurrentLinkedQueue) | `scala.collection.concurrent.TrieMap` | Array + manual index     | Disruptor                |
|-------------------------|----------------------------------------|-------------------------------------------|---------------------------------------|--------------------------|--------------------------|
| **Allocation**          | Single upfront                         | Per-element nodes                         | Per-node allocations                  | Single upfront           | Single upfront           |
| **Lock-free**           | Yes (CAS-based)                        | Yes                                       | Yes (CASes on trie nodes)             | Yes (if single-threaded) | Yes (CAS)                |
| **GC pressure**         | Minimal (reuses slots)                 | High (node garbage)                       | High (trie node garbage)              | Minimal                  | Minimal                  |
| **Bounded**             | Fixed capacity                         | Unbounded                                 | Unbounded                             | Fixed                    | Fixed (bounded capacity) |
| **Thread patterns**     | Four variants (SPSC, MPSC, SPMC, MPMC) | Multi-producer/multi-consumer             | Multi-reader/multi-writer             | Limited (see impl)       | Multi-producer/consumer  |
| **Predictable latency** | High (no GC)                           | Medium (GC pauses)                        | Medium (GC pauses)                    | High (if no contention)  | High                     |

RingBuffer is ideal when you control both producer and consumer thread counts and want maximum performance. `java.util.Queue` is better if you need unbounded capacity; Disruptor is a comparable JVM alternative with similar guarantees.

## Installation

Add the ZIO Blocks Ring Buffer module to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-ringbuffer" % "0.0.56"
```

For Scala.js cross-platform support:

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-ringbuffer" % "0.0.56"
```

## Implementations

ZIO Blocks provides four optimized ring buffer implementations. Choose the one matching your producer/consumer thread counts:

- **SPSC** — Single Producer, Single Consumer (⚡⚡⚡ **Fastest**)
- **SPMC** — Single Producer, Multiple Consumers
- **MPSC** — Multiple Producers, Single Consumer
- **MPMC** — Multiple Producers, Multiple Consumers

Use the sidebar for the per-implementation guides and the advanced topics page covering thread safety, cache-line padding, design patterns, and performance characteristics.

---

## MPMC RingBuffer

`MpmcRingBuffer[A]` is the fully general-purpose implementation for systems with multiple producer and consumer threads. It uses the **Vyukov/Dmitry sequence-buffer algorithm**, a sophisticated lock-free design that coordinates all access through monotonically increasing indices and per-slot sequence stamps—making every slot's state self-describing at any moment.

## Algorithm

`MpmcRingBuffer` handles the hardest case: **many producers and many consumers** all accessing the same buffer at once. It uses the **Vyukov/Dmitry algorithm**, which is a lock-free MPMC queue design.

The challenge: when multiple producers might try to write to the same slot, and multiple consumers might try to read the same slot, we need a fair way to say "this slot is mine" without using locks.

**The trick: Give each slot a "ticket number" that changes in a predictable cycle.**

The algorithm runs two parallel arrays of the same length:

- `buffer[i]` — holds the actual element at position `i`
- `seqBuf[i]` — holds a *sequence stamp* at position `i`

The sequence stamps are the heart of the algorithm. Each stamp encodes the *ownership state* of its slot: who is allowed to act on it and what they are allowed to do.

On initialization, `seqBuf[i]` is set to `i`. The producer index `pIdx` and consumer index `cIdx` both start at zero. From here, every operation follows the same pattern: read the stamp, compute a single difference (`diff`), and branch on whether `diff` is zero, negative, or positive.

At any moment, slot `i` (where `i = index & mask`) is in exactly one of three states:

| stamp value                 | meaning                                          |
|-----------------------------|--------------------------------------------------|
| if `seq == pIdx`            | Slot is free — the producer at `pIdx` may write  |
| if `seq == cIdx + 1`        | Data written — the consumer at `cIdx` may read   |
| if `seq == cIdx + capacity` | Slot consumed — free for the producer's next lap |

The producer looks for `diff = seq - pIdx == 0`. The consumer looks for `diff = seq - (cIdx + 1) == 0`. In both cases, a negative diff means the other side has fallen behind (buffer full or empty), and a positive diff means another thread already claimed this slot and you should retry.

### Diagram

To see the sequence buffer in action, use this interactive stepper. The component below implements the algorithm faithfully in React. Type any label, click **Offer** to enqueue or **Take** to dequeue, and watch the trace panel show every intermediate variable — `pIdx`, `slot`, `seq`, `diff` — and the exact decision the algorithm makes from them.

<MpmcDiagram />

Click "Step Producer" and "Step Consumer" to see how the algorithm coordinates access handoff without locks.

Here is a complete walkthrough of every variable in the trace, in the order the algorithm computes them.

### `pIdx` / `cIdx` — the monotonic counters

These two numbers are the heartbeat of the entire algorithm. `pIdx` is the producer's counter and `cIdx` is the consumer's counter. They start at zero and *only ever increase* — they never wrap, never reset, never go backwards. After a million operations `pIdx` might be 1,000,000 and `cIdx` might be 999,996. The raw slot position is derived from them rather than stored directly, which is what makes the algorithm safe for multiple concurrent threads.

### `slot = idx & mask` — the circular array index

Because the buffer has a power-of-two capacity (4 in the demo), `mask = capacity - 1 = 3`, which in binary is `0011`. The bitwise AND strips everything above the lowest two bits, giving a number in the range `[0, 3]`. This is mathematically identical to `idx % capacity` but costs a single CPU instruction instead of a division. So `pIdx = 7` maps to slot `7 & 3 = 3`, and `pIdx = 8` maps back to slot `8 & 3 = 0` — that is the wrap-around.

### `seq = seqBuf[slot]` — the sequence stamp

Every slot carries its own stamp, completely independent of the other slots. The stamp is not a lock and not a boolean "occupied/free" flag — it is a number that encodes the *exact generation* of the slot. On construction `seqBuf[i] = i`, so slot 0 starts at 0, slot 1 at 1, and so on. After each write the stamp advances by 1. After each consume it advances by `capacity`. Because of this, slot 0's stamp trail across three laps looks like `0 → 1 → 4 → 5 → 8 → 9 → 12 → 13 …` — it grows forever and never repeats, which is what prevents the ABA problem entirely.

### `expected` (take only) `= cIdx + 1`

The producer, after winning its CAS and writing data, stamps the slot with `pIdx + 1`. So if a producer claimed slot 0 when `pIdx` was 4, it leaves `seqBuf[0] = 5`. The consumer that arrives with `cIdx = 4` therefore looks for `seqBuf[0] == cIdx + 1 == 5`. The `+ 1` is the handshake signal: *"a producer has finished writing here, and you are the right consumer to read it."* The offer trace does not need an `expected` row because the producer compares `seq` directly against `pIdx` (not `pIdx + 1`) — the slot is free when the stamp equals the producer index exactly.

### `diff` — the three-way decision

This is the key insight of the Vyukov algorithm. A single subtraction replaces what would otherwise be a tangle of conditional checks.

For **offer**: `diff = seq − pIdx`
- `diff == 0` — the stamp exactly matches the producer index, meaning no one has touched this slot since it was last released. The slot is yours. The thread does a CAS on `pIdx`, writes the element, then stamps `seqBuf[slot] = pIdx + 1`.
- `diff < 0` — the stamp is *behind* the producer index. This means the slot is still occupied by data from the current lap that has not been consumed yet. The buffer is full. Return `false`.
- `diff > 0` — the stamp is *ahead* of the producer index. Another producer already claimed this slot and advanced past it. Retry the loop with a fresh read of `pIdx`.

For **take**: `diff = seq − expected` where `expected = cIdx + 1`
- `diff == 0` — the stamp matches exactly what the producer left. Data is ready. CAS on `cIdx`, read the element, null out the slot for GC, stamp `seqBuf[slot] = cIdx + capacity` to release the slot for a future producer on the next lap.
- `diff < 0` — the stamp is behind what the consumer expects, meaning the producer has not finished writing yet (or has not written at all). The buffer appears empty from this consumer's perspective. Return `null`.
- `diff > 0` — another consumer already read this slot and advanced past it. Retry.

### Why all three decisions are safe without any lock

The diff check and the subsequent CAS form an atomic claim. Two producers might both read the same `pIdx` and both see `diff == 0`, but only one will win the CAS that advances `pIdx`. The loser sees the CAS fail, loops back, reads the new `pIdx`, and naturally ends up looking at the next slot. No explicit coordination between threads is ever needed — the sequence stamps and the monotonically increasing indices together make every slot's state self-describing at any point in time.

## Creating Instances

`MpmcRingBuffer[A]` is the fully general-purpose implementation supporting any number of producers and consumers. It uses the Vyukov/Dmitry algorithm with a parallel sequence buffer to coordinate access safely.

```scala
object MpmcRingBuffer {
  def apply[A <: AnyRef](capacity: Int): MpmcRingBuffer[A]
}
```

Creates a new MPMC ring buffer with the given capacity. The capacity must be a power of two >= 2 (the sequence buffer algorithm requires at least 2 slots). A capacity of 1 is valid for `SpscRingBuffer`, `SpmcRingBuffer`, and `MpscRingBuffer`, but `MpmcRingBuffer` needs at least 2 because its sequence stamp mechanism requires a second slot to distinguish the written-but-not-consumed state from the empty state.

To create an MPMC buffer:

```scala

val rb = MpmcRingBuffer[String](128)
```

## Operations

`MpmcRingBuffer` provides operations for multiple producers and consumers to exchange elements:

### Inserting Elements — `MpmcRingBuffer#offer`

Use `MpmcRingBuffer#offer` to insert an element with this signature:

```scala
final class MpmcRingBuffer[A <: AnyRef](val capacity: Int) {
  def offer(a: A): Boolean
}
```

Inserts the element without blocking. Returns `true` on successful insertion, `false` when the buffer is full. Raises `NullPointerException` if the element is `null`. Thread-safe; multiple producer threads may call this concurrently.

### Removing Elements — `MpmcRingBuffer#take`

Use `MpmcRingBuffer#take` to remove an element with this signature:

```scala
final class MpmcRingBuffer[A <: AnyRef](val capacity: Int) {
  def take(): A
}
```

Retrieves and removes an element from the front of the buffer. Returns immediately without blocking, providing the element or `null` if the buffer is empty. Thread-safe; multiple consumer threads may call this concurrently.

### Checking State — `size`, `isEmpty`, `isFull`

Ring buffers provide three query methods to check their state. Note that under concurrent access, these results are **approximate** — by the time the method returns, other threads may modify the buffer.

All four implementations provide the same method signatures: `size`, `isEmpty`, and `isFull`. See the [`SpscRingBuffer` Operations](./spsc.mdx#operations) section for detailed descriptions and examples of these methods.

## Examples

### MPMC: General-Purpose Queue

For workloads with multiple producers and consumers, use `MpmcRingBuffer`. This example demonstrates how multiple workers coordinate to process tasks from a shared queue.

```scala title="zio-blocks-examples/src/main/scala/ringbuffer/MpmcExample.scala" 
package ringbuffer

object MpmcExample extends App {
  val buffer = MpmcRingBuffer[String](32)
  val processed = new AtomicInteger(0)
  val latch = new CountDownLatch(2)

  val producers = (0 until 2).map { id =>
    new Thread(() => {
      for (i <- 1 to 5) {
        buffer.offer(s"task-$id-$i")
      }
    })
  }

  val consumers = (0 until 2).map { _ =>
    new Thread(() => {
      while (processed.get() < 10) {
        val task = buffer.take()
        if (task ne null) {
          println(s"Worker processing: $task")
          processed.incrementAndGet()
        }
      }
      latch.countDown()
    })
  }

  (producers ++ consumers).foreach(_.start())
  producers.foreach(_.join())
  latch.await()
  println("All tasks completed")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/zio-blocks-examples/src/main/scala/ringbuffer/MpmcExample.scala))

```bash
cd zio-blocks-examples && sbt "runMain ringbuffer.MpmcExample"
```

---

## MPSC RingBuffer

`MpscRingBuffer[A]` handles the inverse case: multiple producer threads safely offering elements to a single consumer thread. It uses a **hybrid design** combining producer-side CAS coordination with a FastFlow-style relaxed-poll consumer, balancing producer contention while keeping the consumer blazingly fast.

## Algorithm

`MpscRingBuffer` handles multiple producers with a single consumer, based on the **JCTools `MpscArrayQueue`** design. It's a hybrid: producers use CAS among themselves, while the consumer remains FastFlow-style.

**How it works:**

- The **producers** (multiple) coordinate via CAS on the shared `producerIndex`. Each producer claims a slot by atomically incrementing `producerIndex`, then writes its element with release semantics. A cached `producerLimit` (initialized to capacity) reduces volatile reads of `consumerIndex`.
- The **consumer** (single) uses **FastFlow relaxed-poll semantics**: it reads array slots directly. A `null` slot indicates either the buffer is empty or a producer has claimed the slot via CAS but has not yet written the element (mid-write). In both cases, `take` returns `null` rather than spinning. The consumer clears the slot after reading.
- The `producerLimit` is updated opportunistically (racy updates are benign) to reflect approximate available capacity.

**Why hybrid?** Multiple producers need CAS to coordinate their offers, but the single consumer can achieve maximum speed using pure FastFlow (no CAS, no reading producer state).

### Diagram

To see the hybrid algorithm in action, use this interactive stepper. Type any label, click **Offer** to enqueue or **Take** to dequeue, and watch the trace panel show every intermediate variable — `pIdx`, `cIdx`, `size`, `slot`, `value` — and the exact decision the algorithm makes.

<MpscDiagram />

Here is a complete walkthrough of every variable in the trace, in the order the algorithm computes them.

### `pIdx` / `cIdx` — the monotonic counters

Like the MPMC algorithm, both counters start at zero and only ever increase — they never wrap, never reset, never go backwards. `pIdx` is shared by all producer threads: each producer atomically claims the next slot by performing a compare-and-swap (CAS) from `pIdx` to `pIdx + 1`. Whoever wins the CAS owns that slot. `cIdx` is read with a plain load because only one thread ever consumes, so no CAS is needed.

### `size = pIdx − cIdx` — the occupancy check

Before claiming a slot, the producer checks whether the buffer is full: if `pIdx − cIdx == capacity`, every slot is occupied and `offer` returns `false` immediately. The difference `pIdx − cIdx` is the number of elements currently in the buffer. Under real concurrent access, the JVM implementation maintains a cached `producerLimit` to avoid reading the volatile `consumerIndex` on every `offer` call — but the fundamental check is the same occupancy test shown here.

### `slot = pIdx & mask` — the circular array index

Because capacity is a power of two, `mask = capacity - 1`, and the bitwise AND strips high bits to give a slot number in `[0, capacity)`. This is mathematically equivalent to `pIdx % capacity` but costs a single CPU instruction. So `pIdx = 7` maps to slot `7 & 3 = 3`, and `pIdx = 8` wraps back to slot `0`.

### `value = buf[slot]` — the FastFlow slot check

The consumer uses the **FastFlow relaxed-poll pattern**: read the array slot directly with acquire semantics. If the slot is `null`, either the buffer is empty, or a producer has won the CAS but has not yet finished writing the element (mid-write). In both cases, `take` returns `null` rather than blocking or spinning — this is the *relaxed poll* semantic. This is why the consumer never reads `pIdx`; it derives all its information from the slot value itself.

Once the producer wins its CAS, it writes the element with release semantics. The acquire/release pair guarantees the consumer will see the fully written element as soon as the slot is non-null.

## Creating Instances

`MpscRingBuffer[A]` allows multiple producer threads to offer elements concurrently via compare-and-swap on the producer index, while a single consumer thread takes elements efficiently.

```scala
object MpscRingBuffer {
  def apply[A <: AnyRef](capacity: Int): MpscRingBuffer[A]
}
```

Creates a new MPSC ring buffer with the given capacity. The capacity must be a positive power of two.

To create an MPSC buffer:

```scala

val rb = MpscRingBuffer[java.lang.Long](32)
```

## Operations

`MpscRingBuffer` supports inserting elements from multiple producers, removing from a single consumer, and batch operations:

### Inserting Elements — `MpscRingBuffer#offer`

Use `MpscRingBuffer#offer` to insert an element with this signature:

```scala
final class MpscRingBuffer[A <: AnyRef](val capacity: Int) {
  def offer(a: A): Boolean
}
```

Inserts the element without blocking. Returns `true` on successful insertion, `false` when the buffer is full. Raises `NullPointerException` if the element is `null`. Thread-safe; multiple producer threads may call this concurrently.

### Removing Elements — `MpscRingBuffer#take`

Use `MpscRingBuffer#take` to remove an element with this signature:

```scala
final class MpscRingBuffer[A <: AnyRef](val capacity: Int) {
  def take(): A
}
```

Retrieves and removes an element from the front of the buffer. Returns immediately without blocking, providing the element or `null` if the buffer is empty. Call only from the single consumer thread.

### Checking State — `size`, `isEmpty`, `isFull`

Ring buffers provide three query methods to check their state. Note that under concurrent access, these results are **approximate** — by the time the method returns, other threads may modify the buffer.

All four implementations provide the same method signatures: `size`, `isEmpty`, and `isFull`. See the [`SpscRingBuffer` Operations](./spsc.mdx#operations) section for detailed descriptions and examples of these methods.

### Batch Operations — `MpscRingBuffer#drain`

Use `MpscRingBuffer#drain` to consume up to N elements with this signature:

```scala
final class MpscRingBuffer[A <: AnyRef](val capacity: Int) {
  def drain(consumer: A => Unit, limit: Int): Int
}
```

Removes up to `limit` elements from the buffer, passing each to the `consumer` callback. Returns the number of elements actually drained. Raises `IllegalArgumentException` if `limit` is negative. Call only from the consumer thread. O(n) where n is the number of elements drained.

**Note**: Uses relaxed poll semantics and stops at the first `null` slot, which may indicate either an empty buffer or a producer that has claimed a slot but has not yet written its element (mid-write). In the mid-write case, fewer than `limit` elements are returned even though more elements will become available shortly. If the callback throws, all elements passed to it up to that point remain consumed and the buffer is in a consistent state.

## Examples

### MPSC: Multiple Producers, Single Aggregator

When multiple threads produce work for a single processor, use `MpscRingBuffer`. This example shows how three producer threads safely offer items to a single consumer.

```scala title="zio-blocks-examples/src/main/scala/ringbuffer/MpscExample.scala" 
package ringbuffer

object MpscExample extends App {
  val buffer = MpscRingBuffer[java.lang.Integer](16)
  val latch = new CountDownLatch(1)

  val producers = (0 until 3).map { id =>
    new Thread(() => {
      for (i <- 1 to 4) {
        buffer.offer(java.lang.Integer.valueOf(id * 100 + i))
      }
    })
  }

  val consumer = new Thread(() => {
    var received = 0
    while (received < 12) {
      val item = buffer.take()
      if (item ne null) {
        println(s"Processed: $item")
        received += 1
      }
    }
    latch.countDown()
  })

  producers.foreach(_.start())
  consumer.start()
  producers.foreach(_.join())
  latch.await()
  println("All items processed")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/zio-blocks-examples/src/main/scala/ringbuffer/MpscExample.scala))

```bash
cd zio-blocks-examples && sbt "runMain ringbuffer.MpscExample"
```

---

## SPMC RingBuffer

`SpmcRingBuffer[A]` allows a single producer thread to efficiently feed multiple consumer threads. It uses an **index-based algorithm** where slot validity is determined by comparing producer and consumer indices, allowing multiple consumers to coordinate safely via CAS operations on a shared consumer index.

## Algorithm

`SpmcRingBuffer` allows a single producer to feed multiple consumers. The algorithm is **index-based**: slot validity is determined by comparing `producerIndex` and `consumerIndex`, not by null-checking slots.

**How it works:**

- The **producer** is single-threaded and never uses CAS. It maintains a `producerLimit` derived from reading the volatile `consumerIndex`. On the fast path, it checks its local limit; when exhausted, it refreshes by reading `consumerIndex`. This design avoids the producer ever reading consumer state on the fast path.
- The **consumers** (any number) coordinate via a CAS loop on `consumerIndex`. Each consumer reads the element at its claimed index, then attempts to CAS `consumerIndex` forward. If CAS fails, it retries with a refreshed index. This ensures each element is claimed by exactly one consumer.
- **No slot clearing by consumers:** Consumers do not write `null` after reading. Instead, the producer safely overwrites slots once its `producerLimit` check (based on `consumerIndex`) confirms all consumers have advanced past them. This eliminates the race that would occur if a consumer claimed a slot but hadn't yet cleared it.

**Trade-off:** Consumer CAS introduces overhead under contention, but the producer remains extremely fast (no synchronization).

### Diagram

To see the single-producer multi-consumer algorithm in action, use this interactive stepper. Type any label, click **Offer** to enqueue (single producer) or **Take** to dequeue (any consumer), and watch the trace panel show every intermediate variable — `pIdx`, `cIdx`, `size`, `slot`, `element` — and the exact decision each side makes.

<SpmcDiagram />

Here is a complete walkthrough of every variable in the trace, in the order the algorithm computes them.

### `pIdx` — the monotonic producer counter

The producer has its own index that starts at zero and only ever increases. Because only one thread produces, `pIdx` is read and written with plain loads and stores — no CAS is needed. The producer's job is simple: check capacity (`pIdx - cIdx < capacity`), write to the slot, then advance `pIdx`.

### `cIdx` — the monotonic consumer counter (multi-threaded)

All consumers share a single `consumerIndex` counter. Because multiple consumers may be operating concurrently, each consumer reads `cIdx` with volatile semantics to see the most recent updates, then attempts to atomically advance it via CAS. If the CAS fails, another consumer claimed the slot first and the reader discards their read and retries.

### `size = pIdx − cIdx` — the occupancy check

Both sides use the same occupancy formula: `pIdx − cIdx` is the number of elements currently in the buffer. The producer checks if `size == capacity` to detect a full buffer. The consumer checks if `size == 0` to detect an empty buffer.

### `slot = pIdx & mask` or `slot = cIdx & mask` — the circular array index

Identical to all other ring buffer variants: because capacity is a power of two, the bitwise AND `& mask` replaces modulo division, giving a slot number in `[0, capacity)`. So `pIdx = 7` maps to slot `7 & 3 = 3`, and `pIdx = 8` wraps back to slot `0`.

### `element = buf[slot]` — read-before-CAS

This is the critical SPMC distinction. The consumer reads the element *before* attempting the CAS on `cIdx`. This ordering is essential: once the CAS succeeds and `cIdx` advances, the producer is permitted to overwrite that slot on its next lap. By reading the element first, the consumer guarantees it captures the value while the slot is still logically owned.

In the trace, when a consumer takes an element, you will see the `element` row highlighted — this captures the read that must happen before the CAS.

### Why SPMC doesn't clear slots

Unlike MPSC (which uses FastFlow's null-check pattern), SPMC determines slot validity purely by index comparison: if `cIdx ≤ slot < pIdx`, the slot is live. After a consumer advances `cIdx`, the slot becomes "stale" — no longer live, but still holding the old value until the producer wraps around and overwrites it. This eliminates the race that would occur if a consumer had to null out the slot while holding the CAS; instead, the producer simply overwrites when it knows the consumer has advanced.

The diagram marks stale slots with a dashed border and a "stale" label in grey.

## Creating Instances

`SpmcRingBuffer[A]` allows a single producer thread to offer elements while multiple consumer threads concurrently take elements via compare-and-swap on the consumer index. Use the companion object to instantiate a buffer:

```scala
object SpmcRingBuffer {
  def apply[A <: AnyRef](capacity: Int): SpmcRingBuffer[A]
}
```

The capacity must be a positive power of two. To create an SPMC buffer:

```scala

val rb = SpmcRingBuffer[java.lang.Integer](64)
```

## Operations

`SpmcRingBuffer` provides three core operations for sending and receiving elements:

### Inserting Elements — `SpmcRingBuffer#offer`

Use `SpmcRingBuffer#offer` to insert an element with this signature:

```scala
final class SpmcRingBuffer[A <: AnyRef](val capacity: Int) {
  def offer(a: A): Boolean
}
```

Inserts the element without blocking. Returns `true` on successful insertion, `false` when the buffer is full. Raises `NullPointerException` if the element is `null`. Call only from the single producer thread; concurrent calls from multiple threads cause undefined behavior.

### Removing Elements — `SpmcRingBuffer#take`

Use `SpmcRingBuffer#take` to remove an element with this signature:

```scala
final class SpmcRingBuffer[A <: AnyRef](val capacity: Int) {
  def take(): A
}
```

Retrieves and removes an element from the front of the buffer. Returns immediately without blocking. Returns the element, or `null` if the buffer is empty. Thread-safe; multiple consumer threads may call this concurrently.

### Checking State — `size`, `isEmpty`, `isFull`

Ring buffers provide three query methods to check their state. Note that under concurrent access, these results are **approximate** — by the time the method returns, other threads may already modify the buffer.

All four implementations provide the same method signatures: `size`, `isEmpty`, and `isFull`. See the [`SpscRingBuffer` Operations](./spsc.mdx#operations) section for detailed descriptions and examples of these methods.

---

## SPSC RingBuffer

`SpscRingBuffer[A]` is optimized for the simplest and fastest case: exactly one producer thread and one consumer thread. It uses the **FastFlow** algorithm to eliminate all cross-core cache traffic, achieving nanosecond-scale latencies with no volatile reads on the fast path.

## Why FastFlow?

Understanding the FastFlow algorithm helps explain why `SpscRingBuffer` achieves such high performance.

Imagine two threads sharing data with a traditional lock-based queue like `ArrayBlockingQueue`. They use a mutex (lock) to coordinate:

1. Producer thread: acquires lock, adds element, releases lock
2. Consumer thread: acquires lock, removes element, releases lock

This works, but locks have a cost: when threads contend for the same lock, one thread **blocks** (goes to sleep) while waiting. Waking a thread is expensive (thousands of CPU cycles). Even lock-free queues using `synchronized` or `volatile` reads can cause **cache-coherency traffic**: when one CPU core writes to a variable another core reads, the entire cache line must be invalidated and transferred — a costly operation that slows down both cores.

The fundamental issue: **if the producer has to read what the consumer wrote (or vice versa), their CPU caches constantly fight**. This is called *false sharing* and can reduce throughput by 10x or more on heavily loaded systems.

The problem is that *any* read by the producer of the consumer's state (or vice versa) causes cache line bouncing. So FastFlow's radical idea: **neither side should ever read the other's state**.

Instead, the producer simply **writes data into slots** and marks them non-null. The consumer independently **reads slots** and takes any non-null values. The array slot's null/non-null status itself is the only coordination needed — a *happens-before* relationship written once by the producer, read once by the consumer. No locks, no atomic operations on the fast path, no reading the other side's counters.

The **note-passing analogy**:
- You (producer) have a row of empty desks between you and your friend (consumer)
- Empty desk = `null` (available)
- Note on desk = non-null value (message ready)
- **You never look at your friend's side** — you just place notes on empty desks
- **Your friend never looks at your side** — they just pick up notes they see
- No shouting "are you ready?", no waiting, no lock contention

**This is how FastFlow solves the cache-coherency problem**: since producer and consumer never read each other's counters, there's no cache line bouncing between CPU cores. All coordination happens through the slots themselves, which are written once and read once.

The **look-ahead cache** is a further optimization: the producer maintains a local cached limit (`producerLimit`) so they don't have to check every slot individually. It's like glancing ahead at the next N desks to see if they're empty, without actually bending over to look. This keeps the fast path extremely fast — just a local counter check and an array write.

**Why FastFlow is fast**:
- **No locks** — no thread ever blocks another
- **Minimal cache coordination** — producer and consumer touch separate memory locations; the array slot is written once, read once
- **Write-once, read-once semantics** — the slot's null/non-null status is the handshake
- **Cache-line padding** — producer and consumer indices padded to separate cache lines, eliminating false sharing

The result: **lock-free** communication that scales linearly with CPU count and achieves nanosecond-scale latencies. This is why FastFlow is the algorithm of choice for SPSC ring buffers in high-performance systems like trading platforms, game engines, and network stacks. (The slow path's retry on full buffers means the algorithm is not wait-free, though contention is rare.)

## Algorithm

`SpscRingBuffer` uses the **FastFlow** pattern, originally developed for the C++ FastFlow framework and popularized in Java by JCTools and the LMAX Disruptor. The core insight: the array element's `null`/non-`null` state **is** the synchronization signal. The producer never reads `consumerIndex`; the consumer never reads `producerIndex`. This eliminates all cross-core cache traffic between the two sides.

**How it works:**

- The producer walks forward through the array, placing elements into slots that are currently `null`. It marks a slot non-`null` after writing. The producer never reads what the consumer has consumed — it only checks its own cached `producerLimit` to know how many slots are available.
- The consumer walks forward through the array, reading any slot that contains a non-`null` value. After reading, it writes `null` to clear the slot. The consumer never reads the producer's index.
- The slot's nullness itself is the coordination: written once by the producer, read once by the consumer. No locks, no atomic operations on the fast path, no reading the other side's counters.

**Look-ahead cache:** The producer maintains a local `producerLimit` (derived from `consumerIndex` during slow path). This allows the fast path to check a simple local counter instead of reading the volatile `consumerIndex` on every `offer`. The look-ahead step is `max(1, min(capacity/4, 4096))`, balancing between reducing consumer reads and memory usage.

### Diagram

To see the single-producer single-consumer FastFlow algorithm in action, use this interactive stepper. Type any label, click **Offer** to enqueue or **Take** to dequeue, and watch how the producer and consumer coordinate using only release/acquire semantics — no CAS on either side.

<SpscDiagram />

Here is a complete walkthrough of every variable in the trace, in the order the algorithm computes them.

### `pIdx` — the monotonic producer counter (plain load/store)

The producer reads `pIdx` with a plain load — no atomic operations at all. Since only one thread produces, no synchronization is needed. The producer maintains its own count, never reads `consumerIndex`, and advances `pIdx` after writing with a release store.

### `cIdx` — the monotonic consumer counter

SPSC avoids reading `consumerIndex` entirely. Instead of checking the occupancy formula (`pIdx - cIdx`), the producer reads the array slot at `producerIndex + lookAheadStep` to determine if a slot is free (on the slow path when the look-ahead cache is exhausted). This slot-based approach eliminates all counter reads between producer and consumer.

### `size = pIdx − cIdx` — the occupancy check

Both sides use the same occupancy formula, but only during slow paths (rare). On the fast path, neither side reads the other's counter; they rely on slot nullness (FastFlow) for synchronization.

### `slot = pIdx & mask` or `slot = cIdx & mask` — the circular array index

Identical bitmask arithmetic: `& mask` replaces modulo, giving a slot in `[0, capacity)`.

### `element = buf[slot]` — no CAS, plain slot reads

Unlike SPMC and MPMC, the consumer reads the slot directly with an acquire load — because there's only one consumer, no read-before-CAS is needed. The consumer is the sole owner of the take path. After reading, the consumer nulls the slot with a release store, which serves two purposes: (1) signals to the producer that the slot is free, and (2) releases any bookkeeping updates to other cores via the release semantics.

### Why SPSC doesn't read producer state

The consumer never reads `pIdx`. Instead, it reads the slot directly: if the slot is non-null, data is ready. If null, the buffer is empty. The producer wrote a non-null value via a release store, and the consumer reads via an acquire load, forming a happens-before pair. This is the core of FastFlow: **eliminate cross-core reads entirely**.

The diagram marks slots as "live" (still owned by the producer-consumer pair) or "stale" (consumed but not yet overwritten). Unlike SPMC where consumers CAS and have ordering concerns, SPSC clears immediately because only one consumer exists — there's no race.

## Creating Instances

Ring buffers are instantiated via the companion object's `apply` method. `SpscRingBuffer[A]` uses the FastFlow pattern with a look-ahead cache. On the fast path, the producer checks a cached `producerLimit` to avoid reading `consumerIndex`. When the cached limit is exhausted, the slow path reads the array slot at `producerIndex + lookAheadStep` — never `consumerIndex` directly. This keeps the producer and consumer cache lines fully independent. The consumer uses null/non-null slot reads (FastFlow semantics). Together, these avoid repeated volatile reads and minimize cross-core cache traffic.

```scala
object SpscRingBuffer {
  def apply[A <: AnyRef](capacity: Int): SpscRingBuffer[A]
}
```

Creates a new SPSC ring buffer with the given capacity. The capacity must be a positive power of two.

We create an SPSC buffer as follows:

```scala

val rb = SpscRingBuffer[String](16)  // capacity must be power of 2
```

## Operations

`SpscRingBuffer` provides operations for sending and receiving elements, as well as querying buffer state:

### Inserting Elements — `SpscRingBuffer#offer`

Use `SpscRingBuffer#offer` to insert an element with this signature:

```scala
final class SpscRingBuffer[A <: AnyRef](val capacity: Int) {
  def offer(a: A): Boolean
}
```

Inserts the element without blocking. Returns `true` on successful insertion, `false` when the buffer is full. Raises `NullPointerException` if the element is `null`. Call only from the producer thread.

To handle backpressure by checking if insertion succeeds:

```scala

val rb = SpscRingBuffer[String](4)
```

When the buffer becomes full, `offer` returns `false`:

```scala
val result1 = rb.offer("a")
// result1: Boolean = true
val result2 = rb.offer("b")
// result2: Boolean = true
val result3 = rb.offer("c")
// result3: Boolean = true
val result4 = rb.offer("d")
// result4: Boolean = true
val result5 = rb.offer("e")
// result5: Boolean = false
```

### Removing Elements — `SpscRingBuffer#take`

Use `SpscRingBuffer#take` to remove an element with this signature:

```scala
final class SpscRingBuffer[A <: AnyRef](val capacity: Int) {
  def take(): A
}
```

Retrieves and removes an element from the front of the buffer. Returns immediately without blocking, providing the element or `null` if the buffer is empty. Call only from the consumer thread.

To retrieve elements in FIFO order:

```scala
rb.take()
// res2: String = "a"
rb.take()
// res3: String = "b"
rb.take()
// res4: String = "c"
rb.take()
// res5: String = "d"
rb.take()
// res6: String = null
```

### Checking State — `size`, `isEmpty`, `isFull`

Ring buffers provide three query methods to check their state. Note that under concurrent access, these results are **approximate** — by the time the method returns, other threads may modify the buffer.

Use `SpscRingBuffer#size` to get the approximate element count with this signature:

```scala
final class SpscRingBuffer[A <: AnyRef](val capacity: Int) {
  def size: Int
}
```

Returns the approximate number of elements currently in the buffer. Under concurrent access, the result may be stale. O(1).

Use `SpscRingBuffer#isEmpty` to check if the buffer is empty with this signature:

```scala
final class SpscRingBuffer[A <: AnyRef](val capacity: Int) {
  def isEmpty: Boolean
}
```

Returns `true` if the buffer contains no elements (approximate). O(1).

Use `SpscRingBuffer#isFull` to check if the buffer is at capacity with this signature:

```scala
final class SpscRingBuffer[A <: AnyRef](val capacity: Int) {
  def isFull: Boolean
}
```

Returns `true` if the buffer is at capacity (approximate). O(1).

All four implementations provide the same three methods. To check state after operations:

```scala

val rb2 = SpscRingBuffer[String](8)
```

State queries are cheap but approximate under concurrency:

```scala
rb2.offer("x")
// res8: Boolean = true
rb2.offer("y")
// res9: Boolean = true

rb2.size
// res10: Int = 2
rb2.isEmpty
// res11: Boolean = false
rb2.isFull
// res12: Boolean = false
```

### Batch Operations — `SpscRingBuffer#drain` and `SpscRingBuffer#fill`

Use `SpscRingBuffer#drain` to consume up to N elements with this signature:

```scala
final class SpscRingBuffer[A <: AnyRef](val capacity: Int) {
  def drain(consumer: A => Unit, limit: Int): Int
}
```

Removes up to `limit` elements from the buffer, passing each to the `consumer` callback. Returns the number of elements actually drained. Raises `IllegalArgumentException` if `limit` is negative. Call only from the consumer thread. O(n) where n is the number of elements drained. If the callback raises an exception, all elements passed to it up to that point remain consumed and the buffer stays in a consistent state.

Use `SpscRingBuffer#fill` to produce up to N elements with this signature:

```scala
final class SpscRingBuffer[A <: AnyRef](val capacity: Int) {
  def fill(supplier: () => A, limit: Int): Int
}
```

Inserts up to `limit` elements by calling the `supplier` for each new element. Returns the number of elements actually inserted. Raises `IllegalArgumentException` if `limit` is negative. Raises `NullPointerException` if the supplier returns `null`. Call only from the producer thread. O(n) where n is the number of elements inserted.

Batch operations amortize synchronization costs. To drain multiple elements at once:

```scala

val rb3 = SpscRingBuffer[java.lang.Integer](16)
(1 to 5).foreach(i => rb3.offer(Integer.valueOf(i)))

val collected = scala.collection.mutable.Buffer[java.lang.Integer]()
val drained = rb3.drain(collected += _, 10)  // drained = 5

println(s"Drained items: ${collected.mkString(", ")}")
```

To avoid repeated `offer` calls when producing elements, use fill:

```scala

val rb4 = SpscRingBuffer[String](8)
val counter = new AtomicInteger(0)
val filled = rb4.fill(() => { counter.incrementAndGet(); s"item-${counter.get()}" }, 5)

println(s"Filled $filled items")
```

## Primitive variants

The generic `SpscRingBuffer[A]` stores `AnyRef` slots, which boxes Java primitives on the producer side. For SPSC workloads handing off `Int`, `Long`, `Float`, or `Double` values between two threads, four zero-boxing companions are provided:

| Type | Class | Slot encoding |
|---|---|---|
| `Int`    | `IntSpscRingBuffer`    | one `Long` per slot: high 32 bits hold an occupancy tag (`0` = empty, `1` = data, `2` = DONE sentinel), low 32 bits hold the `Int` payload. The whole 64-bit word is `0L` when empty, so any tagged value is non-zero regardless of the payload bits. |
| `Long`   | `LongSpscRingBuffer`   | one `Long` per slot. `Long.MinValue` is reserved as the empty marker and `Long.MinValue + 1L` as the in-band DONE sentinel; offering either value throws `IllegalArgumentException`. All other `Long` values pass through unchanged. |
| `Float`  | `FloatSpscRingBuffer`  | one `Long` per slot: high 32 bits tag (`0` = empty, `1` = data, `2` = DONE), low 32 bits hold `floatToRawIntBits(value)`. The whole word is `0L` when empty. Every `Float` (including `0.0f`, `NaN`, `±Inf`) is representable. |
| `Double` | `DoubleSpscRingBuffer` | one `Long` per slot. `doubleToRawLongBits(value)` is stored directly; two reserved non-canonical `NaN` bit patterns mark empty (`0xFFF8_0000_0000_0001L`) and DONE (`0xFFF8_0000_0000_0002L`). A value whose bits equal one of the two reserved patterns is stored as `Double.NaN` (`0x7ff8000000000000L`) so it can never collide with the markers; every other value, every other `NaN` payload included, is stored bit for bit. |

All four use the same FastFlow algorithm as the generic version — single backing `Array[Long]`, padded indices, look-ahead cache — and expose the same core operations (`offer`, `take`, `peek`, `isEmpty`, `isFull`, `size`, `capacity`). The only API difference is that `take` returns a primitive instead of `AnyRef`, so callers should check `peek()` first to know whether the next read will be a real value or just the empty marker. (The generic `drain`/`fill` batch helpers are not provided on the primitive variants; loop `peek`/`take` directly.)

In addition, each primitive variant exposes two methods for **in-band end-of-stream signalling**, used by the concurrent stream operators to propagate "no more data" through a primitive queue without needing a separate side channel:

- `offerDone(): Boolean` — write the DONE sentinel into the next free slot, returning `true` on success or `false` if the queue is full. Once `offerDone()` succeeds the producer should not offer any further values.
- `pollPacked(): Long` — atomically poll the next slot and return its packed encoding: `EMPTY_PACKED`/`EMPTY` / `EMPTY_BITS` when empty (consumer index NOT advanced), `DONE_PACKED`/`DONE` / `DONE_BITS` for the DONE sentinel (index advanced), or a non-sentinel encoded value otherwise (index advanced). The exact constant names live on each companion object (`IntSpscRingBuffer.DONE_PACKED`, `LongSpscRingBuffer.DONE`, `FloatSpscRingBuffer.DONE_PACKED`, `DoubleSpscRingBuffer.DONE_BITS`).

`pollPacked` is intended for advanced consumers that want a single atomic operation distinguishing empty / data / DONE. Most application code that simply needs primitive throughput is fine using the higher-level concurrent stream operators below.

```scala

val ints = new IntSpscRingBuffer(16)
ints.offer(42)
if (ints.peek()) {
  val v: Int = ints.take()     // 42
}

val longs = new LongSpscRingBuffer(16)
longs.offer(Long.MaxValue)
if (longs.peek()) {
  val w: Long = longs.take()   // Long.MaxValue
}

val doubles = new DoubleSpscRingBuffer(16)
doubles.offer(Double.NaN)      // stored as is: Double.NaN is not one of the reserved patterns
```

These variants underpin the primitive-specialized concurrent stream operators (`mapPar`, `mergeAll`, `flatMapPar`) so primitive streams do not box during cross-thread handoff.

## Examples

### SPSC: Producer-Consumer Ping-Pong

In a single-producer, single-consumer setup, use `SpscRingBuffer` for maximum throughput. This example demonstrates how two threads communicate efficiently using FastFlow signaling.

```scala title="zio-blocks-examples/src/main/scala/ringbuffer/SpscExample.scala" 
package ringbuffer

object SpscExample extends App {
  val buffer = SpscRingBuffer[String](8)
  val latch = new CountDownLatch(1)

  val producer = new Thread(() => {
    for (i <- 1 to 5) {
      buffer.offer(s"message-$i")
    }
  })

  val consumer = new Thread(() => {
    for (_ <- 1 to 5) {
      var msg: String = null
      while ({ msg = buffer.take(); msg.eq(null) }) {}
      println(s"Received: $msg")
    }
    latch.countDown()
  })

  producer.start()
  consumer.start()
  latch.await()
  println("Done")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/zio-blocks-examples/src/main/scala/ringbuffer/SpscExample.scala))

```bash
cd zio-blocks-examples && sbt "runMain ringbuffer.SpscExample"
```

### Batch Fill and Drain (SPSC)

Use `fill` and `drain` for efficient batch operations. This example shows how to amortize synchronization costs by processing multiple elements at once.

```scala title="zio-blocks-examples/src/main/scala/ringbuffer/BatchExample.scala" 
package ringbuffer

object BatchExample extends App {
  val buffer = SpscRingBuffer[java.lang.Integer](64)
  val latch = new CountDownLatch(1)

  val producer = new Thread(() => {
    var batch = 1
    while (batch <= 3) {
      val currentBatch = batch
      val count = buffer.fill(() => java.lang.Integer.valueOf(currentBatch * 100), 10)
      println(s"Filled $count items in batch $currentBatch")
      batch += 1
    }
  })

  val consumer = new Thread(() => {
    val items = scala.collection.mutable.Buffer[java.lang.Integer]()
    while (items.size < 30) {
      val drained = buffer.drain(items += _, 10)
      if (drained > 0) println(s"Drained $drained items")
    }
    println(s"Total items consumed: ${items.size}")
    latch.countDown()
  })

  producer.start()
  consumer.start()
  latch.await()
  println("Done")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/zio-blocks-examples/src/main/scala/ringbuffer/BatchExample.scala))

```bash
cd zio-blocks-examples && sbt "runMain ringbuffer.BatchExample"
```

---

## Allows

`Allows[A, S]` is a compile-time capability token that proves, at the call site, that type `A` satisfies the structural grammar `S`. A capability token is a compile-time phantom proof value — it carries no runtime data and exists solely to pass evidence through the type system that a structural constraint has been satisfied.

`Allows` does **not** require or use `Schema[A]`. It inspects the Scala type structure of `A` directly at compile time, using nothing but the Scala type system. Any `Schema[A]` that appears alongside `Allows` in examples is the library author's own separate constraint — it is not imposed by `Allows` itself.

```scala
sealed abstract class Allows[A, S <: Allows.Structural]
```

## Overview

The gap `Allows` fills is **structural preconditions** at the call site, at compile time, with precise error messages. Structural preconditions are constraints on the shape of a type's fields (e.g., "all fields must be scalars"), unlike runtime checks which happen during execution and produce exceptions or errors.

## Motivation

ZIO Blocks gives library authors a powerful way to build data-oriented DSLs. A library can accept `A: Schema` and use the schema at runtime to serialize, deserialize, query, or transform values of `A`. A data-oriented DSL is a generic API built around a data description (Schema) rather than a fixed interface, allowing one function to serialize, validate, or transform any conforming type. Many generic functions have **structural preconditions** that don't require a schema.

Consider these real-world scenarios:

- A CSV serializer requires flat records of scalars — nested records should fail at the call site, not deep inside the serializer.
- An RDBMS layer cannot handle nested records as column values — the error should name the problematic field.
- An event bus expects a sealed trait of flat record cases — violations should be caught before publishing.
- A JSON document store allows arbitrarily nested records but not `DynamicValue` leaves — the schema validation should be precise. DynamicValue is the schema-less escape hatch that can hold arbitrary data — a DynamicValue leaf bypasses compile-time checking entirely, making it impossible for the compiler to enforce any structural grammar.

Without `Allows`, these constraints can only be checked at runtime, producing confusing errors deep inside library internals. With `Allows[A, S]`, the constraint is verified at the **call site**, at compile time, with precise, path-aware error messages and concrete fix suggestions.

## The Upper Bound Semantics

`Allows[A, S]` is an upper bound. A type `A` that uses only a strict subset of what `S` permits also satisfies it — just as `A <: Foo` does not require that `A` uses every method of `Foo`. Upper bound semantics is the right choice because a lower bound would require using every shape (impractical), exact matching would require naming every shape used (too rigid), whereas upper bound says "your type may use any of these shapes" — a permission, not a mandate.

```scala

// Both satisfy Record[Primitive | Optional[Primitive]] — the upper bound

case class UserRow(name: String, age: Int)
// UserRow satisfies the grammar: all fields are Primitive

case class UserRowOpt(name: String, age: Int, email: Option[String])
// UserRowOpt also satisfies the grammar: all fields are Primitive or Optional[Primitive]

val ev1: Allows[UserRow, Record[Primitive | Optional[Primitive]]] = implicitly
val ev2: Allows[UserRowOpt, Record[Primitive | Optional[Primitive]]] = implicitly
```

## Creating Instances

`Allows[A, S]` is not instantiated directly. Instead, you summon an evidence value at the point where you need the constraint. The macro automatically verifies the constraint at compile time.

<Tabs groupId="scala-version" defaultValue="scala2">
  <TabItem value="scala2" label="Scala 2">

```scala

def toJson[A](doc: A)(implicit ev: Allows[A, Record[Primitive]]): String = ???

// Or summon at the call site:
val evidence = implicitly[Allows[Int, Primitive]]
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3">

```scala

def toJson[A](doc: A)(using Allows[A, Record[Primitive]]): String = ???

// Calling the function:
case class Person(name: String, age: Int)
val json = toJson(Person("Alice", 30))  // Compiles if Person satisfies Record[Primitive]
```

  </TabItem>
</Tabs>

The constraint is checked once, at the call site. If the type `A` does not satisfy `S`, you get a compile-time error with a precise message showing exactly which field violates the grammar.

## Grammar Nodes

All grammar nodes extend `Allows.Structural`.

| Node | Matches |
|---|---|
| `Primitive` | **Any** scalar — catch-all for all 30 Schema 2 primitive types |
| `Primitive.Boolean` | `scala.Boolean` only |
| `Primitive.Int` | `scala.Int` only |
| `Primitive.Long` | `scala.Long` only |
| `Primitive.Double` | `scala.Double` only |
| `Primitive.Float` | `scala.Float` only |
| `Primitive.String` | `java.lang.String` only |
| `Primitive.BigDecimal` | `scala.BigDecimal` only |
| `Primitive.BigInt` | `scala.BigInt` only |
| `Primitive.Unit` | `scala.Unit` only |
| `Primitive.Byte` | `scala.Byte` only |
| `Primitive.Short` | `scala.Short` only |
| `Primitive.Char` | `scala.Char` only |
| `Primitive.UUID` | `java.util.UUID` only |
| `Primitive.Currency` | `java.util.Currency` only |
| `Primitive.Instant` / `LocalDate` / `LocalDateTime` / … | Each specific `java.time.*` type |
| | |
| `Record[A]` | A case class / product type whose every field satisfies `A`. Vacuously true for zero-field records. Sealed traits and enums are **automatically unwrapped**: each case is checked individually, so no `Variant` node is needed. |
| `Sequence[A]` | Any collection (`List`, `Vector`, `Set`, `Array`, `Chunk`, …) whose element type satisfies `A` |
| `Map[K, V]` | `Map`, `HashMap`, … whose key satisfies `K` and value satisfies `V` |
| `Optional[A]` | `Option[X]` where the inner type `X` satisfies `A` |
| `Wrapped[A]` | A ZIO Prelude `Newtype`/`Subtype` wrapper, and on Scala 3 also an opaque type or a neotype newtype, whose underlying type satisfies `A` |
| | |
| `Self` | Recursive self-reference back to the entire enclosing `Allows[A, S]` grammar |
| `Dynamic` | `DynamicValue` — the schema-less escape hatch |
| `IsType[A]` | Exact nominal type match: satisfied only when the checked type is exactly `A` (`=:=`) |
| `` `\|` `` | Union of two grammar nodes: `A \| B`. In Scala 2 write `` A `\|` B `` in infix position. |

Every specific `Primitive.Xxx` node also satisfies the top-level `Primitive` node (which matches any of the 30 primitive types). This means a type annotated with `Primitive.Int` is valid wherever `Primitive` or `Primitive | Primitive.Long` is required.

## Core Operations

`Allows[A, S]` is a **proof token**, not an ordinary value. It carries zero public methods that you call directly. Instead, you use it in three ways:

1. **As a constraint in function signatures** — Declare `Allows[A, S]` as an implicit/using parameter to require that callers pass only types satisfying the grammar.
2. **To summon evidence** — Use `implicitly[Allows[A, S]]` (Scala 2) or `summon[Allows[A, S]]` (Scala 3) at a call site to check the constraint and get an error message if it fails.
3. **In type aliases** — Define type aliases like `type FlatRecord = Allows[_, Record[Primitive | Optional[Primitive]]]` to name constraints and reuse them across functions.

The macro that powers `Allows` checks the constraint **at compile time** and emits nothing but a reference to a single shared singleton at runtime, so there is zero per-call-site overhead.

## Specific Primitives

The `Primitive` parent class is the catch-all: it accepts any of the 30 Schema 2 primitive types. For stricter control — such as when the target serialisation format only supports a subset — use the specific subtype nodes in `Allows.Primitive`:

```scala

// Only JSON-representable scalars (no UUID, Char, java.time.*)
type JsonPrimitive =
  Primitive.Boolean | Primitive.Int | Primitive.Long |
  Primitive.Double  | Primitive.String | Primitive.BigDecimal |
  Primitive.BigInt  | Primitive.Unit

def toJson[A](doc: A)(using Allows[A, Record[JsonPrimitive | Self]]): String = ???

// Only numeric types
type Numeric = Primitive.Int | Primitive.Long | Primitive.Double | Primitive.Float |
               Primitive.BigInt | Primitive.BigDecimal

def aggregate[A](data: A)(using Allows[A, Record[Numeric]]): Double = ???
```

A type annotated with `Primitive.Int` satisfies `Primitive` (the catch-all) because `Primitive.Int extends Primitive`:

```scala

val ev: Allows[Int, Primitive] = implicitly  // Primitive (catch-all) — ✓
val sp: Allows[Int, Primitive.Int] = implicitly  // Primitive.Int (specific) — ✓
```

### JSON Document Store Example

JSON's primitive value set is `null | boolean | number | string`. Types such as `UUID`, `Char`, and all `java.time.*` types have no native JSON representation and must be encoded as strings at the application layer. Using `JsonPrimitive` instead of the catch-all `Primitive` enforces this at compile time.

A JSON document grammar is straightforward: a JSON value is either a record (JSON object) or a sequence (JSON array), and `Self` handles all nesting:

```scala

type JsonPrimitive =
  Primitive.Boolean | Primitive.Int | Primitive.Long | Primitive.Double |
  Primitive.String  | Primitive.BigDecimal | Primitive.BigInt | Primitive.Unit

type Json = Record[JsonPrimitive | Self] | Sequence[JsonPrimitive | Self]

def toJson[A](doc: A)(using Allows[A, Json]): String = ???
```

`Self` recurses back to `Json` at every nested position, so `List[String]` satisfies `Sequence[JsonPrimitive | Self]` (String is JsonPrimitive), `List[Author]` satisfies it too (Author satisfies `Record[JsonPrimitive | Self]` via Self), and top-level arrays work directly.

A type with a UUID or Instant field fails at compile time with this error:

```
[error] Schema shape violation at WithUUID.id: found Primitive(java.util.UUID),
        required Primitive.Boolean | Primitive.Int | ... | Primitive.String | ...
        UUID is not a JSON-native type — encode it as Primitive.String.
```

## Union Syntax

Union types express "or" in the grammar.

<Tabs groupId="scala-version" defaultValue="scala2">
  <TabItem value="scala2" label="Scala 2">

Uses the infix operator `` Primitive `|` Optional[Primitive] `` from `Allows`:

```scala

def writeCsv[A](rows: Seq[A])(implicit
  ev: Allows[A, Record[Primitive | Optional[Primitive]]]
): Unit = ???
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3">

Uses native union type syntax:

```scala

def writeCsv[A](rows: Seq[A])(using
  Allows[A, Record[Primitive | Optional[Primitive]]]
): Unit = ???
```

  </TabItem>
</Tabs>

Both spellings compile and produce the same semantic behavior. The grammar is identical — the only difference is how the union type is expressed.

## Use Cases

### Flat Record (CSV, RDBMS Row)

```scala

// Flat record: only primitives and optional primitives allowed
def writeCsv[A: Schema](rows: Seq[A])(using
  Allows[A, Record[Primitive | Optional[Primitive]]]
): Unit = ???

// RDBMS INSERT: primitives, optional primitives, or string-keyed maps (JSONB)
def insert[A: Schema](value: A)(using
  Allows[A, Record[Primitive | Optional[Primitive] | Allows.Map[Primitive, Primitive]]]
): String = ???
```

If a user passes a type with nested records, they get a precise compile-time error like this:

```
[error] Schema shape violation at UserWithAddress.address: found Record(Address),
        required Primitive | Optional[Primitive] | Map[Primitive, Primitive]
```

### Event Bus / Message Broker

Published events are typically sealed traits of flat record cases. No `Variant` node is needed — sealed traits are automatically unwrapped:

```scala

// DomainEvent is a sealed trait; its cases must each satisfy Record[Primitive | Sequence[Primitive]]
def publish[A: Schema](event: A)(using
  Allows[A, Record[Primitive | Optional[Primitive] | Sequence[Primitive]]]
): Unit = ???
```

If a case of the sealed trait has a nested record field, the error names that case and field like this:

```
[error] Schema shape violation at DomainEvent.OrderPlaced.items.<element>:
        found Record(OrderItem), required Primitive | Optional[Primitive] | Sequence[Primitive]
```

### JSON Document Store (Recursive)

A document store accepts arbitrarily nested records but not `DynamicValue` leaves. The `Self` node expresses the recursive grammar:

```scala

type JsonDocument =
  Record[Primitive | Self | Optional[Primitive | Self] | Sequence[Primitive | Self] | Allows.Map[Primitive, Primitive | Self]]

def toJson[A: Schema](doc: A)(using Allows[A, JsonDocument]): String = ???
```

This grammar allows:
- `case class Author(name: String, email: String)` — Record[Primitive] ✓
- `case class Book(title: String, author: Author, tags: List[String])` — Record with Self-nested record and Sequence[Primitive] ✓
- `case class Category(name: String, subcategories: List[Category])` — recursive ✓

But rejects:
- `case class Bad(name: String, payload: DynamicValue)` — DynamicValue is not in the grammar ✗

### GraphQL / Tree Structures (Self)

```scala

def graphqlType[A: Schema]()(using
  Allows[A, Record[Primitive | Optional[Self] | Sequence[Self]]]
): String = ???

// Works:
case class TreeNode(value: Int, children: List[TreeNode])
object TreeNode { implicit val schema: Schema[TreeNode] = Schema.derived }
// graphqlType[TreeNode]() — compiles fine
```

## Sequence Subtypes

The `Sequence[A]` node accepts any collection type. When a DSL needs to restrict to a specific kind of collection — for example, a DynamoDB `Set` operation that is only valid on sets, not lists — use the `Sequence` subtypes:

```scala

// Only an immutable List is accepted
val listOnly: Allows[List[Int], Sequence.List[Primitive]] = implicitly

// Only an immutable Set is accepted
val setOnly: Allows[Set[Int], Sequence.Set[Primitive]] = implicitly

// Only a Vector
val vecOnly: Allows[Vector[String], Sequence.Vector[Primitive]] = implicitly

// Only an Array
val arrOnly: Allows[Array[Int], Sequence.Array[Primitive]] = implicitly

// Only a Chunk

val chkOnly: Allows[Chunk[String], Sequence.Chunk[Primitive]] = implicitly
```

Each subtype extends `Sequence[A]`, so a grammar written with the parent `Sequence` still accepts all collection kinds. A grammar written with a subtype rejects other kinds at compile time:

```
// Set[Int] does NOT satisfy Sequence.List[Primitive] — compile error:
[error] Shape violation at Set: found Sequence[scala.Int], required Sequence.List[...]
val bad: Allows[Set[Int], Sequence.List[Primitive]] = implicitly
```

### DynamoDB-style set operations

A DynamoDB grammar can encode the distinction between set types and list types exactly, without any additional runtime proof. We use `Sequence.Set` to narrow the grammar to sets-only operations:

```scala

type N  = Primitive.Int | Primitive.Long | Primitive.Float | Primitive.Double | Primitive.Short
type S  = Primitive.String
type NS = Sequence.Set[N | Wrapped[N]]
type SS = Sequence.Set[S | Wrapped[S]]

// addSet is only callable with a Set — List[Int] or Vector[Int] would fail at compile time
def addSet[A](set: scala.collection.immutable.Set[A])(implicit
  ev: Allows[scala.collection.immutable.Set[A], NS | SS]
): String = "ok"
```

## `IsType[A]`

`IsType[A]` is a nominal type predicate. It is satisfied only when the checked Scala type is exactly `A` (i.e. `checked =:= A`). It is most useful as an element constraint inside `Sequence` subtypes, where it aligns the element type of a collection with a polymorphic method type parameter.

The primary motivation (GitHub issue #1172) is DSL methods that must constrain both the container kind and the element type in a single `Allows` expression. Without `IsType`, a separate type class (like `Containable`) is needed to connect `A` in `contains[A]` to the element type of the collection. With `IsType`, the connection is expressed directly in the grammar.

To use `IsType[A]` with a polymorphic `A`, require `IsNominalType[A]` from `zio-blocks-typeid` at the call site. This ensures the macro always sees a concrete type when it evaluates `IsType[A]`:

```scala

// `To` must be a Set whose element type is exactly `A`.
// IsNominalType[A] ensures A is concrete at the call site — an unresolved
// type parameter would fail to produce IsNominalType and the call site
// would not compile.
def contains[To, From, A: IsNominalType](a: A)(implicit
  ev: Allows[To, Sequence.Set[IsType[A]]]
): Boolean = ev.ne(null)

// Compiles: Set[Int] satisfies Sequence.Set[IsType[Int]]
val r1: Boolean = contains[Set[Int], Nothing, Int](42)

// Compiles: Set[String] satisfies Sequence.Set[IsType[String]]
val r2: Boolean = contains[Set[String], Nothing, String]("hello")
```

A mismatch between the element type and `A` is a compile-time error:

```
[error] Shape violation at ...<element>: found Primitive(java.lang.String),
        required IsType[Int]
```

`IsType[A]` can also appear as a standalone constraint, or anywhere a grammar node is accepted:

```scala

// Int satisfies IsType[Int] exactly
val ev: Allows[Int, Allows.IsType[Int]] = implicitly

// List[String] satisfies Sequence[IsType[String]]
val ev2: Allows[List[String], Allows.Sequence[Allows.IsType[String]]] = implicitly
```

## The `Self` Grammar Node

`Self` refers back to the entire enclosing `Allows[A, S]` grammar. It allows the grammar to describe recursive data structures.

**Non-recursive types** satisfy a `Self`-containing grammar whenever their parts match: `Self` checks the part at its position against the whole grammar again, and since no type comes back, the check ends. For example, `Allows[Book, Record[Primitive | Self]]` checks an `Author` field of `Book` through `Self` as a `Record[Primitive | Self]`.

**Recursion** is accepted when it leads back to the checked type `A`, directly or through other types: mutually recursive `Forest` and `Tree` checked as `Allows[Forest, Record[Primitive | Sequence[Self] | Self]]` are accepted. A recursive type below `A` is rejected, even when `Self` accepts it on its own: a non-recursive `Library` with a field of type `Topic(name: String, subtopics: List[Topic])` fails that grammar; on Scala 3 the error is a shape violation at `Library.topics`.

## `Wrapped[A]` and Newtypes

The `Wrapped[A]` node matches ZIO Prelude `Newtype` and `Subtype` wrappers and, on Scala 3, opaque types and neotype newtypes. The underlying type must satisfy `A`. A case class with one field, even an `AnyVal` one, is a `Record`, not a wrapper. Here's an example:

```scala

// ZIO Prelude Newtype pattern:
object ProductCode extends Newtype[String]
type ProductCode = ProductCode.Type

given Schema[ProductCode] =
  Schema[String].transform(_.asInstanceOf[ProductCode], _.asInstanceOf[String])

// ProductCode satisfies Wrapped[Primitive] — its underlying String is Primitive
val ev: Allows[ProductCode, Wrapped[Primitive]] = implicitly
```

**Scala 3 opaque types** are seen as their underlying type inside the scope that defines them, so there an opaque alias like this satisfies `Primitive` directly. Everywhere else the macro sees the opaque type itself, which satisfies `Wrapped[Primitive]` but not `Primitive`:

```scala
opaque type UserId = java.util.UUID
```

## Sealed Traits and Enums (Auto-Unwrap)

Sealed traits and enums are **automatically unwrapped** by the macro. Whenever a sealed type is encountered at any grammar check position, the macro recursively checks every case against the same grammar. This makes a `Variant` grammar node unnecessary.

```scala

sealed trait Shape
case class Circle(radius: Double)                    extends Shape
case class Rectangle(width: Double, height: Double)  extends Shape
case object Point                                    extends Shape

// No Variant node — Shape is auto-unwrapped, all cases checked against Record[Primitive]
val ev: Allows[Shape, Record[Primitive]] = implicitly
```

Auto-unwrap is recursive: if a case is itself a sealed trait, its cases are unwrapped too, to any depth.

Union branches (`A | B`) work naturally with auto-unwrap: unused branches are fine under `Allows` upper-bound semantics.

## Error Messages

When a type does not satisfy the grammar, the macro reports:

1. **The path** to the violating field: `Order.items.<element>`
2. **What was found**: `Record(OrderItem)`
3. **What was required**: `Primitive | Sequence[Primitive]`
4. **A hint** where applicable

Multiple violations are reported in a single compilation pass — the user sees all problems at once, for example:

```
[error] Schema shape violation at UserWithAddress.address: found Record(Address),
        required Primitive | Optional[Primitive] | Map[Primitive, Primitive]
[error]   Hint: Type 'Address' does not match any allowed shape
```

## Singleton / Zero-Field Records

`Record[A]` is vacuously true for case objects and zero-field records, since there are no fields to violate the constraint:

```scala

case object EmptyEvent
implicit val schema: Schema[EmptyEvent.type] = Schema.derived

val ev: Allows[EmptyEvent.type, Record[Primitive]] = implicitly  // vacuously true
```

## Runtime Cost

`Allows[A, S]` carries **zero runtime overhead**. The macro emits a reference to the single shared singleton `Allows.instance` cast to the required type. There is no per-call-site allocation.

## Scala 2 vs Scala 3

| Feature | Scala 2 | Scala 3 |
|---|---|---|
| Union syntax | `` A `\|` B `` infix | `A \| B` native union type |
| Summon syntax | `implicitly[Allows[A, S]]` | `summon[Allows[A, S]]` or `implicitly` |
| Evidence parameter | `(implicit ev: Allows[A, S])` | `(using Allows[A, S])` |
| Opaque type detection | ZIO Prelude only | Scala 3 opaque types + ZIO Prelude + neotype |
| Derivation keyword | `Schema.derived` implicit | `Schema.derived` or `derives Schema` |

Both Scala versions produce the same macro behavior and the same error messages.

## Integration with Schema

`Allows` and `Schema` are complementary but independent:

- **`Schema[A]`** describes what an `A` looks like at runtime — how to serialize, deserialize, introspect, or transform it. It requires explicit derivation and handles the full type signature.
- **`Allows[A, S]`** describes what an `A` *may* look like at compile time — a structural grammar that `A` must satisfy. It requires no schema and uses only the Scala type system.

You can use `Allows` **without** `Schema`:

```scala

// Pure shape constraint, no Schema required
def writeCsv[A](rows: Seq[A])(using Allows[A, Record[Primitive | Optional[Primitive]]]): Unit = ???
```

Or combine them when runtime encoding **and** shape validation are both needed:

```scala

// Shape constraint + runtime encoding
def writeCsv[A: Schema](rows: Seq[A])(using
  Allows[A, Record[Primitive | Optional[Primitive]]]
): Unit = ???
```

When combined, `Allows` enforces the structural guarantee that `Schema` can use — for example, a CSV serializer can assume that every field is a primitive or optional primitive and skip defensive type checks.

See [Schema](./schema.md) for more on runtime encoding and decoding with schemas.

## Running the Examples

All code from this guide is available as runnable examples in the `schema-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

**CSV serializer with flat record compile-time constraints**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/comptime/AllowsCsvExample.scala))

```bash
cd schema-examples && sbt "runMain comptime.AllowsCsvExample"
```

```scala title="schema-examples/src/main/scala/comptime/AllowsCsvExample.scala" 
package comptime

// ---------------------------------------------------------------------------
// CSV serializer example using Allows[A, S] compile-time shape constraints
//
// A CSV row is a flat record: every field must be a primitive scalar or an
// optional primitive (for nullable columns). Nested records, sequences, and
// maps are all rejected at compile time.
// ---------------------------------------------------------------------------

// Compatible: flat record of primitives and optional primitives
case class Employee(name: String, department: String, salary: BigDecimal, active: Boolean)
object Employee { implicit val schema: Schema[Employee] = Schema.derived }

case class SensorReading(sensorId: String, timestamp: Long, value: Double, unit: Option[String])
object SensorReading { implicit val schema: Schema[SensorReading] = Schema.derived }

object CsvSerializer {

  type FlatRow = Primitive | AOptional[Primitive]

  /** Serialize a sequence of flat records to CSV format. */
  def toCsv[A](rows: Seq[A])(implicit schema: Schema[A], ev: Allows[A, Record[FlatRow]]): String = {
    val reflect = schema.reflect.asRecord.get
    val header  = reflect.fields.map(_.name).mkString(",")
    val lines   = rows.map { row =>
      val dv = schema.toDynamicValue(row)
      dv match {
        case DynamicValue.Record(fields) =>
          fields.map { case (_, v) => csvEscape(dvToString(v)) }.mkString(",")
        case _ => ""
      }
    }
    (header +: lines).mkString("\n")
  }

  private def dvToString(dv: DynamicValue): String = dv match {
    case DynamicValue.Primitive(PrimitiveValue.String(s))     => s
    case DynamicValue.Primitive(PrimitiveValue.Boolean(b))    => b.toString
    case DynamicValue.Primitive(PrimitiveValue.Int(n))        => n.toString
    case DynamicValue.Primitive(PrimitiveValue.Long(n))       => n.toString
    case DynamicValue.Primitive(PrimitiveValue.Double(n))     => n.toString
    case DynamicValue.Primitive(PrimitiveValue.Float(n))      => n.toString
    case DynamicValue.Primitive(PrimitiveValue.BigDecimal(n)) => n.toString
    case DynamicValue.Primitive(v)                            => v.toString
    case DynamicValue.Null                                    => ""
    case DynamicValue.Variant(tag, inner) if tag == "Some"    => dvToString(inner)
    case DynamicValue.Variant(tag, _) if tag == "None"        => ""
    case DynamicValue.Record(fields)                          =>
      fields.headOption.map { case (_, v) => dvToString(v) }.getOrElse("")
    case other => other.toString
  }

  private def csvEscape(s: String): String =
    if (s.contains(",") || s.contains("\"") || s.contains("\n"))
      "\"" + s.replace("\"", "\"\"") + "\""
    else s
}

// ---------------------------------------------------------------------------
// Demonstration
// ---------------------------------------------------------------------------

object AllowsCsvExample extends App {

  // Flat records of primitives — compiles fine
  val employees = Seq(
    Employee("Alice", "Engineering", BigDecimal("120000.00"), true),
    Employee("Bob", "Marketing", BigDecimal("95000.50"), true),
    Employee("Carol", "Engineering", BigDecimal("115000.00"), false)
  )

  // CSV output for a flat record of primitives
  show(CsvSerializer.toCsv(employees))

  // Flat record with optional fields — also compiles
  val readings = Seq(
    SensorReading("temp-01", 1709712000L, 23.5, Some("celsius")),
    SensorReading("temp-02", 1709712060L, 72.1, None)
  )

  // Optional fields become empty CSV cells when None
  show(CsvSerializer.toCsv(readings))

  // The following would NOT compile — uncomment to see the error:
  //
  // case class Nested(name: String, address: Address)
  // object Nested { implicit val schema: Schema[Nested] = Schema.derived }
  // CsvSerializer.toCsv(Seq(Nested("Alice", Address("1 Main St", "NY", "10001"))))
  //   [error] Schema shape violation at Nested.address: found Record(Address),
  //           required Primitive | Optional[Primitive]
}
```

**Event bus with sealed trait auto-unwrap and nested hierarchies**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/comptime/AllowsEventBusExample.scala))

```bash
cd schema-examples && sbt "runMain comptime.AllowsEventBusExample"
```

```scala title="schema-examples/src/main/scala/comptime/AllowsEventBusExample.scala" 
package comptime

// ---------------------------------------------------------------------------
// Event bus / message broker example using Allows[A, S]
//
// Published events are typically sealed traits of flat record cases. Sealed
// traits are automatically unwrapped by the Allows macro — each case is
// checked individually against the grammar. No Variant node is needed.
//
// This example also shows nested sealed traits (auto-unwrap is recursive).
// ---------------------------------------------------------------------------

// Domain events — a sealed trait hierarchy
sealed trait AccountEvent
case class AccountOpened(accountId: String, owner: String, initialBalance: BigDecimal) extends AccountEvent
case class FundsDeposited(accountId: String, amount: BigDecimal)                       extends AccountEvent
case class FundsWithdrawn(accountId: String, amount: BigDecimal)                       extends AccountEvent
case class AccountClosed(accountId: String, reason: Option[String])                    extends AccountEvent
object AccountEvent { implicit val schema: Schema[AccountEvent] = Schema.derived }

// Nested sealed trait — InventoryEvent has a sub-hierarchy
sealed trait InventoryEvent
case class ItemAdded(sku: String, quantity: Int)   extends InventoryEvent
case class ItemRemoved(sku: String, quantity: Int) extends InventoryEvent

sealed trait InventoryAlert                      extends InventoryEvent
case class LowStock(sku: String, remaining: Int) extends InventoryAlert
case class OutOfStock(sku: String)               extends InventoryAlert

object InventoryEvent { implicit val schema: Schema[InventoryEvent] = Schema.derived }

// Event with sequence fields (e.g. tags or batch items)
sealed trait BatchEvent
case class BatchImport(batchId: String, itemIds: List[String]) extends BatchEvent
case class BatchComplete(batchId: String, count: Int)          extends BatchEvent
object BatchEvent { implicit val schema: Schema[BatchEvent] = Schema.derived }

object EventBus {

  type EventShape = Primitive | AOptional[Primitive]

  /**
   * Publish a domain event. All cases of the sealed trait must be flat records.
   */
  def publish[A](event: A)(implicit schema: Schema[A], ev: Allows[A, Record[EventShape]]): String = {
    val dv                  = schema.toDynamicValue(event)
    val (typeName, payload) = dv match {
      case DynamicValue.Variant(name, inner) => (name, inner.toJson.toString)
      case _                                 => (schema.reflect.typeId.name, dv.toJson.toString)
    }
    s"PUBLISH topic=${schema.reflect.typeId.name} type=$typeName payload=$payload"
  }

  /**
   * Publish events that may contain sequence fields (e.g. batch operations).
   */
  def publishBatch[A](event: A)(implicit
    schema: Schema[A],
    ev: Allows[A, Record[Primitive | Sequence[Primitive]]]
  ): String = {
    val dv                  = schema.toDynamicValue(event)
    val (typeName, payload) = dv match {
      case DynamicValue.Variant(name, inner) => (name, inner.toJson.toString)
      case _                                 => (schema.reflect.typeId.name, dv.toJson.toString)
    }
    s"PUBLISH topic=${schema.reflect.typeId.name} type=$typeName payload=$payload"
  }
}

// ---------------------------------------------------------------------------
// Demonstration
// ---------------------------------------------------------------------------

object AllowsEventBusExample extends App {

  // Flat sealed trait — all cases are records of primitives/optionals
  show(EventBus.publish[AccountEvent](AccountOpened("acc-001", "Alice", BigDecimal("1000.00"))))
  show(EventBus.publish[AccountEvent](FundsDeposited("acc-001", BigDecimal("500.00"))))
  show(EventBus.publish[AccountEvent](AccountClosed("acc-001", Some("customer request"))))

  // Nested sealed trait — auto-unwrap is recursive
  // InventoryAlert extends InventoryEvent, both are unwrapped
  show(EventBus.publish[InventoryEvent](ItemAdded("SKU-100", 50)))
  show(EventBus.publish[InventoryEvent](LowStock("SKU-100", 3)))
  show(EventBus.publish[InventoryEvent](OutOfStock("SKU-100")))

  // Events with sequence fields use a wider grammar
  show(EventBus.publishBatch[BatchEvent](BatchImport("batch-42", List("item-1", "item-2", "item-3"))))
  show(EventBus.publishBatch[BatchEvent](BatchComplete("batch-42", 3)))
}
```

**GraphQL / tree structures using Self for recursive grammars**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/comptime/AllowsGraphQLTreeExample.scala))

```bash
cd schema-examples && sbt "runMain comptime.AllowsGraphQLTreeExample"
```

```scala title="schema-examples/src/main/scala/comptime/AllowsGraphQLTreeExample.scala" 
package comptime

// ---------------------------------------------------------------------------
// GraphQL / tree structure example using Self for recursive grammars
//
// Self refers back to the entire enclosing Allows[A, S] grammar, allowing
// the constraint to describe recursive data structures like trees, linked
// lists, and nested menus.
//
// Non-recursive types also satisfy Self-containing grammars — the Self
// position is never reached, so the constraint is vacuously satisfied.
// ---------------------------------------------------------------------------

// Recursive tree: children reference the same type
case class TreeNode(value: Int, children: List[TreeNode])
object TreeNode { implicit val schema: Schema[TreeNode] = Schema.derived }

// Recursive category hierarchy (common in e-commerce, CMS, etc.)
case class NavCategory(name: String, slug: String, subcategories: List[NavCategory])
object NavCategory { implicit val schema: Schema[NavCategory] = Schema.derived }

// Linked list via Optional[Self]
case class Chain(label: String, next: Option[Chain])
object Chain { implicit val schema: Schema[Chain] = Schema.derived }

// Non-recursive type — satisfies Self-containing grammars vacuously
case class FlatNode(id: Int, label: String)
object FlatNode { implicit val schema: Schema[FlatNode] = Schema.derived }

object GraphQL {

  type TreeShape = Primitive | Sequence[ASelf] | AOptional[ASelf]

  /** Generate a simplified GraphQL type definition for a recursive type. */
  def graphqlType[A](implicit schema: Schema[A], ev: Allows[A, Record[TreeShape]]): String = {
    val reflect = schema.reflect.asRecord.get
    val fields  = reflect.fields.map { f =>
      s"  ${f.name}: ${gqlType(resolve(f.value), schema.reflect.typeId.name)}"
    }
    s"type ${schema.reflect.typeId.name} {\n${fields.mkString("\n")}\n}"
  }

  /** Unwrap Deferred to get the actual Reflect node. */
  private def resolve(r: Reflect.Bound[_]): Reflect.Bound[_] = r match {
    case d: Reflect.Deferred[_, _] => resolve(d.value.asInstanceOf[Reflect.Bound[_]])
    case other                     => other
  }

  private def gqlType(r: Reflect.Bound[_], selfName: String): String = r match {
    case _: Reflect.Sequence[_, _, _] => s"[$selfName]"
    case p: Reflect.Primitive[_, _]   =>
      p.primitiveType match {
        case PrimitiveType.Int(_)     => "Int"
        case PrimitiveType.Long(_)    => "Int"
        case PrimitiveType.Float(_)   => "Float"
        case PrimitiveType.Double(_)  => "Float"
        case PrimitiveType.String(_)  => "String"
        case PrimitiveType.Boolean(_) => "Boolean"
        case _                        => "String"
      }
    case _ => selfName
  }
}

// ---------------------------------------------------------------------------
// Demonstration
// ---------------------------------------------------------------------------

object AllowsGraphQLTreeExample extends App {

  // Recursive tree with Sequence[Self]
  show(GraphQL.graphqlType[TreeNode])

  // Recursive categories — same grammar, different domain
  show(GraphQL.graphqlType[NavCategory])

  // Linked list via Optional[Self]
  show(GraphQL.graphqlType[Chain])

  // Non-recursive type also satisfies the grammar (vacuously — Self is never reached)
  show(GraphQL.graphqlType[FlatNode])

  // Show that recursive data actually works at runtime
  val tree = TreeNode(
    1,
    List(
      TreeNode(2, List(TreeNode(4, Nil), TreeNode(5, Nil))),
      TreeNode(3, Nil)
    )
  )
  show(Schema[TreeNode].toDynamicValue(tree).toJson.toString)

  val nav = NavCategory(
    "Electronics",
    "electronics",
    List(
      NavCategory("Phones", "phones", Nil),
      NavCategory(
        "Laptops",
        "laptops",
        List(
          NavCategory("Gaming", "gaming", Nil)
        )
      )
    )
  )
  show(Schema[NavCategory].toDynamicValue(nav).toJson.toString)
}
```

**Sealed trait auto-unwrap with nested hierarchies and case objects**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/comptime/AllowsSealedTraitExample.scala))

```bash
cd schema-examples && sbt "runMain comptime.AllowsSealedTraitExample"
```

```scala title="schema-examples/src/main/scala/comptime/AllowsSealedTraitExample.scala" 
package comptime

// ---------------------------------------------------------------------------
// Sealed trait auto-unwrap example
//
// Sealed traits and enums are automatically unwrapped by the Allows macro.
// Each case is checked individually against the grammar — no Variant node
// is needed.
//
// Auto-unwrap is recursive: if a case is itself a sealed trait, its cases
// are unwrapped too, to any depth.
//
// Zero-field records (case objects) are vacuously true for any Record[A]
// constraint.
// ---------------------------------------------------------------------------

// Simple sealed trait with case classes and a case object
sealed trait Shape
case class Circle(radius: Double)                   extends Shape
case class Rectangle(width: Double, height: Double) extends Shape
case object Point                                   extends Shape
object Shape { implicit val schema: Schema[Shape] = Schema.derived }

// Nested sealed trait hierarchy — two levels deep
sealed trait Expr
sealed trait BinaryOp                            extends Expr
case class Add(left: Double, right: Double)      extends BinaryOp
case class Multiply(left: Double, right: Double) extends BinaryOp
case class Literal(value: Double)                extends Expr
case object Zero                                 extends Expr
object Expr { implicit val schema: Schema[Expr] = Schema.derived }

// All-singleton enum (all case objects)
sealed trait Color
case object Red   extends Color
case object Green extends Color
case object Blue  extends Color
object Color { implicit val schema: Schema[Color] = Schema.derived }

object SealedTraitValidator {

  /** Validate that a value's type has a flat record structure. */
  def validate[A](value: A)(implicit schema: Schema[A], ev: Allows[A, Record[Primitive]]): String = {
    val dv = schema.toDynamicValue(value)
    dv match {
      case DynamicValue.Variant(caseName, inner) =>
        s"Valid variant case '$caseName': ${inner.toJson}"
      case DynamicValue.Record(fields) =>
        s"Valid record with ${fields.size} field(s): ${fields.map(_._1).mkString(", ")}"
      case _ =>
        s"Valid: ${dv.toJson}"
    }
  }
}

// ---------------------------------------------------------------------------
// Demonstration
// ---------------------------------------------------------------------------

object AllowsSealedTraitExample extends App {

  // Simple sealed trait — all cases checked against Record[Primitive]
  // Circle: Record(radius: Double) — satisfies Record[Primitive]
  // Rectangle: Record(width: Double, height: Double) — satisfies Record[Primitive]
  // Point: zero-field case object — vacuously true
  show(SealedTraitValidator.validate[Shape](Circle(3.14)))
  show(SealedTraitValidator.validate[Shape](Rectangle(4.0, 5.0)))
  show(SealedTraitValidator.validate[Shape](Point))

  // Nested sealed trait — auto-unwrap is recursive
  // BinaryOp is itself sealed with Add and Multiply
  // All leaf cases have only Double fields — satisfies Record[Primitive]
  show(SealedTraitValidator.validate[Expr](Add(1.0, 2.0)))
  show(SealedTraitValidator.validate[Expr](Multiply(3.0, 4.0)))
  show(SealedTraitValidator.validate[Expr](Literal(42.0)))
  show(SealedTraitValidator.validate[Expr](Zero))

  // All-singleton enum — every case is a zero-field record (vacuously true)
  show(SealedTraitValidator.validate[Color](Red))
  show(SealedTraitValidator.validate[Color](Green))
  show(SealedTraitValidator.validate[Color](Blue))
}
```

**RDBMS library with CREATE TABLE and INSERT using flat record constraints** (compile-only)
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/comptime/RdbmsExample.scala))

Demonstrates how Allows constraints are verified at compile time — the code below shows valid examples that compile successfully, and includes comments showing which patterns would be rejected:

```scala title="schema-examples/src/main/scala/comptime/RdbmsExample.scala" 
package comptime

// ---------------------------------------------------------------------------
// Realistic RDBMS example using Allows[A, S] compile-time shape constraints
//
// Demonstrates how a library can require that user-supplied types have a
// structure compatible with what a relational database can represent:
//   - Flat records of primitives, optional primitives, or primitive-valued maps
//   - Top-level variants (enum tables) whose cases are flat records
//
// Incompatible types (nested records, sequences of records, etc.) are
// rejected at the call site with a precise compile-time error message.
// ---------------------------------------------------------------------------

// ---------------------------------------------------------------------------
// Domain types — vary from compatible to incompatible
// ---------------------------------------------------------------------------

// Compatible: flat record of primitives and optional primitives
case class UserRow(
  id: java.util.UUID,
  name: String,
  email: Option[String],
  age: Int,
  active: Boolean
)
object UserRow {
  implicit val schema: Schema[UserRow] = Schema.derived
}

// Compatible: flat record with a string-keyed map column (stored as JSON/JSONB)
case class ProductRow(
  id: java.util.UUID,
  name: String,
  price: BigDecimal,
  attributes: scala.collection.immutable.Map[String, String]
)
object ProductRow {
  implicit val schema: Schema[ProductRow] = Schema.derived
}

// Compatible: event table — a variant (sealed trait) of flat record cases
sealed trait DomainEvent
case class UserCreated(id: java.util.UUID, name: String, email: String)               extends DomainEvent
case class UserDeleted(id: java.util.UUID)                                            extends DomainEvent
case class OrderPlaced(id: java.util.UUID, userId: java.util.UUID, total: BigDecimal) extends DomainEvent
object DomainEvent {
  implicit val schema: Schema[DomainEvent] = Schema.derived
}

// Incompatible: contains a nested record (Address is not a primitive)
case class Address(street: String, city: String, zip: String)
object Address { implicit val schema: Schema[Address] = Schema.derived }

case class UserWithAddress(
  id: java.util.UUID,
  name: String,
  address: Address // ← incompatible: nested record
)
object UserWithAddress {
  implicit val schema: Schema[UserWithAddress] = Schema.derived
}

// ---------------------------------------------------------------------------
// Simulated RDBMS library API
//
// The Allows constraint is checked at the CALL SITE — the library author
// writes these signatures once. Users get a compile-time error if their type
// doesn't match, with a message pointing to the exact violating field.
// ---------------------------------------------------------------------------

object Rdbms {

  // A flat record row: primitives, optional primitives, or string-keyed maps
  type FlatRow = Primitive | AOptional[Primitive] | AMap[Primitive, Primitive]

  /** Generate a CREATE TABLE DDL statement for a flat record type. */
  def createTable[A](implicit
    schema: Schema[A],
    ev: Allows[A, Record[FlatRow]]
  ): String = {
    val fields = schema.reflect.asRecord.get.fields
    val cols   = fields.map { f =>
      val tpe = sqlType(f.value)
      s"  ${f.name} $tpe"
    }
    s"CREATE TABLE ${tableName(schema)} (\n${cols.mkString(",\n")}\n)"
  }

  /** Generate an INSERT statement for a single flat record row. */
  def insert[A](value: A)(implicit
    schema: Schema[A],
    ev: Allows[A, Record[FlatRow]]
  ): String = {
    val dv = schema.toDynamicValue(value)
    dv match {
      case DynamicValue.Record(fields) =>
        val cols = fields.map(_._1).mkString(", ")
        val vals = fields.map { case (_, v) => sqlLiteralDv(v) }.mkString(", ")
        s"INSERT INTO ${tableName(schema)} ($cols) VALUES ($vals)"
      case _ => s"INSERT INTO ${tableName(schema)} VALUES (?)"
    }
  }

  /**
   * Insert an event into an event-sourcing table.
   *
   * The type must be a sealed trait / enum whose cases are flat records. No
   * explicit Variant node is needed — sealed traits are auto-unwrapped by the
   * macro.
   */
  def insertEvent[A](event: A)(implicit
    schema: Schema[A],
    ev: Allows[A, Record[FlatRow]]
  ): String = {
    val typeName = schema.toDynamicValue(event) match {
      case DynamicValue.Variant(caseName, _) => caseName
      case _                                 => schema.reflect.typeId.name
    }
    val payload = schema.toDynamicValue(event).toJson.toString
    s"INSERT INTO events (type, payload) VALUES ('$typeName', '$payload')"
  }

  // ---------------------------------------------------------------------------
  // Helpers
  // ---------------------------------------------------------------------------

  private def tableName[A](schema: Schema[A]): String =
    schema.reflect.modifiers.collectFirst {
      case Modifier.config(k, v) if k == "sql.table_name" => v
    }.getOrElse(schema.reflect.typeId.name.toLowerCase + "s")

  private def sqlLiteralDv(dv: DynamicValue): String = dv match {
    case DynamicValue.Primitive(PrimitiveValue.String(s))  => s"'${s.replace("'", "''")}'"
    case DynamicValue.Primitive(PrimitiveValue.Boolean(b)) => if (b) "TRUE" else "FALSE"
    case DynamicValue.Primitive(v)                         => v.toString
    case DynamicValue.Null                                 => "NULL"
    case other                                             => s"'${other.toString.replace("'", "''")}'"
  }

  private def sqlType(reflect: Reflect.Bound[_]): String = reflect match {
    case p: Reflect.Primitive[_, _] =>
      p.primitiveType match {
        case PrimitiveType.Int(_)           => "INTEGER"
        case PrimitiveType.Long(_)          => "BIGINT"
        case PrimitiveType.String(_)        => "TEXT"
        case PrimitiveType.Boolean(_)       => "BOOLEAN"
        case PrimitiveType.Double(_)        => "DOUBLE PRECISION"
        case PrimitiveType.Float(_)         => "REAL"
        case PrimitiveType.BigDecimal(_)    => "NUMERIC"
        case PrimitiveType.UUID(_)          => "UUID"
        case PrimitiveType.Instant(_)       => "TIMESTAMPTZ"
        case PrimitiveType.LocalDate(_)     => "DATE"
        case PrimitiveType.LocalDateTime(_) => "TIMESTAMP"
        case _                              => "TEXT"
      }
    case _: Reflect.Map[_, _, _, _]   => "JSONB"
    case _: Reflect.Sequence[_, _, _] => "TEXT[]"
    case _                            => "TEXT"
  }
}

// ---------------------------------------------------------------------------
// Demonstration — these all compile
// ---------------------------------------------------------------------------

object RdbmsDemo {

  // Flat rows compile fine
  val createUser: String    = Rdbms.createTable[UserRow]
  val createProduct: String = Rdbms.createTable[ProductRow]
  val insertUser: String    = Rdbms.insert(UserRow(new java.util.UUID(0, 0), "Alice", Some("a@b.com"), 30, true))
  val insertEvent: String   = Rdbms.insertEvent[DomainEvent](UserCreated(new java.util.UUID(0, 0), "Alice", "a@b.com"))

  // The following would NOT compile — uncomment to see the error:
  //
  // val bad = Rdbms.createTable[UserWithAddress]
  //   [error] Schema shape violation at UserWithAddress.address: found Record(Address), required
  //           Primitive | Optional[Primitive] | Map[Primitive, Primitive]
}
```

**JSON document store with specific primitives and recursive Self grammar** (compile-only)
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/comptime/DocumentStoreExample.scala))

Demonstrates how Allows enforces recursive schema constraints at compile time:

```scala title="schema-examples/src/main/scala/comptime/DocumentStoreExample.scala" 
package comptime

// ---------------------------------------------------------------------------
// Realistic JSON document-store example using Allows[A, S]
//
// JSON has a limited primitive value set:
//   - null    → Unit / Option
//   - boolean → Boolean
//   - number  → Int, Long, Float, Double, BigDecimal, BigInt
//   - string  → String
//
// Notably absent from JSON: Char, Byte, Short, UUID, Currency,
// and ALL java.time.* types. A JSON library should use SPECIFIC primitive
// nodes to reject non-JSON scalars at the call site.
//
// The grammar is simply:
//   type Json = Record[JsonPrimitive | Self] | Sequence[JsonPrimitive | Self]
//
// Self handles all nesting: a field may be a primitive or another Json value.
// Sequences of primitives (List[String]) and sequences of records
// (List[Author]) both satisfy Sequence[JsonPrimitive | Self].
// Optional fields (Option[X]) satisfy Record[...] because Option is recognised
// by the Optional grammar node which falls through to the field constraint.
// ---------------------------------------------------------------------------

// ---------------------------------------------------------------------------
// Domain types
// ---------------------------------------------------------------------------

// Compatible: flat document
case class Author(name: String, email: String)
object Author { implicit val schema: Schema[Author] = Schema.derived }

// Compatible: document with nested documents and sequences
case class BookChapter(title: String, wordCount: Int)
object BookChapter { implicit val schema: Schema[BookChapter] = Schema.derived }

case class Book(
  title: String,
  author: Author,              // nested record — satisfied via Self
  chapters: List[BookChapter], // sequence of records — satisfied via Sequence[Self]
  tags: List[String],          // sequence of primitives — satisfied via Sequence[JsonPrimitive]
  rating: Option[Double]       // optional primitive — satisfied via Optional[JsonPrimitive | Self]
)
object Book { implicit val schema: Schema[Book] = Schema.derived }

// Compatible: recursive document
case class Category(name: String, subcategories: List[Category])
object Category { implicit val schema: Schema[Category] = Schema.derived }

// Compatible: variant of search results (for indexing)
sealed trait SearchResult
case class BookResult(title: String, score: Double)   extends SearchResult
case class AuthorResult(name: String, bookCount: Int) extends SearchResult
object SearchResult { implicit val schema: Schema[SearchResult] = Schema.derived }

// INCOMPATIBLE: UUID is not a JSON-native scalar
case class WithUUID(id: java.util.UUID, name: String)
object WithUUID { implicit val schema: Schema[WithUUID] = Schema.derived }

// INCOMPATIBLE: Instant is not a JSON-native scalar
case class WithTimestamp(name: String, createdAt: java.time.Instant)
object WithTimestamp { implicit val schema: Schema[WithTimestamp] = Schema.derived }

// ---------------------------------------------------------------------------
// Document store library API
// ---------------------------------------------------------------------------

object DocumentStore {

  /**
   * JSON-representable scalar types.
   *
   * Excludes Char, Byte, Short, UUID, Currency, and all java.time.* types —
   * none of these have a native JSON encoding. Authors who need java.time
   * values in JSON should store them as Primitive.String (ISO-8601 etc.).
   */
  type JsonPrimitive =
    Allows.Primitive.Boolean | Allows.Primitive.Int | Allows.Primitive.Long | Allows.Primitive.Float |
      Allows.Primitive.Double | Allows.Primitive.String | Allows.Primitive.BigDecimal | Allows.Primitive.BigInt |
      Allows.Primitive.Unit

  /**
   * A JSON value is either a JSON object (Record) or a JSON array (Sequence).
   * Self recurses back to this same grammar, so nesting works at any depth.
   * Optional covers nullable fields (JSON null / absent key).
   */
  type Json = Record[JsonPrimitive | AOptional[JsonPrimitive | ASelf] | ASelf] | Sequence[JsonPrimitive | ASelf]

  /**
   * Encode a value to its JSON string representation.
   *
   * Accepts both JSON objects (records) and JSON arrays (sequences) at the top
   * level. Fields may be primitives, nested documents, sequences, or optionals
   * — all handled via Self and the JsonPrimitive constraint. Types containing
   * UUID, Instant, Char etc. are rejected at compile time.
   */
  def toJson[A: Schema](doc: A)(implicit ev: Allows[A, Json]): String =
    Schema[A].toDynamicValue(doc).toJson.toString

  /** Serialize to DynamicValue for further processing. */
  def serialize[A: Schema](doc: A)(implicit ev: Allows[A, Json]): DynamicValue =
    Schema[A].toDynamicValue(doc)

  /**
   * Index a search result. The type must be a sealed trait of flat JSON
   * records. No explicit Variant node needed — sealed traits are
   * auto-unwrapped.
   */
  def index[A: Schema](result: A)(implicit
    ev: Allows[A, Record[JsonPrimitive | AOptional[JsonPrimitive | ASelf] | Sequence[JsonPrimitive | ASelf]]]
  ): String = {
    val typeName = Schema[A].toDynamicValue(result) match {
      case DynamicValue.Variant(name, _) => name
      case _                             => Schema[A].reflect.typeId.name
    }
    val payload = Schema[A].toDynamicValue(result).toJson.toString
    s"""{"_type":"$typeName","_doc":$payload}"""
  }
}

// ---------------------------------------------------------------------------
// Demonstration
// ---------------------------------------------------------------------------

object DocumentStoreDemo {

  // JSON objects compile fine
  val bookJson: String = DocumentStore.toJson(
    Book(
      "ZIO Blocks",
      Author("John", "john@example.com"),
      List(BookChapter("Intro", 1000)),
      List("scala", "zio"),
      Some(4.9)
    )
  )

  val categoryJson: String = DocumentStore.toJson(
    Category("Programming", List(Category("Scala", Nil)))
  )

  // JSON arrays also satisfy Json (top-level Sequence)
  val tagListJson: String    = DocumentStore.toJson(List("scala", "zio", "functional"))
  val authorListJson: String = DocumentStore.toJson(List(Author("Alice", "a@b.com"), Author("Bob", "b@b.com")))

  val indexed: String = DocumentStore.index[SearchResult](BookResult("ZIO Blocks", 0.99))

  // The following would NOT compile — uncomment to see the errors:
  //
  // DocumentStore.toJson(WithUUID(new java.util.UUID(0, 0), "Alice"))
  //   [error] Schema shape violation at WithUUID.id: found Primitive(java.util.UUID),
  //           required JsonPrimitive (Boolean | Int | Long | Float | Double | String | ...)
  //   UUID is not a JSON-native type — encode it as Primitive.String.
  //
  // DocumentStore.toJson(WithTimestamp("Alice", java.time.Instant.EPOCH))
  //   [error] Schema shape violation at WithTimestamp.createdAt: found Primitive(java.time.Instant)
  //   Instant is not a JSON-native type — encode it as Primitive.String (ISO-8601).
}
```

---

## BindingResolver

`BindingResolver` is the **read-only interface for looking up bindings by type identity** during schema rebinding. Given a type `A`, a resolver searches its internal storage and returns either the matching `Binding` or `None`.

```scala
trait BindingResolver {
  def resolveRecord[A](implicit typeId: TypeId[A]): Option[Binding.Record[A]]
  def resolveVariant[A](implicit typeId: TypeId[A]): Option[Binding.Variant[A]]
  def resolvePrimitive[A](implicit typeId: TypeId[A]): Option[Binding.Primitive[A]]
  def resolveWrapper[A](implicit typeId: TypeId[A]): Option[Binding.Wrapper[A, _]]
  def resolveDynamic(implicit typeId: TypeId[DynamicValue]): Option[Binding.Dynamic]
  def resolveSeq[X](implicit typeId: TypeId[X], u: UnapplySeq[X]): Option[Binding.Seq[u.C, u.A]]
  def resolveSeqFor[C[_], A](typeId: TypeId[C[A]]): Option[Binding.Seq[C, A]]
  def resolveMap[X](implicit typeId: TypeId[X], u: UnapplyMap[X]): Option[Binding.Map[u.M, u.K, u.V]]
  def resolveMapFor[M[_, _], K, V](typeId: TypeId[M[K, V]]): Option[Binding.Map[M, K, V]]
  final def ++(that: BindingResolver): BindingResolver
}
```

`BindingResolver`:

- Uses `TypeId` as the lookup key for all resolution methods.
- Stores sequence and map bindings by their *unapplied type constructor*, so one `List` binding covers `List[Int]`, `List[String]`, and any other element type.
- Composes via `BindingResolver#++` with left-biased precedence: the left resolver is tried first, the right serves as fallback.

## Motivation

ZIO Blocks separates schema **structure** from schema **behavior**:

- A `Reflect.Unbound[A]` carries only structural metadata—field names, type names, documentation. It contains no Scala functions and is fully serializable.
- A `Reflect.Bound[A]` pairs each structural node with its `Binding`, enabling actual construction and deconstruction of values.

This separation powers a key workflow: serialize a schema as a `DynamicSchema` (unbound), transmit or store it, then **rebind** it on the other side using a `BindingResolver` to recover a fully operational `Schema[A]`.

```
  Schema[Person]
       │  toDynamicSchema
       ▼
  DynamicSchema          ◄── serializable, no Scala functions
  (Reflect.Unbound[_])
       │  rebind[Person](resolver)
       ▼
  Schema[Person]         ◄── operational, can encode and decode
  (Reflect.Bound[Person])
```

The `BindingResolver` is what bridges the final step: it supplies the `Binding` for each node in the unbound reflect tree.

A minimal end-to-end example:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

val dynamic: DynamicSchema = Schema[Person].toDynamicSchema

val resolver: BindingResolver =
  BindingResolver.empty.bind(Binding.of[Person]) ++ BindingResolver.defaults

val rebound: Schema[Person] = dynamic.rebind[Person](resolver)
```

## Predefined Resolvers

ZIO Blocks ships three ready-made resolvers. We almost always compose them with `BindingResolver#++` rather than using them in isolation.

### `BindingResolver.empty`

`BindingResolver.empty` is an empty `Registry` with no bindings. It is the starting point for building a custom registry with `Registry#bind`:

```scala

val empty: BindingResolver.Registry = BindingResolver.empty
```

### `BindingResolver.defaults`

`BindingResolver.defaults` is a pre-populated `Registry` covering all primitive types, `java.time` types, `java.util.UUID`, `java.util.Currency`, `DynamicValue`, common sequence types (`List`, `Vector`, `Set`, `IndexedSeq`, `Seq`, `Chunk`), and `Map`. In practice we place it at the right end of a `BindingResolver#++` chain so custom bindings can override it when needed.

The types covered by `defaults` include:

| Category    | Types                                                                                                                                                                                                       |
|-------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Primitives  | `Unit`, `Boolean`, `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `Char`, `String`, `BigInt`, `BigDecimal`                                                                                             |
| `java.time` | `DayOfWeek`, `Duration`, `Instant`, `LocalDate`, `LocalDateTime`, `LocalTime`, `Month`, `MonthDay`, `OffsetDateTime`, `OffsetTime`, `Period`, `Year`, `YearMonth`, `ZoneId`, `ZoneOffset`, `ZonedDateTime` |
| `java.util` | `Currency`, `UUID`                                                                                                                                                                                          |
| Dynamic     | `DynamicValue`                                                                                                                                                                                              |
| Sequences   | `List`, `Vector`, `Set`, `IndexedSeq`, `Seq`, `Chunk`                                                                                                                                                      |
| Maps        | `Map`                                                                                                                                                                                                       |

```scala

val defaults: BindingResolver.Registry = BindingResolver.defaults

defaults.resolvePrimitive[Int]               // Some(...)
defaults.resolvePrimitive[java.time.Instant] // Some(...)
defaults.resolveSeq[List[Int]]               // Some(...)
defaults.resolveMap[Map[String, Int]]        // Some(...)
```

### `BindingResolver.reflection` (JVM only)

`BindingResolver.reflection` derives `Binding.Record` instances at runtime using Java reflection for case classes. Derived bindings are cached per `TypeId` in a `ConcurrentHashMap`, so the reflection cost is paid only once per type.

On Scala.js, `BindingResolver.reflection` is a no-op resolver that returns `None` for every query.

```scala

case class Order(id: Long, item: String, quantity: Int)

object Order {
  implicit val schema: Schema[Order] = Schema.derived[Order]
}

// reflection handles the record; defaults handles Long, String, Int
val resolver: BindingResolver = BindingResolver.reflection ++ BindingResolver.defaults

val rebound: Schema[Order] = Schema[Order].toDynamicSchema.rebind[Order](resolver)
```

:::warning
`BindingResolver.reflection` only derives `Binding.Record` for case classes. It returns `None` for primitives, variants, wrappers, sequences, and maps. Always compose it with `BindingResolver.defaults` to cover those types.
:::

## Building a Registry

`BindingResolver.Registry` is an immutable, map-backed resolver. Every `Registry#bind` call returns a **new** `Registry` with the binding added; the original registry is unchanged.

### `Registry#bind` for proper types

The unified `Registry#bind` method accepts any proper binding—`Record`, `Variant`, `Primitive`, `Wrapper`, or `Dynamic`—and dispatches to the correct internal storage slot automatically. We typically call `Binding.of[A]` to derive the right binding at compile time:

```scala

sealed trait Color
case object Red  extends Color
case object Blue extends Color

object Color {
  implicit val schema: Schema[Color] = Schema.derived[Color]
}

case class Palette(primary: Color, name: String)

object Palette {
  implicit val schema: Schema[Palette] = Schema.derived[Palette]
}

val registry: BindingResolver.Registry =
  BindingResolver.empty
    .bind(Binding.of[Color])    // derives Binding.Variant[Color]
    .bind(Binding.of[Palette])  // derives Binding.Record[Palette]
```

`Binding.of[A]` is a compile-time macro that selects the appropriate binding kind based on the type of `A`:

- Case class → `Binding.Record`
- Sealed trait or enum → `Binding.Variant`
- Scalar type (`Int`, `String`, …) → `Binding.Primitive`
- Single-field wrapper with a smart constructor → `Binding.Wrapper`

:::warning
Passing a `Binding.Seq` or `Binding.Map` to the unified `Registry#bind` method throws `IllegalArgumentException` at runtime. Use the specialized overloads shown below for collection types.
:::

### `Registry#bind` for sequence types

Sequence bindings are keyed by their unapplied type constructor. We supply `[C[_]]` as the explicit type parameter so the compiler uses the constructor—not a specific applied type—as the lookup key:

```scala

val registry: BindingResolver.Registry =
  BindingResolver.empty.bind[List](Binding.Seq.list[Nothing])

// One binding resolves any element type
registry.resolveSeq[List[Int]]    // Some(...)
registry.resolveSeq[List[String]] // Some(...)
```

### `Registry#bind` for map types

Map bindings follow the same pattern with `[M[_, _]]` as the type parameter:

```scala

val registry: BindingResolver.Registry =
  BindingResolver.empty.bind[Map](Binding.Map.map[Nothing, Nothing])

registry.resolveMap[Map[String, Int]] // Some(...)
registry.resolveMap[Map[Int, String]] // Some(...)
```

## Combining Resolvers

The `BindingResolver#++` operator composes two resolvers into a left-biased fallback chain. The left resolver is consulted first; the right is used only when the left returns `None`.

```scala
trait BindingResolver {
  final def ++(that: BindingResolver): BindingResolver
}
```

This makes it straightforward to layer custom bindings over the built-in defaults:

```scala

case class UserId(value: Long)

object UserId {
  implicit val schema: Schema[UserId] =
    Schema[Long].transform(UserId(_), _.value)
}

// Custom bindings shadow matching entries in defaults
val resolver: BindingResolver =
  BindingResolver.empty.bind(Binding.Wrapper[UserId, Long](UserId(_), _.value)) ++ BindingResolver.defaults
```

`BindingResolver#++` is associative for resolution outcomes: `(a ++ b) ++ c` and `a ++ (b ++ c)` always resolve to the same binding for any type.

## Resolution Methods

All `resolve*` methods return an `Option` and never throw. They return `None` when no binding is registered for the requested type.

### `BindingResolver#resolveRecord`

`BindingResolver#resolveRecord` returns the `Binding.Record` for a product type (case class, tuple, module object):

```scala
trait BindingResolver {
  def resolveRecord[A](implicit typeId: TypeId[A]): Option[Binding.Record[A]]
}
```

The `TypeId[A]` witness is satisfied implicitly; we only need to supply the type parameter:

```scala

case class Point(x: Double, y: Double)

object Point {
  implicit val schema: Schema[Point] = Schema.derived[Point]
}

val registry = BindingResolver.empty.bind(Binding.of[Point])
val binding: Option[Binding.Record[Point]] = registry.resolveRecord[Point]
```

### `BindingResolver#resolveVariant`

`BindingResolver#resolveVariant` returns the `Binding.Variant` for a sum type (sealed trait, Scala 3 enum):

```scala
trait BindingResolver {
  def resolveVariant[A](implicit typeId: TypeId[A]): Option[Binding.Variant[A]]
}
```

```scala

sealed trait Shape
case class Circle(radius: Double)          extends Shape
case class Rectangle(w: Double, h: Double) extends Shape

object Shape {
  implicit val schema: Schema[Shape] = Schema.derived[Shape]
}

val registry = BindingResolver.empty.bind(Binding.of[Shape])
val binding: Option[Binding.Variant[Shape]] = registry.resolveVariant[Shape]
```

### `BindingResolver#resolvePrimitive`

`BindingResolver#resolvePrimitive` returns the `Binding.Primitive` for scalar types such as `Int`, `String`, `java.time.Instant`, and `java.util.UUID`:

```scala
trait BindingResolver {
  def resolvePrimitive[A](implicit typeId: TypeId[A]): Option[Binding.Primitive[A]]
}
```

```scala

val binding: Option[Binding.Primitive[Int]] = BindingResolver.defaults.resolvePrimitive[Int]
```

### `BindingResolver#resolveWrapper`

`BindingResolver#resolveWrapper` returns the `Binding.Wrapper` for wrapper types, such as the result of `Schema#transform`. The binding holds a `wrap: B => A` and an `unwrap: A => B` function, converting between the inner type `B` and the outer type `A`:

```scala
trait BindingResolver {
  def resolveWrapper[A](implicit typeId: TypeId[A]): Option[Binding.Wrapper[A, _]]
}
```

```scala

case class Email(value: String)

object Email {
  implicit val schema: Schema[Email] =
    Schema[String].transform(Email(_), _.value)
}

// Binding.of[Email] would derive a Binding.Record, so we bind the wrapper explicitly
val registry = BindingResolver.empty.bind(Binding.Wrapper[Email, String](Email(_), _.value))
val binding: Option[Binding.Wrapper[Email, _]] = registry.resolveWrapper[Email]
```

`Binding.of` derives a `Binding.Wrapper` only for a class with a smart constructor (a companion `apply` that returns `Either`), a zio-prelude newtype or, on Scala 3, an opaque type. For a plain single-field case class such as `Email` it derives a `Binding.Record`, which `resolveWrapper` does not return, so a schema built with `Schema#transform` needs an explicit `Binding.Wrapper`.

### `BindingResolver#resolveDynamic`

`BindingResolver#resolveDynamic` returns the `Binding.Dynamic` singleton, used for `DynamicValue` nodes in the reflect tree. `BindingResolver.defaults` already includes it, so manual registration is rarely needed:

```scala
trait BindingResolver {
  def resolveDynamic(implicit typeId: TypeId[DynamicValue]): Option[Binding.Dynamic]
}
```

```scala

val binding: Option[Binding.Dynamic] = BindingResolver.defaults.resolveDynamic
```

### `BindingResolver#resolveSeq` and `BindingResolver#resolveSeqFor`

`BindingResolver#resolveSeq` uses `UnapplySeq` evidence to decompose an applied type like `List[Int]` into its constructor `List` and element type `Int`, then looks up the binding by constructor. `BindingResolver#resolveSeqFor` is the explicit variant when the constructor and element types are already known as separate parameters:

```scala
trait BindingResolver {
  def resolveSeq[X](implicit typeId: TypeId[X], u: UnapplySeq[X]): Option[Binding.Seq[u.C, u.A]]
  def resolveSeqFor[C[_], A](typeId: TypeId[C[A]]): Option[Binding.Seq[C, A]]
}
```

Because the binding is stored by type constructor, a single registered `List` binding handles any element type:

```scala

val defaults = BindingResolver.defaults

val listIntBinding: Option[Binding.Seq[List, Int]]    = defaults.resolveSeq[List[Int]]
val listStrBinding: Option[Binding.Seq[List, String]] = defaults.resolveSeq[List[String]]

// Explicit form using resolveSeqFor
val explicit: Option[Binding.Seq[List, Int]] =
  defaults.resolveSeqFor[List, Int](TypeId.of[List[Int]])
```

### `BindingResolver#resolveMap` and `BindingResolver#resolveMapFor`

`BindingResolver#resolveMap` and `BindingResolver#resolveMapFor` follow the same pattern as their sequence counterparts, applied to key-value collection types:

```scala
trait BindingResolver {
  def resolveMap[X](implicit typeId: TypeId[X], u: UnapplyMap[X]): Option[Binding.Map[u.M, u.K, u.V]]
  def resolveMapFor[M[_, _], K, V](typeId: TypeId[M[K, V]]): Option[Binding.Map[M, K, V]]
}
```

```scala

val binding: Option[Binding.Map[Map, String, Int]] =
  BindingResolver.defaults.resolveMap[Map[String, Int]]
```

## Registry Inspection

`Registry` exposes several methods to interrogate its state without performing a full resolution. `Registry#contains` checks whether a proper binding (Record, Variant, Primitive, Wrapper, or Dynamic) is registered for type `A`. `Registry#containsSeq` and `Registry#containsMap` check for sequence and map type constructors respectively:

```scala
final class Registry {
  def contains[A](implicit typeId: TypeId[A]): Boolean
  def containsSeq[X](implicit typeId: TypeId[X], u: UnapplySeq[X]): Boolean
  def containsMap[X](implicit typeId: TypeId[X], u: UnapplyMap[X]): Boolean
  def size: Int
  def isEmpty: Boolean
  def nonEmpty: Boolean
}
```

```scala

val registry = BindingResolver.defaults

registry.contains[Int]              // true — primitive binding exists
registry.containsSeq[List[String]]  // true — List constructor is bound
registry.containsMap[Map[Int, Int]] // true — Map constructor is bound
registry.size                       // total number of registered bindings
```

## Integration with `DynamicSchema`

The primary consumer of `BindingResolver` is `DynamicSchema#rebind`. Given an unbound `DynamicSchema`, `DynamicSchema#rebind` walks the `Reflect` tree and queries the resolver for each node's binding, then returns a fully operational `Schema[A]`:

```scala

case class Product(sku: String, price: Double, tags: List[String])

object Product {
  implicit val schema: Schema[Product] = Schema.derived[Product]
}

val dynamic: DynamicSchema = Schema[Product].toDynamicSchema

// The resolver must cover every concrete type that appears in the schema tree
val resolver: BindingResolver =
  BindingResolver.empty.bind(Binding.of[Product]) ++ BindingResolver.defaults

val rebound: Schema[Product] = dynamic.rebind[Product](resolver)
```

When using `BindingResolver.reflection` on the JVM, individual record types do not need to be registered explicitly—the resolver derives their bindings on demand from the class structure:

```scala

case class Address(street: String, city: String, zip: String)

object Address {
  implicit val schema: Schema[Address] = Schema.derived[Address]
}

// No explicit bind(Binding.of[Address]) needed on JVM
val resolver: BindingResolver = BindingResolver.reflection ++ BindingResolver.defaults

val rebound: Schema[Address] = Schema[Address].toDynamicSchema.rebind[Address](resolver)
```

:::warning
If `DynamicSchema#rebind` cannot find a binding for any type present in the unbound schema tree, it throws at runtime. Make sure the resolver covers every concrete type—records, variants, wrappers, primitives, and collections—that appears in the schema.
:::

See [Binding](binding.md) for details on each binding kind, and [Schema](schema.md) for the overall structure of the schema system.

See also: [How to Build a Schema Registry with DynamicSchema](../../guides/build-a-schema-registry.md), a guide that rebinds stored schemas to Scala types with a `BindingResolver`.

---

## Binding

`Binding` is a sealed trait in ZIO Blocks that provides the operational machinery for constructing and deconstructing values of schema-described types. While `Reflect` describes the **structure** of data types, `Binding` provides the **behavior** needed to work with those types at runtime.

In other words, a binding is used to attach non-serializable Scala functions, such as **constructors**, **deconstructors**, and **matchers**, to a reflection type.

The combination of `Reflect` and `Binding` forms the foundation of `Schema`: a bound reflect (one where `F[_, _] = Binding`) that can both describe and manipulate data.

```scala
sealed trait Binding[T, +A]
```

The `Binding` trait is parameterized by:

- **`T`**: A phantom type, one of the `BindingType` members, that identifies the kind of binding (Record, Variant, Seq, Map, etc.)
- **`A`**: The Scala type that this binding operates on

## The Role of Binding in the Schema Architecture

ZIO Blocks separates operational behavior from structural metadata using two key abstractions:

1. **Reflect** (`Reflect[F, A]`): A generic data structure describing the shape of type `A`. It's parameterized by `F[_, _]`, which can be plugged with different "binding strategies".
2. **Binding**: The concrete binding strategy that embeds construction/deconstruction capabilities within a `Reflect` structure. The binding is the operational behavior attached to the reflect.

When `F[_, _] = Binding`, you get a **bound reflect** that can actually construct and deconstruct values. When `F[_, _] = NoBinding`, you get an **unbound reflect** that contains only structural metadata and is fully **serializable**.

```scala
// Type aliases for clarity
type Reflect.Bound[A] = Reflect[Binding, A]     // Can construct/deconstruct A
type Reflect.Unbound[A] = Reflect[NoBinding, A] // Pure metadata, serializable
```

A `Schema[A]` is simply a wrapper around `Reflect.Bound[A]`:

```scala
final case class Schema[A](reflect: Reflect.Bound[A])
```

## Binding Variants

The `Binding` sealed trait has several case class variants, each corresponding to a different kind of `Reflect` node:

### 1. `Binding.Record`

Provides construction and deconstruction capabilities for product types—case classes, tuples, and any type composed of multiple named or positional fields.

```scala
final case class Record[A](
  constructor: Constructor[A],
  deconstructor: Deconstructor[A]
) extends Binding[BindingType.Record, A]
```

**Components:**

- `constructor`: Builds an `A` from primitive components stored in `Registers`
- `deconstructor`: Breaks down an `A` into primitive components in `Registers`

Default values and examples are not part of a binding: they are stored on the `Reflect` node (`Reflect#defaultValue`, `Reflect#examples`).

Here is an example of a `Binding.Record` for a simple `Person` case class:

```scala

case class Person(name: String, age: Int)

val initialRegisters = RegisterOffset(objects = 1, ints = 1)

val personRecord: Binding.Record[Person] =
  Binding.Record[Person](
    constructor = new Constructor[Person] {
      def usedRegisters = initialRegisters

      def construct(in: Registers, offset: RegisterOffset): Person =
        Person(
          in.getObject(offset).asInstanceOf[String],
          in.getInt(offset + RegisterOffset(objects = 1))
        )
    },
    deconstructor = new Deconstructor[Person] {

      def usedRegisters = initialRegisters

      def deconstruct(out: Registers, offset: RegisterOffset, p: Person): Unit = {
        out.setObject(offset, p.name)
        out.setInt(offset + RegisterOffset(objects = 1), p.age)
      }
    }
  )
```

### 2. `Binding.Variant`

The `Binding.Variant` data type provides discrimination capabilities and matchers for sum types—sealed traits, Scala 3 enums, and any type that represents one of several possible alternatives:

```scala
final case class Variant[A](
  discriminator: Discriminator[A],
  matchers: Matchers[A]
) extends Binding[BindingType.Variant, A]
```

**Discriminator** and **Matchers** are complementary components in ZIO Blocks that work together to handle sum types (ADTs/sealed traits/enums):

- `Discriminator`: Given a value of a sum type, determine which case it belongs to by returning a numerical index.
- `Matchers`: Given a value and a case index, safely downcast the value to the specific case type, or return null if it doesn't match.

```scala

sealed trait Shape extends Product with Serializable
case class Circle(radius: Double) extends Shape
case class Rectangle(width: Double, height: Double) extends Shape

val shapeBinding = Binding.Variant[Shape](
  discriminator = new Discriminator[Shape] {
    override def discriminate(a: Shape): Int = a match {
      case Circle(_) => 0
      case Rectangle(_, _) => 1
    }
  },
  matchers = Matchers(
    new Matcher[Circle] {
      override def downcastOrNull(any: Any): Circle =
        any match {
          case c: Circle => c
          case _ => null.asInstanceOf[Circle]
        }
    },
    new Matcher[Rectangle] {
      override def downcastOrNull(any: Any): Rectangle =
        any match {
          case r: Rectangle => r
          case _ => null.asInstanceOf[Rectangle]
        }
    }
  )
)
```

Please note that the `Variant` is only responsible for discrimination and downcasting. The actual construction and deconstruction of each case is handled by the individual case's `Binding.Record` (or other appropriate binding).

### 3. Binding.Seq

The `Binding.Seq` data type provides efficient construction and deconstruction capabilities for sequence/collection types—`List`, `Vector`, `Array`, `Set`, `Chunk`, and any ordered collection:

```scala
final case class Seq[C[_], +A](
  constructor: SeqConstructor[C],
  deconstructor: SeqDeconstructor[C]
) extends Binding[BindingType.Seq[C], C[A]]
```

The type parameter is `C[_]` (the collection type constructor) rather than `C[A]` (a specific element type). This allows a single `Binding.Seq[List]` to work for `List[Int]`, `List[String]`, `List[Person]`, etc. Inside the companion object of `Binding.Seq`, there are implementations for `Set`, `List`, `Vector`, `IndexedSeq`, `Seq` and `Chunk`; `Binding.of[ArraySeq]`, `Binding.of[Array]` and, on Scala 3, `Binding.of[IArray]` create the array-backed ones.

Sequences are ubiquitous in data modeling, and their efficient handling is critical for performance. The `Seq` binding abstracts over the specific collection type using the **builder pattern**, enabling collection-builder and array-backed construction strategies.

- `constructor` is a sequence constructor which provides how to build the sequence of type `C[_]`. It provides collection-specific builder operations. Different collections have different constructor strategies (e.g., an `Array` is built in an array of the element type that doubles when it is full and is trimmed at the end, while a `List` is built in a `ListBuffer`).
- `deconstructor` is a sequence deconstructor that provides iteration and size information for the collection type `C[_]`, which enables us to tear down the collection into its elements efficiently. The size hints enable codecs to pre-allocate output buffers and builders.

Let's take a deep dive into these two main components.

#### SeqConstructor

The `SeqConstructor` trait builds a collection through a mutable builder:

```scala
trait SeqConstructor[C[_]] {
  type Builder[_]

  def newBuilder[A](sizeHint: Int = 4)(implicit ct: ClassTag[A]): Builder[A]

  def add[A](builder: Builder[A], a: A): Unit

  def addBoolean(builder: Builder[Boolean], a: Boolean): Unit = add(builder, a)
  // ... addByte, addShort, addInt, addLong, addFloat, addDouble and addChar, also calling add

  def result[A](builder: Builder[A]): C[A]

  def empty[A](implicit ct: ClassTag[A]): C[A]
}
```

To implement a sequence constructor, we define the builder type, how to create a new builder, how to add an element to it, how to produce the final collection, and the empty collection. Here is the `SeqConstructor` for `List`:

```scala
implicit val listConstructor: SeqConstructor[List] = new SeqConstructor[List] {
  type Builder[A] = scala.collection.mutable.ListBuffer[A]

  def newBuilder[A](sizeHint: Int)(implicit ct: ClassTag[A]): Builder[A] = new ListBuffer[A]

  def add[A](builder: Builder[A], a: A): Unit = builder.addOne(a)

  def result[A](builder: Builder[A]): List[A] = builder.toList

  def empty[A](implicit ct: ClassTag[A]): List[A] = Nil
}
```

There are two kinds of implementations:

1. **Collection-builder constructors** for `List`, `Vector`, `Set`, `IndexedSeq`, `Seq` and `Chunk`. They use the collection's own mutable builder, and their primitive adders such as `addInt` pass the value to that builder like any other element.

2. **Array-backed constructors** for `Array`, `ArraySeq` and, on Scala 3, `IArray`. They store the elements in an array of the element type (that is what the `ClassTag` is for), and override the primitive adders to write into that array without boxing.

All the implementations are found in the companion object of the `SeqConstructor` trait.

#### SeqDeconstructor

The `SeqDeconstructor` has a simple interface with two methods:

```scala
trait SeqDeconstructor[C[_]] {
  def deconstruct[A](c: C[A]): Iterator[A]

  def size[A](c: C[A]): Int
}
```

The `deconstruct` method provides an iterator over the elements of the collection, while the `size` method returns the number of elements.

Inside the companion object of the `SeqDeconstructor` there are implementations for standard Scala collections like `List`, `Vector`, `Seq`, and `IndexedSeq`.

For efficiency, there are specialized implementations for array-based sequences like `Array`, `ArraySeq`, and `IArray` which provide direct access to elements, besides the iterator-based approach. As a result, they are much more performant than the generic iterator-based deconstruction. When an `Optic` needs to access a specific element by index, it can use these specialized methods to avoid the overhead of iterator traversal.

This is done through the `SpecializedIndexed` trait:

```scala
sealed trait SpecializedIndexed[C[_]] extends SeqDeconstructor[C] {
  def elementType[A](c: C[A]): RegisterType[A]
  def objectAt[A](c: C[A], index: Int): A
  def booleanAt(c: C[Boolean], index: Int): Boolean
  def byteAt(c: C[Byte], index: Int): Byte
  def shortAt(c: C[Short], index: Int): Short
  def intAt(c: C[Int], index: Int): Int
  def longAt(c: C[Long], index: Int): Long
  def floatAt(c: C[Float], index: Int): Float
  def doubleAt(c: C[Double], index: Int): Double
  def charAt(c: C[Char], index: Int): Char
}
```

You can find implementations of `SpecializedIndexed` for `Array`, `ArraySeq`, and `IArray` in the companion object of `SeqDeconstructor`.

### 4. Binding.Map

To describe the construction and deconstruction of key-value collections, use `Binding.Map`:

```scala
final case class Map[M[_, _], +K, +V](
  constructor: MapConstructor[M],
  deconstructor: MapDeconstructor[M]
) extends Binding[BindingType.Map[M], M[K, V]]
```

It has two main components for building maps and tearing down maps to their key-value pairs:

- `constructor`: Builds maps from key-value pairs using builders
- `deconstructor`: Provides key-value iteration for maps

Their interfaces are similar to `SeqConstructor` and `SeqDeconstructor`, but specialized for key-value pairs.

ZIO Blocks has implementations of `Binding.Map` in the companion object of `Binding.Map`: `Binding.Map.map` for the standard Scala `Map` and `Binding.Map.chunkMap` for `ChunkMap`.

### 5. Binding.Primitive

The `Binding.Primitive` provides metadata for primitive/scalar types—`Int`, `Long`, `Double`, `String`, `Boolean`, `Byte`, `Short`, `Char`, `Float`, `BigInt`, `BigDecimal`, `UUID`, and temporal types.

```scala
final case class Primitive[A]() extends Binding[BindingType.Primitive, A]
```

Primitives require minimal binding logic since they are atomic values that don't need construction from parts. You can access all built-in primitive bindings in the companion object of `Binding.Primitive`, such as `Primitive.int`, `Primitive.string`, etc.

### 6. Binding.Wrapper

The `Binding.Wrapper` provides wrap/unwrap capabilities for newtype patterns—opaque types, value classes, validated wrappers, and single-field case class wrappers.

`Newtypes` are extremely common in well-designed codebases. Instead of passing raw String values for emails, you define `Email` as a distinct type. Instead of `Int` for user IDs, you define `UserId`. This provides type safety without runtime overhead.

The Wrapper binding handles the bidirectional transformation between the wrapper type and its underlying representation. `wrap` can reject a value by throwing; `Reflect#fromDynamicValue` and the JSON codec report that as a decoding error:

```scala
final case class Wrapper[A, B](
  wrap: B => A,
  unwrap: A => B
) extends Binding[BindingType.Wrapper[A, B], A]
```

**Components:**

- `wrap`: Converts from the underlying type `B` to the wrapper type `A`
- `unwrap`: Extracts the underlying `B` from an `A`

Here is an example of a `Binding.Wrapper` for an `Email` newtype:

```scala

case class Email(value: String)

object Email {
  new Binding.Wrapper[Email, String](
    wrap = value => new Email(value),
    unwrap = email => email.value
  )
}
```

### 7. Binding.Dynamic

To bind untyped dynamic values, use `Binding.Dynamic`. It provides a binding for dynamically typed values whose structure is not known at compile time:

```scala
final case class Dynamic() extends Binding[BindingType.Dynamic, DynamicValue]
```

Used when the schema of the data is not known at compile time, such as JSON payloads with arbitrary structure.

## Binding and Schema Serialization

When `F[_, _] = NoBinding` in `Reflect[F[_, _], A]` the `Reflect` structure contains only pure data (no functions) and becomes fully serializable. This enables:

1. **Schema serialization**: Convert schemas to JSON Schema or other formats, making them portable
2. **Schema rebinding**: Deserialize a schema and rebind it using a `BindingResolver`, so it becomes type-safe and operational again

[ReflectTransformer](./reflect-transformer.md) covers schema serialization and rebinding in full detail — the transformer that strips bindings, the one that reattaches them, and how each is exposed as `Schema#toDynamicSchema`/`DynamicSchema#rebind`. For the full API of the binding lookup mechanism used during rebinding, see [BindingResolver](binding-resolver.md).

## Summary

The `Binding` data type is the operational heart of ZIO Blocks' schema system:

| Reflect Node        | Binding Variant     | Construction and Deconstruction of |
|---------------------|---------------------|------------------------------------|
| `Reflect.Record`    | `Binding.Record`    | Product Types                      |
| `Reflect.Variant`   | `Binding.Variant`   | Sum Types                          |
| `Reflect.Sequence`  | `Binding.Seq`       | Sequence Collection Types          |
| `Reflect.Map`       | `Binding.Map`       | Key-value Collection Types         |
| `Reflect.Primitive` | `Binding.Primitive` | Primitive Types                    |
| `Reflect.Wrapper`   | `Binding.Wrapper`   | Wrap/unwrap newtypes               |
| `Reflect.Dynamic`   | `Binding.Dynamic`   | Untyped Dynamic Values             |

A key innovation is the clean separation between structure (`Reflect`) and behavior (`Binding`), enabling us to achieve serializable schemas with unbound reflects, while the `Reflect` remains pluggable with the `Binding` for runtime operations.

---

## Avro Codec Module

`zio-blocks-schema-avro` is a **schema-driven Avro codec module** for serializing and deserializing Scala types to and from Avro binary format. It provides comprehensive encoding and decoding with support for 30 primitive types, records, variants, sequences, maps, and recursive types. Core types: `AvroCodec`, `AvroCodecDeriver`, `AvroFormat`.

The module integrates with Apache Avro to provide native binary serialization with automatic schema generation and support for recursive types, such as trees.

See also the [How to Encode and Decode Records with Avro](../../../guides/encode-and-decode-records-with-avro.md) guide, which writes the `.avsc` schema for a sealed trait, streams records, lists how each Scala type maps to Avro, and reads the bytes with Apache Avro.

## Motivation

Avro is a powerful serialization format prevalent in distributed systems, messaging platforms, and data pipelines. Manually writing Avro encoders and decoders is error-prone and repetitive, especially for complex types with records, nested structures, and recursive definitions. `zio-blocks-schema-avro` eliminates this friction by deriving codec instances directly from your Scala types using ZIO Blocks Schema. You describe your data shape once, and the module handles:
- Full Avro type support (records, unions, arrays, maps, nested structures)
- Automatic Avro schema generation from Scala types
- Sum types as Avro unions, one record per case
- Precise error reporting with location traces showing the path to errors
- Recursive type support, such as trees
- Multiple encoding paths: ByteBuffer, byte arrays, and streams
- Multiple decoding paths: ByteBuffer, byte arrays, and streams
- JVM support (not available for Scala.js)

Rather than writing custom encoders or relying on string-based Avro schema configuration, you work with strongly-typed schemas that the compiler validates.

## Installation

Add the module to your `build.sbt` (JVM-only, not available for Scala.js):

```sbt
libraryDependencies += "dev.zio" %% "zio-blocks-schema-avro" % "0.0.56"
```

This artifact is for Scala 2.13 and Scala 3.3 to 3.8. On Scala 3.9 and newer, use the `zio-blocks-next-schema-avro` module instead and add `scalacOptions += "-experimental"`, as [How to Add ZIO Blocks to Your Build](../../../getting-started/installation.md) explains.

## Introduction

The module provides a complete pipeline for Avro codec derivation and usage:

1. **Define your type** — Any Scala type with a `Schema` instance
2. **Derive a codec** — Use `Schema#derive` with `AvroFormat` to obtain an `AvroCodec[A]`
3. **Encode or decode** — Call `codec.encode(value)` or `codec.decode(bytes)`
4. **Handle errors** — `decode` returns the `SchemaError`, with the path to the bad value, in a `Left`

The derivation process is automatic for all supported types (all 30 primitives, records, variants, sequences, maps). The module automatically generates Avro schemas and handles encoding/decoding without manual configuration.

## How They Work Together

The Avro codec pipeline flows through these layers:

```
1. User defines Schema[A] for their type
                 ↓
2. Schema[A].derive(AvroFormat) creates AvroCodec[A]
                 ↓
3. AvroCodecDeriver derives Encoder and Decoder implementations
   - For primitives: type-specific Avro encoders/decoders
   - For records: field-by-field composition with Avro record schema
   - For variants: a union of the case records, written as the branch index and the case
   - For sequences: array encoding/decoding
   - For maps: map encoding/decoding
                 ↓
4. AvroCodec provides multiple encoding paths
   - encode(value) → Array[Byte]
   - encode(value, output: OutputStream) → Unit
   - encode(value, buffer: ByteBuffer) → Unit
                 ↓
5. AvroCodec provides multiple decoding paths
   - decode(bytes: Array[Byte]) → Either[SchemaError, A]
   - decode(input: InputStream) → Either[SchemaError, A]
   - decode(buffer: ByteBuffer) → Either[SchemaError, A]
                 ↓
6. AvroCodec.avroSchema exposes generated Avro schema
   Useful for compatibility checks and schema documentation
                 ↓
7. Errors include location traces
   Shows path (.field.at(index).nested) to error location
```

**Typical workflow:**

A user type flows through the derivation and encoding pipeline as follows:

```
User type (e.g., case class Person)
    ↓
Schema.derived (automatic via macro)
    ↓
Schema[Person].derive(AvroFormat) → AvroCodec[Person]
    ↓
Use codec.encode(person) to serialize → Array[Byte]
Use codec.decode(bytes) to deserialize → Either[SchemaError, Person]
    ↓
Handle SchemaError with location trace on failure
```

### Type Relationships

- **`AvroCodec[A]`** — Main public API; contains encoder and decoder for bidirectional serialization
- **`AvroCodecDeriver`** — Configuration and derivation system; generates codecs from Schema
- **`AvroFormat`** — Integration with ZIO Blocks Schema format system; enables `Schema[A].derive(AvroFormat)`
- **`SchemaError`** — Error type with location traces; renders as `DynamicOptic` paths like `.field.at(0).nested`

## Common Patterns

This section shows practical patterns for working with Avro codecs in real-world scenarios.

### Pattern 1: Derive and Encode a Simple Record

To derive and use an Avro codec for a record type:

```scala

case class Person(name: String, age: Int, email: String)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Person.schema.derive(AvroFormat)
val person = Person("Alice", 30, "alice@example.com")
val bytes = codec.encode(person)
```

### Pattern 2: Decode Avro with Error Handling

When decoding Avro data, errors include location traces showing where the problem occurred.

To decode bytes and handle errors with location information:

```scala

case class Employee(id: Int, name: String, salary: Double)

object Employee {
  implicit val schema: Schema[Employee] = Schema.derived
}

val codec = Employee.schema.derive(AvroFormat)
val bytes = Array[Byte](1, 4, 6) // truncated data

val result = codec.decode(bytes)

result match {
  case Right(employee) => println(s"Decoded: $employee")
  case Left(error) =>
    println(s"Error at ${error.getMessage}")
}
```

### Pattern 3: Inspect the Generated Avro Schema

Access the derived Avro schema to verify compatibility or document the serialization format.

To inspect the Avro schema for a type:

```scala

case class Product(name: String, price: Double, inStock: Boolean)

object Product {
  implicit val schema: Schema[Product] = Schema.derived
}

val codec = Product.schema.derive(AvroFormat)
val avroSchema = codec.avroSchema
println(avroSchema.toString)
```

### Pattern 4: Handle Recursive Types

Recursive types (types that reference themselves) are supported: the Avro schema refers to the record by name. The codec does not detect cycles in values, and a case class value cannot be cyclic anyway.

To define and encode a recursive data structure:

```scala

sealed trait Tree
case class Leaf(value: Int) extends Tree
case class Branch(left: Tree, right: Tree) extends Tree

object Tree {
  implicit val schema: Schema[Tree] = Schema.derived
}

val codec = Tree.schema.derive(AvroFormat)
val tree: Tree = Branch(Leaf(1), Branch(Leaf(2), Leaf(3)))
val bytes = codec.encode(tree)
```

---

## AvroCodec[A]

Main codec type for encoding and decoding values to and from Avro binary format. Contains encoder and decoder for bidirectional serialization.

### Overview

`AvroCodec[A]` holds both an encoder and decoder, providing a complete solution for serializing and deserializing values in Avro binary format. The codec is derived automatically from a `Schema[A]` using `AvroFormat`.

### Accessing the Avro Schema

To get the derived Avro schema from a codec:

```scala

case class User(id: Int, name: String)

object User {
  implicit val schema: Schema[User] = Schema.derived
}

val codec = User.schema.derive(AvroFormat)
val avroSchema = codec.avroSchema
```

### Encoding Values to Byte Array

Use the codec to convert values to byte arrays:

```scala

case class Product(name: String, price: Double)

object Product {
  implicit val schema: Schema[Product] = Schema.derived
}

val codec = Product.schema.derive(AvroFormat)
val product = Product("Widget", 9.99)
val bytes = codec.encode(product)
```

### Encoding Values to OutputStream

Write encoded values directly to an output stream:

```scala

case class Item(name: String, quantity: Int)

object Item {
  implicit val schema: Schema[Item] = Schema.derived
}

val codec = Item.schema.derive(AvroFormat)
val item = Item("Gadget", 42)
val output = new ByteArrayOutputStream()
codec.encode(item, output)
val bytes = output.toByteArray
```

### Decoding Values from Byte Array

Use the codec to convert byte arrays back to values:

```scala

case class Record(id: Int, value: String)

object Record {
  implicit val schema: Schema[Record] = Schema.derived
}

val codec = Record.schema.derive(AvroFormat)
// In real usage, bytes would come from a previous encoding or external source
val record = Record(123, "test")
val buffer = java.nio.ByteBuffer.allocate(256)
codec.encode(record, buffer)
buffer.flip()
val bytes = new Array[Byte](buffer.remaining())
buffer.get(bytes)

val result: Either[zio.blocks.schema.SchemaError, Record] = codec.decode(bytes)
```

### Decoding Values from InputStream

Read and decode values from an input stream:

```scala

case class Data(timestamp: Long, payload: String)

object Data {
  implicit val schema: Schema[Data] = Schema.derived
}

val codec = Data.schema.derive(AvroFormat)
// Use encoded bytes from a previous encoding
val data = Data(System.currentTimeMillis(), "example payload")
val buffer = java.nio.ByteBuffer.allocate(256)
codec.encode(data, buffer)
buffer.flip()
val bytes = new Array[Byte](buffer.remaining())
buffer.get(bytes)
val input = new ByteArrayInputStream(bytes)
val result = codec.decode(input)
```

### Decoding Trusts the Reader's Schema

Avro binary carries no field names and no type name, so `AvroCodec#decode` reads the bytes with the schema of the type that you asked for. It does not check that the bytes were written for that type. Bytes written for `Shipped(orderId: Long, carrier: String)` decode without error as `Priority(orderId: Long, priority: Int)` and return wrong values. Avro binary has no type information to check, so the codec cannot detect this, and zio-blocks adds no check of its own (a known limitation in 0.0.56). Keep the writer and the reader on the same schema version, and check a version field before you decode when the two sides can drift apart.

A known bug in 0.0.56: `AvroCodec#decode(ByteBuffer)` on a sliced heap buffer (one with a non-zero array offset) reads the wrong bytes, because the codec ignores `ByteBuffer#arrayOffset`; pass an array or an unsliced buffer.

A second known bug in 0.0.56: `AvroCodec#decode` decodes one value and returns `Right` without checking that the input is used up. A message with extra bytes at the end decodes as the first record, and the extra bytes are ignored. Until the bug is fixed, the codec cannot reject such input, so check the message length in your own code when a message must be exact.

The how-to guide [Encode and Decode Records with Avro](../../../guides/encode-and-decode-records-with-avro.md) shows the full round trip with another Avro library.

---

## AvroCodecDeriver

Configuration and derivation system for creating `AvroCodec[A]` instances from `Schema[A]`.

### Overview

`AvroCodecDeriver` implements the schema-driven derivation of Avro codecs. It automatically handles 30 primitive types and complex types (records, variants, sequences, maps), generating appropriate Avro schemas and encoder/decoder implementations.

### How Derivation Works

To create a codec from a schema:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Person.schema.derive(AvroFormat)
```

### Primitive Type Support

All 30 ZIO Blocks Schema primitives are supported. Each one maps to an Avro type, and the table in [Avro Types for Scala Types](#avro-types-for-scala-types) lists the shape of the common ones:
- Numeric: `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, `BigDecimal`
- Character and text: `Char` (written as an `int` code point), `Boolean`, `String`
- Temporal: `Instant`, `LocalDate`, `LocalDateTime`, `LocalTime`, `Duration`, `Period`, `Year`, `YearMonth`, `MonthDay`, `Month`, `DayOfWeek`, `ZonedDateTime`, `OffsetDateTime`, `OffsetTime`, `ZoneId`, `ZoneOffset`
- Special: `UUID`, `Currency`, `Unit`

### Avro Types for Scala Types

A reader in Java, Python or another Avro library works from the `.avsc` schema, so it sees each Scala type as the Avro type that the codec derives. To print that Avro type for common Scala types:

```scala

def avsc[A](implicit schema: Schema[A]): String = schema.derive(AvroFormat).avroSchema.toString

sealed trait Status
case object Active extends Status
case object Closed extends Status

case class CustomerId(value: Long)

case class Sku(value: String)

implicit val statusSchema: Schema[Status]         = Schema.derived
implicit val customerIdSchema: Schema[CustomerId] = Schema.derived
implicit val skuSchema: Schema[Sku]               = Schema[String].transform(Sku(_), _.value)
implicit val bytesSchema: Schema[Array[Byte]]     = Schema.derived
implicit val chunkSchema: Schema[Chunk[Byte]]     = Schema.derived
```

To print the Avro type of each one:

```scala
List(
  "Int"                 -> avsc[Int],
  "Char"                -> avsc[Char],
  "BigInt"              -> avsc[BigInt],
  "Array[Byte]"         -> avsc[Array[Byte]],
  "Instant"             -> avsc[Instant],
  "LocalDate"           -> avsc[LocalDate],
  "UUID"                -> avsc[UUID],
  "Option[Int]"         -> avsc[Option[Int]],
  "Either[String, Int]" -> avsc[Either[String, Int]],
  "Status"              -> avsc[Status],
  "CustomerId"          -> avsc[CustomerId],
  "Sku"                 -> avsc[Sku]
).foreach { case (scalaType, avroType) => println(s"$scalaType: $avroType") }
// Int: "int"
// Char: "int"
// BigInt: "bytes"
// Array[Byte]: {"type":"array","items":"int"}
// Instant: {"type":"record","name":"Instant","namespace":"java.time","fields":[{"name":"epochSecond","type":"long"},{"name":"nano","type":"int"}]}
// LocalDate: {"type":"record","name":"LocalDate","namespace":"java.time","fields":[{"name":"year","type":"int"},{"name":"month","type":"int"},{"name":"day","type":"int"}]}
// UUID: {"type":"fixed","name":"UUID","namespace":"java.util","size":16}
// Option[Int]: [{"type":"record","name":"None","namespace":"scala","fields":[]},{"type":"record","name":"Some","namespace":"scala","fields":[{"name":"value","type":"int"}]}]
// Either[String, Int]: [{"type":"record","name":"Left","namespace":"scala.util","fields":[{"name":"value","type":"string"}]},{"type":"record","name":"Right","namespace":"scala.util","fields":[{"name":"value","type":"int"}]}]
// Status: [{"type":"record","name":"Active","namespace":"repl.MdocSession.MdocApp10","fields":[]},{"type":"record","name":"Closed","namespace":"repl.MdocSession.MdocApp10","fields":[]}]
// CustomerId: {"type":"record","name":"CustomerId","namespace":"repl.MdocSession.MdocApp10","fields":[{"name":"value","type":"long"}]}
// Sku: "string"
```

The table below sums up the Avro shape of each type and what a reader in another language should know about it:

| Scala type                            | Avro type                                                 | What a reader in another language sees                                                        |
| ------------------------------------- | --------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| `Int`, `Short`, `Byte`                | `int`                                                     | A 32-bit number; `Short` and `Byte` are widened                                               |
| `Long`                                | `long`                                                    |                                                                                               |
| `Float`, `Double`                     | `float`, `double`                                         |                                                                                               |
| `Boolean`                             | `boolean`                                                 |                                                                                               |
| `String`                              | `string`                                                  |                                                                                               |
| `Char`                                | `int`                                                     | The character's code point, not a one-letter string                                           |
| `Unit`                                | `null`                                                    |                                                                                               |
| `BigInt`                              | `bytes`                                                   | Two's-complement big-endian bytes                                                             |
| `BigDecimal`                          | record `scala.BigDecimal`                                 | `mantissa` bytes plus `scale`, `precision` and `roundingMode`; not the `decimal` logical type |
| `Array[Byte]`, `Chunk[Byte]`          | array of `int`                                            | One number per byte, not `bytes`                                                              |
| `List[A]`                             | array of `A`                                              |                                                                                               |
| `Map[String, A]`                      | map of `A`                                                |                                                                                               |
| `Instant`                             | record `java.time.Instant`                                | `epochSecond` and `nano`; not the `timestamp-millis` logical type                             |
| `LocalDate`                           | record `java.time.LocalDate`                              | `year`, `month` and `day`; not the `date` logical type                                        |
| `UUID`                                | `fixed` of 16 bytes named `java.util.UUID`                | Not the `uuid` logical type                                                                   |
| `Option[A]`                           | union of records `scala.None` and `scala.Some`            | Not `["null", A]`; `None` is branch 0 and `Some` holds `value`                                |
| `Either[A, B]`                        | union of records `scala.util.Left` and `scala.util.Right` | Each side holds `value`                                                                       |
| Sealed trait                          | union of one record per case, in declaration order        | A case object is a record with no fields                                                      |
| Case class                            | record named after the class, namespace from its package  | A one-field class such as `CustomerId` stays a record                                         |
| Newtype built with `Schema#transform` | the wrapped type                                          | `Sku` is a plain `string`                                                                     |

Four rows surprise readers in other languages. `Option` is a union of two records, not the `["null", T]` union that most Avro libraries write for an optional field, so a reader that expects `null` for a missing value does not find one. `Char` is an `int`, so a reader expecting a string gets a number. The temporal types are records and `UUID` is a `fixed` of 16 bytes, not the Avro logical types, so a reader that expects `timestamp-millis` or `uuid` needs a custom mapping.

A known bug in 0.0.56: `Array[Byte]` and `Chunk[Byte]` encode as an Avro array of `int` instead of `bytes`, as the table shows. Each byte becomes its own array item, so the output is larger than the payload, and a reader in another language sees a list of numbers instead of a byte buffer.

The how-to guide [Encode and Decode Records with Avro](../../../guides/encode-and-decode-records-with-avro.md) walks through reading these schemas with another Avro library.

### Record Type Support

Case classes (records) are fully supported. Each field becomes a named field in the Avro record schema:

```scala

case class Address(street: String, city: String, zip: String)

object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

val codec = Address.schema.derive(AvroFormat)
```

### Variant Type Support

Sealed traits and sum types are encoded as Avro union types:

```scala

sealed trait Status
case class Active(since: String) extends Status
case class Inactive(reason: String) extends Status

object Status {
  implicit val schema: Schema[Status] = Schema.derived
}

val codec = Status.schema.derive(AvroFormat)
```

---

## AvroFormat

Integration point with ZIO Blocks Schema's format system. Provides `BinaryFormat[AvroCodec]` to enable `Schema[A].derive(AvroFormat)` for any supported type.

### Using AvroFormat

To derive an Avro codec using the standard format:

```scala

case class Sensor(id: Long, temperature: Double, humidity: Float)

object Sensor {
  implicit val schema: Schema[Sensor] = Schema.derived
}

val codec = Sensor.schema.derive(AvroFormat)
```

`AvroFormat` is a singleton object extending `BinaryFormat[AvroCodec]` with the MIME type `"application/avro"` and the `AvroCodecDeriver` as its derivation strategy.

---

## Error Handling

Avro decoding errors include location traces showing the path through nested structures where the error occurred.

### Understanding Error Traces

Errors render as `DynamicOptic` paths such as `.field.at(0).nested.value`: a record field is `.name`, a sequence index is `.at(0)`, a map value is `.atKey(...)` and a variant case is `.when[Case]`. The path shows exactly where decoding failed:

```scala

case class Contact(emails: Seq[String])

object Contact {
  implicit val schema: Schema[Contact] = Schema.derived
}

val codec = Contact.schema.derive(AvroFormat)
// Example invalid Avro bytes that will fail decoding
val invalidBytes: Array[Byte] = Array(0xFF.toByte, 0xFF.toByte)

val result = codec.decode(invalidBytes)

result match {
  case Right(contact) => println(s"Success: $contact")
  case Left(error) =>
    println(s"Error: ${error.getMessage}")
}
```

### Zero-Overhead Error Handling

Errors use zero-overhead exceptions (no stack traces) for efficient error reporting in stream processing scenarios where errors are expected and handled inline.

---

## BSON Codec Module

`zio-blocks-schema-bson` is a **schema-driven BSON codec module** for serializing and deserializing Scala types to and from BSON (Binary JSON) format. It provides comprehensive encoding and decoding with support for 30 primitive types, records, variants, sequences, maps, and recursive types. 

Core types: `BsonCodec`, `BsonEncoder`, `BsonDecoder`, `BsonCodecDeriver`, and the compatibility facade `BsonSchemaCodec`.

The module integrates with org.bson to provide native BSON type support including special handling for `ObjectId`, `Decimal128`, and other BSON-specific types.

## Motivation

BSON is the native wire format for MongoDB and appears widely across modern applications. Manually writing BSON encoders and decoders is error-prone and repetitive, especially for complex types with records, nested structures, and recursive definitions. `zio-blocks-schema-bson` eliminates this friction by deriving codec instances directly from your Scala types using ZIO Blocks Schema. You describe your data shape once, and the module handles:
- BSON documents, arrays, strings, numbers, `ObjectId` and `Decimal128`. The `java.time` types (`Instant`, `LocalDate`, `LocalDateTime`, `OffsetDateTime`, `ZonedDateTime`, `Duration` and the rest) and `UUID` are written as ISO-8601 or text `STRING`s, never as BSON `DATE_TIME` or `TIMESTAMP`, so MongoDB date queries and TTL indexes don't apply to them
- Configurable sum type handling (discriminator fields, wrapper types, or no discriminator)
- Flexible field name mapping for compatibility with MongoDB conventions
- Native ObjectId support with automatic detection and encoding
- Error reporting with location traces showing the path to errors, with a known bug in 0.0.56: a malformed string (such as a bad `UUID`) and, through `decode`, a BSON value of the wrong type throw instead of returning a `Left`
- Recursive type support with automatic cycle detection
- JVM support (not available for Scala.js)

Rather than writing custom encoders or relying on string-based configuration, you work with strongly-typed schemas that the compiler validates.

To store case classes in MongoDB with the official Java driver, step by step, see [How to Store Case Classes in MongoDB](../../../guides/store-case-classes-in-mongodb.md).

## Installation

Add the module to your `build.sbt`:

```sbt
libraryDependencies += "dev.zio" %% "zio-blocks-schema-bson" % "0.0.56"
```

**Note:** This module is JVM-only and is not available for Scala.js.

Supported Scala versions: 2.13.x and 3.x. On Scala 3.9 and newer, use the `zio-blocks-next-schema-bson` module instead and add `scalacOptions += "-experimental"`, as [How to Add ZIO Blocks to Your Build](../../../getting-started/installation.md) explains.

## Introduction

The module provides a complete pipeline for BSON codec derivation and usage:

1. **Define your type** — Any Scala type with a `Schema` instance
2. **Derive a codec** — Use `schema.derive(BsonCodecDeriver)` to obtain a `BsonCodec[A]`
3. **Encode or decode** — Call `codec.encoder.toBsonValue()` or `codec.decoder.fromBsonValue()`
4. **Handle errors** — `decode` and `fromBsonValue` return a `BsonDecoder.Error`, with the path to the bad value, in a `Left`; only the `*Unsafe` methods throw it

The derivation process is automatic for all supported types (all 30 primitives, records, variants, sequences, maps). The module provides flexible configuration for sum type handling, field mapping, and ObjectId support.

## How They Work Together

The BSON codec pipeline flows through these layers:

```
1. User defines Schema[A] for their type
                 ↓
2. Schema[A].derive(BsonCodecDeriver) creates BsonCodec[A]
                 ↓
3. DerivationBuilder traverses the schema and BsonCodecDeriver composes Encoder and Decoder implementations
   - For primitives: type-specific BSON encoders/decoders
   - For records: field-by-field composition
   - For variants: discriminator-based selection
   - For sequences: array encoding/decoding
   - For maps: document/object encoding/decoding
                 ↓
4. BsonEncoder writes values to BsonValue or BsonWriter
   BsonDecoder reads BsonValue or BsonReader to values
                 ↓
5. BsonCodecDeriver configuration methods customize behavior
   - Sum type handling (discriminator, wrapper, or none)
   - Field name mapping for MongoDB conventions
   - ObjectId detection and native BSON encoding
   - Extra field handling (ignore or error)
                 ↓
6. BsonTrace provides location information in errors
   Shows the path (.field[index].nested) to error location
```

**Typical workflow:**

A user type flows through the derivation and encoding pipeline as follows:

```
User type (e.g., case class Person)
    ↓
Schema.derived (automatic via macro)
    ↓
schema.derive(configuredBsonCodecDeriver) → BsonCodec[Person]
    ↓
Use codec.encoder.toBsonValue(person) to serialize
Use codec.decoder.fromBsonValue(bsonValue) to deserialize
    ↓
Handle the Left(BsonDecoder.Error) with its location trace on failure
```

## Common Patterns

This section shows practical patterns for working with BSON codecs in real-world scenarios.

### Pattern 1: Derive and Encode a Simple Record

To derive and use a BSON codec for a record type:

```scala

case class Person(name: String, age: Int, email: String)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Person.schema.derive(BsonCodecDeriver)
val person = Person("Alice", 30, "alice@example.com")
val bsonValue = codec.encoder.toBsonValue(person)
```

### Pattern 2: Decode BSON with Error Handling

When decoding BSON, errors include location traces showing where the problem occurred.

To decode a BSON document and handle errors with location information:

```scala

case class Employee(id: Int, name: String, salary: Double)

object Employee {
  implicit val schema: Schema[Employee] = Schema.derived
}

val codec = BsonSchemaCodec.bsonCodec(Employee.schema)
val doc = new BsonDocument()
doc.put("id", new BsonInt32(123))
doc.put("name", new BsonString("Bob"))
doc.put("salary", new BsonInt32(50000)) // Wrong type - should be number with decimal

val result = codec.decoder.fromBsonValue(doc)

result match {
  case Right(employee) => println(s"Decoded: $employee")
  case Left(error) =>
    println(s"Error at ${error.trace}: ${error.message}")
}
```

### Pattern 3: Configure Sum Type Handling

Sum types (sealed traits with variants) can be encoded in different ways. Configure which approach you prefer.

To configure how variants are encoded in BSON documents:

```scala

sealed trait Status
case class Active(since: String) extends Status
case class Inactive(reason: String) extends Status

object Status {
  implicit val schema: Schema[Status] = Schema.derived
}

val discriminatorConfig = BsonSchemaCodec.Config
  .withSumTypeHandling(BsonSchemaCodec.SumTypeHandling.DiscriminatorField("type"))

val codec = BsonSchemaCodec.bsonCodec(Status.schema, discriminatorConfig)
```

### Pattern 4: Use ObjectId with Native BSON Encoding

ObjectId fields are encoded using BSON's native ObjectId type for compatibility with MongoDB. Importing `ObjectIdSupport._` is all it takes; `BsonSchemaCodec.Config#withNativeObjectId` is not needed and has no effect in 0.0.56.

To derive a codec that stores an `ObjectId` field natively:

```scala

case class Document(id: ObjectId, title: String)

object Document {
  implicit val schema: Schema[Document] = Schema.derived
}

val codec = BsonSchemaCodec.bsonCodec(Document.schema)
```

## BsonCodec[A]

Main codec type for encoding and decoding values to and from BSON format. Contains an encoder and decoder for bidirectional serialization.

### Overview

`BsonCodec[A]` holds both a `BsonEncoder[A]` and `BsonDecoder[A]`, providing a complete solution for serializing and deserializing values. The codec is derived automatically from a `Schema[A]` using `BsonSchemaCodec.bsonCodec()`.

### Access Encoder and Decoder

To access the encoder and decoder from a codec:

```scala

case class User(id: Int, name: String)

object User {
  implicit val schema: Schema[User] = Schema.derived
}

val codec = BsonSchemaCodec.bsonCodec(User.schema)
val encoder = codec.encoder
val decoder = codec.decoder
```

### Encoding Values

Use the encoder to convert values to BsonValue:

```scala

case class Product(name: String, price: Double)

object Product {
  implicit val schema: Schema[Product] = Schema.derived
}

val codec = BsonSchemaCodec.bsonCodec(Product.schema)
val product = Product("Widget", 9.99)
val bsonValue = codec.encoder.toBsonValue(product)
```

### Decoding Values

Use the decoder to convert BsonValue back to values:

```scala

case class Item(name: String, quantity: Int)

object Item {
  implicit val schema: Schema[Item] = Schema.derived
}

val codec = BsonSchemaCodec.bsonCodec(Item.schema)
// Create a BsonValue from an encoded item
val item = Item("Widget", 42)
val bsonValue = codec.encoder.toBsonValue(item)

val result: Either[BsonDecoder.Error, Item] = codec.decoder.fromBsonValue(bsonValue)
```

---

## BsonEncoder[A]

Trait for encoding Scala values to BSON format. Provides methods for writing to BsonWriter or converting to BsonValue directly.

### Overview

`BsonEncoder[A]` is responsible for converting values of type `A` to BSON representation. It provides two encoding paths: direct BsonValue conversion or streaming to a BsonWriter.

### Key Methods

To encode a value to BsonValue:

```scala

trait BsonEncoder[A] {
  def encode(writer: org.bson.BsonWriter, value: A, ctx: BsonEncoder.EncoderContext): Unit
  def toBsonValue(value: A): org.bson.BsonValue
  def isAbsent(value: A): Boolean = false
}
```

`BsonEncoder#isAbsent` exists, but no code in the module calls it, so no field is ever skipped because of it. A field is left out only when it carries `Modifier.transient`.

### Contramap for Transformations

To encode `WrappedInt` through the existing `Int` encoder, use `BsonEncoder#contramap`:

```scala

case class WrappedInt(value: Int)

val intEncoder: BsonEncoder[Int]            = BsonSchemaCodec.bsonCodec(Schema[Int]).encoder
val wrappedEncoder: BsonEncoder[WrappedInt] = intEncoder.contramap(_.value)
```

---

## BsonDecoder[A]

Trait for decoding BSON values to Scala types. Provides error handling with precise location information.

### Overview

`BsonDecoder[A]` converts BSON values back to Scala types. It provides two decoding paths: from BsonValue or streaming from a BsonReader. Errors include location traces showing the path where decoding failed.

### Key Methods

To decode a BsonValue:

```scala

trait BsonDecoder[A] {
  def decodeUnsafe(reader: org.bson.BsonReader, trace: List[BsonTrace], ctx: BsonDecoder.BsonDecoderContext): A
  def fromBsonValueUnsafe(value: BsonValue, trace: List[BsonTrace], ctx: BsonDecoder.BsonDecoderContext): A
}
```

### Error Handling

`BsonDecoder#fromBsonValue` reports a failed decode as a `Left` holding `BsonDecoder.Error(trace: List[BsonTrace], message: String)`, and `BsonTrace.render` turns its trace into a path:

```scala

// Innermost step first, the order the decoder builds the trace in
val trace = List(BsonTrace.Field("age"), BsonTrace.Array(0), BsonTrace.Field("user"))
val rendered = BsonTrace.render(trace) // ".user[0].age"
```

---

## BsonCodecDeriver and BsonSchemaCodec

`BsonCodecDeriver` implements the shared `Deriver[BsonCodec]` contract. It supports direct derivation as well as `Schema.deriving` instance and modifier overrides by type, exact optic, or parent type and term name.

`BsonSchemaCodec` retains `BsonSchemaCodec.bsonCodec`, `BsonSchemaCodec.bsonEncoder`, `BsonSchemaCodec.bsonDecoder`, and `BsonSchemaCodec.Config` as compatibility and convenience APIs. These methods delegate to `BsonCodecDeriver`, so both entry points have the same BSON behavior.

### Configuration

The `Config` class controls codec behavior:

```scala

val defaultConfig = BsonSchemaCodec.Config
val customConfig = defaultConfig
  .withSumTypeHandling(BsonSchemaCodec.SumTypeHandling.DiscriminatorField("_type"))
  .withClassNameMapping(_.toLowerCase)
  .withIgnoreExtraFields(false)
  .withNativeObjectId(true) // no effect in 0.0.56, see Pattern 4
```

### Sum Type Handling Options

Choose how variants are encoded in BSON documents:

```scala

// Wrapper with class name field (default)
val wrapper = BsonSchemaCodec.SumTypeHandling.WrapperWithClassNameField

// Discriminator field approach
val discriminator = BsonSchemaCodec.SumTypeHandling.DiscriminatorField("type")

// No discriminator - encode variant directly
val none = BsonSchemaCodec.SumTypeHandling.NoDiscriminator
```

### Deriving Codecs

To create a codec from a schema:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Person.schema.derive(BsonCodecDeriver)
val configuredCodec = Person.schema.derive(BsonCodecDeriver.withIgnoreExtraFields(false))

// Compatibility facade
val compatibleCodec = BsonSchemaCodec.bsonCodec(Person.schema)
val configuredCompatibleCodec = BsonSchemaCodec.bsonCodec(Person.schema, BsonSchemaCodec.Config)
```

### Derivation Overrides

Use `Schema.deriving` when part of a schema needs a custom codec or runtime modifier:

```scala

// customIntCodec: BsonCodec[Int], defined by you
val codec = Person.schema
  .deriving(BsonCodecDeriver)
  .instance(Person.schema.reflect.typeId, "age", customIntCodec)
  .modifier(Person.schema.reflect.typeId, "name", Modifier.rename("fullName"))
  .derive
```

Type-level overrides affect every matching occurrence. Parent-type and term-name overloads are also available for fields and variant cases. String map keys remain BSON document field names; override map values or the whole map codec when custom map behavior is needed.

Record-level `Modifier.fieldNaming` and `Modifier.noExtraFields` annotations override the corresponding default BSON behavior. Variant-level `Modifier.caseNaming` and `Modifier.discriminator` annotations override class-name mapping and sum-type handling for the annotated variant.

### Semantic Json Values

`zio.blocks.schema.json.Json` objects, arrays, scalars, and nulls map directly to their semantic BSON equivalents. Finite BSON doubles other than signed zero retain their exact value when converted through `Json`; BSON signed zero is normalized to JSON zero because `Json.Number` does not retain its sign. Duplicate `Json.Object` field names are rejected because `BsonDocument` cannot represent them without data loss.

---

## BsonTrace

Error location information for BSON decoding errors. Shows the path to the error in the document.

### Overview

`BsonTrace` elements build up a path through nested documents and arrays. When an error occurs, the complete trace is rendered as a path like `.field[0].nested.value`.

### Trace Elements

The two types of trace elements:

```scala

sealed trait BsonTrace

case class Field(name: String) extends BsonTrace  // Document field
case class Array(idx: Int) extends BsonTrace      // Array index
```

### Rendering Traces

Convert a trace list to a human-readable path. The list holds the innermost step first, the order the decoder builds it in, and `BsonTrace.render` prints it outermost first:

```scala

val trace = List(
  BsonTrace.Field("email"),
  BsonTrace.Array(0),
  BsonTrace.Field("user")
)

val path = BsonTrace.render(trace) // ".user[0].email"
```

---

## ObjectIdSupport

Special support for `org.bson.types.ObjectId` with automatic detection for native BSON ObjectId encoding.

### Overview

`ObjectIdSupport` provides a Schema instance for ObjectId that enables native BSON ObjectId type encoding (12-byte format) when detected by BsonSchemaCodec.

### Using ObjectId

To use ObjectId in your schema, import ObjectIdSupport:

```scala

case class MongoDocument(id: ObjectId, title: String)

object MongoDocument {
  implicit val schema: Schema[MongoDocument] = Schema.derived
}

// ObjectId will automatically use native BSON ObjectId encoding
```

### ObjectId and Configuration

When ObjectIdSupport is imported, ObjectId automatically uses native BSON encoding regardless of the `useNativeObjectId` configuration setting.

---

## CSV Codec Module

`zio-blocks-schema-csv` is a **schema-driven CSV codec module** for serializing and deserializing Scala types to and from CSV format. It provides RFC 4180-compliant parsing and generation with zero dependencies and support for 30 primitive types plus flat record (case class) types. Core types: `CsvCodec`, `CsvConfig`, `CsvError`, `CsvReader`, `CsvWriter`, `CsvFormat`.

See also the [How to Read and Write CSV and TSV Files](../../../guides/read-and-write-csv-and-tsv-files.md) guide, which reads files by header name, reports every bad row, writes TSV through `CsvReader` and `CsvWriter`, and processes large files one row at a time.

The main public API is `CsvCodec[A]`, which extends `TextCodec[A]` and provides CSV-specific header support:

```scala

// API signature (conceptual - simplified for clarity)
trait CsvCodec[A] {
  def headerNames: IndexedSeq[String]
  def encode(value: A, output: CharBuffer): Unit
  def decode(input: CharBuffer): Either[SchemaError, A]
}
```

## Motivation

CSV is the de facto standard for tabular data exchange across systems, but parsing and serialization are often error-prone when done manually. `zio-blocks-schema-csv` eliminates this friction by deriving codec instances directly from your Scala types using ZIO Blocks Schema. You describe your data shape once, and the module handles:
- RFC 4180-compliant parsing and generation with proper quote escaping
- Error messages that quote the bad value, and parse errors that carry a row and column, with the limits described in [CsvError](#csverror)
- Zero-overhead errors (no stack traces) optimized for CSV stream processing
- Cross-platform support (JVM and Scala.js)

Rather than writing custom parsers or relying on string-based configuration, you work with strongly-typed schemas that the compiler validates.

## Installation

Add the module to your `build.sbt`:

```sbt
libraryDependencies += "dev.zio" %% "zio-blocks-schema-csv" % "0.0.56"
```

For Scala.js, use `%%%` instead of `%%`:

```sbt
libraryDependencies += "dev.zio" %%% "zio-blocks-schema-csv" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x. On Scala 3.9 and newer, use the `zio-blocks-next-schema-csv` module instead and add `scalacOptions += "-experimental"`, as [How to Add ZIO Blocks to Your Build](../../../getting-started/installation.md) explains.

## Introduction

The module provides a complete pipeline for CSV codec derivation and usage:

1. **Define your type** — Any Scala type with a `Schema` instance
2. **Derive a codec** — Use `CsvFormat` to obtain a `CsvCodec[A]`
3. **Parse or serialize** — Call `CsvCodec#decode` or `CsvCodec#encode`
4. **Handle errors** — `CsvReader` returns a `CsvError` in a `Left`; `CsvCodec#decode` returns a `SchemaError` in a `Left` for a malformed row, a wrong number of cells or a cell that doesn't convert to its field's type. Nothing throws either error

The derivation process is automatic for supported types (all 30 primitives and flat records). Unsupported types (sealed traits, `Option` fields, sequences, maps) are rejected at derivation time, when `Schema#derive` runs, with an `UnsupportedOperationException`.

## How They Work Together

The CSV codec pipeline flows through these layers:

```
1. User defines Schema[A] for their type
                 ↓
2. Schema.derive(CsvFormat) creates CsvCodec[A]
                 ↓
3. CsvCodecDeriver converts Schema to codec implementation
   - For primitives: type-specific encoders/decoders
   - For records: field-by-field composition
                 ↓
4. CsvCodec#encode writes values to CSV rows (via CsvWriter)
   CsvCodec#decode reads CSV rows to values (via CsvReader)
                 ↓
5. CsvReader/CsvWriter handle RFC 4180 quote escaping
   CsvConfig customizes delimiters, terminators, quoting
                 ↓
6. CsvError reports ParseError for malformed input; CsvCodec#decode reports a bad cell as SchemaError
```

**Typical workflow:**

A user type flows through the derivation and encoding pipeline as follows:

```
User type (e.g., case class Person)
    ↓
Schema.derived (automatic via macro)
    ↓
.derive(CsvFormat) → CsvCodec[Person]
    ↓
Use codec.headerNames for column names
    ↓
Use codec.encode(person, buffer) to serialize
Use codec.decode(buffer) to deserialize
    ↓
Handle the Left: CsvError (CsvReader) or SchemaError (CsvCodec#decode)
```

## Common Patterns

This section shows 4 practical patterns for working with CSV codecs in real-world scenarios.

### Pattern 1: Derive and Use a Simple Codec

To derive and use a CSV codec for a record type:

```scala

case class Person(name: String, age: Int, email: String)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Person.schema.derive(CsvFormat)
val person = Person("Alice", 30, "alice@example.com")
val buffer = CharBuffer.allocate(256)
codec.encode(person, buffer)
buffer.flip()
val encoded = buffer.toString
```

### Pattern 2: Customize CSV Format with CsvConfig

A codec derived with `Schema#derive(CsvFormat)` always uses `CsvConfig.default`: a comma between fields, double quotes and a `\r\n` line ending. `CsvFormat` takes no configuration. When you need tab-separated values (TSV) or a different delimiter, pass a `CsvConfig` to `CsvWriter` and `CsvReader`, which work on rows of field strings.

To write and read a tab-separated document:

```scala

val tsv: String = CsvWriter.writeAll(
  IndexedSeq("id", "value"),
  List(IndexedSeq("1", "first"), IndexedSeq("2", "second")),
  CsvConfig.tsv
)

val parsed: Either[CsvError, (IndexedSeq[String], IndexedSeq[IndexedSeq[String]])] =
  CsvReader.readAll(tsv, CsvConfig.tsv)
```

### Pattern 3: Report Bad Rows by Number

A decode error names the bad value, so a file with several bad rows can report each one. `CsvCodec#decode` decodes one row, not the header, so the example reads the document with `CsvReader.readAll` and decodes each row, counting the header as row 1:

```scala

case class Employee(id: Int, name: String, salary: BigDecimal)

object Employee {
  implicit val schema: Schema[Employee] = Schema.derived
}

val codec   = Employee.schema.derive(CsvFormat)
val csvData = "id,name,salary\n1,Alice,50000\n2,Bob,invalid\n"
```

The example joins each row's cells back into a line, which is safe here because no cell contains a comma:

```scala
CsvReader.readAll(csvData, CsvConfig.default) match {
  case Left(error) => println(s"Parse error: ${error.getMessage}")
  case Right((_, rows)) =>
    rows.zipWithIndex.foreach { case (cells, index) =>
      codec.decode(CharBuffer.wrap(cells.mkString(","))) match {
        case Right(employee) => println(s"Row ${index + 2}: $employee")
        case Left(error)     => println(s"Row ${index + 2}: ${error.getMessage}")
      }
    }
}
// Row 2: Employee(1,Alice,50000)
// Row 3: Cannot parse 'invalid' as BigDecimal: Character i is neither a decimal digit number, decimal point, nor "e" notation exponential mark. at: .
```

The message for row 3 quotes `invalid`, but it does not name the `salary` column. A known bug in 0.0.56: the decode error ends in the bare path `.`, so the column is missing from the message.

### Pattern 4: Working with Headers

Access the derived header names and use them to construct CSV output or validate input.

To get the column headers for a record type:

```scala

case class Product(sku: String, name: String, price: Double)

object Product {
  implicit val schema: Schema[Product] = Schema.derived
}

val codec = Product.schema.derive(CsvFormat)
val headers = codec.headerNames
// headers: IndexedSeq[String] = Vector("sku", "name", "price")
```

## CsvCodec[A]

Abstract codec for encoding and decoding values to and from CSV format. Extends `TextCodec[A]` and provides CSV-specific header support.

### Construction

Codecs are derived automatically from `Schema[A]` using `CsvFormat`:

```scala

case class User(id: Int, username: String)

object User {
  implicit val schema: Schema[User] = Schema.derived
}

val codec: CsvCodec[User] = User.schema.derive(CsvFormat)
```

### Header Names

To access the CSV column headers derived from a record type:

```scala

case class Book(title: String, author: String, isbn: String)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

val codec = Book.schema.derive(CsvFormat)
val headers: IndexedSeq[String] = codec.headerNames
```

### Encoding Values

To serialize a value to CSV format in a `CharBuffer`:

```scala

case class Item(name: String, quantity: Int)

object Item {
  implicit val schema: Schema[Item] = Schema.derived
}

val codec = Item.schema.derive(CsvFormat)
val item = Item("Widget", 42)
val buffer = CharBuffer.allocate(512)
codec.encode(item, buffer)
buffer.flip()
val csvLine = buffer.toString
```

### Decoding Values

To parse one CSV row from a `CharBuffer` into a value, with its cells in the order of `CsvCodec#headerNames`, and without the header line:

```scala

case class Sales(product: String, amount: BigDecimal)

object Sales {
  implicit val schema: Schema[Sales] = Schema.derived
}

val codec = Sales.schema.derive(CsvFormat)
```

Printing `codec.headerNames` shows the cell order that `codec.decode` expects for the row `Widget,99.99`:

```scala
println(codec.headerNames)
// Vector(product, amount)
println(codec.decode(CharBuffer.wrap("Widget,99.99\r\n")))
// Right(Sales(Widget,99.99))
```

---

## CsvConfig

Configuration for CSV parsing and generation, controlling delimiters, quoting, line termination, and header handling.

### Overview

`CsvConfig` is a case class with sensible RFC 4180 defaults. Customize it when you need different delimiters (e.g., tabs), custom line endings, or different quoting behavior. A custom `CsvConfig` applies to `CsvReader` and `CsvWriter`; codecs derived with `CsvFormat` always use `CsvConfig.default`.

### Defaults and Presets

The standard RFC 4180 CSV format:

```scala

val config: CsvConfig = CsvConfig.default
// CsvConfig(',', '"', "\r\n", hasHeader = true, nullValue = "")
```

A tab-separated values (TSV) preset with tabs as delimiters:

```scala

val tsvConfig: CsvConfig = CsvConfig.tsv
// CsvConfig('\t', '"', "\r\n", hasHeader = true, nullValue = "")
```

### Configuration Fields

- **`delimiter`** (default: `','`) — Character separating fields in a row
- **`quoteChar`** (default: `'"'`) — Character to quote fields containing delimiters or newlines; escapes quotes within quoted fields by doubling
- **`lineTerminator`** (default: `"\r\n"`) — Sequence terminating each row (RFC 4180 standard is CRLF)
- **`hasHeader`** (default: `true`) — Whether the first row contains column headers
- **`nullValue`** (default: `""`) — String representation for null values. A known bug in 0.0.56: no code in the module reads this field, so changing it has no effect on `CsvReader`, `CsvWriter` or the derived codecs.

### Custom Configuration

To use a pipe-delimited format:

```scala

val customConfig = CsvConfig(
  delimiter = '|',
  quoteChar = '"',
  lineTerminator = "\n",
  hasHeader = true
)
```

---

## CsvError

Sealed abstract class representing errors during CSV parsing and type conversion. Designed for high-performance CSV processing with zero-overhead exceptions (no stack traces).

### Error Hierarchy

**ParseError** — Occurs when CSV format is invalid. It carries a `row` and a `column`, both 1-based. A known bug in 0.0.56: `CsvReader.readAll` counts the row from the start of the row that failed, not from the start of the document, so a stray character on line 4 of a file is reported as row 1:

```scala

val parseError = CsvError.ParseError("Unclosed quoted field", row = 2, column = 15)
```

**TypeError** — Defined for a field that cannot be converted to the expected type, but no code in 0.0.56 constructs it. A cell that doesn't convert decodes to a `SchemaError` from `CsvCodec#decode`, which does not name the column (see [Pattern 3](#pattern-3-report-bad-rows-by-number)). Do not match on `CsvError.TypeError` in your code.

### Error Information

`ParseError` exposes its message, row and column:

```scala

val error = CsvError.ParseError("Unexpected character after closing quote", row = 1, column = 10)
val message: String = error.getMessage
val row: Int = error.row
val column: Int = error.column
```

---

## CsvReader

Low-level CSV row parser implementing RFC 4180 with a state machine approach. Provides stateless utility methods for parsing individual rows, headers, and complete documents.

### Parsing a Single Row

To parse one CSV row starting at a given offset in the input:

```scala

val input = "Alice,30,alice@example.com\nBob,25,bob@example.com\n"
val config = CsvConfig.default

val result: Either[CsvError, (IndexedSeq[String], Int)] =
  CsvReader.readRow(input, offset = 0, config)

result match {
  case Right((fields, nextOffset)) =>
    println(s"Fields: $fields")
    println(s"Next offset: $nextOffset")
  case Left(err) =>
    println(s"Parse error: ${err.getMessage}")
}
```

### Parsing the Header Row

To parse just the first row as column headers:

```scala

val input = "name,age,email\nAlice,30,alice@example.com\n"
val config = CsvConfig.default

val result: Either[CsvError, (IndexedSeq[String], Int)] =
  CsvReader.readHeader(input, config)
```

### Parsing Complete CSV

To parse a complete CSV document (header and all data rows):

```scala

val csv = "name,age\nAlice,30\nBob,25\n"
val config = CsvConfig.default

val result: Either[CsvError, (IndexedSeq[String], IndexedSeq[IndexedSeq[String]])] =
  CsvReader.readAll(csv, config)

result match {
  case Right((header, rows)) =>
    println(s"Header: $header")
    rows.foreach(row => println(s"Row: $row"))
  case Left(err) =>
    println(s"Parse error: ${err.getMessage}")
}
```

---

## CsvWriter

Low-level CSV row serializer implementing RFC 4180 with proper field escaping.

### Writing a Single Row

To serialize one row of fields to CSV format:

```scala

val fields: IndexedSeq[String] = Vector("Alice", "30", "alice@example.com")
val config = CsvConfig.default

val csvLine: String = CsvWriter.writeRow(fields, config)
// Result: "Alice,30,alice@example.com\r\n"
```

### Writing Headers

To write a header row (semantically identical to `CsvWriter#writeRow` but communicates intent):

```scala

val columnNames: IndexedSeq[String] = Vector("name", "age", "email")
val config = CsvConfig.default

val csvHeader: String = CsvWriter.writeHeader(columnNames, config)
// Result: "name,age,email\r\n"
```

### Writing Complete CSV

To write a complete CSV document with headers and data rows:

```scala

val header: IndexedSeq[String] = Vector("product", "quantity")
val rows: Seq[IndexedSeq[String]] = Seq(
  Vector("Widget", "10"),
  Vector("Gadget", "5"),
  Vector("Gizmo", "8")
)
val config = CsvConfig.default

val csv: String = CsvWriter.writeAll(header, rows, config)
```

### RFC 4180 Escaping

`CsvWriter` automatically escapes fields according to RFC 4180:
- Fields containing the delimiter, quote character, or newlines are wrapped in quotes
- Quote characters within quoted fields are doubled (e.g., `Alice "The Expert" Smith` becomes `"Alice ""The Expert"" Smith"`)
- Fields not requiring escaping are written as-is for readability

---

## CsvCodecDeriver

Implements schema-driven derivation of `CsvCodec[A]` instances. Automatically handles 30 primitive types and flat record types, rejecting unsupported types (sealed traits, `Option` fields, sequences, maps) with clear error messages.

### Primitive Type Support

All 30 ZIO Blocks Schema primitives are supported:
- Numeric: `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, `BigDecimal`
- Logical: `Boolean`, `Char`, `String`
- Temporal: `Instant`, `LocalDate`, `LocalDateTime`, `LocalTime`, `Duration`, `Period`, `Year`, `YearMonth`, `MonthDay`, `Month`, `DayOfWeek`, `ZonedDateTime`, `OffsetDateTime`, `OffsetTime`, `ZoneId`, `ZoneOffset`
- Special: `UUID`, `Currency`, `Unit`

### Record Type Support

Flat case classes (records with no variant or sequence fields) are supported. Each field becomes a CSV column with the field name as the header:

```scala

case class Address(street: String, city: String, zip: String)

object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

val codec = Address.schema.derive(CsvFormat)
```

Printing `codec.headerNames` shows one header per `Address` field:

```scala
println(codec.headerNames)
// Vector(street, city, zip)
```

A field whose type is another case class is not rejected, but it does not become separate columns either. A known bug in 0.0.56: the deriver writes the nested record as one cell, including its `\r\n` line terminator, so the cell spans two lines in a spreadsheet. Flatten a nested record into separate fields before you derive the codec. A field of an `AnyVal` case class type counts as a nested record too, because `Schema.derived` gives a value class a record schema, so a typed id such as `case class UserId(value: Long) extends AnyVal` breaks the export in the same way; a schema built with `Schema#transform` is a wrapper and is written as a plain cell.

A known bug in 0.0.56: the derived codec ignores `Modifier.rename` and `Modifier.transient`. The header is always the Scala field name, and a transient field is still written and read as a column.

### Unsupported Types

Attempting to derive a codec for unsupported types (sealed traits, `Option` fields, sequences, maps, dynamic types) throws an `UnsupportedOperationException` at run time, when `Schema#derive` runs; the code still compiles. `Option` is rejected as a variant, with the message `CSV does not support variant/sum types. Use a flat case class instead.` The deriver rejects these to prevent silent failures in CSV processing.

---

## CsvFormat

Integration point with ZIO Blocks Schema's format system. Provides `TextFormat[CsvCodec]` to enable `Schema[A].derive(CsvFormat)` for any supported type.

To derive a CSV codec using the standard format:

```scala

case class Sensor(id: Long, temperature: Double, humidity: Float)

object Sensor {
  implicit val schema: Schema[Sensor] = Schema.derived
}

val codec = Sensor.schema.derive(CsvFormat)
```

`CsvFormat` is a singleton object extending `TextFormat[CsvCodec]` with the MIME type `"text/csv"` and the `CsvCodecDeriver` as its derivation strategy.

---

## Built-in Formats and Codecs

ZIO Blocks Schema provides codec derivation for multiple serialization formats. Once you have a `Schema[A]`, formats can be passed to `Schema.derive`, while BSON is derived with `Schema[A].derive(BsonCodecDeriver)`. The older `BsonSchemaCodec.bsonCodec(schema)` API remains a compatibility facade. See the [Format documentation](../format.md) for details.

## Built-in Codecs

Here's a summary of the codecs currently supported by ZIO Blocks. Most codecs provide a `BinaryFormat` or `TextFormat`; BSON provides a `Deriver` because it operates on native BSON readers, writers, and values. See the dedicated documentation for details:

| Derivation API      | Codec Type            | MIME Type             | Module                          | Documentation                   |
|---------------------|-----------------------|-----------------------|---------------------------------|---------------------------------|
| `JsonFormat`        | `JsonCodec[A]`        | `application/json`    | `zio-blocks-schema`             | [JSON](./json/index.md)         |
| `AvroFormat`        | `AvroCodec[A]`        | `application/avro`    | `zio-blocks-schema-avro`        | [Avro](./avro.md)               |
| `BsonCodecDeriver`  | `BsonCodec[A]`        | `application/bson`    | `zio-blocks-schema-bson`        | [BSON](./bson.md)               |
| `CsvFormat`         | `CsvCodec[A]`         | `text/csv`            | `zio-blocks-schema-csv`         | [CSV](./csv.md)                 |
| `MessagePackFormat` | `MessagePackCodec[A]` | `application/msgpack` | `zio-blocks-schema-messagepack` | [MessagePack](./messagepack.md) |
| `ThriftFormat`      | `ThriftCodec[A]`      | `application/thrift`  | `zio-blocks-schema-thrift`      | [Thrift](./thrift.md)           |
| `ToonFormat`        | `ToonCodec[A]`        | `text/toon`           | `zio-blocks-schema-toon`        | [TOON](./toon.md)               |
| `XmlFormat`         | `XmlCodec[A]`         | `application/xml`     | `zio-blocks-schema-xml`         | [XML](./xml.md)                 |
| `YamlFormat`        | `YamlCodec[A]`        | `application/yaml`    | `zio-blocks-schema-yaml`        | [YAML](./yaml.md)               |

The module names above are for Scala 2.13 and Scala 3.3 to 3.8. On Scala 3.9 and newer, use the `zio-blocks-next-` modules instead, for example `zio-blocks-next-schema-yaml`, and add `scalacOptions += "-experimental"` (see [How to Add ZIO Blocks to Your Build](../../../getting-started/installation.md)).

## Supported Types

Most formats support the full set of ZIO Blocks Schema primitive types. Some formats have limitations:
- **CSV** supports only flat records and primitive types (no variants, `Option` fields, sequences, maps, or dynamic types)
- **XML** supports records, sequences, and variants with configurable discriminator strategies
- Other formats support all composite types listed below

For format-specific limitations, see the dedicated codec documentation.

**Numeric Types**:
- `Boolean`, `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `Char`
- `BigInt`, `BigDecimal`

**Text Types**:
- `String`

**Special Types**:
- `Unit`, `UUID`, `Currency`

**Java Time Types**:
- `Instant`, `LocalDate`, `LocalTime`, `LocalDateTime`
- `OffsetTime`, `OffsetDateTime`, `ZonedDateTime`
- `Duration`, `Period`
- `Year`, `YearMonth`, `MonthDay`
- `DayOfWeek`, `Month`
- `ZoneId`, `ZoneOffset`

**Composite Types**:
- Records (case classes)
- Variants (sealed traits)
- Sequences (`List`, `Vector`, `Set`, `Array`, etc.)
- Maps (`Map[K, V]`)
- Options (`Option[A]`)
- Eithers (`Either[A, B]`)
- Wrappers (newtypes)

## Cross-Platform Support

| Format      | JVM | Scala.js |
|-------------|-----|----------|
| JSON        | ✓   | ✓        |
| TOON        | ✓   | ✓        |
| MessagePack | ✓   | ✓        |
| CSV         | ✓   | ✓        |
| XML         | ✓   | ✓        |
| YAML        | ✓   | ✓        |
| Avro        | ✓   | ✗        |
| Thrift      | ✓   | ✗        |
| BSON        | ✓   | ✗        |

## Error Handling

All formats return `Either[SchemaError, A]` for decoding operations. Errors include path information for debugging, showing exactly where in nested structures a decoding failure occurred. See individual codec documentation for format-specific error details.

## See Also

- [One Type, Many Formats](../../../guides/one-type-many-formats.md) - a tutorial that derives JSON, YAML, XML, TOON, MessagePack, Avro and CSV codecs from one `Schema` and round-trips a catalog through each.

---

## JSON Codec

The JSON codec module provides complete, type-safe support for working with JSON data in ZIO Blocks. It includes an ADT for representing JSON values, a fluent navigation API (`JsonSelection`), configurable encoding/decoding (`JsonCodec` with `WriterConfig`/`ReaderConfig`), composable patches for transformations, a diff algorithm for computing minimal changes, and full JSON Schema 2020-12 support for validation and code generation.

**Core types:** `Json`, `JsonCodec`, `JsonSelection`, `JsonPatch`, `JsonDiffer`, `JsonSchema`, `JsonType`.

```scala

// Represent any JSON value
val person = Json.Object("name" -> Json.String("Alice"), "age" -> Json.Number(30))

// Navigate with fluent API
val age = person.get("age")  // JsonSelection

// Encode to string
val encoded = person.print(WriterConfig)

// Compute minimal patches
val updated = person.set(p".age", Json.Number(31))
val patch = JsonPatch.diff(person, updated)

// Validate against a schema
case class Person(name: String, age: Int)
object Person { implicit val schema: Schema[Person] = Schema.derived }
val schema: JsonSchema = Schema[Person].toJsonSchema
schema.conforms(person)  // true
```

## Installation

The JSON codec is included in the ZIO Blocks Schema module. Add it to your `build.sbt`:

```scala
// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56"
scalacOptions += "-experimental"
```

For Scala.js projects, use `%%%` instead of `%%`.

**Supported Scala versions:** 2.13.x and 3.x. See [How to Add ZIO Blocks to Your Build](../../../../getting-started/installation.md) for details.

The JSON codec is fully integrated into ZIO Blocks Schema and provides complete type-safe JSON support with no external dependencies beyond core ZIO libraries.

## How They Work Together

The JSON module workflow moves through representation, navigation, encoding/decoding, diffing, patching, and validation:

```
Parse/Create JSON
    ↓
Json (ADT: Object, Array, String, Number, Boolean, Null)
    ├─ Navigate with JsonSelection (fluent query API)
    ├─ Understand with JsonType (type information)
    ├─ Transform via JsonPatch (composable operations)
    └─ Merge with MergeStrategy (field conflict resolution)
    ↓
JsonCodec (encode/decode)
    ├─ WriterConfig (indent step, preferred buffer size, Unicode escaping)
    └─ ReaderConfig (buffer sizes, end-of-input check)
    ↓
JsonDiffer (diff algorithm)
    └─ Produces JsonPatch (minimal changes)
    ↓
JsonPatch.apply (transform JSON)
    ↓
JsonSchema (validate or generate)
    ├─ Derive from Scala types
    ├─ Validate JSON conformance
    └─ Generate documentation
```

**Type Relationships:**

- `Json` is the central value type; all operations start with or produce it
- `JsonSelection` provides fluent navigation through nested `Json` structures
- `JsonCodec` bridges Scala types ↔ `Json` with configurable `WriterConfig` and `ReaderConfig`
- `JsonDiffer` computes `JsonPatch` by comparing two `Json` values
- `JsonPatch` transforms `Json` values with composable operations
- `JsonSchema` validates `Json` conformance and derives from Scala `Schema` types
- `JsonType` provides type information for `Json` values at runtime

## Common Patterns

### Pattern 1: Parse, Transform, Serialize

Read JSON, modify it, and write it back:

```scala

val jsonString = """{"name": "Alice", "age": 30}"""
val json = Json.parseUnsafe(jsonString)

// Transform
val updated = json.modify(p".age") {
  case Json.Number(n) => Json.Number(n + 1)
  case other => other
}

// Serialize
val output = updated.print(WriterConfig.withIndentionStep2)
```

### Pattern 2: Diff and Apply

Compute changes and apply them:

```scala

val original = Json.Object("count" -> Json.Number(0))
val target = Json.Object("count" -> Json.Number(1), "active" -> Json.Boolean(true))

// Compute the diff
val patch = JsonPatch.diff(original, target)

// Apply to reconstruct
val result = patch.apply(original)
// Right({"count": 1, "active": true})
```

### Pattern 3: Validate with Schema

Generate and validate using schemas:

```scala

case class User(name: String, email: String, age: Int)
object User {
  implicit val schema: Schema[User] = Schema.derived
}

val jsonSchema: JsonSchema = Schema[User].toJsonSchema
val validJson = Json.Object(
  "name" -> Json.String("Alice"),
  "email" -> Json.String("alice@example.com"),
  "age" -> Json.Number(30)
)

jsonSchema.conforms(validJson)  // true
```

### Pattern 4: Navigate and Extract with JsonSelection

Use fluent API to navigate nested structures and extract values:

```scala

val data = Json.Object(
  "users" -> Json.Array(
    Json.Object("name" -> Json.String("Alice"), "age" -> Json.Number(30)),
    Json.Object("name" -> Json.String("Bob"), "age" -> Json.Number(25))
  )
)

// Navigate to first user's age
val age: Either[SchemaError, Int] = data
  .get("users")(0)    // first element of the users array
  .get("age")         // Navigate to age field
  .as[Int]            // Decode to Int

// Check values
val ageValue = data.get("users")(0).get("age").one  // Right(Json.Number(30))
```

### Pattern 5: Encode and Decode with Configuration

Control output formatting and parsing behavior:

```scala

case class Config(host: String, port: Int)
object Config {
  implicit val schema: Schema[Config] = Schema.derived
}

val config = Config("localhost", 8080)
val json = config.toJson

// Encode with custom formatting
val pretty = json.print(WriterConfig.withIndentionStep(2))

// Decode: parse back to Json, then decode
val parsed: Json = Json.parseUnsafe(pretty)
val config2: Either[SchemaError, Config] = parsed.as[Config]
```

### Pattern 6: Compose Multiple Patches

Chain multiple transformations into a single patch:

```scala

val original = Json.Object("x" -> Json.Number(1), "y" -> Json.Number(2))

// Build patches independently
val patchX = JsonPatch.diff(
  original,
  original.set(p".x", Json.Number(10))
)

val patchY = JsonPatch.diff(
  original.set(p".x", Json.Number(10)),
  original.set(p".x", Json.Number(10)).set(p".y", Json.Number(20))
)

// Compose them
val combined = patchX ++ patchY
combined.apply(original)
// Right({"x": 10, "y": 20})
```

## Integration Points

**With other codecs:** `Json` values convert to/from other formats (Avro, TOON, etc.) via `DynamicValue`:

```scala

val json = Json.parseUnsafe("""{"name": "Alice"}""")
val dynamic: DynamicValue = json.toDynamicValue

// Convert to other formats using DynamicValue
```

**With Schema system:** `Schema` enables automatic JSON encoding/decoding:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Encode using Schema-based extension methods
val person = Person("Alice", 30)
val json = person.toJson
val jsonString = person.toJsonString

// Decode: parse and extract with Schema
val parsed: Json = Json.parseUnsafe(jsonString)
val decoded: Either[SchemaError, Person] = parsed.as[Person]
```

**With patching system:** `JsonPatch` and `JsonDiffer` integrate with generic `Patch` infrastructure:

```scala

val jsonPatch: JsonPatch = JsonPatch.root(JsonPatch.Op.Set(Json.Null))
val dynamicPatch: DynamicPatch = jsonPatch.toDynamicPatch
```

**With optics and navigation:** `JsonSelection` provides query-like access complementary to `Optic` system:

- `JsonSelection` — runtime navigation through unknown JSON structures
- `Optic` — compile-time, type-safe navigation through known Scala types
- Both support nested access, filtering, and composition

**With validation:** `JsonSchema` integrates with Schema `Validation` system:

- Derive `JsonSchema` from Scala `Schema[A]`
- Validate raw JSON before type-safe decoding
- Compose with logical operators (allOf, anyOf, oneOf)

## Type Pages

- **[Json](./json.md)** — The core ADT representing JSON values with six cases (Object, Array, String, Number, Boolean, Null). Covers construction, navigation, modification, transformation, filtering, merging, and encoding.

- **[JsonPatch](./json-patch.md)** — Composable patches for transforming JSON values. Create patches via diff or manually, compose them with `++`, and apply with different failure modes (Strict, Lenient, Clobber).

- **[JsonDiffer](./json-differ.md)** — Diff algorithm computing minimal patches. Uses smart strategies per type: NumberDelta for numbers, LCS-based StringEdit for strings, ArrayEdit with insertion/deletion for arrays, and ObjectEdit for fields.

- **[JSON Schema](./json-schema.md)** — Full JSON Schema 2020-12 support for validation and code generation. Derive schemas from Scala types, validate JSON, construct schemas manually with builders, and combine with logical operators.

See also the [Your First Schema](../../../../guides/your-first-schema.md) tutorial, which round-trips a type through JSON end to end.

For throughput, see [How to Tune JSON Encoding and Decoding for Throughput](../../../../guides/tune-json-throughput.md).

Coming from another JSON library? [Coming from circe, zio-json or jsoniter-scala](../../../../guides/json-library-migration.md) maps their codecs, configuration and annotations onto ZIO Blocks.

---

## JSON Configuration

The JSON codec module provides four configuration types for controlling encoding and decoding behavior: `WriterConfig`, `ReaderConfig`, `MergeStrategy`, and `NameMapper`. These types allow fine-grained control over how JSON is serialized and deserialized.

## WriterConfig

`WriterConfig` controls the formatting and content of encoded JSON output. Use it when calling `Json#print` on `Json` values or when encoding values with specific formatting requirements.

### Configuration Options

| Option             | Type    | Default | Purpose                                                       |
|--------------------|---------|---------|---------------------------------------------------------------|
| `indentionStep`    | Int     | 0       | Spaces per indentation level (0 = compact JSON)               |
| `escapeUnicode`    | Boolean | false   | Escape non-ASCII characters as `\uXXXX` for ASCII-only output |
| `preferredBufSize` | Int     | 32768   | Internal buffer size in bytes for streaming                   |

### Usage Examples

Pretty-print JSON output with configurable indentation:

```scala

case class Config(server: String, port: Int, debug: Boolean)
object Config { implicit val schema: Schema[Config] = Schema.derived }

val config = Config("localhost", 8080, true)
val json = config.toJson

// Pretty-printed with 2-space indentation
val pretty = json.print(WriterConfig.withIndentionStep(2))
// {
//   "server": "localhost",
//   "port": 8080,
//   "debug": true
// }
```

Configure custom indentation for JSON output:

```scala

case class User(name: String, email: String, age: Int)
object User { implicit val schema: Schema[User] = Schema.derived }

val user = User("Alice", "alice@example.com", 30)
val json = user.toJson

// 4-space indentation
val sorted = json.print(WriterConfig
  .withIndentionStep(4)
)
// {
//     "name": "Alice",
//     "email": "alice@example.com",
//     "age": 30
// }
```

Escape non-ASCII characters for ASCII-only transmission:

```scala

case class Message(text: String)
object Message { implicit val schema: Schema[Message] = Schema.derived }

val msg = Message("Hello 世界")
val json = msg.toJson

// Escape non-ASCII characters - non-ASCII become \uXXXX sequences
val ascii = json.print(WriterConfig.withEscapeUnicode(true))
// Output: {"text":"Hello \\u4e16\\u754c"}
```

## ReaderConfig

`ReaderConfig` controls how JSON input is parsed and validated during decoding. Use it when calling `Json.parse()` with custom parsing behavior.

### Configuration Options

| Option                 | Type    | Default  | Purpose                                                                                   |
|------------------------|---------|----------|-------------------------------------------------------------------------------------------|
| `checkForEndOfInput`   | Boolean | true     | Reject anything but whitespace after the value                                            |
| `preferredCharBufSize` | Int     | 4096     | Starting size of the character buffer for strings                                         |
| `maxCharBufSize`       | Int     | 4194304  | Longest string value or key, in chars                                                     |
| `preferredBufSize`     | Int     | 32768    | Starting size of the byte buffer for `InputStream` and direct (not array-backed) `ByteBuffer` input; an array-backed `ByteBuffer` is read in place |
| `maxBufSize`           | Int     | 33554432 | Largest size of that byte buffer, for example while looking for a late discriminator field |

### Usage Examples

Stop after the first value instead of rejecting the text after it. Whitespace after the value is always allowed:

```scala

// Lenient: ignore whatever follows the first value
val lenientConfig = ReaderConfig.withCheckForEndOfInput(false)

val jsonString = """{"name": "Alice"} {"name": "Bob"}"""
val result = Json.parse(jsonString, lenientConfig)
// Right({"name": "Alice"}); with the default ReaderConfig this is a Left
```

## MergeStrategy

`MergeStrategy` determines how field conflicts are resolved when merging two JSON objects. It provides strategies for different use cases, from strict validation to lenient concatenation.

**Available Strategies:**
- `Auto` (the default) — Merges objects by key and arrays by index at every depth; the right value wins elsewhere
- `Shallow` — Merges only the top-level object or array; nested values are replaced by the right ones
- `Concat` — Merges objects by key and appends the right array to the left one
- `Replace` — The right value replaces the left one
- `Custom(f, r)` — `r` decides where to descend, and `f` merges everything else

For details on merge strategies and their usage, see the Merging section of the [Json](./json.md) documentation.

## NameMapper

`NameMapper` customizes how field names are transformed between Scala types and JSON representation. It enables patterns like camelCase ↔ snake_case conversion.

### Built-in Mappers

Use identity mapping to keep field names unchanged:

```scala

case class User(firstName: String, lastName: String)
object User { 
  implicit val schema: Schema[User] = Schema.derived
}

// Default: no transformation
val user = User("Alice", "Smith")
val json = user.toJson
// {"firstName": "Alice", "lastName": "Smith"}
```

**Snake Case:**
Convert camelCase to snake_case:

```scala

@Modifier.fieldNaming("snake_case")
case class User(firstName: String, lastName: String)
object User { 
  implicit val schema: Schema[User] = Schema.derived
}

val user = User("Alice", "Smith")
val json = user.toJson
// {"first_name": "Alice", "last_name": "Smith"}
```

**Kebab Case:**
Convert camelCase to kebab-case:

```scala

@Modifier.fieldNaming("kebab-case")
case class User(firstName: String, lastName: String)
object User { 
  implicit val schema: Schema[User] = Schema.derived
}

val user = User("Alice", "Smith")
val json = user.toJson
// {"first-name": "Alice", "last-name": "Smith"}
```

**Custom Mappers:**
Besides `Identity`, `SnakeCase` and `KebabCase`, `NameMapper` has `CamelCase`, `PascalCase` and `Custom(f)`, which applies any `String => String` function. An implicit `NameMapper` in scope has no effect: annotate the type with `@Modifier.fieldNaming` (or `@Modifier.caseNaming` for case names), or derive a codec with `JsonCodecDeriver.withFieldNameMapper` and `Schema#derive`.

## Combining Configuration

Use multiple configuration options together for fine-grained control:

```scala

case class Config(server: String, port: Int, timeout: Option[Int])
object Config { implicit val schema: Schema[Config] = Schema.derived }

val config = Config("example.com", 443, None)
val json = config.toJson

// Pretty-printed with 4-space indentation
val output = json.print(WriterConfig
  .withIndentionStep(4)
)
// {
//     "server": "example.com",
//     "port": 443
// }
```

## Performance Implications

- **WriterConfig:** Indentation increases output size but has minimal performance impact
- **ReaderConfig:** `checkForEndOfInput` skips the whitespace after the value to the end of the input
- **MergeStrategy:** Strategy choice affects merge performance based on use case
- **NameMapper:** Applied once per field during schema derivation; zero runtime cost if identity mapping

## Best Practices

1. **Use WriterConfig for human-readable output only** — Compact output is faster for wire transmission
2. **Choose MergeStrategy based on use case** — `Auto` for deep merges of defaults, `Concat` to accumulate lists, `Replace` to override
3. **Standardize on one NameMapper** across your codebase to avoid confusion
4. **Cache WriterConfig/ReaderConfig instances** — They're immutable and can be reused

## See Also

- [How to Customize JSON Field Names, ADT Encoding and Optional Fields](../../../../guides/customize-json-encoding.md): a task-oriented guide that applies these settings to a real model.
- [Coming from circe, zio-json or jsoniter-scala](../../../../guides/json-library-migration.md): maps circe `Configuration`, zio-json annotations and jsoniter-scala `CodecMakerConfig` options onto these settings.
- [How to Tune JSON Encoding and Decoding for Throughput](../../../../guides/tune-json-throughput.md): decode from bytes and streams, reuse codecs and size buffers for throughput.

---

## JsonDiffer

`JsonDiffer` is a **diff algorithm for JSON values** that computes the minimal [`JsonPatch`](./json-patch.md) transforming one `Json` value into another. It is the foundation of `JsonPatch.diff`.

```scala
object JsonDiffer {
  def diff(source: Json, target: Json): JsonPatch
}
```

## Overview

`JsonDiffer` solves a fundamental problem: given two `Json` values, what is the most compact representation of the changes from one to the other?

Instead of transmitting or storing the entire new value, `JsonDiffer` emits a `JsonPatch` containing only the differences. For each type of change — numeric updates, string mutations, array reordering, field additions — it selects the most space-efficient representation:

```
Original JSON              Target JSON
┌──────────────────┐      ┌──────────────────┐
│ {                │      │ {                │
│   "name": "Alice"│      │   "name": "Alice"│
│   "age": 25      │─────→│   "age": 26      │
│   "city": "NYC"  │ diff │   "city": "NYC"  │
│ }                │      │ }                │
└──────────────────┘      └──────────────────┘
        │
        ▼
   JsonPatch {
     ObjectEdit(
       Modify("age", NumberDelta(1))
     )
   }
```

The core guarantee: `JsonDiffer.diff(source, target).apply(source) == Right(target)`, as long as neither value has an object with duplicate keys (a known bug in 0.0.56: a duplicate key in `source` survives the patch, and one in `target` makes the patch fail).

## Motivation

Transmitting or storing entire JSON documents wastes bandwidth, disk space, and network latency when only a fraction of the data changes. `JsonDiffer` enables:

- **Efficient data synchronization** — send only what changed, not the entire document
- **Change tracking and audit logs** — record every mutation as a first-class value
- **Optimistic concurrency control** — detect conflicts by analyzing patches rather than full documents
- **Replay and time-travel debugging** — reconstruct any historical state by applying patches in sequence
- **Compression** — patches are often 10–100× smaller than the full target value

## The Diffing Algorithm

`JsonDiffer.diff` adapts its strategy per `Json` type, choosing the most compact representation for each kind of change.

### Type Mismatches

When the types differ (e.g., number to string, object to array), `JsonDiffer` emits `Op.Set` to replace the value entirely. This is the only compact choice when the types diverge:

```scala

val numberToString = JsonDiffer.diff(Json.Number(42), Json.String("hello"))
val objectToArray  = JsonDiffer.diff(
  Json.Object("x" -> Json.Number(1)),
  Json.Array(Json.Number(1), Json.Number(2))
)
```

Both patches consist of a single `Set` operation:

```scala

numberToString.apply(Json.Number(42))
objectToArray.apply(Json.Object("x" -> Json.Number(1)))
```

### Numbers

For numeric changes, `JsonDiffer` emits `NumberDelta` — representing the difference between old and new values as a delta rather than replacing the entire number:

```scala

val increment = JsonDiffer.diff(Json.Number(100), Json.Number(105))
val decrement = JsonDiffer.diff(Json.Number(50), Json.Number(48))
```

The patches store deltas, not full new values:

```scala

increment.apply(Json.Number(100))
decrement.apply(Json.Number(50))
```

### Strings

For strings, `JsonDiffer` chooses between two strategies:

1. **LCS (Longest Common Subsequence) edit operations** — if the edits are more compact than the full new string.
2. **Full replacement with `Set`** — if the string has changed so much that storing the entire new value is smaller.

The algorithm computes common characters, then emits `Insert` and `Delete` operations (with `Append` and `Modify` existing as supported patch ops but not currently produced by `JsonDiffer`). It only emits the edits if their byte size is smaller than the new string's length:

```scala

// Prefix insertion — common suffix preserved
val addPrefix = JsonDiffer.diff(Json.String("world"), Json.String("hello world"))

// Complete replacement — almost no common subsequence
val replacement = JsonDiffer.diff(Json.String("abc"), Json.String("xyz"))
```

The first patch uses `StringEdit` (compact), the second uses `Set` (more efficient):

```scala
addPrefix.apply(Json.String("world"))
replacement.apply(Json.String("abc"))
```

### Arrays

For arrays, `JsonDiffer` uses **LCS-aligned insert, delete, and append operations** to transform the old array into the new one. It aligns common elements and generates the minimum sequence of mutations:

```scala

// Append elements at the end
val append = JsonDiffer.diff(
  Json.Array(Json.Number(1), Json.Number(2)),
  Json.Array(Json.Number(1), Json.Number(2), Json.Number(3))
)

// Delete and reorder — LCS finds the common elements
val reorder = JsonDiffer.diff(
  Json.Array(Json.Number(1), Json.Number(2), Json.Number(3)),
  Json.Array(Json.Number(3), Json.Number(2), Json.Number(1))
)
```

Both patches use efficient `ArrayEdit` operations:

```scala
append.apply(Json.Array(Json.Number(1), Json.Number(2)))
reorder.apply(Json.Array(Json.Number(1), Json.Number(2), Json.Number(3)))
```

### Objects

For objects, `JsonDiffer` compares field-by-field:

- Fields in the target but not in the source become `Add` operations.
- Fields in the source but not in the target become `Remove` operations.
- Fields in both are recursively diffed — if the values differ, a `Modify` with a sub-patch is emitted.

```scala

val original = Json.Object(
  "name" -> Json.String("Alice"),
  "age"  -> Json.Number(25),
  "city" -> Json.String("NYC")
)

val updated = Json.Object(
  "name"  -> Json.String("Alice"),
  "age"   -> Json.Number(26),
  "email" -> Json.String("alice@example.com")
)

val patch = JsonDiffer.diff(original, updated)
```

The patch contains `Add`, `Remove`, and `Modify` operations:

```scala
patch.apply(original)
```

### Nested Structures

`JsonDiffer` handles deeply nested objects and arrays recursively. Each field or element is diffed independently, producing compact patches at every level:

```scala

val original = Json.Object(
  "user" -> Json.Object(
    "name" -> Json.String("Alice"),
    "scores" -> Json.Array(Json.Number(95), Json.Number(87))
  )
)

val updated = Json.Object(
  "user" -> Json.Object(
    "name" -> Json.String("Alice"),
    "scores" -> Json.Array(Json.Number(95), Json.Number(88), Json.Number(92))
  )
)

val patch = JsonDiffer.diff(original, updated)
```

The patch navigates to the nested array and emits only the array changes:

```scala
patch.apply(original)
```

## Core Operations

`JsonDiffer` exposes a single public operation: `diff`, which computes the minimal `JsonPatch` that transforms `source` into `target`. Returns an empty patch if the values are already equal:

```scala
object JsonDiffer {
  def diff(source: Json, target: Json): JsonPatch
}
```

For values without duplicate keys, the roundtrip property holds: applying the patch to the source yields the target:

```scala

val source = Json.Object("x" -> Json.Number(10), "y" -> Json.Number(20))
val target = Json.Object("x" -> Json.Number(10), "y" -> Json.Number(21))
val patch  = JsonDiffer.diff(source, target)
```

```scala
patch.apply(source) == Right(target)
// res13: Boolean = true
```

`JsonDiffer.diff` is also available as `JsonPatch.diff` and as the `Json#diff` extension method:

```scala

// Via JsonPatch companion
val p1 = JsonPatch.diff(Json.Number(1), Json.Number(2))

// Via Json extension method
val p2 = Json.Number(1).diff(Json.Number(2))
```

## Integration

`JsonDiffer` is the implementation behind the public `JsonPatch.diff` API. You typically interact with it through `JsonPatch.diff` or the `Json#diff` extension method rather than calling `JsonDiffer.diff` directly.

The relationship is simple: `JsonPatch.diff` delegates to `JsonDiffer.diff` and wraps the result in a `JsonPatch` for further composition and application.

Once you have a `JsonPatch` from `JsonDiffer`, use the full `JsonPatch` API to apply it with different modes, compose multiple patches, or convert to/from `DynamicPatch`:

```scala

val source = Json.Object("count" -> Json.Number(0))
val target = Json.Object("count" -> Json.Number(1))
val patch  = JsonDiffer.diff(source, target)
```

Apply with different failure-handling modes:

```scala
patch.apply(source, PatchMode.Strict)
patch.apply(source, PatchMode.Lenient)
patch.apply(source, PatchMode.Clobber)
```

Compose patches with `++`:

```scala
val patch2 = JsonDiffer.diff(target, Json.Object("count" -> Json.Number(2)))
val combined = patch ++ patch2
```

## Implementation Notes

`JsonDiffer.diff` uses the LCS (Longest Common Subsequence) algorithm for both strings and arrays, ensuring that the number of edit operations is minimized. For strings, it compares character sequences; for arrays, it compares JSON elements by structural equality.

The LCS-based approach is particularly effective for arrays and strings with significant common subsequences — a common pattern in real-world data mutation scenarios (e.g., adding an item to a list, inserting a few characters into a string).

:::note
The LCS algorithm has O(n·m) time complexity, where n and m are the lengths of the two sequences. For very large strings or arrays, consider whether you need the minimal patch or can accept a faster approximation.
:::

## Running the Examples

`JsonDiffer` is the foundation of the `JsonPatch` API. Runnable examples demonstrating `JsonPatch.diff` (which internally uses `JsonDiffer`) are available in the `schema-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run JsonPatch examples with sbt:**

These examples demonstrate how `JsonDiffer.diff` computes minimal patches through the public `JsonPatch.diff` API:

```bash
cd schema-examples && sbt "runMain jsonpatch.JsonPatchDiffAndApplyExample"
```

```bash
cd schema-examples && sbt "runMain jsonpatch.JsonPatchOperationsExample"
```

```bash
cd schema-examples && sbt "runMain jsonpatch.JsonPatchCompositionExample"
```

```bash
cd schema-examples && sbt "runMain jsonpatch.CompleteJsonPatchExample"
```

---

## JsonPatch

`JsonPatch` is an **untyped, composable patch** for [`Json`](./json.md) values. It represents a sequence of operations that transform one `Json` value into another — computed automatically via a diff algorithm or constructed manually. The two fundamental operations are `JsonPatch.diff` to compute a patch between two `Json` values, and `JsonPatch#apply` to apply it.

`JsonPatch`:
- is a pure value — applying it never mutates the input
- is composable via `++`, sequencing two patches one after another
- supports three failure-handling modes: `Strict`, `Lenient`, and `Clobber`
- carries its own `Schema` instances for full serialization support
- converts bidirectionally to/from `DynamicPatch` for use in generic patching pipelines

The `JsonPatch` type wraps a sequence of operations:

```scala
final case class JsonPatch(ops: Chunk[JsonPatch.JsonPatchOp])
```

## Motivation

In most systems, updating JSON data means transmitting the entire new value — even when only a single field changed. `JsonPatch` solves this by representing changes as a **first-class value** that can be:

- **Transmitted efficiently** — send only what changed, not the entire document
- **Stored for audit logs** — record every change for compliance, debugging, or undo
- **Composed** — merge multiple changes into a single atomic patch
- **Serialized** — persist patches to disk or a message queue and replay them later

```
Source JSON                    Target JSON
┌─────────────────────┐        ┌─────────────────────┐
│ { "name": "Alice",  │        │ { "name": "Alice",  │
│   "age": 25,        │──diff──│   "age": 26,        │
│   "city": "NYC" }   │        │   "city": "NYC" }   │
└─────────────────────┘        └─────────────────────┘
           │
           ▼
      JsonPatch {
        ObjectEdit(
          Modify("age", NumberDelta(1))
        )
      }
           │
           ▼ apply
┌─────────────────────┐
│ { "name": "Alice",  │
│   "age": 26,        │
│   "city": "NYC" }   │
└─────────────────────┘
```

The "hello world" for `JsonPatch` is diff-then-apply. We compute a patch from `source` to `target`, then verify that applying it to `source` reproduces `target`:

```scala

val source = Json.Object("name" -> Json.String("Alice"), "age" -> Json.Number(25))
val target = Json.Object("name" -> Json.String("Alice"), "age" -> Json.Number(26))

val patch: JsonPatch = JsonPatch.diff(source, target)
```

Applying the patch to `source` always yields `Right(target)`:

```scala
patch.apply(source) == Right(target)
// res1: Boolean = true
```

## Creating Patches

There are three ways to create a `JsonPatch`: compute one automatically with `JsonPatch.diff`, construct one manually with `JsonPatch.root` or `JsonPatch.apply`, or start from the identity patch `JsonPatch.empty`.

### `JsonPatch.diff`

Computes the minimal `JsonPatch` that transforms `source` into `target`. Uses a smart diff strategy per value type — see the [Diffing Algorithm](#diffing-algorithm) section for details:

```scala
object JsonPatch {
  def diff(source: Json, target: Json): JsonPatch
}
```

`JsonPatch.diff` is also available as the `Json#diff` extension method:

```scala

// Via companion object
val p1 = JsonPatch.diff(Json.Number(10), Json.Number(15))

// Via extension method on Json
val p2 = Json.Number(10).diff(Json.Number(15))

// Nested object diff produces minimal ObjectEdit
val p3 = JsonPatch.diff(
  Json.Object("a" -> Json.Number(1), "b" -> Json.Number(2)),
  Json.Object("a" -> Json.Number(1), "b" -> Json.Number(9))
)
// p3 only touches "b", leaves "a" unchanged
```

### `JsonPatch.root`

Creates a patch with a single operation applied at the root of the value:

```scala
object JsonPatch {
  def root(operation: JsonPatch.Op): JsonPatch
}
```

For example, we can replace the root entirely, increment a number, or add a field to a root object:

```scala

// Replace the entire value
val replaceAll = JsonPatch.root(Op.Set(Json.Null))

// Increment a number at the root
val increment = JsonPatch.root(Op.PrimitiveDelta(PrimitiveOp.NumberDelta(BigDecimal(1))))

// Add a field to a root object
val addField = JsonPatch.root(Op.ObjectEdit(Chunk(ObjectOp.Add("active", Json.Boolean(true)))))
```

### `JsonPatch.apply`

Creates a patch with a single operation applied at the specified `DynamicOptic` path. Use this when targeting a nested location within the value:

```scala
object JsonPatch {
  def apply(path: DynamicOptic, operation: JsonPatch.Op): JsonPatch
}
```

Paths are built fluently on `DynamicOptic.root` using `.field(name)` to navigate object fields and `.at(index)` to navigate array elements. For instance, to increment `age` inside a `user` object:

```scala

val agePath  = DynamicOptic.root.field("user").field("age")
val agePatch = JsonPatch(agePath, Op.PrimitiveDelta(PrimitiveOp.NumberDelta(BigDecimal(1))))
val nested   = Json.Object("user" -> Json.Object("name" -> Json.String("Alice"), "age" -> Json.Number(25)))
```

Applying the patch navigates to the nested `age` field and increments it:

```scala
agePatch.apply(nested)
// res5: Either[SchemaError, Json] = Right(
//   Object(
//     IndexedSeq(
//       (
//         "user",
//         Object(IndexedSeq(("name", String("Alice")), ("age", Number(26))))
//       )
//     )
//   )
// )
```

### `JsonPatch.empty`

The empty patch. Applying it to any `Json` value returns that value unchanged. `JsonPatch.empty` is the identity element for `++`:

```scala
object JsonPatch {
  val empty: JsonPatch
}
```

`JsonPatch.empty` is useful as a neutral starting point when building patches conditionally:

```scala

```

Both `JsonPatch#isEmpty` and applying `JsonPatch.empty` confirm the identity property:

```scala
JsonPatch.empty.isEmpty
// res7: Boolean = true
JsonPatch.empty.apply(Json.Number(42))
// res8: Either[SchemaError, Json] = Right(Number(42))
(JsonPatch.empty ++ JsonPatch.empty).isEmpty
// res9: Boolean = true
```

### `JsonPatch.fromDynamicPatch`

Converts a generic `DynamicPatch` to a `JsonPatch`. Returns `Left[SchemaError]` for operations not representable in JSON:
- Temporal deltas (`InstantDelta`, `DurationDelta`, etc.) — JSON has no native time type
- Non-string map keys — JSON object keys must always be strings

All numeric delta types (`IntDelta`, `LongDelta`, `DoubleDelta`, etc.) are widened to `NumberDelta(BigDecimal)`:

```scala
object JsonPatch {
  def fromDynamicPatch(patch: DynamicPatch): Either[SchemaError, JsonPatch]
}
```

The round-trip through `DynamicPatch` preserves numeric deltas:

```scala

val original: JsonPatch                  = JsonPatch.diff(Json.Number(1), Json.Number(2))
val dynPatch: DynamicPatch               = original.toDynamicPatch
val back: Either[SchemaError, JsonPatch] = JsonPatch.fromDynamicPatch(dynPatch)
```

The roundtrip succeeds and the recovered patch equals the original:

```scala
back == Right(original)
// res11: Boolean = true
```

## Core Operations

`JsonPatch` exposes operations for applying patches, composing them, and converting between formats. The three groups of operations are applying, composing, and converting.

### Applying Patches

The primary way to use a `JsonPatch` is to call `JsonPatch#apply` or the `Json#patch` extension method, both of which accept an optional `PatchMode` argument.

#### `apply`

Applies this patch to a `Json` value. Returns `Right` with the patched value on success, or `Left[SchemaError]` on failure. The `mode` parameter controls failure handling — see [`PatchMode`](#patchmode):

```scala
case class JsonPatch(ops: Chunk[JsonPatch.JsonPatchOp]) {
  def apply(value: Json, mode: PatchMode = PatchMode.Strict): Either[SchemaError, Json]
}
```

`apply` is also available via the `Json#patch` extension method. Both forms produce the same result:

```scala

val applyJson  = Json.Object("score" -> Json.Number(10))
val applyPatch = JsonPatch.root(Op.ObjectEdit(Chunk(
  ObjectOp.Modify("score", JsonPatch.root(Op.PrimitiveDelta(PrimitiveOp.NumberDelta(BigDecimal(5)))))
)))
```

The direct call and the extension method are equivalent:

```scala
applyPatch.apply(applyJson)
// res13: Either[SchemaError, Json] = Right(
//   Object(IndexedSeq(("score", Number(15))))
// )
applyJson.patch(applyPatch)
// res14: Either[SchemaError, Json] = Right(
//   Object(IndexedSeq(("score", Number(15))))
// )
applyJson.patch(applyPatch, PatchMode.Lenient)
// res15: Either[SchemaError, Json] = Right(
//   Object(IndexedSeq(("score", Number(15))))
// )
```

#### `isEmpty`

Returns `true` if this patch contains no operations:

```scala
case class JsonPatch(ops: Chunk[JsonPatch.JsonPatchOp]) {
  def isEmpty: Boolean
}
```

A patch computed between two identical values also produces an empty patch:

```scala

```

```scala
JsonPatch.empty.isEmpty
// res17: Boolean = true
JsonPatch.diff(Json.Number(1), Json.Number(1)).isEmpty
// res18: Boolean = true
```

### Composing Patches

`++` is the principal operator for building complex patches from smaller, focused ones. The `JsonPatch.empty` value is the identity element for `++`:

```scala
case class JsonPatch(ops: Chunk[JsonPatch.JsonPatchOp]) {
  def ++(that: JsonPatch): JsonPatch
}
```

Concatenating two patches applies `this` first, then `that`. This allows building a single patch that updates multiple fields independently:

```scala

val renamePatch = JsonPatch(
  DynamicOptic.root.field("name"),
  Op.Set(Json.String("Bob"))
)
val incrAgePatch = JsonPatch(
  DynamicOptic.root.field("age"),
  Op.PrimitiveDelta(PrimitiveOp.NumberDelta(BigDecimal(1)))
)
val combinedPatch = renamePatch ++ incrAgePatch
val personJson    = Json.Object("name" -> Json.String("Alice"), "age" -> Json.Number(25))
```

The combined patch applies both operations in sequence:

```scala
combinedPatch.apply(personJson)
// res20: Either[SchemaError, Json] = Right(
//   Object(IndexedSeq(("name", String("Bob")), ("age", Number(26))))
// )
```

### Converting

`toDynamicPatch` converts a `JsonPatch` to a `DynamicPatch`. This is always safe — every JSON operation maps to a corresponding dynamic operation. `NumberDelta` widens to `BigDecimalDelta`:

```scala
case class JsonPatch(ops: Chunk[JsonPatch.JsonPatchOp]) {
  def toDynamicPatch: DynamicPatch
}
```

To convert in the opposite direction, use `JsonPatch.fromDynamicPatch` — see [Creating Patches](#creating-patches) above:

```scala

val patch: JsonPatch  = JsonPatch.diff(Json.Number(1), Json.Number(5))
val dyn: DynamicPatch = patch.toDynamicPatch
```

## PatchMode

`PatchMode` controls how `JsonPatch#apply` reacts when an operation's precondition is not met (e.g., a field is missing, or `ObjectOp.Add` targets a key that already exists):

| Mode | Behaviour |
|------|-----------|
| `PatchMode.Strict` (default) | Returns `Left[SchemaError]` on the first failure |
| `PatchMode.Lenient` | Silently skips failing operations; returns `Right` with partial result |
| `PatchMode.Clobber` | Overwrites on conflicts; forces through missing-field errors where possible |

`ObjectOp.Add` fails in `Strict` mode when the key already exists. In `Lenient` mode the conflicting add is silently skipped; in `Clobber` mode it overwrites the existing value:

```scala

val modeJson  = Json.Object("a" -> Json.Number(1))
val modePatch = JsonPatch.root(Op.ObjectEdit(Chunk(ObjectOp.Add("a", Json.Number(99)))))
```

The three modes produce different outcomes for the same conflicting patch:

```scala
modeJson.patch(modePatch, PatchMode.Strict)
// res23: Either[SchemaError, Json] = Left(
//   SchemaError(
//     List(
//       ExpectationMismatch(
//         source = DynamicOptic(ArraySeq()),
//         expectation = "Key 'a' already exists in object"
//       )
//     )
//   )
// )
modeJson.patch(modePatch, PatchMode.Lenient)
// res24: Either[SchemaError, Json] = Right(
//   Object(IndexedSeq(("a", Number(1))))
// )
modeJson.patch(modePatch, PatchMode.Clobber)
// res25: Either[SchemaError, Json] = Right(
//   Object(IndexedSeq(("a", Number(99))))
// )
```

## Operation Types

A `JsonPatch` is a sequence of `JsonPatchOp` values. Each `JsonPatchOp` pairs a `DynamicOptic` path with an `Op`:

```scala
final case class JsonPatchOp(path: DynamicOptic, operation: Op)
```

The full `Op` hierarchy covers five cases, from full replacement to fine-grained array and object edits:

```
Op (sealed trait)
 ├── Op.Set                — replace the target value entirely
 ├── Op.PrimitiveDelta     — numeric increment or string edit
 │    ├── PrimitiveOp.NumberDelta
 │    └── PrimitiveOp.StringEdit
 │         ├── StringOp.Insert
 │         ├── StringOp.Delete
 │         ├── StringOp.Append
 │         └── StringOp.Modify
 ├── Op.ArrayEdit          — insert / append / delete / modify array elements
 │    ├── ArrayOp.Insert
 │    ├── ArrayOp.Append
 │    ├── ArrayOp.Delete
 │    └── ArrayOp.Modify
 ├── Op.ObjectEdit         — add / remove / modify object fields
 │    ├── ObjectOp.Add
 │    ├── ObjectOp.Remove
 │    └── ObjectOp.Modify
 └── Op.Nested             — groups a sub-patch under a shared path prefix
```

### `Op.Set`

Replaces the target value with a new `Json` value, regardless of the current value. Works on any `Json` type:

```scala
final case class Set(value: Json) extends Op
```

`Op.Set` can replace across types — for example, replacing a number with a string, or resetting a nested field to `null`:

```scala

val setString  = JsonPatch.root(Op.Set(Json.String("replaced")))
val setNull    = JsonPatch(DynamicOptic.root.field("status"), Op.Set(Json.Null))
val withStatus = Json.Object("status" -> Json.String("active"), "id" -> Json.Number(1))
```

Both patches replace their target regardless of its current type:

```scala
setString.apply(Json.Number(123))
// res27: Either[SchemaError, Json] = Right(String("replaced"))
setNull.apply(withStatus)
// res28: Either[SchemaError, Json] = Right(
//   Object(IndexedSeq(("status", null), ("id", Number(1))))
// )
```

### `Op.PrimitiveDelta`

Applies a primitive mutation to a scalar value — either a numeric increment (`NumberDelta`) or a sequence of string edits (`StringEdit`):

```scala
final case class PrimitiveDelta(op: PrimitiveOp) extends Op
```

#### `PrimitiveOp.NumberDelta`

Adds `delta` to a `Json.Number`. Use a negative value to subtract. Fails if the target is not a `Json.Number`:

```scala
final case class NumberDelta(delta: BigDecimal) extends PrimitiveOp
```

Positive deltas increment; negative deltas decrement:

```scala

val inc = JsonPatch.root(Op.PrimitiveDelta(PrimitiveOp.NumberDelta(BigDecimal(5))))
val dec = JsonPatch.root(Op.PrimitiveDelta(PrimitiveOp.NumberDelta(BigDecimal(-3))))
```

```scala
inc.apply(Json.Number(10))
// res30: Either[SchemaError, Json] = Right(Number(15))
dec.apply(Json.Number(10))
// res31: Either[SchemaError, Json] = Right(Number(7))
```

#### `PrimitiveOp.StringEdit`

Applies a sequence of `StringOp` operations to a `Json.String`. `JsonPatch.diff` generates `StringEdit` automatically when it is more compact than a full `Set`:

```scala
final case class StringEdit(ops: Chunk[StringOp]) extends PrimitiveOp
```

The `StringOp` cases:

| Case | Parameters | Effect |
|------|-----------|--------|
| `StringOp.Insert(index, text)` | position, text | Inserts `text` before character `index` |
| `StringOp.Delete(index, length)` | position, count | Removes `length` characters starting at `index` |
| `StringOp.Append(text)` | text | Appends `text` to the end |
| `StringOp.Modify(index, length, text)` | position, count, text | Replaces `length` characters at `index` with `text` |

We can insert a prefix before the first character using `StringOp.Insert`:

```scala

val insertPatch = JsonPatch.root(
  Op.PrimitiveDelta(PrimitiveOp.StringEdit(Chunk(StringOp.Insert(0, "Hello, "))))
)
```

```scala
insertPatch.apply(Json.String("world"))
// res33: Either[SchemaError, Json] = Right(String("Hello, world"))
```

:::tip
For most use cases, let `JsonPatch.diff` generate `StringEdit` automatically. The diff algorithm uses an LCS (Longest Common Subsequence) comparison and only emits `StringEdit` when it produces fewer bytes than a plain `Set`.
:::

### `Op.ArrayEdit`

Applies a sequence of `ArrayOp` operations to a `Json.Array`. Operations are applied in order, and each one sees the result of the previous:

```scala
final case class ArrayEdit(ops: Chunk[ArrayOp]) extends Op
```

The `ArrayOp` cases:

| Case | Parameters | Effect |
|------|-----------|--------|
| `ArrayOp.Insert(index, values)` | position, elements | Inserts `values` before `index` |
| `ArrayOp.Append(values)` | elements | Appends `values` to the end |
| `ArrayOp.Delete(index, count)` | position, count | Removes `count` elements starting at `index` |
| `ArrayOp.Modify(index, op)` | position, op | Applies `op` to the element at `index` |

Multiple `ArrayOp`s in a single `ArrayEdit` can be combined — here we transform `[1, 2, 3]` into `[0, 1, 2, 4]` in one pass:

```scala

val arrayPatch = JsonPatch.root(Op.ArrayEdit(Chunk(
  ArrayOp.Insert(0, Chunk(Json.Number(0))),
  ArrayOp.Delete(3, 1),
  ArrayOp.Append(Chunk(Json.Number(4)))
)))
val originalArr = Json.Array(Json.Number(1), Json.Number(2), Json.Number(3))
```

```scala
arrayPatch.apply(originalArr)
// res35: Either[SchemaError, Json] = Right(
//   Array(IndexedSeq(Number(0), Number(1), Number(2), Number(4)))
// )
```

:::note
Array indices in `ArrayOp.Delete` and `ArrayOp.Modify` refer to the state of the array **after** all preceding ops in the same `ArrayEdit` have been applied.
:::

### `Op.ObjectEdit`

Applies a sequence of `ObjectOp` operations to a `Json.Object`. Operations are applied in order:

```scala
final case class ObjectEdit(ops: Chunk[ObjectOp]) extends Op
```

The `ObjectOp` cases:

| Case | Parameters | Effect |
|------|-----------|--------|
| `ObjectOp.Add(key, value)` | field name, value | Adds a new field; fails in `Strict` mode if key exists |
| `ObjectOp.Remove(key)` | field name | Removes an existing field |
| `ObjectOp.Modify(key, patch)` | field name, sub-patch | Applies `patch` recursively to the field value |

A single `ObjectEdit` can add, remove, and modify fields in one operation:

```scala

val originalObj = Json.Object(
  "name"  -> Json.String("Alice"),
  "age"   -> Json.Number(25),
  "city"  -> Json.String("NYC")
)

val objPatch = JsonPatch.root(Op.ObjectEdit(Chunk(
  ObjectOp.Add("email", Json.String("alice@example.com")),
  ObjectOp.Remove("city"),
  ObjectOp.Modify("age", JsonPatch.root(Op.PrimitiveDelta(PrimitiveOp.NumberDelta(BigDecimal(1)))))
)))
```

```scala
objPatch.apply(originalObj)
// res37: Either[SchemaError, Json] = Right(
//   Object(
//     IndexedSeq(
//       ("name", String("Alice")),
//       ("age", Number(26)),
//       ("email", String("alice@example.com"))
//     )
//   )
// )
```

### `Op.Nested`

Applies a whole sub-patch at the operation's path, with the sub-patch's paths relative to that value and in the same `PatchMode`. It groups operations that share a path prefix, so each `JsonPatchOp` doesn't repeat the full path:

```scala
final case class Nested(patch: JsonPatch) extends Op
```

`JsonPatch.diff` never emits `Nested`; it nests object changes with `ObjectOp.Modify` instead. Build a `Nested` operation by hand when you compose patches for a sub-tree.

## Diffing Algorithm

`JsonPatch.diff` (and its alias `Json#diff`) delegate to [`JsonDiffer.diff`](./json-differ.md), which selects the most compact representation for each type of change:

| Value type | Change | Strategy |
|------------|--------|----------|
| Any | No change | No operation emitted |
| `Json.Number` | Value changed | `NumberDelta` — stores the numeric difference |
| `Json.String` | Value changed | `StringEdit` via LCS if smaller; otherwise `Set` |
| `Json.Array` | Elements changed | `ArrayEdit` with LCS-aligned `Insert`/`Delete`/`Append` (a changed element is deleted and inserted again) |
| `Json.Object` | Fields changed | `ObjectEdit` with recursive per-field diff |
| Any | Type changed | `Set` — full replacement |

:::tip
`JsonPatch.diff` followed by `JsonPatch#apply` is a lossless roundtrip: `JsonPatch.diff(source, target).apply(source) == Right(target)`, as long as neither value has an object with duplicate keys. A duplicate key in `source` survives the patch, and one in `target` makes the patch fail in `Strict` mode (a known bug in 0.0.56).
:::

## JsonPatch vs RFC 6902 JSON Patch

ZIO Blocks' `JsonPatch` is **not** an implementation of [RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902). The two share the same motivation but differ in design:

| | ZIO Blocks `JsonPatch` | RFC 6902 JSON Patch |
|---|---|---|
| Operations | Typed ADT (`Op.Set`, `Op.ArrayEdit`, …) | String-tagged JSON objects (`"op": "replace"`) |
| Paths | `DynamicOptic` (typed, composable) | JSON Pointer strings (`"/a/b/0"`) |
| Number changes | `NumberDelta` (stores diff) | `replace` (stores full new value) |
| String changes | LCS-based `StringEdit` | `replace` only |
| Array changes | LCS-aligned insert/delete | `add`, `remove`, `replace` at absolute indices |
| Serialization | Via ZIO Blocks `Schema` in any format | Always JSON |
| Composition | `++` operator | Array concatenation |

Use `JsonPatch` when working within ZIO Blocks. Its `Schema` JSON is not RFC 6902, and reformatting it is not enough: number deltas and string edits must become `replace` operations whose values come from the source document, and list indices shift as earlier operations apply. [How to Compute, Store and Replay Changes with Diff and Patch](../../../../guides/diff-and-patch.md#convert-a-jsonpatch-to-rfc-6902) shows a converter that does this.

## Advanced Usage

`JsonPatch`'s composability and first-class serializability unlock patterns beyond simple point-in-time updates.

### Building a Change Log

Because `JsonPatch` is a pure value with a `Schema`, we can serialize every change and replay or audit it later:

```scala

// Every mutation is a patch — store it instead of overwriting
val v0 = Json.Object("count" -> Json.Number(0))
val v1 = Json.Object("count" -> Json.Number(1))
val v2 = Json.Object("count" -> Json.Number(2))

val log: List[JsonPatch] = List(
  JsonPatch.diff(v0, v1),
  JsonPatch.diff(v1, v2)
)

// Replay: reconstruct any historical state
val replay = log.foldLeft(v0: Json)((state, patch) => patch.apply(state).getOrElse(state))
assert(replay == v2)
```

### Composing Targeted Sub-Patches

We can build a single patch that updates multiple nested fields by combining focused per-field patches with `++`:

```scala

def setField(field: String, value: Json): JsonPatch =
  JsonPatch(DynamicOptic.root.field(field), Op.Set(value))

val fieldPatch =
  setField("status", Json.String("active")) ++
  setField("updatedAt", Json.String("2025-01-01"))

val doc = Json.Object("status" -> Json.String("draft"), "id" -> Json.Number(42))
```

Applying the composed patch updates both fields in one step:

```scala
fieldPatch.apply(doc)
// res40: Either[SchemaError, Json] = Left(
//   SchemaError(
//     List(
//       MissingField(source = DynamicOptic(ArraySeq()), fieldName = "updatedAt")
//     )
//   )
// )
```

## Integration

`JsonPatch` integrates with `Json`, `DynamicPatch`, and the ZIO Blocks serialization system. Each integration point is covered below.

### With `Json`

`Json` exposes two extension methods as entry points into `JsonPatch`:

```scala

val source = Json.Object("x" -> Json.Number(1))
val target = Json.Object("x" -> Json.Number(2))

val patch: JsonPatch  = source.diff(target)          // compute patch
val result            = source.patch(patch)          // apply (Strict)
val lenient           = source.patch(patch, PatchMode.Lenient)
```

See [Json](./json.md) for the complete `Json` API.

### With `DynamicPatch`

`JsonPatch` and `DynamicPatch` are bidirectionally convertible. This is useful when patches originate from the typed `Patch[S]` system and need to be applied to raw JSON:

```scala

val jsonPatch: JsonPatch                  = JsonPatch.diff(Json.Number(1), Json.Number(3))

// JsonPatch → DynamicPatch (always succeeds)
val dynPatch: DynamicPatch               = jsonPatch.toDynamicPatch

// DynamicPatch → JsonPatch (may fail for temporal ops or non-string keys)
val back: Either[SchemaError, JsonPatch] = JsonPatch.fromDynamicPatch(dynPatch)
```

For the typed `Patch[S]` API, see the Patching documentation.

### Serialization

`JsonPatch` ships with `Schema` instances for all nested operation types, enabling round-trip serialization via any ZIO Blocks codec:

```scala

val schema: Schema[JsonPatch] = implicitly[Schema[JsonPatch]]
```

For codec derivation and usage details, see the Codec & Format documentation.

## Examples

Runnable examples are in `schema-examples/src/main/scala/jsonpatch/`:

| File | Topic |
|------|-------|
| `JsonPatchDiffAndApplyExample.scala` | `JsonPatch.diff`, `Json#diff`, `Json#patch`, roundtrip guarantee |
| `JsonPatchManualBuildExample.scala` | `JsonPatch.root`, path-based patches, `JsonPatch.empty` |
| `JsonPatchOperationsExample.scala` | All `Op` types — `Set`, `NumberDelta`, `StringEdit`, `ArrayEdit`, `ObjectEdit` |
| `JsonPatchCompositionExample.scala` | `++`, `PatchMode`, `toDynamicPatch`, `fromDynamicPatch` |
| `CompleteJsonPatchExample.scala` | Collaborative document editor with a full patch log, replay, and sync |

Run any example with:

```bash
cd schema-examples && sbt "runMain jsonpatch.CompleteJsonPatchExample"
```

---

## JSON Schema

`JsonSchema` provides first-class support for [JSON Schema 2020-12](https://json-schema.org/specification-links.html#2020-12) in ZIO Blocks. It enables parsing, construction, validation, and serialization of JSON Schemas as native Scala values.

## Overview

The `JsonSchema` type is a sealed ADT representing all valid JSON Schema documents:

```
JsonSchema
 ├── JsonSchema.True          (accepts all values - equivalent to {})
 ├── JsonSchema.False         (rejects all values - equivalent to {"not": {}})
 └── JsonSchema.Object        (full schema with all keywords)
```

Key features:

- **Full JSON Schema 2020-12 support** - All standard vocabularies (core, applicator, validation, format, meta-data)
- **Type-safe construction** - Smart constructors and builder pattern
- **Validation** - Validate JSON values against schemas with detailed error messages
- **Round-trip serialization** - Parse from JSON and serialize back without loss
- **Combinators** - Compose schemas with `&&` (allOf), `||` (anyOf), `!` (not)
- **817 of 844 official tests passing** (about 97%)

## Deriving JSON Schema from Schema

The most common use case is deriving a JSON Schema from an existing `Schema[A]`. You can derive directly from the schema or through a `JsonCodec` for more control.

To derive the JSON Schema of `Person` both ways and check two JSON values against it with `JsonSchema#conforms`:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Direct derivation
val jsonSchema: JsonSchema = Schema[Person].toJsonSchema

// Or derive through JsonCodec for more control
val codec = Schema[Person].derive(JsonFormat)
val jsonSchema2 = codec.toJsonSchema

// The derived schema validates JSON values
val valid = Json.Object("name" -> Json.String("Alice"), "age" -> Json.Number(30))
val invalid = Json.Object("name" -> Json.Number(123))

jsonSchema.conforms(valid)   // true
jsonSchema.conforms(invalid) // false
```

## Creating Schemas

To build a schema by hand instead of deriving it, `JsonSchema` provides values and constructors for each kind of schema: boolean, type, string, numeric, array, object, enum and const schemas.

### Boolean Schemas

The simplest schemas accept or reject all values:

```scala

// Accepts any valid JSON value
val acceptAll = JsonSchema.True

// Rejects all JSON values
val rejectAll = JsonSchema.False
```

### Type Schemas

Create schemas that validate specific JSON types:

```scala

// Single type
val stringSchema = JsonSchema.ofType(JsonSchemaType.String)
val numberSchema = JsonSchema.ofType(JsonSchemaType.Number)
val integerSchema = JsonSchema.ofType(JsonSchemaType.Integer)
val booleanSchema = JsonSchema.ofType(JsonSchemaType.Boolean)
val arraySchema = JsonSchema.ofType(JsonSchemaType.Array)
val objectSchema = JsonSchema.ofType(JsonSchemaType.Object)
val nullSchema = JsonSchema.ofType(JsonSchemaType.Null)

// Convenience aliases
val isNull = JsonSchema.nullSchema
val isBoolean = JsonSchema.boolean
```

### String Schemas

Create schemas for string validation:

```scala

// String with length constraints (compile-time validated literals)
val username = JsonSchema.string(
  NonNegativeInt.literal(3),
  NonNegativeInt.literal(20)
)

// String with pattern
val hexColor = JsonSchema.string(
  pattern = RegexPattern.unsafe("^#[0-9a-fA-F]{6}$")
)

// String with format
val email = JsonSchema.string(format = Some("email"))
val dateTime = JsonSchema.string(format = Some("date-time"))
val uuid = JsonSchema.string(format = Some("uuid"))
```

### Numeric Schemas

Create schemas for number validation:

```scala

// Number with range
val percentage = JsonSchema.number(
  minimum = Some(BigDecimal(0)),
  maximum = Some(BigDecimal(100))
)

// Integer with exclusive bounds
val positiveInt = JsonSchema.integer(
  exclusiveMinimum = Some(BigDecimal(0))
)

// Number divisible by a value
val evenNumber = JsonSchema.integer(
  multipleOf = PositiveNumber.fromInt(2)
)
```

### Array Schemas

Create schemas for array validation:

```scala

// Array of strings
val stringArray = JsonSchema.array(
  items = Some(JsonSchema.ofType(JsonSchemaType.String))
)

// Array with length constraints
val shortList = JsonSchema.array(
  JsonSchema.ofType(JsonSchemaType.Number),
  NonNegativeInt.literal(1),
  NonNegativeInt.literal(5)
)

// Array with unique items
val uniqueNumbers = JsonSchema.array(
  items = Some(JsonSchema.ofType(JsonSchemaType.Number)),
  uniqueItems = Some(true)
)

// Tuple-like array with prefixItems
val point2D = JsonSchema.array(
  prefixItems = Some(NonEmptyChunk(
    JsonSchema.ofType(JsonSchemaType.Number),
    JsonSchema.ofType(JsonSchemaType.Number)
  ))
)
```

### Object Schemas

Create schemas for object validation:

```scala

// Object with properties and constraints
val person = JsonSchema.obj(
  properties = Some(ChunkMap(
    "name" -> JsonSchema.ofType(JsonSchemaType.String),
    "age" -> JsonSchema.ofType(JsonSchemaType.Integer)
  )),
  required = Some(Set("name")),
  // Reject properties not defined above
  additionalProperties = Some(JsonSchema.False)
)
```

### Enum and Const

To restrict a value to a fixed set of strings, a fixed set of mixed JSON values, or a single constant value:

```scala

// Enum of string values
val status = JsonSchema.enumOfStrings(NonEmptyChunk("pending", "active", "completed"))

// Enum of mixed values
val mixed = JsonSchema.enumOf(NonEmptyChunk(
  Json.String("auto"),
  Json.Number(0),
  Json.Boolean(true)
))

// Constant value
val alwaysTrue = JsonSchema.constOf(Json.Boolean(true))
```

## Schema Combinators

Combinators build a new schema from existing ones, either by logical composition or by making a schema also accept `null`.

### Logical Composition

Combine schemas using logical operators:

```scala

val stringSchema = JsonSchema.ofType(JsonSchemaType.String)
val numberSchema = JsonSchema.ofType(JsonSchemaType.Number)
val nullSchema = JsonSchema.ofType(JsonSchemaType.Null)

// allOf - must match all schemas
val stringAndNotEmpty = stringSchema && JsonSchema.string(
  minLength = Some(zio.blocks.schema.json.NonNegativeInt.literal(1))
)

// anyOf - must match at least one schema
val stringOrNumber = stringSchema || numberSchema

// not - must not match the schema
val notNull = !nullSchema

// Combining operators
val nullableString = stringSchema || nullSchema
```

### Nullable Schemas

Make any schema nullable:

```scala

val stringSchema = JsonSchema.ofType(JsonSchemaType.String)

// Accepts string or null
val nullableString = stringSchema.withNullable
```

## Conditional Schemas

Conditional schemas apply a subschema only when a condition holds: `if`/`then`/`else` tests the value against a schema, and dependent schemas test which properties are present.

### if/then/else

Apply different schemas based on conditions:

```scala

// If type is string, require minLength
val conditionalSchema = JsonSchema.Object(
  `if` = Some(JsonSchema.ofType(JsonSchemaType.String)),
  `then` = Some(JsonSchema.string(minLength = Some(NonNegativeInt.literal(1)))),
  `else` = Some(JsonSchema.True)
)
```

### Dependent Schemas

Apply schemas when properties are present:

```scala

// If "credit_card" exists, require "billing_address"
val paymentSchema = JsonSchema.Object(
  properties = Some(ChunkMap(
    "credit_card" -> JsonSchema.ofType(JsonSchemaType.String),
    "billing_address" -> JsonSchema.ofType(JsonSchemaType.String)
  )),
  dependentRequired = Some(ChunkMap(
    "credit_card" -> Set("billing_address")
  ))
)
```

## Validation

A `JsonSchema` validates `Json` values directly: `JsonSchema#check` returns the error, if any, and `JsonSchema#conforms` returns a `Boolean`.

### Basic Validation

To validate `Json` values with `JsonSchema#check` and `JsonSchema#conforms`, and to turn format checking on or off with `ValidationOptions`:

```scala

val schema = JsonSchema.obj(
  properties = Some(ChunkMap(
    "name" -> JsonSchema.ofType(JsonSchemaType.String),
    "age" -> JsonSchema.integer(minimum = Some(BigDecimal(0)))
  )),
  required = Some(Set("name"))
)

val validJson = Json.Object(
  "name" -> Json.String("Alice"),
  "age" -> Json.Number(30)
)

val invalidJson = Json.Object(
  "age" -> Json.Number(-5)
)

// Using JsonSchema#check - returns Option[SchemaError]
schema.check(validJson)   // None (valid)
schema.check(invalidJson) // Some(SchemaError(...))

// Using JsonSchema#conforms - returns Boolean
schema.conforms(validJson)   // true
schema.conforms(invalidJson) // false

// Control validation behavior with ValidationOptions
val emailSchema = JsonSchema.string(format = Some("email"))
val invalidEmail = Json.String("not-an-email")

// With format validation (default)
emailSchema.check(invalidEmail, ValidationOptions.formatAssertion) // Some(error)

// Without format validation (format as annotation only)
emailSchema.check(invalidEmail, ValidationOptions.annotationOnly)  // None
```

### Error Messages

Validation errors include path information:

```scala

val schema = JsonSchema.obj(
  properties = Some(ChunkMap(
    "users" -> JsonSchema.array(
      items = Some(JsonSchema.obj(
        properties = Some(ChunkMap(
          "email" -> JsonSchema.string(format = Some("email"))
        ))
      ))
    )
  ))
)
// schema: JsonSchema = Object(
//   $id = None,
//   $schema = None,
//   $anchor = None,
//   $dynamicAnchor = None,
//   $ref = None,
//   $dynamicRef = None,
//   $vocabulary = None,
//   $defs = None,
//   $comment = None,
//   allOf = None,
//   anyOf = None,
//   oneOf = None,
//   not = None,
//   if = None,
//   then = None,
//   else = None,
//   properties = Some(
//     Map(
//       "users" -> Object(
//         $id = None,
//         $schema = None,
//         $anchor = None,
//         $dynamicAnchor = None,
//         $ref = None,
//         $dynamicRef = None,
//         $vocabulary = None,
//         $defs = None,
//         $comment = None,
//         allOf = None,
//         anyOf = None,
//         oneOf = None,
//         not = None,
//         if = None,
//         then = None,
//         else = None,
//         properties = None,
//         patternProperties = None,
//         additionalProperties = None,
//         propertyNames = None,
//         dependentSchemas = None,
//         prefixItems = None,
//         items = Some(
//           Object(
//             $id = None,
//             $schema = None,
//             $anchor = None,
//             $dynamicAnchor = None,
//             $ref = None,
// ...

val invalid = Json.Object(
  "users" -> Json.Array(
    Json.Object("email" -> Json.String("invalid"))
  )
)
// invalid: Object = Object(
//   IndexedSeq(
//     (
//       "users",
//       Array(IndexedSeq(Object(IndexedSeq(("email", String("invalid"))))))
//     )
//   )
// )

schema.check(invalid) match {
  case Some(err) => println(err.message)
  case None => println("Valid")
}
// String 'invalid' is not a valid email address at: .users.at(0).email
```

## Parsing and Serialization

A `JsonSchema` converts to and from JSON, so a schema can be read from an existing document and written out for other tools.

### Parsing from JSON

To parse a schema from a JSON string with `JsonSchema.parse`, or from a `Json` value with `JsonSchema.fromJson`:

```scala

// From JSON string
val parsed = JsonSchema.parse("""
  {
    "type": "object",
    "properties": {
      "name": { "type": "string" }
    },
    "required": ["name"]
  }
""")

// From Json value
val json = Json.Object(
  "type" -> Json.String("string"),
  "minLength" -> Json.Number(1)
)
val fromJson = JsonSchema.fromJson(json)
```

### Serializing to JSON

To turn a schema back into a `Json` value with `JsonSchema#toJson` and print it as a string:

```scala

val schema = JsonSchema.string(
  NonNegativeInt.literal(1),
  NonNegativeInt.literal(100)
)

val json = schema.toJson
// {"type":"string","minLength":1,"maxLength":100}

val jsonString = json.print
```

## Format Validation

The following formats are supported for validation:

| Format          | Description            | Example                                |
|-----------------|------------------------|----------------------------------------|
| `date-time`     | RFC 3339 date-time     | `2024-01-15T10:30:00Z`                 |
| `date`          | RFC 3339 full-date     | `2024-01-15`                           |
| `time`          | RFC 3339 full-time     | `10:30:00Z`                            |
| `email`         | Email address          | `user@example.com`                     |
| `uuid`          | RFC 4122 UUID          | `550e8400-e29b-41d4-a716-446655440000` |
| `uri`           | RFC 3986 URI           | `https://example.com/path`             |
| `uri-reference` | RFC 3986 URI-reference | `/path/to/resource`                    |
| `ipv4`          | IPv4 address           | `192.168.1.1`                          |
| `ipv6`          | IPv6 address           | `2001:db8::1`                          |
| `hostname`      | RFC 1123 hostname      | `example.com`                          |
| `regex`         | ECMA-262 regex         | `^[a-z]+$`                             |
| `duration`      | ISO 8601 duration      | `P3Y6M4DT12H30M5S`                     |
| `json-pointer`  | RFC 6901 JSON Pointer  | `/foo/bar/0`                           |

Format validation is enabled by default. Use `ValidationOptions.annotationOnly` to treat `format` as annotation only (per JSON Schema spec).

## Unevaluated Properties and Items

JSON Schema 2020-12 introduces `unevaluatedProperties` and `unevaluatedItems` for validating properties/items not matched by any applicator keyword:

```scala

// Reject any properties not defined in properties or patternProperties
val strictObject = JsonSchema.Object(
  properties = Some(ChunkMap(
    "name" -> JsonSchema.ofType(JsonSchemaType.String)
  )),
  unevaluatedProperties = Some(JsonSchema.False)
)

// Reject extra array items not matched by prefixItems or items
val strictArray = JsonSchema.Object(
  prefixItems = Some(NonEmptyChunk(
    JsonSchema.ofType(JsonSchemaType.String),
    JsonSchema.ofType(JsonSchemaType.Number)
  )),
  unevaluatedItems = Some(JsonSchema.False)
)
```

## Schema Object Fields

`JsonSchema.Object` supports all JSON Schema 2020-12 keywords:

### Core Vocabulary
- `$id`, `$schema`, `$anchor`, `$dynamicAnchor`
- `$ref`, `$dynamicRef` (limited support - see Limitations)
- `$defs`, `$comment`

### Applicator Vocabulary
- `allOf`, `anyOf`, `oneOf`, `not`
- `if`, `then`, `else`
- `properties`, `patternProperties`, `additionalProperties`
- `propertyNames`, `dependentSchemas`
- `prefixItems`, `items`, `contains`

### Unevaluated Vocabulary
- `unevaluatedProperties`, `unevaluatedItems`

### Validation Vocabulary
- `type`, `enum`, `const`
- `multipleOf`, `minimum`, `maximum`, `exclusiveMinimum`, `exclusiveMaximum`
- `minLength`, `maxLength`, `pattern`
- `minItems`, `maxItems`, `uniqueItems`, `minContains`, `maxContains`
- `minProperties`, `maxProperties`, `required`, `dependentRequired`

JSON Schema applies `pattern` anywhere in the string, so `"[0-9]{3}"` matches `"a123b"`. A known bug in 0.0.56: `JsonSchema#check` enforces `pattern` only when it starts with `^` and ends with `$`, and it ignores any other pattern. Write the pattern with both anchors, such as `"^[0-9]{3}$"`, until the bug is fixed, so that the check and the JSON Schema agree.

### Format Vocabulary
- `format`

### Content Vocabulary
- `contentEncoding`, `contentMediaType`, `contentSchema`

### Meta-Data Vocabulary
- `title`, `description`, `default`, `deprecated`
- `readOnly`, `writeOnly`, `examples`

## Limitations

`JsonSchema` does not implement every part of JSON Schema 2020-12; the subsections below list the unsupported features, the known edge cases and the official test-suite results.

### Not Implemented

The following features require reference resolution and are **not supported**:

| Feature                          | Description                               |
|----------------------------------|-------------------------------------------|
| `$ref` to external URIs          | References to other files or URLs         |
| `$dynamicRef` / `$dynamicAnchor` | Dynamic reference resolution              |
| `$id` resolution                 | Base URI changing and resolution          |
| Remote references                | Fetching schemas from URLs                |
| Recursive schemas via `$ref`     | Self-referential schemas using references |

Local `$ref` within the same schema is partially supported for `#/$defs/...` references only.

### Known Edge Cases

| Case                                    | Behavior                                                |
|-----------------------------------------|---------------------------------------------------------|
| Float/integer numeric equality          | `1.0` is not treated as equal to `1` for `const`/`enum` |
| String length                           | Measured in codepoints, not grapheme clusters           |
| Some `unevaluatedItems` with `contains` | Edge cases involving item evaluation tracking           |

### Test Suite Compliance

The implementation passes **817 of 844 tests** (about 97%) from the official JSON Schema Test Suite for draft2020-12. The remaining tests require reference resolution features listed above.

## Complete Example

To combine string formats, numeric ranges, an enum-valued array and a closed object in one user schema, then validate two users against it and serialize it:

```scala

// Define a complex schema
val userSchema = JsonSchema.obj(
  properties = Some(ChunkMap(
    "id" -> JsonSchema.string(format = Some("uuid")),
    "email" -> JsonSchema.string(format = Some("email")),
    "name" -> JsonSchema.string(
      NonNegativeInt.literal(1),
      NonNegativeInt.literal(100)
    ),
    "age" -> JsonSchema.integer(
      minimum = Some(BigDecimal(0)),
      maximum = Some(BigDecimal(150))
    ),
    "roles" -> JsonSchema.array(
      items = Some(JsonSchema.enumOfStrings(
        NonEmptyChunk("admin", "user", "guest")
      )),
      minItems = Some(NonNegativeInt.literal(1)),
      uniqueItems = Some(true)
    ),
    "metadata" -> JsonSchema.obj(
      additionalProperties = Some(JsonSchema.ofType(JsonSchemaType.String))
    )
  )),
  required = Some(Set("id", "email", "name", "roles")),
  additionalProperties = Some(JsonSchema.False)
)

// Validate some data
val validUser = Json.Object(
  "id" -> Json.String("550e8400-e29b-41d4-a716-446655440000"),
  "email" -> Json.String("alice@example.com"),
  "name" -> Json.String("Alice"),
  "roles" -> Json.Array(Json.String("admin"), Json.String("user"))
)

val invalidUser = Json.Object(
  "id" -> Json.String("not-a-uuid"),
  "email" -> Json.String("invalid-email"),
  "name" -> Json.String(""),
  "roles" -> Json.Array(),
  "extra" -> Json.String("not allowed")
)

userSchema.conforms(validUser)   // true
userSchema.conforms(invalidUser) // false

// Get detailed errors
userSchema.check(invalidUser) match {
  case Some(err) => println(err.message)
  case None => println("Valid!")
}

// Serialize the schema
val schemaJson = userSchema.toJson.print
// Can be sent to other tools, stored, or shared
```

## Cross-Platform Support

`JsonSchema` works on the JVM with full functionality and on Scala.js in both the browser and Node.js. All features work identically across platforms.

## See Also

- [How to Generate JSON Schema from a Type and Validate Payloads Against It](../../../../guides/generate-json-schema-and-validate-payloads.md) - a step-by-step guide to deriving a JSON Schema from a `Schema` and validating payloads against it

---

## JsonSelection

`JsonSelection` is a fluent wrapper type that enables composable, chainable navigation through JSON structures. It wraps `Either[SchemaError, Chunk[Json]]`, allowing operations that may fail gracefully or return multiple values.

## Overview

`JsonSelection` makes it easy to navigate unknown or deeply nested JSON at runtime without needing to match on `Either` at each step. Chain operations like `JsonSelection#get`, `JsonSelection#apply`, `JsonSelection#filter`, and `JsonSelection#as` to build powerful queries.

**Key characteristics:**
- **Fluent chaining:** Operations chain naturally without unwrapping intermediate results
- **Multi-value support:** Can contain zero, one, or many `Json` values
- **Error propagation:** Errors short-circuit further operations; querying an error selection returns the same error
- **Type extraction:** Use `JsonSelection#as` to decode to Scala types with Schema-based derivation

## Creating JsonSelection

You can create `JsonSelection` instances in multiple ways: from existing `Json` values, or using companion object constructors. Each approach is useful for different scenarios—direct navigation for existing JSON, and explicit construction for building selections programmatically.

### From Json Values

Create a `JsonSelection` directly from a `Json` value by calling navigation methods:

```scala

val json = Json.parseUnsafe("""{"name": "Alice"}""")

// Get a single field  
val name: JsonSelection = json.get("name")

// Get array element by index
val values = Json.parseUnsafe("[1, 2, 3]")
val selected: JsonSelection = values.get(0)
```

### From Companion Object

Construct `JsonSelection` instances programmatically using the companion object methods for empty, successful, and failed selections:

```scala

// Empty successful selection
val empty = JsonSelection.empty

// Succeed with a single value
val success = JsonSelection.succeed(Json.String("hello"))

// Succeed with multiple values
val many = JsonSelection.succeedMany(
  zio.blocks.chunk.Chunk.from(Seq(Json.Number(1), Json.Number(2)))
)

// Fail with an error
val failure = JsonSelection.fail(SchemaError("not found"))
```

## Navigation Operations

Navigate through JSON structures using three complementary approaches: field access for object properties, array indexing for elements, and path expressions for complex nested navigation.

### Field Navigation

Navigate to object fields using `JsonSelection#get` with a field name:

```scala

val data = Json.parseUnsafe("""{
  "user": {
    "name": "Bob",
    "email": "bob@example.com"
  }
}""")

// Single field access
val user = data.get("user")

// Chained field access
val name = data.get("user").get("name")

// Multiple levels of nesting
val email = data.get("user").get("email")
```

### Array Navigation

Access array elements by index with `Json#get` on a `Json` value, or with `JsonSelection#apply` on a selection:

```scala

val data = Json.parseUnsafe("""[
  {"id": 1, "name": "Alice"},
  {"id": 2, "name": "Bob"}
]""")

// Index access (0-based)
val first = data.get(0)

// Chained index and field access
val firstName = data.get(0).get("name")
val secondId = data.get(1).get("id")
```

### Path-Based Navigation

Use path interpolators (e.g., `p".company.employees[0].name"`) to navigate deeply nested structures in a single operation:

```scala

val data = Json.Object(
  "company" -> Json.Object(
    "employees" -> Json.Array(
      Json.Object("name" -> Json.String("Alice"), "department" -> Json.String("Engineering")),
      Json.Object("name" -> Json.String("Bob"), "department" -> Json.String("Sales"))
    )
  )
)

// Navigate using path interpolator
val path = p".company.employees[0].name"
val firstEmpName = data.get(path)  // JsonSelection(Right(Chunk(Json.String("Alice"))))
```

## Filtering and Querying

Reduce selections to only the values you need by filtering by JSON type or custom predicates. This enables working with heterogeneous JSON arrays where values may be of different types.

### Filter by Type

Keep only values of a specific JSON type using type-filtering methods like `JsonSelection#strings`, `JsonSelection#numbers`, and `JsonSelection#booleans`:

```scala

val mixed = Json.parseUnsafe("""["text", 42, true, "more text"]""")

// Create a selection containing all array elements, then filter by type
val allElements = JsonSelection.succeedMany(mixed.elements)
val strings = allElements.strings    // JsonSelection with string values
val numbers = allElements.numbers    // JsonSelection with number values
val booleans = allElements.booleans  // JsonSelection with boolean values
```

### Filter with Predicate

Use custom predicates with `JsonSelection#filter` to keep only values that match specific conditions:

```scala

val data = Json.parseUnsafe("[1, 2, 3, 4]")

// Create selection and filter with predicate
val allElements = JsonSelection.succeedMany(data.elements)
val evenOnly = allElements.filter { json =>
  json match {
    case Json.Number(n) => n.toInt % 2 == 0
    case _ => false
  }
}
```

## Extracting Values

Extract concrete values from selections by decoding them to Scala types, checking single vs. multiple values, or inspecting selection state without extraction.

### Type Decoding

Decode a single selected value to a Scala type using `JsonSelection#as`, which fails if more than one value is selected:

```scala

// Decode to Scala types
val selection = Json.parseUnsafe("""{"count": 42}""")

val count: Either[SchemaError, Int] = selection.get("count").as[Int]
val str: Either[SchemaError, String] = selection.get("count").as[String]  // Left (type mismatch)
```

### Multiple Decoding

Decode all selected values to a collection using `JsonSelection#asAll`, which succeeds even if the selection is empty:

```scala

val data = Json.parseUnsafe("""["Alice", "Bob", "Charlie"]""")

// Create selection of all array elements and decode
val allElements = JsonSelection.succeedMany(data.elements)
val names: Either[SchemaError, Seq[String]] = allElements.asAll[String].map(_.toSeq)
```

### Extract Single Value

Use `JsonSelection#one` to extract exactly one value, failing if the selection contains zero or more than one value:

```scala

val arr = Json.parseUnsafe("[1, 2]")

// Get exactly one value from a selection (fails if 0 or more than 1)
val single = arr.get(0).one    // Right(Json.Number(1))
val multiple = JsonSelection.succeedMany(arr.elements).one  // Left(SchemaError(...))
```

### Check and Get

Inspect selection state without extraction using properties like `JsonSelection#values`, `JsonSelection#error`, `JsonSelection#isSuccess`, and `JsonSelection#isFailure`:

```scala

val data = Json.Object("x" -> Json.Number(1))

// Safe option extraction
val x: Option[Json] = data.get("x").values.flatMap(_.headOption)

// Check if selection succeeded
val success = data.get("x").isSuccess          // true: field "x" exists
val failed = data.get("notFound").isFailure   // true if field not found
```

## Size and Existence Checks

Query selection size and emptiness using `JsonSelection#size`, `JsonSelection#isEmpty`, and `JsonSelection#nonEmpty`:

```scala

val data = Json.parseUnsafe("[1, 2, 3]")

// Create selection and check size
val selection = JsonSelection.succeedMany(data.elements)
val size = selection.size        // 3
val isEmpty = selection.isEmpty  // false
val nonEmpty = selection.nonEmpty  // true
```

## Modifying Values

To update the value at a path, call `Json#set` to replace it or `Json#modify` to transform it; both return a new `Json`:

```scala

val original = Json.Object("count" -> Json.Number(0))

// Set a new value
val updated = original.set(p".count", Json.Number(1))

// Modify with a function
val modified = original.modify(p".count") {
  case Json.Number(n) => Json.Number(n + 1)
  case other => other
}
```

## Error Handling

Selections propagate errors through the chain:

```scala

val data = Json.Object("name" -> Json.String("Alice"))

// Missing field returns error
val missing = data.get("age").error  // Some(zio.blocks.schema.SchemaError: Key 'age' not found)

// Chain continues with error
val stillMissing = data.get("age").get("nested")  // Still carries the error

// Check error state
val result = data.get("x")
val maybeError: Option[SchemaError] = result.error
val maybeValues = result.values      // None if error
```

## Integration with Codecs

`JsonSelection` works with `JsonCodec` and `Schema`:

```scala

case class User(name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val users = Json.parseUnsafe("""[
  {"name": "Alice", "email": "alice@example.com"},
  {"name": "Bob", "email": "bob@example.com"}
]""")

// Navigate and decode in one chain
val firstUser: Either[SchemaError, User] = users.get(0).as[User]
val allUsers: Either[SchemaError, Chunk[User]] = JsonSelection.succeedMany(users.elements).asAll[User]
```

## Performance Notes

- **Value class wrapper:** `JsonSelection` extends `AnyVal`, so wrapping the underlying `Either` adds no extra object in most uses
- **Eager chaining:** Each step builds a new `Chunk` of results, so intermediate selections are allocated
- **Error short-circuiting:** Failed selections don't execute further operations
- **Selective navigation:** For large JSON, call `JsonSelection#get` to reach only the values you need rather than iterating over every element

## See Also

- [How to Read and Modify JSON Without Case Classes](../../../../guides/json-without-case-classes.md) - A task-oriented guide to navigating and changing JSON by path, then decoding only the part you need

---

## Json

`Json` is a type-safe, schema-free representation of JSON values that enables navigation, transformation, merging, and querying without losing fidelity.

## Overview

The `Json` type represents all valid JSON values with six cases:

```
Json
 ├── Json.Object   (key-value pairs, order-preserving)
 ├── Json.Array    (ordered sequence of values)
 ├── Json.String   (text)
 ├── Json.Number   (a BigDecimal)
 ├── Json.Boolean  (true/false)
 └── Json.Null     (null)
```

Key design decisions:

- **Objects use `Chunk[(String, Json)]`** to preserve insertion order while providing order-independent equality
- **Numbers use `BigDecimal`** to keep decimal digits that a `Double` would lose. Parsing rounds a number to 34 significant digits (`MathContext.DECIMAL128`) and rejects one with 308 or more digits, so a longer number does not keep every digit
- **All navigation returns `JsonSelection`** for fluent, composable chaining

## Creating JSON Values

A `Json` value comes from one of three sources: the case constructors, `Json.parse` on a string, or the `json"..."` interpolator.

### Using Constructors

`Json.Object` takes key-value pairs, `Json.Array` takes its elements, and `Json.String`, `Json.Number`, and `Json.Boolean` each wrap one Scala value:

```scala

// Object with named fields
val person = Json.Object(
  "name" -> Json.String("Alice"),
  "age" -> Json.Number(30),
  "active" -> Json.Boolean(true)
)

// Array of values
val numbers = Json.Array(Json.Number(1), Json.Number(2), Json.Number(3))

// Primitive values
val name = Json.String("Bob")
val count = Json.Number(42)
val flag = Json.Boolean(false)
val nothing = Json.Null
```

### Parsing JSON Strings

`Json.parse` returns a parse failure as a `SchemaError` in a `Left`, while `Json.parseUnsafe` throws it:

```scala

// Safe parsing (returns Either)
val parsed: Either[SchemaError, Json] = Json.parse("""{"name": "Alice", "age": 30}""")

// Unsafe parsing (throws on error)
val json = Json.parseUnsafe("""{"items": [1, 2, 3]}""")
```

### String Interpolators

ZIO Blocks provides compile-time validated string interpolators for JSON:

```scala

// JSON literal with compile-time validation
val person = json"""{"name": "Alice", "age": 30}"""

// With Scala value interpolation
val name = "Bob"
val age = 25
val person2 = json"""{"name": $name, "age": $age}"""

// Path interpolator for navigation
val path = p".users[0].name"
```

The `json"..."` interpolator validates JSON syntax at compile time, catching errors before runtime. A known bug in 0.0.56: on Scala 2.13, the interpolator does not compile in user code outside the `zio.blocks.schema` package, so the examples above fail there unless the code is inside that package. Use `Json.parseUnsafe` on Scala 2.13 until the bug is fixed.

## Type Testing and Access

Once you hold a `Json`, you can check which case it is with `Json#is`, narrow it with `Json#as`, and read an object's fields or an array's elements without pattern matching.

### Unified Type Operations

The `Json` type provides unified methods for type testing and narrowing with path-dependent return types.
`JsonType` also implements `Json => Boolean`, so it can be used directly as a predicate for filtering.

The example tests `json` with `Json#is`, narrows it with `Json#as`, extracts Scala values with `Json#unwrap`, and passes `JsonType.String` to `JsonSelection#query` as a predicate:

```scala

val json: Json = Json.parseUnsafe("""{"count": 42}""")

// Type testing with is()
json.is(JsonType.Object)  // true
json.is(JsonType.Array)   // false

// Type narrowing with as() - returns Option[jsonType.Type]
val obj: Option[Json.Object] = json.as(JsonType.Object)  // Some(Json.Object(...))
val arr: Option[Json.Array] = json.as(JsonType.Array)    // None

// Value extraction with unwrap() - returns Option[jsonType.Unwrap]
val str: Json = Json.String("hello")
val strValue: Option[String] = str.unwrap(JsonType.String)  // Some("hello")

val num: Json = Json.Number(42)
val numValue: Option[BigDecimal] = num.unwrap(JsonType.Number)  // Some(42)

// JsonType as predicate - use directly in selection query
val strings = json.select.query(JsonType.String)  // all string values in the JSON tree
```

### Direct Value Access

`Json#fields` returns an object's key-value pairs and `Json#elements` returns an array's values, each as a `Chunk` (empty for any other case):

```scala

val obj = Json.Object("a" -> Json.Number(1))
obj.fields  // Chunk(("a", Json.Number(1)))

val arr = Json.Array(Json.Number(1), Json.Number(2))
arr.elements  // Chunk(Json.Number(1), Json.Number(2))
```

## Navigation

Navigation reaches into nested objects and arrays, either one step at a time with `Json#get` and an index, or in one call with a `DynamicOptic` path; both return a `JsonSelection`.

### Simple Navigation

Navigate into objects by key and arrays by index:

```scala

val json = Json.parseUnsafe("""{
  "users": [
    {"name": "Alice", "age": 30},
    {"name": "Bob", "age": 25}
  ]
}""")

// Navigate to a field
val users = json.get("users")  // JsonSelection

// Navigate to an array element
val firstUser = json.get("users")(0)  // JsonSelection

// Chain navigation
val firstName = json.get("users")(0).get("name")  // JsonSelection

// Extract the value
val name: Either[SchemaError, String] = firstName.as[String]  // Right("Alice")
```

### Path-Based Navigation with DynamicOptic

Use `DynamicOptic` paths for complex navigation:

```scala

val json = Json.parseUnsafe("""{
  "company": {
    "employees": [
      {"name": "Alice", "department": "Engineering"},
      {"name": "Bob", "department": "Sales"}
    ]
  }
}""")

// Using path interpolator
val path = p".company.employees[0].name"
val name = json.get(path).as[String]  // Right("Alice")

// Equivalent to chained navigation
val sameName = json.get("company").get("employees")(0).get("name").as[String]
```

## JsonSelection

`JsonSelection` is a fluent wrapper for navigation results, enabling composable chaining:

```scala

val json = Json.parseUnsafe("""{"users": [{"name": "Alice"}]}""")

// Fluent chaining
val result: JsonSelection = json
 .get("users")
 .arrays
 .apply(0)
 .get("name")
 .strings

// Extract values
result.as[String]  // Right("Alice")
result.one         // Right(Json.String("Alice"))
result.isSuccess   // true
result.isFailure   // false
```

### Terminal Operations

```scala

val json = Json.parseUnsafe("""{"name": "Alice", "age": 30}""")
val selection: JsonSelection = json.get("name")

// Get single value (exactly one required)
val oneValue: Either[SchemaError, Json] = selection.one
// Get any single value (first of many)
val anyValue: Either[SchemaError, Json] = selection.any
// Get all values condensed (wraps multiple in array)
val allValues: Either[SchemaError, Json] = selection.all

// Get underlying result
val underlying: Option[zio.blocks.chunk.Chunk[Json]] = selection.values
val asChunk: zio.blocks.chunk.Chunk[Json] = selection.toChunk  // empty on error

// Decode to specific types
val asString: Either[SchemaError, String] = selection.as[String]
val asBigDecimal: Either[SchemaError, BigDecimal] = selection.as[BigDecimal]
val asBoolean: Either[SchemaError, Boolean] = selection.as[Boolean]
val asInt: Either[SchemaError, Int] = selection.as[Int]
val asLong: Either[SchemaError, Long] = selection.as[Long]
val asDouble: Either[SchemaError, Double] = selection.as[Double]
```

## Modification

`Json` is immutable, so `Json#set`, `Json#modify`, `Json#delete`, and `Json#insert` each return a new `Json` with the change at a `DynamicOptic` path; their `OrFail` variants return a `SchemaError` in a `Left` when the change cannot be made at that path.

### Setting Values

```scala

val json = Json.parseUnsafe("""{"user": {"name": "Alice", "age": 30}}""")

// Set a value at a path
val updated = json.set(p".user.name", Json.String("Bob"))
// {"user": {"name": "Bob", "age": 30}}

// Set with failure handling
val result = json.setOrFail(p".user.email", Json.String("alice@example.com"))
// Left(SchemaError) - path doesn't exist
```

### Modifying Values

```scala

val json = Json.parseUnsafe("""{"count": 10}""")

// Modify with a function
val incremented = json.modify(p".count") {
  case Json.Number(n) => Json.Number(n + 1)
  case other => other
}
// {"count": 11}

// Modify with failure on missing path
val result = json.modifyOrFail(p".count") {
  case Json.Number(n) => Json.Number(n * 2)
}
// Right({"count": 20})
```

### Deleting Values

```scala

val json = Json.parseUnsafe("""{"a": 1, "b": 2, "c": 3}""")

// Delete a field
val withoutB = json.delete(p".b")
// {"a": 1, "c": 3}

// Delete with failure handling
val result = json.deleteOrFail(p".missing")
// Left(SchemaError) - path doesn't exist
```

### Inserting Values

```scala

val json = Json.parseUnsafe("""{"existing": 1}""")

// Insert a new field
val withNew = json.insert(p".newField", Json.String("value"))
// {"existing": 1, "newField": "value"}
```

## Transformation

The transform methods rewrite the whole tree with a function that receives each value's path; `Json#transformUp` and `Json#transformDown` differ in whether children or parents are visited first, and `Json#transformKeys` rewrites object keys instead of values.

### Transform Up (Bottom-Up)

Transform children before parents:

```scala

val json = Json.parseUnsafe("""{"values": [1, 2, 3]}""")

// Double all numbers
val doubled = json.transformUp { (path, value) =>
  value match {
    case Json.Number(n) => Json.Number(n * 2)
    case other => other
  }
}
// {"values": [2, 4, 6]}
```

### Transform Down (Top-Down)

Transform parents before children:

```scala

val json = Json.parseUnsafe("""{"items": [{"x": 1}, {"x": 2}]}""")

// Add a field to all objects
val withId = json.transformDown { (path, value) =>
  value match {
    case Json.Object(fields) if !fields.exists(_._1 == "id") =>
      new Json.Object(("id" -> Json.String(path.toString)) +: fields)
    case other => other
  }
}
```

### Transform Keys

Rename object keys throughout the structure:

```scala

val json = Json.parseUnsafe("""{"user_name": "Alice", "user_age": 30}""")

// Convert snake_case to camelCase
val camelCase = json.transformKeys { (path, key) =>
  key.split("_").zipWithIndex.map {
    case (word, 0) => word
    case (word, _) => word.capitalize
  }.mkString
}
// {"userName": "Alice", "userAge": 30}
```

## Filtering

Filtering keeps or removes parts of a `Json` by predicate or by path, and returns a new `Json`.

### Filter Values

Keep only values matching a predicate using `Json#retain`, or remove values using `Json#prune`:

```scala

val json = Json.parseUnsafe("""{"a": 1, "b": null, "c": 2, "d": null}""")

// Remove nulls using prune (removes values matching predicate)
val noNulls = json.prune(_.is(JsonType.Null))
// {"a": 1, "c": 2}

// Keep only numbers using retain (keeps values matching predicate)
val onlyNumbers = json.retain(_.is(JsonType.Number))
// {"a": 1, "c": 2}
```

### Project Paths

Extract only specific paths:

```scala

val json = Json.parseUnsafe("""{
  "user": {"name": "Alice", "email": "alice@example.com", "password": "secret"},
  "metadata": {"created": "2024-01-01"}
}""")

// Keep only specific fields
val projected = json.project(p".user.name", p".user.email")
// {"user": {"name": "Alice", "email": "alice@example.com"}}
```

### Partition

Split based on a predicate:

```scala

val json = Json.parseUnsafe("""{"a": 1, "b": "text", "c": 2}""")

// Separate numbers from non-numbers
val (numbers, nonNumbers) = json.partition(_.is(JsonType.Number))
// numbers: {"a": 1, "c": 2}
// nonNumbers: {"b": "text"}
```

## Folding

Folding walks the whole tree and combines every value, together with its path, into one result; `Json#foldUp` and `Json#foldDown` differ in visit order.

### Fold Up (Bottom-Up)

Accumulate values from children to parents:

```scala

val json = Json.parseUnsafe("""{"values": [1, 2, 3, 4, 5]}""")

// Sum all numbers
val sum = json.foldUp(BigDecimal(0)) { (path, value, acc) =>
  value match {
    case n: Json.Number => acc + n.value
    case _ => acc
  }
}
// sum = 15
```

### Fold Down (Top-Down)

Accumulate values from parents to children:

```scala

val json = Json.parseUnsafe("""{"a": {"b": {"c": 1}}}""")

// Collect all paths
val paths = json.foldDown(Vector.empty[DynamicOptic]) { (path, value, acc) =>
  acc :+ path
}
```

## Merging

Combine two JSON values using different strategies:

```scala

val base = Json.parseUnsafe("""{"a": 1, "b": {"x": 10}}""")
val overlay = Json.parseUnsafe("""{"b": {"y": 20}, "c": 3}""")

// Auto strategy (default) - deep merge objects, merge arrays by index
val merged = base.merge(overlay)
// {"a": 1, "b": {"x": 10, "y": 20}, "c": 3}

// Shallow merge (only top-level)
val shallow = base.merge(overlay, MergeStrategy.Shallow)

// Replace (right wins)
val replaced = base.merge(overlay, MergeStrategy.Replace)
// {"b": {"y": 20}, "c": 3}

// Concat: deep merge objects, append arrays
val concat = base.merge(overlay, MergeStrategy.Concat)

// Custom strategy
val custom = base.merge(overlay, MergeStrategy.Custom { (path, left, right) =>
  // Your merge logic here
  right
})
```

### Merge Strategies

| Strategy       | Objects                           | Arrays                                       | Primitives   |
|----------------|-----------------------------------|----------------------------------------------|--------------|
| `Auto`         | Merge by key at every depth       | Merge by index (`[1,2,3]` + `[9]` = `[9,2,3]`) | Right wins   |
| `Shallow`      | Merge top-level keys; nested right wins | Merge top-level by index; nested right wins | Right wins   |
| `Replace`      | Right wins                        | Right wins                                   | Right wins   |
| `Concat`       | Merge by key at every depth       | Append right to left                         | Right wins   |
| `Custom(f, r)` | `r` decides whether to descend    | `r` decides whether to descend               | `f` decides  |

## Normalization

Clean up JSON values:

```scala

val json = Json.parseUnsafe("""{
  "z": 1,
  "a": null,
  "m": {"empty": {}},
  "b": 2
}""")

// Sort object keys alphabetically
val sorted = json.sortKeys
// {"a": null, "b": 2, "m": {"empty": {}}, "z": 1}

// Remove null values
val noNulls = json.dropNulls
// {"z": 1, "m": {"empty": {}}, "b": 2}

// Remove empty objects and arrays
val noEmpty = json.dropEmpty
// {"z": 1, "a": null, "b": 2}

// Apply all normalizations
val normalized = json.normalize
// {"b": 2, "z": 1}
```

## Encoding and Decoding

Any type with a `Schema` can be encoded to and decoded from JSON, either through its `JsonCodec` or through extension methods on the value.

### Built-in Codecs

The built-in schemas of primitives, standard collections and `java.time`/`java.util` types each give a `JsonCodec` through `Schema#jsonCodec`:

```scala

// Primitives
Schema[String].jsonCodec
Schema[Int].jsonCodec
Schema[Long].jsonCodec
Schema[Double].jsonCodec
Schema[Boolean].jsonCodec
Schema[BigDecimal].jsonCodec

// Collections
Schema[List[Int]].jsonCodec
Schema[Vector[String]].jsonCodec
Schema[Map[String, Int]].jsonCodec
Schema[Option[String]].jsonCodec

// Java time/util types
Schema[java.time.Instant].jsonCodec
Schema[java.time.LocalDate].jsonCodec
Schema[java.time.ZonedDateTime].jsonCodec
Schema[java.util.UUID].jsonCodec
```

### Extension Syntax

When a `Schema` is in scope, you can use convenient extension methods on any Scala value to encode and decode JSON. These work on primitives, case classes, and all other types:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val person = Person("Alice", 30)

// Convert to Json AST
val json = person.toJson  // Json.Object(...)

// Convert directly to JSON string
val jsonString = person.toJsonString  // {"name":"Alice","age":30}

// Convert to UTF-8 bytes
val jsonBytes = person.toJsonBytes  // Array[Byte]

// Parse JSON string back to a typed value
val parsed = """{"name":"Bob","age":25}""".fromJson[Person]  // Right(Person("Bob", 25))

// Parse from bytes
val fromBytes = jsonBytes.fromJson[Person]  // Right(Person("Alice", 30))

// Works on primitives too
val intJson = 42.toJson  // Json.Number(42)
val strJson = "hello".toJson  // Json.String("hello")
val intResult = intJson.as[Int]  // Right(42)
```

These extension methods provide a more ergonomic API compared to explicitly creating encoders/decoders, and work consistently across all types.

## Printing JSON

`Json#print` and `Json#printBytes` turn a `Json` into text or bytes, `WriterConfig` controls the output layout, and `ReaderConfig` is the matching setting for parsing.

### Basic Printing

`Json#print` with no arguments renders `json` as compact text with no whitespace:

```scala

val json = Json.Object("name" -> Json.String("Alice"), "age" -> Json.Number(30))

// Compact output
val compact: String = json.print
// {"name":"Alice","age":30}
```

### With Writer Config

Passing a `WriterConfig` to `Json#print` changes the layout; `WriterConfig.withIndentionStep2` gives 2-space indentation:

```scala

val json = Json.Object("name" -> Json.String("Alice"))

// Pretty-printed output (2-space indentation)
val pretty = json.print(WriterConfig.withIndentionStep2)
// {
//   "name": "Alice"
// }

// Custom indentation
val indented4 = json.print(WriterConfig.withIndentionStep(4))
```

### WriterConfig Options

`WriterConfig` controls JSON output formatting:

| Option | Default | Description |
|--------|---------|-------------|
| `indentionStep` | `0` | Spaces per indentation level (0 = compact) |
| `escapeUnicode` | `false` | Escape non-ASCII characters as `\uXXXX` |
| `preferredBufSize` | `32768` | Internal buffer size in bytes |

Each option has a `with` method on `WriterConfig`, and the calls chain to combine options:

```scala

// Compact output (default)
val compact = WriterConfig

// Pretty-printed with 2-space indentation
val pretty = WriterConfig.withIndentionStep(2)

// Escape Unicode for ASCII-only output
val ascii = WriterConfig.withEscapeUnicode(true)

// Combine options
val custom = WriterConfig
  .withIndentionStep(2)
  .withEscapeUnicode(true)
  .withPreferredBufSize(65536)
```

### ReaderConfig Options

`ReaderConfig` controls JSON parsing behavior:

| Option | Default | Description |
|--------|---------|-------------|
| `preferredBufSize` | `32768` | Preferred byte buffer size |
| `preferredCharBufSize` | `4096` | Preferred char buffer size for strings |
| `maxBufSize` | `33554432` | Maximum byte buffer size (32MB) |
| `maxCharBufSize` | `4194304` | Maximum char buffer size (4MB) |
| `checkForEndOfInput` | `true` | Error on trailing non-whitespace |

`ReaderConfig` has the same kind of `with` methods, for example to allow trailing content or to enlarge the buffers:

```scala

// Default configuration
val default = ReaderConfig

// Allow trailing content (useful for streaming)
val lenient = ReaderConfig.withCheckForEndOfInput(false)

// Increase buffer sizes for large documents
val largeDoc = ReaderConfig
  .withPreferredBufSize(65536)
  .withPreferredCharBufSize(8192)
```

### To Bytes

`Json#printBytes` encodes `json` as a byte array instead of a `String`:

```scala

val json = Json.Object("x" -> Json.Number(1))

// As byte array
val bytes: Array[Byte] = json.printBytes
```

## Query Operations

Query operations search the whole tree for values that match a condition, or flatten it into path-value pairs.

### Query with Predicate

Find all values matching a condition:

```scala

val json = Json.parseUnsafe("""{
  "users": [
    {"name": "Alice", "active": true},
    {"name": "Bob", "active": false},
    {"name": "Charlie", "active": true}
  ]
}""")

// Find all active users using queryBoth on a selection
val activeUsers = json.select.queryBoth { (path, value) =>
  value.get("active").as[Boolean].getOrElse(false)
}
```

### Convert to Key-Value Pairs

Flatten to path-value pairs:

```scala

val json = Json.parseUnsafe("""{"a": {"b": 1, "c": 2}}""")

val pairs: Chunk[(DynamicOptic, Json)] = json.toKV
// Chunk((.a.b,1),(.a.c,2))
```

## Comparison and Equality

`Json` values compare by content: object equality ignores key order, and `Json#compare` gives a total ordering across all cases.

### Object Equality

Objects are compared **order-independently** (keys are compared as sorted sets):

```scala

val obj1 = Json.parseUnsafe("""{"a": 1, "b": 2}""")
val obj2 = Json.parseUnsafe("""{"b": 2, "a": 1}""")

obj1 == obj2  // true (order-independent)
```

### Ordering

JSON values have a total ordering for sorting:

```scala

val values = List(
  Json.String("z"),
  Json.Number(1),
  Json.Null,
  Json.Boolean(true)
)

// Sort by type, then by value
val sorted = values.sortWith((a, b) => a.compare(b) < 0)
// [null, true, 1, "z"]
```

Type ordering: Null < Boolean < Number < String < Array < Object

## JSON Diffing

`JsonDiffer` computes the difference between two JSON values, producing a [`JsonPatch`](./json-patch.md) that transforms the source into the target:

```scala

val source = Json.parseUnsafe("""{"name": "Alice", "age": 30}""")
val target = Json.parseUnsafe("""{"name": "Alice", "age": 31, "active": true}""")

// Create a patch describing the differences
val patch: JsonPatch = JsonPatch.diff(source, target)

// The patch describes the minimal changes:
// - NumberDelta for age: 30 -> 31
// - Add field "active": true
```

The differ uses optimal operations:
- **NumberDelta** for numeric changes (stores the delta, not the new value)
- **StringEdit** for string changes when edits are more compact than replacement
- **ArrayEdit** with LCS-based Insert/Delete operations for arrays
- **ObjectEdit** with Add/Remove/Modify operations for objects

## JSON Patching

`JsonPatch` represents a sequence of operations that transform a JSON value. Patches are composable and can be applied with different failure modes.

### Computing and Applying Patches

```scala

val original = Json.parseUnsafe("""{"count": 10, "items": ["a", "b"]}""")
val modified = Json.parseUnsafe("""{"count": 15, "items": ["a", "b", "c"]}""")

// Compute the patch
val patch = JsonPatch.diff(original, modified)

// Apply with default (Strict) mode - fails on any precondition violation
val result1: Either[SchemaError, Json] = patch(original)

// Apply with Lenient mode - skips failing operations
val result2 = patch(original, PatchMode.Lenient)

// Apply with Clobber mode - forces changes on conflicts
val result3 = patch(original, PatchMode.Clobber)
```

### Patch Modes

| Mode | Behavior |
|------|----------|
| `Strict` | Fail immediately on any precondition violation |
| `Lenient` | Skip operations that fail preconditions |
| `Clobber` | Force changes, overwriting on conflicts |

### Composing Patches

`JsonPatch#++` joins two patches into one that applies the first and then the second:

```scala

val patch1 = JsonPatch.diff(
  Json.parseUnsafe("""{"x": 1}"""),
  Json.parseUnsafe("""{"x": 2}""")
)

val patch2 = JsonPatch.diff(
  Json.parseUnsafe("""{"x": 2}"""),
  Json.parseUnsafe("""{"x": 2, "y": 3}""")
)

// Compose patches - applies patch1, then patch2
val combined = patch1 ++ patch2

// Apply the combined patch
val result = combined(Json.parseUnsafe("""{"x": 1}"""))
// Right({"x": 2, "y": 3})
```

### Converting to DynamicPatch

`JsonPatch` can be converted to and from `DynamicPatch` for interoperability with the typed patching system:

```scala

val source = Json.parseUnsafe("""{"value": 1}""")
val target = Json.parseUnsafe("""{"value": 2}""")
val jsonPatch: JsonPatch = JsonPatch.diff(source, target)

// Convert to DynamicPatch
val dynamicPatch: DynamicPatch = jsonPatch.toDynamicPatch

// Convert from DynamicPatch (may fail for unsupported operations)
val restored: Either[SchemaError, JsonPatch] = JsonPatch.fromDynamicPatch(dynamicPatch)
```

## Conversion to DynamicValue

Convert JSON to ZIO Blocks' semi-structured `DynamicValue`:

```scala

val json = Json.parseUnsafe("""{"name": "Alice"}""")

val dynamic: DynamicValue = json.toDynamicValue
```

This enables interoperability with other ZIO Blocks formats (Avro, TOON, etc.).

## Error Handling

Navigation and decoding report failures as a `SchemaError` in a `Left` rather than throwing, and the error records the path where the failure happened.

### SchemaError

Errors include path information for debugging:

```scala

val json = Json.parseUnsafe("""{"users": [{"name": "Alice"}]}""")

val result = json.get("users")(5).get("name").as[String]
// Left(zio.blocks.schema.SchemaError: Index 5 out of bounds (size: 1) at: [5])
```

### Error Properties

```scala

case class User(id: Int, name: String)
object User {
  implicit val schema: Schema[User] = Schema.derived
}

val codec = User.schema.derive(JsonFormat)
val invalidJson = """{"id": "not-a-number", "name": "Alice"}"""
val result = codec.decode(invalidJson)

// Extract error properties when decoding fails
result match {
  case Left(error: SchemaError) =>
    error.message            // Error description
    error.errors.head.source // DynamicOptic path to error location
  case Right(_) => ()
}
```

## Cross-Platform Support

The `Json` type works on two platforms: the JVM, with full functionality, and Scala.js, in the browser and on Node.js.

String interpolators use compile-time validation that works on both platforms too.

## See Also

- [How to Read and Modify JSON Without Case Classes](../../../../guides/json-without-case-classes.md) - A task-oriented guide to parsing, reading, changing, and printing JSON as a tree

---

## MessagePack Codec Module

`zio-blocks-schema-messagepack` is a **schema-driven MessagePack codec module** for serializing and deserializing Scala types to and from MessagePack binary format. It provides comprehensive encoding and decoding with support for 30 primitive types, records, variants, sequences, maps, and recursive types. Core types: `MessagePackCodec`, `MessagePackCodecDeriver`, `MessagePackFormat`.

The module integrates with MessagePack specification to provide compact binary serialization with automatic schema generation and optimized reader/writer pools for high-performance streaming.

## Motivation

MessagePack is a compact binary serialization format that achieves smaller payload sizes than JSON while maintaining compatibility and flexibility. It appears widely across distributed systems, real-time streaming, and space-constrained environments. Manually writing MessagePack encoders and decoders is error-prone and repetitive, especially for complex types with records, nested structures, and recursive definitions. `zio-blocks-schema-messagepack` eliminates this friction by deriving codec instances directly from your Scala types using ZIO Blocks Schema. You describe your data shape once, and the module handles:
- Support for the MessagePack types that Scala values map to (fixint, fixarray, fixmap, strings, binary, floats, integers); the extension (`ext`) types are not supported
- Automatic schema generation from Scala types
- Highly optimized encoding with minimal overhead
- Reader/writer pool management for efficient coding
- Precise error reporting with location traces showing the path to errors
- Recursive type support with automatic cycle detection
- Multiple encoding paths: byte arrays and ByteBuffer
- Multiple decoding paths: byte arrays and ByteBuffer
- Cross-platform compatibility (JVM and Scala.js)

Rather than writing custom encoders or relying on string-based schema configuration, you work with strongly-typed schemas that the compiler validates.

## Installation

Add the module to your `build.sbt`:

```sbt
libraryDependencies += "dev.zio" %% "zio-blocks-schema-messagepack" % "0.0.56"
```

For Scala.js, use `%%%` instead of `%%`:

```sbt
libraryDependencies += "dev.zio" %%% "zio-blocks-schema-messagepack" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x

## Introduction

The module provides a complete pipeline for MessagePack codec derivation and usage:

1. **Define your type** — Any Scala type with a `Schema` instance
2. **Derive a codec** — Use `Schema.derive(MessagePackFormat)` to obtain a `MessagePackCodec[A]`
3. **Encode or decode** — Call `codec.encode(value)` or `codec.decode(bytes)`
4. **Handle errors** — `decode` returns the `SchemaError`, with a path to the bad value, in a `Left`

The derivation process is automatic for all supported types (all 30 primitives, records, variants, sequences, maps). The module automatically generates MessagePack-compatible formats and handles encoding/decoding without manual configuration.

## How They Work Together

The MessagePack codec pipeline flows through these layers:

```
1. User defines Schema[A] for their type
                 ↓
2. Schema[A].derive(MessagePackFormat) creates MessagePackCodec[A]
                 ↓
3. MessagePackCodecDeriver derives Encoder and Decoder implementations
   - For primitives: type-specific MessagePack encoders/decoders
   - For records: field-by-field composition with map encoding
   - For variants: tagged union encoding
   - For sequences: array encoding/decoding
   - For maps: map encoding/decoding
                 ↓
4. MessagePackCodec provides multiple encoding paths
   - encode(value) → Array[Byte]
   - encode(value, buffer: ByteBuffer) → Unit
                 ↓
5. MessagePackCodec provides multiple decoding paths
   - decode(bytes: Array[Byte]) → Either[SchemaError, A]
   - decode(buffer: ByteBuffer) → Either[SchemaError, A]
                 ↓
6. MessagePackReader/Writer manage I/O with pooling
   - Reader pool: reuses instances for streaming decoding
   - Writer pool: reuses instances for streaming encoding
   - Automatic format detection and type-specific optimization
                 ↓
7. Errors include location traces
   Shows path (.field.at(index).nested) to error location
```

**Typical workflow:**

A user type flows through the derivation and encoding pipeline as follows:

```
User type (e.g., case class Person)
    ↓
Schema.derived (automatic via macro)
    ↓
Schema[Person].derive(MessagePackFormat) → MessagePackCodec[Person]
    ↓
Use codec.encode(person) to serialize → Array[Byte]
Use codec.decode(bytes) to deserialize → Either[SchemaError, Person]
    ↓
Handle SchemaError with location trace on failure
```

### Type Relationships

- **`MessagePackCodec[A]`** — Main public API; contains encoder and decoder for bidirectional serialization
- **`MessagePackCodecDeriver`** — Configuration and derivation system; generates codecs from Schema
- **`MessagePackFormat`** — Integration with the ZIO Blocks Schema format system; enables `Schema[A].derive(MessagePackFormat)`
- **`MessagePackReader`** — Low-level binary parsing; stateful format-aware decoder
- **`MessagePackWriter`** — Low-level binary encoding; optimized format-aware encoder
- **`SchemaError`** — Error type with location traces; renders as `DynamicOptic` paths like `.field.at(0).nested`

## Common Patterns

This section shows practical patterns for working with MessagePack codecs in real-world scenarios.

### Pattern 1: Derive and Encode a Simple Record

To derive and use a MessagePack codec for a record type:

```scala

case class Person(name: String, age: Int, email: String)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Person.schema.derive(MessagePackFormat)
val person = Person("Alice", 30, "alice@example.com")
val bytes = codec.encode(person)
```

### Pattern 2: Decode MessagePack with Error Handling

When decoding MessagePack data, errors include location traces showing where the problem occurred.

To decode bytes and handle errors with location information:

```scala

case class Employee(id: Int, name: String, salary: Double)

object Employee {
  implicit val schema: Schema[Employee] = Schema.derived
}

val codec = Employee.schema.derive(MessagePackFormat)
val bytes = Array[Byte](1, 2, 3) // truncated data

val result = codec.decode(bytes)

result match {
  case Right(employee) => println(s"Decoded: $employee")
  case Left(error) =>
    println(s"Error: ${error.getMessage}")
}
```

### Pattern 3: Stream Large Datasets with Reader Pool

The reader pool efficiently handles streaming decoding of multiple messages from a large dataset.

To decode a sequence of values from a stream with pooled readers:

```scala

case class Event(id: Long, timestamp: Long, action: String)

object Event {
  implicit val schema: Schema[Event] = Schema.derived
}

val codec = Event.schema.derive(MessagePackFormat)
// Encode an event to get sample bytes
val event1 = Event(1L, System.currentTimeMillis(), "click")
val bytes1 = codec.encode(event1)

// Decode the bytes back to an event
val result = codec.decode(bytes1)
result match {
  case Right(decoded) => println(s"Decoded: $decoded")
  case Left(error) => println(s"Error: ${error.getMessage}")
}
```

### Pattern 4: Handle Recursive Types

Recursive types (types that reference themselves) are fully supported with automatic cycle detection.

To define and encode a recursive data structure:

```scala

sealed trait Node
case class Leaf(value: String) extends Node
case class Branch(left: Node, right: Node) extends Node

object Node {
  implicit val schema: Schema[Node] = Schema.derived
}

val codec = Node.schema.derive(MessagePackFormat)
val tree: Node = Branch(Leaf("A"), Branch(Leaf("B"), Leaf("C")))
val bytes = codec.encode(tree)
```

---

## MessagePackCodec[A]

Main codec type for encoding and decoding values to and from MessagePack binary format. Contains encoder and decoder for bidirectional serialization.

### Overview

`MessagePackCodec[A]` holds both an encoder and decoder, providing a complete solution for serializing and deserializing values in MessagePack binary format. The codec is derived automatically from a `Schema[A]` using `MessagePackFormat`.

### Encoding Values to Byte Array

Use the codec to convert values to byte arrays:

```scala

case class Product(name: String, price: Double)

object Product {
  implicit val schema: Schema[Product] = Schema.derived
}

val codec = Product.schema.derive(MessagePackFormat)
val product = Product("Widget", 9.99)
val bytes = codec.encode(product)
```

### Encoding Values to ByteBuffer

Write encoded values directly to a ByteBuffer:

```scala

case class Item(name: String, quantity: Int)

object Item {
  implicit val schema: Schema[Item] = Schema.derived
}

val codec = Item.schema.derive(MessagePackFormat)
val item = Item("Gadget", 42)
val buffer = ByteBuffer.allocate(256)
codec.encode(item, buffer)
val bytes = java.util.Arrays.copyOf(buffer.array(), buffer.position())
```

### Decoding Values from Byte Array

Use the codec to convert byte arrays back to values:

```scala

case class Record(id: Int, value: String)

object Record {
  implicit val schema: Schema[Record] = Schema.derived
}

val codec = Record.schema.derive(MessagePackFormat)
// In real usage, you would have encoded bytes from a previous encoding or external source
val record = Record(123, "test value")
val buffer = java.nio.ByteBuffer.allocate(256)
codec.encode(record, buffer)
buffer.flip()
val bytes = new Array[Byte](buffer.remaining())
buffer.get(bytes)

val result: Either[zio.blocks.schema.SchemaError, Record] = codec.decode(bytes)
```

### Decoding Values from ByteBuffer

Read and decode values from a ByteBuffer:

```scala

case class Data(timestamp: Long, payload: String)

object Data {
  implicit val schema: Schema[Data] = Schema.derived
}

val codec = Data.schema.derive(MessagePackFormat)
// Encode a value first
val sampleData = Data(System.currentTimeMillis(), "example")
val encBuffer = ByteBuffer.allocate(256)
codec.encode(sampleData, encBuffer)
encBuffer.flip()
// Then decode from the buffer
val result = codec.decode(encBuffer)
```

---

## MessagePackCodecDeriver

Configuration and derivation system for creating `MessagePackCodec[A]` instances from `Schema[A]`.

### Overview

`MessagePackCodecDeriver` implements the schema-driven derivation of MessagePack codecs. It automatically handles 30 primitive types and complex types (records, variants, sequences, maps), generating appropriate MessagePack encoders and decoders.

### How Derivation Works

To create a codec from a schema:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Person.schema.derive(MessagePackFormat)
// codec: MessagePackCodec[Person] = zio.blocks.schema.msgpack.MessagePackCodecDeriver$$anon$3@26b498fb
```

### Primitive Type Support

All 30 primitives of ZIO Blocks Schema are supported:
- Numeric: `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, `BigDecimal`
- Logical: `Boolean`, `Char`, `String`
- Temporal: `Instant`, `LocalDate`, `LocalDateTime`, `LocalTime`, `Duration`, `Period`, `Year`, `YearMonth`, `MonthDay`, `Month`, `DayOfWeek`, `ZonedDateTime`, `OffsetDateTime`, `OffsetTime`, `ZoneId`, `ZoneOffset`
- Special: `UUID`, `Currency`, `Unit`

### Record Type Support

Case classes (records) are fully supported. Each field becomes a named key in the MessagePack map encoding:

```scala

case class Address(street: String, city: String, zip: String)

object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

val codec = Address.schema.derive(MessagePackFormat)
```

### Variant Type Support

A sealed trait is encoded as two consecutive MessagePack values: the index of the case (an integer, in declaration order) followed by the case's own encoding. The case name is not written. Only `Either` is a map (`{"left": ...}` or `{"right": ...}`), and `Option` is an array of zero or one element. Writing the index and the case as two separate values is a known bug in 0.0.56; the layout is not a single value per variant:

```scala

sealed trait Status
case class Active(since: String) extends Status
case class Inactive(reason: String) extends Status

object Status {
  implicit val schema: Schema[Status] = Schema.derived
}

val codec = Status.schema.derive(MessagePackFormat)
```

---

## MessagePackFormat

Integration point with the ZIO Blocks Schema format system. Provides `BinaryFormat[MessagePackCodec]` to enable `Schema[A].derive(MessagePackFormat)` for any supported type.

### Using MessagePackFormat

To derive a MessagePack codec using the standard format:

```scala

case class Sensor(id: Long, temperature: Double, humidity: Float)

object Sensor {
  implicit val schema: Schema[Sensor] = Schema.derived
}

val codec = Sensor.schema.derive(MessagePackFormat)
```

`MessagePackFormat` is a singleton object extending `BinaryFormat[MessagePackCodec]` with the MIME type `"application/msgpack"` and the `MessagePackCodecDeriver` as its derivation strategy.

---

## MessagePackReader

Low-level binary parser implementing MessagePack format with type-specific optimizations and pooling support.

### Overview

`MessagePackReader` provides stateful reading of MessagePack-encoded data with automatic format detection and efficient handling of the MessagePack types that Scala values map to (fixint, fixarray, fixmap, string, binary, float, etc.). It does not accept extension types: the typed readers fail with `Expected ... value, got: 212`, `MessagePackReader#skipValue` with `Unknown MessagePack type: ...`, and the `DynamicValue` codec with `Unsupported MessagePack type for DynamicValue: 212`.

### Reading Primitives

To read individual values from a MessagePack-encoded byte array:

```scala

val bytes = Array[Byte](42) // MessagePack-encoded integer

// Use the codec API for public access to decoding
case class Value(data: Int)
object Value {
  implicit val schema: Schema[Value] = Schema.derived
}
val codec = Value.schema.derive(MessagePackFormat)
val result = codec.decode(bytes)
```

---

## MessagePackWriter

Low-level binary encoder implementing MessagePack format with optimizations for compact representation.

### Overview

`MessagePackWriter` provides efficient writing of Scala values into MessagePack binary format with automatic selection of compact encodings (fixint, fixarray, fixmap, etc.).

### Writing Primitives

To write individual values to MessagePack format:

```scala

// Use the codec API for public access to encoding
case class Value(data: Int)
object Value {
  implicit val schema: Schema[Value] = Schema.derived
}
val codec = Value.schema.derive(MessagePackFormat)
val value = Value(42)

// MessagePackCodec.encode returns Array[Byte] directly
val bytes = codec.encode(value)
```

---

## Error Handling

MessagePack decoding errors include location traces showing the path through nested structures where the error occurred.

### Understanding Error Traces

Errors render as `DynamicOptic` paths such as `.emails.at(1)`: a record field is `.name`, a sequence index is `.at(1)`, a map value is `.atKey(...)` and a variant case is `.when[Case]`. The path shows exactly where decoding failed:

```scala

case class Contact(emails: Seq[String])

object Contact {
  implicit val schema: Schema[Contact] = Schema.derived
}

val codec = Contact.schema.derive(MessagePackFormat)
// Example invalid bytes - in real code, this might come from untrusted input
val invalidBytes: Array[Byte] = Array(0xFF.toByte, 0xFF.toByte) // Invalid MessagePack data

val result = codec.decode(invalidBytes)

result match {
  case Right(contact) => println(s"Success: $contact")
  case Left(error) =>
    println(s"Error: ${error.getMessage}")
}
```

### Zero-Overhead Error Handling

Errors use zero-overhead exceptions (no stack traces) for efficient error reporting in streaming scenarios where errors are expected and handled inline.

---

## Thrift Codec Module

`zio-blocks-schema-thrift` is a **schema-driven Thrift codec module** for serializing and deserializing Scala types to and from Thrift binary format. It provides comprehensive encoding and decoding with support for 30 primitive types, records, variants, sequences, maps, and recursive types. Core types: `ThriftCodec`, `ThriftCodecDeriver`, `ThriftFormat`.

The module integrates with Apache Thrift to provide native binary serialization using TBinaryProtocol with automatic schema generation and optimized performance for distributed systems.

## Motivation

Thrift is a compact binary serialization format that appears widely across distributed systems, microservices, and real-time messaging. Manually writing Thrift encoders and decoders is error-prone and repetitive, especially for complex types with records, nested structures, and recursive definitions. `zio-blocks-schema-thrift` eliminates this friction by deriving codec instances directly from your Scala types using ZIO Blocks Schema. You describe your data shape once, and the module handles:
- Full Thrift type support (all primitives, records, unions, arrays, maps)
- No `.thrift` file needed: the codec derives the Thrift layout from your Scala types (it generates no IDL)
- Optimized encoding with minimal overhead using TBinaryProtocol
- Precise error reporting with location traces showing the path to errors
- Recursive type support, such as trees
- Multiple encoding paths: ByteBuffer and byte arrays
- Multiple decoding paths: ByteBuffer and byte arrays
- JVM support (not available for Scala.js)

Rather than writing custom encoders or relying on code generation from .thrift files, you work with strongly-typed schemas that the compiler validates.

## Installation

Add the module to your `build.sbt`:

```sbt
libraryDependencies += "dev.zio" %% "zio-blocks-schema-thrift" % "0.0.56"
```

**Note:** This module is JVM-only and is not available for Scala.js.

Supported Scala versions: 2.13.x and 3.x

## Introduction

The module provides a complete pipeline for Thrift codec derivation and usage:

1. **Define your type** — Any Scala type with a `Schema` instance
2. **Derive a codec** — Use `Schema.derive(ThriftFormat)` to obtain a `ThriftCodec[A]`
3. **Encode or decode** — Call `codec.encode(value)` or `codec.decode(bytes)`
4. **Handle errors** — `decode` returns the `SchemaError`, with the path to the bad value, in a `Left`

The derivation process is automatic for all supported types (all 30 primitives, records, variants, sequences, maps). The module handles encoding and decoding without a `.thrift` file or manual configuration.

## How They Work Together

The Thrift codec pipeline flows through these layers:

```
1. User defines Schema[A] for their type
                 ↓
2. Schema[A].derive(ThriftFormat) creates ThriftCodec[A]
                 ↓
3. ThriftCodecDeriver derives Encoder and Decoder implementations
   - For primitives: type-specific Thrift encoders/decoders
   - For records: field-by-field composition with Thrift struct encoding
   - For variants: a struct with one field, whose id is the case index plus one
   - For sequences: array encoding/decoding
   - For maps: map encoding/decoding
                 ↓
4. ThriftCodec provides multiple encoding paths
   - encode(value) → Array[Byte]
   - encode(value, buffer: ByteBuffer) → Unit
                 ↓
5. ThriftCodec provides multiple decoding paths
   - decode(bytes: Array[Byte]) → Either[SchemaError, A]
   - decode(buffer: ByteBuffer) → Either[SchemaError, A]
                 ↓
6. TBinaryProtocol manages I/O with Thrift wire format
   - Efficient binary encoding with type information
   - Always `TBinaryProtocol`; nothing detects or switches the protocol
   - Support for unknown fields (schema evolution)
                 ↓
7. Errors include location traces
   Shows path (.field.at(index).nested) to error location
```

**Typical workflow:**

A user type flows through the derivation and encoding pipeline as follows:

```
User type (e.g., case class Person)
    ↓
Schema.derived (automatic via macro)
    ↓
Schema[Person].derive(ThriftFormat) → ThriftCodec[Person]
    ↓
Use codec.encode(person) to serialize → Array[Byte]
Use codec.decode(bytes) to deserialize → Either[SchemaError, Person]
    ↓
Handle SchemaError with location trace on failure
```

### Type Relationships

- **`ThriftCodec[A]`** — Main public API; contains encoder and decoder for bidirectional serialization
- **`ThriftCodecDeriver`** — Configuration and derivation system; generates codecs from Schema
- **`ThriftFormat`** — Integration with the ZIO Blocks Schema format system; enables `Schema[A].derive(ThriftFormat)`

## Common Patterns

This section shows practical patterns for working with Thrift codecs in real-world scenarios.

### Pattern 1: Derive and Encode a Simple Record

To derive a Thrift codec for a record type and encode a value:

```scala

case class Person(name: String, age: Int, email: String)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Person.schema.derive(ThriftFormat)
val person = Person("Alice", 30, "alice@example.com")
val bytes = codec.encode(person)
```

### Pattern 2: Decode Thrift with Error Handling

When decoding Thrift data, errors include location traces showing where the problem occurred.

To decode bytes and handle errors with location information:

```scala

case class Employee(id: Int, name: String, salary: Double)

object Employee {
  implicit val schema: Schema[Employee] = Schema.derived
}

val codec = Employee.schema.derive(ThriftFormat)
val bytes = Array[Byte](1, 2, 3) // truncated data

val result = codec.decode(bytes)

result match {
  case Right(employee) => println(s"Decoded: $employee")
  case Left(error) =>
    println(s"Error: ${error.getMessage}")
}
```

### Pattern 3: Handle Recursive Types

Recursive types (types that reference themselves) are supported: a recursive type refers to itself by name. The codec does not detect cycles in values, and a case class value cannot be cyclic anyway.

To define and encode a recursive data structure:

```scala

sealed trait Node
case class Leaf(value: String) extends Node
case class Branch(left: Node, right: Node) extends Node

object Node {
  implicit val schema: Schema[Node] = Schema.derived
}

val codec = Node.schema.derive(ThriftFormat)
val tree: Node = Branch(Leaf("A"), Branch(Leaf("B"), Leaf("C")))
val bytes = codec.encode(tree)
```

### Pattern 4: Work with Variants (Sealed Traits)

A variant is a Thrift struct with one field. The field id is the case index plus one, so the case name is not written.

To derive a codec for a sealed trait:

```scala

sealed trait Status
case class Active(since: String) extends Status
case class Inactive(reason: String) extends Status

object Status {
  implicit val schema: Schema[Status] = Schema.derived
}

val codec = Status.schema.derive(ThriftFormat)
val status: Status = Active("2024-01-01")
val bytes = codec.encode(status)
```

---

## ThriftCodec[A]

Main codec type for encoding and decoding values to and from Thrift binary format. Contains encoder and decoder for bidirectional serialization.

### Overview

`ThriftCodec[A]` holds both an encoder and decoder, providing a complete solution for serializing and deserializing values in Thrift binary format. The codec is derived automatically from a `Schema[A]` using `ThriftFormat`.

### Encoding Values to Byte Array

Use the codec to convert values to byte arrays:

```scala

case class Product(name: String, price: Double)

object Product {
  implicit val schema: Schema[Product] = Schema.derived
}

val codec = Product.schema.derive(ThriftFormat)
val product = Product("Widget", 9.99)
val bytes = codec.encode(product)
```

### Encoding Values to ByteBuffer

Write encoded values directly to a ByteBuffer:

```scala

case class Item(name: String, quantity: Int)

object Item {
  implicit val schema: Schema[Item] = Schema.derived
}

val codec = Item.schema.derive(ThriftFormat)
val item = Item("Gadget", 42)
val buffer = ByteBuffer.allocate(1024)
codec.encode(item, buffer)
val bytes = java.util.Arrays.copyOf(buffer.array, buffer.position)
```

### Decoding Values from Byte Array

Use the codec to convert byte arrays back to values:

```scala

case class Record(id: Int, value: String)

object Record {
  implicit val schema: Schema[Record] = Schema.derived
}

val codec = Record.schema.derive(ThriftFormat)
val record = Record(123, "test")
val buffer = java.nio.ByteBuffer.allocate(256)
codec.encode(record, buffer)
buffer.flip()
val bytes = new Array[Byte](buffer.remaining())
buffer.get(bytes)

val result: Either[zio.blocks.schema.SchemaError, Record] = codec.decode(bytes)
```

### Decoding Values from ByteBuffer

Read and decode values from a ByteBuffer:

```scala

case class Data(timestamp: Long, payload: String)

object Data {
  implicit val schema: Schema[Data] = Schema.derived
}

val codec = Data.schema.derive(ThriftFormat)
// Use encoded data from a previous encoding
val data = Data(System.currentTimeMillis(), "example")
val encBuffer = java.nio.ByteBuffer.allocate(256)
codec.encode(data, encBuffer)
encBuffer.flip()
val result = codec.decode(encBuffer)
```

---

## ThriftCodecDeriver

Configuration and derivation system for creating `ThriftCodec[A]` instances from `Schema[A]`.

### Overview

`ThriftCodecDeriver` implements the schema-driven derivation of Thrift codecs. It automatically handles 30 primitive types and complex types (records, variants, sequences, maps), generating appropriate Thrift encoders and decoders.

### How Derivation Works

To create a codec from a schema:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Person.schema.derive(ThriftFormat)
// codec: ThriftCodec[Person] = zio.blocks.schema.thrift.ThriftCodecDeriver$$anon$3@1af7d21f
```

### Primitive Type Support

All 30 primitives of ZIO Blocks Schema are supported:
- Numeric: `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, `BigDecimal`
- Logical: `Boolean`, `Char`, `String`
- Temporal: `Instant`, `LocalDate`, `LocalDateTime`, `LocalTime`, `Duration`, `Period`, `Year`, `YearMonth`, `MonthDay`, `Month`, `DayOfWeek`, `ZonedDateTime`, `OffsetDateTime`, `OffsetTime`, `ZoneId`, `ZoneOffset`
- Special: `UUID`, `Currency`, `Unit`

`Float` is written as a Thrift `double`. `UUID`, `Currency` and most of the `java.time` types are Thrift strings. A `DynamicValue` cannot be decoded.

### Record Type Support

Case classes (records) are fully supported. Each field becomes a numbered field of the Thrift struct (ids 1, 2, 3 and so on, in declaration order; the name is not written). The decoder accepts the fields in any order and skips ids it doesn't know:

```scala

case class Address(street: String, city: String, zip: String)

object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

val codec = Address.schema.derive(ThriftFormat)
```

Known bugs in 0.0.56 that matter when you read data you did not write yourself:

- A record field missing from the input decodes as `null`, `0` or `false` instead of failing, so a missing `Option` field comes back as `null`, not `None`.
- The decoder does not check the Thrift type written before a field, so a field of another type can decode without an error as a wrong value.
- `decode` ignores any bytes after the first value.

### Variant Type Support

A sealed trait is encoded as a struct with one field whose id is the index of the case plus one (in declaration order). The case name is not written:

```scala

sealed trait Status
case class Active(since: String) extends Status
case class Inactive(reason: String) extends Status

object Status {
  implicit val schema: Schema[Status] = Schema.derived
}

val codec = Status.schema.derive(ThriftFormat)
```

---

## ThriftFormat

Integration point with the ZIO Blocks Schema format system. Provides `BinaryFormat[ThriftCodec]` to enable `Schema[A].derive(ThriftFormat)` for any supported type.

### Using ThriftFormat

To derive a Thrift codec using the standard format:

```scala

case class Sensor(id: Long, temperature: Double, humidity: Float)

object Sensor {
  implicit val schema: Schema[Sensor] = Schema.derived
}

val codec = Sensor.schema.derive(ThriftFormat)
```

`ThriftFormat` is a singleton object extending `BinaryFormat[ThriftCodec]` with the MIME type `"application/thrift"` and the `ThriftCodecDeriver` as its derivation strategy.

---

## Error Handling

Thrift decoding errors include location traces showing the path through nested structures where the error occurred.

### Understanding Error Traces

Errors render as `DynamicOptic` paths such as `.field.at(0).nested.value`: a record field is `.name`, a sequence index is `.at(0)`, a map value is `.atKey(...)` and a variant case is `.when[Case]`. Errors from the Thrift library start with `Thrift protocol error:`. The path shows exactly where decoding failed:

```scala

case class Contact(emails: Seq[String])

object Contact {
  implicit val schema: Schema[Contact] = Schema.derived
}

val codec = Contact.schema.derive(ThriftFormat)
// Example invalid Thrift bytes
val invalidBytes: Array[Byte] = Array(0xFF.toByte, 0xFF.toByte)

val result = codec.decode(invalidBytes)

result match {
  case Right(contact) => println(s"Success: $contact")
  case Left(error) =>
    println(s"Error: ${error.getMessage}")
}
```

### Zero-Overhead Error Handling

Errors use zero-overhead exceptions (no stack traces) for efficient error reporting in streaming scenarios where errors are expected and handled inline.

---

## TOON Codec Module

`zio-blocks-schema-toon` is a **schema-driven TOON codec module** for serializing and deserializing Scala types to and from TOON format. It provides comprehensive encoding and decoding with support for 30 primitive types, records, variants, sequences, maps, and recursive types. Core types: `ToonCodec`, `ToonCodecDeriver`, `ToonFormat`, `ToonReader`, `ToonWriter`.

The module integrates with a pure-Scala TOON parser and writer to provide line-oriented, indentation-based serialization that is 30-60% more compact than JSON. TOON (Token-Oriented Object Notation) appears widely across LLM prompts, configuration files, and data exchange where compactness and human readability matter equally.

See also the [How to Send Compact Data to an LLM with TOON and Parse the Reply](../../../guides/llm-data-with-toon.md) guide, which encodes records as tabular TOON for a prompt and decodes the model's reply into case classes.

## Motivation

TOON is a compact, line-oriented text format that encodes the JSON data model with explicit structure and minimal quoting. It appears widely in modern LLM applications, configuration management, and streaming data scenarios where bandwidth and readability are both critical. Manually writing TOON encoders and decoders is error-prone and repetitive, especially for complex types with records, nested structures, and recursive definitions. `zio-blocks-schema-toon` eliminates this friction by deriving codec instances directly from your Scala types using ZIO Schema. You describe your data shape once, and the module handles:
- Full TOON type support (records, sequences, primitives, null)
- Four array formats: inline (`tags[3]: a,b,c`), tabular (`users[2]{id,name}:`), list (`- item`), and auto, the default, which picks inline or list from the element type
- Automatic schema generation from Scala types
- Customizable field/case name mapping (identity, snake_case, kebab-case, etc.)
- Configurable discriminator strategies for algebraic data types
- Precise error reporting with location traces showing the path to errors
- Recursive type support with automatic cycle detection
- Multiple encoding paths: byte arrays, streams, ByteBuffers, and strings
- Multiple decoding paths: byte arrays, streams, ByteBuffers, and strings
- Cross-platform compatibility (JVM and Scala.js)

Rather than hand-writing TOON parsing logic or relying on external libraries with limited Scala support, you work with strongly-typed schemas that the compiler validates.

## Installation

Add the module to your `build.sbt`:

```sbt
libraryDependencies += "dev.zio" %% "zio-blocks-schema-toon" % "0.0.56"
```

For Scala.js, use `%%%` instead of `%%`:

```sbt
libraryDependencies += "dev.zio" %%% "zio-blocks-schema-toon" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x

## Introduction

The module provides a complete pipeline for TOON codec derivation and usage:

1. **Define your type** — Any Scala type with a `Schema` instance
2. **Derive a codec** — Use `Schema[A].derive(ToonFormat)` to obtain a `ToonCodec[A]`
3. **Encode or decode** — Call `codec.encode(value)` or `codec.decode(toonBytes)`
4. **Handle errors** — `decode` returns the `SchemaError`, with the path to the bad value, in a `Left`
5. **Customize output** — Use `WriterConfig` for indentation and formatting, `ReaderConfig` for parsing behavior

The derivation process is automatic for all supported types (all 30 primitives, records, variants, sequences, maps). The module automatically generates TOON-compatible formats and handles encoding/decoding without manual configuration.

## How They Work Together

The TOON codec pipeline flows through these layers:

```
1. User defines Schema[A] for their type
                 ↓
2. Schema[A].derive(ToonFormat) creates ToonCodec[A]
                 ↓
3. ToonCodecDeriver derives Encoder and Decoder implementations
   - For primitives: type-specific TOON scalar encoders/decoders
   - For records: field-by-field composition with indented nesting
   - For variants: union encoding with optional discriminators
   - For sequences: one of four formats (auto, inline, tabular, list)
   - For maps: record-like encoding with key-value pairs
                 ↓
4. ToonCodec provides multiple encoding paths
   - encode(value) → Array[Byte]
   - encode(value, output: OutputStream) → Unit
   - encode(value, buffer: ByteBuffer) → Unit
   - encodeToString(value) → String
                 ↓
5. ToonCodec provides multiple decoding paths
   - decode(bytes: Array[Byte]) → Either[SchemaError, A]
   - decode(input: InputStream) → Either[SchemaError, A]
   - decode(buffer: ByteBuffer) → Either[SchemaError, A]
   - decode(toon: String) → Either[SchemaError, A]
                 ↓
6. WriterConfig controls output formatting
   - Indentation (default: 2 spaces per level)
   - Key folding strategy, delimiter and discriminator field (DynamicValue codec only)
                 ↓
7. ReaderConfig controls parsing behavior
   - Strict parsing mode (enforce TOON compliance)
   - Path expansion and discriminator field recognition (DynamicValue codec only)
   - Array delimiters come from each array's header, not from ReaderConfig
                 ↓
8. Errors include location traces
   Shows path (.field.at(index).nested) to error location
```

A user type flows through the derivation and encoding pipeline as follows:

```
User type (e.g., case class Person)
    ↓
Schema.derived (automatic via macro)
    ↓
Schema[Person].derive(ToonFormat) → ToonCodec[Person]
    ↓
Use codec.encode(person) to serialize → Array[Byte]
Use codec.decode(toonBytes) to deserialize → Either[SchemaError, Person]
    ↓
Handle SchemaError with location trace on failure
```

### Type Relationships

- **`ToonCodec[A]`** — Main public API; contains encoder and decoder for bidirectional serialization
- **`ToonCodecDeriver`** — Configuration and derivation system; generates codecs from Schema
- **`ToonFormat`** — Integration with ZIO Schema format system; enables `Schema[A].derive(ToonFormat)`
- **`WriterConfig`** — Output formatting configuration (indentation for every codec; delimiter, key folding and discriminator field for the `DynamicValue` codec)
- **`ReaderConfig`** — Input parsing configuration (indentation and strict mode for every codec; path expansion and discriminator field for the `DynamicValue` codec)
- **`ToonReader`, `ToonWriter`** — Low-level TOON parsing and serialization
- **`NameMapper`** — Field and case name transformation strategies
- **`ArrayFormat`** — Controls sequence encoding (Inline, Tabular, List, Auto)
- **`Delimiter`** — Specifies separator for inline arrays (Comma, Tab, Pipe)

## Common Patterns

This section shows practical patterns for working with TOON codecs in real-world scenarios.

### Pattern 1: Derive and Encode a Configuration Record

For a case class with primitive fields, derive a codec and encode to TOON string.

To derive a TOON codec for a record type and encode a value:

```scala

case class AppConfig(name: String, port: Int, debug: Boolean)

object AppConfig {
implicit val schema: Schema[AppConfig] = Schema.derived[AppConfig]
}

val codec = AppConfig.schema.derive(ToonFormat)
val config = AppConfig("MyApp", 8080, true)
val toonBytes = codec.encode(config)
// Output:
// name: MyApp
// port: 8080
// debug: true
```

### Pattern 2: Decode TOON with Error Handling

When decoding TOON data, errors include location traces showing where the problem occurred.

To decode TOON and handle errors with location information:

```scala

case class DatabaseConfig(host: String, port: Int, username: String)

object DatabaseConfig {
  implicit val schema: Schema[DatabaseConfig] = Schema.derived
}

val codec = DatabaseConfig.schema.derive(ToonFormat)
val toon = """
host: localhost
port: invalid
username: admin
"""

val result = codec.decode(toon)

result match {
  case Right(config) => println(s"Loaded: $config")
  case Left(error) =>
    println(s"Error: ${error.getMessage}")
    // Error: Expected int, got: invalid at: .port
}
```

### Pattern 3: Work with Arrays in Different Formats

TOON has four formats for encoding sequences, selectable via `ArrayFormat` (see [ArrayFormat](#arrayformat)). With the default, `ArrayFormat.Auto`, an array of primitive values is written inline:

```scala

case class Tags(items: List[String])

object Tags {
  implicit val schema: Schema[Tags] = Schema.derived
}

println(Tags.schema.derive(ToonFormat).encodeToString(Tags(List("scala", "zio", "functional"))))
// items[3]: scala,zio,functional
```

To encode an array of records as a table, derive the codec with `ArrayFormat.Tabular`:

```scala

case class User(id: Int, name: String)
case class Team(members: List[User])

object User {
  implicit val schema: Schema[User] = Schema.derived
}

object Team {
  implicit val schema: Schema[Team] = Schema.derived
}

println(
  Team.schema
    .derive(ToonCodecDeriver.withArrayFormat(ArrayFormat.Tabular))
    .encodeToString(Team(List(User(1, "Alice"), User(2, "Bob"))))
)
// members[2]{id,name}:
//   1,Alice
//   2,Bob
```

### Pattern 4: Customize Field Names and Discriminators

Configure the deriver to map field names and control variant encoding.

To derive a codec with custom field name transformation:

```scala

case class Person(firstName: String, lastName: String, emailAddress: String)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Use snake_case for field names
val customDeriver = ToonCodecDeriver.withFieldNameMapper(NameMapper.SnakeCase)
val codec = Person.schema.derive(customDeriver)

val person = Person("Alice", "Smith", "alice@example.com")
val toonBytes = codec.encodeToString(person)
// Output:
// first_name: Alice
// last_name: Smith
// email_address: alice@example.com
```

### Pattern 5: Handle Recursive Types

Recursive types (types that reference themselves) are fully supported with automatic cycle detection.

To define and encode a recursive data structure:

```scala

sealed trait TreeNode
case class Leaf(value: String) extends TreeNode
case class Branch(label: String, children: List[TreeNode]) extends TreeNode

object TreeNode {
  implicit val schema: Schema[TreeNode] = Schema.derived
}

val codec = TreeNode.schema.derive(ToonFormat)
val tree: TreeNode = Branch("root", List(Leaf("a"), Branch("b", List(Leaf("c")))))
val toonBytes = codec.encodeToString(tree)
// Output:
// Branch:
//   label: root
//   children[2]:
//     - Leaf:
//         value: a
//     - Branch:
//         label: b
//         children[1]:
//           - Leaf:
//               value: c
```

---

## ToonCodec[A]

Main codec type for encoding and decoding values to and from TOON format. Contains encoder and decoder for bidirectional serialization.

### Overview

`ToonCodec[A]` holds both an encoder and decoder, providing a complete solution for serializing and deserializing values in TOON format. The codec is derived automatically from a `Schema[A]` using `ToonFormat`.

### Encoding Values to Byte Array

Use the codec to convert values to byte arrays:

```scala

case class Person(name: String, email: String)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}
```

```scala
val codec = Person.schema.derive(ToonFormat)
```

```scala
val person = Person("Alice", "alice@example.com")
// person: Person = Person(name = "Alice", email = "alice@example.com")
val bytes = codec.encode(person)
// bytes: Array[Byte] = Array(
//   110,
//   97,
//   109,
//   101,
//   58,
//   32,
//   65,
//   108,
//   105,
//   99,
//   101,
//   10,
//   101,
//   109,
//   97,
//   105,
//   108,
//   58,
//   32,
//   97,
//   108,
//   105,
//   99,
//   101,
//   64,
//   101,
//   120,
//   97,
//   109,
//   112,
//   108,
//   101,
//   46,
//   99,
//   111,
//   109
// )
```

### Encoding Values to String

Convert values to human-readable TOON strings:

```scala

case class Config(host: String, port: Int)

object Config {
  implicit val schema: Schema[Config] = Schema.derived
}

val codec = Config.schema.derive(ToonFormat)
val config = Config("localhost", 8080)
val toonString = codec.encodeToString(config)
```

### Encoding Values to Stream

Write encoded values directly to an output stream:

```scala

case class Data(timestamp: Long, value: String)

object Data {
  implicit val schema: Schema[Data] = Schema.derived
}

val codec = Data.schema.derive(ToonFormat)
val data = Data(System.currentTimeMillis(), "sample")
val output = new ByteArrayOutputStream()
codec.encode(data, output)
val bytes = output.toByteArray
```

### Decoding Values from Byte Array

Use the codec to convert byte arrays back to values:

```scala

case class Settings(debug: Boolean, timeout: Int)

object Settings {
  implicit val schema: Schema[Settings] = Schema.derived
}

val codec = Settings.schema.derive(ToonFormat)
// Create bytes from a previous encoding
val settings = Settings(debug = true, timeout = 30)
val bytes = codec.encode(settings)

val result: Either[zio.blocks.schema.SchemaError, Settings] = codec.decode(bytes)
```

### Decoding Values from String

Parse and decode values from TOON strings:

```scala

case class ServerConfig(name: String, port: Int)

object ServerConfig {
  implicit val schema: Schema[ServerConfig] = Schema.derived
}

val codec = ServerConfig.schema.derive(ToonFormat)
val toon = """
name: api-server
port: 8080
"""

val result = codec.decode(toon)
```

### Decoding Values from Stream

Read and decode values from bytes:

```scala

case class Message(id: Long, text: String)

object Message {
  implicit val schema: Schema[Message] = Schema.derived
}

val codec = Message.schema.derive(ToonFormat)
val message = Message(42, "Hello TOON")
val encoded = codec.encode(message)
val result = codec.decode(encoded)
```

---

## ToonCodecDeriver

Configuration and derivation system for creating `ToonCodec[A]` instances from `Schema[A]`.

### Overview

`ToonCodecDeriver` implements the schema-driven derivation of TOON codecs. It automatically handles 30 primitive types and complex types (records, variants, sequences, maps), generating appropriate TOON encoders and decoders with comprehensive customization options.

### How Derivation Works

To create a codec from a schema:

```scala

case class User(id: Int, name: String, active: Boolean)

object User {
  implicit val schema: Schema[User] = Schema.derived
}

val codec = User.schema.derive(ToonFormat)
```

### Primitive Type Support

All 30 primitives of ZIO Blocks Schema are supported:
- Numeric: `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, `BigDecimal`
- Logical: `Boolean`, `Char`, `String`
- Temporal: `Instant`, `LocalDate`, `LocalDateTime`, `LocalTime`, `Duration`, `Period`, `Year`, `YearMonth`, `MonthDay`, `Month`, `DayOfWeek`, `ZonedDateTime`, `OffsetDateTime`, `OffsetTime`, `ZoneId`, `ZoneOffset`
- Special: `UUID`, `Currency`, `Unit`

### Record Type Support

Case classes (records) are fully supported. Each field becomes an indented key-value pair in the TOON output:

```scala

case class Address(street: String, city: String, zipCode: String)

object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

val codec = Address.schema.derive(ToonFormat)
```

:::caution[Known bug in 0.0.56]
With the default deriver (`withTransientEmptyCollection(true)`), a record field of type `Array[...]` is never written, even when the array has elements, so decoding the output gives an empty array and the data is lost silently. `List`, `Vector` and `Map` fields are written as expected, and so is an `Array` field once you set `withTransientEmptyCollection(false)`. Prefer `List` or `Vector` for fields that you encode with this codec.
:::

### Variant Type Support

Sealed traits and sum types are encoded as discriminated variants, with customizable discriminator strategies:

```scala

sealed trait Status
case class Running(pid: Int) extends Status
case class Stopped(exitCode: Int) extends Status

object Status {
  implicit val schema: Schema[Status] = Schema.derived
}

val codec = Status.schema.derive(ToonFormat)
```

### Configuration Methods

To customize derivation behavior, chain configuration methods on `ToonCodecDeriver`:

To apply field name transformation:

```scala

val snakeCaseDeriver = ToonCodecDeriver.withFieldNameMapper(NameMapper.SnakeCase)
```

To select array format:

```scala

val tabularDeriver = ToonCodecDeriver.withArrayFormat(ArrayFormat.Tabular)
```

To customize variant discriminators:

```scala

val fieldDeriver = ToonCodecDeriver.withDiscriminatorKind(DiscriminatorKind.Field("type"))
```

---

## ToonFormat

Integration point with ZIO Schema's format system. Provides `BinaryFormat[ToonCodec]` to enable `Schema[A].derive(ToonFormat)` for any supported type.

### Using ToonFormat

To derive a TOON codec using the standard format:

```scala

case class Sensor(id: Long, temperature: Double, humidity: Float)

object Sensor {
  implicit val schema: Schema[Sensor] = Schema.derived
}

val codec = Sensor.schema.derive(ToonFormat)
```

`ToonFormat` is a singleton object extending `BinaryFormat[ToonCodec]` with the MIME type `"text/toon"` and the `ToonCodecDeriver` as its derivation strategy.

---

## WriterConfig

Configuration for TOON output formatting. Controls indentation for every codec, and key folding, the delimiter and the discriminator field for the `DynamicValue` codec.

### Overview

`WriterConfig` provides predefined configurations and factory methods for customizing TOON serialization output. Every codec reads its indentation; the other settings (key folding, flatten depth, delimiter and discriminator field) apply only to the `DynamicValue` codec, `Schema[DynamicValue].derive(ToonFormat)`. A codec derived for your own type takes them from `ToonCodecDeriver` instead.

### Indentation Configuration

Control spacing for nested indentation:

```scala

val config = WriterConfig.withIndent(4)
// Uses 4 spaces per indentation level (default: 2)
```

### Key Folding Strategy

`WriterConfig.withKeyFolding` folds a chain of nested single-field records into one dot-separated key. Only the `DynamicValue` codec reads it; a codec derived for a case class ignores it (see [KeyFolding](#keyfolding)):

```scala

val config = WriterConfig.withKeyFolding(KeyFolding.Safe)
```

`WriterConfig.withDelimiter` likewise applies only to the `DynamicValue` codec; a derived codec takes its delimiter from `ToonCodecDeriver.withDelimiter` (see [Delimiter](#delimiter)).

### Discriminator Field Configuration

`WriterConfig.withDiscriminatorField` makes the `DynamicValue` codec write a variant as a record with a discriminator field (`_type: Running`) instead of a key. A derived codec ignores it; choose its discriminator with `ToonCodecDeriver.withDiscriminatorKind` (see [DiscriminatorKind](#discriminatorkind)):

```scala

val config = WriterConfig.withDiscriminatorField(Some("_type"))
```

---

## ReaderConfig

Configuration for TOON input parsing. Controls path expansion and strict parsing mode.

### Overview

`ReaderConfig` provides factory methods for customizing TOON deserialization behavior. It controls whether dot-separated keys expand to nested records and whether parsing is strict.

### Path Expansion Configuration

`ReaderConfig.withExpandPaths` makes the reader turn a dot-separated key such as `settings.server.port` into nested records. Only the `DynamicValue` codec reads it; a codec derived for a case class ignores it (see [PathExpansion](#pathexpansion)):

```scala

val config = ReaderConfig.withExpandPaths(PathExpansion.Safe)
```

### Delimiter Configuration

`ReaderConfig.withDelimiter` exists, but it has no effect on decoding arrays: the reader takes the delimiter of each array from the array's header, where the writer records any delimiter other than the comma (see [Delimiter](#delimiter)).

### Strict Parsing Mode

Enable or disable strict TOON parsing:

```scala

val config = ReaderConfig.withStrict(false)
// Allows lenient parsing (default: true for strict compliance)
```

---

## ArrayFormat

Controls how sequences are encoded in TOON output.

### Overview

`ArrayFormat` selects one of four strategies for writing sequences: `Auto` (the default), `Inline`, `Tabular` and `List`. Set it on the deriver with `ToonCodecDeriver.withArrayFormat`. `Inline` and `Tabular` each suit one kind of element; for other elements the codec writes list form instead. Every format keeps the length in the header, so the decoder can check it.

The examples in this section encode the same list of strings and list of records with each format:

```scala

case class Tags(items: List[String])
object Tags { implicit val schema: Schema[Tags] = Schema.derived }

case class User(id: Int, name: String)
object User { implicit val schema: Schema[User] = Schema.derived }

case class Team(members: List[User])
object Team { implicit val schema: Schema[Team] = Schema.derived }

val tags = Tags(List("scala", "zio", "functional"))
val team = Team(List(User(1, "Alice"), User(2, "Bob")))

def encodeBoth(format: ArrayFormat): Unit = {
  val deriver = ToonCodecDeriver.withArrayFormat(format)
  println(Tags.schema.derive(deriver).encodeToString(tags))
  println(Team.schema.derive(deriver).encodeToString(team))
}
```

### Inline Format

`ArrayFormat.Inline` writes an array of primitive values on one line, separated by the delimiter. An array of records is still written in list form:

```scala
encodeBoth(ArrayFormat.Inline)
// items[3]: scala,zio,functional
// members[2]:
//   - id: 1
//     name: Alice
//   - id: 2
//     name: Bob
```

### Tabular Format

`ArrayFormat.Tabular` writes an array of records as a table: a header with the field names, then one row per record. An array of primitive values is written in list form:

```scala
encodeBoth(ArrayFormat.Tabular)
// items[3]:
//   - scala
//   - zio
//   - functional
// members[2]{id,name}:
//   1,Alice
//   2,Bob
```

:::caution[Known bugs in 0.0.56]
Three known bugs affect tables in 0.0.56. A record with a non-primitive field, such as `tags: List[String]`, is not written in list form as the `ArrayFormat.Tabular` Scaladoc says; the codec writes a row that it cannot decode ([#1778](https://github.com/zio/zio-blocks/issues/1778)). A top-level `List` of records written as a table cannot be decoded, even when the same codec wrote it ([#1779](https://github.com/zio/zio-blocks/issues/1779)). And `ToonCodecDeriver.withFieldNameMapper` is not applied to the field names in a table header ([#1780](https://github.com/zio/zio-blocks/issues/1780)).
:::

### List Format

`ArrayFormat.List` writes every element as a `- ` item on its own line, under a header that keeps the length, such as `items[3]:`:

```scala
encodeBoth(ArrayFormat.List)
// items[3]:
//   - scala
//   - zio
//   - functional
// members[2]:
//   - id: 1
//     name: Alice
//   - id: 2
//     name: Bob
```

### Auto Format

`Auto` is the default. It chooses the format from the element type: arrays of primitive values are written inline, and arrays of records are written as lists. The one exception is an array of flat records (records whose fields are all primitives) nested inside a list item, which is written as a table. To write record arrays as tables everywhere, use `ArrayFormat.Tabular`:

```scala
encodeBoth(ArrayFormat.Auto)
// items[3]: scala,zio,functional
// members[2]:
//   - id: 1
//     name: Alice
//   - id: 2
//     name: Bob
```

---

## Delimiter

Specifies the separator character for inline arrays and table rows.

### Overview

`Delimiter` has three values for writing: `Comma` (the default), `Tab` and `Pipe`; a fourth, `Delimiter.None`, matches no character and is used inside the reader. For a derived codec, set the delimiter with `ToonCodecDeriver.withDelimiter`. `WriterConfig.withDelimiter` applies only to the `DynamicValue` codec, and `ReaderConfig.withDelimiter` has no effect on decoding arrays.

### Available Delimiters

To encode the same tags with each of the three delimiters:

```scala
List(Delimiter.Comma, Delimiter.Tab, Delimiter.Pipe).foreach { delimiter =>
  println(Tags.schema.derive(ToonCodecDeriver.withDelimiter(delimiter)).encodeToString(tags))
}
// items[3]: scala,zio,functional
// items[3	]: scala	zio	functional
// items[3|]: scala|zio|functional
```

A delimiter other than the comma is written into the array header, as in `items[3|]:`. The reader takes the delimiter from there, so a codec with the default settings decodes the pipe-separated output:

```scala
Tags.schema.derive(ToonFormat).decode("items[3|]: scala|zio|functional")
// res14: Either[SchemaError, Tags] = Right(
//   Tags(List("scala", "zio", "functional"))
// )
```

---

## ToonReader

Low-level, line-oriented text parser implementing TOON format with state management and efficient token scanning.

### Overview

`ToonReader` provides stateful reading of TOON-encoded data with automatic token recognition and error reporting by path (for example `at: .port`). It handles indentation tracking, quoted string parsing, and numeric value extraction.

### Reading from Byte Array

To parse TOON data from a byte array:

```scala

case class Person(name: String, age: Int)
object Person { implicit val schema: Schema[Person] = Schema.derived }

val toonBytes = "name: Alice\nage: 30".getBytes("UTF-8")
// Use the codec API for reading (ToonReader is an internal implementation detail)
val codec = Person.schema.derive(ToonFormat)
val result = codec.decode(toonBytes)
```

### Low-Level Parsing

Direct token-by-token parsing for custom scenarios:

```scala

// ToonReader is an internal implementation detail of the codec
// Use the public codec API instead:
case class Data(id: Int, name: String)
object Data { implicit val schema: Schema[Data] = Schema.derived }

val codec = Data.schema.derive(ToonFormat)
// Codec provides token-aware parsing and composition for reading
val toonString = "id: 1\nname: test"
val result = codec.decode(toonString)
```

---

## ToonWriter

Low-level, line-oriented text writer implementing TOON format with optimization for compact representation.

### Overview

`ToonWriter` provides efficient writing of Scala values into TOON text with automatic formatting, indentation, and type-specific encoding.

### Writing to Byte Array

To encode values to TOON format:

```scala

```

### Custom Encoding Strategies

The writer selects encoding strategies based on value type and configuration:

```scala

```

---

## NameMapper

Transformation strategies for field and case names during encoding and decoding.

### Overview

`NameMapper` provides predefined transformation functions for converting between Scala naming conventions and TOON field names. This enables encoding data with different naming styles without changing your Scala types.

### Identity Mapping

Use Scala field names as-is without transformation:

```scala

val deriver = ToonCodecDeriver.withFieldNameMapper(NameMapper.Identity)
// firstName → firstName
```

### Snake Case Mapping

Convert camelCase field names to snake_case:

```scala

val deriver = ToonCodecDeriver.withFieldNameMapper(NameMapper.SnakeCase)
// firstName → first_name
// emailAddress → email_address
```

### Kebab Case Mapping

Convert camelCase field names to kebab-case:

```scala

val deriver = ToonCodecDeriver.withFieldNameMapper(NameMapper.KebabCase)
// firstName → first-name
// emailAddress → email-address
```

---

## DiscriminatorKind

Controls how variant (sealed trait) discriminators appear in TOON output.

### Overview

`DiscriminatorKind` determines whether variant types are identified by a key discriminator, a field discriminator, or no discriminator at all.

### Key Discriminator

Use the variant case name as the TOON key (default):

```scala

val deriver = ToonCodecDeriver.withDiscriminatorKind(DiscriminatorKind.Key)
// Running:
//   pid: 42
```

### Field Discriminator

Include a dedicated field for the variant type:

```scala

val deriver = ToonCodecDeriver.withDiscriminatorKind(DiscriminatorKind.Field("type"))
// type: Running
// pid: 42
```

### No Discriminator

Omit discriminators entirely (for unambiguous contexts):

```scala

val deriver = ToonCodecDeriver.withDiscriminatorKind(DiscriminatorKind.None)
// pid: 42
```

:::caution[Known bugs in 0.0.56]
Two known bugs affect `DiscriminatorKind.Field` and `DiscriminatorKind.None`. With `DiscriminatorKind.Field`, a case object of a sealed trait that also has case classes is written as an empty string, with no `type: Idle` line, so decoding the output fails with `Missing discriminator field: type`. And a variant stored in a record field is written with its first line on the key's line and its other fields at the level of the parent record, so decoding treats every field after it as part of the variant, and a later field fails with `Missing required field`. The default `DiscriminatorKind.Key` round-trips both cases.
:::

---

## KeyFolding

Strategy for folding chains of nested single-field records into dot-separated keys during encoding.

### Overview

`KeyFolding` is set with `WriterConfig.withKeyFolding`. Only the `DynamicValue` codec, `Schema[DynamicValue].derive(ToonFormat)`, reads it; a codec derived for a case class ignores it and always writes nested records as indented blocks. Folding follows records that have exactly one field: below, `settings` holds a record whose only field is `server`, which holds a record whose only field is `port`, so the three nested lines become `settings.server.port: 8080`. A field whose record has two or more fields stays an indented block.

The examples in this section use a shop whose settings nest single-field records:

```scala

case class Server(port: Int)
object Server { implicit val schema: Schema[Server] = Schema.derived }

case class Settings(server: Server)
object Settings { implicit val schema: Schema[Settings] = Schema.derived }

case class Shop(name: String, settings: Settings)
object Shop { implicit val schema: Schema[Shop] = Schema.derived }

val shop                    = Shop("bookshop", Settings(Server(8080)))
val shopValue: DynamicValue = Shop.schema.toDynamicValue(shop)

val dynamicCodec = Schema[DynamicValue].derive(ToonFormat)
val shopCodec    = Shop.schema.derive(ToonFormat)
```

### Off (No Folding)

`KeyFolding.Off`, the default, writes every nested record as an indented block:

```scala
println(dynamicCodec.encodeToString(shopValue, WriterConfig.withKeyFolding(KeyFolding.Off)))
// name: bookshop
// settings:
//   server:
//     port: 8080
```

### Safe Folding

`KeyFolding.Safe` folds the chain into one dotted key when the `DynamicValue` codec writes it, while the codec derived for `Shop` ignores the setting:

```scala
println(dynamicCodec.encodeToString(shopValue, WriterConfig.withKeyFolding(KeyFolding.Safe)))
// name: bookshop
// settings.server.port: 8080

println(shopCodec.encodeToString(shop, WriterConfig.withKeyFolding(KeyFolding.Safe)))
// name: bookshop
// settings:
//   server:
//     port: 8080
```

---

## PathExpansion

Strategy for parsing dot-separated keys as nested records during decoding.

### Overview

`PathExpansion` is set with `ReaderConfig.withExpandPaths` and controls whether a key such as `settings.server.port` is read as nested records or as one key with dots in its name. Like key folding, only the `DynamicValue` codec reads it. To decode folded TOON into a case class, decode it as a `DynamicValue` with path expansion, then convert the result with `Schema#fromDynamicValue`.

The examples below decode this folded document:

```scala
val folded = "name: bookshop\nsettings.server.port: 8080"
// folded: String = """name: bookshop
// settings.server.port: 8080"""
```

### Off (No Expansion)

`PathExpansion.Off`, the default, keeps the dotted key as one field name:

```scala
println(dynamicCodec.decode(folded, ReaderConfig.withExpandPaths(PathExpansion.Off)))
// Right({
//   name: "bookshop",
//   `settings.server.port`: 8080
// })
```

### Safe Expansion

`PathExpansion.Safe` expands the dotted key into nested records. The codec derived for `Shop` ignores the setting and fails, but the expanded `DynamicValue` converts to a `Shop`:

```scala
println(dynamicCodec.decode(folded, ReaderConfig.withExpandPaths(PathExpansion.Safe)))
// Right({
//   name: "bookshop",
//   settings: {
//     server: {
//       port: 8080
//     }
//   }
// })

println(shopCodec.decode(folded, ReaderConfig.withExpandPaths(PathExpansion.Safe)))
// Left(zio.blocks.schema.SchemaError: Missing required field: settings at: .)

println(dynamicCodec.decode(folded, ReaderConfig.withExpandPaths(PathExpansion.Safe)).flatMap(Shop.schema.fromDynamicValue))
// Right(Shop(bookshop,Settings(Server(8080))))
```

---

## Integration Points

The TOON codec module integrates seamlessly with ZIO Schema and other ZIO Blocks modules:

**With ZIO Schema:**
- The module uses `Schema[A]` to derive codecs automatically
- Supports all schema types: primitives, records, variants, sequences, maps, options
- Integrates through the standard `BinaryFormat` mechanism

**Internal Type Relationships:**
- `ToonCodecDeriver` drives all codec derivation, through `Schema#derive(ToonCodecDeriver)` or `Schema#derive(ToonFormat)`
- `WriterConfig` controls `ToonWriter` behavior during encoding
- `ReaderConfig` controls `ToonReader` behavior during decoding
- `NameMapper`, `ArrayFormat`, `Delimiter`, `DiscriminatorKind` all flow through `ToonCodecDeriver` configuration
- `KeyFolding` and `PathExpansion` affect how the `DynamicValue` codec writes and reads nested records

**With Other Modules:**
- TOON codecs work alongside other codec modules (JSON, Thrift, YAML, MessagePack) via the format system
- Can be combined with `Schema` evolution utilities for schema versioning
- Integrates with `DynamicValue` for generic data handling

---

## Error Handling

TOON decoding errors include location traces showing the path through nested structures where the error occurred.

### Understanding Error Traces

Errors render as `DynamicOptic` paths such as `.field.at(0).nested.value`: a record field is `.name`, a sequence index is `.at(0)`, a map value is `.atKey(...)` and a variant case is `.when[Case]`. The path shows exactly where decoding failed. Here a list field holds a plain value instead of an array:

```scala

case class Team(name: String, members: List[String])

object Team {
  implicit val schema: Schema[Team] = Schema.derived
}

val invalidToon = """
name: Engineering
members: invalid
"""
// invalidToon: String = """
// name: Engineering
// members: invalid
// """

Team.schema.derive(ToonFormat).decode(invalidToon) match {
  case Right(team) => println(s"Loaded: $team")
  case Left(error) => println(s"Error: ${error.getMessage}")
}
// Error: Expected array header with [ at: .members
```

### Zero-Overhead Error Handling

Errors use zero-overhead exceptions (no stack traces) for efficient error reporting in scenarios where errors are expected and handled inline.

---

## XML Codec Module

`Xml` is a **sealed trait representing XML nodes**. It provides a type-safe, immutable representation of all valid XML document structures including elements, text nodes, CDATA sections, comments, and processing instructions.

```scala
sealed trait Xml {
  def xmlType: XmlType
  def is(xmlType: XmlType): Boolean
  def as(xmlType: XmlType): Option[xmlType.Type]
  def unwrap(xmlType: XmlType): Option[xmlType.Unwrap]
  def print: String
  def printPretty: String
  def select: XmlSelection
}
```

`Xml` supports:
- Type-safe representation of all XML node types
- Fluent navigation and querying with XmlSelection
- Schema-derived automatic codec generation
- Zero external dependencies
- Full cross-platform support (JVM and Scala.js)

## Overview

Zero-dependency XML codec for ZIO Blocks Schema with cross-platform support.

The schema-xml module provides automatic XML codec derivation for any type with a `Schema`. It includes a complete XML AST, fluent navigation API, and support for XML-specific features like attributes and namespaces.

Key features:

- **Zero Dependencies**: No external XML libraries required
- **Cross-Platform**: Full support for JVM and Scala.js
- **Schema-Based**: Automatic codec derivation from Schema definitions
- **XML AST**: Complete representation of XML documents
- **Fluent API**: Navigation and transformation with XmlSelection
- **Attributes**: First-class support for XML attributes via annotations
- **Namespaces**: XML namespace support with prefix handling

## Installation

To use the schema-xml module, add the following dependency to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema-xml" % "0.0.56"
```

## Basic Usage

Start by deriving an XML codec from your Schema definition:

### Deriving Codecs

To create an XML codec, use `Schema[A].derive(XmlFormat)`:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Derive XML codec using the unified format API
val codec = Schema[Person].derive(XmlFormat)
```

### Encoding to XML

Encode your values to XML using the codec:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(XmlFormat)
val person = Person("Alice", 30)
```

Encode to XML bytes:

```scala
val bytes: Array[Byte] = codec.encode(person)
// bytes: Array[Byte] = Array(
//   60,
//   80,
//   101,
//   114,
//   115,
//   111,
//   110,
//   62,
//   60,
//   110,
//   97,
//   109,
//   101,
//   62,
//   65,
//   108,
//   105,
//   99,
//   101,
//   60,
//   47,
//   110,
//   97,
//   109,
//   101,
//   62,
//   60,
//   97,
//   103,
//   101,
//   62,
//   51,
//   48,
//   60,
//   47,
//   97,
//   103,
//   101,
//   62,
//   60,
//   47,
//   80,
//   101,
//   114,
//   115,
//   111,
//   110,
//   62
// )
```

Encode to XML string:

```scala
val xmlString: String = codec.encodeToString(person)
// xmlString: String = "<Person><name>Alice</name><age>30</age></Person>"
```

Encode to pretty-printed XML:

```scala
val prettyXml = codec.encodeToString(person, WriterConfig.pretty)
// prettyXml: String = """<Person>
//   <name>Alice</name>
//   <age>30</age>
// </Person>"""
```

### Decoding from XML

Decode XML strings or bytes back to your typed values:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(XmlFormat)

// Decode from XML string
val xml = "<Person><name>Alice</name><age>30</age></Person>"
```

Decode the XML string and see the result:

```scala
val result: Either[SchemaError, Person] = codec.decode(xml)
// result: Either[SchemaError, Person] = Right(
//   Person(name = "Alice", age = 30)
// )
```

You can also decode from bytes:

```scala
val bytes = xml.getBytes("UTF-8")
```

```scala
val fromBytes: Either[SchemaError, Person] = codec.decode(bytes)
// fromBytes: Either[SchemaError, Person] = Right(
//   Person(name = "Alice", age = 30)
// )
```

## XML AST

The `Xml` ADT represents all valid XML node types:

```
Xml
 ├── Xml.Element          (element with name, attributes, children)
 ├── Xml.Text             (character data)
 ├── Xml.CData            (unparsed character data)
 ├── Xml.Comment          (XML comment)
 └── Xml.ProcessingInstruction (processing instruction)
```

### Creating XML Nodes

Construct XML nodes directly using the case class constructors:

```scala

// Create elements
val simple = Xml.Element("person")
val withChildren = Xml.Element("person", Xml.Text("Alice"))

// Create with XmlName (for namespaces)
val namespaced = Xml.Element(
  XmlName("person", "http://example.com"),
  Chunk.empty,
  Chunk.empty
)

// Create text nodes
val text = Xml.Text("Hello, World!")
val cdata = Xml.CData("<script>...</script>")

// Create comments and processing instructions
val comment = Xml.Comment("This is a comment")
val pi = Xml.ProcessingInstruction("xml-stylesheet", "href=\"style.css\"")
```

### XmlName

`XmlName` represents an element or attribute name with optional namespace. Create instances with different namespace configurations:

```scala

// Local name only
val simple = XmlName("person")
val localName = simple.localName     // "person"
val namespace = simple.namespace     // None
val prefix = simple.prefix        // None

// With namespace URI
val ns = XmlName("person", "http://example.com/ns")

// With prefix (for prefixed elements like atom:feed)
val prefixed = XmlName("feed", Some("atom"), None)
val prefixedName = prefixed.localName              // "feed"
prefixed.prefix.contains("atom") // true
```

## XmlBuilder

Construct XML documents programmatically with a fluent API:

```scala

// Build an element with attributes and children
val doc = XmlBuilder.element("person")
  .attr("id", "123")
  .attr("status", "active")
  .child(XmlBuilder.element("name").text("Alice").build)
  .child(XmlBuilder.element("age").text("30").build)
  .build

// Result (compact; nothing pretty-prints it unless you ask):
// <person id="123" status="active"><name>Alice</name><age>30</age></person>

// Create other node types
val textNode = XmlBuilder.text("content")
val cdataNode = XmlBuilder.cdata("raw content")
val commentNode = XmlBuilder.comment("comment text")
```

## Configuration

The schema-xml module provides configuration options for both parsing and writing:

### WriterConfig

Use `WriterConfig` to control XML output formatting:

```scala

// Compact output (default)
val compact = WriterConfig.default
// <Person><name>Alice</name></Person>

// Pretty-printed with 2-space indentation
val pretty = WriterConfig.pretty
// <Person>
//   <name>Alice</name>
// </Person>

// With XML declaration
val withDecl = WriterConfig.withDeclaration
// <?xml version="1.0" encoding="UTF-8"?>
// <Person><name>Alice</name></Person>

// Custom configuration
val custom = WriterConfig(
  indentStep = 4,
  includeDeclaration = true,
  encoding = "UTF-8"
)
```

| Option | Default | Description |
|--------|---------|-------------|
| `indentStep` | `0` | Spaces per indentation level (0 = compact) |
| `includeDeclaration` | `false` | Include XML declaration |
| `encoding` | `"UTF-8"` | Character encoding in declaration |

### ReaderConfig

Controls XML parsing behavior and security limits:

```scala

// Default configuration
val default = ReaderConfig.default

// Custom limits
val custom = ReaderConfig(
  maxDepth = 100,
  maxAttributes = 50,
  maxTextLength = 1000000,
  preserveWhitespace = true
)
```

| Option | Default | Description |
|--------|---------|-------------|
| `maxDepth` | `1000` | Maximum element nesting depth |
| `maxAttributes` | `1000` | Maximum attributes per element |
| `maxTextLength` | `10000000` | Maximum text content length |
| `preserveWhitespace` | `false` | Preserve whitespace in text nodes |

## Attributes

A field is encoded as an XML attribute when it carries `Modifier.config("xml.attribute", name)`; an empty name uses the field name:

```scala

case class Person(
  @Modifier.config("xml.attribute", "") id: String,
  @Modifier.config("xml.attribute", "status") active: String,
  name: String,
  age: Int
)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(XmlFormat)
val person = Person("123", "active", "Alice", 30)
```

Encode the person; `id` and `status` are attributes of `Person`:

```scala
val xml = codec.encodeToString(person)
// xml: String = "<Person id=\"123\" status=\"active\"><name>Alice</name><age>30</age></Person>"
```

The value of the modifier is the attribute name:

- `@Modifier.config("xml.attribute", "")` uses the field name as the attribute name
- `@Modifier.config("xml.attribute", "customName")` uses the provided name as the attribute name

:::warning[`@xmlAttribute` has no effect in 0.0.56]
The `@xmlAttribute` and `@xmlNamespace` annotations exist, but `Schema.derived` records only `Modifier` annotations, so it drops them: a field marked `@xmlAttribute()` is still written as a child element. This is a known bug in 0.0.56. Use the `Modifier.config` form shown on this page, which the codec reads. On decoding, a missing attribute becomes `None` for an `Option` field and fails with `Missing required attribute: <name>` otherwise.
:::

## Namespaces

A record is written in an XML namespace when it carries the modifiers `Modifier.config("xml.namespace.uri", uri)` and, optionally, `Modifier.config("xml.namespace.prefix", prefix)`:

```scala

@Modifier.config("xml.namespace.uri", "http://www.w3.org/2005/Atom")
@Modifier.config("xml.namespace.prefix", "atom")
case class Feed(
  title: String,
  updated: String
)

object Feed {
  implicit val schema: Schema[Feed] = Schema.derived
}

val codec = Schema[Feed].derive(XmlFormat)
val feed = Feed("My Blog", "2024-01-01T00:00:00Z")
```

Encode the feed with a namespace prefix. The element is `atom:Feed` and carries an `xmlns:atom` attribute:

```scala
val xml = codec.encodeToString(feed)
// xml: String = "<atom:Feed xmlns:atom=\"http://www.w3.org/2005/Atom\"><title>My Blog</title><updated>2024-01-01T00:00:00Z</updated></atom:Feed>"
```

Without a prefix, the record gets a default `xmlns` attribute:

```scala
@Modifier.config("xml.namespace.uri", "http://www.w3.org/2005/Atom")
case class Feed(title: String)

object Feed {
  implicit val schema: Schema[Feed] = Schema.derived
}

val codec = Schema[Feed].derive(XmlFormat)
val feed = Feed("My Blog")
```

Encode the feed with the default namespace:

```scala
val xml = codec.encodeToString(feed)
// xml: String = "<Feed xmlns=\"http://www.w3.org/2005/Atom\"><title>My Blog</title></Feed>"
```

Decoding does not check or resolve namespaces: `XmlReader` keeps the prefix in `XmlName#prefix`, leaves `XmlName#namespace` empty and keeps `xmlns` declarations as ordinary attributes, so a namespaced element read back has no `namespace` (a known bug in 0.0.56).

## XmlSelection

`XmlSelection` provides a fluent API for navigating and querying XML structures. `Xml#select` selects the root element itself, so `get(name)` looks at the root's children: for a `<library>` document, start with `get("books")`, not `get("library")`.

### Navigation

Navigate to child elements, filter by type, and extract content:

```scala

val xml = XmlReader.read("""
<library>
  <books>
    <book id="1">
      <title>Functional Programming</title>
      <author>Alice</author>
    </book>
    <book id="2">
      <title>Advanced Scala</title>
      <author>Bob</author>
    </book>
  </books>
</library>
""")

// Navigate to child elements
val books = xml.select.get("books")

// Pick the first of the selected books
val firstBook = XmlSelection.succeed(books.get("book").toChunk.head)
```

Extract text content from the first book:

```scala
val title: Either[SchemaError, String] = firstBook.get("title").text
// title: Either[SchemaError, String] = Right("Functional Programming")
```

`XmlSelection#apply(index)` is not "the nth selected value": it returns the child at that index of every selected element. To pick one element of a selection, take it from `toChunk` as above.

You can also navigate descendants recursively:

```scala
val allTitles = xml.select.descendant("title")
// allTitles: XmlSelection = XmlSelection(
//   Right(
//     IndexedSeq(
//       Element(
//         name = XmlName(localName = "title", prefix = None, namespace = None),
//         attributes = IndexedSeq(),
//         children = IndexedSeq(Text("Functional Programming"))
//       ),
//       Element(
//         name = XmlName(localName = "title", prefix = None, namespace = None),
//         attributes = IndexedSeq(),
//         children = IndexedSeq(Text("Advanced Scala"))
//       )
//     )
//   )
// )
```

### Filtering

Filter selections by node type or custom predicates:

```scala

val xml = XmlReader.read("<root><item>text1</item><comment>note</comment></root>")
val selection = xml.select.descendant("item")

// Filter by type
val elements = selection.elements
val texts = selection.texts
val comments = selection.comments

// Custom filtering
val filtered = selection.filter(xml => xml.is(XmlType.Element))
```

### Terminal Operations

Execute a selection to extract values or convert to other formats:

```scala

val xml = XmlReader.read("<root><item>A</item><item>B</item></root>")
val selection = xml.select.get("item")

// Get single value (fails if not exactly one)
val one: Either[SchemaError, Xml] = selection.one

// Get any value (first of many)
val any: Either[SchemaError, Xml] = selection.any

// Get all values as a single XML element
val all: Either[SchemaError, Xml] = selection.all

// Convert to chunk
val chunk = selection.toChunk

// Extract text content
val text: Either[SchemaError, String] = selection.text
val allText: String = selection.textContent
```

### Combinators

Combine and transform selections using monadic operations:

```scala

val xml1 = XmlReader.read("<root><item>X</item></root>")
val xml2 = XmlReader.read("<root><item>Y</item></root>")
val selection1 = xml1.select.get("item")
val selection2 = xml2.select.get("item")

// Map over selections
val mapped = selection1.map(xml => xml)

// FlatMap for chaining
val nested = selection1.flatMap(xml => XmlSelection.succeed(xml))

// Combine selections
val combined = selection1 ++ selection2

// Alternative on failure
val withFallback = selection1.orElse(selection2)
```

## XmlPatch

`XmlPatch` provides composable XML modification operations. A patch path starts at the children of the root element, as `XmlSelection` does: for `<person>...</person>`, `DynamicOptic.root` is the `<person>` element itself and `p".name"` is its `<name>` child. A path that starts with the root's own name, such as `p".person"`, fails with `Element 'person' not found`.

### Creating Patches

Create patches for add, remove, replace, and attribute operations:

```scala

// Patch paths start at the children of the root element; DynamicOptic.root is the root itself
val path = p".books.book"

// Add content
val addPatch = XmlPatch.add(
  path,
  XmlBuilder.element("book").attr("id", "3").build,
  XmlPatch.Position.AppendChild
)

// Remove element
val removePatch = XmlPatch.remove(path)

// Replace element
val replacePatch = XmlPatch.replace(
  path,
  XmlBuilder.element("book").attr("id", "999").build
)

// Set attribute
val attrPatch = XmlPatch.setAttribute(path, "featured", "true")

// Remove attribute
val removeAttrPatch = XmlPatch.removeAttribute(path, "id")
```

### Position Options

Position options control where new content is inserted relative to the target:

```scala

val before = Position.Before         // Insert before the target element
val after = Position.After          // Insert after the target element
val prepend = Position.PrependChild   // Insert as first child of target
val append = Position.AppendChild    // Insert as last child of target
```

### Applying Patches

Apply a patch to an XML document to produce a modified result:

```scala

val xml = XmlReader.read("<person><name>Alice</name></person>")
val patch = XmlPatch.setAttribute(DynamicOptic.root, "active", "true")

// Apply the patch
val result: Either[SchemaError, Xml] = patch(xml)
```

### Composing Patches

Combine multiple patches to apply transformations in sequence:

```scala

val patch1 = XmlPatch.setAttribute(DynamicOptic.root, "id", "123")
val patch2 = XmlPatch.add(
  DynamicOptic.root,
  XmlBuilder.element("email").text("alice@example.com").build,
  XmlPatch.Position.AppendChild
)

// Compose patches - applies patch1, then patch2
val combined = patch1 ++ patch2
```

## Extension Syntax

When a `Schema` is in scope, use convenient extension methods:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val person = Person("Alice", 30)

// Encode to XML AST
val xml: Xml = person.toXml

// Encode to XML string
val xmlString: String = person.toXmlString
// <Person><name>Alice</name><age>30</age></Person>

// Encode to bytes
val bytes: Array[Byte] = person.toXmlBytes

// Decode from XML string
val parsed: Either[SchemaError, Person] = 
  "<Person><name>Bob</name><age>25</age></Person>".fromXml[Person]

// Decode from bytes
val fromBytes: Either[SchemaError, Person] = bytes.fromXml[Person]
```

## Printing XML

Format XML documents using compact or pretty-printed output. You can convert XML to string with different formatting options:

```scala

val xml = Xml.Element("person", Xml.Element("name", Xml.Text("Alice")))
```

Compact output:

```scala
val compact: String = xml.print
// compact: String = "<person><name>Alice</name></person>"
```

Pretty-printed output:

```scala
val pretty: String = xml.printPretty
// pretty: String = """<person>
//   <name>Alice</name>
// </person>"""
```

Custom configuration:

```scala
val custom: String = xml.print(WriterConfig(indentStep = 4))
```

## Type Testing and Access

Test and extract values from XML nodes using type guards and unwrapping:

```scala

val xml: Xml = Xml.Element("person")

// Type testing
xml.is(XmlType.Element)  // true
xml.is(XmlType.Text)     // false

// Type narrowing (returns Option)
val elem: Option[Xml.Element] = xml.as(XmlType.Element)

// Value extraction (returns Option)
val (name, attrs, children) = xml.unwrap(XmlType.Element).get
```

## Supported Types

All standard ZIO Blocks Schema types are supported:

**Numeric Types**:
- `Boolean`, `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `Char`
- `BigInt`, `BigDecimal`

**Text Types**:
- `String`

**Special Types**:
- `Unit`, `UUID`, `Currency`

**Java Time Types**:
- `Instant`, `LocalDate`, `LocalTime`, `LocalDateTime`
- `OffsetTime`, `OffsetDateTime`, `ZonedDateTime`
- `Duration`, `Period`
- `Year`, `YearMonth`, `MonthDay`
- `DayOfWeek`, `Month`
- `ZoneId`, `ZoneOffset`

**Composite Types**:
- Records (case classes)
- Variants (sealed traits), at the top level only: a sealed trait inside a record, sequence or map loses its case name and does not decode back (a known bug in 0.0.56)
- Sequences (`List`, `Vector`, `Set`, etc.)
- Maps (`Map[K, V]`)
- Options (`Option[A]`), except inside a sequence or a map, where an `Option` does not decode back (a known bug in 0.0.56)
- Wrappers (newtypes)

## XmlCodec

`XmlCodec[A]` is the low-level codec interface that bridges Schema definitions with XML serialization. While usually derived automatically, you can work with it directly:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Get the underlying XML codec
val codec: XmlCodec[Person] = Schema[Person].derive(XmlFormat)

// Encode to Xml directly
val person = Person("Alice", 30)
val xml: Xml = codec.encodeValue(person)

// Decode from Xml directly
val decoded: Person = codec.decodeValue(xml)
```

**XmlCodec supports all Schema types:**
- Primitives (Int, String, Boolean, etc.)
- Java time types (Instant, LocalDate, Duration, etc.)
- Records (case classes with field-level configuration)
- Variants (sealed traits with discriminators)
- Collections (List, Vector, Map, etc.)
- Optional fields (Option[A])
- Custom wrappers and dynamic values

## Error Handling

All decoding operations return `Either[SchemaError, A]`. The `SchemaError` type provides detailed error information:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(XmlFormat)

// Decoding invalid XML
val invalid = "<Person><name>Alice</name></Person>"  // missing age
val result = codec.decode(invalid)

result match {
  case Right(person) => println(s"Decoded: $person")
  case Left(error) =>
    println(s"Error: ${error.getMessage}")
    // Malformed XML reports (line: L, column: C); a bad value reports a path such as `at: .qty`; a missing field reports `Missing required field: age`
}
```

`SchemaError` provides detailed error information for debugging:

```scala

val error = SchemaError("Parse failed")

// Error message
val message: String = error.getMessage

// Get error message for inspection
val errorMsg: String = error.getMessage
```

Error handling best practices:

```scala

case class Config(database: String, port: Int)
object Config {
  implicit val schema: Schema[Config] = Schema.derived
}

def loadConfig(xml: String): Either[String, Config] = {
  val codec = Schema[Config].derive(XmlFormat)
  codec.decode(xml).left.map { error =>
    s"Configuration error: ${error.getMessage}\nCheck XML format and required fields."
  }
}
```

## Cross-Platform Support

The XML module works across all platforms:

- **JVM** - Full functionality
- **Scala.js** - Browser and Node.js

All features including parsing, writing, navigation, and patching work identically on both platforms.

## Examples

These examples demonstrate common use cases and patterns with the XML module:

### Complete Example with Attributes and Namespaces

Define a schema with attributes and namespaces, then encode and decode:

```scala

@Modifier.config("xml.namespace.uri", "http://www.w3.org/2005/Atom")
@Modifier.config("xml.namespace.prefix", "atom")
case class Entry(
  @Modifier.config("xml.attribute", "") id: String,
  title: String,
  updated: String,
  author: Author
)

case class Author(name: String, email: String)

object Entry {
  implicit val authorSchema: Schema[Author] = Schema.derived
  implicit val schema: Schema[Entry] = Schema.derived
}

val codec = Schema[Entry].derive(XmlFormat)

val entry = Entry(
  id = "entry-1",
  title = "First Post",
  updated = "2024-01-01T00:00:00Z",
  author = Author("Alice", "alice@example.com")
)
```

Encode the entry with pretty printing:

```scala
val xmlStr = codec.encodeToString(entry, WriterConfig.pretty)
// xmlStr: String = """<atom:Entry xmlns:atom="http://www.w3.org/2005/Atom" id="entry-1">
//   <title>First Post</title>
//   <updated>2024-01-01T00:00:00Z</updated>
//   <author>
//     <name>Alice</name>
//     <email>alice@example.com</email>
//   </author>
// </atom:Entry>"""
```

Decode the XML back to a typed value. Decoding ignores the namespace, so it reads the `atom:Entry` element as an `Entry`:

```scala
val decoded = codec.decode(xmlStr)
// decoded: Either[SchemaError, Entry] = Right(
//   Entry(
//     id = "entry-1",
//     title = "First Post",
//     updated = "2024-01-01T00:00:00Z",
//     author = Author(name = "Alice", email = "alice@example.com")
//   )
// )
```

### Navigation and Transformation

Find elements, extract data, and apply patches to modify XML. Selections and patch paths start below the root element, so the `<library>` name does not appear in them:

```scala

val xmlString = """
<library>
  <books>
    <book id="1">
      <title>Functional Programming</title>
      <price>49.99</price>
    </book>
    <book id="2">
      <title>Advanced Scala</title>
      <price>59.99</price>
    </book>
  </books>
</library>
"""

val xml = XmlReader.read(xmlString)

// Find all books
val books = xml.select.get("books").get("book")

// Extract all titles
val titles = books.get("title").toChunk.map { titleElem =>
  titleElem.asInstanceOf[Xml.Element].children.head match {
    case Xml.Text(text) => text
    case _ => ""
  }
}

// Apply a patch to add a new book
val patch = XmlPatch.add(
  p".books",
  XmlBuilder.element("book")
    .attr("id", "3")
    .child(XmlBuilder.element("title").text("ZIO Essentials").build)
    .child(XmlBuilder.element("price").text("39.99").build)
    .build,
  XmlPatch.Position.AppendChild
)

val updated = patch(xml)
```

The extracted titles and the patched document:

```scala
println(titles)
// Chunk(Functional Programming,Advanced Scala)
println(updated.map(_.print))
// Right(<library><books><book id="1"><title>Functional Programming</title><price>49.99</price></book><book id="2"><title>Advanced Scala</title><price>59.99</price></book><book id="3"><title>ZIO Essentials</title><price>39.99</price></book></books></library>)
```

## Real-World Examples

Learn by examining practical examples of XML codecs in action:

### RSS Feed Parsing

Parse an RSS-style feed by defining a schema and decoding XML. The codec writes each element of a list as an `item` element under the field's element, and decoding ignores the `rss:` namespace, so the input below decodes into a `Channel`:

```scala

case class Item(
  guid: String,
  title: String,
  link: String,
  pubDate: String,
  description: String
)

@Modifier.config("xml.namespace.uri", "http://www.rss.org/")
@Modifier.config("xml.namespace.prefix", "rss")
case class Channel(
  title: String,
  link: String,
  description: String,
  items: List[Item]
)

object Channel {
  implicit val itemSchema: Schema[Item] = Schema.derived
  implicit val schema: Schema[Channel] = Schema.derived
}

// Parse RSS feed from string
val feedXml = """<rss:Channel xmlns:rss="http://www.rss.org/">
  <title>Tech Blog</title>
  <link>https://example.com</link>
  <description>Latest tech articles</description>
  <items>
    <item>
      <guid>1</guid>
      <title>Functional Programming in Scala</title>
      <link>https://example.com/fp</link>
      <pubDate>2024-01-01</pubDate>
      <description>Deep dive into FP concepts</description>
    </item>
  </items>
</rss:Channel>"""
// feedXml: String = """<rss:Channel xmlns:rss="http://www.rss.org/">
//   <title>Tech Blog</title>
//   <link>https://example.com</link>
//   <description>Latest tech articles</description>
//   <items>
//     <item>
//       <guid>1</guid>
//       <title>Functional Programming in Scala</title>
//       <link>https://example.com/fp</link>
//       <pubDate>2024-01-01</pubDate>
//       <description>Deep dive into FP concepts</description>
//     </item>
//   </items>
// </rss:Channel>"""

val codec = Schema[Channel].derive(XmlFormat)
// codec: XmlCodec[Channel] = zio.blocks.schema.xml.XmlCodecDeriver$$anon$3@75f69ffe
val result: Either[SchemaError, Channel] = codec.decode(feedXml)
// result: Either[SchemaError, Channel] = Right(
//   Channel(
//     title = "Tech Blog",
//     link = "https://example.com",
//     description = "Latest tech articles",
//     items = List(
//       Item(
//         guid = "1",
//         title = "Functional Programming in Scala",
//         link = "https://example.com/fp",
//         pubDate = "2024-01-01",
//         description = "Deep dive into FP concepts"
//       )
//     )
//   )
// )

println(result)
// Right(Channel(Tech Blog,https://example.com,Latest tech articles,List(Item(1,Functional Programming in Scala,https://example.com/fp,2024-01-01,Deep dive into FP concepts))))
```

### Atom Feed with Advanced Features

Work with Atom feeds using attributes, multiple entries, and custom configurations. The output below shows that each list element is written as an `item` element, and that the nested record repeats the `xmlns:atom` declaration:

```scala

@Modifier.config("xml.namespace.uri", "http://www.w3.org/2005/Atom")
@Modifier.config("xml.namespace.prefix", "atom")
case class Entry(
  @Modifier.config("xml.attribute", "") id: String,
  title: String,
  author: String,
  updated: String,
  @Modifier.config("xml.attribute", "href") link: String
)

@Modifier.config("xml.namespace.uri", "http://www.w3.org/2005/Atom")
@Modifier.config("xml.namespace.prefix", "atom")
case class Feed(
  @Modifier.config("xml.attribute", "") id: String,
  title: String,
  updated: String,
  entries: List[Entry]
)

object Feed {
  implicit val entrySchema: Schema[Entry] = Schema.derived
  implicit val schema: Schema[Feed] = Schema.derived
}

// Encode feed to XML with custom formatting
val feed = Feed(
  id = "urn:uuid:60a76c80-d399-11d9-b91C-0003939e0af6",
  title = "Example Feed",
  updated = "2024-01-01T18:30:02Z",
  entries = List(
    Entry(
      id = "urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a",
      title = "Atom-Powered Robots Run Amok",
      author = "John Doe",
      updated = "2024-01-01T18:30:02Z",
      link = "http://example.org/2024/01/entry"
    )
  )
)
// feed: Feed = Feed(
//   id = "urn:uuid:60a76c80-d399-11d9-b91C-0003939e0af6",
//   title = "Example Feed",
//   updated = "2024-01-01T18:30:02Z",
//   entries = List(
//     Entry(
//       id = "urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a",
//       title = "Atom-Powered Robots Run Amok",
//       author = "John Doe",
//       updated = "2024-01-01T18:30:02Z",
//       link = "http://example.org/2024/01/entry"
//     )
//   )
// )

val codec = Schema[Feed].derive(XmlFormat)
// codec: XmlCodec[Feed] = zio.blocks.schema.xml.XmlCodecDeriver$$anon$3@6896ac9a
val xmlOutput = codec.encodeToString(feed, WriterConfig.pretty)
// xmlOutput: String = """<atom:Feed xmlns:atom="http://www.w3.org/2005/Atom" id="urn:uuid:60a76c80-d399-11d9-b91C-0003939e0af6">
//   <title>Example Feed</title>
//   <updated>2024-01-01T18:30:02Z</updated>
//   <entries>
//     <item xmlns:atom="http://www.w3.org/2005/Atom" id="urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a" href="http://example.org/2024/01/entry">
//       <title>Atom-Powered Robots Run Amok</title>
//       <author>John Doe</author>
//       <updated>2024-01-01T18:30:02Z</updated>
//     </item>
//   </entries>
// </atom:Feed>"""

println(xmlOutput)
// <atom:Feed xmlns:atom="http://www.w3.org/2005/Atom" id="urn:uuid:60a76c80-d399-11d9-b91C-0003939e0af6">
//   <title>Example Feed</title>
//   <updated>2024-01-01T18:30:02Z</updated>
//   <entries>
//     <item xmlns:atom="http://www.w3.org/2005/Atom" id="urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a" href="http://example.org/2024/01/entry">
//       <title>Atom-Powered Robots Run Amok</title>
//       <author>John Doe</author>
//       <updated>2024-01-01T18:30:02Z</updated>
//     </item>
//   </entries>
// </atom:Feed>
```

### Sitemap XML

Generate sitemap XML with URLs and optional metadata fields:

```scala

case class Url(
  loc: String,
  lastmod: Option[String],
  changefreq: Option[String],
  priority: Option[Double]
)

case class Urlset(
  urls: List[Url]
)

object Urlset {
  implicit val urlSchema: Schema[Url] = Schema.derived
  implicit val schema: Schema[Urlset] = Schema.derived
}

// Build and encode sitemap
val sitemap = Urlset(List(
  Url("https://example.com", Some("2024-01-01"), Some("monthly"), Some(1.0)),
  Url("https://example.com/about", Some("2024-01-01"), Some("monthly"), Some(0.8)),
  Url("https://example.com/contact", None, Some("monthly"), Some(0.5))
))

val codec = Schema[Urlset].derive(XmlFormat)
val sitemapXml = codec.encodeToString(sitemap, WriterConfig(
  indentStep = 2,
  includeDeclaration = true
))
```

---

## YAML Codec Module

`zio-blocks-schema-yaml` is a **schema-driven YAML codec module** for serializing and deserializing Scala types to and from YAML format. It provides encoding and decoding for the ZIO Blocks Schema primitive types, records, variants, sequences and maps, with optional JSON interoperability, configuration options for pretty-printing, and automatic schema generation.

The module integrates with a pure-Scala YAML parser and writer to provide human-readable serialization. Its reader is lenient: it does not report most YAML syntax errors, so read the [known limits](#known-limits) before you load files written by people.

See also the [How to Read and Write YAML Documents](../../../guides/read-and-write-yaml.md) guide, which reads YAML into case classes, writes CI-style YAML, converts it to JSON, and lists the YAML forms the reader does not support.

## Motivation

YAML is a human-readable data format that appears widely across configuration files, Kubernetes manifests, CI/CD pipelines, and data serialization. Manually writing YAML encoders and decoders is error-prone and repetitive, especially for complex types with records, nested structures, and sum types.

The YAML codec module provides:

- Full YAML type support (mappings, sequences, scalars, null)
- Automatic schema generation from Scala types
- Pretty-printed and compact formatting options
- JSON interoperability for seamless conversion
- Precise error reporting with location traces showing the path to errors
- Recursive type support with automatic cycle detection
- Multiple encoding paths: byte arrays and strings
- Multiple decoding paths: byte arrays and strings
- Cross-platform compatibility (JVM and Scala.js)

Rather than hand-writing YAML parsing logic or relying on external libraries with limited Scala support, you work with strongly-typed schemas that the compiler validates.

## Installation

Add the module to your `build.sbt`:

```sbt
libraryDependencies += "dev.zio" %% "zio-blocks-schema-yaml" % "0.0.56"
```

For Scala.js, use `%%%` instead of `%%`:

```sbt
libraryDependencies += "dev.zio" %%% "zio-blocks-schema-yaml" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x. On Scala 3.9 and newer, use the `zio-blocks-next-schema-yaml` module instead and add `scalacOptions += "-experimental"`, as [How to Add ZIO Blocks to Your Build](../../../getting-started/installation.md) explains.

## Introduction

The module provides a complete pipeline for YAML codec derivation and usage:

1. **Define your type**: Any Scala type with a `Schema` instance
2. **Derive a codec**: Use `Schema[A].derive(YamlFormat)` to obtain a `YamlCodec[A]`
3. **Encode or decode**: Call `codec.encodeToString(value)` or `codec.decode(yamlString)`
4. **Handle errors**: Catch `SchemaError` with location traces showing where the error occurred
5. **Configure output**: Use `YamlOptions` with `YamlWriter` for pretty-printing or compact formatting

The derivation process is automatic for all supported types (primitives, records, variants, sequences, maps). The module generates YAML from the schema and handles encoding and decoding without manual configuration.

## How They Work Together

The YAML codec pipeline flows through these layers:

```
1. User defines Schema[A] for their type
                 ↓
2. Schema[A].derive(YamlFormat) creates YamlCodec[A]
                 ↓
3. YamlCodecDeriver derives Encoder and Decoder implementations
   - For primitives: type-specific YAML scalar encoders/decoders
   - For records: field-by-field composition with YAML mapping
   - For variants: a one-key mapping named after the case
   - For sequences: YAML sequence encoding/decoding
   - For maps: YAML mapping encoding/decoding
                 ↓
4. YamlCodec provides multiple encoding paths
   - encodeToString(value) → String
   - encode(value) → Array[Byte]
                 ↓
5. YamlCodec provides multiple decoding paths
   - decode(yaml: String) → Either[SchemaError, A]
   - decode(bytes: Array[Byte]) → Either[SchemaError, A]
                 ↓
6. YamlOptions controls output formatting, applied by YamlWriter
   - Pretty-printing with document markers
   - Compact output without markers
   - Custom indentation and style
                 ↓
7. Yaml AST provides structured representation
   - Mapping (key-value pairs)
   - Sequence (ordered lists)
   - Scalar (string values with optional tags)
   - NullValue (YAML null)
                 ↓
8. Errors include location traces
   Shows path (.field.at(index).nested) to error location
```

**Typical workflow:**

A user type flows through the derivation and encoding pipeline as follows:

```
User type (e.g., case class Config)
    ↓
Schema.derived (automatic via macro)
    ↓
Schema[Config].derive(YamlFormat) → YamlCodec[Config]
    ↓
Use codec.encodeToString(config) to serialize → String
Use codec.decode(yamlString) to deserialize → Either[SchemaError, Config]
    ↓
Handle SchemaError with location trace on failure
```

### Type Relationships

- **`Yaml`**: Sealed trait AST representing YAML data structures
- **`YamlCodec[A]`**: Main public API; contains encoder and decoder for bidirectional serialization
- **`YamlCodecDeriver`**: Configuration and derivation system; generates codecs from Schema
- **`YamlOptions`**: Configuration for output formatting (pretty, compact, indentation)
- **`YamlReader`, `YamlWriter`**: Low-level YAML parsing and serialization

## Common Patterns

This section shows practical patterns for working with YAML codecs in real-world scenarios.

### Pattern 1: Derive and Encode a Configuration Record

To derive a YAML codec for a record and encode a value to a string:

```scala

case class AppConfig(name: String, port: Int, debug: Boolean)

object AppConfig {
  implicit val schema: Schema[AppConfig] = Schema.derived
}

val codec = AppConfig.schema.derive(YamlFormat)
```

`YamlCodec#encodeToString` writes the record as a block mapping, with one line per field:

```scala
println(codec.encodeToString(AppConfig("MyApp", 8080, debug = true)))
// name: MyApp
// port: 8080
// debug: true
```

### Pattern 2: Decode YAML with Error Handling

When decoding YAML data, errors include location traces showing where the problem occurred.

To decode YAML and handle errors with location information:

```scala

case class DatabaseConfig(host: String, port: Int, username: String)

object DatabaseConfig {
  implicit val schema: Schema[DatabaseConfig] = Schema.derived
}

val codec = DatabaseConfig.schema.derive(YamlFormat)
val yaml = """
host: localhost
port: invalid
username: admin
"""
```

`YamlCodec#decode` returns `Left` with the path of the field that failed:

```scala
codec.decode(yaml) match {
  case Right(config) => println(s"Loaded: $config")
  case Left(error)   => println(s"Error: ${error.getMessage}")
}
// Error: Invalid int value: invalid at: .port
```

### Pattern 3: Pretty-Print with Configuration

Use `YamlOptions` to control output formatting with indentation and document markers. `YamlCodec#encodeToString` takes no options, so encode the value to the `Yaml` tree with `YamlCodec#encodeValue`, then write the tree with `YamlWriter.write`.

To encode with pretty-printing:

```scala

case class Server(name: String, endpoints: List[String], timeout: Int)

object Server {
  implicit val schema: Schema[Server] = Schema.derived
}

val codec  = Server.schema.derive(YamlFormat)
val server = Server("api-server", List("GET /health", "POST /data"), 30)
```

`YamlOptions.pretty` turns on the leading `---` line:

```scala
println(YamlWriter.write(codec.encodeValue(server), YamlOptions.pretty))
// ---
// name: api-server
// endpoints:
//   - GET /health
//   - POST /data
// timeout: 30
```

### Pattern 4: Handle Recursive Types

Recursive types (types that reference themselves) are fully supported with automatic cycle detection.

To define and encode a recursive data structure:

```scala

sealed trait TreeNode
case class Leaf(value: String) extends TreeNode
case class Branch(label: String, children: List[TreeNode]) extends TreeNode

object TreeNode {
  implicit val schema: Schema[TreeNode] = Schema.derived
}

val codec = TreeNode.schema.derive(YamlFormat)
val tree: TreeNode = Branch("root", List(Leaf("a"), Branch("b", List(Leaf("c")))))
```

The encoded tree nests each `Branch` under its `children` key:

```scala
println(codec.encodeToString(tree))
// Branch:
//   label: root
//   children:
//     - Leaf:
//         value: a
//     - Branch:
//         label: b
//         children:
//           - Leaf:
//               value: c
```

### Pattern 5: YAML-JSON Interoperability

Convert between YAML and JSON representations seamlessly using `Yaml#toJson`.

To convert the YAML text to a `Json` value and print it:

```scala
println(YamlReader.read("name: Alice\nage: 30\n").toJson.print)
// {"name":"Alice","age":30}
```

---

## Yaml

Sealed trait representing a YAML node. Provides a complete AST for YAML data with four possible cases: `Mapping`, `Sequence`, `Scalar`, and `NullValue`.

### Overview

`Yaml` is a sealed trait that represents any valid YAML value. It can be constructed directly or derived from codecs.

### YAML AST Structure

To build a mapping and a sequence directly from the AST:

```scala

val mapping = Yaml.Mapping.fromStringKeys(
  "name" -> Yaml.Scalar("Alice"),
  "age"  -> Yaml.Scalar("30")
)

val sequence = Yaml.Sequence(
  Yaml.Scalar("item1"),
  Yaml.Scalar("item2")
)
```

`Yaml#print` writes the node on one document without a leading `---` line:

```scala
println(mapping.print)
// name: Alice
// age: "30"
println(sequence.print)
// - item1
// - item2
```

### Printing YAML

`Yaml#print` writes the compact form, and `Yaml#printPretty` writes the same node with a leading `---` line. A `Yaml.Scalar` that holds `"true"` has no boolean tag, so the writer quotes it:

```scala
val flags = Yaml.Mapping.fromStringKeys(
  "name"   -> Yaml.Scalar("Bob"),
  "active" -> Yaml.Scalar("true")
)
```

```scala
println(flags.print)
// name: Bob
// active: "true"
println(flags.printPretty)
// ---
// name: Bob
// active: "true"
```

A known bug in 0.0.56: `YamlReader` keeps the text of each scalar but not whether it was quoted, so a tree read from text quotes every number and boolean when you print it again. This changes the types of the fields in a read-modify-write:

```scala
println(YamlReader.read("replicas: 3\n").print)
// replicas: "3"
```

### Converting to JSON

`Yaml#toJson` transforms a `Yaml` node into a `Json` value for interoperability:

```scala
println(Yaml.Scalar("42").toJson.print)
// 42
```

---

## YamlCodec[A]

Main codec type for encoding and decoding values to and from YAML. Contains encoder and decoder for bidirectional serialization.

### Overview

`YamlCodec[A]` holds both an encoder and decoder, providing a complete solution for serializing and deserializing values in YAML format. The codec is derived automatically from a `Schema[A]`.

### Encoding Values to String

To convert a value to a YAML string:

```scala

case class Person(name: String, email: String)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec  = Person.schema.derive(YamlFormat)
val person = Person("Alice", "alice@example.com")
```

`YamlCodec#encodeToString` returns the YAML text of the record:

```scala
println(codec.encodeToString(person))
// name: Alice
// email: alice@example.com
```

### Encoding Values to Byte Array

To convert a value to YAML bytes, encode it with `YamlCodec#encode`:

```scala

case class Data(timestamp: Long, value: String)

object Data {
  implicit val schema: Schema[Data] = Schema.derived
}

val codec = Data.schema.derive(YamlFormat)
val bytes = codec.encode(Data(1700000000000L, "sample"))
```

The bytes are the UTF-8 text of the same document:

```scala
println(new String(bytes, "UTF-8"))
// timestamp: 1700000000000
// value: sample
```

### Decoding Values from String

To convert a YAML string back to a value:

```scala

case class Config(host: String, port: Int)

object Config {
  implicit val schema: Schema[Config] = Schema.derived
}

val codec = Config.schema.derive(YamlFormat)
val yaml = """
host: localhost
port: 8080
"""
```

`YamlCodec#decode` returns `Right` with the decoded value:

```scala
println(codec.decode(yaml))
// Right(Config(localhost,8080))
```

### Decoding Values from Byte Array

To decode YAML bytes, pass the bytes to `YamlCodec#decode`:

```scala

case class Settings(debug: Boolean, timeout: Int)

object Settings {
  implicit val schema: Schema[Settings] = Schema.derived
}

val codec    = Settings.schema.derive(YamlFormat)
val settings = Settings(debug = true, timeout = 30)
val bytes    = codec.encodeToString(settings).getBytes("UTF-8")
```

Decoding the bytes gives back the same value:

```scala
println(codec.decode(bytes))
// Right(Settings(true,30))
```

---

## YamlCodecDeriver

Configuration and derivation system for creating `YamlCodec[A]` instances from `Schema[A]`.

### Overview

`YamlCodecDeriver` implements the schema-driven derivation of YAML codecs. It handles the ZIO Blocks Schema primitive types and complex types (records, variants, sequences, maps), generating the matching YAML scalars, mappings and sequences.

### How Derivation Works

To create a codec from a schema:

```scala

case class User(id: Int, name: String, active: Boolean)

object User {
  implicit val schema: Schema[User] = Schema.derived
}

val codec = User.schema.derive(YamlFormat)
```

### Primitive Type Support

The deriver supports the ZIO Blocks Schema primitive types in these groups:

- Numeric: `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, `BigDecimal`
- Logical: `Boolean`, `Char`, `String`
- Temporal: `Instant`, `LocalDate`, `LocalDateTime`, `LocalTime`, `Duration`, `Period`, `Year`, `YearMonth`, `MonthDay`, `Month`, `DayOfWeek`, `ZonedDateTime`, `OffsetDateTime`, `OffsetTime`, `ZoneId`, `ZoneOffset`
- Special: `UUID`, `Currency`, `Unit`

A known bug in 0.0.56: `BigDecimal` and `BigInt` fields are written as quoted strings, although the JSON codec writes them as numbers. A YAML reader that follows the spec then sees a string. The `amount` field below is quoted:

```scala

case class Price(amount: BigDecimal, count: Int)

object Price {
  implicit val schema: Schema[Price] = Schema.derived
}

val codec = Price.schema.derive(YamlFormat)
```

```scala
println(codec.encodeToString(Price(BigDecimal("49.99"), 2)))
// amount: "49.99"
// count: 2
```

### Record Type Support

Case classes (records) are fully supported. Each field becomes a named key in the YAML mapping, and the key is the field name converted to kebab-case, so `zipCode` is written as `zip-code`:

```scala

case class Address(street: String, city: String, zipCode: String)

object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

val codec = Address.schema.derive(YamlFormat)
```

```scala
println(codec.encodeToString(Address("Unter den Linden 1", "Berlin", "10117")))
// street: Unter den Linden 1
// city: Berlin
// zip-code: "10117"
```

A known bug in 0.0.56: the derived codec always uses kebab-case keys and ignores field modifiers, so `@Modifier.rename` has no effect and a `@Modifier.transient` field is written and read. A YAML file with camelCase keys cannot be decoded into a case class with this codec. The JSON codec honours the same modifiers.

### Variant Type Support

Sealed traits and sum types are encoded as a mapping with one key, named after the case, and the case's fields nested under that key. There is no discriminator field:

```scala

sealed trait Status
case class Running(pid: Int) extends Status
case class Stopped(exitCode: Int) extends Status

object Status {
  implicit val schema: Schema[Status] = Schema.derived
}

val codec = Status.schema.derive(YamlFormat)
```

```scala
println(codec.encodeToString(Stopped(1)))
// Stopped:
//   exit-code: 1
```

A known bug in 0.0.56: a case with no fields, such as an enum value, is written as `Ebook: {}`, and a plain `Ebook` scalar in a YAML file fails to decode with `Expected mapping with single key for variant`. The guide [How to Read and Write YAML Documents](../../../guides/read-and-write-yaml.md) shows a `Schema[String].transform` workaround for enumerations.

---

## YamlOptions

Configuration for YAML output formatting. Controls indentation, document markers, and presentation style. Options apply only to `YamlWriter.write` and to `Yaml#print` and `Yaml#printPretty`; `YamlCodec#encodeToString` always uses the default style.

### Overview

`YamlOptions` has three settings:

| Setting           | Default | Effect                                                                     |
| ----------------- | ------- | -------------------------------------------------------------------------- |
| `indentStep`      | `2`     | Spaces per nesting level                                                   |
| `documentMarkers` | `false` | Start the output with a `---` line                                         |
| `flowStyle`       | `false` | Write every mapping as `{k: v}` and every sequence as `[a, b]` on one line |

`YamlOptions.default`, `YamlOptions.pretty` (document markers on) and `YamlOptions.flow` (flow style on) are ready-made values.

### Pretty-Printing Configuration

To write with four-space indentation and a leading `---` line, build the options and pass them to `YamlWriter.write`:

```scala

case class Endpoint(path: String, method: String)

object Endpoint {
  implicit val schema: Schema[Endpoint] = Schema.derived
}

val codec    = Endpoint.schema.derive(YamlFormat)
val endpoint = codec.encodeValue(Endpoint("/health", "GET"))
```

```scala
println(YamlWriter.write(endpoint, YamlOptions(indentStep = 4, documentMarkers = true)))
// ---
// path: /health
// method: GET
```

### Flow Style

With `YamlOptions.flow`, every mapping and sequence is written on one line:

```scala
println(YamlWriter.write(endpoint, YamlOptions.flow))
// {path: /health, method: GET}
```

A known bug in 0.0.56: flow style writes a string that contains a comma, `]` or `}` without quotes. Inside a flow collection those characters end the item, so the output is invalid YAML for other readers, and `YamlReader` reads it back as different data without an error. A title such as `Scala, the Language` comes back as `Scala`. Use flow style only for values that contain none of these characters.

---

## Error Handling

YAML decoding errors include location traces showing the path through nested structures where the error occurred.

### Understanding Error Traces

Errors render as `DynamicOptic` paths such as `.field.at(0).nested.value`: a record field is `.name`, a sequence index is `.at(0)`, a map value is `.atKey(...)` and a variant case is `.when[Case]`. The path shows exactly where decoding failed:

```scala

case class Team(name: String, members: List[String])

object Team {
  implicit val schema: Schema[Team] = Schema.derived
}

val codec = Team.schema.derive(YamlFormat)
val invalidYaml = """
name: Engineering
members: invalid
"""
```

The `members` field holds a scalar where a sequence is expected, so decoding fails at that field:

```scala
codec.decode(invalidYaml) match {
  case Right(team) => println(s"Loaded: $team")
  case Left(error) => println(s"Error: ${error.getMessage}")
}
// Error: Expected sequence at: .members
```

When the bad value is an element of a sequence, the path includes its index in the `.at(index)` form, not `[index]`. Searching logs for `[1]` finds nothing:

```scala
case class Ports(ports: List[Int])

object Ports {
  implicit val schema: Schema[Ports] = Schema.derived
}

val portsCodec = Ports.schema.derive(YamlFormat)
```

Here the second element is not a number:

```scala
portsCodec.decode("ports: [80, abc]") match {
  case Right(value) => println(s"Loaded: $value")
  case Left(error)  => println(s"Error: ${error.getMessage}")
}
// Error: Invalid int value: abc at: .ports.at(1)
```

### Zero-Overhead Error Handling

Errors use zero-overhead exceptions (no stack traces) for efficient error reporting in scenarios where errors are expected and handled inline.

---

## YamlReader and YamlWriter

`YamlReader` parses YAML text into a `Yaml` tree, and `YamlWriter` writes a tree back to text. The codec uses both, and you can call them directly when you work with the tree instead of a case class.

### Reading and Writing the Tree

`YamlReader.read` returns the `Yaml` tree of a document, and `YamlWriter.write` prints it with the options of your choice:

```scala
println(YamlWriter.write(YamlReader.read("name: api\nport: 8080\n"), YamlOptions.default))
// name: api
// port: "8080"
```

The reader does not keep the quotes of a scalar, and the writer quotes a number it reads, so the `8080` comes back quoted. This is the known bug described in [Printing YAML](#printing-yaml).

### Known Limits

The reader does not report most syntax errors. It stops at the first line it cannot read, returns the lines before it, and drops the rest without an error. A key without a space after its colon, such as `pageSize:20`, makes the reader drop that line and every line after it, so `ConfigSource.fromYaml` in `config-yaml` loses every later setting. The reader also accepts an unterminated `"`, `'`, `[` or `{` and drops the last character of the value.

The writer leaves some strings unquoted that other readers treat as YAML syntax. A string that starts with `#`, such as an OpenAPI `$ref` (`#/components/schemas/Book`), is read by other YAML readers as a comment, so the value is lost. A known bug in 0.0.56 also affects the `yaml"..."` interpolator: it does not compile on Scala 2.13 in user code, while it compiles on Scala 3.

The guide [How to Read and Write YAML Documents](../../../guides/read-and-write-yaml.md) has a table of the YAML forms that the reader and writer do not support, with what each one does:

[What schema-yaml Does Not Support](../../../guides/read-and-write-yaml.md#what-schema-yaml-does-not-support)

See also [How to Write an OpenAPI Document and Export It as JSON or YAML](../../../guides/write-and-export-openapi.md), which renders an OpenAPI document as YAML with `Yaml.fromJson`.

---

## Codec

`Codec[DecodeInput, EncodeOutput, Value]` is the base abstraction for encoding and decoding values between a specific input representation and a specific output representation. It forms the foundation of ZIO Blocks' multi-format serialization system, enabling a single `Schema[A]` to derive codecs for JSON, Avro, TOON, MessagePack, Thrift, and other formats integrated through `Codec`/`Format`. BSON's `BsonCodec` is not a subtype of `codec.Codec` because it operates on native `BsonReader`, `BsonWriter`, and `BsonValue` APIs. It nevertheless participates in shared type-class derivation through `Schema[A].derive(BsonCodecDeriver)`; `BsonSchemaCodec` remains a compatibility facade.

## Overview

`Codec` defines two abstract methods that every concrete codec must implement:

```scala
abstract class Codec[DecodeInput, EncodeOutput, Value] {
  def encode(value: Value, output: EncodeOutput): Unit
  def decode(input: DecodeInput): Either[SchemaError, Value]
}
```

- **`encode`** writes the encoded form of `value` into `output`. The output parameter is typically a mutable buffer (`ByteBuffer`, `CharBuffer`) that the caller provides.
- **`decode`** reads from `input` and returns either a `SchemaError` describing the failure or the decoded value.

End users rarely interact with `Codec` directly. Instead, they work with format-specific subclasses like `JsonCodec[A]` or `ToonCodec[A]`, which add convenience methods for common input/output types.

Given a `Schema[A]`, you can derive a codec for any supported format by calling `Schema[A].derive(format)`, which uses the `Deriver` associated with that format to generate the appropriate codec instance. For example, to derive a JSON codec:

```scala

case class Person(name: String, age: Int)

object Person {
  // Derive a schema for Person (required for codec derivation)
  implicit val schema: Schema[Person] = Schema.derived
   // Derive a JSON codec from the schema 
  val codec: JsonCodec[Person] = schema.derive(JsonFormat)
}

// Encode
val bytes: Array[Byte] = Person.codec.encode(Person("Alice", 30))

// Decode
val result: Either[SchemaError, Person] = Person.codec.decode(bytes)
```

## Installation

To include the base schema module with JSON support, add the following dependency to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"
```

Additional format modules are separate artifacts:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema-avro"        % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-bson"        % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-csv"         % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-messagepack" % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-thrift"      % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-toon"        % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-xml"         % "0.0.56"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-yaml"        % "0.0.56"
```

For cross-platform projects (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-schema" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x.

## BinaryCodec and TextCodec

The codec system in ZIO Blocks is organized as a layered hierarchy:

```
Codec[DecodeInput, EncodeOutput, Value]        
├── BinaryCodec[A] =  Codec[ByteBuffer, ByteBuffer, A]   (ByteBuffer ↔ A)
│   ├── AvroCodec[A]                   
│   ├── JsonCodec[A]                    
│   ├── MessagePackCodec[A]         
│   ├── ThriftCodec[A]               
│   ├── ToonCodec[A]                  
│   ├── XmlCodec[A]         
│   └── YamlCodec[A]         
└── TextCodec[A] = Codec[CharBuffer, CharBuffer, A]      (CharBuffer ↔ A)
    └── CsvCodec[A]                    
```

1. **`BinaryCodec[A]`** fixes both the input and output to `ByteBuffer` and is the base class for all codecs that operate on binary data:

```scala
abstract class BinaryCodec[A] extends Codec[ByteBuffer, ByteBuffer, A]
```

2. **`TextCodec[A]`** fixes both the input and output to `CharBuffer`:

```scala
abstract class TextCodec[A] extends Codec[CharBuffer, CharBuffer, A]
```

Most built-in codecs (JSON, Avro, TOON, MessagePack, Thrift, XML and YAML) extend `BinaryCodec`. Despite JSON being a text format, the JSON codec operates on UTF-8 encoded bytes for performance. BSON extends neither: its `BsonCodec` is a pair of a BSON encoder and a BSON decoder.

`TextCodec` is for formats that operate on character data rather than raw bytes. The built-in CSV codec, `CsvCodec`, extends it, and custom text-based formats can extend it too.

To add a format of your own, extend one of these two classes; [How to Write a Codec for a New Wire Format](../../guides/write-a-codec-for-a-new-wire-format.md) builds a `BinaryCodec` and its deriver step by step.

## Deriving Codecs

### Using Schema.derive

The primary way to obtain a codec is through `Schema[A].derive`:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Pass a Format object to get a codec for that format
val jsonCodec: JsonCodec[Person] = Schema[Person].derive(JsonFormat)
```

This works with any format:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val jsonCodec = Schema[Person].derive(JsonFormat)
val toonCodec = Schema[Person].derive(ToonFormat)
```

### Using Schema.deriving for Customization

For more control over the derived codec, use `deriving` to get a `DerivationBuilder`. This lets you override instances for specific substructures or inject modifiers before finalizing:

```scala

case class Person(name: String, age: Int)
object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived
  val name = $(_.name)
  val age  = $(_.age)
}

// Override the codec for the "name" field
val customNameCodec = new JsonCodec[String] {
  def decodeValue(in: JsonReader): String = in.readString()
  def encodeValue(x: String, out: JsonWriter): Unit = out.writeVal(x.toUpperCase)
}

val codec: JsonCodec[Person] = Schema[Person]
  .deriving(JsonFormat.deriver)
  .instance(Person.name, customNameCodec)
  .derive
```

To apply the same override to every codec you derive, register it on the deriver instead (see [Overriding a Type on a Deriver](#overriding-a-type-on-a-deriver)).

### Using Schema#decode and Schema#encode

`Schema` also provides `decode` and `encode` methods that internally call `derive` (with caching) and then delegate to the codec:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Encode directly from Schema
val buffer = ByteBuffer.allocate(1024)
Schema[Person].encode(JsonFormat)(buffer)(Person("Alice", 30))

// Decode directly from Schema
buffer.flip()
val result: Either[SchemaError, Person] = Schema[Person].decode(JsonFormat)(buffer)
```

### Using a Deriver Directly

Each `Format` object contains a `Deriver[TC]` that can also be passed to `derive`:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// These are equivalent:
val codec1 = Schema[Person].derive(JsonFormat)
val codec2 = Schema[Person].derive(JsonFormat.deriver)
```

Passing a `Deriver` directly is useful when working with custom or configured derivers (see [Configuring Codecs](#configuring-codecs)).

### Overriding a Type on a Deriver

`Deriver#withInstance` returns a new `Deriver[TC]` that uses a codec you supply wherever a given type appears, at any depth, and derives everything else as before. Use it when the same override must apply to every codec derived through that deriver.

`DerivationBuilder#instance` (see [Using Schema.deriving for Customization](#using-schemaderiving-for-customization)) changes a single `DerivationBuilder#derive` call, so an override that every type needs, such as money written as a JSON string to keep its precision in JavaScript clients, would have to be repeated for each type. An `implicit` codec in scope doesn't help either: derivation never searches for codec implicits. `Deriver#withInstance` registers the override once on the deriver instead:

| Situation                                                    | Right choice                                    |
|--------------------------------------------------------------|-------------------------------------------------|
| One override for a single derived codec                      | `Schema[A].deriving(deriver).instance(...)`     |
| The same override in every codec derived through a deriver   | `deriver.withInstance(...)`                     |

`Deriver#withInstance` has two overloads, one for every occurrence of a type and one for a single field of a parent type, and `Deriver#withModifier` has matching ones:

```scala
trait Deriver[TC[_]] {
  final def withInstance[A](instance: => TC[A])(implicit typeId: TypeId[A]): Deriver[TC]
  final def withInstance[A, B](typeId: TypeId[A], termName: String, instance: => TC[B]): Deriver[TC]
  final def withModifier[A](typeId: TypeId[A], modifier: Modifier.Reflect): Deriver[TC]
  final def withModifier[A](typeId: TypeId[A], termName: String, modifier: Modifier.Term): Deriver[TC]
}
```

The following billing service writes every `BigDecimal` as a JSON string, in invoices and in the line items nested inside them, and renames one field:

```scala

case class LineItem(sku: String, unitPrice: BigDecimal, quantity: Int)
object LineItem {
  implicit val schema: Schema[LineItem] = Schema.derived
}

case class Invoice(number: String, total: BigDecimal, items: List[LineItem])
object Invoice {
  implicit val schema: Schema[Invoice] = Schema.derived
}

val moneyAsString: JsonCodec[BigDecimal] = new JsonCodec[BigDecimal] {
  def decodeValue(in: JsonReader): BigDecimal = {
    val s = in.readString()
    try BigDecimal(s)
    catch { case _: NumberFormatException => error(s"expected a decimal string, got: $s") }
  }

  def encodeValue(x: BigDecimal, out: JsonWriter): Unit =
    out.writeVal(x.bigDecimal.toPlainString)
}

val billingDeriver: Deriver[JsonCodec] =
  JsonCodecDeriver
    .withFieldNameMapper(NameMapper.SnakeCase)
    .withInstance[BigDecimal](moneyAsString)
    .withModifier(TypeId.of[Invoice], "number", Modifier.rename("invoice_no"))

val invoiceCodec: JsonCodec[Invoice]   = Schema[Invoice].deriving(billingDeriver).derive
val lineItemCodec: JsonCodec[LineItem] = Schema[LineItem].deriving(billingDeriver).derive
```

`Deriver#withInstance[A]` finds the type through an implicit `TypeId[A]`, which ZIO Blocks provides for built-in types. Keep these points in mind:

- `Deriver#withInstance` and `Deriver#withModifier` return a plain `Deriver[TC]`, so call format options such as `JsonCodecDeriver#withFieldNameMapper` first, as above, and the overrides last.
- The field-level overload doesn't check at compile time that the codec's type matches the field's type.
- When the deriver and a `DerivationBuilder` override the same type or field, the deriver's override wins.
- `toJsonString` and `fromJson` always use the default codec from `Schema#jsonCodec`, so call `JsonCodec#encodeToString` and `JsonCodec#decode` on the codec derived through your deriver.

See also: [Customizing Derivation with Instance and Modifier Overrides](./type-class-derivation.md#customizing-derivation-with-instance-and-modifier-overrides) for every override and its resolution order, and the guide [How to Override the Codec for One Type Everywhere](../../guides/override-codec-for-one-type.md) for a step-by-step walkthrough. For decoding and encoding speed, see [How to Tune JSON Encoding and Decoding for Throughput](../../guides/tune-json-throughput.md).

## Convenience Methods on Format-Specific Codecs

While the base `Codec` class defines only `encode(value, output)` and `decode(input)`, format-specific subclasses like `JsonCodec` and `ToonCodec` add convenience overloads for common I/O types.

### JsonCodec Convenience Methods

`JsonCodec[A]` provides the following overloads beyond the base `ByteBuffer` API:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(JsonFormat)
val person = Person("Alice", 30)

// Array[Byte]
val bytes: Array[Byte] = codec.encode(person)
val fromBytes1: Either[SchemaError, Person] = codec.decode(bytes)
val fromBytes2: Either[SchemaError, Person] = codec.decode(bytes, 0, bytes.length)

// String
val jsonStr: String = codec.encodeToString(person)
val fromStr: Either[SchemaError, Person] = codec.decode("""{"name":"Alice","age":30}""")

// InputStream / OutputStream

val os = new ByteArrayOutputStream()
codec.encode(person, os)

val is = new ByteArrayInputStream(os.toByteArray)
val fromStream: Either[SchemaError, Person] = codec.decode(is)
```

### ToonCodec Convenience Methods

`ToonCodec[A]` provides the same set of overloads:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(ToonFormat)
val person = Person("Alice", 30)

// Array[Byte]
val bytes: Array[Byte] = codec.encode(person)
val fromBytes: Either[SchemaError, Person] = codec.decode(bytes)

// String
val toonStr: String = codec.encodeToString(person)
val fromStr: Either[SchemaError, Person] = codec.decode("name: Alice\nage: 30")
```

### Summary of Convenience Methods

`BinaryCodec` subclasses (JSON, TOON, MessagePack, Avro, Thrift) expose the following convenience overloads (availability may vary by format):

| Method                                                                   | Description                                      |
|--------------------------------------------------------------------------|--------------------------------------------------|
| `encode(value): Array[Byte]`                                             | Encode to a byte array                           |
| `decode(input: Array[Byte]): Either[SchemaError, A]`                     | Decode from a byte array                         |
| `decode(input: Array[Byte], from: Int, to: Int): Either[SchemaError, A]` | Decode from a byte array slice                   |
| `encode(value, output: ByteBuffer): Unit`                                | Encode into a `ByteBuffer`                       |
| `decode(input: ByteBuffer): Either[SchemaError, A]`                      | Decode from a `ByteBuffer`                       |
| `encode(value, output: OutputStream): Unit`                              | Encode into an `OutputStream` (JSON, TOON, Avro) |
| `decode(input: InputStream): Either[SchemaError, A]`                     | Decode from an `InputStream` (JSON, TOON, Avro)  |
| `encodeToString(value): String`                                          | Encode to a `String` (JSON, TOON)                |
| `decode(input: String): Either[SchemaError, A]`                          | Decode from a `String` (JSON, TOON)              |

The `String`-based methods are available on text-oriented binary codecs (JSON, TOON) but not on purely binary formats like Avro or Thrift.

## Configuring Codecs

Format-specific derivers support configuration options that control encoding behavior. Instead of passing a `Format` object to `derive`, you pass a configured `Deriver`:

### JSON Configuration

```scala

case class Person(
  firstName: String,
  lastName: String,
  middleName: Option[String] = None
)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val customDeriver = JsonCodecDeriver
  .withFieldNameMapper(NameMapper.SnakeCase)
  .withTransientNone(true)
  .withRejectExtraFields(true)

val codec = Schema[Person].derive(customDeriver)

// Encodes as: {"first_name":"Alice","last_name":"Smith"}
// (middleName omitted because it is None and transientNone is true)
val json = codec.encodeToString(Person("Alice", "Smith"))
```

| Option                          | Description                                            | Default    |
|---------------------------------|--------------------------------------------------------|------------|
| `withFieldNameMapper`           | Transform field names (Identity, SnakeCase, KebabCase) | `Identity` |
| `withCaseNameMapper`            | Transform variant/case names                           | `Identity` |
| `withDiscriminatorKind`         | ADT discriminator style (Key, Field, None)             | `Key`      |
| `withRejectExtraFields`         | Error on unknown fields during decoding                | `false`    |
| `withEnumValuesAsStrings`       | Encode enum values as strings                          | `true`     |
| `withTransientNone`             | Omit `None` values from output                         | `true`     |
| `withTransientEmptyCollection`  | Omit empty collections from output                     | `true`     |
| `withTransientDefaultValue`     | Omit fields with default values                        | `true`     |
| `withRequireOptionFields`       | Require optional fields in input                       | `false`    |
| `withRequireCollectionFields`   | Require collection fields in input                     | `false`    |
| `withRequireDefaultValueFields` | Require fields with defaults in input                  | `false`    |

### TOON Configuration

```scala

case class Person(
  firstName: String,
  lastName: String
)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val customDeriver = ToonCodecDeriver
  .withFieldNameMapper(NameMapper.SnakeCase)
  .withArrayFormat(ArrayFormat.Tabular)
  .withDiscriminatorKind(DiscriminatorKind.Field("type"))

val codec = Schema[Person].derive(customDeriver)
```

## Error Handling

All `decode` operations return `Either[SchemaError, A]`. `SchemaError` includes path information that pinpoints where in the data structure decoding failed:

```scala

case class Address(street: String, city: String)

object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

case class Person(name: String, address: Address)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(JsonFormat)

// Missing required field
val result = codec.decode("""{"name":"Alice","address":{}}""")

result match {
  case Right(person) => println(person)
  case Left(error)   => error.errors.foreach(e => println(s"Error: ${e.message}"))
}
```

---

## DynamicOptic

`DynamicOptic` is a runtime path through nested data structures in ZIO Blocks. It is the untyped,
dynamically-constructed counterpart to the typed [`Optic[S, A]`](./optics.md). Where typed optics are bound to specific
Scala types at compile time, a `DynamicOptic` is a sequence of **navigation steps** that can be built, composed, and
applied at runtime:

```scala

// Build a path: .users[0].name
val path = DynamicOptic.root.field("users").at(0).field("name")

// Navigate a DynamicValue
val data = DynamicValue.Record(
  "users" -> DynamicValue.Sequence(
    DynamicValue.Record("name" -> DynamicValue.string("Alice"))
  )
)

val result = data.get(path).one
// Right(DynamicValue.Primitive(PrimitiveValue.String("Alice")))
```

:::tip
For a practical example of extracting column names from `DynamicOptic` paths to generate SQL, see the [Query DSL Part 2: SQL Generation](../../guides/query-dsl-sql.md) guide.
:::

## Motivation

Most of the time you work with typed `Optic[S, A]` values — they are statically verified and provide type-safe
get/set/modify operations. `DynamicOptic` is useful when:

- **The path is determined at runtime** — user input, configuration, query parameters
- **You are working with `DynamicValue`** — navigating, modifying, or querying schema-less data
- **You need schema introspection** — walking a `Schema` or `Reflect` structure programmatically
- **Serializing optic paths** — `DynamicOptic` has its own `Schema[DynamicOptic]` and can be persisted

The relationship between typed and dynamic optics:

```
   Typed World                        Dynamic World
┌──────────────────┐             ┌──────────────────┐
│  Optic[S, A]     │─────────────│  DynamicOptic    │
│                  │  toDynamic  │                  │
│  Lens, Prism,    │             │  Sequence of     │
│  Optional,       │             │  Node steps      │
│  Traversal       │             │                  │
├──────────────────┤             ├──────────────────┤
│  Operates on     │             │  Operates on     │
│  typed values    │             │  DynamicValue,   │
│  (case classes,  │             │  Schema,         │
│  sealed traits)  │             │  Reflect         │
└──────────────────┘             └──────────────────┘
```

The `Optic[S, A]` and `DynamicOptic` types serve complementary roles in ZIO Blocks' optics system. `Optic[S, A]` provides compile-time type safety through macros or manual construction, operates on typed Scala values, and can be converted to a `DynamicOptic` via the `optic.toDynamic` method. In contrast, `DynamicOptic` performs runtime type checking and is constructed through a builder API or [path interpolator](./path-interpolator.md), operating directly on [DynamicValue](./dynamic-value.md), [Schema](./schema.md), and [Reflect](./reflect.md) representations. 

## Design & Structure

`DynamicOptic` is modeled as a case class wrapping an `IndexedSeq[Node]`, where each `Node` represents one step in a navigation path:

```
DynamicOptic(nodes: IndexedSeq[Node])

sealed trait DynamicOptic.Node
 ├── Field(name: String)            — named field in a record
 ├── Case(name: String)             — specific case in a variant
 ├── AtIndex(index: Int)            — element at index in a sequence
 ├── AtIndices(index: Seq[Int])     — elements at multiple indices
 ├── AtMapKey(key: DynamicValue)    — value at a specific map key
 ├── AtMapKeys(keys: Seq[DynamicValue]) — values at multiple map keys
 ├── Elements                       — all elements in a sequence (wildcard)
 ├── MapKeys                        — all keys in a map (wildcard)
 ├── MapValues                      — all values in a map (wildcard)
 └── Wrapped                        — inner value of a wrapper/newtype
```

Each of the `DynamicOptic` nodes represents a different way to navigate through a data structure. Here's a reference table for the nodes, their navigation semantics, and string representations:

| Node                  | Navigates             | `toString`   | `toScalaString`     |
|-----------------------|-----------------------|--------------|---------------------|
| `Field("name")`       | Record field          | `.name`      | `.name`             |
| `Case("Email")`       | Variant case          | `<Email>`    | `.when[Email]`      |
| `AtIndex(0)`          | Sequence element      | `[0]`        | `.at(0)`            |
| `AtIndices(Seq(0,2))` | Multiple elements     | `[0,2]`      | `.atIndices(0, 2)`  |
| `AtMapKey(k)`         | Map entry by key      | `{"host"}`   | `.atKey("host")`    |
| `AtMapKeys(ks)`       | Multiple entries      | `{"a", "b"}` | `.atKeys("a", "b")` |
| `Elements`            | All sequence elements | `[*]`        | `.each`             |
| `MapKeys`             | All map keys          | `{*:}`       | `.eachKey`          |
| `MapValues`           | All map values        | `{*}`        | `.eachValue`        |
| `Wrapped`             | Wrapper inner value   | `.~`         | `.wrapped`          |

Key design decisions:

- **Map keys as `DynamicValue`** — Map keys can be any type (`String`, `Int`, `Boolean`, etc.), so `AtMapKey` stores the key as a `DynamicValue` to remain type-agnostic.
- **Dual rendering** — `toString` produces a compact interpolator syntax (`.field[0]{key}`), while `toScalaString` produces Scala method-call syntax (`.field.at(0).atKey(key)`). The compact format is designed to be copy-pasteable into the `p"..."` string interpolator (except for wrapper steps `.~`, the empty path `.` and `Char` keys with an escape, which the interpolator rejects, a known bug in 0.0.56), while the Scala format is used in error messages.
- **Every typed `Optic` has a `toDynamic` method** — This bridges the typed and untyped worlds, allowing any `Lens`, `Prism`, `Optional`, or `Traversal` to produce its `DynamicOptic` representation.

## Construction

### Starting Points

Every `DynamicOptic` starts from `root` (the identity/empty path) or one of the pre-built singletons:

```scala

val root = DynamicOptic.root // empty path: "."
val elems = DynamicOptic.elements // "[*]"
val keys = DynamicOptic.mapKeys // "{*:}"
val values = DynamicOptic.mapValues // "{*}"
val inner = DynamicOptic.wrapped // ".~"
```

### Builder Methods

Chain builder methods on `DynamicOptic.root` (or any existing optic) to construct paths fluently:

```scala

// Navigate into a record field, then a sequence index, then another field
val path = DynamicOptic.root.field("users").at(0).field("name")
// toString: .users[0].name

// Navigate into a map with a typed key
val configPath = DynamicOptic.root.field("config").atKey("host")
// toString: .config{"host"}

// Navigate into a variant case, then a field
val resultPath = DynamicOptic.root.field("result").caseOf("Success").field("value")
// toString: .result<Success>.value

// Select all elements, then a field on each
val allEmails = DynamicOptic.root.field("users").elements.field("email")
// toString: .users[*].email
```

| Method           | Node Produced        | Example               |
|------------------|----------------------|-----------------------|
| `.field(name)`   | `Field(name)`        | `.field("street")`    |
| `.caseOf(name)`  | `Case(name)`         | `.caseOf("Email")`    |
| `.at(index)`     | `AtIndex(index)`     | `.at(0)`              |
| `.atIndices(i*)` | `AtIndices(indices)` | `.atIndices(0, 2, 5)` |
| `.atKey[K](key)` | `AtMapKey(dv)`       | `.atKey("host")`      |
| `.atKeys[K](k*)` | `AtMapKeys(dvs)`     | `.atKeys("a", "b")`   |
| `.elements`      | `Elements`           | `.elements`           |
| `.mapKeys`       | `MapKeys`            | `.mapKeys`            |
| `.mapValues`     | `MapValues`          | `.mapValues`          |
| `.wrapped`       | `Wrapped`            | `.wrapped`            |

Note: `.atKey` and `.atKeys` require an implicit `Schema[K]` to convert the typed key to a `DynamicValue`.

### Path Interpolator

The [`p"..."` path interpolator](./path-interpolator.md) provides a concise compile-time syntax for building
`DynamicOptic` values:

```scala

// Equivalent builder vs interpolator
val builderPath     : DynamicOptic = DynamicOptic.root.field("users").at(0).field("name")
val interpolatorPath: DynamicOptic = p".users[0].name"
// Both produce the same DynamicOptic

// Wildcards
val allEmails = p".users[*].email"

// Map access
val host = p""".config{"host"}"""

// Variant cases
val success = p".result<Success>.value"

// Complex path
val complex = p""".groups[*].members[0].contacts{"email"}"""
```

See [Path Interpolator](./path-interpolator.md) for the full syntax reference.

### From Typed Optics

Every typed `Optic[S, A]` can be converted to a `DynamicOptic` via `toDynamic`:

```scala

case class Address(street: String, city: String)

object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

case class Person(name: String, address: Address)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived
  val name: Lens[Person, String] = optic(_.name)
  val street: Lens[Person, String] = optic(_.address.street)
}

val dynamicName: DynamicOptic = Person.name.toDynamic
// toString: .name

val dynamicStreet: DynamicOptic = Person.street.toDynamic
// toString: .address.street
```

See [Optics](./optics.md) for more on typed optics.

## Operations

### Composition

`DynamicOptic` values compose via the `apply` method, which concatenates their node sequences:

```scala

val users = DynamicOptic.root.field("users")
val first = DynamicOptic.root.at(0)
val name = DynamicOptic.root.field("name")

// Compose three paths into one
val fullPath = users(first)(name)
// toString: .users[0].name

// Compose with pre-built singletons
val allUserNames = users(DynamicOptic.elements)(name)
// toString: .users[*].name

// Compose with interpolator-built paths
val emails = users(p"[*].email")
// toString: .users[*].email
```

### DynamicValue Operations

`DynamicOptic` is the path argument for all `DynamicValue` operations: `get(path)` for retrieval (with `.one` or `.toChunk`), `modify(path)(f)` for transformation, `set(path, value)` for replacement, `delete(path)` for removal, and `insert(path, value)` for addition. By default, the mutating operations (`modify`, `set`, `delete`, `insert`) are lenient and return the original value unchanged if the path doesn't resolve, whereas `get(path)` yields a failing `DynamicValueSelection` on a missing path (though calling `.toChunk` on it will produce an empty chunk). Each operation also has a strict `*OrFail` variant returning `Either[SchemaError, DynamicValue]` with error details on failure.

For example, the `DynamicValue#get` method uses `DynamicOptic` to navigate and extract values:

```scala

val data = DynamicValue.Record(
  "users" -> DynamicValue.Sequence(
    DynamicValue.Record("name" -> DynamicValue.string("Alice"), "age" -> DynamicValue.int(30)),
    DynamicValue.Record("name" -> DynamicValue.string("Bob"), "age" -> DynamicValue.int(25))
  )
)

// Get a single value
val firstName = data.get(p".users[0].name").one
// Right(DynamicValue.Primitive(PrimitiveValue.String("Alice")))

// Get all matching values (wildcard)
val allNames = data.get(p".users[*].name").toChunk
// Chunk(DynamicValue.string("Alice"), DynamicValue.string("Bob"))
```

### Schema & Reflect Operations

`DynamicOptic` can navigate schema structures, not just values. This is useful for schema introspection and
metaprogramming.

1. The `Schema#get` method takes a `DynamicOptic` path and returns the `Reflect` for the nested component at that path, if it exists. This allows you to programmatically explore the structure of a schema:

```scala

case class Address(street: String, city: String)

object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

case class Person(name: String, address: Address)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Get the Reflect for the "street" field inside Person
val streetReflect: Option[Reflect.Bound[?]] =
  Schema[Person].get(p".address.street")
```

2. The `DynamicSchema#get` method works similarly, allowing you to navigate a `DynamicSchema` structure:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val dynSchema = Schema[Person].toDynamicSchema

val nameReflect: Option[Reflect.Unbound[_]] =
  dynSchema.get(p".name")
```

3. By applying a `DynamicOptic` directly to a `Reflect` value, you can navigate the reflected structure of a type:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val reflect = Schema[Person].reflect
val path = p".name"

// Using the DynamicOptic's apply method
val result: Option[Reflect[?, ?]] = path(reflect)
```

## Failure Path in OpticCheck

Typed optics use `DynamicOptic` internally for error reporting. When a typed optic operation fails (e.g., a `Prism` encounters the wrong case), the error includes the `DynamicOptic` path to pinpoint exactly where the failure occurred.

```
OpticCheck(errors: ::[Single])
 └── Single (sealed trait)
      ├── Error (sealed trait)
      │    ├── UnexpectedCase    — prism matched wrong variant case
      │    ├── WrappingError     — wrapper conversion failed
      │    └── DynamicConversionError — SchemaExpr#eval could not decode its result
      └── Warning (sealed trait)
           ├── EmptySequence     — traversal over empty sequence
           ├── SequenceIndexOutOfBounds — index beyond sequence length
           ├── MissingKey        — map key not found
           └── EmptyMap          — traversal over empty map
```

Every `OpticCheck.Single` carries two `DynamicOptic` paths (for `DynamicConversionError`, which has no step, both are the root path):

- **`full`** — The complete optic path that was being evaluated
- **`prefix`** — The path up to the point where the error occurred

[Error Reporting with OpticCheck](./optics.md#error-reporting-with-opticcheck) describes each problem and its fields. Error messages use `toScalaString` for human-readable output:

```
During attempted access at .when[Email].subject,
encountered an unexpected case at .when[Email]:
expected Email, but got Push
```

## Path String Syntax

`DynamicOptic` provides two string formats for different contexts:

- **`toString`** — Compact path syntax matching the [`p"..."` interpolator](./path-interpolator.md) format, except for wrapper steps (`.~`), the root (`.`) and `Char` keys with an escape, which the interpolator rejects (a known bug in 0.0.56)
- **`toScalaString`** — Scala method call syntax used in error messages

| Node                      | `toString`   | `toScalaString`     |
|---------------------------|--------------|---------------------|
| `Field("name")`           | `.name`      | `.name`             |
| `Case("Email")`           | `<Email>`    | `.when[Email]`      |
| `AtIndex(0)`              | `[0]`        | `.at(0)`            |
| `AtIndices(Seq(0, 2))`    | `[0,2]`      | `.atIndices(0, 2)`  |
| `AtMapKey(string "host")` | `{"host"}`   | `.atKey("host")`    |
| `AtMapKey(int 42)`        | `{42}`       | `.atKey(42)`        |
| `AtMapKey(bool true)`     | `{true}`     | `.atKey(true)`      |
| `AtMapKey(char 'a')`      | `{'a'}`      | `.atKey('a')`       |
| `AtMapKeys(strings)`      | `{"a", "b"}` | `.atKeys("a", "b")` |
| `Elements`                | `[*]`        | `.each`             |
| `MapKeys`                 | `{*:}`       | `.eachKey`          |
| `MapValues`               | `{*}`        | `.eachValue`        |
| `Wrapped`                 | `.~`         | `.wrapped`          |
| root                      | `.`          | `.`                 |

## Serialization

`DynamicOptic` has an implicit `Schema[DynamicOptic]` defined in its companion object, which means it can be serialized and deserialized just like any other schema-equipped type. This enables storing optic paths in databases, sending them over the wire, or including them in configuration files:

```scala

// Schema[DynamicOptic] is available implicitly
val opticSchema: Schema[DynamicOptic] = Schema[DynamicOptic]

// Convert to/from DynamicValue for serialization
val path = p".users[0].name"
val serialized: DynamicValue = opticSchema.toDynamicValue(path)
```

## Search Optics

A **Search optic** recursively traverses a data structure to find **all occurrences** matching a type or schema. It produces a `Traversal[S, A]` that collects matches in depth-first, left-to-right order. This section covers the two DSLs for building a search path; for the `SearchTraversal` engine that executes one — `fold`/`modify`/`check`, composing a search with other optics, and the `SchemaMatch` predicate behind `#Pattern` matching — see [Schema Search and Update](./schema-search.md).

### Motivation

Search optics address scenarios where you need to:

- **Find all values of a specific type** across deeply nested structures (e.g., all `String` fields in a nested record)
- **Extract all data matching a structural pattern** (e.g., all records with `{ name: string, age: int }` schema)
- **Transform all matching occurrences** in untyped or partially-typed data
- **Query data without knowing the exact path** — the search discovers all paths automatically

### Typed API: `.searchFor[T]`

Use the `.searchFor[T]` extension method on a `CompanionOptics` to search for all values of type `T`:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

case class Address(city: String)
object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

case class Company(name: String, employees: List[Person], hq: Address)
object Company extends CompanionOptics[Company] {
  implicit val schema: Schema[Company] = Schema.derived
}

// Find all Person instances within a Company
val personSearch: Traversal[Company, Person] = Company.optic(_.searchFor[Person])

val company = Company(
  "Acme",
  List(Person("Alice", 30), Person("Bob", 25)),
  Address("NYC")
)

// Modify all persons (increment their age)
val updated: Company = personSearch.modify(company, p => p.copy(age = p.age + 1))
// Company("Acme", List(Person("Alice", 31), Person("Bob", 26)), Address("NYC"))
```

### Dynamic API: Path Strings with `#` Prefix

Use the `#` prefix in path strings to specify type or schema patterns:

```scala

val data = DynamicValue.Record(
  "company" -> DynamicValue.Record(
    "name" -> DynamicValue.string("Acme"),
    "employees" -> DynamicValue.Sequence(
      DynamicValue.Record("name" -> DynamicValue.string("Alice")),
      DynamicValue.Record("name" -> DynamicValue.string("Bob"))
    )
  )
)

// Find all strings in the data
val allStrings = data.get(p"#string").toChunk
// Chunk(
//   DynamicValue.string("Acme"),
//   DynamicValue.string("Alice"),
//   DynamicValue.string("Bob")
// )

// Find all records matching a schema
val recordsWithName = data.get(p"#record { name: string }").toChunk

// Modify all matching values
val modified = data.modify(p"#string")(dv =>
  dv match {
    case DynamicValue.Primitive(PrimitiveValue.String(s)) =>
      DynamicValue.string(s.toUpperCase)
    case other => other
  }
)
```

### The `SchemaRepr` Pattern Type

Every `#Pattern` string above parses into a `SchemaRepr` — a plain, immutable ADT you can also build by hand, independent of the interpolator. `DynamicOptic#searchSchema(schemaRepr)` takes one directly, and `Node.SchemaSearch(schemaRepr)` is the node it produces, exactly the node `#Pattern` syntax builds under the hood:

```scala

val pattern: SchemaRepr = SchemaRepr.Record(
  IndexedSeq("name" -> SchemaRepr.Primitive("string"), "age" -> SchemaRepr.Primitive("int"))
)
```

`SchemaRepr#toString` renders the same syntax `#Pattern` strings use, so a pattern built by hand and one parsed from a literal print identically:

```scala
pattern.toString
// res13: String = "record { name: string, age: int }"

val path: DynamicOptic = DynamicOptic.root.searchSchema(pattern)
// path: DynamicOptic = DynamicOptic(
//   IndexedSeq(
//     SchemaSearch(
//       Record(Vector(("name", Primitive("string")), ("age", Primitive("int"))))
//     )
//   )
// )
path == p"#record { name: string, age: int }"
// res14: Boolean = true
```

The eight cases cover every shape the grammar recognizes:

| Case | Renders as | Matches |
|------|------------|---------|
| `Nominal(name)`         | the bare name, e.g. `Person`         | a value of that Scala type — only against a `Reflect` tree, see below |
| `Primitive(name)`       | the bare name, e.g. `string`         | a primitive of that kind (case-insensitive) |
| `Record(fields)`        | `record { name: string, ... }`       | a record with at least those fields, by subset match |
| `Variant(cases)`        | `variant { Case: type, ... }`        | a variant whose active case matches one of those entries |
| `Sequence(element)`     | `list(...)`                          | a sequence whose elements all match — parses from `list(...)`, `set(...)`, or `vector(...)`, all three producing the same case |
| `Map(key, value)`       | `map(key, value)`                    | a map whose entries all match both patterns |
| `Optional(inner)`       | `option(...)`                        | `Null`, or a value matching the inner pattern |
| `Wildcard`              | `_`                                   | anything |

Parsing a `#Pattern` string is itself layered: the path interpolator recognizes the `#` prefix and hands the substring after it to a small recursive-descent parser, which reports a specific error (unexpected character, an empty `record {}`/`variant {}`, an unterminated expression) at the exact offset where parsing failed — that's what turns a malformed pattern like `#record { name string }` into a compile-time interpolator error. A misspelled type name is not an error: `#record { name: strnig }` parses, with `strnig` as a `Nominal` type that matches nothing.

### Supported Patterns

Search optics support both **type-based** and **schema-based** patterns:

| Pattern                        | Matches                                | Example                    |
|--------------------------------|----------------------------------------|---------------------------|
| `#Nominal` (type name)         | Values of that type name               | `#Person`                  |
| `#int`, `#string`, `#boolean`  | Specific primitive types (case-insensitive) | `#int`, `#string`, `#boolean` |
| `#record { ... }`              | Records with matching fields           | `#record { name: string }` |
| `#variant { ... }`             | Variant cases with matching content    | `#variant { Error: string }` |
| `#list(...)`, `#set(...)`, `#vector(...)` | Sequences with a matching element type, regardless of which of the three keywords appears | `#list(string)` |
| `#map(...)`                    | Maps with key and value types          | `#map(string, int)`        |
| `#option(...)`                 | `null`, or values of the inner type (a `Schema`-encoded `Option` is a `Some`/`None` variant, which it does not match) | `#option(int)`             |

### Traversal Order

Results are collected in **depth-first, left-to-right order**:

```scala

val nested = DynamicValue.Record(
  "a" -> DynamicValue.int(1),
  "b" -> DynamicValue.Record(
    "c" -> DynamicValue.int(2),
    "d" -> DynamicValue.int(3)
  ),
  "e" -> DynamicValue.int(4)
)

// Depth-first, left-to-right: 1, 2, 3, 4
val allInts = nested.get(p"#int").toChunk
// Chunk(1, 2, 3, 4)
```

### Known Limitation: Nominal Matching in Untyped Contexts

`Nominal` pattern matching (e.g., `#Person`) returns `false` against a `DynamicValue` or `Json` value because these untyped structures carry no type identity. The one exception is searching a `Schema`/`Reflect` tree itself (as `DynamicSchema#get` does) rather than a value — there, `Nominal` matches by comparing against the tree's own `TypeId`, since a `Reflect` node still carries the type identity a plain `DynamicValue`/`Json` value never does. For searching values, use one of these approaches instead:

- **Use the typed API**: `optic(_.searchFor[Person])`
- **Or use structural patterns**: `p"#record { name: string, age: int }"`

Here's how the two approaches differ in practice:

```scala

val dynamicData: DynamicValue = ???

// This won't match — no type identity available
val nominalMatch = dynamicData.get(p"#Person").toChunk
// Chunk()

// This works — structural matching
val structuralMatch = dynamicData.get(p"#record { name: string, age: int }").toChunk
// Chunk(...matching records...)
```

## See Also

- [DynamicSchema](./dynamic-schema.md) — use `DynamicSchema#get` with a `DynamicOptic` to navigate schema trees and inspect nested type structures.
- [Optics](./optics.md) — reference page for typed `Optic[S, A]` and reflective optics concepts.

---

## DynamicSchema

`DynamicSchema` is a **type-erased schema container** that wraps a `Reflect.Unbound[_]` tree — all the structural information from a `Schema[A]` (field names, case names, type identities, validations, annotations) with no Scala functions attached. The two fundamental uses are validating `DynamicValue` instances at runtime, and transporting schemas across process boundaries.

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_])
```

`DynamicSchema`:

- Holds the full structural shape of a type without capturing constructors or deconstructors
- Can validate any `DynamicValue` against its structure using `DynamicSchema#check` and `DynamicSchema#conforms`
- Can be converted to a `DynamicValue` and back, and stored or sent as Avro bytes (see [Storing a DynamicSchema](#storing-a-dynamicschema))
- Can be rehydrated into a fully operational `Schema[A]` by supplying runtime bindings via `DynamicSchema#rebind`

## Motivation

A `Schema[A]` carries both *structure* (field names, type identities, validations) and *behaviour* (constructors and deconstructors as Scala closures). Closures cannot cross process boundaries. `DynamicSchema` is the structural half — a schema stripped of all Scala functions:

```
Schema[OrderPlaced]   ←── compile-time type, closures attached
       │
       │  Schema[A]#toDynamicSchema
       ▼
DynamicSchema         ←── no closures, stored in a registry
(Reflect.Unbound[_])       or sent over the wire as Avro bytes
       │
       │  DynamicSchema.toDynamicValue / DynamicSchema.fromDynamicValue
       ▼
DynamicValue          ←── in-memory form that a codec writes
       │
       │  DynamicSchema.fromDynamicValue (on the consumer side)
       ▼
DynamicSchema         ←── reconstructed from storage
       │
       │  DynamicSchema#rebind[OrderPlaced](resolver)
       ▼
Schema[OrderPlaced]   ←── operational again, can encode and decode
```

This pattern enables schema registries: the Checkout Service registers its event schema on startup; the Fulfillment Service fetches it and rebinds it against its own type definitions, guaranteeing it uses the exact schema that was in effect when the event was encoded. See [BindingResolver](binding-resolver.md) for the complete rebinding API.

## Creating a DynamicSchema

There are two ways to obtain a `DynamicSchema`: strip bindings from an existing typed schema, or reconstruct one from a serialized blob.

### `Schema[A]#toDynamicSchema`

`Schema[A]#toDynamicSchema` strips all runtime bindings from a typed schema, returning the structural skeleton as a `DynamicSchema`. The method is defined on `Schema`:

```scala
trait Schema[A] {
  def toDynamicSchema: DynamicSchema
}
```

This is the standard entry point. The structural information — field names, case names, type IDs, `Validation` constraints, `Modifier` annotations, docs, and stored examples and default values — is preserved. The default values of case class fields (`edition: Int = 1`) are not:

```scala

case class Address(street: String, city: String)
case class Person(name: String, age: Int, address: Address)

object Address { implicit val schema: Schema[Address] = Schema.derived[Address] }
object Person  { implicit val schema: Schema[Person]  = Schema.derived[Person]  }

val dynamic: DynamicSchema = Schema[Person].toDynamicSchema
```

We can inspect the schema's structure by printing it:

```scala
dynamic
// res1: DynamicSchema = DynamicSchema(
//   Record(
//     fields = IndexedSeq(
//       Term(
//         name = "name",
//         value = Primitive(
//           primitiveType = String(None),
//           typeId = String,
//           primitiveBinding = zio.blocks.schema.binding.NoBinding$$anon$1@51fa16c9,
//           doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//           modifiers = List(),
//           storedDefaultValue = None,
//           storedExamples = List()
//         ),
//         doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//         modifiers = List()
//       ),
//       Term(
//         name = "age",
//         value = Primitive(
//           primitiveType = Int(None),
//           typeId = Int,
//           primitiveBinding = zio.blocks.schema.binding.NoBinding$$anon$1@51fa16c9,
//           doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//           modifiers = List(),
//           storedDefaultValue = None,
//           storedExamples = List()
//         ),
//         doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//         modifiers = List()
//       ),
//       Term(
//         name = "address",
//         value = Record(
//           fields = IndexedSeq(
//             Term(
//               name = "street",
//               value = Primitive(
//                 primitiveType = String(None),
//                 typeId = String,
//                 primitiveBinding = zio.blocks.schema.binding.NoBinding$$anon$1@51fa16c9,
//                 doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                 modifiers = List(),
//                 storedDefaultValue = None,
//                 storedExamples = List()
//               ),
// ...
```

### `DynamicSchema.fromDynamicValue`

`DynamicSchema.fromDynamicValue` reconstructs a `DynamicSchema` from a previously serialized `DynamicValue` blob. This is the consumer-side entry point when schemas are loaded from a registry or database:

```scala
object DynamicSchema {
  def fromDynamicValue(dv: DynamicValue): DynamicSchema
}
```

Example showing the full store/retrieve round-trip:

```scala

case class Product(id: Long, name: String, price: Double)
object Product { implicit val schema: Schema[Product] = Schema.derived[Product] }

val original: DynamicSchema = Schema[Product].toDynamicSchema
val blob:     DynamicValue  = DynamicSchema.toDynamicValue(original)
val restored: DynamicSchema = DynamicSchema.fromDynamicValue(blob)
```

We inspect the restored schema's type name to confirm the round-trip preserved it:

```scala
restored.typeId.name
// res3: String = "Product"
```

## Serializing a DynamicSchema

A `DynamicSchema` turns into a `DynamicValue` in memory, and a codec turns it into bytes for storage or transmission. The format matters: only Avro reads a stored schema back intact in 0.0.56.

### `DynamicSchema.toDynamicValue`

`DynamicSchema.toDynamicValue` converts a `DynamicSchema` to a `DynamicValue`. The result contains only field names, type names, validations, and annotations — no Scala closures:

```scala
object DynamicSchema {
  def toDynamicValue(ds: DynamicSchema): DynamicValue
}
```

The following example converts a simple case class schema to its `DynamicValue` form:

```scala

case class Tag(name: String)
object Tag { implicit val schema: Schema[Tag] = Schema.derived[Tag] }

val ds:   DynamicSchema = Schema[Tag].toDynamicSchema
val blob: DynamicValue  = DynamicSchema.toDynamicValue(ds)
```

We inspect the blob's value type to confirm the record structure was preserved:

```scala
blob.valueType
// res5: DynamicValueType = <function1>
```

### Storing a DynamicSchema

`DynamicSchema.schema` is the `Schema[DynamicSchema]` that every format derives a codec from, and it is found implicitly as `Schema[DynamicSchema]`. In 0.0.56, only the Avro codec from the `zio-blocks-schema-avro` module round-trips a `DynamicSchema`. The JSON, YAML, TOON and MessagePack codecs write each tagged node of the schema as a plain record and narrow its numbers, and `DynamicSchema.fromDynamicValue` falls back to an accept-anything `Reflect.Dynamic` node instead of reporting an error, so the schema they read back accepts any value. This is a known bug. The same happens when the `DynamicValue` form goes through `Json`.

The following example writes a schema with both codecs and checks a value with the wrong field type against each copy:

```scala

case class Tag(name: String, weight: Int)
object Tag { implicit val schema: Schema[Tag] = Schema.derived[Tag] }

val ds: DynamicSchema = Schema[Tag].toDynamicSchema
val wrongType = DynamicValue.Record(Chunk(
  "name"   -> DynamicValue.int(7),
  "weight" -> DynamicValue.int(1)
))

val avroCodec = Schema[DynamicSchema].derive(AvroFormat)
val jsonCodec = Schema[DynamicSchema].derive(JsonFormat)

val fromAvro = avroCodec.decode(avroCodec.encode(ds))
val fromJson = jsonCodec.decode(jsonCodec.encode(ds))
```

The copy read from Avro equals the original and rejects the value, while the copy read from JSON decodes without an error, is a `Dynamic` node, and accepts it:

```scala
fromAvro.map(_ == ds)
// res7: Either[SchemaError, Boolean] = Right(true)
fromAvro.map(_.conforms(wrongType))
// res8: Either[SchemaError, Boolean] = Right(false)
fromJson.map(_.reflect.isDynamic)
// res9: Either[SchemaError, Boolean] = Right(true)
fromJson.map(_.conforms(wrongType))
// res10: Either[SchemaError, Boolean] = Right(true)
```

Store Avro bytes, and after loading a schema, reject one whose root is `Dynamic` (`DynamicSchema#reflect` and `Reflect#isDynamic`), so a later change of format can't silently turn the registry into one that accepts anything. Also in 0.0.56, the schema of a recursive type can't be stored: `DynamicSchema.toDynamicValue` and the codecs follow the recursion until they throw a `StackOverflowError`. [How to Build a Schema Registry with DynamicSchema](../../guides/build-a-schema-registry.md) builds a registry that stores Avro files.

## Validating DynamicValues

`DynamicSchema` can check whether a `DynamicValue` conforms to its structure. Validation is recursive: field counts and names, variant case names, collection element types, and primitive type + validation constraints are all checked.

### `DynamicSchema#check`

`DynamicSchema#check` returns `None` if the value is valid, or `Some(SchemaError)` describing the first validation failure:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def check(value: DynamicValue): Option[SchemaError]
}
```

The following example shows all three outcomes — a valid value, a missing field, and a type mismatch:

```scala

case class Point(x: Int, y: Int)
object Point { implicit val schema: Schema[Point] = Schema.derived[Point] }

val dynSchema = Schema[Point].toDynamicSchema

val valid = DynamicValue.Record(Chunk(
  "x" -> DynamicValue.int(3),
  "y" -> DynamicValue.int(7)
))

val missing = DynamicValue.Record(Chunk(
  "x" -> DynamicValue.int(3)
  // "y" missing
))

val wrongType = DynamicValue.Record(Chunk(
  "x" -> DynamicValue.int(3),
  "y" -> DynamicValue.string("not an int")
))
```

We evaluate all three cases to observe the outcomes:

```scala
dynSchema.check(valid)
// res12: Option[SchemaError] = None
dynSchema.check(missing)
// res13: Option[SchemaError] = Some(
//   SchemaError(
//     List(MissingField(source = DynamicOptic(ArraySeq()), fieldName = "y"))
//   )
// )
dynSchema.check(wrongType)
// res14: Option[SchemaError] = Some(
//   SchemaError(
//     List(
//       ExpectationMismatch(
//         source = DynamicOptic(ArraySeq(Field("y"))),
//         expectation = "Expected Int, got String"
//       )
//     )
//   )
// )
```

Validation rules:

- **Records**: every field in the schema must be present; extra fields in the value are rejected.
- **Variants**: the case name must be valid; the case payload is validated recursively.
- **Sequences**: every element is validated against the element schema.
- **Maps**: every key and every value is validated against their respective schemas.
- **Primitives**: the `PrimitiveType` must match, and any `Validation` constraints (range, pattern, etc.) must pass.

### `DynamicSchema#conforms`

`DynamicSchema#conforms` is a convenience method that returns `true` when the value is valid:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def conforms(value: DynamicValue): Boolean
}
```

We pass a well-formed record to confirm the return value:

```scala

case class Tag(name: String)
object Tag { implicit val schema: Schema[Tag] = Schema.derived[Tag] }

val dynSchema = Schema[Tag].toDynamicSchema
val value     = DynamicValue.Record(Chunk("name" -> DynamicValue.string("scala")))
```

```scala
dynSchema.conforms(value)
// res16: Boolean = true
```

## Rebinding to a Typed Schema

`DynamicSchema#rebind` converts a structural `DynamicSchema` back into a fully operational `Schema[A]` by walking the `Reflect.Unbound` tree and attaching runtime bindings from a `BindingResolver`:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def rebind[A](resolver: BindingResolver): Schema[A]
}
```

The resolver must provide a binding for every concrete type referenced in the schema tree: record types, variant types, and wrapper types must be covered explicitly; primitives, sequences, and maps are covered by `BindingResolver.defaults`:

```scala

case class OrderId(value: String)
case class LineItem(sku: String, quantity: Int)
case class Order(id: OrderId, items: List[LineItem])

object OrderId  { implicit val schema: Schema[OrderId]  = Schema.derived[OrderId]  }
object LineItem { implicit val schema: Schema[LineItem] = Schema.derived[LineItem] }
object Order    { implicit val schema: Schema[Order]    = Schema.derived[Order]    }

val blob: DynamicValue     = DynamicSchema.toDynamicValue(Schema[Order].toDynamicSchema)
val dynamic: DynamicSchema = DynamicSchema.fromDynamicValue(blob)

val resolver: BindingResolver =
  BindingResolver.empty
    .bind(Binding.of[OrderId])
    .bind(Binding.of[LineItem])
    .bind(Binding.of[Order])
    ++ BindingResolver.defaults

val rebound: Schema[Order] = dynamic.rebind[Order](resolver)
val order                  = Order(OrderId("ORD-1"), List(LineItem("SKU-A", 2)))
val encoded                = rebound.toDynamicValue(order)
```

We decode the encoded value to confirm the round-trip is lossless:

```scala
rebound.fromDynamicValue(encoded)
// res18: Either[SchemaError, Order] = Right(
//   Order(
//     id = OrderId("ORD-1"),
//     items = List(LineItem(sku = "SKU-A", quantity = 2))
//   )
// )
```

:::warning
If `DynamicSchema#rebind` cannot find a binding for any type in the unbound schema tree, it throws a `RebindException` at runtime. Ensure the resolver covers every concrete type — records, variants, wrappers, primitives, and standard collections — that appears in the schema. `BindingResolver.defaults` covers all standard primitives, `java.time` types, `List`, `Vector`, `Set`, `Map` and the other standard collections, but not `Option`.
:::

Rebinding a schema with an `Option` field and only `BindingResolver.defaults` behind the record fails. Consider a book with two optional fields:

```scala

case class Book(title: String, edition: Option[Int], subtitle: Option[String])
object Book { implicit val schema: Schema[Book] = Schema.derived[Book] }

val stored: DynamicSchema = Schema[Book].toDynamicSchema
```

The rebind fails on an `Option` case it can't bind and names its type and path:

```scala
try stored.rebind[Book](BindingResolver.empty.bind(Binding.of[Book]) ++ BindingResolver.defaults)
catch { case e: RebindException => e.getMessage }
// res20: Schema[Book] | String = "Missing Record binding for type scala.Some at path: .subtitle<Some>"
```

Each `Option[A]` needs three bindings: `Binding.of[Option[A]]` for the `Option` itself, a `Some` case for its element type, and `Binding.Record.none`, which serves every element type; leaving out the last one fails the same way, naming `scala.None`. `Binding.Record.some[A]` builds the `Some` case for a reference type such as `String`, and `Binding.Record.someInt`, `Binding.Record.someLong`, `Binding.Record.someDouble` and their siblings build it for primitives:

```scala
val withOptions: BindingResolver =
  BindingResolver.empty
    .bind(Binding.of[Book])
    .bind(Binding.of[Option[Int]])
    .bind(Binding.Record.someInt)
    .bind(Binding.of[Option[String]])
    .bind(Binding.Record.some[String])
    .bind(Binding.Record.none)
    ++ BindingResolver.defaults

val reboundBook: Schema[Book] = stored.rebind[Book](withOptions)
```

The rebound schema converts a `Book` both ways:

```scala
reboundBook.fromDynamicValue(reboundBook.toDynamicValue(Book("Zionomicon", Some(1), None)))
// res21: Either[SchemaError, Book] = Right(
//   Book(title = "Zionomicon", edition = Some(1), subtitle = None)
// )
```

See [BindingResolver](binding-resolver.md) for the full resolver API including `BindingResolver.reflection` for automatic binding via reflection.

## Structural Navigation

`DynamicSchema#get` navigates into the schema tree using a `DynamicOptic` path, returning the nested `Reflect.Unbound[_]` at that location:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def get(optic: DynamicOptic): Option[Reflect.Unbound[_]]
}
```

The following example navigates to a field nested two levels deep:

```scala

case class Address(street: String, city: String)
case class Person(name: String, age: Int, address: Address)

object Address { implicit val schema: Schema[Address] = Schema.derived[Address] }
object Person  { implicit val schema: Schema[Person]  = Schema.derived[Person]  }

val dynSchema = Schema[Person].toDynamicSchema
val streetSchema: Option[Reflect.Unbound[_]] =
  dynSchema.get(DynamicOptic.root.field("address").field("street"))
```

We map over the result to read the type name at that path:

```scala
streetSchema.map(_.typeId.name)
// res23: Option[String] = Some("String")
```

See [DynamicOptic](./dynamic-optic.md) for the full path DSL.

## Metadata Access and Updates

`DynamicSchema` provides read and write access to the metadata stored in the `Reflect` tree.

### `DynamicSchema#typeId`

Returns the `TypeId` of the root type:

```scala

case class Event(id: Long, kind: String)
object Event { implicit val schema: Schema[Event] = Schema.derived[Event] }

val dynSchema = Schema[Event].toDynamicSchema
```

```scala
dynSchema.typeId.name
// res25: String = "Event"
dynSchema.typeId.fullName
// res26: String = "repl.MdocSession.MdocApp24.Event"
```

### `DynamicSchema#doc`

Reads and writes the documentation annotation on the schema. The zero-argument form reads the current `Doc`; the single-argument form returns a copy with updated documentation:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def doc: Doc
  def doc(value: String): DynamicSchema
}
```

We attach a description and read it back to confirm it was applied:

```scala

case class Event(id: Long, kind: String)
object Event { implicit val schema: Schema[Event] = Schema.derived[Event] }

val updated: DynamicSchema = Schema[Event].toDynamicSchema.doc("An event in the event log")
```

```scala
updated.doc
// res28: Doc = Doc(
//   blocks = IndexedSeq(Paragraph(IndexedSeq(Text("An event in the event log")))),
//   metadata = Map()
// )
```

### `DynamicSchema#modifiers` and `DynamicSchema#modifier`

`DynamicSchema#modifiers` returns the `Modifier.Reflect` annotations attached to the root node. `DynamicSchema#modifier` returns a copy with an additional modifier appended:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def modifiers: Seq[Modifier.Reflect]
  def modifier(m: Modifier.Reflect): DynamicSchema
}
```

### `DynamicSchema#getDefaultValue` and `DynamicSchema#defaultValue`

`DynamicSchema#getDefaultValue` returns the stored default `DynamicValue`, if one is set. `DynamicSchema#defaultValue` returns a copy with the given default:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def getDefaultValue: Option[DynamicValue]
  def defaultValue(value: DynamicValue): DynamicSchema
}
```

We attach a default value and then retrieve it:

```scala

case class Config(retries: Int)
object Config { implicit val schema: Schema[Config] = Schema.derived[Config] }

val withDefault: DynamicSchema =
  Schema[Config].toDynamicSchema
    .defaultValue(DynamicValue.Record(
      zio.blocks.chunk.Chunk("retries" -> DynamicValue.int(3))
    ))
```

```scala
withDefault.getDefaultValue
// res30: Option[DynamicValue] = Some(
//   Record(IndexedSeq(("retries", Primitive(Int(3)))))
// )
```

### `DynamicSchema#examples`

The zero-argument form returns stored examples as a `Seq[DynamicValue]`. The multi-argument form returns a copy with the given examples set:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def examples: Seq[DynamicValue]
  def examples(value: DynamicValue, values: DynamicValue*): DynamicSchema
}
```

## Converting to a Typed Schema

`DynamicSchema#toSchema` returns a `Schema[DynamicValue]` — it stays fully in the dynamic world and requires no bindings. Use it when you have received a `DynamicSchema` over the wire and need a codec-compatible schema that enforces structural conformance without binding any Scala types.

After transporting a `DynamicSchema`, you may not have the Scala types that `DynamicSchema#rebind` requires. For example, when the consumer is a validation gateway or format converter that handles arbitrary event shapes, any codec pipeline that accepts `Schema[DynamicValue]` can use the result directly. The conversion runs `DynamicSchema#check`, which compares primitive types exactly, so `DynamicSchema#toSchema` suits input whose `DynamicValue`s carry the exact primitive types, such as values made by `Schema#toDynamicValue` or read with `Schema[DynamicValue].derive(AvroFormat)`.

JSON input is a poor fit. The JSON codec of `Schema[DynamicValue]` reads a string as a `String`, a whole number as an `Int` or a `Long`, every other number as a `BigDecimal`, and an `Option` as a bare value or `null`, so a valid JSON event is rejected when it has a field of another primitive type (such as `Double`, `Float`, `BigInt`, `Instant` or `UUID`), a `Long` field holding a value that fits in an `Int`, a `BigDecimal` field holding a whole number, or an `Option` field. For JSON input, rebind the schema and decode with the rebound schema's JSON codec.

:::warning
Use `DynamicSchema#rebind` instead when you have a `BindingResolver` and need a fully operational `Schema[A]` for typed encoding and decoding.
:::
`DynamicSchema#toSchema` is defined as:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def toSchema: Schema[DynamicValue]
}
```

The following example shows a schema-validation gateway: we receive a `DynamicSchema` from a registry, convert it to a `Schema[DynamicValue]`, and use the result to validate an incoming payload, rejecting it on structural mismatch:

```scala

case class OrderEvent(orderId: String, amount: Double)
object OrderEvent { implicit val schema: Schema[OrderEvent] = Schema.derived[OrderEvent] }

val blob: DynamicValue          = DynamicSchema.toDynamicValue(Schema[OrderEvent].toDynamicSchema)
val received: DynamicSchema     = DynamicSchema.fromDynamicValue(blob)
val gatewaySchema: Schema[DynamicValue] = received.toSchema

val incoming: DynamicValue = DynamicValue.Record(
  zio.blocks.chunk.Chunk(
    "orderId" -> DynamicValue.string("ORD-42"),
    "amount"  -> DynamicValue.double(99.95)
  )
)
```

We validate the payload and observe the result:

```scala
gatewaySchema.fromDynamicValue(incoming)
// res32: Either[SchemaError, DynamicValue] = Right(
//   Record(
//     IndexedSeq(
//       ("orderId", Primitive(String("ORD-42"))),
//       ("amount", Primitive(Double(99.95)))
//     )
//   )
// )
```

The same order sent as JSON is rejected, because the JSON codec reads `99.95` as a `BigDecimal` while the schema expects a `Double`:

```scala
gatewaySchema.jsonCodec.decode("""{"orderId":"ORD-42","amount":99.95}""").left.map(_.getMessage)
// res33: Either[String, DynamicValue] = Left(
//   "Expected Double, got BigDecimal at: .amount at: .wrapped"
// )
```

## Integration

`DynamicSchema` connects to several other ZIO Blocks types, each serving a distinct role in the dynamic layer.

### With `DynamicValue`

`DynamicSchema` and `DynamicValue` are the two halves of ZIO Blocks' dynamic layer. `DynamicValue` holds runtime data without compile-time types; `DynamicSchema` holds structural metadata without runtime bindings. Together they enable fully type-erased validation and serialization pipelines. See [DynamicValue](./dynamic-value.md).

### With `BindingResolver`

The primary consumer of `DynamicSchema` in a type-safe context is `DynamicSchema#rebind`, which requires a `BindingResolver` to reattach runtime bindings. See [BindingResolver](binding-resolver.md) for the complete rebinding API, including `BindingResolver.reflection` for automatic binding discovery.

### With `DynamicOptic`

`DynamicSchema#get` accepts a `DynamicOptic` path to navigate the structural tree. This allows you to inspect nested schemas, extract type information, or validate sub-trees independently. See [DynamicOptic](./dynamic-optic.md).

### With `Schema`

`Schema[A]#toDynamicSchema` is defined on `Schema` and is the standard way to obtain a `DynamicSchema`. See [Schema](./schema.md).

See also: [How to Build a Schema Registry with DynamicSchema](../../guides/build-a-schema-registry.md), a guide that stores versioned schemas as data and checks incoming events against them.

## Running the Examples

All code from this guide is available as runnable examples in the `schema-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

**Validate DynamicValues against a DynamicSchema**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/dynamicschema/DynamicSchemaValidationExample.scala))

```bash
cd schema-examples && sbt "runMain dynamicschema.DynamicSchemaValidationExample"
```

**Serialize and deserialize a DynamicSchema**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/dynamicschema/DynamicSchemaSerializationExample.scala))

```bash
cd schema-examples && sbt "runMain dynamicschema.DynamicSchemaSerializationExample"
```

**Rebind a restored DynamicSchema to a typed Schema**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/dynamicschema/DynamicSchemaRebindExample.scala))

```bash
cd schema-examples && sbt "runMain dynamicschema.DynamicSchemaRebindExample"
```

**Complete schema registry pipeline**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/dynamicschema/DynamicSchemaRegistryExample.scala))

```bash
cd schema-examples && sbt "runMain dynamicschema.DynamicSchemaRegistryExample"
```

---

## DynamicValue

`DynamicValue` is a schema-less, dynamically-typed representation of any structured value in ZIO Blocks. It provides a universal data model that can represent any value without requiring compile-time type information, serving as an intermediate representation for serialization, schema evolution, data transformation, and cross-format conversion.

## Overview

The `DynamicValue` type represents all structured values with six cases:

```
DynamicValue
 ├── DynamicValue.Primitive   (scalar values: strings, numbers, booleans, temporal types, etc.)
 ├── DynamicValue.Record      (named fields, analogous to case classes or JSON objects)
 ├── DynamicValue.Variant     (tagged unions, analogous to sealed traits)
 ├── DynamicValue.Sequence    (ordered collections: lists, arrays, vectors)
 ├── DynamicValue.Map         (key-value pairs where keys are also DynamicValues)
 └── DynamicValue.Null        (absence of a value)
```

Key design decisions:

- **Type-agnostic** — Works without compile-time type information
- **Preserves structure** — Maintains full fidelity of the original data
- **Supports rich primitives** — All Java time types, BigDecimal, UUID, Currency, etc.
- **Path-based navigation** — Uses `DynamicOptic` for traversal and modification
- **EJSON toString** — Human-readable output format with type annotations

## DynamicValue Variants

### Primitive

Wraps scalar values in a `PrimitiveValue`:

```scala

// Using convenience constructors
val str = DynamicValue.string("hello")
val num = DynamicValue.int(42)
val flag = DynamicValue.boolean(true)
val pi = DynamicValue.double(3.14159)

// Using the Primitive case directly

val instant = DynamicValue.Primitive(
  PrimitiveValue.Instant(java.time.Instant.now())
)
```

### Record

A collection of named fields, analogous to case classes or JSON objects:

```scala

// Using varargs constructor
val person = DynamicValue.Record(
  "name" -> DynamicValue.string("Alice"),
  "age" -> DynamicValue.int(30),
  "active" -> DynamicValue.boolean(true)
)

// Using Chunk constructor
val point = DynamicValue.Record(Chunk(
  ("x", DynamicValue.int(10)),
  ("y", DynamicValue.int(20))
))

// Empty record
val empty = DynamicValue.Record.empty
```

Field order is preserved and significant for equality. Use `sortFields` to normalize for order-independent comparison.

### Variant

A tagged union value, analogous to sealed traits:

```scala

// A Some variant containing a value
val some = DynamicValue.Variant(
  "Some",
  DynamicValue.string("hello")
)

// A None variant with an empty record
val none = DynamicValue.Variant("None", DynamicValue.Record.empty)

// Access case information
some.caseName   // Some("Some")
some.caseValue  // Some(DynamicValue.Primitive(...))
```

### Sequence

An ordered collection of values:

```scala

// Using varargs constructor
val numbers = DynamicValue.Sequence(
  DynamicValue.int(1),
  DynamicValue.int(2),
  DynamicValue.int(3)
)

// Using Chunk constructor
val items = DynamicValue.Sequence(Chunk(
  DynamicValue.string("a"),
  DynamicValue.string("b")
))

// Empty sequence
val empty = DynamicValue.Sequence.empty
```

### Map

Key-value pairs where both keys and values are `DynamicValue`:

```scala

// String keys (common case)
val config = DynamicValue.Map(
  DynamicValue.string("host") -> DynamicValue.string("localhost"),
  DynamicValue.string("port") -> DynamicValue.int(8080)
)

// Non-string keys (unlike Record)
val mapping = DynamicValue.Map(
  DynamicValue.int(1) -> DynamicValue.string("one"),
  DynamicValue.int(2) -> DynamicValue.string("two")
)

// Empty map
val empty = DynamicValue.Map.empty
```

Unlike `Record` which uses String keys, `Map` supports arbitrary `DynamicValue` keys.

### Null

Represents the absence of a value:

```scala

val absent = DynamicValue.Null
```

## PrimitiveValue Types

`PrimitiveValue` is a sealed trait representing all scalar values that can be wrapped in `DynamicValue.Primitive`. Each case preserves full type information:

| Type | Description | Example |
|------|-------------|---------|
| `Unit` | Unit value | `PrimitiveValue.Unit` |
| `Boolean` | Boolean | `PrimitiveValue.Boolean(true)` |
| `Byte` | 8-bit integer | `PrimitiveValue.Byte(127)` |
| `Short` | 16-bit integer | `PrimitiveValue.Short(32767)` |
| `Int` | 32-bit integer | `PrimitiveValue.Int(42)` |
| `Long` | 64-bit integer | `PrimitiveValue.Long(9999999999L)` |
| `Float` | 32-bit float | `PrimitiveValue.Float(3.14f)` |
| `Double` | 64-bit float | `PrimitiveValue.Double(3.14159)` |
| `Char` | Unicode character | `PrimitiveValue.Char('A')` |
| `String` | Text | `PrimitiveValue.String("hello")` |
| `BigInt` | Arbitrary precision integer | `PrimitiveValue.BigInt(BigInt("999..."))` |
| `BigDecimal` | Arbitrary precision decimal | `PrimitiveValue.BigDecimal(BigDecimal("3.14159"))` |
| `Instant` | Timestamp | `PrimitiveValue.Instant(Instant.now())` |
| `LocalDate` | Date without time | `PrimitiveValue.LocalDate(LocalDate.now())` |
| `LocalDateTime` | Date and time | `PrimitiveValue.LocalDateTime(LocalDateTime.now())` |
| `LocalTime` | Time without date | `PrimitiveValue.LocalTime(LocalTime.now())` |
| `Duration` | Time duration | `PrimitiveValue.Duration(Duration.ofHours(1))` |
| `Period` | Date-based period | `PrimitiveValue.Period(Period.ofDays(30))` |
| `DayOfWeek` | Day of week | `PrimitiveValue.DayOfWeek(DayOfWeek.MONDAY)` |
| `Month` | Month | `PrimitiveValue.Month(Month.JANUARY)` |
| `Year` | Year | `PrimitiveValue.Year(Year.of(2024))` |
| `YearMonth` | Year and month | `PrimitiveValue.YearMonth(YearMonth.of(2024, 1))` |
| `MonthDay` | Month and day | `PrimitiveValue.MonthDay(MonthDay.of(1, 15))` |
| `ZoneId` | Time zone | `PrimitiveValue.ZoneId(ZoneId.of("UTC"))` |
| `ZoneOffset` | Time zone offset | `PrimitiveValue.ZoneOffset(ZoneOffset.UTC)` |
| `ZonedDateTime` | Date/time with zone | `PrimitiveValue.ZonedDateTime(ZonedDateTime.now())` |
| `OffsetDateTime` | Date/time with offset | `PrimitiveValue.OffsetDateTime(OffsetDateTime.now())` |
| `OffsetTime` | Time with offset | `PrimitiveValue.OffsetTime(OffsetTime.now())` |
| `UUID` | Universally unique ID | `PrimitiveValue.UUID(UUID.randomUUID())` |
| `Currency` | Currency | `PrimitiveValue.Currency(Currency.getInstance("USD"))` |

## Creating DynamicValues from Typed Values

Use `Schema.toDynamicValue` to convert typed Scala values to `DynamicValue`:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val person = Person("Alice", 30)
val dynamic: DynamicValue = Schema[Person].toDynamicValue(person)
// Record with "name" and "age" fields

// Works with any type that has a Schema
val listDynamic = Schema[List[Int]].toDynamicValue(List(1, 2, 3))
// Sequence of Primitive(Int) values
```

## Converting DynamicValues Back to Typed Values

Use `Schema.fromDynamicValue` to convert `DynamicValue` back to typed Scala values:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val dynamic = DynamicValue.Record(
  "name" -> DynamicValue.string("Bob"),
  "age" -> DynamicValue.int(25)
)

val result: Either[SchemaError, Person] = Schema[Person].fromDynamicValue(dynamic)
// Right(Person("Bob", 25))

// Type mismatch produces an error
val badDynamic = DynamicValue.string("not a person")
val error = Schema[Person].fromDynamicValue(badDynamic)
// Left(SchemaError(...))
```

## Type Information

### DynamicValueType

Each `DynamicValue` has a corresponding `DynamicValueType` for runtime type checking:

```scala

val dv = DynamicValue.Record("x" -> DynamicValue.int(1))

// Check type
dv.is(DynamicValueType.Record)    // true
dv.is(DynamicValueType.Sequence)  // false

// Narrow to specific type
val record: Option[DynamicValue.Record] = dv.as(DynamicValueType.Record)
// Some(Record(...))

// Extract underlying value

val fields: Option[Chunk[(String, DynamicValue)]] = 
  dv.unwrap(DynamicValueType.Record)
```

### Extracting Primitive Values

```scala

val dv = DynamicValue.int(42)

// Extract with specific primitive type
val intValue: Option[Int] = dv.asPrimitive(PrimitiveType.Int(Validation.None))
// Some(42)

val stringValue: Option[String] = dv.asPrimitive(PrimitiveType.String(Validation.None))
// None (type mismatch)
```

## Navigation

### Simple Navigation

Navigate using `get` methods that return `DynamicValueSelection`:

```scala

val data = DynamicValue.Record(
  "users" -> DynamicValue.Sequence(
    DynamicValue.Record("name" -> DynamicValue.string("Alice")),
    DynamicValue.Record("name" -> DynamicValue.string("Bob"))
  )
)

// Navigate to a field
val users = data.get("users")  // DynamicValueSelection

// Navigate to an array element
val firstUser = data.get("users").apply(0)

// Chain navigation
val firstName = data.get("users").apply(0).get("name")

// Extract the value
val name = firstName.one  // Either[SchemaError, DynamicValue]
```

### Path-Based Navigation with DynamicOptic

Use `DynamicOptic` for complex path expressions:

```scala

val data = DynamicValue.Record(
  "company" -> DynamicValue.Record(
    "employees" -> DynamicValue.Sequence(
      DynamicValue.Record("name" -> DynamicValue.string("Alice"))
    )
  )
)

// Build a path
val path = DynamicOptic.root.field("company").field("employees").at(0).field("name")

// Navigate using the path
val result = data.get(path).one  // Right(DynamicValue.Primitive(String("Alice")))
```

### DynamicValueSelection

`DynamicValueSelection` wraps navigation results and provides fluent chaining:

```scala

val selection: DynamicValueSelection = ???

// Terminal operations
selection.one      // Either[SchemaError, DynamicValue] - exactly one value
selection.any      // Either[SchemaError, DynamicValue] - first of many
selection.all      // Either[SchemaError, DynamicValue] - wrap multiple in Sequence
selection.toChunk  // Chunk[DynamicValue] - empty on error

// Type filtering
selection.primitives  // Only Primitive values
selection.records     // Only Record values
selection.sequences   // Only Sequence values
selection.maps        // Only Map values

// Combinators
selection.map(dv => ???)      // Transform values
selection.filter(dv => ???)   // Filter values
selection.flatMap(dv => ???)  // Chain selections
```

## Path-Based Modification

### Modify

Update values at a path:

```scala

val data = DynamicValue.Record(
  "user" -> DynamicValue.Record(
    "name" -> DynamicValue.string("Alice")
  )
)

val path = DynamicOptic.root.field("user").field("name")

// Modify value at path
val updated = data.modify(path)(dv => DynamicValue.string("Bob"))
// Record("user" -> Record("name" -> "Bob"))
```

### Set

Replace a value at a path:

```scala

val data = DynamicValue.Record("x" -> DynamicValue.int(1))
val path = DynamicOptic.root.field("x")

val updated = data.set(path, DynamicValue.int(99))
// Record("x" -> 99)
```

### Delete

Remove a value at a path:

```scala

val data = DynamicValue.Record(
  "a" -> DynamicValue.int(1),
  "b" -> DynamicValue.int(2)
)

val updated = data.delete(DynamicOptic.root.field("a"))
// Record("b" -> 2)
```

### Insert

Add a value at a path (fails if path exists):

```scala

val data = DynamicValue.Record("a" -> DynamicValue.int(1))

val updated = data.insert(
  DynamicOptic.root.field("b"),
  DynamicValue.int(2)
)
// Record("a" -> 1, "b" -> 2)
```

### Fallible Operations

Use `*OrFail` variants for operations that should fail explicitly:

```scala

val data = DynamicValue.Record("x" -> DynamicValue.int(1))
val badPath = DynamicOptic.root.field("nonexistent")

val result: Either[SchemaError, DynamicValue] = 
  data.setOrFail(badPath, DynamicValue.int(99))
// Left(SchemaError("Path not found"))
```

## EJSON-like toString Format

`DynamicValue.toString` produces an EJSON (Extended JSON) format that:

- Uses unquoted field names for Records (like Scala syntax)
- Uses quoted string keys for Maps
- Adds `@ {tag: "..."}` annotations for Variants
- Adds `@ {type: "..."}` annotations for typed primitives (Instant, Duration, etc.)

```scala

val person = DynamicValue.Record(
  "name" -> DynamicValue.string("Alice"),
  "age" -> DynamicValue.int(30)
)

println(person.toString)
// {
//   name: "Alice",
//   age: 30
// }

val variant = DynamicValue.Variant(
  "Some",
  DynamicValue.string("hello")
)

println(variant.toString)
// "hello" @ {tag: "Some"}

val timestamp = DynamicValue.Primitive(
  PrimitiveValue.Instant(java.time.Instant.ofEpochMilli(1700000000000L))
)

println(timestamp.toString)
// 1700000000000 @ {type: "instant"}
```

Use `toEjson(indent)` to control indentation level.

## Merging Strategies

Merge two `DynamicValue` structures using configurable strategies:

```scala

val left = DynamicValue.Record(
  "a" -> DynamicValue.int(1),
  "b" -> DynamicValue.int(2)
)

val right = DynamicValue.Record(
  "b" -> DynamicValue.int(99),
  "c" -> DynamicValue.int(3)
)

// Deep merge (default): recursively merge containers
val merged = left.merge(right, DynamicValueMergeStrategy.Auto)
// Record("a" -> 1, "b" -> 99, "c" -> 3)
```

### Available Strategies

| Strategy | Behavior |
|----------|----------|
| `Auto` | Deep merge: Records by field, Sequences by index, Maps by key. Right wins at leaves. |
| `Replace` | Complete replacement: right value replaces left entirely |
| `KeepLeft` | Always keep left value |
| `Shallow` | Merge only at root level, nested containers replaced |
| `Concat` | Concatenate Sequences instead of merging by index |
| `Custom(f, r)` | Custom function with custom recursion control |

```scala

val list1 = DynamicValue.Sequence(DynamicValue.int(1), DynamicValue.int(2))
val list2 = DynamicValue.Sequence(DynamicValue.int(3))

// Concat sequences instead of index-based merge
val concatted = list1.merge(list2, DynamicValueMergeStrategy.Concat)
// Sequence(1, 2, 3)
```

## Normalization

Transform `DynamicValue` structures for comparison or serialization:

```scala

val data = DynamicValue.Record(
  "z" -> DynamicValue.int(1),
  "a" -> DynamicValue.Null,
  "m" -> DynamicValue.int(2)
)

// Sort fields alphabetically
data.sortFields
// Record("a" -> null, "m" -> 2, "z" -> 1)

// Remove null values
data.dropNulls
// Record("z" -> 1, "m" -> 2)

// Remove empty containers
data.dropEmpty

// Remove Unit primitives
data.dropUnits

// Apply all normalizations
data.normalize
// Sorted, no nulls, no units, no empty containers
```

## Transformation

### Transform Up/Down

Apply functions to all values in a structure:

```scala

val data = DynamicValue.Record(
  "values" -> DynamicValue.Sequence(
    DynamicValue.int(1),
    DynamicValue.int(2)
  )
)

// Bottom-up: children transformed before parents
val doubled = data.transformUp { (path, dv) =>
  dv match {
    case DynamicValue.Primitive(pv: PrimitiveValue.Int) =>
      DynamicValue.int(pv.value * 2)
    case other => other
  }
}

// Top-down: parents transformed before children
val topDown = data.transformDown { (path, dv) => ??? }
```

### Transform Field Names

Rename all record fields:

```scala

val data = DynamicValue.Record(
  "first_name" -> DynamicValue.string("Alice"),
  "last_name" -> DynamicValue.string("Smith")
)

// Convert snake_case to camelCase
val camelCase = data.transformFields { (path, name) =>
  name.split("_").zipWithIndex.map {
    case (word, 0) => word
    case (word, _) => word.capitalize
  }.mkString
}
// Record("firstName" -> "Alice", "lastName" -> "Smith")
```

## Folding

Aggregate values from a `DynamicValue` tree:

```scala

val data = DynamicValue.Record(
  "a" -> DynamicValue.int(1),
  "b" -> DynamicValue.int(2),
  "c" -> DynamicValue.int(3)
)

// Sum all integers
val sum = data.foldUp(0) { (path, dv, acc) =>
  dv match {
    case DynamicValue.Primitive(pv: PrimitiveValue.Int) => acc + pv.value
    case _ => acc
  }
}
// 6
```

## Converting to/from JSON

### To JSON

```scala

val dynamic = DynamicValue.Record(
  "name" -> DynamicValue.string("Alice"),
  "age" -> DynamicValue.int(30)
)

val json: Json = dynamic.toJson
// Json.Object with "name" and "age" fields
```

### From JSON

```scala

val json = Json.parseUnsafe("""{"name": "Alice", "age": 30}""")

val dynamic: DynamicValue = json.toDynamicValue
// DynamicValue.Record with "name" and "age" fields
```

## Querying

Search recursively for values matching a predicate:

```scala

val data = DynamicValue.Record(
  "users" -> DynamicValue.Sequence(
    DynamicValue.Record("name" -> DynamicValue.string("Alice"), "active" -> DynamicValue.boolean(true)),
    DynamicValue.Record("name" -> DynamicValue.string("Bob"), "active" -> DynamicValue.boolean(false))
  )
)

// Find all string values
val strings = data.select.query(_.is(DynamicValueType.Primitive))
  .filter(_.primitiveValue.exists(_.isInstanceOf[PrimitiveValue.String]))

// Query with path predicate
val atDepth2 = data.select.queryPath(path => path.nodes.length == 2)
```

## Use Cases

### Schema-less Operations

Work with data when the schema isn't known at compile time:

```scala

def processAnyData(data: DynamicValue): DynamicValue = {
  // Add a timestamp to any record
  data match {
    case r: DynamicValue.Record =>
      DynamicValue.Record(
        r.fields :+ ("processedAt" -> DynamicValue.Primitive(
          PrimitiveValue.Instant(java.time.Instant.now())
        ))
      )
    case other => other
  }
}
```

### Schema Migrations

Transform data between schema versions:

```scala

def migrateV1toV2(data: DynamicValue): DynamicValue = {
  data.transformFields { (path, name) =>
    // Rename deprecated field
    if (name == "userName") "name"
    else name
  }.transformUp { (path, dv) =>
    // Add default for new required field
    dv match {
      case r: DynamicValue.Record if path.nodes.isEmpty =>
        DynamicValue.Record(r.fields :+ ("version" -> DynamicValue.int(2)))
      case other => other
    }
  }
}
```

### Dynamic Queries

Build queries at runtime:

```scala

def buildPath(fields: List[String]): DynamicOptic =
  fields.foldLeft(DynamicOptic.root)(_.field(_))

def getValue(data: DynamicValue, path: List[String]): Option[DynamicValue] =
  data.get(buildPath(path)).one.toOption

// Usage
val data = DynamicValue.Record(
  "user" -> DynamicValue.Record(
    "profile" -> DynamicValue.Record(
      "email" -> DynamicValue.string("alice@example.com")
    )
  )
)

val email = getValue(data, List("user", "profile", "email"))
// Some(DynamicValue.Primitive(String("alice@example.com")))
```

### Cross-Format Conversion

Use `DynamicValue` as an intermediate format:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// JSON -> DynamicValue -> Typed
val json = Json.parseUnsafe("""{"name": "Alice", "age": 30}""")
val dynamic = json.toDynamicValue
val person = Schema[Person].fromDynamicValue(dynamic)

// Typed -> DynamicValue -> JSON
val dynamic2 = Schema[Person].toDynamicValue(Person("Bob", 25))
val json2 = dynamic2.toJson
```

## Comparison and Ordering

`DynamicValue` has a total ordering for sorting and comparison:

```scala

val a = DynamicValue.int(1)
val b = DynamicValue.int(2)

a.compare(b)  // negative
a < b         // true
a >= b        // false

// Type ordering: Primitive < Record < Variant < Sequence < Map < Null
val primitive = DynamicValue.int(1)
val record = DynamicValue.Record.empty
primitive < record  // true
```

## Diff and Patch

Compute differences between `DynamicValue` instances:

```scala

val old = DynamicValue.Record(
  "name" -> DynamicValue.string("Alice"),
  "age" -> DynamicValue.int(30)
)

val new_ = DynamicValue.Record(
  "name" -> DynamicValue.string("Alice"),
  "age" -> DynamicValue.int(31)
)

val patch: DynamicPatch = old.diff(new_)
// Patch that updates "age" from 30 to 31
```

## See Also

- [DynamicSchema](./dynamic-schema.md) — validate `DynamicValue` instances against a structural schema, or transport schemas as `DynamicValue` blobs.

---

## Format Type

A `Format` is an abstraction that bundles together everything needed to serialize and deserialize data in a specific format (JSON, Avro, MessagePack, etc.). It unifies metadata related to serialization formats, such as MIME type and codec deriver, in a single place.

## Overview

The `Format` trait defines the structure for any serialization format. Each format specifies the types for decoding input, encoding output, the codec typeclass, MIME type, and the deriver used to generate codecs from schemas:

```scala

sealed trait Format {
  type DecodeInput
  type EncodeOutput
  type TypeClass[A] <: Codec[DecodeInput, EncodeOutput, A]
  def mimeType: String
  def deriver: Deriver[TypeClass]
}
```

This design allows for a consistent API across different formats when deriving codecs from schemas. Having MIME type information helps with runtime content negotiation and format routing, for example in HTTP servers or message queues.

## Using a Format

You can easily call [`Schema[A].derive(format)`](./type-class-derivation.md#using-the-deriver-to-derive-type-class-instances) for any format that implements the `Format` trait to obtain a codec that can encode and decode values of type `A` according to that format's rules:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(ToonFormat)
val bytes: Array[Byte] = codec.encode(Person("Alice", 30))
val decoded: Either[SchemaError, Person] = codec.decode(bytes)
```

## Format Categories

Formats are categorized into `BinaryFormat` and `TextFormat`, which specify the types of input and output for encoding and decoding:

```scala

abstract class BinaryFormat[TC[A] <: BinaryCodec[A]](val mimeType: String, val deriver: Deriver[TC]) extends Format {
  type DecodeInput  = ByteBuffer
  type EncodeOutput = ByteBuffer
}

abstract class TextFormat[TC[A] <: TextCodec[A]](val mimeType: String, val deriver: Deriver[TC]) extends Format {
  type DecodeInput  = CharBuffer
  type EncodeOutput = CharBuffer
}
```

**BinaryFormat** — Encodes and decodes data as bytes (`ByteBuffer`). Used for binary formats (Avro, Thrift, MessagePack) and also for the text-based formats whose codecs read and write UTF-8 bytes (JSON, YAML, XML, TOON).

**TextFormat** — Encodes and decodes data as characters (`CharBuffer`). Among the built-in formats, only CSV (`CsvFormat`) is a `TextFormat`.

### Example: JsonFormat

The `JsonFormat` is a `BinaryFormat` that represents JSON serialization, where both the input for decoding and output for encoding are `ByteBuffer`, the MIME type is `application/json`, and the deriver for generating codecs from schemas is `JsonCodecDeriver`:

```scala

object MyJsonFormat extends zio.blocks.schema.codec.BinaryFormat("application/json", JsonCodecDeriver)
```

## Defining a Custom Format

To add a new serialization format, define a codec base class that extends `BinaryCodec` (or `TextCodec`), a `Deriver` that builds those codecs from a schema, and a `BinaryFormat` (or `TextFormat`) singleton that ties them to a MIME type:

```scala

// The codec type of the format: every codec the deriver builds extends it
abstract class MyCodec[A] extends BinaryCodec[A]

// One method per kind of schema node: primitive, record, variant, sequence, map, wrapper and dynamic value
lazy val myDeriver: Deriver[MyCodec] = ???

object MyFormat extends BinaryFormat[MyCodec]("application/x-myformat", myDeriver)
```

`MyFormat` then works wherever a built-in format does, as in `Schema[A].derive(MyFormat)` and `Schema#decode(MyFormat)`. The guide [How to Write a Codec for a New Wire Format](../../guides/write-a-codec-for-a-new-wire-format.md) implements every `Deriver` method for a small binary format, with decode errors that carry a path, and [Type-class Derivation](./type-class-derivation.md) explains each method's parameters.

## Available Formats

ZIO Blocks provides multiple built-in formats. See the [Built-in Formats and Codecs](./built-in-codecs/index.md) section for a complete list of supported serialization formats and their respective codec modules.

## See Also

- [One Type, Many Formats](../../guides/one-type-many-formats.md) - a tutorial that derives several built-in formats from one `Schema` and round-trips data through each.
- [Built-in Formats and Codecs](./built-in-codecs/index.md) - every built-in format with its codec type and module.

---

## ZIO Blocks Schema

## Introduction

ZIO Blocks Schema is the core type system and serialization framework that provides reified structural metadata for Scala data types. It enables type-safe schema definition, validation, optics-based data access, multi-format serialization, and runtime introspection — all derived from a single `Schema` definition.

**Core Type System:**
- [`Schema`](./schema.md) — Primary data type containing reified structure of a Scala data type
- [`Reflect`](./reflect.md) — Foundational data structure containing reified structural information
- [`Binding`](./binding.md) — Operational machinery for constructing and deconstructing values
- [`ReflectTransformer`](./reflect-transformer.md) — Generic mechanism for rewriting a Reflect tree's binding type, powering schema (de)serialization and derivation
- [`Registers`](./registers.md) — Register-based design for zero-allocation construction and deconstruction
- [`Structural Types`](./structural-types.md) — Duck typing with ZIO Blocks schemas

**Dynamic & Runtime Data:**
- [`DynamicValue`](./dynamic-value.md) — Schema-less, dynamically-typed representation of any structured value
- [`DynamicSchema`](./dynamic-schema.md) — Type-erased schema container for serialization and transport
- [`Lazy`](./lazy.md) — Deferred computation with monadic abstraction, memoization, and stack-safe evaluation

**Navigation & Transformation:**
- [`Optics`](./optics.md) — Reflective optics for type-safe, composable access to nested data structures
- [`DynamicOptic`](./dynamic-optic.md) — Runtime path through nested data structures
- [`Path Interpolator`](./path-interpolator.md) — Compile-time string interpolator `p"..."` for constructing `DynamicOptic` instances
- [`Schema Search and Update`](./schema-search.md) — Executing type/schema-pattern searches with `SearchTraversal`, and rewriting schema metadata with `Reflect.Updater`/`Term.Updater`
- [`SchemaExpr`](./schema-expr.md) — Schema-aware expressions for evaluation and query language translation
- [`Patch`](./patch.md) — Type-safe, serializable transformations of data structures
- [`Modifier`](./modifier.md) — Mechanism to attach metadata and configuration to schema elements

**Serialization:**
- [`Codec`](./codec.md) — Base abstraction for encoding and decoding values between formats
- [`Format`](./format.md) — Unified abstraction bundling serialization and deserialization for a specific format
- [`Type Class Derivation`](./type-class-derivation.md) — Automatic generation of type class instances from schemas
- [`Syntax`](./syntax.md) — Extension methods for fluent JSON encoding/decoding and patching

**Formats:**
- [`JSON Codec`](./built-in-codecs/json/) — Complete JSON support: ADT for JSON values, fluent navigation, encoding/decoding, diffs, patches, and JSON Schema 2020-12 validation
- [`Xml`](./built-in-codecs/xml.md) — Type-safe, immutable representation of XML document structures

**Validation & Errors:**
- [`Validation`](./validation.md) — Declarative constraints on primitive values
- [`SchemaError`](./schema-error.md) — Structured error type for schema operations
- [`Allows`](./allows.md) — Compile-time capability token proving a type satisfies a structural grammar

## Overview

The ZIO Blocks Schema module revolves around the `Schema` and `Reflect` types, which capture the complete structural description of Scala data types at runtime. From a single schema definition, you get automatic derivation of codecs (`Codec`, `Formats`), type-safe data access (`Optics`, `DynamicOptic`), structural validation (`Validation`, `SchemaError`), patching (`Patch`, `JsonPatch`), and serialization to multiple formats (`Json`, `Xml`, `JSON Schema`).

The type system is powered by a register-based architecture (`Registers`) that eliminates boxing overhead, and supports both compile-time (`Allows`) and runtime (`DynamicValue`, `DynamicSchema`) type manipulation. The `Deriver` system (`Type Class Derivation`) enables automatic generation of any type class instance from schema metadata.

---

## Lazy

The `Lazy[A]` data type represents a deferred computation that produces a value of type `A`. Unlike Scala's built-in `lazy val`, ZIO Blocks' `Lazy` provides a powerful, monadic abstraction with explicit error handling, memoization, and stack-safe evaluation through trampolining:

```scala
sealed trait Lazy[+A] {
  def force: A
  def isEvaluated: Boolean
  def map[B](f: A => B): Lazy[B]
  def flatMap[B](f: A => Lazy[B]): Lazy[B]
  // ... more operations
}

object Lazy {
  def apply[A](expression: => A): Lazy[A]
  def fail(throwable: Throwable): Lazy[Nothing]
  // ... more constructors
}
```

Once a `Lazy` computation is evaluated, the result is cached and `isEvaluated` becomes `true`. Subsequent calls to `force` return the cached value without re-executing the computation, as long as the computed result is not `null`.

Note: the current implementation uses `null` internally as the sentinel for “not evaluated”. If a `Lazy` computation legitimately returns `null`, it will be recomputed on every `force` call and `isEvaluated` will remain `false`. To benefit from memoization, prefer non-null results (for example, use `Option[A]` instead of returning `null`).
The `force` method uses trampolining (an explicit stack) to evaluate deeply nested `Lazy` computations without risking stack overflow.

## Why Lazy Exists?

During type-class derivation, instances for nested types must be created before they are used. `Lazy` allows the derivation machinery to build a structure of deferred computations, resolving them only when the final instance is forced.

When deriving a type-class instance for a complex type, the derivation machinery needs to:

1. **Traverse the schema tree**: Visit each node (records, variants, sequences, etc.)
2. **Create instances for nested types**: Before creating an instance for a parent type, instances for child types must exist
3. **Handle recursion**: For recursive types, the instance for the recursive reference must be deferred

Here's a simplified view of how `Lazy` enables this:

```scala
// In DerivationBuilder
def transformRecord[A](
  fields: IndexedSeq[Term[F, A, ?]],
  metadata: F[BindingType.Record, A],
  // ...
): Lazy[Reflect.Record[G, A]] = Lazy {
  // Get instances for field types (may trigger evaluation of other Lazy values)
  val fieldInstances = fields.map { field =>
    D.instance(field.value.metadata)  // Returns Lazy[TC[FieldType]]
  }

  // Create the record instance
  val instance = deriver.deriveRecord(fields, /* ... */)

  new Reflect.Record(fields, ..., new BindingInstance(metadata, instance), ...)
}
```

The `BindingInstance` class wraps both a `Binding` and a `Lazy[TC[A]]`:

```scala
final case class BindingInstance[TC[_], T, A](
  binding: Binding[T, A],
  instance: Lazy[TC[A]]
)
```

This allows the derivation to build a complete tree of `Lazy` instances, which are only forced when the final type-class instance is needed.

## Creating Lazy Values

### Basic Construction

Create a `Lazy` value by passing a by-name expression to `Lazy.apply`:

```scala

// The expression is NOT evaluated here
val lazyInt: Lazy[Int] = Lazy {
  println("Computing...")
  42
}

println(lazyInt.isEvaluated)  // false

// Now the expression is evaluated
val result = lazyInt.force    // prints "Computing..."
println(result)               // 42
println(lazyInt.isEvaluated)  // true

// Subsequent calls return the cached value
val result2 = lazyInt.force   // does NOT print "Computing..."
println(result2)              // 42
```

### Creating Failed Lazy Values

Use `Lazy.fail` to create a `Lazy` that will throw an exception when forced:

```scala
val failingLazy: Lazy[Int] = 
  Lazy.fail(new RuntimeException("Something went wrong"))
```

## Transforming Lazy Values

`Lazy` is a monad, supporting `map`, `flatMap`, and other familiar operations.

### map

Transform the value inside a `Lazy` without forcing it:

```scala
val lazyInt: Lazy[Int] = Lazy(42)
val lazyString: Lazy[String] = lazyInt.map(_.toString)

println(lazyString.force)       // "42"
```

### flatMap

Chain `Lazy` computations together:

```scala
val lazyA: Lazy[Int] = Lazy(10)
val lazyB: Lazy[Int] = Lazy(20)

val lazySum: Lazy[Int] = lazyA.flatMap(a => lazyB.map(b => a + b))

println(lazySum.force)  // 30
```

Using for-comprehension syntax:

```scala
val result: Lazy[String] = for {
  x <- Lazy(10)
  y <- Lazy(20)
  z <- Lazy(30)
} yield s"Sum: ${x + y + z}"

println(result.force)  // "Sum: 60"
```

### as

Replace the value with a constant, discarding the original:

```scala
val lazy42: Lazy[Int] = Lazy(42)
val lazyHello: Lazy[String] = lazy42.as("Hello")

println(lazyHello.force)  // "Hello"
```

### unit

Discard the value, keeping only the side effects:

```scala
val lazyWithSideEffect: Lazy[Int] = Lazy {
  println("Side effect!")
  42
}

val lazyUnit: Lazy[Unit] = lazyWithSideEffect.unit
lazyUnit.force  // prints "Side effect!", returns ()
```

### flatten
 
Flatten a nested `Lazy[Lazy[A]]` into `Lazy[A]`:

```scala
val nested: Lazy[Lazy[Int]] = Lazy(Lazy(42))
val flat: Lazy[Int] = nested.flatten

println(flat.force)  // 42
```

### zip

Combine two `Lazy` values into a tuple:

```scala
val lazyA: Lazy[Int] = Lazy(1)
val lazyB: Lazy[String] = Lazy("hello")

val lazyPair: Lazy[(Int, String)] = lazyA.zip(lazyB)

println(lazyPair.force)  // (1, "hello")
```

## Error Handling

### catchAll

Recover from errors by providing an alternative `Lazy`:

```scala
val failing: Lazy[Int] = Lazy(throw new RuntimeException("oops"))
val recovered: Lazy[Int] = failing.catchAll(_ => Lazy(0))

println(recovered.force)  // 0
```

The error handler receives the thrown exception:

```scala
val failing: Lazy[Int] = Lazy(throw new RuntimeException("specific error"))
val handled: Lazy[Int] = failing.catchAll { error =>
  println(s"Caught: ${error.getMessage}")
  Lazy(-1)
}

println(handled.force)  // prints "Caught: specific error", returns -1
```

### ensuring

Run a finalizer regardless of success or failure:

```scala
var resourceClosed = false

val computation: Lazy[Int] = Lazy {
  42
}.ensuring(Lazy {
  resourceClosed = true
})

println(computation.force)  // 42
println(resourceClosed)     // true
```

The finalizer runs even when the main computation fails:

```scala
var finalizerRan = false

val failing: Lazy[Int] = Lazy[Int] {
  throw new RuntimeException("error")
}.ensuring(Lazy {
  finalizerRan = true
})

try {
  failing.force
} catch {
  case _: RuntimeException => ()
}

println(finalizerRan)  // true
```

## Working with Collections

### collectAll

Convert an `IndexedSeq[Lazy[A]]` into a `Lazy[IndexedSeq[A]]`:

```scala
val lazies: IndexedSeq[Lazy[Int]] = IndexedSeq(Lazy(1), Lazy(2), Lazy(3))
val collected: Lazy[IndexedSeq[Int]] = Lazy.collectAll(lazies)

println(collected.force)  // IndexedSeq(1, 2, 3)
```

### foreach

Apply a function that returns `Lazy` to each element of a collection:

```scala
val numbers: IndexedSeq[Int] = IndexedSeq(1, 2, 3)
val doubled: Lazy[IndexedSeq[Int]] = Lazy.foreach(numbers)(n => Lazy(n * 2))

println(doubled.force)  // IndexedSeq(2, 4, 6)
```

## How `force` Works: A Deep Dive

The `force` method is the heart of the `Lazy` data type. It evaluates the deferred computation and returns the result. Understanding how it works is essential for understanding `Lazy`.

### Internal Structure

`Lazy` has three internal components:

1. **`Defer[A](thunk: () => A)`**: Represents a deferred computation. The `thunk` is a function that, when called, produces the value.

2. **`FlatMap[A, B](first: Lazy[A], cont: Cont[A, B])`**: Represents a chained computation where `first` must be evaluated, then its result is passed to the continuation `cont`.

3. **`Cont[A, B](ifSuccess: A => Lazy[B], ifError: Throwable => Lazy[B])`**: A continuation that handles both success and error cases.

Additionally, each `Lazy` instance has mutable fields for memoization:
- `value: Any` - Stores the cached successful result
- `error: Throwable` - Stores the cached exception

### Why Trampolining?

Without trampolining, deeply nested `flatMap` chains would overflow the stack:

```scala
// This would overflow without trampolining
var lazy: Lazy[Int] = Lazy(0)
for (_ <- 1 to 100000) {
  lazy = lazy.flatMap(n => Lazy(n + 1))
}
lazy.force  // Works! Returns 100000
```

The trampolining approach converts recursive calls into an iterative loop with an explicit stack (`List[Cont[Any, Any]]`), consuming constant stack space regardless of nesting depth.

## Comparison with Scala's `lazy val`

| Feature            | `Lazy[A]`                    | Scala `lazy val`             |
|--------------------|------------------------------|------------------------------|
| Monadic operations | Yes (`map`, `flatMap`)       | No                           |
| Error handling     | Yes (`catchAll`, `ensuring`) | No                           |
| Stack safety       | Yes (trampolined `flatMap`)  | N/A (no monadic chaining)    |
| Composable         | Yes                          | Limited                      |

## API Reference

### Constructors

| Method                    | Description                                                         |
|---------------------------|---------------------------------------------------------------------|
| `Lazy(expr: => A)`        | Create a `Lazy` from a by-name expression                          |
| `Lazy.fail(e: Throwable)` | Create a `Lazy` that fails with the given exception                |
| `Lazy.collectAll(values)` | Combine an `IndexedSeq` of `Lazy` into a single `Lazy`             |
| `Lazy.foreach(values)(f)` | Apply a function to each element of an `IndexedSeq`, collecting results |

Note: For other collection types, convert them using `.toIndexedSeq` before calling `Lazy.collectAll` or `Lazy.foreach`.
### Instance Methods

| Method                                       | Description                               |
|----------------------------------------------|-------------------------------------------|
| `force: A`                                          | Evaluate and return the result (memoized) |
| `isEvaluated: Boolean`                              | Check if the value has been computed      |
| `map[B](f: A => B): Lazy[B]`                        | Transform the value                       |
| `flatMap[B](f: A => Lazy[B]): Lazy[B]`              | Chain computations                        |
| `flatten: Lazy[B]`                                  | Flatten nested `Lazy`                     |
| `as[B](b: => B): Lazy[B]`                           | Replace the value with a constant         |
| `unit: Lazy[Unit]`                                  | Discard the value                         |
| `zip[B](that: Lazy[B]): Lazy[(A, B)]`               | Combine with another `Lazy`               |
| `catchAll[B >: A](f: Throwable => Lazy[B]): Lazy[B]` | Handle errors                            |
| `ensuring(finalizer: Lazy[Any]): Lazy[A]`           | Run finalizer on completion               |

### Equality and Hashing

`Lazy` values are compared by forcing both values and comparing the results:

```scala
Lazy(42) == Lazy(42)  // true (forces both)
Lazy(42).hashCode == Lazy(42).hashCode  // true
```

Note: Comparing `Lazy` values will force their evaluation. Also, `hashCode` forces the value and then calls `.hashCode` on it; if a `Lazy` evaluates to `null`, `hashCode` will throw a `NullPointerException`, and repeated equality checks may re-run the thunk because `null` is used internally as the “not yet evaluated” sentinel.

---

## Migration

`Migration[A, B]` is ZIO Blocks Schema's typed API for evolving data from one schema version to another. It wraps a [`DynamicMigration`](#dynamicmigration) core, made of inspectable, composable data, with typed source and target schemas, giving you a builder-based workflow for adding fields, dropping fields, renaming fields, changing types, migrating nested values, transforming collections and maps, and composing migrations.

## Overview

The migration system is split into three layers:

- **`Migration[A, B]`** — typed migration between source type `A` and target type `B`
- **`DynamicMigration`** — untyped runtime representation built from actions that can be inspected, composed and reversed
- **`MigrationBuilder[A, B, Changeset]`** — macro-validated builder that accumulates actions and checks that the target shape is fully handled before `build`

```
Migration[A, B]
  ├── sourceSchema: Schema[A]
  ├── targetSchema: Schema[B]
  └── dynamicMigration: DynamicMigration
         └── actions: Chunk[MigrationAction]
                ├── AddField / DropField / RenameField
                ├── TransformField / ChangeFieldType
                ├── MandateField / OptionalizeField
                ├── RenameCase / TransformCase
                ├── MigrateField
                ├── TransformElements / TransformKeys / TransformValues
                └── Irreversible
```

`Migration` is the user-facing entry point. `DynamicMigration` is the data representation you can inspect, compose, reverse, or apply dynamically. It has no `Schema` yet, so it cannot be encoded with a codec the way a `DynamicPatch` can.

## Creating a Migration

The normal entry point is `Migration.newBuilder[A, B]`:

```scala

case class PersonV1(name: String)
case class PersonV2(name: String, age: Int)

object PersonV1 {
  implicit val schema: Schema[PersonV1] = Schema.derived
}

object PersonV2 {
  implicit val schema: Schema[PersonV2] = Schema.derived
}

val migration: Migration[PersonV1, PersonV2] =
  Migration
    .newBuilder[PersonV1, PersonV2]
    .addField(_.age, SchemaExpr.literal(0))
    .build
```

Applying the migration converts the source value to [`DynamicValue`](./dynamic-value.md), applies the dynamic actions, then converts the result back into the target type:

```scala

case class PersonV1(name: String)
case class PersonV2(name: String, age: Int)

object PersonV1 {
  implicit val schema: Schema[PersonV1] = Schema.derived
}

object PersonV2 {
  implicit val schema: Schema[PersonV2] = Schema.derived
}

val migration = Migration
  .newBuilder[PersonV1, PersonV2]
  .addField(_.age, SchemaExpr.literal(0))
  .build

val result = migration(PersonV1("Alice"))
// Right(PersonV2("Alice", 0))
```

:::caution Scala 2
On Scala 2.13 the builder methods and `build` do not compile outside ZIO Blocks itself, a known bug in 0.0.56. Build the actions as a `DynamicMigration` and wrap it with `Migration.fromDynamic`, which works on both Scala versions:

```scala

case class PersonV1(name: String)
case class PersonV2(name: String, age: Int)

object PersonV1 {
  implicit val schema: Schema[PersonV1] = Schema.derived
}

object PersonV2 {
  implicit val schema: Schema[PersonV2] = Schema.derived
}

val migration: Migration[PersonV1, PersonV2] = Migration.fromDynamic[PersonV1, PersonV2](
  DynamicMigration(MigrationAction.AddField(DynamicOptic.root.field("age"), SchemaExpr.literal(0).toDynamic))
)
```

`fromDynamic` does not check at compile time that the actions produce a `PersonV2`; a mismatch shows up as a `Left` when the migration runs.
:::

## Core Operations

### `Migration#apply`

Transforms a typed value:

```scala
def apply(value: A): Either[SchemaError, B]
```

### `Migration#reverse`

Returns the **structural reverse** of the migration:

```scala
def reverse: Migration[B, A]
```

This is best-effort at runtime. Actions that lose information (for example `TransformField`, `ChangeFieldType`, `OptionalizeField`, or collection-wide transforms) reverse to `Irreversible`, which causes reverse execution to fail with a descriptive error instead of silently fabricating data.

### Composition

Migrations compose with `++` or `andThen`:

```scala

case class PersonV1(name: String)
case class PersonV2(name: String, age: Int)
case class PersonV3(name: String, age: Int, city: String)

object PersonV1 { implicit val schema: Schema[PersonV1] = Schema.derived }
object PersonV2 { implicit val schema: Schema[PersonV2] = Schema.derived }
object PersonV3 { implicit val schema: Schema[PersonV3] = Schema.derived }

val migration = Migration
  .newBuilder[PersonV1, PersonV2]
  .addField(_.age, SchemaExpr.literal(0))
  .build

val addCity = Migration
  .newBuilder[PersonV2, PersonV3]
  .addField(_.city, SchemaExpr.literal(""))
  .build

val combined: Migration[PersonV1, PersonV3] = migration ++ addCity
```

## Builder Operations

`MigrationBuilder` uses selector expressions made of field accesses, such as `_.field` and `_.nested.field`. The selector syntax `each`, `eachKey`, `eachValue`, `at`, `atKey`, `when` and `wrapped` exists in the API but does not compile in any builder method, a known bug in 0.0.56; migrate nested collections with `migrateField` or with `MigrationAction`s. The builder's third type parameter, `Changeset`, tracks which operations have already been applied so `build` can validate completeness.

### Record operations

- `addField(_.target, default)`
- `dropField(_.source, defaultForReverse)`
- `renameField(_.from, _.to)`
- `transformField(_.from, _.to, expr)`
- `mandateField(_.optionalSource, _.target, default)`
- `optionalizeField(_.source, _.optionalTarget)`
- `changeFieldType(_.source, expr)`

```scala

case class UserV1(firstName: String, nickname: Option[String], age: Int)
case class UserV2(fullName: String, nickname: String, age: String)

object UserV1 {
  implicit val schema: Schema[UserV1] = Schema.derived
}

object UserV2 {
  implicit val schema: Schema[UserV2] = Schema.derived
}

val userMigration = Migration
  .newBuilder[UserV1, UserV2]
  .renameField(_.firstName, _.fullName)
  .mandateField(_.nickname, _.nickname, SchemaExpr.literal("anonymous"))
  .changeFieldType(_.age, SchemaExpr.literal("30"))
  .build
```

:::warning
`mandateField` and `optionalizeField` fail at run time in 0.0.56 (a known bug): `mandateField` fails for every `Some`, and `optionalizeField` produces a value the target schema cannot decode. The example above compiles, but applying `userMigration` to a `UserV1` with a `Some` nickname returns a `Left`.
:::

`transformField` and `changeFieldType` evaluate their `SchemaExpr` against the currently focused field value, not the entire root record. In practice, that means the expression should be written in terms of the value being replaced.

### Nested migration with `migrateField`

When a nested value has its own migration, use `migrateField`:

```scala

case class AddressV1(street: String, city: String)
case class AddressV2(street: String, city: String, zip: String)
case class PersonWithAddressV1(name: String, address: AddressV1)
case class PersonWithAddressV2(name: String, address: AddressV2)

object AddressV1 { implicit val schema: Schema[AddressV1] = Schema.derived }
object AddressV2 { implicit val schema: Schema[AddressV2] = Schema.derived }
object PersonWithAddressV1 { implicit val schema: Schema[PersonWithAddressV1] = Schema.derived }
object PersonWithAddressV2 { implicit val schema: Schema[PersonWithAddressV2] = Schema.derived }

val addressMigration = Migration
  .newBuilder[AddressV1, AddressV2]
  .addField(_.zip, SchemaExpr.literal("00000"))
  .build

val personMigration = Migration
  .newBuilder[PersonWithAddressV1, PersonWithAddressV2]
  .migrateField(_.address, addressMigration)
  .build
```

There is no `transformNested` builder method in the current API. Nested structural changes are expressed by building a migration for the nested type and applying it with `migrateField`.

### Enum and collection operations

- `renameCase(from, to)`
- `transformCase(caseName)(...)`
- `transformElements(_.items, expr)`
- `transformKeys(_.data, expr)`
- `transformValues(_.data, expr)`

Collection and map transforms are map-like operations. The `SchemaExpr` is evaluated once per matched value:

- `transformElements` evaluates against each collection element
- `transformKeys` evaluates against each map key
- `transformValues` evaluates against each map value

For example, `transformElements(_.scores, expr)` runs `expr` separately for every element in `scores`, and `transformValues(_.metadata, expr)` runs it separately for every map value.

## `build`

```scala
def build(using ev: MigrationComplete[A, B, Changeset]): Migration[A, B]
```

`build` runs macro validation. It checks that every field needed to transform `A` into `B` is either:

- auto-mapped by identical source/target structure, or
- explicitly handled by one of the builder operations

If the migration is incomplete, the code fails to compile.

This validated `build` step is the public completion path for `MigrationBuilder`. If you need unvalidated or dynamically assembled migrations, work at the `DynamicMigration` layer and wrap it with `Migration.fromDynamic`.

## `DynamicMigration`

`DynamicMigration` is the untyped runtime representation:

```scala
final case class DynamicMigration(actions: Chunk[MigrationAction]) {
  def apply(value: DynamicValue): Either[SchemaError, DynamicValue]
  def ++(that: DynamicMigration): DynamicMigration
  def andThen(that: DynamicMigration): DynamicMigration
  def reverse: DynamicMigration
  def isEmpty: Boolean
  def size: Int
}
```

Use it when you need to inspect or apply migrations without keeping the original Scala types around, or to build a migration on Scala 2, where the builder does not compile (see [Creating a Migration](#creating-a-migration)).

In 0.0.56 there is no way to store or send a `DynamicMigration`: no codec can write it. Keep migrations in code instead, for example as a map from a stored data version to the `Migration` that upgrades it, and store only the version number next to the data.

## `MigrationAction`

`MigrationAction` is the ADT used by `DynamicMigration`. Its cases are data that can be inspected and compared, but no `Schema` is provided for them, and a literal inside an action (`DynamicSchemaExpr.Literal`, which every `SchemaExpr.literal` becomes) holds the `Schema` of its value, so an action can't be encoded either.

| Action | Purpose | Reverse behavior |
|---|---|---|
| `AddField` | add a field with a default expression | `DropField` |
| `DropField` | remove a field | `AddField` using stored reverse default |
| `RenameField` | rename a record field | structural inverse rename |
| `TransformField` | replace a field by evaluating an expression against the current field value | `Irreversible` |
| `MandateField` | convert `Option[A]` to `A` with default for `None`; fails for every `Some` (a known bug in 0.0.56) | `OptionalizeField`, which has the same bug |
| `OptionalizeField` | wrap a value in `Some`; the result does not decode (a known bug in 0.0.56) | `Irreversible` |
| `ChangeFieldType` | replace a field with a converted value computed from the current field value | `Irreversible` |
| `RenameCase` | rename enum case | inverse rename |
| `TransformCase` | run actions inside a specific case | reverse inner actions in reverse order |
| `MigrateField` | apply a nested `DynamicMigration` to a field | reverse nested migration |
| `TransformElements` | replace each collection element by evaluating the expression on that element | `Irreversible` |
| `TransformKeys` | replace each map key by evaluating the expression on that key | `Irreversible` |
| `TransformValues` | replace each map value by evaluating the expression on that value | `Irreversible` |
| `Irreversible` | explicit non-invertible sentinel | itself |

## Errors

Migration execution returns [`SchemaError`](./schema-error.md), with migration-specific kinds including:

- path not found
- type mismatch
- missing default
- transform failure
- field/case not found
- invalid value
- mandate failure

Errors carry a [`DynamicOptic`](./dynamic-optic.md) path so failures can be traced to the precise field or nested location that failed.

## Relationship to Other Schema-Evolution APIs

`Into` and `As` derive structural conversions automatically. `Migration` is lower-level and more explicit:

- use [`Into`](./schema-evolution/into.md) for one-way derived schema evolution
- use [`As`](./schema-evolution/as.md) for bidirectional derived evolution
- use `Migration` when you need explicit, inspectable, composable transformation steps

:::tip
For a step-by-step walkthrough rather than a reference, follow the [Evolve Your Data](../../guides/evolve-your-data.md) tutorial, which builds a `Migration` that renames a field and shows the compile-time completeness check.
:::

---

## Modifier

`Modifier` is a sealed trait that provides a mechanism to attach metadata and configuration to schema elements. Modifiers serve as annotations for record fields, variant cases, and reflect values, enabling format-specific customization without polluting domain types.

Modifiers are designed to be **pure data** values that can be serialized, making them ideal for runtime introspection and cross-process schema exchange. When deriving schemas, modifiers are collected and attached to the corresponding fields or types, allowing codecs to read and interpret them accordingly. They are extended with `StaticAnnotation` to also support annotation syntax:

```scala
sealed trait Modifier extends StaticAnnotation
object Modifier {
  sealed trait Term    extends Modifier
  // ... term modifiers (transient, encodeTransient, rename, alias, id, config) ...
  sealed trait Reflect extends Modifier
  // ... reflect modifiers (config, discriminator, noExtraFields, fieldNaming, caseNaming) ...
}
```

Modifiers can be applied in two ways:

1. **Programmatic API**: Using the `Schema#modifier` and `Schema#modifiers` methods to attach modifiers to the entire schema or, for field-level modifiers, attach them to specific fields using optics when deriving codecs. This approach keeps your domain types clean and allows you to separate schema configuration from your data model:

```scala

// Clean domain type - zero dependencies
case class User(
  id: String,
  name: String,
  cache: Map[String, String] = Map.empty
)

// Modifiers applied separately to schema and codecs
object User extends CompanionOptics[User] {
  implicit val schema: Schema[User] = Schema
    .derived[User]
    .modifier(Modifier.config("db.table-name", "users"))

   implicit val jsonCodec: JsonCodec[User] =
     schema
       .deriving(JsonCodecDeriver)
       .modifier(User.name, Modifier.rename("username"))
       .modifier(User.cache, Modifier.transient())
       .derive

  lazy val id   : Lens[User, String]              = $(_.id)
  lazy val name : Lens[User, String]              = $(_.name)
  lazy val cache: Lens[User, Map[String, String]] = $(_.cache)
}
```

In the above example, we derive a JSON codec for `User` and apply the `rename` and `transient` modifiers to the `name` and `cache` fields respectively, while keeping the domain type free of any schema-related annotations. Now when encoding a `User` to JSON, the `name` field will be serialized as `username`, and the `cache` field will be omitted. During decoding, the codec will look for `username` in the input JSON and populate the `name` field accordingly:

```scala
val user = User(
  id = "123",
  name = "Alice",
  cache = Map("lastLogin" -> "2024-06-01T12:00:00Z")
)
val json: String = User.jsonCodec.encodeToString(user)
println(json)
// Prints: {"id":"123","username":"Alice"}
val decodedUser: Either[SchemaError, User] = User.jsonCodec.decode(json)
println(decodedUser)
// Prints: Right(User(123,Alice,Map()))
```

When deriving codecs, you can access these modifiers programmatically, allowing you to build custom logic based on the presence of certain modifiers. For example, your SQL codec could check for the presence of `db.table-name` in the schema modifiers to determine which table to read from or write to.

2. **Annotation Syntax**: Using the `@` syntax to annotate fields and cases directly in your case classes and sealed traits. These annotations are processed during schema derivation to attach the corresponding modifiers to the schema elements. At runtime, you can access these modifiers through the `Reflect` structure of the schema. The `User` below carries the same modifiers as annotations:

```scala

@Modifier.config("db.table-name", "users")
case class User(
  id: String,
  @Modifier.rename("username") name: String,
  @Modifier.transient() cache: Map[String, String] = Map.empty
)

object User extends CompanionOptics[User] {
  implicit val schema: Schema[User] =
    Schema.derived[User]

   implicit val jsonCodec: JsonCodec[User] =
     schema
       .derive(JsonCodecDeriver)
}
```

In this example, we apply the same modifiers as in the programmatic example, but using annotation syntax directly on the case class fields. Let's try encoding and decoding a `User` instance:

```scala
val user = User(
  id = "123",
  name = "Alice",
  cache = Map("lastLogin" -> "2024-06-01T12:00:00Z")
)

val json: String = User.jsonCodec.encodeToString(user)
println(json)
// Prints: {"id":"123","username":"Alice"}
val decodedUser: Either[SchemaError, User] = User.jsonCodec.decode(json)
println(decodedUser)
// Prints: Right(User(123,Alice,Map()))
```

## Modifier Hierarchy

Modifiers are organized into two main categories. **Term modifiers** annotate record fields or variant cases (the data structure elements), and **Reflect modifiers** annotate schemas/reflect values themselves (the metadata about types). The tree below lists the modifiers in each category:

```
Modifier
 ├── Modifier.Term     (annotates record fields and variant cases)
 │    ├── transient()        : exclude from serialization
 │    ├── encodeTransient()  : exclude from encoding only
 │    ├── rename(name)       : change serialized name
 │    ├── alias(name)        : add alternative name
 │    ├── id()               : mark a primary-key field
 │    └── config(key, val)   : attach key-value metadata
 └── Modifier.Reflect  (annotates reflect values / types)
      ├── config(key, val)       : attach key-value metadata
      ├── discriminator(name)    : JSON discriminator field of a variant
      ├── noExtraFields()        : reject unknown JSON fields of a record
      ├── fieldNaming(strategy)  : JSON naming strategy for record fields
      └── caseNaming(strategy)   : JSON naming strategy for variant cases
```

`config` is the only modifier that extends both `Term` and `Reflect`, so it applies to both fields and types.

## Term Modifiers

Term modifiers annotate record fields and variant cases. They are used to control how individual fields or cases are serialized and deserialized, as well as to attach additional metadata that can be interpreted by codecs or other tools.

### transient

The `transient` modifier marks a field as transient, meaning it will be excluded from serialization. This is useful for computed fields, caches, or sensitive data that shouldn't be persisted.

On decode, a transient field's value in the input is skipped, as for any unknown key, so it is rejected when `Modifier.noExtraFields` is on. The field takes its default value. Because of that, a transient field needs a default value unless it is an `Option` or a collection.

### rename

The `rename` modifier changes the serialized name of a field or variant case. This is useful when the field name in your Scala code differs from the expected name in the serialized format. `Person` renames both of its fields:

```scala

case class Person(
  @Modifier.rename("user_name") name: String,
  @Modifier.rename("user_age") age: Int
)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}
```

You can also use `rename` on variant cases to customize the discriminator value:

```scala

sealed trait PaymentMethod

object PaymentMethod {
  @Modifier.rename("credit_card")
  case class CreditCard(number: String, cvv: String) extends PaymentMethod

  @Modifier.rename("bank_transfer")
  case class BankTransfer(iban: String) extends PaymentMethod

  implicit val schema: Schema[PaymentMethod] = Schema.derived
}
```

### alias

The `alias` modifier provides an alternative name for a term during decoding. This is useful for supporting multiple names during schema evolution or data migration. `MyClass` renames its field to `NewName` and still accepts two older names:

```scala

case class MyClass(
  @Modifier.rename("NewName")
  @Modifier.alias("OldName")
  @Modifier.alias("LegacyName")
  value: String
)

object MyClass {
  implicit val schema: Schema[MyClass] = Schema.derived
}
```

With this configuration, encoding always uses the `rename` value, `"NewName"`, and decoding accepts any of `"NewName"`, `"OldName"`, or `"LegacyName"`.

This pattern is particularly useful when migrating data formats without breaking compatibility with existing data.

### id

The `id` modifier marks a record field as its primary key. It is interpreted by `zio-blocks-sql` when deriving a `Repo[E, ID]`, and takes precedence over name- and type-based ID discovery. In `User`, `@Modifier.id()` makes `externalKey` the primary key:

```scala

case class User(
  email: String,
  @Modifier.id() externalKey: Long
)
```

Mark exactly one field with `@Modifier.id()` for a derived repository.

### config

The `config` modifier attaches arbitrary key-value metadata to a term (record fields or variant cases) or a type itself. The convention for keys is `<format>.<property>`, allowing format-specific configuration. `Event` gives each of its fields a `protobuf.field-id` entry:

```scala

case class Event(
  @Modifier.config("protobuf.field-id", "1") id: Long,
  @Modifier.config("protobuf.field-id", "2") name: String
)

object Event {
  implicit val schema: Schema[Event] = Schema.derived
}
```

The `config` modifier extends both `Term` and `Reflect`, making it usable on both fields and types. The Reflect Modifiers section below covers using `config` on types.

## Reflect Modifiers

Reflect modifiers annotate reflect values (types themselves). Alongside `config`, JSON derivation also understands dedicated reflect modifiers for discriminator selection, field naming, case naming, and rejecting extra fields.

### config

You can attach configuration to the type itself using the `Schema#modifier` method:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
    .modifier(Modifier.config("db.table-name", "person_table"))
    .modifier(Modifier.config("schema.version", "v2"))
}
```

### JSON Reflect Modifiers

These reflect modifiers are consumed by `JsonCodecDeriver` and let you keep JSON-specific settings next to the annotated type instead of passing them programmatically at every call site. `Event` sets its discriminator field and case naming, and `UserProfile` rejects extra fields and applies snake_case to its field names:

```scala

@Modifier.discriminator("type")
@Modifier.caseNaming("snake_case")
sealed trait Event

@Modifier.noExtraFields()
@Modifier.fieldNaming("snake_case")
case class UserProfile(
  @Modifier.rename("userId") id: String,
  @Modifier.encodeTransient() checksum: Int = 0,
  displayName: String
)
```

- `Modifier.discriminator(name)` overrides the JSON discriminator field for sealed hierarchies.
- `Modifier.noExtraFields()` rejects unknown object fields during decoding.
- `Modifier.fieldNaming(strategy)` applies a naming strategy to direct record fields.
- `Modifier.caseNaming(strategy)` applies a naming strategy to variant case names.

For field-level JSON behavior, `Modifier.encodeTransient()` skips a field during encoding but still accepts it during decoding; like `transient`, it requires a default value when the field is neither optional nor a collection.

Or add multiple modifiers at once:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
    .modifiers(
      Seq(
        Modifier.config("db.table-name", "person_table"),
        Modifier.config("schema.version", "v2")
      )
    )
}
```

Or annotate the case class directly:

```scala

@Modifier.config("db.table-name", "person_table")
@Modifier.config("schema.version", "v2")
case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}
```

## Programmatic Modifier Access

You can access modifiers programmatically through the `Reflect` structure:

```scala

case class Person(
  @Modifier.rename("full_name") name: String,
  @Modifier.transient cache: String = ""
)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Access field modifiers through the reflect
val reflect = Schema[Person].reflect
reflect match {
  case record: Reflect.Record[_, _] =>
    record.fields.foreach { field =>
      println(s"Field: ${field.name}")
      println(s"Modifiers: ${field.modifiers}")
    }
  case _ => ()
}
```

## Built-in Schema Support

All modifier types have built-in `Schema` instances, enabling them to be serialized and deserialized:

```scala

// Schema instances for individual modifiers
Schema[Modifier.transient]
Schema[Modifier.encodeTransient]
Schema[Modifier.rename]
Schema[Modifier.id]
Schema[Modifier.alias]
Schema[Modifier.discriminator]
Schema[Modifier.noExtraFields]
Schema[Modifier.fieldNaming]
Schema[Modifier.caseNaming]
Schema[Modifier.config]

// Schema instances for modifier traits
Schema[Modifier.Term]
Schema[Modifier.Reflect]
Schema[Modifier]
```

This means you can serialize modifiers as part of your schema metadata, allowing you to persist and exchange schema information with full modifier details.

[//]: # (## Format Support)
[//]: # ()
[//]: # (Different serialization formats interpret modifiers according to their semantics.)
[//]: # ()
[//]: # (TODO: Add a table comparing how each modifier is supported across formats like JSON, BSON, Avro, Protobuf, etc. For example:)
[//]: # (| Modifier    | JSON                 | BSON                 | Avro              | Protobuf          |)
[//]: # (|-------------|----------------------|----------------------|-------------------|-------------------|)
[//]: # (| `transient` | Field omitted        | Field omitted        | Field omitted     | Field omitted     |)
[//]: # (| `rename`    | Custom field name    | Custom field name    | Custom field name | Custom field name |)
[//]: # (| `alias`     | Accepts alternatives | Accepts alternatives | -                 | -                 |)
[//]: # (| `config`    | Format-specific      | Format-specific      | Format-specific   | Format-specific   |)

## Best Practices

1. **Use `rename` for external APIs**: When integrating with external systems that use different naming conventions (snake_case vs camelCase), use `rename` to match the expected format.
2. **Use `alias` for migrations**: When evolving your data model, add `alias` modifiers to support reading old data while writing with new names.
3. **Use `transient` sparingly**: Only mark fields as transient when they are truly derived or temporary. Transient fields need default values unless they are `Option`s or collections.
4. **Use namespaced keys for `config`**: Follow the `<format>.<property>` convention to avoid conflicts between different formats or tools.

## See Also

- [How to Customize JSON Field Names, ADT Encoding and Optional Fields](../../guides/customize-json-encoding.md): a task-oriented guide that applies `rename`, `alias` and `transient` to a real model.

---

## Optics

Optics are a fundamental feature of ZIO Blocks that enable type-safe, composable access and modification of nested data structures. What sets ZIO Blocks apart is its implementation of **reflective optics** — a novel construct that combines the operational capabilities of traditional optics with embedded structural metadata, enabling both data manipulation AND introspection.

:::tip
For a practical walkthrough of building query DSLs with optics, see the [Writing a Query DSL with Reified Optics](../../guides/query-dsl-reified-optics.md) guide. If you are new to optics, start with the [Navigate and Update Nested Data with Optics](../../guides/nested-data-with-optics.md) tutorial.
:::

## What Are Optics?

Optics are abstractions that allow you to focus on a specific part of a data structure. They provide a way to **view**, **update**, and **traverse** nested fields in immutable data types without boilerplate code:

Every optic has two type parameters:

```
Optic[S, A]
      │  │
      │  └── Focus: The "little thing" being accessed
      └───── Source: The "big thing" containing the focus
```

The `S` is the source type from which data is accessed or modified. The `A` is the focus type or the target type of the optic.

The terminology comes from physical optics — like using a magnifying glass to focus on a small part of something larger.

## Reflective Optics

[//]: # (Optics are **reified fields, cases, and for-loops**.)

Traditional optics are defined purely by functions — they give you **capabilities** (get/set) but no **knowledge** about the structure being accessed. ZIO Blocks introduces **reflective optics** which embed the structure of both the source and focus as first-class reified schemas:

```
sealed trait Optic[S, A] {
  def source: Reflect.Bound[S]
  def focus: Reflect.Bound[A]
  
  // Other common operations
}
```

The `source` and `focus` methods return `Reflect.Bound` instances that describe the schema of the source and focus types, respectively. This means that every optic carries with it detailed information about the data shapes it operates on.

This enables capabilities that ordinary optics have never had before:

- **Amazing Error Messages** — Know exactly where and why an optic operation failed, because you know which part of the structure was being accessed during the failure
- **DSL Integration** — Optics as first-class values enables us to write query DSLs that can introspect the data model being queried. So we can integrate that DSL with underlying storage systems (SQL, NoSQL, etc.) without losing type safety.

The `OpticCheck` data type is an error reporting mechanism for reflective optics. It captures detailed diagnostic information when optic operations fail, solving a long-standing pain point in traditional optics libraries where failures silently return `None` without explanation. It says where and why the operation failed: every problem carries the full optic path and the path up to the failing step, and a prism step that found another case also names the expected and actual cases and carries the value it found. [Error Reporting with OpticCheck](#error-reporting-with-opticcheck) lists the problems it can report.

## The Optic Type Hierarchy

ZIO Blocks provides four primary optic types, each designed for a specific data shape:

1. **`Lens[S, A]`** — Focuses on a single field within a record (case class)
2. **`Prism[S, A <: S]`** — Focuses on a specific case within a sum type (sealed trait/enum)
3. **`Optional[S, A]`** — It behaves like a combination of `Lens` and `Prism`; focuses on a value that may or may not exist. Like a `Lens`, it can focus on a part of a larger structure and (if present) get and set it. Like a `Prism`, the focus may or may not exist.
4. **`Traversal[S, A]`** — Focuses on zero or more elements within a collection (List, Vector, Set, Map, etc.)

```text
                              ┌─────────────────┐ 
                              │   Optic[S, A]   │
                              │   (abstract)    │
                              └────────┬────────┘
                                       │
         ┌───────────────────┬───────────────────┬───────────────────┐
         │                   │                   │                   │         
         ▼                   ▼                   ▼                   ▼         
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│   Lens[S, A]    │ │  Prism[S, A]    │ │ Optional[S, A]  │ │Traversal[S, A]  │
│                 │ │   (A <: S)      │ │                 │ │                 │
│ Fields in       │ │ Cases in        │ │ Combinations of │ │ Elements in     │
│ Records         │ │ Enums/Variants  │ │ Lens + Prism    │ │ Collections     │
└─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘
```

## Lens

A **Lens** focuses on a single field within a record (case class). It always succeeds because records always have all their fields.

Lens has two primary operations: 

| Method    | Signature     | Description                             |
|-----------|---------------|-----------------------------------------|
| `get`     | `S => A`      | Extract the field value                 |
| `replace` | `(S, A) => S` | Create new structure with updated field |

```scala
sealed trait Lens[S, A] extends Optic[S, A] {
  def get(s: S): A
  def replace(s: S, a: A): S
}
```

Once you have a lens, you can use it to extract or replace the focused field immutably. There are two main approaches to creating lenses: manual construction and automatic macro-based derivation. Let's explore each.

### Manual Lens Construction

To create a lens manually for a field, you can use the following constructor:

```scala
object Lens {
  def apply[S, A](
    source   : Reflect.Record.Bound[S],
    focusTerm: Term.Bound[S, A]
  ): Lens[S, A] = ???
}
```

It takes a `Reflect.Record.Bound[S]` representing the schema of the source type `S`, and a `Term.Bound[S, A]` representing the specific field within `S` that the lens will focus on, and finally returns a lens of type from `S` to `A`.

Assume you have an `Address` case class like this with the schema derived:

```scala
case class Address(street: String, city: String, zipCode: String)
object Address {
  implicit val schema: Schema[Address] = Schema.derived[Address]
}
```

You can create a lens to focus on the `street` field like this:

```scala
object Address {
  implicit val schema: Schema[Address] = Schema.derived[Address]
  
  val street: Lens[Address, String] =
    Lens[Address, String](
      Schema[Address].reflect.asRecord.get,
      Schema[String].reflect.asTerm("street")
    )
}
```

Now you can use the lens to get or replace the `street` field:

```scala
val address = Address("123 Main St", "Springfield", "12345")
val streetName: String = Address.street.get(address) // "123 Main St"
val updatedAddress: Address = Address.street.replace(address, "456 Elm St")
// Address("456 Elm St", "Springfield", "12345")
```

### Automatic Macro-Based Lens Derivation

While manual lens construction gives you fine-grained control, ZIO Blocks provides macro-based derivation as the **preferred approach** for creating lenses.

The `optic` macro inside the `CompanionOptics` trait creates a lens using intuitive selector syntax that mirrors standard Scala field access. In the rest of the schema docs you may also see the symbolic `$(_.field)` form; it is the same optic-construction API presented with symbolic syntax instead of the named `optic(_.field)` helper.

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived
  
  val name: Lens[Person, String] = optic(_.name)
  val age : Lens[Person, Int   ] = optic(_.age)
}
```

The macro inspects the selector expression `_.name`, validates that it corresponds to a valid field path, and generates the appropriate lens. This approach is type-safe—the compiler verifies that the field exists and has the correct type.

For nested structures, the `optic` macro can create composed lenses by chaining field accesses:

```scala
case class Address(street: String, city: String)
case class Person(name: String, address: Address)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived[Person]
  
  // Lens directly to the nested street field
  val street: Lens[Person, String] = optic(_.address.street)
}
```

Now you can use the `street` lens to access the nested `street` field directly and update it:

```scala
val person = Person("John", Address("123 Main St", "Springfield"))
val street = Person.street.get(person)
val updatedPerson = Person.street.replace(person, "456 Elm St")
// => Person("John", Address("456 Elm St", "Springfield"))
```

## Prism

A **`Prism[S, A <: S]`** focuses on a specific case within a sum type (sealed trait/enum). It may fail if the value is a different case.

The key operations of a prism are summarized in the table below:

| Method          | Signature                         | Description                                                    |
|-----------------|-----------------------------------|----------------------------------------------------------------|
| `getOption`     | `(S)    => Option[A]`             | Extracts if value matches subtype `A`, otherwise `None`        |
| `getOrFail`     | `(S)    => Either[OpticCheck, A]` | Extracts with detailed error info on type mismatch             |
| `reverseGet`    | `(A)    => S`                     | Upcasts subtype `A` back to supertype `S`                      |
| `replace`       | `(S, A) => S`                     | Replaces if value matches subtype, otherwise returns unchanged |
| `replaceOption` | `(S, A) => Option[S]`             | Replaces if value matches subtype, otherwise `None`            |
| `replaceOrFail` | `(S, A) => Either[OpticCheck, S]` | Replaces with detailed error info on type mismatch             |

Before discussing these operations, let's explore how we can create prisms. Similar to lenses, we have two approaches: manual construction and automatic macro-based derivation.

### Manual Prism Construction

To create a prism for a case, you can use the following constructor:

```scala
object Prism {
  def apply[S, A <: S](
    source   : Reflect.Variant.Bound[S],
    focusTerm: Term.Bound[S, A]
  ): Prism[S, A] = ???
}
```

It takes a `Reflect.Variant.Bound[S]` representing the schema of the sum type `S`, and a `Term.Bound[S, A]` representing the specific case within `S` that the prism will focus on. The result is a prism from `S` to `A`.

Assume you have a `Notification` sealed trait representing different notification types:

```scala

sealed trait Notification

object Notification {
  case class Email(to: String, subject: String, body: String)                    extends Notification
  case class SMS(phoneNumber: String, message: String)                           extends Notification
  case class Push(deviceId: String, title: String, payload: Map[String, String]) extends Notification

  implicit val schema: Schema[Notification] = Schema.derived
}
```

First, we need to define schemas for each case class and then write a prism for each case (here we define a prism for the `Email` case):

```scala

sealed trait Notification

object Notification {
  case class Email(to: String, subject: String, body: String) extends Notification
  object Email {
    implicit val schema: Schema[Email] = Schema.derived
  }
  
  case class SMS(phoneNumber: String, message: String) extends Notification
  object SMS {
    implicit val schema: Schema[SMS] = Schema.derived
  }
  
  case class Push(deviceId: String, title: String, payload: Map[String, String]) extends Notification
  object Push {
    implicit val schema: Schema[Push] = Schema.derived
  }

  implicit val schema: Schema[Notification] = Schema.derived

  val email: Prism[Notification, Email] =
    Prism[Notification, Email](
      Schema[Notification].reflect.asVariant.get,
      Schema[Email].reflect.asTerm("Email")
    )
  
  val sms: Prism[Notification, SMS] =
    Prism[Notification, SMS](
      Schema[Notification].reflect.asVariant.get,
      Schema[SMS].reflect.asTerm("SMS")
    )
  
  val push: Prism[Notification, Push] =
    Prism[Notification, Push](
      Schema[Notification].reflect.asVariant.get,
      Schema[Push].reflect.asTerm("Push")
    )
}
```

### Automatic Macro-Based Prism Derivation

The `optic` macro inside the `CompanionOptics` trait creates a prism using intuitive selector syntax with the `when[CaseType]` method:

```scala

sealed trait Notification

object Notification extends CompanionOptics[Notification] {
  case class Email(to: String, subject: String, body: String)                    extends Notification
  case class SMS(phoneNumber: String, message: String)                           extends Notification
  case class Push(deviceId: String, title: String, payload: Map[String, String]) extends Notification

  implicit val schema: Schema[Notification] = Schema.derived

  // Macro-derived prisms using when[CaseType] syntax
  val email: Prism[Notification, Email] = optic(_.when[Email])
  val sms  : Prism[Notification, SMS]   = optic(_.when[SMS])
  val push : Prism[Notification, Push]  = optic(_.when[Push])
}
```

The macro inspects the selector expression `_.when[Email]`, validates that `Email` is a valid case of the sum type `Notification`, and generates the appropriate prism. This approach is type-safe—the compiler verifies that the case type exists and is a subtype of the sum type.

For nested structures, the `optic` macro can compose prisms with lenses by chaining case selection with field access:

```scala
sealed trait Response
object Response extends CompanionOptics[Response] {
  case class Success(data: Data)    extends Response
  case class Failure(error: String) extends Response
  
  case class Data(id: Int, value: String)
  
  implicit val schema: Schema[Response] = Schema.derived
  
  // Prism to the Success case
  val success: Prism[Response, Success] = optic(_.when[Success])
  
  // Composed optic: Prism + Lens = Optional
  // Focus on the 'value' field inside the Success case
  val successValue: Optional[Response, String] = optic(_.when[Success].data.value)
}
```

When you compose a prism with a lens using the `optic` macro, the result is an `Optional`—an optic that may fail to focus (because the sum type might be a different case) but if it succeeds, always finds the field:

```scala
val response = Response.Success(Response.Data(1, "hello"))

Response.successValue.getOption(response)
// => Some("hello")

Response.successValue.getOption(Response.Failure("error"))
// => None (response is Failure, not Success)
```

### Operations

Now let's explore prism operations. Assume we have some sample notifications:

```scala
// Sample notifications
val emailNotif: Notification = Notification.Email("user@example.com", "Hello", "Welcome!")
// emailNotif: Notification = Email(
//   to = "user@example.com",
//   subject = "Hello",
//   body = "Welcome!"
// )
val smsNotif: Notification   = Notification.SMS("+1234567890", "Your code is 1234")
// smsNotif: Notification = SMS(
//   phoneNumber = "+1234567890",
//   message = "Your code is 1234"
// )
val pushNotif: Notification  = Notification.Push("device-abc", "Alert", Map("action" -> "open"))
// pushNotif: Notification = Push(
//   deviceId = "device-abc",
//   title = "Alert",
//   payload = Map("action" -> "open")
// )
```

1. **`Prism#getOption`** — Extract the case if it matches, otherwise None

```scala
// getOption: Extract the case if it matches, otherwise None
Notification.email.getOption(emailNotif)
// => Some(Email(to = "user@example.com", subject = "Hello", body = "Welcome!"))

Notification.email.getOption(smsNotif)
// => None (smsNotif is an SMS, not an Email)

Notification.sms.getOption(smsNotif)
// Option[SMS] = Some(SMS(phoneNumber = "+1234567890", message = "Your code is 1234"))
```

2. **`Prism#getOrFail`** — Extract with detailed error information on failure:

```text
trait Prism[S, A <: S] {
  def getOrFail(s: S): Either[OpticCheck, A]
}
```

It returns `Right(a)` if the source `s` matches the expected case `A`, or `Left(opticCheck)` containing detailed error information if it does not:

```scala
Notification.email.getOrFail(emailNotif)
// => Right(Email(to = "user@example.com", subject = "Hello", body = "Welcome!"))

Notification.email.getOrFail(pushNotif)
// res5: Either[OpticCheck, Email] = Left(
//   OpticCheck(
//     List(
//       UnexpectedCase(
//         expectedCase = "Email",
//         actualCase = "Push",
//         full = DynamicOptic(ArraySeq(Case("Email"))),
//         prefix = DynamicOptic(ArraySeq(Case("Email"))),
//         actualValue = Push(
//           deviceId = "device-abc",
//           title = "Alert",
//           payload = Map("action" -> "open")
//         )
//       )
//     )
//   )
// )
```

3. **`Prism#reverseGet`** — Upcast a specific case back to the parent sum type:

```text
trait Prism[S, A <: S] {
  def reverseGet(a: A): S
}
```

An example of upcasting an `Email` back to `Notification`:

```scala
val email = Notification.Email("alice@example.com", "Meeting", "See you at 3pm")
val notification: Notification = Notification.email.reverseGet(email)
// => Email("alice@example.com", "Meeting", "See you at 3pm") as Notification
```

4. **`Prism#replace`** — Replace the value if it matches the case:

```scala
trait Prism[S, A <: S] {
  def replace(s: S, a: A): S
}
```

In the following example, we are replacing an existing `Email` notification with a new one:

```scala
val newEmail = Notification.Email("new@example.com", "Updated", "New content")

Notification.email.replace(emailNotif, newEmail)
// => Email("new@example.com", "Updated", "New content")
```

If the original value is NOT the expected case, replace returns unchanged:
```scala
Notification.email.replace(smsNotif, newEmail)
// res7: Notification = SMS(
//   phoneNumber = "+1234567890",
//   message = "Your code is 1234"
// )
// => SMS("+1234567890", "Your code is 1234") (unchanged, it was an SMS)
```

5. **`Prism#replaceOption`** — Replace returning Some on success, None on mismatch:

```scala
trait Prism[S, A <: S] {
  def replaceOption(s: S, a: A): Option[S]
}
```

The following example shows replacing an `Email` notification with a new one, returning `Some` on success and `None` if the original value is not an `Email`:

```scala
Notification.email.replaceOption(emailNotif, newEmail)
// => Some(Email("new@example.com", "Updated", "New content"))

Notification.email.replaceOption(smsNotif, newEmail)
// => None (cannot replace, smsNotif is not an Email)
```

6. **`Prism#replaceOrFail`** — Replace with detailed error on mismatch:

```scala
trait Prism[S, A <: S] {
  def replaceOrFail(s: S, a: A): Either[OpticCheck, S]
}
```

The following example shows replacing an `Email` notification with a new one, returning `Right(newValue)` on success and `Left(opticCheck)` with detailed error information if the original value is not an `Email`:

```scala
Notification.email.replaceOrFail(emailNotif, newEmail)
// => Right(Email("new@example.com", "Updated", "New content"))

Notification.email.replaceOrFail(pushNotif, newEmail)
// res11: Either[OpticCheck, Notification] = Left(
//   OpticCheck(
//     List(
//       UnexpectedCase(
//         expectedCase = "Email",
//         actualCase = "Push",
//         full = DynamicOptic(ArraySeq(Case("Email"))),
//         prefix = DynamicOptic(ArraySeq(Case("Email"))),
//         actualValue = Push(
//           deviceId = "device-abc",
//           title = "Alert",
//           payload = Map("action" -> "open")
//         )
//       )
//     )
//   )
// )
```

## Optional

You can think of **`Optional[S, A]`** as the composition of lenses and prisms — it focuses on a value that may or may not exist. Used for accessing fields through variant types, optional fields, or elements at specific indices.

```scala
sealed trait Optional[S, A] extends Optic[S, A] {
  def getOption(s: S): Option[A]
  def getOrFail(s: S): Either[OpticCheck, A]
  def replace(s: S, a: A): S
  def replaceOption(s: S, a: A): Option[S]
  def replaceOrFail(s: S, a: A): Either[OpticCheck, S]
}
```

| Method          | Signature                         | Description                                                  |
|-----------------|-----------------------------------|--------------------------------------------------------------|
| `getOption`     | `(S)    => Option[A]`             | Extracts the focused value if accessible, otherwise `None`   |
| `getOrFail`     | `(S)    => Either[OpticCheck, A]` | Extracts with detailed error info on failure                 |
| `replace`       | `(S, A) => S`                     | Replaces if accessible, otherwise returns original unchanged |
| `replaceOption` | `(S, A) => Option[S]`             | Replaces if accessible, otherwise `None`                     |
| `replaceOrFail` | `(S, A) => Either[OpticCheck, S]` | Replaces with detailed error info on failure                 |

### Manual Optional Construction

Unlike `Lens` and `Prism` which have direct constructors, `Optional` is primarily constructed through **composition** of other optics, mainly from `Lens` and `Prism`:

```scala
object Optional {
  // Compose Lens with Prism (yields Optional)
  def apply[S, T, A <: T](first: Lens[S, T], second: Prism[T, A]): Optional[S, A]

  // Compose Prism with Lens (yields Optional)
  def apply[S, T <: S, A](first: Prism[S, T], second: Lens[T, A]): Optional[S, A]
}
```

| First         | Second        | Output           |
|---------------|---------------|------------------|
| `Lens[S, T]`  | `Prism[T, A]` | `Optional[S, A]` |
| `Prism[S, T]` | `Lens[T, A]`  | `Optional[S, A]` |

These two basic compositions allow you to build `Optional` optics by combining lenses and prisms in either order. 

In addition, `Optional` can also be composed with other optics to create more complex optionals, all these composition methods are supported via the `Optional.apply` method overloads:

| First              | Second             | Output           |
|--------------------|--------------------|------------------|
| `Lens[S, T]`       | `Optional[T, A]`   | `Optional[S, A]` |
| `Optional[S, T]`   | `Lens[T, A]`       | `Optional[S, A]` |
| ------------------ | ------------------ | ---------------- |
| `Prism[S, T]`      | `Optional[T, A]`   | `Optional[S, A]` |
| `Optional[S, T]`   | `Prism[T, A]`      | `Optional[S, A]` |
| --------------     | ---------------    | ---------------- |
| `Optional[S, T]`   | `Optional[T, A]`   | `Optional[S, A]` |

Beside the above composition methods, `Optional` provides specialized constructors for common patterns such as index-based access in sequences, key-based access in maps, and accessing wrapped types:

| Constructor        | Signature                                                  | Description                             |
|--------------------|------------------------------------------------------------|-----------------------------------------|
| `Optional.at`      | `(Reflect.Sequence.Bound[A, C], Int) => Optional[C[A], A]` | Accesses element at index in a sequence |
| `Optional.atKey`   | `(Reflect.Map.Bound[K, V, M], K) => Optional[M[K, V], V]`  | Accesses value at key in a map          |
| `Optional.wrapped` | `(Reflect.Wrapper.Bound[A, B]) => Optional[A, B]`          | Accesses inner value of a wrapper type  |

#### Example: Composing Lens and Prism

The most common way to manually construct an `Optional` is by composing a `Lens` with a `Prism`. Assume you have a `PaymentMethod` sum type which has multiple cases, and we have written a prism for one of its cases, e.g., `CreditCard`:

```scala

sealed trait PaymentMethod
object PaymentMethod extends CompanionOptics[PaymentMethod] {
  case class CreditCard(number: String, expiry: String) extends PaymentMethod
  case class Cryptocurrency(walletAddress: String) extends PaymentMethod
  
  implicit val schema: Schema[PaymentMethod] = Schema.derived
  
  val creditCard     : Prism[PaymentMethod, CreditCard]     = optic(_.when[CreditCard])
  val cryptocurrency : Prism[PaymentMethod, Cryptocurrency] = optic(_.when[Cryptocurrency])
}
```

And also assume we have a `Customer` record that has a `payment` field of type `PaymentMethod` and we have written a lens for that field:

```scala
case class Customer(name: String, payment: PaymentMethod)
object Customer extends CompanionOptics[Customer] {
  implicit val schema: Schema[Customer] = Schema.derived
  
  val name   : Lens[Customer, String       ] = optic(_.name)
  val payment: Lens[Customer, PaymentMethod] = optic(_.payment)
}
```

We can now compose the `payment` lens with the `creditCard` prism to create an `Optional` that focuses on the `CreditCard` details within a `Customer`:

```scala
// Compose lens and prism manually to get Optional
val creditCard: Optional[Customer, PaymentMethod.CreditCard] =
  Optional(Customer.payment, PaymentMethod.creditCard)
```

#### Example: Index-Based Access

For accessing elements at specific indices in sequences:

```scala

case class Order(id: String, items: List[String])
object Order extends CompanionOptics[Order] {
  implicit val schema: Schema[Order] = Schema.derived[Order]
  
  val items: Lens[Order, List[String]] = optic(_.items)
  
  // Manual Optional for first item
  val firstItem: Optional[Order, String] = {
    val atFirst = Optional.at(
      Schema[List[String]].reflect.asSequence.get,
      index = 0
    )
    Optional(items, atFirst)
  }
}
```

### Automatic Macro-Based Optional Derivation

The `optic` macro inside the `CompanionOptics` trait creates optionals automatically through several syntactic patterns. The macro intelligently determines when an `Optional` is needed based on the path expression.

#### Accessing Inner Values of ADTs

By combining the `.when[Case]` (prism) and `.field-name` (lens) syntax of optic macro, you can create optionals that focus on the inner values of ADTs:

```scala

case class ApiResponse(
  requestId: String,
  timestamp: Option[Long],
  result: Either[String, Int] // Left = error message, Right = status code
)
object ApiResponse extends CompanionOptics[ApiResponse] {
  implicit val schema: Schema[ApiResponse] = Schema.derived[ApiResponse]

  // Optional to the inner Long value (may not exist if None)
  val timestamp: Optional[ApiResponse, Long] = optic(_.timestamp.when[Some[Long]].value)

  // Optional to the Left value (may not exist if Right)
  val errorMessage: Optional[ApiResponse, String] = optic(_.result.when[Left[String, Int]].value)

  // Optional to the Right value (may not exist if Left)
  val statusCode: Optional[ApiResponse, Int] = optic(_.result.when[Right[String, Int]].value)
}
```

Here is another example focusing on sum types:

```scala

sealed trait Response
object Response extends CompanionOptics[Response] {
  case class Success(data: String, code: Int) extends Response
  case class Failure(error: String)           extends Response
  
  implicit val schema: Schema[Response] = Schema.derived
  
  // Prism to the Success case
  val success: Prism[Response, Success] = optic(_.when[Success])
  
  // Optional: Prism + Lens = Optional
  // Focuses on the 'data' field inside Success case
  val successData : Optional[Response, String] = optic(_.when[Success].data)
  val successCode : Optional[Response, Int   ] = optic(_.when[Success].code)
  val failureError: Optional[Response, String] = optic(_.when[Failure].error)
}
```

#### Index-based Access with `.at(index)` Syntax

We can access elements at specific indices in sequences using `.at(index)` syntax in `optic` macro:

```scala

case class OrderItem(sku: String, quantity: Int)
object OrderItem {
  implicit val schema: Schema[OrderItem] = Schema.derived
}

case class Order(id: String, items: List[OrderItem])
object Order extends CompanionOptics[Order] {
  implicit val schema: Schema[Order] = Schema.derived
  
  // Optional for specific indices
  val firstItem : Optional[Order, OrderItem] = optic(_.items.at(0))
  val secondItem: Optional[Order, OrderItem] = optic(_.items.at(1))
  
  // Chain with field access
  val firstItemSku     : Optional[Order, String] = optic(_.items.at(0).sku)
  val firstItemQuantity: Optional[Order, Int   ] = optic(_.items.at(0).quantity)
}
```

Usage:

```scala
val order = Order("ord-1", List(
  OrderItem("SKU-A", 2),
  OrderItem("SKU-B", 1)
))
val emptyOrder = Order("ord-2", List.empty)

Order.firstItem.getOption(order)       // => Some(OrderItem("SKU-A", 2))
Order.secondItem.getOption(order)      // => Some(OrderItem("SKU-B", 1))
Order.firstItem.getOption(emptyOrder)  // => None (empty list)

Order.firstItemSku.getOption(order)    // => Some("SKU-A")
```

#### Key-based Access with `.atKey(key)`

To access values at a specific key, we can use `.atKey(key)` syntax in `optic` macro:

```scala

case class Config(settings: Map[String, String])
object Config extends CompanionOptics[Config] {
  implicit val schema: Schema[Config] = Schema.derived
  
  // Optional for specific keys
  def setting(key: String): Optional[Config, String] = optic(_.settings.atKey(key))
  
  // Pre-defined optionals for common keys
  val hostSetting: Optional[Config, String] = optic(_.settings.atKey("host"))
  val portSetting: Optional[Config, String] = optic(_.settings.atKey("port"))
}
```

Here is how you can use these optionals:

```scala
val config = Config(Map("host" -> "localhost", "port" -> "8080"))

Config.hostSetting.getOption(config)        // => Some("localhost")
Config.setting("timeout").getOption(config) // => None (key not present)
```

#### Wrapper Type Access with `.wrapped[T]` Syntax

To access the inner value of wrapper types (newtypes, opaque types) you can use the `.wrapped[T]` syntax in `optic` macro:

```scala

// Assume Email is a wrapper around String with validation
case class Email private (value: String)
object Email {
  implicit val schema: Schema[Email] = Schema.derived
  
  def apply(s: String): Either[String, Email] =
    if (s.contains("@")) Right(new Email(s)) else Left("Invalid email")
}

case class Contact(name: String, email: Email)
object Contact extends CompanionOptics[Contact] {
  implicit val schema: Schema[Contact] = Schema.derived
  
  // Optional to access the wrapped String inside Email
  val emailString: Optional[Contact, String] = optic(_.email.wrapped[String])
}
```

## Traversal

A **`Traversal[S, A]`** focuses on zero or more elements within a collection (List, Vector, Set, Map, etc.). Unlike `Lens` (exactly one element) or `Optional` (zero or one element), a `Traversal` can target any number of elements simultaneously.

The key methods of `Traversal` are:

| Method         | Signature                                   | Description                                      |
|----------------|---------------------------------------------|--------------------------------------------------|
| `fold`         | `(S)(Z, (Z, A) => Z) => Z`                  | Aggregates all focused values                    |
| `reduceOrFail` | `(S)((A, A) => A) => Either[OpticCheck, A]` | Reduces with error handling                      |
| `modify`       | `(S, A => A)      => S`                     | Applies function to all focused values           |
| `modifyOption` | `(S, A => A)      => Option[S]`             | Modifies if any elements exist, otherwise `None` |
| `modifyOrFail` | `(S, A => A)      => Either[OpticCheck, S]` | Modifies with detailed error info on failure     |

### Manual Traversal Construction

Traversals are primarily constructed through the `Traversal` companion object which provides specialized constructors for different collection types:

#### Sequence Traversals

The two basic constructors for creating traversals over sequences are `seqValues` and `atIndices`:

```scala
object Traversal {
  // Traverse all elements in a sequence (List, Vector, Set, ArraySeq, etc.)
  def seqValues[A, C[_]](seq: Reflect.Sequence.Bound[A, C]): Traversal[C[A], A]

  // Traverse specific indices
  def atIndices[A, C[_]](seq: Reflect.Sequence.Bound[A, C], indices: Seq[Int]): Traversal[C[A], A]
}
```

The `seqValues` method creates a traversal that focuses on all elements in a sequence, while `atIndices` creates a traversal that focuses only on elements at the specified indices. 

There are also convenience methods for common collection types:

```scala
object Traversal {
  def listValues[A]    (reflect: Reflect.Bound[A]): Traversal[List[A],     A]
  def vectorValues[A]  (reflect: Reflect.Bound[A]): Traversal[Vector[A],   A]
  def setValues[A]     (reflect: Reflect.Bound[A]): Traversal[Set[A],      A]
  def arraySeqValues[A](reflect: Reflect.Bound[A]): Traversal[ArraySeq[A], A]
}
```

For example, to create a traversal over all items in a shopping cart represented as a list and quantities as a vector, you can do the following:

```scala

case class ShoppingCart(items: List[String], quantities: Vector[Int])
object ShoppingCart {
  implicit val schema: Schema[ShoppingCart] = Schema.derived

  // Manual traversal for all items in the cart
  val allItems: Traversal[List[String], String] =
    Traversal.listValues(Schema[String].reflect)

  // Manual traversal for all quantities
  val allQuantities: Traversal[Vector[Int], Int] =
    Traversal.vectorValues(Schema[Int].reflect)
}
```

#### Map Traversals

To create traversals over keys or values in a map you can use the following constructors:

```scala
object Traversal {
  // Traverse all keys in a map
  def mapKeys[K, V, M[_, _]](map: Reflect.Map.Bound[K, V, M]): Traversal[M[K, V], K]
  
  // Traverse all values in a map
  def mapValues[K, V, M[_, _]](map: Reflect.Map.Bound[K, V, M]): Traversal[M[K, V], V]
  
  // Traverse values at specific keys
  def atKeys[K, V, M[_, _]](map: Reflect.Map.Bound[K, V, M], keys: Seq[K]): Traversal[M[K, V], V]
}
```

Let's say you have an inventory represented as a map of product names to stock counts. You can create traversals for both keys and values as follows:

```scala

case class Inventory(stock: Map[String, Int])
object Inventory {
  implicit val schema: Schema[Inventory] = Schema.derived
  
  // Manual traversal for all product names (keys)
  val productNames: Traversal[Map[String, Int], String] =
    Traversal.mapKeys(Schema[Map[String, Int]].reflect.asMap.get)
  
  // Manual traversal for all stock counts (values)
  val stockCounts: Traversal[Map[String, Int], Int] =
    Traversal.mapValues(Schema[Map[String, Int]].reflect.asMap.get)
}
```

### Automatic Macro-Based Traversal Derivation

The `optic` macro inside the `CompanionOptics` trait creates traversals using intuitive selector syntax with the `.each`, `.eachKey`, and `.eachValue` methods.

1. Use `.each` to traverse all elements in a sequence (List, Vector, Set, ArraySeq):

```scala

case class Order(id: String, items: List[String], prices: Vector[Double])
object Order extends CompanionOptics[Order] {
  implicit val schema: Schema[Order] = Schema.derived
  
  // Traversal over all items
  val allItems: Traversal[Order, String] = optic(_.items.each)
  
  // Traversal over all prices
  val allPrices: Traversal[Order, Double] = optic(_.prices.each)
}
```

2. Use `.eachKey` to traverse all keys and `.eachValue` to traverse all values in a map:

```scala

case class UserScores(scores: Map[String, Int])
object UserScores extends CompanionOptics[UserScores] {
  implicit val schema: Schema[UserScores] = Schema.derived
  
  // Traversal over all user names (keys)
  val allUserNames: Traversal[UserScores, String] = optic(_.scores.eachKey)
  
  // Traversal over all scores (values)
  val allScores: Traversal[UserScores, Int] = optic(_.scores.eachValue)
}
```

3. Use `.atIndices(indices)` to traverse elements at specific indices:

```scala

case class Matrix(rows: Vector[Vector[Int]])
object Matrix extends CompanionOptics[Matrix] {
  implicit val schema: Schema[Matrix] = Schema.derived
  
  // Traverse elements at indices 0, 2, and 4
  val selectedRows: Traversal[Matrix, Vector[Int]] = optic(_.rows.atIndices(0, 2, 4))
}
```

4. Use `.atKeys(keys)` to traverse values at specific keys:

```scala

case class Environment(variables: Map[String, String])
object Environment extends CompanionOptics[Environment] {
  implicit val schema: Schema[Environment] = Schema.derived
  
  // Traverse values for specific environment variables
  val criticalVars: Traversal[Environment, String] = optic(_.variables.atKeys("PATH", "HOME", "USER"))
}
```

Please note that the `optic` macro supports chaining traversals with field access for deep navigation:

```scala

case class LineItem(sku: String, price: Double, quantity: Int)
object LineItem {
  implicit val schema: Schema[LineItem] = Schema.derived
}

case class Invoice(id: String, items: List[LineItem])
object Invoice extends CompanionOptics[Invoice] {
  implicit val schema: Schema[Invoice] = Schema.derived
  
  // Traverse to get all SKUs from all items
  val allSkus: Traversal[Invoice, String] = optic(_.items.each.sku)
  
  // Traverse to get all prices from all items
  val allPrices: Traversal[Invoice, Double] = optic(_.items.each.price)
  
  // Traverse to get all quantities from all items
  val allQuantities: Traversal[Invoice, Int] = optic(_.items.each.quantity)
}
```

### Operations

Let's explore traversal operations with a practical example. Assume we have a `Team` case class with a list of members and a map of scores:

```scala

case class Team(name: String, members: List[String], scores: Map[String, Int])
object Team extends CompanionOptics[Team] {
  implicit val schema: Schema[Team] = Schema.derived[Team]
  
  val allMembers    : Traversal[Team, String] = optic(_.members.each)
  val allScores     : Traversal[Team, Int   ] = optic(_.scores.eachValue)
  val allPlayerNames: Traversal[Team, String] = optic(_.scores.eachKey)
}

val team = Team(
  "Alpha",
  List("Alice", "Bob", "Charlie"),
  Map("Alice" -> 100, "Bob" -> 85, "Charlie" -> 92)
)
```

1. `Traversal#fold` aggregates all focused values:

```scala
// Count all members
Team.allMembers.fold(team)(0, (count, _) => count + 1)
// => 3

// Sum all scores
Team.allScores.fold(team)(0, _ + _)
// => 277

// Concatenate all member names
Team.allMembers.fold(team)("", (acc, name) => if (acc.isEmpty) name else s"$acc, $name")
// => "Alice, Bob, Charlie"
```

2. `Traversal#reduceOrFail` reduces with error handling:

```scala
// Find the maximum score
Team.allScores.reduceOrFail(team)(math.max)
// => Right(100)

// Find the minimum score
Team.allScores.reduceOrFail(team)(math.min)
// => Right(85)

// Attempt to reduce an empty collection
val emptyTeam = Team("Empty", Nil, Map.empty)
Team.allMembers.reduceOrFail(emptyTeam)(_ + _)
// => Left(OpticCheck(...)) with EmptySequence error
```

3. `Traversal#modify` — Apply function to all focused values:

```scala
// Convert all member names to uppercase
Team.allMembers.modify(team, _.toUpperCase)
// => Team("Alpha", List("ALICE", "BOB", "CHARLIE"), Map(...))

// Double all scores
Team.allScores.modify(team, _ * 2)
// => Team("Alpha", List(...), Map("Alice" -> 200, "Bob" -> 170, "Charlie" -> 184))

// Add prefix to all player names (keys)
Team.allPlayerNames.modify(team, name => s"Player: $name")
// => Team("Alpha", List(...), Map("Player: Alice" -> 100, ...))
```

4. `Traversal#modifyOption` — Modify returning Option

```scala
// Modify non-empty collection
Team.allMembers.modifyOption(team, _.toUpperCase)
// => Some(Team("Alpha", List("ALICE", "BOB", "CHARLIE"), Map(...)))

// Modify empty collection
val emptyTeam = Team("Empty", Nil, Map.empty)
Team.allMembers.modifyOption(emptyTeam, _.toUpperCase)
// => None
```

5. `Traversal#modifyOrFail` — Modify with detailed error on failure

```scala
// Successful modification
Team.allMembers.modifyOrFail(team, _.toUpperCase)
// => Right(Team("Alpha", List("ALICE", "BOB", "CHARLIE"), Map(...)))

// Failed modification (empty collection)
val emptyTeam = Team("Empty", Nil, Map.empty)
Team.allMembers.modifyOrFail(emptyTeam, _.toUpperCase)
// => Left(OpticCheck(List(EmptySequence(...))))
```

## Error Reporting with OpticCheck

The `...OrFail` methods, such as `Prism#getOrFail`, `Optional#getOrFail`, `Optional#replaceOrFail`, `Traversal#modifyOrFail` and `Traversal#reduceOrFail`, return `Left(OpticCheck)` when the optic cannot reach its focus. An `OpticCheck` holds a non-empty list of problems, `OpticCheck#errors`. Each problem is an `OpticCheck.Error`, which stops the optic, or an `OpticCheck.Warning`, which reports a missing part:

| Problem                    | Kind    | Reported when                                                               | Extra fields                                |
| -------------------------- | ------- | --------------------------------------------------------------------------- | ------------------------------------------- |
| `UnexpectedCase`           | Error   | A prism step found another case of the variant                              | `expectedCase`, `actualCase`, `actualValue` |
| `WrappingError`            | Error   | A wrapper step's conversion failed                                          | `error` (a `SchemaError`)                   |
| `DynamicConversionError`   | Error   | `SchemaExpr#eval` could not decode its result, or failed for another reason | `error` (a `String`)                        |
| `EmptySequence`            | Warning | A step over every element found an empty sequence                           | none                                        |
| `SequenceIndexOutOfBounds` | Warning | An index step went past the end of the sequence                             | `index`, `size`                             |
| `MissingKey`               | Warning | A key step found a map without the key                                      | `key`                                       |
| `EmptyMap`                 | Warning | A step over every key or value found an empty map                           | none                                        |

Every problem has `full`, the path of the whole optic, `prefix`, the path up to and including the step that failed, and `OpticCheck.Single#message`. `DynamicConversionError` has no step, so both of its paths are the root and its message gives only the reason; every other message names both paths. Only `UnexpectedCase` carries the value it found. `OpticCheck#message` joins the messages of all problems, one per line, and `OpticCheck#hasError` and `OpticCheck#hasWarning` tell the two kinds apart.

To see two of them, we define a parcel that is either delivered or in transit, with an `Optional` into the delivered case and a `Traversal` over the stops of the in-transit case:

```scala
sealed trait Parcel
case class Delivered(signedBy: String)    extends Parcel
case class InTransit(stops: List[String]) extends Parcel

object Parcel extends CompanionOptics[Parcel] {
  implicit val schema: Schema[Parcel] = Schema.derived

  val signedBy: Optional[Parcel, String] = optic(_.when[Delivered].signedBy)
  val stops: Traversal[Parcel, String]   = optic(_.when[InTransit].stops.each)
}
```

Reading the signature of a parcel in transit fails with `UnexpectedCase`, and modifying the stops of a parcel with no stops reports `EmptySequence`, a warning:

```scala
Parcel.signedBy.getOrFail(InTransit(List("Berlin"))).left.foreach { check =>
  println(check.message)
  check.errors.head match {
    case e: OpticCheck.UnexpectedCase => println(s"actual value: ${e.actualValue}")
    case _                            => ()
  }
}
// During attempted access at .when[Delivered].signedBy, encountered an unexpected case at .when[Delivered]: expected Delivered, but got InTransit
// actual value: InTransit(List(Berlin))

Parcel.stops.modifyOrFail(InTransit(Nil), _.toUpperCase).left.foreach { check =>
  println(s"${check.message} (error: ${check.hasError}, warning: ${check.hasWarning})")
}
// During attempted access at .when[InTransit].stops.each, encountered an empty sequence at .when[InTransit].stops.each (error: false, warning: true)
```

## Debug-Friendly toString

All optic types (`Lens`, `Prism`, `Optional`, `Traversal`) have a custom `toString` that mirrors the `optic` macro syntax, which shows at a glance what path an optic represents:

```scala
case class Warehouse(street: String, city: String)
case class Depot(name: String, warehouse: Warehouse, shelves: List[String], stock: Map[String, Int])

object Depot extends CompanionOptics[Depot] {
  implicit val schema: Schema[Depot] = Schema.derived

  val street: Lens[Depot, String]           = optic(_.warehouse.street)
  val shelves: Traversal[Depot, String]     = optic(_.shelves.each)
  val firstShelf: Optional[Depot, String]   = optic(_.shelves.at(0))
  val stockOfBooks: Optional[Depot, Int]    = optic(_.stock.atKey("books"))
  val someShelves: Traversal[Depot, String] = optic(_.shelves.atIndices(0, 2))
}

println(Depot.street)
// Lens(_.warehouse.street)
println(Depot.shelves)
// Traversal(_.shelves.each)
println(Depot.firstShelf)
// Optional(_.shelves.at(0))
println(Depot.stockOfBooks)
// Optional(_.stock.atKey(<key>))
println(Depot.someShelves)
// Traversal(_.shelves.atIndices(<indices>))
```

The output is not always code you can paste back into `optic`: a step that focuses a map key prints the placeholder `atKey(<key>)` instead of the key, steps for several indices or keys print `atIndices(<indices>)` and `atKeys(<keys>)`, and a traversal built with `searchFor` prints only the type it searches for. `DynamicOptic#toString`, the path syntax of the `p"..."` interpolator, has gaps of its own, a known bug in 0.0.56, listed in [Path Interpolator](./path-interpolator.md#debug-friendly-tostring).

**Examples by optic type:**

| Optic                              | toString Output                       |
|------------------------------------|---------------------------------------|
| `Lens` for field                   | `Lens(_.name)`                        |
| `Lens` for nested field            | `Lens(_.address.street)`              |
| `Prism` for variant case           | `Prism(_.when[CreditCard])`           |
| `Optional` combining prism + lens  | `Optional(_.when[Success].data)`      |
| `Optional` at a sequence index     | `Optional(_.items.at(0))`             |
| `Optional` at a map key            | `Optional(_.metadata.atKey(<key>))`   |
| `Traversal` over sequence          | `Traversal(_.items.each)`             |
| `Traversal` over map keys          | `Traversal(_.metadata.eachKey)`       |
| `Traversal` over map values        | `Traversal(_.metadata.eachValue)`     |

## Composing Optics

All optics can be composed together to create more complex access paths. All optics that extend the base `Optic` trait support composition via the `apply` method, which takes another optic as an argument and returns a new optic representing the combined access path:
```scala
trait Optic[S, A] {
  def apply[B](that: Lens[A, B]): Optic[S, B]
  def apply[B <: A](that: Prism[A, B]): Optic[S, B]
  def apply[B](that: Optional[A, B]): Optic[S, B]
  def apply[B](that: Traversal[A, B]): Traversal[S, B]
}
```

All optics (`Lens`, `Prism`, `Optional`, `Traversal`) implement the above `apply` methods to support composition with other optics. Additionally, they have `apply` overloads on their companion objects that take two optics as arguments and return the composed optic.

The following table summarizes the composition rules for combining different optic types:

| `this` ↓ / `that` → | **`Lens`**  | **`Prism`** | **`Optional`** | **`Traversal`** |
|---------------------|-------------|-------------|----------------|-----------------|
| **`Lens`**          | `Lens`      | `Optional`  | `Optional`     | `Traversal`     |
| **`Prism`**         | `Optional`  | `Prism`     | `Optional`     | `Traversal`     |
| **`Optional`**      | `Optional`  | `Optional`  | `Optional`     | `Traversal`     |
| **`Traversal`**     | `Traversal` | `Traversal` | `Traversal`    | `Traversal`     |

Here are some important notes regarding optic composition:

1. As the table demonstrates, `Traversal` is the most general optic type, as composing with it always results in a `Traversal`. This behavior occurs because `Traversal` focuses on zero or more elements, and once you have that level of generality, you cannot return to a more specific optic type. This is the **absorption property** encoded in the type system.

2. Using `Optic#apply` instead of the `apply` methods on companion objects allows for a more natural DSL-like syntax for composing optics. For example, instead of writing `Lens(Person.address, Address.postalCode)`, we can write `Person.address(Address.postalCode)`, which reads naturally as "address's postal code".

3. `Optional` is an absorbing optic for `Lens` and `Prism`—once you have partiality in your access path, composing with total optics (`Lens`) or other partial optics (`Prism`, `Optional`) preserves that partiality. `Traversal` is the most general optic, as composing with it always results in a `Traversal`.

### Examples

In this section, we explore the composition of `Lens` with other optics. The composition of other optics follows similar patterns.

#### Composing a Lens with Another Lens

We can chain two lenses to focus deeper into nested structures using the following composition operators:

```scala
object Lens {
  def apply[S, T, A](
    first : Lens[S, T],   // S => T
    second: Lens[T, A]    // T => A
  ): Lens[S, A] = ???     // S => A
}
```

This method takes two lenses: the first from `S` to `T`, and the second from `T` to `A`, and returns a new lens from `S` to `A`.

For example, if we have a `Person` case class that contains an `Address` and we want to create a lens to access the `street` field of the `Address` within `Person`, we can do so as follows:

```scala

case class Address(street: String, city: String)
object Address extends CompanionOptics[Address] { 
  implicit val schema: Schema[Address] = Schema.derived 
  
  val street: Lens[Address, String] =
    Lens[Address, String](
      Schema[Address].reflect.asRecord.get,
      Schema[String].reflect.asTerm("street")
    )
    
  val city: Lens[Address, String] =
    Lens[Address, String](
      Schema[Address].reflect.asRecord.get,
      Schema[String].reflect.asTerm("city")
    )
}
case class Person(name: String, age: Int, address: Address)
object Person {

  implicit val schema: Schema[Person] = Schema.derived[Person]

  val address: Lens[Person, Address] =
    Lens[Person, Address](
      Schema[Person].reflect.asRecord.get,
      Schema[Address].reflect.asTerm("address")
    )
    
  val street: Lens[Person, String] = 
    Lens[Person, Address, String](
      Person.address, // Lens from Person to Address
      Address.street  // Lens from Address to String (street field)
    )
}
```

To make the DSL more convenient, we can use the `Lens#apply` method:

```scala
object Person {
  val street: Lens[Person, String] = Person.address(Address.street)
}
```

The `optic` macro (or its alias `$`) provides a more concise way to derive composed lenses. By extending `CompanionOptics[T]` and using selector syntax, you can derive the same lenses with significantly less boilerplate:

```scala

case class Address(street: String, city: String)
object Address extends CompanionOptics[Address] 
case class Person(name: String, age: Int, address: Address)
object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  // Simple field lens
  val address: Lens[Person, Address] = optic(_.address)
  
  // Composed lens - directly access nested field via path syntax
  val street: Lens[Person, String] = optic(_.address.street)
  val city  : Lens[Person, String] = optic(_.address.city)
}
```

The `optic` macro inspects the selector path and automatically composes the necessary lenses. The path `_.address.street` is expanded into a composition of `Person → Address` and `Address → String` lenses.

#### Composing a Lens with a Prism

When you compose a `Lens` with a `Prism`, the result is an `Optional`. This makes sense because a `Lens` always succeeds (fields always exist in records), whereas a `Prism` may fail (a value may not match the particular case being targeted). Therefore, the composition may fail, which is exactly what `Optional` represents:

```scala
sealed trait Lens[S, A] extends Optic[S, A] {
  def apply[B <: A](that: Prism[A, B]): Optional[S, B]
}
```

Consider a scenario where you have a record containing a field whose type is a sealed trait (sum type), and you want to focus on a specific case of that sum type.

For instance, suppose we have an `Employee` case class that contains a `ContactInfo` field, where `ContactInfo` is a sealed trait with different cases:

```scala

sealed trait ContactInfo

object ContactInfo {
  case class Email(address: String) extends ContactInfo

  object Email {
    implicit val schema: Schema[Email] = Schema.derived[Email]
    
    val address: Lens[Email, String] =
      Lens[Email, String](
        Schema[Email].reflect.asRecord.get,
        Schema[String].reflect.asTerm("address")
      )
  }

  case class Phone(number: String) extends ContactInfo

  object Phone {
    implicit val schema: Schema[Phone] = Schema.derived[Phone]
  }

  case object NoContact extends ContactInfo

  implicit val schema: Schema[ContactInfo] = Schema.derived[ContactInfo]

  // Prism to focus on the Email case
  lazy val email: Prism[ContactInfo, Email] =
    Prism[ContactInfo, Email](
      Schema[ContactInfo].reflect.asVariant.get,
      Schema[Email].reflect.asTerm("Email")
    )

  // Prism to focus on the Phone case
  lazy val phone: Prism[ContactInfo, Phone] =
    Prism[ContactInfo, Phone](
      Schema[ContactInfo].reflect.asVariant.get,
      Schema[Phone].reflect.asTerm("Phone")
    )
}

case class Employee(name: String, contact: ContactInfo)

object Employee {
  implicit val schema: Schema[Employee] = Schema.derived[Employee]

  // Lens to focus on the contact field
  val contact: Lens[Employee, ContactInfo] =
    Lens[Employee, ContactInfo](
      Schema[Employee].reflect.asRecord.get,
      Schema[ContactInfo].reflect.asTerm("contact")
    )

  // Compose Lens with Prism to get an Optional
  // This focuses on the email address, but only if contact is an Email
  val contactEmail: Optional[Employee, ContactInfo.Email] =
    Employee.contact(ContactInfo.email)
  
  val emailAddress: Optional[Employee, String] =
    Employee.contactEmail(ContactInfo.Email.address)
}
```

The `Employee.contactEmail` is an `Optional[Employee, ContactInfo.Email]` that allows you to access or modify the email contact of an employee, but only if their contact information is indeed an `Email`. If the contact information is a `Phone` or `NoContact`, the operations will fail gracefully, returning `None`:

```scala
val employee1 = Employee("Alice", ContactInfo.Email("alice@example.com"))
// getOption returns Some when the contact is an Email
Employee.contactEmail.getOption(employee1) // => Some(Email("alice@example.com"))

// replace only succeeds if the contact is already an Email
Employee.contactEmail.replaceOption(employee1, ContactInfo.Email("newalice@example.com"))
// => Some(Employee("Alice", Email("newalice@example.com")))

val employee2 = Employee("Bob", ContactInfo.Phone("555-1234"))
// getOption returns None when the contact is NOT an Email
Employee.contactEmail.getOption(employee2) // => None

Employee.contactEmail.replaceOption(employee2, ContactInfo.Email("bob@example.com"))
// => None (Bob's contact is a Phone, not an Email)
```

You can further compose this `Optional` to reach deeper into the structure:

```scala
// Now you can get/set the email address string directly
Employee.emailAddress.getOption(employee1)
// => Some("alice@example.com")

Employee.emailAddress.getOption(employee2)
// => None
```

The `optic` macro supports case selection using the `.when[T]` syntax, which makes composing lenses with prisms much more concise:

```scala

sealed trait ContactInfo

object ContactInfo extends CompanionOptics[ContactInfo] {
  case class  Email(address: String) extends ContactInfo
  case class  Phone(number: String)  extends ContactInfo
  case object NoContact              extends ContactInfo

  implicit val schema: Schema[ContactInfo] = Schema.derived

  // Derive prisms using the optic macro with .when[T] syntax
  val email: Prism[ContactInfo, Email] = optic(_.when[Email])
  val phone: Prism[ContactInfo, Phone] = optic(_.when[Phone])
}

case class Employee(name: String, contact: ContactInfo)

object Employee extends CompanionOptics[Employee] {
  implicit val schema: Schema[Employee] = Schema.derived

  // Simple field lens
  val contact: Lens[Employee, ContactInfo] = optic(_.contact)

  // Compose Lens with Prism using path syntax - result is an Optional
  val contactEmail: Optional[Employee, ContactInfo.Email] = 
    optic(_.contact.when[ContactInfo.Email])
  
  // Chain even deeper to get the email address string
  val emailAddress: Optional[Employee, String] = 
    optic(_.contact.when[ContactInfo.Email].address)
    
  // Similarly for phone
  val phoneNumber: Optional[Employee, String] = 
    optic(_.contact.when[ContactInfo.Phone].number)
}
```

The path `_.contact.when[Email].address` is automatically composed into a `Lens → Prism → Lens` chain, producing an `Optional[Employee, String]` optic.

#### Composing a Lens with an Optional

When you compose a `Lens` with an `Optional`, the result is also an `Optional`. This follows naturally: a `Lens` always succeeds, and an `Optional` may fail; therefore, the composition may fail.

Building on our previous example, suppose we have a `Company` that contains an `Employee`:

```scala
case class Company(name: String, ceo: Employee)
object Company {
  implicit val schema: Schema[Company] = Schema.derived[Company]

  // Lens to focus on the CEO
  val ceo: Lens[Company, Employee] =
    Lens[Company, Employee](
      Schema[Company].reflect.asRecord.get,
      Schema[Employee].reflect.asTerm("ceo")
    )

  // Compose Lens with Optional to get another Optional
  // This chains: Company -> ceo (Lens) -> contactEmail (Optional)
  val ceoEmailContact: Optional[Company, ContactInfo.Email] =
    Company.ceo(Employee.contactEmail)

  // Further composition to get the email address string
  val ceoEmailAddress: Optional[Company, String] =
    Company.ceo(Employee.emailAddress)
}
```

Now you can work with the CEO's email contact through the company:

```scala
val techCorp = Company("TechCorp", Employee("Alice", ContactInfo.Email("alice@tech.com")))
val retailCo = Company("RetailCo", Employee("Bob", ContactInfo.Phone("555-9999")))

// Get the CEO's email contact
Company.ceoEmailContact.getOption(techCorp)
// => Some(Email("alice@tech.com"))

Company.ceoEmailContact.getOption(retailCo)
// => None (Bob's contact is a Phone)

// Get the CEO's email address string
Company.ceoEmailAddress.getOption(techCorp)
// => Some("alice@tech.com")

// Update the CEO's email address
Company.ceoEmailAddress.replaceOption(techCorp, "ceo@tech.com")
// => Some(Company("TechCorp", Employee("Alice", Email("ceo@tech.com"))))

Company.ceoEmailAddress.replaceOption(retailCo, "bob@retail.com")
// => None (cannot replace because Bob does not have an Email contact)
```

This composition pattern is powerful for navigating through complex nested structures where some paths may not be valid for all values.

With the `optic` macro, you can express the entire path in a single selector expression:

```scala

case class Company(name: String, ceo: Employee)
object Company extends CompanionOptics[Company] {
  implicit val schema: Schema[Company] = Schema.derived

  // Simple field lens
  val ceo: Lens[Company, Employee] = optic(_.ceo)

  // Compose through multiple levels using path syntax
  // Lens (ceo) + Lens (contact) + Prism (when[Email]) = Optional
  val ceoEmailContact: Optional[Company, ContactInfo.Email] =
    optic(_.ceo.contact.when[ContactInfo.Email])

  // Go even deeper to the email address string
  val ceoEmailAddress: Optional[Company, String] =
    optic(_.ceo.contact.when[ContactInfo.Email].address)
    
  // Similarly for phone number
  val ceoPhoneNumber: Optional[Company, String] =
    optic(_.ceo.contact.when[ContactInfo.Phone].number)
}
```

The macro automatically determines the correct output type based on the composition rules: since the path includes a `.when[Email]` prism selector, the result is an `Optional`.

#### Composing a Lens with a Traversal

When you compose a `Lens` with a `Traversal`, the result is a `Traversal`. This combination allows you to:

- First zoom into a specific field of a record (via the `Lens`)
- Then iterate over all elements in a collection at that field (via the `Traversal`)

If you have a record containing a collection field and you want to operate on all elements of that collection, you can use this composition.

For example, let's create a `Department` that has a list of employees, and we want to access all employee names:

```scala
case class Department(name: String, employees: List[Employee])
object Department {
  implicit val schema: Schema[Department] = Schema.derived[Department]

  // Lens to focus on the employees list
  val employees: Lens[Department, List[Employee]] =
    Lens[Department, List[Employee]](
      Schema[Department].reflect.asRecord.get,
      Schema[List[Employee]].reflect.asTerm("employees")
    )

  // Traversal to iterate over all elements in the list
  val eachEmployee: Traversal[List[Employee], Employee] =
    Traversal.listValues(Schema[Employee].reflect)

  // Compose Lens with Traversal
  // This focuses on all employees in the department
  val allEmployees: Traversal[Department, Employee] =
    Department.employees(Department.eachEmployee)
}
```

With this `Traversal`, you can fold over all employees or modify them:

```scala
val engineering = Department(
  "Engineering",
  List(
    Employee("Alice", ContactInfo.Email("alice@company.com")),
    Employee("Bob", ContactInfo.Phone("555-1234")),
    Employee("Charlie", ContactInfo.Email("charlie@company.com"))
  )
)

// Fold to collect all employee names
Department.allEmployees.fold(engineering)(List.empty[String], (acc, emp) => acc :+ emp.name)
// => List("Alice", "Bob", "Charlie")

// Modify all employees (e.g., update their contact to NoContact)
Department.allEmployees.modify(engineering, emp => emp.copy(contact = ContactInfo.NoContact))
// => Department("Engineering", List(
//      Employee("Alice", NoContact),
//      Employee("Bob", NoContact),
//      Employee("Charlie", NoContact)
//    ))
```

You can chain further to go even deeper. For example, to access all employee names in a department:

```scala
object Employee {
  val name: Lens[Employee, String] =
    Lens[Employee, String](
      Schema[Employee].reflect.asRecord.get,
      Schema[String].reflect.asTerm("name")
    )
}

object Department {
  // Chain: Department -> employees (Lens) -> each (Traversal) -> name (Lens)
  // Result type: Traversal[Department, String]
  val allEmployeeNames: Traversal[Department, String] =
    Department.allEmployees(Employee.name)
}

// Fold to get all names
Department.allEmployeeNames.fold(engineering)(List.empty[String], (acc, name) => acc :+ name)
// => List("Alice", "Bob", "Charlie")

// Modify all names (e.g., convert them to uppercase)
Department.allEmployeeNames.modify(engineering, _.toUpperCase)
// => Department("Engineering", List(
//      Employee("ALICE", ...),
//      Employee("BOB", ...),
//      Employee("CHARLIE", ...)
//    ))
```

You can even compose a `Traversal` with a `Prism` to filter elements. For example, to retrieve only the email addresses from employees who have email contacts:

```scala
object Department {
  // Chain: Department -> allEmployees (Traversal) -> contactEmail (Optional)
  // Traversal + Optional = Traversal
  val allEmailContacts: Traversal[Department, ContactInfo.Email] =
    Department.allEmployees(Employee.contactEmail)

  // Further chain to get email address strings
  val allEmailAddresses: Traversal[Department, String] =
    Department.allEmailContacts(ContactInfo.Email.address)
}

// This only folds over employees who have Email contacts
Department.allEmailAddresses.fold(engineering)(List.empty[String], (acc, addr) => acc :+ addr)
// => List("alice@company.com", "charlie@company.com")
// Note: Bob is skipped because he has a Phone contact
```

This example demonstrates the power of optics composition: you can build complex data access paths by combining simple, reusable building blocks, and the type system ensures the resulting optic has the correct semantics (always succeeding, potentially failing, or iterating over multiple values).

The `optic` macro supports collection traversal using the `.each` syntax. This allows you to express traversals over lists, vectors, and other sequences in a concise path expression:

```scala

case class Department(name: String, employees: List[Employee])
object Department extends CompanionOptics[Department] {
  implicit val schema: Schema[Department] = Schema.derived

  // Lens to the employees list
  val employees: Lens[Department, List[Employee]] = optic(_.employees)

  // Traversal over all employees using .each syntax
  val allEmployees: Traversal[Department, Employee] = optic(_.employees.each)

  // Chain deeper: Lens + Traversal + Lens = Traversal
  val allEmployeeNames: Traversal[Department, String] = optic(_.employees.each.name)
  
  // Lens + Traversal + Lens + Prism = Traversal
  val allEmailContacts: Traversal[Department, ContactInfo.Email] = 
    optic(_.employees.each.contact.when[ContactInfo.Email])

  // Go even deeper to get email address strings
  val allEmailAddresses: Traversal[Department, String] = 
    optic(_.employees.each.contact.when[ContactInfo.Email].address)
}
```

The path `_.employees.each.contact.when[Email].address` composes:
1. `employees` — a `Lens` to the `List`
2. `.each` — a `Traversal` over list elements
3. `contact` — a `Lens` to `ContactInfo`
4. `.when[Email]` — a `Prism` to the `Email` case
5. `address` — a `Lens` to the `String`

The result is a `Traversal[Department, String]` that focuses on all email addresses in the department.

For maps, the macro provides `.eachKey` and `.eachValue` for traversing keys and values, respectively:

```scala

case class Inventory(items: Map[String, Int])
object Inventory extends CompanionOptics[Inventory] {
  implicit val schema: Schema[Inventory] = Schema.derived

  // Lens to the items map
  val items: Lens[Inventory, Map[String, Int]] = optic(_.items)

  // Traversal over all keys
  val allItemNames: Traversal[Inventory, String] = optic(_.items.eachKey)
  
  // Traversal over all values
  val allQuantities: Traversal[Inventory, Int] = optic(_.items.eachValue)
}
```

---

## Patching

The Patching system in ZIO Blocks provides a type-safe, serializable way to describe and apply transformations to data structures. Unlike direct mutations or lens-based updates, patches are **first-class values** that can be serialized, transmitted over the network, stored for audit logs, and composed together.

## Overview

A `Patch[S]` represents a sequence of operations that transform a value of type `S`. Because patches use serializable operations and reflective optics for navigation, they enable powerful use cases:

- **Remote Patching** — Send patches over the network to update remote state without transmitting entire objects
- **Audit Logs** — Record patches as a log of changes for compliance, debugging, or undo functionality
- **CRDT-like Operations** — Use commutative operations like `Patch.increment` that can be safely applied in any order
- **Optimistic Updates** — Apply patches locally while syncing with a server
- **Schema Evolution** — Patches address fields by name, so a stored patch still applies after a field is added, but fails in `Strict` mode once a field it touches is renamed or removed

:::note
For **untyped JSON patching** without a schema, use [`JsonPatch`](./built-in-codecs/json/json-patch.md) instead. `JsonPatch` is optimized for diff-and-apply workflows on raw JSON values and provides compact delta representations without requiring typed optics.

For an end-to-end workflow (diffing two versions, storing patches as JSON, choosing a `PatchMode` when the data changed, and replaying an edit history), see [How to Compute, Store and Replay Changes with Diff and Patch](../../guides/diff-and-patch.md).
:::

To build a patch, we point smart constructors at fields through [optics](./optics.md), compose the results with `Patch#++`, and apply the composed patch to a value:

```scala

case class Person(name: String, age: Int)
object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived
  val name: Lens[Person, String] = optic(_.name)
  val age: Lens[Person, Int] = optic(_.age)
}

// Create patches using smart constructors
val patch1 = Patch.set(Person.name, "John")
val patch2 = Patch.increment(Person.age, 1)

// Compose patches
val combined = patch1 ++ patch2

val jane = Person("Jane", 25)
```

Applying the composed patch to `jane` sets the name and adds one to the age:

```scala
combined(jane)
// res1: Person = Person(name = "John", age = 26)
```

## Creating Patches

The `Patch` companion object provides smart constructors for patches. Each constructor takes an optic to specify the target location and the operation parameters.

### Setting Values

`Patch.set` replaces the value at the specified location:

```scala

case class Address(street: String, city: String, zip: String)
object Address extends CompanionOptics[Address] {
  implicit val schema: Schema[Address] = Schema.derived
  val street: Lens[Address, String] = optic(_.street)
  val city: Lens[Address, String] = optic(_.city)
}

case class Person(name: String, address: Address)
object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived
  val name: Lens[Person, String] = optic(_.name)
  val address: Lens[Person, Address] = optic(_.address)
  val city: Lens[Person, String] = optic(_.address.city)
}

// Set a simple field
val setName = Patch.set(Person.name, "Alice")

// Set a nested field
val setCity = Patch.set(Person.city, "San Francisco")

// Set an entire nested record
val newAddress = Address("123 Main St", "NYC", "10001")
val setAddress = Patch.set(Person.address, newAddress)
```

Setting the nested `city` field leaves the rest of the address untouched:

```scala
setCity(Person("Bob", Address("1 Market St", "Boston", "02108")))
// res3: Person = Person(
//   name = "Bob",
//   address = Address(
//     street = "1 Market St",
//     city = "San Francisco",
//     zip = "02108"
//   )
// )
```

`Patch.set` accepts every optic type. What happens when the optic does not match the value depends on the [application mode](#application-modes):

- **Lens** — Sets a single field.
- **Optional** — Sets the value when the path exists. When it does not (for example, the path goes through a case the value is not), `Strict` mode fails and `Lenient` and `Clobber` leave the value unchanged.
- **Traversal** — Sets every matching element to the same value. On an empty sequence, `Strict` mode fails with `encountered an empty sequence` and the other modes leave the value unchanged.
- **Prism** — Replaces the value only when it is already that case. When the value is another case, `Strict` mode fails and `Lenient` and `Clobber` leave the value unchanged; `Patch.set` never switches a value from one case to another.

To see the prism rule, we define a sum type and a patch that sets the `Circle` case:

```scala

sealed trait Shape
case class Circle(radius: Double) extends Shape
case class Square(side: Double) extends Shape

object Shape extends CompanionOptics[Shape] {
  implicit val schema: Schema[Shape] = Schema.derived
  val circle: Prism[Shape, Circle] = optic(_.when[Circle])
}

val setCircle = Patch.set(Shape.circle, Circle(5.0))
```

The patch replaces a circle, fails on a square in `Strict` mode, and leaves the square alone in `Lenient` mode:

```scala
setCircle(Circle(1.0), PatchMode.Strict)
// res5: Either[SchemaError, Shape] = Right(Circle(5.0))
println(setCircle(Square(2.0), PatchMode.Strict))
// Left(zio.blocks.schema.SchemaError: Expected case Circle but got Square at: .when[Circle])
setCircle(Square(2.0), PatchMode.Lenient)
// res7: Either[SchemaError, Shape] = Right(Square(2.0))
```

### Numeric Increments

`Patch.increment` adds a delta to numeric fields. This is a **commutative operation** — applying increments in any order produces the same result, making it ideal for distributed systems:

```scala

case class Counter(count: Int, total: Long, balance: BigDecimal)
object Counter extends CompanionOptics[Counter] {
  implicit val schema: Schema[Counter] = Schema.derived
  val count: Lens[Counter, Int] = optic(_.count)
  val total: Lens[Counter, Long] = optic(_.total)
  val balance: Lens[Counter, BigDecimal] = optic(_.balance)
}

// Increment various numeric types
val addOne = Patch.increment(Counter.count, 1)
val addTen = Patch.increment(Counter.count, 10)
val subtractFive = Patch.increment(Counter.count, -5)

// Works with all numeric types
val addToTotal = Patch.increment(Counter.total, 1000L)
val addToBalance = Patch.increment(Counter.balance, BigDecimal("99.99"))

// Compose multiple increments
val combined = addOne ++ addTen ++ subtractFive
val counter = Counter(0, 0L, BigDecimal(0))
```

The three increments on `count` add up to 6, and the other fields stay as they were:

```scala
combined(counter)
// res9: Counter = Counter(count = 6, total = 0L, balance = 0)
```

Supported numeric types:

- `Int`, `Long`, `Short`, `Byte`
- `Float`, `Double`
- `BigInt`, `BigDecimal`

### Temporal Operations

Patches support temporal arithmetic with `Patch.addDuration`, `Patch.addPeriod` and `Patch.addPeriodAndDuration`:

```scala

case class Event(
  timestamp: Instant,
  scheduledDate: LocalDate,
  scheduledTime: LocalDateTime,
  duration: Duration
)
object Event extends CompanionOptics[Event] {
  implicit val schema: Schema[Event] = Schema.derived
  val timestamp: Lens[Event, Instant] = optic(_.timestamp)
  val scheduledDate: Lens[Event, LocalDate] = optic(_.scheduledDate)
  val scheduledTime: Lens[Event, LocalDateTime] = optic(_.scheduledTime)
  val duration: Lens[Event, Duration] = optic(_.duration)
}

// Add duration to an Instant
val postpone1Hour = Patch.addDuration(Event.timestamp, Duration.ofHours(1))

// Add period to a LocalDate
val postpone1Week = Patch.addPeriod(Event.scheduledDate, Period.ofWeeks(1))

// Add both period and duration to a LocalDateTime
val postpone = Patch.addPeriodAndDuration(
  Event.scheduledTime,
  Period.ofDays(1),
  Duration.ofHours(2)
)

// Add duration to a Duration field
val extendDuration = Patch.addDuration(Event.duration, Duration.ofMinutes(30))

val event = Event(
  Instant.parse("2026-03-01T09:00:00Z"),
  LocalDate.of(2026, 3, 1),
  LocalDateTime.of(2026, 3, 1, 9, 0),
  Duration.ofMinutes(60)
)
```

Applying all four patches shifts every field by its own amount:

```scala
(postpone1Hour ++ postpone1Week ++ postpone ++ extendDuration)(event)
// res11: Event = Event(
//   timestamp = 2026-03-01T10:00:00Z,
//   scheduledDate = 2026-03-08,
//   scheduledTime = 2026-03-02T11:00,
//   duration = PT1H30M
// )
```

### String Edits

`Patch.editString` applies character-level edits to strings. It takes the edits as a [`Chunk`](../chunk.md) of `Patch.StringOp` values:

```scala

case class Document(content: String)
object Document extends CompanionOptics[Document] {
  implicit val schema: Schema[Document] = Schema.derived
  val content: Lens[Document, String] = optic(_.content)
}

// Insert text at position
val insertHello = Patch.editString(
  Document.content,
  Chunk(Patch.StringOp.Insert(0, "Hello "))
)

// Delete characters
val deleteFirst5 = Patch.editString(
  Document.content,
  Chunk(Patch.StringOp.Delete(0, 5))
)

// Append text
val appendBang = Patch.editString(
  Document.content,
  Chunk(Patch.StringOp.Append("!"))
)

// Replace a range (delete then insert)
val replaceWorld = Patch.editString(
  Document.content,
  Chunk(Patch.StringOp.Modify(6, 5, "Universe"))
)

// Combine multiple edits (applied in sequence)
val edits = Patch.editString(
  Document.content,
  Chunk(
    Patch.StringOp.Delete(5, 6),
    Patch.StringOp.Append(" there!")
  )
)

val doc = Document("Hello World")
```

Each patch edits the content of the document in place:

```scala
insertHello(Document("World"))
// res13: Document = Document("Hello World")
deleteFirst5(doc)
// res14: Document = Document(" World")
appendBang(doc)
// res15: Document = Document("Hello World!")
replaceWorld(doc)
// res16: Document = Document("Hello Universe")
edits(doc)
// res17: Document = Document("Hello there!")
```

String edit operations:

- `Patch.StringOp.Insert(index, text)` — Insert text at the given position
- `Patch.StringOp.Delete(index, length)` — Delete characters starting at index
- `Patch.StringOp.Append(text)` — Append text to the end
- `Patch.StringOp.Modify(index, length, text)` — Replace a range with new text

## Working with Collections

The sequence operations add, insert, delete and modify elements of `Vector`, `List`, `Seq` and `IndexedSeq` fields.

### Appending Elements

`Patch.append` adds elements to the end of a sequence:

```scala

case class TodoList(items: Vector[String])
object TodoList extends CompanionOptics[TodoList] {
  implicit val schema: Schema[TodoList] = Schema.derived
  val items: Lens[TodoList, Vector[String]] = optic(_.items)
}

val addItems = Patch.append(
  TodoList.items,
  Vector("Buy groceries", "Walk the dog")
)
```

The new items land after the existing one:

```scala
addItems(TodoList(Vector("Existing task")))
// res19: TodoList = TodoList(
//   Vector("Existing task", "Buy groceries", "Walk the dog")
// )
```

`Patch.append`, `Patch.insertAt`, `Patch.deleteAt` and `Patch.modifyAt` each have an overload for `Vector`, `List`, `Seq` and `IndexedSeq` fields. The compiler picks the overload from the field's type, so no import is needed. `LazyList` fields are not supported: `Schema.derived` cannot derive a schema for a case class with a `LazyList` field.

### Inserting at Index

`Patch.insertAt` inserts elements at a specific position:

```scala
val insertAtStart = Patch.insertAt(
  TodoList.items,
  0,
  Vector("First priority")
)

val insertInMiddle = Patch.insertAt(
  TodoList.items,
  1,
  Vector("Second item", "Third item")
)
```

Inserting at index 1 shifts `"B"` and `"C"` to the right:

```scala
insertInMiddle(TodoList(Vector("A", "B", "C")))
// res20: TodoList = TodoList(
//   Vector("A", "Second item", "Third item", "B", "C")
// )
```

### Deleting Elements

`Patch.deleteAt` removes `count` elements starting at an index:

```scala
// Delete one element at index 1
val deleteOne = Patch.deleteAt(TodoList.items, 1, 1)

// Delete three elements starting at index 0
val deleteThree = Patch.deleteAt(TodoList.items, 0, 3)

val abcd = TodoList(Vector("A", "B", "C", "D"))
```

Applying both patches to the same four-element list:

```scala
deleteOne(abcd)
// res21: TodoList = TodoList(Vector("A", "C", "D"))
deleteThree(abcd)
// res22: TodoList = TodoList(Vector("D"))
```

### Modifying Elements

`Patch.modifyAt` applies a nested patch to the element at a specific index:

```scala

case class Task(title: String, priority: Int)
object Task extends CompanionOptics[Task] {
  implicit val schema: Schema[Task] = Schema.derived
  val title: Lens[Task, String] = optic(_.title)
  val priority: Lens[Task, Int] = optic(_.priority)
}

case class Project(tasks: Vector[Task])
object Project extends CompanionOptics[Project] {
  implicit val schema: Schema[Project] = Schema.derived
  val tasks: Lens[Project, Vector[Task]] = optic(_.tasks)
}

// Create a patch for the nested Task type
val increasePriority = Patch.increment(Task.priority, 1)

// Modify the task at index 0
val modifyFirst = Patch.modifyAt(Project.tasks, 0, increasePriority)

val project = Project(Vector(
  Task("Build feature", 1),
  Task("Write tests", 2)
))
```

Only the first task's priority changes:

```scala
modifyFirst(project)
// res24: Project = Project(
//   Vector(
//     Task(title = "Build feature", priority = 2),
//     Task(title = "Write tests", priority = 2)
//   )
// )
```

## Working with Maps

The map operations add, remove and modify entries of `Map` fields by key.

### Adding Keys

`Patch.addKey` adds a new key-value pair to a map:

```scala

case class Config(settings: Map[String, Int])
object Config extends CompanionOptics[Config] {
  implicit val schema: Schema[Config] = Schema.derived
  val settings: Lens[Config, Map[String, Int]] = optic(_.settings)
}

val addTimeout = Patch.addKey(Config.settings, "timeout", 30)
val addRetries = Patch.addKey(Config.settings, "retries", 3)

val combined = addTimeout ++ addRetries
```

Both keys are added next to the existing `port` entry:

```scala
combined(Config(Map("port" -> 8080)))
// res26: Config = Config(Map("port" -> 8080, "timeout" -> 30, "retries" -> 3))
```

### Removing Keys

`Patch.removeKey` removes a key from a map:

```scala
val removePort = Patch.removeKey(Config.settings, "port")
```

Removing `port` keeps the other entry:

```scala
removePort(Config(Map("port" -> 8080, "timeout" -> 30)))
// res27: Config = Config(Map("timeout" -> 30))
```

### Modifying Values

`Patch.modifyKey` applies a nested patch to the value at a specific key. The nested patch is a `Patch[Int]` here; the simplest way to get one that adds 10 is to diff `0` against `10`, which produces a numeric delta:

```scala
case class UserSettings(preferences: Map[String, Int])
object UserSettings extends CompanionOptics[UserSettings] {
  implicit val schema: Schema[UserSettings] = Schema.derived
  val preferences: Lens[UserSettings, Map[String, Int]] = optic(_.preferences)
}

// A Patch[Int] that adds 10
val incrementValue: Patch[Int] = Schema[Int].diff(0, 10)

val increaseVolume = Patch.modifyKey(UserSettings.preferences, "volume", incrementValue)

val settings = UserSettings(Map("volume" -> 50, "brightness" -> 80))
```

The nested patch is a delta of `+10`, and only the `volume` entry changes:

```scala
println(incrementValue)
// DynamicPatch {
//   . += 10
// }
increaseVolume(settings)
// res29: UserSettings = UserSettings(Map("volume" -> 60, "brightness" -> 80))
```

## Composing Patches

Patches compose with the `Patch#++` operator. The result applies the first patch, then the second:

```scala

case class Account(name: String, balance: Int, active: Boolean)
object Account extends CompanionOptics[Account] {
  implicit val schema: Schema[Account] = Schema.derived
  val name: Lens[Account, String] = optic(_.name)
  val balance: Lens[Account, Int] = optic(_.balance)
  val active: Lens[Account, Boolean] = optic(_.active)
}

val rename = Patch.set(Account.name, "Premium Account")
val deposit = Patch.increment(Account.balance, 100)
val activate = Patch.set(Account.active, true)

// Compose all patches
val upgrade = rename ++ deposit ++ activate

val account = Account("Basic", 50, false)
```

The composed patch renames, deposits and activates in one step:

```scala
upgrade(account)
// res31: Account = Account(
//   name = "Premium Account",
//   balance = 150,
//   active = true
// )
```

Composition is associative: `(a ++ b) ++ c` equals `a ++ (b ++ c)`.

`Patch.empty` is the identity element, and `Patch#isEmpty` reports whether a patch has no operations:

```scala
val noChange = Patch.empty[Account]
```

Composing with the empty patch on either side changes nothing:

```scala
(deposit ++ noChange)(account) == deposit(account)
// res32: Boolean = true
(noChange ++ deposit)(account) == deposit(account)
// res33: Boolean = true
noChange.isEmpty
// res34: Boolean = true
```

## Applying Patches

We apply a patch with `Patch#apply`, with or without an explicit `PatchMode`, or with `Patch#applyOption`.

### Basic Application

The simplest way to apply a patch is `Patch#apply(value)`, which applies in `Lenient` mode: it skips operations whose preconditions fail and applies the rest. It returns the original value only when it cannot rebuild a value of the type from the result:

```scala

case class Account(name: String, balance: Int, active: Boolean)
object Account extends CompanionOptics[Account] {
  implicit val schema: Schema[Account] = Schema.derived
  val balance: Lens[Account, Int] = optic(_.balance)
}

val addHundred = Patch.increment(Account.balance, 100)
val testAccount = Account("Test", 50, true)
```

Applying the patch returns the updated value directly:

```scala
addHundred(testAccount)
// res36: Account = Account(name = "Test", balance = 150, active = true)
```

### Application Modes

For more control, use the overload that takes a `PatchMode`. It returns an `Either` of a [`SchemaError`](./schema-error.md) or the patched value:

```scala
addHundred(testAccount, PatchMode.Strict)
// res37: Either[SchemaError, Account] = Right(
//   Account(name = "Test", balance = 150, active = true)
// )
```

The three modes differ only in how they treat an operation whose precondition fails.

#### PatchMode.Strict

`Strict` fails on the first operation whose structural precondition fails: an index out of range, a missing map key, a map key that already exists, a case mismatch, or a string index out of range:

```scala
case class Data(items: Vector[Int])
object Data extends CompanionOptics[Data] {
  implicit val schema: Schema[Data] = Schema.derived
  val items: Lens[Data, Vector[Int]] = optic(_.items)
}

// Try to delete at an invalid index
val badDelete = Patch.deleteAt(Data.items, 10, 1)
val data = Data(Vector(1, 2, 3))
```

`Strict` reports the out-of-range delete, while `Lenient` skips it:

```scala
println(badDelete(data, PatchMode.Strict))
// Left(zio.blocks.schema.SchemaError: Delete range [10, 11) out of bounds for sequence of length 3 at: .items)
badDelete(data, PatchMode.Lenient)
// res39: Either[SchemaError, Data] = Right(Data(Vector(1, 2, 3)))
```

Use `Strict` when you need to know if every operation succeeded.

:::warning
`Strict` checks structure, not content. It does not detect that the target value changed since the patch was made: an index-based delete removes whatever sits at that index now, and a numeric delta adds to the current number. For example, a diff that removes the element `2` from `Vector(1, 2, 3)` records "delete index 1":

```scala
val removeTwo = Schema[Data].diff(Data(Vector(1, 2, 3)), Data(Vector(1, 3)))
```

If someone has meanwhile prepended `0`, index 1 now holds `1`, and `Strict` deletes it without complaint:

```scala
println(removeTwo)
// DynamicPatch {
//   .items:
//     - [1]
// }
removeTwo(Data(Vector(0, 1, 2, 3)), PatchMode.Strict)
// res41: Either[SchemaError, Data] = Right(Data(Vector(0, 2, 3)))
```
:::

#### PatchMode.Lenient

`Lenient` skips operations that fail preconditions and continues with the rest:

```scala
val patch1 = Patch.deleteAt(Data.items, 10, 1)    // out of range for 3 elements
val patch2 = Patch.append(Data.items, Vector(4))

val combined = patch1 ++ patch2
```

`Lenient` skips the invalid delete and still appends, while `Strict` stops at the delete:

```scala
combined(data, PatchMode.Lenient)
// res42: Either[SchemaError, Data] = Right(Data(Vector(1, 2, 3, 4)))
println(combined(data, PatchMode.Strict))
// Left(zio.blocks.schema.SchemaError: Delete range [10, 11) out of bounds for sequence of length 3 at: .items)
```

Use `Lenient` for best-effort patching where partial success is acceptable.

#### PatchMode.Clobber

`Clobber` behaves like `Lenient` except in three places:

- `Patch.addKey` for a key that already exists overwrites the existing value instead of failing.
- `Patch.removeKey` for a missing key is a no-op, which is the same observable result as `Lenient`.
- Out-of-range `Patch.insertAt` and `Patch.deleteAt` are clamped to the sequence instead of failing: inserting past the end appends, and deleting past the end removes only the elements that exist.

Everything else that fails a precondition, such as `Patch.modifyAt` out of range, `Patch.modifyKey` on a missing key, an out-of-range string edit, or a prism case mismatch, is skipped exactly as in `Lenient`. To compare the modes, we add a key that already exists and insert past the end of a sequence:

```scala
case class Config(settings: Map[String, Int])
object Config extends CompanionOptics[Config] {
  implicit val schema: Schema[Config] = Schema.derived
  val settings: Lens[Config, Map[String, Int]] = optic(_.settings)
}

// Adding a key that already exists
val addExisting = Patch.addKey(Config.settings, "port", 9000)
val config = Config(Map("port" -> 8080))

// Inserting past the end of a three-element sequence
val insertFar = Patch.insertAt(Data.items, 10, Vector(4))
```

`Strict` rejects the existing key, `Lenient` keeps the old value, and `Clobber` overwrites it; the out-of-range insert becomes an append under `Clobber`:

```scala
println(addExisting(config, PatchMode.Strict))
// Left(zio.blocks.schema.SchemaError: Key already exists in map at: .settings)
addExisting(config, PatchMode.Lenient)
// res45: Either[SchemaError, Config] = Right(Config(Map("port" -> 8080)))
addExisting(config, PatchMode.Clobber)
// res46: Either[SchemaError, Config] = Right(Config(Map("port" -> 9000)))
insertFar(data, PatchMode.Clobber)
// res47: Either[SchemaError, Data] = Right(Data(Vector(1, 2, 3, 4)))
```

Use `Clobber` when the patch should win over the current state of map keys and when out-of-range sequence edits should be clamped rather than skipped.

### Option-Based Application

`Patch#applyOption` applies in `Strict` mode and returns `None` if any operation fails:

```scala
addHundred.applyOption(testAccount)
// res48: Option[Account] = Some(
//   Account(name = "Test", balance = 150, active = true)
// )
combined.applyOption(data)
// res49: Option[Data] = None
```

## Diffing Values

[`Schema`](./schema.md) provides `Schema#diff(oldValue: A, newValue: A): Patch[A]`, which computes a minimal patch that transforms one value into another, and `Schema#patch(value: A, patch: Patch[A]): Either[SchemaError, A]`, which applies a patch in `Strict` mode. We diff two versions of a person whose age changed:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val oldValue = Person("Alice", 25)
val newValue = Person("Alice", 26)

val patch: Patch[Person] = Schema[Person].diff(oldValue, newValue)
```

The patch holds a single delta on `age`, and applying it to the old value yields the new one:

```scala
println(patch)
// DynamicPatch {
//   .age += 1
// }
Schema[Person].patch(oldValue, patch)
// res52: Either[SchemaError, Person] = Right(Person(name = "Alice", age = 26))
```

The same diff is available one level down, on [`DynamicValue`](./dynamic-value.md), where `DynamicValue#diff` returns a `DynamicPatch`:

```scala
val dynamicOld = Schema[Person].toDynamicValue(oldValue)
val dynamicNew = Schema[Person].toDynamicValue(newValue)
val dynamicPatch: DynamicPatch = dynamicOld.diff(dynamicNew)
```

Applying the `DynamicPatch` to the old dynamic value yields the new one:

```scala
dynamicPatch(dynamicOld, PatchMode.Strict) == Right(dynamicNew)
// res53: Boolean = true
```

The differ produces minimal patches using type-appropriate operations:

- **Numeric fields** — Uses delta operations (`+1` instead of `set 26`)
- **Strings** — Uses edit operations when more compact than replacement
- **Records** — Only includes changed fields
- **Sequences** — Uses LCS algorithm to compute minimal insert/delete operations
- **Maps** — Produces add/remove/modify operations for changed entries

A user whose scores and metadata both changed shows the sequence and map operations together:

```scala
case class User(
  name: String,
  scores: Vector[Int],
  metadata: Map[String, String]
)
object User {
  implicit val schema: Schema[User] = Schema.derived
}

val v1 = User(
  "Alice",
  Vector(10, 20, 30),
  Map("level" -> "1", "status" -> "active")
)

val v2 = User(
  "Alice",
  Vector(10, 25, 30, 40),
  Map("level" -> "2", "status" -> "active")
)
```

The `scores` edit deletes index 1, inserts `25` at index 1 and appends `40`. The changed map value is a string, and strings never get a numeric delta; for a one-character string an edit is not shorter than a replacement, so the patch sets `"level"` to `"2"`:

```scala
println(Schema[User].diff(v1, v2))
// DynamicPatch {
//   .scores:
//     - [1]
//     + [1: 25]
//     + 40
//   .metadata:
//     ~ {"level"}:
//       . = "2"
// }
```

## Serializable Operations

All patch operations are serializable through the schema system: `DynamicPatch` has an implicit `Schema`, so it converts to `DynamicValue` and to any format with a schema-based codec (JSON, Avro, MessagePack, and so on). There is no `Schema[Patch[S]]`; we store `Patch#dynamicPatch`, the untyped part of the patch:

```scala

case class Account(name: String, balance: Int, active: Boolean)
object Account extends CompanionOptics[Account] {
  implicit val schema: Schema[Account] = Schema.derived
  val name: Lens[Account, String] = optic(_.name)
  val balance: Lens[Account, Int] = optic(_.balance)
}

val patch = Patch.increment(Account.balance, 100)

// Patches can be converted to DynamicValue
val asDynamicValue: DynamicValue = Schema[DynamicPatch].toDynamicValue(patch.dynamicPatch)

// Serialize to JSON
val json: String = Schema[DynamicPatch].jsonCodec.encodeToString(patch.dynamicPatch)
```

The JSON names the path and the operation:

```scala
println(json)
// {"ops":[{"path":{"nodes":[{"Field":{"name":"balance"}}]},"operation":{"PrimitiveDelta":{"op":{"IntDelta":{"delta":100}}}}}]}
```

To apply a decoded patch, we convert the target to a `DynamicValue`, apply the `DynamicPatch`, and convert the result back:

```scala
val restored = for {
  decoded <- Schema[DynamicPatch].jsonCodec.decode(json)
  patched <- decoded(Schema[Account].toDynamicValue(Account("Test", 50, true)), PatchMode.Strict)
  account <- Schema[Account].fromDynamicValue(patched)
} yield account
// restored: Either[SchemaError, Account] = Right(
//   Account(name = "Test", balance = 150, active = true)
// )
```

This enables storing patches in databases, sending them over APIs, or logging them for audit purposes.

:::caution
Patches address fields by name. A stored patch does not adapt when the data type evolves: if a field it touches is renamed, `Strict` mode fails with `Missing field` and `Lenient` and `Clobber` skip the operation. Here a patch that sets `name` meets a later version of the type in which `name` became `owner`:

```scala
case class AccountV2(owner: String, balance: Int, active: Boolean)
object AccountV2 {
  implicit val schema: Schema[AccountV2] = Schema.derived
}

val setName = Patch.set(Account.name, "Savings")
val storedValue = Schema[AccountV2].toDynamicValue(AccountV2("Basic", 50, true))
```

Applying the stored patch to the new shape fails in `Strict` mode:

```scala
println(setName.dynamicPatch(storedValue, PatchMode.Strict))
// Left(zio.blocks.schema.SchemaError: Missing field 'name' at: .)
```
:::

## Operation Reference

The tables below list every smart constructor on the `Patch` companion object, grouped by the kind of value they target.

### Value Operations

| Operation                     | Description                   |
|-------------------------------|-------------------------------|
| `Patch.set(optic, value)`     | Set a value at the optic path |
| `Patch.replace(optic, value)` | Alias for `Patch.set`         |
| `Patch.empty[S]`              | Empty patch (identity)        |

### Numeric Operations

| Operation                       | Description                |
|---------------------------------|----------------------------|
| `Patch.increment(optic, delta)` | Add delta to numeric field |

### Temporal Operations

| Operation                                             | Description                             |
|-------------------------------------------------------|-----------------------------------------|
| `Patch.addDuration(optic, duration)`                  | Add duration to `Instant` or `Duration` |
| `Patch.addPeriod(optic, period)`                      | Add period to `LocalDate` or `Period`   |
| `Patch.addPeriodAndDuration(optic, period, duration)` | Add both to `LocalDateTime`             |

### String Operations

| Operation                                               | Description                  |
|---------------------------------------------------------|------------------------------|
| `Patch.editString(optic, edits: Chunk[Patch.StringOp])` | Apply string edit operations |
| `Patch.StringOp.Insert(index, text)`                    | Insert text at position      |
| `Patch.StringOp.Delete(index, length)`                  | Delete characters            |
| `Patch.StringOp.Append(text)`                           | Append text                  |
| `Patch.StringOp.Modify(index, length, text)`            | Replace range                |

### Sequence Operations

| Operation                                | Description                 |
|------------------------------------------|-----------------------------|
| `Patch.append(optic, elements)`          | Append elements to end      |
| `Patch.insertAt(optic, index, elements)` | Insert at index             |
| `Patch.deleteAt(optic, index, count)`    | Delete elements             |
| `Patch.modifyAt(optic, index, patch)`    | Apply nested patch at index |

### Map Operations

| Operation                            | Description                 |
|--------------------------------------|-----------------------------|
| `Patch.addKey(optic, key, value)`    | Add key-value pair          |
| `Patch.removeKey(optic, key)`        | Remove key                  |
| `Patch.modifyKey(optic, key, patch)` | Apply nested patch to value |

## Best Practices

1. **Use increment for counters** — Increment operations are commutative and merge-friendly
2. **Prefer fine-grained patches** — Instead of replacing entire records, patch individual fields
3. **Check `Patch#isEmpty` before applying** — Skip no-op patches for efficiency
4. **Use Strict mode for validation** — Catch errors early in development
5. **Use Lenient mode for resilience** — Handle partial updates gracefully in production
6. **Serialize patches for audit** — Store the patch representation, not just before/after snapshots
7. **Revisit stored patches after renaming fields** — Patches address fields by name, so a rename makes `Strict` fail and `Lenient` skip the operation

---

## Path Interpolator

:::tip
To see optics applied step by step to nested data, follow the [Navigate and Update Nested Data with Optics](../../guides/nested-data-with-optics.md) tutorial.
:::

:::note
To use the path interpolator `p"..."`, you must import the schema package: `import zio.blocks.schema._`
:::

The path interpolator `p"..."` is a compile-time string interpolator for constructing `DynamicOptic` instances in ZIO Blocks. It provides a clean, concise syntax for building optic paths that navigate through complex data structures, with all parsing and validation happening at compile time for zero runtime overhead.

**Why use the path interpolator?**

Instead of manually constructing optics like this:

```scala

DynamicOptic(Vector(
  DynamicOptic.Node.Field("users"),
  DynamicOptic.Node.Elements,
  DynamicOptic.Node.Field("email")
))
```

You can write:

```scala

p".users[*].email"
```

The interpolator is **type-safe**, **compile-time validated**, and **performance-optimized** with zero runtime parsing overhead.

## Getting Started

Import the schema package to enable the path interpolator:

```scala

// Now you can use p"..." anywhere
val path = p".users[0].name"
```

## Key Features

- **✅ Zero Runtime Overhead**: All parsing happens at compile time
- **✅ Cross-Platform**: Works on Scala 2.13.x and Scala 3.x
- **✅ Compile-Time Safety**: Invalid paths are rejected during compilation
- **✅ No Runtime Interpolation**: Prevents accidental use of runtime values
- **✅ Rich Syntax**: Supports all `DynamicOptic` operations

## Syntax Reference

This section documents the complete path syntax with examples for each component type.

### Field Access

Access fields in records using dot notation. The leading dot is optional:

```scala
// With leading dot
p".name"           // Field("name")
p".firstName"      // Field("firstName")

// Without leading dot
p"name"            // Field("name")
p"firstName"       // Field("firstName")

// Chained fields
p".user.address.street"
// Equivalent to: Field("user") → Field("address") → Field("street")
```

Field names can also include special characters and keywords:

```scala
p"._private"       // Fields starting with underscore
p".field123"       // Fields with digits
p".café"           // Unicode field names
p".true"           // Keywords as field names (true, false, null)
```

### Index Access

Access sequence elements by index, multiple indices, or ranges.

To access a single element by index:

```scala
p"[0]"             // AtIndex(0)
p"[42]"            // AtIndex(42)
p"[2147483647]"    // AtIndex(Int.MaxValue)
```

To access multiple elements at specific indices:

```scala
p"[0,1,2]"         // AtIndices(Seq(0, 1, 2))
p"[0, 2, 5]"       // AtIndices(Seq(0, 2, 5)) - spaces allowed
p"[5,2,8,1]"       // Order preserved
```

To select a range of consecutive elements:

```scala
p"[0:5]"           // AtIndices(Seq(0, 1, 2, 3, 4))
p"[5:8]"           // AtIndices(Seq(5, 6, 7))
p"[3:4]"           // AtIndices(Seq(3)) - single element
p"[5:5]"           // AtIndices(Seq.empty) - empty range
p"[10:5]"          // AtIndices(Seq.empty) - inverted range
```

### Element Selectors

Select all elements in a sequence using wildcard syntax:

```scala
p"[*]"             // Elements - all elements
p"[:*]"            // Elements - alternative syntax
```

To navigate nested sequences:

```scala
p"[*][*]"          // Nested sequences: all elements of all elements
p"[*][0]"          // First element of each sequence
```

### Map Access

Access map values by key, where keys can be strings, integers, booleans, or characters.

To use string keys:

```scala
p"""{"host"}"""              // AtMapKey(String("host"))
p"""{"foo bar"}"""           // Keys with spaces
p"""{"日本語"}"""             // Unicode keys
p"""{"🎉"}"""                 // Emoji keys
p"""{""}"""                  // Empty string key
```

To use integer keys:

```scala
p"{42}"                      // AtMapKey(Int(42))
p"{0}"                       // AtMapKey(Int(0))
p"{-42}"                     // AtMapKey(Int(-42))
p"{2147483647}"              // AtMapKey(Int.MaxValue)
p"{-2147483648}"             // AtMapKey(Int.MinValue)
```

To use boolean keys:

```scala
p"{true}"                    // AtMapKey(Boolean(true))
p"{false}"                   // AtMapKey(Boolean(false))
```

To use character keys:

```scala
p"{'a'}"                     // AtMapKey(Char('a'))
p"{' '}"                     // AtMapKey(Char(' '))
p"{'9'}"                     // AtMapKey(Char('9'))
```

To use multiple keys of the same or mixed types:

```scala
p"""{"foo", "bar", "baz"}""" // AtMapKeys(Seq(...))
p"{1, 2, 3}"                 // Multiple integer keys
p"{true, false}"             // Multiple boolean keys

// Mixed types
p"""{"foo", 42}"""           // AtMapKeys(Seq(String("foo"), Int(42)))
p"""{"s", 'c', 42, true}"""  // All supported types
```

### Map Selectors

Select all keys or all values in a map:

```scala
p"{*}"             // MapValues - all values
p"{:*}"            // MapValues - alternative syntax
p"{*:}"            // MapKeys - all keys
```

To apply map selectors to nested maps:

```scala
p"{*}{*}"          // Nested maps: all values of all values
p"{*:}{*:}"        // All keys of all keys
```

### Variant Case Access

Navigate into a specific variant case using angle brackets:

```scala
p"<Left>"          // Case("Left")
p"<Right>"         // Case("Right")
p"<Some>"          // Case("Some")
p"<None>"          // Case("None")
```

Variant case names can include special characters and keywords:

```scala
p"<_Empty>"        // Cases starting with underscore
p"<Case1>"         // Cases with digits
p"<café>"          // Unicode case names
```

To navigate nested variant cases:

```scala
p"<C>"       // Nested variants
```

### Schema Search

Search for values matching a schema pattern anywhere in a data structure using the `#` prefix.

To search for nominal types by name:

```scala
p"#Person"         // Find all values of type Person
p"#User"           // Find all values of type User
p"#Address"        // Find all values of type Address
```

To search for primitive types:

```scala
p"#string"         // Find all string values
p"#int"            // Find all integer values
p"#boolean"        // Find all boolean values
p"#uuid"           // Find all UUID values
```

To search for records with specific field structures:

```scala
p"#record { name: string }"                // Find records with a string 'name' field
p"#record { name: string, age: int }"      // Find records with both fields
p"#record { items: list(Person) }"         // Nested schema
```

To search for variants with specific case structures:

```scala
p"#variant { Left: int, Right: string }"   // Find Either-like variants
```

To search for collection types:

```scala
p"#list(string)"                           // Find lists of strings
p"#list(Person)"                           // Find lists of Person
p"#map(string, int)"                       // Find maps from string to int
p"#option(Person)"                         // Find optional Person values
```

`set(...)` and `vector(...)` work as synonyms for `list(...)` — all three parse to the same `SchemaRepr.Sequence` pattern, since the pattern only cares about the element type, not which collection it comes from.

To match any value regardless of type:

```scala
p"#_"              // Find any value (matches everything)
```

To combine path navigation with schema search:

```scala
p".users#Person"               // Search for Person in users field
p"#Person.name"                // Find all Person values, then get name
p".items[*]#Person.email"      // Elements then search then field
p"#list(Person)#Person"        // Chained searches
```

## Escape Sequences

String and character literals accept a small set of escape sequences, and the two kinds of literal accept different quote escapes:

| Escape | Result  | In `"..."` strings | In `'...'` chars |
|--------|---------|--------------------|------------------|
| `\n`   | newline | Yes                | Yes              |
| `\t`   | tab     | Yes                | Yes              |
| `\r`   | return  | Yes                | Yes              |
| `\\`   | `\`     | Yes                | Yes              |
| `\"`   | `"`     | Yes                | No               |
| `\'`   | `'`     | No                 | Yes              |

Any other escape, including `\'` inside a string and `\"` inside a char, is a parse error, so the `p"..."` interpolator rejects it at compile time.

Here are some examples of escape sequences in use:

```scala
p"""{"foo\nbar"}"""          // String key with newline
p"""{"foo\tbar"}"""          // String key with tab
p"""{'\n'}"""                // Char key with newline
p"""{"foo\"bar"}"""          // Escaped quote in string
p"""{"foo\\bar"}"""          // Escaped backslash in string
```

## Combined Paths

Combine different path elements to navigate complex nested structures.

### Field → Sequence

Access sequence elements by index or range through a field:

```scala
p".items[0]"                 // First item
p".items[*]"                 // All items
p".items[0,1,2]"             // Items at indices 0, 1, 2
p".items[0:5]"               // Items 0 through 4
```

### Field → Map

Access map values by key through a field:

```scala
p""".config{"host"}"""       // Map lookup
p".settings{42}"             // Integer key
p".lookup{*}"                // All map values
p".lookup{*:}"               // All map keys
```

### Field → Variant

Navigate into variant cases through a field:

```scala
p".result<Success>"          // Variant case
p".response<Ok>"             // HTTP response variant
```

### Nested Structures

Combine different path elements to build complex navigation through nested data:

```scala
// Record in sequence
p".users[0].name"
// Equivalent to: Field("users") → AtIndex(0) → Field("name")

// All elements then field
p".users[*].email"
// Equivalent to: Field("users") → Elements → Field("email")

// Map values then field
p".lookup{*}.value"
// Equivalent to: Field("lookup") → MapValues → Field("value")

// Variant then field
p".response<Ok>.body"
// Equivalent to: Field("response") → Case("Ok") → Field("body")
```

### Deeply Nested Paths

Chain multiple path operators for deeply nested data structures:

```scala
// Complex nested navigation
p""".root.children[*].metadata{"tags"}[0]"""
// Field("root") → Field("children") → Elements → 
// Field("metadata") → AtMapKey("tags") → AtIndex(0)

// All node types in one path
p""".a[0]{"k"}<V>.b[*]{*}.c{*:}"""
// Field("a") → AtIndex(0) → AtMapKey("k") → Case("V") →
// Field("b") → Elements → MapValues → Field("c") → MapKeys
```

## Root and Empty Paths

An empty path refers to the root of the data structure:

```scala
p""                          // Empty path = root
```

## Compile-Time Safety

The path interpolator **rejects runtime interpolation** to prevent unsafe dynamic path construction.

### Examples of Safety Checks

Runtime interpolation will fail to compile:

```scala

val fieldName = "email"
val path = p".$fieldName"
// Error: Path interpolator does not support runtime arguments.
// error:
// Path interpolator does not support runtime arguments. Use only literal strings like p".field[0]"
// val path = p".$fieldName"
//            ^^^^^^^^^^^^^
```

Array indices also cannot be interpolated at runtime:

```scala

val idx = 5
val path = p"[$idx]"
// Error: Path interpolator does not support runtime arguments.
// error:
// Path interpolator does not support runtime arguments. Use only literal strings like p".field[0]"
// val path = p"[$idx]"
//            ^^^^^^^^^
```

Instead, use only literal strings at compile time:

```scala
val path = p".users[0].email"  // ✓ Works
```

### Parse Error Examples

Invalid syntax is caught at compile time. Here are examples of common errors:

```scala

// Unterminated string
p"""{"foo"""
// Error: Unterminated string literal

// Invalid escape sequence  
p"""{"foo\x"}"""
// Error: Invalid escape sequence

// Unexpected character
p".field@"
// Error: Unexpected character '@'

// Invalid identifier
p"."
// Error: Invalid identifier
// error: 
// Unterminated string literal starting at position 1
// error: 
// Invalid escape sequence '\x' at position 6
// error: 
// Unexpected character '@' at position 6. Expected field, index, map, or variant accessor
// error: 
// Invalid identifier at position 1
// error: 
// Conflicting definitions:
// val path: zio.blocks.schema.DynamicOptic in object MdocApp at line 14 and
// val path: zio.blocks.schema.DynamicOptic in object MdocApp at line 114
//
```

## Performance

**Zero Runtime Overhead**

All path parsing and validation occurs at **compile time**. The interpolator generates the exact same bytecode as manual `DynamicOptic` construction. Here's the comparison:

```scala
// These produce identical bytecode:
p".users[*].email"

DynamicOptic(Vector(
  DynamicOptic.Node.Field("users"),
  DynamicOptic.Node.Elements,
  DynamicOptic.Node.Field("email")
))
```

There is **no runtime parsing**, **no reflection**, and **no performance penalty**.

## Examples

This section shows practical examples of using the path interpolator with realistic data structures.

### Accessing Nested Fields

To access deeply nested fields in a data structure:

```scala

case class Address(street: String, city: String, zipCode: String)
case class Person(name: String, age: Int, address: Address)

// Access nested street field
val streetPath = p".address.street"

// Use with DynamicValue - example (requires actual DynamicValue instance)
val person: DynamicValue = ???
val street = person.get(streetPath)
```

### Working with Collections

To work with sequences and access elements by index or range:

```scala

case class User(id: Int, email: String, tags: Seq[String])
case class Company(name: String, users: Seq[User])

// Get all user emails
val emailsPath = p".users[*].email"

// Get first user's first tag
val firstTagPath = p".users[0].tags[0]"

// Get specific users by index
val specificUsersPath = p".users[0,2,5]"
```

### Map Lookups

To work with map data structures and access values by key:

```scala

case class Config(
  settings: Map[String, String],
  ports: Map[Int, String]
)

// Lookup by string key
val hostPath = p"""settings{"host"}"""

// Lookup by integer key
val httpPortPath = p"ports{80}"

// Get all config values
val allValuesPath = p"settings{*}"

// Get all port numbers (keys)
val allPortsPath = p"ports{*:}"
```

### Variant Case Handling

To navigate variant cases in your data structures:

```scala

sealed trait Result[+A]
case class Success[A](value: A) extends Result[A]
case class Failure(error: String) extends Result[Nothing]

case class Response(result: Result[User])
```

Use paths to navigate into specific cases:

```scala
// Navigate into Success case
val successValuePath = p".result<Success>.value"

// Navigate into Failure case
val errorPath = p".result<Failure>.error"
```

### Real-World Example: API Response

To extract specific data from complex nested API response structures:

```scala

case class Metadata(tags: Seq[String], version: Int)
case class Item(id: String, data: String, metadata: Metadata)
case class ApiResponse(
  status: String,
  items: Seq[Item],
  config: Map[String, String]
)

// Get the version from the first item's metadata
val versionPath = p".items[0].metadata.version"

// Get all item IDs
val allIdsPath = p".items[*].id"

// Get the first tag of each item
val firstTagsPath = p".items[*].metadata.tags[0]"

// Lookup config value
val apiKeyPath = p"""config{"api_key"}"""
```

## Before & After Comparison

This comparison shows how the path interpolator simplifies optic path construction compared to manual approaches.

### Manual Construction (Before)

Manual path construction requires verbose imports and careful construction:

```scala

// Simple path - verbose and error-prone
val path1 = DynamicOptic(Vector(
  Node.Field("users"),
  Node.AtIndex(0),
  Node.Field("email")
))

// Complex path - extremely verbose
val path2 = DynamicOptic(Vector(
  Node.Field("root"),
  Node.Field("children"),
  Node.Elements,
  Node.Field("metadata"),
  Node.AtMapKey(DynamicValue.Primitive(PrimitiveValue.String("tags"))),
  Node.AtIndex(0)
))

// Map with multiple keys
val path3 = DynamicOptic(Vector(
  Node.Field("data"),
  Node.AtMapKeys(Seq(
    DynamicValue.Primitive(PrimitiveValue.String("foo")),
    DynamicValue.Primitive(PrimitiveValue.String("bar")),
    DynamicValue.Primitive(PrimitiveValue.Int(42))
  ))
))
```

### Path Interpolator (After)

```scala

// Simple path - clean and readable
val path1 = p".users[0].email"

// Complex path - still clean and readable
val path2 = p""".root.children[*].metadata{"tags"}[0]"""

// Map with multiple keys - concise
val path3 = p"""data{"foo", "bar", 42}"""
```

The path interpolator provides significant benefits:

- **90% less code** for typical paths
- **Easier to read** and understand intent
- **Easier to write** and maintain
- **Compile-time validated** - catches errors immediately
- **No performance difference** - identical bytecode

## Practical Usage Patterns

These patterns demonstrate effective ways to use the path interpolator in real-world scenarios.

### Building Paths Dynamically (at Compile Time)

While runtime variables are not supported, you can compose literal paths at compile time:

```scala

// You can't use runtime variables, but you can compose literal paths:
val basePath = p".data.items"
val emailPath = basePath(p"[*].email")
// Same as: p".data.items[*].email"
```

### Working with DynamicValue

`DynamicValue#get` takes a path and returns a `DynamicValueSelection`, which can hold any number of values because a path with `[*]` or a search can match several. `DynamicValueSelection#one` extracts the value when the path matched exactly one, and `DynamicValue#set` replaces the value at a path:

```scala
val directory: DynamicValue = DynamicValue.Record(
  "users" -> DynamicValue.Sequence(
    DynamicValue.Record("email" -> DynamicValue.string("alice@example.com"), "age" -> DynamicValue.int(30))
  )
)
// directory: DynamicValue = Record(
//   IndexedSeq(
//     (
//       "users",
//       Sequence(
//         IndexedSeq(
//           Record(
//             IndexedSeq(
//               ("email", Primitive(String("alice@example.com"))),
//               ("age", Primitive(Int(30)))
//             )
//           )
//         )
//       )
//     )
//   )
// )

val firstEmail = directory.get(p".users[0].email").one
// firstEmail: Either[SchemaError, DynamicValue] = Right(
//   Primitive(String("alice@example.com"))
// )

val olderDirectory = directory.set(p".users[0].age", DynamicValue.int(31))
// olderDirectory: DynamicValue = Record(
//   IndexedSeq(
//     (
//       "users",
//       Sequence(
//         IndexedSeq(
//           Record(
//             IndexedSeq(
//               ("email", Primitive(String("alice@example.com"))),
//               ("age", Primitive(Int(31)))
//             )
//           )
//         )
//       )
//     )
//   )
// )
```

### Integration with Schema Optics

To use path interpolators with schema-based optics:

```scala

case class UserRecord(name: String, email: String)
object UserRecord extends CompanionOptics[UserRecord] {
  implicit val schema: Schema[UserRecord] = Schema.derived
  
  // Use path interpolator for complex lenses
  val email = $(_.email)
}

// DynamicOptic can be used for runtime path resolution
val dynamicPath = p".email"
```

## Tips and Best Practices

1. **Use the leading dot for clarity**: While optional, `p".field"` is more explicit than `p"field"`:

   ```scala mdoc:compile-only
   val userPath = p".users[0]"
   val emailPath = userPath(p".email")
   ```

4. **Use raw strings for map keys**: Triple-quoted strings avoid escape hell:

   ```scala mdoc:compile-only
   p"""config{"api.key"}"""  // p"config{\"api.key\"}" does not compile: the macro does not process escapes
   ```

5. **Document complex paths**: Add comments explaining what nested paths navigate:

   ```scala mdoc:compile-only
   // Get the first tag from each user's metadata
   val tagsPath = p".users[*].metadata.tags[0]"
   ```

## Limitations

- **No runtime interpolation**: You cannot use variables in paths (this is by design for safety)
- **No arithmetic in ranges**: Ranges must be literal integers (e.g., `[0:5]` not `[0:n]`)
- **No string interpolation**: Only literal strings work with the interpolator
- **Map keys limited to primitives**: Only String, Int, Char, and Boolean keys are supported

These limitations ensure compile-time safety and zero runtime overhead.

## Debug-Friendly toString

`DynamicOptic` instances have a custom `toString` that produces output matching the `p"..."` interpolator syntax. This makes debugging easier because you can copy the output directly into your code, except for wrapper steps (`.~`), the empty path (`.`) and `Char` keys with an escape, which the interpolator rejects, a known bug in 0.0.56:

```scala
val optic = DynamicOptic.root.field("users").elements.field("email")
// optic: DynamicOptic = DynamicOptic(
//   IndexedSeq(Field("users"), Elements, Field("email"))
// )
println(optic)
// .users[*].email

// The output can be copy-pasted into p"..."
val same = p".users[*].email"
// same: DynamicOptic = DynamicOptic(
//   Vector(Field("users"), Elements, Field("email"))
// )
```

**Examples:**

| DynamicOptic Construction                            | Interpolator Syntax |
|------------------------------------------------------|---------------------|
| `DynamicOptic.root.field("name")`                    | `p".name"`           |
| `DynamicOptic.root.field("address").field("street")` | `p".address.street"` |
| `DynamicOptic.root.caseOf("Some")`                    | `p"<Some>"`          |
| `DynamicOptic.root.at(0)`                             | `p"[0]"`             |
| `DynamicOptic.root.atIndices(0, 2, 5)`                 | `p"[0,2,5]"`         |
| `DynamicOptic.elements`                                | `p"[*]"`             |
| `DynamicOptic.root.atKey("host")`                      | `p"""{"host"}"""`    |
| `DynamicOptic.root.atKey(80)`                          | `p"{80}"`            |
| `DynamicOptic.mapValues`                               | `p"{*}"`             |
| `DynamicOptic.mapKeys`                                 | `p"{*:}"`            |
| `DynamicOptic.wrapped`                                 | none (`toString` prints `.~`, which `p"..."` rejects) |
| `DynamicOptic.root.searchSchema(SchemaRepr.Nominal("Person"))` | `p"#Person"` |
| `DynamicOptic.root.searchSchema(SchemaRepr.Primitive("string"))` | `p"#string"` |

## Summary

The `p"..."` path interpolator provides:

- **Concise syntax** for building optic paths
- **Compile-time parsing** with zero runtime overhead  
- **Type-safe navigation** through complex data structures
- **Cross-platform support** for Scala 2 and Scala 3
- **Rich feature set** covering all DynamicOptic operations

Use it whenever you need to construct `DynamicOptic` paths for navigating dynamic data structures in ZIO Blocks.

---

## ReflectTransformer

`ReflectTransformer[F, G]` rewrites a `Reflect[F, A]` tree into a `Reflect[G, A]` tree, changing the binding-metadata type constructor from `F` to `G` at every node while leaving the shape — field names, type IDs, docs, modifiers, defaults, examples — untouched. `Reflect#transform` walks the tree and recurses into fields, cases, elements, and keys/values for you; a `ReflectTransformer` only has to say how to rebuild the metadata-bearing leaf of each node kind once its children are already transformed.

Three things in the codebase are built on it: stripping bindings to serialize a [`Schema`](./schema.md) as a [`DynamicSchema`](./dynamic-schema.md), attaching bindings back with `RebindTransformer`, and [type class derivation](./type-class-derivation.md), which pairs each node's binding with a derived instance. This page covers all three, plus `OnlyMetadata`, the base class most transformers use instead of implementing the full interface.

## Design & Structure

`ReflectTransformer` declares one method per [`Reflect`](./reflect.md) node kind — seven of the eight; `Deferred` is handled separately (see [Cycle Safety](#cycle-safety-and-the-transform-cache)) because it has no metadata of its own to transform. `transformRecord` is representative of all seven — each takes the node's already-transformed children plus its untouched shape fields, and returns a `Lazy` of the rebuilt node:

```scala
trait ReflectTransformer[-F[_, _], G[_, _]] {
  def transformRecord[A](
    path: DynamicOptic,
    fields: IndexedSeq[Term[G, A, ?]],
    typeId: TypeId[A],
    metadata: F[BindingType.Record, A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    storedDefaultValue: Option[DynamicValue],
    storedExamples: collection.immutable.Seq[DynamicValue]
  ): Lazy[Reflect.Record[G, A]]

  // ...one more method per remaining node kind, same shape
}
```

The other six follow the identical pattern — a `path`, the already-transformed children, the shape fields, and the still-untransformed `metadata: F[...]` — differing only in which children they carry and which `Reflect` node they return:

| Node kind | Method | Transformed children it receives |
| ---------- | -------- | ----------------------------------- |
| `Record`     | `transformRecord`    | `fields: IndexedSeq[Term[G, A, ?]]` |
| `Variant`    | `transformVariant`   | `cases: IndexedSeq[Term[G, A, ? <: A]]` |
| `Sequence`   | `transformSequence`  | `element: Reflect[G, A]` |
| `Map`        | `transformMap`       | `key: Reflect[G, Key]`, `value: Reflect[G, Value]` |
| `Dynamic`    | `transformDynamic`   | none — `DynamicValue` has no children |
| `Primitive`  | `transformPrimitive` | none |
| `Wrapper`    | `transformWrapper`   | `wrapped: Reflect[G, B]` |

Every result is wrapped in [`Lazy`](./lazy.md) because a transformer built for a recursive schema must be able to build a node before its own children are fully forced.

## `OnlyMetadata` — The Common Case

Most transformers only need to change the metadata and leave every field, case, and element exactly as it already was. `ReflectTransformer.OnlyMetadata[F, G]` implements all seven `transformX` methods for you in terms of one method:

```scala
abstract class OnlyMetadata[F[_, _], G[_, _]] extends ReflectTransformer[F, G] {
  def transformMetadata[K, A](f: F[K, A]): Lazy[G[K, A]]
}
```

Each `transformX` implementation `OnlyMetadata` provides is one line: transform the metadata, then rebuild the same node with the same children and shape, swapping in the new metadata. `HasBinding[F]` — the type class that extracts a `Binding[T, A]` out of an arbitrary `F[T, A]` — is itself a `ReflectTransformer.OnlyMetadata[F, Binding]` whose `transformMetadata` calls the one method a `HasBinding[F]` instance supplies.

## Stripping Bindings: `ReflectTransformer.noBinding`

`ReflectTransformer.noBinding[F]()` is a predefined, stateless transformer that discards whatever metadata `F` holds and replaces it with the singleton `NoBinding` value, regardless of what `F` was. It is what `Reflect#noBinding` and, through it, `Schema#toDynamicSchema` use to turn a bound, operational schema into a pure-data one safe to serialize:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Reflect.Bound[Person] -> Reflect.Unbound[Person], via ReflectTransformer.noBinding()
val unbound: Reflect.Unbound[Person] = Person.schema.reflect.noBinding

// The public entry point most code uses instead
val dynamicSchema: DynamicSchema = Person.schema.toDynamicSchema
```

Because `ReflectTransformer.noBinding` never inspects the metadata it's replacing, one singleton instance (cast per call) serves every `F`, with no allocation per use.

## Attaching Bindings: `RebindTransformer`

`RebindTransformer` is the transformer in the opposite direction — `NoBinding` to `Binding` — that [`DynamicSchema#rebind`](./dynamic-schema.md) constructs internally to turn a deserialized, pure-data schema back into an operational one. It is `private[schema]`; the public surface is `DynamicSchema#rebind`, which builds one for you from a [`BindingResolver`](./binding-resolver.md):

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val dynamicSchema: DynamicSchema = Person.schema.toDynamicSchema

val resolver = BindingResolver.defaults.bind(Binding.of[Person])
val rebound: Schema[Person] = dynamicSchema.rebind[Person](resolver)
```

Unlike `ReflectTransformer.noBinding`, `RebindTransformer` implements every `transformX` method individually rather than extending `OnlyMetadata` — each one must call a different resolver lookup (`resolveRecord`, `resolveVariant`, `resolveSeqFor`, `resolveMapFor`, `resolveDynamic`, `resolvePrimitive`, `resolveWrapper`) and fail in a way that names which kind of binding was missing.

### `RebindException`

A resolver that's missing a binding for some type in the tree makes `DynamicSchema#rebind` throw `RebindException` rather than return quietly, since a `Schema` with an unresolved node isn't safe to hand back:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val dynamicSchema = Person.schema.toDynamicSchema

try dynamicSchema.rebind[Person](BindingResolver.empty)
catch {
  case e: RebindException =>
    // path: the DynamicOptic where the lookup failed
    // typeId: the missing type
    // expectedKind: "Record", "Variant", "Sequence", "Map", "Dynamic", "Primitive", or "Wrapper"
    println(s"${e.expectedKind} binding missing for ${e.typeId.fullName} at ${e.path}")
}
```

## Writing a Full `ReflectTransformer`: Type Class Derivation

`OnlyMetadata` covers every transformer described so far, but the interface exists in full for a reason: automatic type class derivation implements every `transformX` method directly, because deriving an instance for a `Record` needs the already-derived instances of its *fields*, not just a metadata swap. `DerivationBuilder.derive` walks a bound schema with a custom `ReflectTransformer[Binding, BindingInstance[TC, ?, ?]]` that, at each node, either substitutes a user-supplied override or invokes a [`Deriver[TC]`](./type-class-derivation.md) with the node's already-transformed children, then pairs the result with the original binding. This is the pattern to reach for when a transformation genuinely depends on more than the metadata at a single node.

## Cycle Safety and the Transform Cache

A recursive schema — one whose `Reflect` tree refers back to itself — reaches that self-reference through a `Reflect.Deferred` node. `Deferred#transform` doesn't call any `transformX` method; it looks up the pair `(this Deferred, this transformer)` in a per-thread memoization map first, and only builds a new deferred node on a miss. This is what makes `Reflect#transform` terminate on a recursive schema, and what lets a `Deferred` reached through multiple paths share one transformed result instead of being rebuilt repeatedly.

The two public entry points — `Reflect#noBinding` and `DynamicSchema#rebind` — wrap their call in an internal scope that clears this cache once the outermost `Reflect#transform` finishes, so a transformer's captured state (override maps, resolvers) doesn't stay pinned on the thread once its call returns. Reaching either entry point is enough to get this for free; it isn't something you configure.

## See Also

- [Reflect](./reflect.md) — the tree `ReflectTransformer` rewrites, and its eight node kinds
- [Binding](./binding.md) — what `F`/`G` are in practice: `Binding` and `NoBinding`
- [BindingResolver](./binding-resolver.md) — the lookup `RebindTransformer` delegates to
- [DynamicSchema](./dynamic-schema.md) — `Schema#toDynamicSchema` and `DynamicSchema#rebind`, the public entry points
- [Type Class Derivation](./type-class-derivation.md) — the fullest real use of the interface

---

## Reflect

The `Reflect` data type is the foundational data structure underlying ZIO Blocks. While `Schema[A]` is the user-facing API that wraps a `Reflect`, the `Reflect` type itself contains the actual reflective metadata that describes the structure of Scala data types at runtime:

```scala
// Simplified definition of Schema and Reflect
final case class Schema[A](reflect: Reflect.Bound[A])

sealed trait Reflect[F[_, _], A]
object Reflect {
  case class Record   [F[_, _], A](???)             extends Reflect[F, A]
  case class Variant  [F[_, _], A](???)             extends Reflect[F, A]
  case class Sequence [F[_, _], A, C[_]](???)       extends Reflect[F, C[A]]
  case class Map      [F[_, _], K, V, M[_, _]](???) extends Reflect[F, M[K, V]]
  case class Dynamic  [F[_, _]](???)                extends Reflect[F, DynamicValue]
  case class Primitive[F[_, _], A](???)             extends Reflect[F, A]
  case class Wrapper  [F[_, _], A, B](???)          extends Reflect[F, A]
  case class Deferred [F[_, _], A](???)             extends Reflect[F, A]
}
```

The `Reflect` type has two type parameters:

1. **`F[_, _]`** The binding type constructor. When `F = Binding`, the Reflect is "bound" and contains runtime functions for construction/deconstruction. When `F = NoBinding`, the Reflect is "unbound" and contains only pure structural data. 
2. **`A`** The Scala type that this Reflect describes.                                                                         

## Bound and Unbound Reflects

Each of the eight case class nodes of `Reflect` corresponds to a different category of Scala types (records, variants, sequences, maps, primitives, ... types). They may contain a field of type `F[BindingType.X, A]` that holds the runtime binding information for that type. For example the `Record` variant contains a `F[BindingType.Record, A]` instance that holds the `Constructor[A]` and `Deconstructor[A]` functions:

```scala
case class Record[F[_, _], A](
  fields: IndexedSeq[Term[F, A, ?]],
  typeId: TypeId[A],
  recordBinding: F[BindingType.Record, A],  // Binding info
  doc: Doc = Doc.empty,
  modifiers: Seq[Modifier.Reflect] = Nil
) extends Reflect[F, A]
```

We have discussed the binding system in detail on [the Binding System page](./binding.md).

This dual-nature design enables schema serialization:

- **Bound Reflect (`Reflect.Bound[A]`)**: Contains runtime bindings (constructors, deconstructors, functions) that allow actual construction and deconstruction of values. This is what `Schema` wraps.
- **Unbound Reflect**: Contains only pure data with no functions or closures. Can be serialized and deserialized identically, enabling transmission of schemas across the wire.

```scala
type Reflect.Bound[A]   = Reflect[Binding, A]
type Reflect.Unbound[A] = Reflect[NoBinding, A]

// Schemas are always bound
final case class Schema[A](reflect: Reflect.Bound[A])
```

## Reflect Nodes

`Reflect` is a sealed trait with eight nodes, each representing a different category of Scala types.

### 1. Record

`Reflect.Record` represents case classes and other product types:

```scala
case class Record[F[_, _], A](
  fields: IndexedSeq[Term[F, A, ?]],
  typeId: TypeId[A],
  recordBinding: F[BindingType.Record, A],
  doc: Doc = Doc.empty,
  modifiers: Seq[Modifier.Reflect] = Nil
) extends Reflect[F, A]
```

**Key Components:**

- **fields**: An indexed sequence of `Term` objects, each describing a field with its name, type, and nested `Reflect`.
- **typeId**: The type ID of `A`, with its name and owner.
- **recordBinding**: Contains the `Constructor[A]` and `Deconstructor[A]` for building and tearing apart values.
- **doc**: Optional documentation.
- **modifiers**: Metadata modifiers for customization.

The following example shows a `Person` case class represented as a `Reflect.Record`:

```scala

case class Person(
  name: String,
  email: String,
  age: Int,
  height: Double,
  weight: Double
)

object Person {
  implicit val schema: Schema[Person] =
    Schema {
      Reflect.Record[Binding, Person](
        fields = Vector(
          Term("name", Schema.string.reflect),
          Term("email", Schema.string.reflect),
          Term("age", Schema.int.reflect),
          Term("height", Schema.double.reflect),
          Term("weight", Schema.double.reflect)
        ),
        typeId = TypeId.of[Person],
        recordBinding = Binding.Record[Person](
          constructor = new Constructor[Person] {
            override def usedRegisters: RegisterOffset =
              RegisterOffset(objects = 2, ints = 1, doubles = 2)
            override def construct(in: Registers, offset: RegisterOffset): Person =
              Person(
                in.getObject(offset).asInstanceOf[String],
                in.getObject(offset + RegisterOffset(objects = 1)).asInstanceOf[String],
                in.getInt(offset + RegisterOffset.Zero),
                in.getDouble(offset + RegisterOffset(ints = 1)),
                in.getDouble(offset + RegisterOffset(ints = 1, doubles = 1))
              )
          },
          deconstructor = new Deconstructor[Person] {
            override def usedRegisters: RegisterOffset =
              RegisterOffset(objects = 2, ints = 1, doubles = 2)
            override def deconstruct(out: Registers, offset: RegisterOffset, in: Person): Unit = {
              out.setObject(offset, in.name)
              out.setObject(offset + RegisterOffset(objects = 1), in.email)
              out.setInt(offset, in.age)
              out.setDouble(offset + RegisterOffset(ints = 1), in.height)
              out.setDouble(offset + RegisterOffset(ints = 1, doubles = 1), in.weight)
            }
          }
        )
      )
    }
}
```

### 2. Variant

`Reflect.Variant` represents sealed traits, enums, and other sum types—data structures that can be one of several cases:

```scala
case class Variant[F[_, _], A](
  cases: IndexedSeq[Term[F, A, ? <: A]],
  typeId: TypeId[A],
  variantBinding: F[BindingType.Variant, A],
  doc: Doc = Doc.empty,
  modifiers: Seq[Modifier.Reflect] = Nil
) extends Reflect[F, A]
```

**Key Components:**

- **cases**: An indexed sequence of `Term` objects, each describing one of the possible cases with its name, type, and nested `Reflect`.
- **typeId**: The type ID of `A`, with its name and owner.
- **variantBinding**: Contains the binding information for the variant, such as a discriminator and matcher functions.

The following example shows a `Shape` sealed trait represented as a `Reflect.Variant`. We assume the schema for the `Circle` and `Rectangle` case classes are defined elsewhere:

```scala
sealed trait Shape extends Product with Serializable

object Shape {
  case class Circle(radius: Double) extends Shape
  object Circle {
    implicit val schema: Schema[Circle] = ???
  }

  case class Rectangle(width: Double, height: Double) extends Shape
  object Rectangle {
    implicit val schema: Schema[Rectangle] = ???
  }

  implicit val schema: Schema[Shape] =
    new Schema[Shape](
      Reflect.Variant[Binding, Shape](
        cases = IndexedSeq(
          Term(
            name = "Circle",
            value = Circle.schema.reflect
          ),
          Term(
            name = "Rectangle",
            value = Rectangle.schema.reflect
          )
        ),
        typeId = TypeId.of[Shape],
        variantBinding = Binding.Variant[Shape](
          discriminator = new Discriminator[Shape] {
            override def discriminate(a: Shape): Int = a match {
              case Circle(_)       => 0
              case Rectangle(_, _) => 1
            }
          },
          matchers = Matchers(
            new Matcher[Circle] {
              override def downcastOrNull(any: Any): Circle =
                any match {
                  case c: Circle => c
                  case _         => null.asInstanceOf[Circle]
                }
            },
            new Matcher[Rectangle] {
              override def downcastOrNull(any: Any): Rectangle =
                any match {
                  case r: Rectangle => r
                  case _            => null.asInstanceOf[Rectangle]
                }
            }
          )
        ),
        modifiers = Seq(Modifier.config("protobuf.field-id", "1"))
      ).doc("A geometric shape")
    )
}
```

### 3. Sequence

`Reflect.Sequence` represents sequential collections like `List`, `Vector`, `Array`, `Set`, and other `Iterable` types:

```scala
case class Sequence[F[_, _], A, C[_]](
  element: Reflect[F, A],
  typeId: TypeId[C[A]],
  seqBinding: F[BindingType.Seq[C], C[A]],
  doc: Doc = Doc.empty,
  modifiers: scala.Seq[Modifier.Reflect] = Nil
) extends Reflect[F, C[A]]
```

**Key components**:
- **element**: The `Reflect` describing the element type.
- **seqBinding**: Contains the corresponding sequence binding

### 4. Map

`Reflect.Map` represents key-value collections like `Map` etc:

```scala
case class Map[F[_, _], K, V, M[_, _]](
  key: Reflect[F, K],
  value: Reflect[F, V],
  typeId: TypeId[M[K, V]],
  mapBinding: F[BindingType.Map[M], M[K, V]],
  doc: Doc = Doc.empty,
  modifiers: Seq[Modifier.Reflect] = Nil
) extends Reflect[F, M[K, V]]
```

**Key Components:**

- **key**: The `Reflect` describing the key type.
- **value**: The `Reflect` describing the value type.
- **mapBinding**: Contains the corresponding map binding.

### 5. Dynamic

`Reflect.Dynamic` represents values whose types are not known at compile time. This is essential for handling JSON payloads, schemaless data, or any scenario where the structure is determined at runtime.

```scala
case class Dynamic[F[_, _]](
  dynamicBinding: F[BindingType.Dynamic, DynamicValue],
  typeId: TypeId[DynamicValue] = TypeId.of[DynamicValue],
  doc: Doc = Doc.empty,
  modifiers: Seq[Modifier.Reflect] = Nil
) extends Reflect[F, DynamicValue]
```

### 6. Primitive

`Reflect.Primitive` represents primitive and scalar types. This includes numeric types (`Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, `BigDecimal`), text types (`String`, `Char`), and `Boolean`. It also covers the full range of `java.time` temporal types: instants and dates (`Instant`, `LocalDate`, `LocalDateTime`, `LocalTime`, `OffsetDateTime`, `OffsetTime`, `ZonedDateTime`), durations and periods (`Duration`, `Period`), calendar components (`Year`, `YearMonth`, `Month`, `MonthDay`, `DayOfWeek`), and time zones (`ZoneId`, `ZoneOffset`). Additionally, it supports `Currency`, `UUID`, and `Unit`, and can be extended to support custom primitive types as needed.

```scala
case class Primitive[F[_, _], A](
  primitiveType: PrimitiveType[A],
  typeId: TypeId[A],
  primitiveBinding: F[BindingType.Primitive, A],
  doc: Doc = Doc.empty,
  modifiers: Seq[Modifier.Reflect] = Nil
) extends Reflect[F, A]
```

You can access all built-in primitive types inside companion object of `Reflect` data type, e.g. `Reflect.int`, `Reflect.string`, `Reflect.instant`, etc.

### 7. Wrapper

Modern Scala development often involves creating domain-specific types that add semantic meaning or validation to primitive types:

```scala
// Opaque type in Scala 3
opaque type Age = Int

// Newtype pattern
case class Email(value: String)

// ZIO Prelude newtypes
object UserId extends Newtype[String]
type UserId = UserId.Type
```

Each of these patterns shares a common characteristic: they wrap an underlying type (`Int`, `String`) to create a new type with distinct semantics.

`Reflect.Wrapper` is a specialized node type that models the relationship between a wrapper type and its underlying representation. It provides a unified abstraction for opaque types, newtypes, wrapper classes, and similar patterns where one type wraps another with optional validation logic:

```
case class Wrapper[F[_, _], A, B](
  wrapped: Reflect[F, B],
  typeId: TypeId[A],
  wrapperBinding: F[BindingType.Wrapper[A, B], A],
  doc: Doc = Doc.empty,
  modifiers: Seq[Modifier.Reflect] = Nil
) extends Reflect[F, A]
```

The `Wrapper` has three type parameters:

- **`F[_, _]`**: The binding type constructor (typically `Binding` for bound schemas)
- **`A`**: The wrapper type (e.g., `Age`, `Email`)
- **`B`**: The underlying/wrapped type (e.g., `Int`, `String`)

So we can say a wrapper of type `Wrapper[F[_, _], A, B]` wraps a type `B` (described by `wrapped: Reflect[F, B]`) to create a new type `A`. 

Assume we have a positive integer type `PosInt` that wraps an `Int` but enforces a validation rule that the value must be non-negative. We can define its schema using `Reflect.Wrapper` as follows:

```scala

case class PosInt private (value: Int) extends AnyVal

object PosInt {
  def apply(value: Int): Either[String, PosInt] =
    if (value >= 0) Right(new PosInt(value))
    else Left("Expected positive value")

  implicit val schema: Schema[PosInt] = Schema(
    Reflect.Wrapper(
      wrapped = Schema[Int].reflect,
      typeId = TypeId.of[PosInt],
      wrapperBinding = Binding.Wrapper[PosInt, Int](
        wrap = v => PosInt(v), 
        unwrap = _.value
      )
    )
  )
}
```

To create schemas for wrapper types, use `transform`:

```scala

case class PosInt private (value: Int) extends AnyVal

object PosInt {
  def unsafeApply(value: Int): PosInt =
    if (value >= 0) new PosInt(value)
    else throw SchemaError.validationFailed("Expected positive value")

  implicit val schema: Schema[PosInt] = 
    Schema[Int].transform(PosInt.unsafeApply, _.value)
}
```

### 8. Deferred

`Reflect.Deferred` introduces laziness to handle recursive and mutually recursive data types. Without deferred evaluation, recursive types would cause infinite loops:

```scala
case class Deferred[F[_, _], A](
  _value: () => Reflect[F, A],
  _typeId: Option[TypeId[A]] = None
) extends Reflect[F, A]
```

For example, if we have a recursive data type like below:

```scala
case class Tree(value: Int, children: List[Tree])
```

We can define its schema using `Reflect.Deferred` as follows:

```scala

// Recursive data type
case class Tree(value: Int, children: List[Tree])

object Tree {
  implicit val schema: Schema[Tree] = {
    lazy val treeReflect: Reflect.Bound[Tree] = Reflect.Record[Binding, Tree](
      fields = Vector(
        Schema[Int].reflect.asTerm("value"),
        Reflect.Deferred(() => Schema.list(new Schema(treeReflect)).reflect).asTerm("children")
      ),
      typeId = TypeId.of[Tree],
      recordBinding = Binding.Record(
        constructor = new Constructor[Tree] {
          def usedRegisters: RegisterOffset = RegisterOffset(ints = 1, objects = 1)

          def construct(in: Registers, offset: RegisterOffset): Tree =
            Tree(
              in.getInt(offset),
              in.getObject(offset).asInstanceOf[List[Tree]]
            )
        },
        deconstructor = new Deconstructor[Tree] {
          def usedRegisters: RegisterOffset = RegisterOffset(ints = 1, objects = 1)

          def deconstruct(out: Registers, offset: RegisterOffset, in: Tree): Unit = {
            out.setInt(offset, in.value)
            out.setObject(offset, in.children)
          }
        }
      )
    )

    new Schema(treeReflect)
  }
}
```

## Debug-Friendly toString

`Reflect` types have a custom `toString` that produces a human-readable SDL (Schema Definition Language) format. This makes debugging schemas much easier compared to the default case class output.

```scala

case class Person(name: String, age: Int, address: Address)
case class Address(street: String, city: String)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

println(Schema[Person].reflect)
// Output:
// record Person {
//   name: String
//   age: Int
//   address: record Address {
//     street: String
//     city: String
//   }
// }
```

**Format by Reflect type:**

| Type | Format |
|------|--------|
| `Primitive` | Type name (e.g., `String`, `Int`, `java.time.Instant`) |
| `Record` | `record Name { fields... }` |
| `Variant` | `variant Name { \| Case1 \| Case2... }` |
| `Sequence` | `sequence List[Element]` or multiline for complex elements |
| `Map` | `map Map[Key, Value]` or multiline for complex types |
| `Wrapper` | `wrapper Name(underlying)` |
| `Deferred` | `deferred => TypeName` (breaks recursive cycles) |
| `Dynamic` | `DynamicValue` |

The printer writes `deferred => TypeName` for every record, variant, sequence, map or wrapper node it has already printed in the same `toString` call, not only for a recursive reference; primitives are always printed in full. A case class schema used for two fields is printed in full for the first field and as `deferred => TypeName` for the second.

**Variant example:**

```scala
sealed trait PaymentMethod
case object Cash extends PaymentMethod
case class CreditCard(number: String, cvv: String) extends PaymentMethod

// toString output:
// variant PaymentMethod {
//   | Cash
//   | CreditCard(
//       number: String,
//       cvv: String
//     )
// }
```

**Recursive type example:**

```scala
case class Tree(value: Int, children: List[Tree])

// toString output:
// record Tree {
//   value: Int
//   children: sequence List[
//     deferred => Tree
//   ]
// }
```

## Auto-Derivation

While you can manually construct `Reflect` instances as shown in the examples above, ZIO Blocks provides powerful auto-derivation capabilities that can automatically generate `Schema` instances (and thus `Reflect` instances) for most Scala types using macros and implicit resolution.

The auto-derivation mechanism inspects the structure of your data types at compile time and generates the appropriate `Reflect` representation, including nested types, collections, options, and more.

To leverage auto-derivation, simply define an implicit `Schema` for your type using `Schema.derived`:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}
```

The above will automatically generate a `Reflect.Record` for the `Person` case class, including fields for `name` and `age`, along with the necessary bindings for construction and deconstruction. The same applies to more complex types, including variants, collections, and recursive structures.

---

## Register System

The register system is one of the key innovations in ZIO Blocks (ZIO Schema 2) that enables **zero-allocation, box-free construction and deconstruction** of data types. 

## The Problem: Boxing/Unboxing Overheads

When building generic abstractions over data types (like serialization libraries), you need to describe all possible constructions and deconstructions uniformly. The traditional approach uses tuples and boxed primitives. For example assume we have a simple record data type:

```scala
case class Person(name: String, age: Int)
```

A traditional library might represent it as tuple when serializing/deserializing:

```scala
trait Tuple
case class Tuple2[A, B]( _1: A, _2: B)          extends Tuple
case class Tuple3[A, B, C](_1: A, _2: B, _3: C) extends Tuple
// ...
```

Tuple is generic data structure that can hold values of any type. So serializing `Person` would involve converting it to/from `Tuple2[String, Int]`:

```scala
case class Person(name: String, age: Int)
val person = Person("john", 42)

// Generic construction via tuples:
val tuple: (String, Int) = ("john", 42)  // Tuple2[String, Int]
```

The problem is that `Int` is a primitive type, and in order to fit it into a tuple, it must be **boxed** into `java.lang.Integer`. Why? Because tuples can only hold references to objects, not raw primitive values. They are generic containers that work uniformly for any type.

So the actual memory representation of the tuple looks like this:

```text
Stack:                          Heap:

                     
                     ┌────────────────────────┐      ┌─────────────────┐
┌─────────────┐      │ Tuple2 object          │      │ String "john"   │
│ tuple (ref) │─────▶│  _1: ────────────────────────▶│  char[]/byte[]: ───────▶[j][o][h][n]
└─────────────       │  _2: ──────┐           │      └─────────────────┘
                     └────────────────────────┘      
                                  │
                                  ▼
                          ┌─────────────────┐
                          │ Integer object  │
                          │  value: 42      │  ← 4 bytes for int
                          └─────────────────┘
```

This boxing creates significant runtime overhead because:

1. **Primitive boxing**: Values like `Int`, `Long`, `Double` must be wrapped in heap-allocated objects (`java.lang.Integer`, etc.)
2. **Tuple allocation**: All constructor arguments get wrapped in tuple objects. This creates extra allocations for every construction/deconstruction.
3. **Garbage collection pressure**: Each serialization/deserialization creates temporary objects

## The Solution: Register-Based Architecture

ZIO Blocks introduces a novel register-based design that completely eliminates tupling and boxing:

> "Zero allocation, zero boxing, construction and deconstruction of records. It doesn't get faster than this. You can't make generic code in Scala faster than what's been done here."
> — John De Goes, LambdaConf 2025

Instead of tuples, ZIO Blocks uses the `Registers` data structure, which contains:

1. A **byte array** for storing primitives (Int, Long, Double, Float, Boolean, Byte, Char, Short)
2. An **object array** for storing references to heap-allocated objects (AnyRef, which is the supertype of all reference types in Scala including String, custom classes, etc.)

This classification determines where values are stored in the `Registers` data structure. All primitive types use the same `bytes` register to store raw values, and all reference types use the same `objects` register to store references:

```scala
// Conceptual structure of Registers
class Registers {
  var bytes: Array[Byte]     = new Array[Byte](byteArrayLength)      // Stores all primitives efficiently
  var objects: Array[AnyRef] = new Array[AnyRef](objectArrayLength)  // Stores object references
}
```

The `Registers` class is a mutable data structure that serves as an intermediate buffer for **construction** and **deconstruction** operations. So it has methods to set and get values for each primitive type and for object references:

```scala
class Registers {
  private var bytes: Array[Byte]     = new Array[Byte](byteArrayLength)
  private var objects: Array[AnyRef] = new Array[AnyRef](objectArrayLength)

  // Methods to get/set primitive values in byte array (getInt/setInt, getBoolean/setBoolean, etc.)
  def getInt(offset: RegisterOffset): Int = ???
  def setInt(offset: RegisterOffset, value: Int): Unit = ???
  
  def setBoolean(offset: RegisterOffset, value: Boolean): Unit = ???
  def getBoolean(offset: RegisterOffset): Boolean = ???

  // Two methods to get/set object references in object array
  def getObject(offset: RegisterOffset): AnyRef = ???
  def setObject(offset: RegisterOffset, value: AnyRef): Unit = ???
}
```

This design allows primitives to be stored directly in their native binary representation without boxing, while objects are stored as simple references. So, when you use registers, the library adds zero overhead to construction and deconstruction — no tuples and no boxing of primitives.

```
┌─────────────────────────────────────────────────────────────────┐
│                        Registers                                │
├─────────────────────────────────────────────────────────────────┤
│  Byte Array (primitives)          │  Object Array (references)  │
│  ┌───┬───┬───┬───┬───┬───┬───┐    │  ┌───────┬───────┬───────┐  │
│  │ B │ B │ S │ S │ I │ I │...│    │  │ Obj0  │ Obj1  │ Obj2  │  │
│  └───┴───┴───┴───┴───┴───┴───┘    │  └───────┴───────┴───────┘  │
│  (raw bytes, no boxing)           │  (String, etc.)             │
└─────────────────────────────────────────────────────────────────┘
```

One powerful aspect of the register system is that you can reuse registers. If you reuse registers, then without any allocation you can construct and deconstruct things all day long. This is particularly valuable in high-throughput scenarios like:
- Deserializing streams of records
- Batch processing operations
- Real-time data pipelines

## RegisterOffset: Tracking Positions

`RegisterOffset` is a compact way to track positions within the `Registers` structure. It uses a single `Long` to encode two pieces of information:
1. Byte offset (for primitives) in the upper 32 bits
2. Object offset (for references) in the lower 32 bits

You can think of `RegisterOffset` as a cursor that tells you where to read/write the next primitive or object value. For example:
1. `RegisterOffset.Zero` represents the starting position with zero indexes for both primitives and objects.
2. `RegisterOffset(objects = 1)` indicates that the next object reference should be stored at index 1 in the object array, with zero byte offset for primitives.
3. `RegisterOffset(bytes = 10, objects = 3)` indicates that the next primitive value should be retrieved/stored starting at byte index 10 in the byte array, and the next object reference should be stored at index 3 in the object array.
4. `RegisterOffset(bytes = 4, ints = 2, objects = 3)` indicates that the next primitive value should be retrieved/stored starting at byte index 12 (4 bytes + 2 ints × 4 bytes each) in the byte array, and the next object reference should be stored at index 3 in the object array.

The byte offset is calculated by weighting each type by its size:

```
primitiveBytes = booleans + bytes 
               + (chars + shorts) × 2 
               + (floats + ints) × 4 
               + (doubles + longs) × 8
```

The object offset is simply the count of object references.

You don't need to do these calculations manually; the `RegisterOffset.getBytes` method computes the byte offset of the primitive register, and `RegisterOffset.getObjects` computes the offset of the object register.

```scala
RegisterOffset.getBytes(RegisterOffset(bytes = 4, ints = 2, objects = 3)) // output = (4 + 2*4) = 12
RegisterOffset.getObjects(RegisterOffset(bytes = 4, ints = 2, objects = 3)) // output = 3
```

## Creating Registers

`Registers` is a mutable container that holds values. It's created with an initial capacity:

```scala

// Create registers with space for 3 bytes, 1 ints, and 2 objects
val registers = Registers(RegisterOffset(bytes = 3, ints = 1, objects = 2))
```

While creating `Registers`, you specify how much space is needed for primitives and objects, using `RegisterOffset`. The library allocates the necessary arrays internally. After creation, when you set values, the library ensures that it has sufficient capacity, growing the arrays if necessary.

## Setting and Getting Values

The `Registers` data type has methods to set and get values for each primitive type and for object references. For example, the `setInt` method sets an `Int` value in the byte array, while `setObject` sets an object reference in the object array:

```scala
class Registers {
  private var bytes: Array[Byte]     = new Array[Byte](byteArrayLength)
  private var objects: Array[AnyRef] = new Array[AnyRef](objectArrayLength)

  // Methods to get/set primitive values in byte array (getInt/setInt, getBoolean/setBoolean, etc.)
  def getInt(offset: RegisterOffset): Int = ???
  def setInt(offset: RegisterOffset, value: Int): Unit = ???

  def setBoolean(offset: RegisterOffset, value: Boolean): Unit = ???
  def getBoolean(offset: RegisterOffset): Boolean = ???

  // Two methods to get/set object references in object array
  def getObject(offset: RegisterOffset): AnyRef = ???
  def setObject(offset: RegisterOffset, value: AnyRef): Unit = ???
}
```

Here are all the available methods:
- `setBoolean` / `getBoolean`
- `setByte` / `getByte`
- `setShort` / `getShort`
- `setInt` / `getInt`
- `setLong` / `getLong`
- `setFloat` / `getFloat`
- `setDouble` / `getDouble`
- `setChar` / `getChar`
- `setObject` / `getObject`

When setting or getting a value, you are required to provide one or two of the following parameters:

1. `offset`: The `RegisterOffset` indicating where the specific field starts in the `Registers`. It can be used to handle nested structures to point to where inner record starts, or for multiple records to point to each record's starting position, or for variant types to anchor the position of different cases.
2. `value`: The actual value to set. For the primitives, this value is a typed primitive (e.g. `Int`, `Double`); for objects, this is an `AnyRef`.

## Example: Encoding/Decoding a Record Data Type

Encoding data instances into registers involves mapping each field of a data type to its corresponding position in the two register arrays (byte array for primitives and object array for references). For example, assume we have a `Person` data type as below:

```scala
case class Person(
  name: String,
  email: String,
  age: Int,
  height: Double,
  weight: Double
)
```

We can encode it with registers, as follows:

```scala

// Person("John", "john@example.com", 42, 180.0, 67.0)
val registers = Registers(RegisterOffset(objects = 2, ints = 1, doubles = 2))

registers.setObject(
  RegisterOffset.Zero, // Object index: 0
  "John"
)

registers.setObject(
  RegisterOffset(objects = 1), // Object index: 1
  "john@example.com"
)

registers.setInt(
  RegisterOffset(objects = 2), // Byte index: 0
  42
)

registers.setDouble(
  RegisterOffset(objects = 2, ints = 1), // Byte index: (1 * 4) = 4
  180.0
)

registers.setDouble(
  RegisterOffset(objects = 2, ints = 1, doubles = 1), // Byte index: (1 * 4) + (1 * 8) = 12
  67.0
)
```

Conversely, to decode the `Person` data type from registers, you would read the values back from their respective positions:

```scala

// Decode Person from registers
val name = registers.getObject(
  RegisterOffset.Zero, // Object index: 0
).asInstanceOf[String]

val email = registers.getObject(
  RegisterOffset(objects = 1) // Object index: 1
).asInstanceOf[String]

val age = registers.getInt(
  RegisterOffset(objects = 2) // Object index: 2
)

val height = registers.getDouble(
  RegisterOffset(objects = 2, ints = 1) // Byte index: (1 * 4) = 4
)

val weight = registers.getDouble(
  RegisterOffset(objects = 2, ints = 1, doubles = 1) // Byte index: (1 * 4) + (1 * 8) = 12
)

val person = Person(name, email, age, height, weight)
```

---

## SchemaError

`SchemaError` is a **structured error type** for schema operations in ZIO Blocks. It represents one or more validation, conversion, or structural failures that occurred while decoding, encoding, or transforming data, each annotated with a [`DynamicOptic`](./dynamic-optic.md) path that pinpoints the failing location in the data structure.

```scala
final case class SchemaError(errors: ::[SchemaError.Single])
  extends Exception with NoStackTrace
```

Here is the full structure of `SchemaError`:

```
SchemaError
 └── errors: ::[Single]       (non-empty list — always at least one failure)
      │
      └── Single (sealed trait)
           │ ├── source: DynamicOptic   (path to the failing location)
           │ └── message: String        (human-readable description)
           │
           ├── ConversionFailed       (type or value conversion failed)
           ├── MissingField           (required field absent)
           ├── DuplicatedField        (same field key appears more than once)
           ├── ExpectationMismatch    (wrong DynamicValue variant encountered)
           ├── UnknownCase            (unrecognised sealed-trait discriminator)
           └── Message                (free-form message, optional path)
```

`SchemaError`:

- Aggregates multiple independent failures into a single error value
- Annotates every failure with a precise traversal path through the data
- Extends `Exception` so it can be thrown and caught with standard JVM mechanisms
- Suppresses stack traces via `NoStackTrace` — error location is conveyed through the path, not the JVM stack

## Motivation

When decoding a complex nested value, a single structural problem — a missing field, a type mismatch, an unknown case discriminator — must be reported together with the location where it occurred. In a large schema, multiple independent problems can coexist, and surfacing them all at once saves the caller round-trips.

Every `Schema#fromDynamicValue`, every `Codec#decode`, and every optic traversal that can fail returns `Either[SchemaError, A]`. The JSON and TOON codecs stop at the first failure, so their `Codec#decode` returns one error; only `Schema#fromDynamicValue` and `Into.derived` collect every error. The same type carries both structural errors (missing fields, wrong types) and domain validation errors (value out of range, blank string), so callers deal with a single error channel.

`SchemaError` was introduced to replace the earlier `JsonError` and `DynamicValueError` types that existed as separate error channels for each format. Those types used string concatenation (`"error1; error2"`) to combine failures via `++`, which silently discarded path information from the second error onward. `SchemaError` solves this by maintaining a non-empty list (`::`) of `Single` failures — each one independently annotated with its own `DynamicOptic` path — so no information is lost during aggregation.

Here is a quick taste of how `SchemaError` behaves:

```scala

// Create a simple message error
val err = SchemaError("Age must be positive")

// Annotate with the location in the data
val located = SchemaError.missingField(Nil, "email").atField("user")
println(located.message)   // Missing field 'email' at: .user

// Combine two independent failures
val combined = SchemaError("name is blank") ++ SchemaError("age is negative")
println(combined.errors.length)  // 2
```

## Construction / Creating Instances

The `SchemaError` companion has one factory per kind of failure, each returning a `SchemaError` with a single error inside.

### `SchemaError.apply`

The simplest constructor — creates a free-form `Message` error at the root path:

```scala
object SchemaError {
  def apply(details: String): SchemaError
}
```

Passing a description to `SchemaError.apply` gives an error whose message is that description alone:

```scala

val err = SchemaError("Value must be positive")
println(err.message)  // Value must be positive
```

### `SchemaError.message`

Creates a `Message` error with a free-form description and an optional `DynamicOptic` path. When no path is supplied it defaults to the root:

```scala
object SchemaError {
  def message(details: String, path: DynamicOptic = DynamicOptic.root): SchemaError
}
```

Here we create a message error at the root, and another with an explicit path:

```scala

// Root-level message (same as SchemaError.apply)
val atRoot = SchemaError.message("Unexpected null")
println(atRoot.message)  // Unexpected null

// Message with an explicit path
val path   = DynamicOptic.root.field("address")
val atPath = SchemaError.message("Unexpected null", path)
println(atPath.message)  // Unexpected null at: .address
```

### `SchemaError.validationFailed`

Convenience factory for validation failures. Equivalent to `SchemaError.conversionFailed(Nil, message)`, designed for smart constructors that return string-based error messages:

```scala
object SchemaError {
  def validationFailed(message: String): SchemaError
}
```

Here is an example:

```scala

val err = SchemaError.validationFailed("Age must be between 0 and 150")
println(err.message)  // Age must be between 0 and 150
```

### `SchemaError.conversionFailed`

Creates a `ConversionFailed` error for a failed type or value conversion. Two overloads exist:

```scala
object SchemaError {
  def conversionFailed(trace: List[DynamicOptic.Node], details: String): SchemaError
  def conversionFailed(contextMessage: String, cause: SchemaError): SchemaError
}
```

The first overload is used by codecs — `trace` is the list of path nodes accumulated during decoding. Pass `Nil` when constructing an error manually and use the `at*` methods to set the path. The second overload wraps a nested `SchemaError` with additional context; the nested failures are rendered under a "Caused by:" section. The block below builds a root-level failure with the first overload and wraps two failures with the second:

```scala

// Root-level conversion failure
val err = SchemaError.conversionFailed(Nil, "Expected a positive integer")
println(err.message)  // Expected a positive integer

// Wrapping a nested failure with context
val inner = SchemaError("name must not be empty") ++
            SchemaError("age must be positive")
val outer = SchemaError.conversionFailed("Person construction failed", inner)
println(outer.message)
// Person construction failed
//   Caused by:
//   - name must not be empty
//   - age must be positive
```

### `SchemaError.missingField`

Creates a `MissingField` error indicating that a required field was absent from the decoded representation. The JSON codec reports the same failure with its own wording, `missing required field "zip" at: .address`, so its message differs from the one this constructor produces.

`SchemaError.missingField` takes the trace and the name of the missing field:

```scala
object SchemaError {
  def missingField(trace: List[DynamicOptic.Node], fieldName: String): SchemaError
}
```

To report that the `user` record has no `email` field:

```scala

val err = SchemaError.missingField(Nil, "email").atField("user")
println(err.message)  // Missing field 'email' at: .user
```

### `SchemaError.duplicatedField`

Creates a `DuplicatedField` error indicating that the same field key appeared more than once in the encoded form:

```scala
object SchemaError {
  def duplicatedField(trace: List[DynamicOptic.Node], fieldName: String): SchemaError
}
```

To report that the `id` key appears twice inside `record`:

```scala

val err = SchemaError.duplicatedField(Nil, "id").atField("record")
println(err.message)  // Duplicated field 'id' at: .record
```

### `SchemaError.expectationMismatch`

Creates an `ExpectationMismatch` error indicating that the encountered `DynamicValue` variant does not match what the schema expected:

```scala
object SchemaError {
  def expectationMismatch(trace: List[DynamicOptic.Node], expectation: String): SchemaError
}
```

To report that the `data` field holds a `Sequence` where a `Record` was expected:

```scala

val err = SchemaError
  .expectationMismatch(Nil, "Expected Record, got Sequence")
  .atField("data")
println(err.message)  // Expected Record, got Sequence at: .data
```

### `SchemaError.unknownCase`

Creates an `UnknownCase` error indicating that the decoded discriminator value does not correspond to any known variant of a sealed trait:

```scala
object SchemaError {
  def unknownCase(trace: List[DynamicOptic.Node], caseName: String): SchemaError
}
```

To report that the `shape` field names a `Triangle` case the sealed trait does not have:

```scala

val err = SchemaError.unknownCase(Nil, "Triangle").atField("shape")
println(err.message)  // Unknown case 'Triangle' at: .shape
```

## Core Operations

The operations on a `SchemaError` value render its messages, combine it with other errors, and add path segments to every failure inside it.

### Error Messages

`SchemaError` has two ways to render its failures as text, and both return the same string.

#### `message`

Returns all individual error messages joined with newlines:

```scala
final case class SchemaError(errors: ::[SchemaError.Single]) {
  def message: String
}
```

Combining two errors with `++` gives a message with one line per failure:

```scala

val err = SchemaError("first failure") ++ SchemaError("second failure")
println(err.message)
// first failure
// second failure
```

#### `getMessage`

Delegates to `message`. Because `SchemaError` extends `Exception`, `getMessage` is called by the JVM when the exception is printed or logged by frameworks.

`getMessage` overrides the method of `Exception`:

```scala
final case class SchemaError(errors: ::[SchemaError.Single]) {
  def getMessage: String
}
```

For any `SchemaError`, `getMessage` and `message` return the same string:

```scala

val err = SchemaError("something went wrong")
assert(err.getMessage == err.message)
```

### Error Aggregation

To report several independent failures together, combine them into one `SchemaError`.

#### `++`

Combines two `SchemaError` values into one, preserving all individual `Single` failures from both sides. We use `++` to accumulate errors from independent parts of a schema — for example, multiple record fields that are each decoded independently.

`++` takes the other `SchemaError` and returns a new one that holds the failures of both:

```scala
final case class SchemaError(errors: ::[SchemaError.Single]) {
  def ++(other: SchemaError): SchemaError
}
```

Combining a missing-field error with a conversion failure keeps both failures, in order:

```scala

val nameError = SchemaError.missingField(Nil, "name")
val ageError  = SchemaError.conversionFailed(Nil, "Age must be positive")
val combined  = nameError ++ ageError

println(combined.errors.length)   // 2
println(combined.message)
// Missing field 'name' at: .
// Age must be positive
```

`++` is associative: `(a ++ b) ++ c` and `a ++ (b ++ c)` produce the same set of errors.

### Path Annotation

Path annotation methods prepend a path segment to the `source` of every `SchemaError.Single` inside the error. Codecs call these methods as they unwind the call stack — the innermost call adds the innermost path segment, and the outermost call adds the outermost one.

#### `atField`

Prepends a record field access to the path of all errors:

```scala
final case class SchemaError(errors: ::[SchemaError.Single]) {
  def atField(name: String): SchemaError
}
```

When nested codecs each call `atField` as they unwind, the outer field ends up first in the path:

```scala

// Codec decoding 'city' inside 'address' inside 'user'
val err = SchemaError.missingField(Nil, "city")
  .atField("address")  // called by the address codec
  .atField("user")     // called by the user codec
println(err.message)   // Missing field 'city' at: .user.address
```

#### `atIndex`

Prepends a sequence index access to the path of all errors:

```scala
final case class SchemaError(errors: ::[SchemaError.Single]) {
  def atIndex(index: Int): SchemaError
}
```

To place an error at the third element of the `phones` field:

```scala

val err = SchemaError("invalid phone number").atIndex(2).atField("phones")
println(err.message)  // invalid phone number at: .phones[2]
```

#### `atCase`

Prepends a sealed-trait case access to the path of all errors. In the compact path notation used by `Message`, the case name appears as `<CaseName>`.

`atCase` takes the name of the case:

```scala
final case class SchemaError(errors: ::[SchemaError.Single]) {
  def atCase(name: String): SchemaError
}
```

To place an error inside the `value` field of the `Right` case:

```scala

val err = SchemaError("conversion failed").atField("value").atCase("Right")
println(err.message)  // conversion failed at: <Right>.value
```

#### `atKey`

Prepends a map key access to the path of all errors. The key is a [`DynamicValue`](./dynamic-value.md), rendered with `{key}` in the compact path notation.

`atKey` takes the key as a `DynamicValue`:

```scala
final case class SchemaError(errors: ::[SchemaError.Single]) {
  def atKey(key: DynamicValue): SchemaError
}
```

To place an error at the `"config"` key of a map:

```scala

val key = DynamicValue.string("config")
val err = SchemaError("missing required entry").atKey(key)
println(err.message)  // missing required entry at: {"config"}
```

### Path Chaining

All path methods can be chained. Each call prepends to the existing path, so the outermost call appears as the leftmost segment in the rendered message:

```scala

val err = SchemaError("value out of range")
  .atField("amount")       // innermost — added first
  .atIndex(0)
  .atCase("Credit")
  .atField("transactions") // outermost — added last
println(err.message)
// value out of range at: .transactions<Credit>[0].amount
```

Path annotation applies to **every** `Single` inside the error, so combined errors accumulate paths correctly:

```scala

val error1 = SchemaError.missingField(Nil, "name")
val error2 = SchemaError.conversionFailed(Nil, "age must be positive")
val combined = (error1 ++ error2).atField("person")

// Both errors now include the "person" field prefix
println(combined.errors.head.source.nodes.nonEmpty)  // true (name)
println(combined.errors.tail.head.source.nodes.nonEmpty)  // true (age)
```

## Subtypes / Variants

`SchemaError.Single` is the sealed base trait for every individual failure. Each variant carries a `source: DynamicOptic` and a `message: String`, and the variants form this tree:

```
SchemaError.Single (sealed trait)
 ├── SchemaError.IntoError (sealed sub-trait — marks conversion errors)
 │   └── ConversionFailed(source, details, cause: Option[SchemaError])
 ├── MissingField(source, fieldName)
 ├── DuplicatedField(source, fieldName)
 ├── ExpectationMismatch(source, expectation)
 ├── UnknownCase(source, caseName)
 └── Message(source, details)
```

| Subtype               | Factory                                | Typical cause                                                                             |
|-----------------------|----------------------------------------|-------------------------------------------------------------------------------------------|
| `ConversionFailed`    | `conversionFailed`, `validationFailed` | Type conversion or smart-constructor failure; may carry a nested `SchemaError` as `cause` |
| `MissingField`        | `missingField`                         | Required field absent in the decoded representation                                       |
| `DuplicatedField`     | `duplicatedField`                      | Same field key appears more than once                                                     |
| `ExpectationMismatch` | `expectationMismatch`                  | Wrong `DynamicValue` variant encountered                                                  |
| `UnknownCase`         | `unknownCase`                          | Discriminator names an unrecognised sealed-trait variant                                  |
| `Message`             | `message`, `apply`                     | Free-form error with an optional path                                                     |

We can pattern match on `errors` to handle specific failure kinds:

```scala

val err = SchemaError.missingField(Nil, "email") ++
          SchemaError.conversionFailed(Nil, "age must be positive")

err.errors.foreach {
  case SchemaError.MissingField(source, fieldName) =>
    println(s"Missing '$fieldName' at ${source.toScalaString}")
  case SchemaError.ConversionFailed(source, details, _) =>
    println(s"Conversion failed: $details")
  case other =>
    println(other.message)
}
```

### `SchemaError.IntoError`

`IntoError` is a sealed sub-trait of `Single` that marks errors produced during `Into` (type conversion) operations. Its only current implementation is `ConversionFailed`. Codec code pattern-matches on `IntoError` to distinguish conversion errors from structural schema errors:

```scala
sealed trait IntoError extends SchemaError.Single {
  def source: DynamicOptic
}
```

### `SchemaError.ConversionFailed`

Represents a failed type or value conversion. When a `cause: Option[SchemaError]` is present, the rendered `message` includes a "Caused by:" section showing the nested failures, with one cause on the same line and several as a list:

```scala

// Single nested cause
val inner1 = SchemaError.conversionFailed(Nil, "name is blank")
val outer1 = SchemaError.conversionFailed("Person construction failed", inner1)
println(outer1.message)
// Person construction failed
//   Caused by: name is blank

// Multiple nested causes
val inner2 = SchemaError.conversionFailed(Nil, "name is blank") ++
             SchemaError.conversionFailed(Nil, "age is negative")
val outer2 = SchemaError.conversionFailed("Person construction failed", inner2)
println(outer2.message)
// Person construction failed
//   Caused by:
//   - name is blank
//   - age is negative
```

## Integration

`SchemaError` is the error type that decoding, `Schema#transform`, `Validation` and optic operations all report through.

### With Schema Decoding

`Schema#fromDynamicValue` returns `Either[SchemaError, A]`. Every codec accumulates path nodes during decoding and calls `atField`, `atIndex`, or `atCase` as it unwinds, producing a fully-annotated `SchemaError` on failure. The `Person` round trip below decodes a value that `Schema#toDynamicValue` just built:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val result: Either[SchemaError, Person] =
  Schema[Person].fromDynamicValue(Schema[Person].toDynamicValue(Person("Alice", 30)))

result match {
  case Right(person) => println(s"Decoded: $person")
  case Left(err)     => println(s"Error:\n${err.message}")
}
```

See [Schema](./schema.md) and [DynamicValue](./dynamic-value.md) for the full encoding and decoding API.

### With Schema#transform

`Schema#transform` accepts `to` and `from` functions that can throw `SchemaError` to signal validation failures during encoding or decoding. We use `SchemaError.validationFailed` (which wraps a `ConversionFailed`) to turn a smart-constructor rejection into a structured schema error:

```scala

case class PositiveInt private (value: Int)

object PositiveInt {
  def make(n: Int): PositiveInt =
    if (n > 0) PositiveInt(n)
    else throw SchemaError.validationFailed("must be positive")

  implicit val schema: Schema[PositiveInt] =
    Schema[Int].transform(make, _.value)
}
```

When `make` throws, `Schema#fromDynamicValue` and `Codec#decode` catch the exception and return `Left` with a new error that keeps only the exception's message, at a path that ends in `.wrapped` (`must be positive at: .wrapped`): a `SchemaError.ConversionFailed` from `Schema#fromDynamicValue`, and a `SchemaError.ExpectationMismatch` from the JSON codec. See [Schema](./schema.md) for the full `transform` API.

A runnable version of this example, including composite types and error aggregation, is available in the `schema-examples` module:

```bash
cd schema-examples && sbt "runMain schemaerror.SchemaErrorExample"
```

### With Validation

The [Validation](./validation.md) system uses `SchemaError` to report constraint violations. When a `PrimitiveType` carries a `Validation`, `DynamicSchema#check` reports a value that breaks it as a `SchemaError.ConversionFailed` (`Validation failed: value must be positive`) at the value's path. The JSON codec and `Schema#fromDynamicValue` do not check `Validation` constraints; only the schema that `DynamicSchema#toSchema` returns runs `check` while decoding.

### With DynamicOptic and Optics

Path annotation (`atField`, `atIndex`, `atKey`, `atCase`) builds a [`DynamicOptic`](./dynamic-optic.md) inside each error. Operations such as `DynamicValue#setOrFail` and `DynamicValue#modifyAtPathOrFail` return `Either[SchemaError, DynamicValue]`, using the same factory methods. Setting index 10 of a two-element sequence fails with a path error:

```scala

val data   = DynamicValue.Sequence(DynamicValue.int(1), DynamicValue.int(2))
val optic  = DynamicOptic.root.at(10)

val result: Either[SchemaError, DynamicValue] = data.setOrFail(optic, DynamicValue.int(99))
result match {
  case Left(err) => println(err.message)  // Path not found at: [10]
  case Right(v)  => println(v)
}
```

### As an Exception

Because `SchemaError` extends `Exception`, it can be thrown and caught with standard try/catch (In functional code, prefer `Either[SchemaError, A]` or `Option[SchemaError]` instead):

```scala

try {
  throw SchemaError("Unexpected data shape")
} catch {
  case e: SchemaError => println(s"Caught schema error: ${e.getMessage}")
}
```

:::note
`SchemaError` extends `scala.util.control.NoStackTrace`. Stack traces are suppressed for performance — error location is conveyed through the `DynamicOptic` path inside each `Single`, not the JVM stack.
:::

See also the [Your First Schema](../../guides/your-first-schema.md) tutorial, which shows how decode failures surface as `SchemaError`, and the [How to Define Validated Newtypes and Collect Every Validation Error](../../guides/validated-newtypes.md) guide, which collects every validation failure into one `SchemaError`.

---

## As

`As[A, B]` is a **bidirectional conversion type class** that extends `Into[A, B]` with a reverse direction. In addition to converting `A → B` via `As#into`, it also converts `B → A` via `As#from`, providing a round-trip guarantee for every field the two types share.

`As`:
- extends `Into[A, B]`, so every `As` can be used wherever an `Into` is expected
- returns `Right(b)` or `Right(a)` on success and `Left(error)` on validation failure in both directions
- derives automatically for case classes, sealed traits, tuples, and Scala 3 enums via `As.derived`
- enforces stricter derivation constraints than `Into` to guarantee that `A → B → A` always restores the original value

```scala
trait As[A, B] extends Into[A, B] {
  def from(input: B): Either[SchemaError, A]
  def reverse: As[B, A]
}
```

The bidirectional data flow looks like this:

```
  ┌──────────────────────────────────────────────────┐
  │               As[A, B]                           │
  │                                                  │
  │     into(a: A) ──────────────────────► B         │
  │                                                  │
  │     from(b: B) ◄────────────────────── B         │
  │                                                  │
  │     reverse: As[B, A]  (flips directions)        │
  └──────────────────────────────────────────────────┘
```

`As` is the right choice when the conversion must be safe to run in both directions — for example when synchronising data between a local model and a remote representation, or when migrating a database schema that must remain rollback-capable.

## Installation

`As` is part of `zio-blocks-schema`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"
```

For Scala.js and Scala Native, use `%%%`:

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-schema" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x.

## Creating Instances

There are three ways to obtain an `As[A, B]`: construct it from a pair of `Into` instances, derive it automatically with the macro, or summon an implicit already in scope.

### `As.apply` — Manual Construction

`As.apply(intoAB, intoBA)` composes two `Into` instances into one `As`:

```scala
object As {
  def apply[A, B](intoAB: Into[A, B], intoBA: Into[B, A]): As[A, B]
}
```

We build an `As[Int, Long]` by supplying both directions explicitly:

```scala

val intoAB: Into[Int, Long] = a => Right(a.toLong)
val intoBA: Into[Long, Int] = b =>
  if (b >= Int.MinValue && b <= Int.MaxValue) Right(b.toInt)
  else Left(SchemaError.validationFailed("overflow"))

val manualAs: As[Int, Long] = As(intoAB, intoBA)
```

With `manualAs` in scope we can convert in both directions and verify overflow detection:

```scala
manualAs.into(42)
// res0: Either[SchemaError, Long] = Right(42L)
manualAs.from(100L)
// res1: Either[SchemaError, Int] = Right(100)
manualAs.from(Long.MaxValue)
// res2: Either[SchemaError, Int] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(ArraySeq()),
//         details = "overflow",
//         cause = None
//       )
//     )
//   )
// )
```

### `As.derived` — Macro Derivation

`As.derived[A, B]` generates an `As[A, B]` at compile time by deriving both `Into[A, B]` and `Into[B, A]` and running bidirectional compatibility checks:

```scala
object As {
  def derived[A, B]: As[A, B]
}
```

The macro works with case classes, sealed traits, Scala 3 enums, tuples, ZIO Prelude newtypes, Scala 3 opaque types, and structural types (JVM only). We derive an `As` for two case classes with matching fields:

```scala

case class PersonA(name: String, age: Int)
case class PersonB(name: String, age: Long)

val personAs: As[PersonA, PersonB] = As.derived[PersonA, PersonB]
```

Both `As#into` and `As#from` are now available, and we can verify that `A → B → A` restores the original value:

```scala
personAs.into(PersonA("Alice", 30))
// res3: Either[SchemaError, PersonB] = Right(
//   PersonB(name = "Alice", age = 30L)
// )
personAs.from(PersonB("Bob", 25L))
// res4: Either[SchemaError, PersonA] = Right(PersonA(name = "Bob", age = 25))
personAs.into(PersonA("Alice", 30)).flatMap(personAs.from)
// res5: Either[SchemaError, PersonA] = Right(
//   PersonA(name = "Alice", age = 30)
// )
```

### `As.apply[A, B]` — Summoning

`As.apply[A, B]` (with no arguments) summons an implicit `As[A, B]` already in scope — the same pattern used by `Into.apply`:

```scala
object As {
  def apply[A, B](implicit ev: As[A, B]): As[A, B]
}
```

This is useful when you want to retrieve a type-class instance by type rather than by variable name:

```scala

case class Foo(x: Int)
case class Bar(x: Int)

implicit val fooBarAs: As[Foo, Bar] = As.derived[Foo, Bar]

// Summon the implicit instance
val summoned = As[Foo, Bar]
summoned.into(Foo(1))
summoned.from(Bar(2))
```

## Core Operations

`As` exposes three operations: `As#into`, `As#from`, and `As#reverse`.

### `As#into` — Forward Conversion

`As#into` is inherited from `Into[A, B]` and converts an `A` into `Either[SchemaError, B]`:

```scala
trait As[A, B] extends Into[A, B] {
  def into(a: A): Either[SchemaError, B]
}
```

### `As#from` — Reverse Conversion

`As#from` is the operation that distinguishes `As` from `Into`. It converts a `B` back to `Either[SchemaError, A]`:

```scala
trait As[A, B] {
  def from(b: B): Either[SchemaError, A]
}
```

We define two simple wrapper types and derive an `As` between them to show both directions:

```scala

case class IntBox(value: Int)
case class LongBox(value: Long)

val boxAs: As[IntBox, LongBox] = As.derived[IntBox, LongBox]
```

`As#into` widens the value while `As#from` narrows it, validating that the result fits in the target type:

```scala
boxAs.into(IntBox(42))
// res8: Either[SchemaError, LongBox] = Right(LongBox(42L))
boxAs.from(LongBox(99L))
// res9: Either[SchemaError, IntBox] = Right(IntBox(99))
boxAs.from(LongBox(Long.MaxValue))
// res10: Either[SchemaError, IntBox] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(IndexedSeq()),
//         details = "converting field LongBox.value to IntBox.value failed",
//         cause = Some(
//           SchemaError(
//             List(
//               ConversionFailed(
//                 source = DynamicOptic(ArraySeq()),
//                 details = "Value 9223372036854775807 is out of range for Int [-2147483648, 2147483647]",
//                 cause = None
//               )
//             )
//           )
//         )
//       )
//     )
//   )
// )
```

### `As#reverse` — Flipping Directions

`As#reverse` returns an `As[B, A]` whose `As#into` and `As#from` are swapped:

```scala
trait As[A, B] {
  def reverse: As[B, A]
}
```

`As#reverse` creates a new `As` without touching the original:

```scala
val revAs: As[LongBox, IntBox] = boxAs.reverse
// revAs: As[LongBox, IntBox] = zio.blocks.schema.As$$anon$1@62525c99

revAs.into(LongBox(5L))
// res11: Either[SchemaError, IntBox] = Right(IntBox(5))
revAs.from(IntBox(10))
// res12: Either[SchemaError, LongBox] = Right(LongBox(10L))
```

## Using `As` as `Into`

Because `As[A, B]` extends `Into[A, B]`, any `As` instance can be passed wherever an `Into` is expected — with no casts or wrapping needed.

We write a generic migration helper that requires only an `Into`, then pass an `As` directly:

```scala

case class P2D(x: Int, y: Int)
case class Coord(x: Int, y: Int)

def migrate[A, B](data: A)(implicit into: Into[A, B]): Either[SchemaError, B] =
  into.into(data)

implicit val pointAs: As[P2D, Coord] = As.derived[P2D, Coord]
```

Passing `pointAs` where the function expects `Into[P2D, Coord]` works because `As` is a subtype of `Into`:

```scala
migrate(P2D(1, 2))
// res13: Either[SchemaError, Coord] = Right(Coord(x = 1, y = 2))
```

## `As.reverseInto` Implicit

`AsLowPriorityImplicits` provides `As.reverseInto`, an implicit that materialises an `Into[B, A]` from any `As[A, B]` in scope. This lets libraries that only require `Into` automatically benefit from `As` instances without any extra wiring:

```scala
trait AsLowPriorityImplicits {
  implicit def reverseInto[A, B](implicit as: As[A, B]): Into[B, A]
}
```

With an `As[String, Int]` in scope, `As.reverseInto` synthesises `Into[Int, String]` automatically:

```scala

implicit val stringIntAs: As[String, Int] = new As[String, Int] {
  def into(s: String): Either[SchemaError, Int] =
    try Right(s.toInt)
    catch { case _: NumberFormatException => Left(SchemaError.validationFailed("not an int")) }
  def from(n: Int): Either[SchemaError, String] = Right(n.toString)
}
```

We import `As.reverseInto` and use it to obtain the reverse `Into[Int, String]`:

```scala

val intToStr: Into[Int, String] = reverseInto[String, Int]
// intToStr: Into[Int, String] = zio.blocks.schema.AsLowPriorityImplicits$$Lambda/0x000000004a52e2d8@75e7c5b3
intToStr.into(42)
// res14: Either[SchemaError, String] = Right("42")
```

## Derivation Rules

`As.derived` applies the same rules as `Into.derived` in both directions and adds bidirectional compatibility checks on top. The derivation supports the same type categories as `Into`.

### Products (Case Classes and Tuples)

For two case classes `A` and `B`, the macro checks:
- fields with matching names must be convertible in **both** directions
- fields present in one type but absent from the other must be `Option` (defaults are not allowed — see [Restrictions](#restrictions))

We derive `As` for two structurally compatible case classes:

```scala

case class UserV1(name: String, age: Int)
case class UserV2(name: String, age: Long)

val userAs: As[UserV1, UserV2] = As.derived[UserV1, UserV2]
```

Tuples are matched positionally, so field name checks are skipped:

```scala

val tupleAs: As[(Int, String), (Long, String)] = As.derived[(Int, String), (Long, String)]
```

### Coproducts (Sealed Traits and Enums)

`As.derived` handles sealed traits and Scala 3 enums the same way `Into.derived` does: each subtype is matched as `Into.derived` matches it, by name, then by convertible field types, then by position (see the [Into](into.md) page), and its fields are converted:

```scala

sealed trait ShapeV1
object ShapeV1 {
  case class Circle(radius: Int)  extends ShapeV1
  case class Rect(w: Int, h: Int) extends ShapeV1
}

sealed trait ShapeV2
object ShapeV2 {
  case class Circle(radius: Long) extends ShapeV2
  case class Rect(w: Long, h: Long) extends ShapeV2
}

val shapeAs: As[ShapeV1, ShapeV2] = As.derived[ShapeV1, ShapeV2]
```

### Numeric Coercions

All numeric primitive types (`Byte`, `Short`, `Int`, `Long`, `Float`, `Double`) are bidirectionally coercible. Widening always succeeds; narrowing validates at runtime and returns a `Left` on overflow:

```scala

case class IntModel(value: Int)
case class LongModel(value: Long)

val numericAs: As[IntModel, LongModel] = As.derived[IntModel, LongModel]
```

A value within `Int` range round-trips without loss; one outside it fails on the way back:

```scala
numericAs.into(IntModel(1000)).flatMap(numericAs.from)
// res15: Either[SchemaError, IntModel] = Right(IntModel(1000))
numericAs.from(LongModel(Long.MaxValue))
// res16: Either[SchemaError, IntModel] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(IndexedSeq()),
//         details = "converting field LongModel.value to IntModel.value failed",
//         cause = Some(
//           SchemaError(
//             List(
//               ConversionFailed(
//                 source = DynamicOptic(ArraySeq()),
//                 details = "Value 9223372036854775807 is out of range for Int [-2147483648, 2147483647]",
//                 cause = None
//               )
//             )
//           )
//         )
//       )
//     )
//   )
// )
```

## Restrictions

`As` enforces constraints that `Into` does not. Because `As.derived` must produce valid conversions in both directions, it rejects most configurations that would lose data during a round-trip. `Option` fields that only one side has are the exception, as shown below.

**Default values on asymmetric fields are rejected.** A field with a default that has no counterpart in the other type cannot be round-tripped: when converting back, the field is missing and there is no way to distinguish a real default from a missing value:

```scala

case class WithDefault(name: String, age: Int = 25)
case class NoDefault(name: String)

// Does NOT compile — age has a default but is absent from NoDefault:
// As.derived[WithDefault, NoDefault]
```

Default values are allowed when the field exists in **both** types, because the value is never discarded during the round-trip:

```scala

case class PersonA(name: String, age: Int = 25)
case class PersonB(name: String, age: Int)

As.derived[PersonA, PersonB]  // compiles — age is present in both types
```

**`Option` fields on one side are allowed.** An `Option` field absent from the other type is dropped on the way out and comes back as `None`, so `Some(v)` does not survive a round trip:

```scala

case class TypeA(name: String, nickname: Option[String])
case class TypeB(name: String)

val dropsNickname: As[TypeA, TypeB] = As.derived[TypeA, TypeB]
```

Converting a value with a nickname and back loses the nickname:

```scala
dropsNickname.into(TypeA("Ada", Some("Countess"))).flatMap(dropsNickname.from)
// res18: Either[SchemaError, TypeA] = Right(
//   TypeA(name = "Ada", nickname = None)
// )
```

**Numeric coercions must be invertible in both directions.** Widening `Int → Long` is automatically paired with narrowing `Long → Int`. The narrowing validates at runtime, so the round-trip is safe even though it can fail:

```scala

case class IntVersion(value: Int)
case class LongVersion(value: Long)

As.derived[IntVersion, LongVersion]  // compiles — widening + narrowing form a valid pair
```

**Fields present in one type but absent from the other must be `Option`.** A non-optional field that exists only on one side cannot be populated in the reverse direction:

```scala

case class Short_(name: String)
case class Long_(name: String, extra: String)

// Does NOT compile — extra is not Optional and does not exist in Short_:
// As.derived[Short_, Long_]

case class Long2_(name: String, extra: Option[String])

As.derived[Short_, Long2_]  // compiles — extra is Optional
```

## DynamicValue Conversions

Like `Into`, `As` supports bidirectional conversions with `DynamicValue`, allowing you to define a single schema and use it for both type-safe operations and polyglot data handling.

### Bidirectional DynamicValue Support with JSON Round-Trip

You can derive `As[A, DynamicValue]` for any type with a `Schema[A]` and achieve full polyglot round-trips:

```scala

case class Config(host: String, port: Int)

object Config {
  implicit val schema: Schema[Config] = Schema.derived[Config]
  val asDynamic: As[Config, DynamicValue] = As.derived[Config, DynamicValue]
}
```

```scala
// Forward: Config → DynamicValue → JSON
Config.asDynamic.into(Config("localhost", 8080)).map(_.toJsonString)
// res21: Either[SchemaError, String] = Right(
//   "{\"host\":\"localhost\",\"port\":8080}"
// )
```

Now in the reverse direction, deserialize JSON back to Config:

```scala

case class Config(host: String, port: Int)

object Config {
  implicit val schema: Schema[Config] = Schema.derived[Config]
  val asDynamic: As[Config, DynamicValue] = As.derived[Config, DynamicValue]

  // JSON string to parse
  val jsonString = """{"host":"example.com","port":9000}"""
}
```

```scala
// Reverse: JSON → DynamicValue → Config
for {
  dv <- Config.jsonString.fromJson[DynamicValue]
  config <- Config.asDynamic.from(dv)
} yield config
// res22: Either[SchemaError, Config] = Right(
//   Config(host = "example.com", port = 9000)
// )
```

The call to `jsonString.fromJson[DynamicValue]` parses the JSON string into a `DynamicValue`, and `asDynamic.from` converts it back to the strongly-typed `Config`. (Equivalently, you could use `Schema[DynamicValue].getInstance(JsonFormat).decode(jsonString)` for the same decoding step.) This demonstrates the full cycle: **JSON → DynamicValue → Type**, ensuring perfect round-trip fidelity.

### Use Cases

`As` is ideal when data must flow in both directions within the same system, with guarantees that neither direction silently corrupts data, apart from an `Option` field that only one side has, which comes back as `None`.

#### Polyglot configuration systems

Configuration is often stored externally (Consul, etcd, a JSON file) and must be read, modified in-place, and written back. A naive approach requires two separate conversions — `Into[DynamicValue, DatabaseConfig]` to read and `Into[DatabaseConfig, DynamicValue]` to write — with no guarantee they align. `As` solves this by providing a single bidirectional instance that the macro checks will round-trip every field the two types share.

Consider a service that:
1. Reads config from an external store (JSON → `DynamicValue` → typed `DatabaseConfig`)
2. Applies business logic to the typed config (validate, scale, migrate)
3. Writes the updated config back to the store (typed `DatabaseConfig` → `DynamicValue` → JSON)

Without `As`, step 3 might serialize data differently than step 1 read it, causing silent corruption or misalignment. With `As`, the macro guarantees that `config → DynamicValue → config'` preserves the structure.

```scala

case class DatabaseConfig(host: String, port: Int, timeout: Long)

object DatabaseConfig {
  implicit val schema: Schema[DatabaseConfig] = Schema.derived[DatabaseConfig]
  val asDynamic: As[DatabaseConfig, DynamicValue] = As.derived[DatabaseConfig, DynamicValue]
}
```

```scala
// Simulate JSON arriving from the config store (e.g. Consul, etcd, a JSON file)
val storedJson = """{"host":"db.prod.example.com","port":5432,"timeout":30000}"""
// storedJson: String = "{\"host\":\"db.prod.example.com\",\"port\":5432,\"timeout\":30000}"

val result = for {
  stored  <- storedJson.fromJson[DynamicValue]          // Step 1: Read from store
  config  <- DatabaseConfig.asDynamic.from(stored)      // Step 2a: Hydrate into typed config
  updated  = config.copy(timeout = 60000)               // Step 2b: Apply business logic
  written <- DatabaseConfig.asDynamic.into(updated)     // Step 3: Serialize back to store (guaranteed round-trip)
} yield written.toJsonString
// result: Either[SchemaError, String] = Right(
//   "{\"host\":\"db.prod.example.com\",\"port\":5432,\"timeout\":60000}"
// )

result
// res23: Either[SchemaError, String] = Right(
//   "{\"host\":\"db.prod.example.com\",\"port\":5432,\"timeout\":60000}"
// )
```

## Scala 2 vs Scala 3 Differences

| Feature | Scala 2 | Scala 3 |
|---------|---------|---------|
| Derivation syntax | `As.derived[A, B]` | `As.derived[A, B]` |
| Enum support | Sealed traits only | Scala 3 enums + sealed traits |
| Opaque types | N/A | ✅ Supported |
| Structural types | JVM only (reflection) | JVM only (reflection) |
| ZIO Prelude newtypes | ✅ `assert { between(...) }` | ✅ `override def assertion` |
| Error messages | Detailed macro errors | Detailed macro errors |
| DynamicValue ambiguity detection | ✅ Two-pass implicit resolution | ✅ Built-in ambiguity reporting |

## Integration

`As[A, B]` is defined in `zio.blocks.schema` alongside `Into[A, B]`. Because `As` is a subtype of `Into`, the two type classes compose naturally: you can derive an outer `As` from inner `As` instances, or mix `As` and custom `Into` instances when some fields need one-way or custom logic.

For a full reference on one-way conversions and the derivation rules that `As` builds on, see [Into](into.md).

## See Also

- [Evolve Your Data](../../../guides/evolve-your-data.md) - a tutorial that converts a book record between two versions with `As` and checks the round trip in both directions.

---

## Schema Evolution

Schema evolution is the process of changing data structures over time while keeping existing data readable and systems interoperable. ZIO Blocks provides two type classes for this: `Into` for one-way conversions and `As` for bidirectional round-trip conversions.

```
         Into[A, B]                    As[A, B]
  ─────────────────────        ──────────────────────────
  A ──── into(a) ──── B        A ──── into(a) ────► B
                               A ◄─── from(b) ──── B
  One-way, asymmetric           Bidirectional, round-trip
  Allows defaults, drops        Requires fields to match
  extra fields freely           or be Option; no defaults
                                on asymmetric fields
```

## `Into[A, B]` — One-Way Conversion

[`Into[A, B]`](into.md) converts a value of type `A` to `Either[SchemaError, B]`. It is the right choice whenever the migration is asymmetric — for example, when adding a field with a default value, removing a field, or transforming data in a way that cannot be reversed.

Typical use cases:

- Migrating records from an old schema version to a new one
- Translating an external DTO into a validated domain model
- Converting API responses to internal representations

## `As[A, B]` — Bidirectional Round-Trip

[`As[A, B]`](as.md) extends `Into[A, B]` with a `from(b: B): Either[SchemaError, A]` reverse direction. It guarantees that `A → B → A` restores the original value (within the constraints of numeric precision and optional fields). Use `As` when both sides of the conversion must remain in sync.

Typical use cases:

- Synchronising a local model with a remote representation
- Persisting to a data format that must be readable back into the same type
- Bridging two live systems that both produce and consume the same data

## Choosing Between `Into` and `As`

| | `Into[A, B]` | `As[A, B]` |
|---|---|---|
| Reverse conversion | ✗ | ✅ `from(b)` |
| Default values on extra fields | ✅ allowed | ✗ not allowed |
| Optional asymmetric fields | ✅ | ✅ (`Option` only) |
| Numeric coercion | ✅ (widening + narrowing) | ✅ (must be invertible) |
| Use for one-way migrations | ✅ | possible but overly strict |
| Use for bidirectional sync | manual | ✅ |

When in doubt, start with `Into`. Upgrade to `As` only when you need the reverse direction and can satisfy its stricter derivation requirements.

---

## Into

`Into[-A, +B]` is a **one-way conversion type class** that converts values of type `A` into values of type `B`, returning `Either[SchemaError, B]` to represent both successful conversions and validation failures. The fundamental operation is `Into#into`, which performs the conversion at runtime.

`Into`:
- is contravariant in `A` and covariant in `B`, following standard type class variance
- returns `Right(b)` on success and `Left(error)` on validation failure
- accumulates multiple field errors into a single `SchemaError`
- derives automatically for case classes, sealed traits, tuples, and Scala 3 enums via `Into.derived`

```scala
trait Into[-A, +B] {
  def into(a: A): Either[SchemaError, B]
}
```

The variance and data flow can be visualised as:

```
  A ──── into ────► Either[SchemaError, B]
  │                       │
  │                  Left(error)    ← validation failure
  │                  Right(b)       ← successful conversion
  │
  Contravariant in A, Covariant in B
```

## Motivation

`Into` solves a common challenge in Scala applications: **type-safe, validated conversion between structurally similar but different types**. This arises in:

- **Schema evolution**: migrating data from an old API version to a new one
- **Domain boundaries**: converting between external DTOs and internal domain models
- **Type refinement**: promoting raw primitives into validated wrapper types
- **Collection reshaping**: converting between `List`, `Vector`, `Set`, `Array`, etc.

Without `Into`, developers write boilerplate conversion code that silently mismatches fields, misses validation, or accumulates errors inconsistently. `Into.derived` generates all of this automatically at compile time.

```scala

case class PersonV1(name: String, age: Int)
case class PersonV2(name: String, age: Long, email: Option[String])

val migrate = Into.derived[PersonV1, PersonV2]
```

With `migrate` derived, converting a `PersonV1` widens `age` to `Long` and defaults `email` to `None`:

```scala
migrate.into(PersonV1("Alice", 30))
// res0: Either[SchemaError, PersonV2] = Right(
//   PersonV2(name = "Alice", age = 30L, email = None)
// )
```

Compare this to a manual implementation:

| Approach          | Field mismatch detection | Error accumulation      | Collection coercion    |
|-------------------|--------------------------|-------------------------|------------------------|
| Manual conversion | ❌ Compile-time miss      | ❌ Requires custom logic | ❌ Requires custom code |
| `Into.derived`    | ✅ Compile-time check     | ✅ Automatic             | ✅ Automatic            |

## Installation

`Into` is part of the `zio-blocks-schema` module:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"
```

For Scala.js:

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-schema" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x.

## Creating Instances

There are four ways to obtain an `Into[A, B]` instance: summon a pre-existing implicit, derive one at compile time via macro, use the built-in identity instance, or implement the trait directly for custom logic.

### `Into.apply` — Summoning

Summons an implicit `Into[A, B]` instance from the implicit scope. This is the standard way to access a pre-existing instance:

```scala
object Into {
  def apply[A, B](implicit ev: Into[A, B]): Into[A, B]
}
```

We summon the pre-existing `Into[Int, Long]` widening instance and call `Into#into` on it:

```scala

val intToLong: Into[Int, Long] = Into[Int, Long]
```

With `intToLong` in scope, `Into#into` converts the value and returns a `Right`:

```scala
intToLong.into(42)
// res1: Either[SchemaError, Long] = Right(42L)
```

### `Into.derived` — Macro Derivation

Generates an `Into[A, B]` instance at compile time using a macro. This is the primary way to convert between case classes, sealed traits, tuples, and enums:

```scala
object Into {
  def derived[A, B]: Into[A, B] // macro
}
```

We derive the conversion between two case classes and observe the `count` field being widened from `Int` to `Long`:

```scala

case class Source(name: String, count: Int)
case class Target(name: String, count: Long)

val conv = Into.derived[Source, Target]
```

The derived `conv` maps each field by name, coercing types where needed:

```scala
conv.into(Source("events", 100))
// res2: Either[SchemaError, Target] = Right(
//   Target(name = "events", count = 100L)
// )
```

### `Into.identity` — Identity Conversion

A pre-provided implicit `Into[A, A]` that always succeeds. It is always in scope and is resolved automatically when source and target types are the same:

```scala
object Into {
  implicit def identity[A]: Into[A, A]
}
```

Any `Into[A, A]` resolves to this built-in — there is nothing to configure:

```scala

val same: Into[String, String] = Into[String, String]
```

The identity conversion always returns `Right` wrapping the original value:

```scala
same.into("hello")
// res3: Either[SchemaError, String] = Right("hello")
```

### Custom Instances

We can implement `Into` manually for any types that need custom conversion logic:

```scala

case class Celsius(value: Double)
case class Fahrenheit(value: Double)

implicit val celsiusToFahrenheit: Into[Celsius, Fahrenheit] =
  (c: Celsius) => Right(Fahrenheit(c.value * 9.0 / 5.0 + 32.0))
```

With `celsiusToFahrenheit` in implicit scope, `Into[Celsius, Fahrenheit]` resolves to it automatically:

```scala
Into[Celsius, Fahrenheit].into(Celsius(100.0))
// res4: Either[SchemaError, Fahrenheit] = Right(Fahrenheit(212.0))
```

## Predefined Instances

ZIO Blocks ships built-in `Into` instances for all standard numeric types and common container types. These are resolved automatically from implicit scope — no import or explicit call is needed.

### Numeric Widening

These instances always succeed. Most cannot lose information, but `Int` to `Float`, `Long` to `Float` and `Long` to `Double` round a value with more significant digits than the target holds: `Into[Int, Float].into(16777217)` gives `Right(1.6777216E7)`.

| From \ To | `Short` | `Int` | `Long` | `Float` | `Double` |
|-----------|---------|-------|--------|---------|----------|
| `Byte`    | ✅       | ✅     | ✅      | ✅       | ✅        |
| `Short`   |         | ✅     | ✅      | ✅       | ✅        |
| `Int`     |         |       | ✅      | ✅       | ✅        |
| `Long`    |         |       |        | ✅       | ✅        |
| `Float`   |         |       |        |         | ✅        |

Each widening conversion always returns `Right`:

```scala
Into[Byte, Int].into(42.toByte)
// res5: Either[SchemaError, Int] = Right(42)
Into[Int, Long].into(100)
// res6: Either[SchemaError, Long] = Right(100L)
Into[Float, Double].into(3.14f)
// res7: Either[SchemaError, Double] = Right(3.140000104904175)
```

### Numeric Narrowing (With Validation)

These instances check at runtime whether the value fits in the target type. They return `Left(SchemaError)` when the value is out of range or cannot be precisely represented:

| From     | To      | Fails when                                         |
|----------|---------|----------------------------------------------------|
| `Short`  | `Byte`  | value outside `[-128, 127]`                        |
| `Int`    | `Byte`  | value outside `[-128, 127]`                        |
| `Int`    | `Short` | value outside `[-32768, 32767]`                    |
| `Long`   | `Byte`  | value outside `[-128, 127]`                        |
| `Long`   | `Short` | value outside `[-32768, 32767]`                    |
| `Long`   | `Int`   | value outside `[Int.MinValue, Int.MaxValue]`       |
| `Double` | `Float` | value outside Float range                          |
| `Float`  | `Int`   | value is not a whole number, or outside Int range  |
| `Float`  | `Long`  | value is not a whole number, or outside Long range |
| `Double` | `Int`   | value is not a whole number, or outside Int range  |
| `Double` | `Long`  | value is not a whole number, or outside Long range |

A value within range returns `Right`; an overflow or fractional value returns `Left`:

```scala
Into[Long, Int].into(42L)
// res8: Either[SchemaError, Int] = Right(42)
Into[Long, Int].into(Long.MaxValue)
// res9: Either[SchemaError, Int] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(ArraySeq()),
//         details = "Value 9223372036854775807 is out of range for Int [-2147483648, 2147483647]",
//         cause = None
//       )
//     )
//   )
// )
Into[Double, Int].into(3.14)
// res10: Either[SchemaError, Int] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(ArraySeq()),
//         details = "Value 3.14 cannot be precisely converted to Int",
//         cause = None
//       )
//     )
//   )
// )
```

### Container Instances

`Into` composes through standard container types automatically:

#### `Option`

`optionInto` lifts an `Into[A, B]` to work over `Option`, coercing the element when present and passing `None` through unchanged:

```scala
implicit def optionInto[A, B](implicit into: Into[A, B]): Into[Option[A], Option[B]]
```

Both `Some` and `None` are handled:

```scala
Into[Option[Int], Option[Long]].into(Some(42))
// res11: Either[SchemaError, Option[Long]] = Right(Some(42L))
Into[Option[Int], Option[Long]].into(None)
// res12: Either[SchemaError, Option[Long]] = Right(None)
```

#### `Either`

`eitherInto` coerces both branches independently, requiring separate `Into` instances for the left and right types:

```scala
implicit def eitherInto[L1, R1, L2, R2](
  implicit leftInto: Into[L1, L2],
  rightInto: Into[R1, R2]
): Into[Either[L1, R1], Either[L2, R2]]
```

Both `Left` and `Right` branches are coerced independently:

```scala
Into[Either[Int, Int], Either[Long, Long]].into(Right(1))
// res13: Either[SchemaError, Either[Long, Long]] = Right(Right(1L))
Into[Either[Int, Int], Either[Long, Long]].into(Left(2))
// res14: Either[SchemaError, Either[Long, Long]] = Right(Left(2L))
```

#### `Map`

`mapInto` coerces both keys and values, requiring separate `Into` instances for each:

```scala
implicit def mapInto[K1, V1, K2, V2](
  implicit keyInto: Into[K1, K2],
  valueInto: Into[V1, V2]
): Into[Map[K1, V1], Map[K2, V2]]
```

Both keys and values are coerced element-by-element:

```scala
Into[Map[String, Int], Map[String, Long]].into(Map("a" -> 1, "b" -> 2))
// res15: Either[SchemaError, Map[String, Long]] = Right(
//   Map("a" -> 1L, "b" -> 2L)
// )
```

#### Iterables and Arrays

Four overloads cover all combinations of `Iterable` subtypes and arrays as source or target:

```scala
implicit def iterableInto[A, B, F1[X] <: Iterable[X], F2[_]](
  implicit intoAB: Into[A, B],
  factory: Factory[B, F2[B]]
): Into[F1[A], F2[B]]

implicit def arrayToIterable[A, B, F[_]](
  implicit intoAB: Into[A, B],
  factory: Factory[B, F[B]]
): Into[Array[A], F[B]]

implicit def iterableToArray[A, B, F[X] <: Iterable[X]](
  implicit intoAB: Into[A, B],
  ct: ClassTag[B]
): Into[F[A], Array[B]]

implicit def arrayToArray[A, B](
  implicit intoAB: Into[A, B],
  ct: ClassTag[B]
): Into[Array[A], Array[B]]
```

The source and target collection kinds are independent — elements are coerced individually and the target collection is built using its factory:

```scala
Into[List[Int], Vector[Long]].into(List(1, 2, 3))
// res16: Either[SchemaError, Vector[Long]] = Right(Vector(1L, 2L, 3L))
Into[List[Int], Set[Long]].into(List(1, 2, 2, 3))
// res17: Either[SchemaError, Set[Long]] = Right(Set(1L, 2L, 3L))
```

:::note
Converting to `Set` removes duplicates. Converting from `Set` does not guarantee any particular element order.
:::

## Core Operation

`Into` exposes a single abstract method, `Into#into`. All predefined instances, derived instances, and custom implementations reduce to this one operation. It performs the conversion from `A` to `B`, returning a `Right` on success or a `Left` with a `SchemaError` on failure.

```scala
trait Into[-A, +B] {
  def into(a: A): Either[SchemaError, B]
}
```

We derive an `Into[Raw, Narrow]` to show both the success path and the overflow path:

```scala

case class Raw(value: Long)
case class Narrow(value: Int)

val conv = Into.derived[Raw, Narrow]
```

A value that fits in `Int` returns `Right`; a value that overflows returns `Left`:

```scala
conv.into(Raw(42L))
// res18: Either[SchemaError, Narrow] = Right(Narrow(42))
conv.into(Raw(Long.MaxValue))
// res19: Either[SchemaError, Narrow] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(IndexedSeq()),
//         details = "converting field Raw.value to Narrow.value failed",
//         cause = Some(
//           SchemaError(
//             List(
//               ConversionFailed(
//                 source = DynamicOptic(ArraySeq()),
//                 details = "Value 9223372036854775807 is out of range for Int [-2147483648, 2147483647]",
//                 cause = None
//               )
//             )
//           )
//         )
//       )
//     )
//   )
// )
```

## Macro Derivation Rules

`Into.derived[A, B]` generates a conversion by matching fields from `A` to `B` using the following priority:

1. **Exact match**: same field name and same type
2. **Name match with coercion**: same name, types connected by an implicit `Into` (e.g. `Int` → `Long`)
3. **Unique type match**: the type appears exactly once in both `A` and `B`
4. **Position + type match**: fields in the same position with matching types

### Products (Case Classes and Tuples)

Fields are matched by name first; when names differ but types are unique across both types, unique-type matching kicks in:

```scala

case class Source(firstName: String, count: Int)
case class Target(label: String, total: Long)
```

Because `String` and `Long` each appear uniquely, the macro resolves `firstName` → `label` and `count` → `total`:

```scala
Into.derived[Source, Target].into(Source("events", 5))
// res20: Either[SchemaError, Target] = Right(
//   Target(label = "events", total = 5L)
// )
```

Tuples and case classes are interchangeable when their arities and element types match:

```scala

case class Point(x: Int, y: Int)
```

The macro treats a two-element tuple and a two-field case class as structurally equivalent:

```scala
Into.derived[(Int, Int), Point].into((3, 4))
// res21: Either[SchemaError, Point] = Right(Point(x = 3, y = 4))
Into.derived[Point, (Int, Int)].into(Point(3, 4))
// res22: Either[SchemaError, Tuple2[Int, Int]] = Right((3, 4))
```

Target fields missing from the source default to `None` for `Option` types and to their declared default value otherwise:

```scala

case class Source(name: String)
case class Target(name: String, nickname: Option[String], score: Int = 0)
```

Missing fields are filled with `None` or their declared defaults — no extra code is needed:

```scala
Into.derived[Source, Target].into(Source("Alice"))
// res23: Either[SchemaError, Target] = Right(
//   Target(name = "Alice", nickname = None, score = 0)
// )
```

For nested case classes, the macro automatically picks up implicit `Into` instances for the nested types. Defining the inner conversion as an implicit is enough — the outer derivation uses it automatically:

```scala

case class AddressV1(street: String, zip: Int)
case class AddressV2(street: String, zip: Long)

case class PersonV1(name: String, address: AddressV1)
case class PersonV2(name: String, address: AddressV2)

implicit val addressConv: Into[AddressV1, AddressV2] =
  Into.derived[AddressV1, AddressV2]

val personConv = Into.derived[PersonV1, PersonV2]
```

The `personConv` conversion delegates the `address` field to `addressConv` without any extra wiring:

```scala
personConv.into(PersonV1("Alice", AddressV1("123 Main St", 10001)))
// res24: Either[SchemaError, PersonV2] = Right(
//   PersonV2(
//     name = "Alice",
//     address = AddressV2(street = "123 Main St", zip = 10001L)
//   )
// )
```

### Coproducts (Sealed Traits and Enums)

Cases are matched by name; for case classes, field types must be convertible. Target coproducts may introduce new cases that are unreachable from the source. A source case with no target case of the same name is not an error: it goes to the first target case whose fields, position by position, have the same types or types that convert (an implicit `Into`, and on Scala 3 also an opaque type or a newtype; case objects skip this step) or, failing that, to the target case at the same position, so a renamed or removed case can convert to an unrelated one. The macro fails only when no target case is found at all.

For case class variants, fields are coerced just like in product derivation:

```scala

sealed trait ShapeV1
object ShapeV1 {
  case class Circle(radius: Int) extends ShapeV1
  case class Square(side: Int) extends ShapeV1
}

sealed trait ShapeV2
object ShapeV2 {
  case class Circle(radius: Long) extends ShapeV2
  case class Square(side: Long) extends ShapeV2
}

val conv = Into.derived[ShapeV1, ShapeV2]
```

Each case is matched by name and its fields are coerced from `Int` to `Long`:

```scala
conv.into(ShapeV1.Circle(5))
conv.into(ShapeV1.Square(3))
```

For `case object` variants (no fields), the macro matches by name alone. New cases may be added to the target without affecting derivation:

```scala

sealed trait StatusV1
object StatusV1 {
  case object Active   extends StatusV1
  case object Inactive extends StatusV1
}

sealed trait StatusV2
object StatusV2 {
  case object Active   extends StatusV2
  case object Inactive extends StatusV2
  case object Pending  extends StatusV2  // new in V2 — unreachable from V1
}

val conv = Into.derived[StatusV1, StatusV2]
```

Each source case object maps to the identically-named target case object:

```scala
conv.into(StatusV1.Active)
conv.into(StatusV1.Inactive)
```

### ZIO Prelude Newtypes

`Into.derived` automatically detects ZIO Prelude `Newtype` and `Subtype` definitions and validates values through their smart constructors. The syntax for defining the assertion differs between Scala versions.

**Scala 2:**

```scala
object Age extends Subtype[Int] {
  override def assertion = assert {
    between(0, 150)
  }
}
```

**Scala 3:**

```scala
object Age extends Subtype[Int] {
  override def assertion: Assertion[Int] =
    zio.prelude.Assertion.between(0, 150)
}
```

The Scala 3 form is used in the mdoc examples below:

```scala

object Age extends Subtype[Int] {
  override def assertion: zio.prelude.Assertion[Int] =
    zio.prelude.Assertion.between(0, 150)
}
type Age = Age.Type

case class PersonRaw(name: String, age: Int)
case class PersonValidated(name: String, age: Age)

val validate = Into.derived[PersonRaw, PersonValidated]
```

Values within the assertion range succeed; out-of-range values return a `Left` from the smart constructor:

```scala
validate.into(PersonRaw("Alice", 30))
// res26: Either[SchemaError, PersonValidated] = Right(
//   PersonValidated(name = "Alice", age = 30)
// )
validate.into(PersonRaw("Bob", 200))
// res27: Either[SchemaError, PersonValidated] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(IndexedSeq()),
//         details = "converting field PersonRaw.age to PersonValidated.age failed",
//         cause = Some(
//           SchemaError(
//             List(
//               ConversionFailed(
//                 source = DynamicOptic(ArraySeq()),
//                 details = "Validation failed for field 'age': NonEmptyChunk(200 did not satisfy between(0, 150))",
//                 cause = None
//               )
//             )
//           )
//         )
//       )
//     )
//   )
// )
```

### Scala 3 Opaque Types

In Scala 3, `Into.derived` detects opaque types with companion `apply` or `unsafe` methods:

```scala

opaque type Email = String
object Email {
  def apply(s: String): Either[String, Email] =
    if (s.contains("@")) Right(s) else Left(s"Invalid email: $s")
  def unsafe(s: String): Email = s
}

case class UserRaw(name: String, email: String)
case class UserValidated(name: String, email: Email)

val validate = Into.derived[UserRaw, UserValidated]
```

A valid email address succeeds; an invalid one returns the error produced by the `apply` smart constructor:

```scala
validate.into(UserRaw("Alice", "alice@example.com"))
// res29: Either[SchemaError, UserValidated] = Right(
//   UserValidated(name = "Alice", email = "alice@example.com")
// )
validate.into(UserRaw("Alice", "not-an-email"))
// res30: Either[SchemaError, UserValidated] = Right(
//   UserValidated(name = "Alice", email = "not-an-email")
// )
```

The macro looks for `apply(value: Underlying): Either[_, OpaqueType]` first, then falls back to `unsafe(value: Underlying): OpaqueType`.

### Structural Types (JVM Only)

On JVM, `Into.derived` supports structural types (types defined by their members rather than their name). This is not available on Scala.js or Scala Native because structural type access requires runtime reflection.

| Conversion              | JVM | JS/Native |
|-------------------------|-----|-----------|
| Structural → Product    | ✅  | ❌         |
| Product → Structural    | ✅  | ❌         |

On non-JVM platforms, `Into.derived` fails at compile time with a descriptive message:

```
Cannot derive Into[..., Person]: Structural type conversions are not supported on JS.
Structural types require reflection APIs which are only available on JVM.
Consider using a case class or tuple instead.
```

On JVM, we use `scala.language.reflectiveCalls` and create the structural instance at the call site:

```scala
// JVM ONLY — structural types require reflection

def makePerson(n: String, a: Int): { def name: String; def age: Int } = new {
  def name: String = n
  def age: Int = a
}

case class Person(name: String, age: Int)

val into = Into.derived[{ def name: String; def age: Int }, Person]
// into.into(makePerson("Alice", 30)) == Right(Person("Alice", 30))
```

:::warning
For cross-platform code, replace structural types with case classes or tuples.
:::

## Error Handling

All `Into` conversions return `Either[SchemaError, B]`. `SchemaError` carries:

- A human-readable message via `.message` / `.getMessage`
- The field path where the failure occurred
- Accumulated errors from multiple failing fields

```scala

case class Source(a: Long, b: Long, c: Long)
case class Target(a: Int,  b: Int,  c: Int)

val conv    = Into.derived[Source, Target]
val result  = conv.into(Source(Long.MaxValue, Long.MinValue, 42L))
```

We pattern-match on the result to print either the converted value or the accumulated error message:

```scala
result match {
  case Right(t)    => println(s"OK: $t")
  case Left(error) => println(s"Failed:\n${error.message}")
}
// Failed:
// converting field Source.b to Target.b failed
//   Caused by: Value -9223372036854775808 is out of range for Int [-2147483648, 2147483647]
// converting field Source.a to Target.a failed
//   Caused by: Value 9223372036854775807 is out of range for Int [-2147483648, 2147483647]
```

When multiple fields fail, all errors are collected and reported together. The field `c` above succeeds (42 fits in `Int`), so only errors for `a` and `b` appear.

```scala

case class UserRaw(id: Long, email: String, age: Long)

opaque type PositiveId = Long
object PositiveId {
  def apply(n: Long): Either[String, PositiveId] =
    if (n > 0) Right(n) else Left(s"id must be positive, got $n")
  def unsafe(n: Long): PositiveId = n
}

opaque type Email = String
object Email {
  def apply(s: String): Either[String, Email] =
    if (s.contains("@")) Right(s) else Left(s"Invalid email: $s")
  def unsafe(s: String): Email = s
}

case class UserValidated(id: PositiveId, email: Email, age: Int)

val conv = Into.derived[UserRaw, UserValidated]
val res = conv.into(UserRaw(-1L, "not-an-email", 200L))
```

All three field errors are accumulated into a single `SchemaError` with a combined message:

```scala
res match {
  case Left(error) => println(s"Validation failed: ${error.message}")
  case Right(value) => println(s"Validation succeeded: $value")
} 
// Validation succeeded: UserValidated(-1,not-an-email,200)
```

## DynamicValue Conversions

`Into` has special macro support for converting any type with a `Schema` to or from `DynamicValue`, a semi-structured data representation. This is the primary way to achieve polyglot data handling—converting between type-safe domain models and formats like JSON, Avro, or Protobuf.

### Converting to DynamicValue and JSON

The simplest way to convert to DynamicValue and view as JSON:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
  val toDynamic: Into[Person, DynamicValue] = Into.derived[Person, DynamicValue]
}
```

```scala
Person.toDynamic.into(Person("Alice", 30)).map(_.toJsonString)
// res34: Either[SchemaError, String] = Right(
//   "{\"name\":\"Alice\",\"age\":30}"
// )
```

The `toJsonString` method on `DynamicValue` provides a human-readable JSON representation (Extended JSON format with type annotations). The conversion uses `Schema[A].toDynamicValue` internally, ensuring consistency with how the type is serialized to other formats.

**Why this matters:**

- **Format independence**: Convert typed data once to DynamicValue, then serialize to any format (JSON, Avro, MessagePack, etc.)
- **Dynamic pipelines**: Accept or produce semi-structured data in systems that don't have compile-time type information
- **Schema-driven workflows**: Use the same schema definition for both type-safe operations and dynamic transformations

### Converting from DynamicValue with Round-Trip

Given a `DynamicValue` with a matching structure, convert it back to a strongly-typed value:

```scala

case class Person(name: String, age: Int)

val fromDynamic = Into.derived[DynamicValue, Person]
val dv = DynamicValue.Record(
  "name" -> DynamicValue.string("Bob"),
  "age"  -> DynamicValue.int(25)
)
val result = fromDynamic.into(dv)
```

The conversion completes successfully:

```scala
result
// res35: Either[SchemaError, Person] = Right(Person(name = "Bob", age = 25))
```

Conversion fails gracefully if the structure doesn't match:

```scala

case class Person(name: String, age: Int)

val fromDynamic = Into.derived[DynamicValue, Person]
val badDV = DynamicValue.Primitive(PrimitiveValue.String("not a record"))
val result = fromDynamic.into(badDV)
```

```scala
result
// res36: Either[SchemaError, Person] = Left(
//   SchemaError(
//     List(
//       ExpectationMismatch(
//         source = DynamicOptic(ArraySeq()),
//         expectation = "Expected a record"
//       )
//     )
//   )
// )
```

### Collections and DynamicValue Round-Trip

Conversions work seamlessly through collections. Here's a complete round-trip:

```scala

case class Item(id: Int, name: String)

val listToDynamic = Into.derived[List[Item], DynamicValue]
val listFromDynamic = Into.derived[DynamicValue, List[Item]]

val items = List(Item(1, "A"), Item(2, "B"))

// Forward: List[Item] → DynamicValue
val asDV = listToDynamic.into(items)

// Round-trip: DynamicValue → List[Item]
val backToList = asDV.flatMap(listFromDynamic.into)
```

The round-trip restores the original data:

```scala
backToList
// res37: Either[SchemaError, List[Item]] = Right(
//   List(Item(id = 1, name = "A"), Item(id = 2, name = "B"))
// )
```

Similarly for maps:

```scala

val mapToDynamic = Into.derived[Map[String, Int], DynamicValue]
val mapFromDynamic = Into.derived[DynamicValue, Map[String, Int]]

val data = Map("count" -> 42, "total" -> 100)

val asDV = mapToDynamic.into(data)
val backToMap = asDV.flatMap(mapFromDynamic.into)
```

```scala
backToMap
// res38: Either[SchemaError, Map[String, Int]] = Right(
//   Map("count" -> 42, "total" -> 100)
// )
```

## Related Type: `As[A, B]`

`As[A, B]` extends `Into[A, B]` with a reverse direction, enabling round-trip safe bidirectional conversions. Because `As` must restore the original value for every field the two types share, it applies stricter derivation constraints than `Into`; an `Option` field that only one side has comes back as `None`. See [As](as.md) for the full reference.

## Best Practices

Following a few conventions avoids common pitfalls when working with `Into` and `As`.

**Prefer `As` when round-trip correctness is required.** For data sync or bidirectional serialization, use `As`. For one-way migrations or API responses, use `Into`:

```scala

case class LocalModel(id: Long, name: String)
case class RemoteModel(id: Long, name: String)

case class OldFormat(value: Int)
case class NewFormat(value: Long)

val sync: As[LocalModel, RemoteModel]    = As.derived    // round-trip
val migrate: Into[OldFormat, NewFormat]  = Into.derived  // one-way
```

**Use `Option` for truly optional fields, not default values.** Default values prevent `As.derived` when the field is absent from the other type; `Option` always works:

```scala

// Good — Option works with both Into and As
case class V2Good(name: String, email: Option[String])

// Risky — default value prevents As derivation when field is absent from the other side
case class V2Risky(name: String, email: String = "")
```

**Provide explicit implicits for complex nested types.** When nested types need custom validation logic, define the inner `Into` as an implicit before deriving the outer one:

```scala

case class AddressV1(street: String, zip: Int)
case class AddressV2(street: String, zip: Long, country: String = "US")

case class PersonV1(name: String, address: AddressV1)
case class PersonV2(name: String, address: AddressV2)

implicit val addressMigrate: Into[AddressV1, AddressV2] =
  Into.derived[AddressV1, AddressV2]

val personMigrate: Into[PersonV1, PersonV2] =
  Into.derived[PersonV1, PersonV2]  // picks up addressMigrate automatically
```

## Advanced Usage

The real power of `Into` emerges in multi-version schema evolution scenarios where types gain new fields, change numeric precision, and introduce new coproduct cases simultaneously. The following example migrates a two-level object graph from V1 to V2:

```scala

object V1 {
  case class Address(street: String, city: String)
  case class Person(name: String, age: Int, address: Address)
}

object V2 {
  case class Address(street: String, city: String, country: String = "US")
  case class Person(
    name: String,
    age: Long,                  // widened from Int
    address: Address,
    email: Option[String]       // new optional field
  )
}

implicit val addressMigrate: Into[V1.Address, V2.Address] =
  Into.derived[V1.Address, V2.Address]

val personMigrate: Into[V1.Person, V2.Person] =
  Into.derived[V1.Person, V2.Person]
```

A V1 record converts to V2 in one call — all defaults, widenings, and nested conversions are applied automatically:

```scala
val oldPerson = V1.Person("Alice", 30, V1.Address("123 Main St", "NYC"))
// oldPerson: Person = Person(
//   name = "Alice",
//   age = 30,
//   address = Address(street = "123 Main St", city = "NYC")
// )
personMigrate.into(oldPerson)
// res42: Either[SchemaError, Person] = Right(
//   Person(
//     name = "Alice",
//     age = 30L,
//     address = Address(street = "123 Main St", city = "NYC", country = "US"),
//     email = None
//   )
// )
```

## Scala 2 vs Scala 3 Differences

| Feature | Scala 2 | Scala 3 |
|---------|---------|---------|
| Derivation syntax | `Into.derived[A, B]` | `Into.derived[A, B]` |
| Enum support | Sealed traits only | Scala 3 enums + sealed traits |
| Opaque types | N/A | ✅ Supported |
| Structural types | JVM only (reflection) | JVM only (reflection) |
| ZIO Prelude newtypes | ✅ `assert { between(...) }` | ✅ `override def assertion` |
| Error messages | Detailed macro errors | Detailed macro errors |

## See Also

- [Evolve Your Data](../../../guides/evolve-your-data.md) - a tutorial that validates incoming data with `Into`, converts between versions with `As` and renames a field with a `Migration`.

---

## SchemaExpr

`SchemaExpr[A, B]` is a **schema-aware expression** that computes a result of type `B` from an input value of type `A`. It is invariant in both type parameters. The input type `A` must be fully described by a [`Schema`](./schema.md), and the expression is built from [optics](./optics.md), literal values, and operators. The fundamental operations are `eval` and `evalDynamic`.

At runtime, `SchemaExpr` is a typed wrapper around `DynamicSchemaExpr`. The typed layer carries the input and output schemas, while the dynamic layer stores the serializable AST. This split is why you will sometimes see both `.dynamic` and `DynamicSchemaExpr` in advanced examples: `SchemaExpr` is the public typed API, and `DynamicSchemaExpr` is the untyped transport/runtime form underneath it.

`SchemaExpr`:
- represents expressions as a reified AST, enabling introspection and serialization
- supports relational (`===`, `>`, `<`, `>=`, `<=`, `!=`), logical (`&&`, `||`, `!`), arithmetic (`+`, `-`, `*`), and string (`concat`, `matches`, `length`) operations
- evaluates to `Either[OpticCheck, Seq[B]]`, handling failures and multi-valued results from traversals
- preserves both input and output schemas at the type level

```scala
final case class SchemaExpr[A, B](
  dynamic: DynamicSchemaExpr,
  inputSchema: Schema[A],
  outputSchema: Schema[B]
) {
  def eval[B1 >: B](input: A)(implicit schema: Schema[B1]): Either[OpticCheck, Seq[B1]]
  def evalDynamic(input: A): Either[OpticCheck, Seq[DynamicValue]]
}
```

:::tip
For practical walkthroughs of building with `SchemaExpr`, see [Query DSL Part 1: Expressions](../../guides/query-dsl-reified-optics.md), [Part 2: SQL Generation](../../guides/query-dsl-sql.md), and [Part 3: Extending the Expression Language](../../guides/query-dsl-extending.md).
:::

## Motivation

When working with schema-described data, we often need to express computations over that data — comparisons, arithmetic, string operations — in a way that can be both **evaluated at runtime** and **inspected as data**. This is essential for:

1. **Persistence DSLs** — Third-party libraries can translate `SchemaExpr` trees into SQL `WHERE` clauses, NoSQL filters, or other query languages, because the expression structure is reified (not opaque functions).
2. **Validation** — Express constraints like "age must be greater than 18" or "name must match a pattern" as composable, inspectable expressions.
3. **Data Migration** — Define transformation rules that can be analyzed and optimized before execution.

```text
                              SchemaExpr[A, B]
                                     │
          ┌──────────┬───────────────┼───────────────┬──────────────────┐
          │          │               │               │                  │
     Leaf Nodes   Unary Ops     Binary Ops    StringRegexMatch   StringLength
          │          │               │
    ┌─────┴─────┐   Not    ┌────────┼────────┐
  Literal    Optic       Relational Logical  Arithmetic
                                             StringConcat
```

The typical way to build expressions is through the operator syntax on [Optic](./optics.md) values:

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String] = $(_.name)
  val age: Lens[Person, Int]     = $(_.age)
}

// Build expressions using optic operators
val isAdult: SchemaExpr[Person, Boolean]    = Person.age >= 18
val isAlice: SchemaExpr[Person, Boolean]    = Person.name === "Alice"
val combined: SchemaExpr[Person, Boolean]   = isAdult && isAlice

// Evaluate against a value
val alice = Person("Alice", 30)
val result: Either[OpticCheck, Seq[Boolean]] = combined.eval(alice)
// Right(Seq(true))
```

## Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"
```

For cross-platform (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-schema" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x.

## Creating Instances

`SchemaExpr` instances are typically created through **operator syntax on optics** rather than by constructing AST nodes directly. Each operator on `Optic[S, A]` returns a `SchemaExpr[S, B]`.

There are three common construction styles:

- optic/operator syntax such as `Person.age >= 18`
- literal expressions such as `SchemaExpr.literal[Person, Int](18)`
- advanced direct construction when you intentionally need to work with the underlying dynamic AST

### Via Relational Operators on Optics

The comparison operators `===`, `>`, `>=`, `<`, `<=`, and `!=` on `Optic[S, A]` create `SchemaExpr.Relational` nodes. Each operator has two overloads — one comparing against a literal value, and one comparing against another optic:

```scala

case class Product(name: String, price: Double, stock: Int)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]  = $(_.name)
  val price: Lens[Product, Double] = $(_.price)
  val stock: Lens[Product, Int]    = $(_.stock)
}

// Compare optic against a literal value
val expensive: SchemaExpr[Product, Boolean] = Product.price > 100.0
val inStock: SchemaExpr[Product, Boolean]   = Product.stock > 0
val named: SchemaExpr[Product, Boolean]     = Product.name === "Widget"

// Compare optic against another optic
// (e.g., stock > price — contrived, but shows the syntax)
```

### Via Logical Operators on Optics

The `&&`, `||`, and `!` (unary) operators on boolean-focused optics create `SchemaExpr.Logical` and `SchemaExpr.Not` nodes:

```scala

case class User(name: String, active: Boolean, verified: Boolean)

object User extends CompanionOptics[User] {
  implicit val schema: Schema[User] = Schema.derived

  val name: Lens[User, String]       = $(_.name)
  val active: Lens[User, Boolean]    = $(_.active)
  val verified: Lens[User, Boolean]  = $(_.verified)
}

// Logical operators on boolean optics
val activeAndVerified: SchemaExpr[User, Boolean] = User.active && User.verified
val eitherOne: SchemaExpr[User, Boolean]         = User.active || User.verified
val notActive: SchemaExpr[User, Boolean]         = !User.active
```

### Via Arithmetic Operators on Optics

The `+`, `-`, and `*` operators on numeric-focused optics create `SchemaExpr.Arithmetic` nodes. These require an implicit `IsNumeric[A]` instance, which is provided for `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, and `BigDecimal`:

```scala

case class Order(quantity: Int, unitPrice: Double)

object Order extends CompanionOptics[Order] {
  implicit val schema: Schema[Order] = Schema.derived

  val quantity : Lens[Order, Int]     = $(_.quantity)
  val unitPrice: Lens[Order, Double]  = $(_.unitPrice)
}

// Arithmetic on optic values
val doubled   : SchemaExpr[Order, Int]    = Order.quantity * 2
val discounted: SchemaExpr[Order, Double] = Order.unitPrice - 5.0
val increased : SchemaExpr[Order, Int]    = Order.quantity + 1
```

### Via String Operators on Optics

The `concat`, `matches`, and `length` methods on string-focused optics create `SchemaExpr.StringConcat`, `SchemaExpr.StringRegexMatch`, and `SchemaExpr.StringLength` nodes:

```scala

case class Email(address: String, subject: String)

object Email extends CompanionOptics[Email] {
  implicit val schema: Schema[Email] = Schema.derived

  val address: Lens[Email, String] = $(_.address)
  val subject: Lens[Email, String] = $(_.subject)
}

// String operations
val withDomain: SchemaExpr[Email, String]   = Email.address.concat("@example.com")
val isValid   : SchemaExpr[Email, Boolean]  = Email.address.matches("^[^@]+@[^@]+$")
val subjectLen: SchemaExpr[Email, Int]      = Email.subject.length
```

### Via Logical Operators on SchemaExpr

Boolean-typed `SchemaExpr` values can be combined with `&&` and `||`:

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String] = $(_.name)
  val age: Lens[Person, Int]     = $(_.age)
}

// Compose expressions with && and ||
val isAdult = Person.age >= 18
val isAlice = Person.name === "Alice"

val adultAlice  : SchemaExpr[Person, Boolean] = isAdult && isAlice
val adultOrAlice: SchemaExpr[Person, Boolean] = isAdult || isAlice
```

### Via Direct AST Construction

For advanced use cases, you can construct `SchemaExpr` values directly. In most application code, prefer optic/operator syntax or `SchemaExpr.literal` because they preserve the typed surface and are easier to read.

```scala

case class Item(price: Int)

object Item extends CompanionOptics[Item] {
  implicit val schema: Schema[Item] = Schema.derived

  val price: Lens[Item, Int] = $(_.price)
}

// Construct via factory methods
val lit: SchemaExpr[Item, Int] = SchemaExpr.literal[Item, Int](42)
val opticExpr: SchemaExpr[Item, Int] = SchemaExpr.optic[Item, Int](Item.price.toDynamic, Item.schema)
val comparison: SchemaExpr[Item, Boolean] = SchemaExpr.relational(
  opticExpr,
  lit,
  SchemaExpr.RelationalOperator.GreaterThan
)
```

`SchemaExpr.literal` is the normal way to inject constants into an expression tree. It produces a typed `SchemaExpr[S, A]`, while storing the value internally as a `DynamicSchemaExpr.Literal`. That is usually what you want in user code, including migration builders.

If you need the raw dynamic form for serialization, transport, or custom interpreters, use `.dynamic` on an existing `SchemaExpr` rather than constructing `DynamicSchemaExpr` directly unless you are working on expression internals.

## Core Operations

### Evaluation

#### `eval`

Evaluates the expression against an input value, returning the typed result. The result is a `Seq[B]` because traversal-based expressions can produce multiple values.

```scala
final case class SchemaExpr[A, B](...) {
  def eval[B1 >: B](input: A)(implicit schema: Schema[B1]): Either[OpticCheck, Seq[B1]]
}
```

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  val age: Lens[Person, Int] = $(_.age)
}

val isAdult = Person.age >= 18

val alice = Person("Alice", 30)
val result = isAdult.eval(alice)
// Right(List(true))

val bob = Person("Bob", 12)
val result2 = isAdult.eval(bob)
// Right(List(false))
```

:::note
When an expression wraps a `Traversal` optic, `eval` returns multiple values — one per element in the traversed collection. For `Lens`-based expressions, the result is always a single-element `Seq`.
:::

#### `evalDynamic`

Like `eval`, but converts the result to [`DynamicValue`](./dynamic-value.md) instances. This is useful for serialization or when working with schema-agnostic code.

```scala
final case class SchemaExpr[A, B](...) {
  def evalDynamic(input: A): Either[OpticCheck, Seq[DynamicValue]]
}
```

In the following example we are evaluating a simple optic expression to extract the `name` field from a `Person` and retrieving it as a `DynamicValue`:

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String] = $(_.name)
}

val nameExpr = SchemaExpr.optic[Person, String](Person.name.toDynamic, Person.schema)
val result = nameExpr.evalDynamic(Person("Alice", 30))
// Right(List(DynamicValue.Primitive(PrimitiveValue.String("Alice"))))
```

### Logical Combination

#### `&&`

Combines two boolean-typed expressions with logical AND. Both operands must produce `Boolean` results.

`&&` and `||` are provided via `SchemaExpr.BooleanOps`, an implicit class in the `SchemaExpr` companion, rather than as direct methods on `SchemaExpr`. This lets downstream libraries supply their own `&&`/`||` overloads (e.g. returning a DDB- or SQL-specific expression type) by bringing in their own implicit class via an explicit import, which takes priority over the companion-scope implicit in Scala 2 and 3.

```scala
// In SchemaExpr companion:
implicit final class BooleanOps[A, B](val self: SchemaExpr[A, B]) extends AnyVal {
  def and(that: SchemaExpr[A, Boolean])(implicit ev: B <:< Boolean): SchemaExpr[A, Boolean]
  def &&(that: SchemaExpr[A, Boolean])(implicit ev: B <:< Boolean): SchemaExpr[A, Boolean]
}
```

In the following example we are evaluating a combined expression that checks if a `Person` is an adult (age >= 18) and has the name "Alice". The result of evaluating this expression against a `Person` instance will be `true` if both conditions are met, and `false` otherwise:

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String] = $(_.name)
  val age: Lens[Person, Int]     = $(_.age)
}

val isAdultAlice = (Person.age >= 18) && (Person.name === "Alice")
val result = isAdultAlice.eval(Person("Alice", 30))
// Right(List(true))
```

#### `||`

Combines two boolean-typed expressions with logical OR.

```scala
// In SchemaExpr companion:
implicit final class BooleanOps[A, B](val self: SchemaExpr[A, B]) extends AnyVal {
  def or(that: SchemaExpr[A, Boolean])(implicit ev: B <:< Boolean): SchemaExpr[A, Boolean]
  def ||(that: SchemaExpr[A, Boolean])(implicit ev: B <:< Boolean): SchemaExpr[A, Boolean]
}
```

In the following example we are evaluating a combined expression that checks if a `Person` is an adult (age >= 18) or has the name "Alice". The result of evaluating this expression against a `Person` instance will be `true` if either condition is met, and `false` only if both conditions are not met:

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String] = $(_.name)
  val age: Lens[Person, Int]     = $(_.age)
}

val isAdultOrAlice = (Person.age >= 18) || (Person.name === "Alice")
val result = isAdultOrAlice.eval(Person("Alice", 12))
// Right(List(true)) — Alice, even though not adult
```

## Structure

`SchemaExpr` is a typed wrapper. The actual expression AST lives in `DynamicSchemaExpr`, which is a sealed trait of serializable expression nodes. `SchemaExpr` adds the input and output schemas needed to convert between typed values and [`DynamicValue`](./dynamic-value.md).

### `DynamicSchemaExpr` leaf nodes

#### `DynamicSchemaExpr.Literal`

A constant value represented directly as a `DynamicValue`.

```scala
object DynamicSchemaExpr {
  final case class Literal(value: DynamicValue, schema: Schema[_]) extends DynamicSchemaExpr
}
```

#### `DynamicSchemaExpr.Select`

Selects values from the input using a [`DynamicOptic`](./dynamic-optic.md).

```scala
object DynamicSchemaExpr {
  final case class Select(path: DynamicOptic) extends DynamicSchemaExpr
}
```

`SchemaExpr.optic(...)` and optic operator syntax eventually produce `DynamicSchemaExpr.Select` nodes.

### Unary operations

#### `DynamicSchemaExpr.Not`

Negates a boolean expression.

```scala
object DynamicSchemaExpr {
  final case class Not(expr: DynamicSchemaExpr) extends DynamicSchemaExpr
}
```

Created via the `!` (unary negation) operator on boolean optics:

```scala

case class User(active: Boolean)

object User extends CompanionOptics[User] {
  implicit val schema: Schema[User] = Schema.derived

  val active: Lens[User, Boolean] = $(_.active)
}

val inactive: SchemaExpr[User, Boolean] = !User.active

val result = inactive.eval(User(active = true))
// Right(List(false))
```

### Binary operations

`Relational`, `Logical`, `Arithmetic`, `Bitwise`, and the string binary operations are all represented as `DynamicSchemaExpr` nodes holding child expressions.

#### `DynamicSchemaExpr.Relational`

Compares two expressions using a `RelationalOperator`. Returns a boolean result.

```scala
object DynamicSchemaExpr {
  final case class Relational(
    left: DynamicSchemaExpr,
    right: DynamicSchemaExpr,
    operator: RelationalOperator
  ) extends DynamicSchemaExpr
}
```

The available `RelationalOperator` values are:

| Operator             | Optic Syntax | Description           |
|----------------------|--------------|-----------------------|
| `Equal`              | `===`        | Equality check        |
| `NotEqual`           | `!=`         | Inequality check      |
| `LessThan`           | `<`          | Less than             |
| `LessThanOrEqual`    | `<=`         | Less than or equal    |
| `GreaterThan`        | `>`          | Greater than          |
| `GreaterThanOrEqual` | `>=`         | Greater than or equal |

:::note
Equality and inequality (`===`, `!=`) compare values directly. Ordering operators (`<`, `<=`, `>`, `>=`) compare via `DynamicValue` ordering internally.
:::

#### `DynamicSchemaExpr.Logical`

Combines two boolean expressions with a `LogicalOperator`.

```scala
object DynamicSchemaExpr {
  final case class Logical(
    left: DynamicSchemaExpr,
    right: DynamicSchemaExpr,
    operator: LogicalOperator
  ) extends DynamicSchemaExpr
}
```

The available `LogicalOperator` values are:

| Operator | Syntax | Description         |
|----------|--------|---------------------|
| `And`    | `&&`   | Logical conjunction |
| `Or`     | `\|\|` | Logical disjunction |

#### `DynamicSchemaExpr.Arithmetic`

Performs arithmetic on two numeric expressions using an `ArithmeticOperator`. Requires an `IsNumeric[A]` type class instance.

```scala
object DynamicSchemaExpr {
  final case class Arithmetic(
    left: DynamicSchemaExpr,
    right: DynamicSchemaExpr,
    operator: ArithmeticOperator,
    numericType: NumericTypeTag
  ) extends DynamicSchemaExpr
}
```

The available `ArithmeticOperator` values are:

| Operator   | Optic Syntax | Description    |
|------------|--------------|----------------|
| `Add`      | `+`          | Addition       |
| `Subtract` | `-`          | Subtraction    |
| `Multiply` | `*`          | Multiplication |

Supported numeric types include `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, and `BigDecimal`.

#### Other string and bitwise operations

`DynamicSchemaExpr` also includes:

- `StringConcat`
- `StringRegexMatch`
- `StringLength`
- `StringSubstring`
- `StringTrim`
- `StringToUpperCase`
- `StringToLowerCase`
- `StringReplace`
- `StringStartsWith`
- `StringEndsWith`
- `StringContains`
- `StringIndexOf`
- `PrimitiveConversion`
- `Bitwise`
- `BitwiseNot`

These are the cases downstream interpreters should consider when translating `SchemaExpr.dynamic` into SQL, filters, or other query languages.

## Error Handling

Expression evaluation returns `Either[OpticCheck, Seq[B]]`. The `Left` case contains an [`OpticCheck`](./optics.md) with detailed diagnostic information about what went wrong — for example, an unexpected case in a prism, an empty collection in a traversal, or a missing key.

```scala

case class Shape(kind: String)

object Shape extends CompanionOptics[Shape] {
  implicit val schema: Schema[Shape] = Schema.derived

  val kind: Lens[Shape, String] = $(_.kind)
}

val expr = Shape.kind === "circle"
val result = expr.eval(Shape("circle"))

result match {
  case Right(values) => println(s"Result: ${values.head}")
  case Left(check)   => println(s"Error: ${check.message}")
}
```

## Advanced Usage: Building Query DSLs

Because `SchemaExpr.dynamic` is a sealed, inspectable AST, third-party libraries can translate expressions into other languages. The public entry point should still accept `SchemaExpr`; only the interpreter internals need to cross into `DynamicSchemaExpr`.

```scala
// Pseudocode — keeps SchemaExpr as the public API
def toSql[A, B](expr: SchemaExpr[A, B]): String =
  toSqlDynamic(expr.dynamic)

def toSqlDynamic(expr: DynamicSchemaExpr): String = expr match {
  case DynamicSchemaExpr.Relational(left, right, op) =>
    s"${toSqlValue(left)} ${opToSql(op)} ${toSqlValue(right)}"
  case DynamicSchemaExpr.Logical(left, right, DynamicSchemaExpr.LogicalOperator.And) =>
    s"(${toSqlDynamic(left)}) AND (${toSqlDynamic(right)})"
  case DynamicSchemaExpr.Logical(left, right, DynamicSchemaExpr.LogicalOperator.Or) =>
    s"(${toSqlDynamic(left)}) OR (${toSqlDynamic(right)})"
  case DynamicSchemaExpr.Not(inner) =>
    s"NOT (${toSqlDynamic(inner)})"
  // ...
}
```

This is the key advantage of reified expressions over plain functions — the same expression can be evaluated locally *and* translated to a remote query language.

## Integration

### Optics

`SchemaExpr` is tightly integrated with [Optics](./optics.md). All operator methods (`===`, `>`, `<`, `>=`, `<=`, `!=`, `&&`, `||`, `!`, `+`, `-`, `*`, `concat`, `matches`, `length`) are defined on `Optic[S, A]` and return `SchemaExpr[S, B]` values. This makes the optic the primary entry point for building expressions.

### Schema

[Schema](./schema.md) provides the type information needed to construct typed `SchemaExpr` values and to convert typed results to and from `DynamicValue` during evaluation.

### DynamicValue

[DynamicValue](./dynamic-value.md) is the output type of `SchemaExpr#evalDynamic`. It provides a schema-less representation that can be serialized, compared, and manipulated uniformly.

### OpticCheck

[OpticCheck](./optics.md) is the error type returned when expression evaluation fails. It provides rich diagnostic information including the optic path, expected vs. actual cases, and the actual value encountered.

---

## Schema Search and Update

`SchemaSearch` and `TypeSearch` are the two `DynamicOptic` node kinds that turn a path into a search: instead of naming one field, they match every value of a shape or type anywhere in a structure. This page covers the machinery behind that search once a path resolves against a value or a schema: `SchemaMatch`, the structural predicate the search matches against; `SearchTraversal`, the `Traversal[S, A]` engine that executes a search over a typed value and lets you fold, modify, or check the matches; and `Updater`, the mechanism for rewriting a `Reflect`/`Term` tree at a resolved path rather than the values it describes.

For the search DSLs themselves — the typed `.searchFor[T]` macro, the `#Pattern` string syntax, and the supported pattern grammar — see [DynamicOptic](./dynamic-optic.md#search-optics). This page assumes you can already build a search path and focuses on what runs once you have one.

## Design & Structure

A search path splits into two phases that operate on different representations of your data:

```
Typed value S                          Schema tree Reflect[F, S]
      │                                        │
      ▼                                        ▼
SearchTraversal[S, A]                  Reflect#updated(path)(Updater)
  (built from .searchFor[T]              (rewrites the Reflect/Term
   or SearchTraversal.apply)              node found at a path)
      │
      ▼
DynamicValue tree, walked depth-first
      │
      ▼
SchemaMatch.matches(pattern, value)     ← only for #Pattern searches;
  (the structural predicate)              .searchFor[T] matches by TypeId instead
```

`SearchTraversal` finds and transforms **values** — it decodes candidate subtrees of a concrete `S` and collects the ones that decode as `A`. `Updater` rewrites **schema metadata** — it walks the `Reflect`/`Term` tree that describes a type and replaces the node at a path, independent of any particular value. The two are easy to conflate because both start from a `DynamicOptic` path, but a `TypeSearch`/`SchemaSearch` node is only meaningful to the value-searching side: `Reflect#updated` does not special-case those nodes, so a path that reaches `Reflect#updated`/`Schema#updated` must resolve through `Field`/`Case`/`AtIndex`/`Elements`/`Wrapped`/`MapKeys`/`MapValues` nodes only.

## SchemaMatch — Structural Pattern Matching

`SchemaMatch.matches` is the predicate a `#Pattern` search node matches against once it reaches a `DynamicValue`: given a `SchemaRepr` pattern and a `DynamicValue`, it returns whether the value has that shape:

```scala
object SchemaMatch {
  def matches(pattern: SchemaRepr, value: DynamicValue): Boolean
}
```

`SchemaRepr` is the small pattern language `#Pattern` strings parse into — `Wildcard`, `Primitive(name)`, `Record(fields)`, `Variant(cases)`, `Sequence(element)`, `Map(key, value)`, `Optional(inner)`, and `Nominal(name)`. Constructing one directly and matching it against a `DynamicValue` shows the same rules the `#record { ... }` syntax compiles down to:

```scala

val personPattern = SchemaRepr.Record(
  IndexedSeq("name" -> SchemaRepr.Primitive("string"), "age" -> SchemaRepr.Primitive("int"))
)

val alice = DynamicValue.Record("name" -> DynamicValue.string("Alice"), "age" -> DynamicValue.int(30))
val bob   = DynamicValue.Record("name" -> DynamicValue.string("Bob"), "role" -> DynamicValue.string("admin"))
```

`Record` matching is a subset match — every field named in the pattern must exist with a matching type, but extra fields on the value are fine, which is why `bob` fails only because `age` is missing, not because of the extra `role` field:

```scala
SchemaMatch.matches(personPattern, alice)
// res0: Boolean = true
SchemaMatch.matches(personPattern, bob)
// res1: Boolean = false
```

`Wildcard` matches anything, and `Sequence`/`Map` patterns match when every element or entry matches — an empty sequence or map always matches, since there is nothing to fail on:

```scala

val ages = DynamicValue.Sequence(DynamicValue.int(1), DynamicValue.int(2))
SchemaMatch.matches(SchemaRepr.Sequence(SchemaRepr.Primitive("int")), ages)
SchemaMatch.matches(SchemaRepr.Wildcard, ages)
```

`Nominal(name)` always returns `false` against a `DynamicValue`, because a decoded value carries no record of the Scala type it came from — the same limitation documented for `#Person`-style patterns in [DynamicOptic's known limitation](./dynamic-optic.md#known-limitation-nominal-matching-in-untyped-contexts). Matching by nominal type requires the typed `.searchFor[T]` API instead, which matches by `TypeId` rather than by structure.

## SearchTraversal — Executing a Search Over Values

`SearchTraversal` is the `Traversal[S, A]` that `.searchFor[T]` and `#Pattern` searches compile down to. Construct one directly from a pair of schemas when you want the traversal without going through the macro or the path interpolator:

```scala

case class Address(city: String)
object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

case class Person(name: String, age: Int, address: Address)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

case class Team(name: String, lead: Person, members: List[Person])
object Team {
  implicit val schema: Schema[Team] = Schema.derived
}

case class Company(name: String, ceo: Person, teams: List[Team])
object Company extends CompanionOptics[Company] {
  implicit val schema: Schema[Company] = Schema.derived
}

val findPeople: Traversal[Company, Person] = SearchTraversal[Company, Person]

val acme = Company(
  "Acme",
  ceo = Person("Alice", 45, Address("NYC")),
  teams = List(
    Team("Platform", Person("Bob", 32, Address("SF")), List(Person("Carol", 28, Address("SF")))),
    Team("Sales", Person("Dave", 39, Address("LA")), Nil)
  )
)
```

`SearchTraversal[Company, Person]` resolves the two implicit `Schema` instances the same way `Company.optic(_.searchFor[Person])` does—both produce the identical traversal, so pick whichever reads better at the call site. `Traversal#fold` walks every match depth-first, left-to-right, which is why the CEO comes before any team, and each team's lead comes before its members:

```scala
findPeople.fold(acme)(List.empty[String], (names, p) => names :+ p.name)
// res3: List[String] = List("Alice", "Bob", "Carol", "Dave")
```

`Traversal#modify` rewrites every match in place and rebuilds the structure around it, `modifyOption` returns `None` instead of a no-op result when nothing matched, and `modifyOrFail` surfaces a decoding failure as `Left` instead of silently keeping the original value:

```scala
findPeople.modify(acme, p => p.copy(age = p.age + 1)).ceo.age
// res4: Int = 46
SearchTraversal[Company, Address].modifyOption(acme, a => a.copy(city = a.city.toUpperCase)).map(_.ceo.address.city)
// res5: Option[String] = Some("NYC")
SearchTraversal[Team, Address].modifyOption(acme.teams.head, a => a).isDefined
// res6: Boolean = true
```

`Traversal#check` reports whether a traversal has at least one match, which is how a search-backed traversal signals "nothing here" without throwing:

```scala
SearchTraversal[Company, Person].check(acme).isEmpty
// res7: Boolean = true
SearchTraversal[Team, Boolean].check(acme.teams.head).isEmpty
// res8: Boolean = false
```

### Composing a Search With Other Optics

A search traversal composes with `Lens`, `Prism`, `Optional`, and other `Traversal`s on either side, so you can narrow the search to part of a structure or refine each match further. Searching from a specific field finds only what is reachable from there, and appending a lens after a search projects each match down to one of its fields:

```scala

object CompanyOptics extends CompanionOptics[Company] {
  implicit val schema: Schema[Company] = Company.schema

  // Search restricted to one team: only Bob and Carol, never Alice or Dave
  val platformMembers: Traversal[Company, Person] = optic(_.teams.at(0).searchFor[Person])

  // Search first, then focus each match's city
  val allCities: Traversal[Company, String] = optic(_.searchFor[Address].city)
}
```

Both directions rebuild the same way a plain path-based traversal does: `Traversal#modify`, `Traversal#fold`, and `Traversal#check` all work on the composed traversal exactly as they do on a bare `SearchTraversal`, because composition produces another `Traversal[S, A]` — the fact that a search sits inside it is an implementation detail, not a different API.

### Recursive Types Are Safe to Search

Because `SearchTraversal` walks the decoded `DynamicValue` of a concrete value rather than the schema definition, searching a recursive type (a tree, a linked structure) terminates naturally — the value itself is always finite, even though its `Reflect` describes an unbounded type. There is nothing extra to opt into; a self-referential case class searches the same way a flat one does.

## Updater — Rewriting Schema Metadata

`Reflect.Updater` and `Term.Updater` are the callbacks behind `Schema#updated` and `Reflect#updated` — the mechanism for rewriting a schema's metadata (documentation, defaults, validations, or a field's shape entirely) at a resolved path, as opposed to rewriting the values that schema describes:

```scala
object Reflect {
  trait Updater[F[_, _]] {
    def update[A](reflect: Reflect[F, A]): Reflect[F, A]
  }
}

object Term {
  trait Updater[F[_, _]] {
    def update[S, A](input: Term[F, S, A]): Option[Term[F, S, A]]
  }
}
```

The two differ in one important way: `Reflect.Updater#update` is total — it always returns a `Reflect`, because a schema node can be re-shaped but not removed. `Term.Updater#update` is partial — returning `None` deletes the field or case the updater targets, which is how `Record#modifyField` and `Variant#modifyCase` support dropping a member rather than only renaming or retyping it.

[Schema](./schema.md#updating-nested-schemas) already covers the common case, updating one field through an optic with a plain function: `Schema[Person].updated(Person.address)(_.doc("Mailing address"))`. That convenience overload builds a `Reflect.Updater` for you. Reaching for `Reflect.Updater` directly is what you need for the case that overload cannot express — a `DynamicOptic` path built at runtime, or a rewrite that needs the full node rather than just its focus:

```scala

case class Config(host: String, port: Int)
object Config {
  implicit val schema: Schema[Config] = Schema.derived
}

// Attach documentation to a field found by a runtime-built DynamicOptic path
val documented: Option[Schema[Config]] =
  Schema[Config].updated(DynamicOptic.root.field("port"))(new Reflect.Updater[Binding] {
    def update[A](reflect: Reflect[Binding, A]): Reflect[Binding, A] =
      reflect.doc("The TCP port the server listens on")
  })
```

`Term.Updater` operates one level up, on the named field or case itself rather than its value, which is what makes rename and delete possible. Renaming reuses the term's existing `value`; deleting a field returns `None` and the field disappears from the record entirely:

```scala

case class LegacyUser(id: Long, username: String, internalNotes: String)
object LegacyUser {
  implicit val schema: Schema[LegacyUser] = Schema.derived
}

val userRecord = Schema[LegacyUser].reflect.asRecord.get

// Rename username -> name
val renamed = userRecord.modifyField("username")(new Term.Updater[Binding] {
  def update[S, A](input: Term[Binding, S, A]): Option[Term[Binding, S, A]] =
    Some(input.copy(name = "name"))
})

// Drop internalNotes entirely by returning None
val withoutNotes = userRecord.modifyField("internalNotes")(new Term.Updater[Binding] {
  def update[S, A](input: Term[Binding, S, A]): Option[Term[Binding, S, A]] = None
})
```

Both updaters ran against the same `userRecord` independently, so `renamed` still has three fields with one renamed, while `withoutNotes` has two:

```scala
renamed.map(_.fields.map(_.name))
// res11: Option[IndexedSeq[String]] = Some(
//   Vector("id", "name", "internalNotes")
// )
withoutNotes.map(_.fields.map(_.name))
// res12: Option[IndexedSeq[String]] = None
```

:::warning[Search nodes are not valid `updated` paths]
`Reflect#updated`/`Schema#updated` walk a `DynamicOptic` through `Field`, `Case`, `AtIndex`/`AtIndices`/`Elements`, `Wrapped`, and `MapKeys`/`MapValues` nodes only. A path containing a `TypeSearch` or `SchemaSearch` node is not rejected outright, but it is not handled either — it falls into the same branch as `MapKeys`/`MapValues` and produces an unspecified result. Build search-based rewrites with `SearchTraversal#modify` on a value instead of `Schema#updated` on a schema. `DynamicMigration`, by contrast, rejects search nodes outright with an explicit error, since a migration requires one statically-known path.
:::

## Where Search Nodes Are (and Aren't) Handled

The same `TypeSearch`/`SchemaSearch` node means different things depending on which API resolves the path it's part of:

| API                       | Search nodes                                             |
| -------------------------- | ---------------------------------------------------------- |
| `SearchTraversal` / `.searchFor[T]` / `#Pattern` on a value | The intended use — this is what builds and executes the search |
| `Reflect#get` / `Schema#get`         | Supported — resolves the first match, then continues the remaining path from there |
| `Json` / `DynamicValue` patch paths (`JsonPatch`) | Supported — rewrites every match, not just the first |
| `Reflect#updated` / `Schema#updated` | Not handled — falls through to the map-key/value branch; use `SearchTraversal#modify` instead |
| `DynamicMigration`          | Rejected outright with `"Type/Schema search nodes are not supported in migration paths"` |

## See Also

- [DynamicOptic](./dynamic-optic.md#search-optics) — the `.searchFor[T]` and `#Pattern` search DSLs, and the full pattern grammar table.
- [Path Interpolator](./path-interpolator.md) — the `p"..."` string syntax that `#Pattern` search nodes parse from.
- [Schema](./schema.md#updating-nested-schemas) — `Schema#updated` and `Schema#@@` for the common, optic-based case of rewriting one field's metadata.
- [Reflect](./reflect.md) — the node types (`Record`, `Variant`, `Sequence`, `Map`, `Wrapper`, `Deferred`) that `Updater` rewrites and `SearchTraversal` decodes against.
- [Optics](./optics.md) — the base `Traversal[S, A]` type that `SearchTraversal` implements and composes with.

---

## Schema

`Schema[A]` is the primary data type in ZIO Blocks (ZIO Schema 2) that contains reified information about the structure of a Scala data type `A`, together with the ability to tear down and build up values of that type.

```scala
final case class Schema[A](reflect: Reflect.Bound[A])
```

```
┌────────────────────────────────────────────────────────────────┐
│                        Schema[A]                               │
├────────────────────────────────────────────────────────────────┤
│                Reflect.Bound[A] (Reflect[Binding, A])          │
│    ┌─────────────────────────────────────────────────────┐     │
│    │  Structure       │  TypeName[A], DynamicValue       │     │
│    │  (ADT nodes)     │  Doc, Examples, Default Value    │     │
│    │                  │  Modifiers, Metadata             │     │
│    └─────────────────────────────────────────────────────┘     │
│    ┌─────────────────────────────────────────────────────┐     │
│    │  Binding[T, A]                                      │     │
│    │  - Constructor[A] (build values)                    │     │
│    │  - Deconstructor[A] (tear down values)              │     │
│    └─────────────────────────────────────────────────────┘     │
└────────────────────────────────────────────────────────────────┘
```

## Reflect: Structure vs Binding

The [`Reflect`](./reflect.md) data type represents the structure of Scala types. It is parameterized by `F[_, _]` which allows plugging in different "binding" strategies:

```
┌────────────────────────────────────────────────────────────────┐
│                    Reflect[F[_, _], A]                         │
├────────────────────────────────────────────────────────────────┤
│  F = Binding      → Reflect.Bound[A]    (with construction/    │
│                                          deconstruction)       │
│  F = NoBinding    → Reflect.Unbound[A]  (pure data,            │
│                                          serializable)         │
└────────────────────────────────────────────────────────────────┘
```

- **Bound Schema**: Contains functions for constructing/deconstructing values (not serializable)
- **Unbound Schema**: Pure data representation (can be serialized across the wire)

Schema is a bound Reflect, meaning that other than structural information, it also contains construction and deconstruction capabilities via the `Binding` type:

```scala
// Schema wraps a bound Reflect
final case class Schema[A](reflect: Reflect.Bound[A])

// Reflect.Bound is a type alias
type Bound[A] = Reflect[Binding, A]
```

## Schema Derivation

ZIO Blocks provides both automatic schema derivation and also ways to manually create schemas. As an end user, you will mostly use automatic derivation. So we will focus on automatic derivation here. If you need to go deeper and create schemas manually, check out the [Reflect](./reflect.md) and [Binding](./binding.md) documentation pages.

To leverage auto-derivation, simply define an implicit `Schema` for your type using `Schema.derived`:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}
```

It will automatically derive the schema for `Person` based on its structure. After that, you can summon the schema using `Schema[Person]` anywhere in your code.

For sealed traits (ADTs), it will derive the schema for all subtypes as well:

```scala

sealed trait Shape
object Shape {
  case class Circle(radius: Double) extends Shape
  case class Rectangle(width: Double, height: Double) extends Shape
  
  implicit val schema: Schema[Shape] = Schema.derived
}
```

Under the hood, the derivation process includes deriving schemas for cases of enum (i.e. `Circle` and `Rectangle`) and finally creating a schema for the `Shape` trait itself.

On Scala 3.7 and newer, `Schema.derived` also works for named tuples such as `(name: String, age: Int)` (on Scala 3.9 and newer, through the `zio-blocks-next-schema` artifact with `-experimental`). A named tuple is usually named with a type alias, and an `object` with the same name as the alias is not a companion, so a `given Schema` inside it is not found implicitly. Define the schema as a top-level `given`, or import it where you use it (for a named tuple aliased as `Reading`, `import Reading.given`). Two named-tuple features have known bugs in 0.0.56: a `when[...]` step inside `$(...)` after a named-tuple field compiles but throws `Expected a variant` at runtime, and `Into.derived` and `As.derived` reject named tuples (convert through `NamedTuple#toTuple` instead). See [How to Use Scala 3 Named Tuples with Schema](../../guides/use-named-tuples-with-schema.md) for deriving, encoding, optics and conversion to case classes.

## Built-in Schemas

ZIO Blocks ships with a comprehensive set of pre-defined schemas for standard Scala and Java types, eliminating boilerplate and ensuring consistent behavior across your application.

### Primitive Types

The foundational building blocks for all data types. These schemas leverage the [register-based architecture](registers.md) for zero-allocation performance:

```scala

Schema[Unit]      // The unit type (singleton value)
Schema[Boolean]   // Boolean values (true/false)
Schema[Byte]      // 8-bit signed integer (-128 to 127)
Schema[Short]     // 16-bit signed integer
Schema[Int]       // 32-bit signed integer
Schema[Long]      // 64-bit signed integer
Schema[Float]     // 32-bit IEEE 754 floating point
Schema[Double]    // 64-bit IEEE 754 floating point
Schema[Char]      // 16-bit Unicode character
Schema[String]    // Immutable character sequence
```

### Arbitrary Precision Numbers

For financial calculations and scenarios requiring exact decimal representation:

```scala

Schema[BigInt]      // Arbitrary precision integer
Schema[BigDecimal]  // Arbitrary precision decimal
```

### Temporal Types (java.time)

Complete coverage of the modern Java Time API for robust date/time handling:

```scala

// Date components
Schema[java.time.LocalDate]      // Date without time (2024-01-15)
Schema[java.time.LocalTime]      // Time without date (14:30:00)
Schema[java.time.LocalDateTime]  // Date and time without timezone

// Timezone-aware types  
Schema[java.time.ZonedDateTime]   // Full date-time with timezone
Schema[java.time.OffsetDateTime]  // Date-time with UTC offset
Schema[java.time.OffsetTime]      // Time with UTC offset
Schema[java.time.ZoneId]          // Timezone identifier (e.g., "America/New_York")
Schema[java.time.ZoneOffset]      // Fixed UTC offset (e.g., +05:30)

// Duration and period
Schema[java.time.Duration]  // Time-based duration (hours, minutes, seconds)
Schema[java.time.Period]    // Date-based period (years, months, days)
Schema[java.time.Instant]   // Point on the timeline (Unix timestamp)

// Calendar components
Schema[java.time.Year]       // Year value (2024)
Schema[java.time.Month]      // Month of year (JANUARY..DECEMBER)
Schema[java.time.DayOfWeek]  // Day of week (MONDAY..SUNDAY)
Schema[java.time.YearMonth]  // Year and month combination
Schema[java.time.MonthDay]   // Month and day combination
```

### Common Java Utility Types

There are also schemas for frequently used Java utility types, UUID and Currency:

```scala

Schema[java.util.UUID]      // 128-bit universally unique identifier
Schema[java.util.Currency]  // ISO 4217 currency code
```

### Optional Values

ZIO Blocks provides specialized `Option` schemas optimized for primitive types. These avoid boxing overhead by storing primitive values directly in registers:

```scala

// Specialized primitive options (no boxing overhead)
Schema[Option[Boolean]]  // Also: Byte, Short, Int, Long, Float, Double, Char, Unit
```

Other than primitive types, ZIO Blocks uses a generic representation for `Option[A]` which works for all reference types:

```scala

// Reference type options (requires A <: AnyRef)
Schema[Option[A]]  // Generic option for reference types
```

### Either Values

An `Either[A, B]` schema is available whenever both branch types have schemas. Primitive values and primitive-backed wrappers use specialized primitive register layouts, while other values use the layouts described by their schemas:

```scala

Schema[Either[String, Int]]
Schema[Either[Int, Long]]
```

### Tuples

Tuples have no implicit `Schema`, unlike `Either`: `Schema[(String, Int)]` does not compile, and neither does a schema for a collection of tuples such as `Schema[List[(String, Int)]]`. A tuple field inside a derived case class needs nothing extra, because derivation handles it. To summon a tuple schema on its own, derive it with `Schema.derived` and make it implicit:

```scala

implicit val pairSchema: Schema[(String, Int)] = Schema.derived[(String, Int)]
```

With that instance in scope, the built-in collection schemas find it, and a list of pairs encodes as a JSON array of two-element arrays:

```scala
println(Schema[List[(String, Int)]].jsonCodec.encodeToString(List(("widget", 2), ("gadget", 5))))
// [["widget",2],["gadget",5]]
```

### Collection Types

ZIO Blocks also provides polymorphic schemas for standard Scala collections. You can summon schemas for collections of any element type `A` (and key/value types `K`/`V` for maps):

```scala

// Built-in Scala collections
Schema[List[A]]        // Immutable singly-linked list
Schema[Vector[A]]      // Immutable indexed sequence (efficient random access)
Schema[Set[A]]         // Immutable set (unique elements)
Schema[Seq[A]]         // General immutable sequence
Schema[IndexedSeq[A]]  // Indexed sequence
Schema[Map[K, V]]      // Immutable key-value mapping

// ZIO Blocks collections (the chunk module)
Schema[zio.blocks.chunk.Chunk[A]]       // Chunk sequence
Schema[zio.blocks.chunk.ChunkMap[K, V]] // ChunkMap key-value mapping
```

To learn how to create custom collection schemas, check out the [`Sequence`](./reflect.md#3-sequence) and [`Map`](./reflect.md#4-map) nodes on the documentation of [`Reflect`](./reflect.md) data type.

### DynamicValue

ZIO Blocks includes a built-in schema for `DynamicValue`, a semi-structured data representation that serves as a superset of JSON:

```scala

val schema = Schema[DynamicValue]  // Semi-structured data (superset of JSON)
```

Having the schema for `DynamicValue` allows seamless encoding/decoding between `DynamicValue` and other formats, such as JSON, Avro, Protobuf, etc. It enables us to convert our type-safe data into a semi-structured representation and serialize it into any desired format.

### DynamicValue toString (EJSON Format)

`DynamicValue` has a custom `toString` that produces EJSON (Extended JSON) format - a superset of JSON that handles non-string keys, tagged variants, and typed primitives:

```scala

// Records have unquoted keys
val record = DynamicValue.Record(Vector(
  "name" -> DynamicValue.Primitive(PrimitiveValue.String("Alice")),
  "age" -> DynamicValue.Primitive(PrimitiveValue.Int(30))
))
println(record)
// {
//   name: "Alice",
//   age: 30
// }

// Maps have quoted string keys
val map = DynamicValue.Map(Vector(
  DynamicValue.Primitive(PrimitiveValue.String("key")) ->
    DynamicValue.Primitive(PrimitiveValue.String("value"))
))
println(map)
// {
//   "key": "value"
// }

// Variants use @ metadata
val variant = DynamicValue.Variant("Some", DynamicValue.Record(Vector(
  "value" -> DynamicValue.Primitive(PrimitiveValue.Int(42))
)))
println(variant)
// {
//   value: 42
// } @ {tag: "Some"}

// Typed primitives use @ metadata
val instant = DynamicValue.Primitive(PrimitiveValue.Instant(java.time.Instant.now))
println(instant)
// 1705312800000 @ {type: "instant"}
```

**Key EJSON properties:**
- **Records**: unquoted keys (`{ name: "John" }`)
- **Maps**: quoted string keys (`{ "name": "John" }`) or unquoted non-string keys (`{ 42: "value" }`)
- **Variants**: postfix `@ {tag: "CaseName"}`
- **Typed primitives**: postfix `@ {type: "instant"}` for types that would lose precision as JSON

## Debug-Friendly toString

`Schema` has a custom `toString` that wraps the underlying `Reflect` output in a `Schema { ... }` block, providing a complete structural view of your data types:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

println(Schema[Person])
// Output:
// Schema {
//   record Person {
//     name: String
//     age: Int
//   }
// }
```

For primitive schemas:

```scala
println(Schema[Int])
// Output:
// Schema {
//   Int
// }
```

This format makes it easy to inspect complex nested schemas during debugging. See the [Reflect documentation](./reflect.md#debug-friendly-tostring) for details on the SDL format used for the inner structure.

## Encoding and Decoding

The `Schema[A]` provides methods to encode and decode values of type `A` to/from various formats using the `Format` abstraction:

```scala
case class Schema[A](reflect: Reflect.Bound[A]) {
  def encode[F <: codec.Format](format: F)(output: format.EncodeOutput)(value: A): Unit = ???
  def decode[F <: codec.Format](format: F)(decodeInput: format.DecodeInput): Either[SchemaError, A] = ???
}
```

The `Format` is the base abstraction for serialization formats in ZIO Blocks, such as Avro, JSON, Protobuf, etc. Each format associates with a specific codec type class which defines the interface for encoding and decoding values and a deriver for deriving codecs for specific types.

ZIO Blocks has built-in support for several popular formats, currently `AvroFormat` and `JsonFormat`. You can also implement your own custom formats by extending the `Format` trait.

The following example demonstrates encoding and decoding a `Person` case class to/from JSON using the built-in `JsonFormat`:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

object EncodeDecodeExample extends App {
  val person = Person("John", 42)

  // Encode to JSON
  val encodedBuffer = {
    val buffer = ByteBuffer.allocate(1024)
    Schema[Person].encode(JsonFormat)(buffer)(person)
    buffer.flip()
    buffer
  }

  // Extract JSON string
  val jsonString = new String(encodedBuffer.duplicate().array())
  println(s"Encoded JSON: $jsonString")

  // Decode back to Person
  val decodedPerson = Schema[Person].decode(JsonFormat)(encodedBuffer.duplicate())
  println(s"Decoded Person: $decodedPerson")
}
```

Please note that both `Schema#encode` and `Schema#decode` cache instances, so using encode and decode in multiple places only performs the derivation process once.

## Type Class Derivation

To derive type class instances for a type `A` based on its schema, you can use the `derive` method on `Schema[A]`. This method takes a `Deriver` for the desired type class and produces an instance of that type class for `A`.

In the following example, we derive a JSON codec for the `Person` case class using the `JsonFormat` deriver:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
  val codec: JsonCodec[Person] = schema.derive(JsonFormat)
}

val person = Person("John", 42)

// Encode to JSON
val json = Person.codec.encode(person)

// Extract JSON string
val jsonString = new String(json)
println(s"Encoded JSON: $jsonString")

// Decode back to Person
val decodedPerson = Person.codec.decode(json)
println(s"Decoded Person: $decodedPerson")
```

## Metadata Operations

ZIO Blocks allows attaching metadata to schemas and their fields, such as documentation, example values, and default values. This metadata can be useful for generating API documentation, client code, or providing hints to serialization formats.

Here is an example of how to set and retrieve documentation values:

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person]{
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String] = optic(_.name)
  val age: Lens[Person, Int]     = optic(_.age)
}

// Add documentation to the schema
val documented: Schema[Person] = Schema[Person].doc("A person entity")

// Get documentation
val doc: Doc = Schema[Person].doc

// Add documentation to a field using optics
val fieldDoc: Schema[Person] = Schema[Person].doc(Person.name, "The person's name")
```

The important thing to note here is that we can use optics to target specific fields within the schema when adding or retrieving metadata. In the last example, we added documentation specifically to the `name` field of the `Person` schema.

Similarly, you can set and get example values:

```scala

// Add example values
val withExamples: Schema[Person] = 
  Schema[Person].examples(Person("Alice", 30), Person("Bob", 25))

// Get examples
val examples: Seq[Person] = Schema[Person].examples

// Add examples to a specific field
val fieldExamples: Schema[Person] =
  Schema[Person].examples(Person.name, "Alice", "Bob", "Charlie")
```

There are also methods for setting and getting default values:

```scala
// Add default value to schema
val withDefault: Schema[Person] = Schema[Person]
  .defaultValue(Person("Unknown", 0))

// Get default value
val default: Option[Person] = Schema[Person].getDefaultValue

// Add default to a specific field
val fieldDefault: Schema[Person] = Schema[Person]
  .defaultValue(Person.age, 18)

// Get default for a field
val ageDefault: Option[Int] = 
  Schema[Person].getDefaultValue(Person.age)
```

## Accessing and Updating Partial Schemas

To access a specific part of a schema, we can use the `Schema#get` method, which takes an optic and returns the reflection of the targeted field. 

```scala

case class Person(name: String, address: Address)

object Person extends CompanionOptics[Person]{
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String]      = optic(_.name)
  val address: Lens[Person, Address]  = optic(_.address)
}

case class Address(street: String, city: String)

object Address extends CompanionOptics[Address]{
  implicit val schema: Schema[Address] = Schema.derived

  val street: Lens[Address, String] = optic(_.street)
  val city  : Lens[Address, String] = optic(_.city)
}

// Get schema for a nested field
val addressSchema: Option[Reflect.Bound[Address]] =
  Schema[Person].get(Person.address)
```

This method enables us to retrieve the schema for any nested field using optics. For example, to get the schema for the `street` field inside the `address` of a `Person`, we can do as follows:

```scala
val streetSchema: Option[Reflect.Bound[String]] =
  Schema[Person].get(Person.address(Address.street))
```

### Updating Nested Schemas

To update a specific part of a schema, we can use the `Schema#updated` method, which takes an optic and a function to transform the targeted schema:

```scala
case class Schema[A](reflect: Reflect.Bound[A]) {
  def updated[B](optic: Optic[A, B])(
    f: Reflect.Bound[B] => Reflect.Bound[B]
  ): Option[Schema[A]]
}
```

Here is an example of updating the documentation of a nested field:

```scala
// Update schema at a specific path
val updated: Option[Schema[Person]] = Schema[Person]
  .updated(Person.address)(_.doc("Mailing address"))
```

For the lower-level `Reflect.Updater`/`Term.Updater` callbacks this method builds on — including how `Term.Updater` can rename or delete a field by returning `None` — see [Schema Search and Update](./schema-search.md#updater--rewriting-schema-metadata).

## Schema Aspects

Schema aspects are a powerful mechanism in ZIO Blocks for transforming schemas. You can think of the schema aspect as a function that takes a reflect and produces a new reflect:

```scala
trait SchemaAspect[-Upper, +Lower, F[_, _]] {
  def apply[A >: Lower <: Upper](reflect: Reflect[F, A]): Reflect[F, A]
}
```

The `Schema` data type has a `@@` method that applies schema aspects, and it's a thin wrapper over two `Reflect#aspect` overloads: applying an aspect to a whole schema just calls `aspect(reflect)` directly, and applying one at a path uses [`Reflect#updated`](./reflect-transformer.md) to rewrite the subtree the optic points to:

```scala
case class Schema[A](reflect: Reflect.Bound[A]) {
  def @@[Min >: A, Max <: A](aspect: SchemaAspect[Min, Max, Binding]): Schema[A] = ???
  def @@[B](part: Optic[A, B], aspect: SchemaAspect[B, B, Binding]) = ???
}

sealed trait Reflect[F[_, _], A] {
  def aspect[Min >: A, Max <: A](aspect: SchemaAspect[Min, Max, F]): Reflect[F, A]
  def aspect[B, Min >: B, Max <: B](optic: Optic[A, B], aspect: SchemaAspect[Min, Max, F]): Reflect[F, A]
}
```

These methods enable us to use `@@` syntax for applying aspects to either the entire schema or a specific path within the schema using optics:

```scala

// Apply aspect to entire schema
val documented: Schema[Person] = Schema[Person] @@ SchemaAspect.doc("A person")

// Apply aspect to specific path
val fieldDoc: Schema[Person] = Schema[Person] @@ (
  Person.name, 
  SchemaAspect.examples("Alice", "Bob")
)
```

Currently, ZIO Blocks provides the following built-in schema aspects:

- `SchemaAspect.identity`: No-op transformation
- `SchemaAspect.doc`: Attach documentation to schema or field
- `SchemaAspect.examples`: Attach example values to schema or field

The path-targeted overload doesn't fail on an optic that doesn't resolve against the schema — it falls back to the original schema unchanged, the same behavior [`Reflect#updated`](./reflect-transformer.md) has when a path finds nothing. This matters most when an optic is built by hand rather than through the `optic(_.field)` macro: a lens pointing at a field name that isn't actually on the record silently leaves the schema untouched rather than throwing.

## Modifiers

Modifiers in ZIO Blocks provide a mechanism to attach metadata and configuration to schema elements without polluting the domain types themselves. They serve as the successor to ZIO Schema 1's annotation system, with the critical advantage of being **pure data** so, unlike Scala annotations, modifiers are runtime values that can be serialized.

The `Schema.modifier` and `Schema.modifiers` methods allow adding one or more modifiers to a schema:

```scala
final case class Schema[A](reflect: Reflect.Bound[A]) {
  def modifier(modifier: Modifier.Reflect)            : Schema[A] = ???
  def modifiers(modifiers: Iterable[Modifier.Reflect]): Schema[A] = ???
}
```

Here is an example of adding modifiers to a schema:

```scala
// Add modifier to schema
  val modified: Schema[Person] = Schema[Person]
    .modifier(Modifier.config("db.table-name", "person_table"))
    .modifier(Modifier.config("schema.version", "v2"))

// Add multiple modifiers
  val multiModified: Schema[Person] = Schema[Person]
    .modifiers(
      Seq(
        Modifier.config("db.table-name", "person_table"),
        Modifier.config("schema.version", "v2")
      )
    )
```

## Wrapper Types

ZIO Blocks provides the `transform` method for creating schemas for wrapper types, such as newtypes, opaque types and value classes:

```scala
final case class Schema[A](reflect: Reflect.Bound[A]) {
  def transform[B](to: A => B, from: B => A)(implicit typeId: TypeId[B]): Schema[B] = ???
}
```

The `transform` method allows you to define transformations that can fail by throwing `SchemaError` exceptions. Use it for both simple wrapper types and types with validation requirements. 

Here are examples of both:

```scala

// For types with validation (may fail)
case class Email(value: String)

object Email {
  private val EmailRegex = "^[A-Za-z0-9+_.-]+@[A-Za-z0-9.-]+$".r
  
  implicit val schema: Schema[Email] = Schema[String]
    .transform(
      {
        case x @ EmailRegex(_*) => Email(x)
        case _                  => throw SchemaError.validationFailed("Invalid email format")
      },
      _.value
    )
}

// For total transformations (never fail)
case class UserId(value: Long)

object UserId {
  implicit val schema: Schema[UserId] = Schema[Long]
    .transform(UserId(_), _.value)
}
```

## Compile-Time Shape Constraints (`Allows`)

ZIO Blocks provides `Allows[A, S]` — a phantom-typed capability token that proves, at compile time, that type `A` satisfies the structural grammar `S`. This lets library authors express and enforce structural preconditions on their generic APIs without writing macros themselves.

```scala

// Require a flat record of scalars (e.g. for CSV or RDBMS)
def writeCsv[A: Schema](rows: Seq[A])(using
  Allows[A, Record[Primitive | Optional[Primitive]]]
): Unit = ???

// Sealed traits auto-unwrap: each case must satisfy Record[...] — no Variant node needed
def publish[A: Schema](event: A)(using
  Allows[A, Record[Primitive | Sequence[Primitive]]]
): Unit = ???

// Recursive grammar (e.g. for a JSON document store)
def toJson[A: Schema](doc: A)(using
  Allows[A, Record[Primitive | Self | Optional[Primitive | Self] | Sequence[Primitive | Self]]]
): String = ???
```

When a type does not satisfy the grammar, the user gets a precise compile-time error naming the violating field and suggesting a fix. No runtime surprises.

See the [`Allows` reference](./allows.md) for the full grammar node table, union syntax, `Self` for recursive types, newtypes, and error message examples.

See also the [Your First Schema](../../guides/your-first-schema.md) tutorial for a hands-on introduction to deriving a `Schema` and round-tripping a type through JSON.

---

## Structural Types

<!--
BLOCKING ISSUE: Scala 3.7.4 Compiler Crash
============================================

This file uses structural type syntax ({ def field: Type }) which triggers a compiler
crash in Scala 3.7.4 during the erasure phase. Therefore, code blocks here are marked
as plain `scala` instead of `mdoc:compile-only` to allow the documentation build to pass.

WORKAROUND: Code examples are NOT compiled/type-checked until Scala 3.8.x is adopted.

TODO (When Scala 3.8.x is adopted):
1. Change all `\`\`\`scala` blocks back to `\`\`\`scala mdoc:compile-only`
2. Run `sbt docs/mdoc` to verify compilation succeeds
3. Delete this comment

Reference:
- Scala 3.7.4 compiler issue: https://github.com/scala/scala3/issues/24598 (or similar)
- Scala 3.8.x migration: PR #1169 "Preparing to migration on Scala 3.8.x"
- This PR: Addresses structural-types.md CI failures
-->

Structural types enable **duck typing** with ZIO Blocks schemas. Instead of requiring a nominal type name (like `class Person`), a structural schema validates based on the **shape** of an object — the fields it provides, regardless of how it was defined.

## Motivation

Consider a common integration scenario:

```scala
// Your system
case class Person(name: String, age: Int)

// External system (same data, different class)
case class User(name: String, age: Int)
```

Without structural types, converting between `Person` and `User` requires manual translation. With structural types, they both have the same structural schema:

```scala

case class Person(name: String, age: Int)
case class User(name: String, age: Int)

val personSchema = Schema.derived[Person]
val personStructural = personSchema.structural
// Schema[{ def name: String; def age: Int }]

val userSchema = Schema.derived[User]
val userStructural = userSchema.structural
// Schema[{ def name: String; def age: Int }]

// Both schemas accept the same data shape
```

## Construction: `Schema#structural`

Use the `Schema#structural` method on any schema to get the corresponding structural schema.

**Scala 3:** Using transparent inline — the return type is inferred to the full refinement type:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

val personSchema: Schema[Person] = Schema.derived[Person]
val structuralSchema: Schema[{ def name: String; def age: Int }] = personSchema.structural
```

**Scala 2:** Implicit derivation — returns `Schema[ts.StructuralType]` (path-dependent type):

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

val personSchema: Schema[Person] = Schema.derived[Person]
val structuralSchema = personSchema.structural
// Type: Schema[ts.StructuralType] (structural type inferred from macro)
```

## Supported Conversions

The following type categories can be converted to structural schemas:

### Product Types (Case Classes)

Both Scala 2 and 3 support structural conversion of case classes:

```scala

case class Address(street: String, city: String, zipCode: Int)
object Address {
  implicit val schema: Schema[Address] = Schema.derived[Address]
}

val schema = Schema.derived[Address]
val structural = schema.structural
// Schema[{ def street: String; def city: String; def zipCode: Int }]
```

### Tuples

Tuples convert to structural records with field names derived from positions:

```scala

type StringIntBool = (String, Int, Boolean)
implicit val schema: Schema[StringIntBool] = Schema.derived[StringIntBool]

val tupleSchema = Schema.derived[(String, Int, Boolean)]
val structuralSchema = tupleSchema.structural
// Schema[{ def _1: String; def _2: Int; def _3: Boolean }]
```

### Nested Products

Nested product fields keep their nominal types; only the outer product is structuralized:

```scala

case class Address(street: String, city: String)
object Address {
  implicit val schema: Schema[Address] = Schema.derived[Address]
}

case class Person(name: String, age: Int, address: Address)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

val personSchema = Schema.derived[Person]
val structuralSchema = personSchema.structural
// Schema[{
//   def name: String
//   def age: Int
//   def address: Address
// }]
```

### Opaque Types (Scala 3)

Opaque type aliases are unwrapped to their underlying type:

```scala

opaque type UserId = String

case class User(id: UserId, name: String)
object User {
  implicit val schema: Schema[User] = Schema.derived[User]
}

val schema = Schema.derived[User]
val structural = schema.structural
// Schema[{ def id: String; def name: String }]
// (UserId unwrapped to String)
```

### Sum Types / Sealed Traits (Scala 3)

Sealed traits and enums convert to union types with nested method syntax:

```scala

sealed trait Shape
object Shape {
  case class Circle(radius: Double) extends Shape
  case class Rectangle(width: Double, height: Double) extends Shape
  implicit val schema: Schema[Shape] = Schema.derived[Shape]
}

val schema = Schema.derived[Shape]
val structural = schema.structural
// Schema[
//   { def Circle: { def radius: Double } } |
//   { def Rectangle: { def height: Double; def width: Double } }
// ]
// (cases sorted alphabetically)
```

**Enum syntax** (Scala 3):

```scala

enum Color {
  case Red, Green, Blue
}
object Color {
  implicit val schema: Schema[Color] = Schema.derived[Color]
}

val schema = Schema.derived[Color]
val structural = schema.structural
// Schema[
//   { def Blue: {} } |
//   { def Green: {} } |
//   { def Red: {} }
// ]
```

Cases appear in **alphabetical order** in the union type. This alphabetical ordering (applied to fields in products and case names in unions) ensures **deterministic, normalized type identity**: two structural types with the same fields but different declaration order produce the same structural type and normalized name. This is essential for predictable schema evolution and cross-system interop.

## Direct Structural Derivation (Scala 3)

Create a schema directly for a structural type without a nominal base:

```scala

// No case class needed — define the schema for the shape directly
val personStructural = Schema.derived[{ def name: String; def age: Int }]

// The schema is ready to use with values matching that structural shape
```

This is only supported in **Scala 3** with the right macro machinery.

## Round-tripping Through DynamicValue

Structural schemas enable **cross-type conversion through `DynamicValue`** — encode a value of one nominal type and decode it as a *different* nominal type with the same structural shape. This is the core benefit of structural types for system integration.

### Motivation

In real integrations, you often receive data from an external system shaped like one type, but you need to work with it as a different type in your system. Without structural types, field-by-field translation is required. With structural types, if both types have identical shape, `DynamicValue` acts as the seamless bridge.

Common scenarios:
- **API gateways** — receive a `PersonDTO` from an external API, decode as your internal `Person` type
- **Message brokers** — consume an event shaped like `UserEvent`, convert to your domain `Account` type
- **Data pipelines** — records with identical fields but different class names from different services

### Cross-type conversion in action

Set up two types with identical structural shape:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

case class Employee(name: String, age: Int)
object Employee {
  implicit val schema: Schema[Employee] = Schema.derived[Employee]
}

val personSchema = Schema.derived[Person]
val employeeSchema = Schema.derived[Employee]
```

Now encode a `Person` to `DynamicValue` and decode it as an `Employee`:

```scala
val person = Person("Alice", 30)
// person: Person = Person(name = "Alice", age = 30)
val dynamic = personSchema.toDynamicValue(person)
// dynamic: DynamicValue = Record(
//   IndexedSeq(("name", Primitive(String("Alice"))), ("age", Primitive(Int(30))))
// )

val employee: Either[SchemaError, Employee] =
  employeeSchema.fromDynamicValue(dynamic)
// employee: Either[SchemaError, Employee] = Right(
//   Employee(name = "Alice", age = 30)
// )
```

The structural shape guarantee ensures type-safe conversion: at compile time, you know both schemas accept the same fields, so round-tripping through `DynamicValue` is safe and zero-cost.

## Integration

Structural types integrate seamlessly with ZIO Blocks' broader ecosystem:

### With Schema Evolution Macros

Structural schemas work with [Schema Evolution](schema-evolution/into.md) macros for cross-type conversion. When two types share the same structural shape, the conversion machinery can work across type boundaries:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

case class PersonDTO(name: String, age: Int)
object PersonDTO {
  implicit val schema: Schema[PersonDTO] = Schema.derived[PersonDTO]
}

// Both types have identical structural schemas
val personSchema = Schema.derived[Person]
val dtoSchema = Schema.derived[PersonDTO]

// They share the same structural shape:
// Schema[{ def name: String; def age: Int }]
```

### With Binding.of (Serialization)

Structural types are also supported by the `Binding.of` macro for high-performance serialization via register-based encoding:

```scala

// Direct structural type serialization (JVM only)
val binding = Binding.of[{ def name: String; def age: Int }]

// Works with nested structural types
val nestedBinding = Binding.of[{
  def name: String
  def address: { def street: String; def city: String }
}]

// Works with containers
val containerBinding = Binding.of[{
  def name: String
  def emails: List[String]
}]
```

This enables anonymous structural types to benefit from ZIO Blocks' high-performance serialization without requiring nominal case class definitions. Like `Schema#structural`, this is **JVM-only**.

See [Binding](./binding.md) for detailed serialization documentation.

## Running the Examples

Example applications demonstrating structural types are available in `schema-examples`:

```sh
# Simple product type
cd schema-examples && sbt "runMain structural.StructuralSimpleProductExample"

# Nested products
cd schema-examples && sbt "runMain structural.StructuralNestedProductExample"

# Sealed trait (Scala 3)
cd schema-examples && sbt "runMain structural.StructuralSealedTraitExample"

# Enum (Scala 3)
cd schema-examples && sbt "runMain structural.StructuralEnumExample"

# Tuples
cd schema-examples && sbt "runMain structural.StructuralTupleExample"

# Integration with Into macro
cd schema-examples && sbt "runMain structural.StructuralIntoExample"
```

---

## Extension Syntax

ZIO Blocks provides convenient extension methods on any value that has a `Schema`. These methods give you fluent, type-safe access to JSON encoding/decoding, pretty-printing, and patching operations directly on your values.

## Import

To use the extension syntax, import the schema package:

```scala

```

This brings the extension methods into scope for any type with an implicit `Schema` instance.

## Quick Example

To encode a `Person` as JSON, parse one back, pretty-print it, and compute and apply a patch between two values:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val alice = Person("Alice", 30)

// Convert to JSON
val json = alice.toJson              // Json AST
val jsonStr = alice.toJsonString     // {"name":"Alice","age":30}
val jsonBytes = alice.toJsonBytes    // Array[Byte]

// Parse from JSON
val parsed = """{"name":"Bob","age":25}""".fromJson[Person]
// Right(Person("Bob", 25))

// Pretty-print
val shown = alice.show               // EJSON text over several lines: { name: "Alice", age: 30 }

// Compute and apply patches
val bob = Person("Bob", 30)
val patch = alice.diff(bob)          // Patch that changes name
val result = alice.applyPatch(patch) // Person("Bob", 30)
```

## JSON Encoding Methods

Three methods encode a value as JSON and differ only in the output: a `Json` AST, a `String`, or UTF-8 bytes.

### toJson

Converts a value to a `Json` AST (abstract syntax tree):

```scala

case class Point(x: Int, y: Int)
object Point {
  implicit val schema: Schema[Point] = Schema.derived
}

val point = Point(10, 20)
val json = point.toJson
// Json.Object(Vector("x" -> Json.Number(10), "y" -> Json.Number(20)))

// Navigate and extract values
json.get("x").as[Int]  // Right(10)
json.get("y").as[Int]  // Right(20)
```

### toJsonString

Converts a value directly to a JSON string:

```scala

case class User(name: String, email: String)
object User {
  implicit val schema: Schema[User] = Schema.derived
}

val user = User("Alice", "alice@example.com")
val jsonStr = user.toJsonString
// {"name":"Alice","email":"alice@example.com"}
```

### toJsonBytes

Converts a value to a UTF-8 encoded byte array. This is useful for efficient serialization when working with binary protocols or network I/O:

```scala

case class Message(id: Long, content: String)
object Message {
  implicit val schema: Schema[Message] = Schema.derived
}

val msg = Message(42, "Hello, world!")
val bytes: Array[Byte] = msg.toJsonBytes

// Useful for sending over the wire
// socket.write(bytes)
```

## JSON Decoding Methods

To decode JSON into a typed value, `StringSchemaOps#fromJson` reads a `String` and `ByteArraySchemaOps#fromJson` reads an `Array[Byte]`; both return `Either[SchemaError, A]`.

### fromJson (on String)

Parses a JSON string into a typed value:

```scala

case class Config(host: String, port: Int)
object Config {
  implicit val schema: Schema[Config] = Schema.derived
}

val jsonStr = """{"host":"localhost","port":8080}"""
val result: Either[SchemaError, Config] = jsonStr.fromJson[Config]
// Right(Config("localhost", 8080))

// Handle parsing errors
val invalid = """{"host":"localhost"}"""  // missing port
val error = invalid.fromJson[Config]
// Left(SchemaError(...))
```

### fromJson (on Array[Byte])

Parses a UTF-8 byte array into a typed value:

```scala

case class Event(name: String, timestamp: Long)
object Event {
  implicit val schema: Schema[Event] = Schema.derived
}

val jsonBytes = """{"name":"click","timestamp":1234567890}"""
  .getBytes(StandardCharsets.UTF_8)

val result: Either[SchemaError, Event] = jsonBytes.fromJson[Event]
// Right(Event("click", 1234567890))
```

## Pretty-Printing

`SchemaValueOps#show` renders any value that has a `Schema` as readable text, for debugging and logging. It converts the value to a `DynamicValue` and prints that:

```scala

case class Address(street: String, city: String, zip: String)
object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

val addr = Address("123 Main St", "Springfield", "12345")
val shown = addr.show
// {
//   street: "123 Main St",
//   city: "Springfield",
//   zip: "12345"
// }
```

This is useful for debugging and logging, as it provides a consistent, schema-aware representation of any value.

## Patching Operations

ZIO Blocks includes a powerful patching system for computing and applying differences between values.

### diff

Computes the difference between two values, returning a `Patch`:

```scala

case class Product(name: String, price: Double, stock: Int)
object Product {
  implicit val schema: Schema[Product] = Schema.derived
}

val before = Product("Widget", 9.99, 100)
val after = Product("Widget", 12.99, 95)

val patch = before.diff(after)
// Patch contains: price changed from 9.99 to 12.99, stock from 100 to 95

patch.isEmpty  // false
```

An identical comparison produces an empty patch:

```scala

case class Item(id: Int, name: String)
object Item {
  implicit val schema: Schema[Item] = Schema.derived
}

val item = Item(1, "Example")
val samePatch = item.diff(item)
samePatch.isEmpty  // true
```

### applyPatch

Applies a patch to a value, returning the modified value. Uses lenient mode by default, which means operations that can't be applied are silently skipped:

```scala

case class Counter(name: String, value: Int)
object Counter {
  implicit val schema: Schema[Counter] = Schema.derived
}

val counter = Counter("hits", 100)
val updated = Counter("hits", 150)

val patch = counter.diff(updated)
val result = counter.applyPatch(patch)
// Counter("hits", 150)
```

### applyPatchStrict

Applies a patch strictly, returning an `Either` that contains an error if any operation fails:

```scala

case class Record(id: String, version: Int)
object Record {
  implicit val schema: Schema[Record] = Schema.derived
}

val record = Record("abc", 1)
val newRecord = Record("abc", 2)
val patch = record.diff(newRecord)

val result: Either[SchemaError, Record] = record.applyPatchStrict(patch)
// Right(Record("abc", 2))

// Empty patch also succeeds
val emptyResult = record.applyPatchStrict(Patch.empty[Record])
// Right(Record("abc", 1))
```

## Roundtrip Examples

A roundtrip turns a value into another form and back again to recover the value; the subsections show one through JSON and one through a patch.

### JSON Roundtrip

```scala

case class Order(id: Long, items: List[String], total: BigDecimal)
object Order {
  implicit val schema: Schema[Order] = Schema.derived
}

val order = Order(12345, List("apple", "banana"), BigDecimal("19.99"))

// String roundtrip
val jsonStr = order.toJsonString
val decoded1 = jsonStr.fromJson[Order]
// Right(Order(12345, List("apple", "banana"), 19.99))

// Bytes roundtrip
val jsonBytes = order.toJsonBytes
val decoded2 = jsonBytes.fromJson[Order]
// Right(Order(12345, List("apple", "banana"), 19.99))
```

### Patch Roundtrip

```scala

case class Settings(theme: String, fontSize: Int, notifications: Boolean)
object Settings {
  implicit val schema: Schema[Settings] = Schema.derived
}

val defaults = Settings("light", 14, true)
val customized = Settings("dark", 16, false)

// Compute patch and apply
val patch = defaults.diff(customized)
val result = defaults.applyPatch(patch)
// Settings("dark", 16, false)

assert(result == customized)
```

## Edge Cases

JSON encoding and decoding also handle inputs that are easy to get wrong: special characters, Unicode text, empty strings and absent optional fields.

### Special Characters

The JSON encoding handles special characters, Unicode, and escape sequences correctly:

```scala

case class Text(content: String)
object Text {
  implicit val schema: Schema[Text] = Schema.derived
}

// Special characters
val special = Text("""John "Jack" O'Brien""")
val json1 = special.toJsonString
val decoded1 = json1.fromJson[Text]
// Right(Text("John \"Jack\" O'Brien"))

// Unicode
val unicode = Text("日本語テキスト")
val json2 = unicode.toJsonString
val decoded2 = json2.fromJson[Text]
// Right(Text("日本語テキスト"))
```

### Empty and Null Values

An empty string and a `None` field both survive a JSON round trip:

```scala

case class Profile(name: String, bio: Option[String])
object Profile {
  implicit val schema: Schema[Profile] = Schema.derived
}

// Empty strings
val empty = Profile("", None)
val json = empty.toJsonString
val decoded = json.fromJson[Profile]
// Right(Profile("", None))

// Optional fields
val withBio = Profile("Alice", Some("Developer"))
val withoutBio = Profile("Bob", None)
```

## Scala 2 vs Scala 3

The extension syntax works identically in both Scala 2 and Scala 3, but the implementation differs:

### Scala 3 (Extension Methods)

```scala
extension [A](self: A) {
  def toJson(using schema: Schema[A]): Json = ...
  def show(using schema: Schema[A]): String = ...
  def diff(that: A)(using schema: Schema[A]): Patch[A] = ...
  // ...
}

extension (self: String) {
  def fromJson[A](using schema: Schema[A]): Either[SchemaError, A] = ...
}

extension (self: Array[Byte]) {
  def fromJson[A](using schema: Schema[A]): Either[SchemaError, A] = ...
}
```

### Scala 2 (Implicit Classes)

```scala
implicit final class SchemaValueOps[A](self: A) {
  def toJson(implicit schema: Schema[A]): Json = ...
  def show(implicit schema: Schema[A]): String = ...
  def diff(that: A)(implicit schema: Schema[A]): Patch[A] = ...
  // ...
}

implicit final class StringSchemaOps(self: String) {
  def fromJson[A](implicit schema: Schema[A]): Either[SchemaError, A] = ...
}

implicit final class ByteArraySchemaOps(self: Array[Byte]) {
  def fromJson[A](implicit schema: Schema[A]): Either[SchemaError, A] = ...
}
```

The API is the same—just import `zio.blocks.schema._` and the appropriate syntax is provided for your Scala version.

## Method Reference

| Method             | Receiver      | Return Type              | Description                    |
|--------------------|---------------|--------------------------|--------------------------------|
| `toJson`           | `A`           | `Json`                   | Convert to JSON AST            |
| `toJsonString`     | `A`           | `String`                 | Convert to JSON string         |
| `toJsonBytes`      | `A`           | `Array[Byte]`            | Convert to UTF-8 bytes         |
| `fromJson[A]`      | `String`      | `Either[SchemaError, A]` | Parse JSON string              |
| `fromJson[A]`      | `Array[Byte]` | `Either[SchemaError, A]` | Parse JSON bytes               |
| `show`             | `A`           | `String`                 | Pretty-print via DynamicValue  |
| `diff`             | `A`           | `Patch[A]`               | Compute patch to another value |
| `applyPatch`       | `A`           | `A`                      | Apply patch (lenient)          |
| `applyPatchStrict` | `A`           | `Either[SchemaError, A]` | Apply patch (strict)           |

All methods except `applyPatch` and `applyPatchStrict` require an implicit/given `Schema[A]` in scope.

See also the [Your First Schema](../../guides/your-first-schema.md) tutorial for a step-by-step walkthrough of `SchemaValueOps#toJsonString` and `StringSchemaOps#fromJson`.

---

## Type Class Derivation

Type classes are one of the most powerful abstraction mechanisms in functional programming. Originating from Haskell, they enable ad-hoc polymorphism—the ability to define generic behavior that can be extended to new types without modifying those types. ZIO Blocks has a robust type class derivation system built around the `Deriver` trait, which allows automatic generation of type class instances for any data type with an associated `Schema`.

The `Deriver` trait is a cornerstone of ZIO Blocks' type class derivation system. It provides a unified, elegant mechanism for automatically generating type class instances (such as codecs) for any data type that has a `Schema`. Unlike traditional macro-based derivation approaches, `Deriver` requires implementing only a few methods to enable full type class derivation with rich reflective metadata support for every use case.

## The Problem

In functional programming, type classes allow us to define generic behavior that can be extended to new types without modifying those types. However, manually writing type class instances for every data type can be tedious and error-prone, especially as the number of types grows. This is where automatic derivation comes in.

Consider a typical application with 50 domain types that needs 4 type classes (JSON codec, Avro codec, hashing, ordering). That's 200 type class instances to write and maintain manually (50 types × 4 type classes). 

Each instance requires understanding both the type's structure and the type class's semantics, then correctly implementing encoding, decoding, or whatever operation is required. This quickly becomes unmanageable as the codebase grows.

Assume we have a simple `JsonTC` type class for JSON serialization and deserialization:

```scala

sealed abstract class JsonError(msg: String) extends Exception(msg)

case class ParseError(details: String) 
  extends JsonError(s"Parse Error: $details")

case class DecodeError(details: String, path: String) 
  extends JsonError(s"Decode Error at '$path': $details")

trait JsonTC[A] {
  def encode(a: A): Json
  def decode(j: Json): Either[JsonError, A]
}
```

A single manual codec for a simple type like `Person` looks like the following code. You can imagine how complex it gets for larger types and more type classes:

```scala
case class Person(name: String, age: Int)

object Person {
  implicit val personCodec: JsonTC[Person] =
    new JsonTC[Person] {
      def encode(c: Person): Json = Json.obj(
        "name" -> Json.str(c.name),
        "age"  -> Json.number(c.age)
      )

      def decode(j: Json): Either[JsonError, Person] =
        for {
          name <- j.get("name").asString.string
          age  <- j.get("age").asNumber.int
        } yield Person(name, age)
    }
}
```

This manual approach is not only time-consuming but also prone to errors and inconsistencies. As the number of types and type classes increases, the maintenance burden grows significantly.

## The Solution: Automatic Derivation with `Deriver`

The `Deriver` trait provides a powerful and flexible way to automatically derive type class instances for any data type with an associated `Schema`. By implementing just seven methods, you can enable full derivation for primitive types, records, variants, sequences, maps, dynamic values, and wrappers.

ZIO Blocks recognizes that all data types reduce to a small set of structural patterns (as outlined in the `Reflect` documentation):

| Pattern       | Description                     | Examples                           |
|---------------|---------------------------------|------------------------------------|
| **Primitive** | Atomic values                   | `String`, `Int`, `UUID`, `Instant` |
| **Record**    | Product types with named fields | Case classes, tuples               |
| **Variant**   | Sum types with named cases      | Sealed traits, enums               |
| **Sequence**  | Ordered collections             | `List`, `Vector`, `Array`          |
| **Map**       | Key-value collections           | `Map`, `HashMap`                   |
| **Dynamic**   | Schema-less data                | `DynamicValue`, arbitrary JSON     |
| **Wrapper**   | Newtypes and opaque types       | `opaque type Age = Int`            |

If you define how to derive type-class instances for all these patterns, then ZIO Blocks has all the pieces needed to build type-class instances for any data type. This is what the `Deriver[TC[_]]` is responsible for. A `Deriver[TC[_]]` defines how to create `TC[A]` instances for each kind of schema node:

```scala
trait Deriver[TC[_]] {
  def derivePrimitive[A](...)                     : Lazy[TC[A]]
  def deriveRecord   [F[_, _], A](...)            : Lazy[TC[A]]
  def deriveVariant  [F[_, _], A](...)            : Lazy[TC[A]]
  def deriveSequence [F[_, _], C[_], A](...)      : Lazy[TC[C[A]]]
  def deriveMap      [F[_, _], M[_, _], K, V](...): Lazy[TC[M[K, V]]]
  def deriveDynamic  [F[_, _]](...)               : Lazy[TC[DynamicValue]]
  def deriveWrapper  [F[_, _], A, B](...)         : Lazy[TC[A]]
}
```

Conceptually, the `Deriver` interface operates at the meta level, acting as a type class for type class derivation. It takes a higher-kinded type parameter `TC[_]`, which represents the type class to be derived (e.g., `JsonCodec`, `Ordering`, `Eq`, etc.), and defines seven methods, each corresponding to the derivation of the type class for one of the structural patterns.

That's it. As a developer who wants to implement automatic derivation for a new type class, you only need to implement these 7 methods. Each receives all the information needed to build a type class instance such as field names, type names, bindings for construction/deconstruction, documentation, and modifiers.

Looking at the return type of each method, you'll notice they all return the type class wrapped in a `Lazy` container, i.e., `Lazy[TC[_]]`, not just `TC[_]`. This is crucial for handling recursive data types safely. While the `Deriver` system traverses the schema structure to generate type-class instances or codecs, it may encounter recursive data types. To prevent stack overflows caused by unbounded recursion and infinite loops, ZIO Blocks uses the `Lazy` data type, which is a trampolined, memoizing lazy evaluation monad that defers computation until `Lazy#force` is called. It provides stack-safe evaluation through continuation-passing style (CPS), along with error-handling capabilities and composable operations.

Each method (except the `derivePrimitive` method) also receives implicit parameters of type class instances for `HasBinding` and `HasInstance`:
1. **`HasBinding[F]`**: Provides access to the structural binding information (constructors, deconstructors, matchers, discriminators, etc.) for the contained types, e.g., fields of a record or cases of a variant, allowing us to understand how to construct and deconstruct values of those types.
2. **`HasInstance[F, TC]`**: Provides access to already-(provided/derived) type class instances for nested types or fields. This allows you to build type class instances for complex types by composing instances of their constituent parts.

As an example, the `deriveRecord` method signature looks like this:

```scala
trait Deriver[TC[_]] {
  // other methods...
  
  def deriveRecord[F[_, _], A](
    fields: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Record[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[TC[A]]
  
  // other methods...
}
```

The other methods follow a similar pattern, each tailored to the specific structural pattern they handle.

The underlying derivation engine takes care of traversing the schema structure, applying the appropriate derivation method for each structural pattern, and composing the resulting type class instances together. This means that once you've implemented a `Deriver` for a specific type class, you can automatically derive instances for any data type with a schema, without writing any additional boilerplate code.

## Using the `Deriver` to Derive Type Class Instances

Given a `Schema[A]`, you can call the `derive` method to get an instance of the type class `TC[A]`:

```scala
case class Schema[A](reflect: Reflect.Bound[A]) {
   def derive[D, TC[_]](d: D)(implicit ev: Derivable[D, TC]): TC[A] = ???
}
```

It takes a `Deriver[TC]` as a parameter and returns a type class instance of type `TC[A]`. For example, in the following code snippet, we derive a `JsonCodec[Person]` instance for the `Person` case class using the `JsonCodecDeriver`:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

val jsonCodec: JsonCodec[Person] = Person.schema.derive(JsonCodecDeriver)

val result: Either[SchemaError, Person] = 
  jsonCodec.decode(
    """
      |{
      |  "name": "Alice",
      |  "age": 30
      |}
      |""".stripMargin
  )
```

There is a `Derivable` type class that enables seamless overloading between `Deriver[TC]` and `Format` arguments, allowing the same `derive` method to work with both:

For example, by calling `Person.schema.derive(JsonFormat)`, we can derive a `JsonCodec[Person]` instance:

```scala

val jsonCodec = Person.schema.derive(JsonFormat)
```

## Customizing Derivation with Instance and Modifier Overrides

By default, `Deriver` automatically derives codecs for all types. But sometimes you need to customize how specific types are encoded or decoded—for example, encoding `LocalDate` as `"dd/MM/yyyy"` instead of ISO format.

ZIO Blocks provides three levels of customization, in progressive order:

1. **Type-level override** (`Deriver.withInstance`) — Override the codec for ALL occurrences of a type. Configure once, use everywhere:

```scala

case class Event(name: String, date: LocalDate)
object Event {
  implicit val schema: Schema[Event] = Schema.derived
}

case class Meeting(topic: String, scheduledOn: LocalDate)
object Meeting {
  implicit val schema: Schema[Meeting] = Schema.derived
}

// Create a custom JsonCodec for LocalDate
val customDateCodec: JsonCodec[LocalDate] = new JsonCodec[LocalDate] {
  private val fmt = DateTimeFormatter.ofPattern("dd/MM/yyyy")

  def decodeValue(in: JsonReader): LocalDate           = LocalDate.parse(in.readString(), fmt)
  def encodeValue(x: LocalDate, out: JsonWriter): Unit = out.writeVal(fmt.format(x))

  // Optional: support for the Json AST, used by `decode(json: Json)` and `encodeToJson`
  override def decodeValue(json: Json): LocalDate = json match {
    case s: Json.String => LocalDate.parse(s.value, fmt)
    case _              => error("expected a JSON string")
  }
  override def encodeValue(x: LocalDate): Json = Json.String(fmt.format(x))
}

// Configure the deriver once
val myDeriver = JsonCodecDeriver.withInstance[LocalDate](customDateCodec)

// Use everywhere — all LocalDate fields use the custom codec
val codec1 = Schema[Event].deriving(myDeriver).derive
val codec2 = Schema[Meeting].deriving(myDeriver).derive
```

2. **Field-level override** (`Deriver.withInstance` with typeId + termName) — Override a specific field only:

```scala

case class Event(name: String, date: LocalDate, createdOn: LocalDate)
object Event {
  implicit val schema: Schema[Event] = Schema.derived
}

val customDateCodec: JsonCodec[LocalDate] = new JsonCodec[LocalDate] {
  private val fmt = DateTimeFormatter.ofPattern("dd/MM/yyyy")

  def decodeValue(in: JsonReader): LocalDate           = LocalDate.parse(in.readString(), fmt)
  def encodeValue(x: LocalDate, out: JsonWriter): Unit = out.writeVal(fmt.format(x))
}

// Only Event.date uses the custom format; Event.createdOn and other LocalDate fields are unchanged
val deriver = JsonCodecDeriver.withInstance[Event, LocalDate](
  TypeId.of[Event], "date", customDateCodec
)
```

3. **Modifier override** (`Deriver.withModifier`) — Rename fields, add aliases:

```scala

case class Person(firstName: String, lastName: String)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val deriver = JsonCodecDeriver.withModifier(
  TypeId.of[Person], "firstName", Modifier.rename("first_name")
)
```

### Chaining Overrides

Overrides compose, allowing you to build complex derivers incrementally:

```scala

case class Event(name: String, date: LocalDate)
object Event {
  implicit val schema: Schema[Event] = Schema.derived
}

val customDateCodec: JsonCodec[LocalDate] = new JsonCodec[LocalDate] {
  private val fmt = DateTimeFormatter.ofPattern("dd/MM/yyyy")

  def decodeValue(in: JsonReader): LocalDate           = LocalDate.parse(in.readString(), fmt)
  def encodeValue(x: LocalDate, out: JsonWriter): Unit = out.writeVal(fmt.format(x))
}

val myDeriver = JsonCodecDeriver
  .withInstance[LocalDate](customDateCodec)
  .withModifier(TypeId.of[Event], "name", Modifier.rename("title"))
```

### Important Notes

- `withInstance` and `withModifier` return a NEW deriver (they are immutable).
- When combined with `DerivationBuilder`, deriver-level instance overrides take precedence over builder-level instance overrides for the same target. Modifier overrides do not replace each other: the deriver receives the builder's modifiers before the deriver's, and `JsonCodecDeriver` uses the first `Modifier.rename` it finds, so for renames the builder's override wins.
- Unknown `termName` values are silently ignored.
- The `B` type parameter in field-level `withInstance` is not statically checked against the actual field type.

See also: [Override the Codec for One Type](../../guides/override-codec-for-one-type.md) for a step-by-step guide to type-level overrides.

See also: [How to Derive Your Own Type Class from a Schema](../../guides/derive-your-own-type-class.md) for a step-by-step guide to writing a `Deriver` for your own type class, and [How to Write a Codec for a New Wire Format](../../guides/write-a-codec-for-a-new-wire-format.md) for a `Deriver` that builds a binary codec, decoding included, and packages it as a `BinaryFormat`.

## Example 1: Deriving a `Show` Type Class Instance

Let's say we want to derive a `Show` type class instance for any type of type `A`:

```scala
trait Show[A] {
  def show(value: A): String
}
```

The implementation of the `Deriver[Show]` would look like the following code. Don't worry about understanding every detail right now; we'll break down the derivation process step by step afterward.

```scala

object DeriveShow extends Deriver[Show] {

  override def derivePrimitive[A](
    primitiveType: PrimitiveType[A],
    typeId: TypeId[A],
    binding: Binding.Primitive[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  ): Lazy[Show[A]] =
    Lazy {
      new Show[A] {
        def show(value: A): String = primitiveType match {
          case _: PrimitiveType.String => "\"" + value + "\""
          case _: PrimitiveType.Char   => "'" + value + "'"
          case _                       => String.valueOf(value)
        }
      }
    }

  override def deriveRecord[F[_, _], A](
    fields: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Record[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
    // Pre-compute structural setup outside Lazy
    val fieldNames    = fields.map(_.name)
    // Cast fields to use Binding as F (we are going to create Reflect.Record with Binding as F)
    val recordFields  = fields.asInstanceOf[IndexedSeq[Term[Binding, A, ?]]]
    // Cast to Binding.Record to access constructor/deconstructor
    val recordBinding = binding.asInstanceOf[Binding.Record[A]]
    // Build a Reflect.Record to get access to the computed registers for each field
    val recordReflect = new Reflect.Record[Binding, A](recordFields, typeId, recordBinding, doc, modifiers)
    Lazy {
      new Show[A] {
        // Defer child-instance resolution to first show() call via lazy val.
        // For recursive types (e.g. case class Tree(children: List[Tree])), accessing
        // field.value.metadata during derivation would re-enter a Deferred node that is
        // still initializing, causing an infinite loop. The lazy val here ensures access
        // only happens after the framework has finished deriving all instances.
        private lazy val resolvedShows: IndexedSeq[Show[Any]] =
          fields.map(field => D.instance(field.value.metadata).asInstanceOf[Lazy[Show[Any]]].force)
        def show(value: A): String = {
          // Create registers with space for all used registers to hold deconstructed field values
          val registers = Registers(recordReflect.usedRegisters)
          // Deconstruct field values of the record into the registers
          recordBinding.deconstructor.deconstruct(registers, RegisterOffset.Zero, value)
          // Build string representations for all fields
          val fieldStrings = fields.indices.map { i =>
            val fieldValue = recordReflect.registers(i).get(registers, RegisterOffset.Zero)
            s"${fieldNames(i)} = ${resolvedShows(i).show(fieldValue)}"
          }
          s"${typeId.name}(${fieldStrings.mkString(", ")})"
        }
      }
    }
  }

  override def deriveVariant[F[_, _], A](
    cases: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Variant[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
    // Collect the Lazy[Show] references for each case outside the Lazy block.
    // Capturing Lazy refs is safe here; we only .force them later inside the lazy val.
    val caseShowLazies: IndexedSeq[Lazy[Show[Any]]] = cases.map { case_ =>
      D.instance(case_.value.metadata).asInstanceOf[Lazy[Show[Any]]]
    }
    // Cast binding to Binding.Variant to access discriminator and matchers
    val variantBinding = binding.asInstanceOf[Binding.Variant[A]]
    Lazy {
      new Show[A] {
        // Force child instances lazily — same recursive-safety rationale as deriveRecord
        private lazy val resolvedShows: IndexedSeq[Show[Any]] = caseShowLazies.map(_.force)
        // Implement show by using discriminator and matchers to find the right case
        // The `value` parameter is of type A (the variant type), e.g. a Shape value
        def show(value: A): String = {
          // Use discriminator to determine which case this value belongs to
          val caseIndex = variantBinding.discriminator.discriminate(value)
          // Use matcher to downcast to the specific case type
          val caseValue = variantBinding.matchers(caseIndex).downcastOrNull(value)
          // Delegate to the case's Show instance — it already knows its own name
          resolvedShows(caseIndex).show(caseValue)
        }
      }
    }
  }

  override def deriveSequence[F[_, _], C[_], A](
    element: Reflect[F, A],
    typeId: TypeId[C[A]],
    binding: Binding.Seq[C, A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[C[A]],
    examples: Seq[C[A]]
  )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[C[A]]] = {
    // Cast binding to Binding.Seq to access the deconstructor
    val deconstructor = binding.asInstanceOf[Binding.Seq[C, A]].deconstructor
    // Sequences are structurally non-recursive, so we can use monadic .map composition.
    // instance(...).map { elementShow => ... } returns a Lazy that, when forced, builds
    // a Show[C[A]] with elementShow already resolved — no .force needed at show()-time.
    D.instance(element.metadata).map { elementShow =>
      new Show[C[A]] {
        def show(value: C[A]): String = {
          // Use deconstructor to iterate over elements and show each one
          val elements = deconstructor.deconstruct(value).map(elementShow.show)
          s"[${elements.mkString(", ")}]"
        }
      }
    }
  }

  override def deriveMap[F[_, _], M[_, _], K, V](
    key: Reflect[F, K],
    value: Reflect[F, V],
    typeId: TypeId[M[K, V]],
    binding: Binding.Map[M, K, V],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[M[K, V]],
    examples: Seq[M[K, V]]
  )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[M[K, V]]] = {
    // Cast binding to Binding.Map to access the deconstructor
    val deconstructor = binding.asInstanceOf[Binding.Map[M, K, V]].deconstructor
    // Maps are non-recursive: use .zip to pair the two child Lazy instances, then .map
    // to build the Show[M[K,V]] with both keyShow and valueShow already resolved.
    D.instance(key.metadata).zip(D.instance(value.metadata)).map { case (keyShow, valueShow) =>
      new Show[M[K, V]] {
        def show(m: M[K, V]): String = {
          // Use deconstructor to iterate over key-value pairs
          val entries = deconstructor.deconstruct(m).map { kv =>
            val k = deconstructor.getKey(kv)
            val v = deconstructor.getValue(kv)
            s"${keyShow.show(k)} -> ${valueShow.show(v)}"
          }.mkString(", ")
          s"Map($entries)"
        }
      }
    }
  }

  override def deriveDynamic[F[_, _]](
    binding: Binding.Dynamic,
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[DynamicValue],
    examples: Seq[DynamicValue]
  )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[DynamicValue]] = Lazy {
    new Show[DynamicValue] {
      def show(value: DynamicValue): String =
        value match {
          case DynamicValue.Primitive(pv) =>
            value.toString

          case DynamicValue.Record(fields) =>
            val fieldStrings = fields.map { case (name, v) =>
              s"$name = ${show(v)}"
            }
            s"Record(${fieldStrings.mkString(", ")})"

          case DynamicValue.Variant(caseName, v) =>
            s"$caseName(${show(v)})"

          case DynamicValue.Sequence(elements) =>
            val elemStrings = elements.map(show)
            s"[${elemStrings.mkString(", ")}]"

          case DynamicValue.Map(entries) =>
            val entryStrings = entries.map { case (k, v) =>
              s"${show(k)} -> ${show(v)}"
            }
            s"Map(${entryStrings.mkString(", ")})"
          case Null =>
            "null"
        }
    }
  }

  override def deriveWrapper[F[_, _], A, B](
    wrapped: Reflect[F, B],
    typeId: TypeId[A],
    binding: Binding.Wrapper[A, B],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
    // Cast binding to Binding.Wrapper to access the unwrap function
    val wrapperBinding = binding.asInstanceOf[Binding.Wrapper[A, B]]
    // Wrappers are non-recursive: use .map so wrappedShow is already resolved
    // when show() is called — no .force needed.
    D.instance(wrapped.metadata).map { wrappedShow =>
      new Show[A] {
        def show(value: A): String = {
          // Unwrap the value to access the underlying type B, then delegate to its Show
          val unwrapped = wrapperBinding.unwrap(value)
          s"${typeId.name}(${wrappedShow.show(unwrapped)})"
        }
      }
    }
  }
}
```

Now let's see how the derivation process works step by step.

### Primitive Derivation

When the derivation process encounters a primitive type (e.g., `String`, `Int`), it calls the `derivePrimitive` method of the `Deriver`. This method receives the `PrimitiveType[A]` information, which allows it to determine how to encode and decode values of that type:

```scala
def derivePrimitive[A](
  primitiveType: PrimitiveType[A],
  typeId: TypeId[A],
  binding: Binding.Primitive[A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
): Lazy[Show[A]] =
  Lazy {
    new Show[A] {
      def show(value: A): String = primitiveType match {
        case _: PrimitiveType.String => "\"" + value + "\""
        case _: PrimitiveType.Char   => "'" + value + "'"
        case _                       => String.valueOf(value)
      }
    }
  }
```

Please note that for our simple `Show` type class, we only need to know the `PrimitiveType` to determine how to show the value. However, for more complex type classes you might require additional information from the other parameters (e.g., documentation, modifiers, default values, examples) to build a more sophisticated type class instance.

To make it simple, we only handle `String` and `Char` differently by adding quotes around them, while for all other primitive types we simply call `String.valueOf(value)` to get their string representation. You can easily extend this logic to handle other primitive types differently if needed.

### Record Derivation

When the derivation process encounters a record type (e.g., a case class), it calls the `deriveRecord` method of the `Deriver`. This method receives an `IndexedSeq[Term[F, A, ?]]` representing the fields of the record, along with other metadata such as the type ID, binding information, documentation, modifiers, default values, and examples. It also receives implicit parameters for accessing structural bindings and already-derived type class instances for nested types:

```scala
def deriveRecord[F[_, _], A](
  fields: IndexedSeq[Term[F, A, ?]],
  typeId: TypeId[A],
  binding: Binding.Record[A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
)(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
  // Pre-compute structural setup outside Lazy — none of this touches Deferred nodes
  val fieldNames    = fields.map(_.name)
  // Cast fields to use Binding as F (we are going to create Reflect.Record with Binding as F)
  val recordFields  = fields.asInstanceOf[IndexedSeq[Term[Binding, A, ?]]]
  // Cast to Binding.Record to access constructor/deconstructor
  val recordBinding = binding.asInstanceOf[Binding.Record[A]]
  // Build a Reflect.Record to get access to the computed registers for each field
  val recordReflect = new Reflect.Record[Binding, A](recordFields, typeId, recordBinding, doc, modifiers)
  Lazy {
    new Show[A] {
      // Defer child-instance resolution to first show() call via lazy val.
      // For recursive types (e.g. case class Tree(children: List[Tree])), accessing
      // field.value.metadata during derivation would re-enter a Deferred node that is
      // still initialising, causing an infinite loop. The lazy val here ensures access
      // only happens after the framework has finished deriving all instances.
      private lazy val resolvedShows: IndexedSeq[Show[Any]] =
        fields.map(field => D.instance(field.value.metadata).asInstanceOf[Lazy[Show[Any]]].force)
      def show(value: A): String = {
        // Create registers with space for all used registers to hold deconstructed field values
        val registers = Registers(recordReflect.usedRegisters)
        // Deconstruct field values of the record into the registers
        recordBinding.deconstructor.deconstruct(registers, RegisterOffset.Zero, value)
        // Build string representations for all fields
        val fieldStrings = fields.indices.map { i =>
          val fieldValue = recordReflect.registers(i).get(registers, RegisterOffset.Zero)
          s"${fieldNames(i)} = ${resolvedShows(i).show(fieldValue)}"
        }
        s"${typeId.name}(${fieldStrings.mkString(", ")})"
      }
    }
  }
}
```

The `deriveRecord` method demonstrates derivation mechanics for record types such as case classes and tuples. To derive the type class for a record type, we follow these steps:
1. First, we pre-compute structural setup (field names, `Reflect.Record`, binding casts) outside the `Lazy` block — none of this accesses `Deferred` nodes.
2. Second, inside `Lazy`, we build a `new Show[A]` with a `private lazy val resolvedShows` that resolves child instances on demand.
3. Third, when `show()` is first called, `resolvedShows` forces the child `Lazy[Show]` instances and then deconstructs the record value into its individual fields.
4. Finally, we assemble the string representation by combining field names and their representations.

During the first step, the structural setup is computed outside the `Lazy` block so the `Lazy` thunk itself is lightweight. This separation is important because structural types like `Reflect.Record` don't access `Deferred` schema nodes that might not yet be initialized.

The key insight is the `private lazy val resolvedShows` inside the `new Show[A]` class. It gathers `Lazy[Show]` instances for each field by calling `D.instance(field.value.metadata)`, then immediately forces them. These calls happen lazily — only on the first invocation of `show()` — which is safe because by that time the framework has completed derivation and all `Lazy` values are fully memoized. Without this deferral, recursive types like `case class Tree(value: Int, children: List[Tree])` would cause an infinite loop: deriving `Show[Tree]` requires `Show[List[Tree]]`, which requires `Show[Tree]` again.

Our goal is to build a `String` representation of the record in the format `TypeName(field1 = value1, field2 = value2, ...)`. To achieve this, we need to access the individual field values of the record at runtime. To do this, we have to deconstruct the record value, which is given to the `show(value: A)` method, into its individual fields.

To deconstruct the record, we use the `Binding.Record[A]` that was provided as a parameter to the `deriveRecord` method. This binding contains a `deconstructor` that knows how to extract all field values from a record of type `A`. To perform the deconstruction, we should first allocate register buffers to hold the deconstructed field values. But how do we know what the size of the register buffer should be? This is where the `Reflect.Record` comes in. By building a `Reflect.Record[Binding, A]` from the field definitions, we can compute the number of registers needed to hold all field values through `Reflect#usedRegisters`. The `Registers(recordReflect.usedRegisters)` call allocates a register buffer with the appropriate size to hold all field values of the record.

Now we are ready to deconstruct the `A` value, using the `Binding.Record#deconstructor.deconstruct(registers, RegisterOffset.Zero, value)` call, which extracts the field values of the record into this register buffer in a single pass. Now the field values are stored in `registers`.

The next question is how we can access the field values from the registers? The `Reflect.Record` we built earlier also computes the register layout for each field, which allows us to retrieve each field value from the appropriate register slot using `recordReflect.registers(i).get(registers, RegisterOffset.Zero)`. This call accesses the `i`-th field's value from the registers based on the register layout computed by `Reflect.Record`.

Finally, we iterate through each field, retrieve its value from the registers, look up the already-resolved `Show` instance for that field's type from `resolvedShows`, and format the result as `fieldName = fieldValue`. The output assembles into the familiar `TypeName(field1 = value1, field2 = value2)` representation.

### Variant Derivation

When the derivation process encounters a variant type (e.g., a sealed trait with case classes), it calls the `deriveVariant` method of the `Deriver`. This method receives an `IndexedSeq[Term[F, A, _]]` representing the cases of the variant, along with other metadata such as the type ID, binding information, documentation, modifiers, default values, and examples:

```scala
def deriveVariant[F[_, _], A](
  cases: IndexedSeq[Term[F, A, ?]],
  typeId: TypeId[A],
  binding: Binding.Variant[A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
)(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
  // Collect the Lazy[Show] references for each case outside the Lazy block.
  // Capturing Lazy refs is safe here; we only .force them later inside the lazy val.
  val caseShowLazies: IndexedSeq[Lazy[Show[Any]]] = cases.map { case_ =>
    D.instance(case_.value.metadata).asInstanceOf[Lazy[Show[Any]]]
  }
  // Cast binding to Binding.Variant to access discriminator and matchers
  val variantBinding = binding.asInstanceOf[Binding.Variant[A]]
  Lazy {
    new Show[A] {
      // Force child instances lazily — same recursive-safety rationale as deriveRecord
      private lazy val resolvedShows: IndexedSeq[Show[Any]] = caseShowLazies.map(_.force)
      // Implement show by using discriminator and matchers to find the right case
      // The `value` parameter is of type A (the variant type), e.g. a Shape value
      def show(value: A): String = {
        // Use discriminator to determine which case this value belongs to
        val caseIndex = variantBinding.discriminator.discriminate(value)
        // Use matcher to downcast to the specific case type
        val caseValue = variantBinding.matchers(caseIndex).downcastOrNull(value)
        // Delegate to the case's Show instance — it already knows its own name
        resolvedShows(caseIndex).show(caseValue)
      }
    }
  }
}
```

The derivation process for variants is similar to records, but instead of fields, we have cases. The `Lazy[Show]` references for all cases are collected outside the `Lazy` block — this is safe because it only captures references without forcing them. Inside `Lazy`, a `private lazy val resolvedShows` forces all the child instances on the first `show()` call, applying the same recursive-safety guarantee as in `deriveRecord`.

At runtime, we use the discriminator to determine which case the value belongs to, and then the matcher to downcast the value to the specific case type. Finally, we look up the already-resolved `Show` instance for that case from `resolvedShows` and delegate to it. Each case's `Show` instance already knows how to format itself (including its own type name), so no additional formatting is needed at the variant level.

### Sequence Derivation

When the derivation process encounters a sequence type (e.g., `List[A]`), it calls the `deriveSequence` method of the `Deriver`. This method receives a `Reflect[F, A]` representing the element type of the sequence, along with other metadata such as the type ID, binding information, documentation, modifiers, default values, and examples:

```scala
def deriveSequence[F[_, _], C[_], A](
  element: Reflect[F, A],
  typeId: TypeId[C[A]],
  binding: Binding.Seq[C, A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[C[A]],
  examples: Seq[C[A]]
)(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[C[A]]] = {
  // Cast binding to Binding.Seq to access the deconstructor
  val deconstructor = binding.asInstanceOf[Binding.Seq[C, A]].deconstructor
  // Sequences are structurally non-recursive, so we can use monadic .map composition.
  // instance(...).map { elementShow => ... } returns a Lazy that, when forced, builds
  // a Show[C[A]] with elementShow already resolved — no .force needed at show()-time.
  D.instance(element.metadata).map { elementShow =>
    new Show[C[A]] {
      def show(value: C[A]): String = {
        // Use deconstructor to iterate over elements and show each one
        val elements = deconstructor.deconstruct(value).map(elementShow.show)
        s"[${elements.mkString(", ")}]"
      }
    }
  }
}
```

The derivation process for sequences is straightforward. Because sequences are structurally non-recursive, we can use monadic `Lazy` composition via `.map` instead of the `private lazy val` pattern needed for records and variants. `D.instance(element.metadata).map { elementShow => ... }` returns a `Lazy[Show[C[A]]]` that, when forced, produces a `Show[C[A]]` with `elementShow` already resolved — no explicit `.force` is needed inside `show()`. At runtime, we use the deconstructor to iterate over the elements of the sequence and call `elementShow.show` on each one. Finally, we combine all element representations into a bracketed string.

### Map Derivation

When the derivation process encounters a map type (e.g., `Map[K, V]`), it calls the `deriveMap` method of the `Deriver`. This method receives `Reflect[F, K]` and `Reflect[F, V]` representing the key and value types of the map, along with other metadata such as the type ID, binding information, documentation, modifiers, default values, and examples:

```scala
def deriveMap[F[_, _], M[_, _], K, V](
  key: Reflect[F, K],
  value: Reflect[F, V],
  typeId: TypeId[M[K, V]],
  binding: Binding.Map[M, K, V],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[M[K, V]],
  examples: Seq[M[K, V]]
)(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[M[K, V]]] = {
  // Cast binding to Binding.Map to access the deconstructor
  val deconstructor = binding.asInstanceOf[Binding.Map[M, K, V]].deconstructor
  // Maps are non-recursive: use .zip to pair the two child Lazy instances, then .map
  // to build the Show[M[K,V]] with both keyShow and valueShow already resolved.
  D.instance(key.metadata).zip(D.instance(value.metadata)).map { case (keyShow, valueShow) =>
    new Show[M[K, V]] {
      def show(m: M[K, V]): String = {
        // Use deconstructor to iterate over key-value pairs
        val entries = deconstructor.deconstruct(m).map { kv =>
          val k = deconstructor.getKey(kv)
          val v = deconstructor.getValue(kv)
          s"${keyShow.show(k)} -> ${valueShow.show(v)}"
        }.mkString(", ")
        s"Map($entries)"
      }
    }
  }
}
```

The derivation process for maps is similar to sequences, but we have two child instances to compose. Maps are non-recursive, so we use `.zip` to pair the two child `Lazy` instances into a single `Lazy[(Show[K], Show[V])]`, then `.map` to build the `Show[M[K,V]]` with both `keyShow` and `valueShow` already resolved. At runtime, we use the deconstructor to iterate over the key-value pairs of the map, calling `keyShow.show` and `valueShow.show` on each pair. No explicit `.force` is needed since `.map` already handles forcing. Finally, we combine all entries into a `Map(...)` string.

### Dynamic Derivation

When the derivation process encounters a dynamic type (e.g., `DynamicValue`), it calls the `deriveDynamic` method of the `Deriver`. This method receives a `Binding.Dynamic` representing the dynamic type, along with other metadata such as documentation, modifiers, default values, and examples:

```scala
def deriveDynamic[F[_, _]](
  binding: Binding.Dynamic,
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[DynamicValue],
  examples: Seq[DynamicValue]
)(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[DynamicValue]] = Lazy {
  new Show[DynamicValue] {
    def show(value: DynamicValue): String =
      value match {
        case DynamicValue.Primitive(pv) =>
          value.toString
        case DynamicValue.Record(fields) =>
          val fieldStrings = fields.map { case (name, v) =>
            s"$name = ${show(v)}"
          }
          s"Record(${fieldStrings.mkString(", ")})"
        case DynamicValue.Variant(caseName, v) =>
          s"$caseName(${show(v)})"
        case DynamicValue.Sequence(elements) =>
          val elemStrings = elements.map(show)
          s"[${elemStrings.mkString(", ")}]"
        case DynamicValue.Map(entries) =>
          val entryStrings = entries.map { case (k, v) =>
            s"${show(k)} -> ${show(v)}"
          }
          s"Map(${entryStrings.mkString(", ")})"
        case Null =>
          "null"
      }
  }
}
```

The derivation process for dynamic types is more complex because the data structure is not known at compile time. Instead, we must handle different cases based on the runtime type of `DynamicValue` using pattern matching. For each subtype: `Primitive` values are converted via `toString`, `Record` fields are recursively shown, `Variant` cases display the name and contained value, `Sequence` elements are shown in bracket notation, `Map` entries are displayed as key-value pairs, and `Null` returns the string "null".

### Wrapper Derivation

When the derivation process encounters a wrapper type (e.g., a value class, opaque type, or any type that wraps another type), it calls the `deriveWrapper` method of the `Deriver`. This method receives a `Reflect[F, B]` representing the wrapped (underlying) type, along with other metadata such as the type ID, binding information, documentation, modifiers, default values, and examples:

```scala
def deriveWrapper[F[_, _], A, B](
  wrapped: Reflect[F, B],
  typeId: TypeId[A],
  binding: Binding.Wrapper[A, B],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
)(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
  // Cast binding to Binding.Wrapper to access the unwrap function
  val wrapperBinding = binding.asInstanceOf[Binding.Wrapper[A, B]]
  // Wrappers are non-recursive: use .map so wrappedShow is already resolved
  // when show() is called — no .force needed.
  D.instance(wrapped.metadata).map { wrappedShow =>
    new Show[A] {
      def show(value: A): String = {
        // Unwrap the value to access the underlying type B, then delegate to its Show
        val unwrapped = wrapperBinding.unwrap(value)
        s"${typeId.name}(${wrappedShow.show(unwrapped)})"
      }
    }
  }
}
```

The derivation process for wrapper types involves unwrapping the value to access the underlying type. Wrappers are non-recursive, so we use `.map` composition: `D.instance(wrapped.metadata).map { wrappedShow => ... }` returns a `Lazy[Show[A]]` that, when forced, produces a `Show[A]` with `wrappedShow` already resolved. At runtime, we use the `unwrap` function from the binding to retrieve the underlying `B` value and delegate to `wrappedShow.show` — no explicit `.force` is needed.

### Example Usages

To see how this derivation works in practice, we can define some simple data types and then derive `Show` instances for them using the `DeriveShow` object we implemented.

1. Example 1: Simple `Person` Record with Two Primitive Fields:

```scala
case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
  implicit val show: Show[Person]     = schema.derive(DeriveShow)
}
```

Now we can use the derived `Show[Person]` instance to convert `Person` values to strings:

```scala
Person.show.show(Person("Alice", 30))
// res11: String = "Person(name = \"Alice\", age = 30)"
```

2. Simple Shape Variant (Circle, Rectangle)

```scala
sealed trait Shape
case class Circle(radius: Double)                   extends Shape
case class Rectangle(width: Double, height: Double) extends Shape

object Shape {
  implicit val schema: Schema[Shape] = Schema.derived[Shape]
  implicit val show: Show[Shape]     = schema.derive(DeriveShow)
}
```

To show a `Shape` value, we can do the following:

```scala
val shape1: Shape = Circle(5.0)
// shape1: Shape = Circle(5.0)
Shape.show.show(shape1)
// res12: String = "Circle(radius = 5.0)"

val shape2: Shape = Rectangle(4.0, 6.0)
// shape2: Shape = Rectangle(width = 4.0, height = 6.0)
Shape.show.show(shape2)
// res13: String = "Rectangle(width = 4.0, height = 6.0)"
```

3. Recursive Tree and Expr

```scala
case class Tree(value: Int, children: List[Tree])
object Tree {
  implicit val schema: Schema[Tree] = Schema.derived[Tree]
  implicit val show: Show[Tree]     = schema.derive(DeriveShow)
}
```

The `Tree` is a record with a recursive field `children` of type `List[Tree]`. Let's see how the derived `Show[Tree]` instance handles this recursive structure:

```scala
val tree = Tree(1, List(Tree(2, List(Tree(4, Nil))), Tree(3, Nil)))
// tree: Tree = Tree(
//   value = 1,
//   children = List(
//     Tree(value = 2, children = List(Tree(value = 4, children = List()))),
//     Tree(value = 3, children = List())
//   )
// )
Tree.show.show(tree)
// res14: String = "Tree(value = 1, children = [Tree(value = 2, children = [Tree(value = 4, children = [])]), Tree(value = 3, children = [])])"
```

4. Example 4: Recursive Sealed Trait (Expr)

```scala
sealed trait Expr
case class Num(n: Int)           extends Expr
case class Add(a: Expr, b: Expr) extends Expr

object Expr {
  implicit val schema: Schema[Expr] = Schema.derived[Expr]
  implicit val show: Show[Expr]     = schema.derive(DeriveShow)
}
```

Similar to `Tree`, `Expr` is a recursive variant type. The derived `Show[Expr]` instance can handle this recursive structure as well:

```scala
val expr: Expr = Add(Num(1), Add(Num(2), Num(3)))
// expr: Expr = Add(a = Num(1), b = Add(a = Num(2), b = Num(3)))
Expr.show.show(expr)
// res15: String = "Add(a = Num(n = 1), b = Add(a = Num(n = 2), b = Num(n = 3)))"
```

5. Example 5: DynamicValue Example

```scala
implicit val dynamicShow: Show[DynamicValue] = Schema.dynamic.derive(DeriveShow)
```

Let's define a `DynamicValue` that represents a record with some primitive fields and a sequence field, then show it using the derived `Show[DynamicValue]` instance:

```scala
val manualRecord = DynamicValue.Record(
  Chunk(
    "id"    -> DynamicValue.Primitive(PrimitiveValue.Int(42)),
    "title" -> DynamicValue.Primitive(PrimitiveValue.String("Hello World")),
    "tags"  -> DynamicValue.Sequence(
      Chunk(
        DynamicValue.Primitive(PrimitiveValue.String("scala")),
        DynamicValue.Primitive(PrimitiveValue.String("zio"))
      )
    )
  )
)
// manualRecord: Record = Record(
//   IndexedSeq(
//     ("id", Primitive(Int(42))),
//     ("title", Primitive(String("Hello World"))),
//     (
//       "tags",
//       Sequence(IndexedSeq(Primitive(String("scala")), Primitive(String("zio"))))
//     )
//   )
// )

dynamicShow.show(manualRecord)
// res16: String = "Record(id = 42, title = \"Hello World\", tags = [\"scala\", \"zio\"])"
```

6. Example 6: Simple Email Wrapper Type

```scala
case class Email(value: String)
object Email {
  implicit val schema: Schema[Email] = Schema[String].transform(
    Email(_),
    _.value
  )
  implicit val show: Show[Email] = schema.derive(DeriveShow)
}
```

The `Email` type is a simple wrapper around `String`. Let's see how it shows an `Email` value:

```scala
val email = Email("alice@example.com")
// email: Email = Email("alice@example.com")
println(s"Email: ${Email.show.show(email)}")
// Email: Email("alice@example.com")
```

## Example 2: Deriving a `Gen` Type Class Instance

Let's say we want to derive a `Gen` type class instance for any type `A`:

```scala

trait Gen[A] {
  def generate(random: Random): A
}
```

Unlike `Show`, which is a type class for converting values of type `A` to something else (a `String`)—so you can think of it as a function of type `A => Output (String)`—the `Gen` type class is for generating values of type `A`. You can think of it as a function of type `Input (Random) => A`.

To implement the `Show` type class, we need to know what components type `A` is made up of, so we can convert each component to a `String` and combine them to form the final `String` representation of `A`. To do this, we need to be able to deconstruct a value of type `A` into its components. On the other hand, to implement the `Gen` type class, we need to know how to generate each component of type `A` using a `Random` input, and then combine those generated components to form a complete value of type `A`. This means that for `Gen`, we need to be able to construct a value of type `A` from its components, rather than deconstructing it. Therefore, in the derivation methods for `Gen`, we will use the `constructor` from the `Binding` to create values of type `A` from generated components.

Here is a simple pedagogical implementation of a `GenDeriver` that can derive `Gen` instances for various types:

```scala

object DeriveGen extends Deriver[Gen] {

  override def derivePrimitive[A](
    primitiveType: PrimitiveType[A],
    typeId: TypeId[A],
    binding: Binding.Primitive[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  ): Lazy[Gen[A]] =
    Lazy {
      new Gen[A] {
        def generate(random: Random): A = primitiveType match {
          case _: PrimitiveType.String  => random.alphanumeric.take(random.nextInt(10) + 1).mkString.asInstanceOf[A]
          case _: PrimitiveType.Char    => random.alphanumeric.head.asInstanceOf[A]
          case _: PrimitiveType.Boolean => random.nextBoolean().asInstanceOf[A]
          case _: PrimitiveType.Int     => random.nextInt().asInstanceOf[A]
          case _: PrimitiveType.Long    => random.nextLong().asInstanceOf[A]
          case _: PrimitiveType.Double  => random.nextDouble().asInstanceOf[A]
          case PrimitiveType.Unit       => ().asInstanceOf[A]
          // For brevity, other primitives default to their zero/empty value
          // In a real implementation, you'd want to handle all primitives and possibly use modifiers for ranges, etc.
          case _ =>
            defaultValue.getOrElse {
              throw new IllegalArgumentException(
                s"Gen derivation not implemented for primitive type $primitiveType " +
                        s"(typeId = $typeId) and no default value provided."
              )
            }
        }
      }
    }

  /**
   * Strategy:
   *   1. Get Gen type class instances for each field
   *   2. Generate random values for each field
   *   3. Use the constructor to build the record
   */
  override def deriveRecord[F[_, _], A](
    fields: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Record[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[A]] = {
    // Pre-compute structural setup outside Lazy — none of this touches Deferred nodes
    // Build Reflect.Record to access registers and constructor
    val recordFields  = fields.asInstanceOf[IndexedSeq[Term[Binding, A, ?]]]
    val recordBinding = binding.asInstanceOf[Binding.Record[A]]
    val recordReflect = new Reflect.Record[Binding, A](recordFields, typeId, recordBinding, doc, modifiers)
    Lazy {
      new Gen[A] {
        // Defer child-instance resolution to first generate() call via lazy val.
        // For recursive types (e.g. case class Tree(children: List[Tree])), accessing
        // field.value.metadata during derivation would re-enter a Deferred node that is
        // still initialising, causing an infinite loop. The lazy val here ensures access
        // only happens after the framework has finished deriving all instances.
        private lazy val resolvedGens: IndexedSeq[Gen[Any]] =
          fields.map(field => D.instance(field.value.metadata).asInstanceOf[Lazy[Gen[Any]]].force)
        def generate(random: Random): A = {
          // Create registers to hold field values
          val registers = Registers(recordReflect.usedRegisters)
          // Generate each field and store in registers
          fields.indices.foreach { i =>
            val value = resolvedGens(i).generate(random)
            recordReflect.registers(i).set(registers, RegisterOffset.Zero, value)
          }
          // Construct the record from registers
          recordBinding.constructor.construct(registers, RegisterOffset.Zero)
        }
      }
    }
  }

  /**
   * Strategy:
   *   1. Get Gen type class instances for all cases
   *   2. Randomly pick a case
   *   3. Generate a value for that case
   */
  override def deriveVariant[F[_, _], A](
    cases: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Variant[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[A]] = {
    // Collect the Lazy[Gen] references for each case outside the Lazy block.
    // Capturing Lazy refs is safe here; we only .force them later inside the lazy val.
    val caseGenLazies: IndexedSeq[Lazy[Gen[A]]] = cases.map { c =>
      D.instance(c.value.metadata).asInstanceOf[Lazy[Gen[A]]]
    }
    Lazy {
      new Gen[A] {
        // Force child instances lazily — same recursive-safety rationale as deriveRecord
        private lazy val resolvedGens: IndexedSeq[Gen[A]] = caseGenLazies.map(_.force)
        def generate(random: Random): A = {
          // Pick a random case and generate its value
          val caseIndex = random.nextInt(cases.length)
          resolvedGens(caseIndex).generate(random)
        }
      }
    }
  }

  /**
   * Strategy:
   *   1. Get Gen type class instances for the element type
   *   2. Generate 0-5 elements
   *   3. Build the collection using the constructor
   */
  override def deriveSequence[F[_, _], C[_], A](
    element: Reflect[F, A],
    typeId: TypeId[C[A]],
    binding: Binding.Seq[C, A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[C[A]],
    examples: Seq[C[A]]
  )(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[C[A]]] = {
    // Cast binding to Binding.Seq to access the constructor
    val constructor  = binding.asInstanceOf[Binding.Seq[C, A]].constructor
    val elemClassTag = element.typeId.classTag.asInstanceOf[scala.reflect.ClassTag[A]]
    // Sequences are structurally non-recursive, so we can use monadic .map composition.
    // instance(...).map { elementGen => ... } returns a Lazy that, when forced, builds
    // a Gen[C[A]] with elementGen already resolved — no .force needed at generate()-time.
    D.instance(element.metadata).map { elementGen =>
      new Gen[C[A]] {
        def generate(random: Random): C[A] = {
          val length = random.nextInt(6) // 0 to 5 elements
          implicit val ct: scala.reflect.ClassTag[A] = elemClassTag

          if (length == 0) {
            constructor.empty[A]
          } else {
            // Build the collection by generating each element and adding it to the builder
            val builder = constructor.newBuilder[A](length)
            (0 until length).foreach { _ =>
              constructor.add(builder, elementGen.generate(random))
            }
            constructor.result(builder)
          }
        }
      }
    }
  }

  /**
   * Strategy:
   *   1. Get Gen type class instances for key and value types
   *   2. Generate 0-5 key-value pairs
   *   3. Build the map using the constructor
   */
  override def deriveMap[F[_, _], M[_, _], K, V](
    key: Reflect[F, K],
    value: Reflect[F, V],
    typeId: TypeId[M[K, V]],
    binding: Binding.Map[M, K, V],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[M[K, V]],
    examples: Seq[M[K, V]]
  )(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[M[K, V]]] = {
    // Cast binding to Binding.Map to access the constructor
    val constructor = binding.asInstanceOf[Binding.Map[M, K, V]].constructor
    // Maps are non-recursive: use .zip to pair the two child Lazy instances, then .map
    // to build the Gen[M[K,V]] with both keyGen and valueGen already resolved.
    D.instance(key.metadata).zip(D.instance(value.metadata)).map { case (keyGen, valueGen) =>
      new Gen[M[K, V]] {
        def generate(random: Random): M[K, V] = {
          val size = random.nextInt(6) // 0 to 5 entries

          if (size == 0) {
            constructor.emptyObject[K, V]
          } else {
            // Build the map by generating each key-value pair and adding it to the builder
            val builder = constructor.newObjectBuilder[K, V](size)
            (0 until size).foreach { _ =>
              constructor.addObject(builder, keyGen.generate(random), valueGen.generate(random))
            }
            constructor.resultObject(builder)
          }
        }
      }
    }
  }

  /**
   * Since DynamicValue can represent any schema type, we generate random
   * dynamic values by randomly choosing a variant and generating appropriate
   * content.
   */
  override def deriveDynamic[F[_, _]](
    binding: Binding.Dynamic,
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[DynamicValue],
    examples: Seq[DynamicValue]
  )(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[DynamicValue]] = Lazy {
    new Gen[DynamicValue] {
      // Helper to generate a random primitive value
      private def randomPrimitive(random: Random): DynamicValue.Primitive = {
        val primitiveType = random.nextInt(5)
        primitiveType match {
          case 0 => DynamicValue.Primitive(PrimitiveValue.Int(random.nextInt()))
          case 1 => DynamicValue.Primitive(PrimitiveValue.String(random.alphanumeric.take(10).mkString))
          case 2 => DynamicValue.Primitive(PrimitiveValue.Boolean(random.nextBoolean()))
          case 3 => DynamicValue.Primitive(PrimitiveValue.Double(random.nextDouble()))
          case 4 => DynamicValue.Primitive(PrimitiveValue.Long(random.nextLong()))
        }
      }

      def generate(random: Random): DynamicValue = {
        // Randomly choose what kind of DynamicValue to generate
        // Weight towards primitives and simpler structures to avoid deep nesting
        val valueType = random.nextInt(10)
        valueType match {
          case 0 | 1 | 2 | 3 | 4 =>
            // 50% chance: generate a primitive
            randomPrimitive(random)

          case 5 | 6 =>
            // 20% chance: generate a record with 1-3 fields
            val numFields = random.nextInt(3) + 1
            val fields    = (0 until numFields).map { i =>
              val fieldName  = s"field$i"
              val fieldValue = randomPrimitive(random)
              (fieldName, fieldValue: DynamicValue)
            }
            DynamicValue.Record(Chunk.from(fields))

          case 7 | 8 =>
            // 20% chance: generate a sequence of 0-3 primitives
            val numElements = random.nextInt(4)
            val elements    = (0 until numElements).map(_ => randomPrimitive(random): DynamicValue)
            DynamicValue.Sequence(Chunk.from(elements))

          case 9 =>
            // 10% chance: generate null
            DynamicValue.Null
        }
      }
    }
  }

  override def deriveWrapper[F[_, _], A, B](
    wrapped: Reflect[F, B],
    typeId: TypeId[A],
    binding: Binding.Wrapper[A, B],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[A]] = {
    // Cast binding to Binding.Wrapper to access the wrap function
    val wrapperBinding = binding.asInstanceOf[Binding.Wrapper[A, B]]
    // Wrappers are non-recursive: use .map so wrappedGen is already resolved
    // when generate() is called — no .force needed.
    D.instance(wrapped.metadata).map { wrappedGen =>
      new Gen[A] {
        def generate(random: Random): A =
          // Generate a value of the underlying type B and wrap it into A
          wrapperBinding.wrap(wrappedGen.generate(random))
      }
    }
  }
}
```

### Primitive Derivation

The `derivePrimitive` method is responsible for deriving a `Gen` instance for primitive types. It matches on the specific primitive type and generates random values accordingly. For example, for `String`, it generates a random alphanumeric string of random length; for `Int`, it generates a random integer; and so on. The generated value is then cast to the appropriate type `A` and returned:

```scala
def derivePrimitive[A](
  primitiveType: PrimitiveType[A],
  typeId: TypeId[A],
  binding: Binding.Primitive[A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
): Lazy[Gen[A]] =
  Lazy {
    new Gen[A] {
      def generate(random: Random): A = primitiveType match {
        case _: PrimitiveType.String  => random.alphanumeric.take(random.nextInt(10) + 1).mkString.asInstanceOf[A]
        case _: PrimitiveType.Char    => random.alphanumeric.head.asInstanceOf[A]
        case _: PrimitiveType.Boolean => random.nextBoolean().asInstanceOf[A]
        case _: PrimitiveType.Int     => random.nextInt(100).asInstanceOf[A]
        case _: PrimitiveType.Long    => random.nextLong().asInstanceOf[A]
        case _: PrimitiveType.Double  => random.nextDouble().asInstanceOf[A]
        case PrimitiveType.Unit       => ().asInstanceOf[A]
        // For brevity, other primitives default to their zero/empty value
        // In a real implementation, you would want to handle all primitives and possibly use modifiers for ranges, etc.
        case _ => defaultValue.getOrElse(null.asInstanceOf[A])
      }
    }
  }
```

To handle all primitive types, you would want to implement cases for each primitive type defined in your schema system. In a real implementation, you might also want to consider using modifiers to allow users to specify constraints on the generated values (e.g., string length, numeric ranges, etc.).

### Record Derivation

The `deriveRecord` method is responsible for deriving a `Gen` instance for record types, such as case classes and tuples. The strategy for deriving a record type involves three main steps:

```scala
def deriveRecord[F[_, _], A](
  fields: IndexedSeq[Term[F, A, ?]],
  typeId: TypeId[A],
  binding: Binding.Record[A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
)(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[A]] = {
  // Pre-compute structural setup outside Lazy — none of this touches Deferred nodes
  // Build Reflect.Record to access registers and constructor
  val recordFields  = fields.asInstanceOf[IndexedSeq[Term[Binding, A, ?]]]
  val recordBinding = binding.asInstanceOf[Binding.Record[A]]
  val recordReflect = new Reflect.Record[Binding, A](recordFields, typeId, recordBinding, doc, modifiers)
  Lazy {
    new Gen[A] {
      // Defer child-instance resolution to first generate() call via lazy val.
      // For recursive types (e.g. case class Tree(children: List[Tree])), accessing
      // field.value.metadata during derivation would re-enter a Deferred node that is
      // still initialising, causing an infinite loop. The lazy val here ensures access
      // only happens after the framework has finished deriving all instances.
      private lazy val resolvedGens: IndexedSeq[Gen[Any]] =
        fields.map(field => D.instance(field.value.metadata).asInstanceOf[Lazy[Gen[Any]]].force)
      def generate(random: Random): A = {
        // Create registers to hold field values
        val registers = Registers(recordReflect.usedRegisters)
        // Generate each field and store in registers
        fields.indices.foreach { i =>
          val value = resolvedGens(i).generate(random)
          recordReflect.registers(i).set(registers, RegisterOffset.Zero, value)
        }
        // Construct the record from registers
        recordBinding.constructor.construct(registers, RegisterOffset.Zero)
      }
    }
  }
}
```

As shown above, the implementation of the `deriveRecord` method for `Gen` follows the same structure as its `Show` counterpart. The structural setup (field casts, `Reflect.Record`) is done outside `Lazy` to keep the thunk lightweight, and `resolvedGens` is a `private lazy val` that defers child-instance resolution until the first `generate()` call for recursive-type safety. The primary difference from `Show` is the data flow: instead of deconstructing an existing record to read its fields, we generate random values for each field via `resolvedGens(i).generate(random)`, store them in registers using `Register#set`, and then invoke the `constructor` from the `Binding` to create an instance of type `A`.

### Variant Derivation

The `deriveVariant` method is responsible for deriving a `Gen` instance for variant types, such as sealed traits with case classes:

```scala
def deriveVariant[F[_, _], A](
  cases: IndexedSeq[Term[F, A, ?]],
  typeId: TypeId[A],
  binding: Binding.Variant[A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
)(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[A]] = {
  // Collect the Lazy[Gen] references for each case outside the Lazy block.
  // Capturing Lazy refs is safe here; we only .force them later inside the lazy val.
  val caseGenLazies: IndexedSeq[Lazy[Gen[A]]] = cases.map { c =>
    D.instance(c.value.metadata).asInstanceOf[Lazy[Gen[A]]]
  }
  Lazy {
    new Gen[A] {
      // Force child instances lazily — same recursive-safety rationale as deriveRecord
      private lazy val resolvedGens: IndexedSeq[Gen[A]] = caseGenLazies.map(_.force)
      def generate(random: Random): A = {
        // Pick a random case and generate its value
        val caseIndex = random.nextInt(cases.length)
        resolvedGens(caseIndex).generate(random)
      }
    }
  }
}
```

The derivation process for `Gen` variants applies the same deferral pattern as `deriveRecord`. The `caseGenLazies` are collected outside the `Lazy` block — capturing `Lazy` references is safe — and `resolvedGens` forces them lazily on the first `generate()` call. At runtime, we randomly select one of the resolved `Gen` instances and call `generate` on it. This is simpler than the record case because there are no registers or constructors involved — each case's `Gen` already knows how to produce a complete value of its type.

### Sequence Derivation

The `deriveSequence` method is responsible for deriving a `Gen` instance for sequence types, such as `List[A]`:

```scala
def deriveSequence[F[_, _], C[_], A](
  element: Reflect[F, A],
  typeId: TypeId[C[A]],
  binding: Binding.Seq[C, A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[C[A]],
  examples: Seq[C[A]]
)(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[C[A]]] = {
  // Cast binding to Binding.Seq to access the constructor
  val constructor  = binding.asInstanceOf[Binding.Seq[C, A]].constructor
  val elemClassTag = element.typeId.classTag.asInstanceOf[scala.reflect.ClassTag[A]]
  // Sequences are structurally non-recursive, so we can use monadic .map composition.
  // instance(...).map { elementGen => ... } returns a Lazy that, when forced, builds
  // a Gen[C[A]] with elementGen already resolved — no .force needed at generate()-time.
  D.instance(element.metadata).map { elementGen =>
    new Gen[C[A]] {
      def generate(random: Random): C[A] = {
        val length = random.nextInt(6) // 0 to 5 elements
        implicit val ct: scala.reflect.ClassTag[A] = elemClassTag

        if (length == 0) {
          constructor.empty[A]
        } else {
          // Build the collection by generating each element and adding it to the builder
          val builder = constructor.newBuilder[A](length)
          (0 until length).foreach { _ =>
            constructor.add(builder, elementGen.generate(random))
          }
          constructor.result(builder)
        }
      }
    }
  }
}
```

A sequence is an object that contains multiple elements of the same type. Because sequences are non-recursive, we use `.map` composition: `D.instance(element.metadata).map { elementGen => ... }` returns a `Lazy[Gen[C[A]]]` that, when forced, builds a `Gen[C[A]]` with `elementGen` already resolved — no explicit `.force` is needed inside `generate()`. At runtime, we generate a random length (0–5). For an empty length, we return `constructor.empty`. Otherwise, we create a new builder using `constructor.newBuilder`, generate random element values with `elementGen.generate(random)`, add them with `constructor.add`, and finalise with `constructor.result`.

### Map Derivation

The `deriveMap` method is responsible for deriving a `Gen` instance for map types, such as `Map[K, V]`:

```scala
def deriveMap[F[_, _], M[_, _], K, V](
  key: Reflect[F, K],
  value: Reflect[F, V],
  typeId: TypeId[M[K, V]],
  binding: Binding.Map[M, K, V],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[M[K, V]],
  examples: Seq[M[K, V]]
)(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[M[K, V]]] = {
  // Cast binding to Binding.Map to access the constructor
  val constructor = binding.asInstanceOf[Binding.Map[M, K, V]].constructor
  // Maps are non-recursive: use .zip to pair the two child Lazy instances, then .map
  // to build the Gen[M[K,V]] with both keyGen and valueGen already resolved.
  D.instance(key.metadata).zip(D.instance(value.metadata)).map { case (keyGen, valueGen) =>
    new Gen[M[K, V]] {
      def generate(random: Random): M[K, V] = {
        val size = random.nextInt(6) // 0 to 5 entries

        if (size == 0) {
          constructor.emptyObject[K, V]
        } else {
          // Build the map by generating each key-value pair and adding it to the builder
          val builder = constructor.newObjectBuilder[K, V](size)
          (0 until size).foreach { _ =>
            constructor.addObject(builder, keyGen.generate(random), valueGen.generate(random))
          }
          constructor.resultObject(builder)
        }
      }
    }
  }
}
```

The derivation process for maps is similar to sequences but requires composing two child instances. We use `.zip` to combine the `Lazy[Gen[K]]` and `Lazy[Gen[V]]` into a single `Lazy`, then `.map` to build the `Gen[M[K,V]]` with both `keyGen` and `valueGen` already resolved. At runtime, a random size (0–5) determines whether we return an empty map or build one entry by entry using `constructor.addObject(builder, keyGen.generate(random), valueGen.generate(random))`.

### Dynamic Derivation

The `deriveDynamic` method is responsible for deriving a `Gen` instance for dynamic types, such as `DynamicValue`. Since `DynamicValue` can represent any schema type, we generate random dynamic values by choosing a variant at random and generating the appropriate content for that variant. The implementation involves pattern matching on the `DynamicValue` type and generating content accordingly:

```scala
def deriveDynamic[F[_, _]](
  binding: Binding.Dynamic,
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[DynamicValue],
  examples: Seq[DynamicValue]
)(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[DynamicValue]] = Lazy {
  new Gen[DynamicValue] {
    // Helper to generate a random primitive value
    private def randomPrimitive(random: Random): DynamicValue.Primitive = {
      val primitiveType = random.nextInt(5)
      primitiveType match {
        case 0 => DynamicValue.Primitive(PrimitiveValue.Int(random.nextInt()))
        case 1 => DynamicValue.Primitive(PrimitiveValue.String(random.alphanumeric.take(10).mkString))
        case 2 => DynamicValue.Primitive(PrimitiveValue.Boolean(random.nextBoolean()))
        case 3 => DynamicValue.Primitive(PrimitiveValue.Double(random.nextDouble()))
        case 4 => DynamicValue.Primitive(PrimitiveValue.Long(random.nextLong()))
      }
    }

    def generate(random: Random): DynamicValue = {
      // Randomly choose what kind of DynamicValue to generate
      // Weight towards primitives and simpler structures to avoid deep nesting
      val valueType = random.nextInt(10)
      valueType match {
        case 0 | 1 | 2 | 3 | 4 =>
          // 50% chance: generate a primitive
          randomPrimitive(random)

        case 5 | 6 =>
          // 20% chance: generate a record with 1-3 fields
          val numFields = random.nextInt(3) + 1
          val fields    = (0 until numFields).map { i =>
            val fieldName  = s"field$i"
            val fieldValue = randomPrimitive(random)
            (fieldName, fieldValue: DynamicValue)
          }
          DynamicValue.Record(Chunk.from(fields))

        case 7 | 8 =>
          // 20% chance: generate a sequence of 0-3 primitives
          val numElements = random.nextInt(4)
          val elements    = (0 until numElements).map(_ => randomPrimitive(random): DynamicValue)
          DynamicValue.Sequence(Chunk.from(elements))

        case 9 =>
          // 10% chance: generate null
          DynamicValue.Null
      }
    }
  }
}
```

Please note that the random generation logic in this example is basic and is intended for illustrative purposes only.

### Wrapper Derivation

The `deriveWrapper` method is responsible for deriving a `Gen` instance for wrapper types, such as value classes or opaque types:

```scala
def deriveWrapper[F[_, _], A, B](
  wrapped: Reflect[F, B],
  typeId: TypeId[A],
  binding: Binding.Wrapper[A, B],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
)(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[A]] = {
  // Cast binding to Binding.Wrapper to access the wrap function
  val wrapperBinding = binding.asInstanceOf[Binding.Wrapper[A, B]]
  // Wrappers are non-recursive: use .map so wrappedGen is already resolved
  // when generate() is called — no .force needed.
  D.instance(wrapped.metadata).map { wrappedGen =>
    new Gen[A] {
      def generate(random: Random): A =
        // Generate a value of the underlying type B and wrap it into A
        wrapperBinding.wrap(wrappedGen.generate(random))
    }
  }
}
```

Wrappers are non-recursive, so we use `.map` composition: `D.instance(wrapped.metadata).map { wrappedGen => ... }` returns a `Lazy[Gen[A]]` that, when forced, builds a `Gen[A]` with `wrappedGen` already resolved. Within `generate()`, we generate a random value of the underlying type `B` and wrap it into `A` using the `wrap` function from the binding — no explicit `.force` is needed.

### Example Usages

To see how this derivation works in practice, we can define some simple data types and then derive `Gen` instances for them using the `DeriveGen` object we implemented.

1. Example 1: Simple `Person` Record with Two Primitive Fields:

```scala
case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
  implicit val gen: Gen[Person]       = schema.derive(DeriveGen)
}
```

Now we can use the derived `Gen[Person]` instance to generate random `Person` values:

```scala
val random = new Random(42) // Seeded for reproducible output
// random: Random = scala.util.Random@21e8c38d

Person.gen.generate(random)
// res18: Person = Person(name = "p", age = -1360544799)
Person.gen.generate(random)
// res19: Person = Person(name = "C7DgX", age = 392236186)
Person.gen.generate(random)
// res20: Person = Person(name = "AM6", age = 1184328952)
```

2. Simple Shape Variant (Circle, Rectangle)

```scala
sealed trait Shape
case class Circle(radius: Double)                   extends Shape
case class Rectangle(width: Double, height: Double) extends Shape

object Shape {
  implicit val schema: Schema[Shape] = Schema.derived[Shape]
  implicit val gen: Gen[Shape]       = schema.derive(DeriveGen)
}
```

To generate random `Shape` values, we can do the following:

```scala
Shape.gen.generate(random)
// res21: Shape = Rectangle(
//   width = 0.46365357580915334,
//   height = 0.7829017787900358
// )
Shape.gen.generate(random)
// res22: Shape = Rectangle(
//   width = 0.15195824856297624,
//   height = 0.43979982659080874
// )
Shape.gen.generate(random)
// res23: Shape = Rectangle(
//   width = 0.38656687435934867,
//   height = 0.17737847790937833
// )
Shape.gen.generate(random)
// res24: Shape = Rectangle(
//   width = 0.338307935145014,
//   height = 0.2506613258416336
// )
```

3. Team with Sequence of Members (List)

```scala
case class Team(members: List[String])

object Team {
  implicit val schema: Schema[Team] = Schema.derived[Team]
  implicit val gen: Gen[Team]       = schema.derive(DeriveGen)
}
```

Let's generate some random `Team` values:

```scala
Team.gen.generate(random)
// res25: Team = Team(List("zZY", "TZlZMZdVjx", "G", "iqf1Pt9", "S1q6qHNj0R"))
Team.gen.generate(random)
// res26: Team = Team(List("b94sbz0WFC"))
Team.gen.generate(random)
// res27: Team = Team(List("nwyT"))
```

4. Example 4: Recursive Tree

```scala
case class Tree(value: Int, children: List[Tree])

object Tree {
  implicit val schema: Schema[Tree] = Schema.derived[Tree]
  implicit val gen: Gen[Tree]       = schema.derive(DeriveGen)
}
```

The `Tree` is a record with a recursive field `children` of type `List[Tree]`. Let's see how the derived `Gen[Tree]` instance handles this recursive structure:

```scala
Tree.gen.generate(random)
// res28: Tree = Tree(value = 1205047495, children = List())
```

5. Example 5: DynamicValue Example

```scala
implicit val dynamicGen: Gen[DynamicValue] = Schema.dynamic.derive(DeriveGen)
```

Let's generate some random `DynamicValue` instances:

```scala
dynamicGen.generate(random)
// res29: DynamicValue = Primitive(Int(769973518))
dynamicGen.generate(random)
// res30: DynamicValue = Primitive(Long(8878934151639676041L))
dynamicGen.generate(random)
// res31: DynamicValue = Sequence(IndexedSeq(Primitive(Int(-1576812231))))
```

6. Example 6: Simple Email Wrapper Type

```scala
case class Email(value: String)

object Email {
  implicit val schema: Schema[Email] = Schema[String].transform(
    Email(_),
    _.value
  )
  implicit val gen: Gen[Email] = schema.derive(DeriveGen)
}
```

The `Email` type is a simple wrapper around `String`. Let's see how it generates random `Email` values:

```scala
Email.gen.generate(random)
// res32: Email = Email("zlLKVaEitt")
Email.gen.generate(random)
// res33: Email = Email("Sa")
```

## Custom Type-class Instances

While automatic derivation generates type class instances for all substructures of a data type, there are times when you need to override the derived instance for a specific substructure. For example, you might want to use a custom `Show` instance for a particular field, provide a hand-written codec for a specific type that the deriver doesn't handle well, or inject a special implementation for testing purposes.

The `DerivationBuilder` provides an `instance` method that allows you to override the automatically derived type class instance for any part of the schema tree. You access the `DerivationBuilder` by calling `Schema#deriving(deriver)` instead of `Schema#derive(deriver)`:

```scala
val schema: Schema[A] = ...
val deriver: Deriver[TC] = ...

// Using derive: fully automatic, no customization
val tc: TC[A] = schema.derive(deriver)

// Using deriving: returns a DerivationBuilder for customization
val tc: TC[A] = schema.deriving(deriver)
  .instance(...)   // override specific instances
  .modifier(...)   // override specific modifiers
  .derive           // finalize the derivation
```

The `DerivationBuilder` offers three overloaded `instance` methods for providing custom type class instances:

```scala
final case class DerivationBuilder[TC[_], A](...) {
  def instance[B](optic: Optic[A, B], instance: => TC[B]): DerivationBuilder[TC, A]
  def instance[B](typeId: TypeId[B], instance: => TC[B]): DerivationBuilder[TC, A]
  def instance[P, B](typeId: TypeId[P], termName: String, instance: => TC[B]): DerivationBuilder[TC, A]
}
```

### Overriding by Optic

The first overload takes an `Optic[A, B]` that precisely targets a specific location within the schema tree. This is useful when you want to override the instance for a particular field or case without affecting other occurrences of the same type:

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived[Person]

  val name: Lens[Person, String] = $(_.name)
  val age: Lens[Person, Int]     = $(_.age)
}
```

Now we can override the `Show[String]` instance specifically for the `name` field of `Person`:

```scala
val customNameShow: Show[String] = new Show[String] {
  def show(value: String): String = value.toUpperCase
}

val personShow: Show[Person] = Person.schema
  .deriving(DeriveShow)
  .instance(Person.name, customNameShow)
  .derive
```

When we show a `Person`, the `name` field will use the custom `Show[String]` instance (showing it in uppercase), while the `age` field will use the automatically derived `Show[Int]` instance:

```scala
personShow.show(Person("Alice", 30))
// res34: String = "Person(name = ALICE, age = 30)"
```

You can also target deeper nested fields using composed optics. For example, if you have a `Company` that contains a `Person`, you can target the `name` field inside the nested `Person`:

```scala
case class Company(ceo: Person, industry: String)

object Company extends CompanionOptics[Company] {
  implicit val schema: Schema[Company] = Schema.derived[Company]

  val ceo: Lens[Company, Person]       = $(_.ceo)
  val ceoName: Lens[Company, String]   = $(_.ceo.name)
  val industry: Lens[Company, String]  = $(_.industry)
}
```

```scala
val companyShow: Show[Company] = Company.schema
  .deriving(DeriveShow)
  .instance(Company.ceoName, customNameShow)
  .derive
```

In this case, the custom `Show[String]` instance only applies to the CEO's name. The `industry` field, which is also a `String`, will use the default derived `Show[String]` instance:

```scala
companyShow.show(Company(Person("Alice", 30), "tech"))
// res35: String = "Company(ceo = Person(name = ALICE, age = 30), industry = \"tech\")"
```

### Overriding by TypeId

The second overload takes a `TypeId[B]` and applies the custom instance to **all occurrences** of type `B` anywhere in the schema tree. This is useful when you want to override the instance for a type globally, without having to specify each location:

```scala
val customIntShow: Show[Int] = new Show[Int] {
  def show(value: Int): String = s"#$value"
}

val personShow: Show[Person] = Person.schema
  .deriving(DeriveShow)
  .instance(TypeId.int, customIntShow)
  .derive
```

All `Int` fields in the `Person` schema (in this case, just `age`) will use the custom `Show[Int]` instance:

```scala
personShow.show(Person("Alice", 30))
// res36: String = "Person(name = \"Alice\", age = #30)"
```

### Overriding by TypeId and Term Name

The third overload takes a `TypeId[P]` identifying the **parent** record or variant and a `termName` string identifying a field or case within it. This provides medium precision between optic-based (exact path) and type-based (all occurrences) overrides, and is useful when you want to override a specific field across all locations where the parent type appears without having to enumerate each path with an optic:

```scala
val customAgeShow: Show[Int] = new Show[Int] {
  def show(value: Int): String = s"age=$value"
}

val personShow: Show[Person] = Person.schema
  .deriving(DeriveShow)
  .instance(Person.schema.reflect.typeId, "age", customAgeShow)
  .derive
```

The `typeId` refers to the parent record/variant type (here `Person`), not the field type. If no term with the given name exists in the parent type, the override is silently ignored.

```scala
personShow.show(Person("Alice", 30))
// res37: String = "Person(name = \"Alice\", age = age=30)"
```

### Resolution Order

When the derivation engine encounters a schema node, it resolves the type class instance using the following priority order:

1. **Optic-based override** (most precise): If an instance override was registered using an optic that matches the current path in the schema tree, that instance is used.
2. **TypeId + term-name override** (medium precision): If no optic-based match is found, it checks for an override registered by parent type ID and term name.
3. **TypeId-based override** (more general): If no term-name match is found, it checks for an instance override registered by type ID.
4. **Automatic derivation** (default): If no override is found, the deriver's method (e.g., `derivePrimitive`, `deriveRecord`) is called to automatically derive the instance.

This means you can set a global override by type and then selectively refine specific fields using optics or term names:

```scala
val companyShow: Show[Company] = Company.schema
  .deriving(DeriveShow)
  .instance(TypeId.string, new Show[String] {
    def show(value: String): String = s"'$value'"
  })
  .instance(Company.ceoName, new Show[String] {
    def show(value: String): String = value.toUpperCase
  })
  .derive
```

In this example, all `String` fields use single quotes, except for the CEO's name which is shown in uppercase:

```scala
companyShow.show(Company(Person("Alice", 30), "tech"))
// res38: String = "Company(ceo = Person(name = ALICE, age = 30), industry = 'tech')"
```

### Chaining Multiple Overrides

The `instance` method returns a new `DerivationBuilder`, so you can chain multiple overrides fluently:

```scala
val personShow: Show[Person] = Person.schema
  .deriving(DeriveShow)
  .instance(Person.name, new Show[String] {
    def show(value: String): String = s"<<$value>>"
  })
  .instance(Person.age, new Show[Int] {
    def show(value: Int): String = s"age=$value"
  })
  .derive
```

```scala
personShow.show(Person("Alice", 30))
// res39: String = "Person(name = <<Alice>>, age = age=30)"
```

## Custom Modifiers

Modifiers are metadata annotations that influence how type class instances behave at runtime. For example, the `Modifier.rename` modifier tells a JSON codec to use a different field name during serialization, and `Modifier.transient` tells it to skip a field entirely.

While modifiers can be attached to schemas directly using Scala annotations (e.g., `@Modifier.transient`) or the `Schema#modifier` method, the `DerivationBuilder` provides a way to inject modifiers **programmatically at derivation time** without modifying the schema itself. This is particularly useful when:

- You don't control the schema definition (e.g., it comes from a library)
- You need different modifiers for different derivation contexts (e.g., one JSON codec with renamed fields, another without)
- You want to keep the schema clean and push format-specific concerns into the derivation layer

The `DerivationBuilder` offers three overloaded `modifier` methods:

```scala
final case class DerivationBuilder[TC[_], A](...) {
  def modifier[B](typeId: TypeId[B], modifier: Modifier.Reflect): DerivationBuilder[TC, A]
  def modifier[B](optic: Optic[A, B], modifier: Modifier): DerivationBuilder[TC, A]
  def modifier[B](typeId: TypeId[B], termName: String, modifier: Modifier.Term): DerivationBuilder[TC, A]
}
```

### Modifier Hierarchy

ZIO Blocks has two categories of modifiers:

- **`Modifier.Reflect`**: Type-level modifiers that apply to the schema node itself (e.g., `Modifier.config`).
- **`Modifier.Term`**: Field-level or case-level modifiers that apply to a specific field of a record or case of a variant (e.g., `Modifier.transient`, `Modifier.rename`, `Modifier.alias`).

Note that `Modifier.config` extends both `Modifier.Term` and `Modifier.Reflect`, so it can be used at both levels.

### Adding Modifiers by Optic

When you pass an optic and a `Modifier.Term` to the `modifier` method, the modifier is attached to the **term** (field or case) identified by the last segment of the optic path. When you pass a `Modifier.Reflect`, it is attached to the **schema node** targeted by the optic:

```scala

case class User(
  id: Long,
  name: String,
  email: String,
  internalScore: Double
)

object User extends CompanionOptics[User] {
  implicit val schema: Schema[User] = Schema.derived[User]

  val id: Lens[User, Long]              = $(_.id)
  val name: Lens[User, String]          = $(_.name)
  val email: Lens[User, String]         = $(_.email)
  val internalScore: Lens[User, Double] = $(_.internalScore)
}
```

Now we can derive a JSON codec with custom modifiers, renaming fields and marking one as transient, without changing the schema itself:

```scala
val jsonCodec: JsonCodec[User] = User.schema
  .deriving(JsonCodecDeriver)
  .modifier(User.name, Modifier.rename("full_name"))
  .modifier(User.email, Modifier.alias("mail"))
  .modifier(User.internalScore, Modifier.transient())
  .derive
```

In this example:
- The `name` field will be serialized as `full_name` in JSON.
- The `email` field will accept both `email` and `mail` as keys during deserialization.
- The `internalScore` field will be excluded from serialization entirely.

```scala
val user = User(1L, "Alice", "alice@example.com", 95.5)
// user: User = User(
//   id = 1L,
//   name = "Alice",
//   email = "alice@example.com",
//   internalScore = 95.5
// )
new String(jsonCodec.encode(user), "UTF-8")
// res40: String = "{\"id\":1,\"full_name\":\"Alice\",\"email\":\"alice@example.com\"}"
```

### Adding Modifiers by TypeId

The `modifier` method with `TypeId` allows you to add a `Modifier.Reflect` to all schema nodes of a given type. This is useful for attaching format-specific configuration metadata to all occurrences of a type:

```scala
val jsonCodec: JsonCodec[User] = User.schema
  .deriving(JsonCodecDeriver)
  .modifier(TypeId.of[User], Modifier.config("json", "camelCase"))
  .modifier(User.internalScore, Modifier.transient())
  .derive
```

### Adding Modifiers by TypeId and Term Name

The `modifier` method with `TypeId` and `termName` allows you to add a `Modifier.Term` to a specific field or case identified by name inside a parent type identified by its `TypeId`. This is useful when you want to target a specific term without constructing an optic for it. The `typeId` refers to the parent record/variant type that owns the term. If no term with the given name exists in the parent type, the modifier is silently ignored:

```scala
val jsonCodec: JsonCodec[User] = User.schema
  .deriving(JsonCodecDeriver)
  .modifier(User.schema.reflect.typeId, "name", Modifier.rename("full_name"))
  .modifier(User.schema.reflect.typeId, "internalScore", Modifier.transient())
  .derive
```

## Derivation Process In-Depth

Until now, we learned how to implement the `Deriver` methods for different schema patterns. But we haven't yet discussed how the overall derivation process works. In this section, we will go through the main steps of derivation in detail.

### PHASE 1: Deriving the Schema for the Target Type

The first step in deriving a type class instance is deriving a `Schema[A]` for the target type `A`. The `Schema[A]` contains a tree of `Reflect[Binding, A]` nodes that represent the structure of `A` using structural bindings:

For example, assume a case class of `Person(name: String, age: Int)`. The derived schema would look like this:

```
Schema[Person]
  └── Reflect.Record[Binding, Person]
        ├── Term("name", Reflect.Primitive[Binding, String])
        └── Term("age", Reflect.Primitive[Binding, Int])
```

Each node of the derived schema tree, carries two pieces of information:
- **Type Metadata**: Structural representation of the type (e.g., record, variant, primitive).
- **Binding Metadata**: Structural binding information for constructing/deconstructing values of that type.

This schema derivation is typically done using `Schema.derived[A]`, which uses Scala's compile-time reflection capabilities to inspect the structure of type `A` and build the corresponding schema.

For example, the following code derives the schema for `Person`:

```scala
case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}
```

### PHASE 2: Schema Tree Transformation

After generating the schema, by calling `Schema[A]#derive(deriver: Deriver[TC])`, the derivation process begins. This process involves transforming the schema tree from one that contains only structural bindings to one that also includes derived type class instances.

Initially, a `Schema[A]` contains `Reflect[Binding, A]` nodes that represent the structure of the type `A` using structural bindings. During derivation, the `Deriver` transforms these nodes into `Reflect[BindingInstance[TC, _, _], A]` nodes, where each node now contains both the structural binding and the derived type class instance for that part of the structure.

This tree transformation process starts at the root of the schema and recursively traverses each node until it reaches the leaf nodes (primitives). Now it can derive the type class instances for each leaf node by calling the `derivePrimitive` deriver method, which returns the derived type class instance wrapped in a `Lazy` container, i.e., `Lazy[TC[A]]`. The derivation builder now converts that schema node from `Reflect[Binding, A]` to `Reflect[BindingInstance[TC, _, _], A]`, where the `BindingInstance` contains both the structural binding and the derived type class instance. After converting all the leaf nodes, it backtracks up the tree, calling the appropriate `Deriver` methods for each structural pattern (record, variant, sequence, map, dynamic, wrapper) to derive type class instances for the composite types. At each step, it transforms the schema nodes from `Reflect[Binding, A]` to `Reflect[BindingInstance[TC, _, _], A]` accordingly. This process continues until it reaches the root of the schema tree, resulting in a final schema of type `Schema[A]` that contains `Reflect[BindingInstance[TC, _, _], A]` nodes throughout the entire structure.

The following diagram illustrates this transformation process:

```
       ┌──────────────────────────────┐
       │      Reflect[Binding,A]      │
       ├──────────────────────────────┤
       │   STRUCTURAL BINDING ONLY    │
       └──────────────────────────────┘
                      │
                      │ transform
                      ▼
       ┌──────────────────────────────┐
       │  Reflect[BindingInstance,A]  │
       ├──────────────────────────────┤
       │      STRUCTURAL BINDING      │
       │  WITH TYPE-CLASS INSTANCE    │
       └──────────────────────────────┘
                      │
                      │ extract
                      ▼
       ┌──────────────────────────────┐
       │         Lazy[TC[A]]          │
       ├──────────────────────────────┤
       │     TYPE-CLASS INSTANCE      │
       │           (TC[A])            │
       └──────────────────────────────┘
```

The `BindingInstance` is a container that bundles together a structural `Binding` and a derived type class instance `TC[A]`:

```scala
case class BindingInstance[TC[_], T, A](
  binding: Binding[T, A],    // Original runtime binding
  instance: Lazy[TC[A]]      // The derived type-class instance
)
```

For example, the transformation sequence for the `Person` data type would look like this:

```scala
case class Person(name: String, age: Int)

object Person {
    implicit val schema: Schema[Person] = Schema.derived[Person]
    implicit val show: Show[Person]     = schema.derive(DeriveShow)
}
```

- Step 1: Transform Primitive "name" (String)
   - deriver.derivePrimitive(String) → Lazy[Show[String]]
   - Creating BindingInstance(Binding.Primitive, Lazy[Show[String]])
   - Converting reflect node of `String` Schema from `Reflect[Binding, String]` to `Reflect[BindingInstance, String]`

- Step 2: Transform Primitive "age" (Int)
   - deriver.derivePrimitive(Int) → Lazy[Show[Int]]
   - Creating BindingInstance(Binding.Primitive, Lazy[Show[Int]])
   - Converting reflect node of `Int` Schema from `Reflect[Binding, Int]` to `Reflect[BindingInstance, Int]`

- Step 3: Transform Record "Person"
   - deriver.deriveRecord(fields with transformed metadata) → Lazy[Show[Person]]
   - Creating BindingInstance(Binding.Record, Lazy[Show[Person]])
   - Converting reflect node of `Person` Schema from `Reflect[Binding, Person]` to `Reflect[BindingInstance, Person]`

### PHASE 3: Extracting the Derived Type Class Instance

After the schema tree has been fully transformed to contain `Reflect[BindingInstance[TC, _, _], A]` nodes, now each node has a `BindingInstance` containing the original binding and the derived type class instance. The metadata container `BindingInstance` of the root node contains the derived type class wrapped in a `Lazy` container, i.e., `Lazy[TC[A]]`. To get the final derived type class instance, we call `force` on the `Lazy[TC[A]]`, which forces the unevaluated computation and retrieves the actual type class instance `TC[A]`.

### Phase 4: Using the Derived Show Instance

After derivation is complete, you can use the derived type class instance as needed. For example, you can use the derived `Show[Person]` instance to display a `Person` object:

```scala
val result = Person.show.show(Person("Alice", 30))
// result: String = "Person(name = Alice, age = 30)"
```

The interesting part here is how the `show` method of the derived `Show[Person]` instance works. It uses the `HasInstance` type class to access the derived `Show` instances for each field of the `Person` record (i.e., `Show[String]` for the `name` field and `Show[Int]` for the `age` field). This allows it to recursively display each field using its respective `Show` instance, demonstrating the composability and reusability of type class instances in the derivation system.

Please note that this happens when either the `Deriver` implementation uses the `HasInstance` implicit parameter or uses the centralized recursive approach to access nested derived instances.

---

## Validation

`Validation` is a sealed trait in ZIO Blocks that represents declarative constraints on primitive values. Validations are attached to `PrimitiveType` instances and are checked by `DynamicSchema#check`. Decoding does not check them; a wrapper type built with `Schema#transform` is the way to enforce a rule while decoding.

## Overview

The validation system in ZIO Blocks provides:

- **Declarative constraints** on numeric and string values, checked by `DynamicSchema#check`
- **Enforcement during decoding** for wrapper types built with `Schema#transform`, whose check runs when a value is decoded
- **Integration with wrapper types** via `Schema#transform` for custom validation logic
- **Schema error reporting** with path information for debugging via [`SchemaError`](./schema-error.md)

The `Validation` cases form a tree of numeric and string constraints:

```
Validation[A]
 ├── Validation.None                    (no constraint)
 │
 ├── Validation.Numeric[A]              (numeric constraints)
 │    ├── Positive                      (> 0)
 │    ├── Negative                      (< 0)
 │    ├── NonPositive                   (<= 0)
 │    ├── NonNegative                   (>= 0)
 │    ├── Range[A](min, max)            (within bounds)
 │    └── Set[A](values)                (one of specific values)
 │
 └── Validation.String                  (string constraints)
      ├── NonEmpty                      (length > 0)
      ├── Empty                         (length == 0)
      ├── Blank                         (whitespace only)
      ├── NonBlank                      (has non-whitespace)
      ├── Length(min, max)              (length bounds)
      └── Pattern(regex)                (regex match)
```

## Built-in Validations

`Validation` has two families of built-in constraints, one for numeric primitives and one for strings.

### Numeric Validations

Numeric validations apply to `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, and `BigDecimal`. The sign constraints are objects with no arguments, and the range and set constraints take their bounds as arguments:

```scala

// Sign constraints
val positive    = Validation.Numeric.Positive     // value > 0
val negative    = Validation.Numeric.Negative     // value < 0
val nonPositive = Validation.Numeric.NonPositive  // value <= 0
val nonNegative = Validation.Numeric.NonNegative  // value >= 0

// Range constraint (inclusive bounds)
val between    = Validation.Numeric.Range(Some(1), Some(100))  // 1 <= value <= 100
val atLeast    = Validation.Numeric.Range(Some(0), None)       // value >= 0 (no upper bound)
val atMost     = Validation.Numeric.Range(None, Some(1000))    // value <= 1000 (no lower bound)

// Set constraint (value must be one of the specified values)
val smallPrimes = Validation.Numeric.Set(Set(1, 2, 3, 5, 8, 13))
```

### String Validations

String validations apply to `String` primitive types. The `Length` and `Pattern` constraints take a bound or a regular expression:

```scala

// Content constraints
val nonEmpty = Validation.String.NonEmpty  // string.nonEmpty (length > 0)
val empty    = Validation.String.Empty     // string.isEmpty (length == 0)
val blank    = Validation.String.Blank     // string.trim.isEmpty (whitespace only)
val nonBlank = Validation.String.NonBlank  // string.trim.nonEmpty (has non-whitespace)

// Length constraint (inclusive bounds)
val lengthBetween = Validation.String.Length(Some(1), Some(255))  // 1 <= length <= 255
val lengthAtLeast = Validation.String.Length(Some(3), None)       // length >= 3
val lengthAtMost  = Validation.String.Length(None, Some(100))     // length <= 100

// Pattern constraint (regex)
val lowercase = Validation.String.Pattern("^[a-z]+$")   // lowercase letters only
val zipCode   = Validation.String.Pattern("^\\d{5}$")   // exactly 5 digits
```

## Validation with PrimitiveType

Validations are attached to `PrimitiveType` instances. Each primitive type carries its validation constraint:

```scala

// Int with no validation
val intType = PrimitiveType.Int(Validation.None)

// Positive Int
val positiveIntType = PrimitiveType.Int(Validation.Numeric.Positive)

// Non-empty String
val nonEmptyStringType = PrimitiveType.String(Validation.String.NonEmpty)

// String matching email pattern
val emailPattern = "^[A-Za-z0-9+_.-]+@[A-Za-z0-9.-]+$"
val emailStringType = PrimitiveType.String(Validation.String.Pattern(emailPattern))

// Int in range 1-10
val rangeIntType = PrimitiveType.Int(Validation.Numeric.Range(Some(1), Some(10)))
```

## Error Handling with SchemaError

When validation fails, ZIO Blocks returns a `SchemaError` that provides detailed information about what went wrong and where.

### SchemaError Structure

A `SchemaError` wraps a non-empty list of `Single` errors. The cases that matter for validation are these; `SchemaError.Single` also has `MigrationError` and the other `IntoError` cases:

```text
SchemaError.Single
 ├── ConversionFailed(source, details, cause)  // transformation/validation failure
 ├── MissingField(source, fieldName)           // required field not present
 ├── DuplicatedField(source, fieldName)        // field appears multiple times
 ├── ExpectationMismatch(source, expectation)  // type mismatch
 ├── UnknownCase(source, caseName)             // unknown variant case
 └── Message(source, details)                  // generic message
```

### Creating Validation Errors

Use the factory methods on `SchemaError` to create errors:

```scala

// For validation failures in transform
val error = SchemaError.validationFailed("must be positive")

// Generic message error
val msgError = SchemaError("Invalid input")

// With path information

val pathError = SchemaError.message("Value out of range", DynamicOptic.root.field("age"))

// Conversion failure with details
val convError = SchemaError.conversionFailed(Nil, "Expected ISO date format")
```

### Error Messages and Paths

`SchemaError` includes path information showing where in the data structure the error occurred. `SchemaError#message` prints the errors together with their paths, and the path methods add a segment to the front of every error:

```scala

val error: SchemaError = SchemaError.validationFailed("must be positive")

// Get the full error message
val msg: String = error.message

// Add path context to errors
val atField = error.atField("name")    // prepend .name to path
val atIndex = error.atIndex(0)         // prepend [0] to path
val atCase  = error.atCase("Some")     // prepend case context

// Combine multiple errors
val combined = error ++ SchemaError("Invalid input")
```

The decoding examples below show real messages with their paths.

## Integration with Wrapper Types

The most common way to use validation in ZIO Blocks is through `Schema#transform`, which creates a schema for a wrapper type with validation logic. The check runs when a value is decoded, so an invalid input never builds the wrapper.

### Basic Wrapper with Validation

To reject non-positive integers in a wrapper type, throw a `SchemaError` in the function that builds the wrapper:

```scala

case class PositiveInt private (value: Int)

object PositiveInt {
  def unsafeMake(n: Int): PositiveInt =
    if (n > 0) PositiveInt(n)
    else throw SchemaError.validationFailed("must be positive")

  implicit val schema: Schema[PositiveInt] =
    Schema[Int].transform(unsafeMake, _.value)
}
```

Decoding a positive number builds the wrapper, and decoding a negative number returns `Left`:

```scala
println(PositiveInt.schema.derive(JsonFormat).decode("5").map(_.value))
// Right(5)
println(PositiveInt.schema.derive(JsonFormat).decode("-5").left.map(_.message))
// Left(must be positive at: .wrapped)
```

### Email Type with Regex Validation

To check a string against a regular expression before building the wrapper:

```scala

case class Email private (value: String)

object Email {
  private val EmailRegex = "^[A-Za-z0-9+_.-]+@[A-Za-z0-9.-]+$".r

  def unsafeMake(s: String): Email =
    s match {
      case EmailRegex(_*) => Email(s)
      case _ => throw SchemaError.validationFailed("Invalid email format")
    }

  implicit val schema: Schema[Email] =
    Schema[String].transform(unsafeMake, _.value)
}
```

### NonEmptyString with a Non-Empty Check

To reject an empty string in a wrapper type:

```scala

case class NonEmptyString private (value: String)

object NonEmptyString {
  def unsafeMake(s: String): NonEmptyString =
    if (s.nonEmpty) NonEmptyString(s)
    else throw SchemaError.validationFailed("String must not be empty")

  implicit val schema: Schema[NonEmptyString] =
    Schema[String].transform(unsafeMake, _.value)
}
```

### Range-Bounded Integer

To keep an integer between 0 and 100, with the bounds in the message:

```scala

case class Percentage private (value: Int)

object Percentage {
  def unsafeMake(n: Int): Percentage =
    if (n >= 0 && n <= 100) Percentage(n)
    else throw SchemaError.validationFailed(s"Percentage must be 0-100, got $n")

  implicit val schema: Schema[Percentage] =
    Schema[Int].transform(unsafeMake, _.value)
}
```

### Bidirectional Validation

Use the two-function `Schema#transform` when both building and unwrapping the value need a check. The `to` function builds the wrapper from the raw value, and the `from` function unwraps it:

```scala

case class BoundedValue(value: Int)

object BoundedValue {
  implicit val schema: Schema[BoundedValue] = Schema[Int].transform(
    to = n =>
      if (n >= 0 && n < 100) BoundedValue(n)
      else throw SchemaError.validationFailed("Value must be in [0, 100)"),
    from = v =>
      if (v.value >= 0) v.value
      else throw SchemaError.validationFailed("Corrupted value")
  )
}
```

## Validation at Decode Time

Decoding runs the check of a wrapper built with `Schema#transform`, but it does not enforce `Validation` constraints; `DynamicSchema#check` does.

### Decoding with Validation

A wrapper built with `Schema#transform` runs its check when a value is decoded from JSON or a `DynamicValue`. In the example below, `Order` contains a `PositiveInt`, so decoding an order with a quantity of `-5` fails inside the wrapper:

```scala

case class PositiveInt(value: Int)
object PositiveInt {
  def unsafeMake(n: Int): PositiveInt =
    if (n > 0) PositiveInt(n)
    else throw SchemaError.validationFailed("must be positive")

  implicit val schema: Schema[PositiveInt] =
    Schema[Int].transform(unsafeMake, _.value)
}

case class Order(quantity: PositiveInt, price: BigDecimal)
object Order {
  implicit val schema: Schema[Order] = Schema.derived
}
```

Decoding the JSON with the codec of `Order` reports the path of the failed check, which ends in `.wrapped`:

```scala
println(Order.schema.derive(JsonFormat).decode("""{"quantity": -5, "price": 99.99}""").left.map(_.message))
// Left(must be positive at: .quantity.wrapped)
```

### DynamicValue Validation

Use `DynamicSchema#check` to validate `DynamicValue` instances against the structure of a schema, and to check its `Validation` constraints. Decoding does not run these checks, so call `DynamicSchema#check` on a value before you use it:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Create a DynamicSchema for validation
val dynamicSchema: DynamicSchema = Schema[Person].toDynamicSchema

// Create a DynamicValue to validate
val value = DynamicValue.Record(
  "name" -> DynamicValue.Primitive(PrimitiveValue.String("Alice")),
  "age"  -> DynamicValue.Primitive(PrimitiveValue.Int(30))
)
```

`DynamicSchema#check` returns `None` when the value conforms, and `Some` with the error otherwise:

```scala
println(dynamicSchema.check(value).map(_.message))
// None
println(dynamicSchema.conforms(value))
// true
```

### Converting DynamicSchema to Validating Schema

`DynamicSchema#toSchema` creates a `Schema[DynamicValue]` that rejects values that don't conform to the structure. Here the `name` field holds an `Int` instead of a `String`:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val dynamicSchema: DynamicSchema = Schema[Person].toDynamicSchema
val validatingSchema: Schema[DynamicValue] = dynamicSchema.toSchema

val invalidValue = DynamicValue.Record(
  "name" -> DynamicValue.Primitive(PrimitiveValue.Int(42))  // wrong type!
)
```

Converting `invalidValue` with `validatingSchema` returns `Left` with the error:

```scala
println(validatingSchema.fromDynamicValue(invalidValue).left.map(_.message))
// Left(Expected String, got Int at: .name at: .wrapped)
```

## Validation in JSON Schema

`Schema#toJsonSchema` derives a JSON Schema from a schema, and it does not include the `Validation` constraints in the output. A `minimum`, `maxLength` or `pattern` keyword is therefore absent from a JSON Schema that ZIO Blocks derives, even when the type has a constraint.

The constraints do come back in the other direction. Parsing a JSON Schema with `Schema.fromJsonSchema` maps keywords such as `minimum`, `maxLength` and `pattern` to `Validation` instances.

`Schema#toJsonSchema` also leaves out the `Validation` rules that a library attaches for you, such as the ones Iron's ZIO Blocks module adds for its refined types. [How to Use Iron Refined Types and Typed IDs with Schema](../../guides/use-iron-refined-types-and-typed-ids.md) shows where those rules appear and where they do not.

## Composing Validations

The current `Validation` ADT does not support combining multiple validations on a single primitive (e.g., both `NonEmpty` and `Pattern`). For complex validation logic, use `Schema#transform`:

```scala

case class Username private (value: String)

object Username {
  private val UsernameRegex = "^[a-z][a-z0-9_]{2,19}$".r

  def unsafeMake(s: String): Username = {
    if (s.isEmpty)
      throw SchemaError.validationFailed("Username cannot be empty")
    else if (s.length < 3)
      throw SchemaError.validationFailed("Username must be at least 3 characters")
    else if (s.length > 20)
      throw SchemaError.validationFailed("Username cannot exceed 20 characters")
    else if (!s.matches(UsernameRegex.regex))
      throw SchemaError.validationFailed("Username must start with a letter and contain only lowercase letters, numbers, and underscores")
    else
      Username(s)
  }

  implicit val schema: Schema[Username] =
    Schema[String].transform(unsafeMake, _.value)
}
```

## Best Practices

The practices below keep validation rules enforced in one place and make their failures easy to act on.

### 1. Use Wrapper Types for Domain Validation

Prefer creating dedicated wrapper types with `Schema#transform` over relying solely on `Validation` constraints:

```scala

// Good: Explicit domain type with validation
case class OrderId private (value: String)
object OrderId {
  def unsafeMake(s: String): OrderId =
    if (s.matches("^ORD-\\d{8}$")) OrderId(s)
    else throw SchemaError.validationFailed("Invalid order ID format")

  implicit val schema: Schema[OrderId] =
    Schema[String].transform(unsafeMake, _.value)
}
```

The alternative keeps the raw `String` and checks it in a separate step, which lets any code path skip the check.

### 2. Provide Clear Error Messages

Include context in error messages to help users understand what went wrong:

```scala

def requireAge(age: Int): Unit =
  if (age < 0 || age > 150)
    throw SchemaError.validationFailed(s"Age must be between 0 and 150, got $age")
```

A message such as `Invalid age` gives the user no value or range to correct.

### 3. Combine Errors When Possible

For multiple validation failures, combine them into a single `SchemaError`:

```scala

case class Input(name: String, age: Int)
case class ValidInput(input: Input)

def validate(input: Input): Either[SchemaError, ValidInput] = {
  val errors = List.newBuilder[SchemaError]

  if (input.name.isEmpty)
    errors += SchemaError.validationFailed("name is required")
  if (input.age < 0)
    errors += SchemaError.validationFailed("age must be non-negative")

  val allErrors = errors.result()
  if (allErrors.isEmpty) Right(ValidInput(input))
  else Left(allErrors.reduce(_ ++ _))
}
```

Validating an `Input` with an empty name and a negative age reports both failures in one `SchemaError`:

```scala
println(validate(Input("", -1)).left.map(_.message))
// Left(name is required
// age must be non-negative)
```

See also the [How to Define Validated Newtypes and Collect Every Validation Error](../../guides/validated-newtypes.md) guide, which builds validated wrapper types with smart constructors and collects every failing check.

---

## Smithy

`zio-blocks-smithy` is a **Smithy IDL parser and AST library** providing a complete representation of Smithy 2.0 API models. It enables parsing Smithy IDL text into rich data structures, querying shape definitions, and pretty-printing models back to valid IDL syntax—all without external dependencies.

## Installation

Add the library to your build configuration:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-smithy" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x

## Quick Start

Parse Smithy IDL text into a model, query shapes, and serialize back:

```scala

val smithyText = """$version: "2"
namespace com.example.api

structure User {
  @required
  id: String
  name: String
}

operation GetUser {
  input: GetUserInput
  output: User
}

structure GetUserInput {
  @required
  id: String
}
"""

// Parse IDL text into a model
val result = SmithyModel.parse(smithyText)

// Access shapes and data
result match {
  case Right(model) =>
    model.findShape("User").foreach { userDef =>
      println(s"Found shape: ${userDef.name}")
    }
  case Left(error) =>
    println(s"Parse error: ${error.message}")
}

// Serialize back to IDL
result.foreach { model =>
  val idlText = model.prettyPrint
  println(idlText)
}
```

## Core Types

The library provides core types that work together to parse, query, and serialize Smithy models. The main types work together in a parsing → querying → serialization pipeline:

```
Smithy IDL Text
      ↓
SmithyModel.parse (public API)
      ↓
SmithyModel (contains shapes, metadata, traits)
      ├─ shapes: List[ShapeDefinition]
      │   └─ shape: Shape (sealed trait — central type)
      │       ├─ StructureShape(members: List[MemberDefinition])
      │       ├─ ListShape(member: MemberDefinition)
      │       ├─ MapShape(key: MemberDefinition, value: MemberDefinition)
      │       ├─ ServiceShape(operations, resources, errors)
      │       ├─ OperationShape(input, output, errors)
      │       ├─ UnionShape(members: List[MemberDefinition])
      │       ├─ EnumShape(members: List[EnumMember])
      │       ├─ ResourceShape(identifiers, create, read, update, delete, list, ...)
      │       ├─ StringShape, BooleanShape, IntegerShape, and 10 more simple shapes
      │       └─ ... (20 subtypes in total — see the Shape Catalog below)
      ├─ MemberDefinition(name: String, target: ShapeId, traits: List[TraitApplication])
      ├─ TraitApplication(id: ShapeId, value: Option[NodeValue])
      ├─ ShapeId (namespace + name identifier)
      └─ NodeValue (metadata values: String, Number, Boolean, Array, Object, Null)
            ↓
SmithyModel.prettyPrint (public API)
      ↓
Smithy IDL Text
```

The root container for a Smithy model. Contains version, namespace, shapes, metadata, and trait applications. The case class and companion object expose the following API:

```scala
case class SmithyModel(
  version: String,           // Smithy version (e.g., "2")
  namespace: String,
  useStatements: List[ShapeId],
  metadata: Map[String, NodeValue],
  shapes: List[ShapeDefinition],
  applyStatements: List[ApplyStatement] = Nil
) {
  def findShape(name: String): Option[ShapeDefinition]
  def allShapeIds: List[ShapeId]
  def prettyPrint: String
  def prettyPrint(indent: Int): String
}

object SmithyModel {
  def parse(input: String): Either[SmithyError, SmithyModel]
}
```

## Shape Catalog

`Shape` is a sealed trait with 20 subtypes, covering the shape categories the Smithy specification defines. Every shape carries a `Shape#name` and a list of applied `Shape#traits`; the families differ in what else they hold:

```scala
sealed trait Shape {
  def name: String
  def traits: List[TraitApplication]
}
```

| Family    | Subtypes | What they add                            |
| --------- | -------: | ---------------------------------------- |
| Simple    |       13 | Nothing — name and traits only            |
| Enum      |        2 | A member list of allowed values           |
| Aggregate |        4 | Member definitions naming target shapes   |
| Service   |        3 | `ShapeId` references to other shapes      |

A parsed shape is wrapped in a `ShapeDefinition`, which pairs the name with the shape. `Shape` also carries its own `Shape#name`, so the two agree and either can be read:

```scala
final case class ShapeDefinition(name: String, shape: Shape)
```

### Simple Shapes

Thirteen shapes carry no structure beyond their name and traits. They differ only in which IDL keyword produces them and what the target protocol is expected to do with them:

| Type              | IDL keyword  | Represents                        |
| ----------------- | ------------ | --------------------------------- |
| `BlobShape`       | `blob`       | Arbitrary binary data              |
| `BooleanShape`    | `boolean`    | True/false values                  |
| `StringShape`     | `string`     | UTF-8 text                         |
| `ByteShape`       | `byte`       | 8-bit signed integer               |
| `ShortShape`      | `short`      | 16-bit signed integer              |
| `IntegerShape`    | `integer`    | 32-bit signed integer              |
| `LongShape`       | `long`       | 64-bit signed integer              |
| `FloatShape`      | `float`      | Single-precision IEEE 754          |
| `DoubleShape`     | `double`     | Double-precision IEEE 754          |
| `BigIntegerShape` | `bigInteger` | Arbitrarily large signed integer   |
| `BigDecimalShape` | `bigDecimal` | Arbitrary-precision decimal        |
| `TimestampShape`  | `timestamp`  | A point in time                    |
| `DocumentShape`   | `document`   | Protocol-agnostic open content     |

Because they share one shape, parsing them produces values that differ only in their type:

```scala

val simpleModel = SmithyModel.parse(
  """$version: "2"
    |namespace com.example
    |blob Payload
    |timestamp CreatedAt
    |document Metadata
    |""".stripMargin
).toOption.get
```

Each definition names the shape and holds the corresponding subtype:

```scala
simpleModel.shapes
// res1: List[ShapeDefinition] = List(
//   ShapeDefinition(
//     name = "Payload",
//     shape = BlobShape(name = "Payload", traits = List())
//   ),
//   ShapeDefinition(
//     name = "CreatedAt",
//     shape = TimestampShape(name = "CreatedAt", traits = List())
//   ),
//   ShapeDefinition(
//     name = "Metadata",
//     shape = DocumentShape(name = "Metadata", traits = List())
//   )
// )
```

`DocumentShape` is the one to reach for when a field's contents are not known at model time — it is the Smithy equivalent of an open JSON value, and no member list constrains it.

### Enum Shapes

`EnumShape` and `IntEnumShape` each hold a fixed set of permitted values. They differ in the value type and in whether the value is optional:

```scala
final case class EnumShape(
  name: String,
  traits: List[TraitApplication] = Nil,
  members: List[EnumMember] = Nil
) extends Shape

final case class IntEnumShape(
  name: String,
  traits: List[TraitApplication] = Nil,
  members: List[IntEnumMember] = Nil
) extends Shape
```

`EnumMember` carries an `Option[String]`, because the IDL allows a bare member name; `IntEnumMember` carries a required `Int`, because an integer enum has no name to fall back on:

```scala
final case class EnumMember(name: String, value: Option[String] = None, traits: List[TraitApplication] = Nil)
final case class IntEnumMember(name: String, value: Int, traits: List[TraitApplication] = Nil)
```

A string enum with explicit values fills in each `value`:

```scala
val colorModel = SmithyModel.parse(
  """$version: "2"
    |namespace com.example
    |enum Color {
    |    RED = "red"
    |    GREEN = "green"
    |}
    |""".stripMargin
).toOption.get
```

Each member pairs the declared name with the string it maps to:

```scala
colorModel.shapes
// res2: List[ShapeDefinition] = List(
//   ShapeDefinition(
//     name = "Color",
//     shape = EnumShape(
//       name = "Color",
//       traits = List(),
//       members = List(
//         EnumMember(name = "RED", value = Some("red"), traits = List()),
//         EnumMember(name = "GREEN", value = Some("green"), traits = List())
//       )
//     )
//   )
// )
```

Omitting the values leaves `value` absent rather than duplicating the name, so a consumer wanting the effective wire value reads `EnumMember#value` and falls back to `EnumMember#name`:

```scala
val suitModel = SmithyModel.parse(
  """$version: "2"
    |namespace com.example
    |enum Suit {
    |    CLUB
    |    HEART
    |}
    |""".stripMargin
).toOption.get
```

The distinction survives parsing, which is what lets a round trip reproduce the original document:

```scala
suitModel.shapes
// res3: List[ShapeDefinition] = List(
//   ShapeDefinition(
//     name = "Suit",
//     shape = EnumShape(
//       name = "Suit",
//       traits = List(),
//       members = List(
//         EnumMember(name = "CLUB", value = None, traits = List()),
//         EnumMember(name = "HEART", value = None, traits = List())
//       )
//     )
//   )
// )
```

An integer enum requires a value for every member, so `IntEnumMember` holds an `Int` rather than an option:

```scala
val cardModel = SmithyModel.parse(
  """$version: "2"
    |namespace com.example
    |intEnum FaceCard {
    |    JACK = 11
    |    QUEEN = 12
    |}
    |""".stripMargin
).toOption.get
```

Reading the members gives the integers directly:

```scala
cardModel.shapes
// res4: List[ShapeDefinition] = List(
//   ShapeDefinition(
//     name = "FaceCard",
//     shape = IntEnumShape(
//       name = "FaceCard",
//       traits = List(),
//       members = List(
//         IntEnumMember(name = "JACK", value = 11, traits = List()),
//         IntEnumMember(name = "QUEEN", value = 12, traits = List())
//       )
//     )
//   )
// )
```

### Aggregate Shapes

Four shapes compose other shapes, and all of them do it through `MemberDefinition` — a name, the `ShapeId` of the target, and any traits on the member itself:

| Type             | IDL keyword | Members                                              |
| ---------------- | ----------- | ---------------------------------------------------- |
| `ListShape`      | `list`      | `member: MemberDefinition`                            |
| `MapShape`       | `map`       | `key` and `value`, both `MemberDefinition`             |
| `StructureShape` | `structure` | `members: List[MemberDefinition]`                     |
| `UnionShape`     | `union`     | `members: List[MemberDefinition]`, one set at a time   |

A structure's members name their targets by `ShapeId`, not by nested shape, so the model stays flat and a member's target is resolved by lookup:

```scala
val userModel = SmithyModel.parse(
  """$version: "2"
    |namespace com.example
    |structure User {
    |    @required
    |    id: String
    |    tags: TagList
    |}
    |list TagList {
    |    member: String
    |}
    |""".stripMargin
).toOption.get
```

Both shapes appear at the top level, and the `tags` member of `User` points at `TagList` by reference:

```scala
userModel.shapes
// res5: List[ShapeDefinition] = List(
//   ShapeDefinition(
//     name = "User",
//     shape = StructureShape(
//       name = "User",
//       traits = List(),
//       members = List(
//         MemberDefinition(
//           name = "id",
//           target = ShapeId(namespace = "", name = "String"),
//           traits = List(
//             TraitApplication(
//               id = ShapeId(namespace = "smithy.api", name = "required"),
//               value = None
//             )
//           )
//         ),
//         MemberDefinition(
//           name = "tags",
//           target = ShapeId(namespace = "", name = "TagList"),
//           traits = List()
//         )
//       )
//     )
//   ),
//   ShapeDefinition(
//     name = "TagList",
//     shape = ListShape(
//       name = "TagList",
//       traits = List(),
//       member = MemberDefinition(
//         name = "member",
//         target = ShapeId(namespace = "", name = "String"),
//         traits = List()
//       )
//     )
//   )
// )
```

:::note[`member` has no default, `traits` does]
`ListShape` and `MapShape` declare their `Shape#traits` parameter with a default *before* the parameters that have none, so positional construction does not work — `ListShape("TagList", member = m)` compiles while `ListShape("TagList", m)` does not. `StructureShape` and `UnionShape` default their member lists, so both forms work there.
:::

### Service Shapes

`ServiceShape`, `OperationShape`, and `ResourceShape` describe an API rather than a value, and all of their cross-references are `ShapeId`s:

```scala
final case class ServiceShape(
  name: String,
  traits: List[TraitApplication] = Nil,
  version: Option[String] = None,
  operations: List[ShapeId] = Nil,
  resources: List[ShapeId] = Nil,
  errors: List[ShapeId] = Nil
) extends Shape

final case class OperationShape(
  name: String,
  traits: List[TraitApplication] = Nil,
  input: Option[ShapeId] = None,
  output: Option[ShapeId] = None,
  errors: List[ShapeId] = Nil
) extends Shape
```

`ResourceShape` is the largest shape in the module, because a Smithy resource binds identifiers, five named lifecycle operations, and three further reference lists:

| Field                  | Type                   | Meaning                                           |
| ---------------------- | ---------------------- | ------------------------------------------------- |
| `identifiers`          | `Map[String, ShapeId]`  | Identifier names mapped to their target shapes     |
| `create`               | `Option[ShapeId]`       | Create lifecycle operation                         |
| `read`                 | `Option[ShapeId]`       | Read lifecycle operation                           |
| `update`               | `Option[ShapeId]`       | Update lifecycle operation                         |
| `delete`               | `Option[ShapeId]`       | Delete lifecycle operation                         |
| `list`                 | `Option[ShapeId]`       | List lifecycle operation                           |
| `operations`           | `List[ShapeId]`         | Instance operations that are not lifecycle ones     |
| `collectionOperations` | `List[ShapeId]`         | Operations on the collection rather than one item   |
| `resources`            | `List[ShapeId]`         | Child resources                                    |

The five lifecycle fields are separate rather than a map, which is what makes "does this resource support deletion?" a field access instead of a lookup:

```scala
val resourceModel = SmithyModel.parse(
  """$version: "2"
    |namespace com.example
    |resource FooResource {
    |    identifiers: {id: FooId}
    |    read: GetFoo
    |    list: ListFoos
    |}
    |""".stripMargin
).toOption.get
```

Unset lifecycle operations stay `None`, so an absent `create` is distinguishable from one bound to an operation:

```scala
resourceModel.shapes
// res6: List[ShapeDefinition] = List(
//   ShapeDefinition(
//     name = "FooResource",
//     shape = ResourceShape(
//       name = "FooResource",
//       traits = List(),
//       identifiers = Map("id" -> ShapeId(namespace = "", name = "FooId")),
//       create = None,
//       read = Some(ShapeId(namespace = "", name = "GetFoo")),
//       update = None,
//       delete = None,
//       list = Some(ShapeId(namespace = "", name = "ListFoos")),
//       operations = List(),
//       collectionOperations = List(),
//       resources = List()
//     )
//   )
// )
```

### Shape References

Every cross-shape reference is a `ShapeRef`, a sealed trait with exactly two cases:

```scala
sealed trait ShapeRef

final case class ShapeId(namespace: String, name: String) extends ShapeRef

object ShapeId {
  final case class Member(shape: ShapeId, memberName: String) extends ShapeRef
  def parse(s: String): Either[String, ShapeRef]
}
```

`ShapeRef` exists to give `ShapeId` and `ShapeId.Member` a common supertype without a Scala 3 union type, so the same signatures compile on 2.13.

Rendering follows the IDL: a shape is `namespace#name`, and a member appends `$` and the member name:

```scala
ShapeId("com.example", "User").toString
// res7: String = "com.example#User"
ShapeId.Member(ShapeId("com.example", "User"), "id").toString
// res8: String = "com.example#User$id"
```

`ShapeId.parse` reads either form back, choosing the case by whether a `$` is present:

```scala
ShapeId.parse("com.example#User")
// res9: Either[String, ShapeRef] = Right(
//   ShapeId(namespace = "com.example", name = "User")
// )
ShapeId.parse("com.example#User$id")
// res10: Either[String, ShapeRef] = Right(
//   Member(
//     shape = ShapeId(namespace = "com.example", name = "User"),
//     memberName = "id"
//   )
// )
```

Malformed input is reported rather than thrown, and the message names the rule that failed:

```scala
ShapeId.parse("User")
// res11: Either[String, ShapeRef] = Left(
//   "ShapeId must contain '#' separator, got: User"
// )
ShapeId.parse("com.example#User$id$extra")
// res12: Either[String, ShapeRef] = Left(
//   "Member reference must have exactly one separator, got: com.example#User$id$extra"
// )
```

#### Resolving a Reference

A reference names a shape; it does not contain one. Resolving means looking the target up in the model, which `SmithyModel#findShape` does by **name** rather than by `ShapeId`:

```scala
final case class SmithyModel(...) {
  def findShape(name: String): Option[ShapeDefinition]
  def allShapeIds: List[ShapeId]
}
```

Following a structure member to its target definition is therefore a lookup on the name inside the `ShapeId`:

```scala
val tagsTarget = userModel.shapes
  .collectFirst { case ShapeDefinition("User", s: StructureShape) => s }
  .flatMap(_.members.find(_.name == "tags"))
  .map(_.target)
```

The member's target resolves to the `list` shape declared alongside it:

```scala
tagsTarget
// res13: Option[ShapeId] = Some(ShapeId(namespace = "", name = "TagList"))
tagsTarget.flatMap(id => userModel.findShape(id.name))
// res14: Option[ShapeDefinition] = Some(
//   ShapeDefinition(
//     name = "TagList",
//     shape = ListShape(
//       name = "TagList",
//       traits = List(),
//       member = MemberDefinition(
//         name = "member",
//         target = ShapeId(namespace = "", name = "String"),
//         traits = List()
//       )
//     )
//   )
// )
```

Looking up by name rather than by `ShapeId` is not a shortcut — it matches what the parser produces. An IDL target written without a namespace prefix becomes a `ShapeId` whose namespace is the **empty string**, not the model's namespace and not `smithy.api`:

```scala
userModel.shapes.collect { case ShapeDefinition(_, s: ListShape) => s.member.target }
// res15: List[ShapeId] = List(ShapeId(namespace = "", name = "String"))
resourceModel.shapes.collect { case ShapeDefinition(_, s: ResourceShape) => s.identifiers }
// res16: List[Map[String, ShapeId]] = List(
//   Map("id" -> ShapeId(namespace = "", name = "FooId"))
// )
```

Trait identifiers are the exception: the parser resolves those against the prelude, so `@required` arrives fully qualified:

```scala
userModel.shapes
  .collect { case ShapeDefinition("User", s: StructureShape) => s }
  .flatMap(_.members.flatMap(_.traits.map(_.id)))
// res17: List[ShapeId] = List(
//   ShapeId(namespace = "smithy.api", name = "required")
// )
```

Two consequences follow. A reference to a prelude shape such as `String` has no definition in the parsed model, so resolving it yields nothing — a consumer generating code has to recognize prelude names itself:

```scala
userModel.findShape("String")
// res18: Option[ShapeDefinition] = None
```

And a `ShapeId` taken from a parsed model does not necessarily round-trip through `ShapeId.parse`, because the renderer emits the empty namespace as a bare `#` that the parser then rejects:

```scala
ShapeId("", "String").toString
// res19: String = "#String"
ShapeId.parse(ShapeId("", "String").toString)
// res20: Either[String, ShapeRef] = Left("ShapeId namespace cannot be empty")
```

:::warning[Namespaces on references are not populated]
Because unprefixed targets carry an empty namespace, comparing `ShapeId#namespace` on a parsed reference tells you nothing unless the IDL spelled the namespace out. `SmithyModel#findShape` matching on name alone is consistent with that, but it also means a model that uses a `use` statement to import `other.ns#User` while declaring its own `User` cannot distinguish the two by reference alone.
:::

## Parsing

Parse Smithy IDL text into structured models using `SmithyModel.parse`, handle errors, and validate round-trips.

### Basic Parsing

Parse Smithy IDL text and handle the result:

```scala

val smithyText = """$version: "2"
namespace com.example

string Name
"""

SmithyModel.parse(smithyText) match {
  case Right(model) =>
    println(s"Parsed ${model.shapes.length} shapes")
  case Left(error) =>
    println(s"Error: ${error.message}")
}
```

### Handling Parse Errors

Access error details including line and column information when parsing fails. `SmithyError` provides detailed context to help locate and fix issues in your Smithy definitions:

```scala

val invalidSmithy = """$version: "2"
namespace com.example

structure User {
  invalid syntax
}
"""

SmithyModel.parse(invalidSmithy) match {
  case Right(_) =>
    println("Unexpected success")
  case Left(error) =>
    println(s"Error at line ${error.line}, column ${error.column}: ${error.message}")
}
```

### Round-Trip Validation

Verify a model parses correctly by round-tripping (parse → serialize → parse again):

```scala

val original = """$version: "2"
namespace com.example

string MyString
"""

val parsed = SmithyModel.parse(original)
val reprinted = parsed.map(_.prettyPrint)
val reparsed = reprinted.flatMap(SmithyModel.parse)

println(reparsed.isRight)  // true if round-trip succeeds
```

## Querying & Traversing Shapes

Once you have a parsed model, query shapes by name, pattern match on shape types, and traverse their members.

### Finding Shapes

Locate shapes by name or retrieve all shape identifiers:

```scala

val model = SmithyModel.parse("""$version: "2"
namespace example

structure User {
  id: String
  name: String
}
""").toOption.get

// Find by name
model.findShape("User").foreach { shapeDef =>
  println(s"Found: ${shapeDef.name}")
}

// Get all shape IDs
val allIds = model.allShapeIds
println(s"Total shapes: ${allIds.length}")
```

### Pattern Matching on Shapes

Determine shape type and access type-specific properties:

```scala

val model = SmithyModel.parse("""$version: "2"
namespace example

structure User { id: String }
list UserIds { member: String }
""").toOption.get

model.findShape("User").foreach { shapeDef =>
  shapeDef.shape match {
    case struct: StructureShape =>
      println(s"Structure with ${struct.members.length} members")
    case list: ListShape =>
      println(s"List of ${list.member.target}")
    case _ =>
      println("Other shape type")
  }
}
```

### Traversing Members

Iterate over structure/union members and inspect their traits:

```scala

val model = SmithyModel.parse("""$version: "2"
namespace example

structure User {
  @required
  id: String
  name: String
}
""").toOption.get

model.findShape("User").foreach { shapeDef =>
  shapeDef.shape match {
    case struct: StructureShape =>
      struct.members.foreach { member =>
        val required = member.traits.exists(_.id.name == "required")
        println(s"${member.name}: ${member.target} (required: $required)")
      }
    case _ => ()
  }
}
```

## Building Models Programmatically

Construct Smithy models in code by creating shapes, adding traits, and assembling them into a complete model.

### Creating Shapes

Programmatically construct shapes and assemble them into a complete model:

```scala

val userStructure = StructureShape(
  "User",
  traits = Nil,
  members = List(
    MemberDefinition(
      "id",
      ShapeId("smithy.api", "String"),
      traits = List(TraitApplication.required)
    ),
    MemberDefinition(
      "name",
      ShapeId("smithy.api", "String"),
      traits = Nil
    )
  )
)

val model = SmithyModel(
  version = "2",
  namespace = "com.example",
  useStatements = Nil,
  metadata = Map.empty,
  shapes = List(ShapeDefinition("User", userStructure))
)
```

### Adding Traits

Attach metadata traits to shapes during construction. `TraitApplication` provides companion object helper methods like `required`, `documentation`, and others for common traits:

```scala

val serviceShape = ServiceShape(
  "UserService",
  traits = List(
    TraitApplication.documentation("User management API")
  ),
  version = Some("1.0"),
  operations = List(
    ShapeId("com.example", "GetUser"),
    ShapeId("com.example", "CreateUser")
  ),
  resources = Nil,
  errors = Nil
)
```

## Serializing Models

Convert models back to valid Smithy IDL text using `prettyPrint`, with options for custom formatting.

### Basic Serialization

Convert a model to valid Smithy IDL text:

```scala

val model = SmithyModel(
  version = "2",
  namespace = "com.example",
  useStatements = Nil,
  metadata = Map.empty,
  shapes = List(
    ShapeDefinition("Name", StringShape("Name"))
  )
)

val idlText = model.prettyPrint
println(idlText)
```

### Custom Indentation

Control indentation width when serializing models:

```scala

val model = SmithyModel(
  version = "2",
  namespace = "com.example",
  useStatements = Nil,
  metadata = Map.empty,
  shapes = List(
    ShapeDefinition("Data", StructureShape(
      "Data",
      traits = Nil,
      members = List(
        MemberDefinition("field1", ShapeId("smithy.api", "String")),
        MemberDefinition("field2", ShapeId("smithy.api", "String"))
      )
    ))
  )
)

val compact = model.prettyPrint(indent = 2)
val verbose = model.prettyPrint(indent = 8)
```

## Common Use-Cases

See how to apply Smithy parsing and querying to real-world workflows: code generation, validation, and model transformation. For a complete, compiled walk-through of code generation, see [Generate Scala Code from a Smithy Model](../guides/generate-scala-code-from-smithy.md).

### Use-Case 1: Code Generation

Load a Smithy model and generate code for each operation:

```scala

val model = SmithyModel.parse("""$version: "2"
namespace api

service MyService {
  operations: [GetUser, CreateUser]
}

@http(method: "GET", uri: "/users/{id}")
operation GetUser {
  input: GetUserInput
  output: User
}

@http(method: "POST", uri: "/users")
operation CreateUser {
  input: CreateUserInput
  output: User
}

structure User { id: String, name: String }
structure GetUserInput { @required id: String }
structure CreateUserInput { @required name: String }
""").toOption.get

// Generate code stubs for each operation by pattern matching:

model.shapes.foreach { shapeDef =>
  shapeDef.shape match {
    case op: OperationShape =>
      println(s"// Generate operation: ${op.name}")
      op.input.foreach(in => println(s"//   input: ${in.name}"))
      op.output.foreach(out => println(s"//   output: ${out.name}"))
    case _ => ()
  }
}
```

### Use-Case 2: Validation & Analysis

Find deprecated shapes and analyze trait coverage:

```scala

val model = SmithyModel.parse("""$version: "2"
namespace example

@deprecated
structure LegacyUser { id: String }

structure ModernUser {
  @required
  id: String
  email: String
}
""").toOption.get

// Find all deprecated shapes:

val deprecated = model.shapes.filter { shapeDef =>
  shapeDef.shape.traits.exists(_.id.name == "deprecated")
}

println(s"Deprecated shapes: ${deprecated.map(_.name)}")
```

### Use-Case 3: Model Transformation

Parse, modify, and re-serialize a model with updated metadata:

```scala

val original = """$version: "2"
namespace com.example

string UserId
"""

val modified = SmithyModel.parse(original).map { model =>
  // Add metadata to the model:

  val newMetadata = model.metadata + ("version" -> NodeValue.String("1.0"))
  model.copy(metadata = newMetadata)
}

modified.foreach { model =>
  println(model.prettyPrint)
}
```

## Running the Examples

All code from this guide is available as runnable examples in the `smithy-examples` module. Examples demonstrate different aspects of the Smithy library.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

### Step 1: Basic Parsing and Querying

Parse Smithy IDL text, find shapes by name, and access their structure and metadata:

```bash
cd smithy-examples && sbt "runMain smithyexample.BasicParsingAndQuerying"
```

### Step 2: Building Models Programmatically

Construct Smithy models in code by creating shapes, adding traits, and assembling them into a complete model:

```bash
cd smithy-examples && sbt "runMain smithyexample.BuildingModelsAndTraits"
```

### Step 3: Validation and Analysis

Analyze Smithy models for completeness, find deprecated shapes, check for documentation, and validate API contracts:

```bash
cd smithy-examples && sbt "runMain smithyexample.ValidationAndAnalysis"
```

### Step 4: Complete Example — Book Store API

A comprehensive end-to-end workflow showing a complete book store API model with parsing, entity analysis, error handling, code generation, and statistics:

```bash
cd smithy-examples && sbt "runMain smithyexample.BookStoreAPI"
```

**3. Or compile all examples at once:**

```bash
cd smithy-examples && sbt "compile"
```

---

## DbCodecDeriver

`DbCodecDeriver` is a schema-driven derivation engine that converts a `Schema[A]` into a `DbCodec[A]`. 

The companion object `DbCodecDeriver` is itself an instance of the class constructed with the default `SqlNameMapper.SnakeCase`. `DbCodec.derived`, `DbCodec.builder`, `DbCodec.derivedWith`, and `Table.derived` all delegate to this deriver for the row codec; application code rarely constructs or calls it directly.

Key properties:
- **Schema-driven** — derivation is entirely data-driven from the `Schema`'s `Reflect` tree; no separate macro or annotation processor is involved beyond `Schema.derived`.
- **Configurable naming** — the `columnNameMapper: SqlNameMapper` constructor parameter controls how Scala field names become SQL column names; the default is `SnakeCase`.
- **Annotation-aware** — `@Modifier.transient` fields are skipped; `@Modifier.rename` overrides the mapper for individual fields; `@Modifier.config("sql.inline","true")` flattens a nested record into the parent's column list.
- **JSON fallback** — non-inline nested records, sequences, maps, and dynamic values are stored as JSON in a single column backed by `DbCodec.jsonb`. The codec binds the JSON as a string (`DbValue.DbString`), which both dialects map to `TEXT`; no dialect emits `JSONB`, so a `JSONB` column on PostgreSQL must be declared by hand. In 0.0.56, `Table.derived` and `Repo.derived` don't follow this fallback, a known bug: they split a non-inline nested record into one column per field and throw for a collection field, so they work only for records whose nested record fields carry `sql.inline` (see the caution under [`Table#createTable`](./table.md#createtable--generate-a-create-table-statement)).

The structural shape of `DbCodecDeriver` is:

```scala
class DbCodecDeriver(columnNameMapper: SqlNameMapper = SqlNameMapper.SnakeCase)
    extends Deriver[DbCodec]

object DbCodecDeriver extends DbCodecDeriver(SqlNameMapper.SnakeCase) {
  def withColumnNameMapper(mapper: SqlNameMapper): DbCodecDeriver
}
```

## Usage

The following example shows the three most common ways to reach `DbCodecDeriver`: through the `derives` clause on a case class, through `DbCodec.derived` (equivalent), and through `DbCodecDeriver.withColumnNameMapper` when a non-default naming strategy is needed.

```scala

// 1. Derives clause — most concise
case class User(userId: Int, fullName: String) derives DbCodec
DbCodec[User].columns
// res0: IndexedSeq[String] = Vector("user_id", "full_name")

// 2. DbCodec.derived — equivalent, explicit
case class Event(eventId: Long, eventType: String)
object Event { implicit val schema: Schema[Event] = Schema.derived }
```

```scala
val eventCodec = DbCodec.derived[Event]
```

```scala
eventCodec.columns
// res1: IndexedSeq[String] = Vector("event_id", "event_type")

// 3. withColumnNameMapper — use Identity when your DB already uses camelCase
case class Widget(widgetId: Int, widgetName: String)
object Widget { implicit val schema: Schema[Widget] = Schema.derived }
```

```scala
val identityDeriver = DbCodecDeriver.withColumnNameMapper(SqlNameMapper.Identity)

val widgetCodec     = Widget.schema.deriving(identityDeriver).derive
```

```scala
widgetCodec.columns
// res2: IndexedSeq[String] = Vector("widgetId", "widgetName")
```

## See Also

For the naming strategy that `DbCodecDeriver` applies, see [SqlNameMapper](./sql-name-mapper.md). For the `DbCodec` type that derivation produces, see [DbCodec](./db-codec.md). For `Table.derived` and DDL generation, see [Table](./table.md).

---

## DbCodec

`DbCodec[A]` is a bidirectional codec between a Scala value of type `A` and one or more database columns. Every read-side operation — fetching rows from a result set — and every write-side operation — binding parameters to a prepared statement — flows through a `DbCodec`. It is the foundational type in the `sql` module: [`Frag`](./frag.md) uses it to decode query results, [`Table`](./table.md) carries it as column metadata, and [`Repo`](./repo.md) relies on it to map entity rows to and from the database.

Key properties:
- **Bidirectional** — the same type handles both encoding (write) and decoding (read), keeping the two directions in sync.
- **Multi-column** — a single codec spans any number of database columns; a case class codec produces one column per field.
- **Label-based and positional** — `readValue` supports order-independent decoding by column label or fast 1-based positional access per the JDBC convention.
- **Schema-driven** — `DbCodec.derived` and `DbCodecDeriver` produce codecs automatically from a `Schema[A]` at compile time, with no runtime reflection.
- **Explicit about `NULL`** — `Option[A]` and `Maybe[A]` codecs read SQL `NULL` as `None` and `Maybe.absent`. Non-optional types do not all reject it: `BigDecimal` and a derived `Char` field throw `IllegalStateException`, but in 0.0.56 a `String` or `Instant` reads `NULL` as `null` and the numeric types and `Boolean` read it as `0` and `false`, a known bug (see [Reading SQL `NULL`](#reading-sql-null)). Map every nullable column to `Option`, except a decimal one: in 0.0.56, `Option[BigDecimal]` also throws on `NULL`.

## Core API

```scala

trait DbCodec[A] {
  // Column inspection
  def columns: IndexedSeq[String]
  def columnCount: Int

  // Read operations
  def readValue(reader: DbResultReader, startIndex: Int): A
  def readValue(reader: DbResultReader, columnLabels: IndexedSeq[String]): A

  // Write operations
  def writeValue(writer: DbParamWriter, startIndex: Int, value: A): Unit
  def toDbValues(value: A): IndexedSeq[DbValue]

  // Transformation
  def transform[B](read: A => B)(write: B => A): DbCodec[B]
}

object DbCodec {
  // Automatic derivation from Schema[A]
  inline given derived[A]: DbCodec[A]
  inline given derivedOpaque[A]: DbCodec[A]

  // Customize derivation
  inline def builder[A]: DerivationBuilder[DbCodec, A]
  inline def derivedWith[A](
    configure: DerivationBuilder[DbCodec, A] => DerivationBuilder[DbCodec, A]
  ): DbCodec[A]

  // Retrieval
  def apply[A](implicit codec: DbCodec[A]): DbCodec[A]

  // Built-in instances
  // given instances for: Int, Long, String, Boolean, Double, Float, Short, Byte,
  //                      BigDecimal, Instant, UUID, ...
}
```

Codecs for JSON/JSONB columns, type conversions, and specialized encoding strategies are also available through additional companion object methods and instances.

## Usage

The following example shows the core lifecycle of a `DbCodec`: deriving one automatically, inspecting its column metadata, encoding a value, handling nullable columns, and adapting the codec to a newtype:

```scala

// Derive a codec automatically — field names map to snake_case columns by default
case class User(id: Int, name: String, email: Option[String]) derives DbCodec
```

```scala
val codec = DbCodec[User]
```

```scala
codec.columns
// res1: IndexedSeq[String] = Vector("id", "name", "email")
codec.columnCount
// res2: Int = 3

// Encode a value for use as SQL parameters
val params = codec.toDbValues(User(1, "Alice", Some("alice@example.com")))
// params: IndexedSeq[DbValue] = Vector(
//   DbInt(1),
//   DbString("Alice"),
//   DbString("alice@example.com")
// )

// None encodes as SQL NULL
val nullParams = codec.toDbValues(User(2, "Bob", None))
// nullParams: IndexedSeq[DbValue] = Vector(DbInt(2), DbString("Bob"), DbNull)

// Adapt any codec to a newtype with transform — no full Schema needed
case class UserId(value: Int)
```

```scala
val userIdCodec: DbCodec[UserId] = DbCodec[Int].transform(UserId(_))(_.value)
```

```scala
userIdCodec.columns
// res3: IndexedSeq[String] = Vector("value")
userIdCodec.toDbValues(UserId(42))
// res4: IndexedSeq[DbValue] = Vector(DbInt(42))
```

## Construction / Creating Instances

We can obtain a `DbCodec[A]` in several ways: automatic schema derivation (the most common path), structured derivation with field-level overrides, JSONB wrapping for complex types, opaque-type support, and manual composition.

### `DbCodec.derived` — Automatic schema-driven derivation

`DbCodec.derived` is a Scala 3 `inline given` that produces a `DbCodec[A]` by internally deriving `Schema[A]` and running it through `DbCodecDeriver`. Because it derives `Schema` itself, no explicit `Schema` needs to be in scope. It also enables the Scala 3 `derives` clause.

```scala
object DbCodec {
  inline given derived[A]: DbCodec[A]
}
```

The two most common spellings are the `derives` clause on the case class and explicit summoning:

```scala

// Option 1: derives clause
case class Product(sku: String, price: BigDecimal, inStock: Boolean) derives DbCodec

// Option 2: explicit given
case class Category(id: Int, name: String)
given DbCodec[Category] = DbCodec.derived

// Columns follow the default SqlNameMapper (SnakeCase):
// Product → "sku", "price", "in_stock"
// Category → "id", "name"
```

`DbCodec.derived` delegates to `DbCodecDeriver`, which handles primitives, case classes, enums, sealed traits, `Option`/`Maybe` fields, and JSONB-encoded complex fields. Enum and sealed-trait variants serialize to their name as a `String` column unless annotated with `@Modifier.rename`.

:::tip
For case classes with fields that need custom codecs or name overrides, prefer `DbCodec.derivedWith` so you can supply those overrides through the `DerivationBuilder` API.
:::

### `DbCodec.derivedWith` — Derivation with field-level overrides

`DbCodec.derivedWith` derives a `DbCodec[A]` and applies caller-supplied overrides before finalizing the codec. This is the right choice when a specific field needs a custom codec — for example, a JSON-encoded value type or a field stored in a non-default format.

```scala
object DbCodec {
  inline def derivedWith[A](
    configure: DerivationBuilder[DbCodec, A] => DerivationBuilder[DbCodec, A]
  ): DbCodec[A]
}
```

The configure function receives a `DerivationBuilder[DbCodec, A]` and returns a modified one. We call `DerivationBuilder#instance` to attach a custom `DbCodec` for a specific field, identified by the enclosing type's `TypeId` and the field name:

```scala

case class Tags(values: List[String])
object Tags { implicit val schema: Schema[Tags] = Schema.derived }

case class Product(id: Int, tags: Tags)
object Product { implicit val schema: Schema[Product] = Schema.derived }
```

```scala
// tags is stored as a JSON string in the "tags" column
val tagsCodec: DbCodec[Tags] =
  DbCodec[String].transform(json => Tags(json.split(",").toList))(_.values.mkString(","))

val productCodec: DbCodec[Product] =
  DbCodec.derivedWith[Product](
    _.instance(TypeId.of[Product], "tags", tagsCodec)
  )
```

```scala
productCodec.columns
// res7: IndexedSeq[String] = Vector("id", "tags")
productCodec.columnCount
// res8: Int = 2
```

### `DbCodec.jsonb` — JSONB column codec

`DbCodec.jsonb` creates a `DbCodec[A]` that stores and retrieves a value of type `A` as a JSON string in a single database column. Two overloads are available: one using an implicit `JsonCodec[A]` for the encode/decode pair, and one accepting explicit functions.

```scala
object DbCodec {
  def jsonb[A](using jsonCodec: JsonCodec[A]): DbCodec[A]
  def jsonb[A](encode: A => String, decode: String => A): DbCodec[A]
}
```

The first overload requires a `JsonCodec[A]` (`zio.blocks.schema.json.JsonCodec`) in implicit scope:

```scala

case class Address(street: String, city: String)
object Address {
  implicit val schema: Schema[Address]         = Schema.derived
  implicit val jsonCodec: JsonCodec[Address] = schema.deriving(JsonCodecDeriver).derive
}
```

```scala
// Address is stored as a JSON string in a single TEXT/JSONB column
val codec: DbCodec[Address] = DbCodec.jsonb[Address]
```

```scala
codec.columns
// res10: IndexedSeq[String] = Vector("value")
codec.toDbValues(Address("Main St", "NYC"))
// res11: IndexedSeq[DbValue] = Vector(
//   DbString("{\"street\":\"Main St\",\"city\":\"NYC\"}")
// )
```

Use the two-argument overload when you supply custom encode/decode logic instead of relying on `JsonCodec`:

```scala
case class Point(x: Double, y: Double)
```

```scala
// Custom JSON encoding using a hand-rolled format
val pointCodec: DbCodec[Point] = DbCodec.jsonb[Point](
  p => s"${p.x},${p.y}",
  s => { val parts = s.split(","); Point(parts(0).toDouble, parts(1).toDouble) }
)
```

```scala
pointCodec.toDbValues(Point(1.0, 2.0))
// res12: IndexedSeq[DbValue] = Vector(DbString("1.0,2.0"))
```

### `DbCodec.jsonbOption` — Nullable JSONB column codec

`DbCodec.jsonbOption` creates a `DbCodec[Option[A]]` that stores `Some(a)` as a JSON string and `None` as SQL `NULL`. Like `jsonb`, it has an implicit `JsonCodec[A]` overload and a two-argument overload:

```scala
object DbCodec {
  def jsonbOption[A](using jsonCodec: JsonCodec[A]): DbCodec[Option[A]]
  def jsonbOption[A](encode: A => String, decode: String => A): DbCodec[Option[A]]
}
```

The codec delegates to `DbCodec[Option[String]]` and applies the JSON encode/decode on the inner `String`, so `NULL` detection uses the underlying `Option[String]` codec's standard null handling:

```scala
// Assume JsonCodec[Address] is in scope from the previous example
val nullableCodec: DbCodec[Option[Address]] = DbCodec.jsonbOption[Address]
```

```scala
nullableCodec.toDbValues(Some(Address("Elm St", "LA")))
// res13: IndexedSeq[DbValue] = Vector(
//   DbString("{\"street\":\"Elm St\",\"city\":\"LA\"}")
// )

nullableCodec.toDbValues(None)
// res14: IndexedSeq[DbValue] = Vector(DbNull)
```

### `DbCodec.derivedOpaque` — Opaque type derivation

`DbCodec.derivedOpaque` is a lower-priority `inline given` that produces a `DbCodec[A]` for Scala 3 opaque types by reusing the codec of the underlying type. Declare it by name: `given DbCodec[PartNumber] = DbCodec.derivedOpaque[PartNumber]`. Summoning `DbCodec[PartNumber]` without that given does not select it, because the compiler picks `DbCodec.derived`, which reads the underlying value directly and never calls the companion's `apply`; this is a known bug in 0.0.56.

```scala
object DbCodec {
  inline given derivedOpaque[A]: DbCodec[A]
}
```

For the decode direction the opaque type's companion `apply` is called. For the encode direction, if the opaque type is declared as a subtype of its underlying type (`opaque type T <: U = U`), the value is used directly; otherwise the companion must expose an `unwrap` method. With the given declared, a read that the companion's `apply` rejects fails. The opaque type lives in its own object, so the alias stays opaque where the given is declared:

```scala

object Ids {
  opaque type PartNumber <: String = String
  object PartNumber {
    def apply(value: String): PartNumber = {
      require(value.startsWith("PN-"), "not a product id: " + value)
      value
    }
  }
}

given DbCodec[PartNumber] = DbCodec.derivedOpaque[PartNumber]

val transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)
// transactor: JdbcTransactor = zio.blocks.sql.JdbcTransactor@211131fa
```

```scala
transactor.connect(sql"SELECT 'PN-7' AS value".query[PartNumber])
// res16: List[PartNumber] = List("PN-7")

Try(transactor.connect(sql"SELECT 'mug' AS value".query[PartNumber])).failed.map(_.getMessage)
// res17: Try[String] = Success("requirement failed: not a product id: mug")
```

A `PartNumber` field of a derived row type (`derives DbCodec`, `Table.derived`, `Repo.derived`) still skips `apply` in 0.0.56, with or without the given.

:::caution
`DbCodec.derivedOpaque` is a Scala 3-only macro. The `sql` module requires Scala 3.
:::

### `DbCodec.dbCodecFromAs` — Codec derivation via `As` conversion

`DbCodec.dbCodecFromAs` is a `given` that derives `DbCodec[B]` from `DbCodec[A]` and an `As[A, B]` conversion. Declare it by name for the type, for example `given DbCodec[Coupon] = DbCodec.dbCodecFromAs[String, Coupon]`. It is not selected automatically for any type that `DbCodec.derived` can derive, which includes every opaque type: summoning `DbCodec[Coupon]` without the given resolves to `derived` and ignores the `As`, a known bug in 0.0.56:

```scala
object DbCodec {
  given dbCodecFromAs[A, B](using conv: As[A, B], base: DbCodec[A]): DbCodec[B]
}
```

`As[A, B]` (from `zio.blocks.schema`) represents a validated conversion from `A` to `B` and from `B` back to `A`. The derived codec applies `As#into` on decode and `As#from` on encode; if either conversion returns a `Left`, an `IllegalStateException` is thrown at runtime:

```scala

// Suppose As[String, EmailAddress] is defined and EmailAddress wraps String;
// declare the codec by name, because it is not selected automatically:
// given DbCodec[EmailAddress] = DbCodec.dbCodecFromAs[String, EmailAddress]
```

For types without an `As` instance, use `DbCodec[A].transform` instead.

### `DbCodec.apply` — Summoning an instance

`DbCodec.apply` summons an implicitly available `DbCodec[A]` from the current scope. It is the standard way to access a codec without writing `implicitly` or `summon`:

```scala
object DbCodec {
  def apply[A](implicit codec: DbCodec[A]): DbCodec[A]
}
```

We use `DbCodec.apply` whenever we need a codec value without knowing its derivation path:

```scala

case class Order(id: Long, status: String) derives DbCodec
```

```scala
// Summon the derived codec
val codec: DbCodec[Order] = DbCodec[Order]
```

```scala
codec.columns
// res20: IndexedSeq[String] = Vector("id", "status")
```

### `DbCodec.builder` — Derivation builder

`DbCodec.builder[A]` returns a `DerivationBuilder[DbCodec, A]` pre-seeded with the derived schema for `A`. Use it when you need to attach multiple field-level overrides before calling `.derive` to finalize the codec, giving you full control over the build process:

```scala
object DbCodec {
  inline def builder[A]: DerivationBuilder[DbCodec, A]
}
```

`DerivationBuilder` exposes `instance` to attach custom codecs for individual fields and `derive` to produce the final codec. `DbCodec.derivedWith` is a one-liner wrapper around `builder`:

```scala

case class Order(id: Long, tags: List[String], metadata: Map[String, String])

// Build manually — equivalent to derivedWith but explicit
val codec: DbCodec[Order] =
  DbCodec
    .builder[Order]
    .instance(TypeId.of[Order], "tags", DbCodec[String].transform(_.split(",").toList)(_.mkString(",")))
    .derive
```

## Predefined Instances

`DbCodec` provides `given` instances for all primitive and common JVM types. Each occupies a single column named `"value"`:

| Scala Type             | Given Name         | `DbValue` variant            | Notes                                              |
|------------------------|--------------------|------------------------------|----------------------------------------------------|
| `Int`                  | `intCodec`         | `DbValue.DbInt`              |                                                    |
| `Long`                 | `longCodec`        | `DbValue.DbLong`             |                                                    |
| `String`               | `stringCodec`      | `DbValue.DbString`           |                                                    |
| `Boolean`              | `booleanCodec`     | `DbValue.DbBoolean`          |                                                    |
| `Double`               | `doubleCodec`      | `DbValue.DbDouble`           |                                                    |
| `Float`                | `floatCodec`       | `DbValue.DbFloat`            |                                                    |
| `Short`                | `shortCodec`       | `DbValue.DbShort`            |                                                    |
| `Byte`                 | `byteCodec`        | `DbValue.DbByte`             |                                                    |
| `BigDecimal`           | `bigDecimalCodec`  | `DbValue.DbBigDecimal`       | Throws on SQL `NULL`; in 0.0.56 `Option[BigDecimal]` throws too (see [Reading SQL `NULL`](#reading-sql-null)). |
| `java.time.Instant`    | `instantCodec`     | `DbValue.DbInstant`          |                                                    |
| `Option[A]`            | `optionCodec`      | inner or `DbValue.DbNull`    | Requires a `DbCodec[A]`; single-column inner only. |
| `Maybe[A]`             | `maybeCodec`       | inner or `DbValue.DbNull`    | Requires a `DbCodec[A]`; single-column inner only. |

Of these, only `BigDecimal` throws on SQL `NULL`; [Reading SQL `NULL`](#reading-sql-null) shows what the others return. All primitive codecs set their single column name to `"value"`. When a primitive codec is used as part of a record derivation, `DbCodecDeriver` replaces the column name with the field name (after applying the `SqlNameMapper`).

## Core Operations

The five abstract and one final method on `DbCodec` divide into four operational groups: column metadata inspection, decoding from a result set, encoding to prepared-statement parameters, and transformation.

### Column Metadata

The column metadata methods expose the names and count of columns a codec spans. They are used by `Table`, `Repo`, and `Frag` to build SQL `SELECT`, `INSERT`, and `UPDATE` clauses without any per-call string assembly.

#### `columns` — Ordered column names

`DbCodec#columns` returns the ordered `IndexedSeq[String]` of database column names for this codec. The sequence matches the order in which `readValue` and `writeValue` consume and produce values.

```scala
trait DbCodec[A] {
  def columns: IndexedSeq[String]
}
```

For a case class codec produced by `DbCodec.derived`, each field maps to one column name after the `SqlNameMapper` (default: `SnakeCase`). Annotating a field with `@Modifier.rename("custom_name")` overrides the mapped name:

```scala

case class BlogPost(
  @Modifier.rename("post_id") id: Int,
  authorName: String
) derives DbCodec

DbCodec[BlogPost].columns
// res23: IndexedSeq[String] = Vector("post_id", "author_name")
```

#### `columnCount` — Number of columns

`DbCodec#columnCount` returns the number of columns this codec spans. It is derived from `columns.size` and provided as a concrete method:

```scala
trait DbCodec[A] {
  def columnCount: Int = columns.size
}
```

We use `columnCount` to validate multi-column usage and to calculate offsets when composing codecs. For all primitive codecs, `columnCount` is `1`. For a case class, it equals the number of non-transient fields (fields annotated with `@Modifier.transient()` are excluded):

```scala

case class Event(name: String, @Modifier.transient() internalFlag: Boolean = false) derives DbCodec

DbCodec[Event].columnCount // "internalFlag" is excluded
// res25: Int = 1
```

### Reading / Decoding

The two `readValue` overloads decode a Scala value from a `DbResultReader`, which abstracts over a JDBC `ResultSet`. Query execution in `Frag` prefers the label-based overload so that result column order can differ from codec column order.

#### `readValue` — Positional read

`DbCodec#readValue(reader, startIndex)` reads a value of type `A` from the result reader starting at the given 1-based column index. For a multi-column codec, it reads `columnCount` consecutive columns beginning at `startIndex`. Internally this overload delegates to the label-based overload by calling `DbResultReader#columnLabel` for each offset.

```scala
trait DbCodec[A] {
  def readValue(reader: DbResultReader, startIndex: Int): A
}
```

The default implementation converts positional access to label-based access automatically, so implementing only the label-based overload is sufficient when writing a custom `DbCodec`:

```scala

// For illustration: a custom single-column String codec
val uppercaseCodec: DbCodec[String] = new DbCodec[String] {
  val columns: IndexedSeq[String] = IndexedSeq("value")
  def readValue(reader: DbResultReader, columnLabels: IndexedSeq[String]): String =
    reader.getString(columnLabels.head).toUpperCase
  def writeValue(writer: DbParamWriter, startIndex: Int, value: String): Unit =
    writer.setString(startIndex, value)
  def toDbValues(value: String): IndexedSeq[DbValue] =
    IndexedSeq(DbValue.DbString(value))
}

// positional overload works automatically
// uppercaseCodec.readValue(reader, 1) → delegates to label-based via columnLabel(1)
```

:::caution
`startIndex` is 1-based per the JDBC convention. Passing `0` will cause an out-of-bounds error in the underlying `ResultSet`.
:::

#### `readValue` — Label-based read

`DbCodec#readValue(reader, columnLabels)` reads a value of type `A` by looking up each column by label, allowing the result set's column order to differ from the codec's column order. `Frag#query` always calls this overload, passing the labels derived from the query's `SELECT` list.

```scala
trait DbCodec[A] {
  def readValue(reader: DbResultReader, columnLabels: IndexedSeq[String]): A
}
```

The caller must supply exactly `columnCount` labels in the logical order matching the codec's `columns` sequence. In practice, `Frag` constructs this sequence automatically from the query result metadata:

```scala

case class User(id: Int, name: String) derives DbCodec

val codec = DbCodec[User]

// Calling with explicit labels — useful for custom result processing
// codec.readValue(reader, IndexedSeq("id", "name")) → User(...)
```

#### Reading SQL `NULL`

A column that may hold `NULL` belongs in an `Option` (or `Maybe`) field. The `Option` codec reads the inner value, then checks `DbResultReader#wasNull` and returns `None` for a `NULL`.

A non-optional field does not reliably reject a `NULL`. Only the `BigDecimal` codec and the `Char` codec that `DbCodecDeriver` uses throw `IllegalStateException`. In 0.0.56, the other primitive codecs return whatever the JDBC getter returns, a known bug: a `String` or `java.time.Instant` field holds `null`, and an `Int`, `Long`, `Double` or other numeric field holds `0` (`false` for a `Boolean`). `Frag#query`, `Frag#queryStream` and `Repo` all decode through these codecs, so a nullable column mapped to a non-`Option` field passes silently.

The two codecs that do throw have the opposite problem. The JDBC reader only learns that a column is `NULL` after the inner codec has read it, and the `BigDecimal` and `Char` codecs throw during that read, so in 0.0.56 `Option[BigDecimal]`, `Option[Char]` and their `Maybe` forms throw on `NULL` too, another known bug. Until it is fixed, read a nullable decimal column as `Option[String]` and convert it with `BigDecimal(_)`.

To see all of this, we read a table with one row of `NULL`s and one complete row into records with plain fields, with `Option` fields, and with a `BigDecimal` price in each form:

```scala

case class Sale(id: Long, title: String, quantity: Int) derives DbCodec
case class NullableSale(id: Long, title: Option[String], quantity: Option[Int]) derives DbCodec
case class Price(id: Long, price: BigDecimal) derives DbCodec
case class NullablePrice(id: Long, price: Option[BigDecimal]) derives DbCodec
case class PriceText(id: Long, price: Option[String]) derives DbCodec

def errorOf[A](read: => A): Either[String, A] = Try(read).toEither.left.map(_.getMessage)

val (plain, nullable, price, nullablePrice, priceFromText) =
  JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite).transact {
    sql"CREATE TABLE sale (id INTEGER NOT NULL, title TEXT, quantity INTEGER, price NUMERIC)".update
    sql"INSERT INTO sale (id) VALUES (1)".update
    sql"INSERT INTO sale VALUES (2, 'Zionomicon', 3, 49.99)".update

    (
      sql"SELECT id, title, quantity FROM sale ORDER BY id".query[Sale],
      sql"SELECT id, title, quantity FROM sale ORDER BY id".query[NullableSale],
      errorOf(sql"SELECT id, price FROM sale ORDER BY id".query[Price]),
      errorOf(sql"SELECT id, price FROM sale ORDER BY id".query[NullablePrice]),
      sql"SELECT id, price FROM sale ORDER BY id".query[PriceText].map(_.price.map(BigDecimal(_)))
    )
  }
// plain: List[Sale] = List(
//   Sale(id = 1L, title = null, quantity = 0),
//   Sale(id = 2L, title = "Zionomicon", quantity = 3)
// )
// nullable: List[NullableSale] = List(
//   NullableSale(id = 1L, title = None, quantity = None),
//   NullableSale(id = 2L, title = Some("Zionomicon"), quantity = Some(3))
// )
// price: Either[String, List[Price]] = Left(
//   "Encountered SQL NULL while decoding non-optional BigDecimal. Use Option[BigDecimal] or Maybe[BigDecimal] for nullable columns."
// )
// nullablePrice: Either[String, List[NullablePrice]] = Left(
//   "Encountered SQL NULL while decoding non-optional BigDecimal. Use Option[BigDecimal] or Maybe[BigDecimal] for nullable columns."
// )
// priceFromText: List[Option[BigDecimal]] = List(None, Some(49.99))
```

The plain record reads the first row as `Sale(1, null, 0)`, which looks like a sale of zero copies with no title, while the `Option` record reports both columns as missing. Both `BigDecimal` forms fail on the `NULL` price, and reading the price as text gives `None` for the first row and the decimal for the second.

### Writing / Encoding

The two encoding methods convert a Scala value into database parameters: `writeValue` binds values directly to a `DbParamWriter` (a prepared statement), while `toDbValues` converts them to the typed `DbValue` ADT for inspection, testing, and logging.

#### `writeValue` — Bind to a prepared statement

`DbCodec#writeValue` writes a value of type `A` to a `DbParamWriter` starting at the given 1-based parameter index. For a multi-column codec, it writes exactly `columnCount` consecutive parameters beginning at `startIndex`.

```scala
trait DbCodec[A] {
  def writeValue(writer: DbParamWriter, startIndex: Int, value: A): Unit
}
```

`Frag` and `Repo` call this method to bind parameters when executing `INSERT` and `UPDATE` statements. For `None` / `Maybe.absent`, the codec calls `DbParamWriter#setNull` with `java.sql.Types.NULL`:

```scala

case class Point(x: Double, y: Double) derives DbCodec

val codec = DbCodec[Point]
// codec.writeValue(writer, 1, Point(3.0, 4.0))
// → writer.setDouble(1, 3.0); writer.setDouble(2, 4.0)
```

:::caution
`startIndex` is 1-based. The codec writes exactly `columnCount` parameters, so if you compose two codecs at offsets `i` and `i + codec.columnCount`, the second start index must be adjusted accordingly.
:::

#### `toDbValues` — Convert to `DbValue` representation

`DbCodec#toDbValues` converts a value of type `A` into an `IndexedSeq[DbValue]`, one element per column. The result is parallel to `columns` — `toDbValues(v)(i)` corresponds to `columns(i)`.

```scala
trait DbCodec[A] {
  def toDbValues(value: A): IndexedSeq[DbValue]
}
```

`toDbValues` is used by `Frag` and `Repo` to inspect or log parameters before binding, and in tests to assert encoding behavior without a real database connection:

```scala

case class Item(id: Int, name: String, price: Option[BigDecimal]) derives DbCodec
```

```scala
val codec = DbCodec[Item]
```

```scala
codec.toDbValues(Item(1, "Widget", Some(BigDecimal("9.99"))))
// res31: IndexedSeq[DbValue] = Vector(
//   DbInt(1),
//   DbString("Widget"),
//   DbBigDecimal(9.99)
// )

codec.toDbValues(Item(2, "Gadget", None))
// res32: IndexedSeq[DbValue] = Vector(DbInt(2), DbString("Gadget"), DbNull)
```

### Transformations

#### `transform` — Map a codec to a new type

`DbCodec#transform` returns a new `DbCodec[B]` by mapping the read direction with `read: A => B` and the write direction with `write: B => A`. The resulting codec shares the same `columns` as the original and is the lightest way to create a codec for a newtype or value wrapper without defining a full `Schema`:

```scala
trait DbCodec[A] {
  final def transform[B](read: A => B)(write: B => A): DbCodec[B]
}
```

Both `read` and `write` must be total functions; any exception they throw propagates to the caller. The transformed codec delegates all column metadata and read/write operations to the inner codec after applying the conversions:

```scala

case class ProductId(value: String)
```

```scala
// Adapt the String codec to ProductId without a Schema
val productIdCodec: DbCodec[ProductId] =
  DbCodec[String].transform(ProductId(_))(_.value)
```

```scala
productIdCodec.columns
// res34: IndexedSeq[String] = Vector("value")
productIdCodec.toDbValues(ProductId("abc-1"))
// res35: IndexedSeq[DbValue] = Vector(DbString("abc-1"))
```

`DbCodec#transform` is also the engine behind `DbCodec.jsonb`, `DbCodec.jsonbOption`, and `DbCodec.dbCodecFromAs` — each of those constructors builds on top of an existing primitive or composite codec and applies `transform` to attach custom encode/decode logic.

## Supporting Types

The two interfaces that `DbCodec` depends on for its read and write operations are `DbResultReader` and `DbParamWriter`. Both abstract over the JDBC layer so the `sql` module's `shared` source compiles on Scala.js as well as the JVM, and so custom backends can substitute their own implementations without touching codec logic.

```
DbCodec[A]
   │  reads via         writes via
   ▼                    ▼
DbResultReader       DbParamWriter
   │                    │
JdbcResultSetReader  JdbcParamWriter
   │                    │
java.sql.ResultSet   java.sql.PreparedStatement
```

### `DbResultReader` — Result set abstraction

`DbResultReader` is the interface through which `DbCodec#readValue` reads column values from a query result. It supports both label-based access (e.g., `DbResultReader#getString("name")`) and 1-based positional access (e.g., `DbResultReader#getInt(1)`), plus `DbResultReader#wasNull` to detect SQL `NULL` after any `get*` call:

```scala
trait DbResultReader {
  def getInt(index: Int): Int
  def getInt(label: String): Int
  def getString(index: Int): String
  def getString(label: String): String
  def getBoolean(label: String): Boolean
  def getBigDecimal(label: String): java.math.BigDecimal
  def getInstant(label: String): java.time.Instant
  // ... and all other column types
  def columnLabel(index: Int): String
  def hasColumn(label: String): Boolean
  def wasNull: Boolean
}
```

`DbResultReader` is rarely used directly in application code. `Frag#query` wraps the JDBC `ResultSet` in a `JdbcResultSetReader` and passes it to the appropriate `DbCodec#readValue` call automatically.

### `DbParamWriter` — Prepared statement abstraction

`DbParamWriter` is the interface through which `DbCodec#writeValue` binds column values to a prepared statement. It follows the JDBC convention of 1-based parameter indexes and includes `setNull` for writing SQL `NULL`:

```scala
trait DbParamWriter {
  def setInt(index: Int, value: Int): Unit
  def setString(index: Int, value: String): Unit
  def setBoolean(index: Int, value: Boolean): Unit
  def setBigDecimal(index: Int, value: java.math.BigDecimal): Unit
  def setInstant(index: Int, value: java.time.Instant): Unit
  // ... and all other parameter types
  def setNull(index: Int, sqlType: Int): Unit
}
```

Like `DbResultReader`, `DbParamWriter` is not used directly in application code. `Frag#update` and `Repo` CRUD methods create a `JdbcParamWriter` wrapping a `java.sql.PreparedStatement` and pass it to `DbCodec#writeValue` internally.

## Integration

`DbCodec` sits at the center of the `sql` module's layered architecture. The diagram below shows how it connects to its neighbours:

```
Schema[A]
    │
    ▼  (DbCodecDeriver)
DbCodec[A] ◄────────────────────────────────┐
    │                                        │
    ├──► Table[A]                            │
    │         └──► Repo[E, ID]               │
    │                   │                    │
    │              CRUD methods              │
    │                   │                    │
    └──► Frag ──────────┘                    │
              (sql"..." interpolator)        │
                   │                         │
         Transactor#connect/transact         │
                   │                         │
             DbCon / DbTx                    │
            /         \                      │
  DbResultReader   DbParamWriter ────────────┘
  (readValue)      (writeValue)
```

See also the [How to Store and Search Vector Embeddings in PostgreSQL](../../guides/store-and-search-vector-embeddings.md) guide, which builds a `DbCodec` on `String` for a pgvector column and binds it as a parameter with the `sql` interpolator.

---

## DbCon

`DbCon` is the implicit execution context that threads database connection state through every SQL operation in the `sql` module. It is a plain trait with three abstract members — `connection`, `dialect`, and `logger` — and carries no type parameters:

```scala
trait DbCon {
  def connection: DbConnection
  def dialect: SqlDialect
  def logger: SqlLogger
}
```

Application code never constructs a `DbCon` directly. Instead, `Transactor#connect` and `Transactor#transact` each create one internally and supply it as a Scala 3 context parameter (`?=>`) to the block they receive. 

All `Frag` execution methods (`query`, `queryOne`, `queryLimit`, `update`, `updateReturningKeys`) and all `Repo` CRUD methods (`all`, `find`, `insert`, `update`, `delete`, and friends) require an implicit `DbCon` in scope — so any code that runs inside a `Transactor#connect` or `Transactor#transact` block automatically has everything it needs to execute SQL.

Key properties:
- **Implicit context type** — carries the active database session without explicit parameter threading.
- **Three concrete members** — `connection` (the active `DbConnection`), `dialect` (the `SqlDialect`), and `logger` (the `SqlLogger`).
- **Extended by `DbTx`** — `DbTx` is a subtype of `DbCon` that marks transactional scope; any method that accepts `DbCon` also accepts `DbTx`.
- **Lifetime managed by `Transactor`** — the connection is closed when the enclosing `connect` or `transact` block returns, whether it succeeds or throws.

## Usage

The following example shows how `DbCon` is obtained from a `Transactor`, how the context is propagated into helper methods, and how the three members can be accessed when needed:

```scala

case class User(id: Int, name: String)
object User {
  implicit val schema: Schema[User] = Schema.derived
}

val tx: Transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)
// tx: Transactor = zio.blocks.sql.JdbcTransactor@2c1cfa3f

// DbCon is supplied automatically by connect — no explicit argument needed
tx.connect {
  // Frag execution picks up the given DbCon implicitly
  Frag.literal("CREATE TABLE users (id INTEGER NOT NULL, name TEXT NOT NULL)").update
  sql"INSERT INTO users (id, name) VALUES (${1}, ${"Alice"})".update
  val users: List[User] = sql"SELECT id, name FROM users".query[User]
  // Access the context members directly when needed
  val dialectName: String = summon[DbCon].dialect.toString
  (users, dialectName)
}
// res1: Tuple2[List[User], String] = (
//   List(User(id = 1, name = "Alice")),
//   "SQLite"
// )
```

## Entry Points

`DbCon` is never instantiated by application code. A `Transactor` creates and supplies the instance automatically. There are two entry points depending on whether transactional behaviour is needed.

### `Transactor#connect` — Non-transactional connection scope

`Transactor#connect` acquires a JDBC connection from the underlying source, wraps it in a `DbCon`, and calls the provided context function. The connection is closed when the block returns, whether it completes normally or throws. Auto-commit remains at the driver default (enabled for most JDBC drivers), so each statement issued inside the block commits independently.

The signature from `Transactor` is:

```scala
trait Transactor {
  def connect[A](f: DbCon ?=> A): A
}
```

The following example shows a non-transactional read that uses the supplied `DbCon` to execute a query:

```scala

val tx: Transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)

// The `DbCon` binding is supplied by `connect` — no manual construction
val names: List[String] = tx.connect {
  sql"SELECT name FROM users ORDER BY name".query[String]
}
```

:::caution
Do not capture the `DbCon` value or the `DbConnection` it contains and use them outside the `connect` block. The connection is closed when the block returns, and any statement prepared or executed on it afterward will throw.
:::

### `Transactor#transact` — Transactional connection scope

`Transactor#transact` acquires a connection, disables auto-commit, and supplies a `DbTx` context — a subtype of `DbCon` — to the block. On normal return the transaction commits; on any thrown exception it rolls back. The connection is closed after commit or rollback. Because `DbTx extends DbCon`, all `Frag` and `Repo` methods that accept `DbCon` work unchanged inside `transact`.

The signature from `Transactor` is:

```scala
trait Transactor {
  def transact[A](f: DbTx ?=> A): A
}
```

The following example shows a transactional write that inserts two rows atomically — if the second insert throws, the first is rolled back:

```scala

val tx: Transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)

tx.transact {
  // Both inserts commit together; an exception rolls back both
  sql"INSERT INTO users (id, name) VALUES (${1}, ${"Alice"})".update
  sql"INSERT INTO users (id, name) VALUES (${2}, ${"Bob"})".update
}
```

## Core Operations

`DbCon` exposes three read-only members. They are rarely accessed directly in application code — the `Transactor` configures them at construction time and `Frag` / `Repo` consume them implicitly — but they become useful when integrating with lower-level JDBC abstractions, logging frameworks, or custom dialect rendering.

### Context Access

The three context-access members return the `DbConnection`, `SqlDialect`, and `SqlLogger` held by the current `DbCon` instance.

#### `DbCon#connection` — Underlying JDBC-abstraction connection

`DbCon#connection` returns the `DbConnection` wrapping the active JDBC `java.sql.Connection`. `DbConnection` exposes `prepareStatement`, `prepareStatementReturningKeys`, transaction-control methods, and `close`. It is consumed internally by every `Frag` and `Repo` operation that executes a statement. Access it directly only when you need to drop below the `Frag` layer to a raw prepared statement.

```scala
trait DbCon {
  def connection: DbConnection
}
```

The following example shows how to retrieve the `DbConnection` and use it to execute a raw prepared statement — useful for operations that the `Frag` API does not cover directly:

```scala

val tx: Transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)

tx.connect {
  val conn: DbConnection = summon[DbCon].connection
  val ps = conn.prepareStatement("SELECT COUNT(*) FROM users")
  try {
    val rs = ps.executeQuery()
    try { rs.next(); rs.reader.getInt(1) }
    finally rs.close()
  } finally ps.close()
}
```

:::caution
Never call `connection.close()` manually. The `Transactor` closes the connection when the enclosing `connect` or `transact` block finishes, whether it returns normally or throws. Closing the connection early will cause all subsequent statements in the block to fail.
:::

#### `DbCon#dialect` — SQL dialect for fragment rendering

`DbCon#dialect` returns the `SqlDialect` used to render `Frag` values to parameterized SQL strings. Every `Frag` execution method calls `frag.sql(dialect)` internally to produce the driver-specific SQL before binding parameters. The dialect also governs how DDL type names are spelled when generating `CREATE TABLE` statements from `Table#createTable`.

```scala
trait DbCon {
  def dialect: SqlDialect
}
```

The following example shows reading the dialect from a context to render a `Frag` explicitly — for instance, to log the SQL before execution:

```scala

val tx: Transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)
// tx: Transactor = zio.blocks.sql.JdbcTransactor@487d3dfa

tx.connect {
  val frag = sql"SELECT id FROM users WHERE id = ${42}"
  frag.sql(summon[DbCon].dialect)
}
// res6: String = "SELECT id FROM users WHERE id = ?"
```

#### `DbCon#logger` — Query execution logger

`DbCon#logger` returns the `SqlLogger` that the `Transactor` was configured with. After each successful statement `SqlLogger#onSuccess` receives the rendered SQL, the parameter list, the execution duration, and the affected-row count. After a failed statement `SqlLogger#onError` receives the same information plus the thrown exception. The default implementation — `SqlLogger.noop` — discards all events; replace it with a custom `SqlLogger` to integrate with your preferred logging framework.

```scala
trait DbCon {
  def logger: SqlLogger
}
```

The following example shows passing a logging `SqlLogger` to `JdbcTransactor` so that every query executed inside `connect` or `transact` is recorded:

```scala

val loggingLogger: SqlLogger = new SqlLogger {
  def onSuccess(event: SqlLogger.SuccessEvent): Unit =
    println(s"OK [${event.duration.toMillis} ms, ${event.rowCount} rows]: ${event.sql}")
  def onError(event: SqlLogger.ErrorEvent): Unit =
    println(s"ERR [${event.duration.toMillis} ms]: ${event.sql} — ${event.error.getMessage}")
}
// loggingLogger: SqlLogger = repl.MdocSession$MdocApp7$$anon$9@1a601326

val tx: Transactor =
  new JdbcTransactor(
    () => java.sql.DriverManager.getConnection("jdbc:sqlite::memory:"),
    SqlDialect.SQLite,
    loggingLogger
  )
// tx: Transactor = zio.blocks.sql.JdbcTransactor@74f73e1

tx.connect {
  // Every Frag execution notifies loggingLogger automatically
  Frag.literal("SELECT 1").query[Int]
}
// OK [0 ms, 1 rows]: SELECT 1
// res8: List[Int] = List(1)
```

## Transactional Scope — `DbTx`

`DbTx` is the only subtype of `DbCon`. It distinguishes code that runs inside `Transactor#transact` from code that runs inside `Transactor#connect` at the type level, and it adds the savepoint members that nested transactions are built on:

```scala
trait DbTx extends DbCon {
  def savepoint(name: String): Unit
  def release(name: String): Unit
  def rollbackTo(name: String): Unit
  def currentDepth: Int
}
```

[DbTx](./db-tx.md) documents the savepoint members and nested transactions. For `DbCon` code, the important part is the type itself: it makes the transactional / non-transactional boundary visible in method signatures. A method that requires `DbTx` compiles only inside a `Transactor#transact` block, while a method that requires `DbCon` works inside both `Transactor#connect` and `Transactor#transact`. The compiler enforces this asymmetry: `DbTx <: DbCon`, so `DbTx` satisfies a `DbCon` requirement, but a plain `DbCon` does not satisfy a `DbTx` requirement.

The following example illustrates how to write a helper method that is restricted to transactional scope:

```scala

// This helper compiles only inside a `transact` block, never inside `connect`
def insertUser(id: Int, name: String)(using DbTx): Unit = {
  sql"INSERT INTO users (id, name) VALUES ($id, $name)".update
  ()
}

val tx: Transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)

tx.transact {
  insertUser(1, "Alice") // OK — DbTx satisfies the `using DbTx` requirement
}

// tx.connect {
//   insertUser(1, "Alice") // Compile error — DbCon does not satisfy `using DbTx`
// }
```

When a helper only reads data and should work in both contexts, declare it with `using DbCon`:

```scala

// Usable inside both `connect` and `transact` blocks
def findUser(id: Int)(using DbCon): Maybe[String] =
  sql"SELECT name FROM users WHERE id = $id".queryOne[String]
```

---

## DbConnection

`DbConnection` is a `trait` that extends `AutoCloseable` and abstracts over a JDBC `java.sql.Connection`. It exposes the minimum surface needed to prepare statements, control transaction boundaries, and query connection state. 

It is part of a three-tier abstraction:
- `DbConnection` manages the connection itself
- `DbPreparedStatement` executes statements with parameters
- `DbResultSet` reads result rows

## Core API

Here is the core API which `DbConnection` exposes:

```scala
trait DbConnection extends AutoCloseable {
  // Statement preparation
  def prepareStatement(sql: String): DbPreparedStatement
  def prepareStatementReturningKeys(sql: String): DbPreparedStatement

  // Transaction control
  def setAutoCommit(autoCommit: Boolean): Unit
  def getAutoCommit: Boolean
  def commit(): Unit
  def rollback(): Unit

  // Connection state
  def isClosed: Boolean
  def close(): Unit
}
```

## Creating Instances

`DbConnection` has no public constructor. Application code never instantiates it. `Transactor#connect` and `Transactor#transact` create the concrete `JdbcConnection` internally and make it available through `DbCon#connection` inside the block.

For testing, implement the trait directly with a mock or use a lightweight in-memory JDBC driver such as SQLite's `:memory:` database.

:::caution
Never call `con.close()` inside a `connect` or `transact` block. The `Transactor` closes the connection after the block returns; closing it early will cause the rest of the block to fail with a "connection already closed" JDBC error.
:::

## Statement Preparation

### `prepareStatement` — Prepare a SQL statement for execution

`prepareStatement(sql: String): DbPreparedStatement` asks the JDBC driver to compile and cache the SQL string and returns a `DbPreparedStatement` that can have parameters bound to it before execution. The `sql` string must use `?` placeholders for parameters.

```scala

val transactor: Transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)

// Inside a connect block:
transactor.connect {
  val con  = summon[DbCon].connection
  val stmt = con.prepareStatement("INSERT INTO tags (name) VALUES (?)")
  stmt.paramWriter.setString(1, "scala")
  val rowCount = stmt.executeUpdate()
  stmt.close()
  rowCount
}
```

### `prepareStatementReturningKeys` — Prepare an insert that returns generated keys

`prepareStatementReturningKeys(sql: String): DbPreparedStatement` prepares a statement with the JDBC `RETURN_GENERATED_KEYS` hint, enabling `executeUpdateReturningKeys` to return an auto-generated primary key. Use this for `INSERT` statements on tables with a database-generated `SERIAL` or `AUTOINCREMENT` primary key.

```scala

val transactor: Transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)

// Inside a connect block:
transactor.connect {
  val con  = summon[DbCon].connection
  val stmt = con.prepareStatementReturningKeys("INSERT INTO orders (amount) VALUES (?)")
  stmt.paramWriter.setBigDecimal(1, new java.math.BigDecimal("99.99"))
  val keyRs  = stmt.executeUpdateReturningKeys()
  val genKey = if (keyRs.next()) keyRs.reader.getLong(1) else -1L
  keyRs.close()
  stmt.close()
  genKey
}
```

## Transaction Control

`setAutoCommit`, `getAutoCommit`, `commit`, and `rollback` manage the connection's transactional state.

`setAutoCommit(autoCommit: Boolean)` switches the connection between auto-commit and manual-commit mode. `getAutoCommit: Boolean` queries the current mode. `commit()` commits the current transaction; `rollback()` rolls it back. These methods wrap the corresponding `java.sql.Connection` calls.

`Transactor#transact` calls `setAutoCommit(false)` before the block and calls `commit()` on success or `rollback()` on exception — so within a `transact` block, you should not call these methods yourself. Use them directly only in a `connect` block where you need fine-grained control over transaction boundaries.

```scala

val transactor: Transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)

// Manual transaction management inside a connect block (uncommon)
transactor.connect {
  val con = summon[DbCon].connection
  con.setAutoCommit(false)
  try {
    // ... execute statements ...
    con.commit()
  } catch {
    case t: Throwable =>
      con.rollback()
      throw t
  }
}
```

## Connection State

### `isClosed` — Check whether the connection is still open

`isClosed: Boolean` returns `true` if the connection has already been closed. Inside a `connect` or `transact` block the connection is always open; this method is primarily useful in test code that verifies the `Transactor` closes connections correctly.

```scala

val transactor: Transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)
// transactor: Transactor = zio.blocks.sql.JdbcTransactor@123c2eb2

var capturedCon: DbConnection = null
// capturedCon: DbConnection = null

transactor.connect {
  capturedCon = summon[DbCon].connection
  capturedCon.isClosed // still open inside the block
}
// res4: Boolean = false

capturedCon.isClosed // closed after the block returned
// res5: Boolean = true
```

### `close` — Close the connection

`close(): Unit` closes the connection and releases the underlying JDBC resources. Normally the `Transactor` calls this automatically after the `connect` or `transact` block completes. Calling it manually inside a block will cause subsequent operations to fail with a "connection already closed" error.

---

## DbParamWriter

`DbParamWriter` is a trait for binding typed Scala values to the `?` placeholders in a SQL prepared statement. It provides methods to set parameters by their 1-based index (following JDBC convention).

Application code does not use `DbParamWriter` directly — the framework calls it internally when executing queries through `Frag` or `Repo`. It is the write-side counterpart to `DbResultReader`, which reads values from result sets.

## Core API

```scala
trait DbParamWriter {
  // Numeric types
  def setInt(index: Int, value: Int): Unit
  def setLong(index: Int, value: Long): Unit
  def setDouble(index: Int, value: Double): Unit
  def setFloat(index: Int, value: Float): Unit
  def setShort(index: Int, value: Short): Unit
  def setByte(index: Int, value: Byte): Unit
  def setBigDecimal(index: Int, value: java.math.BigDecimal): Unit

  // Text and binary
  def setString(index: Int, value: String): Unit
  def setBytes(index: Int, value: Array[Byte]): Unit

  // Date and time
  def setLocalDate(index: Int, value: java.time.LocalDate): Unit
  def setLocalDateTime(index: Int, value: java.time.LocalDateTime): Unit
  def setLocalTime(index: Int, value: java.time.LocalTime): Unit
  def setInstant(index: Int, value: java.time.Instant): Unit
  def setDuration(index: Int, value: java.time.Duration): Unit

  // Other types
  def setBoolean(index: Int, value: Boolean): Unit
  def setUUID(index: Int, value: java.util.UUID): Unit
  def setArray(index: Int, elementType: String, elements: IndexedSeq[Any]): Unit

  // Null handling
  def setNull(index: Int, sqlType: Int): Unit
}
```

## Usage

You access `DbParamWriter` through `DbPreparedStatement.paramWriter` when using raw prepared statements in a `connect` block:

```scala

val transactor: Transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)

transactor.connect {
  val con  = summon[DbCon].connection
  val stmt = con.prepareStatement("INSERT INTO users (name, age) VALUES (?, ?)")
  
  stmt.paramWriter.setString(1, "Alice")  // First ?
  stmt.paramWriter.setInt(2, 30)          // Second ?
  
  stmt.executeUpdate()
  stmt.close()
}
```

Note that indices are 1-based, not 0-based.

## How It Works

When you use higher-level operations like `Frag.query` or `Repo.insert`, `DbCodec` internally calls `DbParamWriter` methods to bind your Scala values to SQL parameters. You see this trait only when dropping down to raw prepared statements for advanced use cases like stored procedures or batch operations.

See [DbConnection](./db-connection.md) for examples of raw prepared-statement usage, and [DbCodec](./db-codec.md) for how parameter binding integrates with the codec system.

---

## DbParam

`DbParam[A]` is a single-method typeclass that converts a Scala value of type `A` to a `DbValue` for use as a bound SQL parameter. Its sole abstract method is `toDbValue(value: A): DbValue`. The `sql"..."` string interpolator summons a `DbParam[A]` instance at compile time for every interpolated expression and delegates to it during execution to produce the typed `DbValue` that is stored in `Frag#params`. 

The structural shape of `DbParam` is:

```scala
trait DbParam[A] {
  def toDbValue(value: A): DbValue
}

object DbParam {
  def apply[A](implicit p: DbParam[A]): DbParam[A]
  // given instances for:
  //   Int, Long, Double, Float, Boolean, String, Short, Byte,
  //   BigDecimal, Array[Byte], LocalDate, LocalDateTime, LocalTime,
  //   Instant, Duration, UUID, DbValue (identity),
  //   Option[A], Maybe[A], and fromDbCodec[A]
}
```

## Usage

The following example shows the three common entry points for `DbParam`: the `sql"..."` interpolator (which calls `toDbValue` behind the scenes), direct instance summoning for inspection:

```scala

// 1. Implicitly used by the sql"..." interpolator — no explicit call needed
val userId  = 42
// userId: Int = 42
val active  = true
// active: Boolean = true
val query   = sql"SELECT * FROM users WHERE id = $userId AND active = $active"
// query: Frag = Frag(
//   parts = Vector("SELECT * FROM users WHERE id = ", " AND active = ", ""),
//   params = Vector(DbInt(42), DbBoolean(true))
// )
query.params
// res0: IndexedSeq[DbValue] = Vector(DbInt(42), DbBoolean(true))

// 2. Summon an instance explicitly with DbParam.apply to inspect conversion
val instant = Instant.parse("2024-01-15T10:00:00Z")
// instant: Instant = 2024-01-15T10:00:00Z
DbParam[Instant].toDbValue(instant)
// res1: DbValue = DbInstant(2024-01-15T10:00:00Z)

DbParam[Option[Int]].toDbValue(Some(7))
// res2: DbValue = DbInt(7)
DbParam[Option[Int]].toDbValue(None)
// res3: DbValue = DbNull
```

---

## DbResultReader

`DbResultReader` is a trait for reading typed column values from a database result set. It provides methods to access columns by either their 1-based index (following JDBC convention) or by column label.

Application code does not use `DbResultReader` directly — the framework creates it automatically when a `Frag` is executed. It is the read-side counterpart to `DbParamWriter`, which writes values to prepared statements.

## Core API

The core API reads numeric, text, binary, boolean, temporal, and UUID values, plus column metadata and NULL status. `getArray` also exists but defaults to throwing `UnsupportedOperationException` in the shared trait; only backends that override it (the JVM implementation does) actually support SQL array access:

```scala
trait DbResultReader {
  // Numeric types
  def getInt(index: Int): Int
  def getInt(label: String): Int
  def getLong(index: Int): Long
  def getLong(label: String): Long
  def getDouble(index: Int): Double
  def getDouble(label: String): Double
  def getFloat(index: Int): Float
  def getFloat(label: String): Float
  def getShort(index: Int): Short
  def getShort(label: String): Short
  def getByte(index: Int): Byte
  def getByte(label: String): Byte
  def getBigDecimal(index: Int): java.math.BigDecimal
  def getBigDecimal(label: String): java.math.BigDecimal

  // Text and binary
  def getString(index: Int): String
  def getString(label: String): String
  def getBytes(index: Int): Array[Byte]
  def getBytes(label: String): Array[Byte]

  // Boolean
  def getBoolean(index: Int): Boolean
  def getBoolean(label: String): Boolean

  // Date and time
  def getLocalDate(index: Int): java.time.LocalDate
  def getLocalDate(label: String): java.time.LocalDate
  def getLocalDateTime(index: Int): java.time.LocalDateTime
  def getLocalDateTime(label: String): java.time.LocalDateTime
  def getLocalTime(index: Int): java.time.LocalTime
  def getLocalTime(label: String): java.time.LocalTime
  def getInstant(index: Int): java.time.Instant
  def getInstant(label: String): java.time.Instant
  def getDuration(index: Int): java.time.Duration
  def getDuration(label: String): java.time.Duration

  // Other types
  def getUUID(index: Int): java.util.UUID
  def getUUID(label: String): java.util.UUID
  def getArray(index: Int): Array[String]    // default: throws UnsupportedOperationException
  def getArray(label: String): Array[String] // default: throws UnsupportedOperationException

  // Metadata and NULL detection
  def columnLabel(index: Int): String
  def hasColumn(label: String): Boolean
  def wasNull: Boolean
  def isNull(index: Int): Boolean  // default: false
  def isNull(label: String): Boolean // default: false
}
```

## Usage

You access `DbResultReader` through `DbResultSet.reader` after calling `executeQuery` on a prepared statement:

```scala

val transactor: Transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)

transactor.connect {
  val con = summon[DbCon].connection
  val stmt = con.prepareStatement("SELECT id, name, age FROM users")
  val rs = stmt.executeQuery()
  
  if (rs.next()) {
    // Read by column label
    val id: Long    = rs.reader.getLong("id")
    val name: String = rs.reader.getString("name")
    val age: Int    = rs.reader.getInt("age")
  }
  
  rs.close()
  stmt.close()
}
```

## NULL Handling

Numeric and boolean getters return zero-like defaults (`0`, `0L`, `false`) for SQL `NULL`. String, decimal, and temporal getters return `null`. After any `get*` call, check `wasNull` to detect `NULL`:

```scala

val transactor: Transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)

transactor.connect {
  val con = summon[DbCon].connection
  val stmt = con.prepareStatement("SELECT bio FROM users LIMIT 1")
  val rs = stmt.executeQuery()
  
  if (rs.next()) {
    val bio: String = rs.reader.getString("bio")
    val bioOption: Option[String] = if (rs.reader.wasNull) None else Some(bio)
  }
  
  rs.close()
  stmt.close()
}
```

:::caution
Always call `wasNull` immediately after the `get*` call whose NULL status you want to check. Do not call another `get*` method in between, or `wasNull` will reflect the wrong result.
:::

## How It Works

When you use high-level operations like `Frag.query` or `Repo.find`, `DbCodec` internally calls `DbResultReader` methods to decode result rows. You only see this trait directly when dropping down to raw prepared statements for advanced use cases.

Label-based access (preferred) lets you ignore column order:

```scala

val reader: DbResultReader = ???
val name = reader.getString("name")  // Works regardless of SELECT order
```

Index-based access (1-based per JDBC) is faster but requires stable column order:

```scala

val reader: DbResultReader = ???
val name = reader.getString(2)  // Assumes name is the 2nd column
```

---

## DbTx

`DbTx` is the transactional connection context supplied by `Transactor#transact`. It extends `DbCon` (so every `Frag`/`Repo` operation that needs `DbCon` also accepts `DbTx`) and adds savepoint-based nested transaction support. You never construct a `DbTx` directly; the `Transactor` creates one and supplies it as a given context to the block passed to `transact`. The connection is always closed when the outermost block exits, whether it commits, rolls back, or throws.

Key properties:
- **Transactional context** — A `DbTx` value in scope guarantees the underlying JDBC connection has auto-commit disabled.
- **Commit-on-success / rollback-on-failure** — The outermost `Transactor.transact` commits on normal return and rolls back on any uncaught exception (with suppressed rollback failures).
- **Savepoint-based nesting** — Inner blocks reuse the same connection via SQL savepoints (`SAVEPOINT` / `RELEASE SAVEPOINT` / `ROLLBACK TO SAVEPOINT`).
- **`transact(isolation, readOnly)`** — The two-arg `Transactor.transact` overload sets isolation level and `readOnly` before disabling auto-commit; `DbTx` nested blocks inherit those settings on the same connection.

The structural declaration of `DbTx` is:

```scala
trait DbTx extends DbCon {
  def savepoint(name: String): Unit
  def release(name: String): Unit
  def rollbackTo(name: String): Unit
  def currentDepth: Int
  private[sql] def currentDepth_=(depth: Int): Unit

  // inherited from DbCon
  def connection: DbConnection
  def dialect: SqlDialect
  def logger: SqlLogger
}
```

## Usage

The following example opens a transaction via `Transactor#transact`, accesses all three context members, and combines a `Repo` CRUD operation with a hand-written `Frag` query — both of which accept `DbTx` transparently in place of `DbCon`:

```scala

case class User(id: Int, name: String, email: String)
object User {
  implicit val schema: Schema[User] = Schema.derived
}

val repo = Repo.derived[User, Int]("users", "id", _.id)
// repo: Repo[User, Int] = zio.blocks.sql.Repo$DerivedRepo@6c1d5c54
val tx   = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)
// tx: JdbcTransactor = zio.blocks.sql.JdbcTransactor@9fa6512

// On normal return: transaction commits and connection closes.
// On any exception: transaction rolls back, then the exception propagates.
tx.transact {
  // All three context members are accessible via summon[DbTx]
  val conn: DbConnection = summon[DbTx].connection // managed JDBC connection — do not close manually
  val d:    SqlDialect   = summon[DbTx].dialect
  val log:  SqlLogger    = summon[DbTx].logger

  // Repo and Frag operations accept DbTx because DbTx extends DbCon
  repo.table.createTable(summon[DbTx].dialect).update
  repo.insert(User(1, "Alice", "alice@example.com"))
  repo.insert(User(2, "Bob",   "bob@example.com"))

  val all:    List[User] = repo.all
  val custom: List[User] =
    sql"SELECT id, name, email FROM users WHERE name LIKE ${"A%"}".query[User]

  (all, custom)
}
// res1: Tuple2[List[User], List[User]] = (
//   List(
//     User(id = 1, name = "Alice", email = "alice@example.com"),
//     User(id = 2, name = "Bob", email = "bob@example.com")
//   ),
//   List(User(id = 1, name = "Alice", email = "alice@example.com"))
// )
```

## Nested Transactions via Savepoints

Nested transactions reuse the same underlying JDBC connection via SQL savepoints. The `DbTx` given in scope exposes an extension `transact` and the `transactNested` helpers:

```scala

val transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)

// Savepoint-based nesting — same connection, isolated rollback
transactor.transact {
  sql"INSERT INTO t VALUES (1)".update

  // Inner block runs inside SAVEPOINT zib_tx_1
  summon[DbTx].transact {
    sql"INSERT INTO t VALUES (2)".update
  }

  // Equivalent using the `using` helper
  // DbTx.transactNested { sql"INSERT INTO t VALUES (3)".update }
  // transactNested { sql"INSERT INTO t VALUES (4)".update }
}
```

Savepoint names are `zib_tx_1 .. zib_tx_N` where `N` is the nesting depth tracked in `currentDepth`. On success the savepoint is released via `RELEASE SAVEPOINT`; on failure it is rolled back via `ROLLBACK TO SAVEPOINT` and the exception is rethrown (with any rollback failure added as suppressed). Depth is decremented in `finally`, so sibling nested blocks reuse the same name sequence without leaking savepoints. `savepoint`/`release`/`rollbackTo` are also available directly for manual control and validate identifiers via `SqlIdentifier` to prevent injection.

:::caution
Only the outermost `Transactor.transact` issues a real `COMMIT`/`ROLLBACK`. Inner `summon[DbTx].transact` blocks are savepoint-scoped — outer commit still decides the final persistence of all work, including inner blocks that succeeded.
:::

---

## DbValue

`DbValue` is a sealed ADT representing every typed SQL column value the `sql` module handles, one `case class` per Scala type. `Frag` stores its bound parameters as `IndexedSeq[DbValue]`, `DbCodec#toDbValues` produces them, and `SqlDialect#typeName` maps a `DbValue` to its DDL type string — it is the common typed currency all three share, with no JDBC imports or dependencies of its own.

| Variant           | Scala field type          | PostgreSQL DDL     | SQLite DDL |
|-------------------|----------------------------|---------------------|------------|
| `DbNull`          | *(none)*                   | `NULL`               | `NULL`     |
| `DbInt`           | `Int`                       | `INTEGER`             | `INTEGER`  |
| `DbLong`          | `Long`                      | `BIGINT`              | `INTEGER`  |
| `DbDouble`        | `Double`                    | `DOUBLE PRECISION`    | `REAL`     |
| `DbFloat`         | `Float`                     | `REAL`                | `REAL`     |
| `DbBoolean`       | `Boolean`                   | `BOOLEAN`             | `INTEGER`  |
| `DbString`        | `String`                    | `TEXT`                | `TEXT`     |
| `DbBigDecimal`    | `scala.BigDecimal`          | `NUMERIC`             | `TEXT`     |
| `DbBytes`         | `Array[Byte]`               | `BYTEA`               | `BLOB`     |
| `DbShort`         | `Short`                     | `SMALLINT`            | `INTEGER`  |
| `DbByte`          | `Byte`                      | `SMALLINT`            | `INTEGER`  |
| `DbChar`          | `Char`                      | `CHAR(1)`             | `TEXT`     |
| `DbLocalDate`     | `java.time.LocalDate`       | `DATE`                | `TEXT`     |
| `DbLocalDateTime` | `java.time.LocalDateTime`   | `TIMESTAMP`           | `TEXT`     |
| `DbLocalTime`     | `java.time.LocalTime`       | `TIME`                | `TEXT`     |
| `DbInstant`       | `java.time.Instant`         | `TIMESTAMPTZ`         | `TEXT`     |
| `DbDuration`      | `java.time.Duration`        | `INTERVAL`            | `TEXT`     |
| `DbUUID`          | `java.util.UUID`            | `UUID`                | `TEXT`     |
| `DbArray`         | `(String, IndexedSeq[Any])` | `TEXT[]`              | `TEXT`     |

`DbNull` is what `DbParam[Option[A]]` produces for `None` and `DbParam[Maybe[A]]` produces for `Maybe.absent`; `DbNull` appears only on the write side: decoding never sees a `DbValue`, because codecs read from a result reader and detect `NULL` themselves. A non-optional codec does not fail on `NULL` either, except for `BigDecimal` and a derived `Char`: `Int` reads `0` and `String` reads `null`. `DbArray` is the one variant with two fields (`elementType`, `elements`) instead of `value`, used for collection-typed columns and mapped by `SqlDialect#typeName` to `TEXT[]` on PostgreSQL and `TEXT` on SQLite.

See [Frag](./frag.md), [DbCodec](./db-codec.md), and the [SQL module index](./index.md).

---

## Ddl

`Ddl` is a helper object that generates DDL (Data Definition Language) SQL fragments for creating and dropping tables. `ColumnDef` is a simple data class pairing a column name with a SQL type string and a nullability flag.

Normally you don't call `Ddl` directly — [`Table#createTable`](./table.md) and `Table#dropTable` use it internally. You may need `Ddl` directly when building custom DDL tooling.

## Core API

To generate CREATE TABLE and DROP TABLE fragments, call the two methods on `Ddl`, with column metadata described by `ColumnDef`:

```scala
object Ddl {
  def createTable(tableName: String, columns: IndexedSeq[ColumnDef]): Frag
  def dropTable(tableName: String): Frag
}

final case class ColumnDef(name: String, sqlType: String, nullable: Boolean)
```

## Usage

Create a `ColumnDef` for each column, then pass them to `Ddl.createTable`:

```scala

```

```scala
val transactor: Transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)
```

```scala
val columns = IndexedSeq(
  ColumnDef("id",         "INTEGER", nullable = false),
  ColumnDef("name",       "TEXT",    nullable = false),
  ColumnDef("created_at", "TEXT",    nullable = true)
)
// columns: IndexedSeq[ColumnDef] = Vector(
//   ColumnDef(name = "id", sqlType = "INTEGER", nullable = false),
//   ColumnDef(name = "name", sqlType = "TEXT", nullable = false),
//   ColumnDef(name = "created_at", sqlType = "TEXT", nullable = true)
// )

val createFrag = Ddl.createTable("users", columns)
// createFrag: Frag = Frag(
//   parts = Vector(
//     """CREATE TABLE IF NOT EXISTS users (
//   id INTEGER NOT NULL,
//   name TEXT NOT NULL,
//   created_at TEXT
// )"""
//   ),
//   params = Vector()
// )
val dropFrag   = Ddl.dropTable("users")
// dropFrag: Frag = Frag(
//   parts = Vector("DROP TABLE IF EXISTS users"),
//   params = Vector()
// )

// Execute the fragments
transactor.transact {
  createFrag.update
  // ... do work ...
  dropFrag.update
}
// res1: Int = 0
```

## How It Works

`Ddl` is typically used by `Table#createTable`, which derives `ColumnDef` from schema metadata:

1. `Table.derived[A]` builds a schema.
2. `Table#createTable(dialect)` converts schema columns to `ColumnDef` using `dialect.typeName`.
3. `Ddl.createTable` receives the `ColumnDef` list and generates the SQL fragment.

The generated statement declares no `PRIMARY KEY` or `UNIQUE` constraint. This is a known bug in 0.0.56: the ID column that [`Repo#insertOrUpdate`](./repo.md) relies on needs one, so until the bug is fixed, write the `CREATE TABLE` statement yourself with [`Frag.literal`](./frag.md).

You call `Ddl` directly only when you need custom DDL that doesn't fit the `Table` abstraction.

See [Table](./table.md) for the high-level DDL API.

---

## Dump

`Dump` is a **compile-time SQL dump**: two macros that write the SQL of a [`SqlQuery`](./sql-query.md) or the `CREATE TABLE` statement of a [`Table`](./table.md) to `.sql` files while the code compiles. The files let you review the SQL a program runs without running it, for example in a pull request, where a change to a join shows up as a diff of its SQL, or for a database administrator who doesn't read Scala. Dumping is opt-in: the macros only write when the JVM system property `zib.sql.dumpDir` is set in the JVM that runs the compiler, and otherwise each call expands to `()` and costs nothing at run time.

`Dump` lives in the `zio.blocks.sql` package of the `zio-blocks-sql` module, which is published for Scala 3 only (see the [SQL module](./index.md) page). Its key properties are:

- Each call writes one file per dialect, for PostgreSQL and for SQLite.
- The macros read the source code of their argument, not its value, so the query must be written out where the macro can see it.
- Problems never fail the build: a part the macro can't read produces a compiler warning, and the file is written without that part or not at all.
- While the property is set, every `sql"..."` fragment writes its own file too.

`Dump` is an object with two inline methods:

```scala
object Dump {
  inline def dumpQuery[A](inline query: SqlQuery[A]): Unit
  inline def dumpTable[A](inline table: Table[A])(using schema: Schema[A]): Unit
}
```

## Usage

We put the queries to review in `inline def`s, so the macro can read them, and call `Dump.dumpQuery` and `Dump.dumpTable` in an object that is compiled with the rest of the code:

```scala

case class Customer(customerId: Int, name: String, city: String, referrerId: Option[Int])
object Customer { implicit val schema: Schema[Customer] = Schema.derived }

case class Invoice(invoiceId: Int, buyerId: Int, year: Int, totalCents: Long)
object Invoice { implicit val schema: Schema[Invoice] = Schema.derived }

object ShopTables {
  val customers = Table.derived[Customer]
  val invoices  = Table.derived[Invoice]
}

object ShopQueries {
  inline def largeInvoices: SqlQuery[Invoice] =
    SqlQuery
      .from(ShopTables.invoices)
      .join(Rel.manyToOne(ShopTables.invoices, _.buyerId, ShopTables.customers, _.customerId))
      .where(Frag(IndexedSeq("""t0."total_cents" > """, ""), IndexedSeq(DbValue.DbLong(10000L))))
      .orderBy("year", SortOrder.Desc)
      .limit(10)
}

object ShopDumps {
  val largeInvoices = Dump.dumpQuery(ShopQueries.largeInvoices)

  Dump.dumpTable(ShopTables.customers)
  Dump.dumpTable(ShopTables.invoices)
}
```

Compiling with the property set writes the files:

```bash
sbt -Dzib.sql.dumpDir=target/sql-dumps clean compile
```

The query's file, `target/sql-dumps/largeInvoices_query-postgresql.sql`, holds the same SQL that `SqlQuery#sql` returns at run time, on one line:

```sql
SELECT t0."invoice_id", t0."buyer_id", t0."year", t0."total_cents", t1."customer_id", t1."name", t1."city", t1."referrer_id" FROM "invoice" AS t0 INNER JOIN "customer" AS t1 ON t0."buyer_id" = t1."customer_id" WHERE t0."total_cents" > ? ORDER BY t0."year" DESC LIMIT 10
```

Each table's file, such as `target/sql-dumps/invoice-postgresql.sql`, holds its `CREATE TABLE` statement:

```sql
CREATE TABLE IF NOT EXISTS invoice (
  invoice_id INTEGER NOT NULL,
  buyer_id INTEGER NOT NULL,
  year INTEGER NOT NULL,
  total_cents BIGINT NOT NULL
)
```

The `.sql` listings on this page are copied from the files that compiling this code against `zio-blocks-sql` 0.0.56 writes.

## Turning Dumps On

The macros look up `zib.sql.dumpDir` when they expand, so the property must be set in the JVM that runs the Scala compiler, and the directory it names is resolved against that JVM's working directory. With sbt, that is the sbt JVM:

```bash
sbt -Dzib.sql.dumpDir=target/sql-dumps clean compile
```

`SBT_OPTS="-Dzib.sql.dumpDir=target/sql-dumps"` works too. A running sbt server (`sbt --client`, or the one an IDE starts) keeps the properties it was started with, so shut it down and start it again with the property. `clean` matters as well: incremental compilation skips unchanged files, and a file that isn't compiled writes no dumps. Passing the property through `scalacOptions` or `javaOptions` does nothing, because those reach the compiler as flags or the forked program at run time, not the compiler's JVM.

With scala-cli, compile without the build server, so the compiler runs in a JVM that you start, and pass the property through `JAVA_TOOL_OPTIONS`:

```bash
JAVA_TOOL_OPTIONS="-Dzib.sql.dumpDir=$PWD/sql-dumps" scala-cli compile --server=false .
```

## Core Operations

`Dump` has two operations, one per kind of SQL it writes, plus the files that the `sql"..."` interpolator writes on its own while the property is set. All of them write through the same file rules, described in [File Handling](#file-handling). `Dump` also has public `Dump.dumpQueryIrImpl` and `Dump.dumpTableImpl` methods: they are the macro implementations behind the two inline methods, and you don't call them directly.

### Dumping a Query

`Dump.dumpQuery` writes a query's `SELECT` statement to `<name>_query-<dialect>.sql`:

```scala
object Dump {
  inline def dumpQuery[A](inline query: SqlQuery[A]): Unit
}
```

#### What the Macro Can Read

The macro rebuilds the SQL from the source code of the query, so each part of the query must be visible at the call site:

- The query is an `inline def`, or the `SqlQuery.from(...)` chain itself written in the call. A query stored in a plain `val` or built at run time writes no file, and the compiler warns `Dump requires inline query value; preconstructed value will emit no file; ...`.
- Each [`Rel`](./rel.md) is built inside the chain. A `Rel` stored in a `val` gives the warning `Dump.dumpQuery: could not decode Rel for join`, and the file is written without that join.
- `SqlQuery#orderBy` columns are string literals, and `SqlQuery#limit` and `SqlQuery#offset` take integer literals. Anything else gives a warning such as `Dump.dumpQuery: limit expects int literal`, and the file is written without that clause.
- An `IN` condition built from a `DbArray` whose size isn't known at compile time writes no file, with a warning that starts `IN placeholder: compile-time DbArray cardinality is indeterminate`.

A query that passes these rules is written with the same SQL as `SqlQuery#sql`, plain `?` placeholders included. Unlike `SqlQuery#explain`, the file has no numbered placeholders and no `-- params:` line.

:::warning
A `SqlQuery#where` condition written with `sql"..."` loses its parameter in the dump, a known bug in 0.0.56. A query written as `SqlQuery.from(ShopTables.invoices).where(sql"""t0."year" = ${2026}""")` is dumped as `... WHERE t0."year" = ` with no `?`, while `SqlQuery#explain` shows `WHERE t0."year" = ?1`. A condition built with the [`Frag`](./frag.md) constructor, as in [Usage](#usage), keeps its `?`. Use `SqlQuery#explain` to check queries with `sql"..."` conditions.
:::

#### How Files Are Named

The `<name>` part of the file name comes from where the call is. For a call assigned to a `val` or made inside a `def`, it is that `val` or `def`'s name: `val largeInvoices = Dump.dumpQuery(...)` writes `largeInvoices_query-postgresql.sql` and `largeInvoices_query-sqlite.sql`. A call inside `def inDef(): Unit` writes `inDef_query-<dialect>.sql`.

:::warning
A `Dump.dumpQuery` call that isn't assigned to a `val`, written directly in an object body, is named after the object as `_local_<Object>___query-<dialect>.sql` (three underscores before `query`), and every other such call in the same object writes the same file, so only the last one survives. This is a known bug in 0.0.56: assign every call to its own `val`.
:::

### Dumping a Table

`Dump.dumpTable` writes a table's `CREATE TABLE IF NOT EXISTS` statement to `-<dialect>.sql`, with each dialect's column types: an `Int` column is `INTEGER` in both files, and a `Long` column is `BIGINT` for PostgreSQL and `INTEGER` for SQLite. It needs the row type's `Schema` in scope:

```scala
object Dump {
  inline def dumpTable[A](inline table: Table[A])(using schema: Schema[A]): Unit
}
```

Like the statement from `Table#createTable`, the dumped statement declares no `PRIMARY KEY`. The SQLite file for the `customer` table from [Usage](#usage) is:

```sql
CREATE TABLE IF NOT EXISTS customer (
  customer_id INTEGER NOT NULL,
  name TEXT NOT NULL,
  city TEXT NOT NULL,
  referrer_id INTEGER
)
```

:::warning
`Dump.dumpTable` doesn't read the `Table` value; it rebuilds the statement from the row type, a known bug in 0.0.56. Every column is the field name in snake case, even when `@Modifier.rename` gives the column another name, and the table name is the first string literal in the argument that is a plain SQL identifier, such as `Table.derived[Invoice]("invoices")`, or else the type name in snake case. A table created with `TableNamingPolicy.Plural`, or named by a string stored elsewhere, is dumped under the singular type name. `Dump.dumpQuery` takes its table and column names the same way. Compare the dumps of such tables with `Table#createTable` before you rely on them.
:::

### Fragment Files

While the property is set, every `sql"..."` fragment also writes a file, `<owner>-<dialect>.sql`, with the fragment's SQL text and a `?` for each interpolated value. The owner is the enclosing `def` or `val`. A fragment in this method:

```scala

def byCity(city: String): Frag =
  sql"""SELECT name FROM customer WHERE city = $city"""
```

writes `byCity-postgresql.sql` and `byCity-sqlite.sql`, each containing:

```sql
SELECT name FROM customer WHERE city = ?
```

A fragment inside an `inline def` writes one file per place the `inline def` is used, named after the owner at that place. The table-checked form of the interpolator, described in [sql(table) Checked Interpolation](../../guides/sql-checked-interpolation.md), writes the same files. Two fragments with the same owner write the same file, and the last one compiled wins.

## File Handling

All dump files follow the same rules:

- The directory and any missing parent directories are created on the first write.
- Files are written as UTF-8 with a trailing newline.
- A file whose content is already the same isn't rewritten, so committed dump files only change when the SQL does.
- A name that isn't a plain SQL identifier is sanitized: every character other than an ASCII letter, digit or `_` becomes `_`, and a name that starts with a digit gets a leading `_`. That is why `largeInvoices-query` becomes `largeInvoices_query`.
- A write that fails gives a compiler warning that starts `[zib.sql.dumpDir] Failed to dump` and doesn't fail the build.
- Nothing is ever deleted, so the files of a removed or renamed query stay until you clear the directory.

To keep committed dumps in step with the code, clear the directory and run `sbt -Dzib.sql.dumpDir=<dir> clean compile` before committing.

## Integration

`Dump` reviews the SQL that the module's other types generate:

- [`SqlQuery`](./sql-query.md) is the query type `Dump.dumpQuery` reads; `SqlQuery#explain` prints the same SQL at run time, with parameter types.
- [`Rel`](./rel.md) describes the joins, and must be built inside a dumped query.
- [`Table`](./table.md) and [`Ddl`](./ddl.md) produce the `CREATE TABLE` statements that `Dump.dumpTable` writes at compile time.
- [How to Write Joins and Inspect the Generated SQL](../../guides/write-joins-and-inspect-generated-sql.md) has a runnable dump example in the `sql-examples` project.

---

## Frag

`Frag` is an immutable SQL fragment — a piece of SQL text with typed parameter values kept safely separate from the literal SQL. The `sql"..."` string interpolator builds fragments by checking at compile time that every interpolated expression can be bound as a parameter. Fragments compose with `++` and execute through methods like `query`, `update`, and `queryOne`, plus the chunked streaming methods `queryStream` and `queryChunked`.

`Frag` is safe from SQL injection because parameter values never appear in the SQL string — they are stored separately and bound to `?` placeholders at execution time.

## Core API

```scala
final case class Frag(parts: IndexedSeq[String], params: IndexedSeq[DbValue]) {
  def ++(other: Frag): Frag
  def sql(dialect: SqlDialect): String
  def queryParams: IndexedSeq[DbValue]
  def isEmpty: Boolean
}

object Frag {
  val empty: Frag
  def literal(sqlStr: String): Frag
  def sequence(frags: Frag*): Frag
  def values[A](rows: Seq[A])(using codec: DbCodec[A]): Frag

  extension (frag: Frag) {
    def query[A](using DbCon, DbCodec[A]): List[A]
    def queryOne[A](using DbCon, DbCodec[A]): Maybe[A]
    def queryLimit[A](limit: Int)(using DbCon, DbCodec[A]): List[A]
    def queryStream[A](using DbCon, DbCodec[A]): Stream[Throwable, Chunk[A]]
    def queryChunked[A](chunkSize: Int)(using DbCon, DbCodec[A]): Stream[Throwable, Chunk[A]]
    def update(using DbCon): Int
    def updateReturningKeys[A](using DbCon, DbCodec[A]): List[A]
  }
}
```

## Usage

Build fragments with the `sql"..."` interpolator and execute them inside a `Transactor` block:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val tx: Transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)

val userId  = 42
val active  = true

tx.connect {
  // Query — returns all matching rows
  val users: List[User] = 
    sql"SELECT id, name, email FROM users WHERE id > $userId AND active = $active".query[User]
  
  // QueryOne — returns at most one row
  val one: Maybe[User] = 
    sql"SELECT id, name, email FROM users WHERE id = ${1}".queryOne[User]
  
  // Update — returns affected row count
  val deleted: Int = 
    sql"DELETE FROM users WHERE active = ${false}".update
  
  // UpdateReturningKeys — returns generated keys after INSERT
  val keys: List[Long] = 
    sql"INSERT INTO users (name, email) VALUES (${"Alice"}, ${"alice@example.com"})".updateReturningKeys[Long]
}
```

## Building Fragments

**`sql"..."` interpolator** — The primary way to build fragments. Compile-time checked: every interpolated value must have a `DbParam` instance (provided for all common Scala and Java types).

```scala

val userId = 42
// userId: Int = 42
val frag = sql"SELECT * FROM users WHERE id = $userId"
// frag: Frag = Frag(
//   parts = Vector("SELECT * FROM users WHERE id = ", ""),
//   params = Vector(DbInt(42))
// )
frag.params
// res2: IndexedSeq[DbValue] = Vector(DbInt(42))
```

**`Frag.literal`** — Wrap static SQL text (no parameters):

```scala
val query = sql"SELECT * FROM users" ++ Frag.literal(" ORDER BY name")
// query: Frag = Frag(
//   parts = Vector("SELECT * FROM users ORDER BY name"),
//   params = Vector()
// )
query.sql(SqlDialect.SQLite)
// res3: String = "SELECT * FROM users ORDER BY name"
```

**`Frag.values`** — Build a multi-row INSERT VALUES clause:

```scala

case class Product(name: String, price: BigDecimal) derives DbCodec

val products = List(Product("Widget", BigDecimal("9.99")), Product("Gadget", BigDecimal("24.99")))
// products: List[Product] = List(
//   Product(name = "Widget", price = 9.99),
//   Product(name = "Gadget", price = 24.99)
// )
val insert = Frag.literal("INSERT INTO product (name, price) VALUES ") ++ Frag.values(products)
// insert: Frag = Frag(
//   parts = Vector(
//     "INSERT INTO product (name, price) VALUES (",
//     ", ",
//     "), (",
//     ", ",
//     ")"
//   ),
//   params = Vector(
//     DbString("Widget"),
//     DbBigDecimal(9.99),
//     DbString("Gadget"),
//     DbBigDecimal(24.99)
//   )
// )
insert.sql(SqlDialect.SQLite)
// res5: String = "INSERT INTO product (name, price) VALUES (?, ?), (?, ?)"
```

**`Frag.empty`** — The identity fragment for composition. Useful for optional clauses:

```scala

val hasFilter = true
// hasFilter: Boolean = true
val where = if (hasFilter) sql" WHERE active = ${true}" else Frag.empty
// where: Frag = Frag(
//   parts = Vector(" WHERE active = ", ""),
//   params = Vector(DbBoolean(true))
// )
val query = sql"SELECT * FROM users" ++ where
// query: Frag = Frag(
//   parts = Vector("SELECT * FROM users WHERE active = ", ""),
//   params = Vector(DbBoolean(true))
// )
query.sql(SqlDialect.SQLite)
// res7: String = "SELECT * FROM users WHERE active = ?"
```

See also the [How to Build a Query from Optional Search Filters](../../guides/build-a-query-from-optional-search-filters.md) guide, which joins the conditions a search form needs into one `WHERE` clause and skips the empty ones.

**`Frag.sequence`** — Concatenate multiple fragments with no separator:

```scala

val status = "active"
// status: String = "active"
val base   = sql"SELECT * FROM users"
// base: Frag = Frag(parts = Vector("SELECT * FROM users"), params = Vector())
val where  = sql" WHERE status = $status"
// where: Frag = Frag(
//   parts = Vector(" WHERE status = ", ""),
//   params = Vector(DbString("active"))
// )
val order  = Frag.literal(" ORDER BY name")
// order: Frag = Frag(parts = Vector(" ORDER BY name"), params = Vector())
val full   = Frag.sequence(base, where, order)
// full: Frag = Frag(
//   parts = Vector("SELECT * FROM users WHERE status = ", " ORDER BY name"),
//   params = Vector(DbString("active"))
// )
full.sql(SqlDialect.SQLite)
// res9: String = "SELECT * FROM users WHERE status = ? ORDER BY name"
```

See also the [How to Upsert Rows and Paginate with Keyset Cursors](../../guides/upsert-and-paginate-with-keyset-cursors.md) guide, which builds upsert statements and pages through a query with `Frag.keysetAfter`.

## Execution Methods

All execution methods require an implicit `DbCon` (provided by `Transactor#connect` or `Transactor#transact`). They render the fragment to SQL, bind parameters, execute, and log the operation.

**`query[A]`** — Execute SELECT and return all rows:

```scala

case class User(id: Int, name: String)
object User { implicit val schema: Schema[User] = Schema.derived }

given DbCon = ???

val users: List[User] = sql"SELECT id, name FROM users".query[User]
```

**`queryOne[A]`** — Execute SELECT and return at most one row:

```scala

case class User(id: Int, name: String)
object User { implicit val schema: Schema[User] = Schema.derived }

given DbCon = ???

val user: Maybe[User] = sql"SELECT id, name FROM users WHERE id = ${1}".queryOne[User]
```

**`queryLimit[A](n)`** — Execute SELECT and return up to n rows (fetched from Scala side):

```scala

case class User(id: Int, name: String)
object User { implicit val schema: Schema[User] = Schema.derived }

given DbCon = ???

val page: List[User] = sql"SELECT id, name FROM users ORDER BY name".queryLimit[User](10)
```

**`queryStream[A]`** — Execute SELECT and return rows as a chunked stream (`zio.blocks.streams.Stream`), batching `DefaultQueryChunkSize` (64) rows per chunk. Acquisition is lazy — nothing touches the connection until the first pull — so consume the stream within the scope that provides the `DbCon`; leaving `Transactor.connect`/`transact` closes the captured connection, and pulling afterwards fails.

```scala

case class User(id: Int, name: String)
object User { implicit val schema: Schema[User] = Schema.derived }

given DbCon = ???

val chunks: Stream[Throwable, Chunk[User]] =
  sql"SELECT id, name FROM users".queryStream[User]
val all: Either[Throwable, List[User]] = chunks.runCollect.map(_.toList.flatten)
```

**`queryChunked[A](n)`** — Like `queryStream`, but with an explicit batch size. The statement and result set are acquired lazily on the first pull and released when the stream is closed or fully drained; each chunk holds up to `n` rows, so memory stays bounded for large result sets. The same lifetime rule applies: consume the stream before the enclosing `Transactor.connect`/`transact` callback returns — afterwards the captured connection is closed and the first pull fails.

```scala

case class User(id: Int, name: String)
object User { implicit val schema: Schema[User] = Schema.derived }

given DbCon = ???

val batches: Stream[Throwable, Chunk[User]] =
  sql"SELECT id, name FROM users".queryChunked[User](500)
```

**`update`** — Execute INSERT, UPDATE, or DELETE and return affected row count:

```scala

given DbCon = ???

val deleted: Int = sql"DELETE FROM users WHERE inactive = ${true}".update
```

**`updateReturningKeys[A]`** — Execute INSERT and return auto-generated primary key(s):

```scala

given DbCon = ???

val keys: List[Long] = sql"INSERT INTO users (name) VALUES (${"Alice"})".updateReturningKeys[Long]
```

## See Also

- **[How to Stream Large Query Results](../../guides/stream-large-query-results.md)** — Guide to aggregating, exporting and error-handling the chunked streams that `queryStream` and `queryChunked` return.
- **[How to Write Joins and Inspect the Generated SQL](../../guides/write-joins-and-inspect-generated-sql.md)** — Guide that runs join queries and prints or dumps the SQL they generate.
- **[SQL Module Overview](./index.md)** — How `Frag`, `Transactor` and the other SQL types fit together.

---

## SQL Module

The `zio-blocks-sql` module provides type-safe, schema-driven SQL query building and execution on relational databases. Its core types — `DbCodec`, `Frag`, `Table`, `Repo`, `Transactor`, `DbCon`, and `DbTx` — form a layered system: codecs map Scala types to database columns, fragments carry parameterized SQL built via string interpolation, repositories expose CRUD operations derived at compile time from a schema, and a transactor manages the connection lifecycle with automatic commit and rollback.

## Motivation

Relational database access in Scala typically involves one of three trade-offs: 

- Raw JDBC requires manual row mapping and string concatenation (SQL injection risk, tedious boilerplate)
- Reflection-based ORMs hide SQL entirely (opaque at compile time, difficult to optimize)
- Lifted-embedding DSLs like Slick require learning a parallel query language on top of SQL. 

The `zio-blocks-sql` module takes a different path: it uses ZIO Blocks' `Schema` system for compile-time column metadata and familiar string interpolation for SQL text, giving you type safety without losing SQL's expressiveness.

Key advantages of the `zio-blocks-sql` module are:

- **Schema-driven derivation.** `DbCodec`, `Table`, and `Repo` all derive from the same `Schema[A]`, keeping column names, SQL types, and nullability in sync with your Scala data model automatically.
- **No runtime reflection.** Derivation runs at compile time through the `Deriver[DbCodec]` framework, producing zero-overhead codecs with no reflective calls at runtime.
- **Compile-time SQL parameterization.** The `sql"..."` macro interpolator verifies that every interpolated value has a `DbParam[A]` instance at compile time, converts it to a `DbValue`, and binds it to a `?` placeholder — no string concatenation, no SQL injection.
- **Composable fragments.** `Frag` values compose via `Frag#++`, letting you build reusable WHERE clauses, ORDER BY expressions, and pagination helpers that render correctly for any `SqlDialect`.
- **Explicit transaction boundaries.** `Transactor#transact` disables auto-commit, commits on success, and rolls back on any exception — the connection is always closed whether the block succeeds or throws.

## Installation

The core SQL module and the ZIO integration module publish separately. Add the artifacts you need to your build:

```scala
// Core SQL module (cross-built for JVM and Scala.js; the JDBC-backed
// JdbcTransactor implementation itself is JVM-only)
libraryDependencies += "dev.zio" %% "zio-blocks-sql" % "0.0.56"

// ZIO integration (lifts JDBC into ZIO effects; JVM-only)
libraryDependencies += "dev.zio" %% "zio-blocks-sql-zio" % "0.0.56"
```

Both modules require a JDBC driver on the classpath (for example, `org.xerial:sqlite-jdbc` for SQLite or `org.postgresql:postgresql` for PostgreSQL). Swap the JDBC URL and `SqlDialect` constant to switch databases.

## Overview

The module is organized into three groups: core types you interact with directly in application code, supporting types that implement the protocol layer, and a ZIO integration layer that lifts synchronous operations into the effect system.

### Core Types

These seven types form the public API for everyday SQL work:

- **[`DbCodec`](./db-codec.md)** — Converts between Scala values and database columns. Read values from result sets by column label or position; write values as bound parameters. Derived automatically from `Schema[A]`.
- **[`Frag`](./frag.md)** — An immutable SQL fragment built with the `sql"..."` interpolator. Holds literal SQL text and typed parameters separately, preventing SQL injection. Executed via `query`, `queryOne`, `update`, and other extension methods.
- **[`Table`](./table.md)** — Binds a Scala type to a database table: table name, codec, and column metadata. Derived from a `Schema` using naming conventions. Provides DDL generation via `createTable` and `dropTable`.
- **[`Repo`](./repo.md)** — Type-safe CRUD repository providing `all`, `find`, `insert`, `update`, `delete`, and other standard operations. All SQL is pre-built at construction time.
- **[`Transactor`](./transactor.md)** — Entry point for executing SQL. `connect` opens a connection; `transact` adds automatic commit/rollback. `JdbcTransactor` is the JDBC implementation.
- **[`DbCon`](./db-con.md)** — Implicit context available inside a `Transactor` block. Carries the connection, dialect, and logger. Threaded through all SQL operations automatically.
- **[`DbTx`](./db-tx.md)** — A `DbCon` subtype marking transactional scope (inside `Transactor#transact`). Extends `DbCon` so transactional and non-transactional operations compose freely.

### Query Builder

These three types, in the `zio.blocks.sql.query` package (`Dump` is in `zio.blocks.sql`), build `SELECT` statements with joins instead of writing them by hand, and let you review the SQL they generate:

- **[`SqlQuery`](./sql-query.md)** — Immutable `SELECT` builder over a `Table`: joins, `WHERE` conditions, `GROUP BY`, `ORDER BY`, `LIMIT` and `OFFSET`. Renders to a `Frag` with `toFrag`, and prints its SQL with `sql` and `explain`.
- **[`Rel`](./rel.md)** — Foreign-key link between two tables, built from field selectors checked at compile time or from column names. Each `Rel` becomes one join's `ON` condition.
- **[`Dump`](./dump.md)** — Compile-time macros that write a query's `SELECT` or a table's `CREATE TABLE` to `.sql` files for review when the `zib.sql.dumpDir` system property is set.

### Supporting Types

These types implement the protocol and derivation layers and are rarely used directly in application code:

- **[`DbValue`](./db-value.md)** — Sealed ADT representing typed database values. Used internally by `DbCodec` and `Frag` to hold parameters before binding.
- **[`DbParam`](./db-param.md)** — Typeclass converting Scala values to `DbValue` for the `sql"..."` interpolator. Instances provided for all common types, `Option[A]`, and `Maybe[A]`.
- **[`SqlDialect`](./sql-dialect.md)** — Encodes database-specific SQL: type names for DDL and parameter placeholder tokens. `PostgreSQL` and `SQLite` are built-in.
- **[`SqlLogger`](./sql-logger.md)** — Hook for observing query execution. Receives SQL, parameters, duration, and row count on success or error.
- **[`SqlNameMapper`](./sql-name-mapper.md)** — Maps Scala field names to SQL column names. `SnakeCase` (default converts `camelCase` to `snake_case`), `Identity`, or `Custom` function.
- **[`TableMetadata`](./table-metadata.md)** — Derives column metadata from a `Schema`: names, types, and nullability. Used by `Table.derived` to populate table structure.
- **[`Ddl`](./ddl.md)** — Generates `CREATE TABLE IF NOT EXISTS` and `DROP TABLE IF EXISTS` fragments from column definitions.
- **[`DbConnection`](./db-connection.md)** — Abstraction over JDBC `Connection`. Prepares statements, controls transactions, and manages lifecycle.
- **[`DbResultReader`](./db-result-reader.md)** — Reads column values from result sets by label or 1-based position. Supports null detection via `wasNull`. Used by `DbCodec`.
- **[`DbParamWriter`](./db-param-writer.md)** — Binds parameter values to prepared statements. Covers all primitive, temporal, and UUID types. Used by `DbCodec`.
- **[`DbCodecDeriver`](./db-codec-deriver.md)** — Schema-driven codec derivation engine. Converts `Schema[A]` to `DbCodec[A]`, handling primitives, records, options, and JSONB types.
- **[`TransactorZIO`](./transactor-zio.md)** — ZIO integration for `Transactor`. Runs SQL effects on the blocking thread pool with proper resource cleanup and interrupt handling. Includes `ZLayer` support.

## How They Work Together

Five interconnected flows define how the module's types cooperate: codec derivation, SQL assembly, execution context, entity mapping, and transaction lifecycle. Understanding these flows shows why each type exists and how to compose them.

The typical workflow proceeds in this order:

1. Define a `Schema[A]` for your domain type (usually with `Schema.derived`).
2. Call `Repo.derived` (or `Table.derived` + `Repo.apply`) to produce a `Repo[E, ID]` — all SQL is pre-built here.
3. Create a `JdbcTransactor` from a JDBC URL or `DataSource`.
4. Open a connection with `Transactor#connect` (for reads) or `Transactor#transact` (for writes), which supplies a `DbCon` or `DbTx` context.
5. Inside the context block, call `Repo` methods or execute `sql"..."` fragments directly.

The following diagram shows the data-flow relationships between types:

```
                        Schema[A]
                            │
           ┌────────────────┼──────────────────┐
           │                │                  │
           ▼                ▼                  ▼
    DbCodecDeriver    TableMetadata      deriveTableName
           │           .columnsFor()    (TableNamingPolicy
           ▼                │            + SqlNameMapper)
      DbCodec[A] ◄──────────┘                  │
           │                                   │
           └───────────────────────────────────┘
                            │
                    Table[A](name, codec, columnsMeta)
                            │
                    Repo[E, ID]                   sql"..." → Frag
                    ─ all / find                   ─ frag.query[A]
                    ─ insert / update / delete     ─ frag.update
                            │                             │
                    ┌───────┴─────────────────────────────┘
                    │
                    ▼
              Transactor
              ├─ .connect  { ... }    (non-transactional, DbCon in scope)
              └─ .transact { ... }    (commit / rollback, DbTx in scope)
                                │
                         DbCon / DbTx
                   ┌────────────┼────────────┐
                   ▼            ▼            ▼
            DbConnection    SqlDialect   SqlLogger
            (JDBC wrapper) (PostgreSQL  (observability)
                            | SQLite)
```

The `sql"..."` interpolator is defined as an extension on `StringContext`. At compile time the macro verifies that every interpolated expression has a `DbParam[A]` instance, converts it to a `DbValue`, and assembles a `Frag` with literal SQL in `parts` and the typed values in `params`. The `Frag` carries no dialect-specific rendering until `Frag#sql(dialect)` is called, so the same fragment works with PostgreSQL and SQLite unchanged.

A complete end-to-end example showing schema definition, table setup, repository construction, and mixed repository/fragment queries follows:

```scala

case class User(id: Int, name: String, email: String)
object User {
  implicit val schema: Schema[User] = Schema.derived
}

// 1. Create transactor and repository
val tx   = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)
// tx: JdbcTransactor = zio.blocks.sql.JdbcTransactor@55580f9
val repo = Repo.derived[User, Int]("users", "id", _.id)
// repo: Repo[User, Int] = zio.blocks.sql.Repo$DerivedRepo@5785a4fc

// 2. Set up schema and run a transactional workflow
tx.transact {
  // Create table using derived DDL
  repo.table.createTable(summon[DbTx].dialect).update

  // Insert via repository (uses pre-built INSERT fragment)
  repo.insert(User(1, "Alice", "alice@example.com"))
  repo.insert(User(2, "Bob", "bob@example.com"))

  // Read via repository
  val alice: Maybe[User] = repo.find(1)

  // Read via raw SQL fragment — composes freely with repo operations
  val aUsers: List[User] =
    sql"SELECT id, name, email FROM users WHERE name LIKE ${"A%"}".query[User]

  // Update and delete
  repo.update(User(1, "Alice Smith", "alice.smith@example.com"))
  repo.delete(2)

  (alice, aUsers)
}
// res1: Tuple2[Maybe[User], List[User]] = (
//   User(id = 1, name = "Alice", email = "alice@example.com"),
//   List(User(id = 1, name = "Alice", email = "alice@example.com"))
// )
```

Inside `transact`, auto-commit is disabled. If any call throws, the entire block rolls back and the exception propagates. On normal return, the transaction commits and the connection closes.

## Common Patterns

The module is designed around a small set of patterns that appear throughout most application code. Recognizing these patterns makes it easy to choose the right approach for each situation.

### Schema-Driven Derivation

`DbCodec`, `Table`, and `Repo` all derive from a single `Schema[A]`. Derivation respects `@Modifier` annotations — `@Modifier.rename` overrides a column name, `@Modifier.transient` excludes a field from the codec, and `@Modifier.config("sql.table_name", "my_table")` overrides the table name. This means your Scala type definition is the single source of truth for column names, types, and nullability:

```scala

case class BlogPost(
  @Modifier.rename("post_id") id: Int,
  title: String,
  @Modifier.transient authorHandle: String = ""  // excluded from SQL
)
object BlogPost {
  implicit val schema: Schema[BlogPost] = Schema.derived
}

val repo = Repo.derived[BlogPost, Int]("post_id", _.id)
// repo: Repo[BlogPost, Int] = zio.blocks.sql.Repo$DerivedRepo@122af0bf
repo.table.name
// res3: String = "blog_post"
repo.table.codec.columns
// res4: IndexedSeq[String] = Vector("post_id", "title")
```

### Implicit Context Threading

Every SQL operation — `Frag` execution and `Repo` CRUD — requires an implicit `DbCon` (or `DbTx`) in scope. The context carries the connection, dialect, and logger, but you never pass it explicitly. Calling code inside `Transactor#connect` or `Transactor#transact` automatically has the context available, and helper methods can propagate it with a `using` parameter:

```scala

case class Product(id: Int, name: String, price: BigDecimal)
object Product { implicit val schema: Schema[Product] = Schema.derived }

def cheapProducts(maxPrice: BigDecimal)(using DbCon): List[Product] =
  sql"SELECT id, name, price FROM product WHERE price < $maxPrice".query[Product]

val tx = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)
tx.connect {
  // `cheapProducts` picks up the DbCon automatically
  val items = cheapProducts(BigDecimal("9.99"))
}
```

### Multi-Column Codecs

A single `DbCodec[A]` can span multiple database columns. When you use a multi-column type directly in an `sql"..."` expression, the `fromDbCodec` `DbParam` instance throws at runtime because it cannot collapse multiple values into a single `?` placeholder. Instead, use `Frag.values` for multi-row inserts or write the columns explicitly in the fragment:

```scala

case class Point(x: Double, y: Double)
object Point { implicit val schema: Schema[Point] = Schema.derived }

// Multi-row insert using Frag.values — one (?, ?) tuple per row
val points = List(Point(1.0, 2.0), Point(3.0, 4.0))
// points: List[Point] = List(Point(x = 1.0, y = 2.0), Point(x = 3.0, y = 4.0))
val frag   = Frag.literal("INSERT INTO point (x, y) VALUES ") ++ Frag.values(points)
// frag: Frag = Frag(
//   parts = Vector("INSERT INTO point (x, y) VALUES (", ", ", "), (", ", ", ")"),
//   params = Vector(DbDouble(1.0), DbDouble(2.0), DbDouble(3.0), DbDouble(4.0))
// )
frag.sql(SqlDialect.SQLite)
// res7: String = "INSERT INTO point (x, y) VALUES (?, ?), (?, ?)"
```

### Type-Safe SQL Parameterization

The `sql"..."` interpolator accepts any Scala value for which a `DbParam[A]` exists. The macro checks this at compile time and binds the value to a `?` placeholder, preventing SQL injection regardless of the value's content. All standard scalar types have built-in instances, `Option[A]` binds to `NULL` or the inner value, and custom types with a `DbCodec[A]` automatically gain a `DbParam[A]`:

```scala

val userId: Int             = 42
// userId: Int = 42
val namePattern: String     = "%alice%"
// namePattern: String = "%alice%"
val active: Option[Boolean] = Some(true)
// active: Option[Boolean] = Some(true)

// All three are compile-time safe — no string concatenation
val frag =
  sql"SELECT * FROM users WHERE id = $userId AND name LIKE $namePattern AND active = $active"
// frag: Frag = Frag(
//   parts = Vector(
//     "SELECT * FROM users WHERE id = ",
//     " AND name LIKE ",
//     " AND active = ",
//     ""
//   ),
//   params = Vector(DbInt(42), DbString("%alice%"), DbBoolean(true))
// )
frag.sql(SqlDialect.SQLite)
// res9: String = "SELECT * FROM users WHERE id = ? AND name LIKE ? AND active = ?"
frag.params
// res10: IndexedSeq[DbValue] = Vector(
//   DbInt(42),
//   DbString("%alice%"),
//   DbBoolean(true)
// )
```

### JSONB Serialization for Complex Types

When a field's type is not directly representable as a single column — such as `List[A]`, `Map[K, V]`, or a sealed trait with multiple variants — `DbCodecDeriver` automatically uses `DbCodec.jsonb[A]` to store and retrieve the value as a JSON-encoded `TEXT` or `JSONB` column. This keeps complex nested data in a single column without requiring a separate table:

```scala

case class Order(id: Int, tags: List[String], metadata: Map[String, String]) derives DbCodec

// `tags` and `metadata` are encoded via DbCodec.jsonb when read/written through Frag/Repo
val codec = DbCodec[Order]
// codec: DbCodec[Order] = zio.blocks.sql.DbCodecDeriver$$anon$10@2f4254b5
codec.columns
// res12: IndexedSeq[String] = Vector("id", "tags", "metadata")
```

### Optional and Nullable Handling

`Option[A]` and `Maybe[A]` map to a single nullable column. Reading a `NULL` from the database produces `None` or `Maybe.absent`; writing `None` or `Maybe.absent` binds `NULL` to the parameter. For non-optional types, encountering an unexpected `NULL` throws `IllegalStateException` at read time, surfacing schema mismatches immediately rather than silently coercing `NULL` to a default:

```scala

case class Profile(userId: Int, bio: Option[String], avatarUrl: Option[String])
object Profile { implicit val schema: Schema[Profile] = Schema.derived }

// bio and avatar_url become nullable TEXT columns in the generated DDL
val repo = Repo.derived[Profile, Int]("user_id", _.userId)

given DbCon = ???
// repo.insert(Profile(1, None, None)) binds NULL for both optional columns
repo.insert(Profile(1, None, None))
```

## Integration Points

The `zio-blocks-sql` module sits at the intersection of several other ZIO Blocks modules and integrates with the JVM JDBC ecosystem.

The primary dependency is **`zio-blocks-schema`**. `Schema[A]` (specifically its `Reflect` tree) is the source of all compile-time type metadata: field names, types, nullability, and annotations. `DbCodecDeriver` extends `Deriver[DbCodec]`, the same framework used by JSON, Avro, and other codec modules in the schema module's built-in codec suite. The `@Modifier.rename`, `@Modifier.transient`, and `@Modifier.config` annotations attach SQL-specific configuration to individual fields and types without coupling the domain model to the `zio-blocks-sql` module.

The module also depends on **`zio-blocks-maybe`** for `Maybe[A]` support. `Maybe[A]` has distinct absent and present semantics (unlike `Option` which cannot distinguish `Some(null)` from a genuinely absent value), and both `DbCodec` and `DbParam` provide `Maybe[A]` instances alongside `Option[A]`.

For transparent opaque-type support the module integrates with **`As[A, B]`** from `zio-blocks-schema`. If an opaque type has a `DbCodec` for its underlying type and an `As` conversion, `DbCodec` derives an instance for the opaque type automatically via `dbCodecFromAs` without requiring a hand-written codec.

The JDBC layer is isolated behind three abstractions — `DbConnection`, `DbResultReader`, and `DbParamWriter` — so the `JdbcTransactor` can be replaced by an alternate backend (for example, a WebSQL or node-postgres adapter on Scala.js) without changing any application code that depends only on the shared `zio-blocks-sql` module.

The **`zio-blocks-sql-zio` module** provides `TransactorZIO`, a ZIO-aware wrapper around `JdbcTransactor`. It exposes two execution models: blocking wrappers (`TransactorZIO#connect`, `TransactorZIO#transact`) that run synchronous bodies with `ZIO.attemptBlocking`, and effect-aware methods (`TransactorZIO#connectZIO`, `TransactorZIO#transactZIO`) that bracket connections via `ZIO.acquireRelease` so connections are always released even on fiber interruption. A `ZLayer` constructor (`TransactorZIO.layer`) integrates with ZIO's dependency injection system.

Queries can also stream: `Frag#queryStream` and `Frag#queryChunked` return a `zio.blocks.streams.Stream` of row chunks, so a large result set is read in bounded batches instead of loaded whole. See [How to Stream Large Query Results](../../guides/stream-large-query-results.md).

## See Also

- **[SQL-ZIO Integration Reference](../sql-zio.md)** — Reference page for `TransactorZIO` and the ZIO `ZLayer` integration.
- **[Build Your First Database Repository on SQLite](../../guides/your-first-sqlite-repository.md)** — Tutorial that connects to SQLite and builds a `Repo` from a derived schema, end to end.
- **[How to Test Database Code Against SQLite](../../guides/test-database-code-against-sqlite.md)** — Guide to giving each ZIO Test spec its own SQLite database and rolling back writes.
- **[How to Connect to PostgreSQL with a Connection Pool](../../guides/connect-to-postgresql-with-a-connection-pool.md)** — Guide that wires a HikariCP pool into `JdbcTransactor` for PostgreSQL.
- **[How to Stream Large Query Results](../../guides/stream-large-query-results.md)** — Guide to reading a large result set as a stream of row chunks with `queryStream` and `queryChunked`.
- **[How to Write Joins and Inspect the Generated SQL](../../guides/write-joins-and-inspect-generated-sql.md)** — Guide to joining related tables with `SqlQuery` and `Rel`, reading rows as tuples, and printing or dumping the SQL each join generates.
- **[Build a Small JSON Backend from Blocks](../../guides/small-json-backend-from-blocks.md)** — Capstone tutorial that combines `schema`, `config`, `scope`, `sql`, `http-model` and `telemetry` into one small JSON backend.
- **[How to Use the sql Module from a ZIO Application](../../guides/sql-from-zio.md)** — Guide to running queries and transactions as ZIO effects with `TransactorZIO`.
- **[Coming from Magnum or doobie](../../guides/migrate-from-magnum-or-doobie.md)** — Migration guide that maps Magnum and doobie entities, transactors, queries and streams onto the `sql` module.

---

## Rel

`Rel[From, To]` is a **foreign-key link between two tables**: it states that a column of one table equals a column of another, `fromTable.fkColumn = toTable.pkColumn`. You describe each link once as a `Rel` and pass it to the join methods of [`SqlQuery`](./sql-query.md), which turn it into an `ON` condition. `From` is the row type of the table that holds the foreign key, and `To` is the row type of the table it points at.

`Rel` lives in the `zio.blocks.sql.query` package of the `zio-blocks-sql` module, which is published for Scala 3 only (see the [SQL module](./index.md) page). Its key properties are:

- The selector constructors check at compile time that each selector names a field of its table's row type.
- Every `Rel` checks when it is built that both column names are plain SQL identifiers and exist in their tables, and throws an `IllegalArgumentException` otherwise.
- A `Rel` has no direction of its own: `Rel.manyToOne` and `Rel.oneToMany` build the same value, and a query can join it from either table.

`Rel` is a case class with four fields:

```scala
final case class Rel[From, To](
  fromTable: Table[From],
  fkColumn: String,
  toTable: Table[To],
  pkColumn: String
)
```

## Usage

We link each invoice to the customer who placed it:

```scala

case class Customer(customerId: Int, name: String, referrerId: Option[Int])
object Customer { implicit val schema: Schema[Customer] = Schema.derived }

case class Invoice(invoiceId: Int, buyerId: Int, totalCents: Long)
object Invoice { implicit val schema: Schema[Invoice] = Schema.derived }

val customers = Table.derived[Customer]
val invoices  = Table.derived[Invoice]

val invoiceBuyer = Rel.manyToOne(invoices, _.buyerId, customers, _.customerId)
```

We read back the column names the `Rel` resolved and use it in a join:

```scala
println(s"${invoiceBuyer.fromTable.name}.${invoiceBuyer.fkColumn} = ${invoiceBuyer.toTable.name}.${invoiceBuyer.pkColumn}")
// invoice.buyer_id = customer.customer_id
println(SqlQuery.from(invoices).join(invoiceBuyer).sql(SqlDialect.SQLite))
// SELECT t0."invoice_id", t0."buyer_id", t0."total_cents", t1."customer_id", t1."name", t1."referrer_id" FROM "invoice" AS t0 INNER JOIN "customer" AS t1 ON t0."buyer_id" = t1."customer_id"
```

The selectors name Scala fields, `buyerId` and `customerId`, and the `Rel` holds their snake-case column names, `buyer_id` and `customer_id`.

## Creating Values

A `Rel` is built from field selectors, which most code uses, or from column names, for tables whose columns don't follow the row type's field names. The examples below reuse the tables from [Usage](#usage).

### From Field Selectors

`Rel.manyToOne` and `Rel.oneToMany` take the table holding the foreign key, a selector for that field, the table it points at and a selector for its key field. They are macros: each selector must be a plain field access such as `_.buyerId`, and the macro turns the field name into a column name with `SqlNameMapper.SnakeCase`:

```scala
object Rel {
  transparent inline def manyToOne[From, To](
    inline fromTable: Table[From],
    inline fkSelector: From => Any,
    inline toTable: Table[To],
    inline pkSelector: To => Any
  ): Rel[From, To]

  transparent inline def oneToMany[From, To](
    inline fromTable: Table[From],
    inline fkSelector: From => Any,
    inline toTable: Table[To],
    inline pkSelector: To => Any
  ): Rel[From, To]
}
```

A misspelled field fails to compile as any unknown member does. A selector that goes on from a field, such as `_.name.length` or `_.buyerId + 1`, also compiles, a known bug in 0.0.56: the macro uses the first field it touches, so the `Rel` joins on `name` or `buyer_id` and the rest of the expression is dropped without a warning. Pass only a plain field access such as `_.buyerId`, or use the string form described in [From Column Names](#from-column-names). A name that isn't a field of the row type, such as a method called on the row, is rejected by the macro, which lists the valid fields:

```scala
Rel.manyToOne(invoices, _.hashCode, customers, _.customerId)
// error: 
// unknown field 'hashCode' for type MdocApp0.this.Invoice. Valid fields: invoiceId, buyerId, totalCents
```

The two constructors differ only in name. Each puts its first table and selector in `Rel#fromTable` and `Rel#fkColumn`, and its second pair in `Rel#toTable` and `Rel#pkColumn`. Pick the one that reads like the link as you think of it: many invoices point at one customer, and one customer has many invoices. Both links below join the same two columns:

```scala
val customerInvoices = Rel.oneToMany(customers, _.customerId, invoices, _.buyerId)
```

We join each of them to a query that starts from `customers`:

```scala
println(SqlQuery.from(customers).join(invoiceBuyer).sql(SqlDialect.SQLite))
// SELECT t0."customer_id", t0."name", t0."referrer_id", t1."invoice_id", t1."buyer_id", t1."total_cents" FROM "customer" AS t0 INNER JOIN "invoice" AS t1 ON t1."buyer_id" = t0."customer_id"
println(SqlQuery.from(customers).join(customerInvoices).sql(SqlDialect.SQLite))
// SELECT t0."customer_id", t0."name", t0."referrer_id", t1."invoice_id", t1."buyer_id", t1."total_cents" FROM "customer" AS t0 INNER JOIN "invoice" AS t1 ON t0."customer_id" = t1."buyer_id"
```

The `ON` condition always writes the column in `Rel#fkColumn` on the left of `=`, so the two conditions compare the same columns in opposite order and return the same rows.

:::warning
The selector macros always convert the field name to snake case, and ignore `@Modifier.rename`, a known bug in 0.0.56. `Table.derived` honours the annotation, so for a field declared as `@Modifier.rename("buyer_id") customerId: Int` the macro asks for `customer_id`, the table has `buyer_id`, and the `Rel` throws `IllegalArgumentException: FK column invoice.customer_id ... not found in table 'invoice'` when it is built. Build a `Rel` on a renamed column from its column names instead, as in the next section.
:::

### From Column Names

`Rel.manyToOne` and `Rel.oneToMany` also have overloads that take the column names as strings, and the case class itself can be built directly; all three build the same value. Use them for renamed columns, for tables built with a name mapper other than snake case, or for tables constructed at run time:

```scala
object Rel {
  def manyToOne[From, To](fromTable: Table[From], fkColumn: String, toTable: Table[To], pkColumn: String): Rel[From, To]
  def oneToMany[From, To](fromTable: Table[From], fkColumn: String, toTable: Table[To], pkColumn: String): Rel[From, To]
}
```

The string forms give the same `Rel` as the selector form:

```scala
println(Rel.manyToOne(invoices, "buyer_id", customers, "customer_id") == invoiceBuyer)
// true
println(Rel(invoices, "buyer_id", customers, "customer_id") == invoiceBuyer)
// true
```

Nothing checks a string column name at compile time. When the `Rel` is built, each name must be a plain SQL identifier (ASCII letters, digits and underscores, starting with a letter or underscore) that appears in its table's `columns`; anything else throws an `IllegalArgumentException` that lists the table's columns:

```scala
Rel(invoices, "buyer", customers, "customer_id")
// java.lang.IllegalArgumentException: FK column invoice.buyer (validated as 'buyer') not found in table 'invoice' columns: invoice_id, buyer_id, total_cents
// 	at zio.blocks.sql.query.Rel.<init>(Rel.scala:68)
// 	at zio.blocks.sql.query.Rel$.apply(Rel.scala:57)
// 	at repl.MdocSession$MdocApp0.$init$$$anonfun$1(rel.md:96)
```

## Using a Rel in a Query

A `Rel` is passed to `SqlQuery#join`, `SqlQuery#innerJoin`, `SqlQuery#leftJoin`, `SqlQuery#joinLeft` or `SqlQuery#join(rel, kind)`. The query looks up which of the `Rel`'s two tables it already has, keeps that table's alias and joins the other one, so the same `Rel` serves queries that start from either table. The [aliases and join order](./sql-query.md#aliases-and-join-order) section of the `SqlQuery` page describes the rules, including what happens when neither table is in the query yet.

A `Rel` between two columns of the same table is a self-join. The query gives the second copy of the table its own alias and reads the foreign-key column from `t0`:

```scala
val referredBy = Rel.manyToOne(customers, _.referrerId, customers, _.customerId)
```

A left join through it lists each customer next to their referrer, if any:

```scala
println(SqlQuery.from(customers).leftJoin(referredBy).sql(SqlDialect.SQLite))
// SELECT t0."customer_id", t0."name", t0."referrer_id", t1."customer_id", t1."name", t1."referrer_id" FROM "customer" AS t0 LEFT JOIN "customer" AS t1 ON t0."referrer_id" = t1."customer_id"
```

Joined rows decode as tuples, which has pitfalls when tables share column names; [reading joined rows](./sql-query.md#reading-joined-rows) on the `SqlQuery` page covers them. To write a join's SQL to a file at compile time with [`Dump`](./dump.md), build the `Rel` inside the dumped query rather than referring to a `Rel` stored in a `val`, so the macro can see its columns.

## Integration

`Rel` connects the module's tables to its query builder:

- [`Table`](./table.md) supplies the table names and the `columns` that a `Rel` checks its column names against.
- [`SqlNameMapper`](./sql-name-mapper.md) documents the snake-case conversion the selector constructors apply.
- [`SqlQuery`](./sql-query.md) turns each `Rel` into a `JOIN ... ON` clause.
- [How to Write Joins and Inspect the Generated SQL](../../guides/write-joins-and-inspect-generated-sql.md) uses `Rel.manyToOne` and `Rel.oneToMany` in a many-to-one, a one-to-many and a left join.

---

## Repo

`Repo[E, ID]` is a type-safe CRUD repository that provides `all`, `find`, `insert`, `update`, `delete`, and related operations for entities of type `E` identified by a primary key of type `ID`. In the `sql` module's layered architecture, `Repo` sits above `Table[E]` (which it wraps) and below the `Transactor` (which supplies the `DbCon` or `DbTx` context each operation requires). All SQL — `SELECT`, `INSERT`, `UPDATE`, and `DELETE` — is assembled from the `Table`'s column metadata, with identifiers validated and quoted once at construction time; each call assembles its statement from them and binds parameters.

Its declaration and public fields have this structure; instances come from the `Repo.apply` and `Repo.derived` factories, or from subclassing `Repo` with the schema and codec givens in scope:

```scala
abstract class Repo[E, ID] protected (metadata: Repo.Metadata[E, ID]) {
  final val table: Table[E]
  final val idColumn: String
  final val idCodec: DbCodec[ID]
  final val getId: E => ID
  // ... all the CRUD methods are defined here ...
}
```

## Motivation

Writing CRUD SQL by hand requires keeping column names, result-set positions, and prepared-statement parameter counts in sync with your Scala types — a maintenance burden that grows with every added field or renamed column. `Repo` eliminates that burden by deriving all standard CRUD statements from the same `Schema[E]` that already describes your domain type: `Repo.derived` inspects the schema's field names, types, and `@Modifier` annotations once at construction time and quotes the resulting identifiers for reuse.

When the standard operations — select-all, select-by-id, insert, update, delete — cover your needs, `Repo` means zero SQL maintenance. When you need custom queries (filtered selects, joins, aggregations), the `sql"..."` interpolator and `Frag` type compose freely alongside `Repo` inside the same `Transactor#connect` or `Transactor#transact` block, so you never have to choose between a repository pattern and hand-written SQL.

## Usage

The example below shows the full lifecycle: deriving a repository from a `Schema`, setting up the table, writing and reading entities, and performing both bulk and single-entity operations — all inside one `Transactor#transact` block:

```scala

case class User(id: Int, name: String, email: String)
object User {
  implicit val schema: Schema[User] = Schema.derived
}
```

```scala
// The table metadata and quoted identifiers are built here, once, from User's Schema.
val repo = Repo.derived[User, Int]("users", "id", _.id)
val tx   = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)
```

```scala
tx.transact {
  repo.table.createTable(summon[DbTx].dialect).update // CREATE TABLE IF NOT EXISTS users …

  repo.insert(User(1, "Alice", "alice@example.com"))
  repo.insert(User(2, "Bob",   "bob@example.com"))

  val allUsers: List[User] = repo.all          // SELECT id, name, email FROM users
  val one: Maybe[User]     = repo.find(1)      // SELECT … WHERE id = ?
  val exists: Boolean      = repo.exists(99)   // SELECT … WHERE id = ?
  val total: Long          = repo.count        // SELECT COUNT(*) FROM users

  repo.update(User(1, "Alice Smith", "alice.smith@example.com"))
  repo.delete(2)

  // Single multi-row VALUES (…),(…) statement; returns the supplied IDs
  val ids: Seq[Int] = repo.insertAll(Seq(User(3, "Carol", "carol@example.com")))

  // JDBC addBatch/executeBatch; returns total row count
  val n: Int = repo.insertBatch(List(User(4, "Dave", "dave@example.com")))

  (allUsers, one, exists, total, ids, n)
}
// res1: Tuple6[List[User], Maybe[User], Boolean, Long, Seq[Int], Int] = (
//   List(
//     User(id = 1, name = "Alice", email = "alice@example.com"),
//     User(id = 2, name = "Bob", email = "bob@example.com")
//   ),
//   User(id = 1, name = "Alice", email = "alice@example.com"),
//   false,
//   2L,
//   List(3),
//   1
// )
```

Operations inside `Transactor#transact` run under auto-commit disabled; on success the block commits and closes the connection, on any exception it rolls back.

## Construction / Creating Instances

`Repo` exposes four factory methods, three of which auto-derive the underlying `Table[E]` from an implicit `Schema[E]`. The `Repo.derived` overloads differ only in how much naming information is supplied; `Repo.apply` accepts a fully pre-built `Table[E]` for cases where you have already constructed or derived a `Table` manually.

### `Repo.derived` — Derive with ID Column and Getter

This overload derives the `Table[E]` from the implicit `Schema[E]` and uses the caller-supplied `idColumn` name and `getId` getter. The table name is computed by applying the default singular-snake-case policy to the type name — for example, `User` becomes `"user"` and `BlogPost` becomes `"blog_post"`.

```scala
object Repo {
  def derived[E, ID](idColumn: String, getId: E => ID)(using schema: Schema[E], idCodec: DbCodec[ID]): Repo[E, ID]
}
```

A `DbCodec[ID]` instance is resolved automatically for all primitive ID types (`Int`, `Long`, `String`, `UUID`, and others) without any additional imports. The typical usage looks like this:

```scala

case class Article(id: Int, title: String, body: String)
object Article {
  implicit val schema: Schema[Article] = Schema.derived
}
```

```scala
val repo = Repo.derived[Article, Int]("id", _.id)
```

```scala
repo.table.name
// res3: String = "article"
repo.table.codec.columns
// res4: IndexedSeq[String] = Vector("id", "title", "body")
```

:::caution
The `idColumn` string must exactly match a column name in the derived table (after `SqlNameMapper` applies, so `camelCase` fields become `snake_case`). A mismatch throws `IllegalArgumentException` (via `require`) at construction time — not at query time.
:::

### `Repo.derived` — Derive with Table Name, ID Column, and Getter

This overload is identical to the previous one except that it accepts an explicit `tableName`, bypassing the naming policy entirely. Use it when the default snake-case convention does not match the actual database table name — for example, when migrating from a legacy schema.

```scala
object Repo {
  def derived[E, ID](tableName: String, idColumn: String, getId: E => ID)(using schema: Schema[E], idCodec: DbCodec[ID]): Repo[E, ID]
}
```

The supplied `tableName` is used verbatim (after SQL identifier validation) in all generated statements:

```scala

case class OrderLine(lineId: Int, qty: Int, unitPrice: BigDecimal)
object OrderLine {
  implicit val schema: Schema[OrderLine] = Schema.derived
}
```

```scala
// Overrides the default "order_line" table name with the legacy one
val repo = Repo.derived[OrderLine, Int]("tbl_order_lines", "line_id", _.lineId)
```

```scala
repo.table.name
// res6: String = "tbl_order_lines"
```

### `Repo.derived` — Fully Automatic Derivation

When called without any term arguments, `Repo.derived` requires an additional implicit `Schema[ID]` and locates the ID field in `E` using a four-priority rule, trying each in order until one matches:

1. A field annotated `@Modifier.id` whose type matches `ID`.
2. The unique field (among all fields) whose type matches `ID`, if there is exactly one.
3. A field literally named `"id"` whose type matches `ID`.
4. A field named `<entity>Id` (for example, `userId` for entity `User`) whose type matches `ID`.

The ID column name respects that field's `@Modifier.rename` annotation if present, or applies `SqlNameMapper.SnakeCase` to the field name otherwise. The table name uses the default naming policy.

```scala
object Repo {
  def derived[E, ID](using schema: Schema[E], idSchema: Schema[ID], idCodec: DbCodec[ID]): Repo[E, ID]
}
```

This is the most concise form when the entity type has exactly one field of the `ID` type:

```scala

case class Tag(id: Long, label: String)
object Tag {
  implicit val schema: Schema[Tag] = Schema.derived
}
```

```scala
// Inspects Tag's Schema, finds the unique Long field "id", maps it to column "id"
val repo = Repo.derived[Tag, Long]
```

```scala
repo.idColumn
// res8: String = "id"
```

:::caution
`Repo.derived` (fully auto) throws `IllegalArgumentException` at runtime if `E` is not a case class, if no field matches any of the four priorities, or if a priority level itself is ambiguous — multiple `@Modifier.id`-annotated fields of type `ID`, or multiple type-matching fields when none of them is named `"id"` or `<entity>Id`. When any ambiguity is possible, annotate the intended field with `@Modifier.id`, or prefer the explicit overload that names the ID column and getter directly.
:::

### `Repo.apply` — Construct from an Explicit Table

`Repo.apply` is the base constructor. It accepts a fully built `Table[E]` together with the ID column name, codec, and getter. Use it when you already have a `Table` — for example, one obtained from `Table.derived` — and want precise control over all four components.

```scala
object Repo {
  def apply[E, ID](table: Table[E], idColumn: String, idCodec: DbCodec[ID], getId: E => ID): Repo[E, ID]
}
```

Combining `Table.derived` with `Repo.apply` gives access to DDL generation via `Table#createTable` and `Table#dropTable` while also enabling full CRUD:

```scala

case class Category(categoryId: Int, name: String)
object Category {
  implicit val schema: Schema[Category] = Schema.derived
}

val table = Table.derived[Category]
val repo  = Repo(table, "category_id", DbCodec[Int], _.categoryId)

given DbCon = ???

repo.table.createTable(summon[DbCon].dialect).update  // DDL from Table
repo.all                                              // SELECT from Repo
```

## Core Operations

Every `Repo` method requires a `DbCon` (or `DbTx`) given in scope, which is supplied by the enclosing `Transactor#connect` or `Transactor#transact` block. Using `DbTx` (from `Transactor#transact`) means all writes in a block are committed atomically or rolled back together on exception.

### Read Operations

The read operations query the database without modifying it. `Repo#all`, `Repo#findAll`, `Repo#find`, and `Repo#pageAfter` decode and return entity values; `Repo#exists` and `Repo#count` return summary information without decoding full rows.

#### `all` — Retrieve All Rows

`Repo#all` executes `SELECT "<columns>" FROM ""` and decodes every result-set row into an `E` using the entity's `DbCodec`, returning all rows as a `List[E]` in database-native order.

```scala
class Repo[E, ID] {
  def all(using con: DbCon): List[E]
}
```

Inside a `Transactor#connect` or `Transactor#transact` block, the call requires no arguments:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

// Inside a Transactor#connect or #transact block:
val users: List[User] = repo.all
```

:::caution
`Repo#all` loads the entire table into memory. For large tables, complement `Repo` with a custom `Frag` query that includes `LIMIT` and `OFFSET` clauses.
:::

#### `findAll` — Retrieve Rows by a Set of Primary Keys

`Repo#findAll` executes `SELECT "<columns>" FROM "" WHERE "<idColumn>" IN (...)` for the given IDs and decodes every matching row into an `E`. It returns an empty `List` immediately, without executing any SQL, when `ids` is empty.

```scala
class Repo[E, ID] {
  def findAll(ids: Iterable[ID])(using con: DbCon): List[E]
}
```

Use it to batch-fetch a known set of rows in a single round-trip instead of calling `Repo#find` in a loop:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

val users: List[User] = repo.findAll(List(1, 2, 3))
```

#### `find` — Find a Row by Primary Key

`Repo#find` executes `SELECT "<columns>" FROM "" WHERE "<idColumn>" = ?`, binding the ID through `idCodec`. It returns `Maybe.absent` if no row with the given key exists, or `Maybe(entity)` if a row is found.

```scala
class Repo[E, ID] {
  def find(id: ID)(using con: DbCon): Maybe[E]
}
```

The ID value is bound as a parameterized `?` — no string formatting or concatenation:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

val alice: Maybe[User] = repo.find(1)
```

#### `exists` — Check Whether a Row Exists

`Repo#exists` returns `true` when a row with the given primary key exists in the table. It delegates to `Repo#find` and tests whether the result is defined, running a single parameterized `SELECT`.

```scala
class Repo[E, ID] {
  def exists(id: ID)(using con: DbCon): Boolean
}
```

Use `exists` when you only need to confirm presence without loading the full entity:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

val exists: Boolean = repo.exists(99)
```

#### `count` — Count All Rows

`Repo#count` executes `SELECT COUNT(*) FROM ""` and returns the row count as a `Long`. The result is `0L` when the table is empty.

```scala
class Repo[E, ID] {
  def count(using con: DbCon): Long
}
```

The `Long` result avoids boxing and handles tables with more than `Int.MaxValue` rows correctly:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

val total: Long = repo.count
```

#### `pageAfter` — Read the Next Page of Rows by Primary Key

`Repo#pageAfter` implements keyset pagination: it returns at most `limit` rows whose primary key is strictly greater than `cursorId`, ordered by primary key ascending. It runs `SELECT … FROM "" WHERE "<id>" > ? ORDER BY "<id>" ASC LIMIT <limit>`, so the cursor row itself never repeats, and a cursor equal to the last ID returns an empty list.

```scala
class Repo[E, ID] {
  def pageAfter(cursorId: ID, limit: Int)(using con: DbCon): List[E]
}
```

To read the table two rows at a time, we pass the ID of the last row on one page as the cursor for the next:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

val firstPage: List[User]  = repo.pageAfter(0, 2)
val secondPage: List[User] = firstPage.lastOption.fold(List.empty[User])(last => repo.pageAfter(last.id, 2))
```

:::caution
`Repo#pageAfter` throws `IllegalArgumentException` when `limit` is zero or negative.
:::

See also the [How to Upsert Rows and Paginate with Keyset Cursors](../../guides/upsert-and-paginate-with-keyset-cursors.md) guide, which walks a table page by page with `Repo#pageAfter` and over your own query with `Frag.keysetAfter`.

### Write Operations

The write operations insert, update, or delete rows in the database. `Repo#insert`, `Repo#insertBatch`, `Repo#insertAll`, `Repo#insertOrUpdate`, `Repo#insertOrUpdateBatch`, `Repo#update`, `Repo#delete`, `Repo#deleteAll`, and `Repo#clear` each return an `Int` row count (`insertAll` returns the inserted IDs instead); `Repo#insertReturning` is the exception and returns the full inserted entity.

#### `insert` — Insert a Single Entity

`Repo#insert` encodes the entity with `DbCodec[E]` and executes `INSERT INTO "" ("<columns>") VALUES (?, …, ?)`, returning the number of affected rows — normally 1 on success.

```scala
class Repo[E, ID] {
  def insert(entity: E)(using con: DbCon): Int
}
```

Each call builds the statement from the validated, pre-quoted identifiers and binds fresh parameter values from the entity:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

val rowsAffected: Int = repo.insert(User(1, "Alice", "alice@example.com"))
```

#### `insertReturning` — Insert and Return the Inserted Entity

`Repo#insertReturning` inserts the entity with its own ID, retrieves the first column of the driver's generated keys through JDBC's `getGeneratedKeys`, and re-fetches the full row by calling `Repo#find` with that key. If the driver returns no generated key, or no row has that key as its ID, it falls back to `Repo#find` with `Repo#getId(entity)`.

```scala
class Repo[E, ID] {
  def insertReturning(entity: E)(using con: DbCon): E
}
```

The insert always writes every column, the ID included, so the database never assigns the key: `insertReturning` is `Repo#insert` followed by a read of the stored row, and the entity must carry the ID it should have. Passing a placeholder such as `0` stores `0`, and a second placeholder insert violates the primary key:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

// The entity carries its own ID; the returned entity is the row read back from the table.
val inserted: User = repo.insertReturning(User(1, "Bob", "bob@example.com"))
```

:::caution
In 0.0.56 the key is read from the first generated-key column, which is not always the ID column (on SQLite it is the `rowid`), so `insertReturning` can return another row, or fail on PostgreSQL; this is a known bug in 0.0.56. `insertReturning` also throws `NoSuchElementException` if the inserted row cannot be found after the insert.
:::

#### `insertBatch` — Batch-Insert Multiple Entities

`Repo#insertBatch` inserts a collection of entities using JDBC batch execution (`addBatch` / `executeBatch`). Sending all parameter sets to the driver in a single round-trip is significantly faster than calling `insert` in a loop for large collections. It returns the total number of affected rows across all batched statements.

```scala
class Repo[E, ID] {
  def insertBatch(entities: Iterable[E])(using con: DbCon): Int
}
```

The method returns 0 immediately when the input is empty, without opening a prepared statement:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

val users = List(
  User(1, "Alice", "alice@example.com"),
  User(2, "Bob",   "bob@example.com"),
  User(3, "Carol", "carol@example.com")
)
val rowsAffected: Int = repo.insertBatch(users)
```

`Repo#insertBatch` and `Repo#insertAll` solve different problems: `Repo#insertBatch` uses JDBC's batch protocol (one prepared statement, one `addBatch` call per entity, and a single `executeBatch` call) and returns a row count; `Repo#insertAll` constructs a single multi-row `VALUES (…), (…)` SQL statement (one round-trip) and returns the primary keys in input order. For large collections or when relying on database-generated keys, prefer `insertBatch`. For moderate-sized collections where the caller controls the keys and wants a single SQL statement, prefer `insertAll`.

#### `insertAll` — Multi-Row Insert Returning Primary Keys

`Repo#insertAll` assembles a single `INSERT INTO "" ("<columns>") VALUES (?, …, ?), …, (?, …, ?)` statement covering all rows and executes it in one database round-trip. It then extracts the primary keys from the input entities via `getId` and returns them in input order.

```scala
class Repo[E, ID] {
  def insertAll(rows: Seq[E])(using con: DbCon): Seq[ID]
}
```

Because the generated SQL grows with the number of rows, `insertAll` is best suited for moderate-sized batches where the caller controls the primary keys:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

val newUsers = Seq(
  User(10, "Dave", "dave@example.com"),
  User(11, "Eve",  "eve@example.com")
)
// Returns Seq(10, 11) — the IDs extracted from the input entities via getId
val ids: Seq[Int] = repo.insertAll(newUsers)
```

:::caution
`Repo#insertAll` throws `IllegalArgumentException` for an empty `Seq`. Always verify the input is non-empty before calling it, or use `Repo#insertBatch`, which accepts empty collections and returns 0 without error.
:::

#### `insertOrUpdate` — Insert or Update by Primary Key

`Repo#insertOrUpdate` writes an entity whether or not its row already exists, an operation often called an upsert. It runs `INSERT INTO "" (…) VALUES (…) ON CONFLICT ("<id>") DO UPDATE SET …`: when no row has the entity's ID, it inserts one; when a row exists, it overwrites every non-ID column with the entity's values. It returns the affected row count, normally 1 on both paths.

```scala
class Repo[E, ID] {
  def insertOrUpdate(entity: E)(using con: DbCon): Int
}
```

The same call creates a user the first time and replaces its name and email the second time:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

val inserted: Int = repo.insertOrUpdate(User(1, "Alice", "alice@example.com"))
val updated: Int  = repo.insertOrUpdate(User(1, "Alice Smith", "alice.smith@example.com"))
```

:::caution
Known bug in 0.0.56: `Table#createTable` declares no `PRIMARY KEY`, so `Repo#insertOrUpdate` fails on a table that `Table#createTable` creates. `ON CONFLICT` needs a `PRIMARY KEY` or `UNIQUE` constraint on the ID column, and SQLite reports `ON CONFLICT clause does not match any PRIMARY KEY or UNIQUE constraint`. The method also throws `IllegalArgumentException` when the table has no columns besides the ID. Until the bug is fixed, create such a table with a hand-written statement, as shown below.
:::

To use `Repo#insertOrUpdate`, we create the table ourselves and declare the ID column as the primary key:

```scala

given DbCon = ???

Frag.literal("CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT NOT NULL, email TEXT NOT NULL)").update
```

#### `insertOrUpdateBatch` — Upsert Many Entities in One Batch

`Repo#insertOrUpdateBatch` runs the same `INSERT … ON CONFLICT … DO UPDATE` statement as `Repo#insertOrUpdate` for every entity, using JDBC batch execution like `Repo#insertBatch`. It returns the total affected row count, returns 0 without touching the database when the input is empty. It has the same `PRIMARY KEY` requirement as `Repo#insertOrUpdate` and likewise throws `IllegalArgumentException` when the table has no columns besides the ID.

```scala
class Repo[E, ID] {
  def insertOrUpdateBatch(entities: Iterable[E])(using con: DbCon): Int
}
```

To insert new users and update existing ones in one batch:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

val rowsAffected: Int = repo.insertOrUpdateBatch(
  List(
    User(1, "Alice Smith", "alice.smith@example.com"),
    User(2, "Bob",         "bob@example.com")
  )
)
```

See also the [How to Upsert Rows and Paginate with Keyset Cursors](../../guides/upsert-and-paginate-with-keyset-cursors.md) guide, which covers `Repo#insertOrUpdate`, `Repo#insertOrUpdateBatch`, and choosing which columns an upsert changes.

#### `update` — Update an Entity's Non-ID Columns

`Repo#update` executes `UPDATE "" SET "<col1>" = ?, …, "<colN>" = ? WHERE "<idColumn>" = ?` for all non-ID columns of the entity, identifying the target row by its primary key. It returns the number of affected rows — 0 when no row with that ID exists.

```scala
class Repo[E, ID] {
  def update(entity: E)(using con: DbCon): Int
}
```

Only non-ID columns appear in the `SET` clause; the ID column appears only in the `WHERE` clause, so it is never overwritten:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

val rowsAffected: Int = repo.update(User(1, "Alice Smith", "alice.smith@example.com"))
```

:::note
`Repo#update` returns 0 when the entity type has only an ID column and no other updatable fields. In that case the generated `SET` clause would be empty, and `Repo` skips the statement entirely.
:::

#### `delete` — Delete by Primary Key

`Repo#delete` executes `DELETE FROM "" WHERE "<idColumn>" = ?`, binding the ID through `idCodec`. It returns the number of deleted rows — 0 if no row with the given ID exists.

```scala
class Repo[E, ID] {
  def delete(id: ID)(using con: DbCon): Int
}
```

The ID value is bound as a parameterized `?`, not interpolated into the SQL string:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

val rowsAffected: Int = repo.delete(42)
```

To delete by an entity value rather than a bare ID, extract the key with `Repo#getId` first: `repo.delete(repo.getId(user))`.

#### `deleteAll` — Delete Rows by a Set of Primary Keys

`Repo#deleteAll` executes `DELETE FROM "" WHERE "<idColumn>" IN (...)` for the given IDs in a single round-trip and returns the total number of deleted rows. It returns `0` immediately, without executing any SQL, when `ids` is empty.

```scala
class Repo[E, ID] {
  def deleteAll(ids: Iterable[ID])(using con: DbCon): Int
}
```

Use it to batch-delete a known set of rows instead of calling `Repo#delete` in a loop:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

val rowsAffected: Int = repo.deleteAll(List(1, 2, 3))
```

#### `clear` — Remove All Rows

`Repo#clear` executes `DELETE FROM ""` without a `WHERE` clause and returns the number of deleted rows.

```scala
class Repo[E, ID] {
  def clear()(using con: DbCon): Int
}
```

Despite the different name, this method issues a `DELETE FROM` statement rather than a SQL `TRUNCATE`, so the operation participates in transactions and the returned row count is exact:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo = Repo.derived[User, Int]("users", "id", _.id)
given DbCon = ???

val rowsDeleted: Int = repo.clear()
```

:::note
`DELETE FROM` is transactional and precise but may be slower than `TRUNCATE` for very large tables on databases that support the `TRUNCATE` statement. When performance for bulk deletes is critical, issue a `TRUNCATE` via a custom `Frag` query instead.
:::

See also the [Build Your First Database Repository on SQLite](../../guides/your-first-sqlite-repository.md) tutorial, which builds a `Repo` step by step and runs it against SQLite.

---

## SqlDialect

`SqlDialect` tells the framework how to render SQL for a specific database. Two dialects are built in: `PostgreSQL` and `SQLite`.

## Core API

```scala
sealed trait SqlDialect {
  def name: String
  def typeName(dbValue: DbValue): String
  def paramPlaceholder(index: Int): String
}

object SqlDialect {
  case object PostgreSQL extends SqlDialect
  case object SQLite     extends SqlDialect
}
```

## Usage

You can specify the dialect when creating your `Transactor`:

```scala

val txPostgres = JdbcTransactor.fromUrl("jdbc:postgresql://...", SqlDialect.PostgreSQL)
val txSqlite   = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)
```

The dialect is available through `DbCon#dialect` inside a transaction:

```scala

val tx = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)
// tx: JdbcTransactor = zio.blocks.sql.JdbcTransactor@79c8941e

tx.connect {
  val dialect = summon[DbCon].dialect

  (
    dialect.typeName(DbValue.DbInt(0)),
    dialect.typeName(DbValue.DbUUID(new java.util.UUID(0L, 0L))),
    dialect.paramPlaceholder(1)
  )
}
// res2: Tuple3[String, String, String] = ("INTEGER", "TEXT", "?")
```

## Type Mapping

PostgreSQL and SQLite map Scala types to SQL differently:

| Type       | PostgreSQL    | SQLite    |
|------------|---------------|-----------|
| Long       | `BIGINT`      | `INTEGER` |
| Boolean    | `BOOLEAN`     | `INTEGER` |
| Instant    | `TIMESTAMPTZ` | `TEXT`    |
| LocalDate  | `DATE`        | `TEXT`    |
| UUID       | `UUID`        | `TEXT`    |
| BigDecimal | `NUMERIC`     | `TEXT`    |
| Bytes      | `BYTEA`       | `BLOB`    |

PostgreSQL uses native types; SQLite uses `INTEGER` for all integers and `TEXT` for temporal/UUID types.

---

## SqlLogger

`SqlLogger` is a callback interface that reports SQL execution events. Each `Frag` operation, and each `Repo` operation (one event for a whole batch, with empty `params`), calls either `onSuccess` or `onError` with details about the execution: rendered SQL, bound parameters, duration, and row count. Savepoint, commit and rollback statements report nothing.

Key points:

- **Synchronous** — Callbacks run on the JDBC thread immediately after statement completion. Keep implementations fast; use a background thread for heavy I/O.
- **SqlLogger.noop** — Pre-built no-op instance that discards all events. Default when no custom logger is configured.

## Core API

```scala
trait SqlLogger {
  def onSuccess(event: SqlLogger.SuccessEvent): Unit
  def onError(event: SqlLogger.ErrorEvent): Unit
}
```

## Usage

Implement the trait to observe SQL execution:

```scala

val myLogger: SqlLogger = new SqlLogger {
  def onSuccess(event: SqlLogger.SuccessEvent): Unit =
    println(s"OK (${event.duration.toMillis}ms, ${event.rowCount} rows): ${event.sql}")

  def onError(event: SqlLogger.ErrorEvent): Unit =
    println(s"FAIL: ${event.error.getMessage}")
}
// myLogger: SqlLogger = repl.MdocSession$MdocApp0$$anon$1@5b1912b
```

Pass the logger when creating a `JdbcTransactor`:

```scala

val dataSource: javax.sql.DataSource = ???
val myLogger: SqlLogger = SqlLogger.noop

val tx = new JdbcTransactor(() => dataSource.getConnection(), SqlDialect.PostgreSQL, myLogger)
```

Use the predefined no-op logger when you don't need logging:

```scala

val dataSource: javax.sql.DataSource = ???
val tx: JdbcTransactor = JdbcTransactor.fromDataSource(dataSource, SqlDialect.PostgreSQL)
```

---

## SqlNameMapper

`SqlNameMapper` converts Scala field names to SQL column names. Three implementations are built in: `SnakeCase` (default, converts `firstName` to `first_name`), `Identity` (no change), and `Custom` (arbitrary function).

## Core API

The simplified structural shape of `SqlNameMapper` is:

```scala
sealed trait SqlNameMapper extends (String => String)

object SqlNameMapper {
  case object SnakeCase                        extends SqlNameMapper
  case object Identity                         extends SqlNameMapper
  final case class Custom(f: String => String) extends SqlNameMapper
}
```

## Usage

Call a mapper like a function:

```scala

SqlNameMapper.SnakeCase("firstName")
// res1: String = "first_name"
SqlNameMapper.SnakeCase("userID")
// res2: String = "user_id"
SqlNameMapper.Identity("firstName")
// res3: String = "firstName"

val upper = SqlNameMapper.Custom(_.toUpperCase)
// upper: Custom = Custom(
//   repl.MdocSession$MdocApp0$$Lambda/0x000000004a5bd000@15d75b21
// )
upper("firstName")
// res4: String = "FIRSTNAME"
```

Use a custom mapper when deriving codecs:

```scala

case class Order(orderId: Int, totalAmount: BigDecimal)
object Order { given schema: Schema[Order] = Schema.derived }

// Use UPPERCASE column names instead of snake_case
val upperDeriver = DbCodecDeriver.withColumnNameMapper(SqlNameMapper.Custom(_.toUpperCase))
// upperDeriver: DbCodecDeriver = zio.blocks.sql.DbCodecDeriver@c94a0ec
val codec: DbCodec[Order] = Order.schema.deriving(upperDeriver).derive
// codec: DbCodec[Order] = zio.blocks.sql.DbCodecDeriver$$anon$10@296e0db
codec.columns
// res6: IndexedSeq[String] = Vector("ORDERID", "TOTALAMOUNT")
```

## Key Points

For full control over individual field names, use `@Modifier.rename("column_name")` on specific fields without changing the global mapper. It takes precedence over any `SqlNameMapper` in use.

---

## SqlQuery

`SqlQuery[A]` is an **immutable `SELECT` builder** in the `sql` module: it starts from a [`Table[A]`](./table.md), adds joins described by [`Rel`](./rel.md) values, `WHERE` conditions, grouping, sorting and paging, and renders the result to a [`Frag`](./frag.md) that runs like any `sql"..."` fragment. It writes the parts of a join that are tedious and easy to get wrong by hand: the full column list of every table, the table aliases and each `ON` condition. The type parameter `A` is the row type of the table the query starts from; joins don't change it, and you choose the type each row decodes into when you run the rendered `Frag`.

`SqlQuery` lives in the `zio.blocks.sql.query` package of the `zio-blocks-sql` module, which is published for Scala 3 only, for the JVM and Scala.js (see the [SQL module](./index.md) page for the dependency). Its key properties are:

- Every column is selected, from every table, in schema order; there is no projection to a subset of columns.
- Every table and column name is checked as a plain SQL identifier and double-quoted.
- Table aliases are deterministic: `t0` is the `FROM` table, and `t1`, `t2` and so on are the joined tables in the order you add them.
- Values never become SQL text: conditions are `Frag`s, so interpolated values bind as `?` parameters.
- Each builder method returns a new `SqlQuery`; the original is unchanged.

The case class and its companion have this shape:

```scala
final case class SqlQuery[A] private[query] (
  source: Table[A],
  joins: Vector[JoinNode],
  filters: Vector[Frag],
  groupBy: Vector[String],
  having: Option[Frag],
  orderBy: Vector[OrderBy],
  limit: Option[Int],
  offset: Option[Int]
)

object SqlQuery {
  def from[A](table: Table[A]): SqlQuery[A]
}

final case class OrderBy(column: String, direction: SortOrder)

sealed trait JoinKind
object JoinKind {
  case object Inner extends JoinKind
  case object Left  extends JoinKind
}

sealed trait SortOrder
object SortOrder {
  case object Asc  extends SortOrder
  case object Desc extends SortOrder
}
```

`SqlQuery` is not the query DSL of the [Query DSL with Reified Optics](../../guides/query-dsl-reified-optics.md) guides, which build their own SQL generator from `SchemaExpr` optics. For a task-oriented walkthrough of joins, see [How to Write Joins and Inspect the Generated SQL](../../guides/write-joins-and-inspect-generated-sql.md).

## Usage

We join each invoice to the customer who placed it, keep the invoices above 100.00, newest first, print the SQL and run the query on a temporary SQLite database:

```scala

case class Customer(customerId: Int, name: String)
object Customer { implicit val schema: Schema[Customer] = Schema.derived }

case class Invoice(invoiceId: Int, buyerId: Int, year: Int, totalCents: Long)
object Invoice { implicit val schema: Schema[Invoice] = Schema.derived }

val customers = Table.derived[Customer]
val invoices  = Table.derived[Invoice]

val largeInvoices = SqlQuery
  .from(invoices)
  .join(Rel.manyToOne(invoices, _.buyerId, customers, _.customerId))
  .where(sql"""t0."total_cents" > ${10000L}""")
  .orderBy("year", SortOrder.Desc)

val usageDb = java.nio.file.Files.createTempFile("usage", ".db")
val usageTx = JdbcTransactor.fromUrl(s"jdbc:sqlite:$usageDb", SqlDialect.SQLite)

usageTx.connect {
  customers.createTable(SqlDialect.SQLite).update
  invoices.createTable(SqlDialect.SQLite).update
  (Frag.literal("INSERT INTO customer VALUES ") ++ Frag.values(List(Customer(1, "Ada"), Customer(2, "Bahram")))).update
  (Frag.literal("INSERT INTO invoice VALUES ") ++ Frag.values(List(Invoice(101, 1, 2025, 12000L), Invoice(102, 2, 2026, 4500L), Invoice(103, 2, 2026, 25000L)))).update
}
```

The rendered SQL lists the columns of both tables, invoices first, so each row decodes as an `(Invoice, Customer)` tuple:

```scala
println(largeInvoices.sql(SqlDialect.SQLite))
// SELECT t0."invoice_id", t0."buyer_id", t0."year", t0."total_cents", t1."customer_id", t1."name" FROM "invoice" AS t0 INNER JOIN "customer" AS t1 ON t0."buyer_id" = t1."customer_id" WHERE t0."total_cents" > ? ORDER BY t0."year" DESC

usageTx.connect {
  largeInvoices.toFrag(SqlDialect.SQLite).query[(Invoice, Customer)].foreach(println)
}
// (Invoice(103,2,2026,25000),Customer(2,Bahram))
// (Invoice(101,1,2025,12000),Customer(1,Ada))
```

The rest of this page covers each part of that query in turn.

## Creating Queries

The examples in the rest of this page share one small shop database. Customers place invoices, each invoice has lines, and a customer may have been referred by another customer. Every key column has a name that no other table uses, which matters when joined rows are decoded (see [Reading Joined Rows](#reading-joined-rows)):

```scala

case class Customer(customerId: Int, name: String, city: String, referrerId: Option[Int])
object Customer { implicit val schema: Schema[Customer] = Schema.derived }

case class Invoice(invoiceId: Int, buyerId: Int, year: Int, totalCents: Long)
object Invoice { implicit val schema: Schema[Invoice] = Schema.derived }

case class InvoiceLine(lineId: Int, invoiceRef: Int, product: String, quantity: Int)
object InvoiceLine { implicit val schema: Schema[InvoiceLine] = Schema.derived }

val customers = Table.derived[Customer]
val invoices  = Table.derived[Invoice]
val lines     = Table.derived[InvoiceLine]

val invoiceBuyer = Rel.manyToOne(invoices, _.buyerId, customers, _.customerId)
val lineInvoice  = Rel.manyToOne(lines, _.invoiceRef, invoices, _.invoiceId)

val dbFile = java.nio.file.Files.createTempFile("shop", ".db")
val tx     = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

tx.connect {
  customers.createTable(SqlDialect.SQLite).update
  invoices.createTable(SqlDialect.SQLite).update
  lines.createTable(SqlDialect.SQLite).update

  val shopCustomers = List(
    Customer(1, "Ada", "Tehran", None),
    Customer(2, "Bahram", "Shiraz", Some(1)),
    Customer(3, "Chloe", "Berlin", Some(1)),
    Customer(4, "Dara", "Tehran", Some(2))
  )
  val shopInvoices = List(
    Invoice(101, 1, 2025, 12000L),
    Invoice(102, 1, 2026, 4500L),
    Invoice(103, 2, 2026, 25000L),
    Invoice(104, 3, 2024, 8000L)
  )
  val shopLines = List(
    InvoiceLine(1, 101, "Keyboard", 2),
    InvoiceLine(2, 101, "Mouse", 1),
    InvoiceLine(3, 103, "Monitor", 1),
    InvoiceLine(4, 104, "Cable", 4)
  )
  (Frag.literal("INSERT INTO customer VALUES ") ++ Frag.values(shopCustomers)).update
  (Frag.literal("INSERT INTO invoice VALUES ") ++ Frag.values(shopInvoices)).update
  (Frag.literal("INSERT INTO invoice_line VALUES ") ++ Frag.values(shopLines)).update
}
```

The database is a temporary file rather than `jdbc:sqlite::memory:`, because each `Transactor#connect` opens a new connection and an in-memory SQLite database starts empty on every connection.

`SqlQuery.from` is the only way to create a query; the case class constructor is private. It takes the `Table` to select from:

```scala
object SqlQuery {
  def from[A](table: Table[A]): SqlQuery[A]
}
```

A query with nothing added selects every column of that table, aliased as `t0`:

```scala
println(SqlQuery.from(customers).sql(SqlDialect.SQLite))
// SELECT t0."customer_id", t0."name", t0."city", t0."referrer_id" FROM "customer" AS t0
```

## Core Operations

The builder methods fall into four groups: joining tables, filtering rows, grouping and sorting, and rendering the result. Every method returns a new `SqlQuery[A]`, so a query can be stored in a `val` and extended in several directions.

### Joining Tables

A join adds a table to the query through a `Rel`, the description of a foreign-key link between two tables. This section covers the join methods, how aliases and `ON` conditions are chosen, self-joins, and how to decode the joined rows.

#### Inner and Left Joins

`SqlQuery#innerJoin` adds an `INNER JOIN`, and `SqlQuery#leftJoin` adds a `LEFT JOIN`. `SqlQuery#join(rel)` is the same method as `innerJoin` under another name, and `SqlQuery#joinLeft` is the same as `leftJoin`. `SqlQuery#join(rel, kind)` takes the kind as a [`JoinKind`](#joinkind-and-sortorder) value, which suits code that chooses it at run time:

```scala
final case class SqlQuery[A] {
  def innerJoin[From, To](rel: Rel[From, To]): SqlQuery[A]
  def join[From, To](rel: Rel[From, To]): SqlQuery[A]
  def leftJoin[From, To](rel: Rel[From, To]): SqlQuery[A]
  def joinLeft[From, To](rel: Rel[From, To]): SqlQuery[A]
  def join[From, To](rel: Rel[From, To], kind: JoinKind): SqlQuery[A]
}
```

The five methods build equal queries for the same kind:

```scala
println(SqlQuery.from(invoices).innerJoin(invoiceBuyer) == SqlQuery.from(invoices).join(invoiceBuyer))
// true
println(SqlQuery.from(customers).leftJoin(invoiceBuyer) == SqlQuery.from(customers).join(invoiceBuyer, JoinKind.Left))
// true
```

A left join keeps the customers who have no invoices. Dara has none, so her row carries SQL `NULL` in every `invoice` column:

```scala
println(SqlQuery.from(customers).leftJoin(invoiceBuyer).sql(SqlDialect.SQLite))
// SELECT t0."customer_id", t0."name", t0."city", t0."referrer_id", t1."invoice_id", t1."buyer_id", t1."year", t1."total_cents" FROM "customer" AS t0 LEFT JOIN "invoice" AS t1 ON t1."buyer_id" = t0."customer_id"
```

#### Aliases and Join Order

Each join gets the next alias, `t1`, `t2` and so on, and its `ON` condition compares the `Rel`'s foreign-key column with its key column. A `Rel` links two tables, and the builder looks up which of them is already in the query: that side keeps its alias, and the other side is the table the join adds. So the same `Rel` works from either end. Starting from `customers` adds `invoice` as `t1`, and the next `Rel` attaches `invoice_line` to `t1`:

```scala
val customerLines = SqlQuery.from(customers).join(invoiceBuyer).join(lineInvoice)
```

The rendered SQL has one `ON` condition per join, each comparing a column of the new alias with a column of a table already joined:

```scala
println(customerLines.sql(SqlDialect.SQLite))
// SELECT t0."customer_id", t0."name", t0."city", t0."referrer_id", t1."invoice_id", t1."buyer_id", t1."year", t1."total_cents", t2."line_id", t2."invoice_ref", t2."product", t2."quantity" FROM "customer" AS t0 INNER JOIN "invoice" AS t1 ON t1."buyer_id" = t0."customer_id" INNER JOIN "invoice_line" AS t2 ON t2."invoice_ref" = t1."invoice_id"
```

When both tables of a `Rel` are already in the query, the `Rel`'s second table (the one holding the key column) is joined again under a new alias.

:::caution
The builder doesn't check that a `Rel` touches a table already in the query. When neither of its tables is there, it reads the `Rel`'s foreign-key column from `t0` and joins the `Rel`'s second table. The SQL is then wrong, and only the database reports it. Here, `lineInvoice` links `invoice_line` and `invoice`, but the query only has `customer`:

```scala
println(SqlQuery.from(customers).join(lineInvoice).sql(SqlDialect.SQLite))
// SELECT t0."customer_id", t0."name", t0."city", t0."referrer_id", t1."invoice_id", t1."buyer_id", t1."year", t1."total_cents" FROM "customer" AS t0 INNER JOIN "invoice" AS t1 ON t0."invoice_ref" = t1."invoice_id"
```

`customer` has no `invoice_ref` column, so this query fails when it runs. Add the joins in an order where each `Rel` reaches a table that is already joined.
:::

#### Self-Joins

A `Rel` whose two tables have the same name is a self-join. The joined copy of the table gets its own alias, and the `ON` condition always takes the foreign-key column from `t0` and the key column from the new alias. Here, each customer's `referrer_id` points at the customer who referred them:

```scala
val referredBy = Rel.manyToOne(customers, _.referrerId, customers, _.customerId)
```

A left join keeps the customers nobody referred, and the SQL selects the `customer` columns twice, once per alias:

```scala
println(SqlQuery.from(customers).leftJoin(referredBy).sql(SqlDialect.SQLite))
// SELECT t0."customer_id", t0."name", t0."city", t0."referrer_id", t1."customer_id", t1."name", t1."city", t1."referrer_id" FROM "customer" AS t0 LEFT JOIN "customer" AS t1 ON t0."referrer_id" = t1."customer_id"
```

:::warning
A self-join can't be decoded into `(Customer, Customer)` in 0.0.56: both halves of the row have the same column names, and a known bug reads every repeated column name from the first table, so the second `Customer` is a copy of the first. Render the SQL with `SqlQuery#sql` to check it, and run a self-join as a hand-written `sql"..."` query that gives the second table's columns distinct names with `AS`.
:::

#### Reading Joined Rows

A rendered join returns the columns of all its tables in one row: the `FROM` table first, then one table per join, in the order they were added. Decode the row as a tuple with one element per table, in the same order. The three-table query above decodes as `(Customer, Invoice, InvoiceLine)`:

```scala
tx.connect {
  customerLines.toFrag(SqlDialect.SQLite).query[(Customer, Invoice, InvoiceLine)].foreach { case (c, i, l) =>
    println(s"${c.name}: invoice ${i.invoiceId}, ${l.quantity} x ${l.product}")
  }
}
// Ada: invoice 101, 2 x Keyboard
// Ada: invoice 101, 1 x Mouse
// Bahram: invoice 103, 1 x Monitor
// Chloe: invoice 104, 4 x Cable
```

In a left join, the missing side's columns are SQL `NULL`. Decoding them as a plain case class doesn't fail: SQL `NULL` decodes to `0` for an `Int` and to `null` for a `String`. To tell a missing row apart, decode that side into a row type with the same fields as `Option`s:

```scala
case class InvoiceColumns(invoiceId: Option[Int], buyerId: Option[Int], year: Option[Int], totalCents: Option[Long])
object InvoiceColumns { implicit val schema: Schema[InvoiceColumns] = Schema.derived }

tx.connect {
  SqlQuery.from(customers).leftJoin(invoiceBuyer).toFrag(SqlDialect.SQLite).query[(Customer, InvoiceColumns)].foreach {
    case (c, i) => println(s"${c.name}: ${i.invoiceId.fold("no invoices")(id => s"invoice $id")}")
  }
}
// Ada: invoice 101
// Ada: invoice 102
// Bahram: invoice 103
// Chloe: invoice 104
// Dara: no invoices
```

:::warning
When two joined tables have a column with the same name, such as an `id` key in every table, a known bug in 0.0.56 reads every column with that name from the first table that has it. The SQL is correct, but the decoded values of the later tables are wrong, and in a left join the missing side's key comes back as the other table's key instead of `None`. Give every column a name that is unique across the tables you join, as `customer_id`, `buyer_id` and `invoice_id` are here.
:::

The [How to Write Joins and Inspect the Generated SQL](../../guides/write-joins-and-inspect-generated-sql.md) guide walks through many-to-one, one-to-many and left joins step by step.

### Filtering Rows

`SqlQuery#where` adds a `WHERE` condition, written as a `Frag`, usually with the `sql"..."` interpolator. `SqlQuery#filter` is the same method. Each call adds a condition, and the conditions are combined with `AND` in the order they were added:

```scala
final case class SqlQuery[A] {
  def where(frag: Frag): SqlQuery[A]
  def filter(frag: Frag): SqlQuery[A]
}
```

The condition is SQL text, so it names columns through the query's aliases, double-quoted as the generated SQL quotes them. Interpolated values become bound parameters. Here are the 2026 invoices of customers from Shiraz:

```scala
val city = "Shiraz"

val shirazIn2026 = SqlQuery
  .from(invoices)
  .join(invoiceBuyer)
  .where(sql"""t1."city" = $city""")
  .filter(sql"""t0."year" = ${2026}""")
```

The two conditions are joined with `AND`, each value becomes a `?`, and only Bahram's invoice matches both:

```scala
println(shirazIn2026.sql(SqlDialect.SQLite))
// SELECT t0."invoice_id", t0."buyer_id", t0."year", t0."total_cents", t1."customer_id", t1."name", t1."city", t1."referrer_id" FROM "invoice" AS t0 INNER JOIN "customer" AS t1 ON t0."buyer_id" = t1."customer_id" WHERE t1."city" = ? AND t0."year" = ?

tx.connect {
  shirazIn2026.toFrag(SqlDialect.SQLite).query[(Invoice, Customer)].foreach { case (i, c) =>
    println(s"${c.name}: invoice ${i.invoiceId}, ${i.totalCents / 100.0}")
  }
}
// Bahram: invoice 103, 250.0
```

:::caution
Aliases and column names inside a `where` fragment are not checked. A typo such as `t1."citty"` only fails when the database runs the query. Check the query with `SqlQuery#explain` while you write it.
:::

See also [How to Build a Query from Optional Search Filters](../../guides/build-a-query-from-optional-search-filters.md), which adds a `SqlQuery#where` condition only for the search fields a user filled in.

### Grouping and Sorting

These methods add the `GROUP BY`, `HAVING`, `ORDER BY`, `LIMIT` and `OFFSET` clauses. Every column name they take refers to the `FROM` table: the renderer writes it as `t0."<column>"`.

#### Grouping Rows

`SqlQuery#groupBy` sets the `GROUP BY` columns, and `SqlQuery#having` sets the `HAVING` condition as a `Frag`. Both replace the value of an earlier call instead of adding to it. `SqlQuery#groupBy` checks each name as a plain SQL identifier and throws an `IllegalArgumentException` for anything else, including a dotted name such as `t1.city`:

```scala
final case class SqlQuery[A] {
  def groupBy(cols: String*): SqlQuery[A]
  def having(frag: Frag): SqlQuery[A]
}
```

The `SELECT` list is always every column, so a grouped query can't select an aggregate such as `COUNT(*)`; `HAVING` can still use one. That makes `SqlQuery#groupBy` useful for keeping the rows whose group passes a test. Here are the customers with at least two invoices, decoded as `Customer` alone, because the first columns of each row are the customer's:

```scala
val repeatBuyers = SqlQuery
  .from(customers)
  .join(invoiceBuyer)
  .groupBy("customer_id")
  .having(sql"COUNT(*) >= ${2}")
```

Ada is the only customer with two invoices:

```scala
println(repeatBuyers.sql(SqlDialect.SQLite))
// SELECT t0."customer_id", t0."name", t0."city", t0."referrer_id", t1."invoice_id", t1."buyer_id", t1."year", t1."total_cents" FROM "customer" AS t0 INNER JOIN "invoice" AS t1 ON t1."buyer_id" = t0."customer_id" GROUP BY t0."customer_id" HAVING COUNT(*) >= ?

tx.connect {
  repeatBuyers.toFrag(SqlDialect.SQLite).query[Customer].foreach(println)
}
// Customer(1,Ada,Tehran,None)
```

:::caution
SQLite accepts a `GROUP BY` that leaves selected columns out of the grouping, and returns the values of an arbitrary row of each group for them; here the invoice columns come from one of Ada's invoices. PostgreSQL accepts such a query only when every selected column depends on the grouped columns, for example a single-table query grouped by its primary key. Otherwise it fails with `column "..." must appear in the GROUP BY clause or be used in an aggregate function`, so the query above fails on PostgreSQL. For aggregates in the result, write the query with the `sql"..."` interpolator.
:::

#### Sorting Rows

`SqlQuery#orderBy` adds one `ORDER BY` column, ascending unless you pass `SortOrder.Desc`; each call adds a column after the earlier ones. `SqlQuery#orderByMany` adds several `OrderBy` values at once:

```scala
final case class SqlQuery[A] {
  def orderBy(col: String, dir: SortOrder = SortOrder.Asc): SqlQuery[A]
  def orderByMany(cols: OrderBy*): SqlQuery[A]
}
```

`SqlQuery#orderBy` checks the column name when you call it, and throws an `IllegalArgumentException` for a name that isn't a plain SQL identifier. `SqlQuery#orderByMany` checks its names only when the query is rendered, so a bad name surfaces in `SqlQuery#toFrag`, `SqlQuery#sql` or `SqlQuery#explain`. The two forms below sort the same way, by city and then by name in reverse:

```scala
val byCityThenName = SqlQuery.from(customers).orderBy("city").orderBy("name", SortOrder.Desc)
val sameOrder      = SqlQuery.from(customers).orderByMany(OrderBy("city", SortOrder.Asc), OrderBy("name", SortOrder.Desc))
```

Both render one `ORDER BY` with the columns in the order they were added, and the two queries are equal:

```scala
println(byCityThenName.sql(SqlDialect.SQLite))
// SELECT t0."customer_id", t0."name", t0."city", t0."referrer_id" FROM "customer" AS t0 ORDER BY t0."city" ASC, t0."name" DESC
println(byCityThenName == sameOrder)
// true
```

:::caution
`SqlQuery#orderBy` and `SqlQuery#groupBy` can only name columns of the `FROM` table, a known bug in 0.0.56. A name like `t1.name` throws `IllegalArgumentException: Invalid SQL column identifier 't1.name'`, and a column that exists only in a joined table renders as `t0."<column>"` and fails when the database runs the query. To sort by a joined table's column, start the query from that table, or sort the decoded rows in Scala.
:::

#### Paging Rows

`SqlQuery#limit` caps the number of rows, and `SqlQuery#offset` skips rows before the first one returned. Each replaces the value of an earlier call. Both require `n >= 0` and throw an `IllegalArgumentException` otherwise, with the message "requirement failed: limit must be >= 0" for `SqlQuery#limit` and "requirement failed: offset must be >= 0" for `SqlQuery#offset`. The numbers are written into the SQL text, not bound as parameters:

```scala
final case class SqlQuery[A] {
  def limit(n: Int): SqlQuery[A]
  def offset(n: Int): SqlQuery[A]
}
```

Paging needs a stable order, so we sort by the key and read the second page of two customers:

```scala
val secondPage = SqlQuery.from(customers).orderBy("customer_id").limit(2).offset(2)
```

`LIMIT` and `OFFSET` come last in the SQL, as literal numbers, and the page holds the third and fourth customers:

```scala
println(secondPage.sql(SqlDialect.SQLite))
// SELECT t0."customer_id", t0."name", t0."city", t0."referrer_id" FROM "customer" AS t0 ORDER BY t0."customer_id" ASC LIMIT 2 OFFSET 2

tx.connect {
  secondPage.toFrag(SqlDialect.SQLite).query[Customer].foreach(println)
}
// Customer(3,Chloe,Berlin,Some(1))
// Customer(4,Dara,Tehran,Some(2))
```

:::caution
`SqlQuery#offset` without `SqlQuery#limit` renders `OFFSET n` with no `LIMIT`, a known bug in 0.0.56. PostgreSQL accepts that, but SQLite only allows `OFFSET` after `LIMIT` and fails with `near "OFFSET": syntax error`. On SQLite, always pair `SqlQuery#offset` with `SqlQuery#limit`.
:::

### Rendering and Running

A query does nothing until it is rendered. `SqlQuery#toFrag` renders it to a `Frag` for a dialect, `SqlQuery#sql` returns the SQL string that `Frag` sends to the JDBC driver, and `SqlQuery#explain` returns a version for logs and code review:

```scala
final case class SqlQuery[A] {
  def toFrag(dialect: SqlDialect): Frag
  def sql(dialect: SqlDialect): String
  def explain(dialect: SqlDialect): String
}
```

#### Running the Rendered Fragment

`SqlQuery` has no methods that run it: the `Frag` from `SqlQuery#toFrag` runs inside a `Transactor` block with `Frag#query`, `Frag#queryOne`, `Frag#queryLimit`, `Frag#queryStream` or `Frag#queryChunked`, as described on the [Frag](./frag.md) page. `Frag#queryOne` returns the first row, if any:

```scala
tx.connect {
  println(SqlQuery.from(invoices).orderBy("total_cents", SortOrder.Desc).toFrag(SqlDialect.SQLite).queryOne[Invoice])
}
// Invoice(103,2,2026,25000)
```

#### Printing the SQL

`SqlQuery#explain` returns the SQL on one line with numbered `?1`, `?2` placeholders, followed by a `-- params:` line that lists each parameter's position and type, never its value. `-- params: (none)` means the query binds nothing:

```scala
println(shirazIn2026.explain(SqlDialect.PostgreSQL))
// SELECT t0."invoice_id", t0."buyer_id", t0."year", t0."total_cents", t1."customer_id", t1."name", t1."city", t1."referrer_id" FROM "invoice" AS t0 INNER JOIN "customer" AS t1 ON t0."buyer_id" = t1."customer_id" WHERE t1."city" = ?1 AND t0."year" = ?2
// -- params: 1:String, 2:Int
println(secondPage.explain(SqlDialect.PostgreSQL))
// SELECT t0."customer_id", t0."name", t0."city", t0."referrer_id" FROM "customer" AS t0 ORDER BY t0."customer_id" ASC LIMIT 2 OFFSET 2
// -- params: (none)
```

PostgreSQL and SQLite render the same SQL: both use `?` placeholders and double-quoted identifiers, and the `dialect` parameter is kept for a future dialect with another placeholder syntax. To write the SQL of each query to a `.sql` file while the code compiles, see [Dump](./dump.md).

#### Inspecting a Query

The case class fields are public, so you can read back what a query holds. Some field names are also builder methods: `SqlQuery#limit` without arguments is the `Option[Int]` field, and `SqlQuery#limit(10)` is the builder:

```scala
println(secondPage.limit)
// Some(2)
println(secondPage.offset)
// Some(2)
println(secondPage.orderBy)
// Vector(OrderBy(customer_id,Asc))
println(shirazIn2026.filters.size)
// 2
println(shirazIn2026.source.name)
// invoice
```

`joins` holds the query's join nodes, whose type is private to the module.

#### Rendering with QueryRenderer

You rarely call this directly; it is the renderer behind `SqlQuery#toFrag`, which you should prefer. `QueryRenderer.render` turns a query into a `Frag` by composing `Frag.literal` pieces and the query's own condition fragments:

```scala
object QueryRenderer {
  def render[A](query: SqlQuery[A], dialect: SqlDialect): Frag
}
```

It renders the clauses in a fixed order, whatever order the builder methods were called in: `SELECT`, `FROM`, the joins, `WHERE`, `GROUP BY`, `HAVING`, `ORDER BY`, `LIMIT` and `OFFSET`. `SqlQuery#toFrag(dialect)` is `QueryRenderer.render(this, dialect)`:

```scala
println(QueryRenderer.render(secondPage, SqlDialect.SQLite) == secondPage.toFrag(SqlDialect.SQLite))
// true
```

## JoinKind and SortOrder

`JoinKind` and `SortOrder` are sealed traits with two case objects each, and `OrderBy` pairs a column with a `SortOrder`:

| Type        | Value or field          | Meaning                                                    |
|-------------|-------------------------|------------------------------------------------------------|
| `JoinKind`  | `JoinKind.Inner`        | `INNER JOIN`: keep only rows with a match in both tables   |
| `JoinKind`  | `JoinKind.Left`         | `LEFT JOIN`: keep every row of the existing side           |
| `SortOrder` | `SortOrder.Asc`         | `ASC`, the default of `SqlQuery#orderBy`                   |
| `SortOrder` | `SortOrder.Desc`        | `DESC`                                                     |
| `OrderBy`   | `column: String`        | A column of the `FROM` table, checked when rendered        |
| `OrderBy`   | `direction: SortOrder`  | The sort direction for that column                         |

A join kind chosen at run time goes through `SqlQuery#join(rel, kind)`:

```scala
def invoicesOfCustomers(includeWithoutInvoices: Boolean): SqlQuery[Customer] = {
  val kind = if (includeWithoutInvoices) JoinKind.Left else JoinKind.Inner
  SqlQuery.from(customers).join(invoiceBuyer, kind)
}

println(invoicesOfCustomers(includeWithoutInvoices = true).sql(SqlDialect.SQLite))
// SELECT t0."customer_id", t0."name", t0."city", t0."referrer_id", t1."invoice_id", t1."buyer_id", t1."year", t1."total_cents" FROM "customer" AS t0 LEFT JOIN "invoice" AS t1 ON t1."buyer_id" = t0."customer_id"
```

## Integration

`SqlQuery` sits between the schema-derived tables and the fragments that run:

- [`Table`](./table.md) supplies each table's name and column list; `Table.derived` names them in snake case.
- [`Rel`](./rel.md) describes each join, with its column names checked at compile time when built from field selectors.
- [`Frag`](./frag.md) carries the conditions in and the rendered query out; its `query` methods and a [`DbCodec`](./db-codec.md) for the row type run it and decode the rows.
- [`Dump`](./dump.md) writes a query's SQL to a file at compile time.
- [`SqlLogger`](./sql-logger.md) records the SQL of every query a transactor runs, including rendered `SqlQuery`s.

---

## TableMetadata

`TableMetadata` derives column metadata from a `Schema`. It returns a list of `ColumnMeta` instances, each describing a column's name, type, and nullability. `TableNamingPolicy` controls how Scala type names become table names.

## Core API

```scala
object TableMetadata {
  def columnsFor[A](
    schema: Schema[A],
    columnNameMapper: SqlNameMapper = SqlNameMapper.SnakeCase
  ): IndexedSeq[ColumnMeta]
}

final case class ColumnMeta(name: String, dbValue: DbValue, nullable: Boolean)

sealed trait TableNamingPolicy {
  def defaultName(typeName: String): String
}

object TableNamingPolicy {
  case object Singular                         extends TableNamingPolicy
  case object Plural                           extends TableNamingPolicy
  final case class Custom(f: String => String) extends TableNamingPolicy
}
```

## Usage

Derive column metadata from a schema:

```scala

case class Product(id: Int, name: String, price: Option[BigDecimal])
object Product { given schema: Schema[Product] = Schema.derived }

val cols = TableMetadata.columnsFor(Product.schema)
// cols: IndexedSeq[ColumnMeta] = Vector(
//   ColumnMeta(name = "id", dbValue = DbInt(0), nullable = false),
//   ColumnMeta(name = "name", dbValue = DbString(""), nullable = false),
//   ColumnMeta(name = "price", dbValue = DbBigDecimal(0), nullable = true)
// )
cols
// res1: IndexedSeq[ColumnMeta] = Vector(
//   ColumnMeta(name = "id", dbValue = DbInt(0), nullable = false),
//   ColumnMeta(name = "name", dbValue = DbString(""), nullable = false),
//   ColumnMeta(name = "price", dbValue = DbBigDecimal(0), nullable = true)
// )

cols.map(_.name)
// res2: IndexedSeq[String] = Vector("id", "name", "price")
cols.map(_.nullable)
// res3: IndexedSeq[Boolean] = Vector(false, false, true)
```

Use the column metadata to get DDL types:

```scala
cols.map(col => SqlDialect.PostgreSQL.typeName(col.dbValue))
// res4: IndexedSeq[String] = Vector("INTEGER", "TEXT", "NUMERIC")
```

Control table naming when deriving a Table:

```scala

// Singular table name (default)
val table1 = Table.derived[Product]
// table1: Table[Product] = Table(
//   name = "product",
//   codec = zio.blocks.sql.DbCodecDeriver$$anon$10@340d63e8,
//   columnsMeta = Vector(
//     ColumnMeta(name = "id", dbValue = DbInt(0), nullable = false),
//     ColumnMeta(name = "name", dbValue = DbString(""), nullable = false),
//     ColumnMeta(name = "price", dbValue = DbBigDecimal(0), nullable = true)
//   )
// )
table1.name
// res5: String = "product"

// Plural table name
val table2 = Table.derived[Product](TableNamingPolicy.Plural)
// table2: Table[Product] = Table(
//   name = "products",
//   codec = zio.blocks.sql.DbCodecDeriver$$anon$10@1b1d6fe3,
//   columnsMeta = Vector(
//     ColumnMeta(name = "id", dbValue = DbInt(0), nullable = false),
//     ColumnMeta(name = "name", dbValue = DbString(""), nullable = false),
//     ColumnMeta(name = "price", dbValue = DbBigDecimal(0), nullable = true)
//   )
// )
table2.name
// res6: String = "products"

// Custom naming
val table3 = Table.derived[Product](TableNamingPolicy.Custom("t_" + _))
// table3: Table[Product] = Table(
//   name = "t_Product",
//   codec = zio.blocks.sql.DbCodecDeriver$$anon$10@44862f37,
//   columnsMeta = Vector(
//     ColumnMeta(name = "id", dbValue = DbInt(0), nullable = false),
//     ColumnMeta(name = "name", dbValue = DbString(""), nullable = false),
//     ColumnMeta(name = "price", dbValue = DbBigDecimal(0), nullable = true)
//   )
// )
table3.name
// res7: String = "t_Product"
```

## Key Points

**`columnsFor`** — Walks a schema's structure and returns metadata for each column, respecting `@Modifier.transient` (skip field), `@Modifier.rename` (override name), and `Option[A]` / `Maybe[A]` (mark nullable).

**ColumnMeta** — Holds the column name, a representative `DbValue` for type inference, and a nullable flag. The actual value in `dbValue` doesn't matter—only its variant is used by `SqlDialect#typeName`.

**TableNamingPolicy** — Controls default table naming: `Singular` converts `"UserAccount"` to `"user_account"`, `Plural` to `"user_accounts"`, `Custom(f)` applies function `f` directly.

## How It Works

`Table.derived` calls `columnsFor` to extract column metadata from the schema. For each `ColumnMeta`, the dialect's `typeName` method is called to get the SQL type string. The metadata tracks which columns are nullable based on `Option[A]` or `Maybe[A]` types. Fields annotated with `@Modifier.rename` use their explicit name instead of the mapper.

For how Table uses this metadata, see [Table](./table.md). For DDL generation, see [Ddl](./ddl.md).

---

## Table

`Table[A]` is the metadata binding between a Scala type `A` and a specific database table in the `sql` module. It holds the table name, a `DbCodec[A]` for reading and writing rows, and an `IndexedSeq[ColumnMeta]` describing each column's name, SQL type representative, and nullability. `Table` provides both type-safe column access and dialect-aware DDL generation without any ORM runtime or session lifecycle.

The structural shape of `Table` is:

```scala
final case class Table[A](name: String, codec: DbCodec[A], columnsMeta: IndexedSeq[ColumnMeta]) {
  def columns: IndexedSeq[String] = ???
  
  def createTable(dialect: SqlDialect): Frag = ???
  def dropTable: Frag = ???
}

object Table {
  def derived[A](implicit schema: Schema[A]): Table[A] = ???
  def derived[A](tableName: String)(implicit schema: Schema[A]): Table[A] = ???
  def derived[A](namingPolicy: TableNamingPolicy)(implicit schema: Schema[A]): Table[A] = ???
}
```

## Usage

The following example illustrates the core workflow: derive a table from a schema-equipped case class, inspect its column names, generate `CREATE TABLE` DDL, and finally generate `DROP TABLE` DDL:

```scala

case class User(id: Int, name: String, email: String)
object User {
  implicit val schema: Schema[User] = Schema.derived
}

// Derive the table binding — columns come from the schema; "user" is a
// reserved word in PostgreSQL, so the table name is overridden explicitly
val table = Table.derived[User]("users")
// table: Table[User] = Table(
//   name = "users",
//   codec = zio.blocks.sql.DbCodecDeriver$$anon$10@513dfb20,
//   columnsMeta = Vector(
//     ColumnMeta(name = "id", dbValue = DbInt(0), nullable = false),
//     ColumnMeta(name = "name", dbValue = DbString(""), nullable = false),
//     ColumnMeta(name = "email", dbValue = DbString(""), nullable = false)
//   )
// )
table.name
// res1: String = "users"
table.columns
// res2: IndexedSeq[String] = Vector("id", "name", "email")

// Emit dialect-aware DDL as Frag values
val createSql = table.createTable(SqlDialect.PostgreSQL).sql(SqlDialect.PostgreSQL)
// createSql: String = """CREATE TABLE IF NOT EXISTS users (
//   id INTEGER NOT NULL,
//   name TEXT NOT NULL,
//   email TEXT NOT NULL
// )"""
val dropSql    = table.dropTable.sql(SqlDialect.PostgreSQL)
// dropSql: String = "DROP TABLE IF EXISTS users"
```

## Construction / Creating Instances

`Table` offers three `derived` factory methods on its companion object and one direct constructor via its primary constructor. All three `derived` overloads require a `Schema[A]` implicit.

### `Table.derived` — Derive using the default naming policy

`Table.derived[A]` inspects the `Schema[A]` implicit and applies `TableNamingPolicy.Singular` to produce the table name. This policy converts `CamelCase` Scala type names to `snake_case` SQL identifiers (for example, `UserProfile` becomes `user_profile`). The table name can be overridden by annotating the type with `@Modifier.config("sql.table_name", "my_table")`.

```scala
object Table {
  def derived[A](implicit schema: Schema[A]): Table[A]
}
```

The following example derives a table for a two-field case class and checks the resulting name and column list:

```scala

case class BlogPost(title: String, body: String)
object BlogPost {
  implicit val schema: Schema[BlogPost] = Schema.derived
}

val table = Table.derived[BlogPost]
// table: Table[BlogPost] = Table(
//   name = "blog_post",
//   codec = zio.blocks.sql.DbCodecDeriver$$anon$10@513aae74,
//   columnsMeta = Vector(
//     ColumnMeta(name = "title", dbValue = DbString(""), nullable = false),
//     ColumnMeta(name = "body", dbValue = DbString(""), nullable = false)
//   )
// )
table.name
// res4: String = "blog_post"
table.columns
// res5: IndexedSeq[String] = Vector("title", "body")
```

### `Table.derived` (with explicit table name) — Bypass naming policy and annotations

`Table.derived[A](tableName: String)` derives a table with the supplied name, ignoring both the `TableNamingPolicy` and any `@Modifier.config("sql.table_name", …)` annotation on the type. The column names and codec are still derived from the schema in the normal way. Use this overload when the desired SQL table name cannot be expressed by any naming policy.

```scala
object Table {
  def derived[A](tableName: String)(implicit schema: Schema[A]): Table[A]
}
```

The following example maps `UserProfile` to a table called `profiles` rather than the default `user_profile`:

```scala

case class UserProfile(firstName: String, lastName: String)
object UserProfile {
  implicit val schema: Schema[UserProfile] = Schema.derived
}

val table = Table.derived[UserProfile]("profiles")
// table: Table[UserProfile] = Table(
//   name = "profiles",
//   codec = zio.blocks.sql.DbCodecDeriver$$anon$10@23ec3e3d,
//   columnsMeta = Vector(
//     ColumnMeta(name = "first_name", dbValue = DbString(""), nullable = false),
//     ColumnMeta(name = "last_name", dbValue = DbString(""), nullable = false)
//   )
// )
table.name
// res7: String = "profiles"
table.columns
// res8: IndexedSeq[String] = Vector("first_name", "last_name")
```

:::caution
The table name is validated as a SQL identifier immediately at construction time. Spaces, hyphens, or any character outside `[A-Za-z0-9_]` (with a letter or underscore as the first character) will cause `Table.derived` to throw `IllegalArgumentException`. For example, `Table.derived[UserProfile]("user profile")` throws with the message `Invalid SQL table identifier 'user profile'. Only ASCII letters, digits, and underscores are supported, and the first character must be a letter or underscore.`
:::

### `Table.derived` (with naming policy) — Control table name derivation

`Table.derived[A](namingPolicy: TableNamingPolicy)` derives a table and applies the supplied `TableNamingPolicy` to the type name when computing the table name. Use `TableNamingPolicy.Plural` for pluralized names, `TableNamingPolicy.Singular` (the default) for singular names, or `TableNamingPolicy.Custom(f)` for arbitrary transformations.

```scala
object Table {
  def derived[A](namingPolicy: TableNamingPolicy)(implicit schema: Schema[A]): Table[A]
}
```

The following example uses `TableNamingPolicy.Plural` so that `Category` maps to the table `categories`:

```scala

case class Category(name: String)
object Category {
  implicit val schema: Schema[Category] = Schema.derived
}

val singular = Table.derived[Category](TableNamingPolicy.Singular)
// singular: Table[Category] = Table(
//   name = "category",
//   codec = zio.blocks.sql.DbCodecDeriver$$anon$10@1648bb79,
//   columnsMeta = Vector(
//     ColumnMeta(name = "name", dbValue = DbString(""), nullable = false)
//   )
// )
singular.name
// res10: String = "category"

val plural = Table.derived[Category](TableNamingPolicy.Plural)
// plural: Table[Category] = Table(
//   name = "categories",
//   codec = zio.blocks.sql.DbCodecDeriver$$anon$10@18e498b9,
//   columnsMeta = Vector(
//     ColumnMeta(name = "name", dbValue = DbString(""), nullable = false)
//   )
// )
plural.name
// res11: String = "categories"

val custom = Table.derived[Category](TableNamingPolicy.Custom(n => s"tbl_$n"))
// custom: Table[Category] = Table(
//   name = "tbl_Category",
//   codec = zio.blocks.sql.DbCodecDeriver$$anon$10@44cccd84,
//   columnsMeta = Vector(
//     ColumnMeta(name = "name", dbValue = DbString(""), nullable = false)
//   )
// )
custom.name
// res12: String = "tbl_Category"
```

### `Table.apply` — Construct directly from codec and column metadata

The primary constructor accepts the table name, a `DbCodec[A]`, and an `IndexedSeq[ColumnMeta]` explicitly. SQL identifier validation runs for the table name and every column name at construction time. Use this constructor when you have a hand-written or externally produced codec rather than a schema-derived one.

```scala
final case class Table[A](name: String, codec: DbCodec[A], columnsMeta: IndexedSeq[ColumnMeta])
```

The following example builds a `Table` manually, supplying a pre-existing `DbCodec` and explicit column metadata:

```scala

case class Tag(id: Int, label: String) derives DbCodec

// columnsMeta must describe the same columns, in the same order, as the codec;
// nullable must match the field's actual optionality (Tag.label is non-optional)
val meta  = IndexedSeq(
  ColumnMeta("id",    DbValue.DbInt(0),    nullable = false),
  ColumnMeta("label", DbValue.DbString(""), nullable = false)
)
// meta: IndexedSeq[ColumnMeta] = Vector(
//   ColumnMeta(name = "id", dbValue = DbInt(0), nullable = false),
//   ColumnMeta(name = "label", dbValue = DbString(""), nullable = false)
// )
val table = Table[Tag]("tag", DbCodec[Tag], meta)
// table: Table[Tag] = Table(
//   name = "tag",
//   codec = zio.blocks.sql.DbCodecDeriver$$anon$10@4dd1dd88,
//   columnsMeta = Vector(
//     ColumnMeta(name = "id", dbValue = DbInt(0), nullable = false),
//     ColumnMeta(name = "label", dbValue = DbString(""), nullable = false)
//   )
// )
table.name
// res14: String = "tag"
table.columns
// res15: IndexedSeq[String] = Vector("id", "label")
```

:::note
When the column `nullable` flag is `true`, the generated `CREATE TABLE` statement omits the `NOT NULL` constraint for that column, allowing the database to store `NULL` in that position. `Table.derived` sets this flag automatically based on whether the corresponding schema field is `Option[A]` or `Maybe[A]`.
:::

## Core Operations

### Element Access

The Element Access category exposes `columns`, which returns the SQL column names carried by the table in codec order.

#### `columns` — Column names in codec order

`Table#columns` returns an `IndexedSeq[String]` of the SQL column names for this table, in the order of `columnsMeta`. The names come from `columnsMeta`, not from the `DbCodec[A]`, so they can differ from the codec's columns for nested and collection fields, and they have already been checked to be legal SQL identifiers at construction time. Each call maps the validated metadata to names, so the cost grows with the number of columns.

```scala
final case class Table[A](...) {
  def columns: IndexedSeq[String]
}
```

The following example shows `columns` reflecting the snake_case field names derived from the schema:

```scala

case class OrderLine(productId: Int, quantity: Int, unitPrice: BigDecimal)
object OrderLine {
  implicit val schema: Schema[OrderLine] = Schema.derived
}

val table = Table.derived[OrderLine]
// table: Table[OrderLine] = Table(
//   name = "order_line",
//   codec = zio.blocks.sql.DbCodecDeriver$$anon$10@12a98404,
//   columnsMeta = Vector(
//     ColumnMeta(name = "product_id", dbValue = DbInt(0), nullable = false),
//     ColumnMeta(name = "quantity", dbValue = DbInt(0), nullable = false),
//     ColumnMeta(name = "unit_price", dbValue = DbBigDecimal(0), nullable = false)
//   )
// )
table.columns
// res17: IndexedSeq[String] = Vector("product_id", "quantity", "unit_price")
```

### DDL Generation

The DDL Generation category provides `createTable` and `dropTable`, which produce `Frag` values containing dialect-specific `CREATE TABLE IF NOT EXISTS` and `DROP TABLE IF EXISTS` statements. Both methods delegate to the `Ddl` helper, which constructs a `Frag` with no bound parameters — only literal SQL text.

#### `createTable` — Generate a CREATE TABLE statement

`Table#createTable` accepts a `SqlDialect` and returns a `Frag` whose SQL text is a `CREATE TABLE IF NOT EXISTS` statement. Each column definition uses the dialect's `typeName` method to convert the column's `DbValue` representative to the appropriate SQL type string (for example, `DbValue.DbString` becomes `TEXT` in PostgreSQL and `TEXT` in SQLite; `DbValue.DbInt` becomes `INTEGER` in both). Non-nullable columns carry a `NOT NULL` constraint; nullable columns do not. The statement declares no `PRIMARY KEY` or `UNIQUE` constraint, not even on the column that a `Repo` uses as its ID, so in 0.0.56 `Repo#insertOrUpdate` fails on a table made by `createTable`, a known bug; create the table yourself for it (see [Repo](./repo.md)).

```scala
final case class Table[A](...) {
  def createTable(dialect: SqlDialect): Frag
}
```

The following example demonstrates the DDL generated for a record with a mix of column types and an optional field:

```scala

case class Product(sku: String, price: BigDecimal, stock: Option[Int])
object Product {
  implicit val schema: Schema[Product] = Schema.derived
}

val table    = Table.derived[Product]
// table: Table[Product] = Table(
//   name = "product",
//   codec = zio.blocks.sql.DbCodecDeriver$$anon$10@31851f01,
//   columnsMeta = Vector(
//     ColumnMeta(name = "sku", dbValue = DbString(""), nullable = false),
//     ColumnMeta(name = "price", dbValue = DbBigDecimal(0), nullable = false),
//     ColumnMeta(name = "stock", dbValue = DbInt(0), nullable = true)
//   )
// )
val createPg = table.createTable(SqlDialect.PostgreSQL).sql(SqlDialect.PostgreSQL)
// createPg: String = """CREATE TABLE IF NOT EXISTS product (
//   sku TEXT NOT NULL,
//   price NUMERIC NOT NULL,
//   stock INTEGER
// )"""
val createSq = table.createTable(SqlDialect.SQLite).sql(SqlDialect.SQLite)
// createSq: String = """CREATE TABLE IF NOT EXISTS product (
//   sku TEXT NOT NULL,
//   price TEXT NOT NULL,
//   stock INTEGER
// )"""
```

:::caution[Nested Records and Collections]
`Table.derived` computes its column list from the `Schema` on its own, not from the row codec, and in 0.0.56 the two disagree for some field types, a known bug. A nested case class field becomes `<field>_<subfield>` columns, but the codec stores it as one JSON column unless the field is annotated with `@Modifier.config("sql.inline", "true")`, so without the annotation the table and the codec have different columns and `Repo#insert` fails. A `List`, `Set`, `Map` or other collection field, and a sealed trait whose cases have fields, make `Table.derived` and `Repo.derived` throw `UnsupportedOperationException` before any SQL runs. An enumeration of case objects becomes a `TEXT` column.

So `Table.derived`, `Repo.derived` and `Table#createTable` work only for records whose nested record fields carry `sql.inline`. To keep a field as a JSON column, build the `Table` with `Table.apply` from the codec's own `DbCodec#columns`, pass it to `Repo.apply`, and write the `CREATE TABLE` statement by hand. No dialect emits `JSONB`, so a JSON column on PostgreSQL must be declared `JSONB` yourself. [How to Map Custom Column Types](../../guides/map-custom-column-types.md#store-a-nested-case-class-as-json) shows both.
:::

#### `dropTable` — Generate a DROP TABLE statement

`Table#dropTable` returns a `Frag` whose SQL text is `DROP TABLE IF EXISTS <name>`, with no parameters and no dialect argument. Because `DROP TABLE` syntax is uniform across the supported dialects, a single `Frag` is correct for any `SqlDialect`. Render the fragment with `Frag#sql` to obtain the final SQL string.

```scala
final case class Table[A](...) {
  def dropTable: Frag
}
```

The following example shows the drop statement for a table derived from a simple case class:

```scala

case class Session(token: String, userId: Int)
object Session {
  implicit val schema: Schema[Session] = Schema.derived
}

val table = Table.derived[Session]
// table: Table[Session] = Table(
//   name = "session",
//   codec = zio.blocks.sql.DbCodecDeriver$$anon$10@3638adf8,
//   columnsMeta = Vector(
//     ColumnMeta(name = "token", dbValue = DbString(""), nullable = false),
//     ColumnMeta(name = "user_id", dbValue = DbInt(0), nullable = false)
//   )
// )
table.dropTable.sql(SqlDialect.PostgreSQL)
// res20: String = "DROP TABLE IF EXISTS session"
table.dropTable.sql(SqlDialect.SQLite)
// res21: String = "DROP TABLE IF EXISTS session"
```

## Supporting Types

`Table` is a monomorphic final case class with no subtypes of its own, but it depends on two supporting types — `ColumnMeta` and `TableNamingPolicy` — that control how column metadata is captured and how table names are derived.

### `ColumnMeta`

`ColumnMeta` is a final case class that carries the per-column metadata consumed by `Table#createTable` for DDL generation. Each primitive field in a schema-derived type produces exactly one `ColumnMeta`; optional fields set `nullable = true`. A nested record field produces one `ColumnMeta` per subfield, named `<field>_<subfield>`, even when the codec stores it as one JSON column (see the caution under [`createTable`](#createtable--generate-a-create-table-statement)).

```scala
final case class ColumnMeta(name: String, dbValue: DbValue, nullable: Boolean)
```

The three fields serve distinct roles. `name` is the SQL column name after `SqlNameMapper` has been applied and SQL identifier validation has been run. `dbValue` is a representative instance of the column's `DbValue` variant (for example, `DbValue.DbInt(0)` for an `Int` column) — it carries no runtime data and is used only to dispatch to `SqlDialect#typeName` during DDL generation. `nullable` reflects whether the Scala field is `Option[A]` or `Maybe[A]`.

### `TableNamingPolicy`

`TableNamingPolicy` is a sealed trait that controls how a Scala type name is translated into a SQL table name when using `Table.derived`. All three `derived` overloads use a naming policy either explicitly or implicitly.

```scala
sealed trait TableNamingPolicy {
  def defaultName(typeName: String): String
}

object TableNamingPolicy {
  case object Singular                      extends TableNamingPolicy
  case object Plural                        extends TableNamingPolicy
  final case class Custom(f: String => String) extends TableNamingPolicy
}
```

The three variants cover the most common conventions:

- **`Singular`** (the default) — converts the Scala type name to `snake_case` using `SqlNameMapper.SnakeCase`. `UserProfile` becomes `user_profile`, `Category` becomes `category`.
- **`Plural`** — applies the same `snake_case` conversion and then appends a simple English pluralization suffix. `Category` becomes `categories`, `User` becomes `users`, `Quiz` becomes `quizzes`.
- **`Custom(f)`** — applies the function `f` to the type name, giving full control over the mapping. The function receives the raw Scala type name (before any case conversion) and must return a valid SQL identifier.

The `Singular` policy is chosen because most databases treat table names as singular nouns by convention, but `Plural` is equally idiomatic in many teams. Pass the desired policy explicitly to `Table.derived[A](namingPolicy)` when the default does not match your project's convention.

## Comparison

### Slick and Doobie

`Table` takes a narrower scope than lifted-embedding ORMs like Slick and functional query builders like Doobie:

| Concern                  | `Table` (this module)                                        | Slick                                                       | Doobie                                               |
|--------------------------|--------------------------------------------------------------|-------------------------------------------------------------|------------------------------------------------------|
| Schema source of truth   | `Schema[A]` (compile-time derivation)                        | `Table` class extending `TableQuery` (explicit column defs) | Hand-written `Get`/`Put` instances or Doobie macros  |
| Query DSL                | Plain SQL via `sql"..."` interpolator + `Frag` composition   | Lifted Scala expressions compiled to SQL                    | Plain SQL via `sql"..."` interpolator                |
| DDL generation           | `Table#createTable` / `Table#dropTable` return `Frag` values | Via `schema.create` / `schema.drop` (requires lifted query) | Not built-in; usually handled by Flyway or Liquibase |
| Runtime overhead         | Zero — derivation is compile-time; no reflection at runtime  | JVM reflection + query compilation per session              | Minimal; `Get`/`Put` are materialized type classes   |
| Effect system dependency | None (the shared API abstracts over `DbConnection`; no JDBC dependency in shared code) | Slick's `DBIO` monad                                        | Cats `IO` or `Sync[F]`                               |

`Table` does not model relationships, joins, or query projection — those concerns belong to hand-written `sql"..."` fragments and the `Repo` type. When you need rich relational queries, compose `Frag` values manually rather than using a lifted embedding.

### Hibernate JPA

`Table` and Hibernate address the same problem from opposite directions:

| Concern             | `Table` (this module)                                            | Hibernate / JPA                                                         |
|---------------------|------------------------------------------------------------------|-------------------------------------------------------------------------|
| Configuration style | Immutable value derived from `Schema[A]` at compile time         | Annotations on mutable entity classes at runtime                        |
| Session lifecycle   | None — connections managed explicitly by `Transactor`            | `EntityManager`, `Session`, first-level cache, lazy proxies             |
| Lazy loading        | Not supported — all column values are loaded eagerly             | Supported via proxy objects and byte-code instrumentation               |
| SQL control         | Full — every query is a `Frag` of literal SQL + typed parameters | Partial — JPQL / Criteria API abstracts SQL; native SQL as escape hatch |
| DDL generation      | `Table#createTable` returns a `Frag`; you execute it explicitly  | `hbm2ddl.auto` may run DDL automatically at startup                     |
| Scala compatibility | First-class; no mutable beans required                           | Requires JavaBean conventions (default constructor, mutable fields)     |

`Table` never manages entity identity, caching, or lazy associations. It is a thin, transparent layer over JDBC — what you write in `sql"..."` is exactly what the database executes.

---

## TransactorZIO

`TransactorZIO` is the ZIO-integrated database transactor from the `zio-blocks-sql-zio` module. It builds on a [`JdbcTransactor`](./transactor.md) — the synchronous JDBC transactor from the core `sql` module — and lifts connection and transaction lifecycle management into the ZIO effect system: `connect` and `transact` delegate to the `JdbcTransactor`, while `connectZIO` and `transactZIO` open their own connections from the same connection factory. The module targets Scala 3 and runs on JVM only.

:::info
This page is a detailed reference for the `TransactorZIO` class specifically. If you only need `ZLayer`-based dependency injection for the plain synchronous `Transactor` (via `JdbcTransactor.postgresLayer` / `sqliteLayer`, extension methods that need `import zio.blocks.sql.zio.*`), see the [ZIO Integration](../sql-zio.md) page instead — it covers both integration styles and when to reach for each.
:::

`TransactorZIO` exposes two pairs of methods. The **blocking wrappers** (`TransactorZIO#connect`, `TransactorZIO#transact`) run a synchronous body on ZIO's blocking thread pool via `ZIO.attemptBlocking` and return `Task[A]`. The **effect-aware methods** (`TransactorZIO#connectZIO`, `TransactorZIO#transactZIO`) accept a body that itself returns a `ZIO[R, E, A]`, bracket the JDBC connection with `ZIO.acquireRelease`, and return `ZIO[R, E | Throwable, A]` — guaranteeing connection cleanup even under fiber interruption.

- **Immutable** — `TransactorZIO` holds no mutable state; each `TransactorZIO#connect` or `TransactorZIO#transact` invocation opens a fresh connection via the provided `connectionFactory`.
- **Interrupt-safe** — `TransactorZIO#connectZIO` and `TransactorZIO#transactZIO` close the underlying JDBC connection even when the enclosing fiber is interrupted, because they use `ZIO.acquireRelease` internally. Closing is not rolling back: in 0.0.56, interrupting a running `TransactorZIO#transactZIO` (for example with `ZIO#timeout`) commits the statements its body already ran, and so does a statement in the body that throws before the body's effect is returned. [How to Use the sql Module from a ZIO Application](../../guides/sql-from-zio.md#mix-database-writes-with-other-effects-in-one-transaction) shows how to avoid both.
- **Thread-safe** — the transactor is safe to share across fibers; each invocation manages its own connection.
- **JDBC-based** — available on JVM only; Scala 3 is required.

The structural shape of the class and its companion is:

```scala
class TransactorZIO(
  connectionFactory: () => Connection,
  val dialect: SqlDialect,
  val logger: SqlLogger = SqlLogger.noop
) {
  def connect[A](f: DbCon ?=> A): Task[A]
  def transact[A](f: DbTx ?=> A): Task[A]
  def transact[A](isolation: TransactionIsolation, readOnly: Boolean)(f: DbTx ?=> A): Task[A]
  def connectZIO[R, E, A](f: DbCon ?=> ZIO[R, E, A]): ZIO[R, E | Throwable, A]
  def transactZIO[R, E, A](f: DbTx ?=> ZIO[R, E, A]): ZIO[R, E | Throwable, A]
  def transactZIO[R, E, A](isolation: TransactionIsolation, readOnly: Boolean)(
    f: DbTx ?=> ZIO[R, E, A]
  ): ZIO[R, E | Throwable, A]
}

object TransactorZIO {
  def fromDataSource(dataSource: javax.sql.DataSource, dialect: SqlDialect): TransactorZIO
  def fromUrl(url: String, dialect: SqlDialect): TransactorZIO
  def fromUrl(url: String, user: String, password: String, dialect: SqlDialect): TransactorZIO
  def layer(url: String, dialect: SqlDialect): ZLayer[Any, Nothing, TransactorZIO]
}
```

## Usage

The following example demonstrates all four execution modes — a read with `TransactorZIO#connect`, an atomic write with `TransactorZIO#transact`, an effect-aware connection with `TransactorZIO#connectZIO`, and a full transactional ZIO pipeline with `TransactorZIO#transactZIO`. The transactor points at a temporary SQLite file: `TransactorZIO.fromUrl` opens a new connection for every call, and each connection to `jdbc:sqlite::memory:` would get its own empty database:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo   = Repo.derived[User, Int]("users", "id", _.id)
val dbFile = java.nio.file.Files.createTempFile("users", ".db")
dbFile.toFile.deleteOnExit()
val tx = TransactorZIO.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

// Synchronous DDL — runs on the blocking thread pool
val createTable: Task[Int] = tx.connect {
  sql"CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT NOT NULL, email TEXT NOT NULL)".update
}

// Synchronous read — runs on the blocking thread pool, no transaction overhead
val readAll: Task[List[User]] = tx.connect {
  repo.all
}

// Atomic write — commits on success, rolls back on any failure
val insertUser: Task[Int] = tx.transact {
  repo.insert(User(1, "Alice", "alice@example.com"))
}

// Effect-aware connection — body returns ZIO; connection closes even on interruption.
// connectZIO/transactZIO only run connection open/close on the blocking pool, so the
// body itself must wrap each blocking JDBC call in ZIO.attemptBlocking.
val effectRead: ZIO[Any, Throwable, List[User]] = tx.connectZIO {
  ZIO.attemptBlocking(repo.all)
}

// Effect-aware transaction — commits when the body succeeds, rolls back when it fails
val effectWrite: ZIO[Any, Throwable, Maybe[User]] = tx.transactZIO {
  for {
    _    <- ZIO.attemptBlocking(repo.insert(User(2, "Bob", "bob@example.com")))
    user <- ZIO.attemptBlocking(repo.find(2))
  } yield user
}

// ZLayer for wiring TransactorZIO through dependency injection
val transactorLayer: ZLayer[Any, Nothing, TransactorZIO] =
  TransactorZIO.layer(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
```

Each value above is a description of the work; nothing touches the database until the effects run. Running them in order creates the table, writes both users and reads them back:

```scala
Unsafe.unsafe { implicit unsafe =>
  Runtime.default.unsafe
    .run(
      for {
        _        <- createTable
        inserted <- insertUser
        bob      <- effectWrite
        all      <- readAll
        allAgain <- effectRead
      } yield (inserted, bob, all, allAgain)
    )
    .getOrThrowFiberFailure()
}
// res1: Tuple4[Int, Maybe[User], List[User], List[User]] = (
//   1,
//   User(id = 2, name = "Bob", email = "bob@example.com"),
//   List(
//     User(id = 1, name = "Alice", email = "alice@example.com"),
//     User(id = 2, name = "Bob", email = "bob@example.com")
//   ),
//   List(
//     User(id = 1, name = "Alice", email = "alice@example.com"),
//     User(id = 2, name = "Bob", email = "bob@example.com")
//   )
// )
```

The table is created with a hand-written statement because `Table#createTable` declares no `PRIMARY KEY` (see [Table](./table.md#createtable--generate-a-create-table-statement)).

## Installation

Add the `zio-blocks-sql-zio` artifact to your build. It depends on `zio-blocks-sql`, so you do not need to declare both:

```scala
// Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-sql-zio" % "0.0.56"

// Scala 3.9 and newer, which also needs -experimental
libraryDependencies += "dev.zio" %% "zio-blocks-next-sql-zio" % "0.0.56"
scalacOptions += "-experimental"
```

The artifact is JVM-only and requires Scala 3.

## Construction / Creating Instances

The `TransactorZIO` companion provides four factory methods. We use `fromDataSource` for production deployments where a connection pool is already in place, `fromUrl` for prototyping or testing, and `layer` when wiring through ZIO's dependency-injection system.

### `TransactorZIO.fromDataSource` — Create from a DataSource

`TransactorZIO.fromDataSource` is the recommended factory for production use. It calls `dataSource.getConnection()` once per `connect` or `transact` invocation, delegating all pooling concerns to the `DataSource` implementation — for example, HikariCP or c3p0:

```scala
object TransactorZIO {
  def fromDataSource(dataSource: javax.sql.DataSource, dialect: SqlDialect): TransactorZIO
}
```

The following shows how to wrap a pre-existing `DataSource` with a PostgreSQL dialect:

```scala

val dataSource: javax.sql.DataSource = ???
val tx: TransactorZIO =
  TransactorZIO.fromDataSource(dataSource, SqlDialect.PostgreSQL)
```

### `TransactorZIO.fromUrl` — Create from a JDBC URL

`TransactorZIO.fromUrl` creates a transactor that opens each connection via `java.sql.DriverManager.getConnection`. The overload without credentials is convenient for databases that embed authentication in the URL or for local testing:

```scala
object TransactorZIO {
  def fromUrl(url: String, dialect: SqlDialect): TransactorZIO
}
```

The following creates a transactor on a SQLite database file, which every call opens again:

```scala

val tx: TransactorZIO =
  TransactorZIO.fromUrl("jdbc:sqlite:app.db", SqlDialect.SQLite)
```

:::caution
`TransactorZIO.fromUrl` opens a new physical connection for every `TransactorZIO#connect` or `TransactorZIO#transact` call and does no pooling. For applications with concurrent request workloads, prefer `TransactorZIO.fromDataSource` backed by a pooling `DataSource`. Because every call gets a new connection, `jdbc:sqlite::memory:` gives each `TransactorZIO#connect`, `TransactorZIO#transact`, `TransactorZIO#connectZIO` or `TransactorZIO#transactZIO` call its own empty database: tables and rows created in one call are gone in the next. Use a file URL such as `jdbc:sqlite:/path/to/app.db` when data must outlive a single call.
:::

### `TransactorZIO.fromUrl` (with credentials) — Create from a JDBC URL with username and password

Use the credential-bearing overload when the database requires a separate username and password rather than URL-embedded authentication:

```scala
object TransactorZIO {
  def fromUrl(url: String, user: String, password: String, dialect: SqlDialect): TransactorZIO
}
```

The following creates a PostgreSQL transactor with explicit credentials:

```scala

val tx: TransactorZIO = TransactorZIO.fromUrl(
  "jdbc:postgresql://localhost/mydb",
  "alice",
  "secret",
  SqlDialect.PostgreSQL
)
```

### `TransactorZIO.layer` — Provide as a ZLayer

`TransactorZIO.layer` wraps `fromUrl` in a `ZLayer` so that `TransactorZIO` can be provided through ZIO's dependency-injection system. The layer type `ZLayer[Any, Nothing, TransactorZIO]` requires no environment and cannot fail at construction time:

```scala
object TransactorZIO {
  def layer(url: String, dialect: SqlDialect): ZLayer[Any, Nothing, TransactorZIO]
}
```

The following shows how to compose the layer with a ZIO program that reads `TransactorZIO` from its environment:

```scala

val transactorLayer: ZLayer[Any, Nothing, TransactorZIO] =
  TransactorZIO.layer("jdbc:sqlite:app.db", SqlDialect.SQLite)

val program: ZIO[TransactorZIO, Throwable, Maybe[Int]] =
  ZIO.serviceWithZIO[TransactorZIO](_.connect {
    sql"SELECT 1".queryOne[Int]
  })

val runnable: ZIO[Any, Throwable, Maybe[Int]] =
  program.provideLayer(transactorLayer)
```

When credentials are required or a `DataSource` is already available, construct the transactor with `fromUrl(url, user, password, dialect)` or `fromDataSource(ds, dialect)` and wrap the result in `ZLayer.succeed`.

## Core Operations

`TransactorZIO` exposes two pairs of methods that differ in how they handle the body and the connection lifecycle. The first pair accepts a synchronous body and delegates to the underlying `JdbcTransactor` via `ZIO.attemptBlocking`; the second pair accepts an effect-returning body and manages the connection via `ZIO.acquireRelease` for interrupt-safe cleanup.

### Connection Management

The `connect` method acquires a JDBC connection, runs a synchronous body on ZIO's blocking thread pool, and closes the connection when the body returns. Use it for read-only queries or any non-transactional access that does not require ZIO effects inside the body.

#### `connect` — Execute a synchronous body within a connection

`TransactorZIO#connect` wraps the underlying `JdbcTransactor#connect` call in `ZIO.attemptBlocking`, moving the blocking JDBC work off the ZIO main thread pool. The body receives a [`DbCon`](./db-con.md) Scala 3 context parameter carrying the active `DbConnection`, `SqlDialect`, and `SqlLogger`:

```scala
class TransactorZIO {
  def connect[A](f: DbCon ?=> A): Task[A]
}
```

Inside the block, `DbCon` is available as a given, so every [`Frag`](./frag.md) execution method and every [`Repo`](./repo.md) CRUD operation works without any explicit argument passing:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo: Repo[User, Int] = Repo.derived[User, Int]("users", "id", _.id)
val tx: TransactorZIO     = TransactorZIO.fromUrl("jdbc:sqlite:app.db", SqlDialect.SQLite)

// DbCon is synthesized by connect and threaded into all automatically
val users: Task[List[User]] = tx.connect {
  repo.all
}
```

:::caution
Inside `connect`, auto-commit is left at its driver default (typically `true`). Any DML executed here is committed immediately. Use `transact` when you need atomic multi-statement writes.
:::

### Transaction Management

The `transact` method acquires a JDBC connection, disables auto-commit, runs a synchronous body on ZIO's blocking thread pool, and commits or rolls back before closing the connection. Use it whenever writes must succeed or fail as a single atomic unit.

#### `transact` — Execute a synchronous body within a transaction

`TransactorZIO#transact` wraps the underlying `JdbcTransactor#transact` call in `ZIO.attemptBlocking`. Auto-commit is disabled before the body runs; on a normal return the connection is committed, and on any exception it is rolled back. The connection is always closed in a `finally` block regardless of outcome. The body receives a [`DbTx`](./db-tx.md) Scala 3 context parameter:

```scala
class TransactorZIO {
  def transact[A](f: DbTx ?=> A): Task[A]
  def transact[A](isolation: TransactionIsolation, readOnly: Boolean)(f: DbTx ?=> A): Task[A]
}
```

The one-argument overload keeps the driver's default isolation level and read-only flag. The two-argument overload sets the `TransactionIsolation` level and the read-only flag for this transaction and restores the previous values afterwards; [SQL Transactions: Isolation, Read-Only and Savepoints](../../guides/sql-transactions.md#transactionisolation) explains the levels. The read-only flag is best-effort: if the driver throws on `setReadOnly`, as some do, the transaction runs without it.

Because `DbTx` extends `DbCon`, any method that works inside `connect` also works inside `transact` without modification — we can promote code to transactional scope at the call site without changing helper signatures:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo: Repo[User, Int] = Repo.derived[User, Int]("users", "id", _.id)
val tx: TransactorZIO     = TransactorZIO.fromUrl("jdbc:sqlite:app.db", SqlDialect.SQLite)

// Both statements run in one transaction; failure in either rolls back the whole block
val result: Task[Maybe[User]] = tx.transact {
  repo.insert(User(1, "Alice", "alice@example.com"))
  repo.find(1)
}
```

### Effect-Aware Connection Management

The `connectZIO` method acquires a JDBC connection via `ZIO.acquireRelease`, runs an effect-returning body with `DbCon` in scope, and guarantees the connection is closed when the scope exits — including on fiber interruption. Use it when the body needs to interleave SQL queries with other ZIO operations such as logging, concurrency, or calls to external services.

#### `connectZIO` — Execute an effect-returning body within a connection

`TransactorZIO#connectZIO` accepts a body of type `DbCon ?=> ZIO[R, E, A]` and returns `ZIO[R, E | Throwable, A]`. The error channel widens to `E | Throwable` because the connection-acquisition step — a blocking JDBC call — can throw `Throwable` independently of the body's declared error type:

```scala
class TransactorZIO {
  def connectZIO[R, E, A](f: DbCon ?=> ZIO[R, E, A]): ZIO[R, E | Throwable, A]
}
```

Inside the body, `DbCon` is available as a given. `connectZIO` only wraps the connection open/close steps in `ZIO.attemptBlocking` — the body itself must wrap each blocking JDBC call the same way to avoid blocking a ZIO compute thread:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo: Repo[User, Int] = Repo.derived[User, Int]("users", "id", _.id)
val tx: TransactorZIO     = TransactorZIO.fromUrl("jdbc:sqlite:app.db", SqlDialect.SQLite)

// The body returns a ZIO; the connection remains open for the full duration of the effect
val result: ZIO[Any, Throwable, List[User]] = tx.connectZIO {
  for {
    users <- ZIO.attemptBlocking(repo.all)
    _     <- ZIO.logInfo(s"Fetched ${users.size} users from the database")
  } yield users
}
```

Unlike `connect`, the connection lifecycle here spans the entire ZIO scope. If the fiber is interrupted after the connection is opened but before the body completes, the connection is still released because `ZIO.acquireRelease` guarantees the release action runs regardless of how the scope exits.

### Effect-Aware Transaction Management

The `transactZIO` method brackets a full JDBC transaction with `ZIO.acquireRelease`: it disables auto-commit, runs an effect-returning body with `DbTx` in scope, commits on success, rolls back on failure, and closes the connection. This is the recommended method for transactional ZIO workflows.

#### `transactZIO` — Execute an effect-returning body within a transaction

`TransactorZIO#transactZIO` accepts a body of type `DbTx ?=> ZIO[R, E, A]` and returns `ZIO[R, E | Throwable, A]`. Auto-commit is disabled in the acquire step and restored to its original value in the release step. The body's exit value drives the commit/rollback decision:

```scala
class TransactorZIO {
  def transactZIO[R, E, A](f: DbTx ?=> ZIO[R, E, A]): ZIO[R, E | Throwable, A]
  def transactZIO[R, E, A](isolation: TransactionIsolation, readOnly: Boolean)(
    f: DbTx ?=> ZIO[R, E, A]
  ): ZIO[R, E | Throwable, A]
}
```

The two-argument overload sets the isolation level and the read-only flag, like the matching `TransactorZIO#transact` overload. When the body's effect succeeds, the connection is committed; when it fails with an error (`ZIO.fail`) or a defect (`ZIO.die`), the transaction is rolled back before the connection is closed. As with `connectZIO`, wrap each blocking JDBC call in the body with `ZIO.attemptBlocking`:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

val repo: Repo[User, Int] = Repo.derived[User, Int]("users", "id", _.id)
val tx: TransactorZIO     = TransactorZIO.fromUrl("jdbc:sqlite:app.db", SqlDialect.SQLite)

// The ZIO for-comprehension runs within a single JDBC transaction
val result: ZIO[Any, Throwable, Maybe[User]] = tx.transactZIO {
  for {
    _    <- ZIO.attemptBlocking(repo.insert(User(1, "Alice", "alice@example.com")))
    user <- ZIO.attemptBlocking(repo.find(1))
    _    <- ZIO.logInfo("Insert and fetch completed atomically")
  } yield user
}
```

:::danger[Interruption commits in 0.0.56]
In 0.0.56, a known bug makes `TransactorZIO#transactZIO` commit, not roll back, in two cases. Interrupting a running `TransactorZIO#transactZIO` from outside, for example with `ZIO#timeout`, `ZIO#race` or a fiber interrupt, commits the statements the body already ran, and the caller sees only the timeout or interruption. A statement written directly in the body, outside `ZIO.attemptBlocking`, runs while the body's effect is being built; if it throws, the statements before it are committed and the caller sees a defect. Wrap every statement in `ZIO.attemptBlocking`, and put time limits inside the body with `ZIO#timeoutFail` instead of around the transaction, as [How to Use the sql Module from a ZIO Application](../../guides/sql-from-zio.md#mix-database-writes-with-other-effects-in-one-transaction) shows.
:::

:::caution
If `commit` itself fails after a successful body, a rollback is attempted. If both `commit` and `rollback` fail, the rollback error is suppressed onto the commit error (via `addSuppressed`), and the commit error is the one that propagates. The body's return value is discarded in either failure case.
:::

## Comparison: `TransactorZIO` vs `JdbcTransactor`

[`JdbcTransactor`](./transactor.md) is the synchronous, non-ZIO transactor from the `sql` module. `TransactorZIO` wraps it and adds a ZIO integration layer on top. The table below summarizes when to use each:

| Aspect                   | `JdbcTransactor` (`sql` module)                                                                  | `TransactorZIO` (`sql-zio` module)                                                                                                                               |
| ------------------------ | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Return type**          | `A` (synchronous, blocking)                                                                      | `Task[A]` or `ZIO[R, E \| Throwable, A]`                                                                                                                         |
| **Thread model**         | Blocks the calling thread directly                                                               | `connect`/`transact` run on ZIO's blocking thread pool via `ZIO.attemptBlocking`                                                                                 |
| **Interruption safety**  | None — body runs to completion                                                                   | `TransactorZIO#connectZIO`/`TransactorZIO#transactZIO` close the connection on fiber interruption (an interrupted `TransactorZIO#transactZIO` commits in 0.0.56) |
| **ZIO dependency**       | None — zero ZIO runtime dependency                                                               | Requires ZIO runtime                                                                                                                                             |
| **Dependency injection** | Manual construction                                                                              | `TransactorZIO.layer` provides a `ZLayer[Any, Nothing, TransactorZIO]`                                                                                           |
| **Scala versions**       | Scala 3 only (the `sql` module is cross-built JVM + JS, but `JdbcTransactor` itself is JVM-only) | Scala 3 only (JVM only)                                                                                                                                          |
| **When to use**          | Synchronous code, scripts, or non-ZIO applications                                               | ZIO-based applications where effects compose across the entire call stack                                                                                        |

The following diagram shows how the two types relate within the `sql` and `sql-zio` modules:

```
sql module                             sql-zio module
──────────────────────────────         ───────────────────────────────────────────────────
JdbcTransactor                         TransactorZIO
  connect  { DbCon ?=> A }: A    ◄──     connect  { DbCon ?=> A }:    Task[A]
  transact { DbTx  ?=> A }: A    ◄──     transact { DbTx  ?=> A }:    Task[A]
  (no ZIO dep)                           connectZIO  { DbCon ?=> ZIO[R,E,A] }
                                         transactZIO { DbTx  ?=> ZIO[R,E,A] }
                                         layer(url, dialect): ZLayer[Any, Nothing, TransactorZIO]
```

`TransactorZIO` holds a private `JdbcTransactor` field and delegates the synchronous `connect` and `transact` calls to it, wrapping each in `ZIO.attemptBlocking`. The effect-aware `connectZIO` and `transactZIO` bypass `JdbcTransactor` entirely and manage the connection lifecycle with `ZIO.acquireRelease` directly.

## See Also

- [How to Use the sql Module from a ZIO Application](../../guides/sql-from-zio.md) — Guide to running `sql` queries and transactions as ZIO effects with `TransactorZIO`.
- [SQL — ZIO Integration](../sql-zio.md) — Overview of both ZIO integration styles and when to reach for each.

---

## Transactor

`Transactor` is the entry point for all SQL execution in the `sql` module. It declares two methods that acquire a JDBC connection, execute a provided body, and guarantee the connection is closed on return — whether the body succeeds or throws:

- **`Transactor#connect`** — supplies a `DbCon` implicit context for non-transactional queries; auto-commit remains at its default state.
- **`Transactor#transact`** — supplies a `DbTx` implicit context with auto-commit disabled; commits on success and rolls back on any exception.

`DbTx` extends `DbCon`, so every `Frag` execution method and every `Repo` CRUD operation that requires `DbCon` also works inside `transact`. The module's other core types — `Frag`, `Table`, and `Repo` — all depend on the context provided by `Transactor` to reach the database.

The structural shape of the trait is:

```scala
trait Transactor {
  def connect[A](f: DbCon ?=> A): A
  def transact[A](f: DbTx ?=> A): A
  def transact[A](isolation: TransactionIsolation, readOnly: Boolean)(f: DbTx ?=> A): A
}
```

The no-arg `transact` preserves the driver’s default isolation level and `readOnly` flag, while the two-arg overload lets callers request an explicit `TransactionIsolation` (`ReadUncommitted(1)`, `ReadCommitted(2)`, `RepeatableRead(4)`, `Serializable(8)`) and read-only mode, plus the savepoint-based nested transaction support described in `DbTx`.

`JdbcTransactor` is the concrete JDBC-backed implementation. Its companion object provides factory methods for all common connection strategies:

```scala
class JdbcTransactor(
  connectionFactory: () => java.sql.Connection,
  val dialect: SqlDialect,
  val sqlLogger: SqlLogger = SqlLogger.noop
) extends Transactor

object JdbcTransactor {
  def fromDataSource(dataSource: javax.sql.DataSource, dialect: SqlDialect): JdbcTransactor
  def fromUrl(url: String, dialect: SqlDialect): JdbcTransactor
  def fromUrl(url: String, user: String, password: String, dialect: SqlDialect): JdbcTransactor
  def postgres(dataSource: javax.sql.DataSource): JdbcTransactor
  def sqlite(dataSource: javax.sql.DataSource): JdbcTransactor
}
```

## Usage

The following block demonstrates the complete workflow: create a transactor, open a transactional scope, and execute both `Repo` operations and raw `sql"..."` fragments inside it:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

implicit val userCodec: DbCodec[User] = User.schema.deriving(DbCodecDeriver).derive

val repo = Repo.derived[User, Int]("users", "id", _.id)

// A file-backed database, so the table created in `transact` is still there in `connect`
val dbFile     = java.nio.file.Files.createTempFile("users", ".db")
val transactor = JdbcTransactor.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

// transact: auto-commit disabled; commits on success, rolls back on exception
transactor.transact {
  repo.table.createTable(summon[DbTx].dialect).update
  repo.insert(User(1, "Alice", "alice@example.com"))
  repo.insert(User(2, "Bob", "bob@example.com"))

  // Raw fragment composes freely with Repo operations inside the same scope
  val active: List[User] =
    sql"SELECT id, name, email FROM users WHERE id > ${0}".query[User]
}

// connect: auto-commit unchanged; no transaction overhead for pure reads
val users: List[User] = transactor.connect {
  repo.all
}
```

## Construction / Creating Instances

We can create a `JdbcTransactor` from a `DataSource`, from a JDBC URL, or using database-specific convenience factories. The factory methods always use `SqlLogger.noop`; to log statements, call the constructor with a `SqlLogger` as the third argument (it defaults to `SqlLogger.noop`).

### `JdbcTransactor.fromDataSource` — Create from a DataSource

`JdbcTransactor.fromDataSource` is the recommended factory for production use. It calls `dataSource.getConnection` once per `connect` or `transact` invocation, so connection pooling is fully controlled by the `DataSource` implementation (for example, HikariCP or c3p0):

```scala
object JdbcTransactor {
  def fromDataSource(dataSource: javax.sql.DataSource, dialect: SqlDialect): JdbcTransactor
}
```

Pass the `DataSource` and the `SqlDialect` constant that matches your database. The following shows a typical setup where the data source is provided externally:

```scala

val dataSource: javax.sql.DataSource = ???
val transactor: JdbcTransactor =
  JdbcTransactor.fromDataSource(dataSource, SqlDialect.PostgreSQL)
```

### `JdbcTransactor.fromUrl` — Create from a JDBC URL

`JdbcTransactor.fromUrl` creates a transactor that opens each connection via `DriverManager.getConnection`. Two overloads are available: one without credentials and one that accepts a username and password:

```scala
object JdbcTransactor {
  def fromUrl(url: String, dialect: SqlDialect): JdbcTransactor
  def fromUrl(url: String, user: String, password: String, dialect: SqlDialect): JdbcTransactor
}
```

Use the credential-free overload for databases that embed authentication in the URL (for example, SQLite in-memory) or when the JDBC driver handles authentication separately:

```scala

// Without credentials — useful for SQLite or URL-embedded auth
val sqlite: JdbcTransactor =
  JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)
// sqlite: JdbcTransactor = zio.blocks.sql.JdbcTransactor@5942e7ec
```

Use the three-argument overload when the database requires a username and password supplied separately:

```scala
// With credentials — typical for PostgreSQL or MySQL
val postgres: JdbcTransactor =
  JdbcTransactor.fromUrl(
    "jdbc:postgresql://localhost/mydb",
    "alice",
    "secret",
    SqlDialect.PostgreSQL
  )
// postgres: JdbcTransactor = zio.blocks.sql.JdbcTransactor@68d11b53
```

:::caution
`fromUrl` opens a new physical connection for every `connect` or `transact` call and does no pooling. For applications with concurrent workloads, prefer `fromDataSource` backed by a pooling `DataSource`. Because every call gets a new connection, `jdbc:sqlite::memory:` gives each `Transactor#connect` or `Transactor#transact` block its own empty database: tables and rows created in one block are gone in the next. Use a file URL such as `jdbc:sqlite:/path/to/app.db` when data must outlive a single block.
:::

### `JdbcTransactor.postgres` — PostgreSQL convenience factory

`JdbcTransactor.postgres` is shorthand for `fromDataSource(dataSource, SqlDialect.PostgreSQL)`. Use it to reduce boilerplate when working exclusively with PostgreSQL:

```scala
object JdbcTransactor {
  def postgres(dataSource: javax.sql.DataSource): JdbcTransactor
}
```

The factory fixes the dialect to `SqlDialect.PostgreSQL` so DDL type names, parameter placeholders, and dialect-specific SQL all render correctly for PostgreSQL:

```scala

val pgDataSource: javax.sql.DataSource = ???
val transactor: JdbcTransactor = JdbcTransactor.postgres(pgDataSource)
```

### `JdbcTransactor.sqlite` — SQLite convenience factory

`JdbcTransactor.sqlite` is shorthand for `fromDataSource(dataSource, SqlDialect.SQLite)`. Use it for SQLite databases where the `DataSource` is already available:

```scala
object JdbcTransactor {
  def sqlite(dataSource: javax.sql.DataSource): JdbcTransactor
}
```

This factory is useful when you wire the `DataSource` through a dependency-injection layer and want to keep dialect selection close to the data source definition rather than at each call site:

```scala

val sqDataSource: javax.sql.DataSource = ???
val transactor: JdbcTransactor = JdbcTransactor.sqlite(sqDataSource)
```

## Core Operations

`Transactor` exposes exactly two public methods. Together they cover the two modes of database access: non-transactional connections for reads and transactional connections for writes.

### Connection Management

`Transactor#connect` is the lightweight entry point for database access without a transaction. It provides a `DbCon` context, which is the implicit argument required by every `Frag` extension method and every `Repo` CRUD operation.

It acquires a connection from the underlying factory, wraps it in a `JdbcConnection`, synthesizes a `DbCon` given value, and executes the provided body. The connection is closed in a `finally` block, so it is released whether the body returns normally or throws:

```scala
trait Transactor {
  def connect[A](f: DbCon ?=> A): A
}
```

Because `DbTx extends DbCon`, a block that compiles under `connect` will also compile under `transact` — so we can promote non-transactional code to transactional scope without changing the body. The following example runs a read query inside `connect` without incurring transaction overhead:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

implicit val userCodec: DbCodec[User] = User.schema.deriving(DbCodecDeriver).derive

val repo       = Repo.derived[User, Int]("users", "id", _.id)
val transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)

// The body receives DbCon as a given — no explicit passing required
val users: List[User] = transactor.connect {
  repo.all
}
```

:::caution
Inside `connect`, auto-commit is left at its default state (typically `true` for JDBC). Any DML statement executed here is committed immediately. If you need atomic multi-statement writes, use `transact` instead.
:::

### Transaction Management

`Transactor#transact` builds on `connect` by adding full transaction semantics: it disables auto-commit before executing the body, commits when the body returns normally, and rolls back when the body throws. The connection is always closed after commit or rollback. Two overloads are available — a no-arg form that preserves the driver’s defaults and a two-arg form that sets isolation and `readOnly` explicitly:

```scala
trait Transactor {
  def transact[A](f: DbTx ?=> A): A
  def transact[A](isolation: TransactionIsolation, readOnly: Boolean)(f: DbTx ?=> A): A
}
```

The two-arg overload acquires a connection, saves the previous isolation level and `readOnly` flag, calls `setTransactionIsolation(isolation.jdbcLevel)` (fail-fast — isolation failures propagate) and best-effort `setReadOnly(readOnly)` (SQLite may throw, which is swallowed), sets `autoCommit = false` (fail-fast), synthesizes a `DbTx` given value, and runs the body. On success it calls `conn.commit()`. On any exception it calls `conn.rollback()`, adds any rollback failure as a suppressed exception, and rethrows the original. Previous isolation, `readOnly`, and `autoCommit` are restored and the connection is closed in the `finally` block regardless of outcome. SQLite natively supports only `SERIALIZABLE` — other levels are accepted via `setTransactionIsolation` but the engine still behaves as serializable.

The no-arg overload preserves driver defaults (e.g. PostgreSQL `READ_COMMITTED`) and does not force `SERIALIZABLE`; it simply sets `autoCommit = false` before running the body, with the same commit/rollback and cleanup guarantees.

Because `DbTx extends DbCon`, the `DbTx` given satisfies any `DbCon ?=>` requirement inside the block. We can mix `Repo` CRUD calls with raw `sql"..."` fragments freely:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

implicit val userCodec: DbCodec[User] = User.schema.deriving(DbCodecDeriver).derive
// userCodec: DbCodec[User] = zio.blocks.sql.DbCodecDeriver$$anon$10@3d23457f

val repo       = Repo.derived[User, Int]("users", "id", _.id)
// repo: Repo[User, Int] = zio.blocks.sql.Repo$DerivedRepo@1df2c8bf
val transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)
// transactor: JdbcTransactor = zio.blocks.sql.JdbcTransactor@58dd5361

transactor.transact {
  repo.table.createTable(summon[DbTx].dialect).update
  repo.insert(User(1, "Alice", "alice@example.com"))

  // If this throws, the INSERT above is rolled back
  val existing: Maybe[User] = repo.find(1)
  existing
}
// res7: Maybe[User] = User(
//   id = 1,
//   name = "Alice",
//   email = "alice@example.com"
// )
```

:::caution
If `commit` itself throws after a successful body, the transaction is rolled back and the commit exception propagates. The body's return value is discarded in that case.
:::

### Nested Transactions (Savepoints)

Inside `transact`, nested work is emulated via SQL savepoints on the same JDBC connection through `DbTx` — `summon[DbTx].transact { ... }`, `DbTx.transactNested`, or the top-level `transactNested` helper. Savepoints are named `zib_tx_1 .. zib_tx_N` with depth tracked in `DbTx.currentDepth`; inner success issues `RELEASE SAVEPOINT`, failure issues `ROLLBACK TO SAVEPOINT` then rethrows. See [DbTx](db-tx) for full savepoint semantics.

## JdbcTransactor

`JdbcTransactor` is the only concrete implementation of `Transactor` in the `sql` module. It holds three constructor parameters: `connectionFactory`, `dialect`, and `sqlLogger`. The factory methods on its companion object cover the most common configurations; the primary constructor is available for custom setups such as test harnesses that inject a pre-existing connection:

```
Transactor  (trait — shared, cross-platform)
    └── JdbcTransactor  (class — JVM only, JDBC-backed)
```

You can subclass `JdbcTransactor` to override `connect` or `transact` — for example, to reuse a single shared connection across multiple calls in a test suite without closing it between calls, as the `TransactorSpec` test helper does internally.

## Transactor vs TransactorZIO

`Transactor` and `JdbcTransactor` are synchronous and suitable for code that does not use the ZIO effect system. `TransactorZIO`, provided by the separate `zio-blocks-sql-zio` artifact, wraps a `JdbcTransactor` and exposes two execution models for ZIO applications:

| Aspect                   | `Transactor` / `JdbcTransactor`                | `TransactorZIO` (sql-zio module)                                                                  |
|--------------------------|------------------------------------------------|---------------------------------------------------------------------------------------------------|
| **Return type**          | `A` (synchronous, blocking)                    | `Task[A]` (or `ZIO[R, E \| Throwable, A]` for ZIO bodies)                                         |
| **Thread model**         | Blocks the calling thread                      | `connect`/`transact` run on the blocking thread pool via `ZIO.attemptBlocking`                    |
| **Interruption safety**  | None — the body runs to completion             | `connectZIO`/`transactZIO` use `ZIO.acquireRelease`; connection closes even on fiber interruption |
| **ZIO dependency**       | None — zero ZIO dependency                     | Requires ZIO runtime                                                                              |
| **Dependency injection** | Manual construction                            | `TransactorZIO.layer` provides a `ZLayer`                                                         |
| **When to use**          | Synchronous imperative code, scripts, or tests | ZIO-based applications where effects compose across the entire call stack                         |

The following diagram shows how the two types relate:

```
JdbcTransactor  ──────────── wraps ───────────►  TransactorZIO
     │                                                  │
     │ connect  { DbCon ?=> A }: A                      │ connect  { DbCon ?=> A }:    Task[A]
     │ transact { DbTx  ?=> A }: A                      │ transact { DbTx  ?=> A }:    Task[A]
     │                                                  │ connectZIO  { DbCon ?=> ZIO[R,E,A] }
     │                                                  │ transactZIO { DbTx  ?=> ZIO[R,E,A] }
     │                                                  │
     └── (no ZIO dep) ────────────────────────── (ZIO runtime required) ──┘
```

Choose `JdbcTransactor` when the codebase is synchronous or when ZIO is not on the classpath. Choose `TransactorZIO` when you want connections bracketed by ZIO's `acquireRelease` so they survive fiber interruption, or when you need `ZLayer`-based dependency injection.

## Advanced Usage: Composing the Context

`DbCon` carries three members — `connection`, `dialect`, and `logger` — and is passed implicitly through every SQL operation. This means application-level helper methods can declare `(using DbCon)` and they are automatically satisfied anywhere inside `connect` or `transact` with no explicit argument passing. The same pattern composes across multiple layers:

```scala

case class User(id: Int, name: String, email: String)
object User { implicit val schema: Schema[User] = Schema.derived }

implicit val userCodec: DbCodec[User] = User.schema.deriving(DbCodecDeriver).derive
// userCodec: DbCodec[User] = zio.blocks.sql.DbCodecDeriver$$anon$10@6a534f56

val repo       = Repo.derived[User, Int]("users", "id", _.id)
// repo: Repo[User, Int] = zio.blocks.sql.Repo$DerivedRepo@734e2a1f
val transactor = JdbcTransactor.fromUrl("jdbc:sqlite::memory:", SqlDialect.SQLite)
// transactor: JdbcTransactor = zio.blocks.sql.JdbcTransactor@43c53042

// Helper that composes two Repo calls — requires only DbCon, not Transactor
def upsertUser(user: User)(using DbCon): Unit = {
  if (repo.exists(user.id)) repo.update(user)
  else repo.insert(user)
}

// The Transactor provides the DbCon; upsertUser picks it up automatically
transactor.transact {
  repo.table.createTable(summon[DbTx].dialect).update
  upsertUser(User(1, "Alice", "alice@example.com"))
  upsertUser(User(1, "Alice Smith", "alice.smith@example.com"))
  repo.find(1)
}
// res9: Maybe[User] = User(
//   id = 1,
//   name = "Alice Smith",
//   email = "alice.smith@example.com"
// )
```

Because `DbTx extends DbCon`, `upsertUser` works unchanged inside both `connect` and `transact`. This lets us design helper functions against the minimal context they need and promote them to transactional scope at the call site without modifying their signatures.

## See Also

- **[How to Test Database Code Against SQLite](../../guides/test-database-code-against-sqlite.md)** — Guide to giving each ZIO Test spec its own SQLite database and rolling back writes.
- **[How to Connect to PostgreSQL with a Connection Pool](../../guides/connect-to-postgresql-with-a-connection-pool.md)** — Builds a HikariCP pool, passes it to `JdbcTransactor.postgres`, sizes the pool and closes it with a `Scope`.
- **[SQL Transactions](../../guides/sql-transactions.md)** — Isolation levels, read-only transactions and nested transactions with savepoints for `Transactor#transact`.
- **[How to Stream Large Query Results](../../guides/stream-large-query-results.md)** — Shows how to consume a `queryStream` inside `connect` or `transact`, before the connection closes.

---

## ZIO Integration

`zio-blocks-sql-zio` is the ZIO adapter for the [`sql` module](./sql/index.md). It lets a ZIO application use the same `sql"..."` fragments, `Repo` and `Table` values as a plain Scala program, and `TransactorZIO#connect` and `TransactorZIO#transact` run their JDBC calls off ZIO's main thread pool.

The module offers two integration styles. The `TransactorZIO` class wraps the synchronous transactor in methods that return ZIO effects; it is the one to use in ZIO code. The `JdbcTransactor.postgresLayer` and `JdbcTransactor.sqliteLayer` extension methods build a `ZLayer` of the synchronous `Transactor` from a `DataSource`, for code that already works with `Transactor`. This page gives an overview of both; [TransactorZIO](./sql/transactor-zio.md) documents each method, and [How to Use the sql Module from a ZIO Application](../guides/sql-from-zio.md) builds a complete ZIO service on it.

## Installation

Add `sql-zio` to your `build.sbt`. It brings in the `sql` module and ZIO:

```scala
// Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-sql-zio" % "0.0.56"

// Scala 3.9 and newer, which also needs -experimental
libraryDependencies += "dev.zio" %% "zio-blocks-next-sql-zio" % "0.0.56"
scalacOptions += "-experimental"
```

`sql` and `sql-zio` are published for Scala 3 only and run on the JVM. You also need the JDBC driver of your database; the examples on this page use SQLite (`"org.xerial" % "sqlite-jdbc"`).

## Imports

The examples import ZIO, the ZIO integration and the `sql` module, in this order:

```scala

```

:::warning[Put `import zio.blocks.sql._` last]
With `sql-zio` on the classpath, `zio.blocks.sql` has a subpackage named `zio`: this module's package, `zio.blocks.sql.zio`. After `import zio.blocks.sql._`, the name `zio` means that subpackage instead of the root `zio` package, so any later import or name that starts with `zio`, such as `import zio.blocks.schema.Schema`, fails with `value blocks is not a member of zio.blocks.sql.zio`. Put `import zio.blocks.sql._` after every other `zio` import, or write `_root_.zio` where you need the root package after it.
:::

We run the effects on this page with a small helper; in an application, `ZIOAppDefault#run` does this for you:

```scala
def runDemo[A](effect: ZIO[Any, Any, A]): A =
  Unsafe.unsafe { implicit unsafe =>
    Runtime.default.unsafe.run(effect).getOrThrowFiberFailure()
  }
```

## Quick Start

`TransactorZIO.fromUrl` takes a JDBC URL and a dialect. We point it at a temporary SQLite file and derive a repository for a `Book` record:

```scala
case class Book(id: Long, title: String, year: Int)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

val books: Repo[Book, Long] = Repo.derived[Book, Long]

val dbFile = java.nio.file.Files.createTempFile("library", ".db")
dbFile.toFile.deleteOnExit()

val transactor: TransactorZIO = TransactorZIO.fromUrl(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)
```

`TransactorZIO#connect` and `TransactorZIO#transact` take the same synchronous body as `JdbcTransactor#connect` and `JdbcTransactor#transact`, run it on ZIO's blocking thread pool and return a `Task`. We create the table, insert two books in one transaction, then read them back:

```scala
runDemo {
  for {
    _   <- transactor.connect {
             sql"CREATE TABLE book (id INTEGER PRIMARY KEY, title TEXT NOT NULL, year INTEGER NOT NULL)".update
           }
    _   <- transactor.transact {
             books.insert(Book(1L, "Zionomicon", 2024))
             books.insert(Book(2L, "Programming in Scala", 2008))
           }
    all <- transactor.connect(books.all)
  } yield all
}
// res1: List[Book] = List(
//   Book(id = 1L, title = "Zionomicon", year = 2024),
//   Book(id = 2L, title = "Programming in Scala", year = 2008)
// )
```

The table is written by hand because `Table#createTable` declares no `PRIMARY KEY` (see [Table](./sql/table.md#createtable--generate-a-create-table-statement)).

:::warning
Don't use `jdbc:sqlite::memory:` with `TransactorZIO.fromUrl`. It opens a new connection for every `TransactorZIO#connect`, `TransactorZIO#transact`, `TransactorZIO#connectZIO` and `TransactorZIO#transactZIO` call, and each new connection to `:memory:` gets its own empty database, so the table created in one call is gone in the next. Use a file, as above, or a pooled `DataSource` with `TransactorZIO.fromDataSource`.
:::

## Effect-Aware Methods

`TransactorZIO#connectZIO` and `TransactorZIO#transactZIO` take a body that returns a `ZIO` effect, so a transaction can include calls to other ZIO services. `TransactorZIO#transactZIO` commits when the effect succeeds and rolls back when it fails.

Only the connection setup, the commit and the rollback run on the blocking pool; the body's effect runs on the fiber's own thread. Wrap every statement in the body in `ZIO.attemptBlocking`, which moves it to the blocking pool and turns its exception into a failure that triggers the rollback:

```scala
runDemo {
  for {
    added <- transactor.transactZIO {
               for {
                 _    <- ZIO.attemptBlocking(books.insert(Book(3L, "Functional Programming in Scala", 2014)))
                 book <- ZIO.attemptBlocking(books.find(3L))
               } yield book
             }
    failed <- transactor.transactZIO {
                ZIO.attemptBlocking(books.insert(Book(4L, "Unpublished", 2030))) *>
                  ZIO.fail(new RuntimeException("payment declined"))
              }.either
    count <- transactor.connect(books.count)
  } yield (added, failed.left.map(_.getMessage), count)
}
// res2: Tuple3[Maybe[Book], Either[String, Nothing], Long] = (
//   Book(id = 3L, title = "Functional Programming in Scala", year = 2014),
//   Left("payment declined"),
//   3L
// )
```

The second transaction inserted a book, then failed, so `TransactorZIO#transactZIO` rolled the insert back and the table holds three books.

:::danger[Two ways `TransactorZIO#transactZIO` commits instead of rolling back]
In 0.0.56, a known bug makes `TransactorZIO#transactZIO` commit, not roll back, in two cases. A statement written directly in the body, outside `ZIO.attemptBlocking`, runs while the body's effect is being built; if it throws, the statements before it are committed. And interrupting a running `TransactorZIO#transactZIO` from outside, for example with `ZIO#timeout` or `ZIO#race`, commits the statements that already ran. Wrap every statement in `ZIO.attemptBlocking`, and put time limits inside the body with `ZIO#timeoutFail` instead of around the transaction, as [How to Use the sql Module from a ZIO Application](../guides/sql-from-zio.md#mix-database-writes-with-other-effects-in-one-transaction) shows.
:::

## Providing a Transactor as a ZLayer

Both integration styles can come from a `ZLayer`, so the rest of the application only names the service it needs.

### `TransactorZIO.layer`

`TransactorZIO.layer` builds a `ZLayer[Any, Nothing, TransactorZIO]` from a JDBC URL and a dialect. Code that needs the database reads `TransactorZIO` from its environment:

```scala
val transactorLayer: ZLayer[Any, Nothing, TransactorZIO] =
  TransactorZIO.layer(s"jdbc:sqlite:$dbFile", SqlDialect.SQLite)

val titles: ZIO[TransactorZIO, Throwable, List[String]] =
  ZIO.serviceWithZIO[TransactorZIO](_.connect(books.all.map(_.title)))
```

Providing the layer gives an effect that needs nothing else:

```scala
runDemo(titles.provide(transactorLayer))
// res3: List[String] = List(
//   "Zionomicon",
//   "Programming in Scala",
//   "Functional Programming in Scala"
// )
```

For a connection pool, build the transactor with `TransactorZIO.fromDataSource` inside a layer, as [How to Use the sql Module from a ZIO Application](../guides/sql-from-zio.md#provide-the-database-as-a-zlayer) does with HikariCP.

### `JdbcTransactor.postgresLayer` and `JdbcTransactor.sqliteLayer`

`JdbcTransactor.postgresLayer` and `JdbcTransactor.sqliteLayer` turn a layer of a `javax.sql.DataSource` into a layer of the synchronous [`Transactor`](./sql/transactor.md), with the PostgreSQL or SQLite dialect. They are extension methods defined in `zio.blocks.sql.zio`, so they need `import zio.blocks.sql.zio._`.

A `Transactor` runs its body on the calling thread, so each call must be wrapped in `ZIO.attemptBlocking`; otherwise the JDBC call blocks one of ZIO's main threads. Here the `DataSource` is SQLite's own `SQLiteDataSource`:

```scala
val dataSourceLayer: ZLayer[Any, Nothing, javax.sql.DataSource] =
  ZLayer.succeed {
    val dataSource = new org.sqlite.SQLiteDataSource()
    dataSource.setUrl(s"jdbc:sqlite:$dbFile")
    dataSource: javax.sql.DataSource
  }

val syncTransactorLayer: ZLayer[Any, Nothing, Transactor] =
  dataSourceLayer >>> JdbcTransactor.sqliteLayer

val bookCount: ZIO[Transactor, Throwable, Long] =
  ZIO.serviceWithZIO[Transactor](syncTransactor => ZIO.attemptBlocking(syncTransactor.connect(books.count)))
```

The effect counts the books through the synchronous transactor:

```scala
runDemo(bookCount.provide(syncTransactorLayer))
// res4: Long = 3L
```

Prefer `TransactorZIO` in ZIO code: it does the `ZIO.attemptBlocking` wrapping for you.

## Thread Safety

`TransactorZIO` holds no mutable state and is safe to share between fibers. It opens a new connection for each `TransactorZIO#connect`, `TransactorZIO#transact`, `TransactorZIO#connectZIO` and `TransactorZIO#transactZIO` call and closes it afterwards. Because each call has its own connection, two calls never share a transaction; to run several operations in one transaction, call them inside one `TransactorZIO#transact` or `TransactorZIO#transactZIO`.

## See Also

- [TransactorZIO](./sql/transactor-zio.md) — Reference for every `TransactorZIO` method.
- [How to Use the sql Module from a ZIO Application](../guides/sql-from-zio.md) — Guide to running `sql` queries and transactions as ZIO effects with `TransactorZIO`.

---

## Core Types

`Stream`, `Pipeline`, and `Sink` are the three types you compose to build a pipeline. Each is a lazy, immutable description rather than a running process: you assemble one with ordinary combinators, and nothing executes until a terminal operation such as `stream.runAsync(sink)` — or `stream.run(sink)`, which is JVM-only — compiles and drives it.

Compiling is what materializes a description into a [`Reader`](../primitives/reader.md), the stateful cursor a sink pulls from. Those live primitives are documented under [Low-Level Primitives](../primitives/index.md); this section covers the descriptions you write.

## Stream

[`Stream[+E, +A]`](./stream.md) describes a source of elements of type `A` that may fail with `E`. Values are produced lazily, so a stream can outlive what fits in memory and can hold an external resource open only while it is being drained.

## Pipeline

[`Pipeline[-In, +Out]`](./pipeline.md) describes a reusable transformation from `In` to `Out`. Reach for it when the same steps apply to more than one stream. Pipelines compose with `andThen`, apply to a stream with `stream.via(pipe)`, and to a sink with `pipe.andThenSink(sink)`.

## Sink

[`Sink[+E, -A, +Z]`](./sink.md) describes how to consume elements of type `A` into a result `Z`, possibly failing with `E` — the endpoint of a pipeline, built once and reusable across streams.

## See Also

- [Streams Reference](../index.md) — module overview
- [Low-Level Primitives](../primitives/index.md) — the `Reader` and `Writer` cursors these compile into

---

## Pipeline

`Pipeline[-In, +Out]` is a **reusable, composable stream transformation** that converts elements of type `In` into elements of type `Out`. Pipelines are first-class values: you can define them once, compose them with `andThen`, and apply them to any [Stream](./stream.md) via `stream.via(pipe)` or to any `Sink` via `pipe.andThenSink(sink)`.

`Pipeline`:
- Is contravariant in `In` and covariant in `Out` (like a function `In => Out`)
- Can be applied to a **Stream** (transforming the output) or a **Sink** (pre-processing the input)
- Participates in JVM primitive specialization to avoid boxing

Here is the structural shape of the `Pipeline` type:

```scala
abstract class Pipeline[-In, +Out] {
  def andThen[C](that: Pipeline[Out, C]): Pipeline[In, C]
  def applyToStream[E](stream: Stream[E, In]): Stream[E, Out]
  def applyToSink[E, Z](sink: Sink[E, Out, Z]): Sink[E, In, Z]
}
```

## Overview

Pipelines solve the problem of reusing stream transformations across different streams and sinks. Without pipelines, you repeat filtering and mapping logic for every stream. With pipelines, you define transformations once as first-class values, compose them freely, and apply them anywhere.

### The Problem

When you build stream processing logic, you often write chains like:

```scala
stream
  .filter(_ > 0)
  .map(_ * 2)
  .take(100)
```

This works, but the transformation is tied to a specific stream. If you want to apply the same logic to a different stream, or to a sink instead, you have to repeat yourself. You cannot pass the chain around as a value, store it in a variable, or compose it with other transformations.

### The Solution

`Pipeline[-In, +Out]` lifts stream transformations into first-class values. You define a pipeline once, compose it with other pipelines using `andThen`, and apply it wherever you need:

```scala

// Define once
val normalize: Pipeline[Int, Int] =
  Pipeline.filter[Int](_ > 0)
    .andThen(Pipeline.map[Int, Int](_ * 2))
    .andThen(Pipeline.take(100))

// Apply to any stream
val stream1 = Stream(1, 2, 3, 4, 5)
val stream2 = Stream(10, 20, 30, 40, 50)
val stream3 = Stream(-5, 3, 7, 2, 8, 1, 9)

val result1 = stream1.via(normalize).runCollect
val result2 = stream2.via(normalize).runCollect

// Apply to a sink (pre-process input before the sink sees it)
val normalizedSink = normalize.andThenSink(Sink.collectAll[Int])
val result3 = stream3.run(normalizedSink)
```

`Pipeline` forms a **category** in the mathematical sense:

| Law            | Statement                                            |
|----------------|------------------------------------------------------|
| Left identity  | `Pipeline.identity andThen p == p`                   |
| Right identity | `p andThen Pipeline.identity == p`                   |
| Associativity  | `(p andThen q) andThen r == p andThen (q andThen r)` |

These laws guarantee that pipelines compose predictably, regardless of how you parenthesize.

### Architecture

`Pipeline` sits between `Stream` and `Sink`, mediating how elements flow:

```
  Applying to a Stream (via):
  ┌──────────────┐     ┌──────────────────┐     ┌──────────────┐
  │ Stream[E, In]│ ──→ │ Pipeline[In, Out]│ ──→ │Stream[E, Out]│
  └──────────────┘     └──────────────────┘     └──────────────┘

  Applying to a Sink (andThenSink):
  ┌──────────────────┐     ┌────────────────┐     ┌──────────────┐
  │ Pipeline[In, Out]│ ──→ │ Sink[E, Out, Z]│ ──→ │Sink[E, In, Z]│
  └──────────────────┘     └────────────────┘     └──────────────┘

  Composing two Pipelines (andThen):
  ┌──────────────────┐     ┌──────────────────┐     ┌─────────────────┐
  │ Pipeline[A, B]   │ ──→ │ Pipeline[B, C]   │ ──→ │ Pipeline[A, C]  │
  └──────────────────┘     └──────────────────┘     └─────────────────┘
```

## Construction

Pipelines are built using factory methods on the `Pipeline` companion object. Each factory creates a pipeline that performs a specific transformation: mapping elements, filtering, collecting, or controlling flow. A factory that applies a user-supplied function to produce a new element type asks for `JvmType.Infer` evidence for the **result** type; the one exception is the `map` overload for a function returning `Nothing`, which takes a `DummyImplicit` and supplies the boxed lane itself. The input representation already belongs to the stream or sink at application time; callers do not supply redundant input evidence. Type-preserving factories, and the structural factories `buffer` and `chunked`, retain or rebuild that representation without call-site evidence.

Three of those factories have asynchronous twins, and exactly three: `mapAsync`, `filterAsync`, and `collectAsync`. They are cross-platform, their callbacks return [`Async`](../../async.md), they are evaluated sequentially as the downstream pulls, and they work through both application routes. Other `*Async` operators such as `tapEachAsync` and `distinctByAsync` exist on `Stream` rather than on `Pipeline`; see [Asynchronous Stream Execution](../execution-and-compatibility/async-execution.md#async-operators) for the full stream-level list. `Writer`'s similarly named deferred methods are something else again — adaptation wrappers around its synchronous operations, described in [Writer — Asynchronous Writes](../primitives/writer.md#asynchronous-writes).

Factory callbacks are protected as user callbacks: synchronous throws and failed callback effects are defects, not typed stream errors. When a pipeline is applied to a sink, both drain paths close the derived reader. Cancellation closes upstream and downstream state, and a cleanup failure is attached to an existing failure rather than hiding it.

### `Pipeline.map[A, B]` — Transform Each Element

Applies a function to every element, producing a new element type. Here is the signature:

```scala
object Pipeline {
  def map[A, B](f: A => B)(implicit jtB: JvmType.Infer[B]): Pipeline[A, B]
}
```

This is the most common pipeline constructor:

```scala

val doubler = Pipeline.map[Int, Int](_ * 2)
// doubler: Pipeline[Int, Int] = zio.blocks.streams.Pipeline$MapPipeline@138ad455
val toStr = Pipeline.map[Int, String](_.toString)
// toStr: Pipeline[Int, String] = zio.blocks.streams.Pipeline$MapPipeline@495e8e9a

val result = Stream(1, 2, 3).via(doubler).runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(2, 4, 6))
```

### `Pipeline.mapAsync[A, B]` — Transform Each Element Asynchronously

The asynchronous twin of `map`. The callback returns an `Async`, and each one is awaited before the next element is pulled, so at most one invocation is in flight and input order is preserved. Here is the signature:

```scala
object Pipeline {
  def mapAsync[A, B](f: A => Async[B])(implicit jtB: JvmType.Infer[B]): Pipeline[A, B]
}
```

Like `map`, it asks for `JvmType.Infer` evidence for its result type only. A failed or defective `Async` fails the stream as a defect rather than through its typed error channel, and cancellation of a suspended callback is propagated:

```scala

val lookup = Pipeline.mapAsync[Int, String](id => Async.succeed(s"user-$id"))
// lookup: Pipeline[Int, String] = zio.blocks.streams.Pipeline$$anon$4@345e7e56

val result = Stream(1, 2, 3).via(lookup).runCollectAsync
// result: Async[Either[Nothing, Chunk[String]]] = zio.blocks.async.Async$slowPath$FoldCausePollable@7af2d0ad
```

### `Pipeline.filter[A]` — Keep Matching Elements

Keeps only elements that satisfy a predicate. Here is the signature:

```scala
object Pipeline {
  def filter[A](pred: A => Boolean): Pipeline[A, A]
}
```

Note that the output type is the same as the input type — filtering does not change the element type:

```scala

val positives = Pipeline.filter[Int](_ > 0)
// positives: Pipeline[Int, Int] = zio.blocks.streams.Pipeline$FilterPipeline@17eb2247

val result = Stream(-2, -1, 0, 1, 2).via(positives).runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2))
```

### `Pipeline.filterAsync[A]` — Keep Matching Elements Asynchronously

The asynchronous twin of `filter`. Elements are tested sequentially and in input order, and an element is emitted only when its `Async` yields `true`. Here is the signature:

```scala
object Pipeline {
  def filterAsync[A](f: A => Async[Boolean]): Pipeline[A, A]
}
```

Like `filter`, it is type-preserving and therefore takes no `JvmType.Infer` evidence — it keeps whatever representation the stream or sink supplies. A failed predicate effect is a defect, and cancellation of a suspended predicate is propagated:

```scala

val allowed = Pipeline.filterAsync[Int](n => Async.succeed(n % 3 == 0))
// allowed: Pipeline[Int, Int] = zio.blocks.streams.Pipeline$$anon$2@66ff1dd2

val result = Stream(1, 2, 3, 4, 5, 6).via(allowed).runCollectAsync
// result: Async[Either[Nothing, Chunk[Int]]] = zio.blocks.async.Async$slowPath$FoldCausePollable@584bb6af
```

### `Pipeline.collect[A, B]` — Partial Function Transformation

Applies a partial function: only elements for which the function is defined pass through, and they are transformed to the output type. This combines filtering and mapping in one step. Here is the signature:

```scala
object Pipeline {
  def collect[A, B](pf: PartialFunction[A, B])(implicit jtB: JvmType.Infer[B]): Pipeline[A, B]
}
```

Use `collect` when you need to filter and transform simultaneously:

```scala

val extractInts = Pipeline.collect[Any, Int] { case n: Int => n }
// extractInts: Pipeline[Any, Int] = zio.blocks.streams.Pipeline$CollectPipeline@6c40363b

val result = Stream(1, "a", 2, "b", 3).via(extractInts).runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2, 3))
```

### `Pipeline.collectAsync[A, B]` — Asynchronous Filtering Transformation

The asynchronous twin of `collect`, with one shape difference worth noticing: where `collect` takes a `PartialFunction[A, B]`, `collectAsync` takes a total function returning `Async[Option[B]]`. A `None` drops the element and a `Some` emits its value. Here is the signature:

```scala
object Pipeline {
  def collectAsync[A, B](f: A => Async[Option[B]])(implicit jtB: JvmType.Infer[B]): Pipeline[A, B]
}
```

Elements are evaluated sequentially and in input order, and like `collect` the factory asks for `JvmType.Infer` evidence for its result type:

```scala

val parsePort = Pipeline.collectAsync[String, Int] { raw =>
  Async.succeed(raw.toIntOption.filter(p => p > 0 && p <= 65535))
}
// parsePort: Pipeline[String, Int] = zio.blocks.streams.Pipeline$$anon$1@43a86d93

val result = Stream("80", "not-a-port", "8080", "99999").via(parsePort).runCollectAsync
// result: Async[Either[Nothing, Chunk[Int]]] = zio.blocks.async.Async$slowPath$FoldCausePollable@2a65412f
```

### `Pipeline.take[A]` — First N Elements

Passes through at most the first `n` elements, then stops. Here is the signature:

```scala
object Pipeline {
  def take[A](n: Long): Pipeline[A, A]
}
```

This naturally short-circuits — upstream stops producing once `n` elements have passed:

```scala

val firstFive = Pipeline.take[Int](5)
// firstFive: Pipeline[Int, Int] = zio.blocks.streams.Pipeline$TakePipeline@506102b2

val result = Stream.range(0, 1000).via(firstFive).runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(0, 1, 2, 3, 4))
```

### `Pipeline.drop[A]` — Skip First N Elements

Skips the first `n` elements, then passes through the rest. Here is the signature:

```scala
object Pipeline {
  def drop[A](n: Long): Pipeline[A, A]
}
```

Use `drop` to skip headers, metadata, or warm-up elements:

```scala

val skipHeader = Pipeline.drop[String](1)
// skipHeader: Pipeline[String, String] = zio.blocks.streams.Pipeline$DropPipeline@2b0949c0

val result = Stream("header", "row1", "row2").via(skipHeader).runCollect
// result: Either[Nothing, Chunk[String]] = Right(IndexedSeq("row1", "row2"))
```

### `Pipeline.identity[A]` — Pass-Through

The identity pipeline that passes all elements through unchanged. This is the neutral element for `andThen` composition. Here is the signature:

```scala
object Pipeline {
  def identity[A]: Pipeline[A, A]
}
```

You rarely construct `identity` explicitly, but it is important as a base case in generic pipeline-building code:

```scala

val noOp = Pipeline.identity[Int]
// noOp: Pipeline[Int, Int] = zio.blocks.streams.Pipeline$$anon$3@5795b0eb

// These are equivalent:
// stream.via(noOp)  ==  stream
// noOp.andThen(p)   ==  p
// p.andThen(noOp)   ==  p
```

## Composing Pipelines

Pipelines compose into larger, more complex transformations using `andThen`. Because `Pipeline` forms a mathematical category, composition is associative and respects identity, so you can build pipelines incrementally or conditionally without worrying about how you parenthesize or combine them.

### `Pipeline#andThen[C]` — Sequential Composition

Composes two pipelines into one, applying `this` first and `that` second. Here is the signature:

```scala
abstract class Pipeline[-In, +Out] {
  def andThen[C](that: Pipeline[Out, C]): Pipeline[In, C]
}
```

`andThen` is the key operation that makes pipelines composable. Because `Pipeline` forms a category, composition is associative — you can group `andThen` calls however you like and get the same result:

```scala

// Individual steps
val filterPositive = Pipeline.filter[Int](_ > 0)
val double         = Pipeline.map[Int, Int](_ * 2)
val takeFirst10    = Pipeline.take[Int](10)

// Compose into a single reusable pipeline
val normalize = filterPositive
  .andThen(double)
  .andThen(takeFirst10)

// Apply to any stream
val result = Stream(-5, 3, -1, 7, 2, 0, 9, 4, 8, 6, 1, 10)
  .via(normalize)
  .runCollect
```

### Building Pipelines Conditionally

Because pipelines are values, you can build them dynamically:

```scala

def buildPipeline(limit: Option[Int], onlyPositive: Boolean): Pipeline[Int, Int] = {
  val base = if (onlyPositive) Pipeline.identity[Int].andThen(Pipeline.filter(_ > 0)) else Pipeline.identity[Int]
  limit.fold(base)(n => base.andThen(Pipeline.take(n.toLong)))
}
```

## Applying to a Stream

Apply a pipeline to a stream using `via` to transform its output elements. This is the most direct way to use a pipeline: define it once and apply it to multiple streams without repeating the transformation logic.

Applying an asynchronous pipeline stage to a synchronous stream makes the resulting stream asynchronous. There is nothing to annotate and no type that changes: `Stream[E, A]` carries no marker for which way it will materialize, so `syncStream.via(Pipeline.mapAsync(...))` has exactly the type `syncStream.via(Pipeline.map(...))` would have. What changes is how the description compiles — one asynchronous node makes the whole graph compile to an asynchronous reader, with the synchronous source lifted in place. On the JVM a blocking terminal still accepts the result; on Scala.js use an `*Async` terminal. See [Mixing synchronous and asynchronous stages](../execution-and-compatibility/async-execution.md#mixing-synchronous-and-asynchronous-stages) for how that widening works.

### `Stream#via[B]` — Apply a Pipeline to a Stream

The primary way to use a pipeline is through `Stream.via`. Here is the signature:

```scala
abstract class Stream[+E, +A] {
  def via[B](pipe: Pipeline[A, B]): Stream[E, B]
}
```

Under the hood, `via` calls `pipe.applyToStream(this)`. Each pipeline type delegates to a specific `Stream` node — for example, `Pipeline.map` creates a `Stream.Mapped`, and `Pipeline.filter` creates a `Stream.Filtered`. Type-preserving stages propagate the incoming stream representation; transforming stages attach the `JvmType.Infer[B]` evidence for their result. Composition preserves this metadata stage by stage rather than rediscovering the original input type at the call site.

The key advantage of `via` over inline methods is **reuse**: define the pipeline once and apply it to multiple streams:

```scala

// A reusable cleaning pipeline for sensor data
val cleanSensorData: Pipeline[Double, Double] =
  Pipeline.filter[Double](d => !d.isNaN && !d.isInfinite)
    .andThen(Pipeline.filter(d => d >= -100.0 && d <= 100.0))
// cleanSensorData: Pipeline[Double, Double] = zio.blocks.streams.Pipeline$Composed@767de072

// Apply to different sensor streams
val sensorStream1 = Stream(45.5, 67.2, Double.NaN, 23.1)
// sensorStream1: Stream[Nothing, Double] = Stream(45.5, 67.2, NaN, 23.1)
val sensorStream2 = Stream(89.9, -200.0, 12.5, 55.0)
// sensorStream2: Stream[Nothing, Double] = Stream(89.9, -200.0, 12.5, 55.0)

val sensor1Result = sensorStream1.via(cleanSensorData).runCollect
// sensor1Result: Either[Nothing, Chunk[Double]] = Right(
//   IndexedSeq(45.5, 67.2, 23.1)
// )
val sensor2Result = sensorStream2.via(cleanSensorData).runCollect
// sensor2Result: Either[Nothing, Chunk[Double]] = Right(
//   IndexedSeq(89.9, 12.5, 55.0)
// )
```

### `Pipeline#applyToStream[E]` — Direct Application

You can also call `applyToStream` directly. This is equivalent to `via` but reads left-to-right from the pipeline's perspective. Here is the signature:

```scala
abstract class Pipeline[-In, +Out] {
  def applyToStream[E](stream: Stream[E, In]): Stream[E, Out]
}
```

`stream.via(pipe)` and `pipe.applyToStream(stream)` are identical in behavior. Prefer `via` for readability in stream chains.

## Applying to a Sink

Apply a pipeline to a sink using `andThenSink` to pre-process the sink's input elements. This is the dual of `via`: instead of transforming a stream's output, you transform what the sink receives before it processes it.

### `Pipeline#andThenSink[E, Z]` — Pre-Process Sink Input

The dual of `via`: instead of transforming a stream's output, you pre-process a sink's input. Here is the signature:

```scala
abstract class Pipeline[-In, +Out] {
  def andThenSink[E, Z](sink: Sink[E, Out, Z]): Sink[E, In, Z]
}
```

This is an alias for `applyToSink`. After calling `andThenSink`, the resulting sink accepts `In` elements, transforms them through the pipeline, and feeds the `Out` elements to the original sink:

```scala

// A pipeline that normalizes strings
val normalize = Pipeline.map[String, String](_.trim.toLowerCase)
// normalize: Pipeline[String, String] = zio.blocks.streams.Pipeline$MapPipeline@18a6a4e3

// Apply to different sinks
val collectNormalized = normalize.andThenSink(Sink.collectAll[String])
// collectNormalized: Sink[Nothing, String, Chunk[String]] = zio.blocks.streams.Sink$Contramapped@6b779117
val countNormalized   = normalize.andThenSink(Sink.count)
// countNormalized: Sink[Nothing, String, Long] = zio.blocks.streams.Sink$Contramapped@2968d79e

val result = Stream("  Hello ", " WORLD  ").run(collectNormalized)
// result: Either[Nothing, Chunk[String]] = Right(IndexedSeq("hello", "world"))
```

### When to Use `andThenSink` Vs `via`

Both achieve the same result. Choose based on which side you want to reuse:

| Approach                             | Use when…                                   |
|--------------------------------------|---------------------------------------------|
| `stream.via(pipe).run(sink)`         | You have a fixed pipeline and varying sinks |
| `stream.run(pipe.andThenSink(sink))` | You want a reusable "pre-processing sink"   |

The laws guarantee equivalence: `stream.via(pipe).run(sink) == stream.run(pipe.andThenSink(sink))`.

### `Pipeline#applyToSink[E, Z]` — Direct Application

`andThenSink` is an alias for `applyToSink`. Here is the signature:

```scala
abstract class Pipeline[-In, +Out] {
  def applyToSink[E, Z](sink: Sink[E, Out, Z]): Sink[E, In, Z]
}
```

Prefer `andThenSink` for readability.

## JVM Primitive Specialization

`Pipeline.map`, `Pipeline.collect`, and their asynchronous counterparts `mapAsync` and `collectAsync` require `JvmType.Infer` for the transformed result type, except for the `map` overload whose function returns `Nothing`, which fixes the boxed lane itself and takes a `DummyImplicit` instead. It is resolved automatically and records which of the nine logical lanes that result belongs to: the eight JVM primitives, or the reference fallback. Those nine logical lanes compact into five interpreter storage lanes — int-like (`Boolean`, `Byte`, `Short`, `Char`, `Int`), `Long`, `Float`, `Double`, and reference — with operator tags selecting the identity-specific reads over them. Nine lanes therefore does not mean nine interpreter arrays; see [Zero-Boxing Streams](../execution-and-compatibility/zero-boxing.md) for how one is chosen. None of these factories request call-site evidence for the input type.

Type-preserving factories such as `filter`, `filterAsync`, `identity`, `take`, `drop`, and `buffer` do not require `JvmType.Infer`: they preserve the representation supplied when the pipeline is applied.

The rules hold identically through both application routes. On the stream route, result evidence is stored on the transformed stream node. On the sink route, mapping passes the same result evidence to the sink's contramap machinery, and structural pipelines use the generic run-via-sink route. So `stream.via(pipe)`, `pipe.andThenSink(sink)`, and `pipe.applyToSink(sink)` preserve the same specialization information as well as the same semantics.

## Integration

`Pipeline` integrates seamlessly with the other core streaming primitives: `Stream` and `Sink`. Understanding these integrations shows how pipelines fit into the broader streaming architecture.

### With Stream

`Pipeline` is the mechanism behind `Stream.via`. Every call to `stream.via(pipe)` delegates to `pipe.applyToStream(stream)`, which constructs the appropriate `Stream` subtype node. See [Stream — Integration with Pipeline and Sink](./stream.md#integration-with-pipeline-and-sink) for the stream-side perspective.

### With Sink

`Pipeline.andThenSink` creates a new `Sink` that pre-processes its input through the pipeline before the original sink consumes it. The `Sink` type provides its own transformation methods (`contramap`, `map`, `mapError`) — `andThenSink` extends these with the full power of pipeline composition (filtering, taking, dropping, collecting).

## Running the Examples

All code from this guide is available as runnable examples in the `streams-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

### Basic Usage

This example demonstrates six of the Pipeline factory methods: `map`, `filter`, `collect`, `take`, `drop`, and `identity`. Here is the source code:

```scala title="streams-examples/src/main/scala/pipeline/PipelineBasicUsageExample.scala" 
package pipeline

object PipelineBasicUsageExample extends App {
  println("=== Pipeline Basic Usage ===\n")

  // 1. Pipeline.map — transform each element
  println("1. Pipeline.map — transform each element:")
  val doubler = Pipeline.map[Int, Int](_ * 2)
  show(Stream(1, 2, 3).via(doubler).runCollect)

  // 2. Pipeline.map — cross-type transformation
  println("\n2. Pipeline.map — type-changing transformation:")
  val intToString = Pipeline.map[Int, String](n => s"item-$n")
  show(Stream(1, 2, 3).via(intToString).runCollect)

  // 3. Pipeline.filter — keep elements matching a predicate
  println("\n3. Pipeline.filter — keep matching elements:")
  val positives = Pipeline.filter[Int](_ > 0)
  show(Stream(-2, -1, 0, 1, 2).via(positives).runCollect)

  // 4. Pipeline.collect — partial function (filter + map)
  println("\n4. Pipeline.collect — partial function transformation:")
  val extractInts = Pipeline.collect[Any, Int] { case n: Int => n * 10 }
  show(Stream(1, "a", 2, "b").via(extractInts).runCollect)

  // 5. Pipeline.take — first n elements
  println("\n5. Pipeline.take — first n elements (short-circuits):")
  val firstThree = Pipeline.take[Int](3)
  show(Stream.range(0, 1000).via(firstThree).runCollect)

  // 6. Pipeline.drop — skip first n elements
  println("\n6. Pipeline.drop — skip first n elements:")
  val skipTwo = Pipeline.drop[String](2)
  show(Stream("header", "subheader", "data1", "data2").via(skipTwo).runCollect)

  // 7. Pipeline.identity — pass-through (no-op)
  println("\n7. Pipeline.identity — pass-through (neutral element):")
  val noOp = Pipeline.identity[Int]
  show(Stream(1, 2, 3).via(noOp).runCollect)

  // 8. Combining drop and take to get a range
  println("\n8. Combining drop and take for pagination:")
  val page2 = Pipeline.drop[Int](3).andThen(Pipeline.take(3))
  show(Stream.range(0, 10).via(page2).runCollect)
}
```

([source](https://github.com/zio/zio-blocks/blob/main/streams-examples/src/main/scala/pipeline/PipelineBasicUsageExample.scala))

Run it with:

```bash
cd streams-examples && sbt "runMain pipeline.PipelineBasicUsageExample"
```

### Pipeline Composition

This example shows how to compose pipelines with `andThen`, apply them to multiple streams, and build pipelines conditionally. Here is the source code:

```scala title="streams-examples/src/main/scala/pipeline/PipelineCompositionExample.scala" 
package pipeline

object PipelineCompositionExample extends App {
  println("=== Pipeline Composition ===\n")

  // 1. Basic andThen composition
  println("1. Composing filter + map with andThen:")
  val filterPositive = Pipeline.filter[Int](_ > 0)
  val double         = Pipeline.map[Int, Int](_ * 2)
  val composed       = filterPositive.andThen(double)

  show(Stream(-3, -1, 0, 2, 5).via(composed).runCollect)

  // 2. Multi-stage pipeline
  println("\n2. Multi-stage pipeline (filter → map → take):")
  val multiStage = Pipeline
    .filter[Int](_ % 2 == 0)
    .andThen(Pipeline.map[Int, Int](_ * 10))
    .andThen(Pipeline.take(3))

  show(Stream.range(1, 11).via(multiStage).runCollect)

  // 3. Reusing the same pipeline across different streams
  println("\n3. Reusing the same pipeline across different streams:")
  val normalize = Pipeline.filter[Int](_ >= 0).andThen(Pipeline.map[Int, Double](_.toDouble / 100.0))

  val dataset1 = Stream(150, -20, 75, 200, -10)
  val dataset2 = Stream(50, 100, -5, 300)

  show(dataset1.via(normalize).runCollect)
  show(dataset2.via(normalize).runCollect)

  // 4. Category law: left identity
  println("\n4. Category law — left identity (identity andThen p == p):")
  val pipe = Pipeline.map[Int, Int](_ + 1)
  val data = Stream(1, 2, 3)

  val withIdentityL   = data.via(Pipeline.identity[Int].andThen(pipe)).runCollect
  val withoutIdentity = data.via(pipe).runCollect
  show(withIdentityL)
  show(withoutIdentity)

  // 5. Category law: right identity
  println("\n5. Category law — right identity (p andThen identity == p):")
  val withIdentityR = data.via(pipe.andThen(Pipeline.identity[Int])).runCollect
  show(withIdentityR)
  show(withoutIdentity)

  // 6. Category law: associativity
  println("\n6. Category law — associativity ((p andThen q) andThen r == p andThen (q andThen r)):")
  val p = Pipeline.filter[Int](_ > 0)
  val q = Pipeline.map[Int, Int](_ * 3)
  val r = Pipeline.take[Int](2)

  val leftGrouped  = (p.andThen(q)).andThen(r)
  val rightGrouped = p.andThen(q.andThen(r))

  val source = Stream(-1, 2, -3, 4, 5, 6)
  show(source.via(leftGrouped).runCollect)
  show(source.via(rightGrouped).runCollect)

  // 7. Building pipelines conditionally
  println("\n7. Building pipelines conditionally:")
  def buildPipeline(limit: Option[Int], onlyPositive: Boolean): Pipeline[Int, Int] = {
    var pipe: Pipeline[Int, Int] = Pipeline.identity[Int]
    if (onlyPositive) pipe = pipe.andThen(Pipeline.filter(_ > 0))
    limit.foreach(n => pipe = pipe.andThen(Pipeline.take(n.toLong)))
    pipe
  }

  val conditionalPipe = buildPipeline(limit = Some(3), onlyPositive = true)
  show(Stream(-1, 2, -3, 4, 5, 6).via(conditionalPipe).runCollect)

  val noPipe = buildPipeline(limit = None, onlyPositive = false)
  show(Stream(-1, 2, -3).via(noPipe).runCollect)
}
```

([source](https://github.com/zio/zio-blocks/blob/main/streams-examples/src/main/scala/pipeline/PipelineCompositionExample.scala))

Run it with:

```bash
cd streams-examples && sbt "runMain pipeline.PipelineCompositionExample"
```

### Sink Integration

This example demonstrates applying pipelines to sinks with `andThenSink`, showing the equivalence between `stream.via(pipe).run(sink)` and `stream.run(pipe.andThenSink(sink))`. Here is the source code:

```scala title="streams-examples/src/main/scala/pipeline/PipelineSinkIntegrationExample.scala" 
package pipeline

object PipelineSinkIntegrationExample extends App {
  println("=== Pipeline ↔ Sink Integration ===\n")

  // 1. Basic andThenSink usage
  println("1. Basic andThenSink — pre-process before collecting:")
  val doubler        = Pipeline.map[Int, Int](_ * 2)
  val collectDoubled = doubler.andThenSink(Sink.collectAll[Int])

  show(Stream(1, 2, 3).run(collectDoubled))

  // 2. Equivalence law: via + run == run + andThenSink
  println("\n2. Equivalence law: stream.via(p).run(sink) == stream.run(p.andThenSink(sink)):")
  val pipe   = Pipeline.filter[Int](_ > 2).andThen(Pipeline.map[Int, Int](_ * 10))
  val source = Stream(1, 2, 3, 4, 5)
  val sink   = Sink.collectAll[Int]

  val viaResult         = source.via(pipe).run(sink)
  val andThenSinkResult = source.run(pipe.andThenSink(sink))
  show(viaResult)
  show(andThenSinkResult)

  // 3. Reusable pre-processing sink
  println("\n3. Reusable pre-processing sink:")
  val cleanString    = Pipeline.map[String, String](_.trim.toLowerCase)
  val collectCleaned = cleanString.andThenSink(Sink.collectAll[String])
  val countCleaned   = cleanString.andThenSink(Sink.count)

  val rawData = Stream("  Hello ", " WORLD  ", "  Scala  ")

  show(rawData.run(collectCleaned))
  show(rawData.run(countCleaned))

  // 4. andThenSink with foldLeft
  println("\n4. Pipeline + foldLeft sink:")
  val sumPositives = Pipeline
    .filter[Int](_ > 0)
    .andThenSink(Sink.foldLeft(0)((acc, x) => acc + x))

  show(Stream(-5, 3, -2, 7, 1).run(sumPositives))

  // 5. andThenSink with head/find
  println("\n5. Pipeline + head/find sinks:")
  val firstEven = Pipeline.filter[Int](_ % 2 == 0).andThenSink(Sink.head[Int])

  show(Stream(1, 3, 4, 6).run(firstEven))

  // 6. Multiple pipelines, same sink
  println("\n6. Multiple pipelines applied to the same sink:")
  val baseSink = Sink.collectAll[Int]

  val evens = Pipeline.filter[Int](_ % 2 == 0).andThenSink(baseSink)
  val odds  = Pipeline.filter[Int](_ % 2 != 0).andThenSink(baseSink)

  val nums = Stream(1, 2, 3, 4, 5, 6)
  show(nums.run(evens))
  show(nums.run(odds))

  // 7. Complex pipeline applied to sink
  println("\n7. Multi-stage pipeline applied to sink:")
  val processingPipe = Pipeline
    .filter[Int](_ > 0)
    .andThen(Pipeline.map[Int, Int](_ * 2))
    .andThen(Pipeline.take(3))

  val processedSum = processingPipe.andThenSink(Sink.foldLeft(0)(_ + _))

  show(Stream(-1, 5, 3, 8, 2, 9).run(processedSum))
}
```

([source](https://github.com/zio/zio-blocks/blob/main/streams-examples/src/main/scala/pipeline/PipelineSinkIntegrationExample.scala))

Run it with:

```bash
cd streams-examples && sbt "runMain pipeline.PipelineSinkIntegrationExample"
```

### Asynchronous Stages Through Both Routes

This example builds `mapAsync`, `filterAsync`, and `collectAsync` into one composed pipeline value, then applies that same value through `stream.via(pipe)`, `pipe.andThenSink(sink)`, and `pipe.applyToSink(sink)` — showing that all three produce the same elements. Here is the source code:

```scala title="streams-examples/src/main/scala/pipeline/PipelineAsyncExample.scala" 
package pipeline

/**
 * The three asynchronous `Pipeline` factories — `mapAsync`, `filterAsync` and
 * `collectAsync` — built once as values and then applied through both routes:
 * `stream.via(pipe)` and `pipe.andThenSink(sink)` (with `pipe.applyToSink` as
 * the third spelling of the second route).
 *
 * The point of running both is that they agree. A pipeline is a description,
 * not a stage bound to one side of the graph, so the same value produces the
 * same elements whichever end it is attached to.
 *
 * JVM only, because it ends in `.block` to turn an `Async` into a value for
 * `main`.
 */
object PipelineAsyncExample {

  /** Each field of a raw record, as a trimmed reading. */
  private val readings: Stream[Nothing, String] =
    Stream(" 17 ", "  4", "31  ", " 8 ", "  ", "150")

  /**
   * `mapAsync` — one asynchronous callback per element, awaited before the next
   * element is pulled. It asks for `JvmType.Infer` evidence for `Int`, its
   * result type, and nothing for its input.
   */
  private val parse: Pipeline[String, Int] =
    Pipeline
      .mapAsync[String, String](raw => Async.succeed(raw.trim))
      .andThen(Pipeline.mapAsync[String, Int](t => Async.succeed(if (t.isEmpty) 0 else t.toInt)))

  /** `filterAsync` — type-preserving, so it needs no result evidence. */
  private val inRange: Pipeline[Int, Int] =
    Pipeline.filterAsync[Int](n => Async.succeed(n > 0 && n <= 100))

  /**
   * `collectAsync` — filter and transform in one callback. It returns
   * `Async[Option[B]]` rather than a `PartialFunction`: a `None` drops the
   * element, a `Some` emits its value.
   */
  private val tensDigit: Pipeline[Int, Int] =
    Pipeline.collectAsync[Int, Int](n => Async.succeed(if (n >= 10) Some(n / 10) else None))

  /** One composed value, applied unchanged through both routes below. */
  private val pipe: Pipeline[String, Int] =
    parse.andThen(inRange).andThen(tensDigit)

  def main(args: Array[String]): Unit = {
    val sink: Sink[Nothing, Int, Chunk[Int]] = Sink.collectAll[Int]

    // Route 1 — attach the pipeline to the stream, then run a plain sink.
    val viaStream = readings.via(pipe).runAsync(sink).block

    // Route 2 — attach the pipeline to the sink, then run the plain stream.
    val viaSink = readings.runAsync(pipe.andThenSink(sink)).block

    // The third spelling: `andThenSink` is an alias for `applyToSink`.
    val viaApplyToSink = readings.runAsync(pipe.applyToSink(sink)).block

    report("stream.via(pipe)", viaStream)
    report("pipe.andThenSink(sink)", viaSink)
    report("pipe.applyToSink(sink)", viaApplyToSink)
    println(s"routes agree: ${viaStream == viaSink && viaSink == viaApplyToSink}")
  }

  private def report[E, Z](label: String, result: Either[E, Z]): Unit =
    result match {
      case Right(value) => println(s"$label -> $value")
      case Left(error)  => println(s"$label -> typed error: $error")
    }
}
```

([source](https://github.com/zio/zio-blocks/blob/main/streams-examples/src/main/scala/pipeline/PipelineAsyncExample.scala))

Run it with:

```bash
cd streams-examples && sbt "runMain pipeline.PipelineAsyncExample"
```

## See Also

- [Asynchronous Stream Execution](../execution-and-compatibility/async-execution.md#async-operators) — the stream-level `*Async` operators the three asynchronous factories delegate to, and how a mixed graph compiles
- [Stream](./stream.md#integration-with-pipeline-and-sink) — the producer side of `via`
- [Sink](./sink.md) — the consumer a pipeline can be attached to instead
- [Zero-Boxing Streams](../execution-and-compatibility/zero-boxing.md) — the lanes `JvmType.Infer` selects between, and why transforming factories need the evidence

---

## Sink

`Sink[+E, -A, +Z]` is a **stream consumer** that reads elements of type `A` and produces a result of type `Z`, potentially failing with an error of type `E`. Every sink has synchronous and asynchronous drain paths: use cross-platform `Stream.runAsync`, or the JVM-only blocking `Stream.run`.

`Sink`:
- Is covariant in `E` (error) and `Z` (result) — these are outputs
- Is contravariant in `A` (input) — a `Sink[_, Any, _]` accepts any element type
- Dispatches on the reader's physical primitive lane, so element pulls avoid boxing
- Provides `Sink#contramap`, `Sink#map`, and `Sink#mapError` for composable transformations

Built-in sinks implement both drain paths. Asynchronous constructors include `existsAsync`, `findAsync`, `forallAsync`, `foreachAsync`, and `foldLeftAsync`; callbacks are sequential and back-pressured. The result/error combinators `contramapAsync`, `mapAsync`, and `mapErrorAsync` likewise select the native async drain when used by `runAsync`. On the JVM, driving such a sink through a plain blocking terminal blocks while awaiting its async drain; Scala.js has no blocking terminal.

Here is the structural shape of the `Sink` type:

```scala
abstract class Sink[+E, -A, +Z] {
  def contramap[A0 <: A, A2](g: A2 => A0)(implicit jtA0: JvmType.Infer[A0]): Sink[E, A2, Z]
  def contramapAsync[A0 <: A, A2](g: A2 => Async[A0])(implicit jtA0: JvmType.Infer[A0]): Sink[E, A2, Z]
  def map[Z2](f: Z => Z2): Sink[E, A, Z2]
  def mapAsync[Z2](f: Z => Async[Z2]): Sink[E, A, Z2]
  def mapError[E2](f: E => E2)(implicit isNothing: Sink.IsNothing[E]): Sink[E2, A, Z]
  def mapErrorAsync[E2](f: E => Async[E2])(implicit isNothing: Sink.IsNothing[E]): Sink[E2, A, Z]
}
```

## Overview

Sink is the terminal piece in the streaming architecture. A [Stream](./stream.md) describes *what* to produce, a [Pipeline](./pipeline.md) describes *how* to transform, and a Sink describes *how to consume*:

```
┌──────────────┐     ┌──────────────────┐     ┌──────────────┐
│ Stream[E, A] │ ──→ │ Pipeline[A, B]   │ ──→ │ Sink[E, B, Z]│
└──────────────┘     └──────────────────┘     └──────────────┘
                                                      │
                                              ┌───────▼──────┐
                                              │ Either[E, Z] │
                                              └──────────────┘
```

When you call `stream.run(sink)`:
1. The stream compiles into a `Reader` (a low-level pull-based source)
2. The sink's internal `Sink#drain` method pulls elements in a tight loop until end-of-stream
3. On success, the result wraps in `Right(z)`
4. Typed errors (`E`) surface as `Left(e)`, while untyped defects propagate as exceptions
5. The reader's `close()` runs after the drain whether it succeeded or threw, and a failure from `close()` is attached to the drain's own error rather than replacing it, ensuring resource safety

### Physical Input Lanes and Ownership

A sink consumes either kind of reader. `Reader` is not one type with a mode flag: it splits into `Reader.SyncReader[A]`, whose pulls return values, and `Reader.AsyncReader[A]`, whose pulls return [`Async`](../../async.md) values. [The reader union](../primitives/reader.md#the-reader-union) describes the split from the reader's side. From the sink's side, the consequence is that every sink is drained through one of two entry points, and the terminal decides which — the next section covers that choice.

Whichever kind arrives, a sink discovers its input representation from the materialized reader's `jvmType`, not from the sink's contravariant static input type. Generic sinks dispatch once per drain and then pull `Boolean`, `Byte`, `Char`, `Short`, `Int`, `Long`, `Float`, and `Double` through library-internal physical pulls. The synchronous drain uses `readBooleanPhysical`, `readBytePhysical`, `readCharPhysical`, `readShortPhysical`, `readIntPhysical`, `readLongsPhysical`, `readFloatPhysical`, and `readDoublesPhysical`, respectively; the asynchronous drain differs on two lanes, pulling `Byte` through `readBytesPhysical` and `Float` through `readFloatsPhysical`. These are the internal counterparts of the public `readBoolean`, `readByte`, `readChar`, `readShort`, `readInt`, `readLongs`, `readFloat`, and `readDoubles`, which a generic sink cannot call because all but `readByte` demand `Elem <:< X` evidence it has no way to supply. Reference inputs use generic `read`. In particular, the four small primitive lanes do not share the `Int` pull.

`Long` and `Double` use one-element primitive arrays with `readLongsPhysical(scratch, 0, 1)` and `readDoublesPhysical(scratch, 0, 1)`, and on the asynchronous drain `Byte` and `Float` use one-element arrays in the same way. The returned count carries end-of-stream status out of band, so every `Long` value and every raw `Double` bit pattern—including NaN payloads—remains valid data. The scratch storage is allocated once per drain, not once per element.

The reader owns this physical representation. Consequently, widening a specialized stream (for example, from `Stream[Nothing, Int]` to `Stream[Nothing, AnyVal]`) does not erase its `Int` lane before it reaches a sink. A sink adapter preserves the wrapped reader's lane and forwards every exact pull method. It also preserves ownership: the run terminal owns and closes the materialized reader; a sink or sink adapter does not independently close it. External destinations supplied to I/O sinks remain caller-owned unless a constructor explicitly says otherwise.

## Consuming Asynchronous Readers

Every sink carries two drains, one per reader kind:

```scala
abstract class Sink[+E, -A, +Z] {
  private[streams] def drain(reader: Reader.SyncReader[_]): Z
  private[streams] def drain(reader: Reader.AsyncReader[_]): Async[Z]
}
```

They are an overload on the argument type, not a runtime branch inside one method, so the selection is a static one made where the reader is known: at the terminal. Both are `private[streams]`, which means you never call either yourself. The terminal you already chose called one of them for you:

```
┌────────────────────────────────────────────────────────────────────┐
│      Stream#run  (JVM only)         Stream#runAsync  (JVM and JS)  │
│                 │ materializes                    │ materializes   │
│                 ▼                                 ▼                │
│       Reader.SyncReader[A]              Reader.AsyncReader[A]      │
│                 │ is handed to                    │ is handed to   │
│                 ▼                                 ▼                │
│       drain(SyncReader): Z          drain(AsyncReader): Async[Z]   │
│                 │ returns                         │ returns        │
│                 ▼                                 ▼                │
│           Either[E, Z]                   Async[Either[E, Z]]       │
└────────────────────────────────────────────────────────────────────┘
```

The asynchronous drain is a native one, not an adapter: `Stream#runAsync` hands the sink a `Reader.AsyncReader` and never converts it to a blocking reader first, so no thread is parked waiting for a pull. [Async Terminals](../execution-and-compatibility/async-execution.md#async-terminals) covers the terminal family that takes this path, and [The `Async[Either[E, Z]]` convention](../execution-and-compatibility/async-execution.md#the-asynceithere-z-convention) explains why the typed error stays inside the `Either`.

Every built-in sink implements both drains, so nothing on this page is available on only one path. A third member, `drainAsync(reader: Reader.SyncReader[_]): Async[Z]`, is likewise `private[streams]`; it presents a synchronous reader through the asynchronous contract by calling `reader.toAsync`, and exists for internal stages that must expose an asynchronous face over synchronous input.

### Choosing a Factory

Because a `Sink` has two abstract drains and both are `private[streams]`, code outside `zio.blocks.streams` cannot implement them — subclassing `Sink` directly is not a supported route, and the compiler will not let you complete the class. Build custom sinks from one of the three factories instead:

```scala
object Sink {
  // JVM and Scala.js
  def createAsync[E, A, Z](f: Reader.AsyncReader[A] => Async[Z]): Sink[E, A, Z]
  def createBoth[E, A, Z](
    sync: Reader.SyncReader[A] => Z,
    async: Reader.AsyncReader[A] => Async[Z]
  ): Sink[E, A, Z]

  // JVM only — absent from the Scala.js companion entirely
  def create[E, A, Z](f: Reader.SyncReader[A] => Z): Sink[E, A, Z]
}
```

Each factory fills in the drain you did not write:

| Factory            | Callbacks you write               | Platforms        | Reach for it when                                                    |
|--------------------|-----------------------------------|------------------|----------------------------------------------------------------------|
| `Sink.createAsync` | one asynchronous                  | JVM and Scala.js | one implementation is enough, and awaiting per element is acceptable |
| `Sink.createBoth`  | one synchronous, one asynchronous | JVM and Scala.js | the synchronous path can be cheaper than the asynchronous one        |
| `Sink.create`      | one synchronous                   | JVM only         | the sink is JVM-only anyway and its callback wants a blocking reader |

`Sink.createAsync` writes the synchronous drain for you by awaiting the asynchronous one at a JVM blocking terminal. `Sink.create` writes the asynchronous drain for you by deferring the blocking callback into a cancellable effect, where cancellation closes the reader — the callback still runs inline on whichever thread drives that effect, so an asynchronous terminal driving a `Sink.create` sink blocks that thread for the duration of the callback. `Sink.createBoth` writes neither: it takes two independent callbacks and the terminal runs exactly one, so the two must agree on how much input they consume and what they produce — nothing compares their results.

:::warning[`Sink.create` does not exist on Scala.js]
It is declared in the JVM copy of `SinkCompanionPlatformSpecific` and simply absent from the Scala.js copy, so cross-built code that calls it fails to compile for the JS target rather than failing at runtime. [`Sink.create` and Custom Sinks](../execution-and-compatibility/platform-differences.md#sinkcreate-and-custom-sinks) states the platform rule.
:::

[Custom sink factories](#custom-sink-factories) shows a worked callback against the reader protocol.

## Predefined Sinks

These are value sinks (no factory arguments). They work on any element type.

### `Sink.drain` — Discard All Elements

Consumes every element and discards them. Returns `Unit`:

```scala
object Sink {
  val drain: Sink[Nothing, Any, Unit]
}
```

Use `Sink.drain` when you only care about side effects (e.g., via `Stream#tapEach`) and not the elements themselves:

```scala

val log = Buffer[String]()
// log: Buffer[String] = ArrayBuffer(
//   "Processing: 1",
//   "Processing: 2",
//   "Processing: 3"
// )
val result = Stream(1, 2, 3)
  .tapEach(x => log += s"Processing: $x")
  .run(Sink.drain)
// result: Either[Nothing, Unit] = Right(())
// result is Right(())
// log contains: ["Processing: 1", "Processing: 2", "Processing: 3"]
```

### `Sink.count` — Count Elements

Counts the total number of elements consumed. Returns `Long`:

```scala
object Sink {
  val count: Sink[Nothing, Any, Long]
}
```

Count all elements in a stream:

```scala

val result = Stream(1, 2, 3, 4, 5).run(Sink.count)
// result: Either[Nothing, Long] = Right(5L)
```

### `Sink.sumInt` / `Sink.sumLong` / `Sink.sumFloat` / `Sink.sumDouble` — Typed Numeric Sums

Returns the sum of all elements as a numeric type. Each sink accepts the corresponding primitive type:

```scala
object Sink {
  val sumInt:    Sink[Nothing, Int, Long]
  val sumLong:   Sink[Nothing, Long, Long]
  val sumFloat:  Sink[Nothing, Float, Double]
  val sumDouble: Sink[Nothing, Double, Double]
}
```

Note that `Sink.sumInt` returns `Long` (to avoid overflow) and `Sink.sumFloat` returns `Double` (to reduce rounding loss):

```scala

val intSum = Stream(1, 2, 3, 4, 5).run(Sink.sumInt)
// intSum: Either[Nothing, Long] = Right(15L)

val doubleSum = Stream(1.5, 2.5, 3.0).run(Sink.sumDouble)
// doubleSum: Either[Nothing, Double] = Right(7.0)
```

## Construction

Sinks are created using factory methods on the companion object. These methods fall into several categories based on what they do:

### Collecting

Gather elements into collections:

#### `Sink.collectAll[A]` — Collect Into a Chunk

Collects all elements into a [`Chunk[A]`](../../chunk.md):

```scala
object Sink {
  def collectAll[A]: Sink[Nothing, A, Chunk[A]]
}
```

This is the sink behind `Stream.runCollect`:

```scala

val result = Stream(1, 2, 3).run(Sink.collectAll[Int])
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2, 3))
```

#### `Sink.take[A]` — Collect First N Elements

Collects at most `n` elements into a `Chunk[A]`, then stops (short-circuiting the upstream):

```scala
object Sink {
  def take[A](n: Int): Sink[Nothing, A, Chunk[A]]
}
```

Collect only the first three elements from a large stream:

```scala

val result = Stream.range(0, 1000).run(Sink.take(3))
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(0, 1, 2))
```

### Aggregation and Search

These sinks combine elements into a single result or search for specific elements within a stream:

#### `Sink.foldLeft[A, Z]` — General Left Fold

Folds all elements using an accumulator function, starting from initial value `z`:

```scala
object Sink {
  def foldLeft[A, Z](z: Z)(f: (Z, A) => Z)(implicit jtZ: JvmType.Infer[Z]): Sink[Nothing, A, Z]
}
```

This is the most general aggregation sink:

```scala

val sum = Stream(1, 2, 3, 4).run(Sink.foldLeft(0)(_ + _))
// sum: Either[Nothing, Int] = Right(10)

val concat = Stream("a", "b", "c").run(Sink.foldLeft("")(_ + _))
// concat: Either[Nothing, String] = Right("abc")
```

#### `Sink.head[A]` — First Element

Returns the first element wrapped in `Some`, or `None` for an empty stream:

```scala
object Sink {
  def head[A]: Sink[Nothing, A, Option[A]]
}
```

Get the first element from a stream, or None if empty:

```scala

val first = Stream(10, 20, 30).run(Sink.head[Int])
// first: Either[Nothing, Option[Int]] = Right(Some(10))

val empty = Stream.empty.run(Sink.head[Int])
// empty: Either[Nothing, Option[Int]] = Right(None)
```

#### `Sink.last[A]` — Last Element

Returns the last element wrapped in `Some`, or `None` for an empty stream. Must consume all elements:

```scala
object Sink {
  def last[A]: Sink[Nothing, A, Option[A]]
}
```

Get the last element from a stream:

```scala

val result = Stream(10, 20, 30).run(Sink.last[Int])
// result: Either[Nothing, Option[Int]] = Right(Some(30))
```

#### `Sink.find[A]` — First Matching Element

Returns the first element satisfying `pred`, or `None`. Short-circuits on first match:

```scala
object Sink {
  def find[A](pred: A => Boolean): Sink[Nothing, A, Option[A]]
}
```

Find the first even number in the stream:

```scala

val found = Stream(1, 3, 4, 6).run(Sink.find[Int](_ % 2 == 0))
// found: Either[Nothing, Option[Int]] = Right(Some(4))
```

#### `Sink.exists[A]` — Any Element Matches

Returns `true` if any element satisfies `pred`. Short-circuits on first match:

```scala
object Sink {
  def exists[A](pred: A => Boolean): Sink[Nothing, A, Boolean]
}
```

Check if any element matches a condition:

```scala

val hasNegative = Stream(1, -2, 3).run(Sink.exists[Int](_ < 0))
// hasNegative: Either[Nothing, Boolean] = Right(true)
```

#### `Sink.forall[A]` — All Elements Match

Returns `true` if all elements satisfy `pred`. Short-circuits to `false` on first failure:

```scala
object Sink {
  def forall[A](pred: A => Boolean): Sink[Nothing, A, Boolean]
}
```

Test whether all elements satisfy a condition:

```scala

val allPositive = Stream(1, 2, 3).run(Sink.forall[Int](_ > 0))
// allPositive: Either[Nothing, Boolean] = Right(true)

val notAll = Stream(1, -2, 3).run(Sink.forall[Int](_ > 0))
// notAll: Either[Nothing, Boolean] = Right(false)
```

### Effectful

These sinks perform side effects during stream consumption:

#### `Sink.foreach[A]` — Apply Side Effect to Each Element

Applies `f` to every element for side effects. Returns `Unit`:

```scala
object Sink {
  def foreach[A](f: A => Unit): Sink[Nothing, A, Unit]
}
```

Print each element as it is processed:

```scala

val result = Stream(1, 2, 3).run(Sink.foreach[Int](x => println(s"Got: $x")))
// Got: 1
// Got: 2
// Got: 3
// result: Either[Nothing, Unit] = Right(())
```

### Async Aggregation and Search

Five of the aggregation, search, and effectful sinks have an `*Async` twin whose callback returns `Async` instead of a value. The twin consumes the same input and returns the same result type; only the callback's shape differs:

```scala
object Sink {
  def existsAsync[A](pred: A => Async[Boolean]): Sink[Nothing, A, Boolean]
  def findAsync[A](pred: A => Async[Boolean]): Sink[Nothing, A, Option[A]]
  def foldLeftAsync[A, Z](z: Z)(f: (Z, A) => Async[Z])(implicit jtZ: JvmType.Infer[Z]): Sink[Nothing, A, Z]
  def forallAsync[A](pred: A => Async[Boolean]): Sink[Nothing, A, Boolean]
  def foreachAsync[A](f: A => Async[Unit]): Sink[Nothing, A, Unit]
}
```

Each stops where its synchronous twin stops:

| Async sink           | Sync twin       | Result      | Stops pulling at                                             |
|----------------------|-----------------|-------------|--------------------------------------------------------------|
| `Sink.existsAsync`   | `Sink.exists`   | `Boolean`   | the first `true`; otherwise end-of-stream, returning `false` |
| `Sink.findAsync`     | `Sink.find`     | `Option[A]` | the first match; otherwise end-of-stream, returning `None`   |
| `Sink.foldLeftAsync` | `Sink.foldLeft` | `Z`         | end-of-stream                                                |
| `Sink.forallAsync`   | `Sink.forall`   | `Boolean`   | the first `false`; otherwise end-of-stream, returning `true` |
| `Sink.foreachAsync`  | `Sink.foreach`  | `Unit`      | end-of-stream                                                |

Three properties hold across all five. At most one callback effect runs at a time, in encounter order, so these are sequential and back-pressured rather than concurrent — for concurrency, put it in the stream with [`Stream#mapParAsync`](./stream.md#streammapparasync) before the sink. No effect is started for input the sink never pulls, so short-circuiting still costs nothing after the decisive element. And a callback that fails or is cancelled fails or cancels the run as a *defect*, not as a value in the sink's typed error channel `E` — all five have `E = Nothing`.

`Sink.foldLeftAsync` carries the same `JvmType.Infer[Z]` evidence as `Sink.foldLeft`. It keeps the asynchronous fold aligned with the specialized synchronous one; the semantics are identical.

```scala

val readings = Stream(12, 7, 30, 21, 5)

// Sequential: one predicate effect at a time, stopping at 30.
val tooHot: Async[Either[Nothing, Boolean]] =
  readings.runAsync(Sink.existsAsync[Int](c => Async.succeed(c > 25)))

// The fold visits every element; the accumulator is threaded through Async.
val total: Async[Either[Nothing, Long]] =
  readings.runAsync(Sink.foldLeftAsync(0L)((sum: Long, c: Int) => Async.succeed(sum + c)))
```

### Failing

These sinks can be used to produce typed errors or fail under specific conditions:

#### `Sink.fail[E]` — Immediately Fail

Creates a sink that fails immediately with a typed error, without consuming any elements:

```scala
object Sink {
  def fail[E](e: E): Sink[E, Any, Nothing]
}
```

Use this in conditional sink construction:

```scala

val sink: Sink[String, Int, Long] =
  if (false) Sink.count
  else Sink.fail("not ready")
// sink: Sink[String, Int, Long] = zio.blocks.streams.Sink$$anon$7@3f942055

val result = Stream(1, 2, 3).run(sink)
// result: Either[String, Long] = Left("not ready")
```

### I/O

Write elements to Java I/O destinations:

#### `Sink.fromOutputStream` — Write Bytes

Writes every `Byte` element to a `java.io.OutputStream`:

```scala
object Sink {
  def fromOutputStream(os: java.io.OutputStream): Sink[Nothing, Byte, Unit]
}
```
The sink does **not** close the stream when done. This is intentional: you own the stream's lifecycle, not the sink. You're responsible for closing it yourself (typically via try-with-resources or explicit `close()` calls) to flush buffers and release system resources. This design gives you flexibility to reuse the stream after the sink finishes, or to coordinate closing with other stream operations:

```scala

val bos = new ByteArrayOutputStream()
// bos: ByteArrayOutputStream = Hi!

// Write first batch of bytes
Stream.fromChunk(zio.blocks.chunk.Chunk[Byte](72, 105)).run(Sink.fromOutputStream(bos))
// res15: Either[Nothing, Unit] = Right(())

// Write second batch to the same stream (reuse it)
Stream.fromChunk(zio.blocks.chunk.Chunk[Byte](33)).run(Sink.fromOutputStream(bos))
// res16: Either[Nothing, Unit] = Right(())

// When done writing all batches, YOU close the stream
bos.close()

// ByteArrayOutputStream ignores close(), so you can still call toByteArray()
val allBytes = bos.toByteArray()
// allBytes: Array[Byte] = Array(72, 105, 33)
// This works because ByteArrayOutputStream doesn't maintain any closeable resources
```

#### `Sink.fromJavaWriter` — Write Characters

Writes every `Char` element to a `java.io.Writer`. Does not close the writer when done — you own its lifecycle:

```scala
object Sink {
  def fromJavaWriter(w: java.io.Writer): Sink[Nothing, Char, Unit]
}
```

Write a stream of characters to a StringWriter and access the accumulated text:

```scala

val writer = new StringWriter()
// writer: StringWriter = Hello World

// Write a stream of individual characters
Stream('H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd')
  .run(Sink.fromJavaWriter(writer))
// res19: Either[Nothing, Unit] = Right(())

// Get the final string
val result = writer.toString()
// result: String = "Hello World"
```

Like `Sink.fromOutputStream`, this sink intentionally does not close the writer. This gives you control over when to flush or close, allowing you to write multiple streams to the same writer or coordinate lifecycle with other operations.

### Custom

Advanced low-level use cases with direct reader protocol access:

#### Custom Sink Factories

`createAsync` is the cross-platform escape hatch for a native asynchronous drain. `createBoth` supplies independent native synchronous and asynchronous drains. The plain `create` factory is JVM-only because its callback consumes a blocking `SyncReader`:

```scala
object Sink {
  def createAsync[E, A, Z](f: Reader.AsyncReader[A] => Async[Z]): Sink[E, A, Z]
  def createBoth[E, A, Z](sync: Reader.SyncReader[A] => Z,
                           async: Reader.AsyncReader[A] => Async[Z]): Sink[E, A, Z]
  // JVM only
  def create[E, A, Z](f: Reader.SyncReader[A] => Z): Sink[E, A, Z]
}
```

:::note
These factories give you direct access to the reader protocol. Await asynchronous pulls sequentially and prefer built-in sinks when possible.
:::

Here's a custom sink that computes the average of all integers in a stream:

```scala

// A custom sink that computes the average of Ints
val average = Sink.create[Nothing, Int, Double] { reader =>
  def loop(sum: Long, count: Long): (Long, Long) = {
    val v = reader.readInt(Long.MinValue)
    if (v == Long.MinValue) (sum, count)
    else {
      val newSum = sum + v
      loop(newSum, count + 1)
    }
  }
  val (sum, count) = loop(0L, 0L)
  if (count == 0) 0.0 else sum.toDouble / count
}
```

This example shows how `Sink.create` works. `readInt` widens an `Int` to `Long`, leaving `Long.MinValue` available as an out-of-domain end marker. For full-domain `Long` and `Double` inputs, do not choose a data sentinel: allocate a reusable one-element primitive array and use `readLongs` or `readDoubles`, whose returned count reports data or end-of-stream without collisions. You'd use `Sink.create` when no built-in sink provides the exact aggregation or transformation logic you need — it is powerful but requires understanding the low-level [Reader protocol](../primitives/reader.md).

## Transforming Sinks

Every sink can be transformed using these instance methods:

### `Sink#contramap[A2]` — Pre-Process Input

Transforms the input elements before they reach the sink. The sink's result and error types are unchanged:

```scala
abstract class Sink[+E, -A, +Z] {
  def contramap[A0 <: A, A2](g: A2 => A0)(implicit jtA0: JvmType.Infer[A0]): Sink[E, A2, Z]
}
```

The evidence describes the callback's **result** type `A0`, not the new sink input `A2`. The bound `A0 <: A` is variance-sound for contravariant `A`, while invariant `JvmType.Infer[A0]` lets the mapped reader advertise the callback's exact physical result lane. Primitive results therefore retain their exact lane; the `AnyRef` fallback deliberately uses the boxed reference lane. `contramapAsync` has the same evidence and representation rules for `A2 => Async[A0]`. Both adapters transform only elements requested by the wrapped sink, preserve short-circuiting, and leave reader closure to the run terminal.

`Sink#contramap` is the dual of `Sink#map`: it transforms what goes *in*, not what comes *out*:

```scala

// A sink that counts the length of strings
val totalLength: Sink[Nothing, String, Long] =
  Sink.sumInt.contramap[Int, String](_.length)
// totalLength: Sink[Nothing, String, Long] = zio.blocks.streams.Sink$Contramapped@1837b17b

val result = Stream("hello", "world").run(totalLength)
// result: Either[Nothing, Long] = Right(10L)
```

### `Sink#map[Z2]` — Transform Result

Transforms the result after the sink finishes draining:

```scala
abstract class Sink[+E, -A, +Z] {
  def map[Z2](f: Z => Z2): Sink[E, A, Z2]
}
```

Transform the result after draining:

```scala

val countAsString: Sink[Nothing, Any, String] =
  Sink.count.map(n => s"Total: $n elements")
// countAsString: Sink[Nothing, Any, String] = zio.blocks.streams.Sink$Mapped@78e53067

val result = Stream(1, 2, 3).run(countAsString)
// result: Either[Nothing, String] = Right("Total: 3 elements")
```

### `Sink#mapError[E2]` — Transform Error

Transforms the error channel of a sink:

```scala
abstract class Sink[+E, -A, +Z] {
  def mapError[E2](f: E => E2)(implicit isNothing: Sink.IsNothing[E]): Sink[E2, A, Z]
}
```

The `IsNothing` evidence records whether `E` is `Nothing`. For an infallible sink the method returns the same sink without allocating a wrapper; otherwise it maps typed errors:

```scala

// No-op: drain never fails, so mapError is free
val mapped = Sink.drain.mapError[String](_.toString)
// At compile time: this is just a cast, no wrapper allocated

// Real mapping: fail can produce errors
val failing = Sink.fail("oops").mapError[RuntimeException](new RuntimeException(_))
```

### `Sink#contramapAsync[A2]` — Pre-Process Input Asynchronously

Applies an asynchronous function to each element before it reaches the sink:

```scala
abstract class Sink[+E, -A, +Z] {
  def contramapAsync[A0 <: A, A2](g: A2 => Async[A0])(implicit jtA0: JvmType.Infer[A0]): Sink[E, A2, Z]
}
```

The evidence is the same as `Sink#contramap`'s and targets the callback's **result** type `A0`, not the new sink input `A2`. `g` runs sequentially, one effect at a time, as the wrapped sink requests elements, so short-circuiting is unchanged and `g` is never started for input the sink does not pull. A failed or cancelled `g` fails or cancels the run as a defect and stops pulling input; it does not appear in the sink's typed error channel.

```scala

// Resolve each identifier to a length before counting it
val totalLength: Sink[Nothing, String, Long] =
  Sink.sumInt.contramapAsync[Int, String](id => Async.succeed(id.length))

val result: Async[Either[Nothing, Long]] =
  Stream("hello", "world").runAsync(totalLength)
```

### `Sink#mapAsync[Z2]` — Transform Result Asynchronously

Transforms the result asynchronously after the sink finishes draining:

```scala
abstract class Sink[+E, -A, +Z] {
  def mapAsync[Z2](f: Z => Async[Z2]): Sink[E, A, Z2]
}
```

`f` runs once, after the drain completes, so it does not change input consumption. When the sink fails, `f` is never started. Failure or cancellation of `f` fails or cancels the run as a defect.

```scala

val persistedCount: Sink[Nothing, Any, String] =
  Sink.count.mapAsync(n => Async.succeed(s"Total: $n elements"))

val result: Async[Either[Nothing, String]] =
  Stream(1, 2, 3).runAsync(persistedCount)
```

### `Sink#mapErrorAsync[E2]` — Transform Error Asynchronously

Transforms the typed error channel asynchronously:

```scala
abstract class Sink[+E, -A, +Z] {
  def mapErrorAsync[E2](f: E => Async[E2])(implicit isNothing: Sink.IsNothing[E]): Sink[E2, A, Z]
}
```

It carries the same `IsNothing` evidence as `Sink#mapError`: for an infallible sink (`E = Nothing`) the method returns the same sink without allocating a wrapper, and `f` can never be evaluated. Successful results, upstream stream errors, and defects pass through untransformed; only the sink's own typed error reaches `f`. Failure or cancellation of `f` is a defect.

```scala

// Infallible: no wrapper is allocated and `f` can never run
val mapped = Sink.drain.mapErrorAsync[String](e => Async.succeed(e.toString))

// Fallible: the typed error is transformed before it reaches the `Left`
val failing = Sink.fail("oops").mapErrorAsync[RuntimeException](e => Async.succeed(new RuntimeException(e)))
```

## Integration with Stream

`Stream.run(sink)` is the primary entry point. ZIO Blocks also provides convenience methods on `Stream` that delegate to built-in sinks:

| Stream method          | Equivalent Sink                   |
|------------------------|-----------------------------------|
| `stream.runCollect`    | `stream.run(Sink.collectAll)`     |
| `stream.runDrain`      | `stream.run(Sink.drain)`          |
| `stream.runForeach(f)` | `stream.run(Sink.foreach(f))`     |
| `stream.runFold(z)(f)` | `stream.run(Sink.foldLeft(z)(f))` |
| `stream.count`         | `stream.run(Sink.count)`          |
| `stream.head`          | `stream.run(Sink.head)`           |
| `stream.last`          | `stream.run(Sink.last)`           |
| `stream.find(pred)`    | `stream.run(Sink.find(pred))`     |
| `stream.exists(pred)`  | `stream.run(Sink.exists(pred))`   |
| `stream.forall(pred)`  | `stream.run(Sink.forall(pred))`   |

The `runFold` method with primitive accumulator types (`Int`, `Long`, `Double`) uses specialized internal sink classes that keep the accumulator unboxed.

See [Stream — Running Streams](./stream.md#running-streams) for more details on terminal operations.

## Integration with Pipeline

A [Pipeline](./pipeline.md) can be applied to a Sink using `Pipeline#andThenSink`, producing a new Sink that pre-processes input elements through the pipeline:

```scala

val cleanAndCollect: Sink[Nothing, String, Chunk[String]] =
  Pipeline.map[String, String](_.trim.toLowerCase)
    .andThenSink(Sink.collectAll[String])
// cleanAndCollect: Sink[Nothing, String, Chunk[String]] = zio.blocks.streams.Sink$Contramapped@1ccb6ba3

val result = Stream("  Hello ", " WORLD  ").run(cleanAndCollect)
// result: Either[Nothing, Chunk[String]] = Right(IndexedSeq("hello", "world"))
```

The equivalence law holds: `stream.via(pipe).run(sink) == stream.run(pipe.andThenSink(sink))`.

See [Pipeline — Applying to a Sink](./pipeline.md#applying-to-a-sink) for more details.

## JVM NIO Sinks

The `NioSinks` object (JVM-only) provides sinks for Java NIO (`java.nio`) buffers and channels. NIO offers efficient buffers and both blocking and selector-based channel APIs, but these sinks use blocking channel writes; they do not expose selector-based non-blocking output. When you're writing to network sockets, files, or other NIO-based resources, these sinks give you a convenient way to drain streams directly into NIO data structures.

Traditional Java I/O (`OutputStream`, `Writer`) blocks threads and requires manual buffering for efficiency. `NioSinks.fromChannel` also blocks, but handles buffer allocation, position management, and flushing automatically (default 8KB), while typed variants like `NioSinks.fromByteBufferInt` and `NioSinks.fromByteBufferLong` eliminate boxing overhead by writing primitives directly to buffers you provide.

Choose `NioSinks.fromChannel` when blocking channel output is acceptable and you want automatic buffering for network sockets or files. Choose typed variants when you control buffer allocation and are streaming millions of primitives where boxing would degrade performance.

Each of these sinks implements both drains, so NIO output works under the blocking `Stream#run` and under `Stream#runAsync` alike. What the asynchronous drain removes is the thread parked waiting for stream *input*; the destination writes are the same blocking `java.nio` calls on either path. `NioSinks.fromChannel` does budget its asynchronous flush loop: when a channel write makes no progress it yields instead of spinning, so a stalled channel does not monopolise the caller. [Reader](../primitives/reader.md#from-native-asynchronous-sources) covers the NIO adapters on the source side in full, and this page does not duplicate them.

Here are the available NIO sinks:

```scala
object NioSinks {
  def fromByteBuffer      (buf: ByteBuffer): Sink[Nothing, Byte,   Unit]
  def fromByteBufferInt   (buf: ByteBuffer): Sink[Nothing, Int,    Unit]
  def fromByteBufferLong  (buf: ByteBuffer): Sink[Nothing, Long,   Unit]
  def fromByteBufferFloat (buf: ByteBuffer): Sink[Nothing, Float,  Unit]
  def fromByteBufferDouble(buf: ByteBuffer): Sink[Nothing, Double, Unit]
  def fromChannel(ch: WritableByteChannel, bufSize: Int = 8192): Sink[IOException, Byte, Unit]
}
```

### From ByteBuffer Sinks

**`NioSinks.fromByteBuffer` and typed variants** — Write primitive streams directly into a pre-allocated NIO ByteBuffer:
- `NioSinks.fromByteBuffer` — writes individual `Byte` elements.
- `NioSinks.fromByteBufferInt`, `NioSinks.fromByteBufferLong`, `NioSinks.fromByteBufferFloat`, `NioSinks.fromByteBufferDouble` — write primitives directly using the buffer's native methods (`putInt`, `putLong`, etc.), one buffer write per element rather than one per byte.

The sinks write at the buffer's position and advance it, encoding multi-byte values in the buffer's byte order, and keep every value, `Long.MaxValue`, `Double.MaxValue` and `NaN` included. They never grow the buffer. When it runs out of room, the `BufferOverflowException` that the buffer throws is not a typed error, so no `Left` comes back: a blocking terminal such as `Stream#run` throws it, and an `*Async` terminal's `Async` fails with it. The values written before it stay in the buffer, so allocate the buffer for the whole stream or bound the stream with `Stream#take`.

Here's an example using ByteBuffer with typed primitive writes:

```scala

val buffer = ByteBuffer.allocate(32).order(ByteOrder.BIG_ENDIAN)
// buffer: ByteBuffer = java.nio.HeapByteBuffer[pos=32 lim=32 cap=32]

// Write a stream of Longs to the buffer
Stream(1L, 2L, 3L, 4L).run(NioSinks.fromByteBufferLong(buffer))
// res29: Either[Nothing, Unit] = Right(())

// After writing, rewind to read
buffer.rewind()
// res30: ByteBuffer = java.nio.HeapByteBuffer[pos=32 lim=32 cap=32]

val readBack = List(
  buffer.getLong(),
  buffer.getLong(),
  buffer.getLong(),
  buffer.getLong()
)
// readBack: List[Long] = List(1L, 2L, 3L, 4L)
```

This example allocates a 32-byte buffer (4 Longs × 8 bytes each), writes four `Long` values using `NioSinks.fromByteBufferLong` (which calls `putLong` on each element), then rewinds and reads them back to verify. The typed variant writes one 8-byte buffer operation per element, where the byte variant would require the stream to carry each byte separately.

When the buffer is too small, the blocking terminal throws `BufferOverflowException` instead of returning `Left`, and the buffer keeps the values that fit:

```scala
val (overflow, bytesWritten) = {
  val small  = ByteBuffer.allocate(16)
  val result = scala.util.Try(Stream(1L, 2L, 3L).run(NioSinks.fromByteBufferLong(small)))
  (result, small.position())
}
// overflow: Try[Either[Nothing, Unit]] = Failure(
//   java.nio.BufferOverflowException
// )
// bytesWritten: Int = 16
```

The following example shows streaming voltage sensor readings through a calibration curve and buffering them for downstream computation. When processing sensor arrays or scientific measurements, pre-allocated buffers with typed sinks let each element be written through its exact primitive lane without boxing.

Here is the complete example:

```scala title="streams-examples/src/main/scala/sink/SinkScientificComputingExample.scala" 
package sink

object SinkScientificComputingExample extends App {
  println("=== Batching Doubles for Scientific Computing ===\n")

  // Simulate raw sensor measurements that need calibration
  println("Scenario: Streaming voltage sensor measurements with calibration curve\n")

  val measurementCount = 10
  val bufferCapacity   = measurementCount * 8 // 8 bytes per Double

  println(s"Processing $measurementCount measurements...\n")

  // Generate raw voltage readings (0.0 to 0.09)
  val rawVoltages = (0 until measurementCount).map(i => (i * 0.01).toDouble).toList

  println("Raw measurements (voltage):")
  rawVoltages.zipWithIndex.foreach { case (v, i) =>
    println(f"  [$i] $v%.4f V")
  }
  println()

  // Process and calibrate measurements
  val buffer = processAndBufferMeasurements(measurementCount, bufferCapacity)

  println("After calibration (applied quadratic correction: V' = V × (1 + 0.05×V²)):\n")

  // Read back and display calibrated values
  buffer.rewind()
  var index = 0
  while (buffer.hasRemaining) {
    val calibrated = buffer.getDouble()
    println(f"  [$index] $calibrated%.6f V")
    index += 1
  }

  println("\n=== Pattern Use Cases ===")
  println("This pattern is used in:")
  println("  • Scientific instrumentation (analog-to-digital conversion)")
  println("  • Machine learning pipelines (sensor data → training batches)")
  println("  • Signal processing (raw signals → preprocessed data → computation)")
  println("\nKey benefits:")
  println("  • Zero-copy batch processing with DirectByteBuffer")
  println("  • Efficient numerical stream transformation")
  println("  • Memory-friendly for large datasets")

  // Process and buffer measurements using NioSinks
  def processAndBufferMeasurements(
    measurementCount: Int,
    bufferCapacity: Int
  ): ByteBuffer = {
    val buffer = ByteBuffer.allocateDirect(bufferCapacity)

    // Stream of raw voltage measurements (need calibration)
    val voltages = Stream.range(0, measurementCount).map(i => (i * 0.01).toDouble)

    // Apply calibration curve: quadratic correction
    // This simulates real sensor calibration with nonlinear response
    val calibrated = voltages.map { raw =>
      val calibrationFactor = 1.0 + (0.05 * pow(raw, 2))
      raw * calibrationFactor
    }

    // Write calibrated values directly to ByteBuffer using typed sink
    // This is much faster than element-by-element byte writing
    calibrated.run(NioSinks.fromByteBufferDouble(buffer))
    buffer.flip()
    buffer
  }
}
```

Run this example with:

```bash
cd streams-examples && sbt "runMain sink.SinkScientificComputingExample"
```

This use case is typical in scientific instrumentation, machine learning data preprocessing, and signal processing pipelines where you need to efficiently batch-process numerical streams into memory-efficient structures for downstream computation.

The typed sinks dispatch through their exact primitive lanes. `Long` and `Double` use collision-free bulk-count status, so `Long.MinValue`, `Long.MaxValue`, every finite or infinite `Double`, signed zero, and every raw NaN payload are written as ordinary data. There is no sentinel-value truncation restriction. The supplied buffer remains caller-owned and is not flipped, rewound, or closed by the sink. [Zero-Boxing Streams](../execution-and-compatibility/zero-boxing.md) carries the per-lane end-of-stream table this rests on.

### From Channel Sink

The **`NioSinks.fromChannel`** constructor performs buffered writes to a `WritableByteChannel` (e.g., a network socket or file channel). This is the general-purpose NIO sink: it accumulates bytes in an internal buffer of size `bufSize` (default 8192), flushes when the buffer is full, and flushes again at end-of-stream. It does not close the caller-owned channel.

It handles `IOException` as a typed error, so failures surface as `Left(IOException)` from `Stream.run`. Use this for network I/O or when you can't pre-allocate a buffer. The channel I/O is blocking—NIO's non-blocking advantage comes when using selectors across many channels, which this sink does not expose.

Suppose you're collecting metrics from thousands of sensors (temperature, pressure, timestamps) and need to write them to a file efficiently. Using `NioSinks.fromChannel` with a file's `WritableByteChannel` gives you automatic buffering and eliminates manual position management.

Here is the complete example:

```scala title="streams-examples/src/main/scala/sink/SinkTelemetryExample.scala" 
package sink

object SinkTelemetryExample extends App {
  println("=== Streaming Telemetry to File Channel ===\n")

  // Simulated sensor readings (timestamp, temperature)
  case class SensorReading(timestamp: Long, temperature: Double) {
    override def toString: String = f"[$timestamp] $temperature%.2f°C"
  }

  // Generate mock sensor data
  val sensorReadings = List(
    SensorReading(1000L, 22.5),
    SensorReading(1100L, 23.1),
    SensorReading(1200L, 22.8),
    SensorReading(1300L, 23.4),
    SensorReading(1400L, 24.0)
  )

  println("Sensor readings to write:")
  sensorReadings.foreach(r => println(s"  $r"))
  println()

  // Write telemetry to file using NioSinks.fromChannel
  val tempFile = Files.createTempFile("telemetry", ".bin")
  val filePath = tempFile.toString

  println(s"Writing to $filePath...")
  Using(new RandomAccessFile(filePath, "rw")) { file =>
    val channel = file.getChannel
    channel.truncate(0) // Clear file

    // Serialize readings into a single buffer: 8 bytes timestamp + 8 bytes temperature per reading
    val buffer = java.nio.ByteBuffer.allocate(sensorReadings.length * 16)
    sensorReadings.foreach { reading =>
      buffer.putLong(reading.timestamp)
      buffer.putDouble(reading.temperature)
    }
    buffer.flip()

    // Write all bytes to file with internal buffering (8KB chunks)
    val bytes      = buffer.array()
    val byteStream = Stream.fromChunk(zio.blocks.chunk.Chunk.fromArray(bytes))
    byteStream.run(NioSinks.fromChannel(channel, bufSize = 8192))

    println(s"✓ Wrote ${file.length()} bytes to disk")
  }.get

  // Read back and verify
  println("\nVerifying written data:")
  Using(new RandomAccessFile(filePath, "r")) { file =>
    val buf = java.nio.ByteBuffer.allocate((8 + 8) * sensorReadings.length)
    file.getChannel.read(buf)
    buf.rewind()

    var count = 0
    while (buf.remaining() >= 16) {
      val timestamp   = buf.getLong()
      val temperature = buf.getDouble()
      println(f"  [$timestamp] $temperature%.2f°C")
      count += 1
    }
    println(s"✓ Read back $count sensor readings")
  }.get

  println("\n=== Pattern Use Cases ===")
  println("This pattern is used in:")
  println("  • IoT telemetry platforms (time-series databases)")
  println("  • High-throughput logging systems")
  println("  • Sensor data aggregation pipelines")
  println("\nKey benefits:")
  println("  • Automatic buffering eliminates manual position management")
  println("  • Integrated with Stream composition (no boilerplate)")
  println("  • Type-safe error handling (IOException as Sink error type)")

  Files.delete(tempFile)
}
```

Run it with:

```bash
cd streams-examples && sbt "runMain sink.SinkTelemetryExample"
```

This pattern is common in high-throughput logging systems, time-series databases, and IoT platforms where you need to write streams of telemetry data to persistent storage without blocking or allocating excessively.

## Running the Examples

All code from this guide is available as runnable examples in the `streams-examples` module.

Start by cloning the repository and navigating to the project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

Run individual examples with sbt:

### Basic Usage

This example demonstrates the most commonly used built-in sinks: `Sink.drain`, `Sink.count`, `Sink.collectAll`, `Sink.head`, `Sink.last`, and `Sink.take`:

```scala title="streams-examples/src/main/scala/sink/SinkBasicUsageExample.scala" 
package sink

object SinkBasicUsageExample extends App {
  println("=== Sink Basic Usage ===\n")

  val data = Stream(1, 2, 3, 4, 5)

  // 1. Sink.drain — discard all elements
  println("1. Sink.drain — discard all elements:")
  show(data.run(Sink.drain))

  // 2. Sink.count — count elements
  println("\n2. Sink.count — count elements:")
  show(data.run(Sink.count))

  // 3. Sink.collectAll — collect into Chunk
  println("\n3. Sink.collectAll — collect into Chunk:")
  show(data.run(Sink.collectAll[Int]))

  // 4. Sink.head — first element
  println("\n4. Sink.head — first element:")
  show(data.run(Sink.head[Int]))

  println("\n   Sink.head on empty stream:")
  show(Stream.empty.run(Sink.head[Int]))

  // 5. Sink.last — last element
  println("\n5. Sink.last — last element:")
  show(data.run(Sink.last[Int]))

  // 6. Sink.take — first n elements
  println("\n6. Sink.take — first n elements:")
  show(data.run(Sink.take(3)))

  // 7. take on a large stream (short-circuits)
  println("\n7. Sink.take short-circuits (only reads 3 of 1000):")
  show(Stream.range(0, 1000).run(Sink.take(3)))

  // 8. Combining stream operations with sinks
  println("\n8. Combining stream operations with explicit sinks:")
  val result = Stream(1, 2, 3, 4, 5)
    .filter(_ % 2 == 0)
    .run(Sink.collectAll[Int])
  show(result)

  // 9. Equivalence with convenience methods
  println("\n9. Stream convenience methods delegate to sinks:")
  show(data.runCollect == data.run(Sink.collectAll[Int]))
  show(data.count == data.run(Sink.count))
}
```

Run this example with:

```bash
cd streams-examples && sbt "runMain sink.SinkBasicUsageExample"
```

### Aggregation and Search

This example shows aggregation sinks (`Sink.foldLeft`, `Sink.sumInt`, `Sink.sumDouble`) and search sinks (`Sink.exists`, `Sink.forall`, `Sink.find`, `Sink.foreach`):

```scala title="streams-examples/src/main/scala/sink/SinkAggregationExample.scala" 
package sink

object SinkAggregationExample extends App {
  println("=== Sink Aggregation and Search ===\n")

  // 1. foldLeft — general accumulation
  println("1. Sink.foldLeft — general accumulation:")
  val sum = Stream(1, 2, 3, 4, 5).run(Sink.foldLeft(0)(_ + _))
  show(sum)

  println("\n   foldLeft with string concatenation:")
  val concat = Stream("a", "b", "c").run(Sink.foldLeft("")(_ + _))
  show(concat)
  // 2. sumInt — typed numeric sum
  println("\n2. Sink.sumInt — returns Long to avoid overflow:")
  val intSum = Stream(1, 2, 3, 4, 5).run(Sink.sumInt)
  show(intSum)

  // 3. sumDouble — typed floating point sum
  println("\n3. Sink.sumDouble:")
  val doubleSum = Stream(1.5, 2.5, 3.0).run(Sink.sumDouble)
  show(doubleSum)

  // 4. exists — short-circuits on first match
  println("\n4. Sink.exists — short-circuits on first match:")
  val hasNegative = Stream(1, 2, -3, 4).run(Sink.exists[Int](_ < 0))
  show(hasNegative)

  val noNegative = Stream(1, 2, 3, 4).run(Sink.exists[Int](_ < 0))
  show(noNegative)

  // 5. forall — all elements must match
  println("\n5. Sink.forall — all elements must match:")
  val allPositive = Stream(1, 2, 3).run(Sink.forall[Int](_ > 0))
  show(allPositive)

  val notAllPositive = Stream(1, -2, 3).run(Sink.forall[Int](_ > 0))
  show(notAllPositive)

  // 6. find — first element matching predicate
  println("\n6. Sink.find — first matching element:")
  val firstEven = Stream(1, 3, 4, 6, 8).run(Sink.find[Int](_ % 2 == 0))
  show(firstEven)

  val noMatch = Stream(1, 3, 5, 7).run(Sink.find[Int](_ % 2 == 0))
  show(noMatch)

  // 7. foreach — side effects
  println("\n7. Sink.foreach — apply side effects:")
  val items  = scala.collection.mutable.Buffer[String]()
  val result = Stream("x", "y", "z").run(Sink.foreach[String](s => items += s))
  show(result)
  show(items.toList)

  // 8. Complex aggregation: combine foldLeft with map
  println("\n8. Complex aggregation — average via foldLeft + map:")
  val average = Sink
    .foldLeft[Int, (Int, Int)]((0, 0)) { case ((sum, count), x) =>
      (sum + x, count + 1)
    }
    .map { case (sum, count) =>
      if (count == 0) 0.0 else sum.toDouble / count
    }

  val avg = Stream(10, 20, 30, 40).run(average)
  show(avg)
}
```

Run this example with:

```bash
cd streams-examples && sbt "runMain sink.SinkAggregationExample"
```

### Transformations and Composition

This example demonstrates `Sink#contramap`, `Sink#map`, `Sink#mapError`, `Sink.fail`, `Sink.create`, and `Pipeline#andThenSink`:

```scala title="streams-examples/src/main/scala/sink/SinkTransformationExample.scala" 
package sink

object SinkTransformationExample extends App {
  println("=== Sink Transformations and Composition ===\n")

  // 1. contramap — pre-process input
  println("1. Sink.contramap — pre-process input elements:")
  val stringLengthSum: Sink[Nothing, String, Long] =
    Sink.sumInt.contramap[Int, String](_.length)

  show(Stream("hello", "world").run(stringLengthSum))

  // 2. contramap — change element type
  println("\n2. contramap to convert types:")
  val parseInts: Sink[Nothing, String, Long] =
    Sink.sumInt.contramap[Int, String](_.toInt)

  show(Stream("10", "20", "30").run(parseInts))

  // 3. map — transform result
  println("\n3. Sink.map — transform the result:")
  val countFormatted: Sink[Nothing, Any, String] =
    Sink.count.map(n => s"Processed $n elements")

  show(Stream(1, 2, 3).run(countFormatted))

  // 4. Chaining contramap + map
  println("\n4. Chaining contramap + map:")
  val pipeline = Sink.sumInt
    .contramap[Int, String](_.length)
    .map(total => s"Total chars: $total")

  show(Stream("hi", "hello").run(pipeline))

  // 5. mapError — transform error channel
  println("\n5. Sink.mapError — transform errors:")

  sealed trait AppError
  case class ParseError(msg: String) extends AppError

  val failingSink = Sink.fail("raw error").mapError[AppError](msg => ParseError(msg))
  show(Stream(1).run(failingSink))

  // 6. fail — immediately fail
  println("\n6. Sink.fail — immediate failure:")
  show(Stream(1, 2, 3).run(Sink.fail("error")))

  // 7. Pipeline.andThenSink integration
  println("\n7. Pipeline.andThenSink — pipeline pre-processes before sink:")
  val cleanAndCollect =
    Pipeline
      .map[String, String](_.trim.toLowerCase)
      .andThenSink(Sink.collectAll[String])

  show(Stream("  Hello ", " WORLD  ").run(cleanAndCollect))

  // 8. Equivalence: via + run == andThenSink + run
  println("\n8. Equivalence law: via + run == andThenSink + run:")
  val pipe   = Pipeline.filter[Int](_ > 2).andThen(Pipeline.map[Int, Int](_ * 10))
  val source = Stream(1, 2, 3, 4, 5)

  val viaResult  = source.via(pipe).run(Sink.collectAll[Int])
  val sinkResult = source.run(pipe.andThenSink(Sink.collectAll[Int]))
  show(viaResult)
  show(sinkResult)

  // 9. Composing multiple transformations into a reusable sink
  println("\n9. Reusable composed sink:")
  case class Metric(name: String, value: Double)

  val metricSumSink: Sink[Nothing, Metric, Double] =
    Sink.foldLeft(0.0)((acc, m: Metric) => acc + m.value)

  val metrics = Stream(
    Metric("cpu", 45.0),
    Metric("cpu", 67.0),
    Metric("cpu", 23.0)
  )
  show(metrics.run(metricSumSink))

  val metricAvgSink: Sink[Nothing, Metric, Double] =
    Sink
      .foldLeft[Metric, (Double, Int)]((0.0, 0)) { case ((sum, count), m) =>
        (sum + m.value, count + 1)
      }
      .map { case (sum, count) => if (count == 0) 0.0 else sum / count }

  show(metrics.run(metricAvgSink))
}
```

Run this example with:

```bash
cd streams-examples && sbt "runMain sink.SinkTransformationExample"
```

### Async Sinks End to End

This example builds a custom mean sink with `Sink.createBoth` — a `while` loop for the synchronous drain, an `Async#flatMap` recursion for the asynchronous one — then drives it and `Sink.existsAsync` through `Stream#runAsync`, unwrapping each `Either` at the end. It is JVM-only, because it finishes with `.block` to turn the `Async` into a value for `main`:

```scala title="streams-examples/src/main/scala/sink/SinkAsyncExample.scala" 
package sink

/**
 * A custom sink built with `Sink.createBoth`, plus one of the `*Async`
 * combinators, both driven from the cross-platform asynchronous terminal
 * `Stream#runAsync`.
 *
 * `createBoth` supplies two independent drains. The terminal picks exactly one:
 * `runAsync` takes the asynchronous callback, and the JVM-only blocking `run`
 * takes the synchronous one. Both must agree on what they consume and what they
 * return.
 *
 * JVM only, because it ends in `.block` to turn the `Async` into a value for
 * `main`.
 */
object SinkAsyncExample {

  /**
   * Mean of an `Int` lane. The synchronous callback is a plain `while` loop
   * over `readInt`, which allocates no `Async` per element; the asynchronous
   * callback threads the same accumulator through `flatMap`.
   */
  val meanCelsius: Sink[Nothing, Int, Double] =
    Sink.createBoth[Nothing, Int, Double](
      sync = { (reader: Reader.SyncReader[Int]) =>
        var sum   = 0L
        var count = 0L
        var value = reader.readInt(Long.MinValue)
        while (value != Long.MinValue) {
          sum += value
          count += 1L
          value = reader.readInt(Long.MinValue)
        }
        mean(sum, count)
      },
      async = { (reader: Reader.AsyncReader[Int]) =>
        def loop(sum: Long, count: Long): Async[Double] =
          reader.readInt(Long.MinValue).flatMap { value =>
            if (value == Long.MinValue) Async.succeed(mean(sum, count))
            else loop(sum + value, count + 1L)
          }

        loop(0L, 0L)
      }
    )

  private def mean(sum: Long, count: Long): Double =
    if (count == 0L) 0.0 else sum.toDouble / count.toDouble

  def main(args: Array[String]): Unit = {
    val celsius: Stream[Nothing, Int] = Stream(12, 7, 30, 21, 5)

    // Nothing has run yet: both terminals are descriptions.
    val average: Async[Either[Nothing, Double]] = celsius.runAsync(meanCelsius)

    // `existsAsync` runs at most one predicate effect at a time and stops at
    // the first `true`; 30 matches, so 21 and 5 are never pulled.
    val tooHot: Async[Either[Nothing, Boolean]] =
      celsius.runAsync(Sink.existsAsync[Int](c => Async.succeed(c > 25)))

    // `.block` parks the calling thread until the value arrives. It belongs
    // here, at the edge of a JVM `main`, and nowhere else.
    report("createBoth mean", average.block)
    report("existsAsync (> 25)", tooHot.block)
  }

  private def report[E, Z](label: String, result: Either[E, Z]): Unit =
    result match {
      case Right(value) => println(s"$label -> $value")
      case Left(error)  => println(s"$label -> typed error: $error")
    }
}
```

Run this example with:

```bash
cd streams-examples && sbt "runMain sink.SinkAsyncExample"
```

## See Also

- [Asynchronous Stream Execution](../execution-and-compatibility/async-execution.md#async-terminals) — the terminals that select a sink's asynchronous drain, and the `Async[Either[E, Z]]` convention
- [Platform Differences](../execution-and-compatibility/platform-differences.md#sinkcreate-and-custom-sinks) — why `Sink.create` is JVM-only and what replaces it in shared source
- [Reader](../primitives/reader.md#the-reader-union) — the two reader kinds a sink drains, from the reader's side
- [Zero-Boxing Streams](../execution-and-compatibility/zero-boxing.md) — how a primitive lane is chosen, and the per-lane end-of-stream table
- [Stream](./stream.md) — the producer a sink consumes
- [Pipeline](./pipeline.md) — transformations that can be attached to a sink's input
- [Your First Streaming Pipeline](../../../guides/your-first-streaming-pipeline.md) — a step-by-step tutorial that sums a stream into a report with a `Sink`
- [How to Stream Files and Sockets Without Leaking Resources](../../../guides/stream-files-and-sockets.md) — drain a file or socket stream into a sink and have the handle closed on every exit path

---

## Stream

`Stream[+E, +A]` is a **lazy, pull-based, typed-error stream** of elements that may fail with an error of type `E`. Nothing executes until a terminal operation is driven. Cross-platform asynchronous terminals return `Async[Either[E, Z]]`; the JVM also provides the existing blocking terminal family returning `Either[E, Z]`. Typed errors surface as `Left(e)`, while defects and cleanup failures fail the outer `Async` or propagate from a JVM blocking terminal:

```scala
abstract class Stream[+E, +A] {
  def runAsync[ES, E3, Z](sink: Sink[ES, A, Z])(implicit
    errorConcat: Concat.WithOut[E, ES, E3]
  ): Async[Either[E3, Z]]
  def runCollectAsync: Async[Either[E, Chunk[A]]]

  // JVM only
  def run[ES, E3, Z](sink: Sink[ES, A, Z])(implicit
    errorConcat: Concat.WithOut[E, ES, E3]
  ): Either[E3, Z]
}
```

`Stream` is purely functional, referentially transparent, and resource-safe:
- **Lazy**: descriptions of pipelines, not eager computations
- **Nonblocking across platforms**: `*Async` terminals drive synchronous or asynchronous readers without blocking JavaScript
- **JVM-compatible**: plain blocking terminals remain available on the JVM
- **Pull-based**: execution is driven from the sink backward through the pipeline
- **Typed errors**: distinguish recoverable errors (`E`) from untyped defects (`Throwable`)
- **Resource-safe**: RAII semantics ensure resources are released in all cases

### Asynchronous Source Constructors

The companion constructors whose names end in `Async` — `attemptAsync`, `attemptEvalAsync`, `deferAsync`, `evalAsync`, `fromAcquireReleaseAsync`, `fromIteratorAsync`, `fromReaderAsync`, and `unfoldAsync` — defer their `Async` thunk until the first reader operation is driven, and `Stream.unwrap` flattens an `Async[Stream[E, A]]` so that ordinary operators such as `flatMap`, `catchAll`, and `flatMapPar` compose with asynchronously produced streams. [Async Source Constructors](../execution-and-compatibility/async-execution.md#async-source-constructors) documents each of them, along with the laziness and error conventions they share; this page does not repeat them.

### Source Compatibility

`Reader` is an ordinary `abstract class`, not a sealed one, but every reader the library hands you is a `Reader.SyncReader[A]` or a `Reader.AsyncReader[A]`, so code that implements or accepts a reader must choose one kind or match both with a fallback case. Custom sinks cannot be written by subclassing `Sink`, whose two abstract drains are `private[streams]`; use `Sink.createAsync`, `Sink.createBoth`, or the JVM-only `Sink.create`. Plain terminals, `start`, `AsyncReader#toSync`, and `Sink.create` are JVM-only, so shared sources should migrate to `run*Async`, `startAsync`/`useReaderAsync`, and `createAsync`.

Async constructor callbacks remain lazy until the first drive, and managed/unmanaged names encode ownership. Do not compensate by eagerly opening a resource before constructing the stream. Cancellation closes an acquired reader and awaits its finalizer; `startAsync` is the exception because it explicitly transfers that responsibility to its caller.

## Motivation

Traditional eager sequences (like Scala `List`) fall short in **three critical dimensions**. Here's what `Stream[E, A]` solves for each:

**1. Efficiency — Wasteful Computation**

With eager evaluation, the entire dataset is processed upfront, regardless of how many elements you actually need. This example shows how much work is wasted:

```scala
// With Scala List (eager evaluation)
val data = (1 to 1_000_000).toList
val result = data
  .map(_ * 2)          // eagerly: 1M multiplications
  .filter(_ > 10)      // eagerly: 1M comparisons
  .take(10)            // finally: keep only 10
// ❌ Wasted work: computed and discarded 999,990 elements!
```

The problem: `List` eagerly applies `.map` and `.filter` to all 1 million elements, even though only the first 10 passing elements matter. In data processing pipelines (parsing CSV files, filtering logs, transforming sensor streams), this is enormously wasteful.

With `Stream[E, A]`, the architecture is **inverted**: the **sink (consumer) pulls** from the stream. If the sink asks for only 10 elements, only ~20 calculations occur (enough to find 10 valid results after filtering):

```scala

// With Stream (lazy, pull-based evaluation)
val result: Either[Nothing, zio.blocks.chunk.Chunk[Int]] =
  Stream.fromRange((1 to 100))
    .map(_ * 2)
    .filter(_ > 10)
    .run(Sink.take(10))
// ✓ Computation stops after 10 valid elements are produced
// Only necessary work: ~20 multiplications, ~20 comparisons
```

This **short-circuiting** behavior is automatic and requires no special syntax.

**2. Resource Management — Error-Prone Cleanup**

When you open resources (file handles, network connections, database cursors), you must release them in **all** code paths—success, error, and even mid-stream cancellation. With eager sequences, this burden falls on the caller:

```
// ❌ With traditional Scala (manual resource management, uses var for mutable state)

var file: BufferedReader = null
try {
  file = new BufferedReader(new FileReader("build.sbt"))
  var count = 0L
  var char = file.read()
  while (char != -1) {
    if (!Character.isWhitespace(char)) {
      count += 1
    }
    char = file.read()
  }
  count
} catch {
  case e: IOException =>
    throw e
} finally {
  if (file != null) file.close()  // ✓ Manual cleanup in finally
}
// ❌ Problem: You must remember the finally block
// ❌ Problem: If an exception occurs in the loop, cleanup must still run (easy to forget!)
// ❌ Problem: Scale to 10 resources? 50 resources? Manually nesting becomes error-prone
```

With `Stream[E, A]`, resource cleanup is **automatic, composable, and guaranteed**—even on error or if the sink cancels early:

```scala

// With Stream (resource-safe RAII)
// Open a file and count non-whitespace characters
val charCount: Either[IOException, Long] =
  Stream
    .fromAcquireRelease(new FileReader("build.sbt"))(Stream.fromJavaReaderUnmanaged)  // opens the file when the stream runs
    .filter(!_.isWhitespace)  // process only non-whitespace
    .count  // count all matching characters
// ✓ The file closes when the stream ends: on success, on a read error, or on an early stop
// ✓ If reading, filter, or count throws after the file opens, it still closes
// ✓ No manual try/finally needed; no resource leak risk
// ✓ Multiple resources (files, connections, etc.) compose naturally
```

The key difference: `Stream` releases resources via **RAII** (Resource Acquisition Is Initialization) — the resource's lifetime is bound to the compiled stream's `close()` method, which the terminal operation (`run`) always calls in a `finally` block.

**3. Error Handling — Untyped Errors**

Traditional error handling conflates two categories: recoverable **domain errors** (e.g., parsing failed, validation failed) and fatal **defects** (e.g., `OutOfMemoryError`, `NullPointerException`). This makes it hard to write correct error recovery code:

```scala

// With Try/catch (untyped errors)
case class ParseError(msg: String)

def parseLines(lines: List[String]): Try[List[Int]] = Try {
  lines.map { line =>
    line.toInt  // throws NumberFormatException (defect, not domain error!)
  }
}

val result = parseLines(List("1", "abc", "3"))
result match {
  case util.Success(nums) => println(s"Parsed: $nums")
  case util.Failure(e) =>
    // ❌ Can't tell if 'e' is a parse error or a JVM defect
    // ❌ Must handle *all* exceptions the same way
    // ❌ Domain logic mixed with system-level exception handling
    println(s"Error: $e")
}
```

With `Stream[E, A]`, typed errors (`E`) are distinct from untyped defects (`Throwable`), enabling proper error recovery:

```scala

case class ParseError(msg: String)

// With Stream (typed errors)
val result: Either[ParseError, zio.blocks.chunk.Chunk[Int]] =
  Stream
    .fromIterable(List("1", "abc", "3"))
    .flatMap { line =>
      try {
        Stream.succeed(line.toInt)  // success path
      } catch {
        case _: NumberFormatException =>
          Stream.fail(ParseError(s"Not a number: $line"))  // typed error
      }
    }
    .runCollect

result match {
  case Left(parseError) =>
    // ✓ This branch is *only* for domain errors we chose to surface
    println(s"Parse error: ${parseError.msg}")
  case Right(nums) =>
    // ✓ Untyped defects (OutOfMemoryError, etc.) propagate as exceptions
    // ✓ Clear separation: Either[E, Z] is for recovery, uncaught exceptions are fatal
    println(s"Parsed: ${nums}")
}
```

The key distinction: `Either[ParseError, Z]` means domain errors are *recoverable* via `Left`; any uncaught `Throwable` defect propagates as an exception, which is correct—you cannot recover from running out of memory, only from bad input.

## Construction

Streams can be created from constants, collections, resources, and pull-based sources:

### Constant Streams

The simplest streams are single-element or empty streams.

#### `Stream.empty`

An empty stream that emits no elements and succeeds immediately:

```scala
object Stream {
  val empty: Stream[Nothing, Nothing]
}
```

The empty stream is useful as a base case in recursive stream builders or as a neutral element when concatenating:

```scala

val emptyStream = Stream.empty
// emptyStream: Stream[Nothing, Nothing] = Stream.empty
val result = emptyStream.runCollect
// result: Either[Nothing, Chunk[Nothing]] = Right(IndexedSeq())
// emptyStream contains no elements
```

#### `Stream.succeed[A]`

Wraps a single value of any type. Specialized overloads avoid boxing for primitives:

```scala
object Stream {
  def succeed[A](a: A): Stream[Nothing, A]
  def succeed(a: Int): Stream[Nothing, Int]
  def succeed(a: Long): Stream[Nothing, Long]
  def succeed(a: Double): Stream[Nothing, Double]
  // ... and Byte, Short, Char, Float, Boolean variants
}
```

When you call `Stream.succeed(value)`, the stream emits exactly one element and completes successfully. This is useful for wrapping a computed value into the stream abstraction:

```scala

val singleElement = Stream.succeed(42)
// singleElement: Stream[Nothing, Int] = Stream.succeed(...)
val result = singleElement.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(42))
```

The `Byte` overload remains a byte stream: `Stream.succeed(1.toByte)` has type `Stream[Nothing, Byte]`, uses the `Byte` representation lane, and compiles through `Reader.singleByte` rather than widening the element type to `Int`.

#### `Stream.fail[E]`

Creates a stream that fails immediately with a typed error:

```scala
object Stream {
  def fail[E](error: E): Stream[E, Nothing]
}
```

Use `fail` when you need to short-circuit a stream with a known error:

```scala

sealed trait ApiError
case class NotFound(id: String) extends ApiError

val failedStream = Stream.fail(NotFound("user-123"))
// failedStream: Stream[NotFound, Nothing] = Stream.fail(...)
val result = failedStream.runDrain
// result: Either[NotFound, Unit] = Left(NotFound("user-123"))
// result is Left(NotFound("user-123"))
```

#### `Stream.die`

Throws an untyped defect (exception) immediately:

```scala
object Stream {
  def die(t: Throwable): Stream[Nothing, Nothing]
}
```

Use `die` for truly exceptional, unrecoverable conditions that should not be caught as typed errors:

```scala

val dieStream = Stream.die(new Exception("System failure"))
// dieStream: Stream[Nothing, Nothing] = Stream.die(...)
```

### From Collections

Streams can be created from existing collections and iterables, making it easy to convert `List`, `Array`, [`Chunk`](../../chunk.md), or custom iterables into lazy streams:

#### `Stream.apply[A]`

Wraps a variable number of arguments into a stream:

```scala
object Stream {
  def apply[A](as: A*)(implicit jt: JvmType.Infer[A]): Stream[Nothing, A]
}
```

This is the most natural way to lift a list of values:

```scala

val numbers = Stream(1, 2, 3, 4, 5)
// numbers: Stream[Nothing, Int] = Stream(1, 2, 3, 4, 5)
val result = numbers.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2, 3, 4, 5))
```

#### `Stream.fromChunk[A]`

Converts a `Chunk` into a stream. Chunks are immutable, indexed sequences optimized for high-performance operations:

```scala
object Stream {
  def fromChunk[A](chunk: Chunk[A])(implicit jt: JvmType.Infer[A]): Stream[Nothing, A]
}
```

Use this when you already have a `Chunk`:

```scala

val chunk = Chunk(10, 20, 30)
// chunk: Chunk[Int] = IndexedSeq(10, 20, 30)
val stream = Stream.fromChunk(chunk)
// stream: Stream[Nothing, Int] = Stream.fromChunk(...)
val result = stream.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(10, 20, 30))
```

#### `Stream.fromIterable[A]`

Converts any `Iterable[A]` (List, Set, Vector, etc.) into a stream:

```scala
object Stream {
  def fromIterable[A](it: Iterable[A])(implicit jtA: JvmType.Infer[A]): Stream[Nothing, A]
}
```

This is useful when integrating with legacy Scala collections:

```scala

val list = List("a", "b", "c")
// list: List[String] = List("a", "b", "c")
val stream = Stream.fromIterable(list)
// stream: Stream[Nothing, String] = Stream.fromIterable(...)
val result = stream.runCollect
// result: Either[Nothing, Chunk[String]] = Right(IndexedSeq("a", "b", "c"))
```

#### `Stream.fromIterator[A]`

Converts an `Iterator[A]` into a stream. The iterator is consumed lazily:

```scala
object Stream {
  def fromIterator[A](it: => Iterator[A])(implicit jtA: JvmType.Infer[A]): Stream[Nothing, A]
}
```

Create a stream from an iterator and collect all elements:

```scala

val iter = Iterator(10, 20, 30, 40)
// iter: Iterator[Int] = empty iterator
val stream = Stream.fromIterator(iter)
// stream: Stream[Nothing, Int] = Stream.fromIterator(...)
val result = stream.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(10, 20, 30, 40))
```

### From Ranges

Streams can be created from numeric ranges, providing an efficient way to generate sequences of integers without allocating memory upfront:

#### `Stream.range`

Emits integers from `from` (inclusive) to `until` (exclusive):

```scala
object Stream {
  def range(from: Int, until: Int): Stream[Nothing, Int]
}
```

This is memory-efficient (does not allocate intermediate collections):

```scala

val nums = Stream.range(0, 5)
// nums: Stream[Nothing, Int] = Stream.range(0, 5)
val result = nums.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(0, 1, 2, 3, 4))
```

#### `Stream.fromRange`

Converts a Scala `Range` object:

```scala
object Stream {
  def fromRange(range: Range): Stream[Nothing, Int]
}
```

Create a stream from a `Range` and collect elements:

```scala

val range = 1 to 10 by 2
// range: Range = Range(1, 3, 5, 7, 9)
val stream = Stream.fromRange(range)
// stream: Stream[Nothing, Int] = Stream.fromRange(...)
val result = stream.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 3, 5, 7, 9))
```

### Generators

These constructors create streams from functions and logic, useful for synthesizing infinite or computed sequences:

#### `Stream.repeat[A]`

Emits the same value infinitely:

```scala
object Stream {
  def repeat[A](a: A)(implicit jt: JvmType.Infer[A]): Stream[Nothing, A]
}
```

Infinite streams are safe because streams are lazy; nothing runs until you call a terminal operation with a stopping condition (like `take`):

```scala

val infinite = Stream.repeat(42)
// infinite: Stream[Nothing, Int] = Stream.repeat(...)
val first5 = infinite.take(5)
// first5: Stream[Nothing, Int] = Stream.repeat(...).take(5)
val result = first5.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(42, 42, 42, 42, 42))
```

#### `Stream.unfold[S, A]`

A stateful generator that emits elements based on a fold-like transition function:

```scala
object Stream {
  def unfold[S, A](s: S)(f: S => Option[(A, S)])(implicit jtA: JvmType.Infer[A]): Stream[Nothing, A]
}
```

Each iteration, `f` receives the current state and returns either `None` (stop) or `Some((element, nextState))`. This is useful for generating Fibonacci numbers or other sequences defined by a recurrence relation:

```scala

val fibonacci = Stream.unfold((0, 1)) {
  case (a, b) => Some((a, (b, a + b)))
}
// fibonacci: Stream[Nothing, Int] = Stream.unfold(...)
val first10 = fibonacci.take(10)
// first10: Stream[Nothing, Int] = Stream.unfold(...).take(10)
val result = first10.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(
//   IndexedSeq(0, 1, 1, 2, 3, 5, 8, 13, 21, 34)
// )
```

### Side Effects

These constructors embed effects and deferred computation into streams, running actions at stream execution time:

#### `Stream.eval[A]`

Runs an arbitrary side effect and emits nothing:

```scala
object Stream {
  def eval(f: => Any): Stream[Nothing, Nothing]
}
```

Use `eval` when you want a side effect in a stream (e.g., logging, metrics) but no element:

```scala

val sideEffect = Stream.eval(println("Executing side effect"))
// sideEffect: Stream[Nothing, Nothing] = Stream.suspend(...)
val result = sideEffect.runDrain
// Executing side effect
// result: Either[Nothing, Unit] = Right(())
```

#### `Stream.attempt[A]`

Wraps a potentially throwing computation, converting non-fatal `Throwable`s into a typed error. Fatal errors (like `OutOfMemoryError`) are not caught and propagate as exceptions:

```scala
object Stream {
  def attempt[A](f: => A)(implicit jtA: JvmType.Infer[A]): Stream[Throwable, A]
}
```

Use `attempt` when you have legacy code that throws exceptions:

```scala

def unsafeJsonParse(s: String): Int = s.toInt

val parsed = Stream.attempt(unsafeJsonParse("42"))
// parsed: Stream[Throwable, Int] = Stream.attempt(...)
val result = parsed.runCollect
// result: Either[Throwable, Chunk[Int]] = Right(IndexedSeq(42))
```

#### `Stream.attemptEval`

Evaluates a side effect and converts any thrown exception into a typed `Throwable` error. Unlike `eval`, this captures exceptions and emits nothing:

```scala
object Stream {
  def attemptEval(f: => Any): Stream[Throwable, Nothing]
}
```

Use `attemptEval` when you need to safely execute an effect that might throw, but you don't need to emit any elements:

```scala

val effect = Stream.attemptEval {
  val file = new java.io.File("nonexistent.txt")
  if (!file.exists()) throw new java.io.FileNotFoundException("File not found")
}
val result = effect.runDrain
```

#### `Stream.defer[A]`

Creates an empty stream that lazily registers `f` as a release action. `f` runs exactly once when each materialization closes, including after failure, early termination, or cancellation; exceptions are defects:

```scala
object Stream {
  def defer(f: => Unit): Stream[Nothing, Nothing]
}
```

Register a release action that runs when the stream closes:

```scala

val deferred = Stream.defer(println("Release action runs when the stream closes"))
// deferred: Stream[Nothing, Nothing] = Stream.defer(...)
val result = deferred.runDrain
// Release action runs when the stream closes
// result: Either[Nothing, Unit] = Right(())
```

#### `Stream.suspend[E, A]`

Defers the creation of a stream until run time, useful for recursive stream definitions:

```scala
object Stream {
  def suspend[E, A](stream: => Stream[E, A]): Stream[E, A]
}
```

Define a recursive stream safely:

```scala

def countDown(n: Int): Stream[Nothing, Int] =
  if (n <= 0) Stream.empty
  else Stream.suspend(Stream.succeed(n) ++ countDown(n - 1))

val result = countDown(5).runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(5, 4, 3, 2, 1))
```

### I/O

Streams can read from external I/O sources like files and readers, automatically managing resource cleanup:

#### `Stream.fromInputStream`

Reads bytes from a Java `InputStream`, managing the resource:

```scala
object Stream {
  def fromInputStream(is: java.io.InputStream): Stream[java.io.IOException, Byte]
}
```

The stream automatically closes the input stream when done:

```scala

val data = new ByteArrayInputStream("Hello".getBytes)
// data: ByteArrayInputStream = java.io.ByteArrayInputStream@6fe49e53
val bytes = Stream.fromInputStream(data)
// bytes: Stream[IOException, Byte] = Stream.fromAcquireRelease(...)
val result = bytes.runCollect
// result: Either[IOException, Chunk[Byte]] = Right(
//   IndexedSeq(72, 101, 108, 108, 111)
// )
```

#### `Stream.fromJavaReader`

Reads characters from a Java `Reader`:

```scala
object Stream {
  def fromJavaReader(r: java.io.Reader): Stream[java.io.IOException, Char]
}
```

Read characters from a string reader:

```scala

val reader = new StringReader("hello world")
// reader: StringReader = java.io.StringReader@53d41b8b
val stream = Stream.fromJavaReader(reader)
// stream: Stream[IOException, Char] = Stream.fromAcquireRelease(...)
val result = stream.runCollect
// result: Either[IOException, Chunk[Char]] = Right(
//   IndexedSeq('h', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd')
// )
```

#### `Stream.fromInputStreamUnmanaged`

Reads bytes from a Java `InputStream` without automatic resource management. The caller is responsible for closing the stream:

```scala
object Stream {
  def fromInputStreamUnmanaged(is: java.io.InputStream): Stream[java.io.IOException, Byte]
}
```

Use this when you need to manage the stream's lifecycle yourself, for example when the stream is created from a long-lived resource:

```scala

val data = new ByteArrayInputStream("Data".getBytes)
val bytes = Stream.fromInputStreamUnmanaged(data)
val result = bytes.runCollect
// Caller must close data when done
```

#### `Stream.fromJavaReaderUnmanaged`

Reads characters from a Java `Reader` without automatic resource management. The caller is responsible for closing the reader:

```scala
object Stream {
  def fromJavaReaderUnmanaged(r: java.io.Reader): Stream[java.io.IOException, Char]
}
```

Use this when you need to manage the reader's lifecycle yourself:

```scala

val reader = new StringReader("managed externally")
val stream = Stream.fromJavaReaderUnmanaged(reader)
val result = stream.runCollect
// Caller must close reader when done
```

## Transformations

Streams provide powerful operations for transforming elements, flattening nested structures, filtering, and managing state:

### Element-wise Transformations

These operations apply functions to stream elements one-by-one, applying the transformation lazily as elements are pulled:

#### `Stream#map[B]`

Applies a function to each element:

```scala
abstract class Stream[+E, +A] {
  def map[B](f: A => B)(implicit jtB: JvmType.Infer[B]): Stream[E, B]
}
```

`map` does not run immediately; it builds up a description of the transformation. Only when you call a terminal operation does the mapping happen:

```scala

val nums = Stream(1, 2, 3)
// nums: Stream[Nothing, Int] = Stream(1, 2, 3)
val doubled = nums.map(_ * 2)
// doubled: Stream[Nothing, Int] = Stream(1, 2, 3).map(...)
val result = doubled.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(2, 4, 6))
```

For bounded concurrency, [`Stream#mapPar`](#streammappar) applies a synchronous function with up to `n` applications active and [`Stream#mapParAsync`](#streammapparasync) keeps up to `n` `Async` callbacks in flight; both are unordered. See [Bounded Concurrency](#bounded-concurrency).

**Key point:** `Stream#map` is covariant in the output type because it preserves the error type and only transforms elements. Output-changing operations such as `map`, `collect`, `flatMap`, `mapAccum`, `scan`, and `zipWith` take `JvmType.Infer` evidence for their result type; that result evidence selects the physical output lane. Type-preserving operations retain the source's known lane, including when the static element type is widened.

#### `Stream#mapError[E2]`

Transforms typed errors without affecting elements:

```scala
abstract class Stream[+E, +A] {
  def mapError[E2](f: E => E2): Stream[E2, A]
}
```

Use `mapError` to convert one error type to another:

```scala

sealed trait ApiError
case class ServerError(msg: String) extends ApiError
case class NetworkError() extends ApiError

val mayFail: Stream[NetworkError, String] = Stream.fail(NetworkError())
val mapped = mayFail.mapError(e => ServerError("Connection failed"))
```

#### `Stream#filter`

Emits only elements that satisfy a predicate:

```scala
abstract class Stream[+E, +A] {
  def filter(pred: A => Boolean): Stream[E, A]
}
```

Short-circuits: as soon as the sink says "stop," filtering stops:

```scala

val nums = Stream(1, 2, 3, 4, 5)
// nums: Stream[Nothing, Int] = Stream(1, 2, 3, 4, 5)
val evens = nums.filter(_ % 2 == 0)
// evens: Stream[Nothing, Int] = Stream(1, 2, 3, 4, 5).filter(...)
val result = evens.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(2, 4))
```

#### `Stream#collect[B]`

Applies a partial function, emitting only defined results:

```scala
abstract class Stream[+E, +A] {
  def collect[B](pf: PartialFunction[A, B])(implicit jtB: JvmType.Infer[B]): Stream[E, B]
}
```

This combines filtering and mapping in one step:

```scala

val mixed = Stream(1, "a", 2, "b", 3)
// mixed: Stream[Nothing, Int | String] = Stream(1, a, 2, b, 3)
val numbers = mixed.collect { case n: Int => n }
// numbers: Stream[Nothing, Int] = Stream(1, a, 2, b, 3).collect(...)
val result = numbers.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2, 3))
```

### Stateful Transformations

These operations maintain internal state while processing elements, allowing you to fold computations into the transformation:

#### `Stream#mapAccum[S, B]`

Maintains state while transforming each element:

```scala
abstract class Stream[+E, +A] {
  def mapAccum[S, B](init: S)(f: (S, A) => (S, B))(implicit jtB: JvmType.Infer[B]): Stream[E, B]
}
```

`mapAccum` threads a state value through the transformation. At each step, you receive the current state and the element, return a new state and output element:

```scala

val nums = Stream(1, 2, 3)
// nums: Stream[Nothing, Int] = Stream(1, 2, 3)
val indexed = nums.mapAccum(0)((idx, x) => (idx + 1, (idx, x)))
// indexed: Stream[Nothing, Tuple2[Int, Int]] = Stream.suspend(...)
val result = indexed.runCollect
// result: Either[Nothing, Chunk[Tuple2[Int, Int]]] = Right(
//   IndexedSeq((0, 1), (1, 2), (2, 3))
// )
```

#### `Stream#mapAccumAsync[S, B]`

The asynchronous twin of `mapAccum`: the step returns an `Async`, and the state is still threaded strictly in order.

```scala
abstract class Stream[+E, +A] {
  def mapAccumAsync[S, B](init: S)(f: (S, A) => Async[(S, B)])(implicit
    jtB: JvmType.Infer[B]
  ): Stream[E, B]
}
```

At most one invocation of `f` is active at a time, which is what keeps the accumulator meaningful — there is no concurrency here to reorder the steps or to hand two invocations the same state. A failure inside `f` is a defect, not a typed error. The implicit `JvmType.Infer[B]` records the physical lane of the new output type and is supplied by the compiler.

```scala

val events   = Stream("open", "write", "close")
val numbered = events.mapAccumAsync(0L)((seq, event) => Async.succeed((seq + 1, s"$seq:$event")))
```

This operator is also catalogued with the rest of the sequential asynchronous family in [Async Operators](../execution-and-compatibility/async-execution.md#streammapaccumasync), and [Stateful Asynchronous Operators](#stateful-asynchronous-operators) runs it end to end alongside `scanAsync`, `takeWhileAsync`, and `ensuringAsync`.

#### `Stream#scan[S]`

Like `mapAccum`, but emits the accumulator rather than a mapped value, starting with `init` — so the output stream has one more element than the input:

```scala
abstract class Stream[+E, +A] {
  def scan[S](init: S)(f: (S, A) => S)(implicit jtS: JvmType.Infer[S]): Stream[E, S]
}
```

This is useful for computing running totals, moving averages, or other cumulative statistics:

```scala

val nums = Stream(1, 2, 3, 4)
// nums: Stream[Nothing, Int] = Stream(1, 2, 3, 4)
val cumsum = nums.scan(0)(_ + _)
// cumsum: Stream[Nothing, Int] = Stream(1, 2, 3, 4).scan(...)
val result = cumsum.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(0, 1, 3, 6, 10))
```

#### `Stream#scanAsync[S]`

The asynchronous twin of `scan`: the fold step returns an `Async`, and the accumulator is still emitted at each step.

```scala
abstract class Stream[+E, +A] {
  def scanAsync[S](init: S)(f: (S, A) => Async[S])(implicit jtS: JvmType.Infer[S]): Stream[E, S]
}
```

The output stream carries one more element than the input, because `init` is emitted before the first step runs. As with `mapAccumAsync`, the steps are sequential and a failure inside `f` is a defect.

```scala

val amounts  = Stream(120, -40, 75)
val balances = amounts.scanAsync(0L)((balance, amount) => Async.succeed(balance + amount))
```

`balances` emits `0`, `120`, `80`, `155`. See [Async Operators](../execution-and-compatibility/async-execution.md#streamscanasync) for the same entry alongside the rest of the asynchronous family.

### Flat-Mapping (Nested Streams)

`flatMap[E2, E3, B]` — Maps each element to a stream and flattens the results.:

```scala
abstract class Stream[+E, +A] {
  def flatMap[E2, E3, B](f: A => Stream[E2, B])(implicit
    errorConcat: Concat.WithOut[E, E2, E3],
    jtB: JvmType.Infer[B]
  ): Stream[E3, B]
}
```

`Stream#flatMap` is sequential: streams are processed one at a time, in order. This is essential for resource safety: if each inner stream acquires a resource, `Stream#flatMap` ensures they are released in proper FIFO order:

```scala

val ids = Stream(1, 2, 3)
// ids: Stream[Nothing, Int] = Stream(1, 2, 3)
val expanded = ids.flatMap(id => Stream(s"${id}-a", s"${id}-b"))
// expanded: Stream[Nothing, String] = Stream(1, 2, 3).flatMap(...)
val result = expanded.runCollect
// result: Either[Nothing, Chunk[String]] = Right(
//   IndexedSeq("1-a", "1-b", "2-a", "2-b", "3-a", "3-b")
// )
```

For the concurrent counterpart, which merges up to `n` inner streams at once in arrival order, see [`Stream#flatMapPar`](#streamflatmappar) in [Bounded Concurrency](#bounded-concurrency).

#### `Stream.flattenAll[E, A]`

Flattens a stream of streams into a single stream, processing them sequentially:

```scala
object Stream {
  def flattenAll[E, A](streams: Stream[E, Stream[E, A]])(implicit jtA: JvmType.Infer[A]): Stream[E, A]
}
```

This is equivalent to `flatMap(identity)`. Use `flattenAll` when you already have a stream of streams and want to flatten it without applying a transformation:

```scala

val nested = Stream.fromIterable(List(
  Stream(1, 2),
  Stream(3, 4)
))
// nested: Stream[Nothing, Stream[Nothing, Int]] = Stream.fromIterable(...)
val flat = Stream.flattenAll(nested)
// flat: Stream[Nothing, Int] = Stream.fromIterable(...).flatMap(...)
val result = flat.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2, 3, 4))
```

To flatten a stream of streams concurrently instead of sequentially, the companion also offers [`Stream.mergeAll`](#streammergeall), which drains up to `maxOpen` inner streams at a time. See [Bounded Concurrency](#bounded-concurrency).

## Windowing

Streams can be grouped, sliced, and scanned to process data in temporal windows. These operations group elements into chunks and slide windows over the stream for batch processing:

### `Stream#grouped[A]`

Collects elements into fixed-size chunks:

```scala
abstract class Stream[+E, +A] {
  def grouped(n: Int): Stream[E, Chunk[A]]
}
```

The last chunk may contain fewer than `n` elements:

```scala

val nums = Stream(1, 2, 3, 4, 5)
// nums: Stream[Nothing, Int] = Stream(1, 2, 3, 4, 5)
val groups = nums.grouped(2)
// groups: Stream[Nothing, Chunk[Int]] = Stream(1, 2, 3, 4, 5).chunked(2)
val result = groups.runCollect
// result: Either[Nothing, Chunk[Chunk[Int]]] = Right(
//   IndexedSeq(IndexedSeq(1, 2), IndexedSeq(3, 4), IndexedSeq(5))
// )
```

### `Stream#sliding[A]`

Creates a sliding window of size `n`, optionally stepping by `step` elements:

```scala
abstract class Stream[+E, +A] {
  def sliding(n: Int, step: Int = 1): Stream[E, Chunk[A]]
}
```

This is useful for computing local statistics or detecting patterns in sequences:

```scala

val nums = Stream(1, 2, 3, 4, 5)
// nums: Stream[Nothing, Int] = Stream(1, 2, 3, 4, 5)
val windows = nums.sliding(3, step = 1)
// windows: Stream[Nothing, Chunk[Int]] = Stream(1, 2, 3, 4, 5).sliding(3, 1)
val result = windows.runCollect
// result: Either[Nothing, Chunk[Chunk[Int]]] = Right(
//   IndexedSeq(IndexedSeq(1, 2, 3), IndexedSeq(2, 3, 4), IndexedSeq(3, 4, 5))
// )
```

## Combining Streams

Streams can be sequentially concatenated, zipped together, or merged:

### Sequential Concatenation

`++[E2, E3, A2, A3]` or `concat[E2, E3, A2, A3]` — Emits all elements of the first stream, then all elements of the second stream:

```scala
abstract class Stream[+E, +A] {
  final def ++[E2, E3, A2, A3](that: Stream[E2, A2])(implicit
    errorConcat: Concat.WithOut[E, E2, E3],
    valueConcat: Concat.WithOut[A, A2, A3],
    jtA3: JvmType.Infer[A3]
  ): Stream[E3, A3] = concat(that)
}
```

The result type follows the same widening rules as Scala 3 unions:

- identical types stay unchanged (`A ++ A => A`)
- subtypes widen to the supertype (`Dog ++ Animal => Animal`)
- siblings with a common meaningful supertype widen to that supertype (`Dog ++ Cat => Animal`, when both extend a sealed `Animal`)
- otherwise the result is a disjoint union (`String ++ Int => String | Int`)

On Scala 3, disjoint concat results are native unions. On Scala 2, the same/subtype and sibling cases collapse to the wider existing type (zero-cost, values are reused as-is); only types without a shared meaningful supertype fall back to `Either[L, R]`.

Evaluation is sequential: the second stream only starts when the first completes:

```scala

val first = Stream(1, 2)
// first: Stream[Nothing, Int] = Stream(1, 2)
val second = Stream(3, 4)
// second: Stream[Nothing, Int] = Stream(3, 4)
val combined = first ++ second
// combined: Stream[Nothing, Int] = Stream(1, 2) ++ Stream(3, 4)
val result = combined.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2, 3, 4))
```

For unrelated element types, Scala 3 produces a direct union while Scala 2 produces `Either`:

<Tabs groupId="scala-version" defaultValue="scala2">
  <TabItem value="scala2" label="Scala 2.13">

```scala
val combined: Stream[Nothing, Either[String, Int]] =
  Stream.succeed("left") ++ Stream.succeed(1)
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3.x">

```scala
val combined: Stream[Nothing, String | Int] =
  Stream.succeed("left") ++ Stream.succeed(1)
```

  </TabItem>
</Tabs>

```scala

val concatResult = (Stream.succeed("left") ++ Stream.succeed(1)).runCollect
// concatResult: Either[Nothing, Chunk[String | Int]] = Right(
//   IndexedSeq("left", 1)
// )

assert(concatResult == Right(Chunk[String | Int]("left", 1)))
```

The error channel follows the same rules. Same/subtype errors collapse; unrelated errors remain disjoint:

```scala

sealed trait LeftError
case class Boom(msg: String) extends LeftError
case class Missing(code: Int)

val left: Stream[LeftError, String] = Stream.fail(Boom("boom"))
// left: Stream[LeftError, String] = Stream.fail(...)
val right = Stream.succeed(true)
// right: Stream[Nothing, Boolean] = Stream.succeed(...)

left.runCollect
// res39: Either[LeftError, Chunk[String]] = Left(Boom("boom"))

val failed = left ++ (Stream.fail(Missing(404)): Stream[Missing, Boolean])
// failed: Stream[LeftError | Missing, String | Boolean] = Stream.fail(...) ++ Stream.fail(...)
failed.runCollect
// res40: Either[LeftError | Missing, Chunk[String | Boolean]] = Left(
//   Boom("boom")
// )
```

There is no separate `choice` operator anymore. Use `++` / `concat` for all sequential combination; the result type already reflects the Scala 3-style union semantics.

### Zipping

Zips two streams together as tuples:

```scala
abstract class Stream[+E, +A] {
  def &&[E2, E3, B, C](that: Stream[E2, B])(implicit
    errorConcat: Concat.WithOut[E, E2, E3],
    zip: Stream.Zip[A, B, C],
    jtC: JvmType.Infer[C]
  ): Stream[E3, C]
}
```

The error type `E3` is the `Concat` of the two error types, and the element type `C` is chosen by the `Stream.Zip` evidence, which flattens nested pairs so that `a && b && c` produces a `Stream` of `(A, B, C)`.

The result streams have the same length as the shorter input:

```scala

val nums = Stream(1, 2, 3)
// nums: Stream[Nothing, Int] = Stream(1, 2, 3)
val chars = Stream('a', 'b')
// chars: Stream[Nothing, Char] = Stream(a, b)
val zipped = nums && chars
// zipped: Stream[Nothing, Tuple2[Int, Char]] = Stream(1, 2, 3) && Stream(a, b)
val result = zipped.runCollect
// result: Either[Nothing, Chunk[Tuple2[Int, Char]]] = Right(
//   IndexedSeq((1, 'a'), (2, 'b'))
// )
```

## Bounded Concurrency

Four operators bound the concurrency of a stream. `Stream#mapPar`, `Stream#flatMapPar`, and `Stream.mergeAll` take synchronous callbacks; `Stream#mapParAsync` takes a callback that returns an `Async`. All four run on either of two execution paths — a synchronous one backed by worker threads, or an asynchronous one — and all four carry an `n = 1` degradation guarantee.

The two execution paths are different engines with different bounds, and which one a program gets is decided by the kind of reader its pipeline compiles to — not by which operator it called.

### The Operators

| Operator                            | Element callback                               | What `n` bounds                |
|-------------------------------------|------------------------------------------------|--------------------------------|
| `Stream#mapPar(n)(f)`               | `A => B`                                       | concurrent applications of `f` |
| `Stream#mapParAsync(n)(f)`          | `A => Async[B]`                                | callbacks in flight            |
| `Stream#flatMapPar(n)(f)`           | `A => Stream[E1, B]`                           | open inner streams             |
| `Stream.mergeAll(maxOpen)(streams)` | none; `streams` is a `Stream[E, Stream[E, A]]` | open inner streams             |

Each of the four begins with `require` on its parallelism argument, so passing zero or a negative number raises `IllegalArgumentException` at description time rather than producing an empty or sequential stream.

#### `Stream#mapPar`

```scala
def mapPar[B](n: Int)(f: A => B)(implicit jtB: JvmType.Infer[B]): Stream[E, B]
```

Applies `f` to each element with up to `n` applications active. Output is unordered: elements leave in the order their applications finish, not the order they entered. `f` is synchronous, so this is the operator for CPU-bound or blocking work on the JVM — and, as [Threading and Platform Behaviour](#threading-and-platform-behaviour) explains, the operator that overlaps nothing at all once the pipeline is on the asynchronous lane.

#### `Stream#mapParAsync`

```scala
def mapParAsync[B](n: Int)(f: A => Async[B])(implicit jtB: JvmType.Infer[B]): Stream[E, B]
```

Keeps at most `n` `Async` callbacks in flight and emits each result in completion order. It is the only member of the family whose callback can suspend: `f` returns a description, so the engine holds `n` unfinished effects rather than `n` busy threads. A failure inside the callback is a defect, not a typed error, and it fails the outer terminal effect.

Unlike `mapPar`, this operator has no synchronous materialization at all. It compiles to the shared asynchronous concurrent reader on both platforms, which is why its `n` counts suspended callbacks rather than workers.

#### `Stream#flatMapPar`

```scala
def flatMapPar[E1 >: E, B](n: Int)(f: A => Stream[E1, B])(implicit jtB: JvmType.Infer[B]): Stream[E1, B]
```

Applies `f` to each element to produce an inner stream, then merges up to `n` inner streams concurrently. It has no engine of its own. Past the `n = 1` branch, its body is a single delegation:

```scala
Stream.mergeAll[E1, B](n)(
  this.asInstanceOf[Stream[E1, A]].map(f)(JvmType.Infer.boxed[Stream[E1, B]])
)(jtB)
```

One fan-in engine, not two. Everything below about slots, admission, ordering, and shutdown is stated for `mergeAll`, and `flatMapPar` inherits all of it unchanged.

#### `Stream.mergeAll`

```scala
def mergeAll[E, A](maxOpen: Int)(streams: Stream[E, Stream[E, A]])(implicit jtA: JvmType.Infer[A]): Stream[E, A]
```

Merges up to `maxOpen` inner streams concurrently into one output stream, with elements arriving in completion order. Note the argument: `streams` is a *stream of streams*, not a varargs list, so a fixed collection of sources is fed in through a constructor such as `Stream.fromIterable`.

```scala

val sources: Stream[Nothing, Stream[Nothing, Int]] =
  Stream.fromIterable((0 until 10).map(i => Stream.range(i * 100, (i + 1) * 100)))

val merged: Stream[Nothing, Int] = Stream.mergeAll(4)(sources)
```

### Semantics

#### What `n` Means

`n` is a count of selector slots, not of threads. The concurrent reader allocates one `AsyncSelector` with `n + 1` entries: `n` entries for inner work, and one final entry reserved for the outer source.

```
┌────────────────────────────────────────────────────────────────────┐
│ Async.selectorWithCapacity(n + 1)      one selector, n + 1 entries │
├────────────────────────────────────────────────────────────────────┤
│ entry 0     inner work: callback in flight, or an open inner       │
│ entry 1     inner work: callback in flight, or an open inner       │
│   ...       up to n of these; `active` counts the occupied ones    │
│ entry n-1   inner work: callback in flight, or an open inner       │
├────────────────────────────────────────────────────────────────────┤
│ entry n     the outer source        re-armed only while active < n │
└────────────────────────────────────────────────────────────────────┘
```

The extra entry is what keeps the operator from pulling ahead. The source entry is re-armed only while `active < n`, so the reader stops asking upstream for elements the moment every inner slot is taken, and resumes the instant one frees. Nothing queues behind a full set of slots.

Whether a slot corresponds to a thread depends entirely on which engine materialized. On the JVM's synchronous lane each slot does have a worker thread behind it; on the asynchronous lane a slot is one pending `Async` and there are no threads involved.

#### Ordering

All four operators are unordered with respect to input position. An element leaves when its work finishes, so output is in arrival order. The property tests compare results with `.toSet` for exactly this reason: there is no input-order assertion available to make.

The single exception is `n = 1`, which is exactly sequential — and the `n = 1` tests do assert exact [`Chunk`](../../chunk.md) equality, because at that value the operator is not the concurrent engine at all. See [The `n = 1` Guarantee](#the-n-1-guarantee).

:::warning[Unordered means unordered on Scala.js too]
Single-threaded execution does not restore input order. Wherever the concurrent engine runs on Scala.js, which is always for `Stream#flatMapPar` and `Stream.mergeAll` and for a `Stream#mapPar` after an asynchronous stage, its arrival-order semantics are retained. Code that depends on Scala.js emitting source order is a bug that will not reproduce on the JVM.
:::

If input order is what you need, use sequential `map`, `mapAsync`, or `flatMap`. Sorting afterwards — `.runCollectAsync.map(_.map(_.sorted))` — recovers *a* total order, but not the input one unless the elements happen to sort that way.

#### Boundedness

Two separate things are bounded, and conflating them leads to the wrong buffer size.

The first is the number of open inner streams. Admission is guarded by an active count: `occupied` is a `Boolean` array of length `n` and `active` is the number of `true` entries, and a new element is accepted only into a free index. A full set of slots stops admission at the source rather than accumulating work anywhere.

The second is the number of buffered elements, and it exists only on the JVM's synchronous lane. `ConcurrentMapParReader` allocates `n` input and `n` output `SpscRingBuffer`s, each of `bufferSize` capacity, so a `mapPar(8)` with the default buffer holds at most 1024 elements in transit. The concurrent merge readers allocate one output ring per slot on the same basis.

The asynchronous engine has no rings of its own. `AsyncConcurrentReaders.mapPar` does not even take a buffer size: each slot holds exactly one unfinished effect, so in-flight work is bounded at `n` and nothing more. `AsyncConcurrentReaders.merge` does take one, but spends it on compiling each inner stream (`stream.compile(0, bufferSize)`), where it sizes whatever buffered stages that inner stream contains.

#### Admission and Replenishment

A slot is occupied before the work in it begins, and — for merge — freed only after that work has fully closed.

```
  free
    │  outer element arrives: occupied(i) = true, active += 1
    ▼
  constructing      the Async that builds the child runs HERE, in the slot
    │  child installed
    ▼
  draining          elements reach the consumer in arrival order
    │  MapWorkerEnd: slot entry replaced by the child's close effect
    ▼
  closing           slot still held; no successor may be admitted yet
    │  MapWorkerClosed: releaseSlot(i), then armSource()
    ▼
  free
```

That two-phase ending is the part worth remembering. When an inner stream reaches its end the engine does not free the slot; it replaces the slot's selector entry with the inner reader's own close effect, which resolves to a second signal. Only then does `releaseSlot` clear `occupied(i)`, decrement `active`, and re-arm the source. A slot is therefore never handed to a successor before its predecessor's finalizers have run to completion.

`mapPar` and `mapParAsync` have the shorter version of this, because a callback has no reader to close: the slot is released in the same step that hands the value to the consumer, and the source is re-armed there.

The "constructing" phase is where the slot accounting surprises people. With `flatMapPar(n)(a => Stream.unwrap(f(a)))`, the effect `f(a)` that *produces* the child runs inside the slot the child will later occupy — they share one of the `n`, they do not get one each. [Async children and slot accounting](#async-children-and-slot-accounting) below demonstrates this with a running program.

#### The `n = 1` Guarantee {#the-n-1-guarantee}

At `n = 1` each operator degrades to its sequential twin. This is a guarantee about the code path, not an optimization note: the branch sits in the public operator body, above every allocation.

```scala
def mapPar[B](n: Int)(f: A => B)(implicit jtB: JvmType.Infer[B]): Stream[E, B] = {
  require(n >= 1, s"mapPar requires n >= 1, got $n")
  if (n == 1) map(f)
  else {
    jtB.jvmType
    new Stream.MapPar[E, A, B](this, n, f, elementRepresentation, jtB.jvmType)
  }
}
```

The other three are shaped identically: `mapParAsync(1)` returns `mapAsync(f)`, `flatMapPar(1)` returns `flatMap(f)`, and `mergeAll(1)` returns `streams.flatMap(identity)` — or, when the outer stream is already a `Mapped` node, the fused `mapped.self.flatMap(mapped.f)` that skips the intermediate stream entirely. No reader, no selector, no thread, and no ring is allocated in any of those cases, because the decision is made before the concurrent node is ever constructed.

The practical consequence is that `n` can be a configuration value that is allowed to be `1`. A deployment that dials concurrency down to one gets the sequential operator, with its exact input ordering, rather than a concurrent engine running at width one.

### Buffer Sizing

`Stream.bufferSize(n)` sets the queue size of the concurrent operators built inside it, in place of the default 64 (the library-internal `Stream.DefaultBufferSize`, which is not part of the public API). On the JVM, `n` is the size of each ring-buffer queue of a `Stream#mapPar` whose upstream is synchronous:

```scala

def heavyComputation(n: Int): Int = n * n

val sized: Stream[Nothing, Int] =
  Stream.bufferSize(256) {
    Stream.range(0, 1000000).mapPar(8)(heavyComputation)
  }
```

`Stream.bufferSize(n)` requires a positive power of two and rejects anything else with `IllegalArgumentException`:

```scala
require(n >= 1 && (n & (n - 1)) == 0, s"bufferSize must be a positive power of 2, got $n")
```

Nested regions use the innermost size. Larger buffers absorb bursty producers; smaller ones cut memory when many slots are open at once. The default suits most workloads. On the asynchronous lane its reach is narrower: `mapPar` and `mapParAsync` ignore it entirely, because each slot holds exactly one unfinished effect and the bound is the slot count, while `mergeAll` and `flatMapPar` still pass it down, where it sizes the buffered stages inside each compiled inner stream.

:::warning[Known bug in 0.0.56: blocking terminals ignore `bufferSize`]
A JVM blocking terminal (`Stream#runCollect`, `Stream#run` and the rest) ignores `n` and keeps queues of 64, unless the stream also has an asynchronous stage, `Stream#catchAll`, `Stream#catchDefect` or `Stream#orElse`. The `*Async` terminals (`Stream#runCollectAsync`, `Stream#runAsync` and the rest) use `n`, so run the stream with one of them when the queue size matters.
:::

`Pipeline.buffer(n)` is a different tool for a different job: it inserts a bounded buffer between two stages rather than resizing the queues inside one concurrent reader, and it participates in the asynchronous reader graph on both platforms.

### Error Behaviour

First failure wins. The concurrent reader commits a terminal state exactly once, and the first typed source or inner-stream error to reach that commit becomes the result; the fold short-circuits and the operation surfaces as `Left(e)`.

```scala

// A typed error anywhere upstream terminates every worker.
val fromUpstream: Stream[String, Int] =
  Stream
    .range(0, 1000)
    .flatMap(n => if (n == 500) Stream.fail("bad element") else Stream.succeed(n))
    .mapPar(4)(identity)

// A typed error in one inner stream terminates the merge.
val fromInner: Stream[String, Int] =
  Stream.mergeAll(4)(
    Stream.fromIterable(
      List(Stream.range(0, 100), Stream.fail("inner error"), Stream.range(200, 300))
    )
  )
```

Both of those collect to a `Left`. Elements that were already emitted stay emitted — ordering is arrival-based, so a consumer may well have seen output from other slots before the failing one reached the commit point.

Once a failure is committed, cleanup runs and every sibling is torn down. A failure *during* that cleanup is attached to the primary failure rather than substituted for it: the engine calls `StreamError.attachCleanupReplay(primary, secondary)`, so the error a caller sees is still the one that caused the termination, with the cleanup problem carried alongside it. The same rule holds on the successful path, where a cleanup failure with no primary becomes the failure via `StreamError.attachCleanup(null, cleanupFailure)`.

:::note[What the tests actually prove]
`MergeInnerErrorSpec` asserts `result.isLeft` across the generic, `Int`, `Long`, `Float`, and `Double` lanes under a ten-second timeout, and again with four coordinated simultaneous failures per lane under a thirty-second one. That establishes the general shape — a failing inner terminates the merge promptly and does not hang — but it does not pin down *which* failure wins when several race at `n > 1`. Do not write code that depends on a particular one of several concurrent errors being the one reported.
:::

A defect is different from a typed error. The `mapParAsync` callback failing, or an `ensuring` finalizer throwing, is a defect and fails the outer terminal effect rather than appearing in the `E` channel.

### Cancellation and Shutdown

Shutdown is cooperative and ordered, and it is driven from the consumer end. Closing, failing, or cancelling the reader runs the same owned-cleanup path.

For `mergeAll` and `flatMapPar` that path is three steps, in order: shut the selector down, close every installed inner reader, then close the outer source. The steps are joined rather than sequenced-and-abandoned, so a failure in one does not skip the others — each later close still runs, and its failure is attached to the first. For `mapPar` and `mapParAsync` there are no inner readers, so it is the selector shutdown followed by the upstream close.

Cancellation goes through the `Async` cancellation protocol rather than thread interruption: the in-flight child run is cancelled with cleanup, and the reader settles its completion from the cancellation's outcome. A cancellation that arrives after the reader is already closed is recognised as stale and does not turn a clean close into a failure.

Closing is idempotent and it waits. The reader will not report closed until the cleanup it owns has finished, which is the property the slot lifecycle above depends on — a slot's successor cannot start while the predecessor's finalizers are still running.

[Cancellation](../execution-and-compatibility/async-execution.md#cancellation) covers the protocol these readers participate in, and [Async.Running#cancel](../../async.md#runningcancel) documents the primitive underneath it.

### Threading and Platform Behaviour

Which engine a concurrent operator materializes depends on the kind of reader its upstream compiled to, and the two engines differ far more than the two platforms do.

| Materialization                 | JVM reader                              | Scala.js reader                     | What actually overlaps                     |
|---------------------------------|-----------------------------------------|-------------------------------------|--------------------------------------------|
| `mapPar`, synchronous upstream  | `ConcurrentMapParReader` family         | `Reader.MappedInt` and siblings     | one worker thread per slot on the JVM      |
| `mapPar`, asynchronous upstream | `AsyncConcurrentReaders.mapPar`         | `AsyncConcurrentReaders.mapPar`     | nothing; `f` is wrapped in `Async.succeed` |
| `mapParAsync`, always           | `AsyncConcurrentReaders.mapPar`         | `AsyncConcurrentReaders.mapPar`     | whatever the `Async` callbacks suspend on  |
| `mergeAll` from a `Reader`      | `AsyncConcurrentReaders.merge`          | `AsyncConcurrentReaders.merge`      | whatever the inner streams suspend on      |
| `mergeAll` via the interpreter  | `IntConcurrentMergeReader` and siblings | not reached (no blocking terminals) | one drainer thread per slot on the JVM     |

Read the second row before choosing an operator. Once the upstream compiles to an `AsyncReader`, `Platform.createMapParReaderFromReader` routes `mapPar` to the shared asynchronous engine with the mapping function wrapped as `Async.succeed(f(a))` — an already-complete effect. The slots and the selector are all still there, but there is nothing for them to overlap, on either platform. `mapParAsync` exists precisely because a callback that returns a real `Async` is the only way to get concurrency out of that lane.

#### The JVM

Workers on the synchronous lane run on virtual threads where the runtime provides them. `Platform.startVirtualThread` obtains `Thread.ofVirtual()` reflectively, so the module builds and runs on any supported JDK and uses virtual threads on JDK 21 and later; if the reflective lookup fails for any reason, or starting the virtual thread throws, it starts a named daemon platform thread instead.

The threads are named, which makes them identifiable in a thread dump. `Stream#mapPar` names its workers `zio-blocks-mappar-worker-<n>-<index>` and its dispatcher `zio-blocks-mappar-coordinator-<n>`, where `<n>` is drawn from a counter shared by every thread that lane's reader class starts and `<index>` identifies the worker within one reader; merge uses `zio-blocks-merge-drainer-<n>-<index>` and `zio-blocks-merge-coordinator-<n>` on the same scheme. A known bug in 0.0.56 adds a `0` to every virtual thread's name, because the virtual-thread builder is created with `Thread.ofVirtual().name(prefix, 0L)`, whose two-argument form appends a counter: on JDK 21 and later a worker shows up in a thread dump as `zio-blocks-mappar-worker-<n>-<index>0`. Only the platform-thread fallback, which calls `setName` directly, uses the names as written.

#### Scala.js

There is no parallelism on Scala.js — `Platform.supportsConcurrency` is `false` and `Platform.startVirtualThread` throws `UnsupportedOperationException`. What runs in its place depends on the operator.

`Stream#mapPar` runs as a plain `Stream#map` when every stage before it is synchronous: `Platform.createMapParReaderFromReader` builds an ordinary mapped reader (`Reader.MappedIntInt`, `Reader.MappedInt`, `Reader.MappedLong`, and the rest) that maps one element at a time. When an asynchronous stage, `Stream#catchAll`, `Stream#catchDefect` or `Stream#orElse` comes before it, the upstream compiles to an `AsyncReader` and the shared asynchronous engine runs it with each call wrapped in a completed `Async`, so nothing overlaps.

`Stream#flatMapPar` and `Stream.mergeAll` always run on the shared asynchronous engine (`AsyncConcurrentReaders.merge`), because Scala.js has only the `*Async` terminals and the synchronous merge reader is built only by the JVM's blocking interpreter. The inner streams' asynchronous waits overlap, but synchronous work inside them runs one inner stream at a time, and elements leave in the order they arrive. As on the JVM, none of the three operators promises input order.

[Platform Differences](../execution-and-compatibility/platform-differences.md#concurrency-and-threading) states the same split from the platform side, including the full capability surface of `Platform`.

### Laws

`ConcurrentLawsSpec` asserts three equalities, all of them as set equality (`.toSet == .toSet`), since neither side of any of them is ordered:

- `Stream.mergeAll(1)(streams)` equals `streams.flatMap(identity)`
- `stream.mapPar(1)(f)` equals `stream.map(f)`
- `stream.flatMapPar(n)(f)` equals `Stream.mergeAll(n)(stream.map(f))`

The first two additionally hold as exact equality, and not because the engine happens to preserve order — at `n = 1` the operator body *returns the sequential operator itself*, so the two sides are the same description. The third holds only as a set equality: both sides run the concurrent engine at width `n`, so both are in arrival order and neither has an input order to compare against.

The third law is also a statement about the implementation rather than a coincidence, since `flatMapPar` is defined as that right-hand side. Reading it as "there is one fan-in engine" is more useful than reading it as a property that had to be checked.

### Bounded Concurrency Examples

#### `mapPar`, `mergeAll`, and `flatMapPar` on the JVM

The three snippets below use blocking terminals, which exist only on the JVM. In cross-platform code, swap the terminal for its `*Async` twin — `runCollectAsync`, `runFoldAsync` — and the operator itself is unchanged.

Expensive per-element work across eight slots:

```scala

val doubled = Stream
  .range(0, 1000)
  .mapPar(8) { n =>
    Thread.sleep(1)
    n * 2
  }
  .runCollect
```

`doubled` is a `Right` holding all thousand elements, in arrival order rather than `0, 2, 4, …`.

Ten sources drained four at a time:

```scala

val sources = Stream.fromIterable((0 until 10).map(i => Stream.range(i * 100, (i + 1) * 100)))
val summed  = Stream.mergeAll(4)(sources).runFold(0L)(_ + _)
```

A sum is order-insensitive, which is what makes it a safe thing to compute over an unordered merge: `summed` is `Right(499500)` on every run.

One sub-stream per element, eight drained at a time:

```scala

val flattened = Stream
  .range(0, 50)
  .flatMapPar(8)(i => Stream.range(i * 20, (i + 1) * 20))
  .runFold(0L)(_ + _)
```

Again `Right(499500)`: the same 1000 integers, reached through 50 inner streams instead of 10.

#### A Worked `mapParAsync`

The two runnable files below live in the `streams-examples` module. This one makes arrival order visible rather than asserting it: every callback hands back an unresolved `Completer`, and a driver thread then settles the four of them in reverse. The collected chunk comes back reversed with respect to the input.

```scala title="streams-examples/src/main/scala/stream/MapParAsyncExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 */
package stream

/**
 * `mapParAsync` keeps at most `n` `Async` callbacks in flight and emits each
 * result the moment it completes. Output is therefore in arrival order, not
 * input order.
 *
 * This example makes that visible rather than asserting it. Each callback hands
 * back a `Completer` instead of a value, so nothing completes on its own; a
 * driver thread then settles the four callbacks in reverse order. The collected
 * chunk comes back reversed with respect to the input.
 */
object MapParAsyncExample {
  def main(args: Array[String]): Unit = {
    val inputs = Chunk(1, 2, 3, 4)

    // Every callback registers its completer and reports for duty. Nothing
    // resolves until the driver thread below decides that it should.
    val pending     = new ConcurrentHashMap[Int, Completer[Int]]
    val allInFlight = new CountDownLatch(inputs.length)

    val arrivals: Async[Either[Nothing, Chunk[Int]]] =
      Stream
        .fromChunk(inputs)
        .mapParAsync(4) { value =>
          val completer = new Completer[Int]
          pending.put(value, completer)
          allInFlight.countDown()
          completer
        }
        .runCollectAsync

    // n = 4 and there are four elements, so all four callbacks are admitted
    // before any of them completes. The driver settles 4 first and 1 last.
    val driver = new Thread(() => {
      allInFlight.await()
      List(4, 3, 2, 1).foreach { value =>
        pending.get(value).succeed(value * 10)
        Thread.sleep(100L)
      }
    })
    driver.setDaemon(true)
    driver.start()

    // `.block` is JVM-only; on Scala.js drive the Async with map/flatMap.
    val collected = arrivals.block

    println(s"input order:   ${inputs.map(_ * 10)}")
    println(s"arrival order: $collected")

    // The elements are all there...
    require(
      collected.map(_.toSet) == Right(Set(10, 20, 30, 40)),
      s"unexpected elements: $collected"
    )
    // ...but not in the order they went in.
    require(
      collected != Right(inputs.map(_ * 10)),
      "mapParAsync emitted input order; arrival order was expected"
    )
  }
}
```

Run it with:

```bash
cd streams-examples && sbt "runMain stream.MapParAsyncExample"
```

It prints the two orders side by side:

```
input order:   Chunk(10,20,30,40)
arrival order: Right(Chunk(40,30,20,10))
```

#### Async Children and Slot Accounting

`flatMapPar(n)(a => Stream.unwrap(f(a)))` is the idiom for asynchronously produced children, and this example pins down what a slot covers. A gauge is incremented when child construction starts and decremented by the child's finalizer, so it counts exactly the elements occupying a slot; a `CyclicBarrier(2)` forces each construction to wait for a partner, so the program can only finish if two really are in flight at once.

```scala title="streams-examples/src/main/scala/stream/FlatMapParAsyncChildrenExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 */
package stream

/**
 * `flatMapPar(n)(a => Stream.unwrap(f(a)))` is the idiom for asynchronously
 * produced children. The point of this example is the slot accounting: the
 * effect that *builds* the child and the child that is then *drained* share one
 * of the `n` slots, so a slot is occupied from the moment construction starts
 * until the child has finished closing.
 *
 * `inFlight` is incremented when construction starts and decremented by the
 * child's finalizer, so it counts exactly the elements occupying a slot. With
 * `n = 2` its peak is 2 and never 4, even though four outer elements are
 * available immediately: the engine re-arms the outer source only while
 * `active < n`, so it will not pull ahead to start a third construction.
 *
 * The `CyclicBarrier(2)` proves the lower bound from the other side. Each
 * construction waits for a partner, so the run can only finish if two
 * constructions really are in flight at once — and it pairs off exactly twice.
 */
object FlatMapParAsyncChildrenExample {
  private val inFlight = new AtomicInteger
  private val peak     = new AtomicInteger
  private val pairUp   = new CyclicBarrier(2)

  /**
   * Produces a value on a thread of its own. Blocking inside an `Async` would
   * stall the driver that is running the stream, so the barrier wait has to
   * happen somewhere else.
   */
  private def deferred[A](value: => A): Async[A] = {
    val completer = new Completer[A]
    val thread    = new Thread(() => completer.succeed(value))
    thread.setDaemon(true)
    thread.start()
    completer
  }

  /** The asynchronously produced child for one outer element. */
  private def child(value: Int): Async[Stream[Nothing, Int]] =
    deferred {
      val current = inFlight.incrementAndGet()
      peak.getAndUpdate(seen => if (current > seen) current else seen)
      pairUp.await()
      Stream(value * 10, value * 10 + 1).ensuring {
        inFlight.decrementAndGet()
        ()
      }
    }

  def main(args: Array[String]): Unit = {
    // `.block` is JVM-only; on Scala.js drive the Async with map/flatMap.
    val collected = Stream(1, 2, 3, 4)
      .flatMapPar(2)(value => Stream.unwrap(child(value)))
      .runCollectAsync
      .block

    println(s"elements: ${collected.map(_.toSet.toList.sorted)}")
    println(s"peak slot occupancy: ${peak.get()} of 2")

    require(
      collected.map(_.toSet) == Right(Set(10, 11, 20, 21, 30, 31, 40, 41)),
      s"unexpected elements: $collected"
    )
    // Two slots, so at most two elements are ever being constructed or drained.
    require(peak.get() == 2, s"peak occupancy was ${peak.get()}, expected 2")
    // Every slot was released, which means every child was closed.
    require(inFlight.get() == 0, s"${inFlight.get()} children were left open")
  }
}
```

Run it with:

```bash
cd streams-examples && sbt "runMain stream.FlatMapParAsyncChildrenExample"
```

```
elements: Right(List(10, 11, 20, 21, 30, 31, 40, 41))
peak slot occupancy: 2 of 2
```

Peak occupancy is 2 and not 4, even though all four outer elements are available immediately, because the construction effect holds the slot its child will use. For sequential asynchronous children, `flatMap(a => Stream.unwrap(f(a)))` is the operator you want instead.

### Comparison with Other Libraries

The concurrent surface is small, and what distinguishes it is less the operator names than what a caller has to bring along to use them.

| Feature              | ZIO Blocks Streams                 | fs2                | Kyo               | Ox                    | Pekko                      |
|----------------------|------------------------------------|--------------------|-------------------|-----------------------|----------------------------|
| Concurrent operators | `mapPar`, `mergeAll`, `flatMapPar` | `parEvalMap`       | `mapParUnordered` | `mapPar`              | `mapAsync`, `flatMapMerge` |
| Effect system        | none required                      | cats-effect        | Kyo               | none; virtual threads | Akka                       |
| Typed errors         | `Either[E, Z]`                     | `ApplicativeError` | Kyo effects       | exceptions            | none                       |

The table compares contracts, not speed. Cross-provider throughput rankings require the specific benchmark classes that were built to compare like with like, and none of the numbers from those runs belong in a row next to a feature name.

## Other Operations

Common utilities for deduplication, draining, and error recovery:

### Filtering Duplicates

These operations remove duplicate elements, useful for deduplicating streams before processing:

#### `Stream#distinct[A]`

Emits only unique elements (using a mutable `HashSet` internally):

```scala
abstract class Stream[+E, +A] {
  def distinct: Stream[E, A]
}
```

This consumes memory proportional to the number of unique elements:

```scala

val nums = Stream(1, 2, 2, 3, 3, 3)
// nums: Stream[Nothing, Int] = Stream(1, 2, 2, 3, 3, ...)
val unique = nums.distinct
// unique: Stream[Nothing, Int] = Stream.suspend(...)
val result = unique.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2, 3))
```

#### `Stream#distinctBy[K]`

Emits only elements whose key (computed by `f`) has not been seen before:

```scala
abstract class Stream[+E, +A] {
  def distinctBy[K](f: A => K): Stream[E, A]
}
```

This deduplicates elements by a computed key, keeping only the first occurrence of each key:

```scala

case class Person(id: Int, name: String)

val people = Stream(
  Person(1, "Alice"),
  Person(2, "Bob"),
  Person(1, "Alice2"),  // same id as first, dropped
  Person(3, "Charlie")
)

val unique = people.distinctBy(_.id)
val result = unique.runCollect
```

### Skipping and Taking

These operations skip or limit elements, allowing you to keep or drop unwanted portions of the stream:

#### `Stream#drop`

Skips the first `n` elements:

```scala
abstract class Stream[+E, +A] {
  def drop(n: Long): Stream[E, A]
}
```

Dropping the first 3 elements and collecting the remainder:

```scala

val nums = Stream(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
val remaining = nums.drop(3)
val result = remaining.runCollect
```

#### `Stream#take`

Emits at most the first `n` elements, then stops:

```scala
abstract class Stream[+E, +A] {
  def take(n: Long): Stream[E, A]
}
```

This naturally short-circuits: the stream stops pulling from upstream:

```scala

val nums = Stream.range(0, 1000)
// nums: Stream[Nothing, Int] = Stream.range(0, 1000)
val first10 = nums.take(10)
// first10: Stream[Nothing, Int] = Stream.range(0, 1000).take(10)
val result = first10.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(
//   IndexedSeq(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
// )
```

#### `Stream#takeWhile`

Emits elements while a predicate is true, then stops:

```scala
abstract class Stream[+E, +A] {
  def takeWhile(pred: A => Boolean): Stream[E, A]
}
```

Taking elements while they are less than 6 stops early without processing the rest:

```scala

val nums = Stream(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
// nums: Stream[Nothing, Int] = Stream(1, 2, 3, 4, 5, ...)
val firstFive = nums.takeWhile(_ < 6)
// firstFive: Stream[Nothing, Int] = Stream(1, 2, 3, 4, 5, ...).takeWhile(...)
val result = firstFive.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2, 3, 4, 5))
```

#### `Stream#takeWhileAsync`

The asynchronous twin of `takeWhile`, for a predicate that has to await something before it can answer:

```scala
abstract class Stream[+E, +A] {
  def takeWhileAsync(pred: A => Async[Boolean]): Stream[E, A]
}
```

Elements are tested sequentially, and the first `false` closes upstream — so the element that failed the test is not emitted, and nothing beyond it is pulled. Predicate failure is a defect. No `JvmType.Infer` evidence appears here because the element type does not change.

```scala

val feed      = Stream(120, -40, 75, 0, 999)
val untilZero = feed.takeWhileAsync(amount => Async.succeed(amount != 0))
```

The same entry appears in [Async Operators](../execution-and-compatibility/async-execution.md#streamtakewhileasync).

### Interspersing

`intersperse[A2, A3]` — Inserts a separator value between every two elements.:

```scala
abstract class Stream[+E, +A] {
  def intersperse[A2, A3](sep: A2)(implicit
    valueConcat: Concat.WithOut[A, A2, A3],
    jtA3: JvmType.Infer[A3]
  ): Stream[E, A3]
}
```

This is useful for rendering comma-separated lists or row delimiters:

```scala

val items = Stream("a", "b", "c")
// items: Stream[Nothing, String] = Stream(a, b, c)
val separated = items.intersperse(", ")
// separated: Stream[Nothing, String] = Stream(a, b, c).intersperse(...)
val result = separated.runCollect
// result: Either[Nothing, Chunk[String]] = Right(
//   IndexedSeq("a", ", ", "b", ", ", "c")
// )
```

### Repeating

`repeated` — Repeats the stream forever: when a cycle completes, the next one starts. A typed error or a defect ends the repetition, and closing the result (for example with a downstream `Stream#take`) closes the active cycle:

```scala
abstract class Stream[+E, +A] {
  def repeated: Stream[E, A]
}
```

This creates an infinite repetition of the stream:

```scala

val original = Stream(1, 2)
// original: Stream[Nothing, Int] = Stream(1, 2)
val repeated = original.repeated.take(6)
// repeated: Stream[Nothing, Int] = Stream(1, 2).repeated.take(6)
val result = repeated.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2, 1, 2, 1, 2))
```

:::warning[Known bug in 0.0.56: a new cycle resets readers instead of rebuilding the stream]
With both terminal families, a new cycle usually resets the previous cycle's readers instead of building the stream again. `Stream(...)`, `Stream.fromChunk`, `Stream.fromIterable`, `Stream.range` and `Stream.unfold`, with `Stream#map` or `Stream#filter` over them, repeat correctly, as above. But effects that run when the stream opens (`Stream.suspend`, the `acquire` of `Stream.fromAcquireRelease`) and a `Stream#ensuring` finalizer around the whole cycle can run only once in total, and so does `Stream.eval` under the blocking terminals (the asynchronous terminals run it again in every cycle). `Stream#mapAccum` and `Stream#distinct` keep their state from one cycle to the next, `Stream.fromIterator` hangs or stops after one cycle, and `Stream.fromInputStream` fails with `UnsupportedOperationException`. A cycle that emits nothing makes a blocking pull loop forever.
:::

### Side Effects

`tapEach` — Applies a function to each element for side effects, passing the element through unchanged.:

```scala
abstract class Stream[+E, +A] {
  def tapEach(f: A => Unit): Stream[E, A]
}
```

Use `tapEach` for logging or metrics:

```scala

val nums = Stream(1, 2, 3)
// nums: Stream[Nothing, Int] = Stream(1, 2, 3)
val logged = nums.tapEach(x => println(s"Element: $x"))
// logged: Stream[Nothing, Int] = Stream(1, 2, 3).filter(...)
val result = logged.runCollect
// Element: 1
// Element: 2
// Element: 3
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2, 3))
```

## Error Handling

Streams distinguish between recoverable business errors and unexpected exceptions, providing separate recovery mechanisms for each:

### Typed Error Vs Untyped Defect

ZIO Blocks distinguishes two error channels:

- **Typed errors (`E`)**: Recoverable business logic errors. Returned as `Left(e)` from terminal operations.
- **Untyped defects (`Throwable`)**: Unexpected exceptions (bugs, system failures). Propagate as thrown exceptions.

Internally, typed errors are wrapped in `StreamError` (a non-fatal exception) and caught by the terminal operation to surface as `Left(e)`. Untyped `Throwable`s are not caught and propagate upward.

This separation allows you to:
- Use `catchAll` and `orElse` for business logic errors
- Use `catchDefect` or try-catch for unexpected exceptions
- Avoid accidentally silencing real bugs by catching all errors

Streams distinguish between recoverable domain errors and fatal defects, with flexible recovery patterns:

### Recovering From Typed Errors

These operations handle typed errors gracefully by recovering with alternative streams:

#### `Stream#catchAll[E2, A2, A3]`

Recovers from any typed error by switching to a recovery stream:

```scala
abstract class Stream[+E, +A] {
  def catchAll[E2, A2, A3](f: E => Stream[E2, A2])(implicit
    valueConcat: Concat.WithOut[A, A2, A3],
    jtA3: JvmType.Infer[A3]
  ): Stream[E2, A3]
}
```

The recovery function receives the error and can return a new stream:

```scala

sealed trait Error
case object NotFound extends Error

val mayFail: Stream[Error, String] = Stream.fail(NotFound)
// mayFail: Stream[Error, String] = Stream.fail(...)
val recovered = mayFail.catchAll(_ => Stream.succeed("default"))
// recovered: Stream[Nothing, String] = Stream.fail(...).catchAll(...)
val result = recovered.runCollect
// result: Either[Nothing, Chunk[String]] = Right(IndexedSeq("default"))
```

#### `Stream#orElse[E2, A2, A3]`

If this stream fails, tries the fallback stream. The fallback is evaluated lazily, only on error:

```scala
abstract class Stream[+E, +A] {
  def orElse[E2, A2, A3](that: => Stream[E2, A2])(implicit
    valueConcat: Concat.WithOut[A, A2, A3],
    jtA3: JvmType.Infer[A3]
  ): Stream[E2, A3]
}
```

`||` is an alias for `orElse`:

```scala

val primary = Stream.fail("error")
// primary: Stream[String, Nothing] = Stream.fail(...)
val fallback = Stream.succeed(42)
// fallback: Stream[Nothing, Int] = Stream.succeed(...)
val result = (primary || fallback).runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(42))
```

### Recovering From Defects

`catchDefect[E2, E3, A2, A3]` — Catches untyped defects (exceptions not wrapped as typed errors) using a partial function.:

```scala
abstract class Stream[+E, +A] {
  def catchDefect[E2, E3, A2, A3](
    f: PartialFunction[Throwable, Stream[E2, A2]]
  )(implicit
    errorConcat: Concat.WithOut[E, E2, E3],
    valueConcat: Concat.WithOut[A, A2, A3],
    jtA3: JvmType.Infer[A3]
  ): Stream[E3, A3]
}
```

Use `catchDefect` when you need to handle unexpected exceptions that were not wrapped by `attempt`:

```scala

val risky = Stream.die(new IllegalArgumentException("Not allowed"))
val safe = risky.catchDefect {
  case e: IllegalArgumentException => Stream.succeed(-1)
}
val result = safe.runCollect
```

## Resource Management

**The Problem:** Resources like files, database connections, and network sockets must be explicitly closed after use. If you just process them in a stream and forget to close, you leak resources. If an error occurs during processing, manual cleanup code might be skipped.

**The Solution:** ZIO Blocks streams provide three patterns for safe, automatic resource cleanup:

### `Stream.fromAcquireRelease[R, E, A]`

Acquires a resource, uses it in a stream, and **guarantees cleanup regardless of success or failure**:

```scala
object Stream {
  def fromAcquireRelease[R, E, A](
    acquire: => R,                                    // How to open the resource
    release: R => Unit = (r: R) =>                    // How to close it (defaults to .close())
      r match { 
        case ac: AutoCloseable => ac.close()
        case _ => ()
      }
  )(use: R => Stream[E, A]): Stream[E, A]
}
```

This is the fundamental pattern for safe resource handling:
1. **Acquire** — opens the resource (runs once, before streaming)
2. **Use** — streams elements from the resource
3. **Release** — closes the resource in a `finally` block (always runs, even on error)

In the example below, the connection opens when `Stream#runCollect` starts and closes when the stream ends:

```scala

case class DatabaseConnection(id: String) {
  def close(): Unit = println(s"Closing connection $id")
  def query(q: String): List[String] = List("result1", "result2")
}

val managed = Stream.fromAcquireRelease(
  acquire = {
    println("Opening database connection")
    DatabaseConnection("db-1")
  },
  release = _.close()  // Guaranteed to run even if streaming fails
)(conn => Stream.fromIterable(conn.query("SELECT *")))
// managed: Stream[Nothing, String] = Stream.fromAcquireRelease(...)

val result = managed.runCollect
// Opening database connection
// Closing connection db-1
// result: Either[Nothing, Chunk[String]] = Right(
//   IndexedSeq("result1", "result2")
// )
```

Even if the stream fails, cleanup runs before the `Left` comes back:

```scala

val managed = Stream.fromAcquireRelease(
  acquire = { println("Opening"); "resource" },
  release = { r => println(s"Closing $r") }
)(_ => Stream.fail("error occurred"))
// managed: Stream[String, Nothing] = Stream.fromAcquireRelease(...)

val result = managed.runCollect
// Opening
// Closing resource
// result: Either[String, Chunk[Nothing]] = Left("error occurred")
```

### `Stream.fromResource[R, E, A]`

Uses a ZIO Blocks `Resource[R]` (more abstract, composable resource type) within a stream:

```scala
object Stream {
  def fromResource[R, E, A](resource: Resource[R])(use: R => Stream[E, A]): Stream[E, A]
}
```

Use `fromResource` when you already have a `Resource` value, or when you need resource composition. The resource is acquired at stream start and released when the stream terminates:

```scala

val resource = Resource.acquireRelease(acquire = {
  println("Acquiring resource")
  42
})(release = { value =>
  println(s"Releasing resource with value: $value")
})

val stream = Stream.fromResource(resource) { value =>
  Stream(value, value * 2, value * 3)
}

val result = stream.runCollect
```

### `Stream#ensuring`

Adds a **cleanup action to any stream**, regardless of how it is created. The finalizer runs in a `finally` block:

```scala
abstract class Stream[+E, +A] {
  def ensuring(finalizer: => Unit): Stream[E, A]
}
```

Use `ensuring` for simple cleanup tasks that don't fit the acquire-release pattern:

```scala

val stream = Stream(1, 2, 3)
  .ensuring {
    println("Stream finished (success or error)")
  }

val result = stream.runCollect
```

The finalizer always runs, in a `finally` block:

```scala

val managed = Stream(1, 2, 3)
  .ensuring { println("Cleaned up") }

val result = managed.runCollect
```

### `Stream#ensuringAsync`

The asynchronous twin of `ensuring`, for cleanup that is itself an `Async` — closing a socket, flushing a remote session, releasing a lease:

```scala
abstract class Stream[+E, +A] {
  def ensuringAsync(finalizer: => Async[Unit]): Stream[E, A]
}
```

The finalizer is registered lazily and awaited exactly once when the materialized stream closes: on normal completion, on failure, on early termination such as `take` or `takeWhileAsync` cutting the stream short, and on cancellation. Awaited, not merely started — the close does not complete until the finalizer does. A failure inside the finalizer is a defect, and when the stream had already failed, that defect is attached to the primary failure rather than replacing it.

```scala

val session = Stream(1, 2, 3)
  .ensuringAsync(Async.succeed(println("session closed")))

val closed = session.runCollectAsync
```

What drives that close is the terminal. Under `runCollectAsync` and every other terminal the library closes the reader for you, and so does `useReaderAsync`; under `startAsync` you own the reader, and the finalizer has not run until you await `close()`. [Resource Management](../execution-and-compatibility/async-execution.md#resource-management) works through the three cases.

## Running Streams

There are two terminal families, and which one you reach for is a platform decision rather than a stylistic one.

The **cross-platform** family is the one whose names end in `Async`: `runAsync`, `runCollectAsync`, `runDrainAsync`, `runFoldAsync`, `runForeachAsync`/`foreachAsync`, `countAsync`, `existsAsync`, `findAsync`, `forallAsync`, `headAsync`, and `lastAsync`. Each returns `Async[Either[E, Z]]`, stays lazy until driven, and awaits reader cleanup on success, failure, or cancellation. It drives a synchronous and an asynchronous pipeline alike, and it is the only family that compiles for both targets. [Async Terminals](../execution-and-compatibility/async-execution.md#async-terminals) documents the family and the `Async[Either[E, Z]]` convention it follows.

The **JVM-only** family is everything documented in the rest of this section — `run`, `runCollect`, `runDrain`, `runFold`, `runForeach`/`foreach`, `count`, `exists`, `find`, `forall`, `head`, and `last` — together with `start`, covered under [Manual Pull via `start`](#manual-pull-via-start). Those sixteen members, counting `runFold`'s four overloads, are the entire JVM-only surface of `Stream`: the terminals among them park a thread and return a plain `Either[E, Z]`, and `start` hands back a blocking `Reader.SyncReader[A]`. They live in `StreamPlatformSpecific`, whose Scala.js copy has an empty body, so calling one from shared code fails to compile for the JavaScript target rather than failing at runtime. The [availability matrix](../execution-and-compatibility/platform-differences.md#availability-matrix) lists them member by member.

Each heading below therefore carries a one-line note naming its cross-platform form.

### Collecting Results

These operations accumulate or examine stream results, running the entire stream to completion:

#### `Stream#runCollect`

JVM only. The cross-platform form is `runCollectAsync`.

Collects all elements into a `Chunk[A]`:

```scala
abstract class Stream[+E, +A] {
  def runCollect: Either[E, Chunk[A]]
}
```

This is the most common terminal operation for extracting results:

```scala

val nums = Stream(1, 2, 3, 4, 5)
// nums: Stream[Nothing, Int] = Stream(1, 2, 3, 4, 5)
val result = nums.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2, 3, 4, 5))
// result is Right(Chunk(1, 2, 3, 4, 5))
```

#### `Stream#run[E2, Z]`

JVM only. The cross-platform form is `runAsync`.

Runs the stream with a custom sink, producing result `Z`:

```scala
abstract class Stream[+E, +A] {
  def run[ES, E3, Z](sink: Sink[ES, A, Z])(implicit
    errorConcat: Concat.WithOut[E, ES, E3]
  ): Either[E3, Z]
}
```

Use `run` when you need a specialized sink operation:

```scala

val nums = Stream(1, 2, 3, 4, 5)
// nums: Stream[Nothing, Int] = Stream(1, 2, 3, 4, 5)
val sum = nums.run(Sink.foldLeft(0)((acc, x) => acc + x))
// sum: Either[Nothing, Int] = Right(15)
// sum is Right(15)
```

### Discarding Results

These operations consume streams without collecting their elements, useful when you only care about side effects:

#### `Stream#runDrain`

JVM only. The cross-platform form is `runDrainAsync`.

Consumes all elements and discards them, returning `Unit`:

```scala
abstract class Stream[+E, +A] {
  def runDrain: Either[E, Unit]
}
```

Use `runDrain` when you only care about side effects:

```scala

val nums = Stream(1, 2, 3)
// nums: Stream[Nothing, Int] = Stream(1, 2, 3)
val sideEffect = nums.tapEach(x => println(s"Processing $x"))
// sideEffect: Stream[Nothing, Int] = Stream(1, 2, 3).filter(...)
val result = sideEffect.runDrain
// Processing 1
// Processing 2
// Processing 3
// result: Either[Nothing, Unit] = Right(())
```

#### `Stream#runForeach`

JVM only. The cross-platform forms are `runForeachAsync` and its alias `foreachAsync`.

Applies a function to each element for side effects:

```scala
abstract class Stream[+E, +A] {
  def runForeach(f: A => Unit): Either[E, Unit]
}
```

Alias `foreach` also exists:

```scala

val nums = Stream(1, 2, 3)
val result = nums.foreach(x => println(s"Got: $x"))
```

### Aggregations

These operations reduce streams to single values, aggregating elements into results:

#### `Stream#runFold[Z]`

JVM only. The cross-platform form is `runFoldAsync`.

Folds all elements using an accumulator, returning the final result:

```scala
abstract class Stream[+E, +A] {
  def runFold[Z](z: Z)(f: (Z, A) => Z)(implicit jtZ: JvmType.Infer[Z]): Either[E, Z]
}
```

This is the most general aggregation, equivalent to `reduce` or `fold` on eager sequences:

```scala

val nums = Stream(1, 2, 3, 4)
// nums: Stream[Nothing, Int] = Stream(1, 2, 3, 4)
val sum = nums.runFold(0)(_ + _)
// sum: Either[Nothing, Int] = Right(10)
```

Specialized overloads for primitives avoid boxing:

```scala
def runFold(z: Int)(f: (Int, A) => Int): Either[E, Int]
def runFold(z: Long)(f: (Long, A) => Long): Either[E, Long]
def runFold(z: Double)(f: (Double, A) => Double): Either[E, Double]
```

#### `Stream#count`

JVM only. The cross-platform form is `countAsync`.

Returns the number of elements:

```scala
abstract class Stream[+E, +A] {
  def count: Either[E, Long]
}
```

Counting elements in a stream:

```scala

val nums = Stream(10, 20, 30, 40, 50)
// nums: Stream[Nothing, Int] = Stream(10, 20, 30, 40, 50)
val total = nums.count
// total: Either[Nothing, Long] = Right(5L)
```

#### `Stream#head`

JVM only. The cross-platform form is `headAsync`.

Returns the first element (or `None` if empty):

```scala
abstract class Stream[+E, +A] {
  def head: Either[E, Option[A]]
}
```

Getting the first element without collecting the entire stream:

```scala

val nums = Stream(10, 20, 30, 40, 50)
// nums: Stream[Nothing, Int] = Stream(10, 20, 30, 40, 50)
val first = nums.head
// first: Either[Nothing, Option[Int]] = Right(Some(10))
```

#### `Stream#last`

JVM only. The cross-platform form is `lastAsync`.

Returns the last element (or `None` if empty):

```scala
abstract class Stream[+E, +A] {
  def last: Either[E, Option[A]]
}
```

Getting the final element of the stream:

```scala

val nums = Stream(10, 20, 30, 40, 50)
// nums: Stream[Nothing, Int] = Stream(10, 20, 30, 40, 50)
val last = nums.last
// last: Either[Nothing, Option[Int]] = Right(Some(50))
```

#### `Stream#find[A]`

JVM only. The cross-platform form is `findAsync`.

Returns the first element satisfying a predicate:

```scala
abstract class Stream[+E, +A] {
  def find(pred: A => Boolean): Either[E, Option[A]]
}
```

Finding the first element matching a condition:

```scala

val nums = Stream(10, 20, 30, 40, 50)
// nums: Stream[Nothing, Int] = Stream(10, 20, 30, 40, 50)
val firstEven = nums.find(_ % 2 == 0)
// firstEven: Either[Nothing, Option[Int]] = Right(Some(10))
```

#### `Stream#exists[A]`

JVM only. The cross-platform form is `existsAsync`.

Returns `true` if any element satisfies a predicate, short-circuiting:

```scala
abstract class Stream[+E, +A] {
  def exists(pred: A => Boolean): Either[E, Boolean]
}
```

Checking if any element is greater than 35:

```scala

val nums = Stream(10, 20, 30, 40, 50)
// nums: Stream[Nothing, Int] = Stream(10, 20, 30, 40, 50)
val hasLargeValue = nums.exists(_ > 35)
// hasLargeValue: Either[Nothing, Boolean] = Right(true)
```

#### `Stream#forall[A]`

JVM only. The cross-platform form is `forallAsync`.

Returns `true` if all elements satisfy a predicate, short-circuiting:

```scala
abstract class Stream[+E, +A] {
  def forall(pred: A => Boolean): Either[E, Boolean]
}
```

Checking if all elements are positive:

```scala

val nums = Stream(10, 20, 30, 40, 50)
// nums: Stream[Nothing, Int] = Stream(10, 20, 30, 40, 50)
val allPositive = nums.forall(_ > 0)
// allPositive: Either[Nothing, Boolean] = Right(true)
```

## Integration with Pipeline and Sink

Streams compose with pipelines and sinks to form complete data processing flows:

### Using Pipelines

`via[B]` — Applies a `Pipeline[A, B]` transformation to the stream.:

```scala
abstract class Stream[+E, +A] {
  final def via[B](pipe: Pipeline[A, B]): Stream[E, B]
}
```

Pipelines are composable transformations that can be reused across streams and sinks. Common pipelines include `Pipeline.map`, `Pipeline.filter`, `Pipeline.take`, and `Pipeline.drop`:

```scala

val nums = Stream(1, 2, 3, 4, 5)
// nums: Stream[Nothing, Int] = Stream(1, 2, 3, 4, 5)
val pipe = Pipeline.filter((x: Int) => x > 2).andThen(Pipeline.map((x: Int) => x * 10))
// pipe: Pipeline[Int, Int] = zio.blocks.streams.Pipeline$Composed@187f4358
val result = nums.via(pipe).runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(30, 40, 50))
```

Pipelines are useful when you want to build reusable transformation logic:

```scala

def positiveIntsPipe: Pipeline[Int, Int] =
  Pipeline.filter((x: Int) => x > 0)

val mixed = Stream(-2, -1, 0, 1, 2)
// mixed: Stream[Nothing, Int] = Stream(-2, -1, 0, 1, 2)
val positives = mixed.via(positiveIntsPipe)
// positives: Stream[Nothing, Int] = Stream(-2, -1, 0, 1, 2).filter(...)
val result = positives.runCollect
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2))
```

### Understanding Sinks

A `Sink[+E, -A, +Z]` is a consumer of elements of type `A` that produces a result `Z` or fails with `E`. Sinks are contravariant in `A` (they can accept a supertype of what they expect). Common sinks include:

- `Sink.collectAll: Sink[Nothing, A, Chunk[A]]` — collects all elements
- `Sink.drain: Sink[Nothing, Any, Unit]` — discards all elements
- `Sink.count: Sink[Nothing, Any, Long]` — counts elements
- `Sink.foldLeft: Sink[Nothing, A, Z]` — folds elements with an accumulator
- `Sink.head: Sink[Nothing, A, Option[A]]` — takes the first element
- `Sink.foreach: Sink[Nothing, A, Unit]` — applies a function to each element

When you call `stream.run(sink)`, the stream is compiled to a `Reader` and the sink drains it, consuming all elements and producing the result.

## Low-Level Pull with Reader

`Reader[+Elem]` is the low-level, pull-based source that backs every stream at execution time. Use cross-platform `startAsync` for a caller-owned `Reader.AsyncReader`, or `useReaderAsync` for bracketed access that awaits close on every outcome. The JVM additionally provides blocking `start` with a [`Scope`](../../resource-management/scope.md).

### Manual Pull via `start`

`startAsync` transfers ownership to its caller, which must await `close()`. Prefer `useReaderAsync` when ownership need not escape. On the JVM, `start` opens a blocking reader within a `Scope`, which closes it when the scope exits:

```scala
abstract class Stream[+E, +A] {
  def startAsync: Async[Reader.AsyncReader[A]]
  def useReaderAsync[Z](f: Reader.AsyncReader[A] => Async[Z]): Async[Z]
  // JVM only
  def start(implicit scope: Scope): scope.$[Reader.SyncReader[A]]
}
```

`start` is JVM only, and its result type is `Reader.SyncReader[A]` rather than an undifferentiated `Reader[A]`: a pipeline with asynchronous stages still works under it, because the JVM runtime bridges those boundaries by blocking. Shared code has no such member and must use `startAsync` or `useReaderAsync` instead, both of which hand back a `Reader.AsyncReader[A]`. [Manual Pull Across Platforms](../execution-and-compatibility/platform-differences.md#manual-pull-across-platforms) compares the three, and [`Stream#startAsync`](../execution-and-compatibility/async-execution.md#streamstartasync) covers the ownership rules that come with the asynchronous form.

Use `start` to manually pull elements within a resource scope:

```scala title="streams-examples/src/main/scala/stream/ManualPullUsingStart.scala" 
package stream

object ManualPullUsingStart extends App {
  Scope.global.scoped { scope =>

    // Open a stream for manual pulling
    val reader: $[Reader.SyncReader[Int]] = Stream.range(1, 6).start(using scope)

    $(reader) { r =>
      // Iterate through reader values using the protocol directly
      // (cannot use scoped value in nested function, so use it directly)
      var current = r.read(-1)
      while (current != -1) {
        println(current) // prints 1, 2, 3, 4, 5
        current = r.read(-1)
      }
    }
    // reader is closed automatically when scope exits
  }
}
```

Use these methods when you need element-by-element control rather than running through a Sink. Do not overlap asynchronous pulls: an async reader allows one active pull, and its lifecycle operations must be awaited.

A started reader does not turn a typed error into `Left(e)` the way the terminals do: a failed pull throws the library's internal `zio.blocks.streams.internal.StreamError`, whose `value` is the stream's error. A known bug in 0.0.56: when the stream has a `Stream#flatMap`, every `Reader#close` after a failed pull throws that failure again, and so does the `Scope#scoped` block that owns the reader.

### The Reader Protocol

The pull protocol uses a **sentinel value** to signal end-of-stream:

- `read(sentinel)` — returns the next element, or `sentinel` when exhausted
- `close()` — signals the consumer is done
- `isClosed` — checks whether the reader is closed

For primitive types, specialized methods avoid boxing:

- `readBoolean(sentinel: Int): Int`
- `readByte(): Int`
- `readChar(sentinel: Int): Int`
- `readShort(sentinel: Int): Int`
- `readInt(sentinel: Long): Long`
- `readLong(sentinel: Long): Long`
- `readFloat(sentinel: Double): Double`
- `readDouble(sentinel: Double): Double`

These are the eight exact physical methods selected by `Reader.jvmType`. The tag describes the reader's actual representation and method contract, so a known primitive lane survives type-preserving wrappers and static widening. Operations that produce a different element type select their output lane from result `JvmType.Infer` evidence.

On the `Long` and `Double` lanes, the library never detects end-of-stream with `Reader#readLong(sentinel)` or `Reader#readDouble(sentinel)`, because every `Long` and `Double` value is legal data: it reads one element at a time with `Reader#readLongs(scratch, 0, 1)` or `Reader#readDoubles(scratch, 0, 1)` and stops when the returned count is negative. [EOF Signalling Per Lane](../execution-and-compatibility/zero-boxing.md#eof-signalling-per-lane) lists the end-of-stream signal of every lane.

:::note
Avoid holding references to a `Reader` obtained via `start` outside its `Scope`. The scope guarantees cleanup; escaping the reader defeats that guarantee.
:::

## Implementation Notes

ZIO Blocks Streams achieves zero-boxing via compile-time type detection and dual compilation strategies:

### JVM Primitive Specialization

By default, Scala's type system boxes primitive values into objects, which wastes memory and is slower. ZIO Blocks specializes all eight JVM primitive representations: `Boolean`, `Byte`, `Char`, `Short`, `Int`, `Long`, `Float`, and `Double`. `JvmType.Infer[A]` records a result type's physical lane at construction and at output-changing operations; type-preserving operations carry an already-known lane forward.

For example, an `Int` pipeline uses `readInt(Long.MinValue)` instead of boxing. A `Long` or `Double` internal pull cannot use a collision-free scalar sentinel, so it calls `readLongs` or `readDoubles` with a length-one array and interprets the returned count as EOF/data status. This preserves every `Long` and `Double` value:

```scala
if (jvmType eq JvmType.Int) {
  val i = source.readInt(Long.MinValue)
  // ... unboxed, fast path
} else {
  val o = reader.read(EndOfStream)  // generic boxed path
  // ...
}
```

This optimization is transparent: you write normal, high-level code, and the compiler and runtime automatically use the fast path for primitives.

### Dual Compilation: Recursive Vs Interpreter

Each stream node compiles in two ways:

1. **Recursive (`compile`)**: Builds a tree of `Reader` objects, where each operation wraps the previous one. This is fast for shallow pipelines (< 100 operations).

2. **Flat-Array Interpreter (`compileInterpreter`)**: For deep pipelines (> 100 operations), the recursive approach hits Scala's default stack-depth limit (~100) and risks `StackOverflowError`. Instead, the interpreter compiles the entire pipeline into a flat array of operations, executed iteratively.

The switch happens at `DepthCutoff = 100`. You should never see this in normal use, but it ensures that pipelines of any depth are safe.

A second split sits alongside this one: a graph made only of synchronous nodes compiles to the synchronous engine, and a graph containing any asynchronous node compiles to a separate, heap-allocated asynchronous engine. That choice is made once per materialization, never per element, and never from an annotation you write. [Why two engines](../execution-and-compatibility/async-execution.md#why-two-engines) explains what it buys, and why a purely synchronous stream pays nothing for it.

## Running the Examples

All code from this guide is available as runnable examples in the `streams-examples` module.

Clone the repository and navigate to the project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt.** Here are the available examples:

---

### Basic Usage

This example demonstrates constructing streams from collections, transforming elements with `Stream#map` and `Stream#filter`, and collecting results:

```scala title="streams-examples/src/main/scala/stream/StreamBasicUsageExample.scala" 
package stream

object StreamBasicUsageExample extends App {
  println("=== Stream Basic Usage ===\n")

  // Construction from values
  println("1. Creating a stream from values:")
  val nums = Stream(1, 2, 3, 4, 5)
  show(nums.runCollect)

  // Map transformation
  println("\n2. Transforming with map:")
  val doubled = Stream(1, 2, 3).map(_ * 2)
  show(doubled.runCollect)

  // Filter operation
  println("\n3. Filtering elements:")
  val evens = Stream(1, 2, 3, 4, 5, 6).filter(_ % 2 == 0)
  show(evens.runCollect)

  // Chaining operations
  println("\n4. Chaining multiple operations:")
  val result = Stream(1, 2, 3, 4, 5)
    .map(_ * 2)
    .filter(_ > 4)
    .runCollect
  show(result)

  // Count operation
  println("\n5. Counting elements:")
  val count = Stream(1, 2, 3, 4, 5).count
  show(count)

  // Take operation (short-circuiting)
  println("\n6. Taking first n elements (short-circuits):")
  val first3 = Stream.range(0, 1000).take(3).runCollect
  show(first3)

  // Drop operation
  println("\n7. Dropping first n elements:")
  val afterDrop = Stream(1, 2, 3, 4, 5).drop(2).runCollect
  show(afterDrop)

  // Empty stream
  println("\n8. Working with empty streams:")
  val empty = Stream.empty.runCollect
  show(empty)

  // Concatenation
  println("\n9. Concatenating streams:")
  val combined = (Stream(1, 2) ++ Stream(3, 4)).runCollect
  show(combined)
}
```

To run this example:

```bash
cd streams-examples && sbt "runMain stream.StreamBasicUsageExample"
```

### Flat-Mapping Nested Streams

This example shows how `Stream#flatMap` sequences multiple streams and flattens the results:

```scala title="streams-examples/src/main/scala/stream/StreamFlatMapExample.scala" 
package stream

object StreamFlatMapExample extends App {
  println("=== Stream FlatMap and Nested Streams ===\n")

  // Basic flatMap
  println("1. Basic flatMap - expand each element into a stream:")
  val expanded = Stream(1, 2, 3).flatMap(x => Stream(x, x * 10))
  show(expanded.runCollect)

  // FlatMap with different stream sizes
  println("\n2. FlatMap with varying sizes:")
  val varySizes = Stream(1, 2, 3).flatMap(x => Stream.range(0, x))
  show(varySizes.runCollect)

  // FlatMap with string expansion
  println("\n3. Expanding into string streams:")
  val ids          = Stream("a", "b")
  val expanded_ids = ids.flatMap(id => Stream(s"${id}_1", s"${id}_2", s"${id}_3"))
  show(expanded_ids.runCollect)

  // FlattenAll for deeply nested streams
  println("\n4. Flattening nested streams with flattenAll:")
  val nested = Stream(
    Stream(1, 2),
    Stream(3, 4),
    Stream(5, 6)
  )
  val flat = Stream.flattenAll(nested)
  show(flat.runCollect)

  // Sequential processing guarantees
  println("\n5. Sequential processing (important for side effects and resources):")
  var order   = scala.collection.mutable.Buffer[String]()
  val tracked = Stream(1, 2, 3).flatMap { x =>
    order += s"expand($x)"
    Stream(x, x + 100).tapEach(y => order += s"emit($y)")
  }
  val _ = tracked.runCollect
  show(order.toList)

  // FlatMap with error recovery
  println("\n6. FlatMap can propagate errors:")
  sealed trait Error
  case object InvalidId extends Error

  val mayFail = Stream(1, 2, -1, 3).flatMap { x =>
    if (x < 0) Stream.fail(InvalidId)
    else Stream(x, x * 2)
  }
  show(mayFail.runCollect)
}
```

Run this example:

```bash
cd streams-examples && sbt "runMain stream.StreamFlatMapExample"
```

### Error Handling

This example demonstrates typed error recovery with `fail`, `catchAll`, and `orElse`:

```scala title="streams-examples/src/main/scala/stream/StreamErrorHandlingExample.scala" 
package stream

object StreamErrorHandlingExample extends App {
  println("=== Stream Error Handling ===\n")

  sealed trait ApiError
  case object NotFound                    extends ApiError
  case class ValidationError(msg: String) extends ApiError
  case class ServerError(code: Int)       extends ApiError

  // Basic fail
  println("1. Creating a failing stream:")
  val failed: Stream[ApiError, String] = Stream.fail(NotFound)
  show(failed.runCollect)

  // catchAll for recovery
  println("\n2. Recovering from errors with catchAll:")
  val recovered = Stream.fail(NotFound).catchAll(_ => Stream.succeed("default-value"))
  show(recovered.runCollect)

  // orElse for recovery
  println("\n3. Using orElse (lazy fallback evaluation):")
  val fallback = Stream.fail(NotFound) || Stream(1, 2, 3)
  show(fallback.runCollect)

  // Error transformation with error-producing flatMap
  println("\n4. Producing typed errors in flatMap:")
  val errorExample = Stream(1, 2, 3, 4).flatMap { x =>
    if (x == 3) Stream.fail[ApiError](ValidationError("cannot process"))
    else Stream(x)
  }
  show(errorExample.runCollect)

  // Handling errors in flatMap chains
  println("\n5. Error handling in flatMap chains:")
  val chain = Stream(1, 2, 3, 4).flatMap { x =>
    if (x == 3) Stream.fail(ValidationError(s"Cannot process $x"))
    else Stream(x * 10)
  }
  show(chain.runCollect)

  // Recovering from errors in flatMap
  println("\n6. Recovering from errors with catchAll in chains:")
  val recovered_chain = Stream(1, 2, 3, 4).flatMap { x =>
    if (x == 3) Stream.fail(ValidationError(s"Cannot process $x"))
    else Stream(x * 10)
  }
    .catchAll(_ => Stream.succeed(-1))

  show(recovered_chain.runCollect)

  // Handling typed errors from attempt
  println("\n7. Recovering typed errors from Stream.attempt with catchAll:")
  val risky = Stream.attempt("not-a-number".toInt)
  val safe  = risky.catchAll { case _: NumberFormatException =>
    Stream.succeed(-1)
  }
  show(safe.runCollect)

  // Multiple error branches
  println("\n8. Distinguishing error types in recovery:")
  val multi_errors = Stream(1, 2, 3, 4).flatMap { x =>
    x match {
      case 2 => Stream.fail(NotFound)
      case 3 => Stream.fail(ValidationError("Invalid data"))
      case _ => Stream(x * 10)
    }
  }.catchAll {
    case NotFound             => Stream("missing")
    case ValidationError(msg) => Stream(s"invalid: $msg")
    case _                    => Stream("unknown error")
  }

  show(multi_errors.runCollect)
}
```

Run this example:

```bash
cd streams-examples && sbt "runMain stream.StreamErrorHandlingExample"
```

### Resource Management

This example shows how `fromAcquireRelease` and `ensuring` manage resources safely:

```scala title="streams-examples/src/main/scala/stream/StreamResourceExample.scala" 
package stream

object StreamResourceExample extends App {
  println("=== Stream Resource Management ===\n")

  // Simulated resource type
  case class Database(name: String) {
    private var closed = false

    def query(q: String): List[String] = {
      if (closed) throw new Exception("Database is closed")
      q match {
        case "users" => List("Alice", "Bob", "Charlie")
        case "ids"   => List("1", "2", "3")
        case _       => List()
      }
    }

    def close(): Unit = {
      println(s"  → Closing database: $name")
      closed = true
    }

    def isClosed: Boolean = closed
  }

  // Basic resource management
  println("1. Basic fromAcquireRelease - automatic cleanup:")
  val log = Buffer[String]()

  val managed = Stream.fromAcquireRelease(
    acquire = {
      log += "opened"
      Database("main")
    },
    release = { db =>
      db.close()
      log += "closed"
    }
  )(db => Stream.fromIterable(db.query("users")))

  val result1 = managed.runCollect
  show(result1)
  show(log.toList)

  // Ensuring cleanup
  println("\n2. Using ensuring for guaranteed cleanup:")
  log.clear()
  var finalizing = false

  val withEnsure = Stream(1, 2, 3)
    .tapEach(x => log += s"processing $x")
    .ensuring {
      finalizing = true
      log += "finalizing"
    }

  val result2 = withEnsure.runCollect
  show(result2)
  show(log.toList)
  show(finalizing)

  // Error safety
  println("\n3. Cleanup happens even on error:")
  log.clear()

  sealed trait Error
  case object ProcessingFailed extends Error

  val errorStream = Stream.fromAcquireRelease(
    acquire = {
      log += "opened"
      Database("error-test")
    },
    release = { db =>
      db.close()
      log += "closed"
    }
  )(db =>
    Stream(1, 2, 3).flatMap { x =>
      if (x == 2) Stream.fail(ProcessingFailed)
      else Stream(x)
    }
  )

  val result3 = errorStream.runCollect
  show(result3)
  show(log.toList)

  // Multiple nested resources
  println("\n4. Multiple nested resources with proper cleanup order:")
  log.clear()

  val nested = Stream.fromAcquireRelease(
    acquire = {
      log += "open db1"
      Database("db1")
    },
    release = db => {
      db.close()
      log += "close db1"
    }
  )(db1 =>
    Stream.fromAcquireRelease(
      acquire = {
        log += "open db2"
        Database("db2")
      },
      release = db2 => {
        db2.close()
        log += "close db2"
      }
    ) { db2 =>
      val data = db1.query("users") ++ db2.query("ids")
      Stream.fromIterable(data).tapEach(x => log += s"emit $x")
    }
  )

  val result4 = nested.runCollect
  show(result4)
  show(log.toList)

  // AutoCloseable integration
  println("\n5. Using AutoCloseable for simpler cleanup:")
  log.clear()

  class AutoCloseableDb extends AutoCloseable {
    def close(): Unit =
      log += "auto-closed"
  }

  val autoCloseable = Stream.fromAcquireRelease(
    acquire = {
      log += "acquired"
      new AutoCloseableDb
    }
    // release defaults to calling .close() on AutoCloseable
  )(db => Stream.succeed(42))

  val result5 = autoCloseable.runCollect
  show(result5)
  show(log.toList)
}

object X extends App {

  val charCount: Either[IOException, Long] =
    Stream
      .fromJavaReader(new StringReader("Hello\nWorld")) // lazily acquires reader
      .filter(!_.isWhitespace)                          // process only non-whitespace
      .count                                            // count all matching characters

  println(charCount) // prints Right(10) — count of non-whitespace characters
}
```

Run this example:

```bash
cd streams-examples && sbt "runMain stream.StreamResourceExample"
```

### Windowing and Scanning

This example demonstrates `grouped`, `sliding`, and `scan` for windowing and stateful transformations:

```scala title="streams-examples/src/main/scala/stream/StreamWindowingExample.scala" 
package stream

object StreamWindowingExample extends App {
  println("=== Stream Windowing and Stateful Transformations ===\n")

  // Grouped - fixed-size windows
  println("1. Grouping into fixed-size chunks:")
  val nums    = Stream(1, 2, 3, 4, 5, 6, 7)
  val grouped = nums.grouped(3)
  show(grouped.runCollect)

  // Grouped with incomplete last chunk
  println("\n2. Last chunk may be smaller:")
  val ungrouped = Stream(1, 2, 3, 4, 5).grouped(2)
  show(ungrouped.runCollect)

  // Sliding window
  println("\n3. Sliding window (default step = 1):")
  val sliding1 = Stream(1, 2, 3, 4, 5).sliding(3)
  show(sliding1.runCollect)

  // Sliding with custom step
  println("\n4. Sliding window with step > 1:")
  val sliding2 = Stream(1, 2, 3, 4, 5, 6, 7, 8).sliding(3, step = 2)
  show(sliding2.runCollect)

  // Scan - running aggregate
  println("\n5. Scan for running sum (accumulator pattern):")
  val cumsum = Stream(1, 2, 3, 4, 5).scan(0)(_ + _)
  show(cumsum.runCollect)

  // Scan for running product
  println("\n6. Scan for running product:")
  val cumprod = Stream(1, 2, 3, 4).scan(1)(_ * _)
  show(cumprod.runCollect)

  // MapAccum - accumulator + transform
  println("\n7. MapAccum for indexed transformation:")
  val indexed = Stream("a", "b", "c").mapAccum(0)((idx, x) => (idx + 1, (idx, x)))
  show(indexed.runCollect)

  // MapAccum with state structure
  println("\n8. MapAccum with complex state:")
  case class Stats(count: Int, sum: Int, max: Int)

  val stats = Stream(5, 3, 8, 2, 9).mapAccum(Stats(0, 0, Int.MinValue)) { case (s, x) =>
    (
      Stats(s.count + 1, s.sum + x, math.max(s.max, x)),
      (x, Stats(s.count + 1, s.sum + x, math.max(s.max, x)))
    )
  }
  show(stats.runCollect)

  // Combining windowing with filtering
  println("\n9. Windowing + filtering (only windows with sum > 5):")
  val filtered = Stream(1, 2, 3, 4, 5, 6)
    .sliding(3, step = 1)
    .filter(chunk => chunk.foldLeft(0)(_ + _) > 5)
  show(filtered.runCollect)

  // Chaining scan with other operations
  println("\n10. Scan + filter for conditional processing:")
  val conditional = Stream(1, 1, 2, 1, 1, 3)
    .scan(0)(_ + _)
    .filter(_ >= 3) // emit when cumsum >= 3
  show(conditional.runCollect)

  // Intersperse - useful with grouping
  println("\n11. Intersperse (insert separator between elements):")
  val separated = Stream(1, 2, 3).intersperse(0)
  show(separated.runCollect)

  // Grouped + intersperse for row formatting
  println("\n12. Grouped + intersperse for CSV-like output:")
  val rows = Stream(1, 2, 3, 4, 5, 6)
    .grouped(2)
    .map(chunk => chunk.toList.mkString(","))
    .intersperse("\n")
  show(rows.runCollect)
}
```

Run this example:

```bash
cd streams-examples && sbt "runMain stream.StreamWindowingExample"
```

### Stateful Asynchronous Operators

This example runs `takeWhileAsync`, `mapAccumAsync`, `scanAsync`, and `ensuringAsync` in a single pipeline, with every callback completing on another thread. It is JVM-only, because it ends in `.block`:

```scala title="streams-examples/src/main/scala/stream/StreamAsyncStatefulExample.scala" 
package stream

/**
 * The stateful asynchronous operators in one pipeline: `takeWhileAsync`,
 * `mapAccumAsync`, `scanAsync`, and `ensuringAsync`.
 *
 * A day's ledger feed is cut at its end-of-day marker, numbered, folded into
 * running balances, and closed by an asynchronous finalizer. Every step
 * suspends on another thread, and every step is still sequential: at most one
 * invocation of each callback is active at a time, so the accumulator is never
 * shared across concurrent work.
 *
 * JVM only: the closing `.block` parks the calling thread until the `Async`
 * completes. On Scala.js, keep the `Async` and let the host drive it.
 */
object StreamAsyncStatefulExample {
  final case class Posting(seq: Long, amount: Int)

  /** Completes on another thread, so each callback below really suspends. */
  def deferred[A](value: => A): Async[A] = {
    val completer = new Completer[A]
    val thread    = new Thread(() => completer.succeed(value))
    thread.start()
    completer
  }

  def main(args: Array[String]): Unit = {
    val sessionsClosed = new AtomicInteger

    // `0` is the end-of-day marker, not an amount. Everything after it belongs
    // to the next day and must not be posted.
    val feed: Stream[Nothing, Int] = Stream(120, -40, 75, 0, 999)

    val balances: Stream[Nothing, Long] = feed
      // Stops at the marker and closes upstream, so `999` is never pulled.
      .takeWhileAsync(amount => deferred(amount != 0))
      // Threads a sequence number through while numbering each entry.
      .mapAccumAsync(0L)((seq, amount) => deferred((seq + 1, Posting(seq + 1, amount))))
      // Emits the accumulator at each step, starting with `0L`, so the output
      // carries one more element than the input.
      .scanAsync(0L)((balance, posting) => deferred(balance + posting.amount))
      // Awaited exactly once when the materialized stream closes.
      .ensuringAsync(deferred { sessionsClosed.incrementAndGet(); () })

    val result: Either[Nothing, Chunk[Long]] = balances.runCollectAsync.block

    require(result == Right(Chunk(0L, 120L, 80L, 155L)), s"unexpected balances: $result")
    require(sessionsClosed.get() == 1, s"finalizer ran ${sessionsClosed.get()} times")

    println(s"running balances -> $result")
    println(s"sessions closed  -> ${sessionsClosed.get()}")
  }
}
```

Run this example:

```bash
cd streams-examples && sbt "runMain stream.StreamAsyncStatefulExample"
```

## Native Asynchronous Byte Readers

Each platform ships adapters that turn a native byte source into a `Reader.AsyncReader[Byte]`, which `Stream.fromReader` then lifts into a stream: `AsyncNioReaders.fromChannel` and `fromSocket` on the JVM, `ReadableStreamReaders.fromReadableStream` on Scala.js, each with an `Unmanaged` variant that leaves the native source caller-owned. [Reader](../primitives/reader.md#from-native-asynchronous-sources) documents them, together with their chunking and EOF rules and the way source failures reach the typed error channel.

## See Also

- [Asynchronous Stream Execution](../execution-and-compatibility/async-execution.md) — the full asynchronous constructor, operator, and terminal API, and how one `Stream` type describes both execution modes
- [Platform Differences](../execution-and-compatibility/platform-differences.md) — which members exist on the JVM, which exist on Scala.js, and why the blocking family is JVM-only
- [Reader](../primitives/reader.md) — the `SyncReader` / `AsyncReader` union that decides which engine materializes behind the bounded-concurrency operators, and the adapters behind the native asynchronous byte readers
- [Async Reference](../../async.md) — `Async.promise` and `Completer` bridge callback-based APIs into async values that can feed stream sources; `Async.Running` carries a synchronous cancellation handle that complements stream resource management
- [Async](../../async.md#asyncselector) — `AsyncSelector`, the primitive the bounded-concurrency slot machinery is built from
- [Coming from fs2, ZIO Streams or Ox](../../../guides/streaming-library-migration.md) — a guide that maps fs2, ZIO Streams and Ox code onto ZIO Blocks streams
- [Scope Reference](../../resource-management/scope.md) — compile-time resource safety for stream acquisition and release; `fromAcquireRelease` follows the same ownership rules as Scope-managed resources
- [Your First Streaming Pipeline](../../../guides/your-first-streaming-pipeline.md) — a step-by-step tutorial that builds a sales-log report with a `Stream`, from reading a file to recovering from a bad line
- [How to Stream Files and Sockets Without Leaking Resources](../../../guides/stream-files-and-sockets.md) — `fromAcquireRelease` and the NIO constructors applied to files and sockets, with leak checks for success, failure and early stop
- [How to Use ZIO Blocks in a ZIO or cats-effect Application](../../../guides/use-zio-blocks-in-zio-or-cats-effect.md) — calling streams from ZIO or cats-effect, with cancellation carried across

---

## Asynchronous Stream Execution

One `Stream[E, A]` describes both synchronous and asynchronous pipelines. There is no asynchronous stream type to convert to, no mode parameter to thread through your signatures, and no annotation that marks a description as one or the other. The API supports mixed synchronous and asynchronous stream composition without a second stream type or mode parameter, including dynamic inner streams and platform-specific materialization.

The terminals ending in `Async` are the cross-platform ones. They compile and run on the JVM and on Scala.js, and they are the family shared code should be written against. The blocking terminals (`run`, `runCollect`, `head`, `start`, and their siblings) still exist, but only on the JVM.

## Overview

The asynchronous surface is five things: constructors that produce a stream from an `Async`, element-level operators that take an `Async` callback, the `*Async` terminal family, one bounded-concurrency operator, and the platform adapters that turn native asynchronous I/O into a stream.

| Addition                   | Size                    | Documented in                                           |
|----------------------------|-------------------------|---------------------------------------------------------|
| Async source constructors  | 10 names / 14 overloads | [Async Source Constructors](#async-source-constructors) |
| Sequential async operators | 10 names                | [Async Operators](#async-operators)                     |
| Async terminals            | 12 names / 16 overloads | [Async Terminals](#async-terminals)                     |
| Manual-pull terminals      | 2 names                 | [Manual Pull and Ownership](#manual-pull-and-ownership) |
| Bounded concurrency        | 1 name (`mapParAsync`)  | [Bounded Concurrency](../core/stream.md#bounded-concurrency)  |
| The `Reader` union         | 2 subtypes              | [Reader](../primitives/reader.md)                                   |
| Platform I/O adapters      | JVM NIO and JS streams  | [Reader](../primitives/reader.md#from-native-asynchronous-sources)  |

Every asynchronous addition follows one naming convention: the synchronous name with `Async` appended. There is no `fromAsync` and no `asyncPush`.

## Dependency and Imports

The streams module carries the asynchronous effect type with it — `zio-blocks-streams` depends on `zio-blocks-async`, so one coordinate is all you add:

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-streams" % "0.0.56"
```

Use `%%%` in a cross-built project so the same line resolves for both JVM and Scala.js; `%%` is enough for a JVM-only build.

Every snippet on this page assumes these imports:

```scala

```

Importing `zio.blocks.async._` rather than `zio.blocks.async.Async` matters: `map`, `flatMap`, `block`, `either`, and the rest of the `Async` combinators are extension methods brought into scope by the package import.

## One Stream Type, Two Execution Modes

The central claim of this page is short: the type that decides between synchronous and asynchronous execution is `Reader`, not `Stream`.

`Stream[E, A]` is a description. Nothing in it runs until a terminal is driven, and at that moment the description is compiled into a `Reader[A]`. `Reader` is the union of a synchronous and an asynchronous kind, and that compilation is the only place the two modes part ways.

```
┌───────────────────────────────────────────────────────────────┐
│ Stream[E, A]  -  a description; nothing has run yet           │
└───────────────────────────────────────────────────────────────┘
       │  a terminal is driven
       ▼
┌───────────────────────────────────────────────────────────────┐
│ Stream.compile  -  compile the graph structurally             │
│ once, at materialization; never per element                   │
└───────────────────────────────────────────────────────────────┘
       │                            │
       │ every node compiles        │ any asynchronous node
       │ synchronously              │ is present
       │                            │
       ▼                            ▼
┌────────────────────────┐   ┌──────────────────────────────────┐
│ Reader.SyncReader[A]   │   │ Reader.AsyncReader[A]            │
│ read and close         │   │ read and close return Async;     │
│ return directly        │   │ sync stages lifted in place      │
└────────────────────────┘   └──────────────────────────────────┘
```

### Classification Happens at Compile Time

Compilation is structural and single-pass. `Stream#compile` is an abstract per-node method: each node compiles its upstream and returns a `Reader` directly, so a graph whose every node compiles synchronously yields a `SyncReader`, and a graph containing any asynchronous node yields an `AsyncReader`. Asynchronous operator nodes accept either upstream kind — an already-asynchronous upstream is extended through `AsyncInterpreter.transform`, and a synchronous one is lifted through `AsyncInterpreter.transformSync` — so a synchronous source needs no annotation to sit beneath an asynchronous stage.

Separately, and only on the JVM, a blocking terminal first tries to fuse the whole graph into the flat-array `SyncInterpreter`. Nine node types cannot be represented in that form and throw `AsyncBoundaryRequired`; the fallback then compiles the graph the ordinary way and converts the result back to a `SyncReader` at the terminal. That fusion is a performance path for blocking terminals, not the mechanism that decides between the two execution modes.

This decision is made **once, at materialization**. It is never made per element, and it is never made per pull. A stream that turns out to be entirely synchronous runs through the synchronous engine with no asynchronous machinery in the loop at all.

The same description can be materialized more than once, and each materialization classifies independently. Classification is a property of the graph, not of the value's type.

### The Reader Union

`Reader[+Elem]` is the root over two kinds:

```scala
abstract class Reader[+Elem] {
  def ++[Elem2 >: Elem](next: => Reader[Elem2]): Reader[Elem2]
  def concat[Elem2 >: Elem](next: () => Reader[Elem2]): Reader[Elem2]
  def concatAsync[Elem2 >: Elem](next: () => Async[Reader[Elem2]]): Reader.AsyncReader[Elem2]
  def withReleaseAsync(release: () => Async[Unit]): Reader.AsyncReader[Elem]
  def jvmType: JvmType
}
```

The root carries only kind-independent composition and one piece of metadata. Everything that actually pulls or closes lives on one of the two subtypes: `Reader.SyncReader[Elem]`, whose `read` and `close` return directly, and `Reader.AsyncReader[Elem]`, whose pull and lifecycle operations return `Async`.

A synchronous graph materializes as the former; a graph containing any asynchronous node materializes as the latter. See [Reader](../primitives/reader.md) for the full member list of both kinds, for how to implement a custom reader, and for `SyncReader#toAsync` and the JVM-only `AsyncReader#toSync`.

### Mixing Synchronous and Asynchronous Stages

When a synchronous source meets an asynchronous operator, the asynchronous node compiles to an `AsyncReader` and lifts its synchronous upstream through `AsyncInterpreter.transformSync`. Nothing in user code needs annotating, and no static type changes.

Composition widens. Two synchronous participants stay synchronous; a single asynchronous participant makes the result asynchronous:

```scala

val syncReader  = Reader.singleInt(1)
val asyncReader = Reader.singleInt(2).toAsync

val ss: Reader.SyncReader[Int]  = syncReader ++ Reader.singleInt(2)
val sa: Reader.AsyncReader[Int] = Reader.singleInt(1) ++ asyncReader
val as: Reader.AsyncReader[Int] = asyncReader ++ Reader.singleInt(3)
val aa: Reader.AsyncReader[Int] = asyncReader ++ Reader.singleInt(4).toAsync
```

At the stream level the same widening happens with no visible type at all. Adding one asynchronous stage to a synchronous pipeline leaves the annotation exactly as it was:

```scala

val syncOnly: Stream[Nothing, Int] =
  Stream.fromReader[Nothing, Int](Reader.fromIterable(List(1, 2, 3, 4, 5))).map(_ * 10)

val mixed: Stream[Nothing, Int] =
  syncOnly.filterAsync(i => Async.succeed(i > 20))
```

On the JVM a blocking terminal still accepts `mixed`: the asynchronous reader is converted back at the final boundary. On Scala.js, use an `*Async` terminal.

### Why Two Engines

The synchronous engine keeps its lane registers as stack locals inside a single loop. Stack locals cannot survive a suspension — the moment a callback returns a value that is not yet ready, the loop's frame has to unwind and there is nowhere for those registers to live. The asynchronous path is therefore a separate, heap-allocated engine that keeps the equivalent state in an object it can park and resume.

That is the whole reason classification exists. It is also the reason a purely synchronous stream pays nothing for the library's asynchronous support: a graph with no asynchronous node never touches the heap-allocated engine.

### There Is No Mode Annotation, and No Lane Diagnostic

Two things readers look for here, and will not find:

- **No type-level marker.** `Stream[E, A]` carries no phantom parameter, no `Sync`/`Async` tag, and no evidence that says which way a description will compile. You cannot write a signature that only accepts asynchronous streams, and you cannot ask a `Stream` value whether it will materialize asynchronously.
- **No public lane diagnostic.** A `Stream` exposes no lane of its own; `Reader#jvmType` reports the lane of a reader you already hold, which answers a narrower question than whether every fused stage preserved it. `JvmType.Infer` reports the static type, which is exactly the thing the representation machinery stopped trusting. See [Zero-Boxing Optimization](./zero-boxing.md) for what the lanes are and how one is chosen.

If you need to control the kind rather than observe it, use the union-preserving `Stream.fromReader` overloads below: they let you hand a specific reader kind to the stream.

## Async Source Constructors

These are the companion constructors that turn an `Async` into a stream. Each of them defers its thunk until the first reader operation is driven.

### `Stream.attemptAsync`

```scala
def attemptAsync[A](f: => Async[A])(implicit jtA: JvmType.Infer[A]): Stream[Throwable, A]
```

Lazily evaluates an asynchronous thunk once per materialization and emits its result. Non-fatal synchronous throws and asynchronous failures become typed errors; fatal throwables remain defects.

### `Stream.attemptEvalAsync`

```scala
def attemptEvalAsync(f: => Async[Any]): Stream[Throwable, Nothing]
```

Lazily executes an asynchronous effect once per materialization and emits nothing. Non-fatal synchronous throws and asynchronous failures become typed errors; fatal throwables remain defects. Use it for an effect whose result you do not want in the stream.

### `Stream.deferAsync`

```scala
def deferAsync(finalizer: => Async[Unit]): Stream[Nothing, Nothing]
```

Creates an empty stream that lazily registers an asynchronous release action. It is awaited exactly once when each materialization closes, including after failure, early termination, or cancellation; failure is a defect.

### `Stream.evalAsync`

```scala
def evalAsync(f: => Async[Any]): Stream[Nothing, Nothing]
```

Lazily executes an asynchronous effect once per materialization and emits nothing; synchronous throws and asynchronous failures are defects. This is `attemptEvalAsync` without the typed error channel.

### `Stream.fromAcquireReleaseAsync`

```scala
def fromAcquireReleaseAsync[R, E, A](
  acquire: => Async[R],
  release: R => Async[Unit]
)(use: R => Stream[E, A])(implicit jtA: JvmType.Infer[A]): Stream[E, A]
```

Lazily acquires one resource per materialization, constructs the stream with `use`, and awaits release exactly once after completion, failure, early termination, or cancellation — including cancellation during acquisition, once the resource is obtained. Acquisition, `use`, and release failures are defects.

### `Stream.fromIteratorAsync`

```scala
def fromIteratorAsync[A](it: => Async[Iterator[A]])(implicit jtA: JvmType.Infer[A]): Stream[Nothing, A]
```

Asynchronously obtains one iterator per materialization and consumes it in order. Acquisition and iterator failures are defects.

### `Stream.fromReaderAsync`

```scala
def fromReaderAsync[E, A](mkReader: => Async[Reader[A]])(implicit jtA: JvmType.Infer[A]): Stream[E, A]
```

Lazily runs `mkReader` once per materialization and closes the resulting reader when the stream closes. Effect and thunk failures are defects.

### `Stream.unfoldAsync`

```scala
def unfoldAsync[S, A](s: S)(f: S => Async[Option[(A, S)]])(implicit jtA: JvmType.Infer[A]): Stream[Nothing, A]
```

Lazily unfolds state sequentially, emitting each `A` and continuing with the paired state until `f` returns `None`. At most one callback is active at a time; callback failure is a defect.

### `Stream.unwrap`

```scala
def unwrap[E](stream: => Async[Stream[E, Nothing]])(implicit dummy: DummyImplicit): Stream[E, Nothing]
def unwrap[E, A](stream: => Async[Stream[E, A]])(implicit jtA: JvmType.Infer[A]): Stream[E, A]
```

Flattens an asynchronously produced stream. The effect is evaluated lazily once per materialization; effect failures are defects, while the produced stream retains its typed error channel. The `DummyImplicit` overload exists to preserve inference when the element type is `Nothing`.

`unwrap` is the idiom for feeding an asynchronously produced stream into an operator that has no asynchronous twin. `flatMap`, `catchAll`, `catchDefect`, `orElse`, and `flatMapPar` all compose with it unchanged:

```scala

val stream: Stream[Nothing, Int] = Stream(1, 2, 3)

val flatMapped: Stream[Nothing, Long] =
  stream.flatMap(i => Stream.unwrap(Async.succeed(Stream(i.toLong))))

val recovered: Stream[String, Int] =
  Stream.fail[String]("failure").catchAll(_ => Stream.unwrap(Async.succeed(stream)))
```

### `Stream.fromReader`

Four overloads dispatch on the kind of reader you hand them, so the kind you chose is the kind the stream materializes:

```scala
def fromReader[E, A](mkReader: => Reader[A]): Stream[E, A]

def fromReader[E, A](mkReader: => Reader.SyncReader[A])(implicit
  dummy1: DummyImplicit,
  dummy2: DummyImplicit
): Stream[E, A]

def fromReader[E, A](mkReader: => Nothing)(implicit
  dummy1: DummyImplicit,
  dummy2: DummyImplicit,
  dummy3: DummyImplicit
): Stream[E, A]

def fromReader[E, A](mkReader: => Reader.AsyncReader[A])(implicit dummy: DummyImplicit): Stream[E, A]
```

Each overload lazily obtains one reader per materialization and closes it when the stream closes; reader-thunk failures are defects. The `Reader[A]` overload is documented as advanced: it is the union-preserving escape hatch, and it preserves whether the returned reader is synchronous or asynchronous. The `AsyncReader` overload awaits the reader's close.

### Laziness and Error Conventions

Two rules govern this whole family, and they are worth stating on their own because they are the two things most often assumed backwards.

**Laziness.** Compilation and materialization remain synchronous. Closing a stream before initialization neither invokes the thunk nor acquires a resource — an asynchronous constructor's effect starts only when the stream is first pulled. Do not compensate by eagerly opening a resource before constructing the stream.

**Errors.** Only the two `attempt*` constructors — `attemptAsync` and `attemptEvalAsync` — convert non-fatal callback failures into typed `Throwable` errors. Every other constructor's callback failure remains a defect, which fails the outer `Async` rather than appearing as a `Left`.

:::warning[A defect is not a typed error]
A defect does not surface in the `Either` that a terminal returns. It fails the surrounding `Async`, so a `match` on `Left`/`Right` will never see it. Reach for `attemptAsync` when you want a callback's failure in the `Left` channel.
:::

## Async Operators

These are the sequential, element-level twins of the synchronous operators. Each applies its callback to one element at a time, in order, with at most one invocation active.

### `Stream#mapAsync`

```scala
def mapAsync[B](f: A => Async[B])(implicit jtB: JvmType.Infer[B]): Stream[E, B]
```

Asynchronously transforms each element. Synchronous twin: [`map`](../core/stream.md#streammapb).

### `Stream#mapErrorAsync`

```scala
def mapErrorAsync[E2](f: E => Async[E2]): Stream[E2, A]
```

Asynchronously transforms the typed error channel. It runs only when the source fails with a typed error; callback failure is a defect. It genuinely changes the error type, so a `Stream[String, A]` can become a `Stream[Long, A]`. Synchronous twin: [`mapError`](../core/stream.md#streammaperrore2).

### `Stream#filterAsync`

```scala
def filterAsync(pred: A => Async[Boolean]): Stream[E, A]
```

Tests elements sequentially and emits those satisfying the asynchronous predicate, preserving order. Predicate failure is a defect. Synchronous twin: [`filter`](../core/stream.md#streamfilter).

### `Stream#collectAsync`

```scala
def collectAsync[B](f: A => Async[Option[B]])(implicit jtB: JvmType.Infer[B]): Stream[E, B]
```

Asynchronously transforms defined elements, dropping `None` results. Synchronous twin: [`collect`](../core/stream.md#streamcollectb).

### `Stream#mapAccumAsync`

```scala
def mapAccumAsync[S, B](init: S)(f: (S, A) => Async[(S, B)])(implicit jtB: JvmType.Infer[B]): Stream[E, B]
```

Asynchronously transforms elements while threading state sequentially. At most one invocation of `f` is active at a time. Synchronous twin: [`mapAccum`](../core/stream.md#stateful-transformations).

### `Stream#scanAsync`

```scala
def scanAsync[S](init: S)(f: (S, A) => Async[S])(implicit jtS: JvmType.Infer[S]): Stream[E, S]
```

Asynchronously emits the accumulator at each step, starting with `init`. The output stream has one more element than the input. Synchronous twin: [`scan`](../core/stream.md#stateful-transformations).

### `Stream#takeWhileAsync`

```scala
def takeWhileAsync(pred: A => Async[Boolean]): Stream[E, A]
```

Tests elements sequentially and emits them while the asynchronous predicate holds, then closes upstream at the first `false`. Predicate failure is a defect. Synchronous twin: [`takeWhile`](../core/stream.md#skipping-and-taking).

### `Stream#distinctByAsync`

```scala
def distinctByAsync[K](f: A => Async[K]): Stream[E, A]
```

Sequentially computes keys and emits the first element for each key, preserving order. Key state is per materialization and may grow without bound; asynchronous failures are defects. Synchronous twin: [`distinctBy`](../core/stream.md#streamdistinctbyk).

### `Stream#tapEachAsync`

```scala
def tapEachAsync(f: A => Async[Unit]): Stream[E, A]
```

Runs an asynchronous effect for each element and passes it through. Synchronous twin: [`tapEach`](../core/stream.md#other-operations).

### `Stream#ensuringAsync`

```scala
def ensuringAsync(finalizer: => Async[Unit]): Stream[E, A]
```

Registers an asynchronous finalizer lazily and awaits it exactly once when the materialized stream closes, including normal completion, failure, early termination, and cancellation. Finalizer failure is a defect. Synchronous twin: [`ensuring`](../core/stream.md#streamensuring).

For the one *concurrent* asynchronous operator, `mapParAsync`, see [Bounded Concurrency](../core/stream.md#bounded-concurrency).

## Async Terminals

A terminal is what drives a stream. The cross-platform family all ends in `Async` and all shares one return shape.

### The `Async[Either[E, Z]]` Convention

Every cross-platform terminal returns `Async[Either[E, Z]]`, never `Async[Z]`. The two channels are kept apart deliberately:

- The **typed error channel** `E` stays inside the `Either`. A stream that fails with a typed error still succeeds at the `Async` level: the `Async` completes normally, carrying `Left(e)`.
- `Async`'s own **untyped `Throwable` channel** is reserved for defects — a callback that threw, a finalizer that failed, a cleanup failure. These fail the outer `Async` and never appear as a `Left`.

This one rule explains the shape of every signature in this section:

```scala

val readings: Stream[String, Int] = Stream(12, 7, 30)

val collected: Async[Either[String, Chunk[Int]]] = readings.runCollectAsync

val described: Async[String] = collected.map(result =>
  result match {
    case Right(values) => s"collected ${values.length} readings"
    case Left(error)   => s"typed error: $error"
  }
)
```

### Collecting and Running

```scala
def runAsync[ES, E3, Z](sink: Sink[ES, A, Z])(implicit
  errorConcat: Concat.WithOut[E @uncheckedVariance, ES, E3]
): Async[Either[E3, Z]]

def runCollectAsync: Async[Either[E, Chunk[A]]]

def runDrainAsync: Async[Either[E, Unit]]

def runForeachAsync(f: A => Async[Unit]): Async[Either[E, Unit]]
```

`runAsync` is the general form: it runs the stream through the asynchronous drain of a `Sink`, and materialization and cleanup are lazy, cancellation-safe, and performed exactly once. Its error type is the concatenation of the stream's error type and the sink's, which is why the implicit `Concat` evidence appears.

`runCollectAsync` collects all elements in order. It requires memory proportional to the entire output and does not terminate for an infinite stream. `runDrainAsync` discards them. `runForeachAsync` applies an asynchronous callback to each element sequentially.

### Folding

`runFoldAsync` is a five-member family: four primitive accumulator lanes and one generic.

| Accumulator | Signature                                                  | Blocking `runFold` twin |
|-------------|------------------------------------------------------------|-------------------------|
| `Double`    | `runFoldAsync(z: Double)(f: (Double, A) => Async[Double])` | yes                     |
| `Float`     | `runFoldAsync(z: Float)(f: (Float, A) => Async[Float])`    | none                    |
| `Int`       | `runFoldAsync(z: Int)(f: (Int, A) => Async[Int])`          | yes                     |
| `Long`      | `runFoldAsync(z: Long)(f: (Long, A) => Async[Long])`       | yes                     |
| generic `Z` | `runFoldAsync[Z](z: Z)(f: (Z, A) => Async[Z])`             | yes                     |

The generic overload takes an implicit `JvmType.Infer[Z]`; the four primitive ones do not, and the overload is selected by the static type of `z`. Write `0L` rather than `0` when you want the `Long` lane.

The `Float` lane has no counterpart in the blocking `runFold` family, which offers only `Double`, `Int`, `Long`, and generic. It is new with the asynchronous terminals.

Each fold callback is applied sequentially, one element at a time.

### Queries

The query terminals are one-liners over `runAsync`. Knowing which `Sink` each delegates to tells you its semantics exactly:

| Terminal            | Returns                       | Delegates to             |
|---------------------|-------------------------------|--------------------------|
| `countAsync`        | `Async[Either[E, Long]]`      | `Sink.count`             |
| `existsAsync(pred)` | `Async[Either[E, Boolean]]`   | `Sink.existsAsync(pred)` |
| `findAsync(pred)`   | `Async[Either[E, Option[A]]]` | `Sink.findAsync(pred)`   |
| `forallAsync(pred)` | `Async[Either[E, Boolean]]`   | `Sink.forallAsync(pred)` |
| `foreachAsync(f)`   | `Async[Either[E, Unit]]`      | `runForeachAsync(f)`     |
| `headAsync`         | `Async[Either[E, Option[A]]]` | `Sink.head`              |
| `lastAsync`         | `Async[Either[E, Option[A]]]` | `Sink.last`              |

`existsAsync`, `findAsync`, and `forallAsync` take an `A => Async[Boolean]` predicate; `foreachAsync` is an alias for `runForeachAsync`. `countAsync`, `headAsync`, and `lastAsync` take no callback and therefore reuse the ordinary callback-free sinks, which drain a synchronous or an asynchronous reader alike.

### Blocking Twins Are JVM-only

Thirteen blocking members — `count`, `exists`, `find`, `forall`, `foreach`, `head`, `last`, `run`, `runCollect`, `runDrain`, `runFold`, `runForeach`, and `start` — live on the JVM only. Shared, cross-compiled sources cannot call them; they must use the `*Async` family, `startAsync`, and `useReaderAsync` instead.

For the full platform matrix, including which reader conversions and sink constructors exist on which platform, see [Platform Differences](./platform-differences.md).

### Driving an `Async` From a JVM `main`

An `Async[Either[E, Z]]` is a value. Something has to drive it, and on the JVM that something is `.block`:

```scala

val stream: Stream[String, Int] = Stream(1, 2, 3)

// At the edge of the world, and on the JVM only:
val result: Either[String, Chunk[Int]] = stream.runCollectAsync.block
```

`.block` drives the effect to its value, parking the calling thread until it is ready; a failure is re-thrown as its cause. A ready value returns immediately without parking.

This is the edge-of-the-world idiom, and it is JVM-only. Two rules keep it honest:

1. **`.block` belongs in `main`, or in a test, and nowhere else.** Never call it inside a stream callback or inside a `poll` — blocking the driver from within the loop it is driving deadlocks it.
2. **Scala.js code must not use it at all.** JavaScript cannot block, so unless the effect has already completed synchronously, `.block` throws an `IllegalStateException` there. Cross-platform code should keep the `Async` and hand it to the host: convert it at the boundary (for example with `toFuture`) and let the runtime drive it.

Inside an `Async.async { ... }` block, use the direct-style `.await` instead, which extracts the value without blocking. See [Async](../../async.md) for both.

### A Downstream Adopter: `Body`

`Body` in `http-model` is the clearest in-repo illustration of what the `*Async` convention looks like for a cross-platform consumer, because a body is exactly a `Stream[Nothing, Byte]` that someone eventually wants as bytes or as text.

Each blocking accessor has an asynchronous twin under the library-wide naming convention, five in all: `Body#toChunkAsync`, `Body#toArrayAsync`, `Body#asStringAsync`, `Body#asStringFromContentTypeAsync`, and `Body#textAsync`. The twins are the cross-platform API. The original accessors block, so they compile on Scala.js but throw `IllegalStateException` the moment the stream actually has to suspend — see [Why Blocking Terminals Are JVM-Only](./platform-differences.md#why-blocking-terminals-are-jvm-only). `Body#toChunk` is implemented in terms of the asynchronous one, taking a known-chunk fast path first and otherwise running `runCollectAsync` and blocking on the result.

Writing a cross-platform call site is the rename plus a change of result type:

```scala

// Blocking: works on the JVM; on Scala.js this throws once the stream suspends
def bytesBlocking(body: Body): Chunk[Byte] = body.toChunk

// JVM and Scala.js
def bytes(body: Body): Async[Chunk[Byte]] = body.toChunkAsync
def text(body: Body): Async[String]       = body.textAsync
```

[Body](../../http-model/model.md#body) documents the type itself, its constructors, and the rest of its accessors.

## Manual Pull and Ownership

Sometimes you want the reader rather than a result — to interleave pulls with other work, or to hand the source to a protocol loop. Two terminals give you one, and they differ in exactly one respect: who is responsible for closing it.

### `Stream#startAsync`

```scala
def startAsync: Async[Reader.AsyncReader[A]]
```

Materializes this stream as a caller-owned asynchronous reader. **Ownership transfers to the caller**, who must drive the reader and await `close()`. This is the one place in the API where the library does not close what it opened — if you forget the `close()`, finalizers registered by `ensuringAsync`, `deferAsync`, and `fromAcquireReleaseAsync` never run.

### `Stream#useReaderAsync`

```scala
def useReaderAsync[Z](f: Reader.AsyncReader[A] => Async[Z]): Async[Z]
```

The scoped alternative. Ownership is **retained** by the library: the reader is passed to `f`, and its close is awaited on every outcome — success, typed failure, defect, and cancellation alike. Prefer it whenever the reader's lifetime is bounded by a single block of code.

Note the return type: `Async[Z]`, not `Async[Either[E, Z]]`. `useReaderAsync` hands you the reader, so whatever `f` produces is what you get back; stream errors surface through the reader's own pulls.

### One Active Operation Per Reader

An `AsyncReader` is a single-consumer cursor, not a concurrent work queue. **At most one operation may be in flight at a time**: await the `Async` returned by a `read`, `readAll`, `skip`, or `close` before beginning the next one.

Readers are not thread-safe either. Overlapping pulls, or driving one reader from two threads without external synchronization, is outside the contract — the reader's internal position and lifecycle state are not defended against it, and the result is not specified.

### Cleanup Failures

When cleanup fails on a path that has already failed, the cleanup failure is **attached to** the primary failure rather than replacing it. The original cause is what propagates; the cleanup cause is recorded as a suppressed exception on it.

This means a `Throwable` that reaches you from a failed `Async` may carry more than one story. Inspect `getSuppressed` before concluding that a close error was the only thing that went wrong.

## Cancellation

Cancellation in this library is **cooperative, never preemptive**. Cancelling signals the in-flight operation; it does not interrupt a thread, and it never waits for an in-flight `poll` to return.

Two pieces of the API matter here:

- **`Pollable#cancel()`** signals cancellation of the currently pending operation, and reaches the active leaf operation rather than stopping at the driver. Implementations that own cancellable work override `cancel()` with an idempotent, non-blocking signal; implementations without such work inherit the no-op. A running driver invokes it only when cancellation wins the race against completion.
- **`Async.Running#cancel(onCleanupFailure: Throwable => Unit)`** cancels a run and reports a failure from its asynchronous cleanup. The reporter is retained only when this cancellation wins completion, and it is invoked at most once. Use it when a cleanup failure during cancellation must not be lost.

Cancellation closes an acquired reader and awaits its finalizer. `startAsync` is the deliberate exception, because it has already transferred that responsibility to its caller.

See [Async](../../async.md) for `Pollable`, `Cancelable`, and `Async.Running` themselves.

## Resource Management

Two members carry resources through an asynchronous stream, and they compose with the ownership rules above.

`Stream.fromAcquireReleaseAsync` brackets a resource around a stream: one acquisition per materialization, and release awaited exactly once after completion, failure, early termination, or cancellation — including cancellation that arrives during acquisition, once the resource is obtained.

`Stream#ensuringAsync` registers a finalizer without a resource: awaited exactly once when the materialized stream closes, on every outcome.

Both are driven by the *close* of the materialized reader, which is what ties them to ownership:

- Under `runCollectAsync` and every other terminal, the library closes the reader, so both run without your involvement.
- Under `useReaderAsync`, the library still closes the reader, so both still run — on success and on failure alike.
- Under `startAsync`, **you** close the reader. Until you await `close()`, neither the release action nor the finalizer has run.

A failure inside a finalizer is a defect, and if the stream had already failed, that defect is attached to the primary failure rather than replacing it.

## How This Is Verified

The asynchronous execution path is covered by three-way differential equivalence: for each generated program, the *ready* execution, the *suspended* execution, and an independent reference model must agree on the result and on the materialized reader kind. The generated campaign is frozen at a fixed seed, and a separate sweep runs every physical lane against every logical terminal. A production run records only demand and callback counts; failure provenance, throwable order, ownership transitions, outstanding resources, epoch, and suppressed exceptions are computed by the reference model, and a hand-written scenario asserts them against it — that scenario is also the only one carrying a non-empty fault script, injecting a throw and a late close.

:::note[On allocation figures]
Near-zero allocation numbers observed for this path are profiler noise, not a promise. An allocation profiler is required before claiming that any particular stream program allocates nothing.
:::

## Running the Examples

Every example below is a runnable file in the `streams-examples` module. Clone the repository and run them with sbt:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

### Async Terminals and `.block`

Three terminals on one description, a fourth on a stream that fails with a typed error, the `Either` unwrapped on both branches, and `.block` confined to the edge of `main`:

```scala title="streams-examples/src/main/scala/stream/StreamAsyncTerminalsExample.scala" 
package stream

/**
 * The cross-platform asynchronous terminal family.
 *
 * Every `*Async` terminal returns `Async[Either[E, Z]]`: the typed error
 * channel stays in the `Either`, and `Async`'s own `Throwable` channel is
 * reserved for defects. `.block` drives the `Async` to its value and belongs
 * only here, at the edge of a JVM `main`.
 */
object StreamAsyncTerminalsExample {
  final case class Reading(sensor: String, celsius: Int)

  def main(args: Array[String]): Unit = {
    val readings: Stream[String, Reading] =
      Stream(Reading("north", 12), Reading("south", 7), Reading("east", 30))

    // One description, three terminals. Nothing has run yet.
    val collected: Async[Either[String, Chunk[Reading]]] = readings.runCollectAsync
    val total: Async[Either[String, Long]]               =
      readings.runFoldAsync(0L)((sum, reading) => Async.succeed(sum + reading.celsius))
    val first: Async[Either[String, Option[Reading]]] = readings.headAsync

    // A stream that fails with a typed error surfaces it as `Left`, not as a
    // failure of the outer `Async`.
    val offline: Stream[String, Reading]              = Stream.fail("west sensor is offline")
    val failed: Async[Either[String, Chunk[Reading]]] = offline.runCollectAsync

    // `.block` parks the calling thread until the value is ready. It is JVM
    // only: on Scala.js it throws, so keep the `Async` and let the host drive it.
    report("runCollectAsync", collected.block.map(_.length))
    report("runFoldAsync", total.block)
    report("headAsync", first.block.map(_.map(_.sensor)))
    report("runCollectAsync (failing)", failed.block.map(_.length))
  }

  private def report[Z](label: String, result: Either[String, Z]): Unit =
    result match {
      case Right(value) => println(s"$label -> $value")
      case Left(error)  => println(s"$label -> typed error: $error")
    }
}
```

Run it with:

```bash
cd streams-examples && sbt "runMain stream.StreamAsyncTerminalsExample"
```

### Ownership: `startAsync` Versus `useReaderAsync`

A finalizer that counts its own runs, proving that `useReaderAsync` closes on success *and* on failure, while `startAsync` closes only because the caller does it:

```scala title="streams-examples/src/main/scala/stream/StreamAsyncOwnershipExample.scala" 
package stream

/**
 * Manual pull and close ownership.
 *
 * `startAsync` hands the reader to the caller, who must drive it and await
 * `close()`. `useReaderAsync` keeps ownership and awaits `close()` on every
 * outcome. An observable finalizer makes the difference visible.
 */
object StreamAsyncOwnershipExample {
  def main(args: Array[String]): Unit = {
    // startAsync: ownership transfers. Forget the `close()` and the finalizer
    // never runs.
    val startFinalized                  = new AtomicInteger
    val reader: Reader.AsyncReader[Int] = source(startFinalized).startAsync.block
    val started: Chunk[Int]             =
      try reader.readAll[Int]().block
      finally reader.close().block
    println(s"startAsync              -> $started, finalizer ran ${startFinalized.get()} time(s)")

    // useReaderAsync on success: ownership is retained, close is awaited.
    val useFinalized     = new AtomicInteger
    val used: Chunk[Int] = source(useFinalized).useReaderAsync(r => r.readAll[Int]()).block
    println(s"useReaderAsync          -> $used, finalizer ran ${useFinalized.get()} time(s)")

    // useReaderAsync on failure: close is awaited just the same.
    val failFinalized                         = new AtomicInteger
    val failed: Either[Throwable, Chunk[Int]] =
      source(failFinalized)
        .useReaderAsync[Chunk[Int]](_ => Async.fail(new RuntimeException("consumer gave up")))
        .either
        .block
    val message = failed.left.map(_.getMessage)
    println(s"useReaderAsync (failing) -> $message, finalizer ran ${failFinalized.get()} time(s)")
  }

  /** A stream carrying an asynchronous finalizer that counts its own runs. */
  private def source(finalized: AtomicInteger): Stream[Nothing, Int] =
    Stream(1, 2, 3).ensuringAsync(Async.succeed {
      finalized.incrementAndGet()
      ()
    })
}
```

Run it with:

```bash
cd streams-examples && sbt "runMain stream.StreamAsyncOwnershipExample"
```

### A Mixed Synchronous and Asynchronous Pipeline

A synchronous source, two asynchronous operators, and no change to any annotation:

```scala title="streams-examples/src/main/scala/stream/StreamMixedKindExample.scala" 
package stream

/**
 * Mixing a synchronous source with an asynchronous operator.
 *
 * Adding an asynchronous stage changes no static type and requires no
 * annotation. The description is still `Stream[Nothing, Int]`; what changes is
 * the reader it compiles to, and that happens once, at materialization.
 */
object StreamMixedKindExample {
  def main(args: Array[String]): Unit = {
    // Entirely synchronous: a synchronous reader plus a synchronous callback.
    val syncOnly: Stream[Nothing, Int] =
      Stream.fromReader[Nothing, Int](Reader.fromIterable(List(1, 2, 3, 4, 5))).map(_ * 10)

    // The same source with one asynchronous stage appended. Note that the
    // annotation on the left is identical.
    val mixed: Stream[Nothing, Int] =
      syncOnly.filterAsync(i => Async.succeed(i > 20)).mapAsync(i => Async.succeed(i + 1))

    println(s"syncOnly -> ${syncOnly.runCollectAsync.block}")
    println(s"mixed    -> ${mixed.runCollectAsync.block}")

    // The JVM blocking terminal accepts the mixed graph too: the asynchronous
    // reader is converted back at the final boundary.
    println(s"mixed (blocking terminal) -> ${mixed.runCollect}")
  }
}
```

Run it with:

```bash
cd streams-examples && sbt "runMain stream.StreamMixedKindExample"
```

### A Composed Asynchronous Pipeline

Real suspension through a `Completer`, `Stream.unwrap` feeding `filterAsync`, `mapAsync`, and `ensuringAsync`, 33,000 nested stages to demonstrate that the asynchronous path is stack-safe, and an assertion that the finalizer runs exactly once:

```scala title="streams-examples/src/main/scala/stream/StreamAsyncOrderPipelineExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 */
package stream

/** A composed asynchronous order-validation and audit pipeline. */
object StreamAsyncOrderPipelineExample {
  final case class Order(id: Int, amount: Int)

  def deferred[A](value: => A): Async[A] = {
    val result = new Completer[A]
    val thread = new Thread(() => result.succeed(value))
    thread.start()
    result
  }

  def main(args: Array[String]): Unit = {
    val finalized                         = new AtomicInteger
    val validated: Stream[Nothing, Order] = Stream
      .unwrap(deferred(Stream(Order(1, 20), Order(2, -1), Order(3, 40))))
      .filterAsync(order => Async.succeed(order.amount > 0))
      .mapAsync(order => deferred(order.copy(amount = order.amount + 5)))
      .ensuringAsync(Async.succeed { finalized.incrementAndGet(); () })

    // Real applications commonly assemble reusable generated middleware. Its
    // finite depth must not change the meaning of an identity transformation.
    val withMiddleware =
      (0 until 33000).foldLeft(validated)((orders, _) => orders.map(identity))

    val result = withMiddleware.runCollectAsync.block
    require(result == Right(Chunk(Order(1, 25), Order(3, 45))), s"unexpected result: $result")
    require(finalized.get() == 1, s"finalizer ran ${finalized.get()} times")
    println(result)
  }
}
```

Run it with:

```bash
cd streams-examples && sbt "runMain stream.StreamAsyncOrderPipelineExample"
```

## See Also

- [Reader](../primitives/reader.md) — the `SyncReader` / `AsyncReader` union, custom reader implementations, mixed-kind composition, and the JVM NIO and Scala.js `ReadableStream` adapters
- [Bounded Concurrency](../core/stream.md#bounded-concurrency) — `mapPar`, `mapParAsync`, `mergeAll`, and `flatMapPar`
- [Platform Differences](./platform-differences.md) — what exists on the JVM, what exists on Scala.js, and what throws
- [Async](../../async.md) — `Async[A]`, `Pollable`, `Completer`, `Async.Running`, and cancellation
- [How to Run Streams Without Blocking](../../../guides/run-streams-without-blocking.md) — swapping blocking terminals for their `Async` twins, with a Scala.js run and a `ReadableStream` source
- [Zero-Boxing Optimization](./zero-boxing.md) — primitive lanes, and why async is lane-aware rather than end-to-end allocation-free

---

## Execution and Compatibility

[Core Types](../core/index.md) covers the descriptions you write and [Low-Level Primitives](../primitives/index.md) the machinery they compile into. This section covers neither: four concerns that cut across every `Stream`, `Pipeline`, and `Sink` once one starts running. None adds a type to compose with.

## Asynchronous Stream Execution

[Asynchronous Stream Execution](./async-execution.md) — how one `Stream[E, A]` describes both synchronous and asynchronous pipelines, with no second type or mode parameter to track. Covers the async constructors, operators, and `*Async` terminals, plus cancellation and resource ownership.

## Zero-Boxing Optimization

[Zero-Boxing Optimization](./zero-boxing.md) — how the library dispatches on a primitive element type to select a physical lane, and how each lane signals end of stream. A structural account of the dispatch mechanism, not a benchmark.

## Platform Differences: JVM and Scala.js

[Platform Differences](./platform-differences.md) — which members are JVM-only, which are cross-platform, and what to write instead when a blocking terminal is unavailable on Scala.js. Read it before cross-building.

## Scala 2 Compatibility Design Note

[Scala 2 Compatibility Design Note](./scala-2-compatibility.md) — why streams supports Scala 2.13, the hot-path constraint that shaped the approach, and why the two versions share sources rather than splitting them.

## See Also

- [Streams Reference](../index.md) — module overview
- [Core Types](../core/index.md) — `Stream`, `Pipeline`, and `Sink`
- [Low-Level Primitives](../primitives/index.md) — `Reader` and `Writer`

---

## Platform Differences: JVM and Scala.js

The terminals whose names end in `Async` are the cross-platform API: they compile and run on the JVM and on Scala.js alike. The blocking terminals — `Stream#run`, `Stream#runCollect`, `Stream#start` and their siblings — exist only on the JVM. Code that cross-builds is written against the `*Async` family, and there is no configuration flag, shim, or runtime fallback that changes that.

The split is a compile-time one. A Scala.js source that calls `Stream#runCollect` does not fail when it runs; it fails to compile, because that member does not exist on that platform.

## Availability Matrix

Nineteen members behave differently across the two platforms. The JVM column gives what the member evaluates to there, the Scala.js column gives what happens instead, and the last column names what to write in shared source:

| Member                      | JVM result                      | Scala.js                     | Cross-platform replacement                   |
|-----------------------------|---------------------------------|------------------------------|----------------------------------------------|
| `Stream#count`              | `Either[E, Long]`               | Absent — compile error       | `Stream#countAsync`                          |
| `Stream#exists`             | `Either[E, Boolean]`            | Absent — compile error       | `Stream#existsAsync`                         |
| `Stream#find`               | `Either[E, Option[A]]`          | Absent — compile error       | `Stream#findAsync`                           |
| `Stream#forall`             | `Either[E, Boolean]`            | Absent — compile error       | `Stream#forallAsync`                         |
| `Stream#foreach`            | `Either[E, Unit]`               | Absent — compile error       | `Stream#foreachAsync`                        |
| `Stream#head`               | `Either[E, Option[A]]`          | Absent — compile error       | `Stream#headAsync`                           |
| `Stream#last`               | `Either[E, Option[A]]`          | Absent — compile error       | `Stream#lastAsync`                           |
| `Stream#run`                | `Either[E3, Z]`                 | Absent — compile error       | `Stream#runAsync`                            |
| `Stream#runCollect`         | `Either[E, Chunk[A]]`           | Absent — compile error       | `Stream#runCollectAsync`                     |
| `Stream#runDrain`           | `Either[E, Unit]`               | Absent — compile error       | `Stream#runDrainAsync`                       |
| `Stream#runFold(z: Double)` | `Either[E, Double]`             | Absent — compile error       | `Stream#runFoldAsync(z: Double)`             |
| `Stream#runFold(z: Int)`    | `Either[E, Int]`                | Absent — compile error       | `Stream#runFoldAsync(z: Int)`                |
| `Stream#runFold(z: Long)`   | `Either[E, Long]`               | Absent — compile error       | `Stream#runFoldAsync(z: Long)`               |
| `Stream#runFold(z: Z)`      | `Either[E, Z]`                  | Absent — compile error       | `Stream#runFoldAsync(z: Z)`                  |
| `Stream#runForeach`         | `Either[E, Unit]`               | Absent — compile error       | `Stream#runForeachAsync`                     |
| `Stream#start`              | `scope.$[Reader.SyncReader[A]]` | Absent — compile error       | `Stream#startAsync`, `Stream#useReaderAsync` |
| `Sink.create`               | `Sink[E, A, Z]`                 | Absent — compile error       | `Sink.createAsync`, `Sink.createBoth`        |
| `Reader.AsyncReader#toSync` | `Reader.SyncReader[A]`          | Absent — compile error       | None; drive the `AsyncReader` itself         |
| `Async#block`               | `A`, parking the thread         | Present — throws at run time | `Async#map`, `Async#flatMap`, `Async.start`  |

Three details in that table repay a second look. The replacements that take a callback take an *effectful* one: `Stream#exists` takes `A => Boolean` while `Stream#existsAsync` takes `A => Async[Boolean]`, and the same shift applies to `Stream#findAsync`, `Stream#forallAsync`, `Stream#foreachAsync`, `Stream#runForeachAsync`, and every `Stream#runFoldAsync` overload. The asynchronous fold family is also one overload wider than the blocking one — `Stream#runFoldAsync(z: Float)` has no blocking twin, so the `Float` accumulator lane is reachable only through the cross-platform API. And `Async#block` is the single row whose Scala.js cell says "run time" rather than "compile error", because it is declared in shared source and cannot be removed from the platform it cannot serve.

Porting a blocking call is a rename plus a change of result type, since every replacement wraps its answer in `Async`:

```scala

// JVM only: Either[Nothing, Chunk[Int]]
val onTheJvm = Stream(1, 2, 3).map(_ * 2).runCollect

// JVM and Scala.js: Async[Either[Nothing, Chunk[Int]]]
val everywhere = Stream(1, 2, 3).map(_ * 2).runCollectAsync
```

[Asynchronous Stream Execution](./async-execution.md) documents the full `*Async` surface, including the [`Async[Either[E, Z]]` convention](./async-execution.md#the-asynceithere-z-convention) that the second line above returns.

## Why Blocking Terminals Are JVM-Only

A blocking terminal makes one promise: when it returns, the pipeline has finished and the answer is in hand as an `Either`. On the JVM that promise is kept by parking the calling thread until the pipeline completes, and some other thread does the completing.

JavaScript has no other thread. Its single execution thread is also the thread that would have to run the callbacks that deliver the result, so a terminal that waited for completion would prevent the completion it waits for. There is no implementation of `Stream#runCollect` on Scala.js that both blocks and terminates.

That leaves two ways to express the constraint. The blocking terminals could stay in shared source and throw on Scala.js, or they could be removed from the Scala.js API surface entirely. The library removed them. The cost is that a cross-building source cannot always be compiled unchanged; the benefit is that the compiler reports it, at the call site, before anything ships.

`Async#block` is the one member that had to take the other route, because it is an extension method in shared `async` source with no stream terminal to hide behind. On Scala.js it returns normally when the effect has already completed, and throws `IllegalStateException` the moment a suspension actually has to wait — the message tells the caller to drive the `Pollable` from a non-blocking entry point instead.

## `StreamPlatformSpecific`

The sixteen relocated `Stream` members are not scattered through the platform trees. They live in one trait per platform, `StreamPlatformSpecific[+E, +A]`, which `Stream[E, A]` mixes in under a self-type:

```
┌────────────────────────────────────────────────────────────────┐
│ streams/shared   one Stream[E, A], one Sink, one Reader        │
│ every *Async terminal lives here and compiles on both          │
└────────────────────────────────┬───────────────────────────────┘
                                 │ mixed into Stream by self-type
                ┌────────────────┴────────────────┐
                ▼                                 ▼
┌──────────────────────────────┐  ┌──────────────────────────────┐
│ streams/jvm                  │  │ streams/js                   │
│ StreamPlatformSpecific:      │  │ StreamPlatformSpecific:      │
│   16 blocking members        │  │   empty trait, 0 members     │
│ Sink.create                  │  │ no Sink.create               │
│ AsyncReader#toSync           │  │ no AsyncReader#toSync        │
└──────────────────────────────┘  └──────────────────────────────┘
  a blocking terminal parks       a blocking terminal is not a
  a thread and returns Either     member: it fails to compile
```

On the JVM the trait carries the whole blocking family, each member delegating to a blocking run of a `Sink`:

```scala
trait StreamPlatformSpecific[+E, +A] { self: Stream[E, A] =>
  def count: Either[E, Long]
  def exists(pred: A => Boolean): Either[E, Boolean]
  def find(pred: A => Boolean): Either[E, Option[A]]
  def forall(pred: A => Boolean): Either[E, Boolean]
  def foreach(f: A => Unit): Either[E, Unit]
  def head: Either[E, Option[A]]
  def last: Either[E, Option[A]]
  def run[ES, E3, Z](sink: Sink[ES, A, Z])(implicit
    errorConcat: Concat.WithOut[E, ES, E3]
  ): Either[E3, Z]
  def runCollect: Either[E, Chunk[A]]
  def runDrain: Either[E, Unit]
  def runFold(z: Double)(f: (Double, A) => Double): Either[E, Double]
  def runFold(z: Int)(f: (Int, A) => Int): Either[E, Int]
  def runFold(z: Long)(f: (Long, A) => Long): Either[E, Long]
  def runFold[Z](z: Z)(f: (Z, A) => Z)(implicit jtZ: JvmType.Infer[Z]): Either[E, Z]
  def runForeach(f: A => Unit): Either[E, Unit]
  def start(implicit scope: Scope): scope.$[Reader.SyncReader[A]]
}
```

The Scala.js counterpart is the same trait name, the same type parameters, and the same self-type, with nothing inside it:

```scala
trait StreamPlatformSpecific[+E, +A] { self: Stream[E, A] => }
```

That empty body is the whole mechanism. `Stream` still mixes the trait in on Scala.js, so no type signature anywhere in shared source changes; the sixteen names simply have no definition to resolve to, and the compiler reports each one as "not a member".

## `Sink.create` and Custom Sinks

Custom sinks follow the same rule as terminals, because a custom sink's callback is handed a reader and a reader is where the two execution modes part company. `Sink.create` is declared in `SinkCompanionPlatformSpecific`, which the `Sink` companion extends, and it exists only in the JVM copy of that trait:

```scala
object Sink {
  // JVM only — the callback consumes a blocking SyncReader
  def create[E, A, Z](f: Reader.SyncReader[A] => Z): Sink[E, A, Z]

  // JVM and Scala.js
  def createAsync[E, A, Z](f: Reader.AsyncReader[A] => Async[Z]): Sink[E, A, Z]
  def createBoth[E, A, Z](
    sync: Reader.SyncReader[A] => Z,
    async: Reader.AsyncReader[A] => Async[Z]
  ): Sink[E, A, Z]
}
```

Note the parameter type of `Sink.create`: it is `Reader.SyncReader[A]`, not the wider `Reader[A]`. The callback is guaranteed a synchronous reader, and when a `Sink` built this way is driven from an asynchronous terminal the JVM implementation bridges by calling `Reader.AsyncReader#toSync` — the member that does not exist on Scala.js either.

To write the same aggregation for both platforms, use `Sink.createAsync` and sequence the reader's pulls with `Async#flatMap` instead of a loop:

```scala

val average: Sink[Nothing, Int, Double] =
  Sink.createAsync[Nothing, Int, Double] { reader =>
    def loop(sum: Long, count: Long): Async[(Long, Long)] =
      reader.readInt(Long.MinValue).flatMap { value =>
        if (value == Long.MinValue) Async.succeed((sum, count))
        else loop(sum + value, count + 1L)
      }

    loop(0L, 0L).map { case (sum, count) =>
      if (count == 0L) 0.0 else sum.toDouble / count
    }
  }
```

`Sink.createBoth` is the option to reach for when the synchronous drain is worth keeping: it takes both callbacks, and the terminal selects exactly one of them, so the JVM keeps its direct blocking loop — with no per-pull `Async` to allocate and resume — while Scala.js gets a working implementation. Both callbacks must agree on how much input they consume and what they produce. [Sink](../core/sink.md) documents the reader protocol these callbacks drive.

## Manual Pull Across Platforms

Handing the reader to a protocol loop rather than a sink runs into the same split, and here the replacement changes shape rather than just its name. On the JVM, `Stream#start` is `Scope`-based and yields a synchronous reader:

```scala
trait StreamPlatformSpecific[+E, +A] { self: Stream[E, A] =>
  def start(implicit scope: Scope): scope.$[Reader.SyncReader[A]]
}
```

The reader is allocated into the enclosing [`Scope`](../../resource-management/scope.md) as an acquire-release resource, so closing the scope closes the reader, and the dependent result type `scope.$[Reader.SyncReader[A]]` keeps it from escaping that scope. The return type is `Reader.SyncReader[A]`: `Stream#start` never hands back the `Reader` union, because asynchronous boundaries inside the pipeline are bridged by the JVM runtime before you see it.

The cross-platform pair is `Stream#startAsync` and `Stream#useReaderAsync`, and they differ from `Stream#start` and from each other in who closes the reader:

```scala
abstract class Stream[+E, +A] {
  def startAsync: Async[Reader.AsyncReader[A]]
  def useReaderAsync[Z](f: Reader.AsyncReader[A] => Async[Z]): Async[Z]
}
```

`Stream#startAsync` transfers ownership to the caller, who must await `close()`; `Stream#useReaderAsync` retains it, passing the reader to `f` and awaiting its close on every outcome. The scoped one is the direct analogue of `Stream#start`, so it is the one to reach for when porting:

```scala

val collected: Async[Chunk[Int]] =
  Stream(1, 2, 3).useReaderAsync[Chunk[Int]](reader => reader.readAll[Int]())
```

One constraint carries over from the asynchronous reader contract and has no synchronous counterpart: an `AsyncReader` allows at most one operation in flight at a time. Await each `Async` before beginning the next. [Manual Pull and Ownership](./async-execution.md#manual-pull-and-ownership) covers the ownership rules in full.

## Concurrency and Threading

Underneath the availability split sits a capability split: the JVM has threads and Scala.js does not. That difference decides how the concurrent operators execute, and how a suspended `Async` is resumed.

### The JVM

Concurrent stream operators run their workers on virtual threads when the runtime provides them. `Platform.startVirtualThread` obtains `Thread.ofVirtual()` reflectively, so the module compiles and runs against any supported JDK and uses virtual threads on JDK 21 and later; when the reflective lookup fails for any reason — an older JDK, a security restriction, a linkage error — or when starting the virtual thread throws, it falls back to starting a named daemon platform thread rather than failing.

Workers are named, which makes them identifiable in a thread dump or profiler. The `Stream#mapPar` family names its workers `zio-blocks-mappar-worker-<n>-<index>` and its coordinator `zio-blocks-mappar-coordinator-<n>`, where `<n>` is drawn from a counter shared by every thread that lane's reader class starts and `<index>` identifies the worker within one reader. A known bug in 0.0.56 adds a `0` to every virtual thread's name, so on JDK 21 and later a worker shows up in a thread dump as `zio-blocks-mappar-worker-<n>-<index>0` (for example `zio-blocks-mappar-worker-0-00`) and the coordinator as `zio-blocks-mappar-coordinator-<n>0`; only the platform-thread fallback uses the names as written.

### Scala.js

`Platform.supportsConcurrency` is `false` on Scala.js and `Platform.startVirtualThread` throws `UnsupportedOperationException`. The observable effect is that synchronous work inside `Stream#mapPar`, `Stream#flatMapPar`, and `Stream.mergeAll` does not overlap: there is no second execution context for it to overlap on.

They do not all become `Stream#map` and `Stream#flatMap` on Scala.js. The factories that build the concurrent readers, `Platform.createMergeReaderFromReader` and `Platform.createMapParReaderFromReader`, route to the same shared concurrent engine on Scala.js as they do on the JVM — merge unconditionally, and `Stream#mapPar` whenever the upstream compiles to an asynchronous reader, with the mapping function wrapped in an immediately-ready effect:

```scala
internal.AsyncConcurrentReaders.mapPar(reader, n, (a: A) => zio.blocks.async.Async.succeed(f(a)), outType)
```

Only `Stream#mapPar` has a sequential implementation that a Scala.js program reaches: when every stage before it is synchronous, it becomes a plain mapped reader that maps one element at a time. The synchronous merge reader is built only by the JVM's blocking interpreter, and Scala.js has only the `*Async` terminals, so `Stream#flatMapPar` and `Stream.mergeAll` always run on the shared engine there: the inner streams' asynchronous waits overlap, but synchronous work inside them runs one inner stream at a time.

The consequence matters more than the plumbing. Wherever the concurrent engine runs, **unordered arrival is retained on Scala.js**. `Stream#mapPar`, `Stream#flatMapPar` and `Stream.mergeAll` are documented to emit in completion or arrival order rather than input order, on every platform, and that stays true on a single-threaded platform. Do not write code that relies on Scala.js restoring source order.

### The Async Execution Model

Under the streams layer, the `async` module resumes suspended computations differently on each platform. On the JVM, a suspended run executes as a serialized sequence of tasks on `ForkJoinPool.commonPool()`; only `Async.start(body)` spawns a thread of its own, a daemon thread named `zio-blocks-async-eval`; and `Async#block` parks the caller with `LockSupport` until the result arrives.

On Scala.js, resumptions are queued as microtasks with a `setTimeout(0)` macrotask escape hatch and a ready-resumption limit of 1024, so a long chain of already-complete steps yields to the event loop instead of starving it. `Async#block` throws, as described above. There is no blocking-operations API in the `async` module on either platform — nothing corresponding to a `blocking` executor or an `attemptBlocking` wrapper exists to be looked for.

[Async](../../async.md) documents both execution models in detail; this page states only the part that decides what compiles where.

## Platform Capabilities

The capability surface is deliberately small. `Platform` is a trait in shared source with one flag, one thread starter, and three reader factories, and the `Platform` object extends the platform-specific implementation of it, so call sites use the same `Platform.*` names regardless of platform:

```scala
trait Platform {
  def supportsConcurrency: Boolean
  def startVirtualThread(name: String, task: Runnable): Thread
  def createBufferedReaderFromReader[A](upstream: Reader[A], bufferSize: Int): Reader[A]
  def createMergeReaderFromReader[A](
    outerReader: Reader[?],
    maxOpen: Int,
    bufferSize: Int,
    elemType: JvmType
  ): Reader[A]
  def createMapParReaderFromReader[A, B](
    upstream: Reader[A],
    n: Int,
    f: A => B,
    bufferSize: Int,
    inType: JvmType,
    outType: JvmType
  ): Reader[B]
}

object Platform extends PlatformSpecific
```

Two `final` convenience methods, `Platform.createBufferedReader` and `Platform.createMapParReader`, narrow the corresponding factory's result to `Reader.SyncReader`. They add no capability of their own.

What `Platform` does *not* expose is worth stating, because it is the shape of query a cross-platform codebase tends to reach for first. There is no parallelism count, no executor or execution-context accessor, and no `isJS` flag. `Platform.supportsConcurrency` is the only capability flag, and the only supported way to ask at run time whether concurrent work will actually overlap.

## Enforcement

The placement of the blocking API is not maintained by convention; it is verified by a negative compilation test that asserts each blocking member is *absent* on Scala.js. Scala 3 asserts this in-band, using `scala.compiletime.testing.typeCheckErrors` to require that the compiler rejects each snippet with a "not a member" error:

```scala
private inline def missingMember(inline code: String, member: String): Boolean =
  typeCheckErrors(code).exists(error => error.message.contains(member) && error.message.contains("not a member"))
```

The spec covers `Reader.AsyncReader#toSync` (both directly and through wildcard imports, so a re-export cannot reintroduce it), `Stream#run`, `Stream#runCollect`, `Stream#start`, and `Sink.create`. Scala 2 has no `typeCheckErrors`, so those five references live in a fixture at `streams/js/src/test/negative-scala-2/BlockingApiPlacement.scala`, alongside two further references — `Writer#concatAsync` and `Writer#contramapAsync` — that name members `Writer` has on neither platform. That path sits outside the normal test source directories on purpose: it is not compiled by the ordinary Scala 2 test run, and no build task in this repository currently wires it in, so on Scala 2 the placement is documented by the fixture rather than enforced by it.

For the reader the point is a guarantee rather than a test detail: a cross-platform source that uses a blocking terminal fails at compile time on Scala.js. It will not build a Scala.js artifact that throws in a browser, and it will not pass a JVM build and then fail a JS one for a reason that only shows up under load.

## Scala 2 Versus Scala 3

There is no public-API difference between Scala 2.13 and Scala 3 in the streams module. Each version tree holds exactly two files, `LowPriorityJvmTypeInferPlatform.scala` and `internal/InternalVersionSpecific.scala`, and both declare `private[streams]` traits — implementation detail for `JvmType` inference and for primitive pull loops, invisible from user code.

The axis that changes what you can call is JVM versus Scala.js, which is what the [availability matrix](#availability-matrix) above records. [Scala 2 Compatibility](./scala-2-compatibility.md) covers the syntax differences that do affect how you write against the API, such as wildcard imports and implicit resolution.

## See Also

- [Asynchronous Stream Execution](./async-execution.md) — the full cross-platform `*Async` API
- [Reader](../primitives/reader.md#from-native-asynchronous-sources) — the JVM NIO and Scala.js `ReadableStream` adapters
- [Bounded Concurrency](../core/stream.md#bounded-concurrency) — `Stream#mapPar`, `Stream#flatMapPar`, `Stream.mergeAll`, and `Stream#mapParAsync`
- [Async](../../async.md) — the effect type and both execution models
- [How to Run Streams Without Blocking](../../../guides/run-streams-without-blocking.md) — porting a blocking pipeline to the `*Async` terminals and running it on Scala.js

---

## Scala 2 Compatibility Design Note

This document explains the design of Scala 2.13 support for `zio.blocks.streams` and the constraints that shaped it.

## Motivation

HTTP data types in `zio-blocks` depend on streams. `zio-http` 4 depends on those HTTP data types. Without Scala 2 stream support, `zio-http` cannot offer Scala 2 support. That dependency chain makes Scala 2.13 support for streams a hard requirement, not an optional nicety.

## Non-negotiable Constraint: Scala 3 Performance

When Scala 2 support was first proposed, the streams implementation used Scala 3 features on hot combinator paths, especially in `Stream` and `Sink` methods that participate in specialization, error-channel elimination, and zero-boxing-friendly code generation. The `inline` keyword on performance-sensitive helpers was not cosmetic; it directly affected what the JVM saw.

A Scala 2 compatibility layer is only acceptable if it leaves the Scala 3 hot path structurally unchanged. Concretely, this rules out:

- Moving key instance methods out of the Scala 3 class body into a shared trait
- Removing or weakening `inline` definitions to satisfy Scala 2's lack of that feature
- Introducing extra trait boundaries that alter the generated Scala 3 bytecode

Adapting surface syntax from Scala 3 `using` to Scala 2 `implicit` is fine where it does not touch the hot path. The risk is not the spelling of contextual parameters; the risk is changing the runtime shape of `Stream` and `Sink` under Scala 3.

## Rejected Approach: Version-specific Trait Extraction

An early draft extracted several instance methods into `StreamVersionSpecific` and `SinkVersionSpecific` traits under `scala-2/` and `scala-3/`, with the shared classes extending those traits. The methods moved or routed through these traits included:

- `Stream.++`, `Stream.catchAll`, `Stream.catchDefect`, `Stream.concat`
- `Stream.flatMap`, `Stream.mapError`, `Stream.orElse`, `Stream.&&`
- `Sink.mapError`

This shape localized the syntax differences neatly, but changed the structure of the Scala 3 hot path enough to produce measurable regressions. Benchmarks run with `streams-benchmark` on JDK 25 (recorded against Scala 3.8.3; the project targets Scala 3.9.0):

| Benchmark | Baseline (`main`) | Trait-extraction draft | Change |
|---|---:|---:|---|
| `StreamPipelineBench.zb_flatMap` | 14924.328 ops/s | 1178.997 ops/s | ~12x regression |
| `StreamPipelineBench.zb_concat` | 26003.818 ops/s | 22044.497 ops/s | regression |
| `StreamPipelineBench.zb_filterMap` | 54812.709 ops/s | 50349.471 ops/s | regression |

The `flatMap` result is the clearest signal. Dropping from roughly 14.9k ops/s to 1.18k ops/s is not an acceptable tradeoff for any compatibility layer. The approach was rejected.

## Current Structure: One Shared Source Set

Streams compiles from a single shared source set. `Stream`, `Sink`, `Reader`, `Writer`, and `Pipeline` each have exactly one implementation, under `streams/shared/src/main/scala/`, and the same bytes compile for Scala 2.13 and Scala 3.

What made that possible is that the hot combinator paths no longer use any Scala 3-only construct. There is no `inline def` or `inline val`, no `using` or `given`, no `extension`, `enum`, or `opaque type` anywhere in the module's main sources. The constraint above was met not by mirroring the hot path into two trees but by writing it in the syntax both compilers accept, which leaves nothing for a Scala 2 tree to fork.

Two files remain version-specific, and each exists under both `scala-2/` and `scala-3/`:

```
streams/shared/src/main/
├── scala/                  the whole public API: Stream, Sink, Reader, Writer, Pipeline
├── scala-2/zio/blocks/streams/
│   ├── LowPriorityJvmTypeInferPlatform.scala
│   └── internal/InternalVersionSpecific.scala
└── scala-3/zio/blocks/streams/
    ├── LowPriorityJvmTypeInferPlatform.scala
    └── internal/InternalVersionSpecific.scala
```

Both declare `private[streams]` traits and neither is reachable from user code. `LowPriorityJvmTypeInferPlatform` supplies the lowest-priority `JvmType.Infer[A]` fallback that sends an unrecognized element type to the boxed lane, mixed into `JvmType.Infer`; see [Zero-Boxing](./zero-boxing.md) for the lane machinery it serves. `InternalVersionSpecific` supplies the `pullInt`, `pullLong`, `pullFloat`, and `pullDouble` helpers that read one element from a `Reader.SyncReader` on its primitive lane.

Each pair is currently byte-identical, so the surviving split is directory-only: it is a place where a per-version difference could be expressed, not a difference that exists today. Nobody maintains two implementations of anything.

There is no public-API difference between Scala 2.13 and Scala 3 in streams, as [Platform Differences](./platform-differences.md#scala-2-versus-scala-3) records. What differs for you is surface syntax you write anyway — a wildcard import is `_` rather than `*`, and a contextual parameter is `implicit` rather than `using`. The axis that actually changes which members exist is JVM versus Scala.js, covered by the [availability matrix](./platform-differences.md#availability-matrix).

## Maintenance Notes

Any change to the behavior or public API of `Stream`, `Sink`, `Reader`, `Writer`, or `Pipeline` goes to the shared source under `streams/shared/src/main/scala/`. There is no second tree to mirror it into.

Keep the shared sources inside the syntax both compilers accept. An `inline def`, a `using` clause, or an `extension` method added there compiles under Scala 3 and breaks the published Scala 2.13 build, and the constraint above rules out recovering by forking the affected method into two trees.

Touch a file under `scala-2/` or `scala-3/` only when you intend a genuine per-version difference, and then change the counterpart in the same commit. Because each pair is byte-identical today, editing one alone is a divergence rather than a fix, and nothing in the build will tell you that you meant it.

When making changes that touch hot combinators, re-run `streams-benchmark` and verify that `zb_flatMap`, `zb_concat`, and `zb_filterMap` do not regress relative to the `main` baseline.

## See Also

- [Platform Differences](./platform-differences.md) — the JVM versus Scala.js availability matrix, and the Scala 2 versus Scala 3 summary
- [Zero-Boxing](./zero-boxing.md) — `JvmType.Infer` and the primitive lanes that `LowPriorityJvmTypeInferPlatform` backs

---

## Zero-Boxing Optimization

Working with streams of primitives presents a performance challenge in languages with generic types: **boxing**. Without special care, primitive values get wrapped in objects. ZIO Blocks Streams therefore carries runtime element-type information and provides primitive physical lanes for synchronous interpretation. This removes boxing at important boundaries and hot paths; it is not a promise that every stream program, callback, terminal, or asynchronous operation is allocation-free.

## The Boxing Problem

In Scala, primitive types (`Int`, `Long`, `Double`, `Boolean`) are fundamentally different from their object counterparts (`Integer`, `Long`, `Double`, `Boolean`). When a generic class like `Stream[E, A]` works with primitives, the compiler must box them into objects to satisfy the generic contract:

```scala
// Without optimization, this boxes each Int into an Integer object
val stream: Stream[Nothing, Int] = Stream(1, 2, 3, 4, 5)
val doubled = stream.map(_ * 2)  // Each Int is boxed → Integer → boxed result
val result = doubled.runCollect
// Result: Each element was boxed, unboxed, boxed again — wasteful!
```

**Performance cost:**
- Extra heap allocations (memory pressure, more GC)
- Cache misses (objects spread across memory)
- Slower CPU operations (dereferencing objects instead of primitive registers)

For high-throughput data processing, this overhead is unacceptable.

## ZIO Blocks Streams' Solution: JvmType Dispatch

Instead of using Scala's `@specialized` annotation (which generates separate classes for each primitive type, bloating binaries), ZIO Blocks Streams uses **compile-time type detection + runtime dispatch**. This gives you the speed of specialization without the binary bloat.

### How It Works

**Step 1: Compile-Time Detection**

When you create a stream of primitives, the compiler infers a `JvmType` implicit that identifies the element type:

```scala

val intStream: Stream[Nothing, Int] = Stream(1, 2, 3)
// Compiler infers: JvmType.Infer[Int]
// The stream records its element representation

val doubled = intStream.map(_ * 2)
// map receives JvmType.Infer evidence for its transformed result type.
// Its input representation comes from intStream, not call-site evidence.
```

**Step 2: Runtime Type Dispatch**

Stream and pipeline nodes retain their logical `JvmType`. When a run materializes a reader, the drain loop reads that reader's own `jvmType` tag and branches once onto the matching primitive pull. The tag describes what the reader physically pulls, not what the static type was widened to, so a `Stream[Nothing, Int]` widened to `Stream[Nothing, AnyVal]` and then filtered is still pulled through the `Int` lane. [When Zero-Boxing Applies](#when-zero-boxing-applies) lists the lanes and [How a lane is chosen](#how-a-lane-is-chosen) covers when each half of the decision is made.

**Step 3: Unboxed Accessors**

Instead of a single `read()` method that returns boxed `Any` (where boxed means wrapping primitives in object wrappers like `Integer`, `Long`, `Double`), primitives use specialized accessors that operate directly on primitive values:

```scala
abstract class Reader[+Elem] {
  def jvmType: JvmType = JvmType.AnyRef
}

object Reader {
  abstract class SyncReader[+Elem] extends Reader[Elem] {
    def read[A >: Elem](sentinel: A): A
    def readBoolean(_sentinel: Int)(implicit _ev: Elem <:< Boolean): Int
    def readInt(_sentinel: Long)(implicit _ev: Elem <:< Int): Long
    def readDouble(_sentinel: Double)(implicit _ev: Elem <:< Double): Double
    def readDoubles(buf: Array[Double], offset: Int, maxLen: Int)(implicit ev: Elem <:< Double): Int
  }

  abstract class AsyncReader[+Elem] extends Reader[Elem] with AsyncReaderPlatform[Elem] {
    def read[A >: Elem](sentinel: A): Async[A]
    def readBoolean(_sentinel: Int)(implicit _ev: Elem <:< Boolean): Async[Int]
    def readInt(_sentinel: Long)(implicit _ev: Elem <:< Int): Async[Long]
    def readDouble(_sentinel: Double)(implicit _ev: Elem <:< Double): Async[Double]
    def readDoubles(dest: Array[Double], offset: Int, length: Int)(implicit ev: Elem <:< Double): Async[Int]
  }
}
```

The complete primitive pull surface distinguishes all eight identities: `Boolean`, `Byte`, `Short`, and `Char` use `Int` carriers; `Int` uses a widened `Long`; `Long` uses `Long`; `Float` uses a widened `Double`; and `Double` uses `Double`. The scalar reads name their sentinel parameter `_sentinel` because the base implementations spend it only on the reference path and reject it on a genuine primitive lane, which primitive readers override. `Long` and `Double` never spend it at all — those two lanes detect end of stream through the bulk reads instead, which is why `readDoubles` appears above beside its scalar twin. [EOF signalling per lane](#eof-signalling-per-lane) has the full table.

For `SyncReader`, the carrier is a JVM primitive at the method boundary. `SyncInterpreter` also stores primitive state in `Long`-backed physical arrays (using raw bits where necessary) and invokes lane-adapted operators. By contrast, `AsyncReader` returns generic `Async[T]` values. `AsyncInterpreter` keeps primitive lane state internally, but completed primitive results cross the generic async carrier and may be boxed — see [Async and boxing](#async-and-boxing).

## Practical Benefits

To understand the intended benefit, consider how boxing can accumulate through a pipeline. Compare a hypothetical boxed implementation with ZIO Streams' primitive-lane approach.

### Before (Hypothetical Boxed Streams)

Without optimization, each operation in a pipeline adds boxing overhead:

```scala
val nums = Stream(1, 2, 3, 4, 5)
val result = nums
  .map(_ * 2)        // boxes each Int → Integer, applies *, unboxes result
  .filter(_ > 5)     // boxes again, compares, unboxes
  .map(_ + 1)        // boxes, adds, unboxes
  .runCollect
// 5 elements × 3 operations × boxing overhead = significant waste
```

**Memory profile:** Each element is boxed/unboxed multiple times, creating temporary objects.

### With ZIO Streams (Primitive Lanes)

The synchronous interpreter can keep the same pipeline on primitive physical lanes:

```scala

val nums = Stream(1, 2, 3, 4, 5)
val result = nums
  .map(_ * 2)
  .filter(_ > 5)
  .map(_ + 1)
  .runCollect
```

This design avoids per-stage primitive wrapper storage in the fused synchronous interpreter. It does **not** prove that the whole expression allocates nothing: generic Scala function interfaces, source or terminal construction, result collection, fallback paths, and JIT decisions can still introduce boxing or allocation. `runCollect` above is a JVM-only terminal; in cross-platform source write `runCollectAsync`, which drives the same primitive lanes and returns `Async[Either[E, Chunk[Int]]]`.

## When Zero-Boxing Applies

There are nine logical lanes: the eight JVM primitives — `Boolean`, `Byte`, `Char`, `Short`, `Int`, `Long`, `Float`, `Double` — and `AnyRef` for everything else. Each is a separate *physical pull identity*, because each has its own exact read method and its own end-of-stream convention. `readInt` is not an acceptable substitute for `Boolean`, `Byte`, `Char`, or `Short`, and a generic `read` followed by a cast is never an acceptable primitive pull.

Nine identities do not mean nine interpreter arrays. The interpreter compacts them into five storage lanes — int-like, `Long`, `Float`, `Double`, and reference — where the five small integral types (`Boolean`, `Byte`, `Char`, `Short`, `Int`) all land in the int-like lane. That compaction is visible in the arithmetic of the operator tags: a fused `map` is tagged `inLane * 5 + outLane`, twenty-five crossings rather than eighty-one. The read tag stays deliberately independent of the storage lane, so a `Short` stream is pulled by `readShort` and only afterwards shares int-like storage with an `Int` stream.

Primitive type inference and lane selection are automatic. Primitive streams are eligible for specialized synchronous paths, subject to the operation, source, terminal, and any fallback boundaries:

```scala

// Primitive-specialized logical types
val ints = Stream(1, 2, 3).map(_ * 2)
val longs = Stream(1L, 2L, 3L).filter(_ > 0L)
val doubles = Stream(1.5, 2.5, 3.5).map(_ + 1.0)
val bools = Stream(true, false, true).filter(identity)

// Reference lane: fields are primitive, but Point itself is an object
case class Point(x: Int, y: Int)
val points = Stream(Point(1, 2), Point(3, 4))
  .map(p => Point(p.x * 2, p.y * 2))

// Reference lane: tuples are objects
val pairs = Stream((1, 2), (3, 4))
  .map { case (x, y) => (x + 1, y + 1) }

// Works, but may box for non-primitive types
val strings = Stream("a", "b", "c").map(_.toUpperCase)
```

You don't need to do anything special — the compiler and runtime handle it automatically.

## How a Lane Is Chosen

The decision is hybrid and asymmetric: half of it is made when your code compiles, half when the stream runs.

```
┌────────────────────────────────────────────────────────────────┐
│ COMPILE TIME — output lane                                     │
│ JvmType.Infer[B] resolves where the result of map/collect      │
│ lands. Nothing is generated: no @specialized, no macros.       │
└────────────────────────────────────────────────────────────────┘

    the node stores that tag and is then complete
    ▼

┌────────────────────────────────────────────────────────────────┐
│ RUNTIME, ONCE PER DRAIN — input lane                           │
│ reader.jvmType names the lane the materialized reader          │
│ actually pulls on, whatever the widened static type says.      │
└────────────────────────────────────────────────────────────────┘

    one match picks the loop, then the loop runs
    ▼

┌────────────────────────────────────────────────────────────────┐
│ PER ELEMENT — no dispatch left to do                           │
│ The chosen loop calls one exact pull and one callback.         │
│ The lane is never re-decided while elements flow.              │
└────────────────────────────────────────────────────────────────┘
```

**The output lane is fixed at compile time.** An operation that can change the element representation asks for evidence about what it produces, never about what it consumes:

```scala
def map[B](f: A => B)(implicit jtB: JvmType.Infer[B]): Stream[E, B]
def filter(pred: A => Boolean): Stream[E, A]
```

One implicit, and it is about `B`. Implicit resolution finds one of the eight primitive instances or falls through to `Infer.boxed[A]` and records `AnyRef`; adding a `JvmType.Infer` parameter to a signature is always source-compatible, because the low-priority fallback never fails to resolve. Lane-preserving operations such as `filter` ask for nothing and keep whatever representation they were handed.

`Infer` is invariant in `A`. Contravariance would let primitive evidence satisfy `Infer[Nothing]` by accident and let fallback evidence drag a precise output toward `Any`; invariance plus an explicit `Infer[Nothing]` instance closes both. The fallback itself lives in per-Scala-version `LowPriorityJvmTypeInferPlatform` files whose Scala 2 and Scala 3 bodies are byte-identical, so the split is directory-only and there is no version-dependent inference behavior to reason about.

**The input lane is chosen at runtime, from the reader.** A `Stream` only describes; a run materializes a `Reader`, and that reader reports the lane it physically pulls on through `jvmType`. This is a contract about the reader, not about the static type: widening `Stream[Nothing, Int]` to `Stream[Nothing, AnyVal]` and then filtering, tapping, taking, or buffering still pulls the source through the `Int` lane, because none of those operations changes the representation.

**Dispatch happens once per drain, never per element.** A synchronous fold matches on `reader.jvmType` one time, enters the loop belonging to that lane, and stays in it; the asynchronous puller goes further and resolves its per-element pull function once, when it is constructed. Inside the loop no lane test remains — one exact pull, one callback.

### There Is No Lane Diagnostic for a Stream

Readers reach this section wanting a way to confirm that a pipeline stayed on a primitive lane. Nothing on `Stream` answers that, and it is worth being explicit about why each apparent candidate is not it either.

- `JvmType.Infer[A]` reports the **static** type, which is exactly the thing the representation machinery stopped trusting. A `Stream[Nothing, AnyVal]` whose reader is still on the `Int` lane infers `AnyRef`.
- `ElementRepresentation` — the three-case value (`Known(jvmType)`, `Boxed`, `LateBound`) that stream nodes actually carry — is `private[streams]` and not part of the public API.
- `Reader#jvmType` **is** public, so inside a `Sink.createBoth` synchronous callback, or after `Stream#startAsync` or `Stream#useReaderAsync`, you can read the materialized reader's lane. That answers a narrower question than the one being asked: it names the lane at that one boundary, not whether every fused stage kept it.

For the end-to-end question, allocation profiling is the real diagnostic. Measure the workload under an allocation profiler and read the normalized allocation rate; a throughput number on its own cannot tell you whether a lane was held.

## Generalized Specialization

All eight primitive lanes are specialized — `Boolean`, `Byte`, `Char`, `Short`, `Int`, `Long`, `Float`, and `Double` — across `Stream`, `Sink`, `Pipeline`, and `Reader`, in both the synchronous and the asynchronous interpreter, on the JVM and on Scala.js. Counting the reference fallback, that is nine-lane dispatch.

The specialized path is not reserved for any one element and accumulator pairing. A `Short` stream folded into a `Double`, a `Char` stream merged across workers, a `Boolean` stream collected — each reaches the same shared drivers, the same selector lifecycle, and the same buffer policy as an `Int` stream reduced by a `Long` checksum.

Specialization is about where boxing is avoided by construction, not a throughput promise: it says which lane a pull travels on, not what a given workload will measure.

## EOF Signalling Per Lane

A pull has to be able to say "there are no more elements" without allocating an `Option` to say it in, so every lane reports exhaustion out of band. The schemes differ because the lanes differ. The authority is the nine-branch dispatch in `Sink.foldSyncReader`; the library's other synchronous drain loops — `Reader#readAll`, `Sink.exists`, `Sink.find` — use the same scheme for each lane. The asynchronous puller differs on two lanes: it detects end of stream on `Byte` and `Float` by a negative count from `readBytes` and `readFloats`, rather than by the scalar sentinel.

| Lane      | Public pull for this lane    | Carrier          | End of stream                       |
|-----------|------------------------------|------------------|-------------------------------------|
| `Boolean` | `readBoolean(-1)`            | `Int`            | any negative value                  |
| `Byte`    | `readByte()`                 | `Int`            | `-1` (any negative value); takes no sentinel parameter |
| `Char`    | `readChar(-1)`               | `Int`            | any negative value                  |
| `Short`   | `readShort(Int.MinValue)`    | `Int`            | `Int.MinValue`                      |
| `Int`     | `readInt(Long.MinValue)`     | `Long`           | `Long.MinValue`                     |
| `Long`    | `readLongs(scratch, 0, 1)`   | `Array[Long]`    | **negative returned count**         |
| `Float`   | `readFloat(Double.MaxValue)` | `Double`         | `Double.MaxValue`                   |
| `Double`  | `readDoubles(scratch, 0, 1)` | `Array[Double]`  | **negative returned count**         |
| `AnyRef`  | `read(sentinel)`             | the element type | the sentinel, by reference identity |

Three things are worth reading off that table.

**The sentinel travels in the widened carrier.** `readInt` returns `Long` rather than `Int` precisely so a value outside the element's domain exists to spend; `Boolean`, `Char`, and `Short` widen to `Int` for the same reason. `readByte()` is the exception that proves the rule — it yields unsigned bytes in `0..255`, so it can reserve `-1` permanently and takes no sentinel parameter at all.

**`Long` and `Double` use no sentinel.** For those two lanes the carrier is the element type itself, and there is no `Long` value and no `Double` bit pattern — NaN payloads and signed zero included — that is not also legitimate data, so every candidate sentinel collides with something a stream may legitimately carry. They detect end of stream by count instead: a length-one `readLongs` or `readDoubles` whose returned count is negative. The one-element scratch array is allocated once by the owner, outside the loop, never inside it. That is what makes these two lanes fully lossless — every `Long` value and every `Double` bit pattern stays readable as data.

**`readLong` and `readDouble` still take a sentinel parameter.** They are kept for symmetry with the five other sentinel-taking pulls, and for callers who can prove that a particular sentinel lies outside their own data domain. Without that qualifier the prose would contradict the signatures; with it, the point stands, because nothing can safely fill the parameter in general-purpose code and the library never relies on it. `SinkExactLaneSpec` and `RouteSpecializationProofSpec` install probe readers whose scalar `readLong` and `readDouble` throw an `AssertionError` even on their own matching lane, so a regression that reintroduced sentinel detection fails a test rather than silently truncating a value.

:::warning[Do not detect end of stream with `readLong` or `readDouble`]
Any value you reserve as an end marker on those two lanes is also a value the stream may carry, and the truncation is silent. Use `readLongs(scratch, 0, 1)` or `readDoubles(scratch, 0, 1)` and branch on the returned count.
:::

[Reader](../primitives/reader.md#sentinel-protocol) states the same contract from the implementor's side.

## Comparison: @specialized Vs JvmType Dispatch

ZIO Blocks Streams' approach differs fundamentally from Scala's traditional `@specialized` annotation. Here's how they compare:

**Traditional Scala `@specialized` annotation** generates separate specialized classes at compile time:

```scala
@specialized(Int, Long, Double)
class Stream[+E, +A] { ... }
// Generates separate classes:
// - Stream$mcI$sp (specialized for Int)
// - Stream$mcJ$sp (specialized for Long)
// - Stream$mcD$sp (specialized for Double)
// - Stream (generic fallback)
// Result: additional generated classes and bytecode
```

**ZIO Blocks `JvmType` dispatch** records logical input and output types on stream and pipeline nodes. Compilation turns those types into interpreter lane and operator tags; it does not generate a separate `Stream` class for every primitive.

| Metric | `@specialized` | JvmType |
|--------|---|---|
| **Binary size** | Additional specialized classes | No class-per-primitive specialization |
| **Bytecode complexity** | High | Moderate |
| **Runtime dispatch** | Selected by specialized class | Interpreter lane and operator-tag dispatch |
| **Flexibility** | Fixed at compile time | Adaptive at runtime |
| **Primitive support** | Configurable | Boolean, Byte, Short, Char, Int, Long, Float, Double |
| **Generality** | Good for all generics | Specialized for Stream/Sink |

## Implementation Architecture

Primitive type metadata is carried across ZIO Blocks Streams' three core abstractions. Whether a concrete execution remains on primitive physical lanes depends on the interpreter and operation.

### Stream[E, A]

Stores element representation when a source is constructed and dispatches `Reader` accesses. Transformations request fresh evidence for their result, not for an input whose representation the stream already knows:

```scala

val stream: Stream[Nothing, Int] = Stream(1, 2, 3)
// JvmType.Int is inferred and available to all operations

val asLong = stream.map(_.toLong)
// Infer[Long] records the transformed result representation
```

### Sink[E, A, Z]

Pulls elements from the run's [`Reader`](../primitives/reader.md) with the read that matches the lane (`Reader#readInt`, `Reader#readFloat`, or the count-returning `Reader#readLongs` and `Reader#readDoubles`), so the elements a built-in sink reads never box. A known bug in 0.0.56: `Sink.sumLong`, `Sink.sumDouble`, `Sink.sumInt`, `Sink.sumFloat`, `Sink.count` and `Stream#runFold(z: Double)` over a `Long` or `Double` stream still box their accumulator for every element, while `Stream#runFold(0L)` does not. Custom sinks built with `Sink.create` or `Sink.createBoth` can use the same reads. To fold an `Int` stream through the unboxed `Int` lane:

```scala

val nums = Stream(1, 2, 3)
val sum = nums.runFold(0)(_ + _)
// The synchronous interpreter can feed the fold through an Int lane
```

### Pipeline[A, B]

Preserves the input representation and records evidence for transformed results:

```scala

val pipe = Pipeline.map[Int, Int](_ * 2)
// Infer[Int] describes the result. The input representation is supplied when
// the pipeline is applied to a Stream or Sink.

val throughStream = Stream(1, 2, 3).via(pipe)
val throughSink   = pipe.andThenSink(Sink.sumInt)
// Both application routes propagate the same representation information.
```

## Performance Impact

Performance falls into distinct categories:

- **Scalar CPU pipelines** (`map`, `filter`, folds) benefit most from primitive lanes because boxing, allocation, and dispatch are a large share of the work.
- **Bulk memory paths** (`readInts`, `readLongs`, `readDoubles`, collection) additionally benefit from contiguous arrays and fewer per-element calls.
- **Bounded concurrency** pays queue/selector and scheduling costs; use it when callback or child latency dominates, not for trivial arithmetic.
- **Native asynchronous I/O** is governed mainly by source latency, chunk size, and cancellation/ownership costs; zero-boxing is usually secondary.
- **Writer `*Async` adapters** defer synchronous work and may still block, so they should not be benchmarked as native async I/O.

Do not infer a universal multiplier from these categories. Results depend on JDK, Scala version, platform, element type, pipeline shape, buffer size, and terminal; use the repository JMH benchmarks with a workload representative of the application. This page publishes no speedup figure for generalized specialization: any single multiplier would be drawn from a narrower set of benchmark shapes than the claim it was used to support.

Repository JMH results measure throughput for named benchmark methods and configurations, and a suite is evidence only about the contract it actually measures — a ready-effect suite measures ready effects, not asynchronous stream performance in general, and results for unlike contracts must not be aggregated into one ranking. Unless a run also records an allocation profiler (for example `gc.alloc.rate.norm`), it is **not** evidence of zero allocations; a near-zero figure from a throughput run is profiler noise, not a promise. The lane layout and primitive reader signatures establish where boxing is avoided by construction; claims about callback invocation, complete pipelines, async carriers, or parity with handwritten loops remain unproven until measured with allocation profiling for that exact workload.

## Async and Boxing

The claims above are about the synchronous interpreter, and the asynchronous one has an honest limit that belongs right next to them: **asynchronous execution is lane-aware, not end-to-end zero-boxing.**

Lane awareness is real. `AsyncInterpreter` keeps primitive lane state internally, the asynchronous puller resolves its exact per-element pull once from `reader.jvmType`, and `AsyncReader` exposes the same nine-identity pull surface as its synchronous counterpart, bulk `readLongs` and `readDoubles` included.

What changes is the carrier at the boundary. Every asynchronous result crosses a generic `Async[T]`, and a completed primitive in an `Async[Long]` is a `Long` in a generic position, so it may be boxed. The same applies wherever a primitive passes through another generic carrier: an erased Scala function, an `Option`, an `Either`, or the `Async[Either[E, Z]]` convention the asynchronous terminals use. That carrier-boundary boxing is a different thing from the error the lanes exist to prevent — pulling an upstream primitive through generic `read`, or routing it through the wrong lane — and only the second is ruled out by construction.

The practical consequence is that "zero-boxing" should not be read as a property of an asynchronous pipeline end to end. It describes the pull surface and the interpreter's own state, and stops at the carrier. As everywhere else on this page, whether a particular asynchronous workload allocates is a question for an allocation profiler. [Asynchronous Stream Execution](./async-execution.md#why-two-engines) explains why the two engines exist at all.

## When Polymorphism Is Necessary

If you need polymorphic behavior (e.g., different handling for different types), use `JvmType` directly:

```scala

def processStream[A](stream: Stream[Nothing, A])(implicit jt: JvmType.Infer[A]): Unit = {
  jt.jvmType match {
    case JvmType.Int =>
      println("Processing integers")
    case JvmType.Long =>
      println("Processing longs")
    case _ =>
      println("Processing generic type")
  }
}
```

This gives code runtime type information. It does not by itself guarantee allocation-free execution; the selected operation and interpreter still determine the physical path.

## Summary

ZIO Blocks Streams reduces primitive boxing through:

1. **Compile-time output lanes** fixed by `JvmType.Infer[A]` implicit resolution
2. **Runtime input lanes** read from `reader.jvmType`, dispatched once per drain rather than per element
3. **Primitive synchronous accessors** across all nine identities, with widened sentinel carriers, and count-based end of stream on the two lossless lanes
4. **Lane-aware async interpretation**, while accepting that generic `Async[T]` carriers may box completed primitives

The result is specialized hot paths without `@specialized` class proliferation, available to every lane and every accumulator type rather than to one benchmark-shaped product. Some generic callbacks, reference values, tuples/case classes, and async/concurrent coordination can still allocate or box; verify important workloads with an allocation profiler rather than assuming allocation-free execution end to end.

## See Also

- [Reader](../primitives/reader.md#the-reader-union) — the two reader kinds, and the sentinel protocol from the implementor's side
- [Sink](../core/sink.md) — the typed sinks that drain through these lanes
- [Pipeline](../core/pipeline.md) — why the transforming factories ask for `JvmType.Infer` on their result type
- [Asynchronous Stream Execution](./async-execution.md#one-stream-type-two-execution-modes) — how a graph becomes synchronous or asynchronous, and why there is no mode annotation either
- [Platform Differences](./platform-differences.md) — what changes on Scala.js, where the same lanes apply
- [How to Avoid Allocation on Hot Paths](../../../guides/avoid-allocation-on-hot-paths.md) — keeping a hot loop on a primitive lane and measuring whether it still allocates
- [How to Process Binary Data Without Boxing](../../../guides/process-binary-data-without-boxing.md) — read, transform, aggregate and write `ByteBuffer`s, mapped files and `InputStream`s as unboxed `Long` and `Double` streams

---

## Streams

`zio.blocks.streams` is a **pull-based** streaming library for **Scala 3** (and Scala 2.13) with synchronous and asynchronous readers, typed errors, resource safety, and primitive specialization. Streams are lazy descriptions -- nothing executes until a terminal operation is driven. Cross-platform terminals ending in `Async` return `Async[Either[E, Z]]`; the JVM also provides blocking terminals returning `Either[E, Z]`. The library has zero runtime dependencies beyond ZIO Blocks modules, and avoids boxing on primitive element types through JVM-type-specialized internal readers.

ZIO Blocks Streams is built on three composable primitives:

| Type                                   | Description                                                          | Key operation         |
|----------------------------------------|----------------------------------------------------------------------|-----------------------|
| [`Stream[+E, +A]`](./core/stream.md)        | A lazy, pull-based sequence of elements that may fail with error `E` | `stream.via(pipe)`    |
| [`Pipeline[-In, +Out]`](./core/pipeline.md) | A reusable, composable stream-to-stream transformation               | `pipe.andThen(other)` |
| [`Sink[+E, -A, +Z]`](./core/sink.md)        | A stream consumer that produces a typed result `Z`                   | `stream.run(sink)`    |

## Overview

ZIO Blocks Streams is designed around three core principles:

**Dual execution.** Cross-platform `*Async` terminals drive either synchronous or asynchronous sources without blocking and require no ZIO runtime. On the JVM, plain terminals such as `run`, `runCollect`, and `head` are blocking compatibility twins.

**Pull-based evaluation.** Execution is driven from the consumer (Sink) backward through the pipeline to the source (Stream). This enables natural short-circuiting: if a sink only needs the first three elements, the stream stops producing after three elements — no work is wasted.

**Resource safety via RAII.** Resources acquired during stream construction (file handles, database connections, etc.) are always released in `finally` blocks, whether the stream succeeds, fails, or is short-circuited.

## Quick Start

Here's a minimal JVM example. Streams are lazy descriptions — nothing executes until you call a terminal operation like `runCollect`. Use `runCollectAsync` for the cross-platform form.

Unless a section says otherwise, examples using plain terminals (`run`, `runCollect`, `head`, and their peers) are JVM-only shorthand. Replace them with the matching `*Async` terminal in shared JVM/Scala.js code.

```scala

// Build a lazy stream description
val stream = Stream.range(1, 100)
  .filter(_ % 2 == 0)
  .map(_ * 3)

// Run it — nothing executes until here
val result = stream.take(5).runCollect
// Right(Chunk(6, 12, 18, 24, 30))
```

## Installation

Add the Streams module to your SBT build:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-streams" % "0.0.56"
```

For Scala.js (JavaScript/Node.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-streams" % "0.0.56"
```

Supported Scala versions: 2.13.x and 3.x.

## Why Streams?

Streaming libraries in the Scala ecosystem typically require an effect system. fs2 runs in a `cats.effect`-compatible `F[_]`, Kyo Streams needs the Kyo runtime, and Pekko (formerly Akka) Streams needs the actor runtime. When your code is synchronous and you want streaming without pulling in an effect monad, the options narrow considerably.

`zio.blocks.streams` fills that gap:

| Feature                   | ZB Streams              | fs2                  | Kyo           | Ox                      | Pekko           |
|---------------------------|-------------------------|----------------------|---------------|-------------------------|-----------------|
| Effect system required    | No                      | Yes (cats-effect)    | Yes (Kyo)     | No (virtual threads)    | Yes (Pekko)     |
| Execution model           | Sync/async, pull-based  | Async, pull-based    | Async, chunk  | Synchronous, pull-based | Async, push     |
| Typed errors              | `Either[E, Z]`          | Not verified here    | Not verified here | Not verified here   | Not verified here |
| Primitive specialization  | Yes (zero boxing)       | Not verified here    | Not verified here | Not verified here   | Not verified here |
| Stack-safe deep pipelines | Yes (trampolined)       | Not verified here     | Not verified here | Not verified here       | Not verified here |
| Resource safety           | Scope integration       | Resource/bracket     | Kyo resources | try/finally             | Graph lifecycle |
| Dependencies              | scope, chunk, combinators, ringbuffer, async | fs2-core     | kyo-prelude, kyo-core | ox core         | pekko-stream    |

The provider columns name the artifacts this repository pins for benchmarking. `build.sbt` pins fs2 3.14.0, Pekko 1.7.0, Kyo 1.0.0-RC7 and Ox 1.1.0. `streams-benchmark/benchmark-manifest.json` records the versions its baselines were taken with: fs2 3.14.0, Pekko 1.7.0, Kyo 1.0.0-RC6 and Ox 1.0.6. Nothing outside the ZB Streams column is measured or verified in this repository.

If you are porting existing code, the [migration guide](../../guides/streaming-library-migration.md) maps fs2, ZIO Streams and Ox code onto ZIO Blocks.

## Benchmarks

The repository carries a JMH benchmark suite under `streams-benchmark`. Provider comparisons are
governed by the allowlist and provider versions recorded in
`streams-benchmark/benchmark-manifest.json`; results are only comparable within a single benchmark
class and contract, and are not aggregated into a ranking here. Re-run the benchmarks on your target
environment before drawing any performance conclusion.

If you are evaluating Scala 2 compatibility work, read the [Scala 2 compatibility design note](./execution-and-compatibility/scala-2-compatibility.md) before moving any `Stream` or `Sink` hot-path combinators behind version-specific seams.

## Core Mental Model

To understand ZIO Blocks Streams fully, it's helpful to see how the three primitives fit together and how data flows through a pipeline from source to sink. This section walks through the architecture and explains each component in depth.

### Execution Flow

Operations on streams transform the pipeline and ultimately run it against a sink:

```
┌──────────────────────────────────┐
│ Stream[E, A]                     │
│ (lazy description)               │
└──────────────────┬───────────────┘
                   │
      .flatMap, .map, .filter, etc.
                   │
┌──────────────────▼───────────────┐
│ Pipeline[-In, +Out]              │
│ (stream → stream transformation) │
└──────────────────┬───────────────┘
                   │
        .via(pipe)
                   │
┌──────────────────▼───────────────┐
│ Sink[E, A, Z]                    │
│ (stream consumer → result Z)     │
└──────────────────┬───────────────┘
                   │
         .run(sink)
                   │
┌──────────────────▼───────────────┐
│ Async[Either[E, Z]]              │
│ (or blocking Either on JVM)      │
└──────────────────────────────────┘
```

The last box is where the flow forks. The same `Stream` description is materialized either as a
synchronous reader, drained on the calling thread by a blocking terminal such as `run` or
`runCollect`, or as an asynchronous reader, driven without blocking by the matching `*Async`
terminal. Which engine runs is decided by the source and operators the pipeline is built from, not
by the terminal you call. See [Async Execution](./execution-and-compatibility/async-execution.md) for how that classification
works.

### 1) `Stream[E, A]` -- a Lazy Sequence

A `Stream[+E, +A]` is a **description** of a potentially infinite sequence of elements of type `A` that may fail with an error of type `E`. It is covariant in both type parameters.

Nothing happens when you construct a stream or chain transformations. Execution only begins when you drive a terminal operation. Cross-platform terminals (`runAsync`, `runCollectAsync`, `headAsync`, `countAsync`, etc.) return `Async[Either[E, Z]]`; plain blocking terminals return `Either[E, Z]` on the JVM:

- `Left(e)` -- a typed stream error
- `Right(z)` -- the successful result

Untyped defects (unexpected exceptions) propagate as thrown exceptions, not as `Left` values.

```scala

// This does nothing -- it's just a description
val description: Stream[Nothing, Int] =
  Stream.range(0, 1_000_000)
    .filter(_ % 7 == 0)
    .map(_ * 2)
    .take(100)

// Only this line executes the pipeline
// val result = description.runCollect
```

Streams render their pipeline structure as a human-readable string:

```scala
val s = Stream.range(0, 100).map(_ + 1).filter(_ > 50).take(10)
println(s)  // Stream.range(0, 100).map(...).filter(...).take(10)
```

This makes debugging and logging straightforward -- you can see exactly what transformations a stream applies without running it.

---

### 2) `Sink[E, A, Z]` -- a Consumer

A `Sink[+E, -A, +Z]` consumes elements of type `A` from a stream and produces a final result of type `Z`. Sinks are passed to `Stream.run`:

```scala

val streamSinks = Stream.range(1, 101)

// Built-in sinks
val total = streamSinks.run(Sink.count)
val items = streamSinks.run(Sink.collectAll)
val sum   = streamSinks.run(Sink.sumInt)
val first = streamSinks.run(Sink.head)
```

Most sinks also have convenience methods directly on `Stream`:

```scala
stream.count       // Either[Nothing, Long]
stream.runCollect  // Either[Nothing, Chunk[Int]]
stream.head        // Either[Nothing, Option[Int]]
stream.last        // Either[Nothing, Option[Int]]
```

Sinks compose with `contramap` (pre-process input) and `map` (post-process result):

```scala
val lengthSink: Sink[Nothing, String, Long] =
  Sink.sumInt.contramap[Int, String](_.length)

val doubled: Sink[Nothing, Int, Long] =
  Sink.sumInt.map(_ * 2)
```

---

### 3) `Pipeline[In, Out]` -- Reusable Transformation

A `Pipeline[-In, +Out]` is a reusable stream transformation. It decouples the transformation logic from any specific stream, so you can define it once and apply it many times.

```scala
// Define a reusable pipeline
val normalize: Pipeline[Int, Double] =
  Pipeline.filter[Int](_ > 0)
    .andThen(Pipeline.map[Int, Double](_.toDouble / 100.0))

// Apply to different streams
val result1 = Stream.range(-10, 10).via(normalize).runCollect
val result2 = Stream.fromIterable(List(42, -5, 100, 0)).via(normalize).runCollect
```

Pipelines compose with `andThen`:

```scala
val step1: Pipeline[String, Int] =
  Pipeline.map[String, Int](_.length)

val step2: Pipeline[Int, Int] =
  Pipeline.filter[Int](_ > 3)

val combined: Pipeline[String, Int] =
  step1.andThen(step2)
```

You can also apply a pipeline to a sink with `andThenSink` / `applyToSink`, which pre-processes the sink's input:

```scala
val countLong: Sink[Nothing, String, Long] =
  Pipeline.map[String, Int](_.length)
    .andThenSink(Sink.sumInt)
```

## Synchronous and Asynchronous Execution

There is one `Stream` type. It serves both execution modes, and there is no mode type parameter, no
`AsyncStream`, and no annotation to write.

The type that decides is [`Reader`](./primitives/reader.md), not `Stream`. Materializing a stream yields either
a `Reader.SyncReader[A]`, whose pulls return values directly, or a `Reader.AsyncReader[A]`, whose
pulls return `Async` values. A pipeline that is synchronous end to end materializes as the former; a
single asynchronous source or operator anywhere in it makes the whole pipeline asynchronous.

The `*Async` terminals -- `runAsync`, `runCollectAsync`, `runDrainAsync`, `runFoldAsync`,
`countAsync`, `headAsync`, and their peers -- are the cross-platform API. They all return
`Async[Either[E, Z]]`: the typed error `E` stays inside the `Either`, while the outer `Async` fails
only on a defect. They drive a synchronous pipeline just as correctly as an asynchronous one, so
shared JVM/Scala.js code can use them unconditionally.

The blocking terminals -- `run`, `runCollect`, `runDrain`, `runFold`, `count`, `head`, and their
peers -- are **JVM-only** compatibility twins returning a bare `Either[E, Z]`. They do not exist on
Scala.js, and cross-compiled sources cannot call them.

- [Async Execution](./execution-and-compatibility/async-execution.md) -- the full execution model: classification, the `Reader`
  union, the async source constructors, operators, and terminals.
- [Platform Differences](./execution-and-compatibility/platform-differences.md) -- the availability matrix of every member that
  differs between the JVM and Scala.js.

## Error Handling

Streams distinguish between two kinds of failures:

- **Typed errors** (`E`) — domain errors you expect and handle, returned as `Left` in the result. Use `catchAll`, `orElse`, or `mapError` to recover.
- **Defects** (`Throwable`) — unexpected exceptions from bugs or system failures. Use `catchDefect` to recover, or they propagate as thrown exceptions.

```scala
val failing: Stream[String, Int] =
  Stream.fromIterable(List(1, 2, 3)) ++ Stream.fail("oops") ++ Stream.fromIterable(List(4, 5))

val recovered = failing.catchAll(_ => Stream.fromIterable(List(99)))
recovered.runCollect  // Right(Chunk(1, 2, 3, 99))
```

## Resource Management

Streams integrate with [`zio.blocks.scope.Scope`](../resource-management/scope.md) for deterministic resource cleanup. The `fromAcquireRelease` constructor guarantees that a resource is acquired lazily (when the stream runs), used to produce elements, and then released — even if the stream is short-circuited early via `take()`, fails with an error, or succeeds normally. The release function is wired into a finally block, ensuring cleanup always happens.

```scala

val managed = Stream.fromAcquireRelease(
  acquire = scala.io.Source.fromFile("data.txt"),
  release = _.close()
)(source => Stream.fromIterator(source.getLines()))

managed.take(10).runCollect
// File is closed in finally block regardless of outcome
```

This eliminates the need for manual try/finally when working with resources — the stream handles it for you.

## Primitive Specialization

ZB Streams carries the JVM representation of the element type through the whole pipeline, so a stream of primitives is not boxed at each stage boundary. Specialization is not limited to `Int`, `Long`, `Float`, and `Double`: there are **nine logical lanes** -- the eight primitive pull identities `Boolean`, `Byte`, `Short`, `Char`, `Int`, `Long`, `Float`, and `Double`, plus the reference fallback -- and the synchronous interpreter compacts them into **five storage lanes**: int-like (`Boolean`/`Byte`/`Short`/`Char`/`Int`), `Long`, `Float`, `Double`, and reference. Nine logical lanes therefore does not mean nine interpreter arrays; the operator tag selects the identity-specific reads over the shared storage.

[Zero-Boxing Streams](./execution-and-compatibility/zero-boxing.md) explains how a lane is chosen and what the specialization evidence is for.

```scala

// This entire pipeline runs with ZERO boxing of the Int elements.
// Every step uses specialized readInt/writeInt internally.
val sum: Either[Nothing, Long] =
  Stream.range(0, 1_000_000)   // Int-specialized source
    .filter(_ % 2 == 0)        // Int-specialized filter
    .map(_ * 3)                 // Int->Int specialized map
    .runFold(0L)(_ + _)         // Long-specialized accumulator
```

This matters most for numeric workloads — data processing, statistics, encoding/decoding — where millions of elements flow through multi-stage pipelines.

## Practical Guidance

- **Start with `Stream` constructors and terminal operations.** You can get very far with `Stream.range`, `Stream.fromIterable`, `.map`, `.filter`, and `.runCollect`.
- **Use `Either` pattern matching** to handle the result: `Right(value)` for success, `Left(error)` for typed failures.
- **Prefer `Stream.fromAcquireRelease`** when wrapping resources (files, connections, etc.) over manual try/finally. It guarantees cleanup even on early termination via `take`, `head`, or error.
- **Use the auto-closing I/O constructors** (`fromInputStream`, `fromJavaReader`, `NioStreams.fromChannel`) by default. Only use the `Unmanaged` variants when you need to borrow a resource whose lifetime is managed elsewhere.
- **Use `Pipeline`** when you have a transformation you want to reuse across multiple streams or apply to sinks.
- **Use `&&` for zipping** instead of manual zip calls. Tuples flatten automatically: `a && b && c` produces `(A, B, C)` not `((A, B), C)`.
- **Leverage primitive specialization** for numeric workloads. Streams of any primitive element type avoid boxing automatically on the synchronous path; use `Sink.sumInt`, `runFold(0)(_ + _)`, etc. for allocation-free folds.
- **Use `scan` for running accumulators**, `grouped` for batching, and `sliding` for windowed computations.
- **Use `render`/`toString`** to inspect pipeline structure during debugging — it shows each transformation stage without executing the stream.
- **Use `Sink.create`** (JVM only) as an escape hatch when none of the built-in sinks fit; on Scala.js and in cross-compiled code use `Sink.createAsync` or `Sink.createBoth`.
- **`suspend`** is your friend for recursive or self-referential stream definitions, preventing stack overflow during construction.
- **Typed errors vs. defects**: use `Stream.fail` for expected domain errors and `Stream.die` for programmer errors. Use `catchAll` for the former, `catchDefect` for the latter.

## Usage Examples

This section shows practical examples of using streams in real-world scenarios. Each subsection demonstrates a different aspect of the API with runnable code examples.

### Creating Streams

Here are the most common ways to construct a stream. Choose the constructor that best fits your data source:

```scala

// From explicit elements
Stream.fromIterable(List(1, 2, 3))           // Stream[Nothing, Int]
Stream.fromIterable(List("a", "b", "c"))     // Stream[Nothing, String]

// From collections
Stream.fromChunk(Chunk(1, 2, 3))             // Stream[Nothing, Int]
Stream.fromIterable(List("x", "y", "z"))     // Stream[Nothing, String]
Stream.fromIterator(Iterator.from(1))        // Stream[Nothing, Int] (lazy)

// Ranges
Stream.range(0, 100)                         // 0 to 99
Stream.fromRange(1 to 50)                    // 1 to 50

// Single values (primitive-specialized)
Stream.succeed(42)                           // Stream[Nothing, Int]
Stream.succeed(3.14)                         // Stream[Nothing, Double]
Stream.succeed("hello")                      // Stream[Nothing, String]

// Special streams
Stream.empty                                 // Stream[Nothing, Nothing]
Stream.fail("error")                         // Stream[String, Nothing]
// Stream.die(new Exception("defect"))       // throws on evaluation

// Generators
Stream.repeat(1)                             // infinite stream of 1s
Stream.unfold(0)(n =>                        // 0, 1, 2, ..., 9
  if n < 10 then Some((n, n + 1)) else None
)

// Side-effects
Stream.eval(println("hello"))                   // prints, emits nothing
Stream.attempt(someFallibleCall())              // captures exceptions as typed errors
Stream.attemptEval(riskyEffect())               // same, for Unit-returning effects

// Deferred construction (useful for recursion)
Stream.suspend(expensiveStreamBuilder())

// I/O sources (auto-closing)
Stream.fromInputStream(inputStream)             // Stream[IOException, Byte] (auto-closes)
Stream.fromJavaReader(javaReader)               // Stream[IOException, Char] (auto-closes)

// I/O sources (borrowing -- caller manages lifetime)
Stream.fromInputStreamUnmanaged(inputStream)    // Stream[IOException, Byte] (does NOT close)
Stream.fromJavaReaderUnmanaged(javaReader)      // Stream[IOException, Char] (does NOT close)
```

---

### Transforming Streams

Streams support many transformation operations. Use `map` for element-wise changes, `filter` for selection, and `flatMap` for expanding elements into sub-streams. See the [Stream reference](./core/stream.md) page for comprehensive examples of all transformation methods including `map`, `filter`, `flatMap`, `collect`, `scan`, `mapAccum`, `distinct`, `intersperse`, and more.

---

### Zipping Streams with `&&`

The `&&` operator zips two streams element-by-element into tuples. The resulting stream ends when either input is exhausted.

```scala

val names: Stream[Nothing, String] = Stream.fromIterable(List("Alice", "Bob", "Charlie"))
val ages:  Stream[Nothing, Int]    = Stream.fromIterable(List(30, 25, 35))
val ids:   Stream[Nothing, Long]   = Stream.fromIterable(List(1L, 2L, 3L))

// Two-way zip
val pairs = names && ages
pairs.runCollect // Right(Chunk(("Alice", 30), ("Bob", 25), ("Charlie", 35)))

// Three-way zip -- tuples flatten automatically
val triples = names && ages && ids
triples.runCollect // Right(Chunk(("Alice", 30, 1L), ("Bob", 25, 2L), ("Charlie", 35, 3L)))
```

When the error types differ, they widen through `Concat` — to a union `E1 | E2` on Scala 3, and to a meaningful least upper bound (or `Either[E1, E2]` for disjoint types) on Scala 2.13:

```scala

sealed trait MyError
val s1: Stream[MyError, Int]  = Stream.fromIterable(List(1, 2, 3))
sealed trait OtherError
val s2: Stream[OtherError, Int] = Stream.fromIterable(List(4, 5, 6))
// val zipped = s1 && s2
```

---

### Primitive Specialization

All nine logical lanes are specialized, not only `Int`, `Long`, `Float`, and `Double`. Every intermediate step uses the identity-specific read (`readInt`, `readByte`, `readChar`, and so on), so no `java.lang.Integer` wrappers are allocated between stages.

```scala
// This entire pipeline runs with ZERO boxing of the Int elements.
// Every step uses specialized readInt/writeInt internally.
val sum: Either[Nothing, Long] =
  Stream.range(0, 1_000_000)   // Int-specialized source
    .filter(_ % 2 == 0)        // Int-specialized filter
    .map(_ * 3)                 // Int->Int specialized map
    .runFold(0L)(_ + _)         // Long-specialized accumulator
```

This matters most for numeric workloads -- data processing, statistics, encoding/decoding -- where millions of elements flow through multi-stage pipelines.

---

### Consuming Streams

Terminal operations run the stream and produce a final result. Use `runCollect` to gather all elements, `runDrain` to discard them, or specialized operations like `head`, `count`, and `foldLeft`:

```scala
val s = Stream.range(1, 11) // 1 to 10

// Collect all elements
s.runCollect              // Right(Chunk(1, 2, 3, ..., 10))

// Discard all elements (run for side-effects only)
s.tapEach(println).runDrain

// Fold
s.runFold(0)(_ + _)       // Right(55)              (Int accumulator)
s.runFold(0L)(_ + _)      // Right(55L)             (Long accumulator)
s.runFold(0.0)(_ + _)     // Right(55.0)            (Double accumulator)

// Foreach
s.runForeach(n => println(n))
s.foreach(n => println(n))  // alias

// Aggregates
s.count                   // Right(10L)
s.head                    // Right(Some(1))
s.last                    // Right(Some(10))
s.exists(_ > 5)           // Right(true)
s.forall(_ > 0)           // Right(true)
s.find(_ > 7)             // Right(Some(8))

// Run with an explicit Sink
s.run(Sink.sumInt)        // Right(55L)
s.run(Sink.take(3))       // Right(Chunk(1, 2, 3))
```

---

### Async Execution

`runCollectAsync` is the cross-platform twin of `runCollect`. It returns `Async[Either[E, Chunk[A]]]`
rather than `Either[E, Chunk[A]]`, so it never blocks and compiles on both the JVM and Scala.js:

```scala

val evens: Stream[Nothing, Int] =
  Stream.range(1, 100).filter(_ % 2 == 0).map(_ * 3)

// Still a description -- nothing has run.
val pending: Async[Either[Nothing, Chunk[Int]]] = evens.take(5).runCollectAsync

// Stay in Async: map the result rather than extracting it.
val described: Async[String] = pending.map {
  case Right(values) => s"collected ${values.length} elements"
  case Left(error)   => s"failed: $error"
}
```

Something has to drive the `Async` at the edge of the world. On the JVM that is `.block`, which
belongs in `main` or a test and nowhere else:

```scala

// JVM only: Async#block throws on Scala.js.
val result: Either[Nothing, Chunk[Int]] =
  Stream.range(1, 100).filter(_ % 2 == 0).take(5).runCollectAsync.block
// Right(Chunk(2, 4, 6, 8, 10))
```

Scala.js code keeps the `Async` and hands it to the host runtime instead. See
[Async Execution](./execution-and-compatibility/async-execution.md) for the full terminal family and
[Platform Differences](./execution-and-compatibility/platform-differences.md) for what is available where.

---

### Error Handling Patterns

Streams support two types of failures: typed errors that you can handle explicitly, and defects (exceptions) that propagate. Here are common patterns for dealing with both:

```scala
// Typed error: appears in Either
val result = Stream.fail("not found").runCollect
// result: Left("not found")

// Recover and continue
val safe =
  Stream.fromIterable(List(1, 2)) ++ Stream.fail("oops") ++ Stream.fromIterable(List(3))
val recovered = safe.catchAll(_ => Stream.fromIterable(List(99))).runCollect
// Right(Chunk(1, 2, 99))

// Transform error type by catching and converting
val inputError: Stream[String, Int] = Stream.fail("bad input")
val transformed = inputError.catchAll(msg => Stream.fail(new IllegalArgumentException(msg)))

// Fallback stream
val primary: Stream[String, Int] = Stream.fail("down")
val backup:  Stream[String, Int] = Stream.fromIterable(List(1, 2, 3))
val result2 = (primary || backup).runCollect
// Right(Chunk(1, 2, 3))

// Catch defects (unexpected exceptions)
val risky: Stream[Nothing, Int] =
  Stream.fromIterable(List(1, 2, 3)).map { n =>
    if n == 2 then throw new ArithmeticException("boom")
    else n
  }

val handled = risky.catchDefect {
  case _: ArithmeticException => Stream.fromIterable(List(0))
}.runCollect
// Right(Chunk(1, 0))
```

---

### Resource Safety Patterns

When working with files, network connections, or other resources, use the resource-safe constructors to guarantee cleanup. Here are the most common patterns:

```scala

// Bracket pattern: acquire/use/release
def fileLines(path: String): Stream[Nothing, String] =
  Stream.fromAcquireRelease(
    acquire = scala.io.Source.fromFile(path),
    release = _.close()
  ) { source =>
    Stream.fromIterable(source.getLines().toList)
  }

// Compose resource-safe streams -- both resources are released
val merged =
  fileLines("input1.txt") ++ fileLines("input2.txt")

// Only reads 10 lines; both files are still closed properly
merged.take(10).runCollect

// ensuring: attach a finalizer
var cleaned = false
Stream.range(1, 6)
  .ensuring { cleaned = true }
  .take(2)
  .runDrain
// cleaned == true, even though only 2 of 5 elements were consumed

// defer: register cleanup that runs on stream close
val withDefer =
  Stream.defer(println("releasing lock")) ++
  Stream.range(1, 100)
```

---

### NIO Integration (JVM Only)

On the JVM, `NioStreams` and `NioSinks` provide zero-copy integration with `java.nio` buffers and channels.

#### `NioStreams` -- Creating Streams From NIO Sources

```scala

// From a ByteBuffer
val buf = ByteBuffer.wrap(Array[Byte](1, 2, 3, 4, 5))
NioStreams.fromByteBuffer(buf).runCollect
// Right(Chunk(1, 2, 3, 4, 5))

// Typed buffer views (zero-boxing)
val intBuf = ByteBuffer.allocate(16).putInt(1).putInt(2).putInt(3).putInt(4).flip()
NioStreams.fromByteBufferInt(intBuf).runCollect
// Right(Chunk(1, 2, 3, 4))

// Similarly: fromByteBufferLong, fromByteBufferFloat, fromByteBufferDouble

// From a ReadableByteChannel (auto-closing)
val ch = FileChannel.open(Paths.get("data.bin"), StandardOpenOption.READ)
val bytes = NioStreams.fromChannel(ch, bufSize = 4096).runCollect
// ch is closed automatically when the stream completes

// From a ReadableByteChannel (borrowing -- caller manages lifetime)
val ch2 = FileChannel.open(Paths.get("data.bin"), StandardOpenOption.READ)
val bytes2 = NioStreams.fromChannelUnmanaged(ch2, bufSize = 4096).runCollect
ch2.close() // caller is responsible for closing
```

#### `NioSinks` -- Writing to NIO Targets

```scala

// Write to a ByteBuffer using a typed sink (Int values, zero-boxing)
val outBuf = ByteBuffer.allocate(1024)
Stream.range(1, 5).run(NioSinks.fromByteBufferInt(outBuf))

// Write to a WritableByteChannel using a stream of bytes
val tempPath = Files.createTempFile("zio-blocks-streams-", ".bin")
val outCh = FileChannel.open(
  tempPath,
  StandardOpenOption.WRITE,
  StandardOpenOption.TRUNCATE_EXISTING
)
val bytes = Chunk.fromIterable(List[Byte](1, 2, 3, 4, 5))
try Stream.fromChunk(bytes).run(NioSinks.fromChannel(outCh))
finally {
  outCh.close()
  Files.deleteIfExists(tempPath)
}
```

---

### Pipeline Composition

Pipelines are composable transformations that can be reused across different streams. Build complex transformations by chaining pipelines together with `andThen`:

```scala

// Build reusable transformation steps
val parseInts: Pipeline[String, Int] =
  Pipeline.collect[String, Int] {
    case s if s.matches("-?\\d+") => s.toInt
  }

val positiveOnly: Pipeline[Int, Int] =
  Pipeline.filter[Int](_ > 0)

val doubled: Pipeline[Int, Int] =
  Pipeline.map[Int, Int](_ * 2)

// Compose into a single pipeline
val fullPipeline: Pipeline[String, Int] =
  parseInts
    .andThen(positiveOnly)
    .andThen(doubled)

// Apply to any stream of strings
Stream.fromIterable(List("10", "abc", "-3", "7", "0", "25"))
  .via(fullPipeline)
  .runCollect
// Right(Chunk(20, 14, 50))

// Apply to a sink (pre-process the sink's input)
val sumPositiveDoubled: Sink[Nothing, String, Long] =
  fullPipeline.andThenSink(Sink.sumInt)

Stream.fromIterable(List("10", "abc", "-3", "7", "0", "25"))
  .run(sumPositiveDoubled)
// Right(84L)
```

## See Also

- [Async Execution](./execution-and-compatibility/async-execution.md) -- the synchronous/asynchronous execution model, the `Reader` union, and the `*Async` terminal family
- [Platform Differences](./execution-and-compatibility/platform-differences.md) -- which members exist on the JVM, on Scala.js, and on both
- [Zero-Boxing Streams](./execution-and-compatibility/zero-boxing.md) -- the primitive lanes and how one is chosen
- [Async](../async.md) -- the `Async` effect type the cross-platform terminals return
- [Mux](../mux.mdx) -- coordinating many keyed streams over one shared transport
- [Your First Streaming Pipeline](../../guides/your-first-streaming-pipeline.md) -- a newcomer tutorial: read a sales log, parse, filter and batch it, sum it with a `Sink`, and recover from a bad line
- [How to Stream Files and Sockets Without Leaking Resources](../../guides/stream-files-and-sockets.md) -- read files, NIO channels and sockets as streams that close their handles on every exit path
- [Coming from fs2, ZIO Streams or Ox](../../guides/streaming-library-migration.md) -- a guide that maps fs2, ZIO Streams and Ox code onto ZIO Blocks streams
- [How to Use ZIO Blocks in a ZIO or cats-effect Application](../../guides/use-zio-blocks-in-zio-or-cats-effect.md) -- calling streams from ZIO or cats-effect, with cancellation carried across

---

## Low-Level Primitives

`Reader` and `Writer` are the live, stateful objects a pipeline compiles into and runs on. Where the [Core Types](../core/index.md) are inert descriptions, these hold real resources — an open file, a socket, a position in a buffer — until closed.

A terminal such as `stream.runAsync(sink)` compiles to and drains a `Reader` for you. Work with these types directly only when writing a custom source or sink, wrapping a native I/O API, or driving a compiled stream by hand.

## Reader

[`Reader[+Elem]`](./reader.md) is the pull side: the cursor a `Stream` compiles into. It has two kinds — `Reader.SyncReader[Elem]`, whose pulls return directly, and `Reader.AsyncReader[Elem]`, whose pulls return `Async`. An asynchronous reader is single-consumer, with exactly one owner responsible for awaiting its `close()`.

## Writer

[`Writer[-Elem]`](./writer.md) is the push side: a destination you feed elements into one at a time until it closes or fills. Reach for it when adapting an `OutputStream`-shaped API, or implementing a sink that a producer outside the pipeline writes into.

## See Also

- [Core Types](../core/index.md) — the descriptions that compile into these primitives
- [Streams Reference](../index.md) — module overview
- [Asynchronous Stream Execution](../execution-and-compatibility/async-execution.md) — how a graph decides between a `SyncReader` and an `AsyncReader`

---

## Reader

`Reader[+Elem]` is the **pull-based source that powers ZIO Blocks streams**. When you call a terminal operation like `stream.run(sink)`, the stream compiles into a `Reader`, which yields values one at a time on demand until closed.

`Reader` has two library-provided kinds, `Reader.SyncReader[Elem]` and `Reader.AsyncReader[Elem]`. A synchronous reader's `read` and `close` return directly; an asynchronous reader's pull and lifecycle methods return `Async`. Most users never interact with either kind directly, but understanding them clarifies how streams work internally.

The compilation and execution flow:

```
Stream[E, A] ──(compile)──> Reader[A]
                              │
                              └─(drain via Sink)──> Either[E, Z]
```

`Reader`:
- Is lazy and pull-based — `Stream` transformations don't run until `read()` is called, running in constant space one element at a time
- Is a single-consumer cursor — do not share a `SyncReader` between threads or overlap operations on an `AsyncReader`
- Uses a sentinel protocol where callers specify the end-of-stream value; all eight JVM primitives have exact physical pull methods: `readBoolean`, `readByte`, `readChar`, `readShort`, `readInt`, `readLong`, `readFloat`, and `readDouble`. The `Long` and `Double` lanes are the exception: no sentinel is safe there, so they detect end of stream by the count returned from `readLongs` / `readDoubles`.
- Dispatches on `Reader#jvmType`, which describes the reader's physical representation and therefore the exact pull method it supports, not merely the static or logical element type
- Is the compilation target of `Stream` — when a stream runs, it becomes a `Reader`
- Transfers lifecycle responsibility explicitly: terminals and bracketed APIs close their owned reader, while callers of `startAsync` own the returned reader and must await `close()`
- Supports composition by chaining readers through transformations without materializing intermediate data

Here is the core `Reader` interface with the most essential methods:

```scala
abstract class Reader[+Elem]

abstract class Reader.SyncReader[+Elem] extends Reader[Elem] {
  def read[A >: Elem](sentinel: A): A
  def readAll[A >: Elem](): Chunk[A]
  def readN[A >: Elem](n: Int): Chunk[A]
  def readUpToN[A >: Elem](n: Int): Chunk[A]
  def isClosed: Boolean
  def readable(): Boolean
  def close(): Unit
  def toAsync: Reader.AsyncReader[Elem]
}

abstract class Reader.AsyncReader[+Elem] extends Reader[Elem] {
  def read[A >: Elem](sentinel: A): Async[A]
  def readAll[A >: Elem](): Async[Chunk[A]]
  def readN[A >: Elem](n: Int): Async[Chunk[A]]
  def readUpToN[A >: Elem](n: Int): Async[Chunk[A]]
  def isClosed: Async[Boolean]
  def readable(): Async[Boolean]
  def close(): Async[Unit]
  // JVM only: def toSync: Reader.SyncReader[Elem]
}
```

The root contains only kind-independent composition and metadata (`++`, `concat`, `concatAsync`, `withReleaseAsync`, and `jvmType`); it cannot be pulled, queried, or closed directly. Those operations belong to one of the two reader kinds, described in [The reader union](#the-reader-union). Every primitive, bulk, lifecycle, and pushdown method on `AsyncReader` has the same parameters as its `SyncReader` counterpart but returns its result in `Async`; [Asynchronous reading](#asynchronous-reading) is the full member list. The eight physical primitive methods are `readBoolean`, `readByte`, `readChar`, `readShort`, `readInt`, `readLong`, `readFloat`, and `readDouble`; a primitive `jvmType` is a contract that the corresponding method works, even when covariance has widened the reader's static element type.

An `AsyncReader` permits one active operation at a time, and closing it is the owner's responsibility. `SyncReader#toAsync` and the JVM-only `AsyncReader#toSync` move a reader between the two kinds without copying it.

## Quick Showcase

Here's how to create and drain a `Reader`:

```scala

val r = Reader.fromChunk(Chunk(1, 2, 3, 4, 5))
// r: SyncReader[Int] = zio.blocks.streams.io.Reader$FromChunkInt@1ba786f4
val collected = Buffer[Int]()
// collected: Buffer[Int] = ArrayBuffer(1, 2, 3, 4, 5)

// Pull elements until sentinel
def drainAll(): Unit = {
  val elem = r.read(-1)
  if (elem != -1) {
    collected += elem
    drainAll()
  }
}
drainAll()

println(s"Collected: $collected")
// Collected: ArrayBuffer(1, 2, 3, 4, 5)
```

## Motivation

Imagine you're processing a massive CSV file—millions of rows of customer data. Your first instinct is to load it all into memory as a `List[Row]`, transform it, filter it, and then write the results. This works fine for small files, but one day someone feeds you a 50GB dataset and your application crashes with `OutOfMemoryError`. You've hit the fundamental problem of eager evaluation: **you must load everything before you can do anything**, and if the data is bigger than available memory, you're stuck.

Even if you manage to fit the data in memory, you've paid the startup cost upfront. If your pipeline only needs the first 100 rows to produce a result, you've wasted time and energy materializing the other millions. And if something fails partway through—a database connection drops, a file is corrupted—you've already consumed resources and may have inconsistencies to clean up.

The streaming intuition is different: instead of pulling all data at once, what if the consumer asked the producer "give me the next element?" one at a time? This way, you never hold more than one element in memory, you only do work on elements you actually use, and you can stop immediately when you have enough.

`Reader` embodies this pull-based philosophy. Rather than materializing a `List`, a `Stream` compiles down to a `Reader`—a stateful object that produces one element each time you call `read()`. The consumer (a `Sink`) drives the pace: it calls `read()` when ready, and the `Reader` computes and returns the next value. When the stream is exhausted, `Reader` returns a sentinel—a special value you provide—signaling "no more data." No exceptions, no null, no boxing overhead.

`Reader` shines when you're processing large, unbounded, or expensive-to-produce data sources: database result sets, network streams, log files, sensor data, or any pipeline where memory or time efficiency matters. Instead of hoping your data fits in memory, you pay a constant, predictable cost per element.

## The Reader Union

`Reader[+Elem]` is the root of two kinds. It declares composition and one piece of metadata, and nothing that pulls, queries, or closes:

```scala
abstract class Reader[+Elem] {
  def ++[Elem2 >: Elem](next: => Reader[Elem2]): Reader[Elem2]
  def concat[Elem2 >: Elem](next: () => Reader[Elem2]): Reader[Elem2]
  def concatAsync[Elem2 >: Elem](next: () => Async[Reader[Elem2]]): Reader.AsyncReader[Elem2]
  def withReleaseAsync(release: () => Async[Unit]): Reader.AsyncReader[Elem]
  def jvmType: JvmType = JvmType.AnyRef
}
```

A value typed `Reader[A]` can be concatenated and can report its physical lane, and that is all. To read from it you must know its kind:

- `Reader.SyncReader[Elem]` holds the direct pull API — `read`, `readAll`, `readN`, `readUpToN`, the eight primitive pulls, the array transfers, `skip`, the pushdown operations, `isClosed`, `readable()`, and `close()`, each returning its result immediately.
- `Reader.AsyncReader[Elem]` mirrors that surface method for method, with every result wrapped in `Async`.

The two kinds line up one to one, so a signature written against one translates mechanically to the other:

| Member                                     | `SyncReader` result | `AsyncReader` result |
|--------------------------------------------|---------------------|----------------------|
| `read(sentinel)`                           | `A`                 | `Async[A]`           |
| `readAll()`                                | `Chunk[A]`          | `Async[Chunk[A]]`    |
| `readN(n)`, `readUpToN(n)`                 | `Chunk[A]`          | `Async[Chunk[A]]`    |
| `readInt(_sentinel)`                       | `Long`              | `Async[Long]`        |
| `readBytes(dest, offset, length)`          | `Int`               | `Async[Int]`         |
| `isClosed`                                 | `Boolean`           | `Async[Boolean]`     |
| `readable()`                               | `Boolean`           | `Async[Boolean]`     |
| `close()`                                  | `Unit`              | `Async[Unit]`        |
| `skip(n)`                                  | `Unit`              | `Async[Unit]`        |
| `reset()`                                  | `Unit`              | `Async[Unit]`        |
| `setLimit(n)`, `setRepeat()`, `setSkip(n)` | `Boolean`           | `Async[Boolean]`     |

Which kind a stream materializes as is decided once, when the graph compiles: a fully synchronous graph produces a `SyncReader`, and a graph with any asynchronous node produces an `AsyncReader`. See [Asynchronous Stream Execution](../execution-and-compatibility/async-execution.md#one-stream-type-two-execution-modes) for that decision.

One caveat about the root: `Reader` is declared `abstract class Reader[+Elem]`, not `sealed`. Every reader the library hands you is a `SyncReader` or an `AsyncReader`, and code may rely on that in practice — what it cannot rely on is the compiler proving a `match` over the two kinds exhaustive, so write such a match with a fallback case.

### Mixed-kind Composition

Concatenation keeps both kinds usable through the same `++` and `concat` names. `SyncReader` adds a pair of overloads that are narrowed to synchronous arguments and disambiguated from the inherited ones by a `DummyImplicit` parameter:

```scala
abstract class Reader.SyncReader[+Elem] extends Reader[Elem] {
  final def ++[Elem2 >: Elem](next: => SyncReader[Elem2])(implicit dummy: DummyImplicit): SyncReader[Elem2]
  final def concat[Elem2 >: Elem](next: () => SyncReader[Elem2])(implicit dummy: DummyImplicit): SyncReader[Elem2]

  final override def ++[Elem2 >: Elem](next: => Reader[Elem2]): AsyncReader[Elem2]
  final override def concat[Elem2 >: Elem](next: () => Reader[Elem2]): AsyncReader[Elem2]
}
```

The rule that falls out is simple: **synchronous plus synchronous stays synchronous; every other combination widens to `AsyncReader`**. When the widening overload is chosen, the synchronous side is adapted with `toAsync` and the pair is concatenated on the asynchronous path.

```scala

val sync: Reader.SyncReader[Int]   = Reader.fromChunk(Chunk(1, 2))
val async: Reader.AsyncReader[Int] = Reader.fromChunk(Chunk(3, 4)).toAsync

val syncSync: Reader.SyncReader[Int]    = sync ++ Reader.fromChunk(Chunk(5, 6))
val syncAsync: Reader.AsyncReader[Int]  = sync ++ async
val asyncSync: Reader.AsyncReader[Int]  = async ++ Reader.fromChunk(Chunk(7, 8))
val asyncAsync: Reader.AsyncReader[Int] = async ++ async
```

Because the overloads are selected on the *static* type of the argument, a value already widened to `Reader[Int]` picks the widening overload even when it happens to hold a `SyncReader` at runtime. Keep the narrow type if you want to stay on the synchronous path.

`concatAsync` and `withReleaseAsync` are declared on the root and always produce an `AsyncReader`, whichever kind they are called on — the first because the next reader arrives inside an `Async`, the second because the release action does:

```scala
abstract class Reader[+Elem] {
  def concatAsync[Elem2 >: Elem](next: () => Async[Reader[Elem2]]): Reader.AsyncReader[Elem2]
  def withReleaseAsync(release: () => Async[Unit]): Reader.AsyncReader[Elem]
}
```

### Converting Between Kinds

Two adapters move a reader across the split:

```scala
abstract class Reader.SyncReader[+Elem] extends Reader[Elem] {
  final def toAsync: Reader.AsyncReader[Elem]
}

abstract class Reader.AsyncReader[+Elem] extends Reader[Elem] {
  final def toSync: Reader.SyncReader[Elem]   // JVM only
}
```

`toAsync` is available on every platform. `toSync` is supplied by a JVM-only platform trait; on Scala.js that trait is empty, so the method does not exist and shared code cannot call it. [Platform Differences](../execution-and-compatibility/platform-differences.md#availability-matrix) has the full capability split.

Both adapters are lifecycle-preserving views rather than copies. The adapter wraps the original reader, so the two sides share one position and one lifecycle: closing either one closes the underlying source, and consuming through both interleaves pulls on the same cursor. Pick one view and drive the reader through it.

Both also unwrap a round trip instead of stacking. Calling `toAsync` on a reader that is itself the synchronous view of an `AsyncReader` returns that original asynchronous reader, and `toSync` on a synchronous reader's asynchronous view returns the original synchronous one. Converting back and forth therefore costs nothing and never builds a tower of adapters.

`toAsync` is the adapter to reach for when a helper is written against `Reader.AsyncReader` — the kind that compiles on both platforms — and the reader at the call site happens to be synchronous:

```scala

def firstByte(reader: Reader.AsyncReader[Byte]): Async[Int] = reader.readByte()

def firstByteOfSync(reader: Reader.SyncReader[Byte]): Async[Int] = firstByte(reader.toAsync)
```

What it does not do is make blocking work non-blocking. Driving the view still runs the synchronous reader's pulls on the driving thread.

`toSync` runs the other way. Use it at a JVM edge — an `InputStream`-shaped API, a legacy protocol loop — and not in code that cross-builds:

```scala

def firstByteBlocking(reader: Reader.AsyncReader[Byte]): Int = {
  val sync = reader.toSync
  try sync.readByte()
  finally sync.close()
}
```

Unlike `toAsync`, it has hazards that belong at the call site:

:::warning[`toSync` blocks, serializes, and interrupts]
Every pull blocks the calling thread until the asynchronous work settles. Only one pull runs at a time: a second thread entering the view waits until the first pull completes, so the view is a serialization point, not a way to share a reader. Calling `close()` from another thread interrupts the thread parked in a pull — that pull then returns its closed value (`-1`, the sentinel, or an empty chunk) instead of the value it was waiting for. Closing the view also closes the underlying asynchronous reader. After a close, the control operations — `reset()`, `setLimit`, `setRepeat`, `setSkip`, and `skip` — throw `IOException("Reader is closed")`.
:::

The rule to carry away is that `toSync` belongs at JVM edges, not in shared code.

## Construction

Several ways to create a `Reader`, from predefined singletons to collections and I/O sources:

Every companion constructor states its kind in its return type, so you never have to guess which engine a hand-built reader will drive. Factories such as `closed`, `fromChunk`, `fromIterable`, `fromRange`, `single`, `repeat`, and `unfold` are declared to return `SyncReader` — `def fromRange(range: Range): SyncReader[Int]`, and so on. `unfoldAsync` is the one native asynchronous constructor and returns an `AsyncReader`. `repeated` is overloaded three ways and preserves whether its input is synchronous or asynchronous. Composition also preserves asynchronous work: `concatAsync` lazily acquires the next reader and `withReleaseAsync` awaits asynchronous cleanup. Asynchronous children are supported throughout the reader graph.

### Creating Predefined Readers

`Reader.closed` — An already-closed reader that emits no elements. Useful as a base case or for empty streams:

```scala
object Reader {
  def closed: Reader.SyncReader[Nothing]
}
```

Here's how to create and use a closed reader:

```scala

val r = Reader.closed
// r: SyncReader[Nothing] = zio.blocks.streams.io.Reader$ClosedReader$@65813a85
println(r.isClosed)        // true
// true
println(r.read(-1))        // -1 (the sentinel)
// -1
```

### From Collections

`Reader.fromChunk` — Creates a reader backed by a [`Chunk`](../../chunk.md). Dispatches on the element type to use specialized, unboxed reads for primitives:

```scala
object Reader {
  def fromChunk[A](chunk: Chunk[A])(implicit jt: JvmType.Infer[A]): Reader.SyncReader[A]
}
```

Create a reader from a chunk and drain its elements:

```scala

val chunk = Chunk(10, 20, 30)
// chunk: Chunk[Int] = IndexedSeq(10, 20, 30)
val r = Reader.fromChunk(chunk)
// r: SyncReader[Int] = zio.blocks.streams.io.Reader$FromChunkInt@37801393

def drain(): Unit = {
  val v = r.read(-1)
  if (v != -1) {
    println(v)
    drain()
  }
}
drain()
// 10
// 20
// 30
// Output: 10, 20, 30
```

`Reader.fromIterable` — Creates a reader from any `Iterable`. Works with lists, sets, vectors, and other collections:

```scala
object Reader {
  def fromIterable[A](it: Iterable[A])(implicit jt: JvmType.Infer[A]): Reader.SyncReader[A]
}
```

Create a reader from a list and consume its elements:

```scala

val list = List("a", "b", "c")
// list: List[String] = List("a", "b", "c")
val r = Reader.fromIterable(list)
// r: SyncReader[String] = zio.blocks.streams.io.Reader$FromIterable@6488312c

def drain(): Unit = {
  val v = r.read(null)
  if (v != null) {
    println(v)
    drain()
  }
}
drain()
// a
// b
// c
// Output: a, b, c
```

`Reader.fromRange` — Creates a reader from a Scala `Range`. Optimized for integer ranges without allocation:

```scala
object Reader {
  def fromRange(range: Range): Reader.SyncReader[Int]
}
```

Create a reader from a range and drain the integers:

```scala

val r = Reader.fromRange(1 to 5)
// r: SyncReader[Int] = zio.blocks.streams.io.Reader$FromRange@472661a2

def drain(): Unit = {
  val v = r.read(-1)
  if (v != -1) {
    println(v)
    drain()
  }
}
drain()
// 1
// 2
// 3
// 4
// 5
// Output: 1, 2, 3, 4, 5
```

### From I/O

`Reader.fromInputStream` — Wraps a `java.io.InputStream` as a `SyncReader[Byte]`. `readByte()` exposes the unsigned `0`–`255` view and reserves `-1` exclusively for EOF; ordinary element pulls retain `Byte` values:

```scala
object Reader {
  def fromInputStream(is: InputStream): Reader.SyncReader[Byte]
}
```

Closing the reader does not close `is`. An `IOException` from `is` is thrown wrapped in the library's `zio.blocks.streams.internal.StreamError`, whose `value` is the `IOException`, and every later pull throws it again.

`Reader.fromReader` — Wraps a `java.io.Reader` as a `SyncReader[Char]` for character-based I/O:

```scala
object Reader {
  def fromReader(r: java.io.Reader): Reader.SyncReader[Char]
}
```

It follows the same rules as `Reader.fromInputStream`: closing the reader leaves `r` open, and an `IOException` from `r` is thrown wrapped in `StreamError` and again by every later pull.

`NioReaders` is the `java.nio` counterpart, and it is synchronous throughout. It has no asynchronous twins, and the reason is in the type it wraps: `ReadableByteChannel#read` blocks the calling thread. No wrapper can make it non-blocking, so presenting its result as an `AsyncReader` would have promised something the channel cannot deliver. The two objects split by capability — `AsyncNioReaders` for channels that implement the JDK's asynchronous read protocol, `NioReaders` for the blocking ones and for `ByteBuffer`s.

Every `NioReaders` factory states its kind in its return type:

```scala
object NioReaders {
  def fromByteBuffer(buf: ByteBuffer): Reader.SyncReader[Byte]
  def fromByteBufferDouble(buf: ByteBuffer): Reader.SyncReader[Double]
  def fromByteBufferFloat(buf: ByteBuffer): Reader.SyncReader[Float]
  def fromByteBufferInt(buf: ByteBuffer): Reader.SyncReader[Int]
  def fromByteBufferLong(buf: ByteBuffer): Reader.SyncReader[Long]
  def fromChannel(ch: ReadableByteChannel, bufSize: Int = 8192): Reader.SyncReader[Byte]
}
```

These factories return `Reader.SyncReader[Byte]` rather than the root `Reader[Byte]`, which is what makes pulling and closing available on the result: those members belong to the reader kinds, not to the root type. Two details of this object are worth noting at a call site: its channel factory names the buffer parameter `bufSize`, where the asynchronous one names it `bufferSize`, and there is no public unmanaged channel variant here — `NioReaders.fromChannel` always owns the channel it wraps.

`NioReaders.fromChannel` needs a channel in blocking mode. A read that returns no bytes is taken as the end of the data, so over a non-blocking `SocketChannel` or `Pipe.SourceChannel` the reader reports end-of-stream as soon as no bytes are ready, and the data is cut short without an error (a known bug in 0.0.56). An `IOException` from the channel is thrown wrapped in `StreamError`, whose `value` is the `IOException`, and every later pull throws it again.

File bytes are reachable only through this synchronous side: a `java.nio.channels.FileChannel` is a `ReadableByteChannel`, so `NioReaders.fromChannel` accepts it. That reader blocks, and `SyncReader#toAsync` does not change it — the resulting asynchronous reader still blocks the thread that drives it.

### From Native Asynchronous Sources

Each platform ships a small set of factories that turn a native asynchronous byte source into a `Reader.AsyncReader[Byte]`. On the JVM that source is a `java.nio.channels.AsynchronousByteChannel`; on Scala.js it is a Web Streams API `ReadableStream`. Once wrapped, the result is an ordinary asynchronous reader: pull from it by hand, or hand it to `Stream.fromReader` and run the pipeline with a `*Async` terminal.

Six factories exist across the two platforms, all of them returning `Reader.AsyncReader[Byte]` on the `JvmType.Byte` lane. They differ in what they wrap and in who owns the native source once the reader closes:

| Factory                                             | Platform | Native source               | On reader `close()`                        |
|-----------------------------------------------------|----------|-----------------------------|--------------------------------------------|
| `AsyncNioReaders.fromChannel`                       | JVM      | `AsynchronousByteChannel`   | Closes the channel                         |
| `AsyncNioReaders.fromChannelUnmanaged`              | JVM      | `AsynchronousByteChannel`   | Leaves the channel open                    |
| `AsyncNioReaders.fromSocket`                        | JVM      | `AsynchronousSocketChannel` | Closes the socket                          |
| `AsyncNioReaders.fromSocketUnmanaged`               | JVM      | `AsynchronousSocketChannel` | Leaves the socket open                     |
| `ReadableStreamReaders.fromReadableStream`          | Scala.js | `ReadableStream`            | Cancels the stream, then releases the lock |
| `ReadableStreamReaders.fromReadableStreamUnmanaged` | Scala.js | `ReadableStream`            | Releases the lock, never cancels           |

The two adapters follow the same rules, so a cross-platform consumer sees the same behaviour from either one, except for a source failure, where a known bug in 0.0.56 makes a stream over the Scala.js adapter fail its `Async` instead of producing `Left` (see [Scala.js Invariants](#scalajs-invariants)):

| Situation                   | JVM `AsyncNioReaders`                          | Scala.js `ReadableStreamReaders`           |
|-----------------------------|------------------------------------------------|--------------------------------------------|
| A delivery carries no bytes | The buffer is cleared and the read resubmitted | The chunk is skipped and `read()` reissued |
| End of stream               | A negative completion count                    | `done = true` on the read result           |
| A source failure            | `IOException`; a stream gives `Left`           | A rejection; a stream's `Async` fails      |
| Read buffer size            | `bufferSize`, default 8192                     | Not configurable                           |

Neither module is a general native-I/O layer. `AsyncNioReaders` reads from channels that implement the JDK's asynchronous read protocol, and `ReadableStreamReaders` reads from a browser or Node byte stream. Everything else — files on the JVM, non-byte sources on Scala.js — is outside what these factories accept.

#### JVM: `AsyncNioReaders`

`AsyncNioReaders` is the JVM factory object for genuinely non-blocking reads. Its four factories divide along two axes: the type of the native source, and whether the reader owns it.

Both channel factories wrap an `AsynchronousByteChannel` and take the read buffer size as a defaulted second parameter:

```scala
object AsyncNioReaders {
  def fromChannel(channel: AsynchronousByteChannel, bufferSize: Int = 8192): Reader.AsyncReader[Byte]
  def fromChannelUnmanaged(channel: AsynchronousByteChannel, bufferSize: Int = 8192): Reader.AsyncReader[Byte]
}
```

`bufferSize` is the capacity of the single `ByteBuffer` the reader refills from the channel, and it is validated eagerly: a value of zero or less throws `IllegalArgumentException` with the message `requirement failed: bufferSize must be positive` from the factory call itself, not from the first pull.

To lift a channel into a stream and run it with a cross-platform terminal:

```scala

def collect(channel: AsynchronousByteChannel): Async[Either[Nothing, Chunk[Byte]]] =
  Stream.fromReader[Nothing, Byte](AsyncNioReaders.fromChannel(channel, bufferSize = 4096)).runCollectAsync
```

Driving the reader by hand works the same way, and is what you want when the protocol is framed rather than streamed:

```scala

def header(channel: AsynchronousByteChannel): Async[Chunk[Byte]] = {
  val reader: Reader.AsyncReader[Byte] = AsyncNioReaders.fromChannelUnmanaged(channel, bufferSize = 512)
  reader.readN[Byte](16).flatMap(bytes => reader.close().map(_ => bytes))
}
```

The socket pair narrows the parameter type to `AsynchronousSocketChannel`, which is the `AsynchronousByteChannel` most callers actually hold:

```scala
object AsyncNioReaders {
  def fromSocket(socket: AsynchronousSocketChannel, bufferSize: Int = 8192): Reader.AsyncReader[Byte]
  def fromSocketUnmanaged(socket: AsynchronousSocketChannel, bufferSize: Int = 8192): Reader.AsyncReader[Byte]
}
```

There is no behavioural difference to learn: `AsyncNioReaders.fromSocket` delegates to `AsyncNioReaders.fromChannel` and `AsyncNioReaders.fromSocketUnmanaged` to `AsyncNioReaders.fromChannelUnmanaged`, with the same buffer and the same ownership rule. They exist so a socket-shaped call site reads as one.

#### Managed Versus Unmanaged Ownership

Ownership is the whole of the difference between the two variants, and it is decided when you pick the factory, not later.

A managed reader — `AsyncNioReaders.fromChannel` or `AsyncNioReaders.fromSocket` — closes the underlying channel when the reader closes, and only if the channel is still open. If that channel close fails, the failure surfaces from the reader's own `close()` rather than being swallowed. An unmanaged reader releases the reader and nothing else: the channel stays open for whoever owns it, and a reader close is invisible to the rest of the program apart from the read it cancels.

Closing either kind cancels a read that is still in flight. The reader marks itself closed, cancels the underlying channel operation, and settles the pending pull with its end-of-stream answer — `readByte()` returns `-1`, `read(sentinel)` returns the sentinel — so a consumer parked on a pull is released rather than left waiting for a channel that will never answer.

`close()` is idempotent. The first caller performs the work; every later caller awaits the same memoized outcome, and the channel is closed at most once.

All three facts are observable rather than asserted. The example below is a runnable file in the JVM-only `streams-examples` module of the [zio-blocks repository](https://github.com/zio/zio-blocks). It drives a scripted `AsynchronousByteChannel` — one that counts its own `close()` calls and parks a read it cannot serve — through both ownership modes, so the managed close, the untouched unmanaged channel, the memoized second close, and the cancelled pending read are all printed:

```scala title="streams-examples/src/main/scala/nio/AsyncChannelReaderExample.scala" 
package nio

/**
 * Managed and unmanaged readers over an `AsynchronousByteChannel` (JVM only).
 *
 * `AsyncNioReaders.fromChannel` takes ownership of the channel and closes it
 * when the reader closes. `AsyncNioReaders.fromChannelUnmanaged` releases the
 * reader and leaves the channel alive for its owner. Both memoize `close()`,
 * and both settle a read that is still in flight when the reader closes.
 *
 * The channel below is scripted rather than networked: it counts its own
 * `close()` calls and parks a read that has nothing left to deliver, so
 * ownership, idempotent close, and the cancelled pending read are all
 * observable without a socket.
 */
object AsyncChannelReaderExample {
  def main(args: Array[String]): Unit = {
    managedClosesTheChannel()
    unmanagedLeavesTheChannelOpen()
    closeCancelsAPendingRead()
  }

  /** Managed ownership: the reader closes the channel, and only once. */
  private def managedClosesTheChannel(): Unit = {
    val channel = new ScriptedChannel("async".getBytes(UTF_8))
    val reader  = AsyncNioReaders.fromChannel(channel, bufferSize = 16)

    val bytes: Chunk[Byte] = reader.readN[Byte](5).block
    reader.close().block
    reader.close().block // memoized: the channel is not closed a second time

    println(
      s"managed   -> read=${text(bytes)}, channelOpen=${channel.isOpen}, closes=${channel.closeCount.get()}"
    )
  }

  /** Unmanaged ownership: the channel outlives the reader untouched. */
  private def unmanagedLeavesTheChannelOpen(): Unit = {
    val channel = new ScriptedChannel("async".getBytes(UTF_8))
    val reader  = AsyncNioReaders.fromChannelUnmanaged(channel, bufferSize = 16)

    val bytes: Chunk[Byte] = reader.readN[Byte](5).block
    reader.close().block
    reader.close().block

    println(
      s"unmanaged -> read=${text(bytes)}, channelOpen=${channel.isOpen}, closes=${channel.closeCount.get()}"
    )
  }

  /**
   * Closing an unmanaged reader cancels the read the channel is still holding
   * and hands the consumer the end-of-stream answer instead.
   */
  private def closeCancelsAPendingRead(): Unit = {
    val channel = new ScriptedChannel("hi".getBytes(UTF_8))
    val reader  = AsyncNioReaders.fromChannelUnmanaged(channel, bufferSize = 16)

    val delivered: Chunk[Byte] = reader.readN[Byte](2).block
    val pending                = reader.readByte().start

    // Wait until the channel is genuinely holding a read, so the close races an
    // in-flight operation rather than an unstarted one.
    channel.readParked.await()
    reader.close().block

    println(
      s"pending   -> read=${text(delivered)}, cancelledRead=${pending.block}, " +
        s"channelOpen=${channel.isOpen}, closes=${channel.closeCount.get()}"
    )
  }

  private def text(bytes: Chunk[Byte]): String = new String(bytes.toArray, UTF_8)

  /**
   * A channel that delivers `payload` once and then parks every further read
   * until it is closed, the way a live socket waits between packets.
   */
  private final class ScriptedChannel(payload: Array[Byte]) extends AsynchronousByteChannel {
    val closeCount: AtomicInteger  = new AtomicInteger
    val readParked: CountDownLatch = new CountDownLatch(1)

    private val lock                      = new AnyRef
    private var position                  = 0
    private var open                      = true
    private var parked: Throwable => Unit = null

    def read[A](dst: ByteBuffer, attachment: A, handler: CompletionHandler[Integer, ? >: A]): Unit = {
      val served = lock.synchronized {
        val outcome = serve(dst)
        if (outcome.isEmpty) parked = cause => handler.failed(cause, attachment)
        outcome
      }
      served match {
        case Some(count) => handler.completed(count, attachment)
        case None        => readParked.countDown()
      }
    }

    def read(dst: ByteBuffer): Future[Integer] = {
      val future = new CompletableFuture[Integer]
      val served = lock.synchronized {
        val outcome = serve(dst)
        if (outcome.isEmpty) parked = cause => { future.completeExceptionally(cause); () }
        outcome
      }
      served match {
        case Some(count) => future.complete(count)
        case None        => readParked.countDown()
      }
      future
    }

    def write[A](src: ByteBuffer, attachment: A, handler: CompletionHandler[Integer, ? >: A]): Unit =
      handler.failed(new UnsupportedOperationException("scripted channel is read-only"), attachment)

    def write(src: ByteBuffer): Future[Integer] = {
      val future = new CompletableFuture[Integer]
      future.completeExceptionally(new UnsupportedOperationException("scripted channel is read-only"))
      future
    }

    def isOpen: Boolean = lock.synchronized(open)

    def close(): Unit = {
      val release = lock.synchronized {
        closeCount.incrementAndGet()
        open = false
        val current = parked
        parked = null
        current
      }
      if (release ne null) release(new AsynchronousCloseException)
    }

    private def serve(dst: ByteBuffer): Option[Int] = {
      val remaining = payload.length - position
      if (remaining <= 0) None
      else {
        val count = math.min(remaining, dst.remaining)
        dst.put(payload, position, count)
        position += count
        Some(count)
      }
    }
  }
}
```

([source](https://github.com/zio/zio-blocks/blob/main/streams-examples/src/main/scala/nio/AsyncChannelReaderExample.scala))

Run it with:

```bash
cd streams-examples && sbt "runMain nio.AsyncChannelReaderExample"
```

It prints:

```
managed   -> read=async, channelOpen=false, closes=1
unmanaged -> read=async, channelOpen=true, closes=0
pending   -> read=hi, cancelledRead=-1, channelOpen=true, closes=0
```

#### JVM Invariants

Four properties hold for every reader the `AsyncNioReaders` factories produce, and each one is a rule a hand-written channel wrapper commonly gets wrong.

1. **A zero-byte completion is not end of stream.** When the channel completes a read having transferred nothing, the adapter clears its buffer and resubmits the read; only a negative completion count ends the stream. A channel that yields `0` under backpressure therefore stalls the pull, it does not truncate the stream.
2. **`IOException`s are trusted source failures.** A failure reported by the channel is wrapped as a source failure and surfaces in the typed error channel of the stream built from the reader, not as a defect, and every later pull replays it rather than pretending the source recovered.
3. **Pulls are inert until driven.** Every read method returns a deferred `Async`; building `reader.readByte()` submits nothing to the channel, and the read is issued when the effect is driven. `readable()` follows the same rule — it reports whether bytes are already buffered and never initiates I/O to find out.
4. **One operation may be in flight at a time.** A second pull started while another is active fails with `IllegalStateException`; [One Active Operation at a Time](#one-active-operation-at-a-time) covers the rule and the way to sequence pulls instead.

#### JVM Limitations

The asynchronous NIO surface is exactly the four factories above, and file input is not among them.

:::warning[`AsynchronousFileChannel` is not supported]
No factory accepts an `AsynchronousFileChannel`, and it is not an `AsynchronousByteChannel`, so it cannot be passed to `AsyncNioReaders.fromChannel` either. There is no asynchronous file reader in this module.
:::

File bytes go through the synchronous `NioReaders.fromChannel` instead, as [From I/O](#from-io) describes.

#### Scala.js: `ReadableStreamReaders`

`ReadableStreamReaders` is the Scala.js counterpart, wrapping the byte-reading side of the Web Streams API. The module declares minimal `@js.native` facades for `ReadableStream`, its reader, and a read result, so using it does not pull a DOM library into your build.

Two factories mirror the managed and unmanaged pair on the JVM:

```scala
object ReadableStreamReaders {
  def fromReadableStream(stream: ReadableStream): Reader.AsyncReader[Byte]
  def fromReadableStreamUnmanaged(stream: ReadableStream): Reader.AsyncReader[Byte]
}
```

Both take the *stream*, not a reader. Each factory calls `stream.getReader()` itself and keeps the acquired reader for its own lifetime, which is what makes the lock release on close well defined. Acquiring a reader yourself and passing it in is not part of the API.

The managed factory owns the acquired reader: closing it cancels the JavaScript stream, awaits the read that was in flight, and then releases the lock. The unmanaged factory releases the lock and never cancels, so the underlying stream remains usable by the code that created it. As on the JVM, either kind settles a pending pull with end of stream instead of leaving it outstanding, and drops whatever it had buffered.

#### Scala.js Invariants

The JVM adapter's rules about deferral and exclusivity hold here too — pulls are inert until driven, and one operation may be in flight at a time. The four properties below restate the adapter's end-of-stream, buffering and failure behaviour in the terms of the Web Streams `read()` promise and its `done`/`value` result, which is where this adapter is easiest to get wrong.

1. **An empty chunk is skipped, never treated as end of stream.** A result with `done = false` and a zero-length value causes the adapter to reissue `read()`; only `done = true` ends the stream.
2. **Buffered bytes are preserved across pulls.** A chunk delivered by the stream is consumed byte by byte from the adapter's own index, so a pull that the buffer can satisfy issues no `read()` at all, and a partially consumed chunk survives until it is drained.
3. **End of stream is observed without prefetch.** The adapter never calls `read()` merely to discover whether the stream has finished; it learns that from the read that a pull actually needed.
4. **A rejected promise fails the read.** The rejection fails the pull's `Async` with a `StreamError` whose `value` is the rejection, and every later pull fails with it again.

:::warning[Known bug in 0.0.56: a rejected read does not arrive as `Left`]
In a stream built with `Stream.fromReader` over this reader, a read that the JavaScript stream rejects, for example a dropped connection, does not arrive as `Left` in the `Either` the way the JVM channel adapter's failures do. It fails the `Async` itself with an internal `StreamError`, whose message is `null`, so handle it with `Async#either` or a `catch` at the edge.
:::

#### Scala.js Limitations

Two of the JVM adapter's affordances have no Scala.js equivalent, and one of them cannot be worked around from user code.

:::warning[Byte-only, no BYOB, no buffer size]
Both factories produce `Reader.AsyncReader[Byte]` on the `JvmType.Byte` lane and read `Uint8Array` chunks; there is no factory for another element type. Neither factory offers BYOB support — `getReader()` is called with no arguments, so the adapter never acquires a bring-your-own-buffer reader and cannot read into a caller-supplied `ArrayBuffer`. Neither takes a buffer-size parameter: chunk sizes are whatever the underlying stream produces.
:::

### Single Element

`Reader.single` — Creates a reader that emits exactly one element, then closes. Primitive types use specialized variants for zero-boxing:

```scala
object Reader {
  def single[A](value: A)(implicit jt: JvmType.Infer[A]): Reader.SyncReader[A]
  def singleInt(value: Int): Reader.SyncReader[Int]
  def singleLong(value: Long): Reader.SyncReader[Long]
  def singleFloat(value: Float): Reader.SyncReader[Float]
  def singleDouble(value: Double): Reader.SyncReader[Double]
  def singleChar(value: Char): Reader.SyncReader[Char]
  def singleShort(value: Short): Reader.SyncReader[Short]
  def singleByte(value: Byte): Reader.SyncReader[Byte]
  def singleBoolean(value: Boolean): Reader.SyncReader[Boolean]
}
```

When you use `Reader.single`, behavior differs between reference types and primitives. The `JvmType.Infer[A]` implicit parameter enables compile-time type detection, automatically selecting the appropriate implementation (specialized primitive or reference-type generic).

For reference types like String, `Reader.single("hello")` stores the element directly and tracks a taken/not-taken flag internally. You read via the generic `SyncReader#read[A](sentinel)` method, passing your own sentinel value. On the first call, you get your string; on subsequent calls, you receive the sentinel you provided, allowing you to detect stream closure.

For primitive types, `Reader.single(42)` could naively box the integer, but the library avoids this penalty entirely via `SingletonPrim`—a zero-boxing specialization that stores the primitive unboxed in memory. The `JvmType.Infer` implicit detects this at compile time and routes you through specialized factory methods (`Reader.singleInt`, `Reader.singleLong`, etc.) and specialized read methods (`Reader#readInt`, `Reader#readLong`, etc.). Both storage and retrieval stay unboxed, maintaining zero-copy efficiency.

`Reader.singleByte` returns `SyncReader[Byte]` and reports `JvmType.Byte`. Its physical scalar pull is `readByte(): Int`, which returns the unsigned byte value `0`–`255` or `-1` at EOF.

Create and read from a single-element reference-type reader with a custom sentinel:

```scala

val r = Reader.single("hello")
// r: SyncReader[String] = zio.blocks.streams.io.Reader$SingletonGeneric@25072743
val sentinel = "END"
// sentinel: String = "END"
println(r.read(sentinel))    // hello
// hello
println(r.read(sentinel))    // END (sentinel, reader is closed)
// END
```

For primitive types, use the specialized factory and read methods. `SyncReader#readInt` takes a `Long` sentinel and returns `Long`; `AsyncReader#readInt` takes the same sentinel and returns `Async[Long]`:

```scala

val r = Reader.singleInt(100)
// r: SyncReader[Int] = zio.blocks.streams.io.Reader$SingletonPrim@3020eb84
val sentinel = Long.MinValue
// sentinel: Long = -9223372036854775808L
val v1 = r.readInt(sentinel)
// v1: Long = 100L
println(v1)    // 100
// 100
val v2 = r.readInt(sentinel)
// v2: Long = -9223372036854775808L
println(v2)    // -9223372036854775808 (sentinel, reader is closed)
// -9223372036854775808
```

### Infinite & Repeating

`Reader.repeat` — Creates an infinite reader that always emits the same value:

```scala
object Reader {
  def repeat[A](a: A)(implicit jt: JvmType.Infer[A]): Reader.SyncReader[A]
}
```

Create an infinite reader that repeatedly emits the same value:

```scala

val r = Reader.repeat(1)
// r: SyncReader[Int] = zio.blocks.streams.io.Reader$SingletonPrim@2f26b44d

def drainN(n: Int): Unit = {
  if (n > 0) {
    val v = r.read(-1)
    println(v)
    drainN(n - 1)
  }
}
drainN(3)
// 1
// 1
// 1
```

A known bug in 0.0.56: `SyncReader#close` does not stop a `Reader.repeat` reader. After `SyncReader#close`, `SyncReader#isClosed` stays `false` and every pull still returns the value.

`Reader.repeated` — Restarts an inner reader each time it ends, by calling `SyncReader#reset` on it. Used by `Stream.repeated` to create indefinitely repeating streams:

```scala
object Reader {
  def repeated[A](inner: SyncReader[A]): SyncReader[A]
  def repeated[A](inner: AsyncReader[A]): AsyncReader[A]
  def repeated[A](inner: Reader[A]): Reader[A]
}
```

Each time `inner` ends, the synchronous overload calls `inner.reset()` and keeps reading, so `inner` must support `SyncReader#reset`; one that does not makes the pull throw `UnsupportedOperationException`, and an `inner` that produces nothing between two resets makes the pull loop forever. A known bug in 0.0.56: after `SyncReader#close`, the synchronous overload's next pull resets `inner` and keeps producing elements (the asynchronous overload stops as expected).

### Unfold (State Machine)

`Reader.unfold` — Creates a reader by unfolding state with a function. Returns `None` to signal completion, or `Some((elem, nextState))` to emit an element and advance state:

```scala
object Reader {
  def unfold[S, A](s: S)(f: S => Option[(A, S)])(implicit jt: JvmType.Infer[A]): SyncReader[A]
}
```

Create a reader that unfolds state incrementally until completion:

```scala

val r = Reader.unfold(1) { s =>
  if (s > 3) None else Some((s, s + 1))
}
// r: SyncReader[Int] = zio.blocks.streams.io.Reader$Unfold@21ff83c

def drain(): Unit = {
  val v = r.read(-1)
  if (v != -1) {
    println(v)
    drain()
  }
}
drain()
// 1
// 2
// 3
// Output: 1, 2, 3
```

### `Reader.unfoldAsync`

`Reader.unfoldAsync` is the only native asynchronous constructor in the companion. It has the same shape as `unfold`, with the step function returning its `Option` inside an `Async`, and it produces an `AsyncReader`:

```scala
object Reader {
  def unfoldAsync[S, A](s: S)(f: S => Async[Option[(A, S)]])(implicit jt: JvmType.Infer[A]): AsyncReader[A]
}
```

Use it when producing the next element is itself asynchronous — a network round trip, a callback-based API, a timer. Everything downstream of it compiles on the asynchronous path.

```scala

val ticks: Reader.AsyncReader[Int] =
  Reader.unfoldAsync(1) { s =>
    Async.succeed(if (s > 3) None else Some((s, s + 1)))
  }

val drained: Async[Chunk[Int]] = ticks.readAll()
val closed: Async[Unit]        = ticks.close()
```

The state callback is lazy and generation-aware: exactly one callback may be in flight, and the next state is committed only when that callback succeeds while its reader generation is still current. A callback that completes after a `reset` or a `close` therefore cannot advance state that no longer exists.

## Core Operations

These methods form the primary interface for consuming elements and querying reader state:

### Pulling Elements

`read` pulls the next element, or produces `sentinel` if the reader is closed and empty. This is the fundamental operation. The synchronous and asynchronous signatures are distinct:

```scala
abstract class Reader.SyncReader[+Elem] {
  def read[A >: Elem](sentinel: A): A
}

abstract class Reader.AsyncReader[+Elem] {
  def read[A >: Elem](sentinel: A): Async[A]
}
```

The sentinel value is caller-chosen and should never appear as a real element. For reference types, `null` is convenient. For primitives, use a value outside the domain (e.g., `-1` for unsigned bytes, `Long.MinValue` for `Int`):

```scala

val r = Reader.fromChunk(Chunk(10, 20))
// r: SyncReader[Int] = zio.blocks.streams.io.Reader$FromChunkInt@28ff5f6f
val v1 = r.read(-1)        // 10
// v1: Int = 10
val v2 = r.read(-1)        // 20
// v2: Int = 20
val v3 = r.read(-1)        // -1 (sentinel, reader is closed)
// v3: Int = -1
```

### Primitive Specialization

For primitive types, specialized methods avoid boxing by widening the return type.

`Reader#readInt` — Sentinel-return `Int` pull. Returns the element widened to `Long`, or `sentinel` when closed. The sentinel must lie outside `[Int.MinValue, Int.MaxValue]` (typically `Long.MinValue`):

```scala
abstract class Reader.SyncReader[+Elem] {
  def readInt(_sentinel: Long)(implicit _ev: Elem <:< Int): Long
}

abstract class Reader.AsyncReader[+Elem] {
  def readInt(_sentinel: Long)(implicit _ev: Elem <:< Int): Async[Long]
}
```

Why widen to `Long`? If `Reader#readInt` returned `Int`, you couldn't distinguish a real element from the sentinel—both would fit in the int range. By widening to `Long`, the sentinel (e.g., `Long.MinValue`) lies outside the possible int domain, allowing reliable end-of-stream detection. Cast the result back to `Int` if needed: `r.readInt(Long.MinValue).toInt`.

`Reader#readLong` — Sentinel-return `Long` pull. This low-level scalar method cannot distinguish EOF from a real element equal to the caller's sentinel:

```scala
abstract class Reader.SyncReader[+Elem] {
  def readLong(_sentinel: Long)(implicit _ev: Elem <:< Long): Long
}

abstract class Reader.AsyncReader[+Elem] {
  def readLong(_sentinel: Long)(implicit _ev: Elem <:< Long): Async[Long]
}
```

The scalar API necessarily permits a collision with the caller's sentinel. Collision-free internal pulls preserve the complete `Long` domain by calling `readLongs` with a length-one array and using its returned count (`-1` for EOF, `1` for data) as status. Custom full-domain loops should use the same pattern.

`Reader#readFloat` — Sentinel-return `Float` pull. Returns the element widened to `Double`, or `sentinel` when closed:

```scala
abstract class Reader.SyncReader[+Elem] {
  def readFloat(_sentinel: Double)(implicit _ev: Elem <:< Float): Double
}

abstract class Reader.AsyncReader[+Elem] {
  def readFloat(_sentinel: Double)(implicit _ev: Elem <:< Float): Async[Double]
}
```

Like `Reader#readInt`, widening to `Double` allows the sentinel to lie safely outside the float domain. A float value will always fit in the lower precision bits of the double result, and the sentinel (typically `Double.MaxValue`) occupies the upper range. This ensures you can reliably distinguish real float elements from end-of-stream. Cast back to `Float` if needed: `r.readFloat(Double.MaxValue).toFloat`.

`Reader#readDouble` — Sentinel-return `Double` pull. Returns the element, or `sentinel` when closed. No `Double` bit pattern can be reserved as a collision-free sentinel, so the parameter is kept for symmetry with the other sentinel-taking pulls and for callers who can prove that a particular value lies outside their own data domain:

```scala
abstract class Reader.SyncReader[+Elem] {
  def readDouble(_sentinel: Double)(implicit _ev: Elem <:< Double): Double
}

abstract class Reader.AsyncReader[+Elem] {
  def readDouble(_sentinel: Double)(implicit _ev: Elem <:< Double): Async[Double]
}
```

Like scalar `readLong`, scalar `readDouble` cannot reserve a collision-free value (and NaN comparisons add another trap). Collision-free internal pulls call `readDoubles` with a length-one array and use its returned count as EOF/data status, preserving infinities, every NaN payload, and either zero. Custom full-domain loops should use the same pattern.

These specialized methods are the hot path for primitive streams — they avoid allocation and boxing entirely:

```scala

val r = Reader.fromChunk(Chunk(10, 20, 30))
// r: SyncReader[Int] = zio.blocks.streams.io.Reader$FromChunkInt@33a19bf0
val sentinel = Long.MinValue
// sentinel: Long = -9223372036854775808L

val v = r.readInt(sentinel)
// v: Long = 10L
```

### Byte-Level Reading

`Reader#readByte` — Reads a single byte (0–255), widened to `Int`. Returns `-1` when the reader is closed. Dispatches on `Reader#jvmType` for zero-boxing when the reader is specialized:

```scala
abstract class Reader.SyncReader[+Elem] {
  def readByte(): Int
}

abstract class Reader.AsyncReader[+Elem] {
  def readByte(): Async[Int]
}
```

Read bytes one at a time from a reader until end-of-stream:

```scala

val bytes = Array[Byte](72, 101, 108, 108, 111)  // Hello in ASCII bytes
// bytes: Array[Byte] = Array(72, 101, 108, 108, 111)
val is = new ByteArrayInputStream(bytes)
// is: ByteArrayInputStream = java.io.ByteArrayInputStream@52580030
val r = Reader.fromInputStream(is)
// r: SyncReader[Byte] = zio.blocks.streams.io.Reader$InputStreamReader@666c3f32

def drainBytes(): Unit = {
  val b = r.readByte()
  if (b != -1) {
    println(s"Byte: $b (${b.toChar})")
    drainBytes()
  }
}
drainBytes()
// Byte: 72 (H)
// Byte: 101 (e)
// Byte: 108 (l)
// Byte: 108 (l)
// Byte: 111 (o)
// Output:
// Byte: 72 (H)
// Byte: 101 (e)
// Byte: 108 (l)
// Byte: 108 (l)
// Byte: 111 (o)
```

`Reader#readBytes` — Bulk byte read into a caller-supplied buffer, mirroring `java.io.InputStream#read(byte[], int, int)`. The behavior is:

- A `SyncReader` blocks until at least 1 byte is available; an `AsyncReader` represents that wait in `Async`.
- Returns the number of bytes read (`1 <= r <= len`).
- Returns `-1` when closed and empty.
- Returns `0` immediately when `len == 0`.

The method signature is:

```scala
abstract class Reader.SyncReader[+Elem] {
  def readBytes(buf: Array[Byte], offset: Int, len: Int)(implicit ev: Elem <:< Byte): Int
}

abstract class Reader.AsyncReader[+Elem] {
  def readBytes(dest: Array[Byte], offset: Int, length: Int)(implicit ev: Elem <:< Byte): Async[Int]
}
```

Read multiple bytes into a buffer in bulk with a loop pattern:

```scala

val bytes = Array[Byte](72, 101, 108, 108, 111)  // The word Hello
// bytes: Array[Byte] = Array(72, 101, 108, 108, 111)
val is = new ByteArrayInputStream(bytes)
// is: ByteArrayInputStream = java.io.ByteArrayInputStream@3ef958
val r = Reader.fromInputStream(is)
// r: SyncReader[Byte] = zio.blocks.streams.io.Reader$InputStreamReader@16252522

val buffer = new Array[Byte](3)
// buffer: Array[Byte] = Array(108, 111, 108)

def drainBulk(): Unit = {
  val bytesRead = r.readBytes(buffer, 0, 3)
  if (bytesRead > 0) {
    val chunk = buffer.take(bytesRead).map(_.toChar).mkString
    println(s"Read $bytesRead bytes: $chunk")
    drainBulk()
  }
}
drainBulk()
// Read 3 bytes: Hel
// Read 2 bytes: lo
// Output:
// Read 3 bytes: Hel
// Read 2 bytes: lo
```

### Character and Numeric Specialization

`Reader#readChar` — Sentinel-return `Char` pull. Returns the element widened to `Int`, or `sentinel` when closed. Requires evidence that `Elem <:< Char`:

```scala
abstract class Reader.SyncReader[+Elem] {
  def readChar(_sentinel: Int)(implicit _ev: Elem <:< Char): Int
}

abstract class Reader.AsyncReader[+Elem] {
  def readChar(_sentinel: Int)(implicit _ev: Elem <:< Char): Async[Int]
}
```

`Reader#readShort` — Sentinel-return `Short` pull. Returns the element widened to `Int`, or `sentinel` when closed:

```scala
abstract class Reader.SyncReader[+Elem] {
  def readShort(_sentinel: Int)(implicit _ev: Elem <:< Short): Int
}

abstract class Reader.AsyncReader[+Elem] {
  def readShort(_sentinel: Int)(implicit _ev: Elem <:< Short): Async[Int]
}
```

`Reader#readBoolean` — Sentinel-return `Boolean` pull. Returns `1` for `true`, `0` for `false`, or `sentinel` when closed. The sentinel must lie outside `[0, 1]` (typically `-1`):

```scala
abstract class Reader.SyncReader[+Elem] {
  def readBoolean(_sentinel: Int)(implicit _ev: Elem <:< Boolean): Int
}

abstract class Reader.AsyncReader[+Elem] {
  def readBoolean(_sentinel: Int)(implicit _ev: Elem <:< Boolean): Async[Int]
}
```

### Bulk Operations

`Reader#readAll` — Drains the entire reader into a `Chunk`. Dispatches on `Reader#jvmType` for zero-boxing on primitive readers:

```scala
abstract class Reader.SyncReader[+Elem] {
  def readAll[A >: Elem](): Chunk[A]
}

abstract class Reader.AsyncReader[+Elem] {
  def readAll[A >: Elem](): Async[Chunk[A]]
}
```

The result is a new chunk containing all remaining elements:

```scala

val r = Reader.fromChunk(Chunk(10, 20, 30))
// r: SyncReader[Int] = zio.blocks.streams.io.Reader$FromChunkInt@5ae7f413
val all = r.readAll()
// all: Chunk[Int] = IndexedSeq(10, 20, 30)
println(all)  // Chunk(10,20,30)
// Chunk(10,20,30)
```

`Reader#readN` and `Reader#readUpToN` — Bounded drains. `readN` gathers up to `n` elements, returning early only when the reader is exhausted; `readUpToN` gathers at most `n` elements and stops as soon as the next element is not already available, so it never waits for a slow producer to fill the request. Both produce an empty chunk when `n <= 0` or the reader is at end-of-stream:

```scala
abstract class Reader.SyncReader[+Elem] {
  def readN[A >: Elem](n: Int): Chunk[A]
  def readUpToN[A >: Elem](n: Int): Chunk[A]
}

abstract class Reader.AsyncReader[+Elem] {
  def readN[A >: Elem](n: Int): Async[Chunk[A]]
  def readUpToN[A >: Elem](n: Int): Async[Chunk[A]]
}
```

:::caution[Bound `n` yourself]
`n` is a request, and some readers size a buffer from it before knowing how much data will arrive. The JVM channel-backed byte reader allocates `new Array[Byte](n)` up front in `readUpToN`, and the only guards on that allocation are `n <= 0` and an already-closed reader — there is no upper bound. Passing `Int.MaxValue` therefore asks for a 2 GB array rather than "whatever is ready". Choose a bound that reflects how much you are prepared to hold in memory, such as a page or buffer size.
:::

`Reader#skip` — Eagerly discards the first `n` elements. Dispatches on `Reader#jvmType` for zero-boxing when possible:

```scala
abstract class Reader.SyncReader[+Elem] {
  def skip(n: Long): Unit
}

abstract class Reader.AsyncReader[+Elem] {
  def skip(n: Long): Async[Unit]
}
```

### State Queries

`SyncReader#isClosed` and `AsyncReader#isClosed` report whether the reader is exhausted or closed. Once `true`, the answer stays `true` until `SyncReader#reset` or `AsyncReader#reset` rewinds the reader:

```scala
abstract class Reader.SyncReader[+Elem] {
  def isClosed: Boolean
}

abstract class Reader.AsyncReader[+Elem] {
  def isClosed: Async[Boolean]
}
```

`SyncReader#readable` and `AsyncReader#readable` report whether the reader expects to produce another element. The `SyncReader` default is `!isClosed`, and on either kind `true` does not promise that the next pull returns an element; readers over a collection answer exactly. On `AsyncReader` the answer itself is asynchronous. Buffered readers can override it for an accurate, non-consuming probe:

```scala
abstract class Reader.SyncReader[+Elem] {
  def readable(): Boolean
}

abstract class Reader.AsyncReader[+Elem] {
  def readable(): Async[Boolean]
}
```

Use `readable()` to check if elements are available before calling `read()`:

```scala

val r = Reader.fromChunk(Chunk(1, 2))
// r: SyncReader[Int] = zio.blocks.streams.io.Reader$FromChunkInt@134a93c5
println(r.readable())      // true
// true
r.read(-1)
// res39: Int = 1
println(r.readable())      // true
// true
r.read(-1)
// res41: Int = 2
println(r.readable())      // false
// false
```

## Asynchronous Reading

`Reader.AsyncReader[Elem]` is the kind a stream materializes as whenever its graph contains an asynchronous node. It is not a second API: it is the surface described above with every result moved inside `Async`. What follows is that member list, and the handful of behaviours that are specific to the asynchronous kind.

A custom asynchronous reader supplies four members. Everything else on the class has a working default built on top of them:

```scala
abstract class Reader.AsyncReader[+Elem] extends Reader[Elem] {
  def read[A >: Elem](sentinel: A): Async[A]
  def readable(): Async[Boolean]
  def isClosed: Async[Boolean]
  def close(): Async[Unit]
}
```

That is enough to build a reader the whole stream machinery can drive:

```scala

final class OneShot(value: Int) extends Reader.AsyncReader[Int] {
  private var delivered = false
  private var closed    = false

  def read[A >: Int](sentinel: A): Async[A] =
    if (closed || delivered) Async.succeed(sentinel)
    else { delivered = true; Async.succeed(value) }

  def readable(): Async[Boolean] = Async.succeed(!closed && !delivered)
  def isClosed: Async[Boolean]   = Async.succeed(closed)
  def close(): Async[Unit]       = Async.succeed { closed = true }
}
```

The three bulk reads are derived from `read` and the reader's lane, so an implementation gets them for free and overrides them only to exploit a cheaper native path:

```scala
abstract class Reader.AsyncReader[+Elem] extends Reader[Elem] {
  def readAll[A >: Elem](): Async[Chunk[A]]
  def readN[A >: Elem](n: Int): Async[Chunk[A]]
  def readUpToN[A >: Elem](n: Int): Async[Chunk[A]]
}
```

`readAll()` is `readN(Int.MaxValue)`, `readN` gathers until it has `n` elements or hits end-of-stream, and `readUpToN` additionally stops as soon as the next element is not already available. All three yield to the scheduler after a fixed budget of consecutive pulls, so a fast in-memory reader cannot monopolize the calling thread.

The eight primitive pulls mirror the synchronous lane exactly, including the widened carriers — a `Char`, `Short`, `Boolean`, or `Byte` lane returns its value in an `Int`, an `Int` lane in a `Long`, and a `Float` lane in a `Double` — with the result inside `Async`:

```scala
abstract class Reader.AsyncReader[+Elem] extends Reader[Elem] {
  def readBoolean(_sentinel: Int)(implicit _ev: Elem <:< Boolean): Async[Int]
  def readByte(): Async[Int]
  def readChar(_sentinel: Int)(implicit _ev: Elem <:< Char): Async[Int]
  def readShort(_sentinel: Int)(implicit _ev: Elem <:< Short): Async[Int]
  def readInt(_sentinel: Long)(implicit _ev: Elem <:< Int): Async[Long]
  def readLong(_sentinel: Long)(implicit _ev: Elem <:< Long): Async[Long]
  def readFloat(_sentinel: Double)(implicit _ev: Elem <:< Float): Async[Double]
  def readDouble(_sentinel: Double)(implicit _ev: Elem <:< Double): Async[Double]
}
```

Calling one of these on a reader whose `jvmType` is a different primitive lane does not throw at the call site: the default returns a failed `Async` carrying an `UnsupportedOperationException` that names both lanes. The view that `SyncReader#toAsync` returns is the exception, reachable only through a cast: its `Reader#readLong` and `Reader#readDouble` throw `IllegalStateException` at the call site, and its other primitive pulls fail the `Async` with it. A plain reader on the `AnyRef` lane satisfies all eight by pulling boxed and converting.

Five bulk array transfers fill a caller-supplied array and report how many elements were written, or `-1` when the reader was already at end-of-stream:

```scala
abstract class Reader.AsyncReader[+Elem] extends Reader[Elem] {
  def readBytes(dest: Array[Byte], offset: Int, length: Int)(implicit ev: Elem <:< Byte): Async[Int]
  def readInts(dest: Array[Int], offset: Int, length: Int)(implicit ev: Elem <:< Int): Async[Int]
  def readLongs(dest: Array[Long], offset: Int, length: Int)(implicit ev: Elem <:< Long): Async[Int]
  def readFloats(dest: Array[Float], offset: Int, length: Int)(implicit ev: Elem <:< Float): Async[Int]
  def readDoubles(dest: Array[Double], offset: Int, length: Int)(implicit ev: Elem <:< Double): Async[Int]
}
```

A transfer also stops short of `length` when the next element is not already available, so a partial count is a normal result rather than a sign of end-of-stream. An out-of-range `offset` or `length` surfaces as a failed `Async`, not a thrown exception.

The five control operations complete the mirror:

```scala
abstract class Reader.AsyncReader[+Elem] extends Reader[Elem] {
  def skip(n: Long): Async[Unit]
  def reset(): Async[Unit]
  def setLimit(n: Long): Async[Boolean]
  def setRepeat(): Async[Boolean]
  def setSkip(n: Long): Async[Boolean]
}
```

`skip` has a real default that discards elements through the reader's own lane. The pushdown operations do not: on the base class `reset()` fails with an `UnsupportedOperationException`, and `setLimit`, `setRepeat`, and `setSkip` each succeed with `false`. Those defaults are the honest answer for a reader that cannot rewind or bound itself natively, and callers already handle them — a `false` simply means the interpreter wraps the reader instead of pushing the operation down. Override them only when your reader can genuinely do the work in O(1).

### One Active Operation at a Time

An `AsyncReader` is a single-consumer cursor with one position and one lifecycle. **At most one operation may be in flight at a time.** Await the `Async` returned by a pull, a transfer, a control operation, or `close()` before beginning the next one.

For an implementor this is a contract you may rely on and must not weaken: your `read` will not be re-entered while a previous `read` is still pending, so internal position and buffer state need no defence against overlap. It is also a contract you inherit — a reader you wrap gets the same guarantee only if you preserve it, so never fan a single downstream pull out into concurrent pulls on your source.

Readers are not thread-safe either. Driving one reader from two threads without external synchronization is outside the contract, and the result is not specified. [Asynchronous Stream Execution](../execution-and-compatibility/async-execution.md#one-active-operation-per-reader) states the same rule from the consumer's side.

### Close Ownership

Every asynchronous reader has exactly one owner, and the owner is responsible for awaiting `close()`. Which side holds it is never ambiguous, because the entry point that handed you the reader decides: terminals own and close the reader they compile, `Stream#startAsync` transfers ownership to you, and `Stream#useReaderAsync` retains it. [Manual Pull and Ownership](../execution-and-compatibility/async-execution.md#manual-pull-and-ownership) gives each case in full from the consumer's side, including what a forgotten `close()` costs. The rest of this section is what ownership means for the reader itself.

`close()` is itself an asynchronous operation: it participates in the one-active-operation rule, it cancels or joins work already in flight, and its result must be awaited rather than discarded. Library readers tolerate a repeated close, but the owner should still close exactly once.

For an implementor, `close()` is where release actions and underlying resources are surfaced. A failure during cleanup is reported through the returned `Async` rather than swallowed, so do not let a failing release leave the reader believing it is still open.

```scala

// Ownership retained by the library: the reader is closed on every outcome.
val firstFive: Async[Chunk[Int]] =
  Stream.range(1, 100).useReaderAsync { (r: Reader.AsyncReader[Int]) =>
    r.readN(5)
  }

// Ownership transferred to the caller: closing is now your job.
val owned: Async[Chunk[Int]] =
  Stream.range(1, 100).startAsync.flatMap { r =>
    r.readN(5).flatMap(chunk => r.close().map(_ => chunk))
  }
```

## Composition

Combine multiple readers to build more complex sources:

### Concatenation

`Reader#concat` — Concatenates this reader with `next`. When this reader is exhausted, it is closed and elements are pulled from `next` (evaluated lazily). Optimized for left-associative chains:

```scala
abstract class Reader[+Elem] {
  def concat[Elem2 >: Elem](next: () => Reader[Elem2]): Reader[Elem2]
}
```

`Reader#++` — Alias for `Reader#concat`. Syntactic sugar for composing readers:

```scala
abstract class Reader[+Elem] {
  def ++[Elem2 >: Elem](next: => Reader[Elem2]): Reader[Elem2]
}
```

Here is how concatenation chains multiple readers together:

```scala

val r1 = Reader.fromChunk(Chunk(1, 2))
// r1: SyncReader[Int] = zio.blocks.streams.io.Reader$FromChunkInt@2dacf563
val r2 = Reader.fromChunk(Chunk(3, 4))
// r2: SyncReader[Int] = zio.blocks.streams.io.Reader$FromChunkInt@75e4df6d
val combined = r1 ++ r2
// combined: SyncReader[Int] = zio.blocks.streams.io.Reader$ConcatReader@35724e66

def drain(): Unit = {
  val v = combined.read(-1)
  if (v != -1) {
    println(v)
    drain()
  }
}
drain()
// 1
// 2
// 3
// 4
```

The result is on the `AnyRef` lane, whatever the lanes of its parts. A known bug in 0.0.56: its primitive pulls (`Reader#readInt`, `Reader#readBytes`, ...) throw, or fail with, `IllegalStateException`, so read it with `Reader#read`, `Reader#readAll`, `Reader#readN` or `Reader#readUpToN`, or through a stream.

These are the root's declarations, which answer with a `Reader[Elem2]`. `SyncReader` narrows them with a second pair of overloads so that concatenating two synchronous readers gives back a `SyncReader`; see [Mixed-kind composition](#mixed-kind-composition) for which combination produces which kind.

**Optimization**: If this reader is already a `ConcatReader`, the thunk is appended to its internal array and `this` is returned (mutable append, O(1) amortized). Otherwise a new `ConcatReader` is created. This ensures that left-associative chains like `a ++ b ++ c ++ d` compile into a single flat `ConcatReader` with O(1) per-element read, rather than O(n) nested wrappers.

## Resource Management

Close readers and attach cleanup callbacks:

### Closing

`close` signals end-of-stream from the consumer side and releases any held resources. Implementations set internal closed state and wake or cancel any pending work. A synchronous owner calls it directly; an asynchronous owner must run and await the returned `Async`:

```scala
abstract class Reader.SyncReader[+Elem] {
  def close(): Unit
}

abstract class Reader.AsyncReader[+Elem] {
  def close(): Async[Unit]
}
```

`SyncReader#withRelease` wraps a synchronous reader so that `release` runs when it closes. `withReleaseAsync`, available on the root and therefore on both kinds, returns an `AsyncReader` and awaits asynchronous cleanup:

```scala
abstract class Reader.SyncReader[+Elem] {
  def withRelease(release: () => Unit): Reader.SyncReader[Elem]
}

abstract class Reader[+Elem] {
  def withReleaseAsync(release: () => Async[Unit]): Reader.AsyncReader[Elem]
}
```

Here is how cleanup logic is attached to a reader:

```scala

val cleanupRef = scala.collection.mutable.ListBuffer[String]()
// cleanupRef: ListBuffer[String] = ListBuffer("cleaned")
val r = Reader.fromChunk(Chunk(1, 2)).withRelease { () =>
  cleanupRef += "cleaned"
  println("Cleaned up")
}
// r: SyncReader[Int] = zio.blocks.streams.io.Reader$SyncReader$$anon$3@4f7924f8

r.close()
// Cleaned up
println(cleanupRef.nonEmpty)  // true
// true
```

## Pushdown Operations

Readers can sometimes handle skip, limit, and repeat operations natively (O(1), zero per-element cost). These methods attempt that; if the reader cannot handle it natively, they return `false` and the caller must wrap the reader.

`Reader#setSkip` — Attempts to set a skip (drop) on this reader. Returns `true` if handled natively, `false` if the caller must wrap. When `true`, the next n elements are discarded before producing. After `Reader#reset()`, the skip is re-applied:

```scala
abstract class Reader.SyncReader[+Elem] {
  def setSkip(n: Long): Boolean
}

abstract class Reader.AsyncReader[+Elem] {
  def setSkip(n: Long): Async[Boolean]
}
```

Set a skip to discard the first two elements:

```scala

val r = Reader.fromChunk(Chunk(1, 2, 3, 4, 5))
// r: SyncReader[Int] = zio.blocks.streams.io.Reader$FromChunkInt@10e5ddaf
val handled = r.setSkip(2)
// handled: Boolean = true
println(s"Skip handled natively: $handled")
// Skip handled natively: true

def drain(): Unit = {
  val v = r.read(-1)
  if (v != -1) {
    println(v)
    drain()
  }
}
drain()
// 3
// 4
// 5
// Output:
// Skip handled natively: true
// 3
// 4
// 5
```

`Reader#setLimit` — Attempts to set a limit on this reader so it produces at most `n` elements. Returns `true` if handled natively, `false` if the caller must wrap. After `reset()`, the limit is re-applied from the new start position:

```scala
abstract class Reader.SyncReader[+Elem] {
  def setLimit(n: Long): Boolean
}

abstract class Reader.AsyncReader[+Elem] {
  def setLimit(n: Long): Async[Boolean]
}
```

Set a limit to produce only three elements:

```scala

val r = Reader.fromChunk(Chunk(1, 2, 3, 4, 5))
// r: SyncReader[Int] = zio.blocks.streams.io.Reader$FromChunkInt@27f5bc7d
val handled = r.setLimit(3)
// handled: Boolean = true
println(s"Limit handled natively: $handled")
// Limit handled natively: true

def drain(): Unit = {
  val v = r.read(-1)
  if (v != -1) {
    println(v)
    drain()
  }
}
drain()
// 1
// 2
// 3
// Output:
// Limit handled natively: true
// 1
// 2
// 3
```

`Reader#setRepeat` — Attempts to set this reader into repeat-forever mode, so it restarts from the beginning whenever it would otherwise close. Returns `true` if handled natively, `false` if the caller must wrap:

```scala
abstract class Reader.SyncReader[+Elem] {
  def setRepeat(): Boolean
}

abstract class Reader.AsyncReader[+Elem] {
  def setRepeat(): Async[Boolean]
}
```

Set repeat mode so a reader restarts instead of closing. Not every reader can do this natively — the chunk-, iterable- and range-backed readers cannot, and return `false`; the single-element readers can:

```scala

val r = Reader.single(1)
// r: SyncReader[Int] = zio.blocks.streams.io.Reader$SingletonPrim@4c9ca6b2
val handled = r.setRepeat()
// handled: Boolean = true
println(s"Repeat handled natively: $handled")
// Repeat handled natively: true

def drain(count: Int): Unit = {
  if (count < 4) {
    val v = r.read(-1)
    println(v)
    drain(count + 1)
  }
}
drain(0)
// 1
// 1
// 1
// 1
// Output:
// Repeat handled natively: true
// 1
// 1
// 1
// 1
```

`Reader.fromChunk(Chunk(1, 2)).setRepeat()` returns `false` instead: chunk-backed readers do not implement repeat, so the caller wraps the reader rather than pushing the operation down.

`Reader#reset` — Rewinds this reader to its initial state, as if freshly constructed. After `Reader#reset()`, all elements are available again from the beginning. Not all readers support this; readers backed by one-shot resources (InputStreams, `java.io.Reader`s) throw `UnsupportedOperationException`:

```scala
abstract class Reader.SyncReader[+Elem] {
  def reset(): Unit
}

abstract class Reader.AsyncReader[+Elem] {
  def reset(): Async[Unit]
}
```

After rewinding, the reader starts from the beginning:

```scala

val r = Reader.fromChunk(Chunk(1, 2, 3))
// r: SyncReader[Int] = zio.blocks.streams.io.Reader$FromChunkInt@3464e997
println(r.read(-1))  // 1
// 1
r.reset()
println(r.read(-1))  // 1 (back to the beginning)
// 1
```

## Integration with Stream

`Reader` is the compilation target of `Stream`. When you call a terminal operation, the stream compiles to a `Reader`, which is then consumed.

For cross-platform manual pulling, use caller-owned `Stream#startAsync` or bracketed `Stream#useReaderAsync`. `startAsync` transfers ownership to you, so you must await `close()` on every exit path; `useReaderAsync` retains ownership and closes automatically on success, failure, or cancellation. The JVM-only `Stream#start` returns a scoped blocking reader owned by its scope:

```scala

Scope.global.scoped { scope =>

  val reader: $[Reader.SyncReader[Int]] = Stream.range(1, 6).start(using scope)

  $(reader) { r =>
    def drain(): Unit = {
      val v = r.read(-1)
      if (v != -1) {
        println(v)   // prints 1, 2, 3, 4, 5
        drain()
      }
    }
    drain()
  }
  // reader is closed automatically when scope exits
}
```

:::caution
Avoid holding references to a `SyncReader` obtained via `Stream#start` outside its [`Scope`](../../resource-management/scope.md). The scope guarantees cleanup; escaping the reader defeats that guarantee.
:::

A pull on a started reader throws the stream's typed error wrapped in the internal `StreamError` rather than returning `Left(e)`, and, as a known bug in 0.0.56, a stream with a `Stream#flatMap` throws that failure again from `Reader#close` and from the owning `Scope#scoped`; [Manual Pull via `start`](../core/stream.md#manual-pull-via-start) has the details.

## Integration with Sink

`Reader` and `Sink` are dual: `Reader` is the source, and `Sink` is the consumer. A terminal compiles the stream to the reader kind required by the graph, hands that reader to the sink's drain, and retains ownership of it. On the JVM, plain terminals such as `run` use the blocking `SyncReader` path when the graph is synchronous and bridge genuine asynchronous boundaries at the final edge. Cross-platform `runAsync` drains an `AsyncReader` without blocking. Both terminal families close the owned reader on success, typed failure, defect, or cancellation.

The sink repeatedly pulls from its reader until end-of-stream, transforming the sequence of elements into a result of type `Z`. This kind-selected drain is an implementation detail; callers choose it through `run` or `runAsync` rather than invoking a sink drain method directly.

For example, `Sink.collectAll` drains all elements and returns them as a `Chunk`:

```scala

val result = Stream.range(1, 10)
  .run(Sink.collectAll[Int])
// result: Either[Nothing, Chunk[Int]] = Right(
//   IndexedSeq(1, 2, 3, 4, 5, 6, 7, 8, 9)
// )
```

## Implementation Notes

Understand the design choices and mechanisms that power `Reader`:

### Sentinel Protocol

The `read(sentinel)` method uses a caller-chosen sentinel value to signal end-of-stream. This avoids the allocation and boxing of wrapping results in `Option` or `Either`. The sentinel must be a value that never appears as a real element.

The contract has three parts, and it is the same on both reader kinds:

1. **The caller owns the sentinel.** The reader never invents one. Pick a value that cannot occur in your data — `null` is the usual choice for reference elements.
2. **The sentinel travels in the widened carrier.** A primitive pull returns the lane's widened type, not the element type, precisely so a value outside the element's domain is available to spend as the sentinel. `readInt` takes and returns `Long`; `readChar`, `readShort`, and `readBoolean` take and return `Int`; `readFloat` takes and returns `Double`. `readByte()` is the exception that proves the rule: it takes no sentinel parameter because it yields unsigned bytes in `0..255` and can reserve `-1` permanently.
3. **Getting the sentinel back means exhausted, and nothing else.** It is not an error signal. Failures arrive as thrown exceptions on a `SyncReader` and as failed `Async` values on an `AsyncReader`.

Two lanes sit outside that arrangement. There is no `Long` value and no `Double` bit pattern left over to reserve — the carrier is the element type itself, so every candidate sentinel is also legitimate data. **The `Long` and `Double` lanes therefore use no sentinel.** `readLong` and `readDouble` still take a sentinel parameter, for symmetry with the five other sentinel-taking pulls, but nothing can safely fill it; the library never relies on it. Instead those lanes detect end-of-stream by count: a length-one `readLongs` or `readDoubles` whose returned count is negative. That is what makes those two lanes fully lossless — every `Long` value and every `Double` bit pattern stays readable as data.

For the per-lane end-of-stream detail, including which carrier each lane widens to, see [Zero-Boxing Streams](../execution-and-compatibility/zero-boxing.md), which owns that table.

### JVM Type Dispatch

`Reader` dispatches on `jvmType` to choose between unboxed and boxed pull paths. This is a physical contract: `JvmType.Byte`, for example, means `readByte` is supported and yields this reader's elements, even if covariance has widened its static type to `Reader[AnyVal]`. Type-preserving wrappers and widening operations preserve a known lane; only an actually unknown or mixed representation falls back to `AnyRef`. Subclasses with primitive specialization override `jvmType`:

```scala
abstract class Reader[+Elem] {
  def jvmType: JvmType = JvmType.AnyRef
}
```

`JvmType` has nine lanes: the eight JVM primitives — `Boolean`, `Byte`, `Char`, `Short`, `Int`, `Long`, `Float`, `Double` — and `AnyRef` for everything else. The eight primitive tags map exactly to `readBoolean`, `readByte`, `readChar`, `readShort`, `readInt`, `readLong`, `readFloat`, and `readDouble`; `AnyRef` is the ninth, and a plain reader on that lane answers all eight of those methods itself by pulling boxed and converting. A known bug in 0.0.56 limits that: on an `AnyRef`-lane reader the default bulk reads `Reader#readBytes`, `Reader#readInts` and `Reader#readFloats` throw `IllegalStateException`, and so does every primitive pull of a `Reader#concat` result (which is always on the `AnyRef` lane) and of the library's wrappers around an `AnyRef`-lane reader (`SyncReader#toAsync`, `SyncReader#withRelease`, `Reader.repeated`); `Reader#read`, `Reader#readAll`, `Reader#readN` and `Reader#readUpToN` work on all of them. For example, a `SyncReader[Int]` backed by a `Chunk[Int]` reports `JvmType.Int`, so consumers may use `readInt`; a pull belonging to another lane throws `UnsupportedOperationException` unless that particular reader happens to implement it too.

The lane is a property of the reader, not of the kind. A `SyncReader` and the `AsyncReader` it becomes under `toAsync` report the same `jvmType`, and asynchronous readers expose the corresponding values through `Async`.

### Thread Safety

Readers are single-consumer cursors, not concurrent work queues. In particular, do not overlap pulls on an `AsyncReader`; await one operation before beginning another.

## Running the Examples

All code from this guide is available as runnable examples in the `streams-examples` module. Follow these steps to run them:

**Step 1** — Clone the repository and navigate to the project:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**Step 2** — Run individual examples with sbt:

### Basic Reader Construction

This example demonstrates the most common reader factories: `Reader.fromChunk`, `Reader.fromIterable`, `Reader.fromRange`, and `Reader.single`. Embed the source:

```scala title="streams-examples/src/main/scala/reader/ReaderBasicConstructionExample.scala" 
package reader

/**
 * Demonstrates the most common Reader factories: fromChunk, fromIterable,
 * fromRange, single, and unfold. Each reader is drained manually with read() to
 * show how to consume elements.
 */
object ReaderBasicConstructionExample extends App {

  println("=== Reader.fromChunk ===")
  val chunkReader = Reader.fromChunk(Chunk(10, 20, 30))
  var v           = chunkReader.read(-1)
  while (v != -1) {
    println(s"Read: $v")
    v = chunkReader.read(-1)
  }

  println("\n=== Reader.fromRange ===")
  val rangeReader = Reader.fromRange(1 to 3)
  v = rangeReader.read(-1)
  while (v != -1) {
    println(s"Read: $v")
    v = rangeReader.read(-1)
  }

  println("\n=== Reader.fromIterable ===")
  val listReader = Reader.fromIterable(List("a", "b", "c"))
  var sv         = listReader.read(null: String)
  while (sv != null) {
    println(s"Read: $sv")
    sv = listReader.read(null: String)
  }

  println("\n=== Reader.single ===")
  val singleReader = Reader.single(42)
  println(s"Read: ${singleReader.read(-1)}")
  println(s"Read again (closed): ${singleReader.read(-1)}")

  println("\n=== Reader.unfold ===")
  val unfoldReader = Reader.unfold(1) { s =>
    if (s > 3) None else Some((s * 10, s + 1))
  }
  v = unfoldReader.read(-1)
  while (v != -1) {
    println(s"Read: $v")
    v = unfoldReader.read(-1)
  }

  println("\n=== Reader state ===")
  val stateReader = Reader.fromChunk(Chunk(5, 6))
  println(s"readable before: ${stateReader.readable()}")
  stateReader.read(-1)
  println(s"readable after one read: ${stateReader.readable()}")
  stateReader.read(-1)
  println(s"readable after exhaustion: ${stateReader.readable()}")
  println(s"isClosed: ${stateReader.isClosed}")
}
```

Run it with:

```bash
cd streams-examples && sbt "runMain reader.ReaderBasicConstructionExample"
```

### Primitive Specialization and Bulk Operations

This example shows how primitive readers avoid boxing through `Reader#jvmType` dispatch, and demonstrates `Reader#readAll` and `Reader#skip` for bulk operations. Embed the source:

```scala title="streams-examples/src/main/scala/reader/ReaderPrimitiveSpecializationExample.scala" 
package reader

/**
 * Demonstrates primitive specialization in readers. When a Reader is backed by
 * primitive types (Int, Long, Float, Double), specialized factory methods like
 * singleInt, singleLong, etc. avoid boxing entirely. This example also shows
 * readAll for bulk consumption and skip for advancing the reader.
 */
object ReaderPrimitiveSpecializationExample extends App {

  println("=== singleInt (zero-boxed) ===")
  val intReader = Reader.singleInt(42)
  println(s"Read: ${intReader.read(-1)}")

  println("\n=== singleLong (zero-boxed) ===")
  val longReader = Reader.singleLong(9999999999L)
  println(s"Read: ${longReader.read(Long.MaxValue)}")

  println("\n=== singleFloat (zero-boxed) ===")
  val floatReader = Reader.singleFloat(3.14f)
  println(s"Read: ${floatReader.readFloat(Float.MaxValue)}")

  println("\n=== singleDouble (zero-boxed) ===")
  val doubleReader = Reader.singleDouble(2.718)
  println(s"Read: ${doubleReader.read(Double.MaxValue)}")

  println("\n=== readAll: bulk drain to Chunk ===")
  val bulkReader  = Reader.fromChunk(Chunk(1, 2, 3, 4, 5))
  val allElements = bulkReader.readAll()
  println(s"All elements: $allElements")

  println("\n=== skip: discard n elements ===")
  val skipReader = Reader.fromRange(10 to 15)
  skipReader.skip(2)
  // Should now read from 12 onward
  var v         = skipReader.read(-1)
  val remaining = scala.collection.mutable.ArrayBuffer[Int]()
  while (v != -1) {
    remaining += v
    v = skipReader.read(-1)
  }
  println(s"After skipping 2: ${remaining.toList}")

  println("\n=== reset: rewind to beginning ===")
  val resetReader = Reader.fromChunk(Chunk("x", "y", "z"))
  var elem        = resetReader.read(null: String)
  println(s"First read: $elem")
  resetReader.reset()
  elem = resetReader.read(null: String)
  println(s"After reset: $elem")

  println("\n=== readable: check if elements remain ===")
  val checkReader = Reader.fromChunk(Chunk(100, 200))
  println(s"readable before read: ${checkReader.readable()}")
  checkReader.read(-1)
  println(s"readable after one read: ${checkReader.readable()}")
  checkReader.read(-1)
  println(s"readable after exhaustion: ${checkReader.readable()}")
}
```

Run it with:

```bash
cd streams-examples && sbt "runMain reader.ReaderPrimitiveSpecializationExample"
```

### Composition and Resource Management

This example demonstrates reader composition with `Reader#++`, resource cleanup with `Reader#withRelease`, and integration with `Stream.start` for manual pulling. Embed the source:

```scala title="streams-examples/src/main/scala/reader/ReaderCompositionExample.scala" 
package reader

/**
 * Demonstrates reader composition with ++ (concat), resource cleanup with
 * withRelease, and integration with Stream.start for manual element-by-element
 * pulling within a Scope.
 */
object ReaderCompositionExample extends App {

  println("=== concat: ++ operator ===")
  val r1       = Reader.fromChunk(Chunk(1, 2, 3))
  val r2       = Reader.fromChunk(Chunk(4, 5, 6))
  val combined = r1 ++ r2

  var v           = combined.read(-1)
  val allCombined = scala.collection.mutable.ArrayBuffer[Int]()
  while (v != -1) {
    allCombined += v
    v = combined.read(-1)
  }
  println(s"Combined result: ${allCombined.toList}")

  println("\n=== Multiple concat: a ++ b ++ c ===")
  val ra    = Reader.fromChunk(Chunk("a"))
  val rb    = Reader.fromChunk(Chunk("b"))
  val rc    = Reader.fromChunk(Chunk("c"))
  val multi = ra ++ rb ++ rc

  var sv     = multi.read(null: String)
  val result = scala.collection.mutable.ArrayBuffer[String]()
  while (sv != null) {
    result += sv
    sv = multi.read(null: String)
  }
  println(s"Multiple concat: ${result.toList}")

  println("\n=== withRelease: cleanup on close ===")
  var cleanupCalled  = false
  val resourceReader = Reader.fromChunk(Chunk(10, 20)).withRelease { () =>
    cleanupCalled = true
    println("  Cleanup executed!")
  }
  var res = resourceReader.read(-1)
  while (res != -1) {
    res = resourceReader.read(-1)
  }
  resourceReader.close()
  println(s"Cleanup was called: $cleanupCalled")

  println("\n=== Stream.start: manual pull with Scope ===")
  Scope.global.scoped { scope =>

    // Create a stream and open it for manual pulling
    val reader: scope.$[Reader.SyncReader[Int]] = Stream.range(1, 6).start(using scope)

    $(reader) { r =>
      var streamV      = r.read(-1)
      val manualResult = scala.collection.mutable.ArrayBuffer[Int]()
      while (streamV != -1) {
        manualResult += streamV
        streamV = r.read(-1)
      }
      println(s"Manual stream pull: ${manualResult.toList}")
    }
    // reader is automatically closed when scope exits
  }

  println("\n=== repeat: infinite reader ===")
  val infiniteReader = Reader.repeat(99)
  infiniteReader.setRepeat()

  var repeatCount = 0
  var repV        = infiniteReader.read(-1)
  while (repeatCount < 3) {
    println(s"Infinite read $repeatCount: $repV")
    repV = infiniteReader.read(-1)
    repeatCount += 1
  }
  infiniteReader.close()
}
```

Run it with:

```bash
cd streams-examples && sbt "runMain reader.ReaderCompositionExample"
```

## See Also

- [Asynchronous Stream Execution](../execution-and-compatibility/async-execution.md) — how a graph picks its engine, the `*Async` surface, and close ownership from the stream's side
- [Platform Differences](../execution-and-compatibility/platform-differences.md#availability-matrix) — which reader operations exist on the JVM, on Scala.js, and on both
- [Zero-Boxing Streams](../execution-and-compatibility/zero-boxing.md) — how a primitive lane is chosen, and the per-lane end-of-stream table
- [Stream](../core/stream.md) — the operator and terminal reference for the type that compiles to a `Reader`
- [Sink](../core/sink.md) — the consumer that drains a `Reader`
- [How to Run Streams Without Blocking](../../../guides/run-streams-without-blocking.md) — reads a `fetch` body through `ReadableStreamReaders` and runs the stream with the `*Async` terminals
- [Async](../../async.md#the-pollable-protocol) — `Async[A]`, `Pollable`, and what awaiting an asynchronous result means
- [How to Process Binary Data Without Boxing](../../../guides/process-binary-data-without-boxing.md) — read, transform, aggregate and write `ByteBuffer`s, mapped files and `InputStream`s as unboxed `Long` and `Double` streams

---

## Writer

`Writer[-Elem]` is a **push-based sink for elements** that accepts values one at a time until closed or filled. It is the push-based counterpart to `Reader[+Elem]` (which pulls). Elements are written on demand by the producer, making it ideal for streaming, buffering, and integration with I/O subsystems. The fundamental operations are `write(elem): Boolean` — pushes an element and returns success or closure — and `close()` — signals the end of writing and releases resources.

`Writer[-Elem]` has these key properties:

- **Lazy and Push-Based** — nothing happens until the producer calls `write()`
- **Non-Thread-Safe** — designed for single-threaded production; concurrent access requires external synchronization
- **Explicit Closure Signal** — returns `false` when closed; none of the library's writers throws after `Writer#fail`

Here is the structural shape of the `Writer` type:

```scala
abstract class Writer[-Elem] {
  def write(a: Elem): Boolean
  def close(): Unit
  def isClosed: Boolean
  
  // concrete defaults for fail() and writeable()
  def fail(error: Throwable): Unit = close()
  def writeable(): Boolean = !isClosed
}
```

## Motivation

Imagine you're building a data pipeline where a producer feeds items to a bounded sink. The producer doesn't control the sink's internal state—how much capacity remains, whether it's busy, or if it's permanently closed. You need to know before each write: Is the sink ready? Did the write succeed? Is the sink closed?

With Java's `OutputStream`, you call `write()` and either it succeeds (void return) or throws an exception. This leaves ambiguity: Was the exception transient (try again later) or permanent (the stream is done)? If the buffer fills, the thread blocks—but you don't know how long, or even that it will block beforehand. There's no way to check capacity upfront, so you're forced to either over-allocate buffers (wasting memory) or catch exceptions and guess the right strategy.

`Writer` makes the state explicit and non-throwing. You check readiness with `writeable()`, then push with `write()`, which returns a `Boolean` indicating success or closure. The protocol is explicit: when `write()` returns `false`, the sink is permanently closed and you should stop. It is not exception-free, though: an exception other than an `IOException` from a wrapped stream propagates, and a writer of your own can override `Writer#fail` to keep the error and throw it from later writes (none of the library's writers does).

## Quick Showcase

Here's how to create and push elements to a `Writer`:

```scala

val collected = Buffer[Int]()
// collected: Buffer[Int] = ArrayBuffer(10, 20, 30, 40, 50)
val w = new Writer[Int] {
  private var closed = false
  
  def isClosed = closed
  def write(a: Int) = {
    if (!closed) { collected += a; true }
    else false
  }
  def close() = { closed = true }
  override def fail(error: Throwable) = close()
  override def writeable() = !isClosed
}
// w: Writer[Int] = repl.MdocSession$MdocApp0$$anon$2@5d4b3892

// Push elements, checking writeable() before each write
def pushAll(elements: List[Int]): Unit = {
  elements match {
    case Nil => ()
    case head :: tail =>
      if (w.writeable() && w.write(head)) pushAll(tail)
  }
}

pushAll(List(10, 20, 30, 40, 50))
w.close()

println(s"Collected: $collected")
// Collected: ArrayBuffer(10, 20, 30, 40, 50)
println(s"Writable after close: ${w.writeable()}")
// Writable after close: false
```

## Writing and Closure

The fundamental protocol is: call `write(element)` to push an element. It returns `true` on success, `false` only when the writer is **closed** (not when the buffer is full). Once `write()` returns `false`, the writer is permanently closed—all further writes return `false`. There is no recovery.

```scala

val w = Writer.single[Int]
// w: Writer[Int] = zio.blocks.streams.io.Writer$SingleWriter@337ec1eb
println(s"First write: ${w.write(42)}")
// First write: true
println(s"Second write: ${w.write(99)}")
// Second write: false
println(s"Third write: ${w.write(77)}")
// Third write: false
```

## Asynchronous Writes

Every effectful member of `Writer` has a deferred mirror whose name ends in `Async` and whose result is an `Async`. There are sixteen of them, and together they are the entire asynchronous surface of the type; the structural combinators `concat` and `contramap` are deliberately outside it.

A mirror does one thing. It wraps a single synchronous call in an effect that has not happened yet: constructing `writer.writeAsync(42)` performs no write at all, and driving the returned effect performs `write(42)` exactly once and yields its `Boolean`. All sixteen are `final` and delegate to one private helper, which builds them on the library's internal cancellable-defer primitive `Async.deferCancelable` (`Writer.scala:57`).

The mirrors group exactly as their synchronous twins do on this page:

| Group              | Synchronous member             | Deferred mirror                     | Result                |
|--------------------|--------------------------------|-------------------------------------|-----------------------|
| Lifecycle          | `close()`                      | `closeAsync()`                      | `Async[Unit]`         |
| Lifecycle          | `fail(error)`                  | `failAsync(error)`                  | `Async[Unit]`         |
| Single element     | `write(a)`                     | `writeAsync(a)`                     | `Async[Boolean]`      |
| Bulk               | `writeAll(chunk)`              | `writeAllAsync(chunk)`              | `Async[Chunk[Elem1]]` |
| Specialized        | `writeInt(value)`              | `writeIntAsync(value)`              | `Async[Boolean]`      |
| Specialized        | `writeLong(value)`             | `writeLongAsync(value)`             | `Async[Boolean]`      |
| Specialized        | `writeFloat(value)`            | `writeFloatAsync(value)`            | `Async[Boolean]`      |
| Specialized        | `writeDouble(value)`           | `writeDoubleAsync(value)`           | `Async[Boolean]`      |
| Byte and character | `writeByte(b)`                 | `writeByteAsync(b)`                 | `Async[Boolean]`      |
| Byte and character | `writeBytes(buf, offset, len)` | `writeBytesAsync(buf, offset, len)` | `Async[Int]`          |
| Byte and character | `writeChar(value)`             | `writeCharAsync(value)`             | `Async[Boolean]`      |
| Byte and character | `writeShort(value)`            | `writeShortAsync(value)`            | `Async[Boolean]`      |
| Byte and character | `writeBoolean(value)`          | `writeBooleanAsync(value)`          | `Async[Boolean]`      |
| State checks       | `isClosed`                     | `isClosedAsync`                     | `Async[Boolean]`      |
| State checks       | `writeable()`                  | `writeableAsync()`                  | `Async[Boolean]`      |
| State checks       | `jvmType`                      | `jvmTypeAsync`                      | `Async[JvmType]`      |

Each mirror carries the same parameters and the same implicit evidence as its twin, so the specialized mirrors still ask for the subtype witness their twin asks for:

```scala
abstract class Writer[-Elem] {
  final def closeAsync(): Async[Unit]
  final def writeAsync(a: Elem): Async[Boolean]
  final def writeAllAsync[Elem1 <: Elem](chunk: Chunk[Elem1]): Async[Chunk[Elem1]]
  final def writeIntAsync(value: Int)(implicit ev: Int <:< Elem): Async[Boolean]
  final def writeBytesAsync(buf: Array[Byte], offset: Int, len: Int)(implicit ev: Byte <:< Elem): Async[Int]
}
```

`jvmTypeAsync` is the mirror of `jvmType`, the writer's element representation (`JvmType.AnyRef` unless a subclass overrides it). It is the only mirror whose twin is not otherwise documented on this page.

### What `*Async` Does and Does Not Do

These are **cancellation-aware deferral adapters, not asynchronous I/O**. The library's own scaladoc says so in as many words (`Writer.scala:51`), and it is worth repeating because sixteen methods named `*Async` invite the opposite conclusion.

What a mirror does:

- **It defers one synchronous operation.** The wrapped call is first evaluated when the effect is driven, never when it is constructed, and it runs at most once however many times the effect is composed.
- **It closes the writer on cancellation.** Every mirror installs `close()` as its cancellation hook. If cancellation wins before the operation finishes, the writer is closed and the operation's result is discarded rather than published — the run then delivers nothing at all, so a cancelled handle must never be given to `block`.

What a mirror does not do:

- **It does not move the write to another thread.** Driving `writeAsync` calls `write` on whichever thread is driving.
- **It does not make a blocking write nonblocking.** When `write` blocks — a bounded buffer with no space, a socket with a full send window — driving `writeAsync` blocks in the same place for the same duration. `writeBytesAsync` on a `Writer.fromOutputStream` is a `java.io.OutputStream.write` behind an `Async`, and that call blocks.

:::warning[These methods are not nonblocking I/O]
A `Writer` whose `write` blocks still blocks when you drive its `*Async` mirror. The mirrors buy you deferral and a cancellation hook; they do not buy you a nonblocking writer. If you need writes that genuinely suspend rather than block, that is a different writer, not a different method on this one.
:::

Cancellation is cooperative and interrupts no thread, so the hook cannot abort a call already inside a blocking `write`. It calls `close()`, and that helps exactly when closing the writer is what releases the blocked call — which is true of a writer whose blocking wait is woken by closure, and false of one that ignores its own closed flag while parked. See [Running#cancel](../../async.md#runningcancel) for what a cancelled run does and does not stop.

### Why There Is No `concatAsync` or `contramapAsync`

The structural combinators `concat` and `contramap` have no mirrors, and that is deliberate rather than an omission. The synchronous `Writer` protocol requires `write` to return its `Boolean` immediately. A composition callback that produced an `Async` would have no honest way to report that result: `write` cannot return a pending value, and inventing one — blocking on it, or guessing `true` — would break the very protocol the page opens with. Modelling asynchronous composition needs a separate async-writer architecture, not another method here.

## Capacity and Buffering

The default `writeable()` method returns `!isClosed`—it only tells you if the writer is closed, not whether the buffer has space. Bounded implementations can override `writeable()` to reflect remaining capacity, but this is not guaranteed by the interface. The important distinction:

- **`writeable()` returns `false`**: the writer is closed (permanent state)
- **`writeable()` returns `true` but `write()` would block**: the buffer is full but not closed. What happens next is implementation-defined: a writer backed by a bounded buffer may block the calling thread until space becomes available, while the buffer-backed writers in this library instead close and return `false` once a write finds no room.

The writers behind `NioWriters.fromByteBuffer` and its typed variants close when a write finds no room: the write that fills the buffer exactly still returns `true` and leaves the writer open, and the next write returns `false` and closes it. A writer you implement yourself may instead block indefinitely waiting for space.

## Error Handling

When the writer encounters an error, signal it with `fail(error)`. By default, `fail()` closes the writer; all subsequent `write()` calls return `false`.

If you override `fail()` to store the error internally, `write()` will throw it on the next call:

```scala

class ErrorStoringWriter extends Writer[Int] {
  private var closed = false
  private var storedError: Option[Throwable] = None
  
  def isClosed = closed
  def write(a: Int): Boolean = {
    if (storedError.isDefined) throw storedError.get
    if (closed) false else true
  }
  def close() = { closed = true }
  override def fail(error: Throwable) = {
    storedError = Some(error)
    closed = true
  }
}

val w = new ErrorStoringWriter()
// w: ErrorStoringWriter = repl.MdocSession$MdocApp9$ErrorStoringWriter@6e6f2e52
w.fail(new Exception("Stream error"))
try {
  w.write(42)  // throws the stored error
} catch {
  case e: Exception => println(s"Caught: ${e.getMessage}")
}
// Caught: Stream error
```

This gives you optional error propagation: use the default `fail()` for silent closure, or override it to propagate errors as exceptions.

## Construction

Writers are created using factory methods on the companion object, from adapters wrapping Java I/O, or by direct subclassing for custom implementations:

### Creating Predefined Writers

`Writer.closed` — A pre-closed writer that rejects all writes. Useful as a base case for empty streams:

```scala
object Writer {
  def closed: Writer[Any]
}
```

Create a pre-closed writer that rejects all writes:

```scala

val w = Writer.closed
// w: Writer[Any] = zio.blocks.streams.io.Writer$$anon$1@7f00c7cd
println(w.write(42))        // false (closed)
// false
println(w.isClosed)         // true
// true
```

### Single Element

`Writer.single` — Creates a writer that accepts one element and discards it, then reports closed. It stores nothing and has no accessor for the element:

```scala
object Writer {
  def single[Elem]: Writer[Elem]
}
```

The first write returns `true`, and every later write returns `false`:

```scala

val w = Writer.single[Int]
// w: Writer[Int] = zio.blocks.streams.io.Writer$SingleWriter@7ddf11e7
println(w.write(42))
// true
println(w.write(99))
// false
println(w.isClosed)
// true
```

### Limited Capacity

`Writer.limited` — Creates a writer that accepts at most `n` elements from `inner`, then becomes closed. The dual of `Stream.take`. If `inner` closes before `n` elements are accepted, the limited writer also closes immediately without consuming the remaining capacity. 

:::note
The inner writer is not automatically closed—only the limited wrapper's `isClosed` returns `true` when the limit is reached. The inner writer stays open until someone explicitly calls `close()`.
:::

```scala
object Writer {
  def limited[Elem](inner: Writer[Elem], n: Long): Writer[Elem]
}
```

Limit a writer to accept at most n elements:

```scala

val collected = Buffer[Int]()
// collected: Buffer[Int] = ArrayBuffer(1, 2)
val inner = new Writer[Int] {
  def isClosed = false
  def write(a: Int) = { collected += a; true }
  def close() = ()
}
// inner: Writer[Int] = repl.MdocSession$MdocApp19$$anon$23@9338755

val limited = Writer.limited(inner, 2)
// limited: Writer[Int] = zio.blocks.streams.io.Writer$LimitedWriter@6877cfc2
println(limited.write(1))    // true
// true
println(limited.write(2))    // true (space available)
// true
println(limited.write(3))    // false (limit of 2 reached)
// false
println(s"Collected: $collected")  // Collected: Buffer(1, 2)
// Collected: ArrayBuffer(1, 2)
```

### I/O Adapters

`Writer.fromOutputStream` — Wraps a `java.io.OutputStream` as a `Writer[Byte]`. Calling `Writer#close` flushes and closes the underlying stream. An `IOException` from the stream is not thrown: the write returns `false` (or `Writer#writeBytes` returns `0`) and the writer closes itself. A known bug in 0.0.56: after that failed write, `Writer#close` neither flushes nor closes the stream, so it leaks. An `IOException` from `Writer#close` is ignored:

```scala
object Writer {
  def fromOutputStream(os: OutputStream): Writer[Byte]
}
```

`Writer.fromWriter` — Wraps a `java.io.Writer` as a `Writer[Char]`. Calling `Writer#close` flushes and closes the underlying writer. It handles an `IOException` the same way: the write returns `false` (or `Writer#writeAll` returns the whole chunk), the writer closes itself, and, by the same known bug, a later `Writer#close` leaves the underlying writer unflushed and open:

```scala
object Writer {
  def fromWriter(w: java.io.Writer): Writer[Char]
}
```

## Core Operations

The fundamental operations on `Writer` cover pushing elements one at a time, bulk operations, specialized writes for primitives, and state checks:

Each of these operations also has a deferred mirror, listed in [Asynchronous Writes](#asynchronous-writes) above.

### Writing Elements

`Writer#write` — Pushes one element to the writer. Returns `true` on success, `false` if the writer is closed and cannot accept more elements. A writer of your own that overrides `Writer#fail` to keep the error may throw it here; none of the library's writers does:

```scala
abstract class Writer[-Elem] {
  def write(a: Elem): Boolean
}
```

Write elements and observe the return value indicating success or closure:

```scala

val w = Writer.single[Int]
// w: Writer[Int] = zio.blocks.streams.io.Writer$SingleWriter@62d09b9f
val result1 = w.write(42)
// result1: Boolean = true
val result2 = w.write(99)  // false, already closed
// result2: Boolean = false
println(s"First: $result1, Second: $result2")
// First: true, Second: false
```

### Bulk Writing

`Writer#writeAll` — Writes every element in a chunk. Returns the suffix not delivered. If the writer is already closed, returns the entire chunk. Exceptions from individual writes propagate to the caller:

```scala
abstract class Writer[-Elem] {
  def writeAll[Elem1 <: Elem](chunk: Chunk[Elem1]): Chunk[Elem1]
}
```

Write a chunk and observe how many elements were delivered:

```scala

val w = Writer.single[Int]
// w: Writer[Int] = zio.blocks.streams.io.Writer$SingleWriter@6505801b
val chunk = Chunk(1, 2, 3)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3)
val remaining = w.writeAll(chunk)
// remaining: Chunk[Int] = IndexedSeq(2, 3)
println(s"Remaining: $remaining")  // Chunk(2, 3)
// Remaining: Chunk(2,3)
```

### Specialized Writes

For primitive types, specialized write methods take a subtype witness so that a writer backed by that primitive can override them and write the value without going through the generic `write`. The default bodies delegate to `write(value.asInstanceOf[Elem])`, so a writer that does not override them gains nothing.

`writeInt` — Specialized `Int` write. Requires implicit evidence that `Int` is a subtype of `Elem`:

```scala
abstract class Writer[-Elem] {
  def writeInt(value: Int)(implicit ev: Int <:< Elem): Boolean
}
```

`writeLong` — Specialized `Long` write:

```scala
abstract class Writer[-Elem] {
  def writeLong(value: Long)(implicit ev: Long <:< Elem): Boolean
}
```

`writeFloat` — Specialized `Float` write:

```scala
abstract class Writer[-Elem] {
  def writeFloat(value: Float)(implicit ev: Float <:< Elem): Boolean
}
```

`writeDouble` — Specialized `Double` write:

```scala
abstract class Writer[-Elem] {
  def writeDouble(value: Double)(implicit ev: Double <:< Elem): Boolean
}
```

### Byte and Character Writes

`writeByte` — Specialized byte write. Avoids boxing when `Elem = Byte`. Requires evidence that `Byte` is a subtype of `Elem`:

```scala
abstract class Writer[-Elem] {
  def writeByte(b: Byte)(implicit ev: Byte <:< Elem): Boolean
}
```

`writeBytes` — Blocking bulk byte write. Calls `writeByte` for each byte in `buf[offset, offset+len)`, stopping early if the channel closes. Returns the number of bytes successfully written:

```scala
abstract class Writer[-Elem] {
  def writeBytes(buf: Array[Byte], offset: Int, len: Int)(implicit ev: Byte <:< Elem): Int
}
```

`writeChar` — Specialized `Char` write. Requires evidence that `Char` is a subtype of `Elem`:

```scala
abstract class Writer[-Elem] {
  def writeChar(value: Char)(implicit ev: Char <:< Elem): Boolean
}
```

`writeShort` — Specialized `Short` write. Requires evidence that `Short` is a subtype of `Elem`:

```scala
abstract class Writer[-Elem] {
  def writeShort(value: Short)(implicit ev: Short <:< Elem): Boolean
}
```

`writeBoolean` — Specialized `Boolean` write. Requires evidence that `Boolean` is a subtype of `Elem`:

```scala
abstract class Writer[-Elem] {
  def writeBoolean(value: Boolean)(implicit ev: Boolean <:< Elem): Boolean
}
```

### State Checks

`Writer#isClosed` — Returns `true` if the writer is closed. Monotone: once `true`, never returns `false`:

```scala
abstract class Writer[-Elem] {
  def isClosed: Boolean
}
```

`writeable` — Returns `true` if the next `write()` would accept a value without blocking (space is available and the writer is not closed). Default returns `!isClosed`. A writer backed by a bounded buffer can override it for accuracy; none of the writers in this library does. Note the spelling: it is `writeable()`, not `writable()`; `Reader`'s counterpart is `readable()`.

```scala
abstract class Writer[-Elem] {
  def writeable(): Boolean
}
```

Check writer capacity before writing:

```scala

val w = Writer.single[Int]
// w: Writer[Int] = zio.blocks.streams.io.Writer$SingleWriter@725ef1fc
println(w.writeable())      // true
// true
w.write(42)
// res30: Boolean = true
println(w.writeable())      // false (closed after accepting one)
// false
```

## Composition

Writers can be concatenated to chain multiple sinks together, or transformed to adapt their input types:

### Concatenation

`Writer#concat` — Returns a `Writer` that writes to `this` until a write returns `false`, then switches to `next` and writes the rejected element there. `next` is evaluated once, at the switch, and never if `this` accepts every element. A write that throws propagates the exception, and from then on the writer never switches: later writes go to `this` only, a rejection returns `false`, and `next` is never used. The dual of `Reader#concat`:

```scala
abstract class Writer[-Elem] {
  def concat[Elem1 <: Elem](next: => Writer[Elem1]): Writer[Elem1]
}
```

`Writer#++` — Alias for `Writer#concat`. Syntactic sugar for composing writers:

```scala
abstract class Writer[-Elem] {
  def ++[Elem1 <: Elem](next: => Writer[Elem1]): Writer[Elem1]
}
```

Here is how concatenation switches to the next writer when the first closes:

```scala

val collected = mutable.ArrayBuffer[Int]()
// collected: ArrayBuffer[Int] = ArrayBuffer(5, 200)
val w1 = new Writer[Int] {
  def isClosed = false
  def write(a: Int) = {
    if (a < 10) { collected += a; true; }
    else false
  }
  def close() = ()
}
// w1: Writer[Int] = repl.MdocSession$MdocApp32$$anon$43@615e54e1

val w2 = new Writer[Int] {
  def isClosed = false
  def write(a: Int) = { collected += a * 10; true }
  def close() = ()
}
// w2: Writer[Int] = repl.MdocSession$MdocApp32$$anon$45@1f2f75ce

val combined = w1 ++ w2
// combined: Writer[Int] = zio.blocks.streams.io.Writer$ConcatWith@4fc9abd2
combined.write(5)
// res33: Boolean = true
combined.write(20)  // first writer rejects, switches to second
// res34: Boolean = true
println(collected.toList)  // List(5, 200)
// List(5, 200)
```

### Transformation

`Writer#contramap` — Returns a `Writer` that transforms incoming elements with `g` before passing them to this writer. All other operations (`Writer#isClosed`, `Writer#close`, `Writer#fail`) delegate unchanged:

```scala
abstract class Writer[-Elem] {
  def contramap[Elem2](g: Elem2 => Elem): Writer[Elem2]
}
```

Transform the input type before writing:

```scala

val stringWriter = new Writer[String] {
  def isClosed = false
  def write(a: String) = { println(s"Writing: $a"); true }
  def close() = ()
}
// stringWriter: Writer[String] = repl.MdocSession$MdocApp36$$anon$51@1d87efde

val intWriter = stringWriter.contramap[Int](_.toString)
// intWriter: Writer[Int] = zio.blocks.streams.io.Writer$Contramapped@1318a5f3
intWriter.write(42)   // Prints: Writing: 42
// Writing: 42
// res37: Boolean = true
```

## Closure and Error Handling

Writers support both clean closure and error closure, allowing you to signal end-of-stream gracefully or with an error condition:

### Clean Closure

`Writer#close` — Closes the writer cleanly. After this call, `write()` returns `false` and `Writer#isClosed` returns `true`. Idempotent:

```scala
abstract class Writer[-Elem] {
  def close(): Unit
}
```

### Error Closure

`Writer#fail` — Closes the writer with an error. After this call, `Writer#isClosed` returns `true`. Subclasses that override this method may cause `write()` to throw `error` on subsequent calls; the default simply delegates to `Writer#close`. Both `Writer#close` and `Writer#fail` are idempotent; only the first call wins:

```scala
abstract class Writer[-Elem] {
  def fail(error: Throwable): Unit
}
```

Close a writer with an error:

```scala

val w = Writer.single[Int]
// w: Writer[Int] = zio.blocks.streams.io.Writer$SingleWriter@7d7b6821
w.write(42)
// res39: Boolean = true
w.fail(new RuntimeException("Error"))
println(w.isClosed)     // true
// true
```

## Contravariance

`Writer` is **contravariant** in `Elem`, meaning `Writer[-Elem]` can accept narrower types. If you have a `Writer[Number]`, you can use it as a `Writer[Int]` because every `Int` is a `Number`:

```scala

trait Number
case class IntNum(value: Int) extends Number

val numberWriter = new Writer[Number] {
  def isClosed = false
  def write(a: Number) = { println(s"Number: $a"); true }
  def close() = ()
}
// numberWriter: Writer[Number] = repl.MdocSession$MdocApp42$$anon$59@1c1c97ca

// numberWriter is also a Writer[IntNum] due to contravariance
val intNumWriter: Writer[IntNum] = numberWriter
// intNumWriter: Writer[IntNum] = repl.MdocSession$MdocApp42$$anon$59@1c1c97ca
intNumWriter.write(IntNum(42))
// Number: IntNum(42)
// res43: Boolean = true
```

This is the dual of Reader's covariance: Reader is covariant (`+Elem`) because narrower elements flow out; Writer is contravariant (`-Elem`) because broader element types flow in.

## Integration with Readers and Channels

While Reader is typically used with pull-based stream operations, Writer is used internally by channel-based implementations and as an I/O adapter. The pairing is natural: a Reader pulls from a source, while a Writer pushes to a sink.

For typical stream usage, you'll see Writer indirectly when writing to files, network sockets, or other I/O resources. The `Writer.fromOutputStream` and `Writer.fromWriter` factories adapt standard Java I/O to the Writer interface.

## Implementation Notes

Understanding `Writer`'s design decisions helps you use it correctly and avoid common pitfalls:

### Push Vs Pull

`Writer` is push-based (producer-driven), contrasting with `Reader` which is pull-based (consumer-driven):

| Aspect         | Reader                     | Writer                  |
|----------------|----------------------------|-------------------------|
| **Direction**  | Source → Consumer (pull)   | Producer → Sink (push)  |
| **Variance**   | Covariant (`+Elem`)        | Contravariant (`−Elem`) |
| **Blocking**   | `read()` may block         | `write()` may block     |
| **Signal end** | Caller-supplied sentinel, or a negative count from a bulk read | `close()` or `fail()`   |
| **Dual**       | Sink drains Reader         | Producer feeds Writer   |

### Thread Safety

`Writer` is **not thread-safe** by default. It is designed for single-threaded, push-based production. Do not share a `Writer` across threads without external synchronization. If you need concurrent production, wrap the writer in a thread-safe queue or use a concurrent streaming library.

### Idempotency

Both `close()` and `fail()` are idempotent: only the first call wins. Subsequent calls have no effect. This simplifies error handling in try-finally blocks.

## Running the Examples

All code from this guide is available as runnable examples in the `streams-examples` module.

**1. Clone the repository and navigate to the project:**

Run these commands to set up the examples:

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

### Basic Writer Construction

This example demonstrates the most common writer factories: `Writer.single`, `Writer.limited`, `Writer.closed`, and custom writers via subclassing:

```scala title="streams-examples/src/main/scala/writer/WriterBasicConstructionExample.scala" 
package writer

/**
 * Demonstrates the most common Writer factories: single, limited, closed, and
 * custom writers via subclassing. Each writer is fed manually with write() to
 * show how to produce elements.
 */
object WriterBasicConstructionExample extends App {

  println("=== Writer.single ===")
  val singleWriter = Writer.single[Int]
  println(s"Write 42: ${singleWriter.write(42)}")
  println(s"Write 99 (closed): ${singleWriter.write(99)}")
  println(s"isClosed: ${singleWriter.isClosed}")

  println("\n=== Writer.closed ===")
  val closedWriter = Writer.closed
  println(s"Write to closed: ${closedWriter.write(1)}")
  println(s"isClosed: ${closedWriter.isClosed}")

  println("\n=== Writer.limited ===")
  val limitedWriter = Writer.limited(Writer.single[String], 2)
  val a             = "a"
  val b             = "b"
  val c             = "c"
  println(s"Write 'a': ${limitedWriter.write(a)}")
  println(s"Write 'b': ${limitedWriter.write(b)}")
  println(s"Write 'c': ${limitedWriter.write(c)}")
  println(s"isClosed: ${limitedWriter.isClosed}")

  println("\n=== writeAll: bulk write ===")
  val collected     = mutable.ArrayBuffer[Int]()
  val collectWriter = new Writer[Int] {
    def isClosed      = false
    def write(a: Int) = { collected += a; true }
    def close(): Unit = ()
  }

  val chunk     = Chunk(10, 20, 30)
  val remaining = collectWriter.writeAll(chunk)
  println(s"Collected: ${collected.toList}")
  println(s"Remaining: $remaining")

  println("\n=== writeable: check capacity ===")
  val capWriter = Writer.single[Int]
  println(s"writeable before: ${capWriter.writeable()}")
  capWriter.write(42)
  println(s"writeable after: ${capWriter.writeable()}")

  println("\n=== Custom Writer ===")
  val upperWriter = new Writer[String] {
    private val buffer   = mutable.ArrayBuffer[String]()
    def isClosed         = false
    def write(a: String) = {
      buffer += a.toUpperCase()
      true
    }
    def close(): Unit = println(s"Final buffer: $buffer")
  }

  upperWriter.write("hello")
  upperWriter.write("world")
  upperWriter.close()
}
```

Run this example with:

```bash
cd streams-examples && sbt "runMain writer.WriterBasicConstructionExample"
```

### Composition and Transformation

This example shows writer composition with `Writer#++` (concat), transformation with `Writer#contramap`, and bulk writes with `Writer#writeAll`:

```scala title="streams-examples/src/main/scala/writer/WriterCompositionExample.scala" 
package writer

/**
 * Demonstrates writer composition with ++ (concat), transformation with
 * contramap, and error handling via fail(). Shows how multiple writers can be
 * chained and how transformations are applied before writing.
 */
object WriterCompositionExample extends App {

  println("=== concat: ++ operator ===")
  val results = mutable.ArrayBuffer[Int]()

  val w1 = new Writer[Int] {
    def isClosed      = false
    def write(a: Int) = {
      results += a * 10
      a < 50 // reject values >= 50
    }
    def close(): Unit = ()
  }

  val w2 = new Writer[Int] {
    def isClosed      = false
    def write(a: Int) = {
      results += a * 100
      true
    }
    def close(): Unit = ()
  }

  val combined = w1 ++ w2

  combined.write(10) // accepted by w1 (10 < 50)
  combined.write(60) // rejected by w1, switches to w2
  println(s"Results: ${results.toList}")

  println("\n=== contramap: transform elements ===")
  val stringResults = mutable.ArrayBuffer[String]()
  val stringWriter  = new Writer[String] {
    def isClosed         = false
    def write(a: String) = {
      stringResults += a
      true
    }
    def close(): Unit = ()
  }

  val intWriter = stringWriter.contramap[Int](_.toString)
  intWriter.write(42)
  intWriter.write(99)
  println(s"String results: ${stringResults.toList}")

  println("\n=== Multiple contramap: chained transformations ===")
  val doubleResults      = mutable.ArrayBuffer[String]()
  val doubleStringWriter = new Writer[String] {
    def isClosed         = false
    def write(a: String) = {
      doubleResults += a
      true
    }
    def close(): Unit = ()
  }

  val intDoubleWriter = doubleStringWriter
    .contramap[Double](d => s"${d * 2}")
    .contramap[Int](i => i.toDouble)

  intDoubleWriter.write(5)  // 5 -> 5.0 -> "10.0"
  intDoubleWriter.write(10) // 10 -> 10.0 -> "20.0"
  println(s"Double results: ${doubleResults.toList}")

  println("\n=== fail: error closure ===")
  val failWriter = new Writer[Int] {
    private var closed = false
    def isClosed       = closed
    def write(a: Int)  =
      if (closed) false else { println(s"Write: $a"); true }
    def close(): Unit                         = closed = true
    override def fail(error: Throwable): Unit = {
      closed = true
      println(s"Failed with: ${error.getMessage}")
    }
  }

  failWriter.write(1)
  failWriter.fail(new RuntimeException("Oops"))
  println(s"isClosed after fail: ${failWriter.isClosed}")

  println("\n=== writeAll: bulk operations ===")
  val bulkResults = mutable.ArrayBuffer[Int]()
  val bulkWriter  = Writer.limited(
    new Writer[Int] {
      def isClosed      = false
      def write(a: Int) = { bulkResults += a; true }
      def close(): Unit = ()
    },
    2
  )

  val chunk     = Chunk(1, 2, 3, 4)
  val unwritten = bulkWriter.writeAll(chunk)
  println(s"Bulk results: ${bulkResults.toList}")
  println(s"Unwritten: $unwritten")
}
```

Run this example with:

```bash
cd streams-examples && sbt "runMain writer.WriterCompositionExample"
```

### I/O Adapters

This example demonstrates I/O integration with `Writer.fromOutputStream` and `Writer.fromWriter` for streaming to files or character streams:

```scala title="streams-examples/src/main/scala/writer/WriterIOAdapterExample.scala" 
package writer

/**
 * Demonstrates I/O integration with Writer via fromOutputStream and fromWriter.
 * Shows how to write bytes to streams and characters to writers using the
 * Writer interface.
 */
object WriterIOAdapterExample extends App {

  println("=== Writer.fromOutputStream ===")
  val byteStream = new ByteArrayOutputStream()
  val byteWriter = Writer.fromOutputStream(byteStream)

  byteWriter.write(72.toByte)  // 'H'
  byteWriter.write(105.toByte) // 'i'
  byteWriter.write(33.toByte)  // '!'
  byteWriter.close()

  println(s"Output: ${byteStream.toString("UTF-8")}")

  println("\n=== writeBytes: bulk byte write ===")
  val byteStream2 = new ByteArrayOutputStream()
  val byteWriter2 = Writer.fromOutputStream(byteStream2)

  val message      = "Hello".getBytes("UTF-8")
  val bytesWritten = byteWriter2.writeBytes(message, 0, message.length)
  byteWriter2.close()

  println(s"Bytes written: $bytesWritten")
  println(s"Output: ${byteStream2.toString("UTF-8")}")

  println("\n=== Writer.fromWriter ===")
  val charStream = new StringWriter()
  val charWriter = Writer.fromWriter(charStream)

  charWriter.write('H')
  charWriter.write('e')
  charWriter.write('l')
  charWriter.write('l')
  charWriter.write('o')
  charWriter.close()

  println(s"Output: ${charStream.toString}")

  println("\n=== writeChar: individual character writes ===")
  val charStream2 = new StringWriter()
  val charWriter2 = Writer.fromWriter(charStream2)

  val greeting = "Hi!"
  for (c <- greeting) {
    val result = charWriter2.writeChar(c)
    println(s"Write '$c': $result")
  }
  charWriter2.close()

  println(s"Output: ${charStream2.toString}")

  println("\n=== Specialized numeric writes ===")
  val charStream3 = new StringWriter()
  val charWriter3 = Writer.fromWriter(charStream3)

  // Note: These specialized methods require the writer to be typed to accept them
  // For a demo, we'll just show the interface exists
  println("Specialized write methods available:")
  println("  - writeInt(value: Int)")
  println("  - writeLong(value: Long)")
  println("  - writeFloat(value: Float)")
  println("  - writeDouble(value: Double)")
  println("  - writeBoolean(value: Boolean)")
  println("  - writeShort(value: Short)")

  charWriter3.close()

  println("\n=== Error handling in I/O ===")
  val closedStream = new ByteArrayOutputStream()
  closedStream.close()
  val failingWriter = Writer.fromOutputStream(closedStream)

  val writeResult = failingWriter.write(65.toByte) // 'A'
  println(s"Write to closed stream: $writeResult")
  println(s"Writer is closed: ${failingWriter.isClosed}")
}
```

Run this example with:

```bash
cd streams-examples && sbt "runMain writer.WriterIOAdapterExample"
```

### Bounded Implementation

This example shows how to implement a bounded Writer that wraps a fixed-capacity container and auto-closes when full. It demonstrates the protocol: `write()` returns `false` only on closure (not buffer fullness), and `writeable()` reflects closure state:

```scala title="streams-examples/src/main/scala/writer/WriterBoundedImplementationExample.scala" 
package writer

/**
 * Demonstrates implementing a bounded Writer that auto-closes when capacity is
 * reached. Shows how write() returns false only on closure, not on buffer
 * fullness, and how writeable() reflects the closure state.
 */
object WriterBoundedImplementationExample extends App {

  class BoundedWriter[A](maxCapacity: Int) extends Writer[A] {
    private val buffer = mutable.Buffer[A]()
    private var closed = false

    def isClosed: Boolean = closed

    def write(a: A): Boolean =
      if (closed) false
      else if (buffer.size < maxCapacity) {
        buffer += a
        true
      } else {
        // Buffer full: auto-close and reject
        closed = true
        false
      }

    def close(): Unit = closed = true

    override def fail(error: Throwable): Unit = close()

    def contents: mutable.Buffer[A] = buffer
  }

  println("=== Bounded Writer with auto-close ===")
  val bounded = new BoundedWriter[Int](3)

  println(s"Write 10: ${bounded.write(10)}")
  println(s"Write 20: ${bounded.write(20)}")
  println(s"Write 30: ${bounded.write(30)}")
  println(s"Write 40 (buffer full, auto-closes): ${bounded.write(40)}")
  println(s"Write 50 (closed): ${bounded.write(50)}")

  println(s"\nBuffer contents: ${bounded.contents}")
  println(s"Writer closed: ${bounded.isClosed}")
  println(s"Writeable: ${bounded.writeable()}")

  println("\n=== Behavior summary ===")
  println("• write() returns true while space exists")
  println("• When buffer fills, write() auto-closes and returns false")
  println("• All subsequent write() calls return false (closure is permanent)")
  println("• writeable() reflects closure state, not buffer capacity")
}
```

Run this example with:

```bash
cd streams-examples && sbt "runMain writer.WriterBoundedImplementationExample"
```

### Deferred and Cancellable Writes

This example makes the `*Async` caveat concrete. It shows that constructing a mirror writes nothing while driving it writes once, composes three mirrors into one effect, and then cancels a driven `writeAsync` whose write is parked — observing that cancellation closed the writer, that no element was recorded, and that the run delivers no value at all:

```scala title="streams-examples/src/main/scala/writer/WriterAsyncExample.scala" 
package writer

/**
 * The `*Async` mirrors on `Writer`, and the two properties that define them:
 * each mirror defers exactly one synchronous writer operation until the
 * returned effect is driven, and cancellation closes the writer and suppresses
 * the stale result.
 *
 * Neither property makes a write nonblocking, and neither moves it to another
 * thread. The last section only observes cancellation at all because the
 * writer's own `close()` is what releases the parked write.
 *
 * JVM only, because it ends in `.block` to turn an `Async` into a value for
 * `main`.
 */
object WriterAsyncExample {

  /** Records what actually reached the writer, so deferral is observable. */
  final class RecordingWriter extends Writer[Int] {
    private val recorded       = scala.collection.mutable.ArrayBuffer.empty[Int]
    private var closed         = false
    def isClosed: Boolean      = closed
    def write(a: Int): Boolean = if (closed) false else { recorded += a; true }
    def close(): Unit          = closed = true
    def snapshot: List[Int]    = recorded.toList
  }

  /**
   * A writer whose `write` parks until someone closes it. `close()` is the
   * cancellation hook every `*Async` mirror installs, so cancelling a driven
   * `writeAsync` is what wakes this writer up again.
   */
  final class GatedWriter extends Writer[Int] {
    private val gate             = new CountDownLatch(1)
    private val recorded         = new ConcurrentLinkedQueue[Int]
    @volatile private var closed = false

    /** Counts down once the deferred thunk has entered `write`. */
    val entered = new CountDownLatch(1)

    /** Counts down once `close()` has run. */
    val wasClosed = new CountDownLatch(1)

    /** Counts down once the parked `write` has returned. */
    val finished = new CountDownLatch(1)

    def isClosed: Boolean = closed

    def write(a: Int): Boolean = {
      entered.countDown()
      gate.await()
      val accepted =
        if (closed) false
        else { recorded.add(a); true }
      finished.countDown()
      accepted
    }

    def close(): Unit = {
      closed = true
      gate.countDown()
      wasClosed.countDown()
    }

    def recordedCount: Int = recorded.size
  }

  def main(args: Array[String]): Unit = {
    deferral()
    sequence()
    cancellation()
  }

  /** Constructing a mirror writes nothing; driving it writes exactly once. */
  private def deferral(): Unit = {
    val writer  = new RecordingWriter
    val pending = writer.writeAsync(1)

    println(s"deferral: after construction  recorded=${writer.snapshot}")
    println(s"deferral: after driving once  accepted=${pending.block}, recorded=${writer.snapshot}")
  }

  /** The mirrors compose like any other `Async`, one operation per step. */
  private def sequence(): Unit = {
    val writer = new RecordingWriter

    val program: Async[Chunk[Int]] =
      writer
        .writeAsync(10)
        .flatMap(_ => writer.writeAllAsync(Chunk(20, 30, 40)))
        .flatMap(undelivered => writer.closeAsync().map(_ => undelivered))

    val undelivered = program.block
    println(s"sequence: recorded=${writer.snapshot}, undelivered=$undelivered, closed=${writer.isClosed}")
  }

  /**
   * Cancellation closes the writer and discards the result it was about to
   * produce.
   */
  private def cancellation(): Unit = {
    val writer  = new GatedWriter
    val running = writer.writeAsync(99).start

    // Wait until the deferred thunk is parked inside `write`, so cancellation
    // races a genuinely in-flight operation rather than an unstarted one.
    writer.entered.await()
    running.cancel()

    // Cancellation ran `close()`, which released the parked `write`.
    writer.wasClosed.await()
    writer.finished.await()

    // The `false` that `write` then returned lost the race to publish, so this
    // run never delivers a value. Never call `.block` on a cancelled handle.
    println(s"cancellation: closed=${writer.isClosed}, recorded=${writer.recordedCount}")
  }
}
```

Run this example with:

```bash
cd streams-examples && sbt "runMain writer.WriterAsyncExample"
```

## See Also

- [Asynchronous Stream Execution](../execution-and-compatibility/async-execution.md#cancellation) — how cancellation reaches a stream's resources, and the asynchronous stream API the deferred mirrors sit beside
- [Async Reference](../../async.md#runningcancel) — what `Running#cancel` stops, why a cancelled run never delivers, and why the cancel hook cannot interrupt a blocked thread
- [Reader](./reader.md) — the pull-based dual of this type, and the reader kinds a sink drains
- [Sink](../core/sink.md) — the consumer side of a stream, which drains a `Reader` rather than feeding a `Writer`
- [Zero-Boxing Streams](../execution-and-compatibility/zero-boxing.md) — why the specialized write family exists and how a primitive lane is chosen
- [How to Process Binary Data Without Boxing](../../../guides/process-binary-data-without-boxing.md) — read, transform, aggregate and write `ByteBuffer`s, mapped files and `InputStream`s as unboxed `Long` and `Double` streams

---

## AttributeValue

`AttributeValue` is the closed set of eight kinds an attribute can hold — one case class per kind, so code that accepts "any attribute value" can pattern match on which one it got. It exists at the boundary of the telemetry API: `Attributes` itself never stores one (see [Attributes](./attributes.md) for the unboxed parallel-array representation it uses instead), but constructing or reading one at a boundary — a log call, a span attribute buffer, a metric label map, an exporter — needs a single type that can be any of the eight, and `AttributeValue` is it. Its eight variants:

```scala
sealed trait AttributeValue

object AttributeValue {
  final case class StringValue(value: String)           extends AttributeValue
  final case class BooleanValue(value: Boolean)         extends AttributeValue
  final case class LongValue(value: Long)               extends AttributeValue
  final case class DoubleValue(value: Double)           extends AttributeValue
  final case class StringSeqValue(value: Seq[String])   extends AttributeValue
  final case class LongSeqValue(value: Seq[Long])       extends AttributeValue
  final case class DoubleSeqValue(value: Seq[Double])   extends AttributeValue
  final case class BooleanSeqValue(value: Seq[Boolean]) extends AttributeValue
}
```

The name in this page's title is the Scala identifier; the file uses `any-value` because the type's own doc comment describes it as mirroring OpenTelemetry's `AnyValue` protobuf message, which supports the same eight shapes: a string, a boolean, a 64-bit integer, a double, and an array of each.

## Where It Shows Up

`AttributeValue` is the value type wherever an attribute is accepted without a pre-declared [`AttributeKey`](./attribute-key.md). The `Logger` API is the clearest example — every logging method takes a variadic list of name/value pairs typed against it:

```scala

val provider = LoggerProvider.builder.build()
val logger   = provider.get("com.example")

logger.info(
  "order placed",
  "orderId" -> AttributeValue.StringValue("ord_123"),
  "total"   -> AttributeValue.DoubleValue(49.99),
  "rush"    -> AttributeValue.BooleanValue(true)
)

provider.shutdown()
```

The same ADT appears on the reading side of `Attributes` — `Attributes#foreach` and `Attributes#toMap` box each stored value into an `AttributeValue` so you can match on its kind, which [Attributes](./attributes.md#iterating) covers in full, including why `Attributes#accept` is the zero-allocation alternative for a hot path. `Span#setAttribute`, `Tracer`'s span-creation attribute maps, and the label maps `Counter`/`Histogram`/`Gauge`/`UpDownCounter` accept all convert through `AttributeValue` the same way `Logger` does — construct or match on one of the eight cases, never a ninth.

## Relationship to AttributeType

Each `AttributeValue` variant corresponds to exactly one [`AttributeType`](./attribute-key.md) discriminator and one `AttributeKey` factory — the three are the same eight-way split viewed from three angles: the runtime value, the compile-time type tag, and the typed key constructor:

| `AttributeValue` variant | `AttributeType` | `AttributeKey` factory |
| ------------------------- | ---------------- | ------------------------ |
| `StringValue(String)`             | `StringType`      | `AttributeKey.string`      |
| `BooleanValue(Boolean)`           | `BooleanType`     | `AttributeKey.boolean`     |
| `LongValue(Long)`                 | `LongType`        | `AttributeKey.long`        |
| `DoubleValue(Double)`             | `DoubleType`      | `AttributeKey.double`      |
| `StringSeqValue(Seq[String])`     | `StringSeqType`   | `AttributeKey.stringSeq`   |
| `LongSeqValue(Seq[Long])`         | `LongSeqType`     | `AttributeKey.longSeq`     |
| `DoubleSeqValue(Seq[Double])`     | `DoubleSeqType`   | `AttributeKey.doubleSeq`   |
| `BooleanSeqValue(Seq[Boolean])`   | `BooleanSeqType`  | `AttributeKey.booleanSeq`  |

`AttributeType` never appears on its own in application code — it's the tag `AttributeKey[A]#type` carries so a key and a stored value can be checked against each other without reflecting on `A` itself.

## OTLP JSON Mapping

The [OTLP exporter](../otel/index.md) converts every `Attributes` set to OTLP's `AnyValue` JSON shape by boxing each entry into an `AttributeValue` and matching on it. The eight cases map onto four wire keys, with sequences nesting under `arrayValue`:

| `AttributeValue` variant | OTLP JSON field | Notes |
| -------------------------- | ------------------ | ------- |
| `StringValue`     | `"stringValue"`                              | plain JSON string |
| `BooleanValue`    | `"boolValue"`                                | plain JSON boolean |
| `LongValue`       | `"intValue"`                                 | JSON string, per OTLP's `int64` mapping — `42L` becomes `"42"` |
| `DoubleValue`     | `"doubleValue"`                              | plain JSON number |
| `StringSeqValue`  | `"arrayValue":{"values":[{"stringValue":...}, ...]}` | one wrapped element per entry |
| `LongSeqValue`    | `"arrayValue":{"values":[{"intValue":"..."}, ...]}`  | each element quoted, same as scalar `LongValue` |
| `DoubleSeqValue`  | `"arrayValue":{"values":[{"doubleValue":...}, ...]}` | |
| `BooleanSeqValue` | `"arrayValue":{"values":[{"boolValue":...}, ...]}`   | |

## See Also

- [Attributes](./attributes.md#iterating) — `Attributes#foreach`/`Attributes#toMap` (which box into `AttributeValue`) versus `Attributes#accept` (which doesn't)
- [AttributeKey](./attribute-key.md) — the typed key that pairs with a specific `AttributeValue` variant for allocation-free reads
- [OTLP Export](../otel/index.md) — where `AttributeValue` becomes OTLP's `AnyValue` JSON representation
- [Common Types](./index.md) — the types every telemetry pillar uses

---

## AttributeKey

`AttributeKey[A]` is the type-safe key type for reading and writing entries in an `Attributes` collection. It binds a string name to a specific value type `A` via an `AttributeType` discriminator, so `Attributes#get[A](key)` returns `Option[A]` without casting. There are eight supported value types, each with its own factory method on the companion object.

```scala
sealed trait AttributeKey[A] {
  def name: String           // the string label used as the key
  def `type`: AttributeType  // discriminator: StringType, LongType, DoubleType, BooleanType, + 4 Seq variants
}

object AttributeKey {
  // Scalar factory methods
  def string(name: String):     AttributeKey[String]
  def boolean(name: String):    AttributeKey[Boolean]
  def long(name: String):       AttributeKey[Long]
  def double(name: String):     AttributeKey[Double]

  // Sequence factory methods
  def stringSeq(name: String):  AttributeKey[Seq[String]]
  def longSeq(name: String):    AttributeKey[Seq[Long]]
  def doubleSeq(name: String):  AttributeKey[Seq[Double]]
  def booleanSeq(name: String): AttributeKey[Seq[Boolean]]
}
```

## Usage

```scala

// Typed factory methods
val envKey      = AttributeKey.string("environment")
val retryKey    = AttributeKey.long("retry.count")
val debugKey    = AttributeKey.boolean("debug.enabled")
val latencyKey  = AttributeKey.double("latency.ms")
val tagsKey     = AttributeKey.stringSeq("tags")

// Build an Attributes collection using typed keys
val attrs = Attributes.builder
  .put(envKey, "production")
  .put(retryKey, 3L)
  .put(debugKey, false)
  .put(latencyKey, 12.5)
  .put(tagsKey, Seq("important", "v2"))
  .build
```

## Passing to Span and Metric APIs

`AttributeKey` is used directly in `Span#setAttribute` and `Attributes.of`:

```scala

val provider = TracerProvider.builder.build()
val tracer   = provider.get("com.example")

val dbSystem = AttributeKey.string("db.system")
val rowCount = AttributeKey.long("db.rows")

tracer.span("db.query", SpanKind.Client) { span =>
  span.setAttribute(dbSystem, "postgresql")
  span.setAttribute(rowCount, 42L)
}

provider.shutdown()
```

## Predefined Constants

`Attributes.ServiceName` and `Attributes.ServiceVersion` are predefined `AttributeKey[String]` constants for the standard OTel `"service.name"` and `"service.version"` attribute names. Use them instead of string literals to avoid typos:

```scala

val attrs = Attributes.of(Attributes.ServiceName, "order-service")
assert(attrs.get(Attributes.ServiceName) == Some("order-service"))
```

---

## Attributes

`Attributes` is the metadata carried alongside a signal — the `http.route` on a span, the `orderId` on a log record, the `method=GET` on a measurement. It's a set of key-value pairs, and every signal the telemetry module emits has one.

It exists because a signal without detail can't answer questions. "A request was slow" is useless; "a `GET` on `/orders` that returned `503` was slow" is something you can search for, filter on, and group by. Attributes are what carry that detail into your backend.

Two properties shape the type. Values are reached through a typed [`AttributeKey`](./attribute-key.md), so reading one gives you back a `String` or a `Long` rather than something you must cast. And because a set is built on every span and every measurement — the hottest paths in an instrumented application — primitives are stored without boxing, and a finished set is immutable, so it can be shared rather than copied.

## Building a Set

Three ways in, depending on how many pairs you have:

```scala

val none = Attributes.empty

val one = Attributes.of(Attributes.ServiceName, "payments")

val several = Attributes.builder
  .put(AttributeKey.string("http.route"), "/orders")
  .put("http.status_code", 200L)
  .put("cache.hit", false)
  .build
```

`of` takes exactly one pair, so reach for `builder` as soon as you have two. The builder accepts either a typed key or a plain name with a `String`, `Long`, `Double`, or `Boolean` value, and putting the same key twice keeps the last value. `Attributes.ServiceName` and `ServiceVersion` are predefined keys for the conventional OpenTelemetry names; define your own with [`AttributeKey`](./attribute-key.md).

## Reading a Value

`get` takes a key and returns the value at that key's declared type:

```scala

val route = AttributeKey.string("http.route")
val attrs = Attributes.of(route, "/orders")

val found: Option[String] = attrs.get(route)               // Some("/orders")
val missing = attrs.get(AttributeKey.string("db.system"))  // None
```

The `Option` covers a key that isn't there. `size` and `isEmpty` report how many pairs a set holds without reading any of them.

## Merging Two Sets

`++` combines sets, which is how a signal's own attributes join those of its surroundings:

```scala

val env = AttributeKey.string("deployment.environment")

val base = Attributes.of(env, "staging")
val merged = base ++ Attributes.of(env, "production")

merged.get(env)  // Some("production") — the right side wins
```

Merging appends rather than deduplicates, so a key on both sides stays in the set twice: the right side wins every lookup, but `size` counts it twice and anything iterating the set — an exporter included — sees it twice. Where two sets may share a key, build one set with a builder instead.

## Iterating

Reading a whole set back — to export it, or to check it in a test — comes in two forms, and the difference is worth understanding before you pick one.

A `Long` you store here is kept as a plain number, not as an object. Handing it to code that accepts *any* kind of value means wrapping it in an object first — **boxing** — and that wrapper is a small allocation, per attribute, every time. Fine once in a test; not fine on a path that runs on every request.

`accept` avoids it by asking you for one method per kind of value. Each method's parameter is that exact type, so the number is passed as a number and nothing is wrapped:

```scala

val attrs = Attributes.builder.put("http.route", "/orders").put("http.status_code", 200L).build

attrs.accept(new AttributeVisitor {
  def visitString(key: String, value: String): Unit   = println(s"$key=$value")
  def visitLong(key: String, value: Long): Unit       = println(s"$key=$value")
  def visitDouble(key: String, value: Double): Unit   = println(s"$key=$value")
  def visitBoolean(key: String, value: Boolean): Unit = println(s"$key=$value")
})
```

Those four are the only methods you must write. There are four more for sequence values, and they already do nothing — so if you store a sequence attribute and don't override its method, `accept` passes over it in silence. Nothing fails; the attribute simply never reaches your exporter.

The other form is for when convenience matters more than a few allocations, which is most code outside an exporter. `foreach` and `toMap` do the wrapping for you, handing each value over as an [`AttributeValue`](./any-value.md) — one wrapper per kind, so you can match on which kind you got:

```scala

val attrs = Attributes.of(Attributes.ServiceName, "payments")

attrs.foreach { (key, value) =>
  value match {
    case AttributeValue.StringValue(s) => println(s"$key is text: $s")
    case other                         => println(s"$key is $other")
  }
}

val asMap: Map[String, AttributeValue] = attrs.toMap
```

Two sets holding the same pairs are equal regardless of the order they were built in, so a test can compare an expected set directly against a recorded one.

## See Also

- [AttributeKey](./attribute-key.md) — defining typed keys
- [AttributeValue](./any-value.md) — the boxed wrapper `foreach`/`toMap` hand back
- [Resource](./resource.md) — service identity, itself an `Attributes` set
- [Common Types](./index.md) — the types every pillar uses

---

## Common Types

These four types describe every signal, across all three telemetry pillars — tracing, logging, and metrics. Every span, log record, and metric data point is annotated with `Attributes`; every provider accepts a `Resource`; every scope instance is tagged with an `InstrumentationScope`. Understanding these four types unlocks the rest of the module.

## Types

| Type | Description |
|------|-------------|
| [Attributes](./attributes.md) | Immutable, unboxed parallel-array collection of typed key-value pairs. Carried by every signal: `Span`, `LogRecord`, `Counter`, `Resource`, and `InstrumentationScope`. |
| [AttributeKey](./attribute-key.md) | Typed key for an `Attributes` entry. Binds a string name to one of eight value types (`String`, `Long`, `Double`, `Boolean`, and their `Seq` variants). |
| [AttributeValue](./any-value.md) | Boxed existential wrapper for the same eight kinds, used at API boundaries — `Logger`'s vararg attributes, `Span#setAttribute`, metric label maps — where the type isn't known ahead of time. |
| [Resource](./resource.md) | Immutable `Attributes` wrapper that describes the entity producing telemetry (service name, SDK version, deployment environment). Shared across all three providers. |
| [InstrumentationScope](./instrumentation-scope.md) | Named and versioned identity for the library or component that created a signal. Set when calling `TracerProvider.get`, `LoggerProvider.get`, or `MeterProvider.get`. |

## Design

All four types are designed for hot-path telemetry code:

- **`Attributes`** stores `Long`, `Double`, and `Boolean` values unboxed in a parallel primitive array via bit-casting. Reading a value back boxes it — through `get`, `foreach`, or a conversion to `Map` — so hot paths that only write stay allocation-free.
- **`AttributeKey`** ties a name to a value type, so `Attributes#get` returns a typed `Option`. Define each key once as a `val` and reuse it: two keys with the same name read the same attribute whatever their type.
- **`Resource`** is immutable and stamped onto a signal when it is created, not at export: `Tracer` puts it on every span and `Logger` on every log record. Metric instruments do not carry it — a `MeterProvider` holds a `Resource`, but the `Meter` and the collected `MetricData` never see it.
- **`InstrumentationScope`** is constructed once per scope by the provider and referenced from all signals that scope creates.

## See Also

- [Telemetry](../index.md) — module overview and the three-pillar architecture
- [Tracing](../tracing/index.md) — span lifecycle types that use `Attributes` and `Resource`
- [Logging](../logging/index.md) — structured log record types that carry `Attributes`
- [Metrics](../metrics/index.md) — dimensional measurement types that use `Attributes` as labels

---

## InstrumentationScope

`InstrumentationScope` records which library or component produced a signal. It's what lets a backend tell spans from your own code apart from spans emitted by a dependency, and group or filter by either.

```scala
final case class InstrumentationScope(
  name:       String,
  version:    Option[String] = None,
  attributes: Attributes     = Attributes.empty
)
```

You rarely name the type: it's built for you from the name you pass to `TracerProvider#get`, `LoggerProvider#get`, or `MeterProvider#get`, and every [`Span`](../tracing/span.md), [`SpanData`](../tracing/span-data.md), [`LogRecord`](../logging/log-record.md), [`Tracer`](../tracing/tracer.md), and [`Meter`](../metrics/meter.md) then carries it.

---

## Resource(Common)

`Resource` is an immutable wrapper around `Attributes` that describes the entity producing telemetry — the service, container, host, or runtime. It is stamped on every span and every log record so that a backend can group signals by their origin. All three provider types (`TracerProvider`, `LoggerProvider`, `MeterProvider`) accept a `Resource` at configuration time; `Tracer` and `Logger` propagate it onto the signals they emit, while metric instruments do not carry it — collected `MetricData` holds only the per-measurement `Attributes`.

```scala
final case class Resource(attributes: Attributes)

object Resource {
  val empty:   Resource                       // no attributes
  val default: Resource                       // service.name + telemetry.sdk.* attributes
  def create(attrs: Attributes): Resource

  // merge comes from an implicit class on the companion
  implicit class ResourceOps(val self: Resource) {
    def merge(other: Resource): Resource      // other's attributes take precedence on duplicate keys
  }
}
```

## Predefined Instances

| Instance           | Description                                                                                                                                                          |
|--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `Resource.empty`   | A `Resource` with no attributes. Use as a neutral base.                                                                                                              |
| `Resource.default` | Pre-populated with `service.name = "unknown_service"`, `telemetry.sdk.name = "zio-blocks"`, `telemetry.sdk.language = "scala"`, and the SDK version from build info. |

---

## Telemetry

The telemetry module provides the three pillars of modern observability — distributed tracing, structured logging, and dimensional metrics — with a single coherent API aligned to the OpenTelemetry data model. 

Three global entry points (`trace`, `log`, `metric`) work without any configuration. `trace` and `metric` delegate to an installed provider (`TracerProvider`, `MeterProvider`) that produces scope-specific instances (`Tracer`, `Meter`); `log` holds an installed `Logger` directly, which a `LoggerProvider` is one way to build. 

The same metadata types — `Attributes`, `AttributeKey`, `Resource`, and `InstrumentationScope` — describe signals in all three pillars, and a shared `ContextStorage` enables automatic trace–log correlation out of the box.

The three-pillar structure mirrors the following shape:

```scala
// Global entry points — work without any setup
object trace  { /* span creation, provider install */ }
object log    { /* structured log emission, writer config */ }
object metric { /* instrument factories, reader access */ }

// Provider layer — configure and install once at application startup
final class TracerProvider private[telemetry] (...) extends AutoCloseable
final class LoggerProvider private[telemetry] (...) extends AutoCloseable
final class MeterProvider  private[telemetry] (...) extends AutoCloseable

// Scope-specific instances — created by providers on demand
final class Tracer private[telemetry] (...)
final class Logger private[telemetry] (...)
final class Meter  private[telemetry] (...)
```

## Motivation

Modern services produce three distinct kinds of observability data. Traces capture the causal shape of a request as it traverses services and threads. Logs capture structured events at discrete points in time, ideally correlated to the active trace. Metrics capture aggregated numerical measurements — request counts, latencies, queue depths — that accumulate over time. The telemetry module covers all three in one library that pulls in no third-party dependencies and cross-compiles for the JVM and Scala.js.

## Installation

```scala
// Core: logging, tracing, metrics
libraryDependencies += "dev.zio" %% "zio-blocks-telemetry" % "0.0.56"

// Optional: OTLP JSON export over HTTP (JVM only)
libraryDependencies += "dev.zio" %% "zio-blocks-telemetry-otel" % "0.0.56"
```

For Scala.js (telemetry core only):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-telemetry" % "0.0.56"
```

On Scala 3.9 and newer, use the `zio-blocks-next-telemetry` artifact and enable experimental features:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-next-telemetry" % "0.0.56"
scalacOptions += "-experimental"
```

Supports Scala 2.13.x and 3.x. On the JVM, the module needs JDK 21 or newer, and that includes code that only calls `log.*`: the log macros run inside the compiler, so on JDK 17 even compiling a `log.info` call fails with `UnsupportedClassVersionError` (class file version 65.0).

:::warning[Logging on Scala 2.13]
A known bug in 0.0.56: on Scala 2.13, every `log.*` call fails to compile outside the `zio.blocks.telemetry` package, with `object GlobalLogState in package telemetry cannot be accessed`. Scala 3 is not affected. Tracing and metrics work on Scala 2.13.
:::

## Overview

The module is organized into three areas — tracing, logging, and metrics — that map directly to the OTel signal types, plus the metadata types all three rely on.

### Tracing

[Tracing](./tracing/index.md) covers the full span lifecycle. `TracerProvider` is a factory that shares a `Resource`, a `Sampler`, and a set of `SpanProcessor` hooks across all `Tracer` instances it creates. A `Tracer` opens and closes `Span` scopes, consulting the `Sampler` on each new span and calling `SpanProcessor.onStart` and `SpanProcessor.onEnd` for exporting or in-memory collection. Supporting types — `SpanContext`, `SpanData`, `SpanKind`, `SpanStatus`, `SpanId`, `TraceId`, and `TraceFlags` — capture the identity and state of a span.

### Logging

[Logging](./logging/index.md) covers structured, severity-leveled log emission. `LoggerProvider` is a factory that shares a `Resource` and a set of `LogRecordProcessor` hooks across all `Logger` instances it creates. A `Logger` emits `LogRecord` snapshots through configured processors; processors can write formatted output via a `LogFormatter` and `LogWriter`, export records to an external system, or gate emission by `Severity`. The macro-generated methods on the global `log` object capture source location at compile time and automatically stamp the active span context into every emitted record.

### Metrics

[Metrics](./metrics/index.md) covers dimensional cumulative and instantaneous measurements. `MeterProvider` is a factory for `Meter` instances; each `Meter` creates and registers instruments — `Counter`, `UpDownCounter`, `Histogram`, and `Gauge` — and each instrument keeps one series per `Attributes` set it sees. Instruments themselves are not cached by name; build each once and hold it. A `MetricReader` collects all registered instruments into `MetricData` snapshots on demand for export or inspection.

### Common Types

A span, a log record, and a measurement are different kinds of data, but they answer the same three questions: what happened, which service produced it, and which component inside that service. Four types carry those answers, and because all three pillars use the same four, whatever you learn about them applies everywhere:

- [`Attributes`](./common/attributes.md) — the detail on a signal, as typed key-value pairs: the route on a span, the order id on a log record, the labels on a measurement.
- [`AttributeKey`](./common/attribute-key.md) — a name paired with the type of its value, so reading an attribute back gives you a `String` or a `Long` rather than something to cast.
- [`Resource`](./common/resource.md) — which service produced the signal, attached to every one the provider emits. Give all three providers the same one.
- [`InstrumentationScope`](./common/instrumentation-scope.md) — which library or component produced it, taken from the name you pass to `TracerProvider.get`, `LoggerProvider.get`, or `MeterProvider.get`.

## How They Work Together

All three pillars follow the same structural pattern: a global singleton wraps a provider that acts as a factory for lightweight, scope-specific instances. The provider holds the shared configuration — `Resource`, processor lists, and `ContextStorage` — and creates instances on demand. Signals are enriched by `Attributes` and labeled by `InstrumentationScope`. The `ContextStorage[Option[SpanContext]]` instance that `TracerProvider` and `LoggerProvider` share is what makes trace–log correlation automatic. Give all three providers the *same* `Resource` rather than one each — identical service-identity attributes on every signal are what let a backend line up traces, logs, and metrics as one service.

```
  Global entry points          Provider layer                 Scope instances
                             ┌─────────────────────────┐   ┌──────────────────┐
                             │  TracerProvider         │──▶│  Tracer          │──▶ Span
  ┌──────────────────────┐   │  · Resource             │   └──────────────────┘    │ SpanProcessor
  │  trace  (object)     │──▶│  · Sampler              │         ContextStorage ◀╴╴┤
  └──────────────────────┘   │  · SpanProcessors       │                           │ (shared)
                             │  · ContextStorage ╶╶╶╶╶╶┼╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╯
                             └─────────────────────────┘   ┌──────────────────┐
                             ┌─────────────────────────┐   │  Logger          │
  ┌──────────────────────┐   │  LoggerProvider         │   │  (traceId+spanId │──▶ LogRecord
  │  log    (object)     │──▶│  · Resource             │──▶│   from storage)  │    LogRecordProcessor
  └──────────────────────┘   │  · LogRecordProcessor   │   └──────────────────┘
                             │  · ContextStorage ╶╶╶╶╶╶┼╶╶╶╶╶╶╶(same instance)
                             └─────────────────────────┘
                             ┌─────────────────────────┐   ┌──────────────────┐
  ┌──────────────────────┐   │  MeterProvider          │   │  Meter           │──▶ Counter
  │  metric (object)     │──▶│  · Resource             │──▶│  · counterBuilder│    Histogram
  └──────────────────────┘   │  · MeterRegistry        │   │  · histogramBld. │    Gauge
                             └─────────────────────────┘   └──────────────────┘    UpDownCounter
                                                                    │
                                                              MetricReader ──▶ MetricData

  ──────────────── Shared across all three pillars ─────────────────────────────────
        Attributes  ·  AttributeKey  ·  Resource  ·  InstrumentationScope
```

The following sub-sections walk through the data flow for each pillar and explain the correlation mechanism.

### Tracing Data Flow

When application code calls `trace.span("operation")`, the following sequence occurs:

1. The global `trace` object delegates to the active `TracerProvider`, which creates or retrieves a `Tracer` for the `"default"` instrumentation scope.
2. The `Tracer` consults its `Sampler` via `shouldSample`. If the decision is `RecordAndSample`, a `RecordingSpan` is constructed via `SpanBuilder.startSpan` and `SpanProcessor.onStart` is called; if `Drop`, a `Span.NoOp` is returned — the span records nothing and reaches no processor, though a `SpanContext` is still allocated and scoped so nested spans and log records stay on the same trace.
3. The active `SpanContext` is stored into `ContextStorage` for the duration of the user block so that nested spans can read it as their parent.
4. On exit, `Span.end()` is called, `SpanData` is snapshotted, and `SpanProcessor.onEnd` fires — delivering the completed span to exporters or the in-memory test processor.

The following example shows a minimal production tracing configuration:

```scala

val tracerProvider = TracerProvider.builder
  .setResource(Resource.create(Attributes.of(Attributes.ServiceName, "order-service")))
  .setSampler(AlwaysOnSampler)
  .build()

trace.install(tracerProvider)

trace.span("process-order", SpanKind.Server) { span =>
  span.setAttribute("order.id", "ord-123")
  span.addEvent("validation-passed")
}
```

### Logging Data Flow

When application code calls `log.info("message", ...)`, the macro captures the call site's source file, enclosing class, method name, and line number at compile time. At runtime the severity is compared against the configured minimum; if the call passes the threshold, the captured source-location attributes are written into an `AttributesBuilder`, the current `SpanContext` is read from the shared `ContextStorage`, and `traceIdHi`, `traceIdLo`, `spanId`, and `traceFlags` are stamped into the record as unboxed primitives before being dispatched to all configured processors.

The following example routes log output to standard output in human-readable text format:

```scala

log.writer(TextLogFormatter, StdoutWriter)
log.info("order placed", "orderId" -> "ord-123", "amount" -> 99L)
```

### Metrics Data Flow

When a `Counter`, `Histogram`, `Gauge`, or `UpDownCounter` is created via `Meter.counterBuilder("name").build()`, the instrument is registered in the owning `Meter`'s internal instrument list. The `MeterProvider.reader` property returns a `MetricReader` that, on each `collectAllMetrics()` call, iterates every registered `Meter` and invokes `collect()` on each instrument, aggregating the results into a sequence of `MetricData` variants for export or inspection.

The following example records request counts and then reads the aggregated snapshot:

```scala

val requests = metric.counter("http.requests")
requests.add(1, "method" -> "GET", "status" -> "200")

val snapshots = metric.reader.collectAllMetrics()
snapshots.foreach {
  case MetricData.SumData(points)       => points.foreach(p => println(p.value))
  case MetricData.HistogramData(points) => points.foreach(p => println(p.count))
  case MetricData.GaugeData(points)     => points.foreach(p => println(p.value))
}
```

### Trace–Log Correlation

`TracerProvider` and `LoggerProvider` share the same `ContextStorage[Option[SpanContext]]` instance. When a `Tracer` opens a span it stores the `SpanContext` in that storage; when `Logger` emits a record it reads the same storage and stamps `traceIdHi`, `traceIdLo`, `spanId`, and `traceFlags` into the `LogRecord` fields as primitives — no boxing, no formatting until a `LogFormatter` runs. Because both providers default to the same internal `defaultSpanContextStorage` singleton, correlation is automatic when both are installed with their default settings.

To use an explicit shared `ContextStorage` — for example, to isolate correlation in tests — create one instance and pass it to both builders:

```scala

val sharedStorage = ContextStorage.create[Option[SpanContext]](None)

val tp = TracerProvider.builder.setContextStorage(sharedStorage).build()
val lp = LoggerProvider.builder.setContextStorage(sharedStorage).build()

trace.install(tp)
log.install(lp.get("com.example"))
```

## Common Patterns

The following patterns appear consistently across services that use this module. Each fits into a shared startup and request-handling lifecycle.

### Zero-Setup Global API

All three global entry points work immediately after import: `trace` buffers spans in memory, `log` prints readable text to stdout, and `metric` writes to an in-process `MeterProvider`. The following import is the only requirement for development and testing:

```scala

trace.span("bootstrap") { span =>
  span.setAttribute("step", "init")
}

log.writer(TextLogFormatter, StdoutWriter)
log.info("server started", "port" -> 8080L)

metric.counter("startup.count").add(1)
```

### Production Provider Installation

At application startup we configure all three providers once and install them before serving any requests. Sharing a common `Resource` propagates the service identity through all signals:

```scala

val serviceResource = Resource.default.merge(
  Resource.create(Attributes.of(Attributes.ServiceName, "payments"))
)

val tp = TracerProvider.builder
  .setResource(serviceResource)
  .setSampler(ParentBasedSampler(AlwaysOnSampler))
  .build()

val lp = LoggerProvider.builder.setResource(serviceResource).build()
val mp = MeterProvider.builder.setResource(serviceResource).build()

trace.install(tp)
log.install(lp.get("com.example.payments"))
metric.install(mp)
```

### Library Dependency-Injection Pattern

Library code should accept `Tracer` and `Logger` as constructor parameters rather than using the global singletons. This makes the library testable in isolation and avoids coupling to global state. The `Logger.info` method on the class itself takes `(String, AttributeValue)` pairs rather than the macro enrichment tuples used by the global `log` object:

```scala

final class OrderService(tracer: Tracer, logger: Logger) {
  def placeOrder(id: String): Unit =
    tracer.span("orders.place") { span =>
      span.setAttribute("order.id", id)
      logger.info("order placed", "orderId" -> AttributeValue.StringValue(id))
    }
}
```

Application startup code injects instances by calling `trace.get("com.example.orders")` and `loggerProvider.get("com.example.orders")`.

### Scoped Log Annotations

`log.annotated` propagates key-value pairs to every `log.*` call within the block. Annotations are scoped to the enclosing thread and do not leak outside it:

```scala

def handleRequest(requestId: String): Unit =
  log.annotated("requestId" -> requestId, "env" -> "prod") {
    log.info("processing started")
    log.info("processing done")
  }
```

### Rate-Limited Logging

The `*Every` family of log methods emits at most once every N invocations per call site. The counter is keyed by the call site's file and line, so sites with the same interval value are normally independent — though the keys index a fixed 4096-slot table, so two sites can collide and share one counter:

```scala

def heartbeat(): Unit =
  log.infoEvery(100, "heartbeat tick")
```

### Labeled Instruments

`Meter.labeledCounter`, `Meter.labeledHistogram`, and `Meter.labeledGauge` declare label names once and accept positional string values at recording time, matching the Prometheus-style label pattern:

```scala

val meter    = metric.get("com.example")
val requests = meter.labeledCounter("http.requests", "method", "status")

requests.add(1, "GET",  "200")
requests.add(1, "POST", "500")
```

## Integration Points

Scoped, thread-safe propagation of `SpanContext` through call stacks is handled by the module's own `ContextStorage`, which is backed by JDK `ScopedValue` on the JVM and by a saved-and-restored variable on Scala.js. At the build level the module depends only on the `context` and `chunk` blocks and requires no external libraries.

`SpanProcessor` and `LogRecordProcessor` are open traits. External exporters — for example, an OpenTelemetry SDK bridge — implement these traits to receive span and log data and forward it to collection infrastructure such as OTLP endpoints. Plug a custom `SpanProcessor` into `TracerProvider.builder.addSpanProcessor(...)` or a custom `LogRecordProcessor` into `LoggerProvider.builder.addLogRecordProcessor(...)`.

`LogEnrichment` is a typeclass resolved at compile time by the macro-generated `log.*` methods. Built-in instances cover `Throwable`, `Attributes`, `Severity`, and `(String, A)` pairs for `A ∈ {String, Long, Int, Double, Boolean}`. Adding support for a custom enrichment type requires only a new `implicit val LogEnrichment[MyType]` in scope at the `log.*` call site.

## See Also

- [Instrument a Program with Logs, Traces and Metrics](../../guides/instrument-a-program-with-logs-traces-and-metrics.md) — a step-by-step tutorial for newcomers: logs, nested spans, a counter and a histogram
- [Telemetry Guide](../../guides/telemetry-guide.md) — architecture, design trade-offs, and real-world usage patterns
- [Build a Small JSON Backend from Blocks](../../guides/small-json-backend-from-blocks.md) — capstone tutorial that wraps a request handler with logs, spans and a counter
- [How to Test Code That Logs, Traces and Records Metrics](../../guides/test-telemetry.md) — capture spans, log records and counter values in memory and assert on them in tests
- [Common Types](./common/index.md) — `Attributes`, `AttributeKey`, `Resource`, and `InstrumentationScope` shared across all three pillars
- [Context](../context.md) — `Context[R]`, which the otel module's `OtelContext` carries an active `SpanContext` through

---

## Logging

Logging records what your application did as it runs, so you can understand its behavior and diagnose problems after the fact. Unlike a plain `println`, these logs are **structured** — each entry carries typed key/value fields (an order id, a duration) you can search and filter on, not just a line of text — and **severity-leveled**, tagged `trace`, `debug`, `info`, `warn`, `error`, or `fatal` so you can keep production quiet and turn up detail only when debugging. Each log is also tied back to the trace that produced it, so from one log line you can pull up the whole request it belongs to. 

You write logs through the `log` object: call `log.info("order placed")` (or `debug`, `warn`, `error`, …) anywhere in your code and it works immediately, with no setup. Behind each call, `log` automatically records where the log came from — the file, class, method, and line — and stamps it with the trace and span of whatever [`trace.span`](../tracing/index.md) you happen to be inside. That stamping is what lets logs and traces line up later: no need to thread a request id through your code by hand.

Out of the box those records print to stdout as readable text, so `log.info(...)` shows up on your console with nothing configured. When you want a different destination you add one once, at application startup: `log.writer(...)` adds a channel with a formatter of your choosing — JSON, say — through a [`LogWriter`](./log-writer.md), while `log.addProcessor(...)` wires up a pipeline that ships records to a backend for storage and search. Those two are additive, so the default console output stays alongside whatever you add. `log.install(...)` is the replacing move: it swaps in a whole logger, dropping the default console processor, any writers added earlier, and any per-package severity overrides.

After the message you attach context, and this is where structured logging pays off. The common case is key/value pairs — `log.info("order placed", "orderId" -> "ord-123", "amount" -> 99L)` — which become searchable fields on that entry instead of being buried in the text, so later you can query "all logs where `orderId = ord-123`". Pass a `Throwable` and its type, message, and stack trace are captured for you.

A few other values mean something specific rather than becoming a field: a [`Severity`](./severity.md) overrides the entry's level, an [`Attributes`](../common/attributes.md) set adds many fields at once, and a plain `String` replaces the message body. And when you want to log one of your own types directly, give it a [`LogEnrichment`](./log-enrichment.md) instance that tells `log` how to turn it into fields.

## Example Usage

Logging's core job is to **emit structured, correlated logs**. Point `log` at a writer once, then emit records inside your spans; each record carries its typed key-value context and the active trace and span IDs automatically.

```scala

trace.install(TracerProvider.builder.build())
log.writer(TextLogFormatter, StdoutWriter)

trace.span("checkout") { _ =>
  log.info("order placed", "orderId" -> "ord-123", "amount" -> 99L)
  log.warn("inventory low", "sku" -> "sku-42", "remaining" -> 3L)
}
```

## Emit Structured, Severity-Leveled Records

Six severities — `trace`, `debug`, `info`, `warn`, `error`, `fatal` — cover the whole scale.

```scala
object log {
  def trace(message: String, enrichments: Any*): Unit
  def debug(message: String, enrichments: Any*): Unit
  def info(message: String, enrichments: Any*): Unit
  def warn(message: String, enrichments: Any*): Unit
  def error(message: String, enrichments: Any*): Unit
  def fatal(message: String, enrichments: Any*): Unit
}
```

Pass typed key/value pairs for structured attributes, a `Throwable` to capture an exception, or an `Attributes` set to merge many values at once.

```scala

log.info("order placed", "orderId" -> "ord-123", "amount" -> 99L, "express" -> true)
log.debug("cache lookup", "hit" -> false)

try throw new RuntimeException("payment declined")
catch { case e: Throwable => log.error("charge failed", "orderId" -> "ord-123", e) }
```

## Limit Log Volume at Hot Call Sites

Two rate-limiting families, each spanning all six severities, keep high-frequency sites quiet.

```scala
object log {
  def <level>Every(every: Int, message: String, enrichments: Any*): Unit
  def <level>AtMost(intervalMillis: Long, message: String, enrichments: Any*): Unit
}
```

The `Every` family is count-based — `<level>Every(every, message, enrichments*)` emits on every Nth call at that site. The `AtMost` family is time-based — `<level>AtMost(intervalMillis, message, enrichments*)` emits at most once per interval at that site. Each site gets its own counter and clock, keyed by its file and line into a fixed table of 4096 slots; two distant sites can land in the same slot, in which case they share a counter and one of them logs less often than its `every` suggests.

```scala

// Count-based: one line for every 100th retry
log.warnEvery(100, "retrying upstream call", "endpoint" -> "/inventory")

// Time-based: at most one line per 5 seconds, whatever the call rate
log.infoAtMost(5000L, "processing batch", "size" -> 512L)
log.errorAtMost(1000L, "connection pool exhausted")
```

## Attach Scoped Annotations

Attach key/value pairs to every record emitted inside a block with `annotated`.

```scala
object log {
  def annotated[A](annotations: (String, String)*)(f: => A): A
}
```

The pairs reach every record from the block, including calls in nested methods, without threading them through each `log.*` call.

```scala

log.annotated("requestId" -> "req-42", "tenant" -> "acme") {
  log.info("started")   // both annotations attached
  log.info("finished")  // both annotations attached
}
```

## Correlate Logs with the Active Span

When a `log.*` call runs inside a `trace.span`, the record is stamped with the enclosing span's trace and span IDs automatically, because logging and tracing share the same `ContextStorage`. No extra wiring is required.

```scala

trace.span("checkout") { _ =>
  log.info("order validated", "orderId" -> "ord-123") // carries the checkout span's IDs
}
```

## Filter by Severity

Every log carries a `Severity`, and a **minimum-severity floor** decides which ones actually get recorded: anything below the floor is dropped — cheaply, before the record is even built. The floor starts at `Trace`, so everything passes until you raise it. You use this to control noise: run production at `Info` (dropping the `trace`/`debug` chatter) and turn detail back up only where and when you need it.

```scala
object log {
  def setMinSeverity(severity: Severity): Unit
  def setMinSeverity(prefix: String, severity: Severity): Unit
  def clearMinSeverity(prefix: String): Unit
  def clearAllOverrides(): Unit
  def withMinSeverity[A](severity: Severity)(f: => A): A
}
```

`setMinSeverity(severity)` sets one floor for the whole application. `setMinSeverity(prefix, severity)` overrides it for a package — matched against the call site's namespace — and works both ways: raise the floor on a chatty dependency to quiet it, or lower it on the package you're debugging to see more, without touching the rest of the app. `clearMinSeverity(prefix)` removes one override and `clearAllOverrides()` removes them all, back to the global floor. Set the global floor first: `setMinSeverity(severity)` also discards every prefix override, so calling it after the overrides silently wipes them.

When several prefixes match a call site, the longest one wins — so `"com.example"` at `Warn` plus `"com.example.orders"` at `Debug` gives you debug logs from the orders package and warnings from everything else under `com.example`. Matching is a plain string prefix on the enclosing class name, not a package-boundary check, so `"com.acme"` also matches `com.acmecorp`.

```scala

log.setMinSeverity(Severity.Info)                       // drop trace/debug globally
log.setMinSeverity("com.acme.noisy", Severity.Warn)     // quiet a chatty dependency
log.setMinSeverity("com.example.orders", Severity.Debug) // more detail where you're debugging
```

These floors stay in effect until you change them. When you only need extra detail around a specific operation, `withMinSeverity(severity) { … }` lowers the floor, for every thread, while the block runs, then puts back the logging set-up from before the block. A known bug in 0.0.56: anything the block itself changes, such as a processor added with `log.addProcessor`, is undone too.

```scala

log.withMinSeverity(Severity.Trace) {
  log.trace("visible only inside this block")
}
```

## Route Output

A record isn't useful until it leaves the process. Routing decides two things: how each record becomes text — a [`LogFormatter`](./log-formatter.md), plain lines or JSON — and where those bytes go — a `LogWriter`, such as stdout, stderr, or a file. There are two ways to wire this up: a simple console writer for local development, or a processor pipeline for production.

```scala
object log {
  def writer(formatter: LogFormatter, logWriter: LogWriter): Unit
  def clearWriters(): Unit
  def install(logger: Logger, minSeverity: Severity = Severity.Trace): Unit
  def addProcessor(processor: LogRecordProcessor): Unit
  def removeAll(): Unit
}
```

`log.writer(formatter, writer)` is the simple path: pair a formatter with a writer to add one console sink. It's additive — call it again to send the same records to a second destination (say, human-readable text to stdout and JSON to stderr) — and `log.clearWriters()` removes them all.

For production you usually want a pipeline that batches records and ships them to a backend for storage and search. `log.install(logger)` swaps in a fully configured [`Logger`](./logger.md) (built from a `LoggerProvider` with its export processors) in place of whatever was registered before, including the default console output and any prefix overrides. Writers added earlier with `log.writer` are re-attached the next time you call `log.writer` or `log.clearWriters()`, so call `log.clearWriters()` first if you want them gone — `log.addProcessor(processor)` appends a single [`LogRecordProcessor`](./log-record-processor.md) to the current backend without disturbing the rest, and `log.removeAll()` detaches everything, so `log.*` calls become no-ops. A known bug in 0.0.56: adding an output with `log.writer` or `log.addProcessor` after `removeAll()` does not bring logging back, because the minimum level stays off. Only `log.install(...)` restores logging on its own. `log.setMinSeverity(...)` lifts the level filter but adds no output, so call it after `log.addProcessor` or `log.writer`.

`log.writer` adds to the default console output rather than replacing it. To get only the two writers below, first replace the default with an empty logger, so that each record is written once per writer:

```scala

// install replaces the default console output
log.install(LoggerProvider.builder.build().get("com.example"))

// Human-readable text to stdout, JSON to stderr
log.writer(TextLogFormatter, StdoutWriter)
log.writer(JsonLogFormatter, StderrWriter)
```

Or install a processor-based backend in one step:

```scala

val logger = LoggerProvider.builder
  .addLogRecordProcessor(new ConsoleLogRecordProcessor)
  .build()
  .get("com.example")
log.install(logger, Severity.Info)
```

## See Also

- [Telemetry Guide](../../../guides/telemetry-guide.md) — logging data flow, rate limiting, and production patterns
- [Telemetry Reference](../index.md) — module overview and all three pillars
- [Common Types](../common/index.md) — `Attributes`, `AttributeKey`, `Resource`, and `InstrumentationScope`

---

## LogEnrichment

`LogEnrichment[A]` is what lets you pass a value of your own type straight into a [`log`](./index.md) call and have it become part of the [`LogRecord`](./log-record.md). When the macro behind `log.info("msg", value)` meets an argument whose type it does not handle natively, it looks for an implicit `LogEnrichment[A]`, calls `enrich` to fold the value into the record, and fails to compile if none is in scope. You rarely name the typeclass directly — the built-in instances below cover the everyday types — you define one only to log a domain value without unpacking it by hand at every call site.

```scala
trait LogEnrichment[A] {
  def enrich(record: LogRecord, value: A): LogRecord
}
```

`enrich` takes the record built so far and returns a copy with the value folded in — adding attributes, replacing the body, or setting the severity, depending on `A`.

## Built-in Instances

| Type                                                                                               | Effect on the record                                                                                         |
|----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------|
| `String`                                                                                           | Replaces the message body.                                                                                   |
| `Throwable`                                                                                        | Adds `exception.type` and `exception.message` attributes and stores the throwable for stack-trace rendering. |
| `Attributes`                                                                                       | Merges the whole set into the record's attributes.                                                           |
| `Severity`                                                                                         | Overrides the recorded severity — but not the one used for filtering (see below).                            |
| `(String, String)` / `(String, Long)` / `(String, Int)` / `(String, Double)` / `(String, Boolean)` | Adds one typed key-value attribute (`Int` is widened to `Long`).                                             |

These common types are also recognized directly by the `log.*` macro, so passing them costs nothing at runtime:

```scala

val ex: Throwable = new RuntimeException("connection refused")

log.error(
  "payment failed",
  ex,                        // Throwable — attaches type, message, stack trace
  "orderId"   -> "ord-123",  // (String, String)
  "amount"    -> 99L,        // (String, Long)
  "retryable" -> false       // (String, Boolean)
)
```

Passing a `Severity` is the one case that behaves unexpectedly. It changes the severity written into the record, but filtering already happened using the severity of the method you called — so `log.warn("slow query", Severity.Debug)` is recorded as DEBUG even under an `Info` floor, and `log.info("charge failed", Severity.Error)` is dropped by a `Warn` floor despite arriving as an error. Call the method matching the level you want filtered on.

Types outside the table are rejected at compile time, which is what a missing `LogEnrichment` error means. `Float`, `Short`, `Byte`, `Char`, `UUID`, and `Instant` are the ones people hit first: convert them at the call site (`.toDouble`, `.toLong`, `.toString`) or give the type an instance of its own.

## Custom Instances

Define a `LogEnrichment[MyType]` in implicit scope to accept your own type at a `log.*` call. The macro resolves it at that call site and threads the record through your `enrich`:

```scala

final case class RequestId(value: String)

implicit val requestIdEnrichment: LogEnrichment[RequestId] = new LogEnrichment[RequestId] {
  def enrich(record: LogRecord, value: RequestId): LogRecord =
    record.copy(attributes = record.attributes ++ Attributes.of(AttributeKey.string("request.id"), value.value))
}

log.info("handling request", RequestId("req-001")) // adds request.id="req-001"
```

## Integration

Resolution happens entirely at compile time, so there is no runtime cost for type dispatch. An argument whose type has neither native macro handling nor an implicit `LogEnrichment[A]` is a compile error, which keeps unloggable values out of the call. A custom instance must be in implicit scope wherever the `log.*` call appears.

---

## LogFormatter

`LogFormatter` renders a [`LogRecord`](./log-record.md) into text; its partner, the [`LogWriter`](./log-writer.md), routes that text to a destination. You rarely name a formatter directly — you hand a formatter/writer pair to [`log.writer`](./index.md), and the two built-ins below cover the common cases. Implement the trait yourself when your pipeline expects a shape neither produces.

## Built-in Formatters

Two singletons cover the usual output formats — both can be shared across threads; `TextLogFormatter` caches its timestamp prefix for the current second, and a known bug in 0.0.56 lets a line logged at a second boundary from another thread carry the wrong second:

| Formatter          | Output                                                                                                                                       |
|--------------------|----------------------------------------------------------------------------------------------------------------------------------------------|
| `TextLogFormatter` | Human-readable — `2026-07-29T10:00:00.000Z INFO  [Svc.doWork:42] message {key=val}`                                                          |
| `JsonLogFormatter` | OTLP-compatible JSON — `{"timeUnixNano":"...","severityNumber":9,"severityText":"INFO","body":{"stringValue":"message"},"attributes":[...]}` |

Both render the record's timestamp, severity, source location (from the `code.*` attributes), message body, and an attached throwable's stack trace. They differ in what else they carry: `TextLogFormatter` prints the remaining attributes as a compact `{key=value}` list and drops the trace and span IDs, while `JsonLogFormatter` emits every attribute — including the four `code.*` entries — plus `traceId` and `spanId` when the record was written inside a span.

## Example Usage

Choosing a formatter is choosing who reads your logs. In development that's you, so pick `TextLogFormatter` — one aligned line per record, easy to scan in a terminal. In production it's a log collector, so pick `JsonLogFormatter` — every attribute stays a separate field the collector can index and query. Select it once at startup, alongside a [`LogWriter`](./log-writer.md) for the destination:

```scala

val isDevelopment = sys.env.get("ENV").forall(_ != "production")

if (isDevelopment) log.writer(TextLogFormatter, StdoutWriter)  // readable in a terminal
else log.writer(JsonLogFormatter, StdoutWriter)                // queryable by a collector

log.info("server ready", "port" -> 8080L)
```

You can also register both — `log.writer` is additive, so each call adds another channel and every record goes to all of them. That's how you keep readable console output while also emitting machine-readable JSON:

```scala

log.writer(TextLogFormatter, StdoutWriter)  // for the developer watching the terminal
log.writer(JsonLogFormatter, StderrWriter)  // for the collector tailing stderr

log.info("server ready", "port" -> 8080L)   // written to both channels
```

## Custom Formatter

Sometimes neither built-in shape fits — an existing ingest pipeline expects logfmt, or a report wants CSV. Implement the trait yourself and pass your formatter to `log.writer` like any built-in. Two methods are abstract, but only one of them is yours to write:

```scala
trait LogFormatter {
  def format(
    sb: StringBuilder, timestampNanos: Long, severity: Severity, severityText: String,
    body: String, builder: Attributes.AttributesBuilder,
    traceIdHi: Long, traceIdLo: Long, spanId: Long, traceFlags: Byte,
    throwable: Option[Throwable]
  ): Unit

  def formatRecord(sb: StringBuilder, record: LogRecord): Unit
}
```

`formatRecord` is the one that runs: a channel registered with `log.writer` renders each finished [`LogRecord`](./log-record.md) through it. Put your format there.

`format` serves a lower-allocation path that skips building a `LogRecord`, and the library uses it only for its own default console output — no formatter you register ever reaches it. You still have to define it to satisfy the trait, so mirror the scalar fields and move on; it cannot render attributes anyway, since the accessors for reading them off the builder are internal.

Both methods receive a fresh, empty `StringBuilder` from the caller, which writes it out and discards it. Append your output and return — the caller handles writing.

A CSV formatter — three comma-separated fields in `formatRecord`, with `format` mirroring them:

```scala

object CsvFormatter extends LogFormatter {
  def formatRecord(sb: StringBuilder, record: LogRecord): Unit =
    sb.append(record.timestampNanos).append(',').append(record.severityText).append(',').append(record.body.value)

  def format(
    sb: StringBuilder, timestampNanos: Long, severity: Severity, severityText: String,
    body: String, builder: Attributes.AttributesBuilder,
    traceIdHi: Long, traceIdLo: Long, spanId: Long, traceFlags: Byte,
    throwable: Option[Throwable]
  ): Unit = sb.append(timestampNanos).append(',').append(severityText).append(',').append(body)
}

log.writer(CsvFormatter, StdoutWriter)
log.info("order placed")
// 2026-07-31T13:23:25.137Z INFO  [Main$.main:20] order placed  <- default console output
// 1785504205137002324,INFO,order placed                        <- your channel
```

Two lines, because `log.writer` adds a channel rather than replacing one, and the default console output is still registered.

To render attributes too, pass an `AttributeVisitor` to `record.attributes.accept` — it fires once per attribute with the raw unboxed value, so nothing is boxed on the way out. Only the four scalar methods are abstract; the seq variants default to no-ops. Source location arrives among those attributes as `code.filepath`, `code.namespace`, `code.function`, and `code.lineno`. Decide which convention you want: `TextLogFormatter` lifts them into its `[Svc.doWork:42]` prefix and then skips those keys, while `JsonLogFormatter` emits them as ordinary attributes. Skip any key starting with `code.` if you render the location separately.

---

## LogRecordProcessor

A `LogRecordProcessor` is a destination for your logs. Every time you call `log.info(...)`, the finished [`LogRecord`](./log-record.md) is handed to each registered processor, and what happens next is entirely up to that processor: print it, ship it to a backend, forward it into another logging library, or drop it.

The type exists because where logs belong depends on your deployment, which the library can't know. Registering destinations is how you tell it — and you can register several, each receiving the same record, which is how logs reach your console and a remote backend at once.

Most applications never implement one, because the module ships the common destinations. `ConsoleLogRecordProcessor` prints readable text to stdout — the default, and also the fastest path: when it's the only processor registered, the `Logger` skips building a `LogRecord` and formats straight from the raw values. `LogRecordProcessor.noop` accepts every record and discards it, for tests or as a placeholder. And for formatted output — JSON, a file, your own shape — [`log.writer(formatter, writer)`](./index.md) builds a processor for you out of a [`LogFormatter`](./log-formatter.md) and a [`LogWriter`](./log-writer.md), which is less code than implementing this trait.

Write your own to reach somewhere the module doesn't cover — an OTLP collector, a vendor's API, or an existing logging framework you're migrating from.

To capture log records in memory and assert on their severity, message and fields in tests, see [How to Test Code That Logs, Traces and Records Metrics](../../../guides/test-telemetry.md).

```scala
trait LogRecordProcessor extends AutoCloseable {
  def onEmit(logRecord: LogRecord): Unit   // fires for every record the Logger lets through
  def shutdown(): Unit                     // flush and release; called by LoggerProvider.shutdown()
  def forceFlush(): Unit                   // flush buffered records now
  override def close(): Unit = shutdown()

  def minimumLevel: Int = 1                // lowest Severity.number accepted; default accepts all
}

object LogRecordProcessor {
  val noop: LogRecordProcessor  // ignores every record
}
```

`onEmit` is the method that matters — it runs once per log, **on the thread that logged**, before your code continues. So keep it quick. If you send each record over the network from inside `onEmit`, every `log.info` in your application waits for that round trip; buffer records instead and ship them in batches from a background thread, then flush what's left when `shutdown` or `forceFlush` is called.

`minimumLevel` lets a processor say "only send me warnings and worse" as a `Severity` number. It's a volume hint, not a filter applied per processor: the `Logger` takes the lowest `minimumLevel` across every registered processor and, for anything below it, doesn't even build a `LogRecord` — so a debug line no destination wants costs almost nothing. Anything above that shared floor is delivered to every processor, including ones whose own `minimumLevel` is higher, so a processor that must not see debug records has to re-check `record.severity` in `onEmit`.

Your own processor implements those three methods — with nothing buffered, `shutdown` and `forceFlush` can stay empty — and registers on the global [`log`](./index.md):

```scala

val exporter = new LogRecordProcessor {
  def onEmit(r: LogRecord): Unit = println(s"[EXPORT] ${r.severity.text} ${r.body.value}")
  def shutdown(): Unit           = ()
  def forceFlush(): Unit         = ()
}

log.addProcessor(exporter)
log.info("checkout complete")
```

`log.addProcessor` is the easiest route: your processor joins the destinations already registered, with nothing to rebuild — when you build a provider yourself, pass processors to `LoggerProvider.builder.addLogRecordProcessor(...)` instead. Either way, an exception your processor throws is caught and printed to stderr, and the remaining processors still receive the record, so one broken destination can't take down your logging.

---

## LogRecord

A `LogRecord` is one log entry as a value — everything a single `log.info(...)` call captured, bundled together: the message, its severity, when it happened, the key/value fields attached to it, and which request it belongs to.

The type exists because a log entry has to travel. From the call site it may go to your console, to a file, and to a backend that indexes it, and each of those destinations needs the whole story in one self-contained package — a bare string wouldn't carry the severity or the trace it came from. Because a record is immutable, every destination can read the same one safely.

You never build a `LogRecord` in normal use; the global [`log`](./index.md) (or a [`Logger`](./logger.md)) builds one for you on every call. You meet it when you extend the pipeline: it's the value handed to a [`LogRecordProcessor`](./log-record-processor.md)'s `onEmit` and to a [`LogFormatter`](./log-formatter.md), so writing either means reading fields off a record.

```scala
final case class LogRecord(
  timestampNanos:         Long,
  observedTimestampNanos: Long,
  severity:               Severity,
  severityText:           String,
  body:                   LogMessage,           // LogMessage.Simple(text), or a lazy template
  attributes:             Attributes,           // code.* source location + caller-supplied attrs
  traceIdHi:              Long,                 // 0L when no span is active
  traceIdLo:              Long,                 // 0L when no span is active
  spanId:                 Long,                 // 0L when no span is active
  traceFlags:             Byte,                 // 0x00 when not sampled
  resource:               Resource,
  instrumentationScope:   InstrumentationScope,
  throwable:              Option[Throwable] = None
) {
  def hasTraceId: Boolean  // traceIdHi != 0L || traceIdLo != 0L
  def hasSpanId:  Boolean  // spanId != 0L
}

object LogRecord {
  def builder: LogRecordBuilder
}
```

Each field mirrors what the emission recorded. The source location the macro captured arrives inside `attributes`, as the `code.filepath`, `code.namespace`, `code.function`, and `code.lineno` keys. The trace and span IDs are what tie this entry back to the request that produced it: they're filled in when the call ran inside an active span, and left as `0` when it didn't — so check `hasTraceId`/`hasSpanId` rather than comparing to zero yourself.

---

## LogWriter

`LogWriter` is the sink half of console log output: it takes text that has already been formatted and pushes it to a destination — stdout, stderr, a file. Its partner is the [`LogFormatter`](./log-formatter.md), which turns a [`LogRecord`](./log-record.md) into that text. You rarely name a writer directly — you hand a formatter/writer pair to [`log.writer`](./index.md), and the built-ins below cover the usual destinations. Reach for a custom one only to send output somewhere the built-ins don't.

```scala
trait LogWriter {
  def write(content: CharSequence): Unit
  def flush(): Unit = ()
  def close(): Unit = ()
}
```

`write` receives one fully-formatted line; `flush` and `close` default to no-ops, so a simple writer overrides only `write`.

## Built-in Writers

Three singletons cover the common destinations:

| Writer | Destination |
|--------|-------------|
| `StdoutWriter` | `System.out` |
| `StderrWriter` | `System.err` |
| `NoopWriter` | Discards output — for benchmarking the formatting path |

## Example Usage

Pair a writer with a formatter and hand both to `log.writer`. It is additive — each call adds another channel, and every record is sent to all of them. The default console output stays, so call `log.install(LoggerProvider.builder.build().get("com.example"))` first if you want only these channels:

```scala

log.writer(TextLogFormatter, StdoutWriter)  // human-readable text to stdout
log.writer(JsonLogFormatter, StderrWriter)  // OTLP JSON to stderr

log.info("server ready", "port" -> 8080L)   // written to both channels
```

## Custom Writer

Implement the trait to send formatted output somewhere the built-ins don't — a file, say. Override `write`, plus `flush`/`close` if the destination holds resources:

```scala

val fileWriter = new LogWriter {
  private val out = new PrintWriter("app.log")
  def write(content: CharSequence): Unit = out.println(content)
  override def flush(): Unit             = out.flush()
  override def close(): Unit             = out.close()
}

log.writer(TextLogFormatter, fileWriter)
```

---

## LoggerProvider

A `LoggerProvider` is what makes [`Logger`](./logger.md)s. You build one at startup, and it hands out every logger your application uses.

It exists so that two questions get answered once instead of at every call site: *who is logging* — your service's identity, a [`Resource`](../common/resource.md) — and *where the logs go* — the [`LogRecordProcessor`](./log-record-processor.md)s that render or ship each record. Every logger the provider creates inherits both, so pointing your whole application at a different destination is a one-line change at startup rather than an edit everywhere you log.

Most applications never hold a provider directly: build one, hand a logger from it to `log.install(...)`, and use the global [`log`](./index.md) from then on. Reach for the provider itself when you need that startup configuration under your own control.

```scala
final class LoggerProvider private[telemetry] (...) extends AutoCloseable {
  def get(name: String, version: String = ""): Logger  // a Logger for a named scope

  def shutdown(): Unit        // shut down every processor; call once at exit
  override def close(): Unit  // alias for shutdown() — satisfies AutoCloseable
}

object LoggerProvider {
  def builder: LoggerProviderBuilder
}

final class LoggerProviderBuilder private[telemetry] (...) {
  def setResource(resource: Resource): LoggerProviderBuilder
  def addLogRecordProcessor(processor: LogRecordProcessor): LoggerProviderBuilder
  def setContextStorage(contextStorage: ContextStorage[Option[SpanContext]]): LoggerProviderBuilder
  def build(): LoggerProvider
}
```

## Usage

Build a provider once at startup, set only what you need — a `Resource` for your service identity and the `LogRecordProcessor`s that handle your records — then `get` a `Logger` from it. Anything left unset takes a sensible default (`Resource.default`, and an empty pipeline until you add a processor):

```scala

val provider = LoggerProvider.builder
  .setResource(Resource.create(Attributes.of(Attributes.ServiceName, "payments")))
  .addLogRecordProcessor(LogRecordProcessor.noop) // replace with a console writer or OTLP exporter
  .build()

val logger = provider.get("com.example.payments")
logger.info("order placed", "orderId" -> AttributeValue.StringValue("ord-001"))

provider.shutdown()
```

Note that `get` builds a logger rather than looking one up: call it twice with the same name and you get two separate `Logger` objects that behave identically. So call it once where a component is created and keep the result in a `val`, not inside a request handler that runs thousands of times a second — per-request calls produce the same log output, they just make throwaway objects for the garbage collector:

```scala

class PaymentService(provider: LoggerProvider) {
  private val logger = provider.get("com.example.payments")   // once, at construction

  def handle(orderId: String): Unit =
    logger.info("handling payment", "orderId" -> AttributeValue.StringValue(orderId))
}
```

The name you pass is what identifies where a log came from, so one logger per component — a service, a module — is the usual granularity.

Passing a `Logger` down through every constructor gets tedious, though. Register one on the global [`log`](./index.md) at startup instead, and the rest of your code can log without knowing a provider exists:

```scala

val provider = LoggerProvider.builder
  .setResource(Resource.create(Attributes.of(Attributes.ServiceName, "catalog")))
  .addLogRecordProcessor(LogRecordProcessor.noop)
  .build()

log.install(provider.get("com.example.catalog"))

log.info("server started")
```

## Lifecycle

Logs don't always leave your process the moment you write them. A processor that ships records to a backend usually batches them, so at any instant the most recent logs may still be sitting in a buffer — and if the process exits right then, they're gone. Those are exactly the logs you want when something went wrong at shutdown.

`provider.shutdown()` prevents that. It walks the registered processors in the order you added them and shuts each one down, which is a processor's cue to flush whatever it's holding and release what it owns — a file handle, a network connection. Call it once, on the way out:

```scala

val provider = LoggerProvider.builder
  .addLogRecordProcessor(LogRecordProcessor.noop)
  .build()

try log.install(provider.get("com.example"))
finally provider.shutdown()   // last logs flushed, resources released
```

That `finally` is the easy part to forget. So a provider implements `AutoCloseable` — the standard interface for "something that must be closed when you're done with it," the same one files and sockets use — with `close()` simply calling `shutdown()`. That lets you hand the cleanup to the language rather than writing it yourself. In Scala, `scala.util.Using.resource` does the calling:

```scala

Using.resource(LoggerProvider.builder.addLogRecordProcessor(LogRecordProcessor.noop).build()) { provider =>
  log.install(provider.get("com.example"))
  log.info("server started")
}   // close() — and so shutdown() — runs here, even if the block throws
```

The block marks how long the provider lives, and cleanup happens on the way out whether the code finishes normally or raises an exception. From Java the same interface makes a provider usable in try-with-resources.

## Trace–Log Correlation

This is the part you get for free. Write a log inside a span and it carries that span's trace and span IDs — no request id to thread through your functions, no logger to pass around, nothing to configure. A `LoggerProvider` and a [`TracerProvider`](../tracing/tracer-provider.md) read the same ambient context by default, so each is aware of the span the other opened:

```scala

trace.install(TracerProvider.builder.build())
log.install(
  LoggerProvider.builder
    .addLogRecordProcessor(new ConsoleLogRecordProcessor)  // a provider with no processor emits nothing
    .build()
    .get("com.example")
)

trace.span("checkout") { _ =>
  log.info("order validated", "orderId" -> "ord-123")  // stamped with the checkout span's IDs
  log.warn("inventory low", "sku" -> "sku-42")         // same trace, same span
}
```

That stamping is what makes a log backend useful: filter on one trace id and you have every log line from that single request, in order, across every component that touched it — and from any one of those lines you can open the trace it belongs to.

If you ever need to control that shared context yourself — isolating correlation between test cases, or fitting a custom runtime — pass one `ContextStorage` to `setContextStorage` on both providers.

For the full picture of how `LoggerProvider`, `TracerProvider`, and `MeterProvider` interconnect, see the [Telemetry module index](../index.md).

---

## Logger

A `Logger` writes logs. It's tied to one name — the component or library it belongs to, like `"com.example.OrderService"` — and a [`LoggerProvider`](./logger-provider.md) hands you one; you never construct it yourself.

That name travels with every record the logger's own methods emit; records from the global `log.*` calls keep the scope `zio.blocks.telemetry.log` even after `log.install` (a known bug in 0.0.56). The name marks which part of the system produced it. It's what lets you tell your own logs from a library's and turn one noisy component down without touching the rest. Records also arrive tied to the trace they happened in, so any line leads back to the request that caused it.

Most application code shouldn't use a `Logger` directly — the global [`log`](./index.md) is easier and does more. A `Logger` earns its place when a library accepts one as a parameter: then it's an ordinary dependency, and a test can pass in one whose records it inspects.

To pass a `Logger` that records into memory when testing a service, see [How to Test Code That Logs, Traces and Records Metrics](../../../guides/test-telemetry.md).

```scala
final class Logger {
  // One method per severity: trace, debug, info, warn, error, fatal
  def info(body: String, attrs: (String, AttributeValue)*): Unit
  // def trace/debug/warn/error/fatal are identical, just with different severity

  def emit(logRecord: LogRecord): Unit           // send a pre-built record
  def currentSpanContext(): Option[SpanContext]  // active span, or None
}
```

## Emitting a Log

This is what a `Logger` is for. Pick the method matching the severity, pass a message, and add any attributes worth searching on later — the scope name and the active span's identity are attached for you:

```scala

val provider = LoggerProvider.builder
  .addLogRecordProcessor(LogRecordProcessor.noop)
  .build()

val logger = provider.get("com.example.OrderService", "1.0.0")

logger.info("order placed", "orderId" -> AttributeValue.StringValue("ord-001"))
logger.warn("payment delayed", "durationMs" -> AttributeValue.LongValue(250L))
logger.error("checkout failed", "code" -> AttributeValue.LongValue(503L))

provider.shutdown()
```

Attributes must be wrapped: `AttributeValue.StringValue`, `LongValue`, `DoubleValue`, `BooleanValue`, or one of their `*SeqValue` forms for arrays. That verbosity is the cost of a bare `Logger` — the global [`log`](./index.md) accepts the same values as plain literals.

## Forwarding a Record You Already Have

`emit(record)` sends a [`LogRecord`](./log-record.md) you built yourself, as-is. Where `info(...)` fills in the timestamp, scope name, and span identity, `emit` adds nothing, so a hand-built record without span identity won't correlate with any trace. Use it to forward another framework's logs into the same destinations:

```scala

val logger = LoggerProvider.builder
  .addLogRecordProcessor(new ConsoleLogRecordProcessor)  // a provider with no processor emits nothing
  .build()
  .get("com.example.Bridge")

logger.emit(
  LogRecord.builder
    .setSeverity(Severity.Warn)
    .setBody("from the legacy logger")
    .build
)
```

## Reading the Active Span

`currentSpanContext()` returns the span you're inside, or `None` if there isn't one. Correlation is automatic, so you only need this to carry trace IDs across a boundary yourself — an outgoing HTTP header, a queue message, an error response:

```scala

val logger = LoggerProvider.builder.build().get("com.example.Client")

val traceHeader: Option[String] = logger.currentSpanContext().map(_.traceIdHex)
```

To configure where these logs go, see [`LoggerProvider`](./logger-provider.md).

---

## Severity

`Severity` is the level attached to every [`LogRecord`](./log-record.md) — it says how important the message is and drives which records a severity filter keeps. It follows the OpenTelemetry log data model: 24 numeric levels grouped into six named categories with four gradations each, fine enough for detailed instrumentation yet compatible with the familiar five-level `TRACE`/`DEBUG`/`INFO`/`WARN`/`ERROR` scale. Every level shares its category's `text` (all four `Trace` levels report `"TRACE"`), so backends that expect the coarse names still work.

```scala
sealed trait Severity {
  def number: Int    // 1 to 24
  def text:   String // the category name: "TRACE", "DEBUG", "INFO", "WARN", "ERROR", "FATAL"
}

object Severity {
  case object Trace extends Severity  // number = 1; then Trace2, Trace3, Trace4 (2-4)
  case object Debug extends Severity  // number = 5; then Debug2, Debug3, Debug4 (6-8)
  case object Info  extends Severity  // number = 9; then Info2, Info3, Info4 (10-12)
  case object Warn  extends Severity  // number = 13; then Warn2, Warn3, Warn4 (14-16)
  case object Error extends Severity  // number = 17; then Error2, Error3, Error4 (18-20)
  case object Fatal extends Severity  // number = 21; then Fatal2, Fatal3, Fatal4 (22-24)

  def fromNumber(n: Int): Option[Severity]      // Some when 1 <= n <= 24
  def fromText(text: String): Option[Severity]  // case-insensitive; the six category names
}
```

The six category objects (`Trace`, `Debug`, `Info`, `Warn`, `Error`, `Fatal`) are what you name day to day; the numbered gradations exist for instrumentation that needs finer distinctions.

## Filtering by Severity

The main use of `Severity` is to set a floor below which records are dropped before any `LogRecord` is built. Set a global floor with `log.setMinSeverity`, or a per-namespace override that takes precedence for matching call sites:

```scala

log.setMinSeverity(Severity.Warn)              // globally drop Trace, Debug, Info
log.setMinSeverity("com.example.db", Severity.Debug) // but keep Debug for the query layer

log.debug("suppressed by the global floor")
log.warn("emitted")
```

`fromText` and `fromNumber` parse a level from an external source — an environment variable or an OTLP protobuf field — returning `None` for anything out of range:

```scala

val level = Severity.fromText("ERROR").getOrElse(Severity.Info) // Severity.Error, number 17
log.setMinSeverity(level)
```

## See Also

- [LogRecord](./log-record.md) — the record whose level `Severity` sets.
- [Logging](./index.md) — the `log.*` emit methods and the full filtering API.

---

## Metrics

Metrics are the running numbers that tell you how your application is behaving over time — request rate, error count, latency, queue depth — the data you put on dashboards and alerts. They're **dimensional**: each measurement carries labels (like `method=GET`, `status=200`), and a metric splits into a separate series per label combination, so you can break "requests" down by endpoint or status. You record through four [instruments](./instruments.md), each for a different shape of number — a `Counter` for a total that only climbs, an `UpDownCounter` for one that rises and falls, a `Histogram` for a distribution of values, and a `Gauge` for the latest reading.

You record through the `metric` object: `metric.counter("http.requests").add(1)` works immediately, with no setup — it creates the named instrument and records the measurement. Pick the factory that matches what you're measuring — `metric.counter`, `metric.upDownCounter`, `metric.histogram`, or `metric.gauge` — and pass dimension labels as you record.

Measurements accumulate in memory, and reading them out is a pull: `metric.reader.collectAllMetrics()` returns [`MetricData`](./metric-data.md) snapshots — handy for tests and assertions, and the same call an exporter makes. Metrics have no push pipeline the way tracing and logging do: there is no processor hook on `MeterProvider`, so shipping them means scheduling that pull yourself and handing the snapshots to your backend. `metric.install(provider)` swaps in a provider configured with your own `Resource`.

Day to day you only need `metric.*`, but it helps to know what sits underneath, because each piece is where you go when you want more control. A [`MeterProvider`](./meter-provider.md) is what you build at startup: it holds your service identity as a `Resource` and vends everything else. (Unlike spans and log records, collected metric data does not carry that `Resource`, so an exporter has to attach the service identity itself.) A [`Meter`](./meter.md) is a named handle you take with `metric.get("com.example.orders")` when measurements should be attributed to one component, or when you want to declare an instrument's unit and description, or pre-declare label names for a hot path.

The other two are the recording and reading ends. The four [instruments](./instruments.md) are what you actually record through, however you obtained them. And the `MetricReader` behind `metric.reader` is what turns everything recorded so far into `MetricData` snapshots — the thing you call in tests, and the thing an exporter pulls from in production.

## Usage

Metrics' core job is to **record and export application metrics**. Create instruments from `metric`, record measurements with dimension labels as work happens, then read aggregated snapshots (or `metric.install(...)` a provider to export them):

```scala

val requests = metric.counter("http.requests")
requests.add(1, "method" -> "GET", "status" -> "200")

val latency = metric.histogram("http.latency.ms")
latency.record(12.5, "route" -> "/orders")

val snapshots = metric.reader.collectAllMetrics()
snapshots.foreach {
  case MetricData.SumData(points)       => points.foreach(p => println(p.value))
  case MetricData.HistogramData(points) => points.foreach(p => println(p.count))
  case MetricData.GaugeData(points)     => points.foreach(p => println(p.value))
}
```

## Record Measurements with the Four Instruments

Each factory creates a new instrument on the default meter. Nothing is cached by name, so build each one once and hold it — calling `metric.counter("http.requests")` twice gives you two independent counters and two entries in the collected snapshot.

```scala
object metric {
  def counter(name: String): Counter
  def upDownCounter(name: String): UpDownCounter
  def histogram(name: String): Histogram
  def gauge(name: String): Gauge
}
```

The four instruments differ by what they measure: a `Counter` sums a monotonic total, an `UpDownCounter` tracks a value that rises and falls, a `Histogram` builds a distribution of recorded values, and a `Gauge` holds the latest instantaneous reading. Every recording call accepts dimension labels as `(String, Any)*` pairs, which split the metric into separately-aggregated series.

```scala

metric.counter("http.requests").add(1, "method" -> "GET", "status" -> "200")
metric.upDownCounter("queue.depth").add(-1, "queue" -> "orders")
metric.histogram("http.latency.ms").record(12.5, "route" -> "/orders")
metric.gauge("cache.entries").record(4096.0, "cache" -> "sessions")
```

Note the two verbs. `add(delta: Long)` takes a *change* and the instrument keeps the running total — a `Counter` ignores a negative delta, while an `UpDownCounter` accepts one, so `add(1)`/`add(-1)` tracks queue depth. `record(value: Double)` takes the *measurement itself*: a `Histogram` files each value into buckets, and a `Gauge` keeps only the newest.

## Scope a Meter and Pre-Declare Labels

`metric.get(name)` returns a `Meter` bound to a named instrumentation scope.

```scala
object metric {
  def get(name: String): Meter
}
```

Its builders attach a description and unit, and its labeled variants pre-declare label names so a hot path records positionally instead of naming a key at every call.

```scala

val meter: Meter = metric.get("com.example.orders")

val payments: Counter =
  meter.counterBuilder("payments.total").setUnit("{payment}").setDescription("Payments processed").build()
payments.add(1, "currency" -> "USD")

val byRoute: LabeledCounter = meter.labeledCounter("http.requests", "method", "status")
byRoute.add(1, "GET", "200")
```

`setUnit` labels what the numbers mean, so a dashboard can axis-label `http.latency` as milliseconds rather than guessing. Use the UCUM codes OpenTelemetry expects — `"ms"`, `"s"`, `"By"` for bytes — and for a plain count of things, the thing in braces: `"{payment}"`, `"{request}"`. It's metadata attached to the instrument, not part of any measurement, and it stays on the instrument: the snapshots `metric.reader` returns carry only data points, so a unit reaches a backend through an exporter, not through `MetricData`.

## Read Aggregated Snapshots

Read every registered instrument as a `MetricData` snapshot — a fresh value on each call, including fresh copies of a histogram's `boundaries` and `bucketCounts` arrays.

```scala
object metric {
  def reader: MetricReader
}
```

`metric.reader.collectAllMetrics()` aggregates them — `SumData` for counters, `HistogramData` for histograms, `GaugeData` for gauges — for inspection in tests or manual export.

```scala

val snapshots: Seq[MetricData] = metric.reader.collectAllMetrics()
snapshots.foreach {
  case MetricData.SumData(points)       => points.foreach(p => println(p.value))
  case MetricData.HistogramData(points) => points.foreach(p => println(p.count))
  case MetricData.GaugeData(points)     => points.foreach(p => println(p.value))
}
```

## Install a Provider and Reset

`metric.install(provider)` swaps in a `MeterProvider` configured with your own [`Resource`](../common/resource.md). The builder takes nothing else — there is no exporter or reader hook — so instruments created after the swap register on the new provider's meters and are read back through `metric.reader`.

```scala
object metric {
  def install(provider: MeterProvider): Unit
  def removeAll(): Unit
}
```

Install before creating any instruments. Both calls replace the provider along with its registry, so an instrument built beforehand stays registered with the old one and won't appear in `metric.reader` snapshots. `metric.removeAll()` restores a fresh default provider.

```scala

metric.install(
  MeterProvider.builder
    .setResource(Resource.create(Attributes.of(Attributes.ServiceName, "order-service")))
    .build()
)

// later, at shutdown
metric.removeAll()
```

## See Also

- [Telemetry Guide](../../../guides/telemetry-guide.md) — metrics data flow and labeled instrument patterns
- [Telemetry Reference](../index.md) — module overview and all three pillars
- [Common Types](../common/index.md) — `Attributes` used as metric dimension labels

---

## Metric Instruments

The four synchronous instruments are what application code actually calls to record measurements. They differ by what they measure: a `Counter` sums a monotonic total, an `UpDownCounter` tracks a value that rises and falls, a `Histogram` builds a distribution of observations, and a `Gauge` holds the latest reading. Reach for them through [`metric`](./index.md) (`metric.counter("…")`) or a [`Meter`](./meter.md) builder — either path registers the instrument so `MetricReader.collectAllMetrics()` collects it into a [`MetricData`](./metric-data.md) snapshot.

All four share the same recording shape: `add` (counters) or `record` (histogram, gauge) takes a value plus optional dimension labels, either as an [`Attributes`](../common/attributes.md) set or as `(String, Any)*` tuples — a label value may be a `String`, `Long`, `Int` (widened to `Long`), `Double`, or `Boolean`, and anything else is recorded as its `toString`; `bind` pre-attaches a label set for hot-path reuse; and `collect` snapshots the accumulated data into a [`MetricData`](./metric-data.md) variant.

## Counter

A `Counter` records monotonically increasing values — negative deltas are ignored, so it only ever climbs. Use it for totals like requests served, errors, or bytes sent.

```scala
final class Counter private[telemetry] (
  val name: String, val description: String, val unit: String
) {
  def add(value: Long, attributes: Attributes): Unit
  def add(value: Long, attrs: (String, Any)*): Unit  // convenience vararg overload
  def bind(attributes: Attributes): BoundCounter      // pre-attributed for hot-path reuse
  def collect(): MetricData                           // snapshot → MetricData.SumData
}
```

Record with labels, then read the per-label totals from the collected `SumData`:

```scala

val calls = metric.counter("db.calls")
calls.add(1L, "table" -> "orders")
calls.add(2L, "table" -> "items")

val tableKey = AttributeKey.string("table")
metric.reader.collectAllMetrics().foreach {
  case MetricData.SumData(points) =>
    points.foreach(p => println(s"${p.attributes.get(tableKey)}: ${p.value}"))
  case _ => ()
}
```

## UpDownCounter

An `UpDownCounter` records bidirectional deltas — the same API as `Counter`, but negative values count. Use it for a running total that both rises and falls, such as active connections or queue depth.

```scala
final class UpDownCounter private[telemetry] (
  val name: String, val description: String, val unit: String
) {
  def add(value: Long, attributes: Attributes): Unit
  def add(value: Long, attrs: (String, Any)*): Unit
  def bind(attributes: Attributes): BoundUpDownCounter
  def collect(): MetricData
}
```

Add positive and negative deltas; the running total nets out:

```scala

val active = metric.upDownCounter("active.connections")
active.add(1L)   // new connection
active.add(-1L)  // connection closed

metric.reader.collectAllMetrics().foreach {
  case MetricData.SumData(points) => println(points.head.value)  // 0
  case _ => ()
}
```

## Histogram

A `Histogram` distributes `Double` observations into buckets and accumulates their count, sum, min, and max per label set. Use it for value distributions like request latency or payload size. Observations fall into a fixed set of bucket boundaries, defaulting to `[0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000]`.

Each boundary is the **inclusive** upper bound of its bucket, so with boundaries `[5, 10]` a value of exactly `5` lands in the first bucket, not the second, and anything above the last boundary falls into one final overflow bucket. Note that the default set starts at `0`, so its first bucket holds only values at or below zero.

For different boundaries, construct the histogram directly — the builder has no setter for them:

```scala

val latency = Histogram("http.latency.ms", "Request latency", "ms", Array(10.0, 50.0, 100.0, 500.0))
latency.record(42.0, "route" -> "/orders")

val snapshot = latency.collect()
```

That path trades away registration: an instrument you build yourself belongs to no meter, so `collectAllMetrics()` never returns it and you have to call `collect()` on the instrument yourself.

```scala
final class Histogram private[telemetry] (
  val name: String, val description: String, val unit: String,
  val boundaries: Array[Double]
) {
  def record(value: Double, attributes: Attributes): Unit
  def record(value: Double, attrs: (String, Any)*): Unit
  def bind(attributes: Attributes): BoundHistogram
  def collect(): MetricData                            // snapshot → MetricData.HistogramData
}
```

Record observations, then read `count` and `sum` from the collected `HistogramData`:

```scala

val latency = metric.histogram("request.latency")
latency.record(42.5, "endpoint" -> "/api/orders")
latency.record(1500.0, "endpoint" -> "/api/reports")

metric.reader.collectAllMetrics().foreach {
  case MetricData.HistogramData(points) =>
    points.foreach(p => println(s"count=${p.count} sum=${p.sum}"))
  case _ => ()
}
```

## Gauge

A `Gauge` holds the most recent `Double` value per label set — each `record` overwrites the previous one. Use it for an instantaneous reading like CPU temperature or a current queue-depth snapshot.

```scala
final class Gauge private[telemetry] (
  val name: String, val description: String, val unit: String
) {
  def record(value: Double, attributes: Attributes): Unit
  def record(value: Double, attrs: (String, Any)*): Unit
  def bind(attributes: Attributes): BoundGauge
  def collect(): MetricData                            // snapshot → MetricData.GaugeData
}
```

Each `record` overwrites the last; the snapshot holds the latest value:

```scala

val temp = metric.gauge("cpu.temperature")
temp.record(72.5)
temp.record(74.1)  // overwrites the previous value

metric.reader.collectAllMetrics().foreach {
  case MetricData.GaugeData(points) => println(points.head.value)  // 74.1
  case _ => ()
}
```

## Choosing an Instrument

| Instrument      | Delta constraint | Use when                                |
|-----------------|------------------|-----------------------------------------|
| `Counter`       | Non-negative     | Counting requests, errors, events       |
| `UpDownCounter` | Any              | Active connections, queue-depth changes |
| `Histogram`     | Any `Double`     | Latency, payload-size distributions     |
| `Gauge`         | Any `Double`     | CPU temperature, queue-depth snapshot   |

## Bound Instruments

When one label combination is recorded repeatedly on a hot path, `bind(attrs)` returns a `Bound*` instrument pre-associated with that label set, so recording skips rebuilding `Attributes` on every call:

```scala

val bound = metric.counter("rpc.calls").bind(Attributes.of(AttributeKey.string("method"), "OrderService.place"))
bound.add(1L)
bound.add(1L)  // no Attributes construction per call
```

One caveat on a histogram: `bind` creates the per-label state immediately, so a label set you bind but never record collects as `count = 0` with `min` and `max` at their sentinel extremes (`Double.MaxValue` and `Double.MinValue`). Skip zero-count points when plotting, or bind only where you will record.

For a name-based label API — declare label names once, then pass values positionally — see [Labeled Instruments](./labeled-instruments.md).

## Collection

`MetricReader.collectAllMetrics()` calls each registered instrument's `collect()`, producing one `MetricData` per instrument: `SumData` for `Counter` and `UpDownCounter`, `HistogramData` for `Histogram`, and `GaugeData` for `Gauge`. Pattern-match to read the data points — see [MetricData](./metric-data.md) for the point structure. Only instruments obtained from `metric.*` or a `Meter` builder are registered; do not construct an instrument directly, as an unregistered one never reaches `collectAllMetrics()`.

---

## Labeled Instruments

A labeled instrument is an ordinary counter, histogram, or gauge that already knows its label *names*. You declare them once — `"method"`, `"status"` — and then pass only values: `add(1, "GET", "200")`.

It exists to keep call sites short and consistent: the names live in one place, so every recording site is guaranteed to use the same label keys in the same order, and reading `add(1, "GET", "200")` is quicker than reading a line of repeated pairs.

Do not reach for it as an optimization. The pair form (`add(1, "method" -> "GET", …)`) builds its `Attributes` through a pooled, thread-local builder, while the labeled form allocates a fresh builder and a fresh `Attributes` on every call — so labeled recording allocates more, not less. What you give up is safety: passing the wrong *number* of values throws right at the recording call, but passing them in the wrong *order* records silently — swap two and you get a plausible-looking series that's simply wrong. When a label combination really is hot, `bind` is the answer: it resolves the attribute set once and hands back a `Bound*` that writes straight to the accumulator. A [`Meter`](./meter.md) creates labeled instruments, through `labeledCounter`, `labeledHistogram`, and `labeledGauge`.

```scala
final class LabeledCounter private[telemetry] (...) {
  val labelNames: Array[String]
  def add(value: Long, labelValues: Any*): Unit
  def bind(labelValues: Any*): BoundCounter
  def collect(): MetricData
}

final class LabeledHistogram private[telemetry] (...) {
  val labelNames: Array[String]
  def record(value: Double, labelValues: Any*): Unit
  def bind(labelValues: Any*): BoundHistogram
  def collect(): MetricData
}

final class LabeledGauge private[telemetry] (...) {
  val labelNames: Array[String]
  def record(value: Double, labelValues: Any*): Unit
  def bind(labelValues: Any*): BoundGauge
  def collect(): MetricData
}
```

## Usage

Declare the label names once, then supply values in that same order on each call. The count of `labelValues` must match the declared `labelNames`; a mismatch throws `IllegalArgumentException` at the recording call site (not at construction), so a wrong arity surfaces immediately in tests.

```scala

val meter = MeterProvider.builder.build().get("com.example")

// "GET" → "method", "200" → "status"
val reqs = meter.labeledCounter("http.requests", "method", "status")
reqs.add(1L, "GET", "200")
reqs.add(1L, "POST", "500")

val lat = meter.labeledHistogram("request.latency", "route")
lat.record(15.0, "/orders")

val depth = meter.labeledGauge("queue.depth", "queue")
depth.record(7.0, "orders")
```

For a set of label values recorded over and over, `bind(labelValues*)` builds the `Attributes` once and returns a bound instrument that skips that work on each later call:

```scala

val reqs  = MeterProvider.builder.build().get("com.example").labeledCounter("rpc.calls", "service", "method")
val bound = reqs.bind("OrderService", "place")
bound.add(1L)
bound.add(1L)
```

Values flow through the wrapped `Counter`, `Histogram`, or `Gauge` and are collected by `MetricReader.collectAllMetrics()` like any other instrument — see [MetricData](./metric-data.md).

See also [How to Avoid Allocation on Hot Paths](../../../guides/avoid-allocation-on-hot-paths.md) for pre-binding label values on a hot path and measuring the effect.

---

## MeterProvider

A `MeterProvider` is where metrics get configured, once, at startup. You build one, and it hands out the [`Meter`](./meter.md)s your components take.

It exists because two things can't be decided per instrument. The first is **identity**: an exported measurement has to say what produced it, or a backend shows a number with no owner — and repeating that on every counter would be both tedious and inconsistent. You set it once as a [`Resource`](../common/resource.md) on the provider. Unlike spans and log records, collected metric data does not carry it, so read `provider.resource` when you export and attach it there.

The second is **one place to read from**. Instruments end up scattered across components, and nothing useful happens until something collects them all together. The provider keeps the registry every meter joins, so a single `reader.collectAllMetrics()` returns every measurement from every scope beneath it — you never assemble a list of instruments yourself.

Most applications never hold one: they hand a provider to `metric.install(...)` at startup and use the global [`metric`](./index.md) everywhere after. Build one yourself when you want a `Resource` of your own, or a scope you can collect and shut down independently of the global one.

## Configuring Metrics for Your Service

Build the provider with a `Resource` naming your service:

```scala

val provider = MeterProvider.builder
  .setResource(Resource.create(Attributes.of(Attributes.ServiceName, "payments")))
  .build()
```

Do set the name. Skipping it doesn't fail — the default `Resource` supplies `service.name = "unknown_service"` alongside SDK version attributes — but that placeholder is what a backend will show for every measurement you export.

## Getting a Meter per Component

Ask for a meter by the component's name; that name is what attributes its measurements:

```scala

val provider = MeterProvider.builder.build()

val meter: Meter = provider.get("com.example.payments")
```

The full signature is `get(name: String, version: String = "")`; passing a version records it on the scope, which is how you tell measurements from two releases of the same library apart. Asking twice for the same name *and* version returns the same meter, so components take theirs independently without coordinating. The provider also exposes the `Resource` you built it with as `provider.resource`. See [Meter](./meter.md) for what to do with a meter.

## Reading Everything Back

`reader` is a single [`MetricReader`](./metric-data.md) for the provider's whole lifetime, created at `build()` — there is nothing to register, and no second reader to keep in sync:

```scala

val provider = MeterProvider.builder.build()
provider.get("com.example").counterBuilder("ops").build().add(1L)

val snapshots: Seq[MetricData] = provider.reader.collectAllMetrics()
```

Each call aggregates what has accumulated so far into [`MetricData`](./metric-data.md) — one value per registered instrument, across every meter. This is a pull: nothing is sent anywhere until you or an exporter asks.

## Shutting Down

A provider is `AutoCloseable`, with `close()` calling `shutdown()`, so it fits `scala.util.Using.resource`:

```scala

Using.resource(MeterProvider.builder.build()) { provider =>
  provider.reader.collectAllMetrics()
}
```

Unlike logging, there is nothing buffered here waiting to be flushed — metrics are read on demand, so the built-in reader's shutdown does no work today. Call it anyway at exit: it costs nothing, and it's what an export pipeline would hook into.

---

## Meter

A `Meter` is where one component's [instruments](./instruments.md) come from. You take a meter under your component's name, create counters and histograms from it, and record through those.

It exists to answer a question a bare instrument name can't: *whose measurement is this?* A meter carries a scope name, so the instruments one component builds are grouped under it, and a reader can tell your `requests` counter from the one a library you depend on registered. (The two were never merged into one series — each instrument aggregates on its own — but without scopes there is nothing in the code that says which is which.)

The second problem it solves is reachability. Recording a number is useless if nothing can read it back, and an instrument only reaches collection if something registered it. A meter is that something: it is registered with its [`MeterProvider`](./meter-provider.md) when you obtain it, and it registers every instrument you build from it, so a measurement's path to `reader.collectAllMetrics()` is complete the moment you call `build()` — with no wiring step you could forget.

## Taking a Meter

Ask a provider for one by name, or `metric.get(name)` in application code. Use the component's package as the name:

```scala

val meter: Meter = MeterProvider.builder.build().get("com.example.server")
```

Asking twice for the same name gives you back the *same* meter, not a second one — meters are cached per scope. So separate call sites in one component can each take their meter without coordinating, and their instruments still land under a single scope.

## Creating an Instrument

Each of the four instrument kinds has a builder. Name it, optionally describe it and give it a unit, then `build()`:

```scala

val meter = MeterProvider.builder.build().get("com.example.server")

val requests = meter.counterBuilder("http.requests")
  .setDescription("Total HTTP requests")
  .setUnit("1")
  .build()
requests.add(1L, "method" -> "GET", "status" -> "200")

val latency = meter.histogramBuilder("request.latency").setUnit("ms").build()
latency.record(42.5, "route" -> "/api/orders")
```

The description and unit stay on the instrument — `collect()` returns data points only — so they reach a dashboard through whatever exporter reads the instrument, which is what lets it label an axis in milliseconds instead of showing bare numbers. Build each instrument once and hold the result — a `val` on the component that records through it. Building the same name twice gives you two registered instruments, which splits one logical metric into two series that no consumer can merge back together.

## Recording on a Hot Path

When the same label *names* repeat on every call, declaring them once keeps every call site consistent and short. `labeledCounter`, `labeledHistogram`, and `labeledGauge` fix the names at construction so callers pass just the values, positionally:

```scala

val meter = MeterProvider.builder.build().get("com.example")

val byRoute = meter.labeledCounter("http.requests", "method", "status")
byRoute.add(1L, "GET", "200")
```

See [Labeled Instruments](./labeled-instruments.md) for the trade-offs; for ordinary recording the tuple form above is simpler.

## Reporting a Value You Don't Push

Some numbers aren't events you count — they're state you can read at any time, like a cache's size or a pool's idle connections. Instead of pushing an update whenever it changes, `buildWithCallback` on the counter, up-down counter, and gauge builders makes an instrument that asks *you* for the value at collection time:

```scala

val meter = MeterProvider.builder.build().get("com.example")
val cache = scala.collection.mutable.Map("k" -> "v")

meter.gaugeBuilder("cache.entries").buildWithCallback { observer =>
  observer.record(cache.size.toDouble, Attributes.empty)
}
```

The block doesn't run when you build it — it runs once per collection, reading `cache.size` fresh each time, so the value can't go stale and you need no hook at every mutation. Keep it cheap and side-effect-free; it runs on the collecting thread. You can discard the returned instrument — the meter registered it, so collection finds it — and hold it only if you want to call its own `collect()` directly.

`buildWithCallback` returns an `ObservableCounter`, `ObservableUpDownCounter`, or `ObservableGauge` depending on the builder, and hands your block an `ObservableCallback` to record through. Histograms have no callback form, since a distribution has to see every observation as it happens.

Watch the type on the counters: both observable counters round what you report to a whole number, so a callback recording `1.5` is collected as `2`. Only `ObservableGauge` keeps the `Double` as given.

## Two Ways to Lose Measurements

Both come from an instrument that records into nothing:

1. **Constructing an instrument directly.** `Counter("http.requests", "", "")` compiles, because the companion `apply` is public. It records perfectly well into an object no meter registered, so `collectAllMetrics()` never sees it. Go through a meter or `metric.*` — the one case that justifies direct construction is a [histogram with custom bucket boundaries](./instruments.md), where you accept the loss of registration and call `collect()` yourself.
2. **Crossing providers.** An instrument reaches only the reader of the provider whose meter built it. `metric.install(...)` swaps in a provider with an empty registry, so take your meters and build your instruments *after* installing.

## See Also

- [Instruments](./instruments.md) — the recording API of each instrument kind
- [MeterProvider](./meter-provider.md) — where meters come from, and what configures them
- [MetricData](./metric-data.md) — what collection hands back

---

## MetricData

`MetricData` is an aggregated snapshot of one instrument at one collect cycle — its accumulated points, keyed by label set. You rarely build it directly: it is what `MetricReader.collectAllMetrics()` returns, one value per registered instrument, for a test to assert on or an export pipeline to serialize. Each collection builds fresh points; a histogram point holds its own copies of the bucket counts and boundaries.

One thing it does *not* carry is the instrument's name, description, or unit — a snapshot is only points and their label sets. So `collectAllMetrics()` hands back a `Seq[MetricData]` with no way to tell which instrument each entry came from, which is why an export pipeline pairs them with names of its own before serializing. In a test, collect from a single instrument with its own `collect()` when you need to know the source.

Its three variants line up with the instrument that produced them:

- `SumData` — from a [`Counter`](./instruments.md) or an `UpDownCounter`
- `HistogramData` — from a `Histogram`
- `GaugeData` — from a `Gauge`

So a `match` on `MetricData` tells you both the shape of the numbers and the kind of instrument they came from.

To read a counter's total back from `MetricData.SumData` in a test, see [How to Test Code That Logs, Traces and Records Metrics](../../../guides/test-telemetry.md).

```scala
sealed trait MetricData

object MetricData {
  final case class SumData(points: List[SumDataPoint])             extends MetricData
  final case class HistogramData(points: List[HistogramDataPoint]) extends MetricData
  final case class GaugeData(points: List[GaugeDataPoint])         extends MetricData
}

final case class SumDataPoint(
  attributes:     Attributes,
  startTimeNanos: Long,
  timeNanos:      Long,
  value:          Long          // cumulative sum
)

final case class HistogramDataPoint(
  attributes:     Attributes,
  startTimeNanos: Long,
  timeNanos:      Long,
  count:          Long,         // total number of observations
  sum:            Double,       // sum of all observations
  min:            Double,       // smallest observation
  max:            Double,       // largest observation
  bucketCounts:   Array[Long],  // count per bucket (length == boundaries.length + 1)
  boundaries:     Array[Double] // the bucket upper boundaries
)

final case class GaugeDataPoint(
  attributes: Attributes,
  timeNanos:  Long,
  value:      Double            // most recently recorded value
)
```

---

## Building an Exporter

The exporters that ship with this module are internal, but the pieces they are built from are not. `OtlpJsonEncoder` turns recorded signals into OTLP JSON bytes, [`HttpSender`](./index.md) puts bytes on the wire, and `ExportResult` classifies what came back. Together they are enough to export telemetry today, without waiting for a public exporter API. Supporting types: `NamedMetric`, `HttpResponse`. The public surface of the export path:

```scala
object OtlpJsonEncoder {
  def encodeTraces(spans: Seq[SpanData], resource: Resource, scope: InstrumentationScope): Array[Byte]
  def encodeMetrics(metrics: Seq[NamedMetric], resource: Resource, scope: InstrumentationScope): Array[Byte]
  def encodeLogs(logs: Seq[LogRecord], resource: Resource, scope: InstrumentationScope): Array[Byte]
}

sealed trait ExportResult

object ExportResult {
  case object Success                                           extends ExportResult
  final case class Failure(retryable: Boolean, message: String) extends ExportResult

  def fromHttpResponse(response: HttpResponse): ExportResult
}
```

## Motivation

`OtlpJsonTraceExporter`, `OtlpJsonLogExporter`, `OtlpJsonMetricExporter`, and the `BatchProcessor` behind them are all `private[otel]`, so there is no supported way to construct one. That is a real limitation, and the [module overview](./index.md) says so.

What it does not mean is that you cannot export. Three of the four pieces an exporter needs are public: the encoder that produces the payload, the sender that transmits it, and the result type that says whether to retry. Only the batching and retry loop is missing, and for many deployments that loop is a scheduled task you already have — a periodic flush from whatever scheduler your application runs.

Writing it yourself also buys control the internal exporter does not offer: your own retry policy, your own queue semantics, metrics about the exporter itself, and a payload you can inspect before it leaves the process.

## The Encoder

`OtlpJsonEncoder` produces the OTLP protobuf-JSON mapping directly into a `StringBuilder`, with no JSON library involved. Each method takes the signals plus the `Resource` and `InstrumentationScope` that describe their origin, and returns UTF-8 bytes ready to POST.

### Encoding Rules

The output follows the protobuf JSON mapping rather than a naive rendering, which matters if you plan to compare payloads or write assertions against them:

| Wire concern              | Encoding                                       |
| ------------------------- | ---------------------------------------------- |
| `traceId`, `spanId` bytes | Lowercase hex strings                           |
| `int64` / `uint64`        | Quoted strings, not JSON numbers                |
| Enums                     | Integers                                        |
| Field names               | camelCase                                       |
| Control characters        | `\uXXXX` escapes, and every UTF-16 surrogate      |

The quoted-integer rule is the one that surprises people: OTLP timestamps are `uint64`, so they appear as `"1755990000000000000"` rather than a bare number. A collector expects that; a hand-written comparison usually does not.

### Encoding Traces

`OtlpJsonEncoder.encodeTraces` takes `SpanData` values — the same records the core module's in-memory processor collects:

```scala

val payload: Array[Byte] = OtlpJsonEncoder.encodeTraces(
  trace.collectedSpans,
  Resource.empty,
  InstrumentationScope("checkout-service", Some("1.4.0"))
)
```

`trace.collectedSpans` returns the last 1024 ended spans, oldest first, so a flush loop that lets more than 1024 spans end between flushes loses the oldest ones; a real exporter tracks what it has already sent rather than re-encoding the whole list on every flush.

### Encoding Logs

`OtlpJsonEncoder.encodeLogs` takes `LogRecord` values, and is otherwise identical in shape:

```scala

def exportLogs(records: Seq[LogRecord]): Array[Byte] =
  OtlpJsonEncoder.encodeLogs(records, Resource.empty, InstrumentationScope("checkout-service"))
```

### Encoding Metrics

Metrics need one extra step, because the encoder wants a name and the reader does not supply one. `NamedMetric` pairs the descriptor with the data:

```scala
final case class NamedMetric(
  name: String,
  description: String,
  unit: String,
  data: MetricData
)
```

`MetricReader#collectAllMetrics` returns `Seq[MetricData]`, and `MetricData` — `SumData`, `HistogramData`, or `GaugeData` — carries only data points. The name, description, and unit are not on it:

```scala

val collected: Seq[MetricData] = metric.reader.collectAllMetrics()

val named: Seq[NamedMetric] =
  collected.map(data => NamedMetric("http.server.duration", "Request duration", "ms", data))

val payload = OtlpJsonEncoder.encodeMetrics(named, Resource.empty, InstrumentationScope("checkout-service"))
```

:::warning[`MetricData` does not carry its own name]
`MetricReader#collectAllMetrics` returns metric data with no identifying name, and there is no public way to recover which instrument produced which element. Attaching the right name means tracking the correspondence yourself — record the instruments you created in the same order you read them back, or collect per-instrument rather than in bulk. Mapping a whole batch to one name, as above, is only correct when you registered exactly one instrument.
:::

`OtlpJsonEncoder.NamedMetric` is a type alias and value alias for the same case class, so either spelling compiles.

## Interpreting the Response

`ExportResult` classifies an HTTP response into "delivered", "retry this", and "drop this". `ExportResult.fromHttpResponse` applies the standard OTLP rules, so you do not need to remember which status codes are worth a second attempt:

```scala

def result(status: Int): ExportResult =
  ExportResult.fromHttpResponse(HttpResponse(status, Array.emptyByteArray, Map.empty))
```

Any `2xx` is a success:

```scala
result(200)
// res3: ExportResult = Success
result(204)
// res4: ExportResult = Success
```

Four status codes are treated as retryable — `429`, `502`, `503`, and `504` — because each means "not now" rather than "not ever":

```scala
result(429)
// res5: ExportResult = Failure(retryable = true, message = "HTTP 429")
result(503)
// res6: ExportResult = Failure(retryable = true, message = "HTTP 503")
```

Everything else is a permanent failure, and retrying it only wastes the batch. A `400` means the collector rejected the payload itself, so the same bytes will be rejected again:

```scala
result(400)
// res7: ExportResult = Failure(retryable = false, message = "HTTP 400")
result(401)
// res8: ExportResult = Failure(retryable = false, message = "HTTP 401")
```

The `message` is a short diagnostic rather than the response body, so log the body separately when you need to know *why* a `400` was rejected.

A rejected export arrives as a non-`2xx` `HttpResponse`, not as an exception. A connection that never completes is different: `HttpSender.jdk` lets `IOException` out, so a custom loop must catch throwables and treat them as retryable itself — `ExportResult.fromHttpResponse` never sees them.

:::tip[Honour `Retry-After`]
`HttpResponse#firstHeader` looks a header up case-insensitively, which is what a `429` or `503` needs. `ExportResult` does not read it, so a retry loop that ignores `Retry-After` will keep hammering a collector that just asked it to wait.
:::

## Putting It Together

A minimal exporter is a flush function: take what has accumulated, encode it, send it, and act on the result. This is the whole shape, with the retry decision made from `ExportResult`:

```scala

val config = ExporterConfig(
  endpoint = "https://otlp.example.com:4318",
  headers = Map("Authorization" -> "Bearer <token>"),
  timeout = Duration.ofSeconds(10)
)

val sender = HttpSender.jdk(config.timeout)
val scope  = InstrumentationScope("checkout-service", Some("1.4.0"))

def flushTraces(spans: Seq[SpanData]): ExportResult =
  if (spans.isEmpty) ExportResult.Success
  else {
    val body = OtlpJsonEncoder.encodeTraces(spans, Resource.empty, scope)
    try ExportResult.fromHttpResponse(
      sender.send(config.endpoint + "/v1/traces", config.headers + ("Content-Type" -> "application/json"), body)
    )
    catch {
      case e: java.io.IOException => ExportResult.Failure(retryable = true, message = e.getMessage)
    }
  }
```

Three details in there are easy to get wrong. The signal path is appended to the endpoint — `/v1/traces`, `/v1/logs`, or `/v1/metrics` — because `ExporterConfig#endpoint` is a base URL. `Content-Type: application/json` must be set, since the payload is the JSON mapping rather than protobuf. And the `IOException` catch is not optional: without it, a collector that is simply unreachable takes down whatever thread the flush runs on.

Call `HttpSender#shutdown` when the process is stopping. The JDK sender holds nothing that needs releasing, but a custom sender may, and a flush that never runs at shutdown loses whatever was still queued.

## What You Are Reimplementing

The internal `BatchProcessor` does four things a hand-rolled flush does not, and they are worth deciding about explicitly rather than discovering later:

- **Bounded queueing.** It caps the queue at `ExporterConfig#maxQueueSize` and evicts the oldest record when full, so recording never blocks and memory never grows without bound. A naive accumulator has neither property.
- **Chunking.** It splits a drained queue into `ExporterConfig#maxBatchSize` pieces, so one flush after a traffic spike becomes several right-sized requests instead of one oversized one.
- **Interval flushing.** It flushes every `ExporterConfig#flushIntervalMillis` regardless of how full the batch is, which is what bounds how stale exported data can be.
- **Retry on retryable failures.** It retries a batch whose `ExportResult` was `Failure(retryable = true)` (or whose export threw) up to five more times, waiting 1, 2, 4, 8 and 16 seconds on the flushing thread, and drops it after that or at once when the failure is not retryable.

`ExporterConfig` carries all three sizing fields even though nothing public reads them, so use them as the source of truth for your own loop rather than inventing separate numbers. See [the module overview](./index.md) for what each field means.

## Integration Points

This page uses only public API: `OtlpJsonEncoder`, `NamedMetric`, `ExportResult`, `HttpResponse`, `HttpSender`, and `ExporterConfig` from this module, and `SpanData`, `LogRecord`, `MetricData`, `Resource`, and `InstrumentationScope` from the core telemetry module.

The signals themselves come from the core module's recording APIs — [`trace`](../tracing/index.md) for spans, [`log`](../logging/index.md) for records, and [`metric`](../metrics/index.md) for measurements. For trace context across service boundaries rather than export, see [the module overview](./index.md).

---

## OTLP Export

The core telemetry module records signals inside your process. This module gets them out: it speaks OTLP over HTTP, so anything that accepts OpenTelemetry data — a collector, Jaeger, Tempo, a vendor's endpoint — can receive your traces, logs, and metrics.

It also carries trace context *between* processes. A trace that stops at your service boundary isn't much use, so a propagator reads and writes the standard headers that let two services contribute spans to the same trace.

## Installation

Add the module to your build file:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-telemetry-otel" % "0.0.56"
```

## Configuring the Endpoint and Batching

One `ExporterConfig` answers two questions for an exporter: where to send data, and how much to hold before sending it. Both matter for the same reason — a collector is a network hop away, so sending one span per request would add a round trip to every request you were trying to measure. Records accumulate and go out in batches instead.

```scala
final case class ExporterConfig(
  endpoint: String             = "http://localhost:4318",
  headers: Map[String, String] = Map.empty,
  timeout: Duration            = Duration.ofSeconds(30),
  maxQueueSize: Int            = 2048,
  maxBatchSize: Int            = 512,
  flushIntervalMillis: Long    = 5000
)
```

Every field has a default, so `ExporterConfig()` is valid and points at a collector on localhost — the usual local development setup. `endpoint` is the collector's base URL, `headers` carries whatever it needs to accept you (typically an API key or bearer token), and `timeout` is the value to pass to `HttpSender.jdk`, which bounds a single send attempt (nothing else reads it):

```scala

val config = ExporterConfig(
  endpoint = "https://otlp.example.com:4318",
  headers = Map("Authorization" -> "Bearer <token>"),
  timeout = Duration.ofSeconds(10)
)
```

Headers are treated as sensitive: `toString` prints their count, never their contents, so a config logged at startup can't leak a token.

The three batching fields trade latency against overhead. `flushIntervalMillis` is how long a partial batch waits before going out anyway — lower it to see data sooner, raise it to send fewer, larger requests. `maxBatchSize` is how many records go in one request; filling a batch does not trigger a send, it only chunks what the next flush drains, so the interval is what determines when data leaves. `maxQueueSize` is how many records may be waiting at once: once the queue is over that limit each new record evicts the oldest one still queued, so recording never blocks and the queue never grows without bound. The defaults suit a service with steady moderate traffic; a high-throughput one should raise `maxQueueSize` before it starts dropping.

## Sending the Bytes

`HttpSender` is the one piece of the export path that actually touches the network. An exporter hands it a URL, headers, and a body of OTLP bytes; it performs the request and returns the response.

It's a separate trait so the network is replaceable. Real deployments need things a fixed HTTP client can't know about — an outbound proxy, request signing, a corporate TLS setup — and tests need the opposite: no network at all.

```scala
trait HttpSender {
  def send(url: String, headers: Map[String, String], body: Array[Byte]): HttpResponse
  def shutdown(): Unit
}

object HttpSender {
  def jdk(timeout: Duration = Duration.ofSeconds(30)): HttpSender
}

final case class HttpResponse(statusCode: Int, body: Array[Byte], headers: Map[String, Seq[String]]) {
  def firstHeader(name: String): Option[String]
}
```

`HttpSender.jdk` wraps `java.net.http.HttpClient`, which ships with the JVM, so nothing is added to your dependencies. The timeout applies to both connecting and completing a request. `shutdown()` is where a sender releases what it holds — the JDK one holds nothing that needs closing, so its implementation does nothing:

```scala

val sender = HttpSender.jdk(Duration.ofSeconds(10))
```

`send` reports a rejected export as an `HttpResponse` with a non-`2xx` status rather than as an exception, so an exporter can decide whether a failure is worth retrying. A connection that never completes still throws — `HttpSender.jdk` lets `IOException` out — and the batching layer treats such a throw as a retryable failure. Its `firstHeader(name)` looks a header up case-insensitively, which is what you need for a `Retry-After` on a `429` or `503`.

To write your own, implement the two methods: return a `2xx` status for a send the exporter should treat as delivered, anything else to mark it failed. Most custom senders are wrappers rather than replacements — sign the request, pick a proxy, count attempts, then delegate to `HttpSender.jdk(...)` and pass its response back.

## Propagating Context Across Services

One request usually touches several services. Your API calls an inventory service, which calls a database proxy — and you want that whole journey to read as *one* trace, not three unrelated ones.

That doesn't happen by itself. Each service starts a fresh trace unless the caller tells it which trace it is already part of, and an HTTP call carries nothing but headers. So the caller writes the current trace's identity into a header, and the receiver reads it back out and continues that trace instead of beginning its own.

A propagator does that writing and reading. The identity travels as text in a standard header, which is what lets it cross between services written in different languages:

```scala
trait Propagator {
  def extract[C](carrier: C, getter: (C, String) => Option[String]): Option[SpanContext]
  def inject[C](spanContext: SpanContext, carrier: C, setter: (C, String, String) => C): C
  def fields: Seq[String]
}
```

The carrier is whatever holds your headers — a `Map`, your framework's request type — and you supply the accessor, so no HTTP library is assumed. `fields` names the headers a propagator touches, which is what you pass to a CORS or header-allowlist configuration.

Use `W3CTraceContextPropagator` unless something upstream requires otherwise: it implements the W3C `traceparent` standard, which is what OpenTelemetry emits by default and what most backends expect. `B3Propagator` covers Zipkin's format in two shapes — `B3Propagator.single` puts everything in one `b3` header, `B3Propagator.multi` splits it across `X-B3-TraceId`, `X-B3-SpanId`, and `X-B3-Sampled`, and also reads `X-B3-Flags`, where `1` marks a debug trace that is always sampled.

On the way out, take the active span's context and write it into a header map:

```scala

val tracer = trace.get("api-service")

tracer.span("call-inventory", SpanKind.Client) { span =>
  val headers = W3CTraceContextPropagator.inject(
    span.spanContext,
    Map.empty[String, String],
    (carrier: Map[String, String], k: String, v: String) => carrier + (k -> v)
  )
  // headers now holds "traceparent" -> "00-<traceId>-<spanId>-01"
  headers
}
```

On the way in, `extract` returns `None` for a missing header or one it cannot parse; check `SpanContext#isValid`, since `B3Propagator.single` returns an invalid context for a sampling-only header. A request without trace context simply starts its own trace. An extracted context is marked `isRemote = true`, which is how you tell a continued trace from one that started locally.

Extracting isn't enough on its own, though — the context has to be *active* while your handler runs, or the span you open won't attach to it. Hold the `ContextStorage` you gave the provider and scope the remote context around the work:

```scala

val storage = ContextStorage.create[Option[SpanContext]](None)
trace.install(TracerProvider.builder.setContextStorage(storage).build())

val tracer = trace.get("api-service")

val parent: Option[SpanContext] =
  W3CTraceContextPropagator.extract(
    Map("traceparent" -> "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"),
    (m: Map[String, String], k: String) => m.get(k)
  )

storage.scoped(parent) {
  tracer.span("handle-checkout", SpanKind.Server) { span =>
    span.setAttribute("http.route", "/checkout")
  }
}
```

Create the storage yourself: a provider's own is internal, so there's no reaching it after the fact. Note also that `trace.get` binds to whichever provider is installed when it runs, so take your tracers after `trace.install`.

## Exporting Traces, Logs, and Metrics

The exporters that carry each signal to a collector are internal to this module: `OtlpJsonTraceExporter`, `OtlpJsonLogExporter`, and `OtlpJsonMetricExporter` are `private[otel]`, as is the `BatchProcessor` that batches records before a send. There is no public way to construct one yet.

The pieces they are built from *are* public, though, so exporting does not have to wait for that entry point. `OtlpJsonEncoder` produces OTLP JSON bytes from recorded signals, `HttpSender` transmits them, and `ExportResult` says whether a failed send is worth retrying — see [Building an Exporter](./custom-exporter.md) for the encoder, the result type, and a worked flush function.

For development and testing, the core module's in-process destinations avoid the network entirely: [`log.writer`](../logging/index.md) for formatted output, [`trace.collectedSpans`](../tracing/index.md) for recorded spans, and [`metric.reader`](../metrics/index.md) for measurement snapshots. The [Telemetry Guide](../../../guides/telemetry-guide.md) sketches the intended shape of the wiring once a public exporter lands.

## Threading Context Through `Context[R]`

`Propagator` moves trace context between processes. `OtelContext` moves it *within* one, across code that threads dependencies through `Context[R]` rather than reading an ambient storage:

```scala
final case class OtelContext(spanContext: Option[SpanContext])

object OtelContext {
  def current(storage: ContextStorage[Option[SpanContext]]): OtelContext
  def withSpan[A](span: Span, storage: ContextStorage[Option[SpanContext]])(f: => A): A
}
```

`OtelContext.current` snapshots whatever span context is active in a storage, giving a plain value that can be placed into a `Context[R & OtelContext]` and passed along like any other dependency:

```scala

val storage = ContextStorage.create[Option[SpanContext]](None)
val snapshot = OtelContext.current(storage)
```

`OtelContext.withSpan` is the inverse: it makes a span's context active for the duration of a block and restores the previous value afterwards, including when the block throws:

```scala

val storage = ContextStorage.create[Option[SpanContext]](None)
val tracer  = trace.get("api-service")

tracer.span("handle-request", SpanKind.Server) { span =>
  OtelContext.withSpan(span, storage) {
    // anything reading `storage` here sees this span's context
    OtelContext.current(storage)
  }
}
```

An `IsNominalType[OtelContext]` instance is provided, which is what lets the type be used as a `Context` key.

## See Also

- [How to Correlate Logs and Traces Across Services](../../../guides/correlate-logs-and-traces.md) — inject and extract `traceparent` on an http-model `Request`, continue the trace in the receiving service, and keep log lines correlated across threads
- [Building an Exporter](./custom-exporter.md) — `OtlpJsonEncoder`, `ExportResult`, and a worked flush function
- [AttributeValue](../common/any-value.md) — the eight attribute kinds this exporter maps onto OTLP's `AnyValue` JSON shape
- [Tracing](../tracing/index.md) — opening the spans this module exports, and [`SpanContext`](../tracing/span-context.md), the identity a propagator moves
- [Telemetry Reference](../index.md) — the core module that records what this one exports

---

## Tracing

Tracing follows a single request as it moves through your application — and across services — recording each step as a **span**: a timed unit of work with a name, attributes, and an outcome. Spans nest into a **trace**, a parent/child tree that shows where a request spent its time and where it failed. Because one trace can stretch across several services, it's how you answer "why was this request slow?" when the answer is three network hops away.

You create spans through the `trace` object: `trace.span("handle-request") { span => … }` wraps a block of work — it opens a span, runs the block, and closes the span automatically when the block returns. A `trace.span` opened inside another automatically becomes its child, so the tree builds itself with no manual wiring. Inside the block you annotate the span with attributes, events, and a status, and classify its role with a [`SpanKind`](./span-kind.md) when it crosses a service boundary.

With no setup, `trace` records every span into an in-memory buffer — the default for development and tests, where `trace.collectedSpans` hands back what was recorded. For production, call `trace.install(provider)` once at startup to send spans to a real backend (Jaeger, OTLP, …); a [`Sampler`](./sampler.md) then decides which spans to keep so you aren't exporting everything under load. The types behind `trace` — [`TracerProvider`](./tracer-provider.md), [`Tracer`](./tracer.md), [`Span`](./span.md), and the [`SpanProcessor`](./span-processor.md)s — are configured once at startup; day to day you just call `trace.span`.

## Open a Span and Record Work

Wrap a unit of work in `trace.span(name) { span => … }`.

```scala
object trace {
  def span[A](name: String)(f: Span => A): A
}
```

The block receives a live `Span`; set attributes, add timeline events, and set the completion status on it. The span closes automatically when the block returns.

```scala

trace.span("handle-request") { span =>
  span.setAttribute("http.route", "/orders")
  span.setAttribute("http.status_code", 200L)
  span.addEvent("validated")
  span.setStatus(SpanStatus.Ok)
}
```

## Classify a Span and Pre-Set Attributes

Two overloads add a [`SpanKind`](./span-kind.md) and, optionally, initial [`Attributes`](../common/attributes.md).

```scala
object trace {
  def span[A](name: String, kind: SpanKind)(f: Span => A): A
  def span[A](name: String, kind: SpanKind, attributes: Attributes)(f: Span => A): A
}
```

`SpanKind` marks a span's role in a trace (`Server`, `Client`, `Producer`, `Consumer`, or the default `Internal`), and the `Attributes` set stamps values known at creation.

```scala

trace.span("db-query", SpanKind.Client, Attributes.of(AttributeKey.string("db.system"), "postgresql")) { span =>
  span.setAttribute("db.statement", "SELECT * FROM orders")
}
```

## Nest Spans into a Trace Tree

A `trace.span` opened inside another automatically attaches to the enclosing span as its parent through the shared `ContextStorage`, so nested calls build a parent/child tree with no manual context passing.

```scala

trace.span("checkout", SpanKind.Server) { _ =>
  trace.span("load-cart") { child =>
    child.setAttribute("cart.items", 3L)
  }
  trace.span("charge-card")(_ => ())
}
```

## Scope Spans to an Instrumentation Library

Every span records which code produced it. Spans opened with `trace.span(...)` are all attributed to one scope named `"default"`, which is fine for an application tracing its own work — but it means spans from a library you depend on arrive indistinguishable from your own. `trace.get(name)` returns a [`Tracer`](./tracer.md) that stamps a name of your choosing on every span it records instead, so a backend can group them and someone debugging a slow request can see which component the time went to:

```scala
object trace {
  def get(name: String): Tracer
}
```

Calling this is the application's job, at startup: name the scope after the component it's for, then pass the `Tracer` to that component as a constructor parameter. A library should accept a `Tracer` rather than reach for the global `trace` itself — that keeps it testable in isolation and free of global state. Otherwise the two behave the same for opening spans: same overloads, same automatic parent nesting, same sampler and exporters.

```scala

val tracer: Tracer = trace.get("com.example.orders")

trace.clearSpans()
tracer.span("reserve-inventory")(_ => ())
trace.span("unscoped-work")(_ => ())

// the scope each span was recorded under
trace.collectedSpans.foreach(sd => println(sd.instrumentationScope.name))
// com.example.orders
// default
```

Order matters here. A `Tracer` is bound to whichever provider was installed when `trace.get` ran, while `trace.span` looks up the current provider on every call. Get your tracers *after* `trace.install(...)`, or a tracer created during startup will keep recording into the default in-memory buffer and its spans will never reach the exporter you installed afterwards.

## Inspect Recorded Spans

Until you install a provider, finished spans stay in memory where a test can read them back:

```scala
object trace {
  def collectedSpans: List[SpanData]
  def clearSpans(): Unit
}
```

`collectedSpans` returns each completed span as [`SpanData`](./span-data.md), oldest first, and `clearSpans()` empties the buffer — call it at the start of a test so you assert on that test's spans alone.

Two limits are worth knowing before you rely on this. The buffer is a ring of 1024 spans, so a long run silently overwrites its oldest entries; assert as you go rather than at the end of a large suite. And both methods read the buffer belonging to the **default** provider, so once you call `trace.install(...)`, they stop seeing new spans, except from tracers taken with `trace.get` before the install — a test that installs a provider has to collect through that provider's own processors instead.

## Install a Provider and Reset

`trace.install(provider)` swaps in a production `TracerProvider` — configured with a [`Resource`](../common/resource.md), a `Sampler`, and one or more `SpanProcessor` exporters.

```scala
object trace {
  def install(provider: TracerProvider): Unit
  def removeAll(): Unit
}
```

Build the provider once at startup and install it before any span is opened — name your service in the `Resource` so exported spans can be told apart from other services, and pick a `Sampler` to decide how many spans to keep:

```scala

trace.install(
  TracerProvider.builder
    .setResource(Resource.create(Attributes.of(Attributes.ServiceName, "order-service")))
    .setSampler(AlwaysOnSampler)
    .build()
)

// every trace.span in the application now records through this provider
trace.span("handle-request", SpanKind.Server)(_.setAttribute("http.route", "/orders"))
```

`trace.removeAll()` swaps in a provider whose sampler drops everything. Your `trace.span` blocks still run — they just receive a no-op `Span` that records nothing — so removing tracing never changes what your code does, only what it reports. Note that it leaves the in-memory buffer untouched; use `clearSpans()` to empty that.

## See Also

- [Telemetry Guide](../../../guides/telemetry-guide.md) — tracing data flow, sampling, and production patterns
- [Telemetry Reference](../index.md) — module overview and all three pillars
- [Common Types](../common/index.md) — `Attributes`, `AttributeKey`, `Resource`, and `InstrumentationScope`
- To test code that takes a `Tracer` as a constructor parameter, see [How to Test Code That Logs, Traces and Records Metrics](../../../guides/test-telemetry.md).

---

## Sampler

`Sampler` is the policy that decides, for every new span, whether to record it with the sampled flag set (`RecordAndSample`), record it with the flag cleared (`RecordOnly`), or drop it (`Drop`). Both recording decisions reach every `SpanProcessor` identically; `RecordOnly` differs only in the cleared flag, which is the signal a downstream exporter or service uses to leave the span unexported. It exists to keep tracing volume under control: the sampler runs once per span in the hot path, before any recording work happens. You rarely implement one — the three built-ins cover the common strategies — and you configure a sampler once on the [`TracerProvider`](./tracer-provider.md) rather than calling it yourself. When the decision is `Drop`, [`Tracer`](./tracer.md) hands your block a `Span.NoOp` that records nothing and reaches no processor.

```scala
trait Sampler {
  def shouldSample(
    parentContext: Option[SpanContext],
    traceIdHi:     Long,
    traceIdLo:     Long,
    name:          String,
    kind:          SpanKind,
    attributes:    Attributes,
    links:         Seq[SpanLink]
  ): SamplingResult

  def description: String
}

final case class SamplingResult(
  decision:   SamplingDecision,  // Drop | RecordOnly | RecordAndSample
  attributes: Attributes,        // extra attributes injected into the span
  traceState: String
)
```

## Predefined Samplers

Three built-in samplers cover the strategies most services need, so a custom implementation is rarely necessary:

| Sampler                             | Behaviour                                                                          |
|-------------------------------------|------------------------------------------------------------------------------------|
| `AlwaysOnSampler`                   | Always returns `RecordAndSample`. Default for `TracerProvider.builder`.            |
| `AlwaysOffSampler`                  | Always returns `Drop`, reusing one cached `SamplingResult`.                        |
| `ParentBasedSampler(root: Sampler)` | Follows the parent span's sampled flag; delegates to `root` when no parent exists. |

Set one on the provider at startup — this configures OpenTelemetry-compatible head sampling:

```scala

val provider = TracerProvider.builder
  .setSampler(ParentBasedSampler(AlwaysOnSampler))
  .build()

provider.shutdown()
```

## Custom Samplers

When the built-ins do not fit — for example, sampling only premium tenants or at a probabilistic rate — implement `Sampler` yourself. Cache the `SamplingResult` values so the hot path stays allocation-free:

```scala

object TenantSampler extends Sampler {
  private val sampled = SamplingResult(SamplingDecision.RecordAndSample, Attributes.empty, "")
  private val dropped = SamplingResult(SamplingDecision.Drop, Attributes.empty, "")

  def shouldSample(
    parentContext: Option[SpanContext],
    traceIdHi: Long, traceIdLo: Long,
    name: String, kind: SpanKind, attributes: Attributes,
    links: Seq[SpanLink]
  ): SamplingResult =
    if (attributes.get(AttributeKey.string("tenant")).contains("premium")) sampled
    else dropped

  def description: String = "TenantSampler"
}

val provider = TracerProvider.builder.setSampler(TenantSampler).build()
provider.shutdown()
```

The `attributes` your sampler sees are only the ones passed at creation — `trace.span(name, kind, attributes)` or `tracer.span(name, kind, attributes)`. Anything you set on the span *inside* the block happens after the decision, so a sampler can never route on it. A span started through [`SpanBuilder`](./span-builder.md) never consults a sampler at all.

A dropped span is not nothing. The block still runs with a no-op span, and a valid trace id is still put in scope with the sampled bit cleared — so a nested `trace.span` under `ParentBasedSampler` is dropped too, and an outgoing request still carries the trace with flags `00`. That's what lets a downstream service honor the decision you made at the edge.

---

## SpanBuilder

`SpanBuilder` is a mutable builder for creating a [`Span`](./span.md) by hand, outside the automatic [`trace.span`](./index.md) lifecycle, with control the `span(name, kind, attributes)` overloads cannot express: an explicit start timestamp, links to spans in other traces, or a non-default parent context. 

Crucially, a span it starts bypasses the [`Tracer`](./tracer.md)'s [`Sampler`](./sampler.md) and [`SpanProcessor`](./span-processor.md) list — nothing samples it, no `onStart`/`onEnd` fires, and **it is never exported to a backend**. The span lives only in memory, readable via `toSpanData`, and you must `end()` it yourself. Use `SpanBuilder` for tests or local inspection; for any span that should reach your tracing pipeline, use `trace.span`.

```scala
object SpanBuilder {
  def apply(name: String): SpanBuilder
}

final class SpanBuilder {
  // Configuration
  def setKind(kind: SpanKind): SpanBuilder
  def setParent(parentContext: SpanContext): SpanBuilder
  def setAttribute[A](key: AttributeKey[A], value: A): SpanBuilder
  def addLink(link: SpanLink): SpanBuilder
  def setStartTimestamp(nanos: Long): SpanBuilder
  def setResource(resource: Resource): SpanBuilder
  def setInstrumentationScope(scope: InstrumentationScope): SpanBuilder

  // Finalization
  def startSpan(): Span                                  // parent's trace ID, or a random one at the root
  def startSpan(traceIdHi: Long, traceIdLo: Long): Span  // explicit trace ID
}
```

## Usage

Obtain a pre-configured builder from `Tracer.spanBuilder` — it fills in the tracer's `Resource` and `InstrumentationScope` — set the metadata the `span` overloads cannot, then `startSpan()` and `end()` the span in a `try`/`finally`:

```scala

val tracer = TracerProvider.builder.build().get("com.example")

val span = tracer.spanBuilder("enqueue-message")
  .setKind(SpanKind.Producer)
  .addLink(SpanLink(SpanContext.invalid, Attributes.empty)) // in practice, an upstream context
  .setStartTimestamp(System.nanoTime())
  .startSpan()

try span.setAttribute("message.id", "msg-456")
finally span.end()
```

`SpanBuilder(name)` also builds one standalone, without a tracer. It is rarely what you want — the span it produces is attributed to `Resource.empty` and the scope name `"default"` rather than to your service and component, on top of reaching no sampler or processor. Reach for it only where no tracer exists yet, such as bridging a span in from another system. Note also that `startSpan()` marks the span sampled whenever there is no valid parent, so a standalone span always claims the sampled bit.

---

## SpanContext

`SpanContext` is the propagatable identity of a span — the 128-bit trace ID (split into two `Long` fields to avoid boxing), the `SpanId`, the `TraceFlags`, a W3C trace-state string, and an `isRemote` flag that marks contexts extracted from an upstream request. It exists so telemetry from one request can be correlated across services and signals.

You rarely touch it directly: [`Tracer`](./tracer.md) generates one per span and logging stamps the active context onto every record automatically. Crossing a process boundary is explicit — a `Propagator` from the otel module injects it into outgoing headers and extracts it on the way in — so reach for it when you cross a boundary yourself. `SpanContext.invalid` is the sentinel for "no active span".

```scala
final case class SpanContext(
  traceIdHi:   Long,        // high 64 bits of the 128-bit trace ID
  traceIdLo:   Long,        // low  64 bits of the 128-bit trace ID
  spanId:      SpanId,      // AnyVal wrapping a Long
  traceFlags:  TraceFlags,  // AnyVal wrapping a Byte
  traceState:  String,      // W3C tracestate header value
  isRemote:    Boolean      // true when extracted from an incoming request
) {
  def isValid:    Boolean   // true only when BOTH the trace ID and span ID are non-zero
  def isSampled:  Boolean   // true when the sampled bit of traceFlags is set
  def traceIdHex: String    // 32-char lowercase hex trace ID
}

object SpanContext {
  val invalid: SpanContext  // sentinel: all-zero IDs, not sampled, not remote
  def create(traceIdHi: Long, traceIdLo: Long, spanId: SpanId, traceFlags: TraceFlags,
             traceState: String, isRemote: Boolean): SpanContext
}
```

`SpanContext` is read from `ContextStorage` by `Tracer.currentSpan` and by the logger on every emit for automatic trace–log correlation. [`SpanData`](./span-data.md) carries two of them: the span's own context and its parent's, where `SpanContext.invalid` marks a root span with no parent.

---

## SpanData

`SpanData` is an immutable, fully-populated snapshot of a finished span — its name, timing, attributes, events, status, and identity, captured when the span ends. You rarely handle it directly: it is the read-only record a [`SpanProcessor`](./span-processor.md) receives (`onEnd`) and serializes to a backend, and the value `trace.collectedSpans` returns for test assertions. You produce spans with [`trace.span`](./index.md), never `SpanData` itself.

```scala
final case class SpanData(
  name:                 String,
  kind:                 SpanKind,
  spanContext:          SpanContext,
  parentSpanContext:    SpanContext,        // SpanContext.invalid for a root span
  startTimeNanos:       Long,
  endTimeNanos:         Long,               // 0L if snapshotted before end()
  attributes:           Attributes,
  events:               List[SpanEvent],
  links:                List[SpanLink],
  status:               SpanStatus,
  resource:             Resource,
  instrumentationScope: InstrumentationScope
)
```

Each field mirrors what the span recorded; a `SpanProcessor` or exporter reads them to ship the span to a backend (OTLP or another format).

---

## SpanKind

`SpanKind` is a sealed enumeration that classifies a span by its role in a trace — internal work, the client or server side of a synchronous RPC, or the producer or consumer side of an asynchronous message. It aligns with the [OpenTelemetry SpanKind specification](https://opentelemetry.io/docs/specs/otel/trace/api/#spankind), and a [`Tracer`](./tracer.md) records it on each [`Span`](./span.md) so backends can reconstruct client↔server and producer↔consumer relationships across services.

```scala
sealed trait SpanKind
object SpanKind {
  case object Internal extends SpanKind
  case object Server   extends SpanKind
  case object Client   extends SpanKind
  case object Producer extends SpanKind
  case object Consumer extends SpanKind
}
```

Each kind describes what the span covers and how it relates to a span in another process:

| Kind       | Represents                                                                                                                       | Relationship in a trace                                   |
|------------|----------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------|
| `Internal` | Work performed internally in the application. The default when no kind is given.                                                 | No remote parent or child.                                |
| `Server`   | The server side of a synchronous RPC — begins when the server starts processing the request and ends when it sends the response. | Usually the child of a remote `Client` span.              |
| `Client`   | The client side of a synchronous RPC — begins when the request is sent and ends when the response arrives.                       | Usually the parent of a remote `Server` span.             |
| `Producer` | The producer side of an asynchronous message — begins when the message is sent and ends once it is dispatched.                   | Parent that often completes before the `Consumer` begins. |
| `Consumer` | The consumer side of an asynchronous message — begins when the message is received and ends when processing finishes.            | Child of a `Producer` span.                               |

## Usage

Pass a `SpanKind` as the second argument to [`trace.span`](./index.md) (or `Tracer#span`) to classify a span; omit it to accept the `Internal` default.

```scala

// Server span: handling an inbound request
trace.span("handle-order", SpanKind.Server) { _ =>
  // Client span: an outbound call made while handling it
  trace.span("charge-payment", SpanKind.Client)(_ => ())
}
```

Setting the kind accurately lets a tracing backend pair a `Client` span in one service with the `Server` span it triggers in another, and a `Producer` span with the `Consumer` that eventually handles its message.

## See Also

- [Span](./span.md) — the unit of work a `SpanKind` classifies.
- [Tracing](./index.md) — the tracing overview and the `trace` entry point.
- [OpenTelemetry SpanKind specification](https://opentelemetry.io/docs/specs/otel/trace/api/#spankind) — the cross-language semantics this enumeration follows.

---

## SpanProcessor

`SpanProcessor` is the extension point exporter authors implement to do something with finished spans — ship them to a backend, buffer them, or collect them for tests. A [`Tracer`](./tracer.md) calls `onStart` when a recorded span begins — `RecordOnly` as well as `RecordAndSample` — and `onEnd` with the immutable [`SpanData`](./span-data.md) snapshot when it finishes. You rarely implement this directly: `SpanProcessor.noop` and the built-in in-memory collector cover development and testing, so you write one only to bridge spans to an export target such as OTLP.

To capture finished spans in memory and assert on them in tests, see [How to Test Code That Logs, Traces and Records Metrics](../../../guides/test-telemetry.md).

```scala
trait SpanProcessor extends AutoCloseable {
  def onStart(span: Span): Unit         // span still mutable; RecordOnly + RecordAndSample
  def onEnd(spanData: SpanData): Unit   // immutable snapshot — export from here
  def shutdown(): Unit                  // flush and release; called by TracerProvider.shutdown()
  def forceFlush(): Unit                // flush buffered data now
  override def close(): Unit = shutdown()
}

object SpanProcessor {
  val noop: SpanProcessor  // ignores every event
}
```

## Implementing a Processor

A custom processor forwards each `SpanData` to an export target. Register it on the provider with `addSpanProcessor`; when several are registered, they receive callbacks in insertion order.

`onEnd` runs synchronously, on the thread that just ended the span — so whatever it does lands in that request's latency. Keep it to handing the snapshot off to a queue or buffer, and let a background thread do the network call:

```scala

val exporter = new SpanProcessor {
  def onStart(span: Span): Unit       = ()
  def onEnd(spanData: SpanData): Unit = println(s"[EXPORT] ${spanData.name}")
  def shutdown(): Unit                = ()
  def forceFlush(): Unit              = ()
}

val provider = TracerProvider.builder.addSpanProcessor(exporter).build()
trace.install(provider)
trace.span("checkout") { _ => () } // prints: [EXPORT] checkout
provider.shutdown()
```

## Integration

The global `trace` object's default provider already registers an `InMemorySpanProcessor`, so `trace.collectedSpans` and `trace.clearSpans` are available for test assertions without any additional setup.

---

## SpanStatus

`SpanStatus` records the outcome of the operation a [`Span`](./span.md) represents. The default is `SpanStatus.Unset`; instrumentation sets `Ok` on success or `Error(description)` on failure before the span's `end()`.

```scala
sealed trait SpanStatus
object SpanStatus {
  case object Unset                           extends SpanStatus  // default; outcome unknown or normal
  case object Ok                              extends SpanStatus  // explicit success
  final case class Error(description: String) extends SpanStatus  // explicit failure with description
}
```

Set the status on the live span with `Span#setStatus` at any point before it ends:

```scala

val provider = TracerProvider.builder.build()
val tracer   = provider.get("com.example")

tracer.span("process-payment") { span =>
  val ok = true // result of processing
  if (ok) span.setStatus(SpanStatus.Ok)
  else span.setStatus(SpanStatus.Error("payment-gateway-timeout"))
}

provider.shutdown()
```

`SpanStatus` becomes part of the exported [`SpanData`](./span-data.md) (`SpanData.status`) and is read by [`SpanProcessor`](./span-processor.md) implementations and test assertions.

## See Also

- [Span](./span.md) — the unit of work whose outcome `SpanStatus` records.
- [SpanKind](./span-kind.md) — the companion classification, set once at span creation.
- [Tracing](./index.md) — the tracing overview.

---

## Span

`Span` is a mutable, thread-safe unit of work in a distributed trace — it records the name, kind, attributes, events, and outcome of one operation. You obtain a live `Span` from the higher-order [`trace.span`](./index.md) (or `Tracer#span`), which creates it, samples it, and ends it for you — you rarely create one directly.

Here is the full interface; the sections below cover the methods you reach for day to day:

```scala
trait Span {
  // Identity — fixed at creation
  def spanContext: SpanContext
  def name: String
  def kind: SpanKind

  // Annotate — no-ops once the span has ended
  def setAttribute[A](key: AttributeKey[A], value: A): Unit
  def setAttribute(key: String, value: String): Unit
  def setAttribute(key: String, value: Long): Unit
  def setAttribute(key: String, value: Double): Unit
  def setAttribute(key: String, value: Boolean): Unit
  def addEvent(name: String): Unit
  def addEvent(name: String, attributes: Attributes): Unit
  def addEvent(name: String, timestamp: Long, attributes: Attributes): Unit
  def setStatus(status: SpanStatus): Unit

  // Lifecycle — managed for you by trace.span
  def isRecording: Boolean
  def end(): Unit
  def end(endTimeNanos: Long): Unit
  def toSpanData: SpanData
}
```

## Usage

Open a span with `trace.span`: it hands your block a live `Span` and ends it automatically when the block returns. Inside, annotate the span with attributes, events, and a status:

```scala

trace.span("http.request", SpanKind.Server) { span =>
  span.setAttribute("http.method", "GET")
  span.setAttribute("http.status_code", 200L)
  span.addEvent("auth.passed")
  span.setStatus(SpanStatus.Ok)
}
```

`trace.span` manages the lifecycle: it ends the span when the block returns, and on end the span becomes an immutable [`SpanData`](./span-data.md) snapshot that `SpanProcessor`s receive (end-user code seldom reads it via `toSpanData` directly). For the rare cases `trace.span` can't cover — a custom start timestamp, links, or crossing an async boundary — [`SpanBuilder`](./span-builder.md) starts a span you `end()` yourself; it bypasses the sampler and processors, so prefer `trace.span`.

## Annotating a Span

The four convenience overloads cover the common value types; the typed `setAttribute[A](key, value)` form accepts any [`AttributeKey`](../common/attribute-key.md), including the sequence-valued ones (`AttributeKey.stringSeq` and friends) that have no shorthand.

Everything you do with a span happens inside the `trace.span` block, before it ends. Once the span ends, every `setAttribute`, `addEvent`, and `setStatus` call is silently ignored. If the sampler dropped the span you receive `Span.NoOp` — a no-op you can still call freely, so no guards are needed; check `isRecording` only to skip expensive annotation work for a dropped or ended span. Reading from a `NoOp` gives you blanks rather than errors: an invalid `spanContext`, an empty `name`, `SpanKind.Internal`, and `isRecording = false`.

### Attributes

Attach typed key-value pairs with `setAttribute` — there are convenience overloads for `String`, `Long`, `Double`, and `Boolean`. Setting the same key twice replaces the value:

```scala

trace.span("db.query") { span =>
  span.setAttribute("db.system", "postgresql")
  span.setAttribute("db.rows", 12L)
  span.setAttribute("db.cached", false)
  span.setAttribute("db.latency_ms", 3.14)
}
```

### Events

Record a named point-in-time event with `addEvent`, optionally with its own attributes:

```scala

trace.span("payment.process") { span =>
  span.addEvent("validation.passed")
  span.addEvent("retry", Attributes.of(AttributeKey.long("attempt"), 2L))
}
```

### Status

Mark the outcome with `setStatus` — `SpanStatus.Ok` on success, `SpanStatus.Error(reason)` on failure (the default is `SpanStatus.Unset`):

```scala

trace.span("charge") { span =>
  val paid = true
  if (paid) span.setStatus(SpanStatus.Ok)
  else span.setStatus(SpanStatus.Error("gateway timeout"))
}
```

## Identity

A span's `name`, `kind`, and [`spanContext`](./span-context.md) are set at creation and never change. `spanContext` is the span's identity — the trace and span IDs that let all the telemetry from one request be tied together.

You rarely touch it. Log correlation happens for you: logging stamps the active span's IDs onto every record automatically. Carrying the identity to another service is an explicit step — a `Propagator` from the otel module writes it into outgoing headers. Reach for `span.spanContext.traceIdHex` directly only in the low-level case — correlating with a system that isn't wired in, or crossing a boundary the instrumentation doesn't cover.

## Ending a Span

`trace.span` ends the span for you, including when the block throws — the span is closed on the way out and the exception propagates. Nothing sets an error status for you, though: a span whose block threw still reports `SpanStatus.Unset` unless you call `setStatus(SpanStatus.Error(...))` yourself.

Ending is idempotent. The first `end()` wins and any later call is silently ignored, so a `finally` block that ends a span already ended by `trace.span` is harmless. The `end(endTimeNanos)` overload exists for when you are replaying or bridging spans and need to state the finish time rather than take the clock's.

---

## TracerProvider

`TracerProvider` exists so tracing is configured once, in a single place. You set the service identity ([`Resource`](../common/resource.md)), the sampling policy ([`Sampler`](./sampler.md)), and where finished spans go ([`SpanProcessor`](./span-processor.md)s) when you build it at startup; from then on every [`Tracer`](./tracer.md) it hands out — and every span they create — shares that one configuration, so all your telemetry carries a consistent identity and policy. You build it once, install it, and rarely touch it again.

```scala
final class TracerProvider private[telemetry] (...) extends AutoCloseable {
  def get(name: String, version: String = ""): Tracer  // a Tracer for a named scope

  def shutdown(): Unit        // shut down every processor; call once at exit
  def forceFlush(): Unit      // ask processors to export buffered spans now
  override def close(): Unit  // alias for shutdown() — satisfies AutoCloseable
}

object TracerProvider {
  def builder: TracerProviderBuilder
}

final class TracerProviderBuilder private[telemetry] (...) {
  def setResource(resource: Resource): TracerProviderBuilder
  def setSampler(sampler: Sampler): TracerProviderBuilder
  def addSpanProcessor(processor: SpanProcessor): TracerProviderBuilder
  def setContextStorage(contextStorage: ContextStorage[Option[SpanContext]]): TracerProviderBuilder
  def build(): TracerProvider
}
```

## Usage

Build a provider once at startup, set only what you need — a `Resource` for your service identity, a `Sampler`, and the `SpanProcessor`s that export your spans — then `get` a `Tracer` from it. Unset fields take defaults: `Resource.default` (whose `service.name` is the placeholder `unknown_service`) and `AlwaysOnSampler`, which records every span.

Do add a processor, though. The default processor list is **empty**, so a provider you build yourself samples spans and then discards them — nothing keeps or exports them. The in-memory buffer behind `trace.collectedSpans` belongs to the global `trace` object's own provider, not to yours:

```scala

val provider = TracerProvider.builder
  .setResource(Resource.create(Attributes.of(Attributes.ServiceName, "payments")))
  .setSampler(ParentBasedSampler(AlwaysOnSampler))
  .addSpanProcessor(SpanProcessor.noop) // replace with an OTLP exporter
  .build()

val tracer = provider.get("com.example.payments")
tracer.span("process-payment") { span =>
  span.setAttribute("payment.id", "pay-001")
}

provider.shutdown()
```

Most applications never hold the provider directly. Install it once on the global `trace` object at startup and use [`trace.span`](./index.md) everywhere, removing the setup burden from individual call sites:

```scala

val provider = TracerProvider.builder
  .setResource(Resource.create(Attributes.of(Attributes.ServiceName, "catalog")))
  .addSpanProcessor(SpanProcessor.noop) // replace with your exporter
  .build()

trace.install(provider)
```

Each `get` builds a new `Tracer` — there is no caching — so take one per component at startup and hold it, rather than calling `get` inside a request. Passing a `version` alongside the name records it on the scope, which is how you tell spans from two releases of the same library apart.

## Lifecycle

Call `shutdown()` once when the app exits — it forwards to every registered processor so each can flush what it is holding and release its resources. `forceFlush()` does the flushing without the release, for when you want buffered spans exported now.

`shutdown()` does not disable the provider: it still hands out tracers and still records spans afterwards, they just reach processors that have already been shut down. Treat it as the last thing you call, not a switch you can toggle. Since a provider is `AutoCloseable` — `close()` is an alias for `shutdown()` — you can also let the language make that call:

```scala

Using.resource(TracerProvider.builder.addSpanProcessor(SpanProcessor.noop).build()) { provider =>
  provider.get("com.example").span("startup")(_ => ())
}
```

## Trace–Log Correlation

Logs emitted inside a span are correlated with it automatically: a `TracerProvider` and a `LoggerProvider` share the same default context storage, so every log record carries that span's trace and span IDs with no wiring. (Override `setContextStorage` on both only to isolate correlation — e.g. in tests.) See the [Telemetry module index](../index.md).

---

## Tracer

`Tracer` creates spans within a single instrumentation scope — a named library or component. Call `Tracer#span` to wrap a block of work in a timed, attributed span; the tracer consults the configured [`Sampler`](./sampler.md), propagates the parent context so nested spans inherit the right trace, and notifies each [`SpanProcessor`](./span-processor.md) as spans start and end. 

Library code takes a `Tracer` — obtained from a [`TracerProvider`](./tracer-provider.md) — to tag its spans with its own scope; application code usually prefers the global [`trace.span`](./index.md), which manages a tracer for you. A `Tracer` is always produced by a `TracerProvider` and cannot be constructed directly.

```scala
final class Tracer private[telemetry] (...) {
  val instrumentationScope: InstrumentationScope
  val resource: Resource

  // Wrap a block in a span — three progressive overloads
  def span[A](name: String)(f: Span => A): A
  def span[A](name: String, kind: SpanKind)(f: Span => A): A
  def span[A](name: String, kind: SpanKind, attributes: Attributes)(f: Span => A): A

  // Manual span creation (advanced)
  def spanBuilder(name: String): SpanBuilder

  // The active span context, or None outside any span
  def currentSpan: Option[SpanContext]
}
```

## Usage

Obtain a `Tracer` from a provider (or `trace.get` in application code), then wrap work in `span`. The three overloads add control progressively: `span(name)` defaults to `SpanKind.Internal`, `span(name, kind)` sets the kind, and `span(name, kind, attributes)` also seeds initial attributes. Nested `span` calls inherit the parent trace automatically:

```scala

val tracer = TracerProvider.builder.build().get("com.example.orders")

tracer.span("process-order", SpanKind.Server) { span =>
  span.setAttribute("order.id", "ord-123")

  tracer.span("persist-order", SpanKind.Client) { inner =>
    inner.setAttribute("db.system", "postgresql")
  }
}
```

For advanced needs — a manual span lifecycle via [`SpanBuilder`](./span-builder.md) (local-only, never exported) or reading the active context with `currentSpan` (a [`SpanContext`](./span-context.md)) — see those pages. For all normal work, `span` is enough.

---

## Tooling & Codegen

## Introduction

The blocks in this group are for programs that read or write text formats and source code instead of serving users: generators, documentation tools and API model processors.

## The Blocks

The group has three blocks, each aimed at a different format.

| Block                                  | Artifact               | What It Is For                                                                  |
| -------------------------------------- | ---------------------- | ------------------------------------------------------------------------------- |
| [Code Generation](../codegen/index.md) | `zio-blocks-codegen`   | Generic Scala code generation IR and emitter                                    |
| [Docs](../docs.md)                     | `zio-blocks-markdown`  | GitHub Flavored Markdown parsing, rendering, and programmatic construction      |
| [Smithy](../smithy.md)                 | `zio-blocks-smithy`    | Smithy IDL parser and AST library for API modeling                              |

## How the Blocks Differ

`Code Generation` provides an intermediate representation of Scala code, such as files, case classes, sealed traits and methods, and a pure emitter that turns it into formatted Scala source.

`Docs` is the Markdown library: an immutable model of a document, a strict parser, renderers for Markdown, HTML and the terminal, and a string interpolator validated at compile time. Its artifact is `zio-blocks-markdown`.

`Smithy` parses Smithy 2.0 text into a data structure, lets a program query the shapes in a model, and prints a model back as valid Smithy.

## Where to Start

Start with [Code Generation](../codegen/index.md) to emit Scala source from a model, and [Smithy](../smithy.md) when the input is an API model. Read [Docs](../docs.md) to parse or build Markdown documents.

---

## TypeId

`TypeId[A]` represents the identity of a type or type constructor at runtime — it captures complete type metadata (names, type parameters, parent types, annotations, classification) that would otherwise be erased by the JVM and Scala.js. Use `TypeId` when you need to preserve full type information as data for serialization, code generation, registry lookups, or type-safe dispatching.

In Scala and the JVM, compile-time type information is erased at runtime. This means generic type parameters, sealed trait variants, and even opaque types become indistinguishable at runtime — `List[Int]` and `List[String]` both look like `List` to the JVM. This erasure makes it nearly impossible to implement universal serializers that work across formats (JSON, YAML, XML, MessagePack), code generators, or schema-driven transformations without losing semantic information. `TypeId` solves this by capturing complete type structure at compile time and making it available as a hashable, inspectable value at runtime.

The `TypeId` trait exposes the type's structure through a rich set of properties and predicates:

```scala
// Simplified — some members shown here are derived from abstract members
sealed trait TypeId[A <: AnyKind] {
  // Abstract members
  def name: String
  def owner: Owner
  def typeParams: List[TypeParam]
  def typeArgs: List[TypeRepr]
  def defKind: TypeDefKind
  def selfType: Option[TypeRepr]       // Self-type annotation, if any
  def aliasedTo: Option[TypeRepr]      // Target type for type aliases
  def representation: Option[TypeRepr] // Underlying type for opaque types
  def annotations: List[Annotation]

  // Derived properties
  final def fullName: String     // owner.asString + "." + name
  final def arity: Int           // typeParams.size
  final def isCaseClass: Boolean
  final def isSealed: Boolean
  final def isAlias: Boolean
  // ... many more derived predicates
}
```

> In Scala 3, `A` is bounded by `AnyKind` to support higher-kinded types. In Scala 2, the bound is omitted (`sealed trait TypeId[A]`).

Derive a `TypeId` for any type using the `TypeId.of` macro and then inspect the type's structure at runtime:

```scala

case class Person(name: String, age: Int)

val id = TypeId.of[Person]
```

```scala
id.name
// res1: String = "Person"
id.fullName
// res2: String = "repl.MdocSession.MdocApp0.Person"
id.isCaseClass
// res3: Boolean = true
```

## Motivation

Standard approaches to preserving type information at runtime — `ClassTag`, `TypeTag` (Scala 2), `TypeTest` (Scala 3) — each have limitations. `ClassTag` loses generic type arguments. `TypeTag` depends on `scala-reflect` and is unavailable on Scala.js. `TypeTest` only answers "is this value an instance of T?" without exposing type structure. None of them distinguish opaque types from their underlying representation.

TypeId takes a different approach: the `TypeId.of` macro captures type metadata at compile time and stores it as a plain, immutable data structure — no runtime reflection, no platform-specific APIs. This makes it suitable as a foundation for cross-platform schema systems, code generators, and type-indexed registries.

## Installation

TypeId is included in the `zio-blocks-typeid` module. Add it to your build:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-typeid" % "@VERSION"
```

For cross-platform (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-typeid" % "@VERSION"
```

Supported Scala versions: 2.13.x and 3.x.

## Creating Instances

There are two approaches to creating `TypeId` values: **automatic derivation** (recommended for normal use) and **manual construction** (for advanced metaprogramming scenarios).

### Automatic Derivation

For most users and most types, automatic derivation via `TypeId.of` or implicit `derived` is the right choice. These macros extract complete type metadata at compile time, handling all type variants correctly.

#### `TypeId.of` — Macro Derivation

The primary way to obtain a `TypeId` is through the `TypeId.of[A]` macro, which extracts complete type metadata at compile time:

```scala
object TypeId {
  inline def of[A <: AnyKind]: TypeId[A]  // Scala 3
  def of[A]: TypeId[A]                     // Scala 2 (macro)
}
```

Derive a TypeId using the macro:

```scala

case class User(id: Long, email: String)
```

```scala
val userId = TypeId.of[User]
// userId: TypeId[User] = User
userId.name
// res5: String = "User"
userId.fullName
// res6: String = "repl.MdocSession.MdocApp4.User"
userId.isCaseClass
// res7: Boolean = true
```

#### `TypeId.derived` — Implicit Derivation

TypeId instances are available implicitly through the `derived` macro. Any function that requires a `TypeId[A]` in implicit scope will have it derived automatically — you never need to pass it manually.

From the user's perspective, the API is:

```scala
object TypeId {
  inline given derived[A <: AnyKind]: TypeId[A]  // Scala 3
  implicit def derived[A]: TypeId[A]              // Scala 2 (macro)
}
```

:::note
In Scala 3, the `[A <: AnyKind]` bound allows derivation for type constructors (e.g., `TypeId[List]`). In Scala 2, the bound is `[A]` and type constructor derivation uses `TypeId[List[_]]` syntax instead.
:::

The most common use case is accepting `TypeId[A]` as an implicit parameter:

```scala

case class User(id: Long, email: String)

def describe[A](implicit typeId: TypeId[A]): String =
  s"${typeId.fullName} is a case class: ${typeId.isCaseClass}"
```

Call the function with the type argument — the TypeId is derived automatically:

```scala
describe[User]
// res9: String = "repl.MdocSession.MdocApp8.User is a case class: true"
describe[Int]
// res10: String = "scala.Int is a case class: false"
```

You can also summon a TypeId explicitly with `implicitly` (Scala 2) or `summon` (Scala 3):

```scala
val userTypeId = implicitly[TypeId[User]]
// userTypeId: TypeId[User] = User
userTypeId.name
// res11: String = "User"
```

When you need the TypeId in a single expression, use `TypeId.of[A]`. For generic functions that accept any `A` and need its TypeId alongside other implicit evidence, use implicit derivation instead.

### Manual Derivation (Smart Constructors)

For advanced use cases — unit testing with synthetic metadata, code generators that create types dynamically, or frameworks that construct TypeIds at runtime — the smart constructor functions allow you to manually assemble TypeIds by specifying their components. These are never needed in normal user code, since `TypeId.of` handles all these cases automatically.

#### `TypeId.nominal` — Nominal Types

**Nominal types** are concrete type definitions: classes, traits, and objects. In contrast to [type aliases](#typeidalias--type-aliases) (which are alternative names for existing types) and [opaque types](#typeidopaque--opaque-types) (which have a hidden representation), nominal types stand as distinct, named types in the type system.

For most end users, you don't need to use `TypeId.nominal` directly. The `TypeId.of` macro automatically derives nominal TypeIds from your actual type definitions at compile time. The `nominal` smart constructor exists for **advanced use cases**: unit testing with synthetic type metadata, code generators that create types dynamically at runtime, or frameworks that assemble TypeIds programmatically. Unless you're in one of these scenarios, `TypeId.of` is the right tool.

If you do need to construct nominal TypeIds manually, the API provides two overloads:

```scala
object TypeId {
  def nominal[A <: AnyKind](name: String, owner: Owner, kind: TypeDefKind): TypeId[A]

  def nominal[A <: AnyKind](
    name: String, owner: Owner,
    typeParams: List[TypeParam] = Nil, typeArgs: List[TypeRepr] = Nil,
    defKind: TypeDefKind = TypeDefKind.Unknown,
    selfType: Option[TypeRepr] = None,
    annotations: List[Annotation] = Nil
  ): TypeId[A]
}
```

#### `TypeId.alias` — Type Aliases

**Type aliases** are alternative names for existing types. For example, `type Age = Int` creates an alias for `Int` so code can read `Age` instead of `Int`. TypeIds for type aliases preserve the distinction from their underlying type through the `aliasedTo` property, enabling alias-aware serialization and schema generation.

For normal use, you don't need `TypeId.alias` directly. When you write a type alias in your code (e.g., `type UserId = String`), the `TypeId.of` macro automatically derives the correct TypeId. The `alias` smart constructor is for **advanced use cases**: unit testing with synthetic alias metadata, code generators that create type aliases dynamically at runtime, or frameworks that normalize or transform type aliases during schema processing. Unless you're building one of these, `TypeId.of` is the right tool.

For testing or code generation, construct an alias TypeId:

```scala
object TypeId {
  def alias[A <: AnyKind](
    name: String, owner: Owner,
    typeParams: List[TypeParam] = Nil,
    aliased: TypeRepr,
    typeArgs: List[TypeRepr] = Nil,
    annotations: List[Annotation] = Nil
  ): TypeId[A]
}
```

```scala

```

```scala
val ageId = TypeId.alias[Any]("Age", Owner.fromPackagePath("com.example"), aliased = TypeRepr.Ref(TypeId.int))
// ageId: TypeId[Any] = Age
ageId.isAlias
// res13: Boolean = true
ageId.aliasedTo
// res14: Option[TypeRepr] = Some(Ref(Int))
```

#### `TypeId.opaque` — Opaque Types

**Opaque types** (a Scala 3 feature) are types that have a distinct compile-time identity but a hidden runtime representation. For example, `opaque type UserId = String` creates a type that is distinct from `String` at compile time, but represents `String` at runtime. TypeId preserves this distinction, unlike standard reflection which erases opaque types to their underlying type — a critical capability for type-safe serialization and validation.

For normal use, you don't need `TypeId.opaque` directly. When you define an opaque type in your code, the `TypeId.of` macro automatically derives the correct TypeId with its representation. The `opaque` smart constructor is for **advanced use cases**: unit testing with synthetic opaque type metadata, code generators that create opaque types dynamically, or frameworks that need to construct type metadata for dynamically-discovered opaque types. Unless you're building one of these, `TypeId.of` is the right tool.

For testing or code generation, construct an opaque TypeId:

```scala
object TypeId {
  def opaque[A <: AnyKind](
    name: String, owner: Owner,
    typeParams: List[TypeParam] = Nil,
    representation: TypeRepr,
    typeArgs: List[TypeRepr] = Nil,
    publicBounds: TypeBounds = TypeBounds.Unbounded,
    annotations: List[Annotation] = Nil
  ): TypeId[A]
}
```

#### `TypeId.applied` — Applied Types

**Applied types** are generic types instantiated with type arguments. For example, `List[Int]` is `List` (the type constructor) applied to `Int` (the type argument), and `Map[String, Int]` is `Map` applied to two type arguments. TypeIds for applied types preserve the type arguments so serializers can generate specialized codecs, validators can type-check values, and code generators can emit correct code.

For normal use, you don't need `TypeId.applied` directly. When you write an applied type in your code (e.g., `List[Int]` or `Map[String, User]`), the `TypeId.of` macro automatically derives the correct TypeId with its type arguments preserved. The `applied` smart constructor is for **advanced use cases**: unit testing with synthetic applied type metadata, code generators that construct type expressions dynamically, or frameworks that need to build type metadata at runtime for dynamically-discovered generic types. Unless you're building one of these, `TypeId.of` is the right tool.

For testing or code generation, construct applied TypeIds by combining a type constructor with type argument expressions:

```scala
object TypeId {
  def applied[A <: AnyKind](typeConstructor: TypeId[?], args: TypeRepr*): TypeId[A]
}
```

## Core Operations

This section documents all public methods on `TypeId` and its companion object, organized by category.

### Identity and Naming

These methods provide the type's name and fully qualified path.

#### `name` — Simple Type Name

Returns the unqualified name of the type:

```scala
sealed trait TypeId[A <: AnyKind] {
  def name: String
}
```

```scala

case class Order(id: String, total: Double)
val orderId = TypeId.of[Order]
```

```scala
orderId.name
// res16: String = "Order"
TypeId.int.name
// res17: String = "Int"
TypeId.list.name
// res18: String = "List"
```

#### `fullName` — Fully Qualified Name

Returns `owner.asString + "." + name`, or just `name` if the owner is root:

```scala
sealed trait TypeId[A <: AnyKind] {
  def fullName: String
}
```

```scala
orderId.fullName
// res19: String = "repl.MdocSession.MdocApp15.Order"
TypeId.int.fullName
// res20: String = "scala.Int"
TypeId.string.fullName
// res21: String = "java.lang.String"
```

#### `owner` — Enclosing Namespace

The `TypeId#owner` method returns the `Owner` — the hierarchical path showing exactly where a type is defined. This includes the complete package chain and any enclosing objects or types. `Owner` solves a critical problem: multiple types can have the same name (e.g., `User` in `com.api` and `User` in `com.admin`), and the owner uniquely distinguishes them by their definition location:

```scala
sealed trait TypeId[A <: AnyKind] {
  def owner: Owner
}
```

When we derive a `TypeId` for a custom type, the owner captures its full hierarchical location. We can then use `TypeId#fullName` to see how the owner combines with the type name:

```scala

case class User(id: Long, name: String)
```

```scala
val userId = TypeId.of[User]
// userId: TypeId[User] = User
userId.name
// res23: String = "User"
// The owner shows where this User is defined
userId.owner.asString
// res24: String = "repl.MdocSession.MdocApp22"
// fullName combines owner and name into a qualified path
userId.fullName
// res25: String = "repl.MdocSession.MdocApp22.User"
```

When we construct types from different packages, their owners differ even though the names are identical. This is essential for registries and serializers that need to distinguish between types with conflicting names:

```scala
// A User from the admin domain
val adminUser = TypeId.nominal[Any]("User", Owner.fromPackagePath("com.admin"), TypeDefKind.Unknown)
// adminUser: TypeId[Any] = User
adminUser.name
// res26: String = "User"
// Notice the owner is different
adminUser.owner.asString
// res27: String = "com.admin"
// So the full names are distinct
adminUser.fullName
// res28: String = "com.admin.User"

// Compare: both have name "User" but different owners
userId.name == adminUser.name
// res29: Boolean = true
userId.owner.asString == adminUser.owner.asString
// res30: Boolean = false
```

This distinction enables type-indexed registries where you can safely store types with identical names from different sources without collision. 

#### `toString` — Idiomatic Scala Rendering

Renders the TypeId as idiomatic Scala syntax using `TypeIdPrinter`:

```scala
TypeId.of[List[Int]].toString
// res31: String = "List[Int]"
TypeId.of[Map[String, Int]].toString
// res32: String = "Map[String, Int]"
```

### Type Parameters and Arguments

Methods for inspecting generic type information.

#### `typeParams` — Formal Type Parameters

The `TypeId#typeParams` method returns the list of formal type parameters declared by a type. This is what makes a type generic. For a type like `Box[+A]`, `typeParams` captures the declaration of `A` — its name, its position in the parameter list, and its variance (whether it's covariant `+`, contravariant `−`, or invariant). Bounds are not recorded; see [`TypeParam`](#typeparam--type-parameter-details):

```scala
sealed trait TypeId[A <: AnyKind] {
  def typeParams: List[TypeParam]
}
```

To see how `TypeId` preserves type parameter information, we define several generic types with different variance patterns. Each demonstrates a different type parameter characteristic:

```scala

sealed trait Container[+A]
case class Box[+A](value: A) extends Container[A]

sealed trait Sink[-T]
case class Logger[-T]() extends Sink[T]

sealed trait Cache[K, +V]
case class LRUCache[K, +V](maxSize: Int) extends Cache[K, V]
```

When we derive `TypeId` for these types, we can inspect their type parameters and see the variance that was declared:

```scala
val boxId = TypeId.of[Box]
// boxId: TypeId[[A >: Nothing <: Any] =>> Box[A]] = Box[+A]
// Box declares [+A], so we see one covariant parameter
boxId.typeParams
// res34: List[TypeParam] = List(
//   TypeParam(
//     name = "A",
//     index = 0,
//     variance = Covariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
boxId.typeParams.head.variance
// res35: Variance = Covariant
boxId.typeParams.head.name
// res36: String = "A"

val sinkId = TypeId.of[Sink]
// sinkId: TypeId[[T >: Nothing <: Any] =>> Sink[T]] = Sink[-T]
// Sink declares [-T], so we see one contravariant parameter
sinkId.typeParams
// res37: List[TypeParam] = List(
//   TypeParam(
//     name = "T",
//     index = 0,
//     variance = Contravariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
sinkId.typeParams.head.variance
// res38: Variance = Contravariant

val cacheId = TypeId.of[Cache]
// cacheId: TypeId[[K >: Nothing <: Any, V >: Nothing <: Any] =>> Cache[K, V]] = Cache[K, +V]
// Cache declares [K, +V], so we see two parameters with different variances
cacheId.typeParams
// res39: List[TypeParam] = List(
//   TypeParam(
//     name = "K",
//     index = 0,
//     variance = Invariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   ),
//   TypeParam(
//     name = "V",
//     index = 1,
//     variance = Covariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
cacheId.typeParams.map(p => (p.name, p.variance.symbol))
// res40: List[Tuple2[String, String]] = List(("K", ""), ("V", "+"))
```

#### `TypeParam` — Type Parameter Details

Each element in `TypeId#typeParams` is a `TypeParam` value. A type parameter defines a single formal parameter in a generic type's declaration — its name, position, variance (covariance, contravariance, invariance), bounds, and kind. When you derive a `TypeId` for a generic type like `Box[+A]`, the macro records each declared parameter's name, position, and variance as a `TypeParam` so you can inspect them at runtime. The macro does not record bounds or kinds: every derived parameter has `TypeBounds.Unbounded` bounds and `Kind.Type` (a known bug in 0.0.56). To record a kind, build the parameter by hand, for example with `TypeParam.higherKinded`.

`TypeParam` has these fields:

```scala
final case class TypeParam(
  name: String,                           // "A", "T", "K", "F"
  index: Int,                             // Position: 0, 1, 2, ...
  variance: Variance = Variance.Invariant, // +, -, or none
  bounds: TypeBounds = TypeBounds.Unbounded, // >: Lower <: Upper
  kind: Kind = Kind.Type                  // *, * -> *, etc.
)
```

To inspect individual fields of a type parameter, we can extract and examine each one:

```scala

sealed trait Functor[F[_]]
```

To inspect individual fields of a type parameter, extract and examine each property:

```scala
val functorId = TypeId.of[Functor]
// functorId: TypeId[[F >: Nothing <: [_$1 >: Nothing <: Any] =>> Any] =>> Functor[F]] = Functor[F]
val paramF = functorId.typeParams.head
// paramF: TypeParam = TypeParam(
//   name = "F",
//   index = 0,
//   variance = Invariant,
//   bounds = TypeBounds(lower = None, upper = None),
//   kind = Type
// )

paramF.name
// res42: String = "F"
paramF.index
// res43: Int = 0
paramF.variance
// res44: Variance = Invariant
paramF.isInvariant
// res45: Boolean = true
paramF.kind
// res46: Kind = Type
paramF.isTypeConstructor
// res47: Boolean = false
```

Although `F` is declared as `F[_]`, the last two lines show `Type` and `false`: derivation does not record kinds. A parameter built by hand with `TypeParam.higherKinded` does carry one:

```scala
val handBuiltF = TypeParam.higherKinded("F", 0)
// handBuiltF: TypeParam = TypeParam(
//   name = "F",
//   index = 0,
//   variance = Invariant,
//   bounds = TypeBounds(lower = None, upper = None),
//   kind = Arrow(params = List(Type), result = Type)
// )

handBuiltF.kind
// res48: Kind = Arrow(params = List(Type), result = Type)
handBuiltF.isTypeConstructor
// res49: Boolean = true
```

`TypeParam` provides convenience predicates for checking variance without inspecting the raw `variance` field:

```scala

sealed trait Box[+A]
sealed trait Sink[-T]
sealed trait Cache[K, +V]
```

Using these types, we can check the variance predicates to verify which parameters are covariant, contravariant, or invariant:

```scala
val boxId = TypeId.of[Box]
// boxId: TypeId[[A >: Nothing <: Any] =>> Box[A]] = Box[+A]
boxId.typeParams.head.isCovariant
// res51: Boolean = true

val sinkId = TypeId.of[Sink]
// sinkId: TypeId[[T >: Nothing <: Any] =>> Sink[T]] = Sink[-T]
sinkId.typeParams.head.isContravariant
// res52: Boolean = true

val cacheId = TypeId.of[Cache]
// cacheId: TypeId[[K >: Nothing <: Any, V >: Nothing <: Any] =>> Cache[K, V]] = Cache[K, +V]
val (k, v) = (cacheId.typeParams(0), cacheId.typeParams(1))
// k: TypeParam = TypeParam(
//   name = "K",
//   index = 0,
//   variance = Invariant,
//   bounds = TypeBounds(lower = None, upper = None),
//   kind = Type
// )
// v: TypeParam = TypeParam(
//   name = "V",
//   index = 1,
//   variance = Covariant,
//   bounds = TypeBounds(lower = None, upper = None),
//   kind = Type
// )
k.isInvariant
// res53: Boolean = true
v.isCovariant
// res54: Boolean = true
```

#### `typeArgs` — Applied Type Arguments

**Applied types** are generic types instantiated with concrete type arguments. For example, `List[Int]` is the generic `List` type constructor applied to the `Int` type argument, and `Map[String, Int]` applies two arguments to `Map`. When you derive a `TypeId` for an applied type, the `typeArgs` method returns the concrete type arguments as a list of `TypeRepr` values — allowing you to inspect what types were plugged into the type constructor.

The `typeArgs` method is essential for schema systems and code generators that need to understand the full type structure. For instance, a serializer might need to know that `List[Int]` has `Int` as its element type, or a validator might need to distinguish between `Map[String, Int]` and `Map[String, String]`:

```scala
sealed trait TypeId[A <: AnyKind] {
  def typeArgs: List[TypeRepr]
}
```

`typeArgs` returns a list of `TypeRepr` values. `TypeRepr` is an algebraic data type that represents type expressions in the Scala type system — these can be simple type references (like `Int` or `String`), complex applied types (like `List[String]`), or compound types (like `A & B`). For a detailed breakdown of all `TypeRepr` variants, see [TypeRepr — Type Expressions](#typerepr--type-expressions).

Setup some custom generic types with different type argument patterns:

```scala

case class Pair[A, B](first: A, second: B)
case class Container[T](value: T)
case class Result[E, V](error: Option[E], value: Option[V])
```

Now inspect the type arguments of various applied types:

```scala
// Simple single type argument
val containerIntId = TypeId.of[Container[Int]]
// containerIntId: TypeId[Container[Int]] = Container[Int]
containerIntId.typeArgs
// res56: List[TypeRepr] = List(Ref(Int))

// Multiple type arguments
val pairId = TypeId.of[Pair[String, Double]]
// pairId: TypeId[Pair[String, Double]] = Pair[String, Double]
pairId.typeArgs
// res57: List[TypeRepr] = List(Ref(String), Ref(Double))

// Nested applied types
val resultId = TypeId.of[Result[String, List[Int]]]
// resultId: TypeId[Result[String, List[Int]]] = Result[String, List[Int]]
resultId.typeArgs
// res58: List[TypeRepr] = List(
//   Ref(String),
//   Applied(tycon = Ref(List[+A]), args = List(Ref(Int)))
// )

// Type constructor with no arguments has empty typeArgs
TypeId.of[List].typeArgs
// res59: List[TypeRepr] = List()
```

When you access `typeArgs`, each element is a `TypeRepr` describing that argument. You can inspect them further to understand the structure:

```scala
val mapStringIntId = TypeId.of[Map[String, Int]]
// mapStringIntId: TypeId[Map[String, Int]] = Map[String, Int]
val args = mapStringIntId.typeArgs
// args: List[TypeRepr] = List(Ref(String), Ref(Int))

// First argument: String
args(0)
// res60: TypeRepr = Ref(String)

// Second argument: Int
args(1)
// res61: TypeRepr = Ref(Int)
```

For more complex types, `typeArgs` captures the full structure of the arguments, including unions, intersections, function types, and tuples:

```scala

// Union types (Scala 3)
case class Handler[T](process: T | String)

// Intersection types (Scala 3)
trait Readable { def read(): String }
trait Writable { def write(data: String): Unit }
case class Stream[T](data: T & Readable & Writable)

// Function type arguments
case class Transformer[A, B](f: A => B)

// Tuple type arguments
case class MultiValue[A, B, C](values: (A, B, C))
```

Now inspect the type arguments in these complex types:

```scala
// Union type argument
val handlerStrId = TypeId.of[Handler[Int]]
// handlerStrId: TypeId[Handler[Int]] = Handler[Int]
handlerStrId.typeArgs
// res63: List[TypeRepr] = List(Ref(Int))

// Intersection type argument
val streamId = TypeId.of[Stream[List[String]]]
// streamId: TypeId[Stream[List[String]]] = Stream[List[String]]
streamId.typeArgs
// res64: List[TypeRepr] = List(
//   Applied(tycon = Ref(List[+A]), args = List(Ref(String)))
// )

// Function type as argument
val transformerId = TypeId.of[Transformer[String, Int]]
// transformerId: TypeId[Transformer[String, Int]] = Transformer[String, Int]
transformerId.typeArgs
// res65: List[TypeRepr] = List(Ref(String), Ref(Int))

// Tuple type as argument
val multiValueId = TypeId.of[MultiValue[String, Int, Boolean]]
// multiValueId: TypeId[MultiValue[String, Int, Boolean]] = MultiValue[String, Int, Boolean]
multiValueId.typeArgs
// res66: List[TypeRepr] = List(Ref(String), Ref(Int), Ref(Boolean))
```

#### `arity` — Number of Type Parameters

The **arity** of a type is the number of formal type parameters it declares. A type with arity 0 is fully applied (a "proper type"), while arity > 0 means it's a type constructor that needs to be instantiated with type arguments. Arity is useful for generic programming and type-indexed registries where you need to distinguish between different levels of type abstraction:

```scala
sealed trait TypeId[A <: AnyKind] {
  def arity: Int
}
```

Setup some generic types with different arities:

```scala

case class Single[A](value: A)           // Arity 1
case class Pair[A, B](a: A, b: B)       // Arity 2
case class Triple[A, B, C](a: A, b: B, c: C) // Arity 3
case class Value(x: Int)                 // Arity 0
```

Check the arity of different types:

```scala
TypeId.of[Single].arity
// res68: Int = 1
TypeId.of[Pair].arity
// res69: Int = 2
TypeId.of[Triple].arity
// res70: Int = 3
TypeId.of[Value].arity
// res71: Int = 0

// Applied types have the same arity as their type constructor
TypeId.of[Single[Int]].arity
// res72: Int = 1
TypeId.of[Pair[String, Int]].arity
// res73: Int = 2
```

#### `isProperType` — Has No Type Parameters

A **proper type** (also called a ground type or monomorphic type) is a fully instantiated type with no unresolved type parameters. It's the opposite of a type constructor — you can directly instantiate values of a proper type, whereas a type constructor needs type arguments before it's usable. The `isProperType` predicate returns `true` when `arity == 0`, helping distinguish concrete types from abstract type constructors:

```scala

case class Single[A](value: A)
case class Pair[A, B](a: A, b: B)
case class Value(x: Int)
```

Check which types are proper types:

```scala
// Proper types: fully instantiated, arity == 0
TypeId.of[Value].isProperType
// res75: Boolean = true
TypeId.of[List[Int]].isProperType
// res76: Boolean = false
TypeId.of[Pair[String, Int]].isProperType
// res77: Boolean = false
TypeId.of[Int].isProperType
// res78: Boolean = true

// Type constructors: need type arguments, arity > 0
TypeId.of[Single].isProperType
// res79: Boolean = false
TypeId.of[Pair].isProperType
// res80: Boolean = false
TypeId.of[List].isProperType
// res81: Boolean = false
```

#### `isTypeConstructor` — Has Type Parameters

A **type constructor** is a parameterized type that cannot be instantiated directly — it requires concrete type arguments first. For example, `List` is a type constructor (you can't have a value of type `List`, only `List[Int]` or `List[String]`). The `isTypeConstructor` predicate returns `true` when `arity > 0`, indicating the type needs to be applied with arguments before use. This is useful for generic programming where you work with families of related types:

```scala

case class Single[A](value: A)
case class Pair[A, B](a: A, b: B)
case class Value(x: Int)
```

Identify which types are type constructors:

```scala
// Type constructors: need type arguments, arity > 0
TypeId.of[Single].isTypeConstructor
// res83: Boolean = true
TypeId.of[Pair].isTypeConstructor
// res84: Boolean = true
TypeId.of[List].isTypeConstructor
// res85: Boolean = true
TypeId.of[Map].isTypeConstructor
// res86: Boolean = true

// Proper types: fully instantiated, no type parameters
TypeId.of[Value].isTypeConstructor
// res87: Boolean = false
TypeId.of[List[Int]].isTypeConstructor
// res88: Boolean = true
TypeId.of[Int].isTypeConstructor
// res89: Boolean = false
```

#### `isApplied` — Has Type Arguments

An **applied type** is a generic type that has been instantiated with concrete type arguments. For example, `List[Int]` is an applied type (`List` applied to `Int`), while `List` by itself is a type constructor with no arguments applied. The `isApplied` predicate returns `true` when `typeArgs.nonEmpty`, helping distinguish between abstract type constructors and concrete instantiated types. This is useful for code generators that need to know whether a type is ready for use:

```scala

case class Single[A](value: A)
case class Pair[A, B](a: A, b: B)
```

Check which types are applied:

```scala
// Applied types: have type arguments
TypeId.of[List[Int]].isApplied
// res91: Boolean = true
TypeId.of[Pair[String, Int]].isApplied
// res92: Boolean = true
TypeId.of[Single[Boolean]].isApplied
// res93: Boolean = true
TypeId.of[Map[String, Double]].isApplied
// res94: Boolean = true

// Type constructors: no type arguments applied
TypeId.of[List].isApplied
// res95: Boolean = false
TypeId.of[Single].isApplied
// res96: Boolean = false
TypeId.of[Pair].isApplied
// res97: Boolean = false
TypeId.of[Map].isApplied
// res98: Boolean = false
```

### Type Classification

**Type classification** determines what kind of type definition something is — whether it's a class, trait, object, enum, alias, opaque type, or something else. This is essential for code generators, serializers, and frameworks that need to handle different type categories differently. TypeId provides both a `defKind` property that returns detailed classification information, and convenient predicates (like `isClass`, `isTrait`, `isCaseClass`) for common checks.

#### `defKind` — Type Definition Kind

Returns the `TypeDefKind` classifying this type (class, trait, object, enum, alias, opaque, etc.):

```scala
sealed trait TypeId[A <: AnyKind] {
  def defKind: TypeDefKind
}
```

Define types representing different classifications:

```scala

sealed trait Animal
case class Dog(name: String) extends Animal
case object Sentinel
type UserId = String
opaque type Email = String
enum Color { case Red; case Green; case Blue }
```

Inspect the `defKind` for each type to see how they're classified:

```scala
TypeId.of[Dog].defKind
// res100: TypeDefKind = Class(
//   isFinal = false,
//   isAbstract = false,
//   isCase = true,
//   isValue = false,
//   bases = List(Ref(Animal))
// )
TypeId.of[Animal].defKind
// res101: TypeDefKind = Trait(isSealed = true, bases = List())
TypeId.of[Sentinel.type].defKind
// res102: TypeDefKind = Object(List())
TypeId.of[UserId].defKind
// res103: TypeDefKind = TypeAlias
TypeId.of[Email].defKind
// res104: TypeDefKind = OpaqueType(TypeBounds(lower = None, upper = None))
TypeId.of[Color].defKind
// res105: TypeDefKind = Enum(List(Ref(Enum)))
```

#### Classification Predicates

Each predicate inspects `defKind` for a specific type definition kind. These are convenience methods that save you from pattern matching on `defKind` directly:

| Predicate        | Returns `true` when                                         |
|------------------|-------------------------------------------------------------|
| `isClass`        | `defKind` is `TypeDefKind.Class`                            |
| `isTrait`        | `defKind` is `TypeDefKind.Trait`                            |
| `isObject`       | `defKind` is `TypeDefKind.Object`                           |
| `isEnum`         | `defKind` is `TypeDefKind.Enum`                             |
| `isAlias`        | `defKind` is `TypeDefKind.TypeAlias`                        |
| `isOpaque`       | `defKind` is `TypeDefKind.OpaqueType`                       |
| `isAbstract`     | `defKind` is `TypeDefKind.AbstractType`                     |
| `isSealed`       | `defKind` is `TypeDefKind.Trait(isSealed = true, _)` (sealed traits only) |
| `isCaseClass`    | `defKind` is `TypeDefKind.Class(_, _, isCase = true, _, _)` |
| `isValueClass`   | `defKind` is `TypeDefKind.Class(_, _, _, isValue = true, _)`|

Use the classification predicates to identify type kinds:

```scala
val animalId   = TypeId.of[Animal]
// animalId: TypeId[Animal] = Animal
val dogId      = TypeId.of[Dog]
// dogId: TypeId[Dog] = Dog
val sentinelId = TypeId.of[Sentinel.type]
// sentinelId: TypeId[Sentinel] = Sentinel
val userIdId   = TypeId.of[UserId]
// userIdId: TypeId[UserId] = UserId
val emailId    = TypeId.of[Email]
// emailId: TypeId[Email] = Email
val colorId    = TypeId.of[Color]
// colorId: TypeId[Color] = Color

// Trait classifications
animalId.isTrait
// res106: Boolean = true
animalId.isSealed
// res107: Boolean = true

// Case class
dogId.isCaseClass
// res108: Boolean = true

// Object/Singleton
sentinelId.isObject
// res109: Boolean = true

// Type alias
userIdId.isAlias
// res110: Boolean = true

// Opaque type
emailId.isOpaque
// res111: Boolean = true

// Enum
colorId.isEnum
// res112: Boolean = true
```

:::note
`isSealed` only checks sealed traits. A sealed abstract class or sealed enum will return `false`; use `defKind` directly to inspect those cases by pattern matching on `TypeDefKind.Class(_, isAbstract = true, ...)` or `TypeDefKind.Enum(...)`.
:::

#### Semantic Predicates

**Semantic predicates** check specific semantic properties of the type after normalization, allowing you to identify built-in Scala types like tuples, products, sums, options, and either. These are useful for generic serializers and validators that treat built-in types specially.

**Normalization** resolves type aliases to the types they stand for. For example, if you have `type UserId = String`, normalization reveals that the underlying type is `String`. An opaque type is not unwrapped: `opaque type Email = String` stays `Email`. This allows predicates like `isOption` to work correctly when the type is wrapped in an alias — it looks through the alias to find the actual semantic type — but not when it is wrapped in an opaque type, for which `isOption` returns `false` even if the representation is an `Option`.

| Predicate   | Checks                                                 |
|-------------|--------------------------------------------------------|
| `isTuple`   | Normalized type is `scala.TupleN`                      |
| `isProduct` | Normalized type is `scala.Product` or `scala.ProductN` |
| `isSum`     | Normalized type is named `Either` or `Option`          |
| `isEither`  | Normalized type is `scala.util.Either`                 |
| `isOption`  | Normalized type is `scala.Option`                      |

Check semantic properties with practical examples:

```scala
// Tuples
TypeId.of[(String, Int)].isTuple
// res113: Boolean = true
TypeId.of[(Int, String, Boolean)].isTuple
// res114: Boolean = true

// Options and Either
TypeId.of[Option[String]].isOption
// res115: Boolean = true
TypeId.of[Either[String, Int]].isEither
// res116: Boolean = true

// Products (built-in Scala Product interface, not user case classes)
TypeId.of[Product2[String, Int]].isProduct
// res117: Boolean = true
```

:::note
`isProduct` returns `true` only for Scala's built-in `scala.Product`, `scala.Product1`, etc. -- not for user-defined case classes. Use `isCaseClass` for that.
:::

Understanding the distinction between `isSum`, `isEither`, and `isOption`:

```scala

// For standard library types, use isEither and isOption
TypeId.of[Option[String]].isOption
// res118: Boolean = true
TypeId.of[Option[String]].isSum
// res119: Boolean = true

TypeId.of[Either[String, Int]].isEither
// res120: Boolean = true
TypeId.of[Either[String, Int]].isSum
// res121: Boolean = false
```

[//]: # (:::note)
[//]: # (**Practical guidance:** Always use `isEither` for `scala.util.Either` and `isOption` for `scala.Option`. The `isSum` predicate is rarely needed — it checks for hypothetical types named `"Option"` or `"Either"` placed directly in the `scala` package itself &#40;not in `scala.util` subpackage&#41;, which almost never occurs in real code.)
[//]: # (:::)

### Subtype Relationships

**Subtype relationships** determine the inheritance hierarchy and compatibility between types at runtime. This is essential for type-safe dispatch, generic programming, and validating that a value of one type can be used where another type is expected. TypeId provides methods to check direct and transitive subtyping, supertyping, type equivalence, and inspect the parent types in the hierarchy.

#### `isSubtypeOf` — Check Subtyping

Checks if this type is a subtype of another type. A type is a subtype if it extends or implements the other type, either directly or transitively. This method handles direct inheritance, sealed trait subtypes, enum cases, transitive inheritance chains, and variance-aware subtyping for applied generic types:

```scala
sealed trait TypeId[A <: AnyKind] {
  def isSubtypeOf(other: TypeId[?]): Boolean
}
```

Define a type hierarchy with direct and transitive relationships:

```scala

sealed trait Animal
sealed trait Mammal extends Animal
case class Dog(name: String) extends Mammal
case class Cat(name: String) extends Mammal
case class Fish(species: String) extends Animal
```

Check subtyping relationships:

```scala
val dogId     = TypeId.of[Dog]
// dogId: TypeId[Dog] = Dog
val mammalId  = TypeId.of[Mammal]
// mammalId: TypeId[Mammal] = Mammal
val animalId  = TypeId.of[Animal]
// animalId: TypeId[Animal] = Animal
val fishId    = TypeId.of[Fish]
// fishId: TypeId[Fish] = Fish

// Direct inheritance: Dog extends Mammal
dogId.isSubtypeOf(mammalId)
// res123: Boolean = true

// Transitive inheritance: Dog extends Mammal extends Animal
dogId.isSubtypeOf(animalId)
// res124: Boolean = true

// Not a subtype relationship
dogId.isSubtypeOf(fishId)
// res125: Boolean = false
fishId.isSubtypeOf(mammalId)
// res126: Boolean = false
```

Covariant type constructors preserve subtyping relationships:

```scala
TypeId.of[List[Dog]].isSubtypeOf(TypeId.of[List[Mammal]])
// res127: Boolean = true
TypeId.of[List[Dog]].isSubtypeOf(TypeId.of[List[Animal]])
// res128: Boolean = true
```

**Scala 3 exclusive features:** In Scala 3, `isSubtypeOf` handles advanced type relationships that Scala 2 cannot. These examples show what works in Scala 3:

```scala

// Scala 3: Enum cases
enum Color {
  case Red
  case Green
  case Blue
}

// Scala 3: Union type aliases
type StringOrInt = String | Int

// Scala 3: Intersection type aliases
trait Readable { def read(): String }
trait Writable { def write(data: String): Unit }
type ReadWrite = Readable & Writable
```

In Scala 3, `isSubtypeOf` correctly handles these advanced type cases:

```scala
// Enum cases: Red is a subtype of Color
TypeId.of[Color.Red.type].isSubtypeOf(TypeId.of[Color])
// res130: Boolean = true

// Union type aliases: String is one of the union members
TypeId.of[String].isSubtypeOf(TypeId.of[StringOrInt])
// res131: Boolean = true
TypeId.of[Int].isSubtypeOf(TypeId.of[StringOrInt])
// res132: Boolean = true

// Intersection type aliases: A type implementing both traits is a subtype
val readWriteId = TypeId.of[ReadWrite]
// readWriteId: TypeId[ReadWrite] = ReadWrite
val readableId = TypeId.of[Readable]
// readableId: TypeId[Readable] = Readable
readWriteId.isSubtypeOf(readableId)
// res133: Boolean = true
```

:::note
In Scala 2, `isSubtypeOf` does not handle `EnumCase` subtypes, types aliased to union types, or types aliased to intersection types — only the Scala 3 implementation checks those cases.
:::

#### `isSupertypeOf` — Check Supertyping

The mirror of `isSubtypeOf` — returns `true` if the other type is a subtype of this type. This is useful when you need to check if a type can accept instances of another type, or when validating that a container type can hold values of a more specific type:

```scala
sealed trait TypeId[A <: AnyKind] {
  def isSupertypeOf(other: TypeId[?]): Boolean
}
```

Check supertyping relationships using the same hierarchy:

```scala

sealed trait Animal
sealed trait Mammal extends Animal
case class Dog(name: String) extends Mammal
case class Cat(name: String) extends Mammal
case class Fish(species: String) extends Animal

val dogId     = TypeId.of[Dog]
val mammalId  = TypeId.of[Mammal]
val animalId  = TypeId.of[Animal]
val fishId    = TypeId.of[Fish]
```

Now check supertyping relationships:

```scala
// Mammal is a supertype of Dog (Mammal can hold Dog instances)
mammalId.isSupertypeOf(dogId)
// res135: Boolean = true

// Animal is a supertype of both Dog and Fish (Animal is the most general)
animalId.isSupertypeOf(dogId)
// res136: Boolean = true
animalId.isSupertypeOf(fishId)
// res137: Boolean = true

// Mammal is a supertype of Cat too
mammalId.isSupertypeOf(TypeId.of[Cat])
// res138: Boolean = true

// But Dog is not a supertype of Mammal (can't hold all Mammals as Dogs)
dogId.isSupertypeOf(mammalId)
// res139: Boolean = false

// And Fish is not a supertype of Mammal
fishId.isSupertypeOf(mammalId)
// res140: Boolean = false
```

:::note
**Limitation:** TypeId's subtyping checks currently do not handle contravariance in function types. In type theory, `Mammal => String` should be a supertype of `Dog => String` due to contravariance of input parameters, but `isSupertypeOf` returns `false` for function types with subtype relationships. For practical purposes, rely on `isSupertypeOf` for class and trait hierarchies rather than complex generic type relationships.
:::

#### `isEquivalentTo` — Check Type Equivalence

Returns `true` when two types are structurally equivalent — meaning they are **mutual subtypes** of each other. In other words, both `A.isSubtypeOf(B)` and `B.isSubtypeOf(A)` must be true. Two types are equivalent when they represent the same type through different paths, or when they normalize to the same underlying type (important for type aliases; an opaque type is not unwrapped, so it is not equivalent to its representation):

```scala
sealed trait TypeId[A <: AnyKind] {
  def isEquivalentTo(other: TypeId[?]): Boolean
}
```

Check type equivalence with practical examples:

```scala

sealed trait Animal
sealed trait Mammal extends Animal
case class Dog(name: String) extends Mammal
case class Cat(name: String) extends Mammal

val dogId     = TypeId.of[Dog]
val mammalId  = TypeId.of[Mammal]
val animalId  = TypeId.of[Animal]
val catId     = TypeId.of[Cat]
```

Now check type equivalence:

```scala
// A type is always equivalent to itself
dogId.isEquivalentTo(dogId)
// res142: Boolean = true

// The same type referenced twice is equivalent
val dogId2 = TypeId.of[Dog]
// dogId2: TypeId[Dog] = Dog
dogId.isEquivalentTo(dogId2)
// res143: Boolean = true

// Different types in the hierarchy are NOT equivalent (one-way subtyping only)
dogId.isEquivalentTo(mammalId)
// res144: Boolean = false
mammalId.isEquivalentTo(animalId)
// res145: Boolean = false

// Cat and Dog are different types, even though both extend Mammal
dogId.isEquivalentTo(catId)
// res146: Boolean = false
```

Type aliases normalize to the same type, making them equivalent:

```scala

type UserId = String
type Username = String
```

Both aliases normalize to `String`, so they are equivalent:

```scala
val userIdType = TypeId.of[UserId]
// userIdType: TypeId[UserId] = UserId
val usernameType = TypeId.of[Username]
// usernameType: TypeId[Username] = Username
val stringType = TypeId.of[String]
// stringType: TypeId[String] = String

// Both type aliases are equivalent because they normalize to the same underlying type
userIdType.isEquivalentTo(usernameType)
// res148: Boolean = true

// And both are equivalent to their underlying type
userIdType.isEquivalentTo(stringType)
// res149: Boolean = true
usernameType.isEquivalentTo(stringType)
// res150: Boolean = true
```

#### `parents` — Parent Types

Returns the list of parent type representations as `TypeRepr` values, flattened across the full inheritance hierarchy. Each parent is represented as a `TypeRepr` that captures the parent type, including any type arguments it might have. This is useful for code generators, serializers, and frameworks that need to understand the inheritance structure of a type:

```scala
sealed trait TypeId[A <: AnyKind] {
  def parents: List[TypeRepr]
}
```

```scala

trait Swimmer { def swim(): Unit = () }
trait Flyer   { def fly(): Unit  = () }
trait Duck extends Swimmer with Flyer
case class MallardDuck() extends Duck
```

Parents are flattened across the full hierarchy:

```scala
// Duck extends Swimmer and Flyer directly
TypeId.of[Duck].parents
// res152: List[TypeRepr] = List(Ref(Flyer), Ref(Swimmer))

// MallardDuck extends Duck — parents include Duck, Swimmer, and Flyer
TypeId.of[MallardDuck].parents
// res153: List[TypeRepr] = List(Ref(Duck), Ref(Flyer), Ref(Swimmer))
```

### Metadata

Methods for accessing annotations, self-type, alias target, and opaque representation.

#### `annotations` — Type Annotations

Returns the list of annotations attached to this type at compile time. Each `Annotation` carries the annotation's name and its argument values, making this useful for frameworks that drive behaviour from annotations (e.g. serialization hints, validation rules, or access-control markers):

```scala
sealed trait TypeId[A <: AnyKind] {
  def annotations: List[Annotation]
}
```

```scala

@deprecated("use NewData instead", "2.0")
@transient
case class LegacyData(id: Int, payload: String)
case class Plain(x: Int)
```

A type with annotations reports each annotation by name; an unannotated type returns an empty list:

```scala
// LegacyData has two annotations
TypeId.of[LegacyData].annotations.map(_.name)
// res155: List[String] = List("transient", "deprecated")

// Plain has no annotations
TypeId.of[Plain].annotations
// res156: List[Annotation] = List()
```

#### `selfType` — Self-Type Annotation

Returns `Some(typeRepr)` when the trait declares a self-type (e.g., `trait Foo { self: Bar => ... }`), and `None` otherwise. Self-types express a dependency requirement: a trait that declares `self: Logger =>` can only be mixed into a class that also mixes in `Logger`. This method lets frameworks detect and validate those requirements at runtime:

```scala
sealed trait TypeId[A <: AnyKind] {
  def selfType: Option[TypeRepr]
}
```

```scala

trait Logger { def log(msg: String): Unit }
trait Service { self: Logger => def doWork(): Unit = log("working") }
```

`Service` requires a `Logger` to be mixed in, while `Logger` has no self-type requirement:

```scala
// Service declares a self-type dependency on Logger
TypeId.of[Service].selfType
// res158: Option[TypeRepr] = Some(Logger & Service)

// Logger has no self-type requirement
TypeId.of[Logger].selfType
// res159: Option[TypeRepr] = None
```

#### `aliasedTo` — Alias Target

Returns `Some(typeRepr)` for type aliases pointing to their underlying type, and `None` for nominal and opaque types. This lets you inspect what a type alias expands to without evaluating expressions at runtime:

```scala
sealed trait TypeId[A <: AnyKind] {
  def aliasedTo: Option[TypeRepr]
}
```

```scala

type Age = Int
type Name = String
```

A type alias resolves to its target; a concrete type returns `None`:

```scala
// Age is an alias for Int
TypeId.of[Age].aliasedTo
// res161: Option[TypeRepr] = Some(Ref(Int))

// Name is an alias for String
TypeId.of[Name].aliasedTo
// res162: Option[TypeRepr] = Some(Ref(String))

// Int is a concrete type, not an alias
TypeId.of[Int].aliasedTo
// res163: Option[TypeRepr] = None
```

#### `representation` — Opaque Type Representation

Returns `Some(typeRepr)` for opaque types revealing their underlying representation type, and `None` for all other types. Opaque types hide their implementation behind a new name, but `representation` lets frameworks such as serializers discover what the type is actually stored as:

```scala
sealed trait TypeId[A <: AnyKind] {
  def representation: Option[TypeRepr]
}
```

```scala

opaque type Email = String
opaque type UserId = Int
```

An opaque type exposes its representation; a non-opaque type returns `None`:

```scala
// Email is an opaque type backed by String
TypeId.of[Email].representation
// res165: Option[TypeRepr] = Some(Ref(String))

// UserId is an opaque type backed by Int
TypeId.of[UserId].representation
// res166: Option[TypeRepr] = Some(Ref(Int))

// Int is not opaque
TypeId.of[Int].representation
// res167: Option[TypeRepr] = None
```

### Erasure and Runtime

Methods for type erasure, runtime class lookup, and reflective construction.

#### `erased` — Erase Type Parameter

Erases the phantom type parameter, returning a `TypeId.Erased` (alias for `TypeId[TypeId.Unknown]`). This is useful when you need to store heterogeneous `TypeId` values in a collection or a type-indexed map, where the exact type parameter is unknown or irrelevant at the storage site:

```scala
sealed trait TypeId[A <: AnyKind] {
  def erased: TypeId.Erased
}
```

Different types can be stored together once erased:

```scala
val ids: List[TypeId.Erased] = List(
  TypeId.of[Int].erased,
  TypeId.of[String].erased,
  TypeId.of[Boolean].erased
)
// ids: List[Erased] = List(Int, String, Boolean)

ids.map(_.name)
// res168: List[String] = List("Int", "String", "Boolean")
```

#### `classTag` — Runtime ClassTag

Returns a `ClassTag` for this type. Returns the correct primitive `ClassTag` for Scala primitive types (`Int`, `Long`, `Boolean`, etc.) and `ClassTag.AnyRef` for all reference types. This is useful when you need to create properly-typed arrays or work with generic collections that require implicit `ClassTag` evidence at runtime:

```scala
sealed trait TypeId[A <: AnyKind] {
  lazy val classTag: scala.reflect.ClassTag[?]
}
```

On the JVM, arrays are reified — the element type is part of the array object at runtime, not erased like generics. To create an array of a generic type `T`, Scala requires a `ClassTag[T]` so the runtime knows whether to allocate a primitive array (`int[]`, `double[]`) or an object array (`Object[]`). This matters for memory efficiency: a primitive `int[]` stores 4 bytes per element unboxed, while an `Integer[]` stores heap references plus the cost of boxing each value.

`classTag` returns the correct `ClassTag` for each type:

```scala
// Primitive types have dedicated ClassTags
TypeId.of[Int].classTag
// res169: ClassTag[_ >: Nothing <: Any] = Int
TypeId.of[Double].classTag
// res170: ClassTag[_ >: Nothing <: Any] = Double

// Reference types use ClassTag.AnyRef
TypeId.of[String].classTag
// res171: ClassTag[_ >: Nothing <: Any] = Object
TypeId.of[List[Int]].classTag
// res172: ClassTag[_ >: Nothing <: Any] = Object
```

A concrete use case is a generic storage allocator that creates the right array type from a `TypeId`:

```scala

def makeStorage(size: Int, id: TypeId[?]): Array[?] =
  id.classTag.newArray(size)
```

```scala
// Creates int[] (primitive, unboxed)
makeStorage(100, TypeId.int).getClass.getComponentType
// res174: Class[_ >: Nothing <: <FromJavaObject>] = int

// Creates double[] (primitive, unboxed)
makeStorage(100, TypeId.double).getClass.getComponentType
// res175: Class[_ >: Nothing <: <FromJavaObject>] = double

// Creates Object[] (reference)
makeStorage(100, TypeId.string).getClass.getComponentType
// res176: Class[_ >: Nothing <: <FromJavaObject>] = class java.lang.Object
```

Another use case is detecting primitive types. Without `classTag`, you would need to enumerate every primitive with a chain of `isInstanceOf` checks:

```scala
// Without classTag: every primitive listed explicitly
def isPrimitive(value: Any): Boolean =
  value.isInstanceOf[Int]     ||
  value.isInstanceOf[Long]    ||
  value.isInstanceOf[Float]   ||
  value.isInstanceOf[Double]  ||
  value.isInstanceOf[Boolean] ||
  value.isInstanceOf[Byte]    ||
  value.isInstanceOf[Short]   ||
  value.isInstanceOf[Char]
```

This is fragile: if you forget one primitive (e.g. `Unit`) the check silently breaks. With `classTag` the same question reduces to a single comparison that can never miss a case — `ClassTag.AnyRef` is the universal fallback for every reference type, so anything that is not `AnyRef` must be a primitive:

```scala

def isPrimitive(id: TypeId[?]): Boolean =
  id.classTag != scala.reflect.ClassTag.AnyRef
```

```scala
isPrimitive(TypeId.of[Int])
// res178: Boolean = true
isPrimitive(TypeId.of[Double])
// res179: Boolean = true
isPrimitive(TypeId.of[Boolean])
// res180: Boolean = true
isPrimitive(TypeId.of[String])
// res181: Boolean = false
isPrimitive(TypeId.of[List[Int]])
// res182: Boolean = false
```

:::note
`classTag` returns `ClassTag.AnyRef` for all reference types. For matching or filtering by a specific reference type at runtime, use `clazz` instead.
:::

#### `clazz` — Runtime Class

Returns the runtime `Class[_]` for this type. On the JVM it returns `Some(Class[_])` for nominal and applied types, and `None` for alias and opaque types. On Scala.js it always returns `None` since JVM reflection is unavailable. This is the entry point for reflective operations such as instantiation, field access, or integration with Java libraries:

```scala
sealed trait TypeId[A <: AnyKind] {
  def clazz: Option[Class[?]]
}
```

```scala

type Age = Int
```

```scala
// Nominal and applied types return Some on the JVM
TypeId.of[String].clazz
// res184: Option[Class[_ >: Nothing <: Any]] = Some(class java.lang.String)
TypeId.of[Int].clazz
// res185: Option[Class[_ >: Nothing <: Any]] = Some(int)
TypeId.of[List[Int]].clazz
// res186: Option[Class[_ >: Nothing <: Any]] = Some(
//   class scala.collection.immutable.List
// )

// Alias types return None — the alias has no class of its own
TypeId.of[Age].clazz
// res187: Option[Class[_ >: Nothing <: Any]] = None
```

:::note
On Scala.js, `clazz` always returns `None`. Use `classTag` instead when you need cross-platform runtime type information.
:::

#### `construct` — Reflective Construction

Creates a value of this type on the JVM from a `Chunk[AnyRef]` of arguments. `List`, `Vector`, `Set`, `Seq`, `IndexedSeq`, `Map`, `Option`, `Some`, `None`, `Either`, `Tuple2`, `Tuple3`, and several `java.time` types are built directly; any other type is built by reflection with the first public constructor whose parameter count and types fit the arguments. Returns `Left` with an error message on Scala.js or when construction fails (no constructor fits, the constructor throws, or the type is abstract). Primitive values must be explicitly boxed since the argument type is `AnyRef`:

```scala
sealed trait TypeId[A <: AnyKind] {
  def construct(args: Chunk[AnyRef]): Either[String, Any]
}
```

```scala

// JVM only
case class User(name: String, age: Int)

val userId = TypeId.of[User]
```

```scala
userId.construct(Chunk("Alice", 30: Integer))
// res189: Either[String, Any] = Left(
//   "Cannot construct repl.MdocSession.MdocApp188.User: class not available"
// )
userId.construct(Chunk("Bob"))
// res190: Either[String, Any] = Left(
//   "Cannot construct repl.MdocSession.MdocApp188.User: class not available"
// )
```

##### Collection Types

Collection types accept variadic arguments representing elements. `construct` builds `List`, `Vector`, `Set`, `Seq`, and `IndexedSeq` directly from a sequence of elements:

```scala

```

```scala
TypeId.of[List[String]].construct(Chunk("a", "b", "c"))
// res192: Either[String, Any] = Right(List("a", "b", "c"))
TypeId.of[Vector[Int]].construct(Chunk(1: Integer, 2: Integer, 3: Integer))
// res193: Either[String, Any] = Right(Vector(1, 2, 3))
TypeId.of[Set[String]].construct(Chunk("x", "y", "z"))
// res194: Either[String, Any] = Right(Set("x", "y", "z"))
```

`Array`, `ArraySeq`, and `Chunk` are not among the types `construct` knows, so it falls back to reflection, which cannot build them. `Chunk` and `ArraySeq` give `Left`. `Array` gives `Left` too, except for a single boxed `Int`, where it returns a meaningless `scala.Array` object instead of an array (a known bug in 0.0.56):

```scala
TypeId.of[Chunk[Int]].construct(Chunk(1: Integer, 2: Integer))
// res195: Either[String, Any] = Left(
//   "No matching constructor found for zio.blocks.chunk.Chunk with 2 arguments of types: java.lang.Integer, java.lang.Integer"
// )
TypeId.of[Array[Int]].construct(Chunk(1: Integer, 2: Integer))
// res196: Either[String, Any] = Left(
//   "No matching constructor found for scala.Array with 2 arguments of types: java.lang.Integer, java.lang.Integer"
// )
```

Map types pass interleaved key-value pairs and fail on odd argument counts:

```scala
TypeId.of[Map[String, Int]].construct(Chunk("a", 1: Integer, "b", 2: Integer))
// res197: Either[String, Any] = Right(Map("a" -> 1, "b" -> 2))
```

##### Sum Types

Sum types have special calling conventions. `Option` accepts 1 element to construct `Some(value)`, or 0 elements to construct `None`:

```scala
TypeId.of[Option[String]].construct(Chunk("hello"))
// res198: Either[String, Any] = Right(Some("hello"))
TypeId.of[Option[String]].construct(Chunk())
// res199: Either[String, Any] = Right(None)
```

`Either` requires a Boolean flag as the first argument (`true` for `Right`, `false` for `Left`), followed by the value:

```scala
TypeId.of[Either[String, Int]].construct(Chunk(true: java.lang.Boolean, 42: Integer))
// res200: Either[String, Any] = Right(Right(42))
TypeId.of[Either[String, Int]].construct(Chunk(false: java.lang.Boolean, "error"))
// res201: Either[String, Any] = Right(Left("error"))
```

### Normalization and Equality

**Normalization** resolves type aliases to the underlying concrete types. For example, `type Age = Int` normalizes to `Int`, and chained aliases like `type UserId = NonEmpty; type NonEmpty = List[Int]` both resolve to `List[Int]`. Opaque types are not unwrapped: `opaque type UserId = String` stays `UserId` after normalization. Normalization is crucial because multiple syntactic names often refer to the same underlying type, enabling deduplication and caching strategies.

**Structural Equality** compares two TypeIds by their normalized form, treating types with identical underlying structure as equal. Because normalization leaves opaque types as they are, an opaque type keeps its own identity: `TypeId.of[UserId]` where `opaque type UserId = String` is never equal to `TypeId.of[String]`, preserving runtime type safety. This distinction enables type-safe registries and validators that respect opaque type boundaries.

These concepts are essential for building type-indexed registries that recognize multiple alias names as referring to the same handler, implementing serialization strategies based on normalized type structure, and enforcing opaque type safety in type-indexed maps where different opaque types wrapping the same base type should have separate validators or handlers.

#### `TypeId.normalize` — Resolve Aliases

Resolves chains of type aliases to the underlying type. For example, `type MyList = List[Int]` normalizes to `List[Int]`. An id that is not an alias comes back unchanged, and an opaque type is not unwrapped. An alias of a union or intersection type stays as it is, and on Scala 3 an alias of a tuple type and an applied alias with type parameters also come back unchanged (a known bug in 0.0.56):

```scala
object TypeId {
  def normalize(id: TypeId[?]): TypeId[?]
}
```

```scala

type Age = Int
```

```scala
val ageId = TypeId.of[Age]
// ageId: TypeId[Age] = Age
val norm = TypeId.normalize(ageId)
// norm: TypeId[_ >: Nothing <: AnyKind] = Int
norm.fullName
// res203: String = "scala.Int"
```

To see that an opaque type survives normalization, define one and normalize its id:

```scala

opaque type Email = String
```

```scala
val emailId = TypeId.of[Email]
// emailId: TypeId[Email] = Email
val normalizedEmail = TypeId.normalize(emailId)
// normalizedEmail: TypeId[_ >: Nothing <: AnyKind] = Email
normalizedEmail.isOpaque
// res205: Boolean = true
normalizedEmail == emailId
// res206: Boolean = true
normalizedEmail == TypeId.of[String]
// res207: Boolean = false
```

#### `TypeId.structurallyEqual` — Structural Equality

Checks if two TypeIds are structurally equal after normalization. Semantically equivalent to `==` on TypeId instances; `==` additionally short-circuits on hash mismatch for performance:

```scala
object TypeId {
  def structurallyEqual(a: TypeId[?], b: TypeId[?]): Boolean
}
```

```scala

type UserId = Int
```

```scala
val a = TypeId.of[UserId]
// a: TypeId[UserId] = UserId
val b = TypeId.of[Int]
// b: TypeId[Int] = Int
TypeId.structurallyEqual(a, b)
// res209: Boolean = true
a == b
// res210: Boolean = true
```

#### `TypeId.structuralHash` — Structural Hash Code

Computes a hash code based on the normalized structural representation:

```scala
object TypeId {
  def structuralHash(id: TypeId[?]): Int
}
```

#### `TypeId.unapplied` — Strip Type Arguments

Returns the type constructor by stripping all type arguments; an id that is not applied comes back unchanged. For example, `TypeId.unapplied(TypeId.of[List[Int]])` equals `TypeId.list`. On Scala 2 it also equals `TypeId.of[List[_]]`, but on Scala 3 it does not equal `TypeId.of[List]`, which derives the `scala.package.List` alias instead (a known bug in 0.0.56), so compare with `TypeId.list`:

```scala
object TypeId {
  def unapplied(id: TypeId[?]): TypeId[?]
}
```

```scala
val listInt = TypeId.of[List[Int]]
// listInt: TypeId[List[Int]] = List[Int]
val unapplied = TypeId.unapplied(listInt)
// unapplied: TypeId[_ >: Nothing <: AnyKind] = List[+A]
unapplied.isApplied
// res211: Boolean = false
unapplied.name
// res212: String = "List"
unapplied == TypeId.list
// res213: Boolean = true
unapplied == TypeId.of[List]
// res214: Boolean = false
```

### Pattern Matching Extractors

The companion object provides extractors for pattern matching on TypeId classification:

```scala

case class User(id: Long, email: String)
val userId = TypeId.of[User]
```

```scala
userId match {
  case TypeId.Nominal(name, owner, params, defKind, parents) =>
    s"Nominal type '$name' in ${owner.asString}"
  case TypeId.Alias(name, _, _, aliased) =>
    s"Alias '$name'"
  case TypeId.Opaque(name, _, _, repr, _) =>
    s"Opaque '$name'"
}
// res216: String = "Nominal type 'User' in repl.MdocSession.MdocApp215"
```

The extractors are:

| Extractor                                               | Matches                  |
|---------------------------------------------------------|--------------------------|
| `TypeId.Nominal(name, owner, params, defKind, parents)` | Classes, traits, objects |
| `TypeId.Alias(name, owner, params, aliased)`            | Type aliases             |
| `TypeId.Opaque(name, owner, params, repr, bounds)`      | Opaque types             |
| `TypeId.Sealed(name)`                                   | Sealed traits            |
| `TypeId.Enum(name, owner)`                              | Scala 3 enums            |

## TypeDefKind Reference

`TypeDefKind` classifies every type definition. Access it via the `defKind` property documented in [Core Operations](#type-classification).

The `defKind` property (documented in [Core Operations](#type-classification)) returns one of these variants. Use classification predicates like `isCaseClass`, `isSealed`, `isObject` for simple checks.

`TypeDefKind` has these variants:

| Variant                                              | Description                                  |
|------------------------------------------------------|----------------------------------------------|
| `Class(isFinal, isAbstract, isCase, isValue, bases)` | Class definitions                            |
| `Trait(isSealed, bases)`                             | Trait definitions                            |
| `Object(bases)`                                      | Singleton objects                            |
| `Enum(bases)`                                        | Scala 3 enums                                |
| `EnumCase(parentEnum, ordinal, isObjectCase)`        | Enum cases                                   |
| `TypeAlias`                                          | Type aliases (`type Foo = Bar`)              |
| `OpaqueType(publicBounds)`                           | Opaque types                                 |
| `AbstractType`                                       | Abstract type members                        |
| `Unknown`                                            | Unclassified or unresolvable type definition |

## Type Parameters and Generics

When you derive a TypeId for a generic type, the macro captures its type parameters (name, position, and variance) and any applied type arguments. It does not record the parameters' bounds or kinds (a known bug in 0.0.56).

A **raw type constructor** is a generic type without any type arguments filled in. For example, `List` by itself (without `[Int]` or `[String]`) is a raw type constructor. Scala 3 supports deriving TypeIds directly for raw type constructors, but Scala 2 has restrictions due to its type system:

**Scala 3** allows you to work with raw type constructors directly:

```scala
// Scala 3 only
val listId = TypeId.of[List]  // Works: raw type constructor
```

**Scala 2** requires you to use a wildcard placeholder or implicit derivation since raw type constructors are not valid syntax:

```scala
// Scala 2 alternatives
val listId = TypeId.of[List[_]]  // Use wildcard type argument

// Or retrieve via implicit derivation
implicit val derived: TypeId[List[_]] = TypeId.of[List[_]]
```

This distinction matters when you need to capture the type constructor itself (for higher-kinded type operations) rather than concrete types like `List[Int]`.

### Inspecting Type Parameters

Define your own generic types and derive their TypeIds to see how type parameters are captured:

```scala

sealed trait Container[+A]
case class Box[+A](value: A) extends Container[A]

sealed trait Cache[K, +V]
case class LRUCache[K, +V](maxSize: Int) extends Cache[K, V]
```

A single-parameter type constructor:

```scala
val containerId = TypeId.of[Container]
// containerId: TypeId[[A >: Nothing <: Any] =>> Container[A]] = Container[+A]
containerId.typeParams
// res218: List[TypeParam] = List(
//   TypeParam(
//     name = "A",
//     index = 0,
//     variance = Covariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
containerId.arity
// res219: Int = 1
```

A two-parameter type constructor with mixed variance (invariant `K`, covariant `V`):

```scala
val cacheId = TypeId.of[Cache]
// cacheId: TypeId[[K >: Nothing <: Any, V >: Nothing <: Any] =>> Cache[K, V]] = Cache[K, +V]
cacheId.typeParams
// res220: List[TypeParam] = List(
//   TypeParam(
//     name = "K",
//     index = 0,
//     variance = Invariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   ),
//   TypeParam(
//     name = "V",
//     index = 1,
//     variance = Covariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
cacheId.arity
// res221: Int = 2
```

Each derived `TypeParam` records the parameter's name, position, and variance, and always reports `Kind.Type`:

```scala
val containerParam = containerId.typeParams.head
// containerParam: TypeParam = TypeParam(
//   name = "A",
//   index = 0,
//   variance = Covariant,
//   bounds = TypeBounds(lower = None, upper = None),
//   kind = Type
// )
containerParam.name
// res222: String = "A"
containerParam.variance
// res223: Variance = Covariant
containerParam.kind
// res224: Kind = Type
containerParam.isCovariant
// res225: Boolean = true
```

```scala
val cacheParams = cacheId.typeParams
// cacheParams: List[TypeParam] = List(
//   TypeParam(
//     name = "K",
//     index = 0,
//     variance = Invariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   ),
//   TypeParam(
//     name = "V",
//     index = 1,
//     variance = Covariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
cacheParams.map(p => (p.name, p.variance))
// res226: List[Tuple2[String, Variance]] = List(
//   ("K", Invariant),
//   ("V", Covariant)
// )
```

### Inspecting Type Arguments

When you derive a TypeId for an *applied* type (a generic type with concrete arguments), the type arguments are captured:

```scala
val boxIntId = TypeId.of[Box[Int]]
// boxIntId: TypeId[Box[Int]] = Box[Int]
boxIntId.typeArgs
// res227: List[TypeRepr] = List(Ref(Int))
boxIntId.isApplied
// res228: Boolean = true
```

```scala
val cacheStringIntId = TypeId.of[LRUCache[String, Int]]
// cacheStringIntId: TypeId[LRUCache[String, Int]] = LRUCache[String, Int]
cacheStringIntId.typeArgs
// res229: List[TypeRepr] = List(Ref(String), Ref(Int))
```

### Variance

**Variance** describes how a type parameter's subtyping relationships are preserved. **Covariant** types (`+`) preserve subtyping (if `B <: A` then `Container[B] <: Container[A]`), **contravariant** types (`-`) reverse it, and **invariant** types preserve neither. For example, `Container[+A]` is covariant—a `Container[String]` can be used where `Container[Any]` is expected. In contrast, `Cache[K, V]` where `K` is invariant means `Cache[String, Int]` cannot substitute for `Cache[Any, Int]` even if `String <: Any`.

Variance matters for type safety, polymorphism, and API design. TypeId captures variance information, enabling runtime inspection and validation:

```scala

sealed trait Container[+A]
sealed trait Cache[K, +V]
```

```scala
val containerParams = TypeId.of[Container].typeParams
// containerParams: List[TypeParam] = List(
//   TypeParam(
//     name = "A",
//     index = 0,
//     variance = Covariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
containerParams.map(p => (p.name, p.variance))
// res231: List[Tuple2[String, Variance]] = List(("A", Covariant))

val cacheParams = TypeId.of[Cache].typeParams
// cacheParams: List[TypeParam] = List(
//   TypeParam(
//     name = "K",
//     index = 0,
//     variance = Invariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   ),
//   TypeParam(
//     name = "V",
//     index = 1,
//     variance = Covariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
cacheParams.map(p => (p.name, p.variance))
// res232: List[Tuple2[String, Variance]] = List(
//   ("K", Invariant),
//   ("V", Covariant)
// )
```

You can also work with variance values directly:

```scala
Variance.Covariant.symbol
// res233: String = "+"
Variance.Contravariant.symbol
// res234: String = "-"
Variance.Invariant.symbol
// res235: String = ""
Variance.Covariant.flip
// res236: Variance = Contravariant
```

### Kind

**Kind** describes the "type of a type"—it captures whether something is a concrete type or a type constructor, and how many type parameters it requires. A **proper type** like `Int` or `Box[String]` has kind `*` (zero parameters). A **unary type constructor** like `List` has kind `* -> *` (takes one type parameter). A **binary type constructor** like `Map` has kind `* -> * -> *` (takes two). **Higher-kinded types** like `Runnable[F[_]]` have kinds like `(* -> *) -> *` (takes a type constructor as a parameter). Kind information is essential for generic programming, enforcing API contracts, and enabling advanced patterns like monad transformers.

`TypeId` reports how many type parameters a type declares through `arity`, but derivation does not compute the `Kind` of those parameters: every derived `TypeParam` has kind `Kind.Type`, even when the parameter is itself a type constructor (a known bug in 0.0.56). The `kind` values below therefore all print `Type`. The table at the end of this section lists the predefined `Kind` values, which you can give to a `TypeParam` you build by hand:

```scala

sealed trait Container[+A]
case class Box[+A](value: A) extends Container[A]

sealed trait Cache[K, +V]
case class LRUCache[K, +V](maxSize: Int) extends Cache[K, V]

trait Runnable[F[_]] {
  def run[A](fa: F[A]): A
}
```

A proper type (`*`) — fully concrete with no type parameters:

```scala
val boxIntId = TypeId.of[Box[Int]]
// boxIntId: TypeId[Box[Int]] = Box[Int]
boxIntId.isApplied
// res238: Boolean = true
boxIntId.arity
// res239: Int = 1
```

A unary type constructor (`* -> *`) — takes one type parameter:

```scala
val containerId = TypeId.of[Container]
// containerId: TypeId[[A >: Nothing <: Any] =>> Container[A]] = Container[+A]
containerId.arity
// res240: Int = 1
containerId.typeParams.map(p => (p.name, p.kind))
// res241: List[Tuple2[String, Kind]] = List(("A", Type))
```

A binary type constructor (`* -> * -> *`) — takes two type parameters:

```scala
val cacheId = TypeId.of[Cache]
// cacheId: TypeId[[K >: Nothing <: Any, V >: Nothing <: Any] =>> Cache[K, V]] = Cache[K, +V]
cacheId.arity
// res242: Int = 2
cacheId.typeParams.map(p => (p.name, p.kind))
// res243: List[Tuple2[String, Kind]] = List(("K", Type), ("V", Type))
```

A higher-kinded type (`(* -> *) -> *`) — one whose type parameter is itself a type constructor. `Runnable` declares `F[_]`, yet the derived parameter reports kind `Type` with arity 0:

```scala
val runnableId = TypeId.of[Runnable]
// runnableId: TypeId[[F >: Nothing <: [_$4 >: Nothing <: Any] =>> Any] =>> Runnable[F]] = Runnable[F]
runnableId.typeParams.head.kind
// res244: Kind = Type
runnableId.typeParams.head.kind.arity
// res245: Int = 0
```

To record the kind, build the parameter by hand with `TypeParam.higherKinded`:

```scala
val handBuiltParam = TypeParam.higherKinded("F", 0)
// handBuiltParam: TypeParam = TypeParam(
//   name = "F",
//   index = 0,
//   variance = Invariant,
//   bounds = TypeBounds(lower = None, upper = None),
//   kind = Arrow(params = List(Type), result = Type)
// )
handBuiltParam.kind
// res246: Kind = Arrow(params = List(Type), result = Type)
handBuiltParam.kind.arity
// res247: Int = 1
```

The `Kind` values and the types whose kind each one describes:

| Kind                      | Notation        | Arity | Describes the kind of  |
|---------------------------|-----------------|-------|------------------------|
| `Kind.Type` / `Kind.Star` | `*`             | 0     | `Int`, `Box[Int]`      |
| `Kind.Star1`              | `* -> *`        | 1     | `Container`, `Option`  |
| `Kind.Star2`              | `* -> * -> *`   | 2     | `Cache`, `Either`      |
| `Kind.HigherStar1`        | `(* -> *) -> *` | 1     | `Runnable`             |

## Subtype Relationships

**Subtype relationships** determine if one type is a subtype of another, enabling type-safe polymorphism and dispatch at runtime. This is essential for checking if a value of one type can be used where another type is expected. TypeId handles direct inheritance, transitive inheritance chains, sealed trait cases, and variance-aware subtyping for generic types.

Three key methods work together to express the full range of type relationships:

```scala

sealed trait Animal
sealed trait Mammal extends Animal
case class Dog(name: String) extends Mammal
case class Cat(name: String) extends Mammal
case class Fish(species: String) extends Animal
```

**`isSubtypeOf`** checks if this type is a subtype of another (direct or transitive):

```scala
val dogId     = TypeId.of[Dog]
// dogId: TypeId[Dog] = Dog
val mammalId  = TypeId.of[Mammal]
// mammalId: TypeId[Mammal] = Mammal
val animalId  = TypeId.of[Animal]
// animalId: TypeId[Animal] = Animal
val fishId    = TypeId.of[Fish]
// fishId: TypeId[Fish] = Fish

// Direct inheritance: Dog extends Mammal
dogId.isSubtypeOf(mammalId)
// res249: Boolean = true

// Transitive inheritance: Dog extends Mammal extends Animal
dogId.isSubtypeOf(animalId)
// res250: Boolean = true

// Not a subtype relationship
dogId.isSubtypeOf(fishId)
// res251: Boolean = false
fishId.isSubtypeOf(mammalId)
// res252: Boolean = false
```

**`isSupertypeOf`** is the reverse—checks if this type is a supertype (parent) of another:

```scala
mammalId.isSupertypeOf(dogId)
// res253: Boolean = true
animalId.isSupertypeOf(dogId)
// res254: Boolean = true
dogId.isSupertypeOf(animalId)
// res255: Boolean = false
```

**`isEquivalentTo`** checks if two types are exactly the same:

```scala
dogId.isEquivalentTo(dogId)
// res256: Boolean = true
dogId.isEquivalentTo(mammalId)
// res257: Boolean = false
animalId.isEquivalentTo(animalId)
// res258: Boolean = true
```

**Variance-aware subtyping** for generic types respects covariance and contravariance. Covariant type constructors like `List[+A]` preserve subtyping relationships:

```scala
val listDogId    = TypeId.of[List[Dog]]
// listDogId: TypeId[List[Dog]] = List[Dog]
val listMammalId = TypeId.of[List[Mammal]]
// listMammalId: TypeId[List[Mammal]] = List[Mammal]
val listAnimalId = TypeId.of[List[Animal]]
// listAnimalId: TypeId[List[Animal]] = List[Animal]

listDogId.isSubtypeOf(listMammalId)
// res259: Boolean = true
listDogId.isSubtypeOf(listAnimalId)
// res260: Boolean = true
listMammalId.isSubtypeOf(listAnimalId)
// res261: Boolean = true
```

These methods are essential for building type-safe registries, implementing generic serializers that dispatch based on type hierarchy, and validating API contracts that require specific type relationships.

## Annotations

**Annotations** are metadata attached to types at compile time. TypeId captures them at runtime, making this metadata available for introspection, validation, and dispatch logic. Annotations enable building smart serializers, validators, and code generators that adjust their behavior based on type-level metadata.

TypeId exposes each annotation as an `Annotation` object containing the annotation's type and its arguments. This is essential for frameworks that need to read compile-time metadata (like JPA, validation libraries, or custom serialization frameworks) but want to remain generic and support multiple annotation schemes:

```scala

@transient
case class ImportantData(id: Int, payload: String)

case class Plain(x: Int)
```

Inspect annotations on a type:

```scala
val importantId = TypeId.of[ImportantData]
// importantId: TypeId[ImportantData] = ImportantData
importantId.annotations
// res263: List[Annotation] = List(
//   Annotation(typeId = transient, args = List())
// )
importantId.annotations.map(_.name)
// res264: List[String] = List("transient")

TypeId.of[Plain].annotations
// res265: List[Annotation] = List()
```

Annotations can have arguments and parameters. Create a custom annotation to see how arguments are captured:

```scala

// Custom annotation with parameters
case class ApiEndpoint(version: Int, deprecated: Boolean = false) extends scala.annotation.StaticAnnotation

@ApiEndpoint(version = 2, deprecated = true)
case class UserV2(id: String, name: String)
```

Derive the TypeId and inspect annotation arguments:

```scala
val userV2Id = TypeId.of[UserV2]
// userV2Id: TypeId[UserV2] = UserV2
userV2Id.annotations
// res267: List[Annotation] = List(
//   Annotation(
//     typeId = ApiEndpoint,
//     args = List(
//       Named(name = "version", value = Const(2)),
//       Named(name = "deprecated", value = Const(true))
//     )
//   )
// )
userV2Id.annotations.head.args
// res268: List[AnnotationArg] = List(
//   Named(name = "version", value = Const(2)),
//   Named(name = "deprecated", value = Const(true))
// )
```

Annotations are represented internally as instances of the `Annotation` data class. The `Annotation` contains the annotation's `TypeId` and a list of `AnnotationArg` values representing the arguments:

| Type                                           | Description                                        |
|------------------------------------------------|----------------------------------------------------|
| `Annotation(typeId, args)`                     | An annotation instance with its type and arguments |
| `AnnotationArg.Const(value)`                   | A constant value argument                          |
| `AnnotationArg.Named(name, value)`             | A named parameter                                  |
| `AnnotationArg.ArrayArg(values)`               | An array of arguments                              |
| `AnnotationArg.Nested(annotation)`             | A nested annotation                                |
| `AnnotationArg.ClassOf(tpe)`                   | A `classOf[T]` argument                            |
| `AnnotationArg.EnumValue(enumType, valueName)` | An enum constant                                   |

**Use cases:** Annotations are commonly used to drive serialization strategies, enforce validation rules, mark types for code generation, configure persistence metadata, or enable framework-specific behavior without requiring explicit configuration objects.

## Namespaces and Owners

Every type has an `owner` — the hierarchical path that tells you where the type is defined (its package, enclosing object, or enclosing type). Owner is essential when you need to identify types by their origin, filter schemas by namespace, or build type-indexed registries that respect module boundaries.

When building cross-module systems (middleware, gateways, plugin registries, code generators), you often need to distinguish between types from different packages or modules. For example, you might want to:
- Apply different serialization strategies to types from `com.internal.domain` vs `com.external.api`
- Build a type registry keyed by both type name *and* origin package (to handle name collisions across modules)
- Validate that a deserialized type comes from a trusted package

Owner gives you the tools to make these decisions at runtime.

### Inspecting Owners

When you derive a TypeId, the `owner` property captures where the type is defined:

```scala

case class MyType(x: Int)
```

```scala
val myId = TypeId.of[MyType]
// myId: TypeId[MyType] = MyType
myId.owner
// res270: Owner = Owner(
//   List(Package("repl"), Term("MdocSession"), Type("MdocApp269"))
// )
myId.owner.asString
// res271: String = "repl.MdocSession.MdocApp269"
myId.fullName
// res272: String = "repl.MdocSession.MdocApp269.MyType"
```

Owner provides methods to inspect and navigate the hierarchy:

```scala
myId.owner.parent
// res273: Owner = Owner(List(Package("repl"), Term("MdocSession")))
myId.owner.lastName
// res274: String = "MdocApp269"
myId.owner.isRoot
// res275: Boolean = false
```

### Owner Structure: Packages, Terms, and Types

An Owner is a chain of segments: packages, terms (objects/values), and types. For a type defined as:

```scala
package com.example
object Outer {
  class Inner
}
```

The owner of `Inner` has three segments: `com`, `example` (packages), and `Outer` (term):

```scala

object ExampleModule {
  case class Config(timeout: Int)
}
```

```scala
val configId = TypeId.of[ExampleModule.Config]
// configId: TypeId[Config] = Config
configId.owner.asString
// res277: String = "repl.MdocSession.MdocApp276.ExampleModule"
```

### TermPath

`TermPath` represents paths to term values and is used in TypeRepr expressions for singleton types (like `obj.field.type`). Singleton type information exists at compile time but is **erased at runtime** by the JVM — both `TypeId.of[HttpStatus.OK.type]` and `TypeId.of[HttpStatus.NotFound.type]` resolve to the same underlying `Int` TypeId. TermPath is useful in type representation structures and code generators that need to capture the compile-time singleton distinction for metaprogramming.

When the macro encounters a singleton type (a `TermRef` in Scala's reflection API), it recursively walks the qualifier chain to build the term path and stores it as `TypeRepr.Singleton(path)` for use in type expressions.

Derive TypeIds for singleton values to see them resolve to their underlying type:

```scala

object HttpStatus {
  val OK = 200
  val NotFound = 404
}
```

```scala
val okSingletonId = TypeId.of[HttpStatus.OK.type]
// okSingletonId: TypeId[OK] = Int
okSingletonId.name
// res279: String = "Int"

val notFoundSingletonId = TypeId.of[HttpStatus.NotFound.type]
// notFoundSingletonId: TypeId[NotFound] = Int
notFoundSingletonId.name
// res280: String = "Int"

okSingletonId == notFoundSingletonId
// res281: Boolean = true
```

**When to use TermPath:** In TypeRepr expressions and code generators that need to represent the compile-time path to a value. While singleton types are erased at runtime, TermPath captures this distinction for reflection and metaprogramming scenarios.

## TypeRepr — Type Expressions

`TypeRepr` represents type expressions in the Scala type system. This is fundamentally different from `TypeId`: while `TypeId` identifies a specific type *definition* (like `List` as a class or `Person` as a case class), `TypeRepr` represents how types are *composed and expressed* at runtime — as type arguments, parent types, intersections, unions, functions, and more.

**The Key Distinction:**

A `TypeId` answers the question **"What is this type definition?"** — for example, `TypeId.of[List]` gives you metadata about the `List` class itself. But a `TypeRepr` answers **"How is this type used in context?"** — for example, when you inspect `TypeId.of[List[Int]].typeArgs`, you get a `TypeRepr.Applied(Ref(TypeId.list), List(Ref(TypeId.int)))`, which describes that `List` is applied to the type argument `Int`.

**Practical Examples of the Difference:**

- `TypeId.list` identifies the `List` class definition
- `TypeRepr.Ref(TypeId.list)` is the expression "use List as a type" (standalone)
- `TypeRepr.Applied(TypeId.list, args)` is the expression "List applied to type arguments" (e.g., `List[Int]`)
- `TypeRepr.Function` represents `(A, B) => C` as a first-class type expression (not a method signature)
- `TypeRepr.Union` represents `A | B` without requiring a union type definition to exist

TypeRepr variants like `Intersection`, `Tuple`, `Union`, `TypeLambda`, and `ContextFunction` allow you to represent type expressions that may not have their own `TypeId` definitions — they are *computed expressions* in the type system rather than named definitions.

You encounter `TypeRepr` values when inspecting `typeArgs`, parent types in `defKind`, and alias targets:

```scala

```

When you derive a TypeId for an applied type, the `typeArgs` are `TypeRepr` values representing the type arguments:

```scala
TypeId.of[Int & String].typeArgs
// res283: List[TypeRepr] = List()
TypeId.of[Map[String, Int]].typeArgs
// res284: List[TypeRepr] = List(Ref(String), Ref(Int))
```

Here is a reference of the different `TypeRepr` variants you may encounter when inspecting TypeIds:

| Category     | Variant                                          | Example                                     |
|--------------|--------------------------------------------------|---------------------------------------------|
| **Common**   | `Ref(id)`                                        | `Int`, `String` — reference to a named type |
|              | `ParamRef(param, depth)`                         | `A` — reference to a type parameter         |
|              | `Applied(tycon, args)`                           | `List[Int]` — parameterized type            |
| **Compound** | `Intersection(types)`                            | `A & B` (Scala 3) or `A with B` (Scala 2)   |
|              | `Union(types)`                                   | `A \| B` (Scala 3)                          |
|              | `Tuple(elems)`                                   | `(A, B, C)`                                 |
|              | `Function(params, result)`                       | `(A, B) => C`                               |
|              | `ContextFunction(params, result)`                | `(A, B) ?=> C` (Scala 3)                    |
| **Special**  | `Singleton(path)`                                | `x.type`                                    |
|              | `ThisType(owner)`                                | `this.type`                                 |
|              | `TypeProjection(qualifier, name)`                | `Outer#Inner`                               |
|              | `TypeSelect(qualifier, name)`                    | `qual.Member`                               |
|              | `Structural(parents, members)`                   | `{ def foo: Int }`                          |
| **Advanced** | `TypeLambda(params, body)`                       | `[X] =>> F[X]`                              |
|              | `Wildcard(bounds)`                               | `?`, `? <: Upper`                           |
|              | `ByName(underlying)`                             | `=> A`                                      |
|              | `Repeated(element)`                              | `A*`                                        |
|              | `Annotated(underlying, annotations)`             | `A @anno`                                   |
|              | `Constant.*`                                     | `42`, `"foo"`, `true` (literal types)       |
| **Builtins** | `AnyType`, `NothingType`, `NullType`, `UnitType` | Special types                               |

A Scala 3 named tuple is not a `Tuple` with labelled elements. Its `TypeId` is the type `scala.NamedTuple.NamedTuple` applied to two plain `Tuple`s, one of label constants and one of element types, and every `TupleElement` label is `None`:

```scala
val reading = TypeId.of[(city: String, tempC: Double)]
// reading: TypeId[NamedTuple[Tuple2["city", "tempC"], Tuple2[String, Double]]] = NamedTuple[("city", "tempC"), (String, Double)]
reading.fullName
// res285: String = "scala.NamedTuple.NamedTuple"
reading.typeArgs
// res286: List[TypeRepr] = List(
//   Tuple(
//     List(
//       TupleElement(label = None, tpe = StringConst("city")),
//       TupleElement(label = None, tpe = StringConst("tempC"))
//     )
//   ),
//   Tuple(
//     List(
//       TupleElement(label = None, tpe = Ref(String)),
//       TupleElement(label = None, tpe = Ref(Double))
//     )
//   )
// )
```

The labels are the `Constant.StringConst` elements of the first type argument.

## Erased TypeId

For type-indexed collections where the type parameter doesn't matter, erase it:

```scala

```

```scala
val erased: TypeId.Erased = TypeId.of[Int].erased
// erased: TypeId[Unknown] = Int
erased
// res288: TypeId[Unknown] = Int
```

Erased TypeIds are the key to building type-indexed maps:

```scala
val registry: Map[TypeId.Erased, String] = Map(
  TypeId.of[Int].erased    -> "Integer type",
  TypeId.of[String].erased -> "String type"
)
// registry: Map[Erased, String] = Map(
//   Int -> "Integer type",
//   String -> "String type"
// )

registry.get(TypeId.of[Int].erased)
// res289: Option[String] = Some("Integer type")
registry.get(TypeId.of[Double].erased)
// res290: Option[String] = None
```

## Predefined TypeIds

TypeId provides instances for common types:

**Core Interfaces:** `TypeId.charSequence` (`java.lang`), `comparable` (`java.lang`), `serializable` (`java.io`)

**Primitives:** `TypeId.unit`, `boolean`, `byte`, `short`, `int`, `long`, `float`, `double`, `char`, `string`, `bigInt`, `bigDecimal`

**Collections:** `TypeId.option`, `some`, `none`, `list`, `vector`, `set`, `seq`, `indexedSeq`, `map`, `either`, `array`, `arraySeq`, `chunk`

**java.time:** `TypeId.dayOfWeek`, `duration`, `instant`, `localDate`, `localDateTime`, `localTime`, `month`, `monthDay`, `offsetDateTime`, `offsetTime`, `period`, `year`, `yearMonth`, `zoneId`, `zoneOffset`, `zonedDateTime`

**java.util:** `TypeId.currency`, `uuid`

**Scala 3 only:** `TypeId.iarray` — `IArray[T]`, the immutable array type.

## Integration with Schema

TypeId is central to ZIO Blocks' schema system. Every `Reflect` node carries an associated TypeId:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}
```

You can access the TypeId from a schema's reflection:

```scala
val reflect = Schema[Person].reflect
val typeId = reflect.typeId
typeId.name
typeId.isCaseClass
```

### Schema Transformations

TypeId is automatically attached when transforming schemas. The `transform` method takes an implicit `TypeId[B]` parameter, so the TypeId for the target type is resolved at compile time:

```scala
case class Email(value: String)

object Email {
  implicit val schema: Schema[Email] = Schema[String]
    .transform(Email(_), _.value)
    // TypeId[Email] is resolved implicitly — no extra call needed
}
```

### Schema Derivation

The `Deriver` trait receives a `TypeId` for each node in the schema. Methods like `deriveRecord` and `deriveVariant` include a `typeId: TypeId[A]` parameter alongside fields/cases, bindings, documentation, modifiers, and more. This lets you inspect the type's structure, annotations, and relationships when generating code.

For details on the full `Deriver` API and how to implement custom derivers, see the [Type Class Derivation](./schema/type-class-derivation.md) reference.

## Comparison with Alternatives

TypeId occupies a different niche from the reflection and type-tagging mechanisms in the Scala ecosystem:

| Feature                           | `TypeId` | `ClassTag` | `TypeTag` (Scala 2) | `TypeTest` (Scala 3) | `Mirror` (Scala 3) |
|-----------------------------------|----------|------------|---------------------|----------------------|--------------------|
| Preserves generic type args       | Yes      | No         | Yes                 | No                   | No                 |
| Distinguishes opaque types        | Yes      | No         | No                  | No                   | No                 |
| Available on Scala.js             | Yes      | Partial    | No                  | Yes                  | Yes                |
| Cross-version (2 & 3)             | Yes      | Yes        | Scala 2 only        | Scala 3 only         | Scala 3 only       |
| Pure data (no runtime reflection) | Yes      | No         | No                  | No                   | Yes                |
| Captures annotations              | Yes      | No         | Yes                 | No                   | No                 |
| Captures variance                 | Yes      | No         | Yes                 | No                   | No                 |
| Subtype relationship checks       | Yes      | No         | Yes                 | Yes                  | No                 |

**When to migrate from `ClassTag`:** If you only need `ClassTag` to create arrays of the correct runtime type, keep using it — TypeId does not replace that functionality. If you are using `ClassTag` to identify or dispatch on types, TypeId provides strictly more information (generics, opaque types, annotations) and works identically on JVM and Scala.js.

**When to migrate from `TypeTag` / `WeakTypeTag`:** These are Scala 2-only, depend on `scala-reflect`, and are not available on Scala.js. TypeId captures comparable metadata (full name, type arguments, variance, annotations) as a pure data structure without runtime reflection, and works across Scala 2, Scala 3, JVM, and Scala.js.

**When to migrate from `TypeTest`:** `TypeTest` is a Scala 3 mechanism for safe pattern matching on types. It answers "is this value an instance of T?" but does not expose type structure, annotations, or generic arguments. Use TypeId when you need to inspect or serialize type metadata, not just test membership.

**When to migrate from `Mirror`:** `Mirror` provides structural information about products and sums for derivation in Scala 3. TypeId complements `Mirror` by adding namespace information (owner/package), annotations, opaque type support, and cross-version compatibility. In ZIO Blocks, the schema derivation system uses TypeId rather than `Mirror`.

## Running the Examples

All code from this guide is available as runnable examples in the `schema-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

### Basic Usage

Demonstrates deriving TypeIds for case classes, accessing their properties (name, fullName, owner, arity), using predefined TypeIds for built-in types, and implicit derivation:

```scala title="schema-examples/src/main/scala/typeid/TypeIdBasicExample.scala" 
package typeid

/**
 * TypeId Basic Example
 *
 * Demonstrates deriving TypeIds for case classes, accessing their properties,
 * and using predefined TypeIds for built-in types.
 *
 * Run with: sbt "schema-examples/runMain typeid.TypeIdBasicExample"
 */

object TypeIdBasicExample extends App {

  println("═══════════════════════════════════════════════════════════════")
  println("TypeId Basic Example")
  println("═══════════════════════════════════════════════════════════════\n")

  // Derive TypeId for a case class
  case class Person(name: String, age: Int)

  // Simple derivation
  val personId = TypeId.of[Person]

  println("--- Deriving TypeId for Person ---\n")

  // Accessing name and fullName
  println("TypeId.of[Person].name")
  show(personId.name)

  // Accessing fully qualified name
  println("TypeId.of[Person].fullName")
  show(personId.fullName)

  // Checking if it's a case class
  println("TypeId.of[Person].isCaseClass")
  show(personId.isCaseClass)

  // Accessing the owner (package/enclosing type)
  println("TypeId.of[Person].owner")
  show(personId.owner)

  println("\n--- Type Classification ---\n")

  // Check the kind (arity)
  println("TypeId.of[Person].arity")
  show(personId.arity)

  // Check it's a proper type
  println("TypeId.of[Person].isProperType")
  show(personId.isProperType)

  // Check it's not a type constructor
  println("TypeId.of[Person].isTypeConstructor")
  show(personId.isTypeConstructor)

  println("\n--- Predefined TypeIds ---\n")

  // Use predefined TypeIds for built-in types
  println("TypeId.int.fullName")
  show(TypeId.int.fullName)

  println("TypeId.string.fullName")
  show(TypeId.string.fullName)

  println("TypeId.list.name")
  show(TypeId.list.name)

  // List is a type constructor (arity = 1)
  println("TypeId.list.arity")
  show(TypeId.list.arity)

  println("TypeId.map.arity")
  show(TypeId.map.arity)

  println("\n--- Classification Predicates ---\n")

  // Check various classification predicates
  println("TypeId.option.isTypeConstructor")
  show(TypeId.option.isTypeConstructor)

  println("TypeId.either.arity")
  show(TypeId.either.arity)

  println("\n--- Implicit Derivation ---\n")

  // Implicit derivation
  val personIdImplicit: TypeId[Person] = implicitly[TypeId[Person]]

  println("implicitly[TypeId[Person]].name")
  show(personIdImplicit.name)

  println("\n═══════════════════════════════════════════════════════════════")

  show(
    TypeId.of[scala.util.Either].isSum
  )

  show(
    TypeId.of[Option[String]].isOption
  )
  show(
    TypeId.of[Either[String, Int]].isEither
  )

  show(
    TypeId.of[Option[String]].isSum
  )

  show(
    TypeId.of[Either[String, Int]].isSum
  )

  show(
    TypeId.of[(Int, String)].isProduct
  )

  show(
    TypeId.of[Tuple2[Int, String]].isProduct
  )

  show(
    TypeId.of[Product2[Int, String]].isProduct
  )

  show(
    TypeId.of[(Int, String, Boolean)].isTuple
  )

  sealed trait Animal

  sealed trait Mammal extends Animal

  case class Dog(name: String) extends Mammal

  case class Cat(name: String) extends Mammal

  case class Fish(species: String) extends Animal

  val dogId    = TypeId.of[Dog]
  val mammalId = TypeId.of[Mammal]
  val animalId = TypeId.of[Animal]
  val fishId   = TypeId.of[Fish]

  // Direct inheritance: Dog extends Mammal
  dogId.isSubtypeOf(mammalId)

  // Transitive inheritance: Dog extends Mammal extends Animal
  dogId.isSubtypeOf(animalId)

  // Not a subtype relationship
  dogId.isSubtypeOf(fishId)
  fishId.isSubtypeOf(mammalId)

  TypeId.of[List[Dog]].isSubtypeOf(TypeId.of[List[Mammal]])
  TypeId.of[List[Dog]].isSubtypeOf(TypeId.of[List[Animal]])

  show(
    TypeId.of[Mammal => String].isSupertypeOf(TypeId.of[Dog => String])
  )

  // A generic parent type with concrete type arguments
  case class StringList() extends scala.collection.mutable.ListBuffer[String]

  // A type with multiple generic parents
  case class Entry[K, V](key: K, value: V) extends scala.collection.Map[K, V] {
    def iterator = Iterator((key, value))

    def get(k: K) = if (k == key) Some(value) else None

    override def -(key: K): collection.Map[K, V] = ???

    override def -(key1: K, key2: K, keys: K*): collection.Map[K, V] = ???
  }

  val stringListId = TypeId.of[StringList]
  // StringList extends ListBuffer[String] - the type argument is captured
  show(
    stringListId.parents
  )

  val entryId = TypeId.of[Entry[String, Int]]
  // Entry[String, Int] extends Map[String, Int] - type arguments are preserved

  show(
    entryId.parents
  )

  // A trait can extend multiple traits
  trait Swimmer {
    def swim(): Unit = ()
  }

  trait Flyer {
    def fly(): Unit = ()
  }

  trait Duck extends Swimmer with Flyer

  // A case class can extend a trait
  case class MallardDuck() extends Duck

  show(
    TypeId.of[MallardDuck].parents
  )

  def isPrimitive(id: TypeId[?]): Boolean =
    id.classTag != scala.reflect.ClassTag.AnyRef

  show {
    isPrimitive(TypeId.of[Int])
    isPrimitive(TypeId.of[Double])
    isPrimitive(TypeId.of[Boolean])
    isPrimitive(TypeId.of[String])
    isPrimitive(TypeId.of[List[Int]])
  }

  def makeStorage(size: Int, id: TypeId[?]): Array[?] =
    id.classTag.newArray(size)

  show {
    makeStorage(100, TypeId.int).getClass.getComponentType
    makeStorage(100, TypeId.double).getClass.getComponentType
    makeStorage(100, TypeId.string).getClass.getComponentType
  }

}
```

([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/typeid/TypeIdBasicExample.scala))

```bash
cd schema-examples && sbt "runMain typeid.TypeIdBasicExample"
```

### Subtype Relationships

Demonstrates subtype checking with `isSubtypeOf`, `isSupertypeOf`, and `isEquivalentTo`, including direct inheritance, transitive inheritance, sealed trait cases, and variance-aware subtyping for applied types like `List[Dog] <: List[Animal]`:

```scala title="schema-examples/src/main/scala/typeid/TypeIdSubtypingExample.scala" 
package typeid

/**
 * TypeId Subtyping Example
 *
 * Demonstrates subtype relationships, checking inheritance, and variance-aware
 * subtyping for applied types.
 *
 * Run with: sbt "schema-examples/runMain typeid.TypeIdSubtypingExample"
 */

object TypeIdSubtypingExample extends App {

  println("═══════════════════════════════════════════════════════════════")
  println("TypeId Subtyping Example")
  println("═══════════════════════════════════════════════════════════════\n")

  // Define a sealed trait hierarchy
  sealed trait Animal
  case class Dog(name: String) extends Animal
  case class Cat(name: String) extends Animal
  case object Bird             extends Animal

  println("--- Direct Inheritance ---\n")

  val dogId    = TypeId.of[Dog]
  val animalId = TypeId.of[Animal]

  // Check if Dog is a subtype of Animal
  println("TypeId.of[Dog].isSubtypeOf(TypeId.of[Animal])")
  show(dogId.isSubtypeOf(animalId))

  // Check if Animal is a supertype of Dog
  println("TypeId.of[Animal].isSupertypeOf(TypeId.of[Dog])")
  show(animalId.isSupertypeOf(dogId))

  // Check equivalence
  println("TypeId.of[Dog].isEquivalentTo(TypeId.of[Dog])")
  show(dogId.isEquivalentTo(dogId))

  println("TypeId.of[Dog].isEquivalentTo(TypeId.of[Animal])")
  show(dogId.isEquivalentTo(animalId))

  println("\n--- Transitive Inheritance ---\n")

  val catId = TypeId.of[Cat]

  // Both Dog and Cat are subtypes of Animal
  println("TypeId.of[Cat].isSubtypeOf(TypeId.of[Animal])")
  show(catId.isSubtypeOf(animalId))

  println("TypeId.of[Dog].isSubtypeOf(TypeId.of[Animal]) && TypeId.of[Cat].isSubtypeOf(TypeId.of[Animal])")
  show(dogId.isSubtypeOf(animalId) && catId.isSubtypeOf(animalId))

  println("\n--- Sealed Trait Cases ---\n")

  val birdId = TypeId.of[Bird.type]

  println("TypeId.of[Bird.type].isSubtypeOf(TypeId.of[Animal])")
  show(birdId.isSubtypeOf(animalId))

  println("\n--- Applied Type Subtyping (Covariance) ---\n")

  // List is covariant in its type parameter
  val listDogId    = TypeId.of[List[Dog]]
  val listAnimalId = TypeId.of[List[Animal]]

  // Due to covariance, List[Dog] is a subtype of List[Animal]
  println("TypeId.of[List[Dog]].isSubtypeOf(TypeId.of[List[Animal]])")
  show(listDogId.isSubtypeOf(listAnimalId))

  println("\n--- Applied Type Non-Subtyping (Invariance) ---\n")

  // Array is invariant in its type parameter (on Scala runtime)
  val arrayDogId    = TypeId.of[Array[Dog]]
  val arrayAnimalId = TypeId.of[Array[Animal]]

  println("TypeId.of[Array[Dog]].isSubtypeOf(TypeId.of[Array[Animal]])")
  show(arrayDogId.isSubtypeOf(arrayAnimalId))

  println("\n--- Type Parameters ---\n")

  // Check the type parameters of a generic type
  println("TypeId.of[List[Dog]].typeArgs")
  show(listDogId.typeArgs)

  println("TypeId.of[Map[String, Animal]].typeArgs")
  show(TypeId.of[Map[String, Animal]].typeArgs)

  println("\n═══════════════════════════════════════════════════════════════")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/typeid/TypeIdSubtypingExample.scala))

```bash
cd schema-examples && sbt "runMain typeid.TypeIdSubtypingExample"
```

### Normalization and Registries

Demonstrates type alias handling, normalization to underlying types, structural equality, and building type-indexed registries using erased TypeIds:

```scala title="schema-examples/src/main/scala/typeid/TypeIdNormalizationExample.scala" 
package typeid

/**
 * TypeId Normalization Example
 *
 * Demonstrates type alias handling, normalization, and type-indexed registries
 * using erased TypeIds.
 *
 * Run with: sbt "schema-examples/runMain typeid.TypeIdNormalizationExample"
 */

object TypeIdNormalizationExample extends App {

  println("═══════════════════════════════════════════════════════════════")
  println("TypeId Normalization Example")
  println("═══════════════════════════════════════════════════════════════\n")

  // Define some type aliases
  type UserId = Long
  type Email  = String
  type Age    = Int

  println("--- Type Aliases ---\n")

  // Create TypeIds for the aliases
  val userIdAlias = TypeId.alias[UserId](
    name = "UserId",
    owner = Owner.Root,
    typeParams = Nil,
    aliased = TypeRepr.Ref(TypeId.long)
  )

  show(userIdAlias.name)

  show(userIdAlias.fullName)

  println("\n--- Normalization ---\n")

  // Normalize the alias to its underlying type
  val normalized = TypeId.normalize(userIdAlias)

  show(normalized.name)

  show(normalized.fullName)

  println("\n--- Equality with Normalization ---\n")

  // Structural equality (not considering aliases)
  val anotherUserIdAlias = TypeId.alias[UserId](
    name = "UserId",
    owner = Owner.Root,
    typeParams = Nil,
    aliased = TypeRepr.Ref(TypeId.long)
  )

  show(userIdAlias == anotherUserIdAlias)

  show(TypeId.normalize(userIdAlias) == TypeId.long)

  println("\n--- Erased TypeIds for Registries ---\n")

  // Erased TypeIds are useful for type-indexed maps
  val intErased: TypeId.Erased     = TypeId.int.erased
  val stringErased: TypeId.Erased  = TypeId.string.erased
  val listIntErased: TypeId.Erased = TypeId.of[List[Int]].erased

  show(intErased)

  show(stringErased)

  show(listIntErased)

  println("\n--- Type Registry Using Erased TypeIds ---\n")

  // Build a type registry
  val registry: Map[TypeId.Erased, String] = Map(
    TypeId.int.erased           -> "Integer type",
    TypeId.string.erased        -> "String type",
    TypeId.long.erased          -> "Long type",
    TypeId.of[List[Int]].erased -> "List of integers"
  )

  show(registry.get(TypeId.int.erased))

  show(registry.get(TypeId.string.erased))

  show(registry.get(TypeId.of[List[Int]].erased))

  println("\n--- Querying the Registry ---\n")

  // Look up types in the registry
  val intType    = TypeId.of[Int].erased
  val doubleType = TypeId.of[Double].erased

  show(registry.get(intType))

  show(registry.get(doubleType))

  println("\n═══════════════════════════════════════════════════════════════")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/typeid/TypeIdNormalizationExample.scala))

```bash
cd schema-examples && sbt "runMain typeid.TypeIdNormalizationExample"
```

### Opaque Types

Demonstrates how TypeId preserves the semantic distinction of opaque types, enabling runtime type safety that pure Scala reflection cannot provide. Shows building type-indexed validator registries keyed by opaque type identity:

```scala title="schema-examples/src/main/scala/typeid/OpaqueTypesExample.scala" 
package typeid

/**
 * Opaque Types Example
 *
 * Demonstrates how TypeId preserves the semantic distinction of opaque types,
 * enabling runtime type safety that pure Scala reflection cannot provide.
 *
 * Opaque types allow you to create distinct types that share the same
 * representation at runtime. TypeId captures this distinction, making it
 * possible to:
 *   - Distinguish between different opaque types wrapping the same base type
 *   - Build type-indexed registries that respect opaque type boundaries
 *   - Implement runtime validators specific to each opaque type
 *
 * Run with: sbt "schema-examples/runMain typeid.OpaqueTypesExample"
 */

object OpaqueTypesExample extends App {

  println("═══════════════════════════════════════════════════════════════")
  println("Opaque Types with TypeId")
  println("═══════════════════════════════════════════════════════════════\n")

  // Define domain-specific opaque types wrapping String
  // In a real application, these would enforce different validation rules
  opaque type UserId       = String
  opaque type Email        = String
  opaque type SessionToken = String

  println("--- Opaque Type Definitions ---\n")
  println("opaque type UserId = String")
  println("opaque type Email = String")
  println("opaque type SessionToken = String\n")

  // Derive TypeIds for each opaque type
  val userIdType       = TypeId.of[UserId]
  val emailType        = TypeId.of[Email]
  val sessionTokenType = TypeId.of[SessionToken]
  val stringType       = TypeId.string

  println("--- TypeId Derivation ---\n")

  println("TypeId.of[UserId].name")
  show(userIdType.name)

  println("TypeId.of[Email].name")
  show(emailType.name)

  println("TypeId.of[SessionToken].name")
  show(sessionTokenType.name)

  println("\n--- Opaque Types vs Base Type ---\n")

  // Key insight: opaque types are distinct from their representation type
  println("TypeId.of[UserId].isEquivalentTo(TypeId.string)")
  show(userIdType.isEquivalentTo(stringType))

  println("TypeId.of[Email].isEquivalentTo(TypeId.string)")
  show(emailType.isEquivalentTo(stringType))

  println("TypeId.of[SessionToken].isEquivalentTo(TypeId.string)")
  show(sessionTokenType.isEquivalentTo(stringType))

  println("\n--- Opaque Types are Distinct from Each Other ---\n")

  println("TypeId.of[UserId].isEquivalentTo(TypeId.of[Email])")
  show(userIdType.isEquivalentTo(emailType))

  println("TypeId.of[Email].isEquivalentTo(TypeId.of[SessionToken])")
  show(emailType.isEquivalentTo(sessionTokenType))

  println("TypeId.of[UserId].isEquivalentTo(TypeId.of[SessionToken])")
  show(userIdType.isEquivalentTo(sessionTokenType))

  println("\n--- Real-World Use Case: Type-Safe Registry ---\n")

  // Define validators for each opaque type
  trait Validator {
    def validate(value: String): Boolean
    def errorMessage: String
  }

  val userIdValidator = new Validator {
    def validate(value: String): Boolean = value.nonEmpty && value.forall(_.isDigit)
    def errorMessage                     = "UserId must be non-empty digits"
  }

  val emailValidator = new Validator {
    def validate(value: String): Boolean = value.contains("@") && value.contains(".")
    def errorMessage                     = "Email must contain @ and ."
  }

  val sessionTokenValidator = new Validator {
    def validate(value: String): Boolean = value.length >= 32
    def errorMessage                     = "SessionToken must be at least 32 characters"
  }

  // Build a type-indexed registry of validators
  // This demonstrates the power of TypeId: we can safely dispatch
  // to different validators based on opaque type identity
  val validatorRegistry: Map[TypeId.Erased, Validator] = Map(
    TypeId.of[UserId].erased       -> userIdValidator,
    TypeId.of[Email].erased        -> emailValidator,
    TypeId.of[SessionToken].erased -> sessionTokenValidator
  )

  println("Built validator registry keyed by opaque type")
  println("Validators can enforce different validation rules per type\n")

  // Demonstrate validation dispatch
  def validateString(value: String, typeId: TypeId[_]): Boolean =
    validatorRegistry
      .get(typeId.erased)
      .map(_.validate(value))
      .getOrElse {
        println(s"No validator found for type: ${typeId.fullName}")
        false
      }

  def getValidationError(typeId: TypeId[_]): String =
    validatorRegistry
      .get(typeId.erased)
      .map(_.errorMessage)
      .getOrElse("Unknown validator")

  println("--- Validation Examples ---\n")

  val testUserId = "12345"
  val testEmail  = "user@example.com"
  val testToken  = "a" * 32

  println(s"Validating UserId: '$testUserId'")
  if (validateString(testUserId, TypeId.of[UserId])) {
    println("✓ Valid UserId\n")
  } else {
    println(s"✗ Invalid: ${getValidationError(TypeId.of[UserId])}\n")
  }

  println(s"Validating Email: '$testEmail'")
  if (validateString(testEmail, TypeId.of[Email])) {
    println("✓ Valid Email\n")
  } else {
    println(s"✗ Invalid: ${getValidationError(TypeId.of[Email])}\n")
  }

  println(s"Validating SessionToken: '$testToken'")
  if (validateString(testToken, TypeId.of[SessionToken])) {
    println("✓ Valid SessionToken\n")
  } else {
    println(s"✗ Invalid: ${getValidationError(TypeId.of[SessionToken])}\n")
  }

  println("--- What Pure Scala Cannot Do ---\n")

  println("With pure Scala reflection:")
  println("- classOf[UserId] == classOf[String]  (erased at runtime)")
  println("- classOf[Email] == classOf[String]   (erased at runtime)")
  println("- You cannot distinguish opaque types from their base type\n")

  println("With TypeId:")
  println("- TypeId.of[UserId] != TypeId.of[String]  (preserved)")
  println("- TypeId.of[Email] != TypeId.of[String]   (preserved)")
  println("- You can build type-safe validators and registries\n")

  println("═══════════════════════════════════════════════════════════════")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/typeid/OpaqueTypesExample.scala))

```bash
cd schema-examples && sbt "runMain typeid.OpaqueTypesExample"
```

---

## Web & HTTP

## Introduction

Building a web service involves several separate concerns: representing requests and responses, describing the API, documenting it, authenticating callers and producing pages. The blocks in this group cover each of these as immutable data types. Most of them describe their subject without committing to a particular server or client, and all nine run on the JVM and in Scala.js.

## The Blocks

The nine blocks are listed below in the order the docs present them.

| Block                                   | Artifact                       | What It Is For                                                                              |
| --------------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------- |
| [MediaType](../media-type.md)           | `zio-blocks-mediatype`         | Type-safe IANA media types with 2,600+ predefined types                                     |
| [HTTP Model](../http-model/index.md)    | `zio-blocks-http-model`        | Pure HTTP data model with URL parsing, headers, cookies, and forms                          |
| [HTTP Model Schema](../http-model/schema.md) | `zio-blocks-http-model-schema` | Schema-based typed access to the HTTP model                                                 |
| [Endpoint](../endpoint/index.md)        | `zio-blocks-endpoint`          | Type-safe HTTP endpoint descriptors with composable codecs and typed auth                   |
| [OpenAPI](../openapi.md)                | `zio-blocks-openapi`           | Type-safe OpenAPI 3.1 specification generation and rendering                                |
| [JWT](../jwt.md)                        | `zio-blocks-jwt`               | Zero-dependency JWT signing and verification with HMAC, RSA, ECDSA and EdDSA support        |
| [HTML](../html.md)                      | `zio-blocks-html`              | Type-safe HTML templating with XSS protection                                               |
| [Datastar](../datastar/index.md)        | `zio-blocks-datastar`          | Typed Datastar attribute and signal DSL, plus the SSE events that patch a live page         |
| [HTMX](../htmx/index.md)                | `zio-blocks-http-htmx`         | Typed HTMX DSL for compile-time-checked HTMX attributes                                     |

## How the Blocks Fit Together

[MediaType](../media-type.md), [HTTP Model](../http-model/index.md) and [JWT](../jwt.md) are the protocol layer. The HTTP model represents requests, responses, headers, URLs and cookies and keeps protocol concerns apart from effects such as sending a message, and [HTTP Model Schema](../http-model/schema.md) adds schema-based typed access to it. `MediaType` supplies the content types those messages carry, and `JWT` signs and verifies tokens without any dependency.

[Endpoint](../endpoint/index.md) and [OpenAPI](../openapi.md) describe an API. An endpoint is a description of routes, parameters, headers, bodies, errors and authentication, written once and independent of any server or client implementation. The OpenAPI block is a data model of an OpenAPI 3.1 document that can be built in Scala and exported as JSON for tools such as Swagger UI.

[HTML](../html.md), [Datastar](../datastar/index.md) and [HTMX](../htmx/index.md) produce pages. `HTML` builds markup with automatic XSS protection. `HTMX` and `Datastar` add typed attributes for the two hypermedia libraries, and `Datastar` also produces the server-sent events that patch a live page. Both are available on Scala 3 only.

## Where to Start

Start with [HTTP Model](../http-model/index.md) to see how messages are represented, then [Endpoint](../endpoint/index.md) to describe an API. Pick [HTML](../html.md) first when the goal is to render pages.