How to Compute, Store and Replay Changes with Diff and Patch

In this guide, we track the changes made to a book catalog: who lowered the price of Zionomicon, from what to what, and what the catalog looked like before each edit. Instead of storing a full copy of every version, we compute a patch, a small value that describes only what changed, with Schema#diff. By the end, you will have an edit log that stores each change as JSON, replays it from a base version, undoes the last edit, and sends changes to other systems as JSON patches.

We take an incremental approach:

Diff two versions of a Book and read the patch
Turn a patch into audit-log lines
Apply a patch, and compose two patches into one
Store a patch as JSON text and load it back
Choose a PatchMode for when the data changed after the patch was made
Keep an edit history, replay it and undo the last edit
Exchange JSON patches with other systems, including the standard RFC 6902 format

The Problem

Many programs need to know what changed between two versions of their data, not just the new version. An audit log answers "who changed this price, and from what?". A client that syncs with a server wants to send only the fields that changed. An editor needs undo, and a collaboration feature stores a stream of small edits and replays them.

Without help, we compare the two values by hand, one field at a time. Every new field needs another line, lists and maps need their own comparison code, and nothing turns the result back into an update we can apply later.

For example, an audit function for a Book often looks like this:

final case class Book(id: String, title: String, price: BigDecimal, tags: List[String], edition: Int)

def changes(before: Book, after: Book): List[String] = {
  val title   = if (before.title != after.title) List(s"title: ${before.title} -> ${after.title}") else Nil
  val price   = if (before.price != after.price) List(s"price: ${before.price} -> ${after.price}") else Nil
  val added   = after.tags.filterNot(before.tags.contains).map(t => s"tag added: $t")
  val removed = before.tags.filterNot(after.tags.contains).map(t => s"tag removed: $t")
  val edition = if (before.edition != after.edition) List(s"edition: ${before.edition} -> ${after.edition}") else Nil
  title ++ price ++ added ++ removed ++ edition
}

The function forgets new fields silently, describes changes only as text, and gives us nothing to apply: to replay or undo the edit, we would write a second function that reads these strings back.

In this guide, we let the schema do this work. Schema#diff compares any two values of a type that has a Schema and returns a Patch, which we can print, store, send and apply.

Prerequisites

This guide builds on Your First Schema: you should know how to derive a Schema for a case class. Add the schema module to your build.sbt (see How to Add ZIO Blocks to Your Build for the full setup):

// Scala 2.13 and Scala 3.3 to 3.8
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.56"

// Scala 3.9 and newer
libraryDependencies += "dev.zio" %% "zio-blocks-next-schema" % "0.0.56"
scalacOptions += "-experimental"

Everything in this guide works the same on Scala 2.13 and Scala 3. We use these imports throughout:

import zio.blocks.chunk.Chunk
import zio.blocks.schema._
import zio.blocks.schema.json._
import zio.blocks.schema.patch._

The Book Model

Our catalog holds books. Each Book has an id, a title, a price, a list of tags and an edition number, which covers the field kinds a diff treats differently: strings, numbers and lists. The Schema is all Schema#diff needs:

final case class Book(id: String, title: String, price: BigDecimal, tags: List[String], edition: Int)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

val zionomicon = Book("zionomicon", "Zionomicon", BigDecimal("44.99"), List("zio", "scala"), 1)

Diff Two Versions of a Book

Suppose an editor publishes the second edition of Zionomicon: the price drops by five dollars, the book gets an effects tag and the edition goes up. We pass the old and the new version to Schema#diff:

val secondEdition = zionomicon.copy(price = BigDecimal("39.99"), tags = List("zio", "scala", "effects"), edition = 2)

val pricePatch: Patch[Book] = Book.schema.diff(zionomicon, secondEdition)

A Patch prints as a readable list of operations, one per changed field:

println(pricePatch)
// DynamicPatch {
//   .price -= 5.00
//   .tags:
//     + "effects"
//   .edition += 1
// }

The patch lists only the three fields that changed. It does not store the new values: numbers change by a delta (-= 5.00, += 1), and the list gets an append operation (+ "effects"). Unchanged fields such as title do not appear at all.

A text field changes by an edit at a position, not by replacing the whole string. Adding a subtitle inserts text at index 10:

println(Book.schema.diff(secondEdition, secondEdition.copy(title = "Zionomicon (2nd Edition)")))
// DynamicPatch {
//   .title:
//     + [10: " (2nd Edition)"]
// }

When nothing changed, the diff is empty, which Patch#isEmpty reports:

Book.schema.diff(zionomicon, zionomicon).isEmpty
// res3: Boolean = true

Turn a Patch into Audit Lines

An audit log wants lines like "price: 44.99 -> 39.99", with the old and the new value. The patch knows where each change happened: Patch#dynamicPatch exposes the untyped DynamicPatch, whose operations (DynamicPatch#ops) each carry a path, a DynamicOptic that prints as .price for the price field, and an operation. We look up each path in the old and the new value, converted to a DynamicValue with Schema#toDynamicValue, and print both sides as JSON:

def auditLines[A](schema: Schema[A], before: A, after: A, patch: Patch[A]): List[String] = {
  val old = schema.toDynamicValue(before)
  val now = schema.toDynamicValue(after)
  def show(value: DynamicValue, path: DynamicOptic): String =
    value.get(path).one.fold(_ => "(none)", v => Json.fromDynamicValue(v).print)
  patch.dynamicPatch.ops.toList.map(op => s"${op.path}: ${show(old, op.path)} -> ${show(now, op.path)}")
}

The function works for any type with a Schema, so new fields show up in the audit without changes to it:

auditLines(Book.schema, zionomicon, secondEdition, pricePatch).foreach(println)
// .price: 44.99 -> 39.99
// .tags: ["zio","scala"] -> ["zio","scala","effects"]
// .edition: 1 -> 2

Apply a Patch

Schema#patch applies a patch to a value and returns Either[SchemaError, A]. Applying the patch to the version it was computed from gives back the new version, which is the basic law of diff and patch:

Book.schema.patch(zionomicon, pricePatch)
// res5: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon",
//     price = 39.99,
//     tags = List("zio", "scala", "effects"),
//     edition = 2
//   )
// )

Book.schema.patch(zionomicon, pricePatch) == Right(secondEdition)
// res6: Boolean = true

Schema#patch always uses PatchMode.Strict. To choose the mode, call Patch#apply(value, mode) instead; we look at the modes in Choose a PatchMode When the Data Changed.

warning

Patch#apply(value), with no mode, returns a plain Book, not an Either. It applies the patch in PatchMode.Lenient, which skips the operations that fail, and returns the original value if the result cannot be built. A failed patch then looks the same as a patch that did nothing. Prefer Patch#apply(value, mode) or Schema#patch, which report the error.

Compose Patches into One

Two patches compose with Patch#++: the result applies the left patch first, then the right one. Here, a second edit adds the subtitle:

val withSubtitle  = secondEdition.copy(title = "Zionomicon (2nd Edition)")
val subtitlePatch = Book.schema.diff(secondEdition, withSubtitle)

Applying the composed patch to the first edition gives the book after both edits:

(pricePatch ++ subtitlePatch)(zionomicon, PatchMode.Strict)
// res7: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon (2nd Edition)",
//     price = 39.99,
//     tags = List("zio", "scala", "effects"),
//     edition = 2
//   )
// )

Store a Patch as JSON and Load It Back

To keep a patch in a database, write it to a file or send it over the network, we store its DynamicPatch. DynamicPatch has its own Schema, so it has a JSON codec like any other type:

val patchCodec = Schema[DynamicPatch].jsonCodec

val storedText: String = patchCodec.encodeToString(pricePatch.dynamicPatch)

The JSON names each operation and its path:

println(storedText)
// {"ops":[{"path":{"nodes":[{"Field":{"name":"price"}}]},"operation":{"PrimitiveDelta":{"op":{"BigDecimalDelta":{"delta":-5.00}}}}},{"path":{"nodes":[{"Field":{"name":"tags"}}]},"operation":{"SequenceEdit":{"ops":[{"Append":{"values":["effects"]}}]}}},{"path":{"nodes":[{"Field":{"name":"edition"}}]},"operation":{"PrimitiveDelta":{"op":{"IntDelta":{"delta":1}}}}}]}

Decoding the text gives back an equal DynamicPatch:

patchCodec.decode(storedText).map(_ == pricePatch.dynamicPatch)
// res9: Either[SchemaError, Boolean] = Right(true)

A decoded DynamicPatch is not a Patch[Book], and Patch has no public way to wrap one again. Instead, we apply it to the book's DynamicValue and convert the result back with Schema#fromDynamicValue. A small helper does both steps:

def applyStored[A](schema: Schema[A], value: A, patch: DynamicPatch, mode: PatchMode = PatchMode.Strict): Either[SchemaError, A] =
  patch(schema.toDynamicValue(value), mode).flatMap(schema.fromDynamicValue)

Loading the stored text and applying it to the first edition gives the second edition:

patchCodec.decode(storedText).flatMap(loaded => applyStored(Book.schema, zionomicon, loaded))
// res10: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon",
//     price = 39.99,
//     tags = List("zio", "scala", "effects"),
//     edition = 2
//   )
// )

The JSON refers to fields by name, so a stored patch keeps working when we add fields to Book, as long as the fields it names still exist.

Choose a PatchMode When the Data Changed

A stored or sent patch is often applied to a value that changed after the patch was made: two editors work on the same catalog, or a client syncs a change made while offline. PatchMode decides what happens when an operation no longer fits:

Mode	When an operation does not fit
`Strict`	Stops and returns `Left` with the first error
`Lenient`	Skips that operation and applies the rest
`Clobber`	Like `Lenient`, but overwrites a map key that already exists and clamps list inserts and deletes to the list's end

To see the modes at work, we keep the books in a Catalog, a map from book id to Book:

final case class Catalog(books: Map[String, Book])

object Catalog {
  implicit val schema: Schema[Catalog] = Schema.derived
}

val programmingInScala = Book("programming-in-scala", "Programming in Scala", BigDecimal("59.99"), List("scala"), 5)
val fpInScala          = Book("fp-in-scala", "Functional Programming in Scala", BigDecimal("49.99"), List("scala", "fp"), 2)

val monday  = Catalog(Map(zionomicon.id -> zionomicon, programmingInScala.id -> programmingInScala))
val tuesday = Catalog(Map(zionomicon.id -> secondEdition, fpInScala.id -> fpInScala))

val catalogPatch = Catalog.schema.diff(monday, tuesday)

The patch modifies the Zionomicon entry, adds Functional Programming in Scala and removes Programming in Scala. The added book's fields print without indentation, a known bug in 0.0.56:

println(catalogPatch)
// DynamicPatch {
//   .books:
//     ~ {"zionomicon"}:
//       .price -= 5.00
//       .tags:
//         + "effects"
//       .edition += 1
//     + {"fp-in-scala": {
//   id: "fp-in-scala",
//   title: "Functional Programming in Scala",
//   price: 49.99,
//   tags: [
//     "scala",
//     "fp"
//   ],
//   edition: 2
// }}
//     - {"programming-in-scala"}
// }

A helper applies the patch in all three modes, so we can compare them:

def applyInEveryMode(target: Catalog): Unit =
  List(PatchMode.Strict, PatchMode.Lenient, PatchMode.Clobber).foreach { mode =>
    println(s"$mode: ${catalogPatch(target, mode).map(_.books.values.map(b => s"${b.id} ${b.price}").toList.sorted)}")
  }

Now suppose another editor added Functional Programming in Scala first, at a price of 45.00. Strict refuses the patch, Lenient keeps the other editor's book, and Clobber overwrites it with ours:

applyInEveryMode(monday.copy(books = monday.books + (fpInScala.id -> fpInScala.copy(price = BigDecimal("45.00")))))
// Strict: Left(zio.blocks.schema.SchemaError: Key already exists in map at: .books)
// Lenient: Right(List(fp-in-scala 45.00, zionomicon 39.99))
// Clobber: Right(List(fp-in-scala 49.99, zionomicon 39.99))

If the other editor removed Programming in Scala already, Strict fails because the key is gone, and the other two modes skip the removal:

applyInEveryMode(monday.copy(books = monday.books - programmingInScala.id))
// Strict: Left(zio.blocks.schema.SchemaError: Key not found in map at: .books)
// Lenient: Right(List(fp-in-scala 49.99, zionomicon 39.99))
// Clobber: Right(List(fp-in-scala 49.99, zionomicon 39.99))

Use Strict when a conflict must reach a person or a retry loop, Lenient when the latest writer should not undo other people's additions, and Clobber when our patch should win.

Strict Mode Does Not Detect Every Change

Strict only checks that each operation fits the value: a map key exists, an index is inside the list, an insert position is inside the string. It does not check that the value still holds what it held when the patch was made. Numeric deltas, string edits and list operations by index all apply to whatever value is there now.

For example, the Zionomicon price patch says "minus 5.00", not "set to 39.99". If someone set the price to 42.00 in the meantime, the patch still applies, and the result is 37.00:

pricePatch(zionomicon.copy(price = BigDecimal("42.00")), PatchMode.Strict)
// res14: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon",
//     price = 37.00,
//     tags = List("zio", "scala", "effects"),
//     edition = 2
//   )
// )

Lists are riskier, because a diff removes list elements by index. This patch removes the scala tag, which sits at index 1:

val tagged         = zionomicon.copy(tags = List("zio", "scala", "effects"))
val removeScalaTag = Book.schema.diff(tagged, tagged.copy(tags = List("zio", "effects")))

The patch stores only the position:

println(removeScalaTag)
// DynamicPatch {
//   .tags:
//     - [1]
// }

warning

If someone already removed the scala tag, index 1 now holds effects, and Strict removes it without an error. The book loses a tag that nobody asked to remove:

removeScalaTag(tagged.copy(tags = List("zio", "effects")), PatchMode.Strict)
// res16: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon",
//     price = 44.99,
//     tags = List("zio"),
//     edition = 1
//   )
// )

When the result must be exact, check that the value is still the one the patch was computed from before applying it. If it changed, reject the edit or compute a fresh diff against the current value. Comparing the whole value works for any type; a version number stored next to the value does the same job with less data to send:

def applyIfUnchanged(current: Book, base: Book, patch: Patch[Book]): Either[String, Book] =
  if (current != base) Left(s"${current.id} changed since this patch was made; diff again against the current version")
  else patch(current, PatchMode.Strict).left.map(_.message)

The same patch is now rejected for the changed book, and still applies to the original:

applyIfUnchanged(tagged.copy(tags = List("zio", "effects")), tagged, removeScalaTag)
// res17: Either[String, Book] = Left(
//   "zionomicon changed since this patch was made; diff again against the current version"
// )

applyIfUnchanged(tagged, tagged, removeScalaTag)
// res18: Either[String, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon",
//     price = 44.99,
//     tags = List("zio", "effects"),
//     edition = 1
//   )
// )

Keep an Edit History, Replay It and Undo

An edit history is a list of patches plus whatever we want to know about each one. We record two patches per edit: the forward patch from the old to the new version, and the backward patch from the new to the old version, which undoes the edit. Both are DynamicPatch values, so the case class gets a Schema and a JSON codec like any other:

final case class Edit(author: String, forward: DynamicPatch, backward: DynamicPatch)

object Edit {
  implicit val schema: Schema[Edit] = Schema.derived
}

def record(author: String, before: Book, after: Book): Edit =
  Edit(author, Book.schema.diff(before, after).dynamicPatch, Book.schema.diff(after, before).dynamicPatch)

val history: List[Edit] = List(
  record("alice", zionomicon, secondEdition),
  record("bob", secondEdition, withSubtitle)
)

The whole history is a List[Edit], so one codec stores it as JSON text, in the same shape as the stored patch above:

val historyCodec = Schema[List[Edit]].jsonCodec

val historyText: String = historyCodec.encodeToString(history)

Decoding the text gives back the same history:

historyCodec.decode(historyText) == Right(history)
// res19: Boolean = true

Replaying the history is a fold over the edits, starting from the base version. Each step applies one forward patch with the applyStored helper, and the first failure stops the replay:

def replay(base: Book, edits: List[Edit]): Either[SchemaError, Book] =
  edits.foldLeft[Either[SchemaError, Book]](Right(base))((book, edit) => book.flatMap(applyStored(Book.schema, _, edit.forward)))

Replaying every edit gives the latest version, and replaying a prefix gives any earlier version:

replay(zionomicon, history)
// res20: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon (2nd Edition)",
//     price = 39.99,
//     tags = List("zio", "scala", "effects"),
//     edition = 2
//   )
// )

replay(zionomicon, history.take(1))
// res21: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon",
//     price = 39.99,
//     tags = List("zio", "scala", "effects"),
//     edition = 2
//   )
// )

To undo the last edit, we apply its backward patch to the current version:

applyStored(Book.schema, withSubtitle, history.last.backward)
// res22: Either[SchemaError, Book] = Right(
//   Book(
//     id = "zionomicon",
//     title = "Zionomicon",
//     price = 39.99,
//     tags = List("zio", "scala", "effects"),
//     edition = 2
//   )
// )

A history only replays from the base it was recorded against. Started from a different book, Bob's insert at index 10 no longer fits a three-letter title, and the replay stops with an error:

replay(zionomicon.copy(title = "ZIO"), history)
// res23: Either[SchemaError, Book] = Left(
//   SchemaError(
//     List(
//       ExpectationMismatch(
//         source = DynamicOptic(ArraySeq(Field("title"))),
//         expectation = "String insert index 10 out of bounds for string of length 3"
//       )
//     )
//   )
// )

To store one patch instead of many, for example to squash old history, DynamicPatch#++ composes the forward patches in order:

println(history.map(_.forward).foldLeft(DynamicPatch.empty)(_ ++ _))
// DynamicPatch {
//   .price -= 5.00
//   .tags:
//     + "effects"
//   .edition += 1
//   .title:
//     + [10: " (2nd Edition)"]
// }

Exchange JSON Patches with Other Systems

When the data is JSON rather than a Scala type, for example a document from another service, Json#diff compares two Json values and returns a JsonPatch, and Json#patch applies one. Json.from turns a Book into Json through its schema:

val mondayJson  = Json.from(zionomicon)
val tuesdayJson = Json.from(withSubtitle)

val jsonPatch: JsonPatch = mondayJson.diff(tuesdayJson)

The JsonPatch holds the same kinds of operations as a typed patch, nested by object key:

println(jsonPatch)
// JsonPatch {
//   root:
//     ~ {"title"}:
//       root:
//         + [10: " (2nd Edition)"]
//     ~ {"price"}:
//       root -= 5.00
//     ~ {"tags"}:
//       root:
//         + "effects"
//     ~ {"edition"}:
//       root += 1
// }

mondayJson.patch(jsonPatch).map(_.print)
// res26: Either[SchemaError, String] = Right(
//   "{\"id\":\"zionomicon\",\"title\":\"Zionomicon (2nd Edition)\",\"price\":39.99,\"tags\":[\"zio\",\"scala\",\"effects\"],\"edition\":2}"
// )

A typed patch's DynamicPatch converts to a JsonPatch with JsonPatch.fromDynamicPatch, which returns Either[SchemaError, JsonPatch] because temporal deltas and non-string map keys have no JSON form. JsonPatch has a Schema, so another ZIO Blocks service can receive it with JsonPatch.schema.jsonCodec.

The ZIO Blocks JsonPatch is its own format, not the RFC 6902 "JSON Patch" standard that HTTP PATCH APIs and many JSON libraries understand. RFC 6902 has no number deltas or string edits: it sends operations such as add, remove and replace (it also defines move, copy and test) with JSON Pointer paths such as /tags/1. The JsonPatch reference lists every operation and compares the two formats.

Convert a JsonPatch to RFC 6902

To send a change to a system that expects RFC 6902, we convert the JsonPatch ourselves. The converter below is our own code, not a library API. It walks the operations in order and keeps the current JSON value next to them, applying each operation with JsonPatch.root, so that every index and every computed value refers to the document as it is at that point:

object Rfc6902 {
  def fromJsonPatch(source: Json, patch: JsonPatch): Either[SchemaError, Json] =
    patchOps(source, "", patch).map(ops => Json.Array(Chunk.from(ops)))

  private def patchOps(value: Json, path: String, patch: JsonPatch): Either[SchemaError, Vector[Json]] =
    steps(value, patch.ops.toVector)(op => JsonPatch(Chunk.single(op))) { (current, op) =>
      if (op.path.nodes.nonEmpty) Left(SchemaError(s"only root paths are supported, got ${op.path}"))
      else opOps(current, path, op.operation)
    }

  private def opOps(value: Json, path: String, op: JsonPatch.Op): Either[SchemaError, Vector[Json]] = op match {
    case JsonPatch.Op.Set(v)            => Right(Vector(rfc("replace", path, Some(v))))
    case JsonPatch.Op.PrimitiveDelta(_) => JsonPatch.root(op)(value).map(v => Vector(rfc("replace", path, Some(v))))
    case JsonPatch.Op.Nested(p)         => patchOps(value, path, p)
    case JsonPatch.Op.ArrayEdit(edits)  =>
      steps(value, edits.toVector)(e => JsonPatch.root(JsonPatch.Op.ArrayEdit(Chunk.single(e)))) {
        case (_, JsonPatch.ArrayOp.Insert(i, vs)) =>
          Right(vs.toVector.zipWithIndex.map { case (v, k) => rfc("add", s"$path/${i + k}", Some(v)) })
        case (_, JsonPatch.ArrayOp.Append(vs))     => Right(vs.toVector.map(v => rfc("add", s"$path/-", Some(v))))
        case (_, JsonPatch.ArrayOp.Delete(i, n))   => Right(Vector.fill(n)(rfc("remove", s"$path/$i", None)))
        case (cur, JsonPatch.ArrayOp.Modify(i, o)) =>
          cur.elements.lift(i).toRight(SchemaError(s"no index $i at $path")).flatMap(opOps(_, s"$path/$i", o))
      }
    case JsonPatch.Op.ObjectEdit(edits) =>
      steps(value, edits.toVector)(e => JsonPatch.root(JsonPatch.Op.ObjectEdit(Chunk.single(e)))) {
        case (_, JsonPatch.ObjectOp.Add(k, v))      => Right(Vector(rfc("add", s"$path/${escape(k)}", Some(v))))
        case (_, JsonPatch.ObjectOp.Remove(k))      => Right(Vector(rfc("remove", s"$path/${escape(k)}", None)))
        case (cur, JsonPatch.ObjectOp.Modify(k, p)) =>
          cur.fields.collectFirst { case (`k`, v) => v }.toRight(SchemaError(s"no key $k at $path"))
            .flatMap(patchOps(_, s"$path/${escape(k)}", p))
      }
  }

  // Emits the RFC 6902 operations for each edit, then applies the edit, so later edits see the updated value.
  private def steps[E](value: Json, edits: Vector[E])(asPatch: E => JsonPatch)(
    emit: (Json, E) => Either[SchemaError, Vector[Json]]
  ): Either[SchemaError, Vector[Json]] =
    edits
      .foldLeft[Either[SchemaError, (Json, Vector[Json])]](Right((value, Vector.empty))) { (acc, e) =>
        acc.flatMap { case (current, out) =>
          for {
            ops  <- emit(current, e)
            next <- asPatch(e)(current)
          } yield (next, out ++ ops)
        }
      }
      .map(_._2)

  private def escape(key: String): String = key.replace("~", "~0").replace("/", "~1")

  private def rfc(op: String, path: String, value: Option[Json]): Json =
    Json.Object(Chunk.from(Vector("op" -> Json.String(op), "path" -> Json.String(path)) ++ value.map("value" -> _)))
}

Converting the Zionomicon patch gives standard RFC 6902 operations:

Rfc6902.fromJsonPatch(mondayJson, jsonPatch).map(_.print).foreach(println)
// [{"op":"replace","path":"/title","value":"Zionomicon (2nd Edition)"},{"op":"replace","path":"/price","value":39.99},{"op":"add","path":"/tags/-","value":"effects"},{"op":"replace","path":"/edition","value":2}]

List edits become add and remove operations by index. Here, we remove two tags and append one:

val retagged = Json.from(zionomicon.copy(tags = List("zio", "scala", "effects", "fibers")))
val trimmed  = Json.from(zionomicon.copy(tags = List("zio", "fibers", "streams")))

The result has two remove operations at index 1, because the first removal shifts the next tag into its place:

Rfc6902.fromJsonPatch(retagged, retagged.diff(trimmed)).map(_.print).foreach(println)
// [{"op":"remove","path":"/tags/1"},{"op":"remove","path":"/tags/1"},{"op":"add","path":"/tags/-","value":"streams"}]

The conversion loses some detail, and it needs the source document. A number delta becomes a replace with the new number, which the converter computes from the source, and a string edit becomes a replace of the whole string. The result says "set the price to 39.99", not "lower it by 5.00". The converter accepts the patches that Json#diff and JsonPatch.diff produce, whose operations all sit at the root path, and returns Left for a hand-built patch with a deeper path. Going the other way, reading an RFC 6902 patch that another system sends us, is out of scope for this guide.

Putting It Together

The complete example defines the catalog, records two edits, stores the history as JSON, replays and undoes it, shows PatchMode on a changed catalog, and prints the RFC 6902 form of a change:

schema-examples/src/main/scala/diffandpatch/CompleteExample.scala
package diffandpatch

import zio.blocks.chunk.Chunk
import zio.blocks.schema._
import zio.blocks.schema.json._
import zio.blocks.schema.patch._

/**
 * Diff and Patch: Putting It Together, Complete Example
 *
 * Records two edits to Zionomicon in an edit log, prints the audit lines,
 * stores the log as JSON and loads it back, replays and undoes it, applies a
 * catalog patch to a catalog another editor changed, rejects a stale patch, and
 * prints the RFC 6902 form of a change.
 *
 * Run with: cd schema-examples && sbt "runMain diffandpatch.CompleteExample"
 */
object CompleteExample extends App {

  final case class Book(id: String, title: String, price: BigDecimal, tags: List[String], edition: Int)

  object Book {
    implicit val schema: Schema[Book] = Schema.derived
  }

  final case class Catalog(books: Map[String, Book])

  object Catalog {
    implicit val schema: Schema[Catalog] = Schema.derived
  }

  final case class Edit(author: String, forward: DynamicPatch, backward: DynamicPatch)

  object Edit {
    implicit val schema: Schema[Edit] = Schema.derived
  }

  def auditLines[A](schema: Schema[A], before: A, after: A, patch: Patch[A]): List[String] = {
    val old                                                   = schema.toDynamicValue(before)
    val now                                                   = schema.toDynamicValue(after)
    def show(value: DynamicValue, path: DynamicOptic): String =
      value.get(path).one.fold(_ => "(none)", v => Json.fromDynamicValue(v).print)
    patch.dynamicPatch.ops.toList.map(op => s"${op.path}: ${show(old, op.path)} -> ${show(now, op.path)}")
  }

  def applyStored[A](schema: Schema[A], value: A, patch: DynamicPatch): Either[SchemaError, A] =
    patch(schema.toDynamicValue(value), PatchMode.Strict).flatMap(schema.fromDynamicValue)

  def record(author: String, before: Book, after: Book): Edit =
    Edit(author, Book.schema.diff(before, after).dynamicPatch, Book.schema.diff(after, before).dynamicPatch)

  def replay(base: Book, edits: List[Edit]): Either[SchemaError, Book] =
    edits.foldLeft[Either[SchemaError, Book]](Right(base))((book, edit) =>
      book.flatMap(applyStored(Book.schema, _, edit.forward))
    )

  def applyIfUnchanged(current: Book, base: Book, patch: Patch[Book]): Either[String, Book] =
    if (current != base)
      Left(s"${current.id} changed since this patch was made; diff again against the current version")
    else patch(current, PatchMode.Strict).left.map(_.message)

  object Rfc6902 {
    def fromJsonPatch(source: Json, patch: JsonPatch): Either[SchemaError, Json] =
      patchOps(source, "", patch).map(ops => Json.Array(Chunk.from(ops)))

    private def patchOps(value: Json, path: String, patch: JsonPatch): Either[SchemaError, Vector[Json]] =
      steps(value, patch.ops.toVector)(op => JsonPatch(Chunk.single(op))) { (current, op) =>
        if (op.path.nodes.nonEmpty) Left(SchemaError(s"only root paths are supported, got ${op.path}"))
        else opOps(current, path, op.operation)
      }

    private def opOps(value: Json, path: String, op: JsonPatch.Op): Either[SchemaError, Vector[Json]] = op match {
      case JsonPatch.Op.Set(v)            => Right(Vector(rfc("replace", path, Some(v))))
      case JsonPatch.Op.PrimitiveDelta(_) => JsonPatch.root(op)(value).map(v => Vector(rfc("replace", path, Some(v))))
      case JsonPatch.Op.Nested(p)         => patchOps(value, path, p)
      case JsonPatch.Op.ArrayEdit(edits)  =>
        steps(value, edits.toVector)(e => JsonPatch.root(JsonPatch.Op.ArrayEdit(Chunk.single(e)))) {
          case (_, JsonPatch.ArrayOp.Insert(i, vs)) =>
            Right(vs.toVector.zipWithIndex.map { case (v, k) => rfc("add", s"$path/${i + k}", Some(v)) })
          case (_, JsonPatch.ArrayOp.Append(vs))     => Right(vs.toVector.map(v => rfc("add", s"$path/-", Some(v))))
          case (_, JsonPatch.ArrayOp.Delete(i, n))   => Right(Vector.fill(n)(rfc("remove", s"$path/$i", None)))
          case (cur, JsonPatch.ArrayOp.Modify(i, o)) =>
            cur.elements.lift(i).toRight(SchemaError(s"no index $i at $path")).flatMap(opOps(_, s"$path/$i", o))
        }
      case JsonPatch.Op.ObjectEdit(edits) =>
        steps(value, edits.toVector)(e => JsonPatch.root(JsonPatch.Op.ObjectEdit(Chunk.single(e)))) {
          case (_, JsonPatch.ObjectOp.Add(k, v))      => Right(Vector(rfc("add", s"$path/${escape(k)}", Some(v))))
          case (_, JsonPatch.ObjectOp.Remove(k))      => Right(Vector(rfc("remove", s"$path/${escape(k)}", None)))
          case (cur, JsonPatch.ObjectOp.Modify(k, p)) =>
            cur.fields.collectFirst { case (`k`, v) => v }
              .toRight(SchemaError(s"no key $k at $path"))
              .flatMap(patchOps(_, s"$path/${escape(k)}", p))
        }
    }

    // Emits the RFC 6902 operations for each edit, then applies the edit, so later edits see the updated value.
    private def steps[E](value: Json, edits: Vector[E])(asPatch: E => JsonPatch)(
      emit: (Json, E) => Either[SchemaError, Vector[Json]]
    ): Either[SchemaError, Vector[Json]] =
      edits
        .foldLeft[Either[SchemaError, (Json, Vector[Json])]](Right((value, Vector.empty))) { (acc, e) =>
          acc.flatMap { case (current, out) =>
            for {
              ops  <- emit(current, e)
              next <- asPatch(e)(current)
            } yield (next, out ++ ops)
          }
        }
        .map(_._2)

    private def escape(key: String): String = key.replace("~", "~0").replace("/", "~1")

    private def rfc(op: String, path: String, value: Option[Json]): Json =
      Json.Object(Chunk.from(Vector("op" -> Json.String(op), "path" -> Json.String(path)) ++ value.map("value" -> _)))
  }

  val zionomicon    = Book("zionomicon", "Zionomicon", BigDecimal("44.99"), List("zio", "scala"), 1)
  val secondEdition = zionomicon.copy(price = BigDecimal("39.99"), tags = List("zio", "scala", "effects"), edition = 2)
  val withSubtitle  = secondEdition.copy(title = "Zionomicon (2nd Edition)")

  // Record two edits and print what alice changed.
  val history = List(record("alice", zionomicon, secondEdition), record("bob", secondEdition, withSubtitle))
  auditLines(Book.schema, zionomicon, secondEdition, Book.schema.diff(zionomicon, secondEdition))
    .foreach(line => println(s"alice changed $line"))

  // Store the log as JSON, load it back, replay it and undo the last edit.
  val historyCodec = Schema[List[Edit]].jsonCodec
  val loaded       = historyCodec.decode(historyCodec.encodeToString(history))
  println(loaded.flatMap(replay(zionomicon, _)))
  println(loaded.flatMap(edits => applyStored(Book.schema, withSubtitle, edits.last.backward)))

  // Apply a catalog patch to a catalog that another editor changed.
  val programmingInScala = Book("programming-in-scala", "Programming in Scala", BigDecimal("59.99"), List("scala"), 5)
  val fpInScala          = Book("fp-in-scala", "Functional Programming in Scala", BigDecimal("49.99"), List("scala", "fp"), 2)
  val monday             = Catalog(Map(zionomicon.id -> zionomicon, programmingInScala.id -> programmingInScala))
  val tuesday            = Catalog(Map(zionomicon.id -> secondEdition, fpInScala.id -> fpInScala))
  val catalogPatch       = Catalog.schema.diff(monday, tuesday)
  val changedMonday      = Catalog(monday.books - programmingInScala.id)
  List(PatchMode.Strict, PatchMode.Lenient).foreach { mode =>
    println(s"$mode: ${catalogPatch(changedMonday, mode).map(_.books.keys.toList.sorted)}")
  }

  // Reject a patch whose base changed, instead of removing the wrong tag.
  val tagged         = zionomicon.copy(tags = List("zio", "scala", "effects"))
  val removeScalaTag = Book.schema.diff(tagged, tagged.copy(tags = List("zio", "effects")))
  println(applyIfUnchanged(tagged.copy(tags = List("zio", "effects")), tagged, removeScalaTag))

  // Send the change to a system that speaks RFC 6902.
  val mondayJson = Json.from(zionomicon)
  println(Rfc6902.fromJsonPatch(mondayJson, mondayJson.diff(Json.from(withSubtitle))).map(_.print))
}

Running the Examples

All examples in this guide have corresponding runnable Scala files in the schema-examples project. To get them, clone the repository and change into the examples project:

git clone https://github.com/zio/zio-blocks.git
cd zio-blocks/schema-examples

The step examples share the model in BookModel.scala:

schema-examples/src/main/scala/diffandpatch/BookModel.scala

schema-examples/src/main/scala/diffandpatch/BookModel.scala
package diffandpatch

import zio.blocks.schema._
import zio.blocks.schema.json._
import zio.blocks.schema.patch._

/**
 * Diff and Patch: the shared model
 *
 * The book, the catalog and the edit-log entry used by every step example, the
 * sample books, and two helpers from the guide: `auditLines` and `applyStored`.
 */
final case class Book(id: String, title: String, price: BigDecimal, tags: List[String], edition: Int)

object Book {
  implicit val schema: Schema[Book] = Schema.derived
}

final case class Catalog(books: Map[String, Book])

object Catalog {
  implicit val schema: Schema[Catalog] = Schema.derived
}

final case class Edit(author: String, forward: DynamicPatch, backward: DynamicPatch)

object Edit {
  implicit val schema: Schema[Edit] = Schema.derived
}

object Books {
  val zionomicon: Book = Book("zionomicon", "Zionomicon", BigDecimal("44.99"), List("zio", "scala"), 1)

  val secondEdition: Book =
    zionomicon.copy(price = BigDecimal("39.99"), tags = List("zio", "scala", "effects"), edition = 2)

  val withSubtitle: Book = secondEdition.copy(title = "Zionomicon (2nd Edition)")

  val programmingInScala: Book =
    Book("programming-in-scala", "Programming in Scala", BigDecimal("59.99"), List("scala"), 5)

  val fpInScala: Book =
    Book("fp-in-scala", "Functional Programming in Scala", BigDecimal("49.99"), List("scala", "fp"), 2)
}

object PatchHelpers {

  /**
   * One line per changed path: the path, the old value and the new value as
   * JSON.
   */
  def auditLines[A](schema: Schema[A], before: A, after: A, patch: Patch[A]): List[String] = {
    val old                                                   = schema.toDynamicValue(before)
    val now                                                   = schema.toDynamicValue(after)
    def show(value: DynamicValue, path: DynamicOptic): String =
      value.get(path).one.fold(_ => "(none)", v => Json.fromDynamicValue(v).print)
    patch.dynamicPatch.ops.toList.map(op => s"${op.path}: ${show(old, op.path)} -> ${show(now, op.path)}")
  }

  /** Applies a stored `DynamicPatch` to a typed value. */
  def applyStored[A](
    schema: Schema[A],
    value: A,
    patch: DynamicPatch,
    mode: PatchMode = PatchMode.Strict
  ): Either[SchemaError, A] =
    patch(schema.toDynamicValue(value), mode).flatMap(schema.fromDynamicValue)
}

Run them in order to follow the guide step by step.

Diff and Audit

This example diffs the first and second edition of Zionomicon, prints the patch, and turns it into audit lines.

schema-examples/src/main/scala/diffandpatch/DiffAndAuditExample.scala

schema-examples/src/main/scala/diffandpatch/DiffAndAuditExample.scala
package diffandpatch

import Books._

/**
 * Diff and Patch: Diff Two Versions and Turn the Patch into Audit Lines
 *
 * Diffs the first and second edition of Zionomicon, prints the patch, a string
 * edit and an empty diff, then prints one audit line per changed field.
 *
 * Run with: cd schema-examples && sbt "runMain
 * diffandpatch.DiffAndAuditExample"
 */
object DiffAndAuditExample extends App {

  val pricePatch = Book.schema.diff(zionomicon, secondEdition)
  println(pricePatch)

  println(Book.schema.diff(secondEdition, withSubtitle))

  println(Book.schema.diff(zionomicon, zionomicon).isEmpty)

  PatchHelpers.auditLines(Book.schema, zionomicon, secondEdition, pricePatch).foreach(println)
}