What is Elastic Schema?

StatelyDB features Elastic Schema, which lets you:

• Describe your data model with real types and integrated validation, enforcing that your data has the shape you want.
• Generate code in your favorite language that works with your database types and removes the need to write object/document mapping code.
• Change your data model any way, at any time. The database automatically migrates data where necessary, while maintaining backwards compatibility with all previous versions of schema. This eliminates the need to coordinate deployments or carefully consider the impact of data model changes on existing clients.

Schema describes your data model

A schema is a description of all the data types in your database. This means all the different kinds of objects, and the fields they can have. Many databases have the concept of a schema, though often “NoSQL” or document databases claim to not have one. Even if your database lets you store arbitrary JSON, you still have a schema - it’s just implicit in your application code.

StatelyDB uses an explicit schema language, written in TypeScript, to define your data model and give the database information it can use to help you out, whether that’s by validating data, optimizing storage, or transforming it. Your schema is a single source of truth for what kinds of data your database can contain, and you can share that common definition between clients in multiple languages.

As an example, here’s a simple User object in StatelyDB Elastic Schema:

import {
  itemType,
  string,
  timestampSeconds,
  uint,
  uuid,
} from "@stately-cloud/schema";

/** A user of our fantastic new system. */
itemType("User", {
  keyPath: "/user-:id",
  fields: {
    id: { type: uuid, initialValue: "uuid" },
    displayName: { type: string },
    email: { type: string },
    lastLoginDate: { type: timestampSeconds },
    numLogins: { type: uint },
  },
});

Elastic Schema lets you change your mind

While many databases have some way to express your data’s schema, Elastic Schema takes it further by allowing you to change your schema any way you want, any time. When you change your schema, earlier versions of schema are still valid and can be used by existing or new clients. Since StatelyDB knows about all your different schema versions, it will automatically transform data between the version that’s actually stored in the database and the version that each client expects. In many cases this transformation can be done without having to update anything in the database itself, meaning the “migration” is free.

Always backwards (and forwards) compatible

Backwards compatibility means that a new system (in this case a new database schema) is still compatible with older data and clients. Think of it like a video game console that can still play old games from a previous generation, like the Wii playing GameCube games. Forwards compatibility is more rare—it means that an old system can still handle data created by a newer version of a system. To stretch an analogy, this is like how the original Game Boy could play Game Boy Color games (in greyscale!).

It’s possible to manually maintain backwards and forwards compatibility as you evolve a data model by carefully following a set of rules about which changes are allowed and which aren’t, and then managing the difference in your application code. For example, say you wanted to change a field’s data type from a string to a uint64. You would have to follow a sequence of steps:

1. Add a new uint64 field alongside the original string field.
2. Update your code to read the string field and write both the string and uint64 field. You can’t stop writing the string field if old clients still use it!
3. Eventually after old clients are deprecated, you can stop writing the string field.
4. Write a backfill program to scan through all old records that only have the string field and copy the value over to the uint64 field.
5. Go back and clean up the code that handles both fields—now you can use just the uint64 version.
6. Finally you can write another backfill program to scan through all the old records and remove the string field entirely to save storage space.

As you can imagine, this is easy to mess up, and after you’ve made a lot of changes like this, your application code will grow and grow as more of these cases are added. And this example is for one of the simpler kinds of change!

Elastic Schema handles this all for you in the database. Under the hood, it’s doing much the same kind of work to maintain backwards and forwards compatibility, but your application doesn’t need to worry about any of it. In StatelyDB, you’d just change that string field to a uint64, and update your code to use the new type. Your new code doesn’t need to handle the old data, or worry about compatibility, while old code that expects a string keeps working. And when you eventually deprecate the old schema version, the database will automatically clean up after itself.

More than an ALTER TABLE

In SQL databases there is a system for making changes like that called DDL (Data Definition Language). For example, the data type change above could be done with:

ALTER TABLE table_name
ALTER COLUMN column_name TYPE INTEGER USING column_name::integer;

There are some major differences between this and what Elastic Schema does, though:

1. This statement will immediately lock the table (so no writes can happen) and update every row in the table to use the new format. If this takes a long time, you’ll have an outage.
2. If any values can’t be converted into an integer, the operation will fail.
3. Most importantly, any code that treats this column as a string will now break.

How could you do this safely? Even ignoring the table lock, how could you sequence deploying your code with this operation? There are ways to do it (for example, updating your SQL queries to cast the column back to a string, deploy that change, then run the DDL, then update your SQL queries to remove the cast) but again, it’s very complex and error prone. And what happens if you have to roll back your server after you updated the queries? The older code won’t work against the new database schema and your app will break.

In contrast, Elastic Schema migrations don’t lock the database or disrupt any existing usage. They gracefully handle when existing data isn’t compatible, and they provide automatic backwards compatibility so existing code will keep working, and you can deploy the new code whenever you want—and roll back safely.