Syncing Lists

Many applications want to cache or store a list of items from the database, and keep them up to date. However, it can be wasteful to repeatedly fetch the same list of items from scratch, over and over - especially since it’s likely only a few Items (or none!) have changed since the last poll.

Instead of fetching the whole list from scratch, you can take an existing list token and call SyncList. SyncList tells you about any Items which have been added, changed or deleted within your result set since you got that token. Imagine you’ve already gotten the first few screens of emails in your user’s inbox. You can call SyncList with that token, and it will tell you about any new emails, and any emails which have changed state (e.g. being marked read). SyncList won’t tell you about changes to Items outside your result set window - so you wouldn’t get an update for and email from 100 screens down that changed state, unless you’d called ContinueList enough to include that email in your result set.

SyncList allows for building efficient offline-capable clients, cheaply updating caches, and more that would be difficult to maintain if you had to always get the whole result set fresh every time.

The same token is used for ContinueList and SyncList, so every time you call either one, you can update your saved token to use for the next time.

Sync Results

SyncList returns a stream of results that are each one of the following cases:

1. changed - An item that changed since the last time you saw it. The entire new item is returned with this case.
2. deleted - The item at this key path has been deleted since the last time you saw it. You should remove the item matching the key path from any local cache or storage.
3. reset - If your token is too old, or something else has changed to make Sync impossible, you’ll get a “reset” message from SyncList as the first result. This means you should throw away any cached or stored data and start over. The rest of the results from this SyncList call will be changed cases to help you start your result set over.
4. updatedOutsideWindow - This is a special case when syncing lists managed by a Group Local Index (see Sort Property, and Group Local Index). Unlike the Key Property, index values are mutable, and therefore Stately cannot always determine if an item was previously in a result set but has moved or has never been in a given result set. All that Stately can say for sure is that an item has changed and that it is not currently in the sync window. Imagine you’re listing items by date, and you have the last 24 hours of Items, but one of the Items had its date updated to be a week in the past. That Item is no longer in your result set, so you probably don’t want to display it, but it also wasn’t deleted either. The same applies to an item two weeks in the past which was updated with more information while the date remained unchanged. The item has changed, it is not in your sync window, but it might have been at some point in time. In most situations a key returned in updatedOutsideWindow should be treated just like a key returned in the deleted field; remove this from the cache if it exists. But in some cases, such as those involving a shared cache, it is important to know the difference between a delete and an update outside this window.

Here’s an example of syncing changes to the list we fetched in Listing Items:

Go

package main

import (
  "context"
  "fmt"
  "os"
  "slices"
  "strconv"
  "time"

  "github.com/StatelyCloud/go-sdk/stately"
  // This is the code you generated from schema
  "github.com/StatelyCloud/stately/go-sdk-sample/schema"
)

func sampleSyncList(
  ctx context.Context,
  client stately.Client,
  token *stately.ListToken,
) (*stately.ListToken, error) {
  syncIter, err := client.SyncList(ctx, token.Data)
  if err != nil {
    return nil, err
  }

  for syncIter.Next() {
    switch r := syncIter.Value().(type) {
    case *stately.Reset:
      // This means our token is too old and we need to start over.
      fmt.Println("Sync operation reset")
    case *stately.Changed:
      // This item has changed since we last saw it, we should update our
      // local copy
      fmt.Printf("Item has changed: %s\n", r.Item)
    case *stately.Deleted:
      // This item has been deleted since we last saw it, we should remove
      // it from our local copy
      fmt.Printf("Item has been deleted: %s\n", r.KeyPath)
    }
  }

  return syncIter.Token()
}

Ruby

require 'bundler/setup'
require_relative 'schema/stately'
require 'byebug'


def sample_sync_list(client, token)
  sync_list_result = client.sync_list(token)

  if sync_list_result.is_reset
    # This means our token is too old and we need to start over.
    puts "Sync operation reset"
  end
  sync_list_result.changed_items.each do |item|
    # This item has changed since we last saw it, we should update our
    # local copy
    puts "Item has changed: #{item.id}"
  end
  sync_list_result.deleted_item_paths.each do |item|
    # This item has been deleted since we last saw it, we should remove
    # it from our local copy
    puts "Item has been deleted: #{item.id}"
  end

  return sync_list_result.token
end

Python

from __future__ import annotations

from typing import TYPE_CHECKING

from statelydb import ListToken, SyncChangedItem, SyncDeletedItem, SyncReset, key_path

from .schema import Actor, Change, Character, Client, Movie


async def sample_sync_list(client: Client, token: ListToken) -> None:
    sync_resp = await client.sync_list(token)
    async for item in sync_resp:
        if isinstance(item, SyncReset):
            # This means our token is too old and we need to start over.
            print("Sync operation reset")
        elif isinstance(item, SyncChangedItem):
            # This item has changed since we last saw it, we should
            # update our local copy
            print(f"Item has changed {item}")
        elif isinstance(item, SyncDeletedItem):
            # This item has been deleted since we last saw it, we should
            # remove it from our local copy
            print(f"Item has been deleted {item.key_path}")
    token = sync_resp.token

TypeScript

import { createClient, DatabaseClient, Movie } from "./schema/index.js";
import { keyPath, ListToken } from "@stately-cloud/client";

async function sampleSync(client: DatabaseClient): Promise<ListToken> {
  const syncResp = client.syncList(token.tokenData);
  for await (const change of syncResp) {
    switch (change.type) {
      case "reset":
        // This means our token is too old and we need to start over.
        console.log("Sync operation reset");
        break;
      case "changed":
        // This item has changed since we last saw it, we should update our
        // local copy
        console.log("Item has changed:", change.item);
        break;
      case "deleted":
        // This item has been deleted since we last saw it, we
        // should remove it from our local copy
        console.log(`Item has been deleted: ${change.keyPath}`);
        break;
      default:
        throw new Error("unexpected change type " + change.type);
    }
  }

  return syncResp.token!;
}

Syncing Across Client Upgrades

The list token you get from BeginList is specific to the schema version your client was built with. If you call SyncList with a client that has been upgraded to use a different schema version, you must ensure you properly handle the “reset” change type by clearing out any local state before consuming the updated items. This ensures that your cached result set is all based on a consistent schema version.