Skip to content
StatelyDB

Scanning over a Store

The Scan family of APIs are very similar to List but they allow you to list Items across your entire Store. This can be useful scenarios such as:

  • Migrations and backfills that need to operate on every Item
  • Custom exporters to other datastores
  • Auditing/validation workflows
  • Deleting unwanted data
  • Building global aggregations (e.g. compute the top X blog posts by comments, or counting the number of items meeting some criteria)

Be warned that these operations can be slow and expensive, especially on large Stores. You should use them sparingly and consider using List instead if you can. These results of a Scan operation are not guaranteed to be in any particular order and Items with multiple key paths will only be returned once with their primary key path.

Beginning a Scan

Section titled “Beginning a Scan”

Just like for a List operation, you begin by calling BeginScan, with your desired parameters. Then you can continue to retrieve more Items by calling ContinueScan with the token returned by BeginScan.

For this example we’ll use the schema defined in Example: Movies Schema, which defines these key paths (among others):

Item TypeKey Path Template
Movie/movie-:id
Actor/actor-:id
17 collapsed lines
package main


import (
  "context"
  "fmt"
  "os"
  "slices"
  "strconv"
  "time"


  "github.com/google/uuid"


  "github.com/StatelyCloud/go-sdk/stately"
  // This is the code you generated from schema
  "github.com/StatelyCloud/stately/go-sdk-sample/schema"
)


func sampleScan(
  ctx context.Context,
  client stately.Client,
) (*stately.ListToken, error) {
  iter, err := client.BeginScan(
    ctx,
    stately.ScanOptions{ItemTypes: []string{"Movie", "Actor"}},
  )
  if err != nil {
    return nil, err
  }


  for iter.Next() {
    item := iter.Value()
    switch v := item.(type) {
    case *schema.Movie:
      fmt.Printf("Movie Title: %s\n", v.GetTitle())
    case *schema.Actor:
      fmt.Printf("Actor Name: %s\n", v.GetName())
    }
  }
  // When we've exhausted the iterator, we'll get a token that we
  // can use to fetch the next page of items.
  return iter.Token()
}

Using the List Token to Continue

Section titled “Using the List Token to Continue”

The result from BeginScan includes a list token which you can use to continue in the ContinueScan. Read more about list tokens in Using the List Token to Continue. token.canSync will always be set to false for Scan operations.

17 collapsed lines
package main


import (
  "context"
  "fmt"
  "os"
  "slices"
  "strconv"
  "time"


  "github.com/google/uuid"


  "github.com/StatelyCloud/go-sdk/stately"
  // This is the code you generated from schema
  "github.com/StatelyCloud/stately/go-sdk-sample/schema"
)


func sampleContinueScan(
  ctx context.Context,
  client stately.Client,
  token *stately.ListToken,
) (*stately.ListToken, error) {
  iter, err := client.ContinueScan(ctx, token.Data)
  if err != nil {
    return nil, err
  }
  for iter.Next() {
    item := iter.Value()
    switch v := item.(type) {
    case *schema.Character:
      fmt.Printf("Character Name: %s\n", v.GetName())
    case *schema.Actor:
      fmt.Printf("Actor Name: %s\n", v.GetName())
    }
  }
  // You could save the token to call ContinueScan later.
  return iter.Token()
}

Filters

Section titled “Filters”

Scan results can also be limited by applying filters to the result set. Please note: filters are applied after the initial result set is fetched. This means that you are still charged for reading items which are filtered out. If scanning over an entire table to fetch only a small number of the items becomes a common pattern in your application, it may be more cost-effective to use a dedicated global index; reach out to learn more! There are two kinds of filters StatelyDB supports:

Item Type Filter
Section titled “Item Type Filter”

The first is an item type filter, which allows you to specify which item types you want to include in the result set. This is useful when there are multiple item types in the table that could be returned but when you only care about specific item types. If this filter is not specified, all item types found via the scan operation are included in the result set.

For example, the Example: Movies Schema, has 4 unique item types Movie, Character, Actor, and Change but we could scan the entire table for ALL movies by adding a filter on the Movie item type. We can set an itemType filter accordingly. Note: When an item has multiple key paths, it will only be returned once with its primary key path, even if multiple key paths match the item type filter.

To see language-specific examples of how to use item type filters in a Scan operation, see the example tables under CEL Expression Filter.

CEL Expression Filter
Section titled “CEL Expression Filter”

The second is a CEL expression filter, which allows you to specify any arbitrary conditions that an item type must satisfy to be included in the result set. CEL expression filters use the CEL language spec providing you with a powerful, flexible way to filter items based on their properties, relationships, and other criteria. For example, you could construct a filter that would include movies rated PG-13 less than 2 hours in duration, released in a year that is a multiple of 3.

CEL expression filters only apply to a single item type at a time and do not affect other item types in a result set. This means that if an item type isn’t mentioned in a CEL expression filter and there are no item type filter constraints, it will be included in the result set.

In the context of a CEL expression, the key-word this refers to the item being evaluated, and property properties should be accessed by the names as they appear in schema — not necessarily as they appear in the generated code for a particular language. For example, if you have a Movie item type with the property rating, you could write a CEL expression like this.rating == 'R' to return only movies that are rated R.

See the following examples for how to use filters in a Scan operation by language:

17 collapsed lines
package main


import (
  "context"
  "fmt"
  "os"
  "slices"
  "strconv"
  "time"


  "github.com/google/uuid"


  "github.com/StatelyCloud/go-sdk/stately"
  // This is the code you generated from schema
  "github.com/StatelyCloud/stately/go-sdk-sample/schema"
)


func sampleScanWithFilters(
  ctx context.Context,
  client stately.Client,
) (*stately.ListToken, error) {
  opts := stately.ScanOptions{}.
    // Fetch ONLY Movie itemTypes
    WithItemTypesToInclude("Movie").
    // This filter will ONLY return movies that are...
    // 1. Rated PG-13
    // 2. Have a duration of less than 2 hours
    // 3. Released in a year that is a multiple of 3
    WithCelExpressionFilter("Movie",
      "this.rating == 'PG-13' && this.duration < duration('2h').getSeconds() && this.year % 3 == 0")


  iter, err := client.BeginScan(ctx, opts)
  if err != nil {
    return nil, err
  }


  for iter.Next() {
    movie := iter.Value().(*schema.Movie) // We know this is a Movie because of the item type filter!
    fmt.Printf("Movie Title: %s\n", movie.GetTitle())
  }


  // When we've exhausted the iterator, we'll get a token that we
  // can use to fetch the next page of items.
  return iter.Token()
}

Limits

Section titled “Limits”

Pass a limit to BeginScan to limit the max number of items to retrieve. If limit is set to 0 then the first page of results will be returned which may be empty because all the results were filtered out. Be sure to check token.canContinue to see if there are more results to fetch.

Segmentation

Section titled “Segmentation”

Because a Scan operation can be slow for large data sets, you can segment the operation into smaller chunks by passing a totalSegments and segmentIndex parameter to BeginScan. This will allow you to run multiple Scan operations in parallel, each responsible for a different segment of the Store. You can split your scan into up to 1000000 segments. Note: Some backing storage layers may have limits on the number of segments they support.

Listing Across Client Upgrades

Section titled “Listing Across Client Upgrades”

Just like for List operations, you are not able to use a list token across client versions.