Skip to content
StatelyDB

Listing Items

The List family of APIs are one of the most useful APIs in StatelyDB. While APIs like Put, Get, and Delete allow you to operate on individual Items by their full key path, List lets you fetch multiple Items from the same Group in one go (i.e. Items that share the same Group Key). This can be useful for fetching collections of Items such as a customer’s order history, or a list of messages in a conversation. It’s especially useful for implementing features like infinite scrolling lists, where you want to keep fetching more Items as the user scrolls down.

Beginning a Paginated List

Section titled “Beginning a Paginated List”

You start a list by calling BeginList with a key path prefix and an optional limit to get an initial result set. StatelyDB will return all the Items whose key paths have the given prefix. See Constraining a List Operation for other ways to constrain the result set of a List operation.

Your initial result set might not contain all the Items under that prefix if you set a page size limit. This initial result set is your first “page” of data - imagine it as the first several screens full of emails in your email app.

For this example we’ll use the schema defined in Example: Movies Schema, which defines these key paths (among others):

Item TypeKey Path Template
Movie/movie-:id
Character/movie-:movieId/role-:role/name-:name

Note how both Movie and Character share the same /movie-:id prefix.

17 collapsed lines
package main


import (
  "context"
  "fmt"
  "os"
  "slices"
  "strconv"
  "time"


  "github.com/google/uuid"


  "github.com/StatelyCloud/go-sdk/stately"
  // This is the code you generated from schema
  "github.com/StatelyCloud/stately/go-sdk-sample/schema"
)


func sampleList(
  ctx context.Context,
  client stately.Client,
  movieID uuid.UUID,
) (*stately.ListToken, error) {
  iter, err := client.BeginList(
    ctx,
    // This key path is a prefix of BOTH Movie and Character.
    "/movie-"+stately.ToKeyID(movieID[:]),
    stately.ListOptions{Limit: 10},
  )
  if err != nil {
    return nil, err
  }


  for iter.Next() {
    item := iter.Value()
    switch v := item.(type) {
    case *schema.Movie:
      fmt.Printf("Movie Title: %s\n", v.GetTitle())
    case *schema.Character:
      fmt.Printf("Character Name: %s\n", v.GetName())
    }
  }
  // When we've exhausted the iterator, we'll get a token that we
  // can use to fetch the next page of items.
  return iter.Token()
}


func sampleListWithConstraints(
  ctx context.Context,
  client stately.Client,
  genre string,
  startYear int32,
  endYear int32,
) (*stately.ListToken, error) {
  prefixKey := "/genres-" + stately.ToKeyID(genre) + "/years-"
  iter, err := client.BeginList(
    ctx,
    prefixKey,
    stately.ListOptions{}.
      WithKeyGreaterThanOrEqualTo(prefixKey+stately.ToKeyID(startYear)).
      WithKeyLessThanOrEqualTo(prefixKey+stately.ToKeyID(endYear)).
      WithItemTypesToInclude("Movie"),
  )
  if err != nil {
    return nil, err
  }


  for iter.Next() {
    move := iter.Value().(*schema.Movie) // We know this is a Movie because of the item type filter!
    fmt.Printf("Movie Title: %s\n", move.GetTitle())
  }
  // When we've exhausted the iterator, we'll get a token that we
  // can use to fetch the next page of items.
  return iter.Token()
}


func sampleListWithFilters(
  ctx context.Context,
  client stately.Client,
  genre string,
) (*stately.ListToken, error) {
  prefixKey := "/genres-" + stately.ToKeyID(genre) + "/years-"
  iter, err := client.BeginList(
    ctx,
    prefixKey,
    stately.ListOptions{}.
      WithItemTypesToInclude("Movie").
      // This filter will ONLY return movies that are...
      // 1. Rated PG-13
      // 2. Have a duration of less than 2 hours
      // 3. Released in a year that is a multiple of 3
      WithCelExpressionFilter("Movie",
        "this.rating == 'PG-13' && this.duration < duration('2h').getSeconds() && this.year % 3 == 0"),
  )
  if err != nil {
    return nil, err
  }


  for iter.Next() {
    move := iter.Value().(*schema.Movie) // We know this is a Movie because of the item type filter!
    fmt.Printf("Movie Title: %s\n", move.GetTitle())
  }
  // When we've exhausted the iterator, we'll get a token that we
  // can use to fetch the next page of items.
  return iter.Token()
}

Constraining a List Operation

Section titled “Constraining a List Operation”

While List operations can fetch items in the same group, it is also possible to optimize the result set of a List operation further to fetch only the items you need. Below are some of the ways you can constrain a List operation.

Key Prefix

Section titled “Key Prefix”

Key paths are more than just unique identifiers for items in a database. Their structure defines the questions you can efficiently answer with a List operation. The key prefix you use is a lot like choosing which question you want to ask StatelyDB. This is why it is so important to carefully design your key paths so that your most common questions can be efficiently answered. To illustrate this let’s think about a few questions we might want to ask our Example: Movies Schema about Movie and Character item types. Then we’ll define a key path that can answer that question and a prefix to use. Note: The key paths below are not necessarily defined in the schema example we’re referencing, see parting thought 1 below the table.

#QuestionItem TypeKey PathKey prefix to use
1What were all the movies released in 2023?Movie/years-:year/genres-:genre/movie-:id/years-2023
2.aWhat were all the Comedies released in 2023?Movie/years-:year/genres-:genre/movie-:id/years-2023/genres-Comedy
2.bWhat were all the Comedies released in 2023?Movie/genres-:genre/years-:year/movie-:id/genres-Comedy/years-2023
3What Comedies were released over all time?Movie/genres-:genre/years-:year/movie-:id/genres-Comedy
4What were all Character roles in movie X?Character/movie-:movieId/role-:role/actor-:actorId/movie-:X/role
5Who were all the actors who played role Y?Character/movie-:movieId/role-:role/actor-:actorId/movie-:X/role-:Y/actor
6What are all the roles/characters played by actor Z?Character/actor-:actorId/years-:year/movie-:movieId/role-:role/actor-:Z
7What are all the roles/characters played by actor Z in 2024?Character/actor-:actorId/years-:year/movie-:movieId/role-:role/actor-:Z/year-2024
8.aWhat were all the roles played actor Z in movie X?Character/actor-:actorId/years-:year/movie-:movieId/role-:role/actor-:Z/year-:X.year/movie-:X
8.bWhat were all the roles played actor Z in movie X?Character/actor-:actorId/movie-:movieId/role-:role/actor-:Z/movie-:X

Some parting thoughts:

  1. It is important to recognize that in any schema you may not want to define key paths to answer every question. In DynamoDB-backed stores, the cost of a write is significantly more expensive than the cost of a read, so there is a balance to strike. It might feel counterintuitive, but sometimes a less efficient list operation (e.g. one with other ItemTypes in its path) with filtering can be more cost-effective than writing an additional key path in order to create an index of only the desired items. Thus, choosing which key path(s) to write is a balance between read and write cost efficiency and your application’s access patterns. For example, in the table above, there are two key paths that can answer “What were all the Comedies released in 2023?” (2.a and 2.b). So if you only need to fetch movies by year, and never (or rarely) all years by genre, you might choose to define the key path /years-:year/genres-:genre/movie-:id instead of /genres-:genre/years-:year/movie-:id. This is because the first key path answers question 1, “What were all the movies released in 2023?” more efficiently than the second key path, which would require fetching all genres for each movie in 2023.

  2. Our questions above are a little simplistic because they focus on fetching a single item type. But in practice, you may find that you want to fetch multiple item types at once, or that you want to fetch items that are related to each other. For example, you might want to fetch all the movies in a certain genre AND for each movie fetch all the characters in that movie. A movie key path /genres-:genre/movie-:id and a Character key path of /genres-:genre/movie-:id/role-:role/actor-:actorId can be used to answer this question with the prefix /genres-:genre.

  3. Sometimes multiple key prefixes can be used to answer the same question with the same underlying key path. Question 5 above can be answered without the trailing namespace actor. But that is only because there is currently no other ItemType that shares the /movie-:id/role-:role prefix. As a general rule, you should always use the most specific key prefix that answers your question. This will help you avoid fetching more items than you need, and will also help you avoid potential conflicts with other item types in the future.

Key Path Range

Section titled “Key Path Range”

Sometimes we want to fetch a range of items that share the same key prefix, but we want to limit the results to a specific range of key paths. For example, if we want to know “What are all the Comedies released from 2020 to 2023 (inclusive)?”. This can be answered with the key prefix /genres-Comedy/years but the result set would include all years and need to be filtered not just 2020 to 2023. Here we can use key constraints to narrow the scope of our list operation: GreaterThanEqualTo /genres-Comedy/years-2020 and LessThanEqualTo /genres-Comedy/years-2023. These constraints are applied at the database layer to give you the most optimized list experience, and so you don’t have to filter the results in your application code. Examples:

17 collapsed lines
package main


import (
  "context"
  "fmt"
  "os"
  "slices"
  "strconv"
  "time"


  "github.com/google/uuid"


  "github.com/StatelyCloud/go-sdk/stately"
  // This is the code you generated from schema
  "github.com/StatelyCloud/stately/go-sdk-sample/schema"
)


func sampleListWithConstraints(
  ctx context.Context,
  client stately.Client,
  genre string,
  startYear int32,
  endYear int32,
) (*stately.ListToken, error) {
  prefixKey := "/genres-" + stately.ToKeyID(genre) + "/years-"
  iter, err := client.BeginList(
    ctx,
    prefixKey,
    stately.ListOptions{}.
      WithKeyGreaterThanOrEqualTo(prefixKey+stately.ToKeyID(startYear)).
      WithKeyLessThanOrEqualTo(prefixKey+stately.ToKeyID(endYear)).
      WithItemTypesToInclude("Movie"),
  )
  if err != nil {
    return nil, err
  }


  for iter.Next() {
    move := iter.Value().(*schema.Movie) // We know this is a Movie because of the item type filter!
    fmt.Printf("Movie Title: %s\n", move.GetTitle())
  }
  // When we've exhausted the iterator, we'll get a token that we
  // can use to fetch the next page of items.
  return iter.Token()
}

Filters

Section titled “Filters”

List results can also be limited by applying filters to the result set. Unlike Key Prefix and Key Path Range constraints which are applied at a database layer to reduce the number of items read, filters are applied after the initial result set is fetched. This means that you are still charged for reading items which are filtered out. Therefore, it is still important to use Key Prefix and Key Path Range constraints where possible to optimize your List operations. This doesn’t mean using a filter is bad! In fact, it can sometimes be more cost-effective to apply filters to list operations than to write items in dedicated indexes or with additional key paths just to limit the use of filters. There are two kinds of filters StatelyDB supports:

Item Type Filter
Section titled “Item Type Filter”

The first is an item type filter, which allows you to specify which item types you want to include in the result set. This is useful when there are multiple item types a group which could be returned by the query but when you only care about specific item types. If this filter is not specified, all item types found via the list operation are included in the result set.

For example, Movies and Characters in Example: Movies Schema, have the key paths /movie-:id and /movie-:movieId/role-:role/name-:name, respectively. So a Key Prefix of /movie-<movieID> will return both Movies and Characters for that movie. To ensure only Character item types are returned for a specific movie, you can use an item type filter with the value Character. All other types will be excluded or filtered out of the result set.

To see language-specific examples of how to use item type filters in a List operation, see the example tables under Key Path Range and CEL Expression Filter.

CEL Expression Filter
Section titled “CEL Expression Filter”

The second is a CEL expression filter, which allows you to specify any arbitrary conditions that an item type must satisfy to be included in the result set. CEL expression filters use the CEL language spec providing you with a powerful, flexible way to filter items based on their properties, relationships, and other criteria. For example, you could construct a filter that would include movies rated PG-13 less than 2 hours in duration, released in a year that is a multiple of 3.

CEL expression filters only apply to a single item type at a time, and do not affect other item types in a result set. This means that if an item type isn’t mentioned in a CEL expression filter and there are no item type filter constraints, it will be included in the result set.

In the context of a CEL expression, the key-word this refers to the item being evaluated, and property properties should be accessed by the names as they appear in schema — not necessarily as they appear in the generated code for a particular language. For example, if you have a Movie item type with the property rating, you could write a CEL expression like this.rating == 'R' to return only movies that are rated R.

See the following examples for how to use filters in a List operation by language:

17 collapsed lines
package main


import (
  "context"
  "fmt"
  "os"
  "slices"
  "strconv"
  "time"


  "github.com/google/uuid"


  "github.com/StatelyCloud/go-sdk/stately"
  // This is the code you generated from schema
  "github.com/StatelyCloud/stately/go-sdk-sample/schema"
)


func sampleListWithFilters(
  ctx context.Context,
  client stately.Client,
  genre string,
) (*stately.ListToken, error) {
  prefixKey := "/genres-" + stately.ToKeyID(genre) + "/years-"
  iter, err := client.BeginList(
    ctx,
    prefixKey,
    stately.ListOptions{}.
      WithItemTypesToInclude("Movie").
      // This filter will ONLY return movies that are...
      // 1. Rated PG-13
      // 2. Have a duration of less than 2 hours
      // 3. Released in a year that is a multiple of 3
      WithCelExpressionFilter("Movie",
        "this.rating == 'PG-13' && this.duration < duration('2h').getSeconds() && this.year % 3 == 0"),
  )
  if err != nil {
    return nil, err
  }


  for iter.Next() {
    move := iter.Value().(*schema.Movie) // We know this is a Movie because of the item type filter!
    fmt.Printf("Movie Title: %s\n", move.GetTitle())
  }
  // When we've exhausted the iterator, we'll get a token that we
  // can use to fetch the next page of items.
  return iter.Token()
}

Using the List Token to Continue

Section titled “Using the List Token to Continue”

The result from BeginList includes a token which you can save for later, or use right away. Only the tokenData part of the token needs to be saved. There is also a canContinue property which indicates whether there are more pages still available, and canSync which indicates whether SyncList is supported for this list.

You can pass the token to ContinueList, which lets you fetch more results for your result set, continuing from where you left off. For example, you might call ContinueList to get the next few screens of emails when the user scrolls down in their inbox. Or, you could call it in the background to eventually pull the entire result set into a local database. All you need is the token—the original arguments to BeginList are saved with it.

The token keeps track of the state of the result set, and ContinueList allows you to expand the window of results that you have retrieved. Every time you call ContinueList, you’ll get a new token back, and you can use that token for the next ContinueList or SyncList call, and so on. This is also known as pagination, and it allows you to quickly show results to your users without having to get all the data at once, while still having the ability to grab the next results consistently.

For many applications, you’ll only need to call BeginList once, to set up the initial token, and from then on they’ll call ContinueList (as a user scrolls through results) or SyncList (whenever the user opens or focuses the application, to check for new and updated items).

In this example we’ve passed the token from the first example back into ContinueList to keep getting more Items:

17 collapsed lines
package main


import (
  "context"
  "fmt"
  "os"
  "slices"
  "strconv"
  "time"


  "github.com/google/uuid"


  "github.com/StatelyCloud/go-sdk/stately"
  // This is the code you generated from schema
  "github.com/StatelyCloud/stately/go-sdk-sample/schema"
)


func sampleContinueList(
  ctx context.Context,
  client stately.Client,
  token *stately.ListToken,
) (*stately.ListToken, error) {
  iter, err := client.ContinueList(ctx, token.Data)
  if err != nil {
    return nil, err
  }
  for iter.Next() {
    item := iter.Value()
    switch v := item.(type) {
    case *schema.Character:
      fmt.Printf("Character Name: %s\n", v.GetName())
    case *schema.Movie:
      fmt.Printf("Movie Title: %s\n", v.GetTitle())
    }
  }
  // You could save the token to call ContinueList later.
  return iter.Token()
}

Sort Direction

Section titled “Sort Direction”

By default, the items returned in a List are sorted by their key paths, in ascending order. Namespaces, string IDs, and byte IDs are sorted lexicographically, while number IDs are sorted numerically. For example:

  • /customer-1234
  • /customer-1234/order-9
  • /customer-1234/order-10
  • /customer-1234/order-10/li-abc
  • /customer-1234/order-10/li-bcd

You can specify a SortDirection option to reverse this order (from the default of Ascending to Descending).

Listing All Groups

Section titled “Listing All Groups”

If you have root item type, this can be achieved using the Scan operation to list all Items of the group type.

Listing Across Client Upgrades

Section titled “Listing Across Client Upgrades”

The list token you get from BeginList is specific to the schema version your client was built with. If you save that token, and then upgrade your client to another schema version, ContinueList will return a SchemaVersionMismatch error, since there’s no guarantee that the items you had cached are compatible with the new schema version. In this case you should discard the list token and start a new BeginList call.