SiaSia Developer Portal

List & Sync Objects

The indexer doesn't have a "list my files" query. It has a change feed: objectEvents() returns one event for each pinned object, ordered by when it last changed. Your app keeps its own index of objects, uses the feed to bring that index up to date, and lists objects from the index.

📌 Important

Don't call objectEvents() every time you need to show a list of objects. Keep an index locally, sync it with the feed periodically, and replay the feed from the start only when you need to rebuild your local state.

ℹ Note

The Dart examples need sia_storage 0.4.0 or later. Earlier Dart releases are built on an older version of the SDK core that doesn't support slab versions, so objectEvents() fails with sealed object: invalid signature as soon as your account holds an object uploaded by a current SDK.

How the event feed works

  • One event per object. Each event holds an object's latest state. Replaying the feed from the beginning gives you every live object, plus a deletion event for each object that was deleted.
  • Ordered and resumable. Events are sorted by their feed position, then by object ID. Pass the position and ID of the last event you processed as a cursor, and the next call starts right after it.
  • Paginated. Each call returns at most limit events, and limit can be at most 500. A page shorter than limit means you've caught up.
  • Deletions carry no object. When deleted is true, object is empty. Remove the object from your index.
  • Repeats are possible. If an object changes while you're paging, it shows up again later in the feed with its newer state. Treat every event as an upsert or a delete and repeats are harmless.
  • The event's timestamp is a feed position, not an edit time. The indexer assigns it when it publishes the event, usually within a second or two of the change. Use it only to build the cursor, and use the object's own createdAt and updatedAt for display.

Choose what to keep for each object

Keep a small record for each object with only the fields your app shows, not the object itself. An object holds the key that decrypts its data, so it shouldn't be written to disk as it is, but a record is safe to store. When you need to download or share an object, fetch it by ID with object(id). (If you'd rather keep whole objects, seal them with your App Key before you store them.)

This record keeps the size and the File Name and mime fields from Object Metadata. Metadata is whatever the app that pinned the object attached, so decode it defensively:

rust
use serde::{Deserialize, Serialize}; use sia_storage::Object; /// What the app keeps for each object: only the fields it shows. #[derive(Clone, Serialize, Deserialize)] struct Record { name: String, mime: String, size: u64, } /// Builds a record from an object's metadata. Metadata that isn't a JSON /// object falls back to the defaults below. fn to_record(object: &Object) -> Record { // Requires the serde (with the derive feature) and serde_json crates. let meta: serde_json::Value = serde_json::from_slice(&object.metadata).unwrap_or_default(); Record { name: meta["File Name"].as_str().unwrap_or("(unnamed)").to_owned(), mime: meta["mime"].as_str().unwrap_or("application/octet-stream").to_owned(), size: object.size(), } }

Sync your local index

This function reads every event after cursor, applies each one to index, and returns the cursor to resume from. The same function covers both cases: on first run, pass no cursor to build the index from scratch, and after that, pass the saved cursor to fetch only what changed.

rust
use std::collections::HashMap; use sia_storage::{DateTime, Error, Hash256, ObjectsCursor, Sdk, Utc}; /// The indexer returns at most 500 events per call. const PAGE_SIZE: usize = 500; /// The feed position and ID of the last event applied to the index. type Position = (DateTime<Utc>, Hash256); /// Applies every event after `position` to `index` and returns the /// position to resume from next time. async fn sync_index( sdk: &Sdk, index: &mut HashMap<Hash256, Record>, mut position: Option<Position>, ) -> Result<Option<Position>, Error> { loop { let cursor = position.map(|(after, id)| ObjectsCursor { after, id }); let events = sdk.object_events(cursor, Some(PAGE_SIZE)).await?; let caught_up = events.len() < PAGE_SIZE; for event in events { if event.deleted { index.remove(&event.id); } else if let Some(object) = event.object { index.insert(event.id, to_record(&object)); } position = Some((event.updated_at, event.id)); } if caught_up { return Ok(position); } } }

If a sync fails partway through, retry from the last cursor you saved. Events you already applied will be applied again, which is safe.

List objects from your index

Listing objects means reading your own records, with no network call:

rust
for (id, record) in &index { println!("{id} {} {} {} bytes", record.name, record.mime, record.size); }

Save and load your index

Save the index and the cursor after every sync, and load them when your app starts, so a restart picks up where it left off instead of replaying the whole feed. These functions keep both in one JSON file and replace it in one step (they write a temporary file, then rename it), so the saved index and cursor always match. In a browser, they keep both under one localStorage key.

The saved state belongs to one indexer and one App Key, so it records the indexer URL and the App Key's public key. Loading returns an empty index and no cursor when nothing is saved, when the saved state can't be read, or when it belongs to a different indexer or App Key, and the next sync then rebuilds from scratch. Only the public key is written. Never store the App Key itself next to your index.

rust
/// The index and cursor as written to disk, tagged with the indexer and /// App Key they belong to. #[derive(Deserialize)] struct SavedIndex { indexer: String, app_key: String, position: Option<Position>, records: HashMap<Hash256, Record>, } /// Writes the index and cursor to `path` in one step, so they always match. fn save_index( path: &str, indexer: &str, app_key: &str, index: &HashMap<Hash256, Record>, position: Option<Position>, ) -> std::io::Result<()> { let saved = serde_json::json!({ "indexer": indexer, "app_key": app_key, "position": position, "records": index, }); let tmp = format!("{path}.tmp"); std::fs::write(&tmp, serde_json::to_vec(&saved)?)?; std::fs::rename(&tmp, path) // replaces the old file in one step } /// Reads the index and cursor saved for this indexer and App Key. Returns /// an empty index and no cursor if there's no saved state, or if it /// belongs to a different indexer or App Key, so the next sync rebuilds. fn load_index( path: &str, indexer: &str, app_key: &str, ) -> (HashMap<Hash256, Record>, Option<Position>) { let saved = std::fs::read(path) .ok() .and_then(|bytes| serde_json::from_slice::<SavedIndex>(&bytes).ok()); match saved { Some(saved) if saved.indexer == indexer && saved.app_key == app_key => { (saved.records, saved.position) } _ => (HashMap::new(), None), } }

A JSON file is fine for a few thousand objects, but it's rewritten in full on every save. For larger indexes, keep the records in a database such as SQLite, or IndexedDB in a browser, and write each sync's changes and the new cursor in one transaction. If you can't use a transaction, write the records first and the cursor second: a cursor that lags behind the records only means some events get applied twice, but a cursor that runs ahead of them skips events for good.

Keep the index up to date

Load the saved index when your app starts, then sync on a timer to pick up changes from other devices and apps that share the same App Key, saving after each successful sync. This loop syncs every 30 seconds and keeps the old cursor when a sync fails:

rust
const STATE_FILE: &str = "sia-index.json"; let app_key = sdk.app_key().public_key().to_string(); let (mut index, mut position) = load_index(STATE_FILE, INDEXER_URL, &app_key); let mut ticker = tokio::time::interval(std::time::Duration::from_secs(30)); loop { ticker.tick().await; match sync_index(&sdk, &mut index, position).await { Ok(next) => { position = next; if let Err(e) = save_index(STATE_FILE, INDEXER_URL, &app_key, &index, position) { eprintln!("save failed, retrying next tick: {e}"); } } Err(e) => eprintln!("sync failed, retrying next tick: {e}"), } }

An object your app has just pinned takes a second or two to appear in the feed. Add its record to the index yourself as soon as pinObject() returns. When its event arrives, the event simply overwrites that entry. Do the same for deletes.

Rebuild from scratch

To rebuild, delete the saved index, then sync with an empty index and no cursor. Loading already does this for you when the saved state is missing, can't be read, or belongs to a different indexer or App Key. Rebuild when local state can't be trusted: after storage was cleared or corrupted, or after you change what a record holds. A rebuild also drops any objects the indexer operator has blocked, since blocked objects leave the feed without a deletion event.