Skip to main content
Availability: MongoDB replication pre-filtering is available on the Team and Enterprise plans of PowerSync Cloud and in the Enterprise Self-Hosted Edition.
Replication pre-filtering limits which MongoDB documents enter PowerSync bucket storage. Use it when your instance needs only a subset of a collection, such as active records or records for a fixed set of stores. Pre-filtering can speed up initial replication: with suitable source indexes, the initial snapshot reads only matching documents. Define filters in your Sync Config under config.source_table_options. The Service applies them during the initial snapshot and ongoing change-stream replication, before evaluating Sync Streams queries. A filter applies to the whole instance, independently of which client connects.
Pre-filters do not define client authorization. Keep the Sync Streams query filters that control which documents each client can access. A document excluded by a pre-filter is unavailable to every client, even if a Sync Streams query would select it.

​
Prerequisites

  • PowerSync Service 1.28.0 or later.
  • Sync Config edition 3 (config.edition: 3).
  • MongoDB 6.0 or later. Azure DocumentDB is not supported for pre-filtering.
  • Post Images set to Auto-Configure (post_images: auto_configure) or Read-only (post_images: read_only). Pre-filtering does not support post_images: off.
  • With Auto-Configure, grant the Service permission to enable changeStreamPreAndPostImages. With Read-only, enable it yourself on every replicated collection. See MongoDB Permissions.
  • Source indexes that support your filters. Create them before deploying the Sync Config, because the initial snapshot starts after deployment.

​
Configure Pre-filtering

This example replicates only todos with an explicit archived: false field. Each client receives only its own todos.
1

Create a Source Index

Run the following in mongosh against your source database:
The filtered field narrows the snapshot query. The final _id key lets MongoDB return each snapshot page in _id order without sorting all matching documents again. See Snapshot Indexes for filters on multiple fields.
2

Add the Filter to Your Sync Config

$$doc refers to the source document. The pre-filter handles archived, so the Sync Stream query does not need to repeat that condition. The owner_id condition still controls access for each client.This filter excludes documents where archived is missing. Set archived: false explicitly on every document you want to replicate.
3

Deploy Your Sync Config

Deploy the Sync Config as you normally would. For PowerSync Cloud, use the Dashboard or CLI. For self-hosted instances, see Update Sync Streams. Adding, changing, or removing filters forces a full reprocess: the Service creates a new replication stream and snapshots all replicated collections. Allow for that source database work when planning the deployment.Validate the Sync Config before you deploy it. Use the Validate button in the Dashboard, powersync validate in the CLI, or the /api/sync-rules/v1/validate endpoint for self-hosted instances. If validation reports an inefficient snapshot index, create the recommended index before deployment. Index warnings are advisory and do not block deployment.

​
Deployment and Replication Errors

  • Non-MongoDB source: An active mongodb_filter_expression on a source such as Postgres fails Sync Config validation and blocks deployment.
  • Unsupported Post Images setting: post_images: off blocks deployment. Set Post Images to Auto-Configure or Read-only.
  • Pre-filtering no longer available: If your plan or Service edition stops supporting pre-filtering, replication stops for a saved Sync Config that contains filters, with error PSYNC_S1348. The Service does not ignore the filters. Restore support, or remove the filters and deploy a new Sync Config.

​
Supported Expressions

Filters use a restricted subset of MongoDB aggregation expressions. The supported operators compare source fields against fixed values or combine other filters: For $eq and $in, put the source field first, using $$doc.<path>. Nested fields such as $$doc.customer.tier are supported. Numeric path segments, such as $$doc.items.0, and segments that start with $ are not supported. $in lists and $and or $or operands must be nonempty. Constants can be strings, numbers, booleans, or BSON values expressed as Extended JSON (EJSON). For example, match an ObjectId with:
Supported EJSON wrappers are $oid, $date, $numberInt, $numberLong, $numberDouble, $numberDecimal, $timestamp, $binary, and $uuid. Plain integer constants must fit a signed 32-bit integer; use an explicit numeric wrapper for larger integers. For a literal string that starts with $, use $literal, such as { $literal: '$internal' }. Filters do not support comparisons between fields, computed values, null, array constants, or embedded-document constants. Operators such as $ne, $not, $gt, and $lt are not supported. Use explicit values, such as archived: false, instead of conditions such as “not true” or “field is missing.” YAML aliases are not supported inside filter expressions. Write each condition directly instead of referencing an anchor.

​
Comparison Semantics

Pre-filters use MongoDB aggregation expressions. Their comparison behavior differs from the query predicates you might use in find().

​
Arrays and Nested Paths

$eq compares the field’s whole value against a constant. $in compares that whole value against each constant in the supplied list. Neither operator searches for matching elements inside an array-valued source field: A native query such as find({ store_id: 'store-a' }) can match the array in the last row. The pre-filter excludes that document because its store_id value is an array, not one of the scalar constants. See MongoDB’s aggregation $in reference for how it compares values. When a dotted path crosses an array of documents, MongoDB collects the nested field values into an array. For example, if a document contains items: [{ sku: 'sku-a' }, { sku: 'sku-b' }], $$doc.items.sku evaluates to ['sku-a', 'sku-b']. MongoDB does this automatically when it evaluates the path. A pre-filter compares that whole array against its constant, so $eq: ['$$doc.items.sku', 'sku-a'] does not match. The supported pre-filter operators cannot express “any array element matches”; use a scalar source field for that condition.

​
Numbers and Other Types

MongoDB compares numbers by value across BSON numeric types. For example, $eq: ['$$doc.quantity', 5] matches a quantity stored as Int32, Int64, Double, or Decimal128 when its value is 5. Using { $numberLong: '5' } as the filter constant does not restrict matching to Int64 fields. Other types are not converted to numbers during comparison: Keep the source field’s type consistent with the filter constant. For example, compare an ObjectId field against an EJSON $oid constant, not its hexadecimal string. See MongoDB’s BSON comparison rules.

​
Collection Names and Wildcards

Each source_table_options key identifies a collection. Use collection, database.collection, or default.database.collection. Unqualified names use the connection’s default database. An explicit connection tag must be default. Unquoted identifiers are converted to lowercase. Double-quote identifiers to preserve case or include a dot in a collection name. Retain those double quotes in YAML by enclosing the key in single quotes: '"Todos"' identifies the collection Todos, and '"audit.events"' identifies one collection named audit.events. Use the same identifier quoting in your Sync Streams queries. For a literal double quote inside an identifier, write two double quotes. A final % matches a collection-name prefix. Database wildcards are not supported. For example:
Exact collection names override wildcards, regardless of declaration order. Among wildcards, the first matching entry wins. Put narrower wildcards before broader ones; validation rejects a wildcard made unreachable by an earlier entry. Use mongodb_filter_expression: disabled to exclude a collection from a wildcard filter. An empty options object ({}) inherits the matching wildcard filter. These entries configure collections already selected by your Sync Streams queries. Each filter entry must apply to at least one collection name or pattern referenced by those queries. The entries do not cause additional collections to replicate.

​
Snapshot Indexes

The Service does not create indexes. Without a suitable index, MongoDB can still scan many or all source documents during the snapshot, even though excluded documents do not enter bucket storage. Create indexes on the filtered fields with _id as the final key. Use collation: { locale: 'simple' }, because filtered snapshots use this collation. For example, a filter on store_id can use:
For filters on multiple fields:
  • $or needs a suitable index for every branch. An unindexed branch can cause a collection scan for the whole filter.
  • $and can use an index on one term. Prefer the most selective term. For an $and of equality conditions, a compound index on those fields followed by _id can narrow the scan further.
For an empty collection, validation checks the available index definitions. This confirms whether a compatible index exists, but it cannot confirm which query plan MongoDB will choose with data. After you populate the collection, validate the Sync Config again to check the snapshot query plan. An index on only the filtered field can require MongoDB to read and sort all matching documents for every snapshot page. Including _id avoids that repeated work. MongoDB may still choose an _id index scan when a filter matches a large fraction of the collection. For index options, see MongoDB’s createIndex() reference.

​
Snapshot Progress

For a filtered collection, the snapshot total counts only documents that match the pre-filter. The Service first tries to count matching documents within a timeout. If that count fails or times out, the Service estimates the total by sampling the collection. If sampling also fails, no total is available. The total can therefore be approximate or unavailable while the snapshot continues. An unavailable total does not indicate that replication has stopped.

​
Changes to Filtered Documents

During replication, the Service keeps stored data consistent with the pre-filter: Pre-filtering reduces stored data, but excluded changes still have some network and processing cost. MongoDB sends compact progress events for excluded changes so that the Service can advance its recovery position through a backlog of excluded writes.

​
Pre-Images and Post-Images

When a connection has pre-filters, the Service also requests pre-images on its change stream with fullDocumentBeforeChange: 'whenAvailable'. Pre-filtering uses these images to check whether an updated or deleted document previously matched the filter. The changeStreamPreAndPostImages setting that Post Images requires captures both pre-images and post-images, so pre-filtering needs no additional MongoDB setting. When adding replicated collections, enable changeStreamPreAndPostImages when you create them, as described in Post Images.