Buckets store data as a history of changes, not only the current state.
compact
command in the Docker image can be used to compact all buckets. This can be run manually, or on a regular schedule using Kubernetes CronJob or similar scheduling functionality.
Defragmenting may still be required.
PUT
, REMOVE
, MOVE
and CLEAR
operations. In normal operation, only PUT
and REMOVE
operations are created.
A simplified view of a bucket may look like this:
MOVE
operations. This just indicates that an operation is not needed anymore, since a later PUT
or REMOVE
operation replaces the row.
After this compact step, the bucket may look like this:
CLEAR
, MOVE
and/or REMOVE
operations at the start of the bucket, and replaces them all with a single CLEAR
operation. The CLEAR
operation indicates to the client that “this is the start of the bucket, delete any prior operations that you may have”.
After this compacting step, the bucket may look like this:
CLEAR
operation can only remove operations at the start of the bucket, not in the middle of the bucket, which leads us to the next step.
powersync-service
repo for more technical details on compacting.