Sync to iCloud with CKSyncEngine

Speakers: Tim Mahoney and Aamer Husain, CloudKit Engineers

The state of sync

• Sync is expected

• Sync is not magic

• Simpler is often better

APIs

• NSPersistentCloudKitContainer: Full-stack solution that includes local persistence.

• NEW: CKSyncEngine: Bring your own local persistence.

• CKDatabase and CKOperation: More fine-grained control.

… if you want to sync with CloudKit, and if you’re not using NSPersistentCloudKitContainer, you should use CKSyncEngine. Sync involves many moving parts, and using a higher level API like CKSyncEngine can help reduce complexity and improve your app’s sync experience.

Meet CKSyncEngine

When you use CKSyncEngine, the amount of sync code you have to write becomes much smaller and more focused. You only have to handle the things that are specific to your app, and the sync engine handles the rest.

• Convenient but flexible

• Private and shared data

• Backward compatible

• Used by many Apple apps (including Freeform)

• NSUbiquitousKeyValueStore was rewritten on top of the sync engine

Already have a custom CloudKit sync implementation?

• Consider switching

• Less maintenance

• Future enhancements

• Smaller API

• Submit feedback if you are missing features

General flow of operation

On saving data to CloudKit:

Generally, the sync engine acts as a conduit of data between your app and the CloudKit server. Your app communicates with the sync engine in terms of records and zones.

When there are changes to save, your app gives them to the sync engine. When it fetches these changes on another device, it gives them to your app. That said, when the sync engine has work to do, it doesn’t always do it immediately.

If it needs to communicate with the server, it’ll first consult with the system task scheduler. This is the same scheduler used across the OS for background task management, and it makes sure that the device is ready to sync.

Once the device is ready, the scheduler runs the task, and the sync engine talks with the server.

This is the basic flow of operation for the sync engine. More specifically, what does it look like when the sync engine sends changes to the server?

• First, someone makes a modification to the data. Maybe they typed something or they flipped a switch or deleted an object.

• Then, your app tells the sync engine that there’s a pending change to send to the server. This lets the sync engine know that it has work to do.

• Next, the sync engine submits a task to the scheduler. Once the device is ready, the scheduler runs the task.

• When the task runs, the sync engine starts the process of sending changes to the server. In order to do that, it asks your app for the next batch of changes to send.

  • If someone made a single modification, you might only have one pending change.

  • However, if someone imports a huge database of new data, you might have hundreds or thousands of changes.

• Since there’s a limit on how much can be sent to the server in a single request, the sync engine asks for these changes in batches.

  • This also helps to reduce memory overhead by not bringing any records into memory until they’re actually needed.

• After you provide the next batch, the sync engine sends it to the server.

• The server responds with the result of the operation, including any information about the success or failure of these changes.

• Once the request finishes, the sync engine calls back to your app with the result.

• This is your opportunity to react to the success or failure of the operation.

• If you have any more pending changes, the sync engine will continue to ask for batches until there’s nothing left to send.

On receiving data from CloudKit:

When the server receives a new change, it sends a push notification to the other devices that have access to that data.

CKSyncEngine automatically listens for these push notifications in your app.

• When it receives a notification, it submits a task to the scheduler.

• When the scheduler task runs, the sync engine fetches from the server.

• When it fetches new changes, it gives them to your app. This is your chance to persist these changes locally and show them in the UI.

And that’s the basic flow of operations when using the sync engine.

The scheduler

• Enables automatic sync

• Monitors system condition

• Ensures proper balance between user experience and device resources

• Often fast

Trust the scheduler!

• Rely on automatic sync

• Easier, more efficient

• Manual sync when necessary

• Testing with manual sync

In general, we recommend that you rely on automatic sync scheduling. But, we understand that there are valid use cases for manual syncing, and the sync engine has API to do that when necessary.

Getting started

• Learn about the fundamental types in CloudKit: CKRecord and CKRecordZone.

• Enable the CloudKit capability in your Xcode project.

• Enable the Remote Push notifications capability in your Xcode project as the sync engine relies on push notifications.

• Initialize CKSyncEngine on app launch to start listening for push notifications and scheduler tasks.

  • Provide an object which conforms to the CKSyncEngineDelegate.

  • Provide the last known version fo the sync engine state.

  • The delegate should persist the Event.stateUpdate when received, for next app launch.

actor MySyncManager : CKSyncEngineDelegate {
    
    init(container: CKContainer, localPersistence: MyLocalPersistence) {
        let configuration = CKSyncEngine.Configuration(
            database: container.privateCloudDatabase,
            stateSerialization: localPersistence.lastKnownSyncEngineState,
            delegate: self
        )
        self.syncEngine = CKSyncEngine(configuration)
    }
    
    func handleEvent(_ event: CKSyncEngine.Event, syncEngine: CKSyncEngine) async {
        switch event {
        case .stateUpdate(let stateUpdate):
            self.localPersistence.lastKnownSyncEngineState = stateUpdate.stateSerialization
        }
    }
}

Using CKSyncEngine

• Add changes in CKSyncEngine.State.

• Implement the delegate method nextRecordZoneChangeBatch.

• Handle change events: sentDatabaseChanges and sentRecordZoneChanges.

Sending changes to the server

func userDidEditData(recordID: CKRecord.ID) {
    // Tell the sync engine we need to send this data to the server.
    self.syncEngine.state.add(pendingRecordZoneChanges: [ .save(recordID) ])
}

func nextRecordZoneChangeBatch(
    _ context: CKSyncEngine.SendChangesContext, 
    syncEngine: CKSyncEngine
) async -> CKSyncEngine.RecordZoneChangeBatch? {

    let changes = syncEngine.state.pendingRecordZoneChanges.filter { 
        context.options.zoneIDs.contains($0.recordID.zoneID) 
    }

    return await CKSyncEngine.RecordZoneChangeBatch(pendingChanges: changes) { recordID in
        self.recordToSave(for: recordID)
    }
}

Fetching changes from the server

• CKSyncEngine automatically fetches

• Handle fetch events: fetchedDatabaseChanges and fetchedRecordZoneChanges.

• Handle optional events: willFetchChanges and didFetchChanges.

  • Handing these events may be useful if you want to perform any setup or cleanup tasks before or after fetching changes.

func handleEvent(_ event: CKSyncEngine.Event, syncEngine: CKSyncEngine) async {
    switch event {
        
    case .fetchedRecordZoneChanges(let recordZoneChanges):
        for modifications in recordZoneChanges.modifications {
            // Persist the fetched modification locally
        }

        for deletions in recordZoneChanges.deletions {
            // Remove the deleted data locally
        }

    case .fetchedDatabaseChanges(let databaseChanges):      
        for modifications in databaseChanges.modifications {
            // Persist the fetched modification locally
        }
      
        for deletions in databaseChanges.deletions { 
            // Remove the deleted data locally
        }

    // Perform any setup/cleanup necessary
    case .willFetchChanges, .didFetchChanges:
        break
      
    case .sentRecordZoneChanges(let sentChanges):

        for failedSave in sentChanges.failedRecordSaves {
            let recordID = failedSave.record.recordID

            switch failedSave.error.code {

            case .serverRecordChanged:
                if let serverRecord = failedSave.error.serverRecord {
                    // Merge server record into local data
                    syncEngine.state.add(pendingRecordZoneChanges: [ .save(recordID) ])
                }
            
            case .zoneNotFound: 
                // Tried to save a record, but the zone doesn't exist yet.
                syncEngine.state.add(pendingDatabaseChanges: [ .save(recordID.zoneID) ])
                syncEngine.state.add(pendingRecordZoneChanges: [ .save(recordID) ])
             
            // CKSyncEngine will automatically handle these errors
            case .networkFailure, .networkUnavailable, .serviceUnavailable, .requestRateLimited:
                break
              
            // An unknown error occurred
            default:
                break
            }
        }
      
    case .accountChange(let event):
        switch event.changeType {

        // Prepare for new user
        case .signIn:
            break
          
        // Delete local data
        case .signOut:
            break
          
        // Delete local data and prepare for new user
        case .switchAccounts: 
            break
        }
    }
}

Error handling

• CKSyncEngine handles transient errors

• Automatic retry

• You handle application errors

• Re-schedule sync to retry

func handleEvent(_ event: CKSyncEngine.Event, syncEngine: CKSyncEngine) async {
    switch event {
        
    case .fetchedRecordZoneChanges(let recordZoneChanges):
        for modifications in recordZoneChanges.modifications {
            // Persist the fetched modification locally
        }

        for deletions in recordZoneChanges.deletions {
            // Remove the deleted data locally
        }

    case .fetchedDatabaseChanges(let databaseChanges):      
        for modifications in databaseChanges.modifications {
            // Persist the fetched modification locally
        }
      
        for deletions in databaseChanges.deletions { 
            // Remove the deleted data locally
        }

    // Perform any setup/cleanup necessary
    case .willFetchChanges, .didFetchChanges:
        break
      
    case .sentRecordZoneChanges(let sentChanges):

        for failedSave in sentChanges.failedRecordSaves {
            let recordID = failedSave.record.recordID

            switch failedSave.error.code {

            case .serverRecordChanged:
                if let serverRecord = failedSave.error.serverRecord {
                    // Merge server record into local data
                    syncEngine.state.add(pendingRecordZoneChanges: [ .save(recordID) ])
                }
            
            case .zoneNotFound: 
                // Tried to save a record, but the zone doesn't exist yet.
                syncEngine.state.add(pendingDatabaseChanges: [ .save(recordID.zoneID) ])
                syncEngine.state.add(pendingRecordZoneChanges: [ .save(recordID) ])
             
            // CKSyncEngine will automatically handle these errors
            case .networkFailure, .networkUnavailable, .serviceUnavailable, .requestRateLimited:
                break
              
            // An unknown error occurred
            default:
                break
            }
        }
      
    case .accountChange(let event):
        switch event.changeType {

        // Prepare for new user
        case .signIn:
            break
          
        // Delete local data
        case .signOut:
            break
          
        // Delete local data and prepare for new user
        case .switchAccounts: 
            break
        }
    }
}

Transient errors

[The following are] examples of transient errors that the sync engine will handle for you. You’ll still receive these errors for your awareness but you do not need to take action in response to them. The sync engine will automatically retry for these errors when system conditions permit.

case .sentRecordZoneChanges(let sentChanges):
  case networkFailure, // The network was available but encountered an error
  case .networkUnavailable, // The network was not available
  case .serviceUnavailable, // CloudKit was not available
  case .requestRateLimited: // cloudKit has rate limited work and should be attempted later

Account changes

• CKSyncEngine monitors account status/changes

• You handle .accountChange events

case .accountChange(let event):
  switch event.changeType {
    case .signIn, // Prepare for new user
    case .signOut, // Delete local data
    case .switchAccounts: // Delete local data and prepare for new user
  }

The sync engine will not begin syncing with iCloud until there is an account present on the device.

Sharing data with other users

• Initialize a CKSyncEngine for each database

• Existing CKShare invitation/acceptance flow

let databases = [ container.privateCloudDatabase, container.sharedCloudDatabase ]

let syncEngines = databases.map {
    var configuration = CKSyncEngine.Configuration(
        database: $0,
        stateSerialization: lastKnownSyncEngineState($0.databaseScope),
        delegate: self
    )
    return CKSyncEngine(configuration)
}

See more in the Tech Talk “Get the most out of CloudKit Sharing”.

Testing and Debugging

• You can simulate device-to-device user flows using multiple CKSyncEngine instances.

• Simulate specific edge cases by setting automaticallySync = false

In this example we simulate two devices using MySyncManager. In this example, MySyncManager creates a local database and sync engine:

func testSyncConflict() async throws {
    // Create two local databases to simulate two devices.
    let deviceA = MySyncManager()
    let deviceB = MySyncManager()
    
    // Save a value from the first device to the server.
    deviceA.value = "A"
    try await deviceA.syncEngine.sendChanges()
    
    // Try to save the value from the second device before it fetches changes.
    // The record save should fail with a conflict that includes the current server record.
    // In this example, we expect the value from the server to win.
    deviceB.value = "B"
    XCTAssertThrows(try await deviceB.syncEngine.sendChanges())
    XCTAssertEqual(deviceB.value, "A")
}

Tips for debugging

• Understand the sequence of events

• Logging record/zone IDs and timestamps

• Write tests for user flows

• Look at timestamps when piecing the puzzle together

Sample code

There is sample code available at https://github.com/apple/sample-cloudkit-sync-engine.