Cannot Open Realm in Background Isolate

[rsanford-hh2]

What happened?

See #1055 for a similar issue...

I'm trying to follow the doc from here : https://www.mongodb.com/docs/realm/sdk/flutter/sync/sync-multiple-processes/...

The basic instructions from that are to have a single realm using the flexible sync config and then any other workers using a disconnected sync configuration.

I'm trying to do that and am getting the following exception.
RealmException (RealmException: Error opening realm at path /Users/ebenezer/.hh2mobile.realm. Error code: 2013 . Message: Realm at path '/Users/ebenezer/.hh2mobile.realm' already opened with different sync configurations.)

Repro steps

Look at the code in the code snippet below. It does the following

• Starts up a flexible sync realm
• adds subscriptions to listen for changes to a collection of objects
• spawns an isolate that will open the same realm with a disconnected sync config
  • the isolate will loop to scan for new items to process
• the main thread will loop and run a query every so often

Note that the way the writeRealm is initialized is the same as the issue that I mention in the "What happened?" section except that reporter used a memory config where I'm using a disconnected sync config.

What am I doing wrong?

Version

Dart SDK version: 3.2.3 (stable) (Tue Dec 5 17:58:33 2023 +0000) on "macos_arm64"

What Atlas Services are you using?

Atlas Device Sync

What type of application is this?

Dart standalone application

Client OS and version

macOS Sonoma 14.2

Code snippets

import 'dart:async';
import 'dart:io';
import 'dart:isolate';

import 'package:path/path.dart' as path;
import 'package:realm_dart/realm.dart';

import 'package:mongodart/realmmodels/models.dart';

String get _homeDir => Platform.environment['HOME'] ?? (Platform.environment['APPDATA'] != null ? path.join(Platform.environment['APPDATA']!, 'Local') : null) ?? Platform.environment['USERPROFILE'] ?? '.';
String get _realmPath => path.join(_homeDir, '.hh2mobile.realm');

final schemaObjects = [RealmReceipt.schema, RealmFileRecordLinkChunk.schema, RealmCreditCard.schema, RealmFileRecordLink.schema, RealmUser.schema];
final writeConfig = Configuration.disconnectedSync(schemaObjects, path: _realmPath);
final writeRealm = Realm(writeConfig);

class _tusConfig {
  _tusConfig(this.endpoint, this.cachePath, this.port);

  final String endpoint;
  final String cachePath;
  final SendPort port;
}

Future<void> _tusUploader(_tusConfig config) async {
  final p = ReceivePort();
  config.port.send(p.sendPort);

  do {
    final frl = writeRealm.all<RealmFileRecordLink>().where((frl) => frl.tusEndpoint != null && frl.completedOn == null);

    for (var r in frl) {
      // do some work here... but it doesn't really matter...
    }

    await Future.delayed(Duration(seconds: 60));
  } while (await p.isEmpty);
}

Future<void> _subscribeForSync<T extends RealmObject>(Realm realm) async {
  final subName = T.toString();
  final q = realm.all<T>();
  realm.subscriptions.update((mutableSubscriptions) {
    mutableSubscriptions.add(q, name: subName);
  });
}

Future<void> main() async {
  final app = App(AppConfiguration('devicesync-kjdtl'));
  final user = await app.logIn(Credentials.anonymous());
  final syncConfig = Configuration.flexibleSync(user, schemaObjects, path: _realmPath);
  final syncRealm = Realm(syncConfig);

  // subscribe to the data elements that we care about
  await _subscribeForSync<RealmUser>(syncRealm!);
  await _subscribeForSync<RealmReceipt>(syncRealm!);
  await _subscribeForSync<RealmCreditCard>(syncRealm!);
  await _subscribeForSync<RealmFileRecordLink>(syncRealm!);

  final p = ReceivePort();
  final cachePath = path.join(_homeDir, '.tuscache');
  final tusConfig = _tusConfig("https://tusd.tusdemo.net/files/", cachePath, p.sendPort);
  await Isolate.spawn(_tusUploader, tusConfig);

  final sendPort = await p.first as SendPort;

  var keepRunning = true;
  do {
    final receipts = syncRealm.all<RealmReceipt>();
    if (receipts.isNotEmpty) {
      final first = receipts.first;
    }
    await Future.delayed(const Duration(seconds: 5));
  } while (keepRunning);

  sendPort.send(null);
}

Stacktrace of the exception/crash you're getting

No response

Relevant log output

No response

[nirinchev]

This documentation explains how to open the file from multiple processes. In your case, you are opening it from the same process from different isolates. In this case, you should open it with the same config as the one you used to open it on the main isolate. Note that due to #1433, that won't work with 1.6.0, so you either need to downgrade or wait until 1.7.0 is released.

[rsanford-hh2]

Okay... any idea what the timeframe for 1.7 is?

[nirinchev]

I don't imagine it'll land before the holidays, so realistically - the first half of January.

[richard457]

Any update on this?

[nielsenko]

@richard457 Waiting for #1470. I expect us to release this week.

[richard457]

when this goes to pub.dev?

[nielsenko]

@richard457 1.7 is out

[richard457]

I have tested this out with the following code:

static Future handleRealm(List<dynamic> args) async {
    final rootIsolateToken = args[0] as RootIsolateToken;
    final sendPort = args[1] as SendPort;
    final branchId = args[2] as int;
    final encryptionKey = args[3] as String;
    BackgroundIsolateBinaryMessenger.ensureInitialized(rootIsolateToken);
    // get isar instances
    isar = await isarK();
    final app = App.getById(AppSecrets.appId);
    final user = app?.currentUser!;
    List<int> key = encryptionKey.toIntList();
    final config = Configuration.flexibleSync(
      user!,
      [
        RealmITransaction.schema,
        RealmITransactionItem.schema,
        RealmProduct.schema,
        RealmVariant.schema,
        RealmStock.schema,
        RealmIUnit.schema
      ],
    );

    final realm = Realm(config);
    // final realm = await Realm.open(config);

    // Subscribe to changes for transactions
    final iTransactionsCollection =
        realm.query<RealmITransaction>(r'branchId == $0', [branchId]);

    iTransactionsCollection.changes.listen((changes) async {
      for (final result in changes.results) {
        final model = createmodel(result);

        if (model.action == AppActions.deleted && model.deletedAt == null) {
          model.deletedAt = DateTime.now();
        }
       sendPort.send('data ${model.id}');
      }
    });
}

and Trigered the function like

  @override
  Future<void> pull() async {
    RootIsolateToken? rootIsolateToken = RootIsolateToken.instance;
    int branchId = ProxyService.box.getBranchId()!;
    String key = ProxyService.box.encryptionKey();
    ReceivePort receivePort = ReceivePort();
    await Isolate.spawn(IsolateHandler.handleRealm, [
      rootIsolateToken,
      receivePort.sendPort,
      branchId,
      key,
    ]);
    receivePort.listen((message) {
      log('Isolate: $message');
    });
  }

I can confirm that on Windows the above code works but not on Android device, when there is a data change the sendPort.send('data ${model.id}'); is not triggered on android @nielsenko

[nielsenko]

@richard457 Remember to hold onto the stream subscription so that you can call cancel when you are done with it.

final subscription = iTransactionsCollection.changes.listen(...
// and later
await subscription.cancel(); 

This holds for all streams, not just realm change streams. Otherwise you risk leaking native resources, such as file handles, etc. You can also end up in situations where the subscription is the only thing that prevents the entire stream from being garbage collected.

In general I recommend using await for instead of raw listen to avoid oversights such as this.

await for (final t in iTransactionsCollection.changes.listen(...)) {
  // dart will ensure subscription is cancelled no matter how you exit from here
}

but that is not always possible.

Realm actually tries to help mitigate this issue by using weak handles, and while debugging this, I found a bug that I'm fixing as I write this, so stay tuned..

However - after the fix - it becomes even more important to always remember to cancel the stream subscription, as you are otherwise certain to both leak native- and pin dart- resources.

[richard457]

Thanks, @nielsenko but I find on Android that data changes are not reflected when the connection to the server (flexible sync) is established while on Windows changes are detected automatically, I mean I have two device

1. Windows which makes changes
2. Android the receiver of change
   Please shed more light on this as I can not get the latest android data when there are new changes..

[nielsenko]

@richard457 Could you setup logging and tell us what you see?

Realm.logger.level = RealmLogLevel.trace;

on both windows and android?

Also,

• How does your models look?
• How does your flexible sync subscriptions look?
• Do you see the data in Atlas?

[richard457]

My model

@RealmModel()
class _RealmITransaction {
  late String id;
  @PrimaryKey()
  @MapTo('_id')
  late ObjectId realmId;
  late String reference;
  String? categoryId;
  late String transactionNumber;
  late int branchId;
  late String status;
  late String transactionType;
  late double subTotal;
  late String paymentType;
  late double cashReceived;
  late double customerChangeDue;
  late String createdAt;
}

 final app = App(AppConfiguration(AppSecrets.appId,
        baseUrl: Uri.parse("https://services.cloud.mongodb.com")));
    final user = app.currentUser ?? await app.logIn(Credentials.anonymous());
    List<int> key = ProxyService.box.encryptionKey().toIntList();
    final config = Configuration.flexibleSync(
      user,
      [
        RealmITransaction.schema,
      ],
      // encryptionKey:key,
      path: await absolutePath("db_"));
      realm = Realm(config);

Yes I am able to see data in atlas when data are saved from desktop app.

[nielsenko]

And how does your flexible sync subscriptions look?

realm.subscriptions.update((ms) {
  // what are you subscribing on?
});

.. and what does the logging tell you?

[richard457]

final transaction =
        realm.query<RealmITransaction>(r'branchId == $0', [branchId]);
    //https://www.mongodb.com/docs/realm/sdk/flutter/sync/manage-sync-subscriptions/
    realm.subscriptions.update((sub) {
      sub.clear();
      sub.add(transaction, name: "transactions-${branchId}", update: true);
);

FYI: I am only doing subscription from main tread not in isolate.
Getting this in log:

I/flutter ( 5150): 2024-01-26T11:23:11.276052: [INFO] Realm: Connection[1]: Session[1]: Binding '/data/user/0/rw.app/app_flutter/app-sync/db_' to ''
I/flutter ( 5150): 2024-01-26T11:23:11.278213: [INFO] Realm: Connection[1]: Session[1]: client_reset_config = false, Realm exists = true 
I/flutter ( 5150): 2024-01-26T11:23:11.279519: [INFO] Realm: Connection[1]: Connecting to 'wss://ws.ap-south-1.aws.realm.mongodb.com:443/api/client/v2.0/app/devicesync-ifwtd/realm-sync'
I/flutter ( 5150): 2024-01-26T11:23:11.626326: [INFO] Realm: Connected to endpoint '15.207.151.40:443' (from '10.110.0.187:48752')
I/flutter ( 5150): 2024-01-26T11:23:13.377412: [INFO] Realm: Connection[1]: Connected to app services with request id: "65b37a0123d017c54b8e3f4c"
I/flutter ( 5150): 2024-01-26T11:23:21.771668: [INFO] Realm: Connection[1]: Session[1]: Begin processing pending FLX bootstrap for query version 12. (changesets: 1, original total changeset size: 0)
I/flutter ( 5150): 2024-01-26T11:23:21.783177: [INFO] Realm: Connection[1]: Session[1]: Integrated 1 changesets from pending bootstrap for query version 12, producing client version 159 in 10 ms. 0 changesets remaining in bootstrap

[nielsenko]

The code bits I have seen makes me question if you are using the same realm in both the foreground and the background isolate. I see a path passed to AppConfiguration in one place, but not the other. It is important that the realm you add the subscription to, and the realm you listen for changes on, are actually backed by the same realm file..

[nielsenko]

It is okay to setup the subscription(s) in one isolate, and query/observe the realm in another, but it must be backed by the same physical realm file.

Try to print the realm.config.path in both isolates.

[richard457]

Sorry for late reply, I realized that not setting path in isolate was the root cause of the issue.