Switch to integration tests #1461

nielsenko · 2024-01-08T08:48:37Z

This PR migrates to using integration tests, as recommended by the flutter team.

To run the test use:

flutter test integration_test/all_tests.dart
  --dart-define=BAAS_URL=$BAAS_URL
  --dart-define=BAAS_DIFFERENTIATOR=$BAAS_DIFFERENTIATOR 
  --file-reporter=json:test-results.json 
  --suppress-analytics 
  --device-id=... # choose device here

If you run BAAS on localhost, remember that on android you should use 10.0.2.2 to reach it.

The PR also:

Runs android tests roughly 3 times faster.
Avoid test symlinks, so no duplicated files in your IDE.
Get rid of relative imports to lib folder (import '../lib/...';) following on Avoid lib symlink #1475.
Allows test reports to be generated, for flutter tests as well.

You can (and should) use --dart-define to pass BASS_URL, etc. to the host. Tests will still read these values from the shell environment if available, that is if host and driver runs on the same node (ie. linux, macos, and windows). But you need to use --dart-define when running on the emulators or real devices.

Similar data files used in tests needs to be added as assets to the flutter tests project.

Fixes: #1465

coveralls-official · 2024-01-18T12:33:38Z

Pull Request Test Coverage Report for Build 7667566156

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall first build on kn/integration_test at 87.347%

Totals
Change from base Build 7654644143:	87.3%
Covered Lines:	3721
Relevant Lines:	4260

💛 - Coveralls

nielsenko · 2024-01-25T08:47:09Z

Dropping CHANGELOG entry as this is an internal change to how we structure our tests

nielsenko · 2024-01-25T09:01:49Z

flutter/realm_flutter/tests/integration_test/all_tests.dart

+import '../../../../test/app_test.dart' as app_test;
+import '../../../../test/asymmetric_test.dart' as asymmetric_test;
+import '../../../../test/backlinks_test.dart' as backlinks_test;
+import '../../../../test/client_reset_test.dart' as client_reset_test;
+import '../../../../test/configuration_test.dart' as configuration_test;
+import '../../../../test/credentials_test.dart' as credentials_test;
+import '../../../../test/decimal128_test.dart' as decimal128_test;
+import '../../../../test/dynamic_realm_test.dart' as dynamic_realm_test;
+import '../../../../test/embedded_test.dart' as embedded_test;
+import '../../../../test/geospatial_test.dart' as geospatial_test;
+import '../../../../test/indexed_test.dart' as indexed_test;
+import '../../../../test/list_test.dart' as list_test;
+import '../../../../test/manual_test.dart' as manual_test;
+import '../../../../test/migration_test.dart' as migration_test;
+import '../../../../test/realm_logger_test.dart' as realm_logger_test;
+import '../../../../test/realm_map_test.dart' as realm_map_test;
+import '../../../../test/realm_object_test.dart' as realm_object_test;
+import '../../../../test/realm_set_test.dart' as realm_set_test;
+import '../../../../test/realm_test.dart' as realm_test;
+import '../../../../test/realm_value_test.dart' as realm_value_test;
+import '../../../../test/results_test.dart' as results_test;
+import '../../../../test/session_test.dart' as session_test;
+import '../../../../test/subscription_test.dart' as subscription_test;
+import '../../../../test/user_test.dart' as user_test;


This is still maintained by hand for now. Integration tests can run each file separately just fine, but it is prohibitively slow to build and deploy a separate app per test file.

I have created an issue to fix this: #1490

nielsenko · 2024-01-25T09:05:20Z

flutter/realm_flutter/tests/integration_test/all_tests.dart

+  group('app_test.dart', app_test.main);
+  group('asymmetric_test.dart', asymmetric_test.main);
+  group('backlinks_test.dart', backlinks_test.main);
+  group('client_reset_test.dart', client_reset_test.main);
+  group('configuration_test.dart', configuration_test.main);
+  group('credentials_test.dart', credentials_test.main);
+  group('decimal128_test.dart', decimal128_test.main);
+  group('dynamic_realm_test.dart', dynamic_realm_test.main);
+  group('embedded_test.dart', embedded_test.main);
+  group('geospatial_test.dart', geospatial_test.main);
+  group('indexed_test.dart', indexed_test.main);
+  group('list_test.dart', list_test.main);
+  group('manual_test.dart', manual_test.main);
+  group('migration_test.dart', migration_test.main);
+  group('realm_logger_test.dart', realm_logger_test.main);
+  group('realm_map_test.dart', realm_map_test.main);
+  group('realm_object_test.dart', realm_object_test.main);
+  group('realm_set_test.dart', realm_set_test.main);
+  group('realm_test.dart', realm_test.main);
+  group('realm_value_test.dart', realm_value_test.main);
+  group('results_test.dart', results_test.main);
+  group('session_test.dart', session_test.main);
+  group('subscription_test.dart', subscription_test.main);
+  group('user_test.dart', user_test.main);


Currently we need to add groups manually as new test files are added (similar to how it always was).

I have created an issue to fix this: #1490

nielsenko · 2024-01-25T09:10:42Z

flutter/realm_flutter/tests/integration_test/all_tests.dart

+Future<void> _copyBundledFile(String fromPath, String toPath) async {
+  final data = await rootBundle.load(fromPath);
+  final bytes = data.buffer.asUint8List(data.offsetInBytes, data.lengthInBytes);
+  await File(toPath).writeAsBytes(bytes);
+}
+
+void main() {
+  IntegrationTestWidgetsFlutterBinding.ensureInitialized();
+
+  // To support both dart test and flutter integration test we pass an alternative
+  // copyFile function. Remember to add any needed files as assets in pubspec.yaml.
+  configuration_test.copyFile = _copyBundledFile;


Trick used to get rid of the duplicated flavor_helpers.dart that are imported via relative imports.

nirinchev

Good stuff - only minor nits from me.

flutter/realm_flutter/tests/android/app/build.gradle

flutter/realm_flutter/tests/linux/CMakeLists.txt

flutter/realm_flutter/tests/pubspec.yaml

src/realm_dart_decimal128.cpp

test/app_test.dart

test/baas_helper.dart

test/test.dart

test/configuration_test.dart

Use copyFile function variable instead, and override runtime in driver tests

Use package imports over ../lib relative imports

.. to support both flutter drive and dart test

…e runners

* main: Switch to integration tests (#1461) Always use persistent handles when passing dart objects as userdata to callbacks (#1491) # Conflicts: # test/dynamic_realm_test.dart

github-actions bot assigned nielsenko Jan 8, 2024

nielsenko force-pushed the kn/integration_test branch 6 times, most recently from b09f48c to 0fdb91a Compare January 18, 2024 12:26

nielsenko force-pushed the kn/integration_test branch 10 times, most recently from 5e5c442 to 104dfed Compare January 24, 2024 15:30

cla-bot bot added the cla: yes label Jan 24, 2024

nielsenko force-pushed the kn/integration_test branch from 9092aaa to 0fd9cce Compare January 25, 2024 08:43

nielsenko added the no-changelog Used to skip the changelog check label Jan 25, 2024

nielsenko marked this pull request as ready for review January 25, 2024 08:47

nielsenko requested a review from nirinchev January 25, 2024 08:47

nielsenko commented Jan 25, 2024

View reviewed changes

nirinchev reviewed Jan 25, 2024

View reviewed changes

nielsenko added 4 commits January 25, 2024 21:47

Allow tests to be run as integration tests

70559cc

Adapt flutter driver tests to new structure

1f01226

Drop flavor_helpers.dart

13880dd

Use copyFile function variable instead, and override runtime in driver tests

Drop lib symlinks in tests

a82dee7

Use package imports over ../lib relative imports

nielsenko added 14 commits January 25, 2024 21:47

Use a group per test file

a3f9d34

Remove test symlink as well

3d968f3

Check both dart defined and shell defined env

4bb30fe

.. to support both flutter drive and dart test

Fix CI

219d088

Simplify dartdoc_options.yaml a bit

4f14620

Switch to integration_test

11dd617

bump compileSdkVersion to 33 for android integration tests

5f12f1a

Disable realm_map_test on Android integration tests (TEST)

563e848

Run android flutter tests on ubuntu-latest - faster, cheaper, and mor…

2d93a4f

…e runners

Enable gradle caching

a217c23

Only drop Decimal128 tests on Android (TEMPORARY)

108dc0e

Re-create platforms

1fffe38

Allow network again for macos

69d60b1

Publish Test Report from flutter integration test

20f0c80

nielsenko force-pushed the kn/integration_test branch from 0fd9cce to 446d6d9 Compare January 25, 2024 20:56

Remember to set org to io.realm when creating platforms projects

6abf65d

nielsenko force-pushed the kn/integration_test branch from 446d6d9 to 40ed4e3 Compare January 25, 2024 21:09

nielsenko added 2 commits January 25, 2024 22:16

Move copyFile

c9ea03c

Fix line-length in app_test.dart

58d8617

nielsenko force-pushed the kn/integration_test branch from 40ed4e3 to 58d8617 Compare January 25, 2024 21:16

nielsenko added 2 commits January 25, 2024 22:22

default differentiator to 'local'

1d89e66

Add full directories as assets

ca7b0b3

nielsenko requested a review from nirinchev January 26, 2024 10:08

nirinchev approved these changes Jan 26, 2024

View reviewed changes

nielsenko merged commit 517ace7 into main Jan 26, 2024
53 of 55 checks passed

nielsenko deleted the kn/integration_test branch January 26, 2024 12:13

github-actions bot locked as resolved and limited conversation to collaborators Mar 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to integration tests #1461

Switch to integration tests #1461

nielsenko commented Jan 8, 2024 •

edited

Loading

coveralls-official bot commented Jan 18, 2024 •

edited

Loading

nielsenko commented Jan 25, 2024

nielsenko Jan 25, 2024

nielsenko Jan 25, 2024

nielsenko Jan 25, 2024

nirinchev left a comment

Switch to integration tests #1461

Switch to integration tests #1461

Conversation

nielsenko commented Jan 8, 2024 • edited Loading

coveralls-official bot commented Jan 18, 2024 • edited Loading

Pull Request Test Coverage Report for Build 7667566156

💛 - Coveralls

nielsenko commented Jan 25, 2024

nielsenko Jan 25, 2024

Choose a reason for hiding this comment

nielsenko Jan 25, 2024

Choose a reason for hiding this comment

nielsenko Jan 25, 2024

Choose a reason for hiding this comment

nirinchev left a comment

Choose a reason for hiding this comment

nielsenko commented Jan 8, 2024 •

edited

Loading

coveralls-official bot commented Jan 18, 2024 •

edited

Loading