Major Android Update: Introducing Offline Support and Resolving CORS Issues

[jiongxuan]

Description

This pull request introduces offline support and resolves CORS issues in the Android application. It integrates Capacitor for better performance and a modern architecture, replacing Cordova after evaluating their differences and benefits. To ensure compatibility with existing users, a new CapacitorMainActivity is added, allowing new installations to utilize the offline capabilities without affecting the current FullscreenActivity used by existing users. Additionally, the codebase has been modularized by introducing the LaunchDecider and WebViewRequestHandler classes to manage app launch behavior and WebView request handling respectively.

Problem:

• Lack of Offline Support: The existing Android application requires an active internet connection to function, preventing users from accessing core features when offline.

• CORS (Cross-Origin Resource Sharing) Issues: The app encounters CORS-related problems, limiting its ability to interact securely with local or self-hosted resources.

• Security Concerns: There are significant security vulnerabilities related to network hijacking, which could lead to data breaches or unauthorized access if the app relies solely on external servers without proper safeguards.

---

Objective:

• Enable Offline Functionality: Allow users to utilize the Android app's core features without needing an internet connection, enhancing accessibility and reliability.

• Resolve CORS Issues: Configure WebView appropriately to eliminate CORS-related problems, ensuring secure and flexible interactions with various resources.

• Enhance Security: Implement measures to prevent network hijacking and other security threats, ensuring that data interactions remain secure even when offline.

---

User Experience:

• Enhanced Accessibility: Users can now access and use the app's functionalities even without an internet connection, making the app more reliable and user-friendly.

• Seamless Experience for Existing Users: Existing users upgrading from older versions will continue using the app without any disruptions, maintaining data integrity and consistent functionality.

• Offline Mode for New Installations: New installations will have offline capabilities enabled by default, allowing users to work without constant internet access.

• Future Migration Plans: While the new offline solution is available for fresh installations, plans are in place to develop migration strategies to transition existing users to the new offline mode without affecting their current usage.

---

Technical Changes:

• Introduced Capacitor:

  • Reasoning: Compared to Cordova, Capacitor offers better performance, a more modern architecture, and easier integration with native platforms. Capacitor provides a more robust bridge for web-native interactions, enhancing the app's capabilities and maintainability.

• Added CapacitorMainActivity:

  • Purpose: Isolates the new Capacitor-based implementation from the existing FullscreenActivity, preventing mutual interference and ensuring stable operation for both new and existing users.

• Introduced LaunchDecider Class:

  • Purpose: Manages the app's launch behavior based on whether the user performed a fresh installation or upgraded from a previous version.
  • Functionality: Utilizes SharedPreferences, PackageInfo, and file existence checks to determine the appropriate launch mode (MODE_ONLINE for existing users and MODE_OFFLINE for new installations).
  • Implementation: Ensures that users are directed to the correct activity (FullscreenActivity or CapacitorMainActivity) based on their installation status.

• Extracted WebViewRequestHandler:

  • Purpose: Centralizes the handling of WebView requests, including URL loading and request interception, to avoid code duplication across activities.
  • Functionality: Manages both shouldOverrideUrlLoading and shouldInterceptRequest methods, ensuring consistent behavior and streamlined request processing.

• Updated AndroidManifest.xml:

  • Configuration: Set FullscreenActivity as the default launcher activity and declared CapacitorMainActivity appropriately without a launcher intent filter.
  • Intent Handling: Ensured that FullscreenActivity manages the transition to CapacitorMainActivity based on the launch decision logic.

• Refactored FullscreenActivity:

  • Integration: Incorporated LaunchDecider to determine the appropriate activity to launch during onCreate.
  • Stability Fix: Added checks to prevent wvContainer from causing crashes by verifying its initialization before attempting to remove views in onDestroy.

---

Issues Resolved

• App fails to start without internet connection super-productivity-android#3
• Serve web app locally super-productivity-android#4
• CORS headers should not be required super-productivity-android#16
• Android App #183
• docker-compose.yml for self hosted sync #2309
• Major Android Update Discussion: Introducing Offline Support and Resolving CORS Issues #3527

Check List

• [ ✅ ] New functionality includes testing.
• [ ✅ ] New functionality has been documented in the README if applicable.

[johannesjo]

This is really awesome! Thank you so much for your hard work!! I've been testing this and it seems to work very well.

Some questions:

• Would you like to stick with the git submodule approach or do think moving to a mono repo is better?
• How do we best continue with this? Since this is a big change I would probably go with a "staged" role-out for the play store. Another important step would be tackling build automation to make making new releases as convenient as possible. I definitely would welcome help with this, since I have next to none experience with fastlane and other build tools.

[jiongxuan]

@johannesjo Thank you so much for your kind words and for testing the changes!

• In my current company, we've successfully used git submodules for several years, which is why I initially chose this method. However, after some research, I see that a monorepo could offer better scalability and easier dependency management. I’m still considering this and plan to switch to a monorepo soon if it proves to be the better option.

• I agree that a staged roll-out is the most prudent approach. As discussed, existing users will continue using the current setup without any changes, while new users will benefit from the new offline capabilities. This minimizes impact while allowing us to monitor and address any issues through phased testing. Given that some code changes are involved, staging the release will help ensure stability, and I’m open to feedback to make adjustments as needed.

• Since Google Play isn't supported in my country and we distribute through various manufacturers and channels, my experience with the Google Play release process is limited. I will research the best practices for build automation and releases here. I agree that ensuring the functionality is stable should come first, followed by establishing a reliable release process.

Additionally, I have been thoroughly testing these changes during the holidays and have resolved any issues that arose. I encourage you to test the PR as well to ensure everything works smoothly.

Looking forward to your feedback and any further suggestions! 🍺🎉

[johannesjo]

Btw. I encountered a small issue when using LAUNCH_MODE=2 and ONLINE_SERVICE_HOST=test-app.super-productivity.com on a real android device. Haven't digged into it, just wanted to let you know.

EDIT this just shows up on the first launch.

[jiongxuan]

After further research, I agree that a monorepo would be more suitable, especially given the context of using Capacitor. However, the transition will involve significant changes, particularly with merging the two repositories. Here’s a proposal I’m considering:

• Merge the Android Repo into the Main Project: We could merge the super-productivity-android repository into the main super-productivity project. Using git subtree, we can ensure all commit histories from the Android repo are preserved, making it easier to trace and manage past changes.
• Archive the Android Repo: Once the merge is complete and all logic is centralized in the main project, we can archive the super-productivity-android repository to prevent any accidental changes. The README would be updated to redirect contributors, issues, and PRs to the main repo. Existing issues in the old repo can remain open for gradual resolution.
• Maintain Existing Structure: Aside from this consolidation, the rest of the project structure would remain the same to minimize disruption.

Let me know your thoughts on this approach.

[jiongxuan]

I took a look at Fastlane, and it seems quite interesting. It’s definitely worth exploring further for future automation.

For now, though, I was thinking that we could proceed with manually generating the Android release APK.

This way, we can move forward in parallel: users can start testing the app early in a staged roll-out, while we continue researching and developing the automated build process at our own pace.

To generate the APK, you can run npm run dist:android:prod, and then manually sign it. Alternatively, you can run npx cap sync and continue working in Android Studio as we’ve done before.

What do you think of this approach? 🙂

[jiongxuan]

@johannesjo I tried setting LAUNCH_MODE=2 and ONLINE_SERVICE_HOST=test-app.super-productivity.com on real Android devices (Xiaomi 14 Pro - Android 14 and Huawei Mate 60 Pro - Android 12/HarmonyOS) and emulator (Pixel 8 - Android 14) but couldn’t reproduce the issue.

In fact, when LAUNCH_MODE is set to 2, it forces a switch to the new mode, which doesn’t involve ONLINE_SERVICE_HOST, so it might not be related to ONLINE_SERVICE_HOST.

I’m concerned that another issue may be causing this. Could you please share the Java crash stack trace there? That would help me better investigate the problem.

Btw. I encountered a small issue when using LAUNCH_MODE=2 and ONLINE_SERVICE_HOST=test-app.super-productivity.com on a real android device. Haven't digged into it, just wanted to let you know.

EDIT this just shows up on the first launch.

[johannesjo]

When the error occurred yesterday logcat wasn't showing anything interesting unfortunately. Not sure if the debugger was attached when the error was thrown though.
I tried to dig a little deeper today, but I am also now unable to reproduce this. Since I am using test-app.super-productivity.com with debug android releases on my own phone, it might have been some special edge case that won't affect normal users, so we probably don't have to worry too much.

Thank you for outlining the approach to merge this into one. I think that's exactly how we should do it.

I have to say that I am really impressed by how you approach these problems and how clean and well documented everything is. Your work will have a big impact on how we can move the project forward. I can't stress enough how much I appreciate this! Thank you!

How would you like to proceed? Can I start merging this soon or are there any additional changes you'd like to make first?

[jiongxuan]

Thank you so much for your kind words. Like you, I have a deep passion for programming and contributing to open-source projects. It’s truly inspiring to see how much time and effort you’ve put into this fantastic software, which I use daily myself. I’m honored to be a part of this, and I believe we’re helping each other achieve something great.

Regarding the PR:

Since this PR has been thoroughly tested and validated, I believe it’s ready to be merged. We can proceed with that and manually compile a Release APK for users to start testing right away.

In the coming days, I plan to open two more PRs:

• The first will focus on migrating from git submodules to a monorepo structure, as I previously outlined (merging the super-productivity-android repo into the main one using git subtree to preserve commit history).

• The second will introduce support for automated APK builds. I’ve noticed that Fastlane is already integrated into the android project, and I’m currently looking into how to fully incorporate it into the main project to ensure APKs can be generated automatically. I’ll reflect these updates in the new PR. For now, we can proceed with generating and releasing APKs using the existing method.

Regarding the Java crash issue, I’ll keep an eye on it. If you encounter it again, please share the Java stack trace, and I’ll do my best to take a look when I have time.

Once again, thank you for everything, and I look forward to our continued collaboration!

EDIT: I just submitted a new git commit to update documentation for Connectivity-Free Mode and terminology improvements. There is no change to the code logic, so it can be merged directly.

[johannesjo]

@jiongxuan I encountered an issue and was hoping you might help with it:
#3569

[jiongxuan]

Thanks for bringing this up!

For now, let’s hold off on releasing the new version until we’ve fully resolved this issue. I’ll work on verifying the problem and try to fix it as soon as possible.

Also, thank you for your thorough testing, and I appreciate your efforts to test it further.

@jiongxuan I encountered an issue and was hoping you might help with it: #3569

[jiongxuan]

Regarding the two PRs I mentioned earlier, it seems like you’ve already implemented most of the changes. Is there anything else that still needs to be done, or should we continue waiting to verify the outcome?

Additionally, I found a few issues that might be related. If the validation goes well, we could consider marking these as resolved (closed):

• App fails to start without internet connection super-productivity-android#3
• Serve web app locally super-productivity-android#4
• docker-compose.yml for self hosted sync #2309
• Major Android Update Discussion: Introducing Offline Support and Resolving CORS Issues #3527
• Issues with new android build #3569

This could help keep everything tidy and up to date.

One more thing: If possible, we could also consider marking the super-productivity-android project as archived. This would help prevent unnecessary PRs and ensure contributors and devs understand that the focus has shifted to the main project.

[johannesjo]

No, I think it's done for now. I'm very happy about it! Next step for me would be to setup fastlane to make releasing quicker and hopefully easier.

The next big subject besides creating an iOS app will be decoupling the issue providers integrations from projects and to add an inbox feature along with it. Do you have any thoughts or preferences about this? I'm still in the conceptual phase.

[jiongxuan]

I’m glad to hear you’re happy with the progress! Regarding the next steps, I think focusing on “from scratch” features should take priority—things like the iOS app and Android offline notifications, which address core pain points, would be more impactful.

For the Issue Provider integration, I agree that it could benefit from a redesign. It does feel a bit bloated, but the exact approach needs more thought. As for the Inbox, the current version works well for me, though I’d be interested to hear feedback from other users.

In fact, I believe the app already has a lot of advanced features, and perhaps some could be hidden to streamline the user experience and attract a broader audience.

We could consider opening a discussion on this to gather more user feedback—what do you think?

[johannesjo]

Thank you very much for your feedback. I did exactly that. Should have thought about that before! :D