Organice Android local storage access in android native app

[ieugen]

This issue is to enable organice running as a native Android app to provide access to local files.
IOs should be added later, but is out of scope for this (I don't know IOs).

Workflow should be like this:

• User opens the native file picker dialog to choose a directory where org files live
• (optional) organice will store that location for future use
• organice will read and write org files to that location - the same flow as for web

I believe native android with local storage is achievable.

There is prior work on organice develop branch.
It uses capacitor-js to build a native Android version of organice.
organice runs in a web view and should be able to access native Android (and IOs) filesystem API's via capacitor plugins.

I tried to implement the code with some success using https://github.com/ionic-team/capacitor-plugins/tree/main/filesystem plugin and https://github.com/capawesome-team/capacitor-file-picker .

I managed to create and read files but not control exactly where to put them and also did not manage to use the right native open file dialog (use SD card or other providers) .

I talked with one friend who is familiar with Android dev and also did some checking and I published my findings as comments on

• feat(file-picker): add pickDirectory(...) method capawesome-team/capacitor-plugins#16
• feat: On Android I can't select files on SD card - I see only Photos and Google drive capawesome-team/capacitor-file-picker#60

I think the easiest solution for us long term would be to implement our own capacitor plugin with and API specific to organice.
I believe the Android Java code will be easy to maintain and port to something else then capacitor if there is ever the need.
Also the API surface is small.

The downside of this approach is that the IOs interface should be developed separately (by someone who knows this part)

Related issues:

• Create a plugin: https://capacitorjs.com/docs/plugins/creating-plugins
• Android API's to use https://developer.android.com/guide/topics/providers/document-provider
• Android storage client example code: https://github.com/android/storage-samples/blob/main/StorageClient/Application/src/main/java/com/example/android/storageclient/StorageClientFragment.java
• Provide access to local files - filesystem backend - use web File API #921
• Local File System Adapter #752
• Allow syncing to local filesystem / new sync target #730
• Full offline support #399

[ieugen]

Using Android Storage Access Framework (SAF) we can access any app that implements DocumentProvider.
Thins like Google Drive, NextCloud, DropBox files should be accessible via local filesystem access.

Some of the features offered by the SAF are as follows:
Lets users browse content from all document providers, not just a single app.
Makes it possible for your app to have long term, persistent access to documents owned by a document provider. Through this access users can add, edit, save, and delete files on the provider.
Supports multiple user accounts and transient roots such as USB storage providers, which only appear if the drive is plugged in.

We should use ACTION_OPEN_DOCUMENT_TREE : https://developer.android.com/training/data-storage/shared/documents-files#grant-access-directory

When using ACTION_OPEN_DOCUMENT_TREE, your app gains access only to the files in the directory that the user selects.
Your app can then access any file in the selected directory and any of its sub-directories.

There are some documented restrictions, but I think it does not affect normal user usage:
https://developer.android.com/training/data-storage/shared/documents-files#grant-access-directory

[ieugen]

Capacitor allows us to call an intent with result (e.g. ACTION_OPEN_DOCUMENT_TREE) quite easy:
https://capacitorjs.com/docs/plugins/android#intents-with-results

[ieugen]

I found 2 guides that describe how to use SAF to list and open files.

• https://medium.com/swlh/sample-for-android-storage-access-framework-aka-scoped-storage-for-basic-use-cases-3ee4fee404fc
• https://mateuszteteruk.pl/how-to-use-android-storage-access-framework-with-example

I managed to implement directory picker and directory listing.
See animation bellow:

[danielkrajnik]

Is the native android client still being developed? It would be much easier to use.

[ieugen]

@danielkrajnik : I use a build I did for grocerie shopping, to check off things I bought.
So yes.
I do not use it for much else.

You can build it yourself or install the APK https://github.com/ieugen/organice/releases/tag/android-local-fs.v1 .
Please report how it works so we might get the PR merged.
If I remember, that was one of the conditions ?! cc @munen

[danielkrajnik]

Fantastic, thank you for pointing me to the build. I will give it a try.

I understand that this is the only way for organice to access local files without involving external servers? Have you tried by any chance running a simple webdav server locally (for example dufs via termux) and access them from a locally served react app?

[danielkrajnik]

Sorry, but how do you build this .apk? I couldn't find it mentioned anywhere and I'm not quite sure how it's usually done with react.

[ieugen]

@danielkrajnik : Follow the build instructions: https://github.com/ieugen/organice/tree/android-local-fs#native-app-on-android .
You might have to use the versions there (node 14.x ?!) .

[ieugen]

@danielkrajnik : I did not try local webdav server.
My goal was to use oraganice when I don't have internet / internet is spotty.
When using over internet and link is slow it does not work well at all.

[danielkrajnik]

thank you very much, I've totally missed that. Node version as of June 2023 is 12.13.1. Really curious why it's so old (I'm not a web dev). Will try it out.

Regarding the webdav I was thinking of running it locally on android (localhost from termux), so no internet connection would be needed.

[ieugen]

@danielkrajnik : Share your setup.
I wonder if the webdav server could be packaged as part of organice build so you don't need external services.

[danielkrajnik]

local webdav server is the only thing that I've got working so far:
dufs -A -a admin:123@/:rw -b 127.0.0.1 -p 8008 --tls-cert cert.pem --tls-key key.pem shared/Notes

using dufs

you need to generate TLS certificates:
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -sha256 -days 3650 -nodes -subj "/C=US/ST=Oregon/L=Portland/O=Company Name/OU=Org/CN=localhost." -addext "subjectAltName = DNS:localhost"

However, I couldn't get organice serve on localhost due to this old nodejs version unavailable on native termux. It may work on proot, but haven't got to it yet. I'm reporting updates in this issue by the way.

I suspect that easiest thing to do would be to bundle it up as static assets and use some lightweight http server, but I'm not sure how to do it (I think you need to npm eject react app and then bundle it)

[ERamseth]

@ieugen tried your local storage build. Seemed to work to an extent. Doesn't seem to save preferences (color schemes, default delay setting) or capture templates. Is this expected? I think organice is supposed to save these to the sync directory?

[ieugen]

Feel free to hack on it @ERamseth .
I am not using this.
Switched to orgzly.
Seems to be a better fit for my use case.
Also I did no manage to get this merged upstream so I decided to drop it.

[danielkrajnik]

I ended up using orgzly(-revived) as well. It works okay as long as you stick to specific notebooks (it still doesn't delete notebooks if they were deleted in an upstream location)

[ERamseth]

@ieugen @danielkrajnik yeah I tried orgzly as well. In the end I went back to todo.txt org-mode to heavy for my needs I think. And I don't really like the way it handles recurring todo items. To each their own of course.

[pjhfggij]

@ERamseth funny for you to say that now as I'm just experiencing problems with scheduling recurring events in orgzly...

[ERamseth]

@pjhfggij I didnt necessarily have problems scheduling. I just don't like how when you "check off" a recurring to-do... it doesn't really "check off" it just adds some sub-entry/metadata about the last time it was checked off. Functionally it's fine I guess, but visually, I prefer how todo.txt handles it.

[pjhfggij]

@ERamseth what do you mean by "check-off"? I tend to just remove the notification when a recurring task is due without modifying it in any way (at least if they worked).

I prefer how todo.txt handles it

Interesting, I've never heard of todo.txt before. Thanks.

[ERamseth]

@ERamseth what do you mean by "check-off"?

So, recurring tasks are sort of an extension to the core todo.txt syntaxes, but many editors support them.

Basically when you "check off" a recurring task in todo.txt it gets crossed off your list and a new line is added with updated due date and the old line is still there, just checked off (or crossed off).

Far as I could tell, on organice (and other org mode editors), if the recurrence works at all, it just adds some Metadata to the entry about when it was last done and updates the due date.

Mostly a matter of preference, but I do think the way organice was handling it was not super great because i found myself "checking off" tasks multiple times, not realizing it registered the tap to check it off.

[unhammer]

Some visual confirmation on switching back to TODO would be nice.

On desktop I have a tri-state #+SEQ_TODO: TODO HOLD DONE which makes it easy to see, but it seems (this build of) organice doesn't support that (doesn't go back to TODO with that)