Skip to content

Commit

Permalink
revamped resources handling
Browse files Browse the repository at this point in the history
  • Loading branch information
@XilinJia
XilinJia committed Jan 7, 2024
1 parent 04d8ce7 commit 6d61e03
Show file tree
Hide file tree
Showing 16 changed files with 1,188 additions and 164 deletions.
893 changes: 893 additions & 0 deletions .yarn/releases/yarn-4.0.2.cjs
View file

Large diffs are not rendered by default.

11 changes: 7 additions & 4 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ On the desktop application, the default home directory where the notebooks and n

#### Android app

The default home directory on Android is different. On Android 9 or older, the directory is "/Android/data/ac.mdiq.xilinota/files/Xilinotas". On Android 10 or newer, the directory is chosen by you upon first start of Xilinota. These directories can not be changed at the moment. (With dev version, on Android 9 or older, the directory is "/Android/data/ac.mdiq.xilinota/files/XilinotasDev".
The default home directory on Android is different. On Android 9 or older, the directory is "/Android/data/ac.mdiq.xilinota/files/Xilinotas". On Android 10 or newer, the directory is chosen by you upon first start of Xilinota. These directories can not be changed at the moment. (With dev version, on Android 9 or older, the directory is "/Android/data/ac.mdiq.xilinota.D/files/XilinotasDev".

#### Mechanisms

Expand All @@ -40,7 +40,9 @@ You can create, update, move, delete notes or notebooks in Xilinota and the file

#### Resources

Resources (images or attachments in notes) are now located in the ".resources" sub-directory in the directory of every notebook ("_resources" sub-directory is reserved for future use). Markdown file shown in external viewer now shows the related resources. Resource files are copied when the "Xilinotas" directory first gets populated and are saved on note edit when resources are added to the note. Resources now follow the associated note, i.e., when you move/remove note, the related resource files will be similarly handled.
The whole resources folder is now relocated from the original config directory to the profile's home directory (e.g. "/home/loginname/Documents/Xilinotas/default/.resources"). The relocation of the directory is handled automatically when directory "resources" exists in the original config directory. Same as usual, this directory contains all resource files of notes associated with the profile.

In the directory of every notebook, there is a sub-directory ".resources" that contains all resources related to notes in the notebook. The resources files in this sub-directory are <mark>hard-links</mark> (on desktop) or <mark>copies</mark> (on mobile) to the ones in the whole resources folder. Markdown file shown in external viewer now shows the related resources. Resource files are automatically populated when the "Xilinotas" directory first gets populated and are saved on note updates when resources are added to a note. Resources now follow the associated note, i.e., when you move/remove note (within Xilinota), the related resource files will be similarly handled.

Supported formats of resources in notes are following (this is only for technical info and not a concern for normal usage of Xilinota application):
```
Expand All @@ -49,16 +51,17 @@ Supported formats of resources in notes are following (this is only for technica
<img width="684" height="306" src=":/f5c27bc3b7fb4116a10fbf0f1cbfefef"/>
<img width="684" height="306" src=".resources/f5c27bc3b7fb4116a10fbf0f1cbfefef.xyz"/>
```
"_resources" sub-directory in the directory of every notebook is reserved for future use.

Resource files are currently copied from Xilinota's config directory, so you have duplicate files on your system. Going forward, it appears more reasonable to have the resources close to the note files, so I'm looking into the possibilities of removing the resources directory under Xilinota's config directory. But this will be at a later stage.
Note: due to the relocation of the resources folder, if you migrate from Joplin by feeding Xilinota with renaming Joplin's config directory, and if you rename the resulting config directory back for Joplin, you will also need to manually move the resources folder back.

#### Sync of files/folders and notes/notebooks

Files and folders in the file system are sync'ed back to Xilinota. The process takes place at the start of Xilinota. With Xilinota desktop, similar to the first file population process, there is a popup with an animated bar during the sync process. In the mobile apps, this sync process runs in the background without blocking any other functions of Xilinota.

Any added or deleted note files or folders will be synced into Xilinota (an empty folder added is ignored). A markdown file if edited after the previous exit of Xilinota is also synced. Adding an external folder with markdown files will get all files synced in. Removing a folder also results in getting all notes in the folder removed from Xilinota after sync (though the notebook corresponding to the folder stays).

A positive note: if you remove the home directory, or the profile directory (e.g. default) under the home directory, or all the folders and files under the profile directory, Xilinota will not delete all of your notes and notebooks in the DB, rather it will re-populate the entire home directory.
A positive note: if you remove the home directory, or the profile directory (e.g. default) under the home directory, or all the folders and files under the profile directory, Xilinota will not delete all of your notes and notebooks in the DB, rather it will re-populate the entire home directory. Also note that since the whole resources folder is under the profile directory, if you delete that, it will not be re-generated.

#### Special notice and limitations

Expand Down
2 changes: 1 addition & 1 deletion packages/app-desktop/package.json
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@xilinota/app-desktop",
"version": "2.13.8",
"version": "2.14.0",
"description": "Xilinota for Desktop",
"main": "main.js",
"private": true,
Expand Down
4 changes: 2 additions & 2 deletions packages/app-mobile/android/app/build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -110,8 +110,8 @@ android {
applicationId "ac.mdiq.xilinota"
minSdkVersion rootProject.ext.minSdkVersion
targetSdkVersion rootProject.ext.targetSdkVersion
versionCode 2097725
versionName "2.13.7"
versionCode 2097726
versionName "2.14.0"
ndk {
abiFilters "armeabi-v7a", "x86", "arm64-v8a", "x86_64"
}
Expand Down
250 changes: 134 additions & 116 deletions packages/app-mobile/root.tsx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ import ProfileSwitcher from './components/ProfileSwitcher/ProfileSwitcher';
import ProfileEditor from './components/ProfileSwitcher/ProfileEditor';
import sensorInfo, { SensorInfo } from './components/biometrics/sensorInfo';
import { getCurrentProfile } from '@xilinota/lib/services/profileConfig';
import { getDatabaseName, getProfilesRootDir, getResourceDir, setDispatch } from './services/profiles';
import { getDatabaseName, getProfilesRootDir, setDispatch } from './services/profiles';
import userFetcher, { initializeUserFetcher } from '@xilinota/lib/utils/userFetcher';
import { ReactNode } from 'react';
import { parseShareCache } from '@xilinota/lib/services/share/reducer';
Expand Down Expand Up @@ -479,10 +479,9 @@ async function initialize(dispatch: Function) {
Setting.setConstant('appId', 'ac.mdiq.xilinota-mobile');
Setting.setConstant('appType', 'mobile');
Setting.setConstant('tempDir', await initializeTempDir());
const resourceDir = getResourceDir(currentProfile, isSubProfile);
Setting.setConstant('resourceDir', resourceDir);

await shim.fsDriver().mkdir(resourceDir);
// const resourceDir = getResourceDir(currentProfile, isSubProfile);
// Setting.setConstant('resourceDir', resourceDir);
// await shim.fsDriver().mkdir(resourceDir);

const logDatabase = new Database(new DatabaseDriverReactNative());
await logDatabase.open({ name: 'log.sqlite' });
Expand Down Expand Up @@ -549,156 +548,175 @@ async function initialize(dispatch: Function) {

setRSA(RSA);

try {
let dbName = '';
if (Setting.value('env') === 'prod') {
dbName = getDatabaseName(currentProfile, isSubProfile);
} else {
dbName = getDatabaseName(currentProfile, isSubProfile, '-3');
// try {
let dbName = '';
if (Setting.value('env') === 'prod') {
dbName = getDatabaseName(currentProfile, isSubProfile);
} else {
dbName = getDatabaseName(currentProfile, isSubProfile, '-3');
// await db.clearForTesting();
}
await db.open({ name: dbName });
}
await db.open({ name: dbName });

reg.logger().info('Database is ready.');
reg.logger().info('Loading settings......');
reg.logger().info('Database is ready.');
reg.logger().info('Loading settings......');

await loadKeychainServiceAndSettings(KeychainServiceDriverMobile);
await migrateMasterPassword();
await loadKeychainServiceAndSettings(KeychainServiceDriverMobile);
await migrateMasterPassword();

const lastTimeAlive: number = Setting.value('lastTimeAlive');
Setting.setValue('shutdownTime', lastTimeAlive);
const lastTimeAlive: number = Setting.value('lastTimeAlive');
Setting.setValue('shutdownTime', lastTimeAlive);

if (!Setting.value('clientId')) Setting.setValue('clientId', uuid.create());
reg.logger().info(`Client ID: ${Setting.value('clientId')}`);
if (!Setting.value('clientId')) Setting.setValue('clientId', uuid.create());
reg.logger().info(`Client ID: ${Setting.value('clientId')}`);

BaseItem.syncShareCache = parseShareCache(Setting.value('sync.shareCache'));
BaseItem.syncShareCache = parseShareCache(Setting.value('sync.shareCache'));

if (Setting.value('firstStart')) {
const detectedLocale = shim.detectAndSetLocale(Setting);
reg.logger().info(`First start: detected locale as ${detectedLocale}`);
if (Setting.value('firstStart')) {
const detectedLocale = shim.detectAndSetLocale(Setting);
reg.logger().info(`First start: detected locale as ${detectedLocale}`);

Setting.skipDefaultMigrations();
Setting.setValue('firstStart', 0);
} else {
Setting.applyDefaultMigrations();
}
Setting.skipDefaultMigrations();
Setting.setValue('firstStart', 0);
} else {
Setting.applyDefaultMigrations();
}

if (Setting.value('env') === Env.Dev) {
if (Setting.value('env') === Env.Dev) {
// Setting.setValue('sync.10.path', 'https://api.joplincloud.com');
// Setting.setValue('sync.10.userContentPath', 'https://xilinotausercontent.com');
Setting.setValue('sync.10.path', 'http://api.joplincloud.local:22300');
Setting.setValue('sync.10.userContentPath', 'http://joplinusercontent.local:22300');
Setting.setValue('privateCode', 'MyXilinotaD');
reg.logger().info('privateCode set to', Setting.value('privateCode'));
}
Setting.setValue('sync.10.path', 'http://api.joplincloud.local:22300');
Setting.setValue('sync.10.userContentPath', 'http://joplinusercontent.local:22300');
Setting.setValue('privateCode', 'MyXilinotaD');
reg.logger().info('privateCode set to', Setting.value('privateCode'));
}

if (Setting.value('db.ftsEnabled') === -1) {
const ftsEnabled = await db.ftsEnabled();
Setting.setValue('db.ftsEnabled', ftsEnabled ? 1 : 0);
reg.logger().info('db.ftsEnabled = ', Setting.value('db.ftsEnabled'));
}
if (Setting.value('db.ftsEnabled') === -1) {
const ftsEnabled = await db.ftsEnabled();
Setting.setValue('db.ftsEnabled', ftsEnabled ? 1 : 0);
reg.logger().info('db.ftsEnabled = ', Setting.value('db.ftsEnabled'));
}

if (Setting.value('env') === 'dev') {
Setting.setValue('welcome.enabled', false);
}
if (Setting.value('env') === 'dev') {
Setting.setValue('welcome.enabled', false);
}

PluginAssetsLoader.instance().setLogger(mainLogger);
await PluginAssetsLoader.instance().importAssets();
reg.logger().info('Going to initialize UDP client');
initUDPClient();

// eslint-disable-next-line require-atomic-updates
BaseItem.revisionService_ = RevisionService.instance();
const prepResourcesDir = async () => {
const resourceDir = Setting.value('resourceDir');
const resStat = await shim.fsDriver().exists(resourceDir);

initUDPClient();
const isSubProfile = Setting.value('isSubProfile');
const resourceDirOld = !isSubProfile ? getProfilesRootDir() : `${getProfilesRootDir()}/resources-${currentProfile.id}`;
const resOldStat = await shim.fsDriver().exists(resourceDirOld);
// reg.logger().info('prepResourcesDir resourceDir resourceDirOld', resourceDir, resourceDirOld);

reg.logger().info('Going to initialize local files');
await LocalFile.init(profileConfig.currentProfileId);
if (resOldStat) {
await shim.fsDriver().moveAllFiles(resourceDirOld, resourceDir, ['jpg', 'jpeg', 'png', 'gif', 'bmp', 'mp4', 'mov']);
} else {
if (!resStat) await shim.fsDriver().mkdir(resourceDir);
}
};

// Note: for now we hard-code the folder sort order as we need to
// create a UI to allow customisation (started in branch mobile_add_sidebar_buttons)
Setting.setValue('folders.sortOrder.field', 'title');
Setting.setValue('folders.sortOrder.reverse', false);
reg.logger().info('Going to initialize local files');
LocalFile.prepResourcesDirFunc = prepResourcesDir;
await LocalFile.init(profileConfig.currentProfileId);

reg.logger().info(`Sync target: ${Setting.value('sync.target')}`);
reg.logger().info('Going to import assets');
PluginAssetsLoader.instance().setLogger(mainLogger);
await PluginAssetsLoader.instance().importAssets();

setLocale(Setting.value('locale'));
// eslint-disable-next-line require-atomic-updates
BaseItem.revisionService_ = RevisionService.instance();

if (Platform.OS === 'android') {
const ignoreTlsErrors = Setting.value('net.ignoreTlsErrors');
if (ignoreTlsErrors) {
await setIgnoreTlsErrors(ignoreTlsErrors);
}
// Note: for now we hard-code the folder sort order as we need to
// create a UI to allow customisation (started in branch mobile_add_sidebar_buttons)
Setting.setValue('folders.sortOrder.field', 'title');
Setting.setValue('folders.sortOrder.reverse', false);

reg.logger().info(`Sync target: ${Setting.value('sync.target')}`);

setLocale(Setting.value('locale'));

if (Platform.OS === 'android') {
const ignoreTlsErrors = Setting.value('net.ignoreTlsErrors');
if (ignoreTlsErrors) {
await setIgnoreTlsErrors(ignoreTlsErrors);
}
}

// ----------------------------------------------------------------
// E2EE SETUP
// ----------------------------------------------------------------

EncryptionService.fsDriver_ = fsDriver;
// eslint-disable-next-line require-atomic-updates
BaseItem.encryptionService_ = EncryptionService.instance();
BaseItem.shareService_ = ShareService.instance();
Resource.shareService_ = ShareService.instance();
DecryptionWorker.instance().dispatch = dispatch;
DecryptionWorker.instance().setLogger(mainLogger);
DecryptionWorker.instance().setKvStore(KvStore.instance());
DecryptionWorker.instance().setEncryptionService(EncryptionService.instance());
await loadMasterKeysFromSettings(EncryptionService.instance());
DecryptionWorker.instance().on('resourceMetadataButNotBlobDecrypted', decryptionWorker_resourceMetadataButNotBlobDecrypted);
// ----------------------------------------------------------------
// E2EE SETUP
// ----------------------------------------------------------------

// ----------------------------------------------------------------
// / E2EE SETUP
// ----------------------------------------------------------------
EncryptionService.fsDriver_ = fsDriver;
// eslint-disable-next-line require-atomic-updates
BaseItem.encryptionService_ = EncryptionService.instance();
BaseItem.shareService_ = ShareService.instance();
Resource.shareService_ = ShareService.instance();
DecryptionWorker.instance().dispatch = dispatch;
DecryptionWorker.instance().setLogger(mainLogger);
DecryptionWorker.instance().setKvStore(KvStore.instance());
DecryptionWorker.instance().setEncryptionService(EncryptionService.instance());
await loadMasterKeysFromSettings(EncryptionService.instance());
DecryptionWorker.instance().on('resourceMetadataButNotBlobDecrypted', decryptionWorker_resourceMetadataButNotBlobDecrypted);

await ShareService.instance().initialize(store, EncryptionService.instance());
// ----------------------------------------------------------------
// / E2EE SETUP
// ----------------------------------------------------------------

reg.logger().info('Loading folders...');
await ShareService.instance().initialize(store, EncryptionService.instance());

await FoldersScreenUtils.refreshFolders();
reg.logger().info('Loading folders...');

const tags = await Tag.allWithNotes();
await FoldersScreenUtils.refreshFolders();

dispatch({
type: 'TAG_UPDATE_ALL',
items: tags,
});
const tags = await Tag.allWithNotes();

// const masterKeys = await MasterKey.all();
dispatch({
type: 'TAG_UPDATE_ALL',
items: tags,
});

// dispatch({
// type: 'MASTERKEY_UPDATE_ALL',
// items: masterKeys,
// });
// const masterKeys = await MasterKey.all();

const folderId = Setting.value('activeFolderId');
let folder = await Folder.load(folderId);
// dispatch({
// type: 'MASTERKEY_UPDATE_ALL',
// items: masterKeys,
// });

if (!folder) folder = await Folder.defaultFolder();
const folderId = Setting.value('activeFolderId');
let folder = await Folder.load(folderId);

dispatch({
type: 'FOLDER_SET_COLLAPSED_ALL',
ids: Setting.value('collapsedFolderIds'),
});
if (!folder) folder = await Folder.defaultFolder();

const notesParent = parseNotesParent(Setting.value('notesParent'), Setting.value('activeFolderId'));
dispatch({
type: 'FOLDER_SET_COLLAPSED_ALL',
ids: Setting.value('collapsedFolderIds'),
});

if (notesParent && notesParent.type === 'SmartFilter') {
dispatch(DEFAULT_ROUTE);
} else if (!folder) {
dispatch(DEFAULT_ROUTE);
} else {
dispatch({
type: 'NAV_GO',
routeName: 'Notes',
folderId: folder.id,
});
}
const notesParent = parseNotesParent(Setting.value('notesParent'), Setting.value('activeFolderId'));

await clearSharedFilesCache();
} catch (error) {
alert(`Initialization error: ${error.message}`);
reg.logger().error('Initialization error:', error);
if (notesParent && notesParent.type === 'SmartFilter') {
dispatch(DEFAULT_ROUTE);
} else if (!folder) {
dispatch(DEFAULT_ROUTE);
} else {
dispatch({
type: 'NAV_GO',
routeName: 'Notes',
folderId: folder.id,
});
}

await clearSharedFilesCache();
// } catch (error) {
// alert(`Initialization error: ${error.message}`);
// reg.logger().error('Initialization error:', error);
// }

reg.setupRecurrentSync();

initializeUserFetcher();
Expand Down
8 changes: 4 additions & 4 deletions packages/app-mobile/services/profiles/index.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,10 @@ export const getProfilesConfigPath = () => {
return `${getProfilesRootDir()}/profiles.json`;
};

export const getResourceDir = (profile: Profile, isSubProfile: boolean) => {
if (!isSubProfile) return getProfilesRootDir();
return `${getProfilesRootDir()}/resources-${profile.id}`;
};
// export const getResourceDir = (profile: Profile, isSubProfile: boolean) => {
// if (!isSubProfile) return getProfilesRootDir();
// return `${getProfilesRootDir()}/resources-${profile.id}`;
// };

// The suffix is for debugging only
export const getDatabaseName = (profile: Profile, isSubProfile: boolean, suffix = '') => {
Expand Down
4 changes: 3 additions & 1 deletion packages/app-mobile/utils/ShareExtension.ts
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
import { NativeEventEmitter } from 'react-native';
import { LogBox, NativeEventEmitter } from 'react-native';

const { NativeModules, Platform } = require('react-native');

LogBox.ignoreLogs(['new NativeEventEmitter']); // Ignore log notification by message

export interface SharedData {
title?: string;
text?: string;
Expand Down
Loading

0 comments on commit 6d61e03

Please sign in to comment.