Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to sync group structure with KeeShare #3045

Open
welusa opened this issue Apr 20, 2019 · 57 comments · May be fixed by #11593
Open

Add ability to sync group structure with KeeShare #3045

welusa opened this issue Apr 20, 2019 · 57 comments · May be fixed by #11593
Assignees
Milestone

Comments

@welusa
Copy link

welusa commented Apr 20, 2019

Summary

Hi there,
first of all, sorry for my terrible english and sorry for wasting your time, but I am too much of a noob to deal with this myself. I stumbled across the KeeShare feature in KeePassXC. Now I am trying to get a hang of how this is working. I am able to synchronize two databases with it. Now to my problem. Synchronizing the entrys works just fine, but the groups aren't getting synchronized. Is there any way to get this to work, so that i can create a subgroup in a synchronized group and it is going to synchronize it as well? I really don't know if this is a feature request, maybe it's only me being to stupid to use this. But when I import or synchronize it on a different device i only get the entrys, all in the group i activatet the keeshare feature for.

Desired Behavior

If i create a sub group in a shared group it should be synchronized / imported to other devices as well. Not only the entrys.

Possible Solution

Context

I would really like this, because i kinda like sorting things. In the future i want everyone in the office to have his own database, in which i can synchronize only the groups he needs or he is supposed to have. Giving him the structure of the groups i chose in my database would make it way more easily to use.

@droidmonkey
Copy link
Member

Make sure you have both import and export enabled in the KeeShare settings:

image

@welusa
Copy link
Author

welusa commented Apr 20, 2019

I have enabled those everywhere. Just tried again. If I create a new group in a keeshared group, the entries made in the new one appear in the original keeshared group, without adding the new subgroup.

@welusa
Copy link
Author

welusa commented Apr 20, 2019

grafik
grafik
Maybe those two pictures show my problem a bit better.

@ckieschnick
Copy link
Contributor

In short, the feature was designed to work exactly this way. We thought about synchronizing group structures with KeeShare, but after several discussions we decided against it since there a lot of edge cases where its hard to specify and understand the behavior concerning the synchronization of groups and subgroups.

The advantage of only sharing entries is, that receiver is free to rearrange shared entries (moving them into new subgroups, moving them outside of the importing group the share since the synchronize feature will locate those entries - or even deleting them locally).

@droidmonkey
Copy link
Member

Ah, I did not even know this was the behavior. Perhaps there should be an option to "preserve share group structure" which would imply the behavior of enforcing a particular entry share into a particular location.

@welusa
Copy link
Author

welusa commented Apr 22, 2019 via email

@mrjk
Copy link

mrjk commented Apr 25, 2019

Yep, I agree with droidmonkey, using flat only files limits the interest in my case. It would be nice if keeshare could act as a symlink to another DB, not having the possibility to share the password folders is a no go for me. I hope this feature will be available at some points :) Thanks for you amazing work guys ;)

@ckieschnick
Copy link
Contributor

@mrjk you may have a look at #2658. I think this would be a better fit for your description.

@droidmonkey droidmonkey changed the title KeePassXC Keeshare subfolders not synchronized Add ability to sync group structure with KeeShare Jul 10, 2019
@dsonck92
Copy link

dsonck92 commented Jul 22, 2019

Not entirely sure if it warrants a new bug report but, @ckieschnick, the current import/export (and sync) behavior doesn't exactly match what is described:

  • If I export (export or sync) a group, the group gets flattened, as already known
  • If I export (export or sync) a group, and I move the item outside the group, the item is removed from the container
  • If I import (import or sync) a group, and the keeshare file does have structure, this structure is kept
  • If I move an entry around inside a keeshare group, it stays. If I then directly modify the keeshare database by moving the entry to another place, it will also be moved in the parent database when it imports/syncs again

So as the status is currently:

  • To a keeshare database will flatten, and in fact export will remove anything moved outside
  • From a keeshare database will enforce the keeshare hierarchy if present, but can be moved outside and still update from the container

Interestingly enough, this actually fits me a bit because I can make several local databases that pull from different shared keeshare databases. These databases are the ones I keep active on my local machines to autofill/type passwords. And I simply modify the keeshare database files so they retain hierarchy. However I understand this would not entirely be the correct use case.

I think ideally it should be configurable to keep or flatten the hierarchy. Where, if it's flattened, the destination hierarchy is kept. As I see it currently, this means including a checkbox for export, and modify how it exports based on this state

@crosscodr
Copy link

In my opinion, the flattening of the database, when synchronizing with keeshare is not an expected behavoir. At least, it should be decidable to sync/export with flattening or not.
In my case, I first imported a password-group with many supgroups from a shared keepass-database into my private one. Then I made some changes in my local version of the imported group and found the option to synchronize instead of just import from another keepass-db.
When I did this, the shared keepass-db was flattened without any warning.
I think, this behaviour does not reflect, what the word 'synchronize' means. In my opinion, the synced group should be equal including all subgroups in both locations.
Took me some time to repair the shared db :(
I considered this a bug and just wanted to open a new bug-report, before I found this issue. I would endorse to label this issue as a bug.

@welusa
Copy link
Author

welusa commented Jul 25, 2019

I fully agree to what crosscodr is saying. I startet this issue because i am searching for a solution for a small company. While trying out the share feature I destroyed my database multiple times too. It'd be awesome if there was some way to sync the database without flattening. I am definately not trying this one more time on my main database, since i have learned my lesson now.

@ckieschnick
Copy link
Contributor

I hope I don't repeat the documentation, but it looks like a misunderstanding for using this feature. Therefore I'll try to clarify the usage of KeeShare:

Neither the database used by the synchronize feature nor the import/export of KeeShare is intended to be used as real database. They are just a transfer format which happens to be a database readable by KeePass. The containers should be in between real databases opened by KeePassXC which use KeeShare to pull or push the changes to the transfer container.
The transfer container can an be overwritten or updated anytime by any client - therefore you shouldn't open such a database in a regular scenario - it is intended to be written and read only be KeeShare.

The idea is the following:
Client A <=[synchronizes with KeeShare]=> (Transfer container) <=[synchronizes with KeeShare]=> Client B

There is no explicit support for scenarios like this (unless the KeePassXC team says otherwise):
Client A <=[synchronizes with KeeShare]=> (Transfer container) <=[opens directly]=> Client B
in this case, I think you can use the standard synchronize feature of KeePassXC.

Nonetheless, if you want to open a transfer container directly - make sure it is read only since every change maybe overwritten by another client using KeeShare. At least from my side, there is no guarantee, that transfer format doesn't change, so this may not work in future versions.

@ckieschnick
Copy link
Contributor

Interestingly enough, this actually fits me a bit because I can make several local databases that pull from different shared keeshare databases. These databases are the ones I keep active on my local machines to autofill/type passwords.

@dsonck92 ... this is more or less how we intended to use the feature. It is a bit cumbersome to setup, but in the end it is the most easy way to understand and handle complex scenarios without to much headache. The smaller containers are easier to compose and divide according to responsibilities (which was the main driver behind this feature).

And I simply modify the keeshare database files so they retain hierarchy. However I understand this would not entirely be the correct use case.

Do you edit them by hand? Or are those containers just the leaf groups of your group tree (which is mirrored in the client databases)? The second one was the one had in mind since this provides the most ease. Enforcing the structure of your passwords to other users didn't look that nice - but in a single user multiple computer scenario, the structure sharing would be ok.

@crosscodr
Copy link

Thanks @ckieschnick for clarification. It makes sense when you explain it this way.
I still think this is not the most obvious behaviour for users. Maybe this could be made more clear in the documentation.

@dsonck92
Copy link

I meant by that, If I edit the keeshare container (unsigned) by hand, an import forces the structure of entries that change. This was unexpected for me but does work for me personally and I think would work for others participating in this ticket.

But as you already mentioned (hence I was saying "I understand this would not entirely be the correct use case") that directly modifying the keeshare container is not supported. I was merely doing that to understand specific corner cases:

Modifying the container (importing into a keepass database):

  • Item is added to container
    • What happens if I move this item around in a keeshare import -> container stays the same (expected)
  • Item is modified in container
    • What happens to the moved item -> moved item is still updated (expected)
  • Item is moved/created in container in subgroup
    • What happens now -> moved item is moved back to the shared group (unexpected) or created in a subgroup under the share (unexpected)
  • Item is removed
    • Item disappears from database -> expected

Modifying the sharegroup (exporting to keepass database):

  • Item added to subdir
    • How is it stored -> flattened items (expected)

So I didn't expect import to respect any structure, which comes back to this request. I'm currently trying to write some proof of concept code that adds the aforementioned "Flatten/Keep Structure" functionality to (for now) the Export/Sync function. A checkbox shows up when Sync or Export is chosen which should skip the flattening step if flattening is not desired. The import code can remain as-is as far as I can see now. Though this does not fully take into account every corner case. However, I personally think that adding a checkbox to get the original behavior or the (for some) expected behavior is useful. That gives end users the ability to choose which vision they require.

At a later stage, it could be decided whether this flattening can also happen during import as well, so the "any present structure is forced" can be avoided on demand as well.

Currently having some unexplainable crashing bugs which I expect are due to some library issues, which hold me back from testing my work. I'll give some updates and if you have any input feel free to share those.

@dsonck92
Copy link

I actually got the code to work, turned out to be a silly mistake: I forgot to actually make the change to KeeShareSettings::Reference also affect sorting and equality. The commit dsonck92@2d6ad06 has the changes.

The reason I did not yet create the pull request for it is because there are some open bugs that annoy me:

  • The way flattening works on import will not prevent items being moved back if the container structure changes. I want to know what could cause this before I submit it because I want the "do not keep structure" to allow people to ignore structure on new structured containers. Giving the best of both worlds.
  • When 2 databases sync with keep structure, they always cause one of the two to be "unsaved". Saving touches the container again which makes the other import and become "unsaved". Saving the other export again, etc. etc. This is more of a small inconvenience issue but could completely break autosave and makes using yubikey a chore as it would constantly ask for the yubikey to give the secret.

@dsonck92
Copy link

@crosscodr for your use case I would recommend the following thing (keeping in mind that ckieschnick might still change the format and also noting it won't support secure containers), which I will personally use on my main database until my proposed code changes get polished enough for inclusion and when the developers accept this idea.

  • Create one or more shared containers that only import, noting that importing will retain structure.
  • Also create an AutoOpen group under the root group of the keepassxc database. This is the auto-open feature that is already present for some time.
  • Add entries for each shared container, pointing the URL field to the container file and putting the associated password inside the password boxes

Now, whenever you open up your database, it will also open up the related shares in the background. This gives you (at least in my eyes) all the features you want:

  • Autotype/Browser fill in will work properly as the main database (which contains the imports) is mostly active
  • Structure is kept
  • You can still update the containers directly
  • It is still shared, so you can give those containers to others you trust

Again, note that, as ckieschnick pointed out, the container format is not to be modified directly, mostly because it can also be overwritten by an export or sync. And because format might change (which would essentially mean that at some point you cannot import anymore, and have to use the auto-open feature to access your separate containers, however I don't see a very big threat there)

Another way would actually be to try my fork and see how well it works for you, and possibly give some input on it if there are some things that could be better. Note that it's very alpha, I will try to use it myself carefully but I might create some stupid bug that simply erases everything from a container. Yes, I have had that happen while trying to flatten the database.

@dssouza-ti
Copy link

dssouza-ti commented Oct 9, 2019

Hi @dsonck92

Any new on this feature? I have a use case where keeping the structures would be awsome, so I'm really looking forward to seeing it fully functional. In the mean time, I'll try to use your commit with a test database for a while to see if anything strange happens.

@dsonck92
Copy link

I've been a bit preoccupied with other tasks, mainly as I was very annoyed by bug 2 that I mentioned (always unsaved). However, as I've updated my keepassxc install in the meantime (and thus not use my own patch), it seems like it also happens without my patch so only the first bug remains.

I would certainly appreciate you trying it out, and see if there is more interest in it. Currently the auto-open workflow does work. But I do remember ckieschnick's warning that the format might change, which would mean the patch would be better as you never have to touch the internal format ever.

If I have more time/more interest is shown, I can continue to squash that bug of "flattening will still modify existing items". It would be very helpful to have:

  • keep structure -> items will be forced into structure on import / items will keep structure on export
  • discard structure -> items will be merged with existing on import, not altering structure / items will be flattened on export

@dsonck92
Copy link

At least I've updated my branch to reflect latest changes in develop https://github.com/dsonck92/keepassxc/tree/feature/sharing_keep_structure

@droidmonkey
Copy link
Member

Which bugs are you talking about, please link to the issue reports.

@dssouza-ti
Copy link

@dsonck92 I have been testing your changes with my team at work and it works really well for our use case. Since we are only interested in the "Keep Structure" part of this, the bug you mention does not affect us.

@droidmonkey I just opened #3791, which describes the bug 2 that @dsonck92 has stated above.

@JackDev-cc
Copy link

Any further update to this? I know its open funding but this is a multi-year open issue any timeframe? as i'd also love this feature.

@xmgr
Copy link

xmgr commented Aug 29, 2022

What's the progress for this feature? I'd also love having the ability to preserve the group structure :)

@droidmonkey
Copy link
Member

None made for a while, I've been working on other problems.

@meow464
Copy link

meow464 commented Dec 25, 2022

I used a "syncthing" group to send and receive new entries, which have to be manually organized, then simply copied my database and keeshare file to the new machine to preserve already existing groups.
If you are using syncthing it's important to enable versioning for the sync folder.

Merry Christmas. KeepassXC rocks!

@xstable
Copy link

xstable commented Feb 8, 2023

come from linked issue to here, I do comment to be noticed if anything new here.

@lindhe
Copy link
Contributor

lindhe commented Feb 10, 2023

@xstable No need to comment for that. You could simply have clicked the "Subscribe" button at the right hand side of the page (or at the bottom, in mobile view) to get subscribed for notifications. :)

@SnejPro
Copy link

SnejPro commented May 21, 2023

I would really like to see this feature ... or maybe it could be made easier to do a full sync with another file (For example with presaved credentials).

@evur evur mentioned this issue Aug 22, 2023
3 tasks
@krodelabestiole
Copy link

The advantage of only sharing entries is, that receiver is free to rearrange shared entries (moving them into new subgroups, moving them outside of the importing group the share since the synchronize feature will locate those entries - or even deleting them locally).

I'm sorry I don't understand this point. Actually I don't see any reason in favor of this behavior (flattening synced db), unless it's a technical limitation ?
If the receiver wants to rearrange, well he/she can. And the "remote" changes will reflect in the "local" database, hence the sync.
I don't understand why keys and groups would be treated differently in that regard. Imagine a file manager that would remove every directories and give a bulky bunch of files in no particular arrangement when "syncing" it... (and that would remove the distant tree structure every time it get "synced" again, deleting any arrangement done in the meantime) ?

It is a destructive behavior. Unless I miss something I think this "feat" should be treated a bugfix, not even some kind of option...

@droidmonkey
Copy link
Member

Where did you pull that quote from? I agree this is more of a bugfix because the original behavior doesn't make sense in terms of have a shared "container".

@krodelabestiole
Copy link

Where did you pull that quote from?

I've just "quote replied" this message above : #3045 (comment)

@N0W0RK
Copy link

N0W0RK commented Nov 30, 2023

I quickly want to share my usecase here as it might affect the functionality of the sync DB.
I use a WebDAV server to share one keyfile between all my devices.
My mobile app(Strongbox) stores a local copy of my Database for quick access and syncs it with the server if there are changes and an internet connection present.
I would like simmilar behaviour for the new KeeShare. An added bonus of having the shared file beeing a fully functioning database would be, that my mobile devices could just use that database as the "upstream source" and as far as I understand any change from any device could be properly merged.

@elRicharde
Copy link

elRicharde commented May 23, 2024

[> I hope I don't repeat the documentation, but it looks like a misunderstanding for using this feature. Therefore I'll try to clarify the usage of KeeShare:

Neither the database used by the synchronize feature nor the import/export of KeeShare is intended to be used as real database. They are just a transfer format which happens to be a database readable by KeePass. The containers should be in between real databases opened by KeePassXC which use KeeShare to pull or push the changes to the transfer container. The transfer container can an be overwritten or updated anytime by any client - therefore you shouldn't open such a database in a regular scenario - it is intended to be written and read only be KeeShare.

The idea is the following: Client A <=[synchronizes with KeeShare]=> (Transfer container) <=[synchronizes with KeeShare]=> Client B

There is no explicit support for scenarios like this (unless the KeePassXC team says otherwise): Client A <=[synchronizes with KeeShare]=> (Transfer container) <=[opens directly]=> Client B in this case, I think you can use the standard synchronize feature of KeePassXC.

Nonetheless, if you want to open a transfer container directly - make sure it is read only since every change maybe overwritten by another client using KeeShare. At least from my side, there is no guarantee, that transfer format doesn't change, so this may not work in future versions.](#3045 (comment))

Thanks for clarifying, but i seem to have trouble understanding it correctly.

Is there a User Guide on How to

  • Use a DB on Multiple Devices Correctly (i used one KP-File located in my GoogleDrive with 2 PC with KeePassXC and one Mobile with KeePassDX and always got trouble of Dataloss thru overwriting)
  • Use a Subset of an DB on Multiple Devices

I switched today to Keyshare-principle, but now i read your post and if i Understand correct I do it wrong now (i open the File generated with KeyShare directly in other Device)
I just came to this thread while searching how to handle a third or more devices.

It would be great to have this also Mentioned in the Manual. Thanks in Advance, great tool!

Edit:
Also there is no manual for this and I´ve only saw it her as an Answer in an Issue, sould be part of Documentation as well as showing the limits of this functionality.

The advantage of only sharing entries is, that receiver is free to rearrange shared entries (moving them into new subgroups, moving them outside of the importing group the share since the synchronize feature will locate those entries - or even deleting them locally).

Thanks

@jay7x
Copy link

jay7x commented Sep 24, 2024

Maybe we can get rid of this problem in another way. What if a DB can be "mounted" under some path in the "main" DB? For example, I have my personal DB personal.kdbx and I want a work.kdbx file contents (which is synced from my office workstation e.g.) to appear under the /Work path. Then one can just share the sub-DB and have the proper structure always.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: To triage