macOS client synchronisation failures (macOs and using groupfolders)

[CoMPaTech]

Expected behaviour

Using macOs: 1) Adding new folder and new files should synchronise normally. 2) Opening existing word or excel files (using Microsoft Office suite) and saving should be possible.

Actual behaviour

1. Adding a new folder and placing files in them leads to either working alright, giving the error on 'no permissions to create the folder' or even the folder being alright but not being able to 'place files' in said folder.
2. Creating a new (excel or word) file works fine, but opening such file again leads to the file being saved as 'conflicted'. More to the point we can - for those users - reproduce this by creating a file (either touch or vi on command line) and opening it again and saving. It looks like the original (i.e. the opened file) falls to 444 rights (-r--r--r--) on the client.
3. Some users are unable to delete files (the keep coming back unless deleted using the web-interface)
4. Inspecting the metadata table in the sqlite3 database mostly such files or directories (from 1)) have "empty" remotePerms. If we delete these rows from the sqlite database things start to work, but new problems will keep popping up when adding folder+files again or editing existing documents.

Our setup is outlined below - also see my comment in nextcloud/groupfolders#746 - please note that in the web-interface everything works/seems ok. I've also included potential relevant other issues in my comment on that issue. To us the problem looks to be bound to the client synchronisation in macOs and upgrading one of our users that still was using 10.14 to 10.15 seemed to have increased the probability and reproducability of the issues.

Exception is that one of our users who has all of the above problems only has these in her (as depicted below) department2 set of folders, not in department3.

Steps to reproduce

1. Nextcloud (Ubuntu 18.04 LTS) Server
2. macOs (10.15) desktop
3. Nextcloud (latest) clients

Groupfolders deployed as

/Departement2 - group department2_subgroup1 and department2_subgroup2 have been added, but no rights (write/share/delete disabled)
/Department2/Subgroup1 - group department2_subgroup1 has been added and full rights (all three checked)
/Department2/Subgroup2 - group department2_subgroup2 has been added and full rights (all three checked)
/Department2/Subgroup3 - group department2_subgroup1 and department2_subgroup2 have been added and full rights (all three checked)

/Department3 - group department3 has been added but no rights (write/share/delete disabled)
/Department3/Subgroup1 - group department3_subgroup1 has been added but no rights (write/share/delete disabled)
/Department3/Subgroup1/SharedFolder1 - group department3_subgroup1 has been added and full rights
/Department3/Subgroup1/SharedFolder2 - group department3_subgroup1 has been added and full rights

Below the folders users are free to create their own directories as per usual (example given here to support the debug log included).
/Department3/Subgroup1/SharedFolder1/UserCreatedDir1/UserCreatedDir2

Our problematic clients are all in Department3 - mostly running into what seems the emty remotePerm problems (with side effects as the unable to delete). We can't reproduce the errors with the web-interface or even using webdav directly (although we really don't want to go that way). Do note that a couple of users of Department3 also work for Department2 and they don't experience the issues there.

For one user we've created a new set of groupfolders like below (resembling the Department2 setup) but they problems still exist (though frequently and scripted cleaning empty remotePerms rows in their local database seems to 'fix' it and is usable as a workaround).

/Departement3__test - group department3_subgroup1 has been added, but no rights (write/share/delete disabled)
/Department3_test/Subgroup1 - group department3_subgroup1 has been added and full rights (all three checked)

For a subset of department3 we'll now revert to having all their files in a /subgroupX folder with full rights and no (sub)groupfolders below it and test if that works

Client configuration

Client version: 2.6.5 (client, though still same with 3.0.0)

Operating system: macOs 10.15.5, 10.15.6 and 10.14.6

OS language: happens with either English or Dutch as default lang

Qt version used by client package (Linux only, see also Settings dialog): n/a

Client package (From Nextcloud or distro) (Linux only): n/a

Installation path of client: /Applications/NextCloud.app

Server configuration

Sharing not enabled, using group-folders for distributions
Nextcloud version: 18.0.7 (server)

Storage backend (external storage):
NFS

Logs

Not applicable that we can see, server in debug reveals no obvious errors, client in debug (F12). We're happy to provide more logging if needed and are willing to switch clients between native English/Dutch setup. (Below log is from a Dutch setup).

[OCC::SocketListener::sendMessage 	Sending SocketAPI message --> "STATUS:IGNORE:/Users/username/OurCloud/Department3/Subgroup1/SharedFolder1/UserCreatedDir1/UserCreatedDir2" to SocketApiSocket(0x6000034316a0)
[OCC::ActivityWidget::slotItemCompleted 	Item  "Department3/Subgroup1/SharedFolder1/UserCreatedDir1/UserCreatedDir2/UserCreatedDir3/UserCreatedDir4/somefile.pdf"  retrieved resulted in  "Niet toegestaan omdat u geen rechten hebt om bestanden in die map toe te voegen"

(...)

[OCC::SocketListener::sendMessage 	Sending SocketAPI message --> "STATUS:IGNORE:/Users/username/OurCloud/Department3/Subgroup1/SharedFolder1/UserCreatedDir1/UserCreatedDir2" to SocketApiSocket(0x6000034316a0)
[OCC::ActivityWidget::slotItemCompleted 	Item  "Department3/Subgroup1/SharedFolder1/UserCreatedDir1/UserCreatedDir2/UserCreatedDir3/UserCreatedDir5/somefile.pdf"  retrieved resulted in  "Niet toegestaan omdat u geen rechten hebt om bestanden in die map toe te voegen"
[OCC::ActivityWidget::slotItemCompleted 	Item  "Department3/Subgroup1/SharedFolder1/UserCreatedDir1/UserCreatedDir2/UserCreatedDir3/UserCreatedDir5/somefile.pdf"  retrieved resulted in error  "Niet toegestaan omdat u geen rechten hebt om bestanden in die map toe te voegen"
[OCC::ActivityListModel::addErrorToActivityList 	Error successfully added to the notification list:  "Niet toegestaan omdat u geen rechten hebt om bestanden in die map toe te voegen"
[OCC::PropagateItemJob::scheduleSelfOrChild 	Starting INSTRUCTION_ERROR propagation of "Department3/Subgroup1/SharedFolder1/UserCreatedDir1/UserCreatedDir2/UserCreatedDir3/UserCreatedDir6/somefile.pdf" by OCC::PropagateIgnoreJob(0x600002ae2040)
[OCC::PropagateItemJob::done 	Could not complete propagation of "Department3/Subgroup1/SharedFolder1/UserCreatedDir1/UserCreatedDir2/UserCreatedDir3/UserCreatedDir6" by OCC::PropagateIgnoreJob(0x600002ae2040) with status 2 and error: "Niet toegestaan omdat u geen rechten hebt om bestanden in die map toe te voegen"
[OCC::SocketListener::sendMessage 	Sending SocketAPI message --> "STATUS:IGNORE:/Use/Users/username/OurCloud/Department3" to SocketApiSocket(0x6000034316a0)

We can confirm that we have seen the same errors on an English-language setup.

[er-vin]

Definitely interested from logs of a client 2.6.5 or later (logs provided are clearly from before 2.6.5). Also please make sure they're in debug mode, that'd help.

[CoMPaTech]

Sharing that without disclosing information through all the files is rather difficult @er-vin - I could let loose a couple of seds but you might eventually mis out on the details of which file is which (currently already trying with a couple of statements to make them 'less visible' but I recon if you need deatils like the aboved scrubbed ones it won't work.

[er-vin]

Sharing that without disclosing information through all the files is rather difficult @er-vin

Yes, I know this is not easy. When it is a concern it's better to have a test instance and reproduce the scenario.

I could let loose a couple of seds but you might eventually mis out on the details of which file is which (currently already trying with a couple of statements to make them 'less visible' but I recon if you need deatils like the aboved scrubbed ones it won't work.

Well, we can try just in case. Indeed it's likely we'll have too low information.

[CoMPaTech]

Hi, the redacted version should be at https://gist.github.com/CoMPaTech/c518b9add7b79d5d107b7c9bdb8d2467

[er-vin]

Indeed seems to be permission related, again 2.6.5 client and using both --logfile and --logdebug on the command line might help know more.

You might also want to look at the rights for the corresponding folders in the journal db metadata table.

[CoMPaTech]

Working on less-redacted version (although skipping LEFT JOIN checksumtype|SQL bind to limit to 5k5 lines (instead of 38k and original 424k :))

To be able to search this though - the user used the Department3 folder accordingly renamed with 'naamloze folder' to 'TestFolder' and later 'TestFolder2'.

• Created a new dir (hence 'naamloze map') in SharedFolder1 and named it TestFolder
• Opened a template (word) testform in another dir and save-as it to the TestFolder
• Save-as again, creating a new dir while at it (TestFolder2)

On the first save the client indicated issues, but only a warning (yellow exclamation). On the second action it gave an error (red cross). Testing client is the same as last file submitted, it's mac 10.15.6(19G2021) - NextCloud client is 2.6.5.

Debug file will follow after review by co-worker.

[CoMPaTech]

Debug file available at https://gist.github.com/CoMPaTech/56952455b45396ce5289faa86b0ad5ac

[er-vin]

Alright, even with those logs I can't really pinpoint which permissions ended up in which files. So still confirms this is a permission error. Note that the fact that created entries (during a folder creation for instance) end up with empty remote perms is accounted for in the code. AFAICT when you get a rejection like this it means the entry got something for its perms, otherwise the client just assumes it doesn't know and proceed (assuming the server will reject later on in case of perms violation).

So there's definitely something in the perms of those folders as seen from the client. I'd check in the client database which are those and verify if they match what the server says about them.

[CoMPaTech]

Well in most cases the problematic sqlite rows - client exited, rows deleted, client started - are solved until the next time the user tries this. So it is sort-of-a-workaround but not really doable for administrative tasks with repetitive creation and manipulation of the files. To be clear, problematic files are along the lines of echo \"delete from metadata where (md5 is NULL or md5 = '' ) AND (remotePerm is NULL or remotePerm='')\"

[CoMPaTech]

Any other ideas and/or should we aim to share the file in a non-public way/other form of redaction etc?

[er-vin]

Well in most cases the problematic sqlite rows - client exited, rows deleted, client started - are solved until the next time the user tries this. So it is sort-of-a-workaround but not really doable for administrative tasks with repetitive creation and manipulation of the files. To be clear, problematic files are along the lines of echo \"delete from metadata where (md5 is NULL or md5 = '' ) AND (remotePerm is NULL or remotePerm='')\"

Ah I get your point, from the database perspective they got "no" perms (NULL or empty string). That's odd... that'd mean they get something later on in the pipe...

Still, could you see if the problematic ones have NULL or ''? Right now you're going for a large swipe as workaround so if we could narrow it down a bit.

[er-vin]

Any other ideas and/or should we aim to share the file in a non-public way/other form of redaction etc?

That's an option indeed. You could drop a full debug log there: https://cloud.nextcloud.com/s/BbRrHPpcdzWoXdX

Note however that it might take time until we can look at it, got the official support requests coming first.

[github-actions]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[CoMPaTech]

The issue is still present, but we are working around it nowadays (not sharing more than one level of groupfolders). Lets close it for now.