Externally deleted emails show up in the inbox and cannot be deleted

[LostinSpacetime]

Steps to reproduce

1. Open your email account in Thunderbird with new emails
2. Delete one or more emails before reading them
3. Open the same account in the Mail app on the cloud

Expected behavior

The inbox shouldn't contain the deleted emails.

Actual behavior

The inbox contains the deleted emails and the emails are marked as unread. The overall counter for unread emails is 0.
If one clicks on one of the emails, instead of the content on the email, "not found" is shown. Trying to delete the emails results in failure with the message "Message could not be deleted" and after a few seconds the "ghost message" reappears in the inbox.

Mail app version

3.5.0

Mailserver or service

IMAP

Operating system

Ubuntu 18.04

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database

MariaDB

Additional info

No response

[LostinSpacetime]

Currently, my inbox is filled with non-existent emails, and is basically unusable. I would be thankful for a way to rescan my emails as a temporary workaround.

[3Nigma]

Same here

[alexh-swdev]

Same here:

• Mail 3.5
• NC 28.0.1
• PHP 8.2.14 (fpm w/ Apache 2.4.46)
• Debian bullseye

I tried the commands w/o success:

• mail:clean-up
• mail:sync

I could not figure out a rule which mails I can remove from my local MUA (which is emClient in my case) but remain in NC-Mail. But there should definetly be an easy way to remophe orphanded mails (mail which are not found) as a hotfix ASAP unless this is fixed properly

/Edit: I also have Redis and APCu active (but this is independent from the error, just tried it)

/Edit2: NC Log shows this:

[paulvt]

It is possible to use "Clear mailbox" on my Inbox, and then the messages are gone.
However, because I also suffer from #5323, when I go to the Priority inbox, they are still all there!
Then, when I mark them as Unimportant and reload the page/app, they are finally all gone.

So the issue seems to be purely that threads and prioritization tags keep tracks of messages that are gone.

[thrommel1980]

Same problem here.
I already avoid deleting emails in the K9 Mail app. If I do, they are in my NC app as corpses. But are deleted in my mail provider inbox.

[mayermart]

I do not think this is a Problem caused by K9 itself.
After reading some issues and some experiments i got rid of the problem by simply disable the "Mark E-Mails as Important" Feature in the Nextcloud Mail Settings.

To delete all remaining (Ghost-)E-Mails from the Mailbox i

1. moved all actually existing Mails via Thunderbird (K9 would work the same i guess) in a new temporary Folder
2. used the "Clear mailbox" option on the Inbox folder in Nextcloud Mail and
3. moved all Mails from the temporary Folder back into my Inbox via Thunderbird.

Edit 2024-04-02: The Problem reappeared but less frequently.

[maximelehericy]

same is happening to me, with archived or sent to junk emails.

[violoncelloCH]

posting the full log lines in case that helps:

{"reqId":"YFRxUA0D1JXRPoiRvzhL","level":3,"time":"2024-01-22T20:56:55+00:00","remoteAddr":"172.22.0.4","user":"jonas","app":"mail","method":"DELETE","url":"/apps/mail/api/thread/36311","message":"OCA\\Mail\\IMAP\\MessageMapper::move(): Return value must be of type int, null returned in file '/var/www/html/custom_apps/mail/lib/IMAP/MessageMapper.php' line 334","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0","version":"27.1.5.1","exception":{"Exception":"Exception","Message":"OCA\\Mail\\IMAP\\MessageMapper::move(): Return value must be of type int, null returned in file '/var/www/html/custom_apps/mail/lib/IMAP/MessageMapper.php' line 334","Code":0,"Trace":[{"file":"/var/www/html/lib/private/AppFramework/App.php","line":183,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[["OCA\\Mail\\Controller\\ThreadController"],"delete"]},{"file":"/var/www/html/lib/private/Route/Router.php","line":315,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OCA\\Mail\\Controller\\ThreadController","delete",["OC\\AppFramework\\DependencyInjection\\DIContainer"],["36311","mail.thread.delete"]]},{"file":"/var/www/html/lib/base.php","line":1068,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/apps/mail/api/thread/36311"]},{"file":"/var/www/html/index.php","line":38,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","Line":169,"Previous":{"Exception":"TypeError","Message":"OCA\\Mail\\IMAP\\MessageMapper::move(): Return value must be of type int, null returned","Code":0,"Trace":[{"file":"/var/www/html/custom_apps/mail/lib/Service/MailManager.php","line":381,"function":"move","class":"OCA\\Mail\\IMAP\\MessageMapper","type":"->","args":[["OCA\\Mail\\IMAP\\ImapClientRateLimitingDecorator",["HICenv","HICflags","HIChdrs","HICdate","HICsize","HICstruct"],true,true],"INBOX",1582,"Trash"]},{"file":"/var/www/html/custom_apps/mail/lib/Service/MailManager.php","line":340,"function":"deleteMessageWithClient","class":"OCA\\Mail\\Service\\MailManager","type":"->","args":[["OCA\\Mail\\Account"],["OCA\\Mail\\Db\\Mailbox",8],1582,["OCA\\Mail\\IMAP\\ImapClientRateLimitingDecorator",["HICenv","HICflags","HIChdrs","HICdate","HICsize","HICstruct"],true,true]]},{"file":"/var/www/html/custom_apps/mail/lib/Service/MailManager.php","line":954,"function":"deleteMessage","class":"OCA\\Mail\\Service\\MailManager","type":"->","args":[["OCA\\Mail\\Account"],"INBOX",1582]},{"file":"/var/www/html/custom_apps/mail/lib/Controller/ThreadController.php","line":205,"function":"deleteThread","class":"OCA\\Mail\\Service\\MailManager","type":"->","args":[["OCA\\Mail\\Account"],["OCA\\Mail\\Db\\Mailbox",8],"<[email protected]>"]},{"file":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","line":230,"function":"delete","class":"OCA\\Mail\\Controller\\ThreadController","type":"->","args":[36311]},{"file":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","line":137,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[["OCA\\Mail\\Controller\\ThreadController"],"delete"]},{"file":"/var/www/html/lib/private/AppFramework/App.php","line":183,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[["OCA\\Mail\\Controller\\ThreadController"],"delete"]},{"file":"/var/www/html/lib/private/Route/Router.php","line":315,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OCA\\Mail\\Controller\\ThreadController","delete",["OC\\AppFramework\\DependencyInjection\\DIContainer"],["36311","mail.thread.delete"]]},{"file":"/var/www/html/lib/base.php","line":1068,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/apps/mail/api/thread/36311"]},{"file":"/var/www/html/index.php","line":38,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"/var/www/html/custom_apps/mail/lib/IMAP/MessageMapper.php","Line":334},"message":"OCA\\Mail\\IMAP\\MessageMapper::move(): Return value must be of type int, null returned in file '/var/www/html/custom_apps/mail/lib/IMAP/MessageMapper.php' line 334","exception":[],"CustomMessage":"OCA\\Mail\\IMAP\\MessageMapper::move(): Return value must be of type int, null returned in file '/var/www/html/custom_apps/mail/lib/IMAP/MessageMapper.php' line 334"},"id":"65aed6b3751ba"}
{"reqId":"YFRxUA0D1JXRPoiRvzhL","level":3,"time":"2024-01-22T20:56:55+00:00","remoteAddr":"172.22.0.4","user":"jonas","app":"PHP","method":"DELETE","url":"/apps/mail/api/thread/36311","message":"Undefined array key 1582 at /var/www/html/custom_apps/mail/lib/IMAP/MessageMapper.php#334","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0","version":"27.1.5.1","data":{"app":"PHP"},"id":"65aed6b3751f5"}

[meichthys]

Currently, my inbox is filled with non-existent emails, and is basically unusable. I would be thankful for a way to rescan my emails as a temporary workaround.

This is not a fix, but a workaround that usually helps (granted we should not need to run in debug mode to workaround this issue):

• Enable Debug mode in Nextcloud config. php:
  • 'debug' => true,
• In the Nextcloud Mail app, click the ... menu on the folder exhibiting the issue
• Choose Clear cache (Only available in debug mode)

[LostinSpacetime]

This is not a fix, but a workaround that usually helps (granted we should not need to run in debug mode to workaround this issue):

* Enable Debug mode in Nextcloud `config. php`:
  
  * `'debug' => true,`

* In the Nextcloud Mail app, click the `...` menu on the folder exhibiting the issue

* Choose `Clear cache` (Only available in debug mode)

Thank you! I will try this on the weekend, since I would rather not put the cloud in Debug mode on a working day. We are using Nextcloud in our school and several teachers are complaining about this issue. Is there a possibility to clear the cache for all or certain users?

[caltenhofen]

I agree, it would be great to have this as an option for all users.
Or is there any risk in doing so?

[frederikb96]

I would also highly appreciate if the clear cache mode would be available without debug mode, thanks :)

[HLFH]

This issue is a duplicate of #9048.

[alexh-swdev]

After skimming the other issue: No!

[caltenhofen]

I agree from my understanding it's probably a similar but distinct issue. But as I am not too familiar with the codebase the solutions might be connected.

[violoncelloCH]

I observed that this particularly happens for me for mails that get classified as "important" by Nextcloud Mail. Not sure if this is a necessary requirement, but for the last few occurrences I could observe it was always mails tagged as important that then stick in the Nextcloud Mail UI.
Interesting is also, that I can still mark the mail as "not important" while all other actions don't work on the ghost mail.
So maybe that's somehow related to how that "important" mail feature interacts with the mailbox?

[LostinSpacetime]

I want to add that even after turning off the priority emails in the app-settings, in Thunderbird deleted emails keep showing up in the app.

[YourWishes]

Looking for a solution also. Perhaps related to my use case of running a docker image with both PSQL and redis connected?

[c0r3dump3d]

I also have the same problem, in addition to emails it also happens to me with folders, once they are deleted from the imap server from a external client they are not reflected in Nextcloud. NC 28.0.3 and Mail app 3.5.7.

[meichthys]

Looking for a solution also. Perhaps related to my use case of running a docker image with both PSQL and redis connected?

@YourWishes This is likely not an issue with your environment since this has been a longstanding known issue.

[evilphish]

Experiencing the same with mail 3.5.7 and NC 28.0.4. Other clients I use are K9 Mail which is set to immediately delete messages on the server and Thunderbird which is set to expunge on exit. Between K9 and Thunderbird there are no issues, if I delete a mail on one client it disappears on the other. Nextcloud Mail is the only problem child here displaying ghosts of deleted messages in the inbox.

[meichthys]

I see now. If i choose "Delete Permanently" rather than "Move to Trash" or "Archive", then the mail does seem to be reliably sometimes be removed from the Nextcloud inbox.

Moving mail still seems to be problematic on my end, but permanently deleting mail does seem to be fixed 👍

[violoncelloCH]

After testing this a few times over the last days, I can also confirm this to be fixed in v3.7.2 for the previously affected mailbox.

[cwerstuik]

I also have found it's been working well now, but I came across one scenario where it didn't work.

It seems I deleted an email from an external email client, but then I received a 2 more emails in that thread. In Nextcloud mail, all 3 emails showed up in the thread and email 1 showed the "not found" warning and I couldn't delete the whole thread

When I deleted the emails 2 and 3 individually in Nextcloud, the first email also disappeared properly. So technically there's an easy work around, but it's still a glitch and can be confusing

[untuonac]

After updating to version 3.7.2 [Nextcloud Hub 7 28.0.5), the problem would seem to be solved for me. Thank you all.

[MarZi235]

Unfortunately, the issue is not resolved for me. After upgrading to 3.7.2 (running NC 29.0.3.4) and clearing all the caches for all users (via enabling debug and manually clearing the caches), I still see stale e-mail in my inboxes. Anything I can do to help root cause the issue?

[meichthys]

I'm also still seeing the issue as well, but on some rare occasions I do see it work.

For reference, I am using IMAP on purelymail.com

[rikmeijer]

Yes, I am afraid I still see ghost mails in my inbox :'(

[powerpaul17]

Yes, unfortunately I'm still seeing the "ghost" messages. It seems it mostly affects the important ones.

[violoncelloCH]

Did you all clean the mailbox cache after updating to v3.7.2 ? Without that the old ghost mails will stay, the fix only affects new mails.

[meichthys]

These are the steps I took:
Update the app, enable debugging in NC, clicked the clear cache button of my inbox, deleted a mail in another client, refres nc mail, mail was still in NC mail.

[alexh-swdev]

Did you all clean the mailbox cache after updating to v3.7.2 ? Without that the old ghost mails will stay, the fix only affects new mails.

Isn't that sth the upgrade should do on its own?

[meichthys]

A one time clearing of the cache is required after updating: #9168 (comment)

[frederikb96]

Thanks a lot for the update 😃

The new version is also available in the store now 👍

I updated, enabled debug mode, cleared cache and disabled debug mode again. After that, I got a mail, deleted it on my phone and the mail was also gone in Nextcloud Mail. Looks like there is a good chance that the issue is fixed now, thanks a lot 🥳

After a few days of testing, I sadly have to say that the issue still exists. What I did:

• Having a mail account (from posteo.de) being set up in Nextcloud Mail.
• Going on vacation, so not opening Nextcloud Mail or any other Mail client for some time
• Opening the Mail client K9-Mail on my phone: deleting some mails I received during my days off
• Opening Nextcloud Mail, and the once I already deleted via K9-Mail show up as those ghost mails again, which cannot be deleted

Before my vacation, it was fine for two days or so. Probably because, I opened Nextcloud Mail regularly. So deleting mails via other mail client did not have negative impact.

[philiprenich]

I'll also throw in that I am still experiencing this issue. I am on v3.7.2 and have put NC into debug mode and used the revealed clear cache function in Mail. This issue unfortunately isn't an uncommon occurrence, it is consistently happening. All externally deleted mail stays in NC Mail. It unfortunately make the app unusable. At this point I am only keeping it installed so I can keep testing new versions and maybe helping with bug reports. To that end, if there are logs I can post, do let me know.

[ChristophWurst]

#10036 will fix the edge case where IMAP pushes vanishes messages

[LostinSpacetime]

#10036 will fix the edge case where IMAP pushes vanishes messages

Will this fix also clean up the inbox on all accounts?

[ChristophWurst]

Yes #10038. Stay tuned.

[LostinSpacetime]

My menu looks the same:

Do I need to do anything to see the option to "Repair Mailbox"?

[ChristophWurst]

Wait for the change to be released and update the app.

[LostinSpacetime]

I am already running Mail 4.0.1. I thought the change was included in Mail 4.0.1.