Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: retrieve body structure and process parts on mailbox sync #10046

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

SebastianKrupinski
Copy link
Contributor

@SebastianKrupinski SebastianKrupinski commented Aug 25, 2024

Adjusted mail retrieval logic to pull body structure without contents on partial sync so that attachments (like iMip Messages) can be found.

Part 1 of iMip Messaging Refactoring

Copy link
Member

@ChristophWurst ChristophWurst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's be careful with the performance impact of this change.

The last time I tried, pulling structure increased the data transferred significantly. Hence we offloaded the processing to \OCA\Mail\Service\PreprocessingService::process outside of the initial/partial sync, to keep the sync as fast as possible.
We also don't need to fetch this info for very old messages, because the invites imip messages are probably not relevant anymore.

Do you have any numbers on how much the amount of transferred data increases? An idea for measuring is

  1. Delete data/horde_imap.log
  2. Enable debug mode
  3. Change the imap connection factory to debug_literal=true
  4. Add an account via occ. The account should have at least a few thousand email in the INBOX
  5. Run occ mail:account:sync with blackfire attached

Then inspect the size of horde_imap.log

Do it twice with and without the changes.

@SebastianKrupinski
Copy link
Contributor Author

The last time I tried, pulling structure increased the data transferred significantly.

Well you are correct... sort of... but the data transfer increase has nothing to do with adding BODYSTRUCTURE to the command... sort of... Let me explain....

I tested this with my main account that has about 12K of messages, the difference was about 200MB.

Currently,
- The sync fetches all basic information for all the messages. (does this for every mailbox and stops)

With BODYSTRUCTURE,
- First, the sync fetches all basic information for all the messages including BODYSTRUCTURE.
- Then, this somehow triggers a followup pull on each individual message for BODY.PEEK[Mime Part Number] which pulls the message text body.

So the issue is the followup fetch and no the BODYSTRUCTURE.

I will look in to fixing this.

@SebastianKrupinski
Copy link
Contributor Author

Fixed. The BODYSTRUCTURE no longer triggers another fetch of any body parts.

File sizes difference on 12K of messages is minimal.

image

@kesselb
Copy link
Contributor

kesselb commented Sep 4, 2024

Please update the pull request description what problem it does fix and how to reproduce the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: 🏗️ In progress
Development

Successfully merging this pull request may close these issues.

3 participants