🐛 [Bug] Gmail messages not shown

[indridieinarsson]

🐛 Bug

I have an install using sqlite database, redis and memcached enabled, running as a docker image.

I have 3 different accounts, one of them gmail. The other two accounts (on two separate and different servers) work as expected, both in single-mailbox mode (under the Email tab) and in the combined views.

Gmail, on the other hand has problems. After restarting cypht, it works, showing me mails in single-mode and in the combined views.
After just a few navigations, cypht stops showing any messages from my gmail account.
That goes for both viewing the gmail INBOX directly (which shows no messages, although there are thousands) and also the combined views, which show no message from the gmail account, only from the other two.

It may be of importance here, that my gmail inbox is pretty huge, containing my entire email history for years.

Version & Environment

Rev: 412b5ad
OS: Linux x86 / docker

[indridieinarsson]

I analysed this a bit. Cypth (modules/imap/hm-imap-base.php) is successfully communicating with gmail. Getting settings, and the list of messages. All of it, the first timestamp is 2006. My guess is that it then goes on to fetch a whole lot of messages and something times out, but I haven't confirmed that yet. My feeling is that gmail is not answering imap requests very fast (throttling them even).

Now, perhaps, I should tidy up my inbox, but then again, I'm hardly the only one using the search function instead of organized folders...
It's also worth noting that (at least a couple of weeks ago) everything worked ok on a mailbox which is a mirror of my gmail box. But that one is faster.

[indridieinarsson]

I looked a bit into this. I see the following imap command being sent to gmail imap server :

A8 UID FETCH 53418,53419,53420,53421,53422,53423,53424,53425,53429,53432,53436,53437,53438,53439,53440,53441,53442,5 3443,53444,53445 (FLAGS INTERNALDATE RFC822.SIZE X-GM-MSGID X-GM-THRID X-GM-LABELS BODY.PEEK[HEADER.FIELDS (SUBJECT X-AUTO-BCC FROM DATE CONTENT-TYPE X-PRIORITY TO LIST-ARCHIVE REFERENCES ME SSAGE-ID X-SNOOZED)] BODY.PEEK[0.1])

which then yields the answer :
A8 BAD Could not parse command

[indridieinarsson]

Ok, it's BODY.PEEK[0.1] that's causing this. Apparently not supported in this form by gmail. If I change it to BODY.PEEK[1], then it works.
And also if I disable the preview mail option (which controls the BODY.PEEK imap option), then i get gmail messages again.

[indridieinarsson]

Follow up : after disabling preview mail option (and also locally changing BODY.PEEK[0.1] to BODY.PEEK[1]), things only sort-of work. If I nuke local cache in the browser, I get gmail message in the "Everything" view.
However, after a while, when I refresh the view, the gmail messages are no longer there (but the others are). The dedicated gmail Inbox still has all the gmail messages (but the "Everything" view is sort of the best feature of cypht...)

[christer77]

Hello @indridieinarsson,

thank you for raising the issue,
thank you also for the proposed solution.
We get to it and get back to you early.

[indridieinarsson]

Hello @indridieinarsson,

thank you for raising the issue, thank you also for the proposed solution. We get to it and get back to you early.

Hi,

thanks for looking into it. And my "solution" isn't really a solution yet, as it goes only from "never works" to "works for an hour then no more".

[indridieinarsson]

Now, I'm tracking this a bit. Seems that the xoauth2 logon is unsuccessful. In hm-imap.php, I am landing here:

      if (mb_stristr($response, 'A' . $this->command_count . ' OK')) {
                $authed = true;
                $this->state = 'authenticated';
            } elseif (mb_strtolower($this->auth) == 'xoauth2' && preg_match("/^\+ ([a-zA-Z0-9=]+)$/", $response, $matches)) {
                    $jsonString = base64_decode(mb_substr($response,2));
                    Hm_Functions::error_log(sprintf(' ' . $this->server . ' got response ' . $jsonString));
                $this->send_command("\r\n", true);
                $this->get_response();
            }

the base64 encoded json string in the reply says : {"status":"400","schemes":"Bearer","scope":"https://mail.google.com/"}
which is a bit weird, since I'm successfully logged into gmail imap server if I go to the dedicated gmail inbox. Also "Everything" view successfully logged in an hour earlier.

[indridieinarsson]

So, it appears that this is caused by the oauth2 bearer token expiring.
When I'm in the "gmail dedicated" inbox, there is a function imap_refresh_oauth2_token which is regularly being called to check validity of the oauth2 token, and refresh it if needed.
When in the "Everything" view, this function never gets called, so the token expires (after an hour) and subsequent login attempts are refused.
If I navigate to the gmail-inbox, the token is refreshed, and I can enjoy one more hour of "Everything" view with everything actually in it, but then it expires again.

[christer77]

Hello @indridieinarsson,
thank you for raising the issue, thank you also for the proposed solution. We get to it and get back to you early.

Hi,

thanks for looking into it. And my "solution" isn't really a solution yet, as it goes only from "never works" to "works for an hour then no more".

seen somewhere

[christer77]

Bonjour,
Merci d’avoir soulevé la question, merci aussi pour la solution proposée. Nous nous y rendons et vous revenons plus tôt.

Salut

Merci de vous y être penché. Et ma « solution » n’en est pas encore vraiment une, car elle va seulement de « ne fonctionne jamais » à « fonctionne pendant une heure puis plus maintenant ».

RFC