Supabase SMTP Integration Issue with Gmail - Solution Proposal

[saschTa]

Bug report

• I confirm this is a bug with Supabase, not with my own application.
• I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

SMTP Service degraded

I have setup a custom smtp service (Gmail Workspace relay) with supabase projects some while ago.
All worked flawlessly but now since the last week ( 21st of March 2024) sending out magic links and verification emails are timing out.

In Discord similar issues pop up.

Auth Logs just show:
info: /magiclink | request started

EdgeAPI Logs show this response error by POST /auth/v1/magiclink:

{
...
  "origin_time": 60127,
  "status_code": 504
}

To Reproduce

Configure gmail smtp relay service

Supabase Config

• use smtp-relay.gmail.com:465 as host
• use an admin account of workspace with 2fa enabled and create app password for said account
• use email of this account as user in supabase smtp config
• use app password as password
• use custom smtp server in supabase
• use any sender email from your domain

Gcloud Workspace:

• enable smtp relay service in google workspace and force tls.
• Don't enforce IPs
• allow sending from emails in your domain
• create app password in your account settings

Supabase Emails

• trigger a magic link or try to login with email verification on

Expected behavior

Emails should be sent successfully as it worked before the 21st of March.
Emails work with the exact same credential configuration using telnet or openssl

Solution Proposal

Issue

So I digged through supabase auth code and found the culprit:

• gmail workspace relay service does not accept HELO localhost !
• gomail used in mailme default dialer uses the default dialer which again uses localhost as HELO message and gmail rejects it: gomail default

Example gmail relay with localhost:

220 smtp-relay.gmail.com ESMTP m17-20020a5d6a11000000b0033e55077fd9sm4288wru.69 - gsmtp
helo localhost
421-4.7.0 Try again later, closing connection. (EHLO)
421-4.7.0  For more information, go to
421 4.7.0  https://support.google.com/a/answer/3221692 m17-20020a5d6a11000000b0033e55077fd9sm4288wru.69 - gsmtp
read:errno=0

Solution Proposal

mailme
should do something along this:

	dial := gomail.NewPlainDialer(m.Host, m.Port, m.User, m.Pass)
        dial.LocalName = m.Host
	return dial.DialAndSend(mail)

GoMail PR

There is a gomail pr that aims to fix rejects because of default localhost HELO message but I think supabase / supabase/auth should make sure to not rely on defaults here.
go mail fix pr

Additional context

I created a ticket with google as well on this issue. Trying manual telnet to the smtp relay service works fine with the same credentials provided to supabase and basic setup.
Google adviced me, that the ip of supabase instance was not blocked so something must be wrong with the implementation now of the smtp service at supabase. Which I mentioned above.
Here is a working example using openssl with the same credentials used for supabase. Just HELO message is not localhost

openssl s_client -connect smtp-relay.gmail.com:465 -crlf
CONNECTED(00000005)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = smtp-relay.gmail.com
verify return:1
---
Certificate chain
 0 s:CN = smtp-relay.gmail.com
   i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
---
Server certificate
{CERT}
subject=CN = smtp-relay.gmail.com

issuer=C = US, O = Google Trust Services LLC, CN = GTS CA 1C3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4306 bytes and written 402 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
{SSL INFO}
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
{SSL INFO}
---
read R BLOCK
220 smtp-relay.gmail.com ESMTP cm22-20020a170906f59600b00a4df3dcc089sm75400ejd.153 - gsmtp
HELO smtp-relay.gmail.com        
250 smtp-relay.gmail.com at your service
auth login
334 VXNlcm5hbWU6
{email of workspace user base64 encoded}
334 UGFzc3dvcmQ6
{App Password of workspace user base64 encoded}
235 2.7.0 Accepted
mail from:<{email in domain}>
250 2.1.0 OK cm22-20020a170906f59600b00a4df3dcc089sm75400ejd.153 - gsmtp
rcpt to:<{some other email}>
250 2.1.5 OK cm22-20020a170906f59600b00a4df3dcc089sm75400ejd.153 - gsmtp
data
354  Go ahead cm22-20020a170906f59600b00a4df3dcc089sm75400ejd.153 - gsmtp
This is a test.
.
250 2.0.0 OK  1711554331 cm22-20020a170906f59600b00a4df3dcc089sm75400ejd.153 - gsmtp
quit
221 2.0.0 closing connection cm22-20020a170906f59600b00a4df3dcc089sm75400ejd.153 - gsmtp
read:errno=0

[jkgatt]

Same issue here!

[GabeRoze]

I believe I am having the same issue. Disabling "Enable Custom SMTP" appears to have fixed the issue temporarily.

[GabeRoze]

Including my Supabase logs but I doubt they are of any value.

supabase_logs.csv

[saschTa]

I found the culprit in the codebase. At least I think :). Will try to create a PR if possible.

[saschTa]

I have created an issue on mailme which seems to be supabases mail sending lib. It's an easy fix :)
Mailme PR
Supabase/Auth Issue

[kangmingtay]

Hi @saschTa, thanks for the PRs - i'll look into reproducing the error and see if it fixes things. AFAIK we haven't changed anything related to the SMTP package recently so I'm quite curious why it broke for you suddenly.

Just a sanity check, have you seen this guide on setting up google SMTP (https://github.com/orgs/supabase/discussions/5320#discussioncomment-7836793)? I'm wondering if this is just a configuration issue and can be fixed with better documentation.

[sapphire008]

I am having the same issue. Tried my set up with a simple Python script and it is working okay to send the email. Can confirm that only when switching to Supabase SMTP with Gmail, the server gives 504 time out. We followed the set up from https://github.com/orgs/supabase/discussions/5320#discussioncomment-7836793 as pointed by by @kangmingtay . I can also confirm this set up was working during the week of March 11, 2024.

[GabeRoze]

Hi @saschTa, thanks for the PRs - i'll look into reproducing the error and see if it fixes things. AFAIK we haven't changed anything related to the SMTP package recently so I'm quite curious why it broke for you suddenly.

Just a sanity check, have you seen this guide on setting up google SMTP (https://github.com/orgs/supabase/discussions/5320#discussioncomment-7836793)? I'm wondering if this is just a configuration issue and can be fixed with better documentation.

I followed this guide and had auth email working until suddenly stopping this week.

[saschTa]

Hi @saschTa, thanks for the PRs - i'll look into reproducing the error and see if it fixes things. AFAIK we haven't changed anything related to the SMTP package recently so I'm quite curious why it broke for you suddenly.

Just a sanity check, have you seen this guide on setting up google SMTP (https://github.com/orgs/supabase/discussions/5320#discussioncomment-7836793)? I'm wondering if this is just a configuration issue and can be fixed with better documentation.

I can assure you I have checked everything in the Docs. Also the documentation that is provided has some minor issues.
The Advantage of smtp-relay.google.com is that the sending Email can be any Mail adress within your Domain. Therefore it's way better to use this host vs. normal Gmail SMTP host which relies on a workspace user and E-Mails will only be sent from this users adress.

For why it stopped so suddenly. I assume google just enforces now the rfc standard about helo messages. Localhost is and was never the right thing to send.

At the end my little test using manual smpt through openssl is a minimal reproducable step to test If localhost helo works. It does Not For me with google relay SMTP.
Since supabase uses gomail the behaviour is clearly described about default helo localhost.

I haven't checked if anything else is also off but in general smtp is quite simple and this was the most obvious issue I could find.

Happy to help if you have issues reproducing or if the fix does not Work.

[kangmingtay]

hey everyone, we've merged @saschTa changes into the mailme package and will be testing it out with the auth service before we roll it out to all projects.