Using on Mac OS X

[guibor]

Hi!

When I use this on Mac OS, I get the following error with mbsync

Error performing SASL authentication step: SASL(-1): generic failure: Unable to find a callback: 18948

I know this is probably an issue with mbsync, but perhaps there's a way to avoid it without changing isync code?

Thanks!

[pejaab]

Hi,
I expect this has to do with OS X SASL setup. I am currently trying to figure it out myself. As far as I understand one needs to install https://www.cyrusimap.org/sasl/ (can be done via brew). This is where I am stuck currently, as using pluginviewer the only plugin I am shown is EXTERNAL. I think ideally, one would see all plugins coming with cyrus sasl. My assumption is, if that works, one would need to install this xoauth2 plugin into a place where cyrus sasl finds it as well. Then one should be good to go.
As pointed out, I am not there myself. So the description is only based on assumption and what I have read so far ( mostly: https://www.reddit.com/r/commandline/comments/muhnqb/request_for_help_with_mbsync_and_xoauth2_for/ and the documentation of both cyrus sasl and xoauth2 itself).
Maybe it points you in the correct direction... Any help would be highly appreciated

[guibor]

I have reached more or less the same state where I am shown EXTERNAL and now other plugins.

That's as far as I got.

[leothelocust]

Did anyone discover a solution for this? @guibor @pejaab ?

[guibor]

I haven't. Perhaps @moriyoshi might have an idea?

[moriyoshi]

Sorry for leaving this behind too long. The answer depends on the setup, for example where the accompanying SASL library comes from, either the system provided or the one installed on your own, through homebrew, MacPorts, or by building it yourself. Anyways, the number appeared in the error message Unable to find a callback: 18948 indicates the constant number called "callback ID" that is designated by the header file as SASL_CB_* and instructs the SASL library what operation it should delegate to the underlying implementation. 18948 corresponds to 0x4a04, and what drew my attention the most is there is no such designation in the original Cyrus-SASL, but there is, in the Apple's implementation where you can find SASL_CB_OAUTH2_BEARER_TOKEN for that value.

So all I can assume at the moment is that you are most likely using the system SASL library and I need to know more context about why this callback ID is propagated.

[guibor]

Thanks @moriyoshi - what about the context would be helpful? It does seem that the system library is called but I am unsure how to make mbsync call the right library

[ldavison]

Did you compile mbsync --with-sasl=/path/to/cyrus? It compiles and no longer calls the Apple provided implementation, but I wasn't able to get it to work. Running pluginviewer only shows EXTERNAL.

It wasn't immediately clear to me why this was the only plugin listed, so I ended up patching mbsync. By adding the above mentioned callback, I was able to get mbsync working with the Apple provided implementation. Certainly not ideal, but at least it works until I can figure something else out.

[guibor]

Hi @ldavison this sounds promising, do you have more details on the patch? What exactly did you change in mbsync's source?

[guibor]

Specifically, the area of code is in drv_imap.c and seems to be

static sasl_callback_t sasl_callbacks[] = {
	{ SASL_CB_USER,     NULL, NULL },
	{ SASL_CB_AUTHNAME, NULL, NULL },
	{ SASL_CB_PASS,     NULL, NULL },
	{ SASL_CB_LIST_END, NULL, NULL }
};

static int
process_sasl_interact( sasl_interact_t *interact, imap_server_conf_t *srvc )
{
	const char *val;

	for (;; ++interact) {
		switch (interact->id) {
		case SASL_CB_LIST_END:
			return 0;
		case SASL_CB_USER:  // aka authorization id - who to act as
		case SASL_CB_AUTHNAME:  // who is really logging in
			val = ensure_user( srvc );
			break;
		case SASL_CB_PASS:
			val = ensure_password( srvc );
			break;
		default:
			error( "Error: Unknown SASL interaction ID\n" );
			return -1;
		}
		if (!val)
			return -1;
		interact->result = val;
		interact->len = strlen( val );
	}
}

what changes did you make to this area before compiling --with-sasl=/path/to/cyrus?

Do you mean adding a case to the function as in

   case SASL_CB_OAUTH2_BEARER_TOKEN:
	return  0x4a04;

or something else?

Thanks @ldavison!

[ldavison]

@guibor - The following diff will apply against the mbsync tag for v1.4.4, which will get mbsync working with Apple's implementation:

diff --git a/src/drv_imap.c b/src/drv_imap.c
index c5a7aed..b8ac65a 100644
--- a/src/drv_imap.c
+++ b/src/drv_imap.c
@@ -2195,6 +2195,9 @@ static sasl_callback_t sasl_callbacks[] = {
        { SASL_CB_USER,     NULL, NULL },
        { SASL_CB_AUTHNAME, NULL, NULL },
        { SASL_CB_PASS,     NULL, NULL },
+#ifdef __APPLE__
+       { SASL_CB_OAUTH2_BEARER_TOKEN, NULL, NULL },
+#endif
        { SASL_CB_LIST_END, NULL, NULL }
 };

@@ -2212,6 +2215,9 @@ process_sasl_interact( sasl_interact_t *interact, imap_server_conf_t *srvc )
                        val = ensure_user( srvc );
                        break;
                case SASL_CB_PASS:
+#ifdef __APPLE__
+               case SASL_CB_OAUTH2_BEARER_TOKEN:
+#endif
                        val = ensure_password( srvc );
                        break;
                default:

This is most of the patch from this thread: https://sourceforge.net/p/isync/mailman/isync-devel/thread/20210308170245.kq4ubg24jcxezyhf%40bsd-mbp.dhcp.thefacebook.com/#msg37235467; I removed is the suppression of the warning.

I did not have to change my AuthMechs to OAUTHBEARER though, I used XOAUTH2 and it worked fine on macOS 11.6 (Big Sur). The patch will not work when compiling with cyrus-sasl-xoauth2 as SASL_CB_OAUTH2_BEARER_TOKEN is not defined.

[guibor]

Thanks a lot. So this worked!

What I did was:

1. Made the suggested changes to the source code.
2. Compile with SSL: ./configure --with-ssl=/usr/local/opt/openssl\@1.1/. This is critical because without this imap.gmail.com won't respond and using different versions/packages for SSL fails for some reason (the process starts and seems to work, but at the end of the transaction I get a weird SSL error). Don't forget to make clean before building.

[xukai92]

I also managed to make it work using the code patch (which I made avaiable at https://github.com/xukai92/isync.git).
I simply modified the formula of isync in my Homebrew (i.e. removing url,sha256 and change the Git address in head to the one above with brew edit isync) and reinstall with brew reinstall isync --build.
I didn't find it necessary to do the SSL thing as above.

For anyone just started looking into how to set up mbsync with OAUTH2, I closely followed https://cvanelteren.github.io/post/mu4e/, which lead me to the last step of failing to authenticate with mbsync using the obtained code (due to SASL issue), and then came to this issue which the solution above on how to fix that.

[Cerebus]

Updated: Ignore me, I can't even read my own config. Duplicate AuthMechs further down; removed and all is right with the world. :)

OK, I'm clearly missing something here.

• cyrus-sasl installed w/ brew
• cyrus-sasl-xoauth2 compiled and installed in cyrus (shows in pluginviewer)
• isync compiled --with-sasl and --with-ssl
• mbsync -h reports +HAVE_LIBSASL and +HAVE_LIBSSL
• AuthMechs XOAUTH2

mbsync is still trying SASL PLAIN:

Connection is now encrypted
Logging in...
Authenticating with SASL mechanism PLAIN...
IMAP command 'AUTHENTICATE PLAIN <authdata>' returned an error: NO AUTHENTICATE failed.
C: 1/1  B: 0/0  F: +0/0 *0/0 #0/0  N: +0/0 *0/0 #0/0

[bmclean2]

Successfully got this to work using the above code patch. I needed to use ./configure --with-ssl=/usr/local/opt/openssl\@1.1/

Of note: When I run mbsync, I get the following warning on the oauth2 accounts: Warning: SASL wants more steps despite successful IMAP authentication. Ignoring... Anyone else? I'm guessing that this is due to some peculiarity in Apple's SASL. I'm running Mojave.

[jngeist]

Did you compile mbsync --with-sasl=/path/to/cyrus? It compiles and no longer calls the Apple provided implementation, but I wasn't able to get it to work. Running pluginviewer only shows EXTERNAL.

It wasn't immediately clear to me why this was the only plugin listed, so I ended up patching mbsync. By adding the above mentioned callback, I was able to get mbsync working with the Apple provided implementation. Certainly not ideal, but at least it works until I can figure something else out.

I can't speak for you (much less the you of a year ago), but I think the issue is that the ./configure invocation for this package misses something crucial on macos which can cause it to fail in a way that is very easy to overlook. Specifically, on macos trying to install cyrus-sasl-xoauth2 as described into the docs will attempt to install the plugin to the system implementation of sasl, which will fail, even if cyrus-sasl is properly installed.

The solution to this is to direct it to the correct target: ./configure --with-cyrus-sasl=/path/to/cyrus-sasl. Then, make install will actually install the plugin for cyrus-sasl, after which everything should start working properly. (Given all the other givens, like mbsync being compiled to use cyrus-sasl, etc.) Rather than having to hack mbsync to get it to play nicely with the system sasl implementation, this seems to solve the problem using cyrus-sasl and cyrus-sasl-xoauth2, which is what we're nominally here for.

This might be worth noting in the readme for macos users, given the number of us bumbling around here in the issues. :)

[bmclean2]

Rather than having to hack mbsync to get it to play nicely with the system sasl implementation, this seems to solve the problem using cyrus-sasl and cyrus-sasl-xoauth2, which is what we're nominally here for.

I can confirm this works without the aforementioned code hack. To summarize:

./configure --with-sasl=/usr/local/opt/cyrus-sasl --with-ssl=/usr/local/opt/[email protected]

The above assumes you've installed the homebrew cyrus-sasl package and then compiled this repo (cyrus-sasl-xoauth2) plugin to go along with the homebrew (not macOS) sasl version.

I could not get mbsync to work with the isync homebrew formula edited and built with the same configuration options as above and using the same version.

[adithyabhatkajake]

The following worked for me:

$ sudo port install cyrus-sasl-xoauth2 #install from macports
$ # git clone isync & cd into it
$ # Run export commands specified in `brew info berkeley-db@4`
$ ./configure --with-sasl=/opt/local --with-ssl=/opt/homebrew/opt/[email protected]

Hope it helps someone who is also looking to get this working.

[g-simmons]

I used @bmclean2's comment to build mbsync from source. The paths to cyrus-sasl and openssl matched on my system.

I also used this stackoverflow post to build cyrus-sasl-xoauth2 from source

Editing the Makefile to use the correct CYRUS_SASL_PREFIX was the important part. The SO post uses sed, I just manually edited the Makefile that results from the ./configure step.

On my system the correct CYRUS_SASL_PREFIX turned out to be /usr/local/opt/cyrus-sasl/. I'm on MacOS, and I installed cyrus-sasl using brew.

After successful install,

pluginviewer shows an entry like:

Plugin "xoauth2" [loaded], 	API version: 4
	SASL mechanism: XOAUTH2, best SSF: 0
	security flags: NO_ANONYMOUS|PASS_CREDENTIALS
	features: WANT_CLIENT_FIRST|PROXY_AUTHENTICATION

[com4]

i got this to work with brew. Here's roughly what i did

Install cyrus sasl for mbsync to link against

brew install cyrus-sasl libtool

Build this xoauth2 plugin against cyrus-sasl

git clone https://github.com/moriyoshi/cyrus-sasl-xoauth2.git
cd cyrus-sasl-xoauth2
vi autogen.sh

Change libtoolize to glibtoolize, otherwise you'll get a libtoolize not found error. I'm unsure if this warrants a patch.

./autogen.sh \
  && ./configure --with-cyrus-sasl=/opt/homebrew/opt/cyrus-sasl \
  && make \
  && make install 

You should now have sasl with xoauth support. If this command returns nothing something went wrong.

/opt/homebrew/opt/cyrus-sasl/sbin/pluginviewer  | grep XOAUTH

Update brew formula to use cyrus-sasl instead of apple's

brew edit isync

• Add , "--with-sasl=/opt/homebrew/opt/cyrus-sasl" to the ./configure line in the def install section.
• I also added a depends_on "cyrus-sasl" below depends_on "openssl@3" but I'm unsure if this is required since it was manually installed.

Build isync with your local formula

HOMEBREW_NO_INSTALL_FROM_API=1 brew install -s isync

(You can also brew install --HEAD isync which should build from source and has the added benefit (as of 2024-07-16) of including a fix for error:0A000126:SSL routines::unexpected eof while reading when using mbsync with Gmail. https://sourceforge.net/p/isync/isync/ci/b6c36624f04cd388873785c0631df3f2f9ac4bf0/)

mbsync should now work with XOAUTH2 and an access token as the password.

Note: Homebrew doesn't use the local files when building from source. This might be the missing piece @bmclean2 and others needed (https://github.com/orgs/Homebrew/discussions/4286)

[nxtruong]

I followed the instructions by @com4 exactly, but mbsync still gives the error:

Error performing SASL authentication step: SASL(-1): generic failure: Unable to find a callback: 18948

Did anyone have the same issue?

[jorgerojas26]

i got this to work with brew. Here's roughly what i did

Install cyrus sasl for mbsync to link against

brew install cyrus-sasl libtool

Build this xoauth2 plugin against cyrus-sasl

git clone https://github.com/moriyoshi/cyrus-sasl-xoauth2.git
cd cyrus-sasl-xoauth2
vi autogen.sh

Change libtoolize to glibtoolize, otherwise you'll get a libtoolize not found error. I'm unsure if this warrants a patch.

./autogen.sh \
  && ./configure --with-cyrus-sasl=/opt/homebrew/opt/cyrus-sasl \
  && make \
  && make install 

You should now have sasl with xoauth support. If this command returns nothing something went wrong.

/opt/homebrew/opt/cyrus-sasl/sbin/pluginviewer  | grep XOAUTH

Update brew formula to use cyrus-sasl instead of apple's

brew edit isync

• Add , "--with-sasl=/opt/homebrew/opt/cyrus-sasl" to the ./configure line in the def install section.
• I also added a depends_on "cyrus-sasl" below depends_on "openssl@3" but I'm unsure if this is required since it was manually installed.

Build isync with your local formula

HOMEBREW_NO_INSTALL_FROM_API=1 brew install -s isync

(You can also brew install --HEAD isync which should build from source and has the added benefit (as of 2024-07-16) of including a fix for error:0A000126:SSL routines::unexpected eof while reading when using mbsync with Gmail. https://sourceforge.net/p/isync/isync/ci/b6c36624f04cd388873785c0631df3f2f9ac4bf0/)

mbsync should now work with XOAUTH2 and an access token as the password.

Note: Homebrew doesn't use the local files when building from source. This might be the missing piece @bmclean2 and others needed (https://github.com/orgs/Homebrew/discussions/4286)

This worked flawlessly for me. The key part is in the last note.

[nxtruong]

@jorgerojas26 Could you list the steps you did for the last note (make changes to the isync formula and build it)? I followed the steps in the link in the last note: uninstall isync (mbsync no longer exists), extract the modified formula to my own tap (link), untap homebrew-core, then install/build from source from the modified formula in my own tap. The build log shows ./configure --disable-silent-rules --with-sasl=/opt/homebrew/opt/cyrus-sasl which seems correct. However, I still have the same error message when running the newly built mbsync: Error performing SASL authentication step: SASL(-1): generic failure: Unable to find a callback: 18948.

By the way, running /opt/homebrew/opt/cyrus-sasl/sbin/pluginviewer | grep XOAUTH returned text indicating the xoath plugin was installed properly.

[jvillasante]

I'm on fedora and get a slightly different error:

Authenticating with SASL mechanism XOAUTH2...
Error performing SASL authentication step: SASL(-1): generic failure: Unable to find a callback: 32775

Does anybody knows what 32775 means?