Proxy support?

[houchin]

Does Maestral support web proxies (and in particular proxies that require authentication)? If so, how is that configured when using the Mac bundle? I'm not seeing anything in the documentation about proxies.

[samschott]

Maestral will respect any system-wide proxy settings, as configured under "System Preferences > Network > Advanced...". It will also respect any proxy settings detected from environment variables, though you will need to take care that the process is indeed launched with the correctly configured environment. Apart from that, there is currently no way to configure individual proxy settings for Maestral.

[jshouchin]

Upon launch of Maestral 1.6.2 on macOS 12.3.1, Maestral did ask for permission to access my proxy password from keychain. However, when I hit the link button after entering the authorization token, I got an error "Connection failed. Please make sure that you are connected to the internet and try again."

Not having dived into the Maestral internals, I wonder if you've got two separate network parts going, like the official Dropbox macOS client. The DropboxUpdater tool looks like it does things a more Mac like way; it appears to pull proxy settings for system preferences and asks for access to the proxy keychain. However, the main Dropbox daemon uses its own internal proxy configuration. My proxy password changes every day, and for Dropbox I have to go into Dropbox prefs and manually change it. It rotates through known values, so I've been trying to find where Dropbox actually stores that configuration item and see if I can script up changing it (I was able to do this for Synology Drive; I just had to manually change it to each value and then look in the config for the salted value).

Back to Maestral, is there a missing piece here in getting it to use the proxy password specified in my keychain.

I did set the log mode to DEBUG, but there was still nothing in the log related to the connection attempt.

Alternately to a fix in Maestral itself, I can write my own launch agent to run a script that manually sets the http_proxy and https_proxy environment variables (are those two sufficient?), and then runs Maestral as the official launch agent does. Aside from manually configuring start at login, are there any times when Maestral would overwrite com.samschott.maestral.maestral.plist; I ask because the Synology Drive client overwrites it (at least whenever there's an app update), and I had to remove write permission to prevent it from overwriting my customized version.

[samschott]

Maestral relies on the requests library for all requests which in turn uses Python's urllib.request.getproxies() method to get Proxy settings, either from environment variables or from system settings. This is used for all requests, so none should receive any special treatment.

Could you check in a Python shell what the following snippet returns?

import urllib.request
urllib.request.getproxies()

This should ideally show the correct settings. If it does, something is going wrong with how Maestral creates sessions. If it does not, it means that Python's default logic to get Proxy settings fails in your case.

I can write my own launch agent to run a script that manually sets the http_proxy and https_proxy environment variables (are those two sufficient?), and then runs Maestral as the official launch agent does.

Yes, that should work. Maestral won't overwrite com.samschott.maestral.maestral.plist unless you manually toggle the "start on login" option which will either delete or recreate this file.

[houchin]

That snippet does return the correct proxy host and ports (I moved aside my .zprofile file first to be sure it wasn't getting it from an environment variable that Maestral wouldn't have).

But it doesn't include any indication that proxy authentication is required. As I understand it, authentication is generally handled by whatever is making the connection to the proxy catching the HTTP 407 Proxy Authentication Required error, and then the next time providing the additional header with the proxy user/password.

[samschott]

But it doesn't include any indication that proxy authentication is required. As I understand it, authentication is generally handled by whatever is making the connection to the proxy catching the HTTP 407 Proxy Authentication Required error, and then the next time providing the additional header with the proxy user/password.

Indeed, therein lies the issue. We don't currently handle Proxies that require authentication by passing the correct headers. I'm not particularly familiar with how auth credentials for Proxies are stored on different platforms and how to fetch them in a reliable way.

I'd prefer either using a third party library here, or going down the same route as Dropbox with asking for user input and storing the credentials ourselves.

[jshouchin]

I tried to automate setting the proxy environment variables through a LaunchAgent, but it didn't work.

I have a shell script that sets/exports http_proxy (plus the other variations), PYTHONOPTIMIZE and LC_CTYPE (per the official LaunchAgent plist file). It then runs:

/Applications/Maestral.app/Contents/MacOS/Maestral -m maestral_cocoa -c maestral

just like the official LaunchAgent.

That shell script is run by an AppleScript compiled into an application, and full disk permissions were given to that app (running bash directly from a LaunchAgent is problematic due to those system permissions).

Maestral does start up and the menu bar item runs, but there are two problems:

First, Maestral can't connect. To get it to connect, I had to run "maestral stop" and "maestral start" from a shell that had the proxy environment variables correctly set. I'm thinking that the environment variables set in the shell aren't making it all the way into the code in the app that actually does the work, but I can't confirm this.

Second, that AppleScript app doesn't exit and sits in my dock. I'm guessing that the Maestral app itself isn't quitting (maybe that's the actual GUI, which I don't want to quit). I would try changing the script to run "maestral gui", but that gets me the known issue with it seeing the wrong version of the bundled dropbox code.

Any suggestions?

[samschott]

Could you set the log level to DEBUG with maestral log level DEBUG and check the log output after startup? As part of the output, it will print all environment variables that the sync daemon sees. This should help clarify if they are propagated properly to the daemon process.

[jshouchin]

Ok, something weird is going on. I turned on debug logs, then logged out and back in. I am indeed getting debug level logs. However, when Maestral is started up through "/Applications/Maestral.app/Contents/MacOS/Maestral -m maestral_cocoa -c maestral", it is NOT putting the environment into the log. Only if I "maestral stop;maestral start" do I get the environment; but that environment is coming from the bash shell I'm running the maestral CLI in, which is not necessarily the same as what's being passed on when I run /Applications/Maestral.app/Contents/MacOS/Maestral.

I did try unsetting the four proxy environment variables and then restarting Maestral through the CLI, and in that case it did NOT show them in the environment (expected) and Maestral was reporting proxy authentication errors. Restarted from a fresh shell with the proxy environment variables set, and then launched the Maestral app by double-clicking to get the GUI and things are working again.

That said, even when Maestral was NOT showing the environment (started by running /Applications/Maestral.app/Contents/MacOS/Maestral from the bash script run by a compiled AppleScript app started by the LaunchAgent, I think it was running correctly. I put a test file into the Maestral version of my dropbox folder, and the GUI reported it was synced, and a moment later it showed up in the Dropbox native app version of the folder. I deleted it from the Dropbox native folder and a moment later it was deleted from the Maestral version.

I did log out and log back in several times, clearing the log a few times to verify I was looking at the new log statements, and in no case did the environment show up in the log even though I saw other DEBUG level log statements.

(Where is the actual log file, being able to examine it through BBEdit or some other app would have been helpful.)

So assuming it is working correctly even though it's not printing the environment, the only remaining problem is that the compiled AppleScript app doesn't exit and I have to force quit it. Maestral does continue running, which sort of surprised me. I'm not sure how Maestral is then forking off the sync daemon and GUI, but it appears something isn't exiting, thus the bash shell doesn't exit and thus the AppleScript app doesn't exit. I could try running /Applications/Maestral.app/Contents/MacOS/Maestral in the background, but not sure if that will fully disassociate it from the shell so whatever needs to stay running after the shell exits does stay running.

Do you want me to create a formal bug report on the no environment to log when running /Applications/Maestral.app/Contents/MacOS/Maestral?

I can be patient and wait for a fix to both that and the not exiting issue (if that is indeed a bug), or I can try to install Maestral differently so "maestral gui" works. Suggestions?

Thanks!
Scott

[jshouchin]

To add a few details on what gets run by the launch agent:

Modified com.samschott.maestral.maestral.plist to be:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>Label</key>
        <string>us.houchin.maestral</string>
        <key>ProgramArguments</key>
        <array>
                <string>/usr/bin/open</string>
                <string>/Users/.../local/applescripts/UpdateMaestralProxyAndStart.app</string>
        </array>
        <key>RunAtLoad</key>
        <true/>
</dict>
</plist>

That compiled AppleScript app "UpdateMaestralProxyAndStart.app" runs:

on run
	do shell script "/bin/bash /Users/.../local/scripts/UpdateMaestralProxyAndStart.sh"
end run

And finally the bash script "UpdateMaestralProxyAndStart.sh"

#!/bin/bash

proxy=...
export http_proxy=$proxy
export https_proxy=$proxy
export HTTP_PROXY=$proxy
export HTTPS_PROXY=$proxy

export PYTHONOPTIMIZE="2"
export LC_CTYPE="UTF-8"

/Applications/Maestral.app/Contents/MacOS/Maestral -m maestral_cocoa -c maestral

[houchin]

OK, I'm going to reverse part of what I said yesterday. I do not think it's running correctly when started through "/Applications/Maestral.app/Contents/MacOS/Maestral -m maestral_cocoa -c maestral"

I looked at the log this morning, and there was a proxy authentication error front and center. So I don't think it's getting the environment at all.

But today, I left the compiled AppleScript app running and then ran "maestral stop; maestral start" from the command line and the app exited by itself.