Cannot login through caddy reverse proxy (403) #746
Comments
|
Thanks for the ticket. It seems like you have a working WebUI but not a login and none of your requests hit gotify? Can you look at the network and console tab in the devtools and upload what is intuitively not right? Also a good idea might be to enable the access log on caddy and see how caddy route the request. I will retag this as question for now because I think the fact that local access have logs written but the not working remote access has not which strongly suggests the requests did not even hit gotify. |
eternal-flame-AD
added
question
|
Caddy log for gotify Hitting WebUI on fqdn:443 Entering credentials Edit:
|
|
This is highly unlikely to be a genuine gotify response because the only place 403 is returned is if you used an application token for a client operation or vice versa, and you should see a JSON error message indicating the error on the browser devtools if you look at the corresponding request, I assume the 0 at the end mean content-length 0 but a genuine error message should have a non zero content-length. If it helps I use Caddy too and here is my config and it just works, can you try simplify your setup (like use HTTP for the caddy <-> gotify part) and gradually try add back the additional options? |
|
Also just to not miss the obvious have you tried looking at the log of Caddy itself? |
|
FYI: Gotify does return 403 with empty response when there is a failed server side cors request. |
eternal-flame-AD
changed the title
|
Well I see only brackets... I attached my caddyfile.json and gotify-caddy-log.json. Every reverse_proxy host in my caddyfile.json work as it suppose to. As for gotify the android client is working. I'm using gotify only for proxmox notifications as a backup channel when mailserver gets updates and need to restart. I don't know to fix it so if you are willing to share your insights hot to enable server side CORS... Edit: LogsLog while sending credentials{
"level": "info",
"ts": 1733423132.8591142,
"logger": "http.log.access.log0",
"msg": "handled request",
"request": {
"remote_ip": "178.13.xx.xx",
"remote_port": "60464",
"client_ip": "178.13.xx.xx",
"proto": "HTTP/2.0",
"method": "POST",
"host": "gotify.foo.de",
"uri": "/client",
"headers": {
"Content-Type": [
"application/json"
],
"Sec-Fetch-Mode": [
"cors"
],
"Priority": [
"u=0"
],
"Te": [
"trailers"
],
"User-Agent": [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
],
"Accept-Language": [
"de"
],
"Origin": [
"https://gotify.foo.de"
],
"Sec-Fetch-Dest": [
"empty"
],
"Sec-Gpc": [
"1"
],
"Sec-Fetch-Site": [
"same-origin"
],
"Accept-Encoding": [
"gzip, deflate, br, zstd"
],
"Referer": [
"https://gotify.foo.de/"
],
"Authorization": [
"REDACTED"
],
"Content-Length": [
"26"
],
"Accept": [
"application/json, text/plain, */*"
],
"Dnt": [
"1"
]
},
"tls": {
"resumed": false,
"version": 772,
"cipher_suite": 4865,
"proto": "h2",
"server_name": "gotify.foo.de"
}
},
"bytes_read": 26,
"user_id": "",
"duration": 0.003471511,
"size": 0,
"status": 403,
"resp_headers": {
"Content-Type": [
"application/json"
],
"Content-Length": [
"0"
],
"Date": [
"Thu, 05 Dec 2024 18:25:32 GMT"
],
"Server": [
"Caddy"
],
"Alt-Svc": [
"h3=\":443\"; ma=2592000"
]
}
}caddyfile.json{
"logging": {
"logs": {
"default": {
"exclude": [
"http.log.access.log0",
"http.log.access.log1",
"http.log.access.log2",
"http.log.access.log3",
"http.log.access.log4",
"http.log.access.log5",
"http.log.access.log6"
]
},
"log0": {
"writer": {
"filename": "/var/log/caddy/gotify-access.log",
"output": "file"
},
"include": [
"http.log.access.log0"
]
},
"log1": {
"writer": {
"filename": "/var/log/caddy/local-access.log",
"output": "file"
},
"include": [
"http.log.access.log1"
]
},
"log2": {
"writer": {
"filename": "/var/log/caddy/mail-access.log",
"output": "file"
},
"include": [
"http.log.access.log2"
]
},
"log3": {
"writer": {
"filename": "/var/log/caddy/nextcloud-access.log",
"output": "file"
},
"include": [
"http.log.access.log3"
]
},
"log4": {
"writer": {
"filename": "/var/log/caddy/notify-access.log",
"output": "file"
},
"include": [
"http.log.access.log4"
]
},
"log5": {
"writer": {
"filename": "/var/log/caddy/prism-access.log",
"output": "file"
},
"include": [
"http.log.access.log5"
]
},
"log6": {
"writer": {
"filename": "/var/log/caddy/vaultwarden-access.log",
"output": "file"
},
"include": [
"http.log.access.log6"
]
}
}
},
"apps": {
"crowdsec": {
"api_url": "http://192.168.168.173:8080/",
"api_key": "REDACTED",
"ticker_interval": "10s",
"enable_streaming": true,
"enable_hard_fails": false
},
"http": {
"servers": {
"srv0": {
"listen": [
":443"
],
"routes": [
{
"match": [
{
"host": [
"bitwarden.foo.de"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "error",
"status_code": 403
}
],
"match": [
{
"not": [
{
"maxmind_geolocation": {
"allow_asn": null,
"allow_countries": [
"AD",
"AL",
"AT",
"AX",
"BA",
"BE",
"BG",
"CH",
"CZ",
"DE",
"DK",
"EE",
"ES",
"FI",
"FR",
"GB",
"GG",
"GI",
"GR",
"HR",
"HU",
"IE",
"IM",
"IT",
"JE",
"LI",
"LT",
"LU",
"LV",
"MC",
"MD",
"ME",
"MK",
"MT",
"NL",
"NO",
"PL",
"PT",
"RO",
"RS",
"SE",
"SI",
"SK",
"SM",
"TR",
"VA",
"CA",
"US",
"UNK"
],
"allow_metro_codes": null,
"allow_subdivisions": null,
"db_path": "/var/lib/GeoIP/GeoLite2-Country.mmdb",
"deny_asn": null,
"deny_countries": null,
"deny_metro_codes": null,
"deny_subdivisions": null
}
},
{
"remote_ip": {
"ranges": [
"178.13.xx.xx"
]
}
}
]
}
]
},
{
"handle": [
{
"handler": "crowdsec"
},
{
"handler": "reverse_proxy",
"headers": {
"request": {
"set": {
"Host": [
"{http.reverse_proxy.upstream.hostport}"
],
"X-Real-Ip": [
"{http.request.remote.host}"
]
}
}
},
"transport": {
"protocol": "http",
"tls": {}
},
"upstreams": [
{
"dial": "192.168.168.158:8000"
}
]
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"gotify.foo.de"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "error",
"status_code": 403
}
],
"match": [
{
"not": [
{
"maxmind_geolocation": {
"allow_asn": null,
"allow_countries": [
"AD",
"AL",
"AT",
"AX",
"BA",
"BE",
"BG",
"CH",
"CZ",
"DE",
"DK",
"EE",
"ES",
"FI",
"FR",
"GB",
"GG",
"GI",
"GR",
"HR",
"HU",
"IE",
"IM",
"IT",
"JE",
"LI",
"LT",
"LU",
"LV",
"MC",
"MD",
"ME",
"MK",
"MT",
"NL",
"NO",
"PL",
"PT",
"RO",
"RS",
"SE",
"SI",
"SK",
"SM",
"TR",
"VA",
"CA",
"US",
"UNK"
],
"allow_metro_codes": null,
"allow_subdivisions": null,
"db_path": "/var/lib/GeoIP/GeoLite2-Country.mmdb",
"deny_asn": null,
"deny_countries": null,
"deny_metro_codes": null,
"deny_subdivisions": null
}
},
{
"remote_ip": {
"ranges": [
"178.13.xx.xx"
]
}
}
]
}
]
},
{
"handle": [
{
"handler": "crowdsec"
},
{
"handler": "reverse_proxy",
"headers": {
"request": {
"set": {
"Host": [
"{http.reverse_proxy.upstream.hostport}"
],
"X-Real-Ip": [
"{http.request.remote.host}"
]
}
}
},
"transport": {
"protocol": "http",
"tls": {}
},
"upstreams": [
{
"dial": "192.168.168.159:443"
}
]
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"notify.foo.de"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "error",
"status_code": 403
}
],
"match": [
{
"not": [
{
"maxmind_geolocation": {
"allow_asn": null,
"allow_countries": [
"AD",
"AL",
"AT",
"AX",
"BA",
"BE",
"BG",
"CH",
"CZ",
"DE",
"DK",
"EE",
"ES",
"FI",
"FR",
"GB",
"GG",
"GI",
"GR",
"HR",
"HU",
"IE",
"IM",
"IT",
"JE",
"LI",
"LT",
"LU",
"LV",
"MC",
"MD",
"ME",
"MK",
"MT",
"NL",
"NO",
"PL",
"PT",
"RO",
"RS",
"SE",
"SI",
"SK",
"SM",
"TR",
"VA",
"CA",
"US",
"UNK"
],
"allow_metro_codes": null,
"allow_subdivisions": null,
"db_path": "/var/lib/GeoIP/GeoLite2-Country.mmdb",
"deny_asn": null,
"deny_countries": null,
"deny_metro_codes": null,
"deny_subdivisions": null
}
},
{
"remote_ip": {
"ranges": [
"178.13.xx.xx"
]
}
}
]
}
]
},
{
"handle": [
{
"handler": "crowdsec"
},
{
"handler": "reverse_proxy",
"headers": {
"request": {
"set": {
"Host": [
"{http.reverse_proxy.upstream.hostport}"
],
"X-Real-Ip": [
"{http.request.remote.host}"
]
}
}
},
"transport": {
"protocol": "http",
"tls": {}
},
"upstreams": [
{
"dial": "192.168.168.160:443"
}
]
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"prism.foo.de"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "error",
"status_code": 403
}
],
"match": [
{
"not": [
{
"maxmind_geolocation": {
"allow_asn": null,
"allow_countries": [
"AD",
"AL",
"AT",
"AX",
"BA",
"BE",
"BG",
"CH",
"CZ",
"DE",
"DK",
"EE",
"ES",
"FI",
"FR",
"GB",
"GG",
"GI",
"GR",
"HR",
"HU",
"IE",
"IM",
"IT",
"JE",
"LI",
"LT",
"LU",
"LV",
"MC",
"MD",
"ME",
"MK",
"MT",
"NL",
"NO",
"PL",
"PT",
"RO",
"RS",
"SE",
"SI",
"SK",
"SM",
"TR",
"VA",
"CA",
"US",
"UNK"
],
"allow_metro_codes": null,
"allow_subdivisions": null,
"db_path": "/var/lib/GeoIP/GeoLite2-Country.mmdb",
"deny_asn": null,
"deny_countries": null,
"deny_metro_codes": null,
"deny_subdivisions": null
}
},
{
"remote_ip": {
"ranges": [
"178.13.xx.xx"
]
}
}
]
}
]
},
{
"handle": [
{
"handler": "crowdsec"
},
{
"handler": "reverse_proxy",
"headers": {
"request": {
"set": {
"Host": [
"{http.reverse_proxy.upstream.hostport}"
],
"X-Real-Ip": [
"{http.request.remote.host}"
]
}
}
},
"transport": {
"protocol": "http",
"tls": {}
},
"upstreams": [
{
"dial": "192.168.168.153:2342"
}
]
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"mail.foo.de"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "error",
"status_code": 403
}
],
"match": [
{
"not": [
{
"maxmind_geolocation": {
"allow_asn": null,
"allow_countries": [
"AD",
"AL",
"AT",
"AX",
"BA",
"BE",
"BG",
"CH",
"CZ",
"DE",
"DK",
"EE",
"ES",
"FI",
"FR",
"GB",
"GG",
"GI",
"GR",
"HR",
"HU",
"IE",
"IM",
"IT",
"JE",
"LI",
"LT",
"LU",
"LV",
"MC",
"MD",
"ME",
"MK",
"MT",
"NL",
"NO",
"PL",
"PT",
"RO",
"RS",
"SE",
"SI",
"SK",
"SM",
"TR",
"VA",
"CA",
"US",
"UNK"
],
"allow_metro_codes": null,
"allow_subdivisions": null,
"db_path": "/var/lib/GeoIP/GeoLite2-Country.mmdb",
"deny_asn": null,
"deny_countries": null,
"deny_metro_codes": null,
"deny_subdivisions": null
}
},
{
"remote_ip": {
"ranges": [
"178.13.xx.xx"
]
}
}
]
}
]
},
{
"handle": [
{
"handler": "crowdsec"
},
{
"handler": "reverse_proxy",
"headers": {
"request": {
"set": {
"Host": [
"{http.reverse_proxy.upstream.hostport}"
],
"X-Real-Ip": [
"{http.request.remote.host}"
]
}
}
},
"transport": {
"protocol": "http",
"tls": {}
},
"upstreams": [
{
"dial": "192.168.168.154:443"
}
]
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"nw.foo.de"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "error",
"status_code": 403
}
],
"match": [
{
"not": [
{
"maxmind_geolocation": {
"allow_asn": null,
"allow_countries": [
"AD",
"AL",
"AT",
"AX",
"BA",
"BE",
"BG",
"CH",
"CZ",
"DE",
"DK",
"EE",
"ES",
"FI",
"FR",
"GB",
"GG",
"GI",
"GR",
"HR",
"HU",
"IE",
"IM",
"IT",
"JE",
"LI",
"LT",
"LU",
"LV",
"MC",
"MD",
"ME",
"MK",
"MT",
"NL",
"NO",
"PL",
"PT",
"RO",
"RS",
"SE",
"SI",
"SK",
"SM",
"TR",
"VA",
"CA",
"US",
"UNK"
],
"allow_metro_codes": null,
"allow_subdivisions": null,
"db_path": "/var/lib/GeoIP/GeoLite2-Country.mmdb",
"deny_asn": null,
"deny_countries": null,
"deny_metro_codes": null,
"deny_subdivisions": null
}
},
{
"remote_ip": {
"ranges": [
"178.13.xx.xx"
]
}
}
]
}
]
},
{
"handle": [
{
"handler": "crowdsec"
},
{
"handler": "reverse_proxy",
"headers": {
"request": {
"set": {
"Host": [
"{http.reverse_proxy.upstream.hostport}"
],
"X-Real-Ip": [
"{http.request.remote.host}"
]
}
}
},
"transport": {
"protocol": "http",
"tls": {}
},
"upstreams": [
{
"dial": "192.168.168.151:443"
}
]
}
]
}
]
}
],
"terminal": true
}
],
"errors": {
"routes": [
{
"match": [
{
"host": [
"bitwarden.foo.de"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"body": "{http.error.status_code} {http.error.status_text}",
"handler": "static_response"
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"gotify.foo.de"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"body": "{http.error.status_code} {http.error.status_text}",
"handler": "static_response"
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"notify.foo.de"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"body": "{http.error.status_code} {http.error.status_text}",
"handler": "static_response"
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"prism.foo.de"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"body": "{http.error.status_code} {http.error.status_text}",
"handler": "static_response"
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"mail.foo.de"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"body": "{http.error.status_code} {http.error.status_text}",
"handler": "static_response"
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"nw.foo.de"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"body": "{http.error.status_code} {http.error.status_text}",
"handler": "static_response"
}
]
}
]
}
],
"terminal": true
}
]
},
"logs": {
"logger_names": {
"bitwarden.foo.de": [
"log6"
],
"gotify.foo.de": [
"log0"
],
"mail.foo.de": [
"log2"
],
"notify.foo.de": [
"log4"
],
"nw.foo.de": [
"log3"
],
"prism.foo.de": [
"log5"
]
}
}
},
"srv1": {
"listen": [
":80"
],
"routes": [
{
"handle": [
{
"handler": "vars",
"root": "/usr/share/caddy"
}
]
},
{
"match": [
{
"not": [
{
"maxmind_geolocation": {
"db_path": "/var/lib/GeoIP/GeoLite2-Country.mmdb",
"allow_countries": [
"AD",
"AL",
"AT",
"AX",
"BA",
"BE",
"BG",
"CH",
"CZ",
"DE",
"DK",
"EE",
"ES",
"FI",
"FR",
"GB",
"GG",
"GI",
"GR",
"HR",
"HU",
"IE",
"IM",
"IT",
"JE",
"LI",
"LT",
"LU",
"LV",
"MC",
"MD",
"ME",
"MK",
"MT",
"NL",
"NO",
"PL",
"PT",
"RO",
"RS",
"SE",
"SI",
"SK",
"SM",
"TR",
"VA",
"CA",
"US",
"UNK"
],
"deny_countries": null,
"allow_subdivisions": null,
"deny_subdivisions": null,
"allow_metro_codes": null,
"deny_metro_codes": null,
"allow_asn": null,
"deny_asn": null
}
},
{
"remote_ip": {
"ranges": [
"178.13.xx.xx"
]
}
}
]
}
],
"handle": [
{
"handler": "error",
"status_code": 403
}
]
},
{
"handle": [
{
"handler": "crowdsec"
},
{
"handler": "file_server",
"hide": [
"/etc/caddy/Caddyfile",
"/etc/caddy/sites-enabled/local-caddy"
]
}
]
}
],
"errors": {
"routes": [
{
"handle": [
{
"body": "{http.error.status_code} {http.error.status_text}",
"handler": "static_response"
}
]
}
]
},
"logs": {
"default_logger_name": "log1"
}
}
}
},
"tls": {
"automation": {
"policies": [
{
"subjects": [
"bitwarden.foo.de",
"gotify.foo.de",
"notify.foo.de",
"prism.foo.de",
"mail.foo.de",
"nw.foo.de"
],
"issuers": [
{
"email": "[email protected]",
"module": "acme"
},
{
"ca": "https://acme.zerossl.com/v2/DV90",
"email": "[email protected]",
"module": "acme"
}
]
}
]
}
}
}
} |
|
As @jmattheis pointed out, gotify must see a If you would like Gotify to accept additional alternative Origin headers you can add your other possible origins to the You can also bypass this logic altogether by rolling your own logic on Caddy and removing or rewriting the Origin header, but that is more complicated and not warranted unless you have specific logics. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Can the issue be reproduced with the latest available release? (y/n)
yes
Which one is the environment gotify server is running in?
Docker startup command or config file here (please mask sensitive information)
It is a Proxmox LXC Container
/etc/ssl/gotify/foo.bar.lan.crt generated by custom root-CA for the name and ip. Custom root-ca added to system certs.
Do you have an reverse proxy installed in front of gotify server? (Please select None if the problem can be reproduced without the presense of a reverse proxy)
Reverse proxy configuration (please mask sensitive information)
On which client do you experience problems? (Select as many as you can see)
What did you do?
Trying to login into WebUI on FQDN foo.bar.tld:443 getting "Login failed" and nothing in gotify logs while using any browser I own. Tested Firefox, Opera and Safari on debian, macos and android.
Loging into WebUI on local domain foo.bar.lan:443 or on 192.168.8.159:443 without any issue.
Correct logins and simulated incorrect logins appear in /var/log/gotify/gotify.log. I never saw anything in /var/log/gotify/gotify-error.log
Is there anyway to set loglevel?
Loging in with Android Client works on both fqdn and local domain.
What did you expect to see?
Successful login.
I have identical caddy configurations for vaultwarden, nextcloudpi, photoprism and roundcube. All working without any problems.
What did you see instead? (Include screenshots, android logcat/request dumps if possible)
Failed login
The text was updated successfully, but these errors were encountered: