"Bad gateway" on export

[bersbersbers]

I wanted to used this fork for Export all (watsonbox#56) and starred (watsonbox#37), but it returns "Bad gateway" upon clicking any Export or Export All button, while watsonbox/exportify works fine (well, except these two issues above).

[ocram]

Thanks!

Could you try again? For me it’s working perfectly, without any issues. The only thing that might go wrong is that you could hit the rate limitting, in which case you could try your own application ID.

[bersbersbers]

Just tried again, same thing. I'll try again tomorrow morning.

Some thing now, 11 hours laters (different network).

In the Chrome network tab, I can see that these are the failing pages:

https://api.spotify.com/v1/me/tracks?offset=1800&limit=50

Request URL: https://api.spotify.com/v1/me/tracks?offset=1800&limit=50
Request Method: GET
Status Code: 502

access-control-allow-credentials: true
access-control-allow-headers: Accept, App-Platform, Authorization, Content-Type, Origin, Retry-After, Spotify-App-Version
access-control-allow-methods: GET, POST, OPTIONS, PUT, DELETE, PATCH
access-control-allow-origin: *
access-control-max-age: 604800
alt-svc: clear
cache-control: private, max-age=0
content-encoding: gzip
content-length: 86
content-type: application/json; charset=utf-8
date: Mon, 02 Sep 2019 09:16:19 GMT
status: 502
via: 1.1 google

[bersbersbers]

All other playlists are working fine by the way, including some with 400 entries. Just "Starred" does not like me.

[ocram]

Thank you very much for investigating this further!

So the “Saved” (?) list only fails after 1800 entries. How many entries does it have in total?

By the way, “watsonbox/exportify” not showing this behavior is just because it doesn’t even have the “Saved” export, right?

If the problem was rate limiting, the response should indicate this, which doesn’t seem to be the case.

Not sure if there’s anything we can do here – except perhaps trying the same (individual HTTP) request again.

[bersbersbers]

Thank you very much for investigating this further!

So the “Saved” (?) list only fails after 1800 entries. How many entries does it have in total?

It doesn't show a number. "Liked songs" in Spotify has around 120 pages of ~10 songs, so not more than 1800.

By the way, “watsonbox/exportify” not showing this behavior is just because it doesn’t even have the “Saved” export, right?

That hypothesis is compatible with what I observe, yes :)

If the problem was rate limiting, the response should indicate this, which doesn’t seem to be the case.

Alright - it's not to big a deal for now, I was just about to have music moved between Spotify accounts and wanted to backup before - but it'll probably work out anyway.

[ocram]

It doesn't show a number. "Liked songs" in Spotify has around 120 pages of ~10 songs, so not more than 1800.

Do you use one of Spotify’s mobile apps as well? The exact number of liked songs should be visible there.

Further, as the request URL you reported is https://api.spotify.com/v1/me/tracks?offset=1800&limit=50, due to the offset=1800, we know that the export script had progressed past the first 1800 tracks when it failed for you.

Alright - it's not to big a deal for now, I was just about to have music moved between Spotify accounts and wanted to backup before - but it'll probably work out anyway.

Which is a good idea. Using Spotify’s desktop app, you should be able to move your liked songs to a playlist, perhaps even divided into multiple lists. That could help as well.

[bersbersbers]

It doesn't show a number. "Liked songs" in Spotify has around 120 pages of ~10 songs, so not more than 1800.

Do you use one of Spotify’s mobile apps as well? The exact number of liked songs should be visible there.

Yes: it shows 966 liked songs.

Further, as the request URL you reported is https://api.spotify.com/v1/me/tracks?offset=1800&limit=50, due to the offset=1800, we know that the export script had progressed past the first 1800 tracks when it failed for you.

Which appears wrong with only 966 songs. Maybe that is the problem - the lower request all seem to work, but somehow exportify seems to estimate an incorrect number of total likes songs:

Alright - it's not to big a deal for now, I was just about to have music moved between Spotify accounts and wanted to backup before - but it'll probably work out anyway.

Which is a good idea. Using Spotify’s desktop app, you should be able to move your liked songs to a playlist, perhaps even divided into multiple lists. That could help as well.

I did that, too - that creates a new playlist that Exportify sees as having 1002 songs, and is able to download it. Great!

So my problem is solved, but I'm happy to help out if you want to investigate further.

[ocram]

Thanks, that is really helpful and makes perfect sense.

Spotify might have changed the behavior if you exceed the number of available songs in “Saved”.

The limit here is not even guessed by the application, instead it’s hardcoded at 2,500.

You can, however, adjust that limit by opening your browser’s developer tools, going to the JavaScript console, and executing

window.localStorage.setItem("librarySize", 1002);

or whatever may be the current size of your library.

Does it work after that?

[bersbersbers]

Yes! (Be sure to reload the page after setting this.)

[ocram]

Very happy to hear that it works, and thanks for testing this!

Right now, this is the only workaround already implemented. One alternative would be to show a prompt for all users at the beginning asking for their library size, so that you wouldn’t have to set the value in localStorage manually.

A much better solution would be if we could just assume 10,000 entries for everyone’s library and then inspect the API responses to detect the last valid request and stop before we’re getting any errors because we’ve exceeded the library limit.