The task export_all_contacts collects all contacts in memory.

[hodgestar]

For accounts with lots of contacts, this will run out of RAM and fail. We need to ensure it does not collect all contacts in memory and collects them on disk instead.

Alternatively, we need to add a streaming contacts export.

[jerith]

There are a variety of problems with doing this sort of export by email, especially with large data volumes (which is the cause of the memory usage issue here). We should probably expose the streaming HTTP version and kill the email entirely.

[smn]

I'm a bit -1 on streaming to end users. Even for developers it's a major undertaking to download the kinds of JSON/CSV files we produce in a streaming fashion, Internet dies, computers go to sleep, batteries run out etc... And worst of all, we don't support resuming streaming from where things left off.

I'd be +1 on a streaming API that is used internally to stream to some temporary location and then use nginx to serve it as a static & resumable download after having emailed the download link to the end user.

[rudigiesler]

+1 on smn's idea. A paginated response does solve all the problems of resuming, but does require some sort of client.