generate_cohort fails (only) when expectation population size exceeds 900,000

[annaschultze]

This may well be me doing something strange, but I'm able to run an action to generate_cohort and this completes successfully when I set the expectation population size in the project.yaml to 700,000 (or less), but fails without an error message if I increase the size above this (I tried 900,000 and above). I'd like to make the dataset larger because my code consequently filters on a date equivalence and the resulting dataset ends up quite small.

This is the repo: https://github.com/opensafely/vaccine-characteristics-research. The successful and error metadata are:

Succesful metadata with dummy data size at 700,000

Error metadata with dummy data size at 900,000

[evansd]

Ah, it's running out of memory. I can tell that because of that exit_code: 137 line.

How much RAM do you have on your computer? You probably have enough but might need to assign more of it to Docker (which we'll have to work out how to do on Windows).

[annaschultze]

This makes sense - I was so baffled by it..! I've swapped to a Mac if that helps, which tells my I've got 16GB memory.

[annaschultze]

This is not urgent for this afternoon by the way, I can move ahead with the 700k population size, but would be good to know how to do this for the future..! Is there somewhere where we can check what cohort extractor error codes mean? Alternatively might be useful to add as an error message.

[evansd]

Yes, we should absolutely be adding a proper error message for this and I've just created an issue. This isn't a cohortextractor specific thing by the way, it's just one of those weird bits of background Linux knowledge that you pick up that exit code 137 usually means out-of-memory.

If you're on a mac then @ghickman is probably best placed to tell you how to up the Docker memory limit.

[ghickman]

@annaschultze – You can increase Docker for Mac's memory limit by opening the Docker preferences (you probably have the Docker icon in the top right of your screen, right click that, and then select Preferences…).

In the left-hand menu of the window which pops up you should see the Resources item (1), click that then you'll see some sliders to change the various resources, you should be able to change the Memory (2) one to fix your issue.