"run_cgs_models" runs for a long time without output

[WangRong423]

Hi Scenicplus team,
My "run_cgs_models" step had run for over 3 weeks and with output (n_topics=[10,16,32], n_cpu=40). This is the data size of my cisTopic object with n_cells × n_regions = 412962 × 514566. This is a very large dataset consisting of forty samples combined, but there are some error prompts, is it related to that?

muliti_model_1.log

[SeppeDeWinter]

Hi @WangRong423

Have you considered running the topic modeling step using mallet instead? See: https://pycistopic.readthedocs.io/en/latest/Single_sample_workflow-RTD.html#7b.-Parallel-LDA-with-Mallet.

Mallet is more efficient, certainly for bigger datasets.

Also consider setting save_path to a path of choice, this will save each model intermediately. That way you don't have to restart topic modelling from scratch.

I hope this helps.

Best,
Seppe

[strawberry098]

Hi @SeppeDeWinter

I have set save_path such that each n_topic model is saved intermediately. I split the models=run_cgs_models_mallet() function. For the next step, when calling models, there won't be the full list of n_topic.

Can I do this do "rename" model?
file = open("....Topic2.pkl",'rb')
file1 = open("....Topic5.pkl",'rb')
Topic2 = pickle.load(file)
Topic5 = pickle.load(file1)
models = Topic2 + Topic 5

And then run:
model=evaluate_models(models,
select_model=None,
return_model=True,
metrics=['Arun_2010','Cao_Juan_2009', 'Minmo_2011', 'loglikelihood'],
plot_metrics=False,
save= outDir + 'models/model_selection.pdf')

[SeppeDeWinter]

Hi @strawberry098

Can you clarify what you mean by: "I split the models=run_cgs_models_mallet() function."?

All the best,

Seppe

[strawberry098]

Hi @SeppeDeWinter

To clarify, I ran models=run_cgs_models_mallet() a few times, because the run was long and I lost connection. As a result, the newest models run would overwrite the previous models, but I have all the individual TopicX.pkl files saved. How do I call all those TopicX.pkl files to run the next step?

Thank you.

[SeppeDeWinter]

Hi @strawberry098

Now I get it, you can just read them in a list, like this:

file = open("....Topic2.pkl",'rb')
file1 = open("....Topic5.pkl",'rb')
Topic2 = pickle.load(file)
Topic5 = pickle.load(file1)
models = [Topic2, Topic 5]

I hope this helps?

All the best,

Seppe

[solvi808]

For this step I had the same issue with a very large test dataset (200K cells) and the command run_cgs_models as well as using save_path. This issue is related to running it as a script, and not in a jupyter notebook and not using a conda based environment (which is not even possible or allowed on many clusters).

I also didn't use mallet because it requires me to install and compile something on the computer, which might also require extra libraries and not work easily .

• On an HPC node, the python interpreter will sometimes not produce any standard output until the end of the run , i.e. you will see all the "printed output" be dumped at once the job finishes . See discussion here: https://stackoverflow.com/questions/25897335/why-doesnt-print-output-show-up-immediately-in-the-terminal-when-there-is-no-ne
• For me I had to set 700 GB memory for this step to work, it was going well over 500 GB . You have 400 thousand cells, I am not sure how much memory it will require or if it might be possible if you set up 1TB of memory, it could be more manageable if pycisTopic splits the data into smaller chunks that can be ran in smaller bits somehow.
• I also noticed that IF you hit the memory limit, instead of crashing, a job will just dramatically slow down and no progress really made because it doesn't have enough memory to actually compute. It could play a role here as well