Inconsistent filtered genes before and after scv.utils.merge function

[jipeifeng]

Hi:
Thanks for the helpful tools.
I have came across an issue at the preprocessing step, particularly at the filter_and_normalize function.

Test 1:
I merged all the loom files and run the filter step:

Test 2:
I merged all the velocity output files (loom files), merged the loom files into an already existing AnnData object, and run the filter step:

And you can see that the number of filtered genes are inconsistent with each other. And would you please point what is the difference. Moreover, should I merge the Anndata object with scvelo object (using scv.utils.merge function) before or after performing the preprocessing step (scv.pp.filter_and_normalize).
Thank you very much !

[WeilerP]

You should first merge, then preprocess the data.

Regarding the different results after filtering: Let's say you detected gene A 5 times (in unspliced) in your first dataset but 0 times in the second. Suppose you are now filtering only based on unspliced counts via

scv.filter_and_normalize(adata, min_counts_u=5)

Filtering the two datasets individually, gene A will not be filtered out in the first dataset. It is, however, filtered out in the second one. If you then merge the two datasets using an inner merge, you will not find gene A in the final AnnData object.
Merging the two datasets prior to filtering, will lead to gene A not being filtered as it is detected at least 5 times.