mixcr downsample error: "Require clnx file type, got ./clns"

[jvpon]

Dear mixcr people,

After running
mixcr analyze takara-mouse-tcr-cdr3
I have only .clns files, which I link to the folder ./clns

I'd like to obtain normalized read counts for clonotypes.

I am trying to run
mixcr downsample \ --chains TRA,TRB \ --downsampling count-reads-auto \ --only-productive \ --summary summary.tsv \ ./clns
and get an error
Require clnx file type, got ./clns

Thanks,

Julia

[mizraelson]

Hi,
if you have .clns files in the clns folder the command should be:

mixcr downsample \ 
--chains TRA,TRB \ 
--downsampling count-reads-auto \
--only-productive \
--summary summary.tsv \
 ./clns/*.clns 

[jvpon]

This (above) command produces empty files giving output
[.., clns, IDB01.clns, :, isDropped=, true, nClonesBefore=, 394605, nClonesAfter=, 0, sumWeightBefore=, 981404.0, sumWeightAfter=, 0.0] [.., clns, IDB02.clns, :, isDropped=, true, nClonesBefore=, 466130, nClonesAfter=, 0, sumWeightBefore=, 1294519.0, sumWeightAfter=, 0.0] [.., clns, IDB03.clns, :, isDropped=, true, nClonesBefore=, 393301, nClonesAfter=, 0, sumWeightBefore=, 924685.0, sumWeightAfter=, 0.0] [.., clns, IDB04.clns, :, isDropped=, true, nClonesBefore=, 491381, nClonesAfter=, 0, sumWeightBefore=, 1368943.0, sumWeightAfter=, 0.0] [.., clns, IDB06.clns, :, isDropped=, true, nClonesBefore=, 548327, nClonesAfter=, 0, sumWeightBefore=, 1969945.0, sumWeightAfter=, 0.0] [.., clns, IDB07.clns, :, isDropped=, true, nClonesBefore=, 426671, nClonesAfter=, 0, sumWeightBefore=, 1035123.0, sumWeightAfter=, 0.0] [.., clns, IDB08.clns, :, isDropped=, true, nClonesBefore=, 406755, nClonesAfter=, 0, sumWeightBefore=, 962089.0, sumWeightAfter=, 0.0] [.., clns, IDB09.clns, :, isDropped=, true, nClonesBefore=, 532787, nClonesAfter=, 0, sumWeightBefore=, 1687519.0, sumWeightAfter=, 0.0] [.., clns, IDB10.clns, :, isDropped=, true, nClonesBefore=, 408513, nClonesAfter=, 0, sumWeightBefore=, 951190.0, sumWeightAfter=, 0.0] [.., clns, IDB11.clns, :, isDropped=, true, nClonesBefore=, 421554, nClonesAfter=, 0, sumWeightBefore=, 1021150.0, sumWeightAfter=, 0.0] [.., clns, IDB12.clns, :, isDropped=, true, nClonesBefore=, 379159, nClonesAfter=, 0, sumWeightBefore=, 857995.0, sumWeightAfter=, 0.0]
and generated *.clns files that are empty.

But it works when only one chain is specified:
mixcr downsample --force-overwrite\ --chains TRA \ --downsampling count-reads-auto \ --only-productive \ --summary summary_TRA.tsv \ --verbose \ ../clns/*.clns

the output:
[.., clns, IDB01.clns, :, isDropped=, false, nClonesBefore=, 394605, nClonesAfter=, 147507, sumWeightBefore=, 981404.0, sumWeightAfter=, 391809.0] [.., clns, IDB02.clns, :, isDropped=, false, nClonesBefore=, 466130, nClonesAfter=, 155088, sumWeightBefore=, 1294519.0, sumWeightAfter=, 391809.0] [.., clns, IDB03.clns, :, isDropped=, false, nClonesBefore=, 393301, nClonesAfter=, 150408, sumWeightBefore=, 924685.0, sumWeightAfter=, 391809.0] [.., clns, IDB04.clns, :, isDropped=, false, nClonesBefore=, 491381, nClonesAfter=, 158452, sumWeightBefore=, 1368943.0, sumWeightAfter=, 391809.0] [.., clns, IDB06.clns, :, isDropped=, false, nClonesBefore=, 548327, nClonesAfter=, 154934, sumWeightBefore=, 1969945.0, sumWeightAfter=, 391809.0] [.., clns, IDB07.clns, :, isDropped=, false, nClonesBefore=, 426671, nClonesAfter=, 153947, sumWeightBefore=, 1035123.0, sumWeightAfter=, 391809.0] [.., clns, IDB08.clns, :, isDropped=, false, nClonesBefore=, 406755, nClonesAfter=, 151598, sumWeightBefore=, 962089.0, sumWeightAfter=, 391809.0] [.., clns, IDB09.clns, :, isDropped=, false, nClonesBefore=, 532787, nClonesAfter=, 157983, sumWeightBefore=, 1687519.0, sumWeightAfter=, 391809.0] [.., clns, IDB10.clns, :, isDropped=, false, nClonesBefore=, 408513, nClonesAfter=, 153495, sumWeightBefore=, 951190.0, sumWeightAfter=, 391809.0] [.., clns, IDB11.clns, :, isDropped=, false, nClonesBefore=, 421554, nClonesAfter=, 153514, sumWeightBefore=, 1021150.0, sumWeightAfter=, 391809.0] [.., clns, IDB12.clns, :, isDropped=, false, nClonesBefore=, 379159, nClonesAfter=, 147595, sumWeightBefore=, 857995.0, sumWeightAfter=, 374355.0]

Then
mixcr downsample --force-overwrite\ --chains TRB \ --downsampling count-reads-auto \ --only-productive \ --summary summary_TRB.tsv \ --verbose \ ../clns/*.clns

outputs
[.., clns, IDB01.clns, :, isDropped=, false, nClonesBefore=, 394605, nClonesAfter=, 189898, sumWeightBefore=, 981404.0, sumWeightAfter=, 449224.0] [.., clns, IDB02.clns, :, isDropped=, false, nClonesBefore=, 466130, nClonesAfter=, 200307, sumWeightBefore=, 1294519.0, sumWeightAfter=, 449224.0] [.., clns, IDB03.clns, :, isDropped=, false, nClonesBefore=, 393301, nClonesAfter=, 195120, sumWeightBefore=, 924685.0, sumWeightAfter=, 449224.0] [.., clns, IDB04.clns, :, isDropped=, false, nClonesBefore=, 491381, nClonesAfter=, 204954, sumWeightBefore=, 1368943.0, sumWeightAfter=, 449224.0] [.., clns, IDB06.clns, :, isDropped=, false, nClonesBefore=, 548327, nClonesAfter=, 205009, sumWeightBefore=, 1969945.0, sumWeightAfter=, 449224.0] [.., clns, IDB07.clns, :, isDropped=, false, nClonesBefore=, 426671, nClonesAfter=, 201132, sumWeightBefore=, 1035123.0, sumWeightAfter=, 449224.0] [.., clns, IDB08.clns, :, isDropped=, false, nClonesBefore=, 406755, nClonesAfter=, 199143, sumWeightBefore=, 962089.0, sumWeightAfter=, 449224.0] [.., clns, IDB09.clns, :, isDropped=, false, nClonesBefore=, 532787, nClonesAfter=, 207526, sumWeightBefore=, 1687519.0, sumWeightAfter=, 449224.0] [.., clns, IDB10.clns, :, isDropped=, false, nClonesBefore=, 408513, nClonesAfter=, 200871, sumWeightBefore=, 951190.0, sumWeightAfter=, 449224.0] [.., clns, IDB11.clns, :, isDropped=, false, nClonesBefore=, 421554, nClonesAfter=, 200794, sumWeightBefore=, 1021150.0, sumWeightAfter=, 449224.0] [.., clns, IDB12.clns, :, isDropped=, false, nClonesBefore=, 379159, nClonesAfter=, 188721, sumWeightBefore=, 857995.0, sumWeightAfter=, 417221.0]

But now I struggle to export clones in a human-readable format.
Either
mixcr exportClonesPretty \ --chains TRA \ *.TRA.downsampled.clns
or
mixcr exportClonesPretty *.clns

gives
Unmatched arguments

Also, I am curious how and if I could see the rarefaction curves to make a decision about downsampling.

Thank you.

[jvpon]

I can run any export command for one file at a time.
So, this is clear to me now.
My mistake of being confused with your instructions.

[mizraelson]

Yes, also I would recommend using exportClones instead of exportClonesPretty unless you really need to manually inspect the clones.

As for the --chains parameter, I will check if there is a bug on our side when using two chains. Can you please try updating to the latest version and see if the issue still persists? Notice that in the latest version, your preset is called takara-mouse-rna-tcr-smarter. By default, this preset assembles clones by the full VDJRegion. If you want to replicate the behavior of takara-mouse-tcr-cdr3, please use the following command:

mixcr analyze takara-mouse-rna-tcr-smarter \
      input_R1.fastq.gz \
      input_R2.fastq.gz \
      --assemble-clonotypes-by CDR3 \
      result 

[jvpon]

Thank you. I will try but later on. Now I want to go through the whole process to see if I can get everything I need using mixcr (in the past I used vdjtools and was happy with its functionality).
And it is fine for me to work with TRA and TRB chains separately.

Please help me with the following.
I would like to filter out clones based on some criteria, such as in how many samples among each group of my samples (KO and WT) they are present, or basically by the sum of reads across all samples in the group. How can I do this while still keeping .clns files?
Because exportClonesOverlap doesn't allow, first, to specify this criterion for the overlap and, second, doesn't allow to save into .cnls files which I would like to use for post-analysis.
Also, if I slice and dice tables with clones obtained using exportClones, can I convert them back to .cnls files to use in post analysis? Or is it possible to use tables (.tsv files) directly calling mixcr postanalysis?

[mizraelson]

Unfortunately, this type of filtering is not currently possible in MiXCR. However, it will be available soon in the upcoming update.

[jvpon]

Dear, Thank you for your reply. But is it possible to convert tables back to.cnls files or use tsv files to produce.json file to use in mixcr postsnalysis? If not I would be interested to use vdjtools. Do you know how to make current mixcr files compatible with vdjtools? Or can you suggest me other postanalysis software compatible with current mixcr? Also, I asked about rarefaction curves. Would you be so kind to answer me? Thank you, Julia

…

On Mon, Aug 14, 2023 at 11:34 mizraelson ***@***.***> wrote: Unfortunately, this type of filtering is not currently possible in MiXCR. However, it will be available soon in the upcoming update. — Reply to this email directly, view it on GitHub <#1300 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFUQAKO7LYD75YEV5KXZEJTXVHWKTANCNFSM6AAAAAA3MX3EIA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[mizraelson]

Dear Julia,

1. It's not possible to convert tables back to .clns files, as this is a binary format that holds extensive information on every clone. It's also not possible to use table files as input for mixcr postanalysis.

2. To use the new MiXCR export tables with vdjtools, you should rename the following columns:

   • readCount to cloneCount
   • readFraction to cloneFraction

   I recommend renaming columns using a notebook or another text editor, but not Excel, as Excel may convert numbers (e.g., 0.0000983 -> 9.83E-04).

   Also, sometimes new MiXCR tables have cells with 'region not covered.' We used to leave these columns blank (if the region was not covered), but that confused some of our users, so we added this note. This might cause an issue for VDJtools if present. As a bypass, you can run the following command for the files:

   sed -i 's/region_not_covered//g' input.clones_TRB.tsv
   

   This will remove "region_not_covered" from the file, and it should work.

3. Currently, we don't have rarefaction curves in MiXCR, but they will be added soon with an upcoming update.

Let me know if this helps and if you have any other questions!

Sincerely,
Mark

[jvpon]

Thank you, Marc, for your prompt response. How does this command exactly work? mixcr postanalysis overlap \ --default-weight-function read \ --criteria "VDJRegion|AA" \ --metadata metadata.tsv \ ./downsample_min/clns_TRA ./overlap_by_VDJ_aa/TRA/result.json Does it consider the number of reads for similar clones in all samples, in both KO and WT groups together, and ignores all other clones that are not shared among all samples? But then the number of overlapping clones between ko and wt is very small in comparison to those that are not shared. How that dissimilarity can be explored in mixcr postanalysis? Thank you, Julia

…

On Mon, Aug 14, 2023 at 3:11 PM mizraelson ***@***.***> wrote: Dear Julia, 1. It's not possible to convert tables back to .clns files, as this is a binary format that holds extensive information on every clone. It's also not possible to use table files as input for mixcr postanalysis. 2. To use the new MiXCR export tables with vdjtools, you should rename the following columns: - readCount to cloneCount - readFraction to cloneFraction I recommend renaming columns using a notebook or another text editor, but not Excel, as Excel may convert numbers (e.g., 0.0000983 -> 9.83E-04). Also, sometimes new MiXCR tables have cells with 'region not covered.' We used to leave these columns blank (if the region was not covered), but that confused some of our users, so we added this note. This might cause an issue for VDJtools if present. As a bypass, you can run the following command for the files: sed -i 's/region_not_covered//g' input.clones_TRB.tsv This will remove "region_not_covered" from the file, and it should work. 3. Currently, we don't have rarefaction curves in MiXCR, but they will be added soon with an upcoming update. Let me know if this helps and if you have any other questions! Sincerely, Mark — Reply to this email directly, view it on GitHub <#1300 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFUQAKJI2ELWBIDRMD55HKLXVIPWXANCNFSM6AAAAAA3MX3EIA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[mizraelson]

Hi Julia,

The command above calculates pairwise Distance metrics. In other words, it will compare the repertoires of each pair of samples. According to --criteria "VDJRegion|AA", the clones that share the same VDJRegion amino-acid sequence will be considered equal. In your command, no information about the groups has been passed, so all possible pairs of samples will be cross-compared. If you want to compare two groups, you can add the following parameter --factor-by genotype, where genotype is the name of the column in your metadata.tsv table.

Also, I guess you use pre-downsampled data for the function. You can actually use the full .clns files and downsample the data within this function:

mixcr postanalysis overlap \
  --default-weight-function read \
  --downsampling count-reads-auto \
  --factor-by genotype \
  --criteria "VDJRegion|AA" \
  --metadata metadata.tsv \
  *.clns \
  result.json

This will generate a set of tables with different comparison metrics. You can see an example here and read about the available metrics here. Most of the metrics (except for Relative diversity) are weighted by the number of reads (in the case of --default-weight-function read), and since it operates with frequencies and not absolute read numbers, both overlapped and non-overlapped nucleotides are considered to properly evaluate how close the repertoires are. If you only have two groups (KO and WT), this will generate a single number for every metric. However, mixcr postanalysis overlap does not return clone sequences.

If I understand your question correctly, you are looking for a way to isolate clones that are present in one group (e.g. KO) and absent in the other (e.g. WT). This type of grouping is not available at the moment, but will be available (along with the filtering discussed above) with the next major update which we are going to release soon.

As a workaround at the moment, you can perform exportClonesOverlap separately on groups of WT and KO samples, isolate clones that are present in all samples within a group, and manually compare those lists (using Python, for example).

Let me know if you have any other questions!

Best Regards,
Mark