Rarefying data

[niall142]

Hi Chen,

I was just wondering why it's recommended to rarefy data before calculating measures of diversity?

I used mStat_rarefy_data, and it seems to have dropped some species from my abundance data. Is that normal? I also noticed that some of the columns in my abundance table have had their values all changed to zero. Is that a mistake?

Thanks!
-Niall

[cafferychen777]

Dear Niall,

Thank you for your questions regarding the rarefaction process in MicrobiomeStat. I'm happy to provide some clarification.

1. Rarefaction before diversity calculations:
   We recommend rarefying data before calculating diversity measures to standardize sampling effort across different samples. This helps to mitigate biases that can arise from varying sequencing depths.

2. Dropped species:
   It's normal for some species to be dropped during rarefaction. This typically happens with very rare species or those with extremely low abundance. The process aims to create a more standardized dataset by subsampling to a common depth, which can result in the loss of some low-abundance taxa.

3. Last two columns changed to zero:
   This is unexpected behavior and may indeed be a mistake. Rarefaction shouldn't systematically zero out entire columns, especially if they contained non-zero values before. I'd like to investigate this further:

   • Could you please provide more details about these last two columns? What do they represent in your dataset?
   • If possible, could you share a small sample of your data before and after rarefaction, focusing on these columns?
   • Which version of MicrobiomeStat are you using?

Your observation is valuable, and if there's an issue with the function, we'd like to address it promptly.

Thank you for bringing this to our attention. Your feedback helps us improve MicrobiomeStat.

Best regards,
Chen Yang

[niall142]

Hi Chen, just checking in, did you get a chance to take a look at this issue?

Thanks!
-Niall

[niall142]

Hi Chen,

Thanks for your reply. Looking at the rarefied data again, it's actually 5 out of 36 columns that had their values changed to zeroes, but I can't
see anything special about those columns. Each column in my data represents the soil that was sampled under each tree in an orchard, across three years, with bacterial abundance reported in counts per gram.

Here I've attached the my feature.tab abundance file, my feature.ann file, and my metadata, before and after I rarefied them. Let me know if the attachments didn't work.

Thanks!
-Niall

SEM.bact.abun.copies.per.gram.feature.tab.csv
SEM.bact.abun.meta.dat.csv
SEM.bact.abun.feature.ann.csv

SEM.bact.abun.copies.per.gram.feature.tab.rarefied.csv
SEM.bact.abun.copies.per.gram.feature.ann.rarefied.csv
SEM.bact.abun.meta.dat.rarefied.csv

[niall142]

Oh, and I'm using MicrobiomeStat version 1.2.1.

[cafferychen777]

Dear Niall,

I apologize for the delayed response. The past couple of weeks have been quite challenging with midterm exams, and I regret that I forgot to address your issue earlier.

After carefully reviewing your data, I believe that rarefaction may not be necessary in your specific case. Let me explain why:

Rarefaction in microbiome studies is typically done to account for uneven sequencing depths across samples, which can affect diversity estimates. It's often applied when working with sequence count data, where some samples might have significantly more reads than others due to technical variations in sequencing.

However, your abundance data is reported in counts per gram of soil, which is already a standardized measure. These readings are likely quite large and don't suffer from the same kind of sampling depth bias that raw sequencing data does.

In your case, rarefying the data might actually be counterproductive, as it could lead to unnecessary loss of information, especially for less abundant species. The zeroing out of some columns that you observed is likely an artifact of the rarefaction process trying to standardize data that doesn't require standardization.

For your study, I would recommend using the original, non-rarefied data for your analyses. This will preserve all the information in your carefully measured abundance data.

I apologize again for the confusion this may have caused. If you have any further questions or if you'd like to discuss alternative approaches for your specific dataset, please don't hesitate to ask.

Best regards,
Chen

[niall142]

No worries, I'm sure you're super busy!

Ok, so I'll leave out the rarefaction for the counts per gram data. If I happen to use the percentage data instead at some point, would it be advisable to rarefy it?

Thanks very much!
-Niall

[cafferychen777]

Dear Niall,

Thank you for your understanding. I'm glad I could clarify the issue with the counts per gram data.

Regarding your question about percentage data: If your data is already in the form of percentages, rarefaction is actually not necessary or advisable. Here's why:

1. Percentage data is already a form of normalization. Each sample's microbial composition is represented as proportions that sum to 100%, which inherently accounts for differences in total abundance or sequencing depth between samples.

2. Rarefaction is primarily used to standardize sampling effort across different samples when dealing with count data. With percentage data, this standardization has effectively already been done through the conversion to proportions.

3. Applying rarefaction to percentage data wouldn't change the relative abundances within each sample. The results would be essentially the same as your original percentage data, making the process redundant.

4. Rarefaction can sometimes lead to loss of rare taxa information. With percentage data, you're already preserving all the relative abundance information, including that of less common species.

So, in short, if you're working with percentage data, you can proceed with your analyses without rarefaction. The data is already in a standardized form suitable for most ecological and statistical analyses of microbial communities.

If you have any more questions about handling your data or about specific analyses you're planning to run, please don't hesitate to ask.

Best regards,
Chen