About the final result

[Jdemon233]

Hi,I have finished all pipeline, but i finally get the result a few .parquet files and don't find where my processed text data

[mauriceweber]

Hi @Jdemon233 ,

Can you provide some more context on the steps you have run and the params you used? The parquet files are produced by the deduplication scripts which take as input the zipped jsonl files which contain the text documents.

[Jdemon233]

Hi @Jdemon233 ,

Can you provide some more context on the steps you have run and the params you used? The parquet files are produced by the deduplication scripts which take as input the zipped jsonl files which contain the text documents.

thanks, i resolved my problem,