Code

feat: Add DocumentCleaner 2.0#5976

Code

Merged

julian-risch merged 11 commits intomaindeepset-ai/haystack:mainfrom text-document-cleanerdeepset-ai/haystack:text-document-cleanerCopy head branch name to clipboardOct 13, 2023

+434-1

Commits

Commits on Oct 5, 2023

• remove whitespaces, substrings, regex, empty lines

  julian-risch

  committed
  e87b131

  Copy full SHA for e87b131

• remove repeated substrings

  julian-risch

  committed
  813ee6b

  Copy full SHA for 813ee6b

• reno

  julian-risch

  committed
  c5ff409

  Copy full SHA for c5ff409

Commits on Oct 9, 2023

• return empty string as shortest common ngram

  julian-risch

  committed
  c792b58

  Copy full SHA for c792b58

Commits on Oct 12, 2023

• address first half of review feedback

  julian-risch

  committed
  f202acf

  Copy full SHA for f202acf

• Merge branch 'main' into text-document-cleaner

  julian-risch

  authored
  f5969ca

  Copy full SHA for f5969ca

Commits on Oct 13, 2023

• address second half of review feedback

  julian-risch

  committed
  215da3d

  Copy full SHA for 215da3d

• mention \f page separator for header/footer removal

  julian-risch

  committed
  97bd916

  Copy full SHA for 97bd916

• mention \f page separator for header/footer removal

  julian-risch

  committed
  c3cdd86

  Copy full SHA for c3cdd86

• Merge branch 'main' into text-document-cleaner

  julian-risch

  authored
  b649261

  Copy full SHA for b649261

• mark example usage as python code

  julian-risch

  committed
  365b323

  Copy full SHA for 365b323