Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TLDR-462 gpu for 1.1 #365

Merged
merged 11 commits into from
Nov 10, 2023
Merged

TLDR-462 gpu for 1.1 #365

merged 11 commits into from
Nov 10, 2023

Conversation

raxtemur
Copy link
Collaborator

@raxtemur raxtemur commented Oct 26, 2023

Added gpu config and tested on XGBoost and torch models. Added in:

  • AbstractPickledLineTypeClassifier
  • TxtlayerClassifier
  • ScanParagraphClassifierExtractor
  • PdfImageReader

@raxtemur raxtemur marked this pull request as ready for review October 26, 2023 12:03
dedoc/config.py Outdated Show resolved Hide resolved
@NastyBoget
Copy link
Collaborator

NastyBoget commented Nov 7, 2023

Видимо еще нужно сделать try/except на ValueError, потому что если нет nvidia в принципе, то вылетает ошибка:

ValueError: invalid literal for int() with base 10: "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running."

@NastyBoget NastyBoget force-pushed the TLDR-462_GPU_for_1_1 branch 2 times, most recently from c45b758 to 0d58fdc Compare November 9, 2023 09:49
@NastyBoget NastyBoget force-pushed the TLDR-462_GPU_for_1_1 branch from 0d58fdc to 95dfbce Compare November 9, 2023 10:00
could make function for calling
subprocess.run("nvidia-smi", check=True)
and deal with excepting return value error, but as did
finally made function for calling
subprocess.run("nvidia-smi", check=True)
and dealt with excepting return value error

because previouse approach didn't work
dedoc/config.py Outdated Show resolved Hide resolved
requirements.txt Outdated Show resolved Hide resolved
@NastyBoget NastyBoget merged commit e6abe72 into develop Nov 10, 2023
2 checks passed
@NastyBoget NastyBoget deleted the TLDR-462_GPU_for_1_1 branch November 10, 2023 11:26
NastyBoget added a commit that referenced this pull request Nov 24, 2023
* Use older pydantic version (#364)

* Added rtf format to docx convertor (#366)

Co-authored-by: Alexander Golodkov <[email protected]>

* fix small bugs with docx reader such as non-integer sizes in docx sty… (#367)

* fix small bugs with docx reader such as non-integer sizes in docx style and filename with dots and spaces

* Rename test

---------

Co-authored-by: Nasty <[email protected]>

* TLDR-462 gpu for 1.1 (#365)

* TLDR-462 - test on GPU work

* TLDR-354 images attachments extraction from PDF (#368)

* Benchmarks before changes

* Add image extraction to tabby

* Fix document partial parsing

* Use start_page, end_page in java tabby execution

* Fix txtlayer classification tests

* Fixes in partial parsing

* Fix tests

* TLDR-518: Fix tabby partially read  (#372)

* Fix tabby partially read

* Add more tests

* Fix tabby page slice parameters

* Fix extract table in tabby with page range parameter

---------

Co-authored-by: Nasty <[email protected]>

* TLDR-514 creating document classes tutorial (#369)

* TLDR-517 attachments_dir (#370)

* TLDR-533 extract images from PDF to attachments_dir (#374)

* new version 1.1.1 (#375)

---------

Co-authored-by: Alexander Golodkov <[email protected]>
Co-authored-by: Alexander Golodkov <[email protected]>
Co-authored-by: IlyaKozlov <[email protected]>
Co-authored-by: raxtemur <[email protected]>
Co-authored-by: Andrey Mikhailov <[email protected]>
Co-authored-by: Nikita Shevtsov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants